Studying average associations between income and survey responses on happiness: Be careful about deterministic and causal interpretations that are not supported by these data.

Jonathan Falk writes:

This is an interesting story of heterogeneity of response, and an interesting story of “adversarial collaboration,” and an interesting PNAS piece. I need to read it again later this weekend, though, to see if the stats make sense.

The article in question, by Matthew Killingsworth, Daniel Kahneman, and Barbara Mellers, is called “Income and emotional well-being: A conflict resolved,” and it begins:

Do larger incomes make people happier? Two authors of the present paper have published contradictory answers. Using dichotomous questions about the preceding day, Kahneman and Deaton reported a flattening pattern: happiness increased steadily with log(income) up to a threshold and then plateaued. Using experience sampling with a continuous scale, Killingsworth reported a linear-log pattern in which average happiness rose consistently with log(income). We engaged in an adversarial collaboration to search for a coherent interpretation of both studies. A reanalysis of Killingsworth’s experienced sampling data confirmed the flattening pattern only for the least happy people. Happiness increases steadily with log(income) among happier people, and even accelerates in the happiest group. Complementary nonlinearities contribute to the overall linear-log relationship. . . .

I agree with Falk that the collaboration and evaluation of past published work is great, and I’m happy with the discussion, which is focused so strongly on data and measurement and how they map to conclusions. I don’t know why they call it “adversarial collaboration,” as I don’t see anything adversarial here. That’s a good thing! I’m glad they’re cooperating. Maybe they could just call it “collaboration from multiple perspectives” or something like that.

On the substance, I think the article has two main problems, both of which are exhibited by its very first line:

Do larger incomes make people happier?

Two problems here:

1. Determinism. The question, “Do larger incomes make people happier?”, does not admit variation. Larger incomes are gonna make some people happier in some settings.

2. Causal attribution. If I’m understanding correctly, the data being analyzed are cross-sectional; to put it colloquially, they’re looking at correlation, not causation.

3. Framing in terms of a null hypothesis. Neither of the two articles that motivated this work suggested a zero pattern.

Putting these together, the question, “Do larger incomes make people happier?”, would be more accurately written as, “How much happier are people with high incomes, compared to people with moderate incomes?”

Picky, Picky

You might say that I’m just being picky here; when they ask, “Do larger incomes make people happier?”, everybody knows they’re really talking about averages (not about “people” in general), that they’re talking about association (not about anything “making people happier”), and that they’re doing measurement, not answering a yes-or-no question.

And, sure, I’m a statistician. Being picky is my business. Guilty as charged.

But . . . I think my points 1, 2, 3 are relevant to the underlying questions of interest, and dismissing them as being picky would be a mistake.

Here’s why I say this.

First, the determinism and the null-hypothesis framing leads to a claim about, “Can money buy happiness?” We already know that money can buy some happiness, some of the time. The question, “Are richer people happier, on average?”, that’s not the same, and I think it’s a mistake to confuse one with the other.

Second, the sloppiness about causality ends up avoiding some important issues. Start with the question, “Do larger incomes make people happier?” There are many ways to have larger incomes, and these can have different effects.

One way to see this is to flip the question around and ask, “Do smaller incomes make people unhappier?” The funny thing is, based Kahneman’s earlier work on loss aversion, he’d probably say an emphatic Yes to that question. But we can also see that there are different ways to have a smaller income. You might choose to retire—or be forced to do so. You might get fired. Or you might take time off from work to take care of young children. Or maybe you’re just getting pulled by the tides of the national economy. All sorts of possibilities.

A common thread here is that it’s not necessarily the income causing the mood change; it’s that the change in income is happening along with other major events that can affect your mood. Indeed, it’s hard to imagine a big change in income that’s not associated with other big changes in your life.

Again, nothing wrong with looking at average associations of income and survey responses about happiness and life satisfaction. These average associations are interesting in their own right; no need to try to give them causal interpretations that they cannot bear.

Again, I like a lot of the above-linked paper. Within the context of the question, “How much happier are people with high incomes, compared to people with moderate incomes?”, they’re doing a clean, careful analysis, kinda like what my colleagues and I tried to do when reconciling different evaluations of the Millennium Villages Project, or as I tried to do when tracking down an iffy claim in political science. Starting with a discrepancy, getting into the details and figuring out what was going on, then stepping back and considering the larger implications: that’s what it’s all about.