Setting aside the politics, the debate over the new health-care study reveals that we’re moving to a new high standard of statistical journalism

Pointing to this news article by Megan McArdle discussing a recent study of Medicaid recipients, Jonathan Falk writes:

Forget the interpretation for a moment, and the political spin, but haven’t we reached an interesting point when a journalist says things like:

When you do an RCT with more than 12,000 people in it, and your defense of your hypothesis is that maybe the study just didn’t have enough power, what you’re actually saying is “the beneficial effects are probably pretty small”.

and

A good Bayesian—and aren’t most of us are supposed to be good Bayesians these days?—should be updating in light of this new information. Given this result, what is the likelihood that Obamacare will have a positive impact on the average health of Americans? Every one of us, for or against, should be revising that probability downwards. I’m not saying that you have to revise it to zero; I certainly haven’t. But however high it was yesterday, it should be somewhat lower today.

This is indeed an excellent news article. Also this sensible understanding of statistical significance and effect sizes:

But that doesn’t mean Medicaid has no effect on health. It means that Medicaid had no statistically significant effect on three major health markers during a two-year study. Those are related, but not the same. And in fact, all three markers moved in the right direction. They just weren’t big enough to rule out the possibility that this was just random noise in the underlying data. I’d say this suggests that it’s more likely than not that there is some effect–but also, more likely than not that this effect is small.

The only flaw is this bit:

There was, on the other hand, a substantial decrease in reported depression. But this result is kind of weird, because it’s not coupled with a statistically significant increase in the use of anti-depressants. So it’s not clear exactly what effect Medicaid is having. I’m not throwing this out: depression’s a big problem, and this seems to be a big effect. I’m just not sure what to make of it. Does the mere fact of knowing you have Medicaid make you less depressed?

McArdle is forgetting that the difference between “significant” and “not significant” is not itself statistically significant. I have no idea whether the result is actually puzzling. I just think that she was leaping too quickly from “A is significant and B is not” to “A and B contradict.”

Also I’d prefer she’d talk with some public health experts rather than relying on sources such as, “as Josh Barro pointed out on Twitter.” I have nothing against Josh Barro, I just think it’s good if a journalist can go out and talk with people rather than just grabbing things off the twitter feed.

But these are minor points. Overall the article is excellent.

With regard to the larger questions, I agree with McArdle that ultimately the goals are health and economic security, not health insurance or even health care. She proposes replacing Medicaid with “free mental health clinics, or cash.” The challenge is that we seem to have worked ourselves into an expensive, paperwork-soaked health-care system, and it’s not clear to me that free mental health clinics or even cash would do the trick.

Other perspectives

I did some searching and found this post by Aaron Carroll. I agree with what Carroll wrote, except for the part where he says that he would not say that “p=0.07 is close to significant.” I have no problem with saying p=0.07 is close to significant. I think p-values are often more of a hindrance than a help, but if you’re going to use p=0.05 as a summary of evidence and call it “significant,” then, indded, 0.001 is “very significant,” 0.07 is “close to significant,” and so forth. McArdle was confused on some of these issues too, most notably by mixing statistical significance with a Bayesian attitude. I wouldn’t be so hard on either of these writers, though, as the field of statistics is itself in flux on these points. Every time I write a new article on the topic, my own thinking changes a bit.

I see some specific disagreements between McArdle and Carroll:

1. McArdle writes:

Katherine Baicker, a lead researcher on the Oregon study, noted back in 2011, “people who signed up are pretty sick”.

Carroll writes:

Most people who get health insurance are healthy. They’re not going to get “healthier”.

This seems like a factual (or, at least, a definitional) disagreement.

2. McArdle:

We heard that 150,000 uninsured people had died between 2000 and 2006. Or maybe more. With the implication that if we just passed this new law, we’d save a similar number of lives in the future. Which is one reason why the reaction to this study from Obamacare’s supporters has frankly been a bit disappointing.

Carroll:

This was Medicaid for something like 10,000 people in Oregon. The ACA was supposed to be a Medicaid expansion for 16,000,000 across the country. If 8 people’s lives in the study were saved in some way by the coverage, the total statistic holds.

Indeed, (16,000,000/10,000)*8 = 128,000. I’m guessing that McArdle’s would reply that there’s no evidence that 8 people’s lives were saved in the Oregon study. Thus, numbers such as 150,000 lives saved are possible, but other things are possible too.

The bottom line

What does this all mean in policy terms? McArdle describes Obamacare as “a $1 trillion program to treat mild depression.” I’m not sure where the trillion dollars comes from. A famous graph shows U.S. health care spending at $7000 per person per year, that’s a total of 2.1 trillion dollars a year. I’m assuming that the Obama plan would not increase this to 3.1 trillion! Maybe it is projected to increase annual spending to 2.3 trillion, which would correspond to an additional trillion over a five-year period? In any case, that sounds pretty expensive. Given that other countries with better outcomes spend half as much as we do, I’d hope a new health-care plan would reduce costs, not increase them. But that’s politics: the people who are currently getting these 2.1 trillion dollars don’t want to give up any of their share! The other half of McArdle’s quote (“mild depression”) sounds to me like a bit of rhetoric. If a policy will reduce mild depression, I assume it would have some eventual effect on severe depression too, no?

Beyond this, I can’t comment. I’m like many (I suspect, most) Americans who already have health insurance in that I don’t actually know what’s in that famous health-care bill. I mean, sure, I know there’s something about every American getting coverage, but I don’t know anything beyond that. So I’m in no position to say anything more on the topic. I’ll just link to Tyler Cowen, who, I assume, actually knows what’s in the law and has further comments on the issue.

Let me conclude where I began, with an appreciation of the high quality of statistical journalism today. In her news article, McArdle shows the sort of nuanced understanding of statistics and evidence that I don’t think was out there, twenty years ago. And she’s not the only one. Journalists as varied as Felix Salmon, Nate Silver, and Sharon Begley are all doing the good work, writing about newsworthy topics in a way that acknowledges uncertainty.