Journalists are suckers for anything that looks like science. And selection bias makes it even worse. But I was unfair to NPR.

Journalists are suckers. Marks. Vics. Boobs. Rubes.

You get the picture.

Where are the classically street-trained reporters, the descendants of Ring Lardner and Joe Liebling, the hard-bitten journos who would laugh in the face of a press release?

Today, nowhere in evidence.

I’m speaking, of course, about the reaction in the press to the latest bit of “p less than .05” clickbait to appear in PPNAS. Here’s what I wrote yesterday regarding the article, “Physical and situational inequality on airplanes predicts air rage”:

NPR will love this paper. It directly targets their demographic of people who are rich enough to fly a lot but not rich enough to fly first class, and who think that inequality is the cause of the world’s ills.

This morning I was curious so I googled the name of the article’s first author and NPR. No hits on this study. But a lot from other news organizations:

Let’s go through and take a look:

Deborah Netburn in the L.A. Times presents the story completely uncritically. Zero concerns. From the authors’ lips to God’s ears.

Carina Storrs at CNN: 12 paragraphs of unskeptical parroting of the authors’ claims, followed by three paragraphs of very mild criticism (quoting psychologist Michael McCullough as saying that the study “is provocative, but it does not strike me as an open and shut case”), followed by two more paragraphs by the study’s author.

Gillian Mohney at ABC News: no skepticism at all, she buys into the whole study, hook, line, and sinker.

Bob Weber, CTV News: Again takes it at face value. A regression with p less than .05 in PPNAS is good enough for Bob Weber.

Unsigned, ABC Radio: A short five-paragraph story, the last paragraph of which is, “Although this study points to a link between air rage and first class cabins, it does not prove causation.”

Vanessa Lu, Toronto Star: Straight P.R., no chaser.

Peter Dockrill, Science Alert: A nearly entirely credulous story, marred only by a single paragraph buried in the middle of the story, quoting Michael McCullough from that CNN article.

And Sophie Ryan at the New Zealand Herald buys into the whole story. Again, if it’s published in PPNAS and it tells us something we want to hear, run with it.

LA Times, ABC News, Toronto Star, sure, fine, what can you expect? But the New Zealand Herald? I’m disappointed. You can do better. If NPR dodged this bullet, you can too.

Where were the savvy reporters?

Where were Felix Salmon, Ed Yong, Sharon Begley, Julie Rehmeyer, Susan Perry, etc., in all this? The quantitative and science reporters who know what they’re doing? They didn’t waste their time with this paper. They see the equivalent in Psychological Science each week, and they just tune it out.

You don’t see the most respected pop music critics reviewing the latest Nickelback show, right? OK, maybe at the Toronto Star. But nowhere else.

Where were Nate Silver’s 538 and the New York Times’s Upshot team? They didn’t waste their time with this. They like to analyze their own data. They know that data analysis is hard, and they don’t trust any numbers they haven’t crunched themselves.

We have a classic case of selection bias. The knowledgeable reporters don’t waste their time on this, leaving the suckers to write it up.

Comparison to himmicanes

Here’s a data point, though. This air rage study, like the power pose study, got nearly uniformly positive coverage, whereas the ovulation-and-clothing study and the himmicanes study were accompanied in their news reports with a bit of skepticism (not as much as was deserved, but some). Why?

I suspect a key factor is that the conclusions of this new paper told people what they want to hear: flying sucks, first-class passengers are assholes, social inequality is a bad thing, and it’s been proved by science!

Also, the ovulation-and-clothing and himmicanes studies had particularly obvious errors in their conceptualization and measurement, whereas the statistical flaws in the air rage study are more subtle and have to do with scaling of ratios and the interpretation of multiple regression coefficients.

A template for future news stories

OK, fine, you might say. But what’s a reporter to do? They can’t always call Andrew Gelman at Columbia University for a quote, and they typically won’t have the technical background to evaluate these papers by themselves.

But I do have a suggestion, a template for how reporters can handle PPNAS studies in the future, a template that respects the possibility that these papers can have value.

I’ll share that template in my next post.

P.S. BoingBoing fell for it too. Too bad. You can do better, BoingBoing!

P.P.S. Felix Salmon pointed out that the study was also promoted completely uncritically in Science magazine. Tabloids gonna tabloid.

The post Journalists are suckers for anything that looks like science. And selection bias makes it even worse. But I was unfair to NPR. appeared first on Statistical Modeling, Causal Inference, and Social Science.