How did some of this goofy psychology research become so popular? I think it’s a form of transubstantiation.

OK, more on junk science. Sorry! But it’s been in the news lately, and people keep emailing me about it.

For those of you who want some more technical statistics content, here are some recent unpublished papers to keep you busy:

BISG: When inferring race or ethnicity, does it matter that people often live near their relatives?

Simulation-based calibration checking for Bayesian computation: The choice of test quantities shapes sensitivity

Nested R-hat: Assessing the convergence of Markov chain Monte Carlo when running many short chains

OK, are you satisfied??? Good. Now back to today’s topic: the mysterious popularity of goofy psychology research.

Here’s the deal. We’ve been hearing a lot about glamorous scientists who go on Ted and NPR, write airport bestsellers, get six-figure speaking gigs . . . and then it turns out, first that their work does not replicate, and next that their fame and fortune were based on scientific publications that were fatally flawed. Maybe the data were fabricated, maybe the experiments never happened, maybe the data were analyzed manipulatively or incompetently, often even if everything was on the up-and-up these studies were too noisy for anything useful to be learned.

At this point we usually ask, What happened? How did this bad work get done? Or, How did it not get caught, staying aloft, Wiley E. Coyote-like, for years, with no means of support? Or, What caused the eventual fall?

But today I want to ask a different question: How did this work get all the adoring publicity in the first place?

Sometimes the answer seems clear to me. Brian Wansink, for example, the guy who claimed that if you move items around on the menu or use a smaller plate or whatever, you could get people to eat 28% less for lunch. That’s a big deal. Big if true, as the saying goes. If the work was legit, it deserved all the publicity it got.

Similarly with Freakonomics, which has some strong messages regarding incentives and what you can learn from observing people’s behaviors. Some of the research they promoted was mistaken, but they really were going for important topics much of the time. And the Gladwellverse. No, I don’t believe that therapist’s claim that he can predict with 93% accuracy who’s gonna get divorced—but if it were the case, it would be worth hearing about. Again, big if true.

Or, if a researcher cures cancers in mice and then gives a Ted talk saying how he’s gonna end cancer in humans, ok, sure, that’s an exaggeration, but the relevance is clear.

Other times, though, I didn’t get it.

For example, the scandal everyone’s talking about now is a paper coauthored by two different NPR/Ted talk stars, and the key result of the paper was that if people sign an honesty pledge at the top of a form, they behave more honestly than if they sign at the bottom. Even if this was all real—even if there had not been all these data problems (for example here, and that ain’t the half of it), even if this had not been part of a whole line of research with replication problems—even without all that, why did anyone really care about this stuff in the first place?

I mean, sure, within the world of academic psychology, yes, studying small influences on behavior is important. Years ago when I read the classic collection on judgment uncertainty, edited by Kahneman, Slovic, and Tversky, I was entranced. It was super-cool and changed my understanding of human nature. But . . . research on where people should sign an honesty pledge? OK, I can see how business marketers would care about this, but how does it make you a Ted talk superstar? And one of these researchers made his name by writing books and articles saying that people are “predictably irrational” . . . this is news?

How did this happen? What got these people into the stratosphere of fame? Part of it is that they must be very engaging speakers. But, still, is it enough to just be a good performer? When I think of inspiring Ted talks, I think of biologists who are curing cancer by figuring out how proteins work. I think about people building machines that will remove pollution from the oceans. Developing new ways of teaching that will reach underpriviliged kids. Using technology to build veggie burgers. Training parrots to play the marimba. Amazing stuff. Compared to that, we have . . . running a psychology experiment using a fake paper shredder? Huh?

So I went to some of the Ted talk pages of these psychology researchers and I think I haver some sense of what was going on.

Here’s one: “Are we in control of our own decisions?”

And another: “How to change your behavior for the better”

And how about this: “The Power of Why: Unlocking a Curious Mind”

And: “Rebel Talent: Why it Pays to Break the Rules”

It’s a three-step process: First you do research demonstrating some very specific thing. Actually, the research doesn’t even need to demonstrate it, but it needs to look like it does. Second, you parlay this into a reputation for being a brilliant, innovative academic. Third, you imply that the specific thing you found (or, to be more precise, that you claim to have found) has important general implications. That’s how you get from the angels-dancing-on-the-head-of-a-pin result, “Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end,” to the big ideas such as “How to change your behavior for the better.”

I’m no saying that the researchers who do this are bad people, or that there’s anything bad about these three steps. First, it’s fine to do research on a specific topic; that’s often how we learn, building bricks that become part of the edifice of science. Second, if you do successful work, you get a good reputation; fair enough! If you have a problem with that, the problem is structural (to publish in tabloid journals it helps to make big claims); you can hardly blame researchers for doing their best. Third, if you honestly your work can help the world, then it makes sense to get some publicity for it. Nothing wrong with that!; indeed, it would be irresponsible not to try to publicize important work.

Somewhere along the way, though, there’s a leap, unsupported by data. OK, there could be many leaps, starting with the claims in the published research in step 1. But here I’m thinking about the leap that takes you from oh-so-prosaic “Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end” to the dramatic “Rebel Talent: Why it Pays to Break the Rules.”

Here’s the problem, as I see it: the scientific credentials accrued by the journal publications give the researcher a sort of license to make large claims unsupported by evidence.

An extreme case was disgraced primatologist Mark Hauser, whose research on monkeys had no practical applications at all, but it was enough to get him to be considered an expert on morality and hang out at Jeffrey Epstein’s Edge Foundation. You might think it’s a cheap shot for me to bring up the child-molester connection here, but it’s not! Hauser didn’t just con NPR and, presumably, himself; he also managed to convince people in the Epstein orbit that he was this savant. And a key component of all this was the academic status conveyed in part by the peer-reviewed publications.

But really we see this all the time. Consider power pose, that Ted-talk favorite. The subtitle of the published article was the careful, “Brief Nonverbal Displays Affect Neuroendocrine Levels and Risk Tolerance.” OK, it turns out the research was botched and didn’t actually demonstrate effects on neuroendocrine levels or risk tolerance. But the researchers didn’t know that at the time. What they, and the journal editors, have to take some responsibility for is the title, “Power pose”—a misleading title given that there was nothing about “power” in the research—and the final sentence of the abstract, “That a person can, by assuming two simple 1-min poses, embody power and instantly become more powerful has real-world, actionable implications,” even though the actual study had nothing about anyone “instantly becoming more powerful,” nor did it have any real-world, actionable implications. I’ve brought up this sentence many times, partly because it annoys me and partly because it indicates a big blind spot of the authors and the journal editors, that they’d end the abstract with such a strong claim that had zero support from the experiment.

For another example, there was this report from Freakonomics that “sports participation causes women to be less religious, more likely to have children, and, if they do have children, more likely to be single mothers.” It doesn’t sound so impressive if you find out what they really did: “comparing women in states with greater levels of 1971 male [high school] sports participation . . . to women in states with lower levels of 1971 male sports participation.”

OK, here’s another, from a New York Times op-ed: “Knowing a person’s political leanings should not affect your assessment of how good a doctor she is — or whether she is likely to be a good accountant or a talented architect. But in practice, does it? Recently we conducted an experiment to answer that question. Our study . . . found that knowing about people’s political beliefs did interfere with the ability to assess those people’s expertise in other, unrelated domains.” Oh, interesting, so they asked people to assess the quality of doctors, accountants, and architects, and saw how these assessments were colored by the political beliefs of these professionals? I followed the link to the research article and did a quick search. The words “doctor,” “accountant,” and “architect” appear . . . exactly zero times. Actually, “Participants were required to learn through trial and error to classify shapes as ‘blaps’ or ‘not blaps’, ostensibly based on the shape’s features. Unbeknownst to the participants, whether a shape was a blap was in fact random.” And later they had to choose “who the participant wanted to hear from about blaps and how they used the information they received.”

As always . . .

I have no problem with people doing studies to address very specific research claims. I also have no problem with speculation. My problem is when these get conflated: when a demonstration of claim X gets transmuted into a statement about Y (also problems arise because often X has not really been itself demonstrated as presented). And my problem is when the conflation is done within the scientist rather than the research study: Researcher A has demonstrated very specific claim X, which now turns A into a sort of science pundit who has authority to make strong claims about Y.

I don’t know much about Jesus, but this all sounds like some sort of doctrine of transubstantiation.

Summary

The original question was, how did obscure academic research such as “Brief Nonverbal Displays Affect Neuroendocrine Levels and Risk Tolerance” and “Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end” get such publicity and reach a level of cultural salience to merit all this media attention?

The answer is that there has been a sort of transubstantiation, whereby the success (such as it was) of scientific studies had the effect of elevating the researchers involved into a higher state, granting them authority so that their evidence-free speculations were given the status of scientific findings.