(again) Yeah, yeah, I understand why you’re all talking about accusations of fraud. But for the rest of us, it’s about the non-replication and the bad science, not about possible fraud and blame

There’s been a lot of news coverage lately on the replication crisis in psychology and related fields. Simine Vazire and I wrote something a couple years ago, Why did it take so many decades for the behavioral sciences to develop a sense of crisis around methodology and replication?, exploring more formally the timeline I’d discussed in a much-discussed post from 2016. Why this has been motivating so much discussion just now in the news media is another story, I guess related to their being a big lawsuit in the air. Lawsuit = news, I guess? This article by Gideon Lewis-Kraus seems like a good summary.

Just one thing, though. The subtitle of that article is, “Dan Ariely and Francesca Gino became famous for their research into why we bend the truth. Now they’ve both been accused of fabricating data.” We see two things that should help the reader engage: (a) a focus on individual personalities and life stories, and (b) the accusation of fabricating data. Both these things are worth discussing—people’s choices matter, and fraud is always a concern (as would be mistaken accusations of fraud). Indeed, we had a long discussion just a couple months ago on cheating in science, sports, journalism, business, and art, riffing on a book by financier Dan Davies.

So, sure, but . . . I continue to think that the big problem of non-replicability is not fraud so much as the misguided expectations: the idea that science is supposed to be some endless stream of discoveries and a refusal to admit error. Here are a few relevant posts: – Honesty and transparency are not enough – Psychology needs to get tired of winning – Clarke’s Law: Any sufficiently crappy research is indistinguishable from fraud – Here’s why I don’t trust the Nudgelords . . . – The real problem of that nudge meta-analysis is not that it includes 12 papers by noted fraudsters; it’s the GIGO of it all

One refreshing difference of the current headlines compared to what came in the past is that, for whatever reasons, some of the researchers involved in the latest scandals appear open to admitting that much of their past work is just wrong, that its nonreplication is not just some technical problem but rather is a reflection that in the past they were basically doing the experimental equivalent of generating random numbers and using them to tell stories. I still have unpleasant memories of political scientists insisting, in the face of all evidence, that subliminal smiley faces have large effects on attitudes toward immigration; of a sociologist avoiding looking at careful explanations of how his much-publicized claims were nothing more than noise mining; of the himmicanes and air rage people never giving up; the Freakonomists not coming to terms with their promotion of climate change denial; the nudgelords memory-holing their former adoration of a now-discredited food behavior researcher; etc etc etc.

I guess what I’m saying is an important step forward in the current discussion of replication problems, both in science and in the news media, is to recognize that so much of this research is just no good, at best ridiculously overestimating effect sizes and setting up a false sense of certainty in a world that is highly variable. All this is separate from any questions of fraud and blame.