Rebuilding After the Replication Crisis—Asterisk

"You’d also notice a big rise in scientific papers discussing a “crisis,” as well as all sorts of special issues and debate pieces dedicated to the idea of replicability. Like never before, many scientists are looking inward and questioning the reliability of their work. There are also telling patterns in the tools they’re using. The Open Science Framework, a website where scientists can post their plans, share their data and generally make their whole research process more transparent, had somewhere near zero users in 2012, but by the end of 2021 had hit 400,000. The number of new files posted by those users, and the number of preregistrations, have also risen exponentially. In the past, a major barrier to being open and transparent with research was that it was really difficult to do so (how would you share your data, pre-internet?). It’s still far from super easy, but the technology has substantially improved, and a great many scientists are signing up to use it.

You’d also notice that scientific publishers are changing. One of my formative experiences as a PhD student, in 2011, was submitting a replication study to the Journal of Personality and Social Psychology, only to be told that the journal did not publish replications under any circumstances (you might be thinking, “WTF?” — and we were too). Now at that very same journal and a host of others, replications are encouraged, as is a set of other “open” practices — sharing data, code and materials, and pre-registering hypotheses and analyses before a study is carried out. Some journals now publish an article’s peer reviews alongside its online version, so the whole process is on view — hopefully encouraging reviewers to put in more effort, and allowing us to see where things went wrong in cases where reviewers missed important flaws....

These are all encouraging developments, and represent impressive progress in and of themselves. A scientist from 2012 would find a lot to be optimistic about in 2022 — at least on the surface. But the number of people talking about the crisis, debating open science or signing up to a website isn’t what we really want to know. And journals offering ways to make science better isn’t much use if nobody takes them up. Have these changes actually made the science better?..."