Why I like preregistration (and it’s not about p-hacking). When done right, it unifies the substance of science with the scientific method.

This came up in comments to Jessica’s recent post.

I like preregistration. It’s not something I used to do, and I still don’t always do it. I’ve worked on hundreds of research projects, and only a few of them had had any preregistration at all.

That said, I think preregistration has value, and I’m doing it more and more.

The reason I like preregistration has nothing at all to do with hypothesis tests or p-values or p-hacking or questionable research practices or anything like that.

I like preregistration for two reasons.

1. For me, preregistration implies constructing a hypothetical world–not a “null hypothesis” of no effect, but a possible world corresponding to what I’m actually aiming to study–and then simulating fake data and proposing and trying out analysis methods on those simulated data. I find this sort of commitment–the effort of laying out a complete generative model for the process–to be helpful. Thinking about effect sizes and their variation, all sorts of things, also seeing if the proposed analysis can recover parameters of interest from the simulated data, which is what’s often called power analysis although I prefer the more general term “design analysis.”

2. When other people preregister, that can be useful because then we can see discrepancies between the original plan and what actually got reported. Two examples are here and here–in both those cases, discrepancies between the preregistration and the final paper gave us doubts about the published claims. When these changes happen, it is not a moral failure on anyone’s part–we can learn from data!–it’s just relevant for understanding the theories being promulgated in these papers.

I agree that preregistration is not necessary for good science. I still think it can be a useful tool, both my own workflow in developing scientific hypotheses and gathering data to understand them, and in communication of workflow to others.

Preregistration has a valuable indirect function of making it more difficult to do bad science. It does not directly turn bad science into good science. That doesn’t make preregistration a bad idea–recently I’ve been preregistering studies and, more generally, simulating data before gathering any data–; we should just be aware that this sort of procedural step can only one small part of the story. Ultimately, science is about the substance of science, not just about the scientific method.

There’s something interesting here, though, that links the two perspectives. If you do things right, your preregistration will involve the substance of what you’re studying and will not merely be a procedural step, a form of paperwork that exists to validate the p-values that your study will produce. Rather, doing this preregistration will require simulating fake data, which in turn will require hypothesizing a full model of the underlying process.

I recognize that what I just described is not the usual thing that is meant by “preregistration,” which is more along the lines of: “We will perform this comparison and use a 2-sided test,” etc. But it could be! I think this is a useful connection.