I fear that many people are drawing the wrong lessons from the Wansink saga, focusing on procedural issues such as “p-hacking” rather than scientifically more important concerns about empty theory and hopelessly noisy data. If your theory is weak and your data are noisy, all the preregistration in the world won’t save you.

This came up in the discussion of yesterday’s post.

We’ve discussed theory and measurement in this space before. And here’s a discussion of how the problems of selection bias are magnified when measurements are noisy.

Forking paths and p-hacking do play a role in this story: forking paths (multiple potential analyses on a given experiment) allow researchers to find apparent “statistical significance” in the presence of junk theory and junk data, and p-hacking (selection on multiple analyses on the same dataset) allows ambitious researchers to do this more effectively. P-hacking is turbocharged forking paths.

So, in the absence of forking paths and p-hacking, there’d be much more of an incentive to use stronger theories and better data. But I think the fundamental problem in work such as Wansink’s is that his noise is so much larger than his signal that essentially nothing can be learned from his data. And that’s a problem with lots and lots of studies in the human sciences.

The forking paths and p-hacking are relevant only in the indirect way that it explained how the food behavior researchers (like the beauty-and sex-ratio researchers, the ovulation-and-clothing researchers, the embodied-cognition researchers, the fat-arms-and-voting researchers, etc etc etc) managed to get apparent statistical success out of hopelessly noisy data.

So I hope the lesson that researchers will draw from Pizzagate is not “I should not p-hack,” but, rather, “I should think more seriously about theory and I should work hard to take better measurements and use within-person designs to study within-person effects, and, when forking paths are no longer available to me, I’ll need to get better measurements anyway, so I might as well start now.

The post I fear that many people are drawing the wrong lessons from the Wansink saga, focusing on procedural issues such as “p-hacking” rather than scientifically more important concerns about empty theory and hopelessly noisy data. If your theory is weak and your data are noisy, all the preregistration in the world won’t save you. appeared first on Statistical Modeling, Causal Inference, and Social Science.