Why I no longer use the term “cargo-cult science”

In our new paper, “Interrogating the ‘cargo cult science’ metaphor,'” Megan Higgs and I write:

Over the past fifty years, the term “cargo cult” has been used to describe the actions of scientists who appear to follow forms of scientific inquiry but without the understanding and self-criticism that are essential to real scientific progress. The term has served a useful role by providing a short and catchy label to something that is otherwise difficult to explain. However, the term is also fraught with historical and cultural baggage and, in our opinion, encourages crossing of a subtle line between criticizing discipline methodological norms and criticizing the individuals currently carrying out those norms as part of a complex and context dependent social process. We find that carefully interrogating the term itself holds some important lessons for improvement in the science reform movement.

We conclude:

Better practice in quantitative research is not just about improving our use of statistical tests (advice akin to saying “change the shape of the headphones” or “build bigger runways” in the cargo cult narrative) or even abandoning tests entirely (“fake headphones and runways serve no purpose at all”); it also involves studying real effects, controlling and adjusting for variation, and carefully integrating statistical models with scientific models. It involves a deeper understanding of available methods and when they are well aligned with the bigger goals of the science. In today’s scientific culture, long-established principles of measurement and statistical design are often forgotten, or inadvertently ignored, because statistical significance and publication can be bought with the cheap coin of uncontrolled statistical analysis.

Improving statistical practice can be expected to reduce the rate of absurd claims that appear to be supported by data, but the real gains should arrive indirectly by incentivizing researchers to put in the effort to improve design, measurement, data collection, and the mapping of scientific theories to realistic models of data. Those pushing science reform efforts could benefit from respecting these challenges, rather than labeling other scientists, or their practices, as “cargo cult” and offering easy diagnoses that, while well-meaning, miss the mark.

The full paper is here. Also relevant is this paper on forking paths and workflow in statistical practice and communication.