The turtles stop here. Why we meta-science: a meta-meta-science manifesto

All those postscripts in the previous post . . . this sort of explanation of why I’m writing about the scientific process, it comes up a lot.

I spend a lot of time thinking and writing about the research process, rather than just doing research.

And all too often I often find myself taking time out to explain why I’m spending time on meta-science that I could be spending on science instead.

My answer is that meta-science discussions are, like it or not, necessary conditions for more technical work. Without meta-science, we keep getting caught in deference traps. Remember when we and others criticized silly regression discontinuity analyses? The response in some quarters was a reflexive deference to standard methods in econometrics, without reflection on the applicability of these methods to the problems at hand. Remember when we and others criticized silly psychology experiments? The response in some quarters was a reflexive deference to standard practices in that field (“You have no choice but to accept,” etc.). Remember when we and others criticized ridiculously large estimated causal effects in policy analysis? The response in some quarters was to not respond at all, perhaps based on a savvy judgment that institutional reputations would outlast technical criticism. Remember pizzagate? The response was to duck and weave, which probably would’ve worked had there not been so so so many data anomalies, and also that particular researcher had no powerful friends to go on the counterattack against his critics. Remember that gremlins research? Again, the researcher didn’t give an inch; he relied on the deference given to the economics profession, and the economics profession didn’t seem to care that its reputation was being used in this way. Remember beauty and sex ratio, divorce predictions, etc.? Technical criticism means nothing at all to the Freakonomicses, Gladwells, NPRs, and Teds of the world. Remember how methods of modern survey adjusted were blasted by authority figures such as the president of the American Association for Public Opinion Research and the world’s most famous statistical analyst? Again, our technical arguments didn’t matter one bit to these people. Technical reasoning didn’t come into play at all. It was just one deference trap after another. So, yes, I spend hours dismantling these deference traps so we can get to our real work. Perhaps not the best use of my time, given all my technical training, but somebody’s gotta do it. I’m sick and tired of feeling like I have to explain myself, but the alternative, where people don’t understand why I’m doing this, seems worse. In the words of Auden, “To-morrow, perhaps the future.”

Indeed, not only do we have to spend valuable time on meta-science so we can get to our science, but sometimes we need to spend time on meta-meta-science as in the above paragraph and the postscripts to the previous post. We need the meta-meta-science to explain to people who would otherwise ask why we are wasting our time on meta-science.

THE TURTLES STOP HERE.