Jerzy Neyman, Sigmund Freud, and Milton Friedman walk into a bar . . . (the mistaken association of null hypothesis testing with rigor)

This discussion thread reminded me of the pervasive way in which null hypothesis significance testing is (mistakenly) thought to have some special level of rigor not possessed by other methods in statistics and machine learning. Christian Robert and I discuss this in our article, “Not only defended but also applied”: The perceived absurdity of Bayesian inference.

I see an analogy to strict theories in other fields that have an air of austere rigor. I’m thinking of rational choice theory in political science, monetarism in economics, and Freudian psychoanalysis in the 1940s-1970s (there were all sorts of therapies, but 5-days-a-week Freudian analysis had an air of rigor back then, I think; there was the idea that “strict Freudian” therapy was the best), or what was called “theory” in literary studies a couple decades ago. There was the sense that these were the most rigorous, or coldly analytical, approaches: expensive regimens, difficult to follow, but the most effective. Recall our discussion of how economics in the early 2000s was like Freudian psychology in the 1950s.