“Science does not advance by guessing”

I agree with Deborah Mayo who agrees with Carlo Rovelli that “Science does not advance by guessing. It advances by new data or by a deep investigation of the content and the apparent contradictions of previous empirically successful theories.”

And, speaking as a statistician and statistical educator, I think there’s a big problem with the usual treatment of statistics and scientific discovery in statistics articles and textbooks, in that the usual pattern is for the theory and experimental design to be airlifted in from who-knows-where and then the statistical methods are just used to prove (beyond some reasonable doubt) that the theory is correct, via a p-value or a posterior probability or whatever. As Seth pointed out many times, this skips the most key question of where the theory came from, and in addition it skips the almost-as-key question of how the study is designed.

I do have a bit of a theory of where theories come from, and that is from anomalies: in a statistical sense, predictions from an existing model that do not make sense or that contradict the data. We discuss this in chapter 6 of BDA, and Cosma Shalizi and I frame it from a philosophical perspective in our philosophy paper. Or, for a completely non-technical, “humanistic,” take, see my paper with Thomas Basbøll on the idea that good stories are anomalous and immutable.

The idea is that we have a tentative model of the world, and we push that model, and gather data, and find problems with the model, and it is the anomalies that motivate us to go further.

The new theories themselves, though, where do they come from? That’s another question. It seems to me that new theories often come via analogies from other fields (or from other subfields, within physics, for example). At this point I think I should supply some examples but I don’t quite have the energy.

My real point is that sometimes it does seem like science advances by guessing, no? At least, retrospectively, it seems like Bohr, Dirac, etc., just kept guessing different formulations and equations and pushing them forward and getting results. Or, to put it another way, these guys did do “deep investigation of the content and the apparent contradictions of previous empirically successful theories.” But then they guessed too. But their guesses were highly structured, highly constrained. The guesses of Dirac etc. were mathematically sophisticated, not the sort of thing that some outsider could’ve come up with.

How does this relate to, say, political science or economics? I’m not sure. I do think that outsiders can make useful contributions to these fields but there does need to be some sense of the theoretical and empirical constraints.

The post “Science does not advance by guessing” appeared first on Statistical Modeling, Causal Inference, and Social Science.