“Toward reproducible research: Some technical statistical challenges” and “The political content of unreplicable research” (my talks at Berkeley and Stanford this Wed and Thurs)

Wed 2 Oct 9:30am at Bin Yu’s research group, 1011 Evans Hall, University of California, Berkeley:

Toward reproducible research: Some technical statistical challenges

The replication crisis in social science is not just about statistics; it has also involved the promotion of naive pseudo-scientific ideas and exaggerated “one weird trick” claims in fields ranging from embodied cognition to evolutionary psychology to nudging in economics. In trying to move toward more replicable research, several statistical issues arise; here, we discuss challenges related to design and measurement, modeling of variation, and generalization from available data to new scenarios. Technical challenges include modeling of deep interactions and taxonomic classifications and the incorporation of sampling weights into regression modeling. We use multilevel Bayesian inference, but it should be possible to implement these ideas using other statistical frameworks.

Thurs 3 Oct 10:30am at the Stanford classical liberalism seminar, Stanford Graduate School of Business:

The political content of unreplicable research

Discussion of the replication crisis in the social science has focused on the statistical errors that have led researchers and consumers of research to overconfidence in dubious claims, along with the social structures that incentivize bad work to be promoted, publicized, and left uncorrected. But what about the content of this unreplicable work? Consider embodied cognition, evolutionary psychology, nudging in economics, claimed efficacy of policy interventions, and the manipulable-voter model in political science. These models of the world, if true, would have important implications for politics, supporting certain views held on the left, right, and technocratic center of the political spectrum. Conversely, the lack of empirical support for these models has implications for social science, if people are not so arbitrarily swayed as the models suggest.

The two talks should have very little overlap—which is funny, given that I’ll probably be the only person to attend both of them!

In preparation for both talks, I recommend reading the first three sections of our piranha paper.

The Stanford talk is nontechnical, talking about the social science and policy implications of the replication crisis, and I want to convey that the replication crisis isn’t just about silly Ted talks; it also has implications for how we should understand the world.

The Berkeley talk is for statisticians, talking about how the roots of and solutions to the replication crisis are not just procedural (pregregistration etc.) or data-analytical (p-values etc.) but also involve measurement, design, and modeling.