“What Can we Learn from the Many Labs Replication Project?”

Aki points us to this discussion from Rolf Zwaan:

The first massive replication project in psychology has just reached completion (several others are to follow). . . . What can we learn from the ManyLabs project? The results here show the effect sizes for the replication efforts (in green and grey) as well as the original studies (in blue). The 99% confidence intervals are for the meta-analysis of the effect size (the green dots); the studies are ordered by effect size.

Let’s first consider what we canNOT learn from these data. Of the 13 replication attempts (when the first four are taken together), 11 succeeded and 2 did not (in fact, at some point ManyLabs suggests that a third one, Imagined Contact also doesn’t really replicate). We cannot learn from this that the vast majority of psychological findings will replicate . . .

But even if we had an accurate estimate of the percentage of findings that replicate, how useful would that be? Rather than trying to arrive at a more precise estimate, it might be more informative to follow up the ManyLabs projects with projects that focus on a specific research area or topic . . . So what DO we learn from the ManyLabs project? We learn that for some experiments, the replications actually yield much larger effects that the original studies, a highly intriguing findings that warrants further analysis.

We also learn that the two social priming studies in the sample, dangling at the bottom of the list in the figure, were resoundingly nonreplicated. . . . It is striking how far the effects sizes of the original studies (indicated by an x) are away from the rest of the experiments. . . .

Most importantly, we learn that several labs working together yield data that have an enormous evidentiary power. At the same time, it is clear that such large-scale replication projects will have diminishing returns . . . rather than using the ManyLabs approach retrospectively, we can also use it prospectively: to test novel hypotheses. . . .

P.S. It’s also worth reading this long and detailed discussion from Tal Yarkoni.

The post “What Can we Learn from the Many Labs Replication Project?” appeared first on Statistical Modeling, Causal Inference, and Social Science.