| Times Higher Education -- What have we learned from the Reproducibility Project?

"When 270 researchers spend several years replicating 100 psychology experiments, one expects momentous insights. That only 36 per cent of results could be replicated, and that social psychological research was less reproducible than research in cognitive psychology is, on the face of it, shocking ('Majority of psychology papers are not reproducible, study discovers', News, 3 September). But are the findings of the Reproducibility Project: Psychology really that unexpected, and do they mean that we can no longer believe psychology textbooks? Although these results have made headlines, they should not have been a surprise to research psychologists. In 1962, Jacob Cohen reported in the Journal of Abnormal and Social Psychology that the average statistical power of research in these fields was only 0.48. Several subsequent reviews indicated that the power of this type of research has not increased. The power of a study, which determines whether we can identify valid and reject invalid hypotheses, can be compared to the magnification of a microscope: with too little magnification, we may not see things that are there or believe we see things that are not there. When conducting scientific studies, researchers look for 'significant' results – in technical terms, for a 'p-value' of less than 0.05. If one tried to replicate a study that had a p-value of less than 0.05 and power of 0.50, one would have only a 50 per cent chance of success. So why do psychologists not aim for a maximal power of 1? Power is determined by the study’s sample size, the significance level and the strength of the effect (relationship) studied. In some cases that effect might be large statistically, as when we study the impact of study hours on grades; in other cases the effect may be weaker, as between class size and grades. Since in novel research one often does not know the effect strength, researchers often underestimate the large sample size they need to achieve acceptable levels of power ..."