From black box to white box at open access journals: predictive validity of manuscript reviewing and editorial decisions at Atmospheric Chemistry and Physics | Research Evaluation | Oxford Academic

Abstract: More than 4,500 open access (OA) journals have now become established in science. But doubts exist about the quality of the manuscript selection process for publication in these journals. In this study we investigate the quality of the selection process of an OA journal, taking as an example the journal Atmospheric Chemistry and Physics (ACP). ACP is working with a new system of public peer review. We examine the predictive validity of the ACP peer review system, namely, whether the process selects the best of the manuscripts submitted. We have data for 1,111 manuscripts that went through the complete ACP selection process in the years 2001 to 2006. The predictive validity was investigated on the basis of citation counts for the later published manuscripts. The results of the citation analysis confirm the predictive validity of the reviewers' ratings and the editorial decisions at ACP: Both covary with citation counts for the published manuscripts.