Open Evaluation: 11 sure steps – and 2 maybes – towards a new approach to peer review

"Open Evaluation will improve science. Researchers constantly evaluate each other — when we submit our results for publication, when we apply for grants, and when we apply for new jobs or promotions. Peer evaluation is our quality assurance strategy. And it needs to be better. Open access provides a context to radically reform scientific publishing. The way we evaluate scientific papers must be part of this. Our current system lacks sufficient quality and it lacks transparency. One creative approach to reconceptualizing evaluation comes from Nikolaus Kriegeskorte, Alexander Walther and Diana Deca. These three scholars invited eighteen papers by authors ready to look beyond open access, and they summarized the nascent suggestions in their recent paper, An emerging consensus for open evaluation: 18 visions for the future of scientific publishing, appearing in Frontiers in computational neuroscience. The importance of the Open Evaluation (OE) project is characterized in that article as follows. Evaluation is at the heart of the entire endeavor of science. As the number of scientific publications explodes, evaluation, and selection will only gain importance. A grand challenge of our time is to design the future system, by which we evaluate papers and decide which ones deserve broad attention and deep reading. The eighteen papers they solicited are cooked down to thirteen suggested features for OE. Eleven of the thirteen suggestions were “overwhelmingly endorsed” by the authors of the independent papers. Two of the suggestions were supported by some but doubted by others. Together, these give us a glimpse of a better system that is just around the corner. 13 measures for changing evaluation ..."