Opening up peer review | University Affairs

... “And part and parcel of scholarly publishing is peer review, a centuries-old custom that’s undergoing some revolutionary experiments in the Internet age. As social media become more entrenched in daily life, academic journals and scholars are beginning to test the boundaries of the traditional review model and explore the merits of online “open peer review” – so far with mixed results. In 2010, Shakespeare Quarterly, a prominent U.S. journal of literary studies published by Johns Hopkins University Press, became one of the first humanities journals to conduct an open peer-review trial. For eight weeks, the journal posted several articles on MediaCommons Press, a scholarly website. Katherine Rowe, guest editor of the special SQ edition, declares the experiment ‘an ecstatic success.’ About 40 participants, including a filmmaker and a media historian, posted more than 350 comments. The contributors were mostly academics but represented ‘a much wider sampling of scholarly expertise than we would have gotten had we gone through the traditional process,’ she says. Alan Galey, assistant professor at the University of Toronto’s faculty of information, was one of the authors who participated in the experiment, but not without some initial misgivings. Eventually he was won over by the journal’s prestige, and in the end he was pleased with the ‘range and depth’ of the feedback and the way the process unfolded publicly and in real time. ‘It was like a public conversation,’ he says, similar to one you’d have around a conference table. The editors, too, worried at the outset that a lack of anonymity might inhibit frankness among reviewers. That didn’t happen, but they confronted other challenges. For one thing, the process was more time-consuming for editors, authors and reviewers, says Dr. Rowe. For smaller journals, the increased workload could be a deal-breaker. The experiment also failed to attract the participation of junior scholars, as the editors had hoped... While some of the most recent experiments with open review have been in the humanities, the concept was embraced much earlier by the science community. In 2006, the prestigious journal Nature launched a trial of open peer review – with disappointing results. Few authors agreed to participate, and just half of those who did received critiques. The editors found most of the comments lacked substance and were of little use in making publishing decisions... The physics community adopted the practice – albeit unofficially – earlier still. Since the early 1990s, physicists have posted pre-published drafts of their research papers on arXiv... the idea behind arXiv was to make research findings available to a broad community as quickly as possible, rather than have researchers wait for months for the findings to appear in a journal... It may be that certain fields or subfields are better suited to open review than others. Gunther Eysenbach, editor of the Journal of Medical Internet Research, a Canada-based, online and open-access journal, says its readers have readily accepted open review. “Our audience is very tech-savvy and intrinsically interested in this kind of experiment.” Dr. Eysenbach says the journal adopted the system in 2009 to expand its pool of referees and make its articles accessible to lay readers. In some ways the experiment has fallen short of its goals: most of the self-selected reviewers are still academics, and sometimes the ones who aren’t don’t fully grasp the idea of what’s required. But overall the experiment has been a success, and the journal intends to continue the practice, mainly because authors seem to prefer it... Another benefit has been that the number of self-selected referees helps Dr. Eysenbach gauge reader interest in topics the magazine covers... Postmedieval, a quarterly published by Palgrave Macmillan, conducted an open peer review last year with similarly positive results. Jen Boyle, assistant professor of English at Coastal Carolina University and guest editor of the edition, was pleased that so many commentators took part and with the quality of their feedback. But she admits ‘the comments lacked the critical heft of private reviews.’ ‘It’s not that there weren’t any negative comments,’ she says. ‘They just weren’t quite as feisty as you’d see in a private peer review.’ Canadian scholars are also testing the waters. Researchers at the University of Victoria’s Electronic Textual Cultures Laboratory recently posted on Wikibooks (a Wikipedia sister site) an electronic version of the Devonshire Manuscript, a collection of 16th-century English poetry and verse. The manuscript contains an introduction, biographies and genealogical tables, all of which can be edited and commented on by readers. ‘We thought we might try to engage not only traditional academic sensibilities in review of materials but also the general public, especially what are increasingly becoming known as citizen scholars,’ explains Ray Siemens, head of the lab... The materials will remain on Wikibooks for a year and then the revised manuscript will go through a traditi