Our proposal for scheduled post-publication review: “Even if each review took twice the effort of the average pre-publication review, our system would add only 1 percent to the total reviewing effort, while providing important perspectives on papers representing more than one-quarter of the citations received by these influential journals.”

The current system of scholarly journal review is absolutely nuts. The vast majority of review effort goes to papers that nobody reads. We can do better via scheduled post-publication review, for example every time a paper reaches its 250th citation, the journal commissions an outside review–not with the goal of retracting the paper, but to provide a new perspective. If the paper has been cited 250 times, it’s worth getting that new take–and it’s rare enough that a published article reached that level of citation, so the total cost of these new reviews would be low, only a small fraction of the cost of existing pre-publication review.

Andy King and I present the idea:

Problems with the credibility of empirical research have been discussed for decades.

The most common prescriptions for improving credibility — better review, public critique, and replication — have merit, but they fail to direct scarce resources where they can do the greatest good: those few publications with the greatest impact.

We propose an alternative, using review resources more efficiently and effectively by borrowing the idea of “replay review” from professional sports. The current peer-review system would continue to judge research articles in real time when they are submitted, but the publications that go on to have an outsized impact would be evaluated again, and in more detail, to confirm or refine the initial assessment.

Here’s the key insight:

All proposals for strengthening primary review face an economic challenge: Most of the resources spent on strengthening it are wasted because, for most submissions, the existing review process is already strong enough. At top journals, 90 percent or more of the submissions are rejected for apparent flaws, and thus, a strengthened review process will not change the outcome. Of the 10 percent that are accepted and published, most are lightly read and cited and thus have little influence.

And this is what we recommend:

Once a publication receives a specified number of citations, it would receive an independent review. These reviews would then be published in full, along with author responses, so that readers have additional guidance on how to interpret the initial publication. . . .

We crunched some numbers:

To assess the practicality of our proposal, we evaluated the submission and citation history of articles published in the 2014 cohort of the peer-reviewed empirical journals selected by the Financial Times for determining the research rank of business schools.

Skewness in citation rates means that a large proportion of the citation impact can be checked at a relatively low cost. Replay review of just those articles receiving more than 250 citations would mean that publications accounting for 28 percent of all the citations would be checked through further review. Even if each review took twice the effort of the average pre-publication review, our system would add only 1 percent to the total reviewing effort, while providing important perspectives on papers representing more than one-quarter of the citations received by these influential journals.

This is an elaboration of my efficiency argument for post-publication review.

P.S. The Chronicle of Higher Education gave our article the title, “Social Science Is Broken. Here’s How to Fix It.” We didn’t choose that title! Our recommendation was, “Social Science Needs Replay Review.” But those of you involved in the news media know that the authors of an article are rarely are in charge of the title. We like our suggestion of post-publication review, but we’re under no illusion that it would “fix” social science. It’s just one small part of the picture.

P.P.S. We also thank David Wescott at the Chronicle for editing our article.