Peer Review, Part 5: The Importance of Gatekeepers

The purpose of peer review is to separate the wheat from the chaff, the good from the bad, the brilliant from the clinically insane – you get the picture. But why? Why filter and not just let anybody publish whatever they want?

Why Gatekeepers? And Why Gates?

In the old days, there was the resource argument. A journal that’s published six times a year with so many pages per issue can only print some limited number of papers. The same is true for a conference: there are only so many sessions in which to present, and printing the proceedings also puts limits on the number of papers that can be included.

In the new era of electronic proceedings, online libraries, etc., none of these is true anymore. There are no limits to paper length because it doesn’t matter if your paper is 10 pages long or 100. It also doesn’t matter if a journal issue contains 10 papers or 100. So why insist on limiting the number of papers?

The answer is simple: time. It’s the only resource we can’t make more of, and it is the one that limits what we can possibly consume. Being able to find work that has been vetted, rather than having to vet it all yourself, is hugely valuable. You can now trust that the work has at least a minimum quality level, and is not just a video of somebody’s sleeping cat.

If you want to see what kinds of stuff you end up getting when there is no gatekeeping, just take a stroll around YouTube. Most of the content there is simply awful, pointless dreck. New stuff is also added at a rate that ensures that you could never possibly watch it all, even if you wanted to. And while there is no formalized system for it, most people (like me) don’t waste their time watching random videos, because they are mostly bad. Instead, we wait for the masses of users to find the ones that are worthwhile, and post those on Twitter and other social media. But the end result is the same: somebody has to wade through the flood of crap to find the few gold nuggets.

That is also what can make reviewing so frustrating. Your job as a reviewer is to weed out the bad 75–80% of papers, so the good 20–25% will be accepted. That means that for every one good paper, you will see three to four bad ones. But the result is that the papers in the journal or conference be of a much higher quality.

Alternatives

Today, anybody can publish whatever they want, at no cost. It’s called the web. It’s accessible and easy. So why bother with the gatekeepers and their walled gardens? The answer is, again, quality and time. If you just randomly search for things and don’t pick authoritative sources, you’re likely to end up with things that are not very good.

That is not to say that there isn’t great work out there that isn’t going through peer review. Bret Victor has never published an academic paper, and his work is amazing. But he is an exception. Work does not have to be vetted to be good. But if you’re looking for good work, you end up going to the places that rigorously select and filter.

The science world is looking for some sort of middle ground, too. There are places like arXiv.org, which provides a reasonably structured (though entirely non-reviewed) place to make papers available and discuss them. PLOS ONE, which the Quilt Plots paper was published in, has less stringent reviewing criteria and tends to err on the side of accepting more, rather than less. And there are certainly more.

The issue is not that there aren’t alternatives, it’s that they mostly only work in addition to the established journals, not really as actual alternatives. A paper that is only published on arXiv will not get a lot of attention (unless somebody trustworthy spots it and can make a strong case for it).

Does the Peer Review System Work?

Yes, it does. You can always complain about this paper or that paper getting in or not getting in. But overall, it certainly works. The visualization conferences are competitive enough to produce new and interesting work every year, yet not so insanely competitive that acceptance comes down to luck. There is a lot of variety, and people who have been publishing for a long time don’t automatically get their work accepted.

Reviewers don’t always spot problems, as Stephen Few helpfully points out. But there are few published papers with egregious errors in them. Sure, there should be more rigor in reviewing, and there should be ways of retracting papers when things go wrong. We haven’t had a big plagiarism or other ethics scandal in visualization, but I doubt that we have the mechanisms in place to get such papers out of the respective digital libraries if and when they happen.

If anything, reviewers in visualization (and computer science in general) are some of the harshest and finicky reviewers around. Maria Zemankova once said, “We are computer scientists, we’re trained to find bugs!” I also remember Robert Moorehead speaking at the opening or closing sessions of Vis/VisWeek a number of years ago and asking people not to be so “cut-throat” in their reviews. It sometimes seems like a miracle that anything gets published at all.

Wrapping Up

What I hope this little series has shown is that peer review is a complex process that has many things going for it. While the paper that got me to write over 4,000 words on the topic was clearly not one of the brighter spots in the history or peer review, it’s a good system overall. There are many ways a paper can be bad, but there are also good reasons why those get submitted. I’ve glanced over many issues and details in describing the process, and there are easily ten times as many words to be written about all the issues in academic publishing, especially outside of the mostly well-functioning technical sciences.

Ultimately, the quality of the papers that get accepted are the measure of whether the process works or not. And while I’m the first one to jump on a paper I don’t like, the visualization field as a whole is producing good work and moving forward at a steady pace. But it only does that thanks to a functioning peer review system.

This is the last part of a five-part series on peer review in visualization. One posting a day was posted throughout this week. One or two further parts may follow in the coming weeks.

Teaser image by Paolo Del Signore, used under Creative Commons.