Peer Review, Part 2: How It Works

Peer review is one of the central pillars of academic publishing. But how does it actually work? What is blind review, and what is it good for? This part will answer those questions, and then tell you how to be a good reviewer yourself.

The Process

The basic process is this: you have some work written up that you want to publish at a conference or in a journal. In some fields, you might submit an abstract to a conference, but in visualization, they all want full papers. So you submit your paper, usually through some sort of review management system that lets you upload a PDF, potentially supporting files (video, datasets, etc.), and fill in some data like the kind of work you’re presenting (research vs. application vs. study, etc.).

The paper goes through some minimal checks and is then sent out to reviewers. In the case of a journal, an associate editor picks those reviewers and asks them for the review. In the case of a conference, this is done by the papers chairs. For the VIS conferences and EuroVis, there is an extra layer here, where the papers chairs pick two reviewers for each paper (members of the program committee), who then each pick one additional reviewer. This helps offload some of the work, since the papers chairs have to organize reviews for hundreds of papers in a short amount of time.

The result is that a paper is usually reviewed by three to five reviewers. Some low-quality conferences make do with fewer, but this number is typical. In some cases, there might also be more, for example when outside reviewers are asked because a paper makes claims the usual reviewers don’t feel comfortable judging (e.g., in a particular application domain, field of science, etc.)

Once the reviews come back, the associate editor or primary reviewer make a recommendation that the editor or papers chairs can follow or not. They typically do, though in the case of a conference there can be some back and forth when trying to balance the number of papers accepted with the demands of the program.

There are basically two outcomes: accept the paper or reject it (journals can also require revisions). The way the decision is made is democratic in a way, because the reviewers weigh in, but is ultimately up to the editor or papers chair (who are senior people in the field).

Blind and Double-Blind Review

Almost all reviews are blind, which means that as the paper author, you don’t know who is reviewing your papers. There is also double-blind review, which means that the reviewers don’t know who wrote the paper, either. To do this, you remove all identifying information from the submitted paper. While that’s obvious for things like the list of authors, it becomes a bit tricky when referencing your own prior work. Some people simply ignore this, and it’s generally not required in visualization (i.e., you are free to no do it if you don’t care).

The goal of “blinding” the reviewers is that it helps judge the work on its own merits, rather than having the reviewers be biased by who wrote the paper, or the institution they are from. That bias can be both positive and negative, and neither is good. Since you don’t know who’s reviewing your work, you can’t retaliate against bad reviews, either. This is not a big issue in visualization, where most people are fair and reasonable, but it certainly helps keep things a bit more objective.

Of course, if you know your way around the field and know the people, you can often tell who was likely involved in a particular paper you’re reviewing. You might even find some hints who some of the reviewers are by looking at their style or if they suggest you read lots of papers by a particular author. But in general, there aren’t enough clues, and that’s a good thing.

The Peers

The reason for peer reviewing is that there is no central authority to decide which work is good and which isn’t. You can’t leave it to some arcane committee of the elders of the field or similar, that would be bad (plus they want to publish too!). Instead, every member of a particular field is expected to not just publish papers, but also assist in the review process.

Not having a single person (or a small group) make all the decisions is clearly a good thing, and it works better in some areas than in others. In visualization, the main conferences change papers chairs every year (they’re on two-year terms, so there’s always a new one and one who has done it before), and program committee members get rotated off after three years or so. Journal associate editors and editors also have limited terms. As a result, the field is able to change and evolve, with new people moving up the ranks and taking over the reigns.

Being A Reviewer

Anybody can sign up to be a reviewer, though don’t expect to be given work to review if nobody knows you. You typically have to have something published before people will have a sense of what you do and will ask you to do reviews. Once you’ve done a few, you will get more requests than you probably want, though.

It’s important to be realistic about reviewing workload and to be able to say no. Decide on a number of reviews you’re willing to take on per year or per semester, and then simply say no when you get more. The key is to say no right away, rather than sit on a request forever. By saying no, you give the associate editor or primary reviewer a clear signal that he or she has to look for another reviewer. It’s not a big deal, people say no all the time. But if you drag the decision out because you don’t want to do the review but also don’t want to say no, you’re just wasting everybody’s time. So make a quick decision and click the appropriate link in the email.

Reviewing can be interesting and it can be annoying. But either way, it’s an important part of the scientific process. Without peer review, there is no way to judge the quality of work, and to decide which papers are worthy of publication, and which ones are not.

This is part of a five-part series on peer review in visualization. One posting a day will be posted throughout this week.

Teaser image by Dirk Schaefer, used under Creative Commons.