Link: The NIPS Experiment

The conference on Neural Information Processing Systems (NIPS) has conducted a fascinating experiment: split the program committee into two and get 10% of submissions reviewed by both. The article I’m linking to above has a great analysis of what they found (and it’s not encouraging).

This would be a great experiment to run at VIS. Anybody who has spent any time publishing papers has seen reviews that were unnecessarily harsh or too forgiving. Striking the right balance is difficult, and collecting some data on how well the process works is definitely important.

For some context on NIPS, I wrote about the conference a while ago.