The Psychology of Program Committees

One thing that I frequently tell my grad students is that your chances of getting a paper accepted to a conference depend as much on who the reviewers are, and what kind of mood they are in, than the content of the paper itself. OK, leaving aside those obviously brilliant papers that my group is known to churn out, the paper content does matter -- but on most program committees there is a large pile of "pretty good" papers that all have a roughly equal chance of getting accepted. In these cases it often comes down to the collective mindset of the reviewers during the PC meeting.

There are many subtle psychological effects that influence the disposition towards a given paper. The first has to do with the timing of the paper discussion. At the beginning of a PC meeting, everyone is amped up on caffeine and uncalibrated with the respect to the overall quality of the papers being discussed. Your chances of getting a paper accepted when it is discussed early on in the meeting can vary widely. Most PC meetings discuss the top-ranked papers first, but after a string of (say) five or so papers accepted, people start thinking that it's time to reject a paper, so the sixth paper tends to be a scapegoat to release the pressure and make everyone feel better that the conference is still "selective." Fortunately, most PC chairs recognize this effect and switch gears to discussing some of the lower-ranked papers next, so the committee sees both ends of the spectrum. I've been in PC meetings where the top ranked paper with four strong accepts and a weak accept is tabled for further discussion, just because the committee is thirsty for blood.

Late in the afternoon, everyone is exhausted, cranky, and the power structure of the PC meeting has started to play itself out -- who's willing to throw themselves on the tracks to accept or reject a paper; who's willing to defer to more senior members of the committee; etc. Here we start to see some interesting personality traits emerge that can save or sink a paper:

"I didn't read this paper, but I know the work." This drives me nuts -- someone who was not even a reviewer casting doubt on (or supporting) the paper being discussed because they know the people involved or had some discussion with them at a conference a few months ago. This should be flatly disallowed by the program chairs, but I've seen it happen. A variant on this is...

"I didn't read this paper, but I saw their original failed submission last year." Again, this should be disallowed by PC chairs -- whether a paper was any good last year should have no bearing on the discussion of the present submission.

"I'm not an expert in this area, but I don't think there's anything novel here." Too many times a reviewer who is simply not qualified to review a paper is unwilling to defer to more expert members of the committee. Someone who doesn't know the related work that well might infuse the discussion with a vague sense of unease that taints the rest of the reviewers and makes it harder for someone to champion the paper for acceptance.

"I know way too much about this area and they should have used a value of 1.3 instead of 1.4 for the alpha parameter on page 7." Often, when a paper is too close to a reviewer's area, they tend to nickle-and-dime the paper for small problems that chip away at its credibility. Sometimes this is a poorly disguised attempt at tamping down the competition. These kinds of reviewers often miss the forest for the trees, where a paper has some good ideas but needs some rough edges sanded off, as all papers do.

"I'm a new faculty member and want to prove how smart I am by rejecting most of the papers in my pile." When you are new to program committees there is a real temptation to exercise your new power by rejecting papers left and right, which clearly establishes your intellectual dominance over the poor authors who are at your mercy. Most new faculty fall into this trap, and I've certainly been in this situation before.

"I'm a senior, well-respected faculty member and like to compare all of the papers in my pile to things published in the 1960s." The "there's nothing new here" argument sometimes comes up when you have a senior, somewhat jaded PC member who thinks that all of the good ideas were published back in the good old days when men were men and the women programmed in octal. It's inevitable that good ideas will come up time and time again, and I actually think there is value in reevaluating previously-published ideas in the context of new technology capabilities and application demands. Perspective is a great thing but sometimes you can have too much perspective.

The final point is that it is easy to argue to reject a paper; much harder to argue to accept a paper over other reviewers' objections. If a reviewer is not really sure about the novelty or importance of a paper's contributions, they often defer to the most negative reviewer, since nobody likes looking like an idiot in a PC meeting. Standing up and championing a paper takes a lot of guts, and by doing so you are taking responsibility for any faults in the paper that might arise later (if it turns out, say, that the exact same idea was published before, or there is a flaw in the experimental design). I think it's important that every member of a program committee commit themselves to championing one or two papers during the meeting, even if they aren't so sure about them -- otherwise the process can get to be too negative. One way to view your role as a PC member is to identify that beautiful piece of work that would otherwise have been overlooked due to superficial problems that turn off the reviewers.

So, next time you get a paper rejected, just remember that it's probably because the reviewers were in a bad mood because they hadn't served the afternoon coffee yet.