Tuesday, March 30, 2010

The Psychology of Program Committees

One thing that I frequently tell my grad students is that your chances of getting a paper accepted to a conference depend as much on who the reviewers are, and what kind of mood they are in, than the content of the paper itself. OK, leaving aside those obviously brilliant papers that my group is known to churn out, the paper content does matter -- but on most program committees there is a large pile of "pretty good" papers that all have a roughly equal chance of getting accepted. In these cases it often comes down to the collective mindset of the reviewers during the PC meeting.

There are many subtle psychological effects that influence the disposition towards a given paper. The first has to do with the timing of the paper discussion. At the beginning of a PC meeting, everyone is amped up on caffeine and uncalibrated with the respect to the overall quality of the papers being discussed. Your chances of getting a paper accepted when it is discussed early on in the meeting can vary widely. Most PC meetings discuss the top-ranked papers first, but after a string of (say) five or so papers accepted, people start thinking that it's time to reject a paper, so the sixth paper tends to be a scapegoat to release the pressure and make everyone feel better that the conference is still "selective." Fortunately, most PC chairs recognize this effect and switch gears to discussing some of the lower-ranked papers next, so the committee sees both ends of the spectrum. I've been in PC meetings where the top ranked paper with four strong accepts and a weak accept is tabled for further discussion, just because the committee is thirsty for blood.

Late in the afternoon, everyone is exhausted, cranky, and the power structure of the PC meeting has started to play itself out -- who's willing to throw themselves on the tracks to accept or reject a paper; who's willing to defer to more senior members of the committee; etc. Here we start to see some interesting personality traits emerge that can save or sink a paper:

"I didn't read this paper, but I know the work." This drives me nuts -- someone who was not even a reviewer casting doubt on (or supporting) the paper being discussed because they know the people involved or had some discussion with them at a conference a few months ago. This should be flatly disallowed by the program chairs, but I've seen it happen. A variant on this is...

"I didn't read this paper, but I saw their original failed submission last year." Again, this should be disallowed by PC chairs -- whether a paper was any good last year should have no bearing on the discussion of the present submission.

"I'm not an expert in this area, but I don't think there's anything novel here." Too many times a reviewer who is simply not qualified to review a paper is unwilling to defer to more expert members of the committee. Someone who doesn't know the related work that well might infuse the discussion with a vague sense of unease that taints the rest of the reviewers and makes it harder for someone to champion the paper for acceptance.

"I know way too much about this area and they should have used a value of 1.3 instead of 1.4 for the alpha parameter on page 7." Often, when a paper is too close to a reviewer's area, they tend to nickle-and-dime the paper for small problems that chip away at its credibility. Sometimes this is a poorly disguised attempt at tamping down the competition. These kinds of reviewers often miss the forest for the trees, where a paper has some good ideas but needs some rough edges sanded off, as all papers do.

"I'm a new faculty member and want to prove how smart I am by rejecting most of the papers in my pile." When you are new to program committees there is a real temptation to exercise your new power by rejecting papers left and right, which clearly establishes your intellectual dominance over the poor authors who are at your mercy. Most new faculty fall into this trap, and I've certainly been in this situation before.

"I'm a senior, well-respected faculty member and like to compare all of the papers in my pile to things published in the 1960s." The "there's nothing new here" argument sometimes comes up when you have a senior, somewhat jaded PC member who thinks that all of the good ideas were published back in the good old days when men were men and the women programmed in octal. It's inevitable that good ideas will come up time and time again, and I actually think there is value in reevaluating previously-published ideas in the context of new technology capabilities and application demands. Perspective is a great thing but sometimes you can have too much perspective.

The final point is that it is easy to argue to reject a paper; much harder to argue to accept a paper over other reviewers' objections. If a reviewer is not really sure about the novelty or importance of a paper's contributions, they often defer to the most negative reviewer, since nobody likes looking like an idiot in a PC meeting. Standing up and championing a paper takes a lot of guts, and by doing so you are taking responsibility for any faults in the paper that might arise later (if it turns out, say, that the exact same idea was published before, or there is a flaw in the experimental design). I think it's important that every member of a program committee commit themselves to championing one or two papers during the meeting, even if they aren't so sure about them -- otherwise the process can get to be too negative. One way to view your role as a PC member is to identify that beautiful piece of work that would otherwise have been overlooked due to superficial problems that turn off the reviewers.

So, next time you get a paper rejected, just remember that it's probably because the reviewers were in a bad mood because they hadn't served the afternoon coffee yet.

23 comments:

  1. Matt, you forgot to include "he is not in my clan" fallacy. Some people take their research field as a religion, and also display radical fundamentalism towards papers from other fields. "If the approach the paper proposes does not stem in my research field, it is wrong."

    This is the worst kind, in my view.

    ReplyDelete
  2. Hi Matt! What a timely discussion! I'm sitting in one of those meetings at this very moment. I'll never forget my first PC meeting when I learned about the reality of the situation. At that point, I decided I'm glad I get my sense of self worth from other events in life than conference paper outcomes. Two things I'll add:

    (1) There is a bimodal reaction (MM is going to kill me for misusing theoretical terms) when your paper is sent to a reviewer who does very similar work. They'll either overinflate its importance - "This paper tackles the most important problem known to mankind!" or they try to squash the competition - "No need to accept this paper because the smartest person in the world (me) is already working on the problem."

    (2) People's egos get tangled up in the discussion. Often, once they get going in fighting for/against a paper, it becomes less about the paper and more about winning the battle. This adds to the randomness of the outcomes.

    I recently got invited to be on a PC where each PC member was to be given one "trump card" at the start of the meeting. This would allow the PC member to choose one paper for which they could unilaterally decide the outcome. (I declined the invitation - I'm on too many PCs. But I must admit I was very tempted to accept the invitation just to see how it shook out.)

    The next time I'm in Cambridge, we can share PC meeting stories. I have lots of them.

    ReplyDelete
  3. There are some of us who respect what the PC is trying to do and only send our papers to appropriate venues. There are people who have 2-3 papers a year in the top venue, and still an 80% accept rate. Why are you so concerned with the probabilistic fate of marginal papers? (There are those who, in fact, make a career out of the probabilistic fate of marginal papers.)

    ReplyDelete
  4. Anon - If you've ever been on a PC you'll know that most papers are marginal - there are very few papers that are clear accepts. The issue is that even very, very good papers can sometimes get rejected due to non-technical reasons, and it's important for students (especially) to understand the dynamics behind the scenes.

    ReplyDelete
  5. Kim -- First off, you know I like you far too much to abuse you for misusing theoretical terms! Second, you used it entirely appropriately. :)

    I heard the "trump card" approach, or something like it, was used recently for a theory conference. I admit I'm a bit skeptical of it, but I suppose it depends on how it's used. There's the argument that if someone on the PC really likes the paper, then there's something interesting there so that the paper should be accepted. The counterargument is that (at least in my experience) some PC members get overly enthusiastic about a paper they don't know enough about. I'd feel better if you had to justify you were an expert (or at least above some fairly high threshold) in the subarea if you were going to use your trump card. Also, it probably works better for larger conference (60++ papers, instead of the 30 or so of SIGCOMM...) It does help avoid the "everything's getting rejected!" phenomenon!

    When you and Matt go out for that beer to swap PC stories, please remember to invite me along....

    ReplyDelete
  6. Matt --

    I'd like to echo the point I think anonymous was making. Obviously the process is noisy, but the noise tends to have a much less significant effect on papers at the top and bottom. Where it has the most effect is on papers at the borderline.

    I've been on multiple PCs where there's say 4-10 slots left and maybe a dozen papers still left to discuss. As you say, "Late in the afternoon, everyone is exhausted, cranky, and the power structure of the PC meeting has started to play itself out..." At this point, people are thinking about getting to the airport. My take is that, often, at this point, there's no "right answer" -- if one had an arbitrary amount of time, one could still imagine different outcomes that would be reasonable, and there's nothing like an arbitrary amount of time left. So do the types of things you're talking about start happening and influence the final decisions? Absolutely. And, where possible, they should be avoided. But when there's 5 reviewers with 2 at best lukewarm about a paper, 2 lukecold, and 1 who doesn't feel informed enough to make a decision, I'm actually glad that some of these random personality quirks come out and allow us to move on.

    ReplyDelete
  7. Weird bug here. Says 6 comments above (after this one will say 7), but there are only 4 (now 5). Michael Mitzenmacher posted 2 comments, which I got email notifications for, but they don't show up here. Perhaps the Mitzenmacher Particle and the anti-Mitzenmacher Particle (i.e., the Welsh Particle) collided in the blogosphere and annihilated each other.

    ReplyDelete
  8. Michael MitzenmacherMarch 30, 2010 at 5:27 PM

    Matt --

    I'd like to echo the point I think anonymous was making. Obviously the process is noisy, but the noise tends to have a much less significant effect on papers at the top and bottom. Where it has the most effect is on papers at the borderline.

    I've been on multiple PCs where there's say 4-10 slots left and maybe a dozen papers still left to discuss. As you say, "Late in the afternoon, everyone is exhausted, cranky, and the power structure of the PC meeting has started to play itself out..." At this point, people are thinking about getting to the airport. My take is that, often, at this point, there's no "right answer" -- if one had an arbitrary amount of time, one could still imagine different outcomes that would be reasonable, and there's nothing like an arbitrary amount of time left. So do the types of things you're talking about start happening and influence the final decisions? Absolutely. And, where possible, they should be avoided. But when there's 5 reviewers with 2 at best lukewarm about a paper, 2 lukecold, and 1 who doesn't feel informed enough to make a decision, I'm actually glad that some of these random personality quirks come out and allow us to move on.

    ReplyDelete
  9. Michael MitzenmacherMarch 30, 2010 at 5:33 PM

    Kim --

    Come now. First, you know I like you too much to abuse you for misusing a theoretical term. Second, you used it entirely properly. :)

    I heard a theory conference recently used the "trump card" approach. I admit to some skepticism, although I suppose it depends on how it's used. On the plus side, if I understand the argument, if someone is that positive on a paper, there must be something about it that merits acceptance. On the down side, it seems ripe for abuse (did the "in crowd" just get even more in?). Also, I've certainly been on PCs where people got overly excited about a paper that was not in their area. Perhaps the trump card should be restricted so the reviewer has to be an expert (or above some threshold) in the subarea? Finally, it probably works better for conferences with 60-70++ papers over smaller conferences like SIGCOMM.

    When you and Matt get together to tell PC stories, be sure to let me know....

    ReplyDelete
  10. So, can another PC member use her trump card to annihilate your trump card if they disagree with you? And, if so, can I have the ace?

    -- Maria Ebling

    ReplyDelete
  11. The general psychology you suggest seems spot on, though a good PC chair will switch to the considering the bottom before the first reject. It should happen when the accepts start to require significant debate and the switch should happen whenever things lag.

    A big mistake is for a PC to spend too much time on the obvious rejects and accepts so that too little time is left for borderline papers. The average time per paper spent should be planned to increase monotonically over the course of the meeting as the decisions get more and more difficult. The chair has to be really cognizant of the rate of progress at all times.

    The direction of the reaction to papers that are close to a PC member's work can be very area dependent and part of the socialization in the area. In some areas the members form a mutual admiration society and in others there is a tendency to tear down each others' work (or work that is not by their immediate social circle). This effect is one of the most difficult for PCs to deal with.

    The situation for electronic-only PCs is different but far from better.

    ReplyDelete
  12. Matt, the first 2 points ("I didn't read this paper, but I know the work" and "I didn't read this paper, but I saw their original failed submission last year") can be EASILY canceled with double-blind review process. All the others issues would still be there, but at least these 2 would quickly disappear. My question is: why double-blind review is still not used in most of the most important journals and conferences? Maybe because the "caste" likes the idea of keeping everything under "politically correct" control, so to avoid embarrassing situations in which e.g. you are eventually shooting down the paper of the PC member sitting just in front of you? We already discussed this in one of your previous posts...
    I think that one of the most important things a good supervisor should always do with his grad and post-grad students is to push them towards the best achievable quality of research, even when all these unpleasant behaviours are evident to everyone who has ever attended a PC meeting and result in unfair rejections of good papers.

    ReplyDelete
  13. Anon - actually, double-blind doesn't address problems #1 or #2 at all. If someone is familiar with a piece of work outside of the context of the paper submission, double-blind can't generally hide the authors' identity. And seeing the same paper on several subsequent program committees doesn't have anything to do with whether the authors have been identified.

    Actually, at least in systems, most of the top conferences do use double-blind reviewing. Journals typically do not, and I think it would be very difficult to write a detailed journal paper that covers all of the related work while masking your own identity.

    I absolutely agree that the best way to get your papers accepted is simply to do good research. You can't really game the system.

    ReplyDelete
  14. > You can't really game the system.

    I'm not sure I agree with the last point. It depends what you call "gaming" the system. For example, you can:

    - go around giving talks about your unpublished work. This gives an "unfair" advantage to established researchers in a field who have more, higher-visibility opportunities to give talks. [I put "unfair" in quotes because you can make a case that people with established research records have earned this advantage -- we invite them to give talks precisely because we want to hear what they're currently thinking about.]

    - tell people on the PC whom you know personally about your work. It's remarkable how when I'm on the PC of an important conference suddenly everyone wants to tell me about what they're doing. Of course, I enjoy hearing about what they're doing but I am aware that it biases my reviewing.

    - discuss your work on a blog, post it on the arxiv, etc. Again, this is a way of "getting around" the vexing anonymity of submissions, when you're well-enough known and liked for anonymity to work "against" you.

    - make sure you refer generously to the work of PC members and likely reviewers. [Note: of course, you should refer generously to all previous work, but I've certainly seen people trying to suck up to specific PC members.]

    The first three points only give you a hand up if you're doing reasonably interesting work. So is it gaming the system? Sure it is. There are more "reasonably interesting" papers than there are slots. For borderline papers, the amount of buzz around a paper or line of work is a factor, conscious or not, in PC decisions. And buzz can, to some extent, be manufactured.

    ReplyDelete
  15. If someone is familiar with a piece of work outside of the context of the paper submission, double-blind can't generally hide the authors' identity.

    Oh no, not this canard again. Do you really can guess the author papers that often? Say with more than 50% correct probability? I can't.

    ReplyDelete
  16. Anon - I said, "If someone is familiar with a piece of work **outside the context of the paper submission**" -- meaning, the TPC member already knows the project. It is very difficult to disguise your identity when a reviewer has out-of-band information.

    ReplyDelete
  17. Adam, good points - I intend to write a future blog post on marketing one's research, but it's clearly an important part of building one's reputation and "brand" in the community. Program committees are human beings and they can't be expected (nor would it be desirable) to leave their knowledge of the research community at the door of the PC meeting.

    ReplyDelete
  18. I agree with Mike that the noise for papers in the gray area doesn't really matter. Any subset given however much room is left in the program is likely to be about as good as any other. I would love as PC chair to follow this agenda:
    9:00 AM - Process easy rejects
    9:45 AM - Process easy accepts
    10:30 AM - Decide on how many slots are left in program
    11:00 AM - Randomly choose a subset of papers in the middle
    11:30 AM - Go to lunch
    1:00 PM to 2:00 AM - Play poker and drink

    ReplyDelete
  19. Ketan -- Hi there! Man, first Kim, now you -- seems everyone is reading this blog! (Feel free to stop by mine once in a while...)

    I like your PC meeting schedule. When you're PC chair, please invite me to be on the committee. And I"m not just saying that because I still have some of your money sitting in my wallet from Berkeley days.

    I don't think I've been to a PC meeting where a poker game was on the schedule. The other socializing is often quite nice, but a poker game would be an excellent addition to the mix.

    ReplyDelete
  20. Matt,

    You are of course absolutely right about the timing playing a big role in PC discussions. I was just co PC chair of VEE and we decided to do a preliminary "calibration round" to try to counteract this effect. We (the PC chairs) more or less randomly picked three papers from each tier (one in top-ranked, one in mid-ranked, and one in bottom-ranked).

    Naturally, the top one got in, the bottom one didn't, and I don't remember the fate of the mid-ranked one. But the effect was noticeable: everyone now had a sense of perspective of the whole range of papers, rather than just their pile, and the PC meeting went insanely smoothly. Now, that might be because we managed to put together an insanely great PC, and it was a sunny day in Seattle so the view from the Allen building was superb :), but I think it helped.

    And unlike Kim, I accepted the invitation to be on the PC with the "trump card" (aka "whiteball") - it's OOPSLA 2010. Martin Rinard is the PC chair and he is hoping it will help ensure that edgy, creative papers get in. We will see...

    ReplyDelete
  21. One development that I think has decreased the effectiveness of program committees is the increasing power of "senior PC members" or "meta-reviewers" or whatever you call it. Having senior PC members closely supervise the review process, to make sure every paper gets careful attention from the reviewers, and that all of the reviewers' concerns are discussed in detail can be valuable. However, such a role works best when executed modestly. When senior PC members step in and overrule the will of the reviewers they are dominating the direction of the community in a way that is usually not healthy. Of course, the senior PC should provide feedback to the reviewers, and raise issues for discussion, but if the reviewers are not convinced, the senior PC member should respect their will.

    John

    ReplyDelete
  22. If you could come up with a few more of these, you could make a BINGO card to pass out at PC meetings. When people saw a PC member exhibiting one of these behaviors, they could mark a square. I'm not sure what would happen when you called out BINGO, though -- you might find it makes several PC members surly.

    ReplyDelete
  23. This blog provides a brillant insight and should be accepted in the toughest conference :)
    It does open my frustrated mind after many rejections, thanks ^_^

    ReplyDelete

Startup Life: Three Months In

I've posted a story to Medium on what it's been like to work at a startup, after years at Google. Check it out here.