Skip to main content

The PC Meeting Protocol

I am always surprised at how chaotic program committee meetings tend to be. Although most people have served on several PCs, it seems that a lot of the same procedural questions and issues come up each time, and it would be helpful to establish a common protocol for the community to maintain order. Having just gone through the SIGCOMM TPC meeting (with a whopping 50 PC members - it was like being a delegate at the UN) I started thinking about some of the things we could possibly standardize to make the process run more smoothly. (By the way, Geoff and KK did an awesome job running the meeting - the problems outlined below are common to *all* PCs I have been on!) Michael Mitzenmacher not-live-blogged about this meeting here.

The first is laying down the ground rules. Program chairs tend to assume that PC members know basic things like not to leak information during the PC meeting (emailing your students or colleagues when the paper is being discussed), not to express an opinion on papers you didn't review, and not to compare the paper to the version that was rejected last year. This stuff seems obvious but it's amazing how often people forget.

The order in which papers are discussed is another important decision. In this case there is no one-size-fits-all model. In my opinion, the best model is to group papers by broad topic areas, and within each area discuss the best and worst papers in the group first, followed by those in the middle of the group. This helps to calibrate things and keeps the discussion focused on a set of papers with some commonality. The worst model, I think, is to discuss papers by order of submission number (which is effectively random), since people don't get calibrated that way.

Keeping the discussion to a time limit is very important. Many PC members think that it's their job to hold forth about every minute detail of the paper during the meeting. Once, there was a case where the lead discussant went on for about 6 minutes about a paper, and when finally cut off and asked what he thought the decision should be, said "I don't care!" PC members need to remember that the audience for this discussion is the chairs and the other reviewers (the other PC members are usually reading email or otherwise ignoring discussion of papers they didn't read). So keep it short and sweet, and respect everyone's time.

Dealing with conflicts is always a huge pain. It is disruptive to ask people to leave the room and then call them back in later. It would be awesome if HotCRP could automate this (coming up with a paper discussion schedule to minimize the number of times someone has to leave the room). I'd like an iPhone app that automatically buzzes people when they should leave and come back into the room -- a lot of time is wasted in PC meetings with these context switches.

It has to be recognized that reaching consensus is not always possible. PC members have to accept that they will win some and lose some. I dislike it when the discussion comes down to a vote amongst the reviewers, since that is rarely a good way to decide these things, but at least it is quick and painless.

The last issue is the role of shepherding. This should be explained clearly by the PC chairs at the start of the meeting. Personally, I am in favor of shepherding every paper for a major conference, with the goal being to ensure that the final paper really satisfies the reviewers' main comments and the writing is cleaned up (as it often needs to be). In general, shepherding should not be used to get the authors to run new experiments or change something technical in the design -- it should be limited to changes in wording or positioning of the work. This question comes up at every PC meeting I've been on and setting expectations early would make things easier.

Check out my related post on the psychology of program committees for a behind-the-scenes look at what happens behind the closed doors.


  1. What's the rationale behind making PC members with conflicts leave the room (instead of simply abstaining from the discussion)? Is there a fear that reviewers will not feel that they can be frank in their comments with those people still in earshot? I'd think that would only apply to actual AUTHORS. In other words, I'd ask any of the paper's authors to leave the room, but people with other conflicts (e.g. they read a draft or provided some data, or whatever) could stay (but not discuss) - it doesn't seem like it would be a big deal and would reduce overhead.

  2. Ian, I think the main reason is to avoid any sense of impropriety in terms of how a paper is handled. The other conflicts tend to be people at the same institution or an advisor-advisee relationship. Sometimes this gets out of hand (for example when half of the PC has to leave the room because someone from MSR is an author on a paper). Although it's a pain on balance I think it's good to be conservative about conflicts.

  3. What are the main differences between the journal and conference review processes? Do you think conference papers are more important that journal papers because of the TPC?


Post a Comment

Popular posts from this blog

Why I'm leaving Harvard

The word is out that I have decided to resign my tenured faculty job at Harvard to remain at Google. Obviously this will be a big change in my career, and one that I have spent a tremendous amount of time mulling over the last few months.

Rather than let rumors spread about the reasons for my move, I think I should be pretty direct in explaining my thinking here.

I should say first of all that I'm not leaving because of any problems with Harvard. On the contrary, I love Harvard, and will miss it a lot. The computer science faculty are absolutely top-notch, and the students are the best a professor could ever hope to work with. It is a fantastic environment, very supportive, and full of great people. They were crazy enough to give me tenure, and I feel no small pang of guilt for leaving now. I joined Harvard because it offered the opportunity to make a big impact on a great department at an important school, and I have no regrets about my decision to go there eight years ago. But m…

Rewriting a large production system in Go

My team at Google is wrapping up an effort to rewrite a large production system (almost) entirely in Go. I say "almost" because one component of the system -- a library for transcoding between image formats -- works perfectly well in C++, so we decided to leave it as-is. But the rest of the system is 100% Go, not just wrappers to existing modules in C++ or another language. It's been a fun experience and I thought I'd share some lessons learned.

Why rewrite?

The first question we must answer is why we considered a rewrite in the first place. When we started this project, we adopted an existing C++ based system, which had been developed over the course of a couple of years by two of our sister teams at Google. It's a good system and does its job remarkably well. However, it has been used in several different projects with vastly different goals, leading to a nontrivial accretion of cruft. Over time, it became apparent that for us to continue to innovate rapidly wo…

Running a software team at Google

I'm often asked what my job is like at Google since I left academia. I guess going from tenured professor to software engineer sounds like a big step down. Job titles aside, I'm much happier and more productive in my new role than I was in the 8 years at Harvard, though there are actually a lot of similarities between being a professor and running a software team.

I lead a team at Google's Seattle office which is responsible for a range of projects in the mobile web performance area (for more background on my team's work see my earlier blog post on the topic). One of our projects is the recently-announced data compression proxy support in Chrome Mobile. We also work on the PageSpeed suite of technologies, specifically focusing on mobile web optimization, as well as a bunch of other cool stuff that I can't talk about just yet.

My official job title is just "software engineer," which is the most common (and coveted) role at Google. (I say "coveted&quo…