Running a successful program committee

Yesterday we held the program committee meeting for the 13th Workshop on Hot Topics in Operating Systems (HotOS), for which I am serving as the program chair. This is the premier workshop in the OS community and focuses on short (five page) position papers meant to bring out exciting new research directions for the field. In some years it has been more exciting than others. What tends to happen is that people send five-page versions of an SOSP submission they are working on, which (in my opinion) is not the best use of this venue. When HotOS becomes an SOSP preview I think it misses an important opportunity to discuss new and crazy ideas that would not make it into a regular conference.

We accepted 33 out of 133 submissions. The number of accepted papers is a bit higher in previous years because I wanted to be more inclusive, but also recognized that we could fit more presentations in at the workshop when you don't have 25-minute talk slots. There is no reason a 5-page paper needs a 25-minute talk, and I think it goes against the idea of the workshop to turn it into a conventional conference.

This was, by far, the best program committee experience I ever had, and I was reflecting on some of the things that made it so successful.

I've been on a lot of program committees, and sometimes they can be a very painful experience. Imagine a dozen (or more) people crammed into a room, piles of paper and empty coffee cups, staring at laptops, arguing about papers for 9 or 10 hours. Whether a PC meeting goes well seems to be the result of many factors...

Pick the best people. We had a stellar program committee and I knew going in that everyone was going to take the job very seriously. Everyone did a fantastic job and wrote wonderful and thoughtful reviews. These folks were invested in HotOS as a venue, were the kind of people who often submit papers to the workshop, and care deeply about the systems community as a whole. The discussion at the PC meeting itself was great, nobody seemed to get cranky, and even after 8+ hours of discussing papers there was still a lot of energy in the room. This is helped a lot by the content -- HotOS papers tend to be more "fun" and since they are so short, you can't nitpick every little detail about them.

Set expectations. I tried to be very organized and made sure that the PC knew what was expected of them, in terms of getting reviews done on time, coming to the PC meeting in person, what my philosophy was for choosing papers, and how we were going to run the discussion at the meeting. I think laying out the "rules" up front helps a lot since it keeps things running smoothly. I've blogged about this before but I think establishing some ground rules for the meeting is really useful.

Get everyone in the room. Having a face-to-face PC meeting is absolutely key to success. Everyone came to the PC meeting in person, except for one person whose family fell ill at the last minute and had to cancel, but even he phoned in for the entire meeting (I can't imagine being on the phone for more than eight hours!). I made sure the PC knew they were expected to come in person, and nailed down the meeting date very early, so everyone was able to commit. Letting some people phone in is a slippery slope. I can't count how many PC meetings I've been to that have been hampered by painful broken conference call or Skype sessions.

Use technology. We used HotCRP for managing submissions and reviews, which is by far the best conference management system out there. During the PC meeting itself, I shared a Google spreadsheet with the TPC which had the paper titles, topic area, accept/reject decision, and a one-line summary of the discussion. The summary was really helpful for remembering what we thought about a paper when revisiting it later in the day. Below is a snippet (with the paper numbers and titles blurred out). The "order" column below is the order in which the paper was discussed. This way, everyone in the PC could see the edits being made in real time and there was rarely confusion about which paper we were discussing next.

Pre-reject and pre-accept. I rejected around half of the submissions before the PC meeting and gave the PC a chance to revive any such paper for discussion (none were). I also "pre-accepted" about 10 papers that were noncontroversial; we saved discussion of these for the end of the day, since they were easy cases. We ended up discussing a total of 69 papers at the meeting, which meant we had to go at a pretty good clip.

Be definitive.  With very few exceptions, we tried to reach a clear accept/reject decision on each paper as we discussed it, and did not table any papers for later discussion. There was one case where we were hung on what to do with a paper and decided to push the discussion until the end of the day. In cases where there was disagreement, I would mark a paper as "presumed reject" or "presumed accept" and put down the name of the person who wanted to argue for the opposite outcome later. That gave us a chance to move on when there was an insurmountable debate, and it was clear that the champion (or anti-champion) of a paper would have a chance to have their say.

Take everyone out to a nice dinner afterwards. As far as I'm concerned, this was the best part of hosting the PC meeting.