Skip to main content

Scaling up program committees

As a follow-up to my earlier post on scaling up the number of papers that conferences accept, I wanted to comment on the reviewing load imposed on program committees. Ken Birman and Fred Shneider have a thought-provoking article on this topic in May's issue of CACM (thanks to Yuiry Brun for the pointer). They touch on many points, but one issue they do not explicitly consider is the possibility of increasing the size of the program committee itself to reduce the workload.

The figure below shows the size of the program committee and the number of submissions for the last few years of SOSP and OSDI (OSDI 2002 is left out since I could not find data on the number of submissions). Note that I am not counting program chairs in the PC size, since presumably they do not shoulder the same burden for paper reviews (indeed, they have a much harder job).



I also estimate the number of reviews by each PC member, assuming that -- on average -- every paper gets four reviews. This is a guess and it may in fact be closer to 3 reviews per paper, but many conferences are now doing at least two reviewing rounds, so this seems reasonable. Split the difference if you like. I happened to be on the OSDI 2004 PC when the number of submissions spiked, and indeed I did have to review around 45 papers. (My average review length for that year was 60 lines of ASCII text, or around 3.5 KB per review -- you do the math -- I worked my butt off.)

As the figure shows, in the last couple of years, program chairs have caught on that it is time to increase the PC size to compensate for the increased number of submissions. Prior to 2007, the typical PC size was 12 or 13, whereas in the last couple of years it has spiked to 26, 31, and 33 (for SOSP 2009). Some conferences have adopted a "light" and "heavy" PC model in which the "heavy" members get more papers to review and have to attend the PC meeting.

In general I think it is beneficial to increase the program committee size, within reason. The classic model that the PC was mainly comprised of a "wise council of elders" seems too limiting and, as Ken and Fred point out, cannot scale. Looking at the last few OSDI and SOSP PCs, they are fairly diverse, with quite a few names that I haven't conventionally associated with these program committees, whereas prior to about 2002 there was far more homogeneity. This practice widens the scope of the community and gives more people an opportunity to help shape the direction the conferences take. This is a good thing.

Comments

  1. There's an interesting recent paper on PCs by Crowcroft, Keshav and McKeown, it discusses many (or all?) of the issues with conferences today.

    J. Crowcroft, S. Keshav, and N. McKeown, Scaling Internet Research Publication Processes to Internet Scale (http://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/08/wowcs.pdf), Proc. Workshop on Organizing Workshops, Conferences, and Symposia in Computer Systems, April 2008.

    ReplyDelete

Post a Comment

Popular posts from this blog

Why I'm leaving Harvard

The word is out that I have decided to resign my tenured faculty job at Harvard to remain at Google. Obviously this will be a big change in my career, and one that I have spent a tremendous amount of time mulling over the last few months.

Rather than let rumors spread about the reasons for my move, I think I should be pretty direct in explaining my thinking here.

I should say first of all that I'm not leaving because of any problems with Harvard. On the contrary, I love Harvard, and will miss it a lot. The computer science faculty are absolutely top-notch, and the students are the best a professor could ever hope to work with. It is a fantastic environment, very supportive, and full of great people. They were crazy enough to give me tenure, and I feel no small pang of guilt for leaving now. I joined Harvard because it offered the opportunity to make a big impact on a great department at an important school, and I have no regrets about my decision to go there eight years ago. But m…

Rewriting a large production system in Go

My team at Google is wrapping up an effort to rewrite a large production system (almost) entirely in Go. I say "almost" because one component of the system -- a library for transcoding between image formats -- works perfectly well in C++, so we decided to leave it as-is. But the rest of the system is 100% Go, not just wrappers to existing modules in C++ or another language. It's been a fun experience and I thought I'd share some lessons learned.

Why rewrite?

The first question we must answer is why we considered a rewrite in the first place. When we started this project, we adopted an existing C++ based system, which had been developed over the course of a couple of years by two of our sister teams at Google. It's a good system and does its job remarkably well. However, it has been used in several different projects with vastly different goals, leading to a nontrivial accretion of cruft. Over time, it became apparent that for us to continue to innovate rapidly wo…

Running a software team at Google

I'm often asked what my job is like at Google since I left academia. I guess going from tenured professor to software engineer sounds like a big step down. Job titles aside, I'm much happier and more productive in my new role than I was in the 8 years at Harvard, though there are actually a lot of similarities between being a professor and running a software team.

I lead a team at Google's Seattle office which is responsible for a range of projects in the mobile web performance area (for more background on my team's work see my earlier blog post on the topic). One of our projects is the recently-announced data compression proxy support in Chrome Mobile. We also work on the PageSpeed suite of technologies, specifically focusing on mobile web optimization, as well as a bunch of other cool stuff that I can't talk about just yet.

My official job title is just "software engineer," which is the most common (and coveted) role at Google. (I say "coveted&quo…