Scaling up program committees

As a follow-up to my earlier post on scaling up the number of papers that conferences accept, I wanted to comment on the reviewing load imposed on program committees. Ken Birman and Fred Shneider have a thought-provoking article on this topic in May's issue of CACM (thanks to Yuiry Brun for the pointer). They touch on many points, but one issue they do not explicitly consider is the possibility of increasing the size of the program committee itself to reduce the workload.

The figure below shows the size of the program committee and the number of submissions for the last few years of SOSP and OSDI (OSDI 2002 is left out since I could not find data on the number of submissions). Note that I am not counting program chairs in the PC size, since presumably they do not shoulder the same burden for paper reviews (indeed, they have a much harder job).



I also estimate the number of reviews by each PC member, assuming that -- on average -- every paper gets four reviews. This is a guess and it may in fact be closer to 3 reviews per paper, but many conferences are now doing at least two reviewing rounds, so this seems reasonable. Split the difference if you like. I happened to be on the OSDI 2004 PC when the number of submissions spiked, and indeed I did have to review around 45 papers. (My average review length for that year was 60 lines of ASCII text, or around 3.5 KB per review -- you do the math -- I worked my butt off.)

As the figure shows, in the last couple of years, program chairs have caught on that it is time to increase the PC size to compensate for the increased number of submissions. Prior to 2007, the typical PC size was 12 or 13, whereas in the last couple of years it has spiked to 26, 31, and 33 (for SOSP 2009). Some conferences have adopted a "light" and "heavy" PC model in which the "heavy" members get more papers to review and have to attend the PC meeting.

In general I think it is beneficial to increase the program committee size, within reason. The classic model that the PC was mainly comprised of a "wise council of elders" seems too limiting and, as Ken and Fred point out, cannot scale. Looking at the last few OSDI and SOSP PCs, they are fairly diverse, with quite a few names that I haven't conventionally associated with these program committees, whereas prior to about 2002 there was far more homogeneity. This practice widens the scope of the community and gives more people an opportunity to help shape the direction the conferences take. This is a good thing.