Many prestigious conferences in systems and networking -- such as SOSP, SenSys, and SIGCOMM -- constrain themselves to a single track over two and a half days. This limits the number of papers that can be presented at the conference to around 25-30 at most, assuming 30-minute talk slots.
The problem is that the field has been growing, but the publication venues have not. This means it is becoming increasingly competitive to get a paper accepted to one of these venues. You can check out the stats for yourself here. Although the stats are not yet on that page, SIGCOMM 2009 accepted only 10% of submitted papers. Conference publications in top venues are now a highly prized commodity, and one that is becoming increasingly valued over time. Unfortunately this leads to spiraling inflation, in the sense that the harder it is to get a paper accepted, the more it is seen as a superhuman achievement ("Wow! You got a paper into SIGCOMM!"), causing more people to submit more papers to the conference: a vicious cycle.
This value inflation is evident in the CVs of the hotshot graduate students on the job market last year. Several of us on the hiring committee were amazed at how many freshly-minted Ph.D.s were coming out with multiple papers in places like SOSP, OSDI, and NSDI. Clearly there is a lot more weight placed on getting those papers accepted than there used to be. When I was a grad student, publicatons were important, but nobody pushed me to publish relentlessly -- I submitted papers when I had something to submit. (I'll admit this was not very strategic.) Somewhere along the way the ante has been upped considerably.
Of course, this is a "great" thing for those lucky few who are successful at publishing at these venues. (To be fair, I count myself as part of that cohort.) But it does little to foster the community as a whole. Grad students need publications in top conferences to be taken seriously for faculty jobs. Junior faculty need them for tenure. If the conference venues become more and more selective, I don't see how we can sustain growth in the field as a whole. It seems to me that the number of papers accepted at conferences needs to keep pace with the number of new faculty and students entering the field. Either that or we need to reset our expectations of what constitutes making a research contribution.
There is also a potential loss to the diversity of the research community. Getting a paper into SOSP or NSDI takes a substantial investment of money and time. Research groups with more firepower (like, say, Microsoft Research) have a much easier time than those who might have good ideas but fewer resources. I don't have hard data to back this up, but it feels that it is increasingly rare to see papers from anything other than "top ten" universities and labs in the top venues. One thing that would help would be a quota on the number of papers that a given institution or author could submit to a conference, much as the NSF does for some of its programs. (It seems that everyone I know at MSR is working on at least least three papers for every major deadline. This is insane.)
Now, I am not suggesting that conferences lower their standards. But we need to decide what is reasonable and scale up as the community grows. One way to scale up is to create new venues, but this is not very cost-effective: it is expensive and time-consuming to run a conference, and who has time to go to so many events each year? Accepting a few more papers once you already have the whole program committee in the room incurs much less overhead.
This may mean going to multiple tracks, having shorter talks (my preferred solution), or not presenting every paper at the conference orally. As much as people bemoan multi-track conferences, every "single track" conference is really double-track: there is the track of people who stay in the room and listen to the talks, and the track of those who stand in the hallway and chat.
Finally, it's clear that paper quality varies over time, and it seems unlikely that any specific target for acceptance rates (say, 25%) makes sense for a given conference in a given year. But I think we should take a hard look at where we are headed, and ask whether we shouldn't open up the field more by devaluing the currency of the conference paper.
Subscribe to:
Post Comments (Atom)
-
The word is out that I have decided to resign my tenured faculty job at Harvard to remain at Google. Obviously this will be a big change in ...
-
My team at Google is wrapping up an effort to rewrite a large production system (almost) entirely in Go . I say "almost" because ... -
I'm often asked what my job is like at Google since I left academia. I guess going from tenured professor to software engineer sounds l...
Here's a recent short article on the subject of growing demand for top-conference publications and the negative way in which it is affecting the quality of the accepted papers. I think the issue that, as a result of an increase in the number of submissions, the wrong papers get accepted to these conferences is quite real and important.
ReplyDeletehttp://cacm.acm.org/magazines/2009/5/24644-program-committee-overload-in-systems/abstract
Lots of thoughts in my mind, I caught a couple that are slightly more well-formed :) That's not to say they're bullet-proof, but perhaps good enough for further thought. Hopefully :)
ReplyDeleteI'll start with 3 scenarios, one of my experience, two of other branches of sciences (anyone, please correct if I'm wrong).
o When I was a kid my first ambition was to be a scientist. I used to think then that what's right is right, and what's wrong is wrong. I used to think that science is not subjective, it's as clean a truth as it gets. Then I got to publishing, and realized that that's really not so. I think conferences coupled what's supposed to be non-subjective, to personal opinions. It couples what's correct, with what people think is good, and the latter is subjective.
o I have a friend who did his PhD in organic chemistry at Cambridge, UK. He solved a big problem, how to create organic molecules that, under the right conditions, will emit blue light (the other two primary colors, green and red, are easier and have already been solved). I was curious and attempted to read his groundbreaking paper. While I understood about 1% of his work, what struck me as very different from networking is the lack of "writing style": the report goes something like, "This work reports on how to get compound XYZ to emit light of this frequency. The steps are: take Y ml of compound A, mix with Z ml of compound B, subject to temperature T Kelvins for M minutes." And that's it. No motivations, no fancy intro, just the steps that describe how the experiment can be replicated thereby verifying the correctness.
o I heard that for physics, people submit work that are verified by members of the community for correctness, and thereafter are accepted. I don't remember what exactly they do for conferences, but the main aim is to generate correct works. And it's the number of accepted works that play a bigger role in determining the "worthiness" of a physicist.
When I look at a person's CV, I count the number of publications where the applicant's name is the first. I have the impression that many times people just "tag along", getting themselves onto the authorships to boost their paper counts. Personally, I value originality much more than engineering capability: I tend to think that people who come up with good ideas are harder to find than people who can execute them.
Next, I look at the references, to see if the "originality" part is true, as well as other traits of the applicant. Is he / she motivated? Does he / she have his / her own opinions rather than just following what others are doing?
And of course, there's the subjective parts: is the applicant's area of expertise in line with what the organization needs? I would also imagine that for faculty positions, the question of, "Am I willing to interact with this person for the rest of my life?" also comes to mind :)))
So, taking into account the above, I am thinking that a person's work should be handled differently: the hard truth part (if it's correct, then, it's correct and should be accepted), and the subjective part (do we want to hire a person with this expertise?). The first part, hard truth, can be handled by the community. The subjective part can be handled separately by whoever has additional interests.
And now it gets (even more) fuzzy: how then do we organize conferences? Perhaps they can consider only work that have first been verified to be correct, then, the PCs look through the piles of presentation slides (much like NANOG), and decide which ones tickle their fancy the most.
Cheng - good observations. Every time I read an article in Science or Nature I am struck at how terse and matter-of-fact it is. I am convinced we don't need 14 pages in our systems papers; we should make the limit 3 pages and see how authors respond!
ReplyDeleteYou are right that publication counts should not be the key metric for hiring decisions. On the other hand it does get you noticed.
I would not take the "correctness" line of thinking too far. A lot of CS, especially systems, is really about design, aesthetics, and elegance. I can build any number of "correct" solutions to a problem that are impractical, cumbersome, or just plain ugly. This is one of the many ways in which we differ from the pure sciences and even other areas of engineering: our solutions have to be yield insight into a design space, not just solve a problem.
I disagree with "the number of papers accepted at conferences needs to keep pace with the number of new faculty and students entering the field." The goal should be to keep the quality of the papers fixed over time. So if you have lots of new students but they're all doing crappy work, there's no reason to accept more papers. On the other hand, if you have lots of new students and the average student is doing better work now than they were doing 10 years ago, then you should accept even more papers than the increase in total number of students would suggest.
ReplyDeleteTo be honest, I'm not sure which situation we're in. Papers are certainly more esoteric today (I doubt someone is going to come up with ideas as groundbreaking as virtual memory, caches, RISC, RAID, TCP, etc. very often) so the results are less widely useful, but papers today do seem more technically sophisticated than papers 10 years ago.
I'm certainly not suggesting that we lower our quality standards in order to accept more papers. The constraint on accepting only 25 or so papers to a conference is artificial; I have a very hard time believing that in a given year, only 25 papers are worthy of being published in a top venue, despite the fact that the size of the community has grown so substantially. It stands to reason that as the community grows, so does the (potential) amount of *quality* output. To put this another way, do you think that the many CS departments that have (in some cases) doubled the size of their faculties over the last 10 years have lowered their quality standards for hiring? I certainly hope not.
ReplyDeleteNice post. I agree that both the community size and the value of "top" publications is increasing. However, if we look at conferences that scaled out (Infocom) vs. those that didn't (SOSP), we'd find that scaling out while maintaining quality is hard to do. A "gold standard" can be Nature. They have maintained a publishing rate at around 800 papers since 1997. Has the community increased? Sure. The acceptance rate has dropped from 10% to 7%. But once you have a Nature publication, even people in the other departments know you as "the guy who publishes in Nature".
ReplyDeleteMore competition is not necessarily a bad thing. It forces the students and faculty to bring out their best. A NSDI reject might be an easy accept somewhere else, but it forces the authors to go the extra mile in their work.
Finally, I think that we are at a stage where we need to answer other important questions e.g., how to ensure that only the best papers make it to the program and the "Oh that paper made it to SIGCOMM!?" ones don't. If the currency is getting more expensive then we are obliged to be more careful about who gets his/her hands on the currency. Scaling out, in my view, is a problem when you start leaving out papers that are clear accepts. I am not sure that the top CS conferences are quite there yet.
Your hiring at departments analogy is interesting. The departments have certainly not lowered their standards, but more quality venues have popped up respectively. There was no NSDI or SenSys before 2003. When departments hire, they also expand the scope of research carried out at the department - I think.
All in all it is good to see some debate on this topic! :-)