The Psychology of Program Committees

One thing that I frequently tell my grad students is that your chances of getting a paper accepted to a conference depend as much on who the reviewers are, and what kind of mood they are in, than the content of the paper itself. OK, leaving aside those obviously brilliant papers that my group is known to churn out, the paper content does matter -- but on most program committees there is a large pile of "pretty good" papers that all have a roughly equal chance of getting accepted. In these cases it often comes down to the collective mindset of the reviewers during the PC meeting.

There are many subtle psychological effects that influence the disposition towards a given paper. The first has to do with the timing of the paper discussion. At the beginning of a PC meeting, everyone is amped up on caffeine and uncalibrated with the respect to the overall quality of the papers being discussed. Your chances of getting a paper accepted when it is discussed early on in the meeting can vary widely. Most PC meetings discuss the top-ranked papers first, but after a string of (say) five or so papers accepted, people start thinking that it's time to reject a paper, so the sixth paper tends to be a scapegoat to release the pressure and make everyone feel better that the conference is still "selective." Fortunately, most PC chairs recognize this effect and switch gears to discussing some of the lower-ranked papers next, so the committee sees both ends of the spectrum. I've been in PC meetings where the top ranked paper with four strong accepts and a weak accept is tabled for further discussion, just because the committee is thirsty for blood.

Late in the afternoon, everyone is exhausted, cranky, and the power structure of the PC meeting has started to play itself out -- who's willing to throw themselves on the tracks to accept or reject a paper; who's willing to defer to more senior members of the committee; etc. Here we start to see some interesting personality traits emerge that can save or sink a paper:

"I didn't read this paper, but I know the work." This drives me nuts -- someone who was not even a reviewer casting doubt on (or supporting) the paper being discussed because they know the people involved or had some discussion with them at a conference a few months ago. This should be flatly disallowed by the program chairs, but I've seen it happen. A variant on this is...

"I didn't read this paper, but I saw their original failed submission last year." Again, this should be disallowed by PC chairs -- whether a paper was any good last year should have no bearing on the discussion of the present submission.

"I'm not an expert in this area, but I don't think there's anything novel here." Too many times a reviewer who is simply not qualified to review a paper is unwilling to defer to more expert members of the committee. Someone who doesn't know the related work that well might infuse the discussion with a vague sense of unease that taints the rest of the reviewers and makes it harder for someone to champion the paper for acceptance.

"I know way too much about this area and they should have used a value of 1.3 instead of 1.4 for the alpha parameter on page 7." Often, when a paper is too close to a reviewer's area, they tend to nickle-and-dime the paper for small problems that chip away at its credibility. Sometimes this is a poorly disguised attempt at tamping down the competition. These kinds of reviewers often miss the forest for the trees, where a paper has some good ideas but needs some rough edges sanded off, as all papers do.

"I'm a new faculty member and want to prove how smart I am by rejecting most of the papers in my pile." When you are new to program committees there is a real temptation to exercise your new power by rejecting papers left and right, which clearly establishes your intellectual dominance over the poor authors who are at your mercy. Most new faculty fall into this trap, and I've certainly been in this situation before.

"I'm a senior, well-respected faculty member and like to compare all of the papers in my pile to things published in the 1960s." The "there's nothing new here" argument sometimes comes up when you have a senior, somewhat jaded PC member who thinks that all of the good ideas were published back in the good old days when men were men and the women programmed in octal. It's inevitable that good ideas will come up time and time again, and I actually think there is value in reevaluating previously-published ideas in the context of new technology capabilities and application demands. Perspective is a great thing but sometimes you can have too much perspective.

The final point is that it is easy to argue to reject a paper; much harder to argue to accept a paper over other reviewers' objections. If a reviewer is not really sure about the novelty or importance of a paper's contributions, they often defer to the most negative reviewer, since nobody likes looking like an idiot in a PC meeting. Standing up and championing a paper takes a lot of guts, and by doing so you are taking responsibility for any faults in the paper that might arise later (if it turns out, say, that the exact same idea was published before, or there is a flaw in the experimental design). I think it's important that every member of a program committee commit themselves to championing one or two papers during the meeting, even if they aren't so sure about them -- otherwise the process can get to be too negative. One way to view your role as a PC member is to identify that beautiful piece of work that would otherwise have been overlooked due to superficial problems that turn off the reviewers.

So, next time you get a paper rejected, just remember that it's probably because the reviewers were in a bad mood because they hadn't served the afternoon coffee yet.

The App Store is evil. And I love it.

Apple's App Store is the perhaps the most brilliant innovation in software distribution ever. In case you've been living in a cave, the App Store lets iPhone and iPod Touch (and soon, iPad) users download and install apps directly on their device. This has absolutely revolutionized the way that software applications are marketed and sold. With a single tap you can download an app and the price is automatically billed to your credit card. The best part is that Apple gets to keep 30% of the app price. So, every sale of the $900 iRa Pro app nets Apple $270. They must be raking it in!

Before the App Store, installing apps on mobile devices was a huge pain. My old Windows CE PDA required that you download a ZIP file to your Windows machine (a deal-killer right there), unpack it, run a wizard, physically tether the PDA to the PC, go through several steps to complete the installation, and usually reboot a couple of times for good measure. Any time the PDA's battery ran out I had to create a new device profile in Windows and re-install the apps by hand. It was totally broken and it is unsurprising that the application market did not exactly take off on these platforms. On the App Store, a purchase is just a tap away. I'll admit to have bought quite a few iPhone apps on a whim, maybe because I was about to board a flight and wanted a new game to try out. Dropping 99 cents on a new app does not seem like a big deal at the time, but if I were to add up all my app purchases in the last year, the total is no doubt in the triple digits.

Of course, the App Store is also blatantly, totally evil. It gives Apple a monopoly on the software distribution channel. Now, I'm all for quality control -- it's nice that Apple is trying to screen apps to ensure some modicum of sanity, and perhaps to screen out trojans and such -- but this is going to have a profound effect on how developers and users interact in the future. Essentially, the App Store means that the person owning the device has no control over what software can and can't be installed on that device. This is a huge philosophical shift from our current model, in which the hardware manufacturer, OS developer, and application developers were all separate entities. Apple is doing a great job at consolidating power for its platforms, which of course includes not just apps but also music, books, video, and other media.

I'm a big Apple fan boy so I find myself somewhat unnerved by these developments. When it was limited to these little cheesy mobile devices like the iPhone, the totalitarian App Store model did not seem like such a problem. Now it's being expanded for the iPad, and it would not surprise me to see an App Store-like model for conventional desktops and laptops in the future. This has dire implications for freedom and openness, which I think is important for the future of technology. As much as I like Apple's products, I think it's dangerous to let one company decide what we can and can't install on a device. In five years when everyone is using iPads instead of laptops, how are Computer Science students supposed to tinker and learn to program on their own when they have to go through Steve Jobs' army of goons before they can even run their own code?

Of course, I'm eagerly awaiting my iPad delivery this weekend. And I can't wait to drop $5 for the iPad version of Flight Control -- it's going to be awesome!

Mac tools for profs

Macs seem to be insanely popular amongst CS faculty. Most conferences and faculty meetings I go to are dominated by Mac users. No big surprises, since (a) Macs work, and (b) they're sexy. I switched from Linux to Mac a couple of years ago after I got tired of editing three configuration files and rebooting to join a wireless LAN. That worked when I was a grad student, but now I'm too busy for that kind of crap.

I wanted to share some links to good Mac specific tools that I've found to be very useful in my job. If you have other suggestions, please share them as comments!

OmniGraffle is a great figure drawing program and produces very professional results. It's also easy as hell since it can do most of the layout for you, making sure that the boxes and arrows all line up correctly. The PDF output looks very slick and I've been using it for most of the figures in my papers; see Figure 1 in our SenSys'08 paper on Pixie for an example. Be sure to get the educational pricing.

Papers is one of my favorite Mac apps. It's like iPhoto for PDF files -- it will keep track of all of your papers, index them by title, author, keyword, etc. I use this program to keep track of my ever-growing reading list (rather than printing out a bunch of papers and letting them collect dust on my desk.) It also has a pretty slick interface for matching metadata about a paper (e.g., to pull in full citation information from the ACM Digital Library or Google Scholar) and for exporting to BibTeX and other formats. You can take notes and there's even an iPhone app (and soon, an iPad app) to let you read and take notes on papers on the go. (Yes, reading a two-column paper on the iPhone screen works -- if you zoom in on a single column and go widescreen, the text is the same size as on a printed page.) The only downside is that you have to manually synchronize the Papers library across multiple machines, easily accomplished with Unison, but I wish that were simpler.

BibDesk is a BibTeX library organizer. To import a new BibTeX entry, just copy it to your clipboard and paste it in BibTeX -- everything appears in the correct fields. You can also drag and drop BibTeX entries between files. This is infinitely easier than editing BibTeX files by hand and keeps the formatting right.

OmmWriter is a fantastic little app that is a essentially a Zen text editor -- it clears your entire screen and shows you only the text that you are writing. This is a great way of minimizing distractions and focusing while you write. WriteRoom is similar but I find OmmWriter's interface more appealing.

Finally, Caffeine is a little app I could not do without -- it puts an icon in your menu bar that, when clicked, disables your screensaver. This is very important when giving talks and avoids the embarrassing moment when your screen saver kicks in mid-presentation.

Who pays for conference reviews?

Why not make authors pay to submit papers to conferences?

Serving on a program committee takes a tremendous amount of time. So, one of the frequent complaints that TPC members make is when authors submit half-baked, clearly below-threshold papers a conference just to get some reviews back on their work. Personally, I feel little responsibility to write detailed reviews on papers that are clearly in the "Hail Mary" category, but I still have to read them, and that takes time. Not to mention the long-term psychological damage incurred by having to read a slew of crappy papers one after the other... I'm still in therapy after IPSN 2007 :-)

The problem is that submitting a paper to a conference is free: all it takes is a few clicks of the mouse to upload your PDF file. (Of course, I'm not accounting for the cost of doing the research and writing the paper itself.)

Let's estimate the costs associated with serving on a program committee and reviewing a stack of papers. I spend about an hour reading and writing a review for each paper that I am assigned. A highly competitive conference will assign 25 papers (or so) across one or more reviewing rounds to each TPC member, equating to roughly 25 hours of my time. At my current salary, that is worth around $1900 (give or take). Then there is the PC meeting itself. This will typically involve two days' worth of work plus travel -- let's estimate 3 full days of labor, plus airfare and hotel, adding up to another $2500.

So, with a program committee of 18 people, that works out to around $79,000 to review something like 150 paper submissions. In other words, to recoup its costs, the conference should charge authors $500 just to submit a paper. This seems to make a lot of sense.

Of course, imposing this kind of a fee would no doubt drastically reduce the number of papers that are submitted. But this seems like a good thing: it would reduce the workload for TPC members, allow conferences to operate with smaller, more focused program committees, and vastly improve the quality of the submitted papers. It would potentially also improve the quality of the reviews, since TPC members would now be paid for their time. Although the financial incentive is not that great (e.g., my hourly rate for consulting is something like 5 times my regular salary), getting paid should encourage TPC members to take the process more seriously.

The only downside I can see is people who sign up for a slew of program committees and become "professional paper reviewers", but TPC chairs would clearly have to balance this against the research credentials of the people being asked to serve. Note that many journals impose author fees for publication of the paper, but presumably you are willing to pay once you have done all the work to get the paper accepted. And conferences expect authors to show up at the conference to present the paper, which can get to be pretty expensive as well. But it seems crazy to me that the research community provides this free paper reviewing service with no negative ramifications for submitting totally unpolished work.

The Paperless Tourist

I recently spent a week in Portugal for EWSN'10 and spent a few days in Lisbon and Porto on either end of the conference. I decided this time to go entirely paperless -- that is, not take a paper guidebook. Rather, I was going to rely entirely on my iPhone for all of the travel information. As an experiment it was largely successful, with some caveats.

Normally I take a Rough Guide or Lonely Planet guidebook with me when I travel, but this has two big disadvantages. First, I have to lug the book around wherever I go, which usually means also having a bag or something else just to carry the book when I'm out on the town. Second, having the guidebook out in a bar, restaurant, or on the street immediately pegs you as a tourist and I hate being so conspicuous. I'm all about blending in, as the picture on the right should make absolutely clear. (Pop quiz: Which one is me? Hint: I don't smoke.)

This time, I decided to rely on the iPhone Kindle app and bought the Rough Guide to Portugal for Kindle. So the entire text of the book was in my pocket at all times, and reading the book on the iPhone just makes me look like another cell-phone-obsessed tech junkie, which is fine by me. (At least in most places that I travel, although I have been to some pretty dodgy places where messing with an iPhone in public is likely to garner some unwanted attention.)

The big disappointment was that the resolution of the maps in the Kindle Rough Guide is not good enough to actually read the street names and markers -- even when zooming in on the map. I am not sure if this is unique to the iPhone or whether I'd have the same problem on a proper Kindle (I don't have one so I can't tell). I can say it isn't a problem when using a paper guidebook. So I could not really use the maps in the guidebook at all.

Of course, Google Maps is great on the iPhone and the GPS feature is a huge help when you're trying to get your bearings. However, this requires use of your data plan, which is expensive overseas. I bought a 50MB international data add-on which costs about $60. Other than having to monitor my usage it was a pretty good investment, though I really wish it were not necessary.

There are a couple of iPhone apps allowing you to download maps for offline viewing, including OffMaps (and quite a few rip off apps that simply take the same data and package it for a single city and sell you that alone for 99 cents.) They also permit use of GPS without incurring data charges. Unfortunately, they use free map sources that are much less accurate and complete than Google Maps -- the map for Coimbra was just terrible and only showed a couple of major highways, and none of the smaller back streets of the city. So the quality varies a lot.

The best iPhone app by far was the Lonely Planet Lisbon City Guide, which includes a great map with all of the restuarants, bars, etc. listed and linked to a little page telling you about the place with its hours. You can even search the guide, unlike the Kindle app which has no search capability. It's pretty terse but for a few days in a city was more than adequate. I also like how Web links can be tapped directly -- in case you want to dip into your data plan, say to check out the website for a restaurant or hotel -- the same is true in the Kindle e-books as well.

The final caveat was the limited battery life of the iPhone. The Kindle app does not eat a lot of power (I've read for hours on it with hardly a dent in the battery) but use of the GPS is pretty energy-intensive. While traveling I was using the iPhone a lot more than I usually do, and found that by late afternoon or early evening I was getting into dangerous territory, necessitating a quick recharge at the hotel if I was going to last the evening. Fortunately this coincides with my usual tourist siesta so it was not a problem.

If it were not for the poor resolution of the maps in Kindle Rough Guide, this combination of apps would have been an ideal solution for travel without a paper guidebook. If they can just fix that I'm ready to conquer the world without a book.