Geoff Challen and IDEA

First of all, I'm very pleased to report that my student Geoffrey Werner Challen (formerly known as Geoffrey Werner-Allen, for reasons explained on his own web page) just defended his thesis! Geoff was the lead student on our work on sensor nets for volcano monitoring and developed the Harvard MoteLab testbed (used by dozens of research groups around the world). His work on Lance and IDEA defined new paradigms for resource management in sensor networks. He will be joining the faculty at the University of Buffalo next year and I am very proud of him. Congrats!

Second, I just posted the final version of our MobiSys'10 paper on Integrated Distributed Energy Awareness for Wireless Sensor Networks. This paper deals with the problem of configuring individual nodes in a sensor net to achieve some network-wide energy objective, such as targeting a given network lifetime. Each node generally has many local parameters that can impact overall energy consumption, such as the choice of parent in a routing tree or the MAC protocol duty cycle. We observe that nodes performing a greedy, local decision will often get it wrong; for example, a node deciding to forward packets through a parent with limited energy can rapidly drain the parent's supply. When energy harvesting (such as solar charging) is employed, it becomes very difficult to tune each node to achieve a network-wide objective.

The idea behind IDEA (sorry!) is to enable individual sensor nodes to make decentralized decisions about how to configure their own state. Nodes share information on their own battery level, charging rate, and energy load with other nodes in the network, using an efficient neighborhood broadcast scheme (akin to Abstract Regions). This information coupled with a utility function representing the intrinsic value of each possible state (such as a choice of parent node) allows nodes to make informed decisions that account for the non-local energy impact. In the paper we show that IDEA can significantly improve energy load distribution and network longevity with low overhead.

The PC Meeting Protocol

I am always surprised at how chaotic program committee meetings tend to be. Although most people have served on several PCs, it seems that a lot of the same procedural questions and issues come up each time, and it would be helpful to establish a common protocol for the community to maintain order. Having just gone through the SIGCOMM TPC meeting (with a whopping 50 PC members - it was like being a delegate at the UN) I started thinking about some of the things we could possibly standardize to make the process run more smoothly. (By the way, Geoff and KK did an awesome job running the meeting - the problems outlined below are common to *all* PCs I have been on!) Michael Mitzenmacher not-live-blogged about this meeting here.

The first is laying down the ground rules. Program chairs tend to assume that PC members know basic things like not to leak information during the PC meeting (emailing your students or colleagues when the paper is being discussed), not to express an opinion on papers you didn't review, and not to compare the paper to the version that was rejected last year. This stuff seems obvious but it's amazing how often people forget.

The order in which papers are discussed is another important decision. In this case there is no one-size-fits-all model. In my opinion, the best model is to group papers by broad topic areas, and within each area discuss the best and worst papers in the group first, followed by those in the middle of the group. This helps to calibrate things and keeps the discussion focused on a set of papers with some commonality. The worst model, I think, is to discuss papers by order of submission number (which is effectively random), since people don't get calibrated that way.

Keeping the discussion to a time limit is very important. Many PC members think that it's their job to hold forth about every minute detail of the paper during the meeting. Once, there was a case where the lead discussant went on for about 6 minutes about a paper, and when finally cut off and asked what he thought the decision should be, said "I don't care!" PC members need to remember that the audience for this discussion is the chairs and the other reviewers (the other PC members are usually reading email or otherwise ignoring discussion of papers they didn't read). So keep it short and sweet, and respect everyone's time.

Dealing with conflicts is always a huge pain. It is disruptive to ask people to leave the room and then call them back in later. It would be awesome if HotCRP could automate this (coming up with a paper discussion schedule to minimize the number of times someone has to leave the room). I'd like an iPhone app that automatically buzzes people when they should leave and come back into the room -- a lot of time is wasted in PC meetings with these context switches.

It has to be recognized that reaching consensus is not always possible. PC members have to accept that they will win some and lose some. I dislike it when the discussion comes down to a vote amongst the reviewers, since that is rarely a good way to decide these things, but at least it is quick and painless.

The last issue is the role of shepherding. This should be explained clearly by the PC chairs at the start of the meeting. Personally, I am in favor of shepherding every paper for a major conference, with the goal being to ensure that the final paper really satisfies the reviewers' main comments and the writing is cleaned up (as it often needs to be). In general, shepherding should not be used to get the authors to run new experiments or change something technical in the design -- it should be limited to changes in wording or positioning of the work. This question comes up at every PC meeting I've been on and setting expectations early would make things easier.

Check out my related post on the psychology of program committees for a behind-the-scenes look at what happens behind the closed doors.

Why I am a Computer Scientist

As a counterpoint to my last, somewhat pessimistic post on the future of CS, I thought I'd post something more upbeat today.

Ed Lazowska from UW gave the colloquium at Harvard this week on Computer Science: Past, Present, and Future. This is a talk he has no doubt given many times at many places, though it was the first I had heard it, and it was fantastic. More than anything else, his talk reminded me of why I am a computer scientist, and of how many great problems we have to work on in this field. I can't imagine wanting to do anything else.

Ed's talk started off reviewing what computer science has accomplished in the last 40 years, since the ARPAnet came online in 1969. This New York Times story from 2009 reported on the "Top 20 inventions of the last 30 years" and nearly all of them are derived from computer science in one fashion or another -- the Internet, PC, mobile phones, and email top the list. Although there's no surprise here at all, it was a great reminder of how much impact computing has had on nearly all aspects of the modern world.

This is why I am a computer scientist: because I want to change the world. Computing is the engine that is driving innovation in every field of humanity, and working on computer science puts you at the center of it. Of course, it's easy to forget that on days when I am beating my head against some obscure Objective C type mismatch bug, so it's nice to get a reminder of the bigger purpose.

This is why I am a computer scientist today, but it's not how I got started. My first experience with a computer was sitting down at an Apple II+ (I am dating myself) when I was 8 years old. I remember it clearly: The teacher told me to type my name at the prompt and I got:

]MATT

?SYNTAX ERROR

This was probably not the most rewarding first experience with a computer, but it taught me that this box spoke a different language, and I wanted to learn how to communicate with it. It was not long before I was writing BASIC programs to do lo-res graphics. While most kids in the class just manually translated their static image from a sheet of graph paper into a series of PLOT, HLIN, and VLIN commands, I was writing animated scenes (one was the pyramids with an animated sunset behind them) and even a sword-fighting game with a simple AI algorithm so you could play against the computer (keep in mind this was 3rd grade). I was totally hooked.

At that age, computers represented this amazing magical box that you could control and make it do almost anything. The games I was writing back then rivaled what I could buy in the store, but for me it was much more about writing the code than playing the game.

Now that I have a son, I wonder what his experience with computing will be like, and whether he'll be as turned on as I was by the whole mystery of the thing. Perhaps he will just treat the computer like any other appliance, like the dishwasher or telephone -- not something to be taken apart, programmed, explored. One thing we already do is play with the iPad, and even at 9 months old he loves to touch the screen and manipulate objects on it -- there are a couple of good iPad and iPhone programs for babies and the touch screen interface has tremendous potential there. But will he want to program? And how will he even do it? BASIC and LOGO were great back in the day as you could dive right in. I'm pretty sure I don't want to throw Visual Studio at him -- hopefully there are still good kid-friendly programming environments out there. And of course, the programs he'll be able to write won't be able to hold a candle to whatever the latest 3D immersive video game system he'll be playing then, so it's hard to say whether he'll appreciate the potential for doing it himself.

I am convinced that giving kids this very direct and rewarding experience with computing is important, though. If we turn computers in to just another kind of media consumption device (which most already are) then we'll lose that opportunity to hook the next generation.

Will DARPA save computer science?

I've been doing a lot of thinking lately about the role of academic computer science research vis-à-vis the state of the art in industry. When I was in grad school at Berkeley, we were building "big systems" -- the 200+ node cluster that I did all my research on was on the rough order of the size of sites like Inktomi and Google at the time. Since then, industry has scaled up by orders of magnitude, leaving academics in the dust. These days it's not clear that it makes sense for a university research group to work on systems problems that try to approximate industry: not only do we not have the resources, we simply have no idea what the real problems are at that scale. My friend and colleague Steve Gribble told me that after doing a sabbatical at Google, he decided that there's no way a university group could compete with what they're doing.

So what should academics be spending their time on? Too many academic systems researchers, I think, are doing "industry research in the small", with project horizons on the order of 2-3 years, not things that are radical departures from the state of the art. Is this the right approach? In a large department, it might be possible to approximate industry; David Patterson's PARLab at Berkeley is an example. This takes a tremendous amount of industry funding, though, and it's not scalable in the long run -- and there are not many people like Patterson who can pull that off. So what are the rest of us to do?

Ideally, academics should be pushing the envelope of computer science well beyond where industry is looking today. My research at Harvard has focused on radically new computing platforms -- mostly wireless sensor networks -- and our RoboBees project is pretty out there too. The downside is that industry isn't as interested in these problems, so it's harder to get funding from industry when you're not working on problems that are on their critical path. The other problem is that working on sci-fi it's more difficult to have impact on the real world, unless you're willing to wait for many years.

DARPA could be our savior. When I started my faculty job in 2003 I was told that getting money from DARPA would be no problem, but during the Tether years it mostly dried up. As a result nearly all of my research at Harvard has been funded by relatively small NSF grants plus various industry gifts. I am very hopeful that with Peter Lee heading up the new TCTO office that we'll see bolder initiatives from DARPA to bring back the good old days.

In the meantime, I guess I could find some bugs to squash in the Linux kernel...

Should Harvard's Intro CS class do away with grades?

We've been having some discussion amongst the CS faculty over the last few weeks about whether CS50, the intro computer science class at Harvard, should do away with letter grades and instead switch to SAT/UNSAT. Recently, the Harvard Crimson reported that CS50 is going to do away with letter grades, but this is not true -- the issue is still being debated by the faculty, and has yet to have any formal approval. (I don't know what led the Crimson to report it as though it were a fait accompli.) Given that my opinion seems to differ from most of the CS faculty, I thought I'd put my two cents forth here. Of course, this only represents my own thoughts on the matter, not the CS faculty as a whole (so if you are a Crimson reporter, don't go around reporting this as the final word).

For the record, I used to teach CS61, one of the two follow-on courses to CS50, so the output of CS50 directly feed into my course, and I have a vested interest in the intro course maintaining a certain standard. (My colleague Steve Chong is taking over CS61 next fall.)

David Malan, who teaches CS50, has done a fantastic job at increasing enrollments since taking it over 3 years ago -- I believe the enrollment has more than doubled (possibly tripled) under his watch. CS50 is going great guns and attracting students from all over the University, including many students who would have otherwise never taken a CS class. (Unlike a lot of places, CS is not required for Harvard undergrads as a whole, although it is required for a number of majors, including engineering, applied math, etc.) The course culminates in the CS50 fair where students show off their final projects. It is a great course and David has done amazing things to raise the profile of CS at Harvard.

So, right off the bat, I question the need to change the grading option for CS50, which potentially creates more problems than it solves.

David says that he has been wanting to get rid of letter grades entirely in CS50 for some time, and this year decided to raise the issue for discussion. David's claim is that there are still a lot of Harvard students who are intimidated by CS50 (despite the massive trend in increased enrollment) and that doing away with grades will make those students feel more comfortable trying out CS. While this may be true, I am not sure that we should be designing our foundational intro courses for those students. Harvard already has a "CS for non-majors" course, CS1, to fill that need.

A number of faculty believe that more women will be attracted to CS if they can take the course without worrying about their grade. I can't say whether that is true, but I find it somewhat implausible that what is standing between women and CS is just letter grades.

Even if you believe that doing away with letter grades will increase enrollment, consider the possible downsides. The first is that students who complete CS50 with a "SAT" grade won't have a good idea of their readiness to take more advanced CS courses. Grades do have informational value, and eliminating them makes it much harder to plan one's future course of study. The second is that the course will be less attractive to students who intend to major in CS or engineering, or who are simply used to getting all A's (as many incoming freshmen at Harvard are), so this change could actually decrease enrollment at the top end of the course. It would be fine if the hardcore students could just skip CS50 and take one of the later courses instead, but those courses are not currently designed to supplant CS50 for the better-prepared students. Also, skipping CS50 means missing out on the community experience that I think is important for student cohesion.

The third, and most severe, problem with this proposal is that it will make CS50 no longer suitable for the ABET-accredited engineering degree (which requires letter-graded courses) and the University-wide General Education requirement. So by trying to cast a wider net with CS50, the course no longer satisfies important requirements for certain students, so this approach seems self-defeating.

The compromise proposal currently on the table is to offer CS50 in two flavors: a letter-graded version (call it CS50g) and a SAT/UNSAT version (call it CS50su). It would be the same exact course, just with different grading options. This way students who need the letter grade can take CS50g, and everyone else can take CS50su. It seems clear that this will backfire in several ways. First, many students who take CS50su will later realize they needed a letter grade to satisfy a requirement down the line, but it will be too late. Second, it will create a class distinction between the CS50g and CS50su students. Students taking the course for a letter grade will demand much more detailed feedback on their assignments and more rigor in the grading process (since they will be competing for A's) whereas the course staff, used to dealing with mostly SAT/UNSAT students, will not feel the need to make such fine distinctions.

I have not looked into what other universities do and whether there is any agreement that doing away with letter grades for an intro CS course is a good idea. I know that MIT grades all freshmen pass/fail to alleviate stress associated with grades, which might make sense if done across the board, but it is unclear that changing one class is the right way to do this.

Personally, I'd rather not mess with CS50 right now. It's doing great things and I am concerned that doing away with letter grades would do more harm than good. Of course, many of my colleagues here at Harvard disagree with me, and they are encouraged to weigh in on the comments section below.

Update 4/20/10: The Crimson published a follow-on article this morning discussing the CS50 controversy.

Update #2 4/26/10: Given the many concerns raised, there was simply not enough time for the CS50 SAT/UNSAT proposal to be discussed by the faculty before the deadline for changes to the course catalog for next fall. As a result the proposal has been tabled for now; I expect it will be revisited next year.

HotOS XIII Call for Papers

I am pleased to announce that the call for papers for the Thirteenth Workshop on Hot Topics in Operating Systems (aka HotOS XIII) has been announced. I am fortunate to serve as the program chair and we have put together a fantastic program committee for the workshop, which will be held from May 8-10, 2011 in Napa Valley (think good wine and fantastic food... No promises yet on whether Thomas Keller will be doing the catering). While it's a long ways off, it never hurts to start thinking ahead about what you want to submit! The paper deadline is January 15, 2011.

A few words about HotOS and my own philosophy behind the workshop. HotOS has been the flagship venue for bold new ideas in the systems community over the years. It is often the first place that we hear about new projects and exciting ideas, and it's also a place for grad students to float their crazy thesis plans before submitting full papers to places like SOSP and OSDI. Just to be clear on the format: HotOS submissions are five-page position papers, which are not to be confused with miniature versions of full conference papers (i.e., with half-baked ideas and none of the graphs). The best submissions to HotOS are those that challenge widely-held assumptions, make a clever or unorthodox argument, and get people thinking and talking. In my opinion, the ideal HotOS paper has a strongly-worded idea that gets people working on new problems, without concern for how practical the idea is in the short term. Graphs and early prototype results are actually a negative here -- if your idea has progressed to that point, it is probably not a good candidate for HotOS.

One common complaint about HotOS is that it is sometimes heavily loaded with "SOSP preprints," and indeed, there is a pretty high conversion ratio from HotOS paper to SOSP paper in the same year. Not everyone thinks this is a bad thing and I agree this does serve a certain purpose. As program chair I plan to exert my influence to keep things focused on the bleeding edge, but I also respect that different members of the TPC will have their own taste for different styles of papers.

Beyond the technical content, HotOS serves an important role of building ties between members of our community, and especially to help grad students get a chance to engage, mano a mano, with more senior folks in the field. Mentorship is an incredibly important part of the workshop. I'll never forget my first HotOS and the chance to bat around ideas with such luminaries as Jay Lepreau. There is also a fair bit of, shall we say, enology involved at HotOS, which certainly makes things more interesting (distributed hash tables have never been more fascinating!). There is a huge difference between meeting folks at a 400+ person conference versus a small workshop where people are more likely to let their guard down.

I want to build upon this tradition and make HotOS more inclusive to new folks and less-represented research groups. I am going to be on the lookout for unusual, bold, oddball papers from outside of the "conventional" systems community, with the hope of shaking things up a bit, but it's safe to say that there will be plenty of familiar faces as well. If you have a nutty idea, by all means submit it. I'm going to try to tease apart novelty from "solid work" in the reviewing process, and take a range of papers that have different kinds of strengths. We'll also be inviting some folks who don't submit papers at all just to be sure we have a good mix of disciplines and groups represented.

If you have thoughts or suggestions for HotOS, please feel free to drop me a line or post comments here. And I look forward to your submissions!

Reviewing papers on an iPad

Following up on my recent post on Mac tools for profs, I wanted to share some early thoughts on the use of the iPad for reading and reviewing papers. (This seems to be 60% of my job these days, and it was the main reason I got an iPad in the first place.) For the last year or so I've gone paperless with paper reviews: I read PDFs on the laptop screen and have a separate text editor window open to type up my review. So the iPad seemed like the perfect way to carry the proverbial stack of papers around with me and write up reviews anytime, anywhere.

I've been testing a bunch of iPad apps for paper reading and annotation. Verdict: The software for this is still immature, but it's clear that the potential is there. In a few months I hope this will be a lot more straightforward.

Good news first: Reading PDFs on the iPad screen is fantastic. Although the screen is a little smaller than a 8.5x11" or A4 page, you can still read the text quite clearly and every reading app lets you zoom in (so you can, for example, eliminate the margins on the page or zoom in on a single column). The multitouch interface makes this easy to do and there's something quite visceral about panning and zooming around a paper with your fingers. (On my wish list: A PDF reading app that literally allows you to "poke holes" in papers as you read them, or crumple them up with a multitouch gesture.)

The PDF readers can handle very large documents and graphics and images are rendered just fine. I recently reviewed a 170+ page grant proposal on the iPad and had no problems with it at all.

Now for the bad news: First, annotating PDF files or taking notes is still somewhat crude, depending on the app. Some of the best reader apps, like GoodReader, don't support annotations at all. (I wouldn't be surprised if they added this feature sometime.) The best annotation support I've seen is iAnnotate, which lets you do pretty much anything (notes, highlights, scribbles, bookmarks), and the annotations can be read by any standard PDF reader. However, the user interface is a bit clunky and syncing PDF files is somewhat of a pain (see below). My favorite app so far, Papers, only supports plain text notes that are kept separate from the PDF file, so you can't just email a marked up paper to someone.

The other piece of bad news: Getting PDFs onto the iPad, or getting your notes off, can be a pain. Unfortunately, the iPad OS does not support a common filesystem across apps, so each app needs to have its own way to sync files to the device. This means that if someone emails you a PDF file, you can't just save it and load it up into one of the apps mentioned above.

The best sync support by far is GoodReader, which can pull from darn near anything: the Web, an FTP or WebDAV server, DropBox, Google Docs, even email attachments (which you configure separately from the Mail app). Unfortunately, there is no way to annotate PDFs and hence no way to get documents off the iPad. I hope they change this since they've clearly done a great job with the connectivity options.

iAnnotate and Papers have their own custom sync methods that require that you run an app on your Mac (or PC in the case of iAnnotate) and do the sync over the wireless LAN. They don't let you sync directly through iTunes, though again, this would be an obvious addition in the future. I already use Papers on the Mac to track my ever-growing list of papers to read, so this was the obvious choice for me. The UI needs a little tweaking: when reading in landscape mode, you have the Library pane taking up part of the display and there's no way to hide it. This would be easy to fix.

What is lacking is an all-in-one solution that lets you sync from anywhere, maintain metadata, and annotate. A Frankenstein of GoodReader, iAnnotate, and Papers is what I really want. Even better would be direct integration with HotCRP and the ability to edit and upload the review form directly. Hmm, maybe I'll have to write this app myself...