The Secret Lives of Professors

I came to Harvard 7 years ago with a fairly romantic notion of what it meant to be a professor -- I imagined unstructured days spent mentoring students over long cups of coffee, strolling through the verdant campus, writing code, pondering the infinite. I never really considered doing anything else. At Berkeley, the reigning belief was that the best and brightest students went on to be professors, and the rest went to industry -- and I wanted to be one of those elite. Now that I have students that harbor their own rosy dreams of academic life, I thought it would be useful to reflect on what being a professor is really like. It is certainly not for everybody. It remains to be seen if it is even for me.

To be sure, there are some great things about this job. To first approximation you are your own boss, and even when it comes to teaching you typically have a tremendous amount of freedom. It has often been said that being a prof is like running your own startup -- you have to hire the staff (the students), raise the money (grant proposals), and of course come up with the big ideas and execute on them. But you also have to do a lot of marketing (writing papers and giving talks), and sit on a gazillion stupid committees that eat up your time. This post is mostly for grad students who think they want to be profs one day. A few surprises and lessons from my time in the job...

Show me the money. The biggest surprise is how much time I have to spend getting funding for my research. Although it varies a lot, I guess that I spent about 40% of my time chasing after funding, either directly (writing grant proposals) or indirectly (visiting companies, giving talks, building relationships). It is a huge investment of time that does not always contribute directly to your research agenda -- just something you have to do to keep the wheels turning. To do systems research you need a lot of funding -- at my peak I've had 8 Ph.D. students, 2 postdocs, and a small army of undergrads all working in my group. Here at Harvard, I don't have any colleagues working directly in my area, so I haven't been able to spread the fundraising load around very much. (Though huge props to Rob and Gu for getting us that $10M for RoboBees!) These days, funding rates are abysmal: less than 10% for some NSF programs, and the decision on a proposal is often arbitrary. And personally, I stink at writing proposals. I've had around 25 NSF proposals declined and only about 6 funded. My batting average for papers is much, much better. So, I can't let any potential source of funding slip past me.

Must... work... harder. Another lesson is that a prof's job is never done. It's hard to ever call it a day and enjoy your "free time," since you can always be working on another paper, another proposal, sitting on another program committee, whatever. For years I would leave the office in the evening and sit down at my laptop to keep working as soon as I got home. I've heard a lot of advice on setting limits, but the biggest predictor of success as a junior faculty member is how much of your life you are willing to sacrifice. I have never worked harder than I have in the last 7 years. The sad thing is that so much of the work is for naught -- I can't count how many hours I've sunk into meetings with companies that led nowhere, or writing proposals that never got funded. The idea that you get tenure and sit back and relax is not quite accurate -- most of the tenured faculty I know here work even harder than I do, and they spend more of their time on stuff that has little to do with research.

Your time is not your own. Most of my days are spent in an endless string of meetings. I find almost no time to do any hacking anymore, which is sad considering this is why I became a computer scientist. When I do have some free time in my office it is often spent catching up on email, paper reviews, random paperwork that piles up when you're not looking. I have to delegate all the fun and interesting problems to my students. They don't know how good they have it!

Students are the coin of the realm. David Patterson once said this and I now know it to be true. The main reason to be an academic is not to crank out papers or to raise a ton of money but to train the next generation. I love working with students and this is absolutely the best part of my job. Getting in front of a classroom of 80 students and explaining how virtual memory works never ceases to be thrilling. I have tried to mentor my grad students, though in reality I have learned more from them than they will ever learn from me. My favorite thing is getting undergrads involved in research, which is how I got started on this path as a sophomore at Cornell, when Dan Huttenlocher took a chance on this long-haired crazy kid who skipped his class a lot. So I try to give back.

Of course, my approach to being a prof is probably not typical. I know faculty who spend a lot more time in the lab and a lot less time doing management than I do. So there are lots of ways to approach the job -- but it certainly was not what I expected when I came out of grad school.

Update 4/24/10: Thanks to Mike Belfrage for pointing me to this interview with Niklaus Wirth where he echoes some of the above sentiments.

Proposal: Improving the NSF Review Process

Summary: Let's improve the NSF proposal review process by making it function more like conference program committees.

Intellectual merit: The core problem that this proposal addresses is the poor quality of many reviews submitted by NSF panelists. It is not uncommon for a proposal to be rejected with short, content-free reviews, offering little feedback to the authors. In many cases the scoring of a proposal is poorly justified, leaving the author mystified as to why they got such a low (or high) score. Recently, I had a proposal rejected where one of the reviews was essentially a single sentence in length. Not only does this not help the PI improve the work for later submission, but it leaves the impression that the review process is arbitrary.

(I'd like to emphasize that this is a problem that many NSF program managers have called attention to, but they are powerless as individuals to do much about it. So I believe the fault rests with the research community, not with the NSF PMs.)

A key problem with NSF panels is that there is no community standard for what constitutes a good (or even acceptable) proposal review. I am a strong advocate of the approach used by the systems community, where paper submissions are given extremely detailed reviews with constructive feedback. Given that we spend so much effort reviewing papers, couldn't we also give the same effort to NSF proposals, which arguably are more important than a single paper?

It is my impression that NSF program managers also have a hard time pulling panels together, mainly because people are so busy, and don't have the time to travel to DC. Yet many of the potential panelists freely serve on conference program committees with much higher reviewing loads and an expectation of much more detailed reviews. (A typical panelist will review 8-12 proposals, whereas a competitive conference will require TPC members to review 2-3x as many papers.) Why? One reason, perhaps, is that program committees are recognized for their work, and serving on a TPC is an important indication of one's stature in the research community.

These two issues are related. Since serving on an NSF panel is seen as "paying your dues," rather than an activity you take pride in, there is little incentive to write good reviews. However, if you write a bunch of crappy reviews for a TPC, you can earn a reputation as someone who doesn't take the process seriously might not get invited back in the future. So the public recognition of the TPC and the quality of the reviews go hand in hand.

My proposal: Let's have NSF panels emulate the conference program committee model. First, we should recognize panelists publicly for their work. Being on the "NSF NeTS 2010" panel should be as prestigious as serving on SIGCOMM or SenSys. The NSF should create a web page for each years' competition where they list the panelists and list the proposals funded through that competition (the latter information is available, but a little hard to dig up). So the community can take pride in the effort and see the outcome of the process more directly.

Second, establish expectations for high-quality proposal reviews. If you are a bad panelist, you won't get invited back in the future, so you won't gain the recognition of being asked to serve. Panelists will be chosen from among the best people in the field, where "best" is defined both by research contribution and service.

Third, hold panels somewhere other than Washington DC. Since I live in Boston, it's easy for me to get down there, but for people on the West Coast it is much harder. If panels are run in different locations around the country, the travel burden can be spread around more evenly.

I will be the first to admit that the conference program committee model is not perfect -- see my related posts here and here for thoughts on that. But in my experience it is better than the (typical) NSF panel.

Of course, NSF's conflict-of-interest guidelines will have to be tweaked. Currently, you can't serve on an NSF panel for a program for which you have submitted a proposal. (The upshot is that panels tend to consist of people who didn't get their act together to submit a proposal, which may not be the best group of scholars to evaluate the work.) Recently NSF has separated larger programs into multiple panels and simply ensured that someone can't serve on the specific panel for which their own proposal is under consideration.

Broader impacts: Improving the proposal review process and providing more constructive feedback to PIs will result in better science.

In the comments, please indicate your rating for this proposal:
   Excellent
   Very Good
   Good
   Fair
   Poor

Geoff Challen and IDEA

First of all, I'm very pleased to report that my student Geoffrey Werner Challen (formerly known as Geoffrey Werner-Allen, for reasons explained on his own web page) just defended his thesis! Geoff was the lead student on our work on sensor nets for volcano monitoring and developed the Harvard MoteLab testbed (used by dozens of research groups around the world). His work on Lance and IDEA defined new paradigms for resource management in sensor networks. He will be joining the faculty at the University of Buffalo next year and I am very proud of him. Congrats!

Second, I just posted the final version of our MobiSys'10 paper on Integrated Distributed Energy Awareness for Wireless Sensor Networks. This paper deals with the problem of configuring individual nodes in a sensor net to achieve some network-wide energy objective, such as targeting a given network lifetime. Each node generally has many local parameters that can impact overall energy consumption, such as the choice of parent in a routing tree or the MAC protocol duty cycle. We observe that nodes performing a greedy, local decision will often get it wrong; for example, a node deciding to forward packets through a parent with limited energy can rapidly drain the parent's supply. When energy harvesting (such as solar charging) is employed, it becomes very difficult to tune each node to achieve a network-wide objective.

The idea behind IDEA (sorry!) is to enable individual sensor nodes to make decentralized decisions about how to configure their own state. Nodes share information on their own battery level, charging rate, and energy load with other nodes in the network, using an efficient neighborhood broadcast scheme (akin to Abstract Regions). This information coupled with a utility function representing the intrinsic value of each possible state (such as a choice of parent node) allows nodes to make informed decisions that account for the non-local energy impact. In the paper we show that IDEA can significantly improve energy load distribution and network longevity with low overhead.

The PC Meeting Protocol

I am always surprised at how chaotic program committee meetings tend to be. Although most people have served on several PCs, it seems that a lot of the same procedural questions and issues come up each time, and it would be helpful to establish a common protocol for the community to maintain order. Having just gone through the SIGCOMM TPC meeting (with a whopping 50 PC members - it was like being a delegate at the UN) I started thinking about some of the things we could possibly standardize to make the process run more smoothly. (By the way, Geoff and KK did an awesome job running the meeting - the problems outlined below are common to *all* PCs I have been on!) Michael Mitzenmacher not-live-blogged about this meeting here.

The first is laying down the ground rules. Program chairs tend to assume that PC members know basic things like not to leak information during the PC meeting (emailing your students or colleagues when the paper is being discussed), not to express an opinion on papers you didn't review, and not to compare the paper to the version that was rejected last year. This stuff seems obvious but it's amazing how often people forget.

The order in which papers are discussed is another important decision. In this case there is no one-size-fits-all model. In my opinion, the best model is to group papers by broad topic areas, and within each area discuss the best and worst papers in the group first, followed by those in the middle of the group. This helps to calibrate things and keeps the discussion focused on a set of papers with some commonality. The worst model, I think, is to discuss papers by order of submission number (which is effectively random), since people don't get calibrated that way.

Keeping the discussion to a time limit is very important. Many PC members think that it's their job to hold forth about every minute detail of the paper during the meeting. Once, there was a case where the lead discussant went on for about 6 minutes about a paper, and when finally cut off and asked what he thought the decision should be, said "I don't care!" PC members need to remember that the audience for this discussion is the chairs and the other reviewers (the other PC members are usually reading email or otherwise ignoring discussion of papers they didn't read). So keep it short and sweet, and respect everyone's time.

Dealing with conflicts is always a huge pain. It is disruptive to ask people to leave the room and then call them back in later. It would be awesome if HotCRP could automate this (coming up with a paper discussion schedule to minimize the number of times someone has to leave the room). I'd like an iPhone app that automatically buzzes people when they should leave and come back into the room -- a lot of time is wasted in PC meetings with these context switches.

It has to be recognized that reaching consensus is not always possible. PC members have to accept that they will win some and lose some. I dislike it when the discussion comes down to a vote amongst the reviewers, since that is rarely a good way to decide these things, but at least it is quick and painless.

The last issue is the role of shepherding. This should be explained clearly by the PC chairs at the start of the meeting. Personally, I am in favor of shepherding every paper for a major conference, with the goal being to ensure that the final paper really satisfies the reviewers' main comments and the writing is cleaned up (as it often needs to be). In general, shepherding should not be used to get the authors to run new experiments or change something technical in the design -- it should be limited to changes in wording or positioning of the work. This question comes up at every PC meeting I've been on and setting expectations early would make things easier.

Check out my related post on the psychology of program committees for a behind-the-scenes look at what happens behind the closed doors.

Why I am a Computer Scientist

As a counterpoint to my last, somewhat pessimistic post on the future of CS, I thought I'd post something more upbeat today.

Ed Lazowska from UW gave the colloquium at Harvard this week on Computer Science: Past, Present, and Future. This is a talk he has no doubt given many times at many places, though it was the first I had heard it, and it was fantastic. More than anything else, his talk reminded me of why I am a computer scientist, and of how many great problems we have to work on in this field. I can't imagine wanting to do anything else.

Ed's talk started off reviewing what computer science has accomplished in the last 40 years, since the ARPAnet came online in 1969. This New York Times story from 2009 reported on the "Top 20 inventions of the last 30 years" and nearly all of them are derived from computer science in one fashion or another -- the Internet, PC, mobile phones, and email top the list. Although there's no surprise here at all, it was a great reminder of how much impact computing has had on nearly all aspects of the modern world.

This is why I am a computer scientist: because I want to change the world. Computing is the engine that is driving innovation in every field of humanity, and working on computer science puts you at the center of it. Of course, it's easy to forget that on days when I am beating my head against some obscure Objective C type mismatch bug, so it's nice to get a reminder of the bigger purpose.

This is why I am a computer scientist today, but it's not how I got started. My first experience with a computer was sitting down at an Apple II+ (I am dating myself) when I was 8 years old. I remember it clearly: The teacher told me to type my name at the prompt and I got:

]MATT

?SYNTAX ERROR

This was probably not the most rewarding first experience with a computer, but it taught me that this box spoke a different language, and I wanted to learn how to communicate with it. It was not long before I was writing BASIC programs to do lo-res graphics. While most kids in the class just manually translated their static image from a sheet of graph paper into a series of PLOT, HLIN, and VLIN commands, I was writing animated scenes (one was the pyramids with an animated sunset behind them) and even a sword-fighting game with a simple AI algorithm so you could play against the computer (keep in mind this was 3rd grade). I was totally hooked.

At that age, computers represented this amazing magical box that you could control and make it do almost anything. The games I was writing back then rivaled what I could buy in the store, but for me it was much more about writing the code than playing the game.

Now that I have a son, I wonder what his experience with computing will be like, and whether he'll be as turned on as I was by the whole mystery of the thing. Perhaps he will just treat the computer like any other appliance, like the dishwasher or telephone -- not something to be taken apart, programmed, explored. One thing we already do is play with the iPad, and even at 9 months old he loves to touch the screen and manipulate objects on it -- there are a couple of good iPad and iPhone programs for babies and the touch screen interface has tremendous potential there. But will he want to program? And how will he even do it? BASIC and LOGO were great back in the day as you could dive right in. I'm pretty sure I don't want to throw Visual Studio at him -- hopefully there are still good kid-friendly programming environments out there. And of course, the programs he'll be able to write won't be able to hold a candle to whatever the latest 3D immersive video game system he'll be playing then, so it's hard to say whether he'll appreciate the potential for doing it himself.

I am convinced that giving kids this very direct and rewarding experience with computing is important, though. If we turn computers in to just another kind of media consumption device (which most already are) then we'll lose that opportunity to hook the next generation.

Will DARPA save computer science?

I've been doing a lot of thinking lately about the role of academic computer science research vis-à-vis the state of the art in industry. When I was in grad school at Berkeley, we were building "big systems" -- the 200+ node cluster that I did all my research on was on the rough order of the size of sites like Inktomi and Google at the time. Since then, industry has scaled up by orders of magnitude, leaving academics in the dust. These days it's not clear that it makes sense for a university research group to work on systems problems that try to approximate industry: not only do we not have the resources, we simply have no idea what the real problems are at that scale. My friend and colleague Steve Gribble told me that after doing a sabbatical at Google, he decided that there's no way a university group could compete with what they're doing.

So what should academics be spending their time on? Too many academic systems researchers, I think, are doing "industry research in the small", with project horizons on the order of 2-3 years, not things that are radical departures from the state of the art. Is this the right approach? In a large department, it might be possible to approximate industry; David Patterson's PARLab at Berkeley is an example. This takes a tremendous amount of industry funding, though, and it's not scalable in the long run -- and there are not many people like Patterson who can pull that off. So what are the rest of us to do?

Ideally, academics should be pushing the envelope of computer science well beyond where industry is looking today. My research at Harvard has focused on radically new computing platforms -- mostly wireless sensor networks -- and our RoboBees project is pretty out there too. The downside is that industry isn't as interested in these problems, so it's harder to get funding from industry when you're not working on problems that are on their critical path. The other problem is that working on sci-fi it's more difficult to have impact on the real world, unless you're willing to wait for many years.

DARPA could be our savior. When I started my faculty job in 2003 I was told that getting money from DARPA would be no problem, but during the Tether years it mostly dried up. As a result nearly all of my research at Harvard has been funded by relatively small NSF grants plus various industry gifts. I am very hopeful that with Peter Lee heading up the new TCTO office that we'll see bolder initiatives from DARPA to bring back the good old days.

In the meantime, I guess I could find some bugs to squash in the Linux kernel...

Should Harvard's Intro CS class do away with grades?

We've been having some discussion amongst the CS faculty over the last few weeks about whether CS50, the intro computer science class at Harvard, should do away with letter grades and instead switch to SAT/UNSAT. Recently, the Harvard Crimson reported that CS50 is going to do away with letter grades, but this is not true -- the issue is still being debated by the faculty, and has yet to have any formal approval. (I don't know what led the Crimson to report it as though it were a fait accompli.) Given that my opinion seems to differ from most of the CS faculty, I thought I'd put my two cents forth here. Of course, this only represents my own thoughts on the matter, not the CS faculty as a whole (so if you are a Crimson reporter, don't go around reporting this as the final word).

For the record, I used to teach CS61, one of the two follow-on courses to CS50, so the output of CS50 directly feed into my course, and I have a vested interest in the intro course maintaining a certain standard. (My colleague Steve Chong is taking over CS61 next fall.)

David Malan, who teaches CS50, has done a fantastic job at increasing enrollments since taking it over 3 years ago -- I believe the enrollment has more than doubled (possibly tripled) under his watch. CS50 is going great guns and attracting students from all over the University, including many students who would have otherwise never taken a CS class. (Unlike a lot of places, CS is not required for Harvard undergrads as a whole, although it is required for a number of majors, including engineering, applied math, etc.) The course culminates in the CS50 fair where students show off their final projects. It is a great course and David has done amazing things to raise the profile of CS at Harvard.

So, right off the bat, I question the need to change the grading option for CS50, which potentially creates more problems than it solves.

David says that he has been wanting to get rid of letter grades entirely in CS50 for some time, and this year decided to raise the issue for discussion. David's claim is that there are still a lot of Harvard students who are intimidated by CS50 (despite the massive trend in increased enrollment) and that doing away with grades will make those students feel more comfortable trying out CS. While this may be true, I am not sure that we should be designing our foundational intro courses for those students. Harvard already has a "CS for non-majors" course, CS1, to fill that need.

A number of faculty believe that more women will be attracted to CS if they can take the course without worrying about their grade. I can't say whether that is true, but I find it somewhat implausible that what is standing between women and CS is just letter grades.

Even if you believe that doing away with letter grades will increase enrollment, consider the possible downsides. The first is that students who complete CS50 with a "SAT" grade won't have a good idea of their readiness to take more advanced CS courses. Grades do have informational value, and eliminating them makes it much harder to plan one's future course of study. The second is that the course will be less attractive to students who intend to major in CS or engineering, or who are simply used to getting all A's (as many incoming freshmen at Harvard are), so this change could actually decrease enrollment at the top end of the course. It would be fine if the hardcore students could just skip CS50 and take one of the later courses instead, but those courses are not currently designed to supplant CS50 for the better-prepared students. Also, skipping CS50 means missing out on the community experience that I think is important for student cohesion.

The third, and most severe, problem with this proposal is that it will make CS50 no longer suitable for the ABET-accredited engineering degree (which requires letter-graded courses) and the University-wide General Education requirement. So by trying to cast a wider net with CS50, the course no longer satisfies important requirements for certain students, so this approach seems self-defeating.

The compromise proposal currently on the table is to offer CS50 in two flavors: a letter-graded version (call it CS50g) and a SAT/UNSAT version (call it CS50su). It would be the same exact course, just with different grading options. This way students who need the letter grade can take CS50g, and everyone else can take CS50su. It seems clear that this will backfire in several ways. First, many students who take CS50su will later realize they needed a letter grade to satisfy a requirement down the line, but it will be too late. Second, it will create a class distinction between the CS50g and CS50su students. Students taking the course for a letter grade will demand much more detailed feedback on their assignments and more rigor in the grading process (since they will be competing for A's) whereas the course staff, used to dealing with mostly SAT/UNSAT students, will not feel the need to make such fine distinctions.

I have not looked into what other universities do and whether there is any agreement that doing away with letter grades for an intro CS course is a good idea. I know that MIT grades all freshmen pass/fail to alleviate stress associated with grades, which might make sense if done across the board, but it is unclear that changing one class is the right way to do this.

Personally, I'd rather not mess with CS50 right now. It's doing great things and I am concerned that doing away with letter grades would do more harm than good. Of course, many of my colleagues here at Harvard disagree with me, and they are encouraged to weigh in on the comments section below.

Update 4/20/10: The Crimson published a follow-on article this morning discussing the CS50 controversy.

Update #2 4/26/10: Given the many concerns raised, there was simply not enough time for the CS50 SAT/UNSAT proposal to be discussed by the faculty before the deadline for changes to the course catalog for next fall. As a result the proposal has been tabled for now; I expect it will be revisited next year.