The other side of "academic freedom"

My various blog posts about moving from academia to industry have prompted a number of conversations with PhD students who are considering academic careers. The most oft-cited reason for wanting a faculty job is "academic freedom," which is typically described as "being able to work on anything you want." This is a nice theory, but I think it's important to understand the realities, especially for pre-tenure, junior faculty.

I don't believe that most professors (even tenured ones) can genuinely work on "anything they want." In practice, as a professor you are constrained by at least four things:

• What you can get funding to do;
• What you can publish (good) papers about;
• What you can get students to help you with;
• What you can do better than anyone else in the field.

These are important limitations to consider, and I want to take them one by one.

Funding doesn't come easy. When I was a PhD student at Berkeley, I was fortunate to be a student of David Culler's, who had what seemed like an endless supply of funding from big DARPA and NSF grants, among others. When I went to start my faculty career, he (and many others) told me I would have "no problem" getting plenty of funding. This turned out not to be true. Shortly after I started my faculty job, DARPA all but shut down their programs in computer science, and NSF grants became heavily constrained (and much more competitive). Being a freshly-minted faculty member meant I was essentially a nobody, but that didn't mean that NSF review panels took pity on me -- apart from special programs like the CAREER award, you're competing with the most senior, established people in your field for every grant. To make matters worse, I didn't have a lot of senior colleagues in my area at Harvard to write proposals with, so I mostly had to go it alone.

Now, I will readily admit that I suck at writing grants, although according to my colleagues my hit rate for funding was about on par with other profs in my area. However, there were several projects that I simply could not do because I couldn't get funding for them. I tried for four years to get an NSF grant for our work on monitoring volcanoes with sensor networks -- which was arguably the thing I was most famous for as a professor. I failed. As a result we never did the large-scale, 100-node, multi-month study that we had hoped to do. It was a huge disappointment and taught me a valuable lesson that you can't work on something that you can't get funding for.

Who decides which problems are sexy (and therefore publishable)? I'll tell you: it's the 30-some-odd people who serve on the program committees of the top conferences in your area year after year. It is very rare for a faculty member to buck the trend of which topics are "hot" in their area, since they would run a significant risk of not being able to publish in the top venues. This can be absolutely disastrous for junior faculty who need a strong publication record to get tenure. I know of several faculty who were denied tenure specifically because they chose to work on problems outside of the mainstream, and were not able to publish enough top papers as a result. So, sure, they could work on "anything they wanted," but that ended up getting them fired.

Now, there are some folks (David Culler being one of them) who are able to essentially start new fields and get the community to go along with them. I argue that most professors are not able to do this, even tenured ones. Most people have to go where the funding and the publication venues are.

What can you get students to work on? I don't mean this in a kind of grad-students-won't-write-unit-tests kind of way (although that is also true). What I mean is how likely is it that you will find grad students in your field who have the requisite skills to undertake a particular research agenda? In my case, I would have killed for some students who really knew how to design circuit boards. Or students who had some deep understanding of compiler optimization -- but still wanted to work on (and publish) in the area of operating systems. A bunch of times I felt that the problems I could tackle were circumscribed by my students' (and my own) technical skills. This has nothing to do with the "quality" of the students; it's just the fact that PhD students (by definition) have to be hyper-specialized. This means that grad students in a given area tend to have a fairly narrow set of skills, which can be a limitation at times.

Can you differentiate your research? The final (and arguably most important) aspect of being successful as a faculty member is being able to solve new problems better than anyone else in your area. It is not usually enough to simply do a better job solving the same problem as someone else -- you need to have a new idea, a new spin, a new approach -- or work on a different problem. Hot areas tend to get overcrowded, making it difficult for individual faculty to differentiate themselves. For a while it felt like everyone was working on peer-to-peer networking. A bunch of "me too" research projects started up, most of which were forgettable. Being one of those "me too" researchers in a crowded area would be a very bad idea for a pre-tenure faculty member.

Do things get better after tenure? I didn't stick around long enough to find out, so I don't know. I definitely know some tenured faculty who are coasting and care a lot less about where and how much they publish, or who tend to dabble rather than take a more focused research agenda post-tenure. Certainly you cannot get fired if you are not publishing or bringing in the research dollars anymore, but to me this sounds like an unsatisfying career. Others -- like David Culler -- are able to embark on ambitious, paradigm-shifting projects (like NOW and TinyOS) without much regard to which way the winds are blowing. I think most tenured faculty would agree that they are subject to the same sets of pressures to work on fundable, publishable research as pre-tenure faculty, if they care about having impact.

Okay, but how much freedom do you have in industry? This is worth a separate post on its own, which I will write sometime soon. The short version is that it depends a lot on the kind of job you have and what kind of company you work for. My team at Google has a pretty broad mandate which gives us a fair bit of freedom. But unlike academia, we aren't limited by funding (apart from headcount, which is substantial); technical skills (we can hire people with the skills we need); or the somewhat unpredictable whims of a research community or NSF panel. So, yes, there are limitations, but I think they are no more severe, and a lot more rational, than what you often experience as an academic.

Running a software team at Google

I'm often asked what my job is like at Google since I left academia. I guess going from tenured professor to software engineer sounds like a big step down. Job titles aside, I'm much happier and more productive in my new role than I was in the 8 years at Harvard, though there are actually a lot of similarities between being a professor and running a software team.

LIKE A BOSS.

I lead a team at Google's Seattle office which is responsible for a range of projects in the mobile web performance area (for more background on my team's work see my earlier blog post on the topic). One of our projects is the recently-announced data compression proxy support in Chrome Mobile. We also work on the PageSpeed suite of technologies, specifically focusing on mobile web optimization, as well as a bunch of other cool stuff that I can't talk about just yet.

My official job title is just "software engineer," which is the most common (and coveted) role at Google. (I say "coveted" because engineers make most of the important decisions.) Unofficially, I'm what we call a "Tech Lead Manager," which means I am responsible both for the technical direction of the team as well as doing the people management stuff. (Some people use the alternate term "Über Tech Lead" but this has one too many umlauts for me.) A TLM is not a very common role at Google: most teams have separate people doing the TL and M jobs. I do both in part because, being based out of Seattle, it doesn't make sense to have my team report to a "regular" manager who would likely be in Mountain View. Besides I'm really happy to do both jobs and enjoy the variety.

There are four main aspects to my job: (1) Defining the technical agenda for the team and making sure we're successful; (2) Writing code of my own; (3) Acting as the main liaison between our team and other groups at Google, and (4) Doing the "people management" for the team in terms of hiring, performance reviews, promotion, and so forth.

Academics will immediately recognize the parallels with being a professor. In an academic research group, the professor defines the technical scope of the group as well as mentors and guides the graduate students. The big difference here is that I don't consider the folks on my team to be my "apprentices" as a professor would with graduate students. Indeed, most people on my team are much better software engineers than I am, and I lean on them heavily to do the really hard work of building solid, reliable software. My job is to shield the engineers on my team from distractions, and support them so they can be successful.

There are of course many differences with academic life. Unlike a professor, I don't have to constantly beg for funding to keep the projects going. I have very few distractions in terms of committees, travel, writing recommendation letters, pointless meetings. Of course, I also don't have to teach. (I loved teaching, but the amount of work it requires to do well is gargantuan.) Most importantly, my team's success is no longer defined through an arbitrary and often broken peer review process, which applies to pretty much everything that matters in the academic world. This is the best part. If we can execute well and deliver products that have impact, we win. It no longer comes down to making three grumpy program committee members happy with the font spacing in your paper submissions. But I digress.

I do spend about 50% of my time writing code. I really need to have a few solid hours each day hacking in order to stay sane. Since I don't have as many coding cycles (and service more interrupts) than other people on my team, I tend to take on the more mundane tasks such as writing MapReduce code to analyze service logs and generate reports on performance. I actually like this kind of work as it means dealing with a huge amount of data and slicing and dicing it in various interesting ways. I also don't need to show off my heroic coding skills in order to get promoted at this point, so I let the folks who are better hackers implement the sexy new features.

I do exert a lot of influence over the direction that our team's software takes, in terms of overall design and architecture. Largely this is because I have more experience thinking about systems design than some of the folks on my team, although it does mean that I need to defer to the people writing the actual code when there are hairy details with which I am unfamiliar. A big part of my job is setting priorities and making the call when we are forced to choose between several unappealing options to solve a particular problem. (It also means I am the one who takes the heat if I make the wrong decision.)

I reckon that the people management aspects of my job are pretty standard in industry: I do the periodic performance reviews for my direct reports, participate in compensation planning, work on hiring new people to the team (both internally and externally), and advocate for my team members when they go up for promotion. Of course I meet with each of my direct reports on a regular basis and help them with setting priorities, clearing obstacles, and career development.

The most varied part of my job is acting as the representative for our team and working with other teams at Google to make amazing things happen. My team is part of the larger Chrome project, but we have connections with many other teams from all over the world doing work across Google's technology stack. I am also frequently called into meetings to figure out how to coordinate my team's work with other things going on around the company. So it never gets boring. Fortunately we are pretty efficient at meetings (half an hour suffices for almost everything) and even with all of this, my meeting load is about half of what it was as an academic. (Besides, these meetings are almost always productive; compared to academic meetings where only about 10% of them have any tangible outcome.)

Despite the heavy load and lots of pokers in the fire, my work at Google is largely a 9-to-5 job. I rarely work on the evenings and weekends, unless there's something I'm really itching to do, and the volume of email I get drops to near-zero when it's outside of working hours. (Although I am on our team's pager rotation and recently spent a few hours in the middle of the night fixing a production bug.) This is a huge relief from the constant pressure to work, work, work that is endemic of professors. I also feel that I get much more done now, in less time, due to fewer distractions and being able to maintain a clear focus. The way I see it is this: If I'm being asked to do more than I can get done in a sane work week, we need to hire more people. Fortunately that is rarely a problem.

Disclaimer: Everything in this post is my personal opinion and does not represent the view of my employer.