Startup vs. Big Company: What's "Freedom"?

I was talking with a talented young PhD student today about his career ambitions, and the conversation turned to whether it would be better to do a startup after finishing school rather than joining a big company, like Google, to get some "real world" experience first. I asked him what was so appealing about doing a startup, and he said that it would mean having tremendous freedom to choose what to work on and how you pursue problems. That is true, in a sense, but I question how much "freedom" you really have when starting a new company. Doing a startup is also highly constraining, there are real advantages to being at a larger company where you have more resources and a broader set of problems you can work on.

(Caveat emptor: Note that the last time I tried to talk someone out of doing a startup, it was Mark Zuckerberg, and we all know how that turned out -- so perhaps this post should be taken with a grain of salt.)

It seems to me that many startups these days are working on low-hanging-fruit problems: Things that a few guys in a garage can put together with EC2, Ruby on Rails, and XCode. Yes, you can build amazing products this way, but you're not necessarily doing rocket science. There are limits to what one can build with a small team and limited resources, and this means that most startups are constrained in the set of problems they can reasonably tackle.

Also, startups tend to focus on problems that one can build a standalone business around, something that can be monetized or build value in some direct way, in order to make investors happy. For example, it would be difficult to do a startup around a new programming language, since it's not clear how you sell it, although it might form a component of a larger product.

My point is that not all interesting problems are good startup fodder. This is where larger companies come in. One of the reasons I personally like working at Google is because I can work on problems at a scale that most startups would never achieve. Running jobs on terabytes of data on many thousands of cores is routine. For me, this is where the really interesting problems lie: Not in designing another sepia-tone photo filter app, but in solving fundamental problems of computer science that only emerge at the scale and complexity that I can work on here.

Being at a larger company means I am "free" to work on unsexy problems -- problems a venture capitalist wouldn't touch with a ten foot pole. Network protocol optimization, performance measurement, and infrastructure building may not be everyone's idea of fun, but I really enjoy doing this kind of core systems work that can have huge impact over the long run.

This is not to say that all problems at Google are unsexy. I just tend to gravitate towards infrastructure and networking since that's my background. And damned if I can program JavaScript.

The other side of being at a large company is that you have the freedom to fail. Money is not about to run out in a few weeks, and you don't have to make deals you wish you hadn't in order to keep the cash flowing. You can take your time to try different things, get it wrong, and make big bets that don't pan out. It's unlikely you'll lose your job for doing so, and there are always more cool problems to tackle just around the corner.

On the flip side, doing a startup while you are young and untethered might be exactly the right career move. In some ways I wish I had not sat out the dot-com boom doing a PhD, but taken a risk to join PurpleYogi or one of the other dozen or so startups (including Google) that tried to recruit me back when I was 22, unmarried, childless, and already used to living in a small apartment and eating ramen noodles for dinner. Who knows, maybe I would have learned JavaScript after all.

NCSSM and how it saved my life

I just got back from my 20th high school reunion and was reflecting on how much impact my high school had on my life and my career. You see, I was lucky enough to go to the North Carolina School of Science and Math, also known as NCSSM, or as we lovingly called it back then, "S&M". NCSSM is a public high school in Durham -- not far from Duke -- for juniors and seniors. Around 680 students live on campus, in dorms -- a lot like college, but with curfews, and students aren't allowed to have cars. To get in, you take the SAT and some other tests in 10th grade, and if you're accepted, it's completely free of charge -- no tuition, no housing fees, even the food is paid for. (The food was not, by the way, one of the highlights of the place.)

NCSSM is an utterly amazing place. Everyone I know who has been there has had their lives deeply touched by the experience. Although it has a well-deserved reputation as a school for, well, nerds, it is also full of some of the most interesting and creative people I have ever met. Twenty years later, it is amazing to see what my classmates are doing today: Doing high-end CGI for Hollywood movies; numerous professors and research scientists in areas as diverse as political science, planetology, integrated science and technology, and sociology; working for the Department of Health and Human Services while doing regular club and radio DJ gigs; even serving as an Episcopalian minister. Many of my classmates are not doing "science" or "math" in the conventional sense.

Prior to NCSSM, I lived in a small town called Wilson, about an hour east of Raleigh. (If you're from North Carolina, the correct pronunciation is "WILT-sun".) It would be understatement to say that I did not fit in in Wilson, which is surrounded by a rural tobacco-growing community. There were not a lot of people there like me, and my horizons were severely limited. The main pastime of high-school kids in Wilson those days was driving in circles around the mall parking lot. There were a few great teachers in the schools, but I really needed more than Wilson had to offer.

Coming to NCSSM I found a community of people like me -- a school full of outcasts, geeks, free spirits, lost souls. Not everyone was socially maladjusted, of course, but there were plenty of people there all pushing the boundaries of their humble (often rural and low-middle income) backgrounds. The faculty at NCSSM were (and still are) stellar. I could take Russian, quantum physics, photography, t'ai chi. It was like opening a vista on vast opportunities that I had scant awareness of when I was in Wilson, and I mean it seriously when I say that NCSSM saved my life: there's no way I'd be where I am today without that experience.

For one thing, my exposure to computing was greatly expanded at NCSSM. Along with some other students, I ran the school's VAX minicomputer which powered the school's "intranet" (although it was really a bunch of VT-100 terminals scattered around campus, tied to the single computer). The students and faculty all had primitive email and chat accounts on the VAX -- this was the days before the Internet was widespread. We also had an IBM RT, a high end (at the time) UNIX workstation with 3D (!!) graphics support. A few of us managed to get this machine on the Internet, over a slow ISDN connection, so we could use FTP and email, and the IBM RT was my first UNIX "root" account. At one point, I dusted off an old, unused Data General mainframe sitting in the corner, figured out how to boot it from tape, and set up a series of terminals in the adjacent computer lab, giving any student who asked for it an account, with the provisio that they have no password -- a tribute to RMS' similar practice at the MIT AI Lab. I got to do an internship at nearby Data General, and a volunteer from NC State taught a C programming class after hours. It was incredible.

Outside of conventional academics, NCSSM has tremendous resources for exploring music and the arts. It has the most unbelievable art studio, where we would spend countless hours: in the darkroom, screen printing, making stained glass, paintings, sculptures, ceramics. My major creative outlet there was the electronic music studio. Back then it was a somewhat modest affair: A couple of synthesizers, a drum machine, 8-track reel-to-reel, effects units, MIDI sequencer -- more than enough for me to produce and record two full-length albums (and no, I will not be posting MP3s). I spent hours in that studio every weekend, all thanks to the dear late Ray Church, the music teacher who let me and others run roughshod over "his" gear. The best aspect of this was that the studios were open all the time, and the students were trusted, and encouraged, to make it their own space and use the resources to explore their own projects.

It's important to keep in mind that NCSSM is a public school. It's paid for by the taxpayers of North Carolina, and can only exist because of a state legislature, and state university system, that recognizes the importance of having a high school like this. I can't imagine what my life would be like had I not had the opportunity to go there, and I know a lot of my classmates agree.

In Defense of the Scientific Paper

http://www.flickr.com/photos/openeye/5428831/

Since leaving academia, I still find the time to serve on scientific program committees (recently NSDI, MobiSys, and SOCC) and have plenty of opportunity to read both good and bad scientific papers in various states of preparation. And although I am not required to publish papers in my current job, I certainly hope to do so -- a lot of the work we are doing at Google is imminently publishable -- it's just a matter of finding the time to sit down and write them!

Although I've blogged about how the scientific publication process needs fixing, I still feel that the process of writing a scientific paper is a hugely rewarding experience. Arguably, the primary value of scientific papers isn't in reading them, but writing them. You learn so much in the process.

Writing a paper sharpens your mental focus like nothing else. Like Japanese bonsai art or building a ship in a bottle, paper writing forces you to obsess over every meticulous detail -- word choice, overall tone, readability of graphs -- and of course more mundane details like font size and line spacing. This microscopic attention to every aspect of your work brings out a wonderful, if somewhat exhausting, intellectual rapture. I have never thought so clearly about a piece of research than when I'm in the throes of putting together a paper against a deadline.

You start with nothing, a blank editor window and some LaTeX boilerplate, some half-baked ideas, a few axes to grind and a tremendous apprehension at how much your life is going to suck between now and the deadline. You throw in all of the raw ingredients, the rough ideas, the broken implementation, the confusing data, the missing citations. Over a period of days or weeks you grind it and refine it and throw it out and start over and eventually hone the paper to a razor-sharp, articulate, polished fourteen pages of scientific beauty, and then just hope like hell that you didn't screw up the margins or forget to cite some important piece of related work.

I used to think that writing a paper was something you did after the research was over, but now I realize you should sit down to write the paper as early as possible -- sometimes before even starting the "research work" itself. On a few occasions, it wasn't until I started writing a paper that I knew what the hell the research project was really about. Case in point: Our SenSys 2009 paper on the Mercury wearable sensor platform came out of a project that had been running for nearly two years without a clear set of goals or any real insight into what the interesting research problems were. We had built a prototype and had some stuff working, but we didn't know what was publishable about it, and most of the problems we had to solve seemed mundane.


In a last-ditch measure to revive the project, I got the students together and said, fuck it, let's write a SenSys paper on this. As we started piecing together the story that we wanted to tell in the paper, we realized that none of our work to that point tackled the most important problem: how to ensure that the sensors produced good, and useful, data when there was a hard limit on battery lifetime. With the deadline just weeks away, the students pulled together and reimplemented the system from scratch and cranked out a ton of new measurements. The process of writing the paper resulted in a flood of new ideas, many of which bled over into my other projects, ultimately resulting in a half dozen papers and three PhD theses. It was awesome.


And even if a paper does not get accepted, crystallizing the ideas through the process of putting together the submission can be really energizing. I never assumed any paper I wrote would actually get accepted, so submitting the paper was often the start of a new line of work, riding on that clarity of thought that would emerge post-deadline (and a much-needed break of course).

Google's Hybrid Approach to Research

This month's Communications of the ACM features an article on Google's Hybrid Approach to Research by Alfred Spector, Peter Norvig, and Slav Petrov. Since this is a topic I've blogged about here before, I thought I'd provide a quick pointer to the article:

http://cacm.acm.org/magazines/2012/7/151226-googles-hybrid-approach-to-research/fulltext

Overall I think the article does a nice job of summarizing Google's approach. The key takeaway is that Google doesn't separate its research and engineering activities: most "research" at Google happens during the day-to-day work of building products.

The benefit of this model is that it's easy to have real world impact, and the pace of innovation is fairly rapid, meaning research results get translated into products quickly. The possible downside is that you don't always get a chance to fork off  long-term (multi-year) projects that will take a long time to translate into a product. However, there are exceptions to this rule -- things like Google Glass, for example -- and plenty of things I can't talk about publicly. It is true that Google tends not to do "pure academic" research just for the purpose of publishing papers. We could have a healthy debate about whether this is good or bad, but I'll leave that for the comments...

Startup University

The academic research process is incredibly inefficient when it comes to producing real products that shape the world. It can take decades for a good research idea to turn into a product - and of course most research never reaches this phase. However, I don't think it has to be that way: We could greatly accelerate the research-to-product pipeline if we could fix the academic value system and funding model.

Here's the problem: Some of the smartest people in the world have spent their entire careers building throwaway prototypes. I sure never built anything real until I moved to Google, after nearly ten years of college and grad school, and seven years as a faculty member. And by "real," I don't just mean a prototype that we developed for a couple of years and then threw away as soon as the papers got published. In effect, I "wasted" millions of dollars in funding, and countless man-years of development effort by my students and lab staff -- apart from a bunch of papers, nothing of practical value came out of my entire academic research career. (Maybe I'm being a little hard on myself, but let's take this as a given for sake of argument.) And I don't think my lack of real-world impact is at all unusual in a university setting.

What would the world be like if all of this hard work had actually translated into real, shipping products that people could use? How could we change the structure of academic research to close the gap between playing in the sandbox and making things real?

The plight of the academic is that there is often no direct way to translate ideas into reality -- you don't have the resources to do it at the university, and the academic process forces you to bounce between ideas every few years, rather than sticking it out to turn something into a product. In theory, academics are supposed to be patenting their ideas, and companies are supposed to come along and license the patents and turn them into real products. However, I am not aware of a single project from a computer science department that ever been commercialized through this route. This approach is more commonplace in fields like biotech, but in computer science it is rarely done.

A far more common (and successful) approach is for academics to spin out their own startups. However, this involves a high degree of risk (potentially career-ending for pre-tenure faculty), and many universities do not structure their sabbatical and leave policies to make this easy to do. Most universities also make starting a company painfully difficult when it comes to questions of IP ownership, licensing, and forcing the academic's research to be dissociated with their commercial activities. As a result, you get a bunch of super smart academics who play it safe and stay within their tenured faculty jobs, subsisting on grants and rarely commercializing their work. This means that a lot of great ideas never get beyond the prototype phase.

What I'd like to see is a university with a startup incubator attached to it, taking all of the best ideas and turning them into companies, with a large chunk of the money from successful companies feeding back into the university to fund the next round of great ideas. This could be a perpetual motion machine to drive research. Some universities have experimented with an incubator model, but I'm not aware of any cases where this resulted in a string of successful startups that funded the next round of research projects at that university.

Typically, when a startup spins off, the university gets a tiny slice of the pie, and the venture capitalists -- who fill the much-needed funding gap -- reap most of the benefits. But why not close the air gap between the research lab and the startup? Allow the faculty to stay involved in their offspring companies while keeping their research day job? Leverage the tremendous resources of a university to streamline the commercialization process -- e.g., use of space, equipment, IT infrastructure, etc.? Allow students to work at the startups for course credit or work-study without having to quit school? Maintain a regular staff of "serial entrepreneurs" who help get new startups off the ground? Connect the course curriculum to the fledgling startups, rather than teaching based on artificial problems? One might joke that some universities, like Stanford, effectively already operate in this way, but this is the exception rather than the rule.

It seems to me that bringing together the university model with the startup incubator would be a great benefit both for spinning out products and doing better research.

Do you need a PhD?

Since I decamped from the academic world to industry, I am often asked (usually by first or second year graduate students) whether it's "worth it" to get a PhD in Computer Science if you're not planning a research career. After all, you certainly don't need a PhD to get a job at a place like Google (though it helps). Hell, many successful companies (Microsoft and Facebook among them) have been founded by people who never got their undergraduate degrees, let alone a PhD. So why go through the 5-to-10 year, grueling and painful process of getting a PhD when you can just get a job straight out of college (degree or not) and get on with your life, making the big bucks and working on stuff that matters?

Doing a PhD is certainly not for everybody, and I do not recommend it for most people. However, I am really glad I got my PhD rather than just getting a job after finishing my Bachelor's. The number one reason is that I learned a hell of a lot doing the PhD, and most of the things I learned I would never get exposed to in a typical software engineering job. The process of doing a PhD trains you to do research: to read research papers, to run experiments, to write papers, to give talks. It also teaches you how to figure out what problem needs to be solved. You gain a very sophisticated technical background doing the PhD, and having your work subject to the intense scrutiny of the academic peer-review process -- not to mention your thesis committee.

I think of the PhD a little like the Grand Tour, a tradition in the 16th and 17th centuries where youths would travel around Europe, getting a rich exposure to high society in France, Italy, and Germany, learning about art, architecture, language, literature, fencing, riding -- all of the essential liberal arts that a gentleman was expected to have experience with to be an influential member of society. Doing a PhD is similar: You get an intense exposure to every subfield of Computer Science, and have to become the leading world's expert in the area of your dissertation work. The top PhD programs set an incredibly high bar: a lot of coursework, teaching experience, qualifying exams, a thesis defense, and of course making a groundbreaking research contribution in your area. Having to go through this process gives you a tremendous amount of technical breadth and depth.

I do think that doing a PhD is useful for software engineers, especially those that are inclined to be technical leaders. There are many things you can only learn "on the job," but doing a PhD, and having to build your own compiler, or design a new operating system, or prove a complex distributed algorithm from scratch is going to give you a much deeper understanding of complex Computer Science topics than following coding examples on StackOverflow.

Some important stuff I learned doing a PhD:

How to read and critique research papers. As a grad student (and a prof) you have to read thousands of research papers, extract their main ideas, critique the methods and presentation, and synthesize their contributions with your own research. As a result you are exposed to a wide range of CS topics, approaches for solving problems, sophisticated algorithms, and system designs. This is not just about gaining the knowledge in those papers (which is pretty important), but also about becoming conversant in the scientific literature.

How to write papers and give talks. Being fluent in technical communications is a really important skill for engineers. I've noticed a big gap between the software engineers I've worked with who have PhDs and those who don't in this regard. PhD-trained folks tend to give clear, well-organized talks and know how to write up their work and visualize the result of experiments. As a result they can be much more influential.

How to run experiments and interpret the results: I can't overstate how important this is. A systems-oriented PhD requires that you run a zillion measurements and present the results in a way that is both bullet-proof to peer-review criticism (in order to publish) and visually compelling. Every aspect of your methodology will be critiqued (by your advisor, your co-authors, your paper reviewers) and you will quickly learn how to run the right experiments, and do it right.

How to figure out what problem to work on: This is probably the most important aspect of PhD training. Doing a PhD will force you to cast away from shore and explore the boundary of human knowledge. (Matt Might's cartoon on this is a great visualization of this.) I think that at least 80% of making a scientific contribution is figuring out what problem to tackle: a problem that is at once interesting, open, and going to have impact if you solve it. There are lots of open problems that the research community is not interested in (c.f., writing an operating system kernel in Haskell). There are many interesting problems that have been solved over and over and over (c.f., filesystem block layout optimization; wireless multihop routing). There's a real trick to picking good problems, and developing a taste for it is a key skill if you want to become a technical leader.

So I think it's worth having a PhD, especially if you want to work on the hardest and most interesting problems. This is true whether you want a career in academia, a research lab, or a more traditional engineering role. But as my PhD advisor was fond of saying, "doing a PhD costs you a house." (In terms of the lost salary during the PhD years - these days it's probably more like several houses.)

My love affair with code reviews

One of the most life-altering events in my move from academia to industry was the discovery of code reviews. This is pretty standard fare for developers in the "real world", but I have never heard of an academic research group using them, and had never done code reviews myself before joining Google.

In short: Code reviews are awesome. Everyone should use them. Heck, my dog should use them. You should too.

For those of you not in the academic research community, you have to understand that academics are terrible programmers. (I count myself among this group.) Academics write sloppy code, with no unit tests, no style guidelines, and no documentation. Code is slapped together by grad students, generally under pressure of a paper deadline, mainly to get some graphs to look pretty without regard for whether anyone is ever going to run the code ever again. Before I came to Google, that was what "programming" meant to me: kind of a necessary side effect of doing research, but the result was hardly anything I would be proud to show my mother. (Or my dog, for that matter.) Oh, sure, I released some open source code as an academic, but now I shudder to think of anyone at a place like Google or Microsoft or Facebook actually reading that code (please don't, I'm begging you).

Then I came to Google. Lesson #1: You don't check anything in until it has been reviewed by someone else. This took some getting used to. Even an innocent four-line change to some "throw away" Python script is subject to scrutiny. And of course, most of the people reviewing my code were young enough to be my students -- having considered myself to be an "expert programmer" (ha!), it is a humbling experience for a 23-year-old one year out of college to show you how to take your 40 lines of crap and turn them into one beautiful, tight function -- and how to generalize it and make it testable and document the damn thing for chrissakes.

So there's a bunch of reasons to love code reviews:

Maintain standards. This is pretty obvious but matters tremendously. The way I think of it, imagine you get hit by a truck one day, and 100 years from now somebody who has never heard of your code gets paged at 3 a.m. because something you wrote was suddenly raising exceptions. Not only does your code have to work, but it also needs to make sense. Code reviews force you to write code that fits together, that adheres to the style guide, that is testable.

Catch bugs before you check in. God, I can't count the number of times someone has pointed out an obvious (or extremely subtle) bug in my code during the code review process. Having another pair of eyes (or often several pairs of eyes) looking at your code is the best way to catch flaws early.

Learn from your peers. I have learned more programming techniques and tricks from doing code reviews than I ever did reading O'Reilly books or even other people's code. A couple of guys on my team are friggin' coding ninjas and suggest all kinds of ways of improving my clunky excuse for software. You learn better design patterns, better approaches for testing, better algorithms by getting direct feedback on your code from other developers.

Stay on top of what's going on. Doing code reviews for other people is the best way to understand what's happening in complex codebase. You get exposed to a lot of different code, different approaches for solving problems, and can chart the evolution of the software over time -- a very different experience than just reading the final product.

I think academic research groups would gain a lot by using code reviews, and of course the things that go with them: good coding practices, a consistent style guide, insistence on unit tests. I'll admit that code quality matters less in a research setting, but it is probably worth the investment to use some kind of process.


The thing to keep in mind is that there is a social aspect to code reviews as well. At Google, you need an LGTM from another developer before you're allowed to submit a patch. It also takes a lot of time to do a good code review, so it's standard practice to break large changes into smaller, more review-friendly pieces. And of course the expectation is you've done your due diligence by testing your code thoroughly before sending it for review.

Don't code reviews slow you down? Somewhat. But if you think of code development as a pipeline, with multiple code reviews in the flight at a time you can still sustain a high issue rate, even if each individual patch has higher latency. Generally developers all understand that being a hardass on you during the review process will come back to bite them some day -- and they understand the tradeoff between the need to move quickly and the need to do things right. I think code reviews can also serve to build stronger teams, since everyone is responsible for doing reviews and ensuring the quality of the shared codebase. So if done right, it's worth it.

Okay, Matt. I'm convinced. How can I too join the code review bandwagon? Glad you asked. The tool we use internally at Google was developed by none other than Guido van Rossum, who has graciously released a similar system called Rietveld as open source. Basically, you install Rietveld on AppEngine, and each developer uses a little Python script to upload patches for review. Reviews are done on the website, and when the review is complete, the developer can submit the patch. Rietveld doesn't care which source control system you use, or where the repository is located -- it just deals with patches. It's pretty slick and I've used it for a couple of projects with success.

Another popular approach is to use GitHub's "pull request" and commenting platform as a code review mechanism. Individual developers clone a master repository, and submit pull requests to the owner of that repository for inclusion. GitHub has a nice commenting system allowing for code reviews to be used with pull requests.

I was floored the other day when I met an engineer from a fairly well-known Internet site who said they didn't use code reviews internally -- and complained about how messy the code was and how poorly designed some pieces were. No kidding! Code reviews aren't the ultimate solution to a broken design process, but they are an incredibly useful tool.