Thursday, January 30, 2014

Getting a job at Google for PhD Students

I happen to sit on one of the hiring committees at Google, which looks at interview packets and makes a recommendation about whether we should extend an offer or not. So I've read a lot of packets, and have seen some of the ways in which applicants succeed or fail to get an offer. Ph.D. students, in particular, tend to get tripped up by the Google interview process, so I thought I'd offer some advice.

While I can't be certain, I imagine this same advice would apply to other companies which have a similar interview process that focuses on coding and algorithms.

(Disclaimer: This is all my personal opinion, and nothing I'm saying here is sanctioned or recommended by Google in any way. In fact, it might be totally wrong. Take it with a grain of salt.)

Google's interview process

Google uses a fairly typical industry interview process: Candidates go through one or two phone screens (or possibly an on-campus interview), and if they do well they are brought on campus for a full interview loop. Each interview is an hour and consists largely of problem solving and coding on the whiteboard. Sometimes a laptop is used.

This same process is used for all software engineering positions, regardless of level: undergrads, PhD students, and seasoned industry candidates all get the same style of interview. I had to go through this interview process upon joining Google as a professor. PhD-level candidates will generally spend one interview slot discussing their thesis work, and the questions may be more "researchy", but by and large it's the same for everyone.

The problem

Ph.D. students often tend to do worse on coding interviews than, say, bachelors' or masters' level candidates. Why? Doing a Ph.D. simply does not train you in professional software development skills, and that is (primarily) what a Google interview tests for. Undergrads, paradoxically, often do better because (a) they may have done internships at companies writing code, and (b) have practiced for this style of interview in the past.

There is a widespread belief that doing a Ph.D. somehow elevates you above the need to demonstrate fundamental algorithms and coding skills. Having a Ph.D. from Berkeley is awesome, but you still gotta be able to write good, clean code.

Also, part of the long process of doing a Ph.D. means you get hyper-specialized, so you get farther away from the "basics". Many of the Google interview questions touch on topics you probably first encountered (and mastered) as a sophomore or junior in college. I don't know about you, but I never dealt with binary search trees or graph connectivity problems directly during my Ph.D. and subsequent years as a faculty member. (Then again I'm just a systems guy, so the most sophisticated data structure I ever deal with is a hash table.)

Why the basics matter

Being at Google means writing production-quality software. We don't have "research labs" where people primarily build prototypes or write papers. I have written about Google's hybrid research model elsewhere -- also see this CACM article for more. While there are exceptions, by and large being at Google means being on a product team building and launching real products. That is even true of the more far-flung projects like self-driving cars and high-altitude Internet balloons. The quality and professionalism of the code you develop matters a great deal.

Doing a Ph.D. generally trains you for building research prototypes. There is a vast difference between this and writing production-quality code. First of all, it's not good enough for the code to make sense -- or be maintainable -- only by you or a small number of collaborators. Adherence to good design, avoiding overcomplicated code, conforming to style guidelines, etc. are all super important. In addition, you have to really concern yourself with robustness, scalability, testability, and performance. Corner cases that aren't interesting for publishing your next paper can't be overlooked.

Most of these skills can only be developed by working with a professional software development team. Research and class projects don't give you a chance to develop these skills. Undergrads gain these skills largely through internships. Unfortunately, most PhD students do internships at research labs, which may or may not provide much opportunity to build production-quality software.

Advice for grad students

If you're interviewing at Google, bone up on your basic algorithms and data structures. Go dust off that sophomore-level textbook and try to page it back in. I also highly recommend the book Cracking the Coding Interview, which gives the best description I have seen of Google-style interviews - it was written by a former Googler.

Don't go in with the attitude that you're above all this. Roll up your sleeves and show them what you've got. I know it may feel silly being asked what seem like basic CS questions, but if you're really as good as you think you are, you should knock them out of the park. (Keep in mind that the questions get harder the better you are doing, so no matter what, you will probably feel like crap at the end of the day.)

Every line of code you write on the whiteboard will get written up as part of the interview packet. Make it squeaky clean. Initialize variables. Use semicolons. Don't forget your constructors. Although writing sloppy pseudocode to get your meaning across might seem adequate (after all, we're all professionals here, aren't we?), attention to detail matters. Code in C++ or Java, which shows maturity. If you can only code in Python or bash shell, you're going to have trouble. If you make the slightest suggestion of wanting to code in Haskell or Lisp, the interviewer will push a hidden button which opens a trap door, dropping you into a bottomless pit. (Just kidding.)

Never, ever suggest you are a "C++ expert", either on your resume or in person. You are not.

Unfortunately, Google interviews tend to be a bit one-sided and you will not have as much opportunity to learn about Google (and what projects you might be working on) as you would like. If you do get an offer, you'll have more opportunities to come back and ask those questions. Google is notoriously secretive, so you have to trust me that there are plenty of cool things to work on.

Finally: Remember that the content of the interview has nothing to do with the kind of projects you would work on here. You're not going to get hired by Google and be asked to implement depth-first-search or reverse a linked list -- trust me on that. I'm pretty sure we have library routines for those already.

38 comments:

  1. "...so the most sophisticated data structure I ever deal with is a hash table."

    Hey, no putting down hash tables.

    ReplyDelete
  2. Michael, you might be pleased to know that one of my projects here involved designing a funky stack of counting Bloom filters. It was very clever, at least for me.

    ReplyDelete
    Replies
    1. That sounds like the main idea behind count-min sketch.

      Delete
    2. I am indeed pleased. (Although really, what CS professional hasn't used a stack of counting Bloom filters in one of their projects? That's why you shouldn't put down hash tables.)

      Delete
  3. One more thing I’d add is that candidates shouldn’t feel like their research has been overlooked. Usually only one interviewer out of the entire slate is assigned the job to evaluate their research contributions, and all others are explicitly asked to stick to software development questions, such that the final packet is well-rounded.

    As Ph.D. interviewers, we are super-curious about what you are up to, but we can’t spend the time to ask you about it, lest we harm your chances by spending too much time on it.

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. In addition to what Matt said: before your interview, find some programming questions and practice writing the answers on an actual whiteboard (or at least paper and pencil), as well as talking out loud while coding. It’s surprising how different this is than writing code on a computer, and on the day of your interview, you’ll be more used to it.

    ReplyDelete
  6. "Professional software development skills" never felt like what these sort of interviews are testing for. No professional software developer I've met sits around collaborating unidirectionally, reinventing convex-hull algorithms without looking them up, or writing code longhand on a whiteboard. An interview like that tests broader skills like critical thinking and communication, but it's not a direct test of software development practice.

    As for how to prepare for them: do old ACM programming contest problems. Time yourself. Write some of the solutions on a piece of paper before you type them in and run them. If nothing else, you'll discover where your knowledge gaps are, where you need to review sophomore CS textbooks.

    ReplyDelete
  7. Is the primary job for a non systems PhD at Google to write production code. Is that true for 'research scientist' job title as well.

    ReplyDelete
  8. Once, just once, I would like to meet somebody from google who doesn't think that everybody in the world wants a job at google.

    ReplyDelete
  9. @Muncha Research scientist interviews have fewer algorithms/coding questions and more questions about research subject matter and relates analytic topics (such as probability or convex optimization). Almost always there's a research talk as well. We do expect the research scientists we hire to contribute to production systems.

    ReplyDelete
  10. So I've programmed professionally in multiple companies, and I've never been asked to write clean, correct code in one hour without using any references (heck, I still refer to the BDB documentation while writing BDB code, and I actually know that code base prety well), so how does the interview scenario actually reflect writing real code for a real product?

    ReplyDelete
  11. "Never, ever suggest you are a "C++ expert", either on your resume or in person. You are not."

    Widely true !

    ReplyDelete
  12. Margo - The interview scenario has nothing to do with what the real job is like. But, as a predictor of success on the job, it seems to do well. In other words, we don't see many people who do very well on the interview who don't do well on the job. That is not to say that we don't have false negatives (people who we don't hire that we should), but I think Google is comfortable with that.

    ReplyDelete
  13. One thing that seems to be missing from this is the understanding that the interview is not just for evaluating a candidate, but also for recruiting them.

    Many top PhDs I know have gone through the google interview process and ended up getting offers, but because of the way the process is done they decide not to accept the google offer and go elsewhere (and also discourage other smart PhDs from considering applying to google). Designing an interview process for PhDs that focuses on coding puzzles doesn't seem like a smart way to either evaluate the candidates or make them want to work for you.

    ReplyDelete
    Replies
    1. I actually agree 100% with this. One of my concerns about Google's interview process is that it doesn't give PhD candidates a good idea of what they can work on. I was kind of turned off by my interview at Google. Fortunately I took the job and got to see the other side.

      Delete
  14. I don't quite understand why one will get into trouble if his/her language of choice is Python.

    ReplyDelete
  15. Great write-up, Matt. Here were my interview notes from prepping for Google software engineering interviews at the end of Summer 2011, which touches on similar points:
    http://pgbovine.net/programming-interview-tips.htm

    The high-order bit is the same as yours: it's a coding interview, not a research job talk.

    ReplyDelete
  16. No place for non Ph.D.s ?

    ReplyDelete
  17. Anonymous, it might be problematic if Python was really your only language, because if your software is going to be running on hundreds or thousands of machines, it's very much worth the time to write in a more efficient language. At scale, machine time starts to cost more than engineering time.

    ReplyDelete
  18. @anonymous "Many top PhDs I know..." I extensively involved in PhD hiring. We do lose a few candidates who would prefer a more pure research-y role than what they perceive Google will provide, but those perceptions are partly the result of an idealized view of "research" vs "engineering." We do interview PhDs for research scientist roles in some areas (such as machine learning, computer vision, natural language, auction algorithms, HCI, data management, ...) and the research scientist interview has different balance of subject matter and algorithms/coding (as I said above), but the central point that Matt is making still stands. I like to state it as "the language of successful communication within Google is good code, not slides, proofs, or research prototypes". There are exceptions (think of the optics, electronics, and bioengineering expertise in the recently announced contact lens) where coding is much less relevant, but even in projects like that, the ability to develop effective measurement, simulation and automation software is a great asset.

    ReplyDelete
  19. You said that the important software engineering skills can only be developed by working with a professional software development team. In my case, I only did one internship at a web development company where I worked on simple front end code (JavaScript mostly ). I know the basics, and I'm a decent Java programmer. However, I feel that I suck in software engineering. The only real software engineering experience I have is from this software engineering course that I took last year. Aside from internships at companies that have good software development teams, can working on problems on programming competition sites and completing open source projects prepare someone fresh out of undergrad for these types of interviews? Also, can one acquire the important software engineering skills that you mentioned above by working on good open source projects?

    ReplyDelete
    Replies
    1. Suck at software engineering*

      Delete
    2. I know how you feel. Although I do it every day I still feel that I kind of suck at software engineering :-)

      It is possible to game the interview somewhat by mastering "programming competition" style questions. The Google interview doesn't directly test how you work with a team, but it does test things like communication and design skills. Open source projects are certainly a good way to gain some of these skills, although of course it depends on the project.

      Delete
  20. I've interviewed with Google a number of times. Never really felt like they were respectful or even reviewed my resume. The phone screen is easy they asked questions from a database of simple questions to eliminate dummies. They also ask you not to post the questions online. I've specifically done that from my notes every time on glass door.

    My onsites were interesting but inpersonal lunch 1 hour sessions and I think the random pool of people is a real disservice because its nice to know if you will work with jerks, changing careers is a big deal and its the main reason I rejected my offer from Google other then a big miss on my market value. Didn't even feel like countering.

    By comparison My company does not stress people out in interviews and the people who are interviewing the candidates are really going to be coworkers. We've got over 100,000 employees and didn't have to resort to testing people like caddle. Brand impression forming discussions are important even if you don't pick someone.

    I also think all too often companies get caught up in thinking their brand is a form of compensation for working at a company its not. It's only stupid people who make such decisions about employment like that. In my tech career of interviews I've only seen two companies more full of shit then Google employees about their brands: Nike and Amazon.

    Working for Google isn't all its cracked I up to be. Its just another software company with an early position in its life cycle. Eventually the business will level off and times will be average for employees and investors. Following hype for careers sake gets you a fragmented resume and potentially a few rental houses from lots of moves.

    I only wish college grads were smarter then falling for employment marketing pitches like those coming out of tech companies rather then a more objective approach.

    ReplyDelete
    Replies
    1. You are right that Google's interview process is highly selective and as a result can sometimes come across as arrogant or pretentious. I don't like this, either, but given how many people apply for jobs here there is a very high bar. It will certainly turn some people off. Still, I find it difficult to believe you when you say "Working for Google isn't all it's cracked up to be" when you haven't worked at Google.

      Delete
    2. I don't think "highly selective" is a valid excuse.

      One area where the number of high-quality candidates far outnumbers available positions is top-tier management consulting (please don't flame me for having the gall to compare techies with flaky business people). Their interview process is also very challenging (admittedly for different reasons), but their interviewers are specifically trained to be very courteous, professional, even humble. Similar to what has been mentioned above, they use interviews as an opportunity to further their brand, because it is entirely possible that they could be pissing off a future ceo.

      For instance, I was rejected by BCG, and while it was a huge blow at the time, I feel was treated very nicely, and don't hold that against my interviewers at all.

      Compare that to a tech interview I had at a major tech company (where I was successful), where the interviewer asked me to square a large number in my head, and I blurted it out immediately, simply because I had memorized it from another context. The guy was so impressed that the rest of the 40 minutes or so were basically moot....now, what did this interview actually accomplish?

      Delete
  21. Hey Matt, good article there... could you write one on the structure of the resume required by the recruiters... i have so far gotten two chances, but after the initial phone call, the recruiters seem to lose interest and i suspect it could be with my resume(though i feel it's abit nice). cheers.

    ReplyDelete
    Replies
    1. Good question. What style of resume have you been using? If you're coming from a PhD background I would think a full CV (which might be 2-3 pages for a PhD student with multiple publications) would be appropriate. Mostly however you want to emphasize your *building* skills, not just research/academic background.

      Delete
  22. From an outside view, what makes a company great is its ability to produce innovative products. From an inside view, a great company is the one that cultivates and produces great inventors who will create the next innovative products.
    Microsfot is probably the greatest example of a company struggling to produce its next generation of innovators. A flagship tech company that can attract and hire great minds from any great university in the country is having a hard time to find a replacement for Ballmer. On the other hand, Apple not only has produced great products, but produced great people that have created companies that are at the forefront of innovation. Like the one Google just bought: Nest.
    Google is making a humongous amount of money selling advertising beside internet search results. What happens if Google goes out of business tomorrow? I would say not much other than a lot of people losing their jobs. I might be wrong but I don’t think hiring that next clever programmer will make much of a difference.

    ReplyDelete
    Replies
    1. I'm not exactly sure what your point is. Apple, Microsoft, and Google are all highly innovative, though they operate with vastly different business models.

      Delete
  23. Is there any role suitable for PhD doesn't emphasize on coding? Quantitative analyst?

    Are all positions must go through hiring committee? Even sales, marketing?

    ReplyDelete
    Replies
    1. There are several non-coding-oriented roles that a CS PhD might be interested in. I know several CS PhDs who are Product Managers at Google, which is a very diverse and interesting job - shaping products, working closely with engineering teams on defining requirements and navigating the launch process. UX Researcher also comes to mind, although there are not very many of these. I don't know about the Quantitative Analyst role so much, though given that it deals with data it might involve some coding.

      I don't know how sales and marketing people are hired but I'd be surprised if there was not some kind of committee process for those roles.

      Delete
    2. So-called "Data Science" jobs are pretty common at tech companies these days, including at Google, I think.

      Delete
  24. Hi Matt,
    Question --- you have only mentioned C++ and Java, but not C; how is a candidate using C viewed? Even though I have used C++, I do not feel comfortable coding at a whiteboard with C++ without having references at hand. For people like me, would you recommend using C during interview, or crank on C++ before hand and use C++? Thanks.

    ReplyDelete
  25. Hi Matt,

    Thank you for sharing your experience at google. I used to read your papers when I was doing research in field of sensor network.

    Just out of curiosity, what was the questions like when you interviewed with Google? I am just wondering how Google interview established researchers like yourself. there are number of well known former professors working at Google.

    Thanks,
    H

    Thanks
    H

    ReplyDelete
  26. Its been a good blog and very nice ideas to get in to the google.

    ReplyDelete
  27. Dear Matt and Fernando,

    First off, thank you so much for sharing this. Very interesting read.

    Do you have any suggestion for research scientist candidates at Google?

    Compared to SDE candidates, could you please sketch out how a typical onsite interview looks like and what aspects will the interview feedback be focused on?

    Apart from typical coding interviews, how is the research talk evaluated (by whom)? During the talk, which aspect should we emphasize (e.g., practical concerns? sexy demo? research-y novelty?) What is the general structure for an interview on "analytic topics" (as Fernando mentioned) and how would you suggest us to prepare for it?

    Thanks in advance!

    ReplyDelete

Startup Life: Three Months In

I've posted a story to Medium on what it's been like to work at a startup, after years at Google. Check it out here.