Wednesday, January 6, 2016

Academics, we need to talk.

Although I made the move to industry a bit more than five years ago, I still serve on program committees and review articles for journals and the like. So it's painful for me to see some of my academic colleagues totally botch it when it comes to doing industry-relevant research. Profs, grad students: we need to talk.

(Standard disclaimer: This is my personal blog and the opinions expressed here are mine alone, and most certainly not that of my employer.)

Of course, many academics do a great job of visionary, out-of-the-box, push-the-envelope research that inspires and drives work in industry. I'm talking about stuff like Shwetak Patel's increasingly insane ways of inferring activities from random signals in the environment; Dina Katabi's rethinking of wireless protocols; and pretty much anything that David Patterson has ever done. Those folks (and many others) are doing fine. Keep it up.

But the vast majority of papers and proposals I read are, well, crap. Mostly I'm involved in mobile, wireless, and systems, and in all three of these subfields I see plenty of academic work that tries to solve problems relevant to industry, but often gets it badly wrong. I don't quite know why this happens, but I have some ideas of how we might be able to fix it.

Now, I don't think all academic research has to be relevant to industry. In some sense, the best research (albeit the riskiest and often hardest to fund) is stuff going way beyond where industry is focused today. Many academics kind of kid themselves about how forward-thinking their work is, though. Working on biomolecular computation? That's far out. Working on building a faster version of MapReduce? Not so much. I'd argue most academics work on the latter kind of problem -- and that's fine! -- but don't pretend you're immune from industry relevance just because you're in a university.

My first piece of advice: do a sabbatical or internship in industry. It's probably the best way to get exposed to the real world problems -- and the scale and complexity -- that you want to have as inspiration for your work. It drives me insane to see papers that claim that some problem is "unsolved" when most of the industry players have already solved it, but they didn't happen to write an NSDI or SIGCOMM paper about it. Learn about what's going on in the real world and what problems are both being actively worked on, and what will be important a few years down the line. If you buy me a beer I'll spend a lot of time telling you what's hard at Google and what we need help with. Hint: It ain't yet another multihop routing protocol.

And by "industry", I do not mean "an academic research lab that happens to reside at a company." You can spend time there and learn a lot, but it isn't getting quite the level of product exposure I have in mind.

For this to work, you have to work on a real product team. I know this sounds like a waste of time because you probably won't get a paper out of it, but it can be an extremely eye-opening experience. Not only do you start to understand the constraints that real products have -- like, oh, it actually has to work -- but you also pick up on good engineering practices: bug tracking, code reviews, documentation. You get your hands dirty. You might even launch something that other people use (I know, crazy, right?).

Second: don't get hung up on who invents what. Coming from academia, I was trained to fiercely defend my intellectual territory, pissing all over anything that seemed remotely close to my area of interest. Industry is far more collaborative and credit is widely shared. If another team has good ideas, and can help you achieve your goals faster, join them -- and build upon that common success. So much bad academic work seems to boil down to someone trying to be so differentiated and unique that they paint themselves in a sparkly, rainbow-colored corner that, yes, ensures you stand out, but means you're often going about things the wrong way. Unfortunately, the publish-or-perish culture of academia often forces people to add arbitrary differentiation to their work so they can't be accused of being derivative. That's too bad, because a tremendous amount of value can be derived by refining and improving upon someone else's ideas.

Third: hold yourself to a higher standard. Program committees can be brutal, but I don't think they go far enough when it comes to setting the bar for real-world relevance. Collecting data from a dozen students running your little buggy mobile app is nothing compared to industry scale. Even if you can't get a million or a hundred million people using your app, what would it take? Wouldn't it have to work on a lot of different phone platforms? Not badly impact battery life? Actually be secure? Use a server running somewhere other than under a grad student's desk? Possibly have a unit test or two in the code?

Now, I know what you're thinking -- all of this is a distraction, since you don't need to go this extra mile to get that paper submitted. That's true. But if you're trying to do industry-relevant work, it helps to look at things through an industry lens, which means going beyond the "it ran once and I got the graphs" prototypes you're probably building. Why doesn't Google just rewrite Android in Haskell, or why doesn't Facebook just throw away their filesystem and use yours instead? Maybe it has something to do with some of these annoying "engineering problems".

Finally, try to keep an open mind about how your research can have impact and relevance. A lot of academics get so laser-focused on impressing the next program committee that they fail to see the big picture. It's like being a chef that's only out to impress the restaurant critics and can't cook a decent goddamn hamburger. My PhD advisor never seemed to care particularly about publishing papers; rather, he wanted to move the needle for the field, and he did (multiple times). Over-indexing on what's going to impress the tiny set of people reviewing SOSP or MobiCom papers is a missed opportunity. Racking up publications is fine, but if you want to have impact on the real world, there's a lot more you can do.

43 comments:

  1. Nicely put Matt, and very good points!

    ReplyDelete
  2. well written, having made the move myself, it really is good advice to academics.

    ReplyDelete
  3. Basically you expect academia to be more aligned with industry's challenges and problems. I think industry needs to do (much) more to motivate researchers to steer their focus to the industry's agenda. Google has been doing some interesting things, namely by open sourcing important libraries (e.g. TensorFlow).

    Expecting researchers to go spend their sabbaticals contributing to particular agendas is not a serious proposal. If you want more from academia you need to give more to academia.

    ReplyDelete
    Replies
    1. Matt wasn't asking academics to solve Google's (or any other particular company's) specific problems. He was suggesting they quit wasting their time solving irrelevant (i.e., already solved or solving a non-problem) problems. I read Matt's suggestion that academics try a sabbatical in industry as a way to help the academics themselves, not a backdoor plea for free labor.

      Having made a similar leap from academia to first an industry research lab and now a product unit (IBM Cloud), I completely agree with Matt's points. It's frustrating to read and review papers that no doubt represented a lot of very hard work, but do not move the needle for the field.

      Delete
    2. I'd love to hear ways in which Google (and other companies) can help with this problem. It's something I spend a fair bit of time on. I recently hosted a two-day workshop with a bunch of academics working on mobile, with the goal of doing a "show and tell" of the stuff we're doing at Google and helping to build connections between us and the academics. I do think that sabbaticals and internships (for students) are a great way to get plugged in. We give a lot of funding to academic researchers in the hopes of fostering collaborations as well. Suggestions always welcome.

      Delete
    3. > We give a lot of funding to academic researchers
      Surely you meant to say you give a tiny bit of funding to academic researchers.

      Delete
    4. Millions of dollars a year isn't tiny.

      Delete
    5. I think NSF's budget is about $7 billion with about $800-900M going to CS. DARPA is about $3 billion per year and a significant fraction goes to CS (e.g., I20 is mostly CS focused and their budget is about $500 million). I'm not sure what Google's grant budget is, but I think a "lot of" funding would be in terms of hundreds of millions.

      I do agree with your point that academia could do more to work with industry, but many people in industry have little interest in working with academia. I remember running into this issue after joining a security group in a large software company. It seemed like the focus was so short term on what to do next week, that there was very little interest or opportunity in working with other groups or taking outside ideas. This was true even if they came from a research groups within the same company. Although I learned a fair bit while there, I wouldn't recommend academics join that group. I'm sure your group is different, but there are also academic groups that do a great job at working with people in industry.

      As a side note, I really like the approach Stokes takes in his book "Pasteur's Quadrant" by describing fundamental research and applications as completely separate axes. It seems like Google's based their research funding on this paradigm, which is quite smart in my opinion.

      Delete
  4. Great post Matt. The industry booths at conferences are one way to connect with companies to learn more about their problems, data, and research opportunities. We @YahooLabs have lots to offer, as do lots of industrial labs. Check them out!

    ReplyDelete
  5. A few health care organizations offer paid sabbaticals or summer experiences for faculty, graduate students and post-docs. Why don't more IT companies do this?

    ReplyDelete
    Replies
    1. Most IT companies offer (paid) internships and many offer sabbaticals.

      Delete
  6. Yeah, why doesn't Google just rewrite Android in Haskell?

    ReplyDelete
    Replies
    1. I think it has more to do with the fact that the majority of app developers is willing to program in java but not haskell.

      (IMHO, Haskell has effective ways of programming with side effects if you need them. But I agree that it is a difficult language for newbies.)

      Delete
  7. Isn't it the other way around ?
    The way I see it is, this "pubish-or-perish" kind of harsh environment is forced upon many academia, forcing academic researchers to produce quick results (irrespective of how ugly or differential the work is). I have personally spoken with many academicians who has some very interesting (and probably defining) problems in their minds, but could not work on that as it does not produce quick results. Because of this pressure environment, they cannot afford to see the big picture, even if they want to!

    Adding to that, by actually finding ways to dissuade this "publish-or-perish" pressure, I believe academicians will automatically give up working on well, crap. And they can put efforts into pursuing revolutionary thoughts and also have enough time to focus on those annoying engineering problems.

    While, I sincerely acknowledge and appreciate the presence of few academic labs who have been able to keep themselves out of this rat race, yet be successful - not the majority of academic labs have been successful in doing this.

    ReplyDelete
    Replies
    1. Maybe you shouldn't be an academic if you don't actually have any ideas. Doing the right thing is hard, because people have incentives to do the wrong thing - everybody has constraints. Academics somehow think this is special to them. Also, the main point of this post is that the proposals themselves are bad - where are all the revolutionary ideas? If someone has a truly good idea, it's actually not hard to get funding in my personal experience.

      Delete
  8. Another large issue is the clarity of Academic papers. Often they are incomprehensible to people outside a small research area. This often serves to make the irrelevant seem important while adding to a checklist of publications, but in no way advances the field.

    The best papers advance the field and are written in jargon-free language that's accessible to the expert and the interested novice. But these are few and far between.


    ReplyDelete
    Replies
    1. What you said.

      Like Matt, I read a lot of papers and proposals. This week, I happen to be reading a bunch of proposals from faculty PIs who are asking Google to fund their research. Here's a fact: if we don't understand your proposal, we're not going to fund it. And in many cases, we don't have enough reviewers who are expert in your field, so if you can't explain it to our non-expert reviewers, they aren't going to vote for it.

      Delete
  9. Well put. I mostly agree.

    One counterpoint though: on the question of whether Academia is a good place to build "real" software, I dug up data from the Bureau of Labor Statistics:

    http://colin-scott.github.io/blog/2014/04/29/is-academia-a-good-place-to-build-real-software/

    tl;dr: the number of professional software developers is 3 orders of magnitude larger than the number of academic researchers, and the gap is growing. Even if academics decide to expend time developing "real software" (forgoing the opportunity to work on other things), we can hardly compete with the sheer engineering horsepower of industrial software.

    Matt, I'm curious what you think!

    ReplyDelete
    Replies
    1. I don't think academia's job is to compete with industry; as you point out, that's futile. Academia should be working to advance knowledge. Ideally some of that knowledge will translate into real-world outcomes. The best way to do that (in my opinion) is to ensure that industry-focused research is being done with awareness and understanding of the real problems faced by industry, rather than in a make-believe world.

      Delete
  10. Good points Matt. I've spent all my research career in industry, and faced a different challenge .. problems that are valued in an industrial research setting being considered engineering or integration work by the academic community. I stopped publishing for a few years because it seemed pointless.

    Two requests: can you please write a blog post about how to have a successful research career in CS outside of a University? Also, the workshop you held at Google on mobile computing challenges .. any chance you can make the agenda or any of the material public? Thanks!

    ReplyDelete
    Replies
    1. I'm not a good person to write about having a successful research career outside of a university, because I don't think I'm having a research career anymore :-) But see: http://matt-welsh.blogspot.com/2012/06/googles-hybrid-approach-to-research.html

      I don't think we can make the workshop material public, unfortunately.

      Delete
    2. That is things most PHD want to know but they do not have chance, there are still too few internship from Google or some other industries. But really if Google can reveal more about what industry really needs, at least fresh PHDs get this big picture, otherwise they will be forced to go through paper jungle

      Delete
  11. I loved this post. I am an academic ("trying to be" would be more appropriate) but I come from an engineering world. I learned some of the points you mention as keys that we need to focus on as academics, in an effort to visualize/aim at real products.
    I'm trying to apply that everyday, it's difficult.
    We have biotechnology projects and try to really align with what medical people NEED! The people we collaborate with are reluctant to it, it's natural for them to prove a concept. Don't ask for a field solution. Don't ask for a solution at all...
    I believe we can change that, as students are willing to see their work used, in other people's hands. Especially in healthcare.

    But it's a struggle to ask for any quality tests, or yield evaluation, or repeatability, or debugging, or any optimization of a prototype.

    Thanks for sharing. You have a new follower.

    ReplyDelete
  12. I loved this post. I am an academic ("trying to be" would be more appropriate) but I come from an engineering world. I learned some of the points you mention as keys that we need to focus on as academics, in an effort to visualize/aim at real products.
    I'm trying to apply that everyday, it's difficult.
    We have biotechnology projects and try to really align with what medical people NEED! The people we collaborate with are reluctant to it, it's natural for them to prove a concept. Don't ask for a field solution. Don't ask for a solution at all...
    I believe we can change that, as students are willing to see their work used, in other people's hands. Especially in healthcare.

    But it's a struggle to ask for any quality tests, or yield evaluation, or repeatability, or debugging, or any optimization of a prototype.

    Thanks for sharing. You have a new follower.

    ReplyDelete
  13. Ideally, academics should really stay away from any research that can in any way and form be profitable for a company. Companies should do it all on their own. In fact, Matt is correct, when academics try to do it they usually don't do it well enough and company people get frustrated (e.g. Matt). Best way to have academic+industry mashup is the way it usually happens: academics do something new, some companies benefit from it and take it further.

    ReplyDelete
    Replies
    1. I think this is wrong-headed and not, in general, a good way of having an academic career.

      It's not the only path, but many successful academics work on stuff that can be more or less immediately turned into a product or a startup company. Others work farther out on things that might not be practical for years to come. You *can* do industry-relevant research if you want to; this post is about avoiding the pitfalls.

      Delete
    2. Well you titled your post provocatively. You want to get the eyeballs, well they are here. It is not difficult to squint and see how my answer was within the boundaries of your contrived post: it is those academics that may want to do "industry relevant" research that I am advising against doing it. A very clear cut argument can be made that contributing creatively to any problem is better than toiling in the narrow confines of a short lived business agenda.

      Delete
    3. That's a fair point, and many academics do in fact work outside the immediate interest of industry. I don't think it's the best or only way though.

      Delete
    4. Math? Astronomy? Chemistry? Medicine? Biology? I think you may have an inflated view of how big your niche of the world is. Most scientists rarely worry about industry. I'd say working along with industry is the much less frequent, and most ill advised route for an academic. The nucleus for most impactful advances on human life have and continue to come from people tinkering around in no particular direction.

      Delete
    5. I agree with your last sentence, save for the final 3 words; I would say that the greatest advances have come *as a side effect* of people tinkering around or working on other, though often related, problems. While some advances (e.g. pure mathematics) come from largely decoupled work, most stem from an individual's attempts at solving a different problem entirely.

      Delete
  14. I would say that it is difficult for some reviewers do realize the relevance of a practical research. In my specific case, I took almost two years to get a paper accepted because of this.

    ReplyDelete
    Replies
    1. I guess this depends on the field and how you couch the research. A lot of systems and mobile research is fairly practical, but I don't know how other fields view this.

      Delete
  15. You don't to dive straight in a commit to the length of time that it needs to do an internship or a sabbatical. I negotiated three days in one company, where they arranged that I got to talk to everyone: sales team, support desk, developers, management. That gave me enough context to know what questions to ask and to determine whether a longer internship would be useful. Is this sort of short, intensive, personal experience something that Google (and others) would be willing to copy?

    ReplyDelete
    Replies
    1. A place like Google is large enough that 3 days isn't going to give you a lot of context, but informal visits like this are possible.

      Delete
  16. I fully agree with this article. I spent six months at Google research which was an eye opener for me. Pivotted my research after my visit and my lab is less focused on publishing and more on developing open source software that can make a direct impact on people's lives.

    ReplyDelete
  17. Many good points, Matt. You explicitly focus on engineering and problem-based research driven and weighted by here-and-now size/significance/pain-point/USP status, the natural domain of R&D organizations. I am with you. It leaves implicit the raison d'etre -- and arguably both strength and mission by virtue of being complementary -- for academic institutions: Being independent, in pursuit of genuinely new knowledge (per se) and seeking public enlightenment (of all). These were the guiding principles of Humboldtian universities, whose unexpected consequences were serendipitous technological and commercial successes shifted across fields (pure math to crypto, pure chemistry to pharma, etc), space (somewhere else) and time (much later). No contradiction to what you advocate, expect, maybe, this:

    "I don't think we can make the workshop material public, unfortunately."

    You say academics should take industrial sabbaticals or internships -- a good thing! -- in order to see what's already been solved and what's really important, which I'm a bit worried about if requiring a Hotel-California-style cloak of company secrecy. I'd argue it's fair game to require anybody claiming novelty to produce public, reproducible evidence, including the data. It's the price of secrecy that one might get "scooped" by somebody who's doing is half-bad. A low price for a company (and often paid), but intentionally high (to favor sharing) for an academic pursuing the (Humboldtian) university's rather than the company's mission.

    ReplyDelete

  18. Am here to testify what this great spell caster done for me. i never believe in spell casting, until when i was was tempted to try it. i and my husband have been having a lot of problem living together, he will always not make me happy because he have fallen in love with another lady outside our relationship, i tried my best to make sure that my husband leave this woman but the more i talk to him the more he makes me fell sad, so my marriage is now leading to divorce because he no longer gives me attention. so with all this pain and agony, i decided to contact this spell caster to see if things can work out between me and my husband again. this spell caster who was a woman told me that my husband is really under a great spell that he have been charm by some magic, so she told me that she was going to make all things normal back. she did the spell on my husband and after 5 days my husband changed completely he even apologize with the way he treated me that he was not him self, i really thank this woman her name is Dr Aluta she have bring back my husband back to me i want you all to contact her who are having any problem related to marriage issue and relationship problem she will solve it for you. her email is traditionalspellhospital@gmail.com she is a woman and she is great. wish you good time.
    He cast spells for different purposes

    ReplyDelete
  19. I made a similar point a while back on research on recommender systems, that showing performance of recommendations on a toy problem and small data doesn't help advance the state-of-the-art:

    http://glinden.blogspot.com/2006/01/recommender-systems-and-toy-problems.html

    There's the legitimate response that the academic community doesn't have access to the data they need. Despite some releases on anonymized data, the companies that have data mostly hold on to it tightly, and the main way to get access to it is to work at the company. Not clear how to solve this, but it's a hard problem for academic research.

    ReplyDelete