My experience with Amazon Cloud Player

As I've posted here before, I'm an avid music fan and collector. A few years ago I decided to go all-digital with my music collection, and since then have mostly refused to buy CDs in favor of digital music online -- mostly from Amazon's (excellent) MP3 store, as well as iTunes. However, this created a new problem: where to keep the music, and how to keep it synced between my various devices -- home laptop, work laptop, work desktop, home desktop, phone, iPad. Lots of people have this problem. My music collection is now more than 50 GB and it's no small feat to keep it synchronized between devices.

For a while I had this crazy scheme where I would only buy new music on my home laptop (the "master" library) which I could sync directly to my phone. From the home laptop I would push new music (using rsync) to my home desktop, which would allow me to listen to it on the stereo at home (via a first-generation Squeezebox player). I would also push it to my desktop at Harvard, so I could listen at work. Once I moved to Google, syncing from home to work became more difficult, although not impossible -- but I could only do so when on the Google VPN, which I can only access on my work laptop. So I modified the aforementioned crazy scheme by syncing from the home laptop to my old desktop machine at Harvard, from which I would pull down new music to my work desktop and laptop. In this way, at least I always knew where the master copy was, and the flow of new data along any edge was always unidirectional, thereby avoiding sync conflicts.

Over time the complexity of this scheme became a real annoyance, especially since I could only buy new music when I was using my home laptop, and syncing to my iPhone and iPad required manually plugging them into that same machine.

So, about a month ago I switched over to using Amazon Cloud Drive, in the hopes that it would fix this problem once and for all. In terms of solving the sync problem, it has been a great success: all of my music now simply lives on Amazon’s servers, and (except for my phone) I don’t need to worry about syncing it to any other devices. For that, Amazon has a Cloud Player app for Android which can pull the music down to my phone directly, without having to stage it on any of my other machines. So I can buy music on the phone, which is immediately available in my Cloud Drive, and I can pull it down to the phone if I want to. (I even bought and downloaded an album to my phone while on an Alaskan Airlines flight, using the in-flight WiFi.) There is an amazing instant-gratification factor here: read about an album on Pitchfork, it’s in your library and on all of your devices in under a minute.

Now, I mostly listen to music at work using the Amazon Cloud Player in my web browser. I don’t even bother syncing a copy to my various machines, though from time to time I do download new music to my home laptop just to have a local backup (in case Amazon goes out of business or something).

Unfortunately, as a user experience goes, I think the web-based Cloud Player has a long way to go:

• The web interface is terribly slow, especially with a large music collection such as my own. I have more than 12,000 songs and 1,000 albums in the collection, and opening up the Web-based Cloud Player takes a good 20 seconds. The worst part is scrolling through the music’s “album” view: for some damn reason I always get a yellow spinner when paging through the set of albums, and it can take 20-30 seconds to load the album art for each page so I can see what I want to listen to. Plenty of other websites have solved the problem of previewing large numbers of images, so I can’t understand why Amazon’s site is so slow.
  
• I rarely have problems with the music playback, though keep in mind I am usually listening from Google’s very well provisioned network. (I also happen to live in Seattle, spitting distance from Amazon HQ, though I’m not sure how much that helps.) However, given that I generally have a few dozen other tabs open on my browser in several windows, I have noticed that the Cloud Player running in the background was inducing additional lag. Now, I only run Cloud Player from a different browser (Safari) dedicated to that purpose.
  
• Every 24 hours, the login credentials on the Cloud Player time out and it will stop playback immediately and force me to log in again. This is highly annoying, especially when I’m in the middle of rocking out to Gang Gang Dance. And of course, when I login again, the Cloud Player has forgotten its state so I have to wait 30 seconds to reload my music library and figue out which song I was listening to and fire it up again -- a good minute of rocking-out time lost. I can use Google docs, Gmail, and a bunch of other online services without having to enter my login credentials every day; why can’t Amazon solve this problem? Maybe it’s a licensing issue, but it makes for a painful user experience.
  
• When I first signed up, I had to run Amazon’s MP3 uploader to load my music library into their servers. Given that 99% of the music is available on Amazon’s own MP3 store -- and I suspect a good 15% of my library was originally purchased from same said store -- I was surprised that I had to go through this painful step. To add insult to injury, the uploader for some reason capped bandwidth at 500 Kbps or so, meaning it took nearly a week and a half to upload all my music. (Made even worse because the login credentials would time out every day and would not resume uploading until I logged in again.) I should have been able to upload my entire music library to Amazon in less than 20 hours on my 5 Mbps connection from home, so the artificial cap seems ridiculous.
  
• Somewhat humorously, when I buy new music from Amazon’s MP3 store, there is a delay before it shows up in my library. I can hit reload and watch each song being loaded into the library, and it seems to take a few seconds for each one to appear. I assumed that adding music from Amazon’s MP3 store to my Cloud Drive would be a matter of creating the S3 equivalent of a symlink, so what the hell is going on here?

Just for the record I have not yet tried Google Music, which is in beta, since I am too lazy to move my music collection over yet again. From what I have seen, the web interface is pretty similar so there's not a good enough reason -- yet -- to switch. I'm curious to see what Apple's iCloud is like, but I really like having my music collection liberated from Apple's DRM.

So, I am somewhat surprised at how bare-bones the Amazon Cloud Player is. Given that this is a web app, you would think there would be all kinds of great features that go beyond what you can do inside of a “closed” app like iTunes. For example, there’s no way to share a link to a song or album in Amazon’s MP3 store with Facebook or Twitter -- I have to go search for the listing on Amazon’s MP3 store by hand, and post that manually. This seems like a lost revenue opportunity for Amazon, since it’s hard to share with my friends online that I’m loving the new Bon Iver album and give them a link to buy it (hey, maybe with a referral discount).

Overall it's awesome to use this service and I love having my music everywhere all at once, and not having to manually maintain a library. If the web player were more sophisticated and responsive it would be a slam dunk.

Being Googley

I've been at Google for almost a year now and have been thinking back on what my expectations of the job would be like compared to what it has turned out to be. This got me thinking about corporate culture in general and how important it is for fostering innovation and being successful.

Google is well known for having a creative work environment with tons of perks -- free food, yoga classes, massage, on-site doctor. Here in Seattle, we can borrow kayaks to take out onto the shipping canal next to the building. (I am fond of telling people this but know full well that I am unlikely to ever take advantage of it.) On the surface these things might seem frivolous, but I think they go a long way towards creating an environment where people are passionate about what they do. The term we use is "being Googley," meaning, doing whatever it is that Google people do: thinking big, focusing on the user, not being evil, etc. On a more day-to-day basis, being Googley means turning out the lights when you leave a conference room, being friendly and helpful to new engineers, being a good citizen.

Google is by no means the only company like this: Yahoo, Facebook, and Amazon are great examples, and many other Internet-era startups follow a similar model. But this is miles away from what I thought corporate life would be like before I joined Google. To be sure, most of my prior exposure to corporate life (that is, before doing my PhD and becoming a professor) was through internships I did at a couple of older technology companies. I spent time writing code at a company that built semiconductor testers, as well as a large electronics company in Japan. I had also visited several large industrial research labs in the US, places that in some cases have been around for more than 50 years. I had a very hard time imagining myself taking a job at any of these companies: the sea of cubicles, drab beige walls, terrible cafeteria food, very few people under 40. Like Dilbert in real life. I wonder how those companies continue to attract top talent when there are places that are so much more appealing to work.

The Google culture is not just about lava lamps in the conference rooms though. The thing that surprised me the most is that there is very little hierarchy in the company: every engineer has the opportunity to create and lead new projects. It's not uncommon for something cool to start up with a few engineers working in their "20% time" -- GMail being one famous example. It is rare for a technical edict to be handed down from on high: projects are usually bottom up and are driven by what the engineers want to accomplish. To be sure, there are projects that could benefit from more adult supervision: things can go off the rails when you don't have enough management. But it's amazing what a merry band of hackers can put together without a lot of imposed structure or artificial constraints from the Pointy Haired Boss. I think the result is that engineers feel a lot of ownership for what they create, rather than feeling like they are just building something to make management happy.

When I was in the Google office in Cambridge, I worked on the team that builds Google's content delivery network -- a huge system that carries a significant chunk of the Internet's traffic (mostly YouTube). Almost everyone on the team was a good 10 years younger than I am (and way smarter, too). There was very little oversight and everyone kind of pitched in to keep the thing running, without anyone having to be told explicitly what to do. I was amazed that you could run such a large, complex project like this, but it seems to work. It's a hacker culture at a large scale. Companies like Facebook and Amazon are run pretty much the same way. All of this seems to turn the conventional wisdom about what it takes to run a successful company upside down. This won't be a surprise to anyone who spent time at startups in the last 10 years, but I'm new to this stuff and surprised that it even works.

The changing face of Computer Science education

The New York Times has a great article today on changing attitudes towards CS, driven in part by movies like "The Social Network." Apart from the movie's (glossy and inaccurate) depiction of what it's like to be a hacker, there's something else going on here: the fact that CS students can jump in and apply their knowledge to build great things. At Harvard, countless undergrads taking the introductory CS50 class are producing games, websites, and iPhone apps -- some of which are good enough to turn into commercial products. I don't know of any other field where this is possible after taking just a single semester's worth of courses.

Of course, it wasn't always this way. For a very long time, Computer Science education at most Universities was grounded in the abstract and theoretical. Undergraduates rarely got the opportunity to build "real" applications or products. After all, before the advent of cheap, powerful PCs, a department might have one computer for the entire class, and its main purpose was to sit in a walled-off machine room and spit out numbers on a printout -- hardly inspiring. I did my undergrad degree at Cornell, and the first class I took was taught in Scheme -- a language I have never used since -- although the projects were fun for someone like me (implementing a public key cryptosystem, and doing some neat machine vision algorithms). Of course, this was before the Web, iPhones, and Facebook, so CS class "projects" tended to be somewhat dry back then.

Unfortunately, there are still too many vestiges of this old fashioned approach to Computer Science evident in the curriculum. It is largely a generational thing. At Harvard, I had a hell of a time convincing some of the senior faculty that we should be teaching all CS students the fundamentals of computer systems, like how a process works, what a cache is, how to program using threads. (Of course, like most CS degree programs, Harvard still requires all students to learn the finer points of nondeterministic finite state automata and arcane discrete mathematics. Harry Lewis, who teaches this class, once described this to me as "eating your vegetables.")

A few years ago, I was asked to take over teaching CS50, Harvard's introductory CS course. Since the semester was right around the corner, I didn't have time to revamp the course, and agreed to do it only if I could teach the original course material with few changes. I took a look at the existing syllabus. The first lecture was about the "history of computing" and was full of black and white pictures of Babbage and ENIACS and men in horn-rimmed glasses looking over printouts in a terminal room somewhere. This was not a promising start. The next six lectures explained in painful detail -- down to machine instructions and the bit representation of integers -- how to write a single program: How to convert Fahrenheit to Celsius. This being the only program that students saw for the first month or so of the course, it's no wonder that the course did not have broad appeal. This kind of material probably worked very well in the early 1990's, but not so today -- the field has changed, and what students are looking for has changed too.

I passed on teaching CS50, and it's a good thing I did -- Harvard hired David Malan, who is infinitely more energetic and likable than I am, to teach it instead. David completely reworked the course from the perspective of someone who learned CS in the PC and Internet era. He had students hacking iPhone apps, writing PHP and JavaScript, building websites. Over the next few years, enrollment in the course has nearly quadrupled -- it's become one of the "must take" courses at Harvard. He has done an amazing job.

Of course, there is a risk in going too far down this fun, project-oriented route. Computer Science is not a vocational program, and it's important for students to graduate with a deep understanding of the field. It's true that you can do amazing things with existing languages and tools without learning much about the deeper theory and foundations. Still, I think it's great to attract students with a fun, application-oriented course that gets them excited about the field, and hit them later with the more abstract ideas that might seem less relevant at the outset.

One problem is that the classes that follow CS50 are nowhere near as exciting -- they don't have monthly pizza parties and free Facebook schwag at the end of the semester -- so keeping students in the program beyond the intro course can be a challenge. But I think it's important for universities to consider where CS undergrads are coming from and try to meet them there, rather than to teach the way it was done 30 years ago, on a PDP-11 running LISP.

Reflections on Fast, User-Level Networking

A couple of weeks ago at HotOS, one of the most controversial papers (from Stanford) was entitled "It's Time for Low Latency." The basic premise of the paper is that clusters are stuck using expensive, high-latency network interfaces (generally TCP/IP over some flavor of Ethernet), but it should now be possible to achieve sub-10-microsecond round-trip-times for RPCs. Of course, a tremendous amount of research looked at low-latency, high-bandwidth cluster networking in the mid-1990's, including Active Messages, the Virtual Interface Architecture, and U-Net (which I was involved with as an undergrad at Cornell). A bunch of commercial products were available in this space, including Myrinet (still the best, IMHO) and InfiniBand.

Not much of this work has really taken off in commercial datacenters. John Ousterhout and Steve Rumble argue that this is because the commercial need for low latency networking hasn't been there until now. Indeed, when we were working on this in the 90's, the applications we envisioned were primarily numerical and scientific computing: big matrix multiplies, that kind of thing.

When Inktomi and Google started demonstrating Web search as the "killer app" for clusters, they managed to get away with relatively high-latency, but cheap, Ethernet-based solutions. For these applications, the cluster interconnect was not the bottleneck. Rumble's paper argues that emerging cloud applications are motivating the need for fast intermachine RPC. I'm not entirely convinced of this, but John and Steve and I had a few good conversations about this at HotOS and I've been reflecting on the lessons learned from the "fast interconnect" heyday of the 90's...

Microbenchmarks are evil: There is a risk in focusing on microbenchmarks when working on cluster networking. The standard "ping-pong" latency measurement and bulk transfer throughput measurements rarely reflect the kind of traffic patterns seen in real workloads. Getting something to work on two unloaded machines connected back-to-back says little about whether it will work at a large scale with a complex traffic mix and unexpected load. You often find that real world performance comes nowhere near the ideal two-machine case. For that matter, even "macrobenchmarks" like the infamous NOW-Sort work be misleading, especially when measurements are taken under ideal conditions. Obtaining robust performance under uncertain conditions seems a lot more important than optimizing for the best case that you will never see in real life.

Usability matters:  I'm convinced that one of the reasons that U-Net, Active Messages, and VIA failed to take off is that they were notoriously hard to program to. Some systems, like Fast Sockets, layer a conventional sockets API on top, but often suffered large performance losses as a result, in part because the interface couldn't be tailored for specific traffic patterns. And even "sockets-like" layers often did not work exactly like sockets, being different enough that you couldn't just recompile your application to use them. A common example is not being entirely threadsafe, or not working with mechanisms such as select() and poll(). When you are running a large software stack that depends on sockets, it is not easy to rip out the guts with something that is not fully backwards compatible.

Commodity beats fast: If history has proven anything, there's only so much that systems designers are willing to pay -- in terms of complexity or cost -- for performance. The vast majority of real-world systems are based on some flavor of the UNIX process model, BSD filesystem, relational database, and TCP/IP over Ethernet. These technologies are all commodity and can be found in many (mostly compatible) variants, both commercial and open source; few companies are willing to invest time and money to tailor their design for some funky single-vendor user-level networking solution that might disappear one day.

Conference report: HotOS 2011 in Napa

This week, I served as program chair for the Thirteenth Workshop on Hot Topics in Operating Systems, or HotOS 2011, which took place at the Westin Verasa in Napa, California. HotOS is a unique workshop and one of my favorite venues -- it is the place for systems researchers to put forth their most forward-thinking ideas. Unlike most conferences, HotOS takes 5-page position papers, and it's expected that the submission really represents a position, not a mature piece of technical work condensed into the shorter format.

When it's done right, HotOS is full of great, big sky papers and lots of heated discussions that give the community a chance to think about what's next. In some years, HotOS has been more like an "SOSP preview," with 5-page versions of papers that are likely to appear in a major conference a few months after the workshop. We tried to avoid that this year, and for the most part I think we were successful -- very few papers in this year's HotOS were mature enough to have been considered for SOSP (although that remains to be seen).

I've already blogged about the highly contentious cloud computing panel at HotOS. Here's the rest of the trip report.

Timothy Roscoe holding court at HotOS.

This year I tried to tinker with the conventional conference format in which speakers give 25 minute talks with 5 minutes of questions afterwards. For HotOS, this seems excessive, especially since the papers are so short. Instead, we limited speakers to 10 minutes. There was some pushback on this, but overall I think it was extremely successful: I didn't feel that anyone was rushed, speakers did a great job of staying within the time limits, and by the time a talk started to get boring, it was over.

The other side is we wanted to have room for longer discussions and debates, which often can't happen in the 5 minutes between talks. Too often you hear "let's take that offline," which is code language for "I don't want to get into that in front of the audience." This is a cop-out. At HotOS, after every couple of paper sessions we had a 30-to-45 minute "open mic" session where anybody could ask questions or just rant and rave, which gave plenty of time for more in-depth discussions and debate. At first I was worried that we wouldn't be able to fill up the time, but remarkably there was often plenty of people lined up to take the mic, and lots of great back-and-forth.

A few highlights from this years' HotOS... all of the papers are available online, although they might be limited to attendees only for a while.

Jeff Mogul from HP kicked off the workshop with a talk about reconnecting OS and architecture research. He argued that the systems community is in a rut by demanding that new systems run on commodity hardware, and the architecture community is in a rut by essentially pushing the OS out of the way. He made some great points about the opportunity for OS designs to leverage new hardware features and for the systems community not to be afraid to do so.

To prove this point, Katelin Bailey from UW gave a great talk about how OS designs could leverage fast, cheap NVRAM. The basic idea is to get rid of the gap between memory and disk-based storage altogether, which opens up a wide range of new research directions, like processes which never "die." I find this work very exciting and look forward to following their progress.

Mike Walfish from UT Austin gave a very entertaining talk about "Repair from a Chair." The idea is to allow PC users to have their machines repaired by remote techs by pushing the full software image of their machine into the cloud, where the tech could fix it in a way that the end user can still verify exactly what changes were made to their system. The talk included a nice case study drawn from interviews with Geek Squad and Genius Bar techs -- really cool. My only beef with this idea is that the problem is largely moot when you run applications in the cloud and simply repair the service, rather than the end-user's machine.

Dave Ackley from UNM gave the wackiest, most out-there talk of the conference on "Pursue Robust Indefinite Scalability." I am still not sure exactly what it is about, but the idea seems to be to build modular computers based on a cellular automaton model that can be connected together at arbitrary scales. This is why we have workshops like HotOS -- it would be really hard to get this kind of work into more conventional systems venues. Best quote from the paper: "pledge allegiance to the light cone."

Steve Rumble from Stanford talked about "It's Time for Low Latency," arguing that the time has come to build RPC systems that can achieve 10 microsecond RTTs. Back in the mid-1990s, myself and a bunch of other people spent a lot of time working on this problem, and we called 10 usec the "Culler Constant," since that was the (seemingly unattainable) goal that David Culler set forth for messaging in the Berkeley NOW cluster project. Steve's argument was that the application pull for this -- cloud computing -- is finally here so maybe it's time to revisit this problem in light of modern architectures. I would love to see someone dust off the old work on U-Net and Active Messages and see what kind of performance we can achieve today, and whether there is a role for this kind of approach in modern cluster designs.

Geoff Challen from Univ. Buffalo and Mark Hempstead from Drexel gave the most entertaining talk of the workshop on "The Case for Power-Agile Computing." The idea of the talk was that mobile devices should incorporate multiple hardware components with different power/performance characteristics to support a wide range of applications. As you can see below, Geoff was dressed as a genie and had to say "shazam" a lot.

This might be the first open-shirted presentation ever at HotOS. Let us hope it was the last.

Moises Goldszmidt from MSR gave a really energetic talk on the need for better approaches for modeling and predicting the performance of complex systems. He proposed to use intervention at various points within the system to explore its state space and uncover dependencies. To me, this sounds a lot like the classic system identification problem from control theory, and I would love to see this kind of rigorous engineering approach applied to computer systems performance management.

The traditional Wild and Crazy Ideas session did not disappoint. Margo Seltzer argued that all of the studies assuming users keep cell phones in their pocket (or somewhere on their person) failed to account for the fact that most women keep them in a bag or elsewhere. Good point: I have lost count of how many papers assume that people carry their phones on them at all times. Sam King from UIUC talked about building an app store for household robots, in which the killer app really is a killer app. Dave Andersen from CMU made some kind of extended analogy between systems researchers and an airliner getting ready to crash into a brick wall. (It made more sense with wine.)

We gave away four amazing prizes: Google ChromeOS Laptops! Dave Ackley won the "most outrageous opinion" prize for his wild-eyed thoughts on computer architecture. Vijay Vasudevan from CMU won the best poster award for a poster entitled "Why a Vector Operating System is a Terrible Idea", directly contradicting his own paper in the workshop. Chris Rossbach from MSR and Mike Walfish from UT Austin won the two best talk awards for excellent delivery and great technical content.

Finally, I'd like to thank the program committee and all of the folks at USENIX for helping to make this a great workshop.

How can academics do research on cloud computing?

This week I'm in Napa for HotOS 2011 -- the premier workshop on operating systems. HotOS is in its 24th year -- it started as the Workshop on Workstation Operating Systems in 1987. More on HotOS in a forthcoming blog post, but for now I wanted to comment on a very lively argument discussion that took place during the panel session yesterday.

The panel consisted of Mendel Rosenblum from Stanford (and VMWare, of course); Rebecca Isaacs from Microsoft Research; John Wilkes from Google; and Ion Stoica from Berkeley. The charge to the panel was to discuss the gap between academic research in cloud computing and the realities faced by industry. This came about in part because a bunch of cloud papers were submitted to HotOS from academic research groups. In some cases, the PC felt that the papers were trying to solve the wrong problems, or making incorrect assumptions about the state of cloud computing in the real world. We thought it would be interesting to hear from both academic and industry representatives about whether and how academic researchers can hope to do work on the cloud, given that there's no way for a university to build something at the scale and complexity of a real-world cloud platform. The concern is that academics will be relegated to working on little problems at the periphery, or come up with toy solutions.

The big challenge, as I see it, is how to enable academics to do interesting and relevant work on the cloud when it's nearly impossible to build up the infrastructure in a university setting. John Wilkes made the point that that he never wanted to see another paper submission showing a 10% performance improvement in Hadoop, and he's right -- this is not the right problem for academics to be working on. Not because 10% improvement is not useful, or that Hadoop is a bad platform, but because those kinds of problems are already being solved by industry. In my opinion, the best role for academia is to open up new areas and look well beyond where industry is working. But this is often at odds with the desire for academics to work on "industry relevant" problems, as well as to get funding from industry. Too often I think academics fall into the trap of working on things that might as well be done at a company.

Much of the debate at HotOS centered around the industry vs. academic divide and a fair bit of it was targeted at my previous blog posts on this topic. Timothy Roscoe argued that academia's role was to shed light on complex problems and gain understanding, not just to engineer solutions. I agree with this. Sometimes at Google, I feel that we are in such a rush to implement that we don't take the time to understand the problems deeply enough: build something that works and move onto the next problem. Of course, you have to move fast in industry. The pace is very different than academia, where a PhD student needs to spend multiple years focused on a single problem to get a dissertation written about it.

We're not there yet, but there are some efforts to open up cloud infrastructure to academic research. OpenCirrus is a testbed supported by HP, Intel, and Yahoo! with more than 10,000 cores that academics can use for systems research. Microsoft has opened up its Azure cloud platform for academic research. Only one person at HotOS raised their hand when asked if anyone was using this -- this is really unfortunate. (My theory is that academics have an allergic reaction to programming in C# and Visual Studio, which is too bad, since this is a really great platform if you can get over the toolchain.) Google is offering a billion core hours through its Exacycle program, and Amazon has a research grant program as well.

Providing infrastructure is only one part of the solution. Knowing what problems to work on is the other. Many people at HotOS bemoaned the fact that companies like Google are so secretive about what they're doing, and it's hard to learn what the "real" challenges are from the outside. My answer to this is to spend time at Google as a visiting scientist, and send your students to do internships. Even though it might not result in a publication, I can guarantee you will learn a tremendous amount about what the hard problems are in cloud computing and where the great opportunities are for academic work. (Hell, my mind was blown after my first couple of days at Google. It's like taking the red pill.)

A few things that jump to mind as ripe areas for academic research on the cloud:

• Understanding and predicting performance at scale, with uncertain workloads and frequent node failures.
• Managing workloads across multiple datacenters with widely varying capacity, occasional outages, and constrained inter-datacenter network links.
• Building failure recovery mechanisms that are robust to massive correlated outages. (This is what brought down Amazon's EC2 a few weeks ago.)
• Debugging large-scale cloud applications: tools to collect, visualize, and inspect the state of jobs running across many thousands of cores.
• Managing dependencies in a large codebase that relies upon a wide range of distributed services like Chubby and GFS.
• Handling both large-scale upgrades to computing capacity as well as large-scale outages seamlessly, without having to completely shut down your service and everything it depends on.

What I'm working on at Google: Making the mobile web fast

A bunch of people have asked me what I work on at Google these days. When I joined Google last July in the Cambridge office, I worked with the team that runs Google’s content delivery network, which is responsible for caching a vast amount of (mostly video) content at many sites around the world. It is a fantastic project with some great people. My own work focused on building tools to measure and evaluate wide-area network performance and detect performance problems. This was a great “starter project,” and I got to build and deploy some pretty large systems that now run on Google’s worldwide fleet.

Now that I’m in Seattle, I am heading my own team with the charter to make the mobile web fast. By “mobile web”, I mean the entire web as accessed from all mobile devices, not just Google services and not just Android. While Android is a big focus of our work, we care a lot about improving performance for all mobile devices. This project is an outgrowth of Google’s broader make the web faster initiative, which has (to date) largely been focused on desktop web. The parent project has been involved with the Chrome browser, the SPDY protocol (a replacement for HTTP), the WebP image format, numerous network stack enhancements, and more. I see mobile as the next big frontier for the web and there are some huge opportunities to make impact in this space.

This is a hugely exciting project since it touches on many different platforms and cuts across layers. Everyone knows that mobile web usage is taking off: the growth of the mobile web is much, much faster than the growth of the desktop web was back during the dot-com boom. At the same time, the mobile web is painfully slow for most sites and most users. And of course, not everyone has the benefit of using a bleeding-edge 4G phone with LTE network speeds of 25 Mbps, like I do (my current phone is an HTC Thunderbolt, and it’s awesome). For the vast majority of Internet users, a mobile phone will be the only device they ever interact with.

So what are we working on? At a high level we are planning to tackle problems in three broad areas: The mobile devices themselves; the services they connect to; and the networks that connect them. On the device side, we are looking at a wide range of OS, network stack, and browser enhancements to improve performance. On the service side, we are looking at better ways to architect websites for mobile clients, providing tools to help web developers maximize their performance, and automatic optimizations that can be performed by a site or in a proxy service. Finally, at the network layer we are looking at the (sometimes painful) interactions between different layers of the protocol stack and identifying ways to streamline them. There is a huge amount of work to do and I am really fortunate to work with some amazing people, like Steve Souders and Arvind Jain, on this effort.

Our goal is to share solutions with the broader web development community -- we hope to release most of our code as open source, as many other Google projects have done. I also plan to maintain strong ties with the academic research community, since I feel that there is a great opportunity to open up new avenues for mobile systems research, as well as leverage the great work that universities are doing in this space.

I think Google is in a unique position to solve this problem. And, of course, we are hiring! Drop me a line if you are interested in joining our team.