tag:blogger.com,1999:blog-9186457242428335144.post1380719819644455794..comments2024-03-28T00:36:13.790-07:00Comments on Volatile and Decentralized: Computing at scale, or, how Google has warped my brainMatt Welshhttp://www.blogger.com/profile/04255792550910131960noreply@blogger.comBlogger37125tag:blogger.com,1999:blog-9186457242428335144.post-75138518515261170152010-11-03T05:17:14.838-07:002010-11-03T05:17:14.838-07:00it should really feel awesome if you have that muc...it should really feel awesome if you have that much computing capacity to write code on ..lakshmananhttp://www.rubyinair.comnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-37249589519454684192010-10-28T10:14:35.562-07:002010-10-28T10:14:35.562-07:00Hi Matt, comparing MPI and MapReduce, computations...Hi Matt, comparing MPI and MapReduce, computations would definitely benefit from a loosely coupled fault resilient infrastructure. Can you isolate the infrastructure away from MapReduce algorithm per-se, and provide the infrastructure in terms of passing out blocks of data for computation, returning results (if not crashed), and repeat until all the results show up?<br />We need a resilient computation infrastructure. MapReduce is only one algoritm!! Support all of the "Berkeley 7 Dwarfs" !Rich Altmaiernoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-52826405660164963662010-10-28T06:13:39.031-07:002010-10-28T06:13:39.031-07:00Since you asked, here are the relevant parts of my...Since you asked, here are the relevant parts of my .vimrc, which I have been using for years. I am by no means a vim wizard; I tend to stick to the pretty basic commands, and don't do fancy stuff like split windows, multiple buffers, or macros very well. If I got timewarped back to 1976 and had to edit a file on a VAX I would be very comfortable :-)<br /><br />set noshowmode<br />set ts=8<br />set tw=70<br />set noruler<br />set ai<br />set shiftwidth=2<br />set background=dark<br />set wildmode=longest:full<br />set wildmenu<br />set hlsearch<br /><br />" allow backspacing over everything in insert mode<br />set backspace=indent,eol,start<br />set copyindent " copy the previous indentation on autoindenting<br />"set number " always show line numbers<br />set shiftround " use multiple of shiftwidth when indenting with '<' and '>'<br />set showmatch " set show matching parenthesis<br />set ignorecase " ignore case when searching<br />set smartcase " ignore case if search pattern is all lowercase, case-sensitive otherwise<br />set smarttab " insert tabs on the start of a line according to shiftwidth, not tabstop<br />set incsearch " show search matches as you type<br /><br />set history=1000 " remember more commands and search history<br />set undolevels=1000 " use many muchos levels of undo<br />set wildignore=*.swp,*.bak,*.pyc,*.class<br />set title " change the terminal's title<br />set visualbell " don't beep<br />set noerrorbells " don't beep<br />syntax enable<br />hi SpellBad cterm=bold ctermfg=227 ctermbg=NONE<br />map :syntax enable:set spell<br />map :nohlsearch:redraw:<br />autocmd FileType c set cindent | set tw=0 | map ij<br />autocmd FileType cs set cindent | set tw=0 | map ij<br />autocmd FileType nesc set cindent | set tw=0 | map ij<br />autocmd FileType cpp set cindent | set tw=0 | map ij<br />autocmd FileType java set cindent | set tw=0 | map ij<br />autocmd FileType tex set spell<br />autocmd FileType txt set spell<br />autocmd FileType perl set cindent | set tw=0 | map ij<br />autocmd FileType python set cindent | set tw=0 | set shiftwidth=2 | set expandtab | set ts=2 | map ijMatt Welshhttps://www.blogger.com/profile/07077674014671176946noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-21039797247146986242010-10-28T06:10:23.989-07:002010-10-28T06:10:23.989-07:00The comparison between MPI and MapReduce was simpl...The comparison between MPI and MapReduce was simply to define the two extremes in the parallel programming model space. MPI is designed for tightly-coupled systems where processors and communication rarely fail; MapReduce works on much larger, loosely-coupled systems where failures are commonplace. It's safe to say that MR scales much better than MPI, but is better suited for computations that don't require explicit coordination between processors. In MR, the data is primary; in MPI, the computation is primary. There's a vast difference between the two models.Matt Welshhttps://www.blogger.com/profile/07077674014671176946noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-71521625967805322802010-10-27T10:59:31.455-07:002010-10-27T10:59:31.455-07:00Hi Matt, you mention MapReduce and brittle MPI inf...Hi Matt, you mention MapReduce and brittle MPI infrastructure. Aren't these two different layers? MapReduce is one parallel algorithm. Aren't there others? And MPI is one parallel implementation method underlying an algorithm such as MapReduce or matrix multiply. What is MapReduce based on, and can it be used for matrix multiply? <br />Can you provide a systematic picture here?<br />Thanks for the article!!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-83240655360363939902010-10-26T06:30:45.088-07:002010-10-26T06:30:45.088-07:00very good information,thank youvery good information,thank youJoshttp://www.netmon.canoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-72152455608118015452010-10-25T22:49:40.646-07:002010-10-25T22:49:40.646-07:00Great post. Great comments alike. Matt, I just won...Great post. Great comments alike. Matt, I just wonder if you could share your .vimrc file (a trimmed down version I guess) to learn new tricks from a Vim power user. Thanks.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-20437899714532632782010-10-22T07:27:52.166-07:002010-10-22T07:27:52.166-07:00Automated Unit Testing : http://www.parasoft.com/j...Automated Unit Testing : http://www.parasoft.com/jsp/technologies/unit_testing.jsp?itemId=326Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-30011590862480550412010-10-22T04:54:25.345-07:002010-10-22T04:54:25.345-07:00Hi Matt,
Interesting comments about debugging - u...Hi Matt,<br /><br />Interesting comments about debugging - unit tests are great, success of printf depends on or requires the independence the machines/processes from each other - which may be your case.<br /><br />But for the line "There's little hope of doing real interactive debugging in this kind of environment, [..]" -- take a look at <a href="http://www.allinea.com/index.php?article=34" rel="nofollow">this</a> 200,000 cores debugging at the same time -- it's not the same environment but shows what can be done at scale.Paralloxonhttps://www.blogger.com/profile/07057964814555050069noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-27908900041048063612010-10-21T20:01:09.839-07:002010-10-21T20:01:09.839-07:00Do you use unit tests after you re in google?or yo...Do you use unit tests after you re in google?or you have use it before?<br /><br />Anyway,unit test rocks!fajrhttp://chipz.kandangbuaya.comnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-34594210513947874432010-10-21T18:51:34.278-07:002010-10-21T18:51:34.278-07:00I liked the fact that you mentioned that unit test...I liked the fact that you mentioned that unit testing should be done a priori. Can't stressed that enough. In an environment like the one you've experienced, i can appreciate that it needs to be done.Raymond Tayhttps://www.blogger.com/profile/12435190240911599928noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-54240731585797913282010-10-21T18:51:19.791-07:002010-10-21T18:51:19.791-07:00Cool post! I interned at Google over the summer, a...Cool post! I interned at Google over the summer, and felt the same effect. I am a fan of unit tests now as well.Vijay Chidambaramhttps://www.blogger.com/profile/04004244081389414619noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-92097070635356274502010-10-21T16:50:05.481-07:002010-10-21T16:50:05.481-07:00Interesting post. I think that we've passed fr...Interesting post. I think that we've passed from "The network is the computer" to "The Internet is the database"...Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-4567516567310911592010-10-21T13:01:52.284-07:002010-10-21T13:01:52.284-07:00Google has a possibilty to do than others :)Google has a possibilty to do than others :)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-69005018577452972542010-10-21T12:25:32.598-07:002010-10-21T12:25:32.598-07:00The cloud and its costs don't scale well for a...The cloud and its costs don't scale well for all data sets. <br /><br />http://technews.acm.org/archives.cfm?fo=2008-05-may/may-28-2008.html&hdr=1#363435<br /><br />Admittedly, things have changed since this report came out in 2008 (such as being able to ship physical disks to Amazon for import into their storage systems), but much of it still holds true.<br /><br />TravisTravishttp://www.linearnon.comnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-54727577783654686062010-10-21T12:24:15.822-07:002010-10-21T12:24:15.822-07:00@Edward Benson, or perhaps our IDEs become web-bas...@Edward Benson, or perhaps our IDEs become web-based editors. Our source code, the compiler, and all the other compute-intensive parts live in the cloud. Look up Bespin and Ecco.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-55225045493297330692010-10-21T09:27:21.422-07:002010-10-21T09:27:21.422-07:00Here is a great introduction to this from Google: ...Here is a great introduction to this from Google: http://research.google.com/pubs/pub35290.htmlAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-25729524313001391342010-10-21T07:18:48.131-07:002010-10-21T07:18:48.131-07:00Great post! The cool part is the data center is al...Great post! The cool part is the data center is all Linux and the desktop / laptop computer is running Linux or the Mac OS. <br /><br />http://blogs.computerworld.com/16232/good_bye_windows_hello_linux_mac_says_google<br /><br />http://content.dell.com/us/en/gen/d/large-business/google-data-center.aspxErichttp://mixeduperic.comnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-87351106886894825822010-10-21T06:02:46.767-07:002010-10-21T06:02:46.767-07:00A very practical question: if we move to a world i...A very practical question: if we move to a world in which it more effective for us to use low-powered laptops as windows into remotely managed server farms, do we have to throw away our graphical IDEs in favor of text-mode editors (emacs and vi)?Edward Bensonhttp://www.edwardbenson.comnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-73813057340931942382010-10-21T04:16:19.432-07:002010-10-21T04:16:19.432-07:00While the concept of daily large-scale computing i...While the concept of daily large-scale computing is certainly an important message, as a developer I was more intrigued by how you set up your development environment in a remote machine with a text-based interface. Very old school but, I suspect, very effective as well. My whole career as a programmer was bound to microcomputers and I've always done my development work locally and, in the last 15 years, using IDEs in a graphic environment. I suspect the old way gives you some freedom, though, and of course some serious computing, storage and bandwidth power at your command. No offline work, though, unless you replicate your environment in a local VM.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-81316656592090920632010-10-21T03:49:52.237-07:002010-10-21T03:49:52.237-07:00@Maulik
“When the network becomes as fast as the ...@Maulik<br /><br />“When the network becomes as fast as the processor, the computer hollows out and spreads across the network.” -Eric Schmidt, 1993;<br /><br />This is all about overcoming the Von Neumann bottleneck. You guys may be interested in reading this John Backus Turing Award ACM paper from 1977. Backus was way ahead of his time with this.<br /><br />http://www.thocp.net/biographies/papers/backus_turingaward_lecture.pdfAndrew de Andradehttp://twitter.com/andrewdeandradenoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-35385425627646471222010-10-21T01:48:40.743-07:002010-10-21T01:48:40.743-07:00I am curious -- what has changed in your views on ...I am curious -- what has changed in your views on the client-side role of operating systems?Ballom, Master of Deathhttps://www.blogger.com/profile/09923647886706218295noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-72544951645337908092010-10-20T20:20:46.601-07:002010-10-20T20:20:46.601-07:00Matt, if you're really serious about the 6-mon...Matt, if you're really serious about the 6-month SRE rotation, there's a GWS presence in Cambridge, and I think we could work something out. Email me if you're interested.Astrid Atkinsonhttps://www.blogger.com/profile/18444870973539689296noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-38539390022681716632010-10-20T18:37:33.246-07:002010-10-20T18:37:33.246-07:00Anon re: SREs. Indeed! SREs (site reliability engi...Anon re: SREs. Indeed! SREs (site reliability engineers, for non-Googlers) have one of the most important jobs here. There is a six-month rotation for software engineers to learn the SRE trade that I'd love to do, but unfortunately Cambridge doesn't have a large enough SRE team to host it.Matt Welshhttps://www.blogger.com/profile/07077674014671176946noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-4191392467669473332010-10-20T17:04:45.215-07:002010-10-20T17:04:45.215-07:00Well, aside from tools, there are also a "few...Well, aside from tools, there are also a "few" SREs who make things go.Anonymousnoreply@blogger.com