tag:blogger.com,1999:blog-9186457242428335144.post6803918483818622730..comments2024-03-28T00:36:13.790-07:00Comments on Volatile and Decentralized: The plight of the poor application paperMatt Welshhttp://www.blogger.com/profile/04255792550910131960noreply@blogger.comBlogger14125tag:blogger.com,1999:blog-9186457242428335144.post-81074489665971124182010-11-29T23:53:04.198-08:002010-11-29T23:53:04.198-08:00This problem is worse for technology for developin...This problem is worse for technology for developing regions for several reasons:<br /><br />1) the best solutions are simple, and simple solutions do not do well with reviewers!<br /><br />2) much of the value is the *discovery* of the problem; most fields reward problem discovery, but CS is not one of them. Once a problem is well defined, the solution tends to be too obvious for publication.<br /><br />3) applications in developing regions tend to be harder to execute, which is a kind of tax on the rest of the paper. (Given k hours for a paper, the higher the tax, the less time for innovation.)<br /><br />As we develop the IT for developing regions community (and conferences), we are trying to create a culture that values applications. On the plus side, the CHI community does this, so it is possible.<br /><br />-EricEric Brewerhttp://www.cs.berkeley.edu/~brewernoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-91849440164800957852009-12-06T03:07:11.151-08:002009-12-06T03:07:11.151-08:00Wonderful blog, i recently come to your blog throu...Wonderful blog, i recently come to your blog through Google excellent knowledge keep on posting you guys.dissertation writinghttp://www.ukdissertations.net/dissertation_writing.htmnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-73905868375631640862009-11-20T23:18:56.954-08:002009-11-20T23:18:56.954-08:00Many institutions limit access to their online inf...Many institutions limit access to their online information. Making this information available will be an asset to all.Paper Researchhttp://www.researchpaperspot.com/noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-62320837254126466572009-03-20T12:48:00.000-07:002009-03-20T12:48:00.000-07:00As another note, IMC would only consider papers fo...As another note, IMC would only consider papers for "Best Paper Awards" that agreed to publish their datasets. I always thought that was a great incentive device, one I wish we could somehow extend to other systems and networking conferences.Mike Freedmanhttps://www.blogger.com/profile/13738553299681103099noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-5522772623775650542009-03-20T12:41:00.000-07:002009-03-20T12:41:00.000-07:00I'd agree with Norman here. While building real s...I'd agree with Norman here. While building real systems isn't appropriately recognized in terms of publication count---and indeed, the incentives are in the opposite direction---I think they are well acknowledged by ones' peers at other points of your career (hires, promotions, awards, etc.).<BR/><BR/>Speaking from a personal note, I only really published one real paper on CoralCDN, even though it ended up taking (portions) of many years. On the other hand, I think its "realness" ultimately had a huge impact on my job search. The same should be said, with even stronger emphasis, on Larry's work on PlanetLab. Remarkably few publications, but huge impact on the entire systems and networking community. And a large reason we probably have this thing called GENI (hopefully) around.Mike Freedmanhttps://www.blogger.com/profile/13738553299681103099noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-49385116355275968782009-03-17T15:59:00.000-07:002009-03-17T15:59:00.000-07:00Matt,Welcome to my world. I think that in today's...Matt,<BR/><BR/>Welcome to my world. I think that in today's system, research credit through real deployments comes not so much through traditional measures as through the respect of one's peers, as measured through your tenure letters. As we know, this measurement depends greatly on whom is asked.<BR/><BR/>Over the years I have squeezed out quite a few papers because it is hard to build something serious without learning *something* new, which you can then publish. It is a harder sell to solve an old problem in a new way, but sometimes that can be done, too.<BR/><BR/>NormanAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-25505125583143573192009-03-05T23:17:00.000-08:002009-03-05T23:17:00.000-08:00"I admit I kind of like the idea of "marking" a pa...<I>"I admit I kind of like the idea of "marking" a paper as an applications paper in some way"</I><BR/><BR/>I think Mike's suggestion is good. For what it's worth, SIGCOMM 2009 has "focus" tags such as "system implementation" in addition to the traditional topics classifiers.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-90958869054425967212009-03-05T06:57:00.000-08:002009-03-05T06:57:00.000-08:00Hi Matt. I sympathize. I admit I kind of like th...Hi Matt. I sympathize. I admit I kind of like the idea of "marking" a paper as an applications paper in some way, although one would hope that most people in the area would be able to read and judge such a paper appropriately.<BR/><BR/>I've actually just put up a post I've been tinkering with for a few weeks on the plight of the poor theory paper for networking/systems conferences. Good timing. :)Michael Mitzenmacherhttps://www.blogger.com/profile/02161161032642563814noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-2519864736366971832009-03-03T21:12:00.000-08:002009-03-03T21:12:00.000-08:00How about having people explicitly mark as part of...How about having people explicitly mark as part of the submission process that their paper should be considered as an application paper, then allocating reviewers accordingly? For SenSys in particular, there should be plenty of PC members and subreviewers who have the background and experience to judge these papers. There are plenty of issues in figuring out what is a good application paper, several of which you have raised. Most of these seem to come down to needing reviewers that have the appropriate "taste" in judging the paper.<BR/><BR/>Having authors mark "this is an application paper, judge it by those criteria" at submission time would save time and match the paper to the right people. Of course you would not necessarily have every single PC member on the paper be an "applications person," just to keep things from becoming too inbred. Still, it would be a way to fairly evaluate such papers and give them a fighting chance without splitting off into a separate conference.<BR/><BR/>I agree with the concern about splitting into a separate conference, by the way. I have seen cases where creating a new conference or workshop pays off with great new research (e.g. Privacy Enhancing Technologies, Usenix Electronic Voting Technologies), but in general I worry about fields becoming balkanized. Hard enough as it is to keep up with all the work coming out in the main focus of an area.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-81116850815599245542009-03-03T13:17:00.000-08:002009-03-03T13:17:00.000-08:00True, releasing code wouldn't help during the revi...True, releasing code wouldn't help during the review process. However, it might help on a longer timescale - it would enable repeatability of experiments (something that the community talks about but doesn't do that much in practice), and might keep folks from submitting papers they can't back up with code by the time the camera ready is due. That in turn would (indirectly) help application papers.<BR/><BR/>That said, it might prove too unpopular...Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-83613369767118168352009-03-02T17:01:00.000-08:002009-03-02T17:01:00.000-08:00I'm not sure that releasing source code is going t...I'm not sure that releasing source code is going to help matters much. PC members are overwhelmed with it is and I doubt that anyone would have time to take a serious look at the code/scripts/etc. when evaluating a paper. I guess it's a good idea in the sense that you could be "audited" at any time by a reviewer, but, it also opens up the potential for abuse where a PC member shoots down a paper due to lack of understanding of the code (or not liking the coding style, or some other trivial issue).Matt Welshhttps://www.blogger.com/profile/04255792550910131960noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-56656750602691289082009-03-02T11:26:00.000-08:002009-03-02T11:26:00.000-08:00"Another, perhaps less popular approach might be t...<I>"Another, perhaps less popular approach might be to require that all papers (in OSDI, for example) release the source code / test scripts used in the experiments described in the paper."</I><BR/><BR/>I think SIGMOD made this stuff a mandatory requirement last year. I'm not sure if the results changed significantly.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-88518744348920686422009-03-01T22:03:00.000-08:002009-03-01T22:03:00.000-08:00"Maybe there should be a built-in handicap for rea...<I>"Maybe there should be a built-in handicap for real deployment papers."</I><BR/><BR/>Another, perhaps less popular approach might be to require that all papers (in OSDI, for example) release the source code / test scripts used in the experiments described in the paper. This would shine a light on papers that were based upon unrealistic simulations or that don't deal with hard implementation details, and might tip the balance back toward application papers.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-65466359897175265822009-03-01T20:10:00.000-08:002009-03-01T20:10:00.000-08:00There's a fourth concern I have heard with buildin...There's a fourth concern I have heard with building and deploying artifacts, which is their relative transience vis-a-vis ideas.<BR/><BR/>One way to counter this handicap is for the community to promote application-driven research that either validates or points out significant drawbacks in making ideas work in practice. For e.g., in fields such as experimental Physics, it's even possible to obtain a Ph.D. for repeating prior ideas or claims carefully.Anonymousnoreply@blogger.com