tag:blogger.com,1999:blog-9186457242428335144.post2429955575747904290..comments2024-03-28T00:36:13.790-07:00Comments on Volatile and Decentralized: Death by peer reviewMatt Welshhttp://www.blogger.com/profile/04255792550910131960noreply@blogger.comBlogger12125tag:blogger.com,1999:blog-9186457242428335144.post-77881305872033599132016-06-15T16:27:43.023-07:002016-06-15T16:27:43.023-07:00Industry is vast and has many examples of both mor...Industry is vast and has many examples of both more rational and less rational processes.<br /><br />Zooming in specifically on grant proposals, for example, that could be compared to getting your startup funded by a particular firm. That's just as random, if not more random, than NSF panel recommendations: the outcome of your case will similarly be influenced by whether the reviewers had coffee in the morning, by the order in which they see the proposals, by how combative they are feeling on any particular day, and by two dozen other factors over which you have no control whatsoever. Within an organization, pushing a new project through could be straightforward in some settings, but is extremely difficult in others. In a well-run organization with healthy appetite for risk you probably know most of the rules and the approval process appears non-random to you. In many organizations, though, you will have next to no visibility into why your projects were picked up or not. Indeed, most organizations would default to not letting you propose risky projects requiring substantial funding commitments to begin with :).<br /><br />You make an interesting point re that in industry the decisions are made by people who have vested interest in the work. That's true, but, arguably, NSF's decisions are made by the representatives of the NSF, who hopefully have a vested interest in the success of their overall program, rather than by panelists who only advise the NSF. In industry, similarly, a decision-maker would have plenty of advisers and would depend on the quality of their advice -- no real procedural difference with the approach taken by the NSF. And there are real issues with the advice quality in industry settings as well. Major one being: it is often hard, if not impossible, to find people who are both knowledgeable and unbiased. After all, entire corporate strategy consulting industry exists to solve just this problem: provide senior corporate decision-makers with [most likely] diligent, [hopefully] unbiased, and [at least somewhat] knowledgeable advisers. In general, I do agree that major corporate decisions are treated with somewhat more diligence than NSF funding decisions. I am not sure that this diligence necessarily translates to better decisions. Take the example of the MSR SV lab closure. Microsoft most definitely spent considerable amounts of time and energy investigating whether to do it. Whether at the end the resulting closure was based on realistic projections, biased opinions, or a coin flip, we'll never know. <br /><br />As an aside, I think that current corporate decision-making processes are where networked systems folks could really make a difference right now. There are control and communication networks underlying all these issues that I have not seen anybody study with any kind of rigor (I may not be fully up to date on all the relevant literature though) -- and this is where there is tremendous impact to be had from improving things just a little tiny bit. Anonymoushttps://www.blogger.com/profile/14640429204762234518noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-22132037891738551142016-06-15T02:39:05.379-07:002016-06-15T02:39:05.379-07:00One problem is that the reviewers reject a submiss...One problem is that the reviewers reject a submission by arguing some of their favorate papers are not cited. This is so annoying.Anonymoushttps://www.blogger.com/profile/03557850672543014273noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-62477686349582889342016-06-13T20:49:46.592-07:002016-06-13T20:49:46.592-07:00Another factor is that for really novel work a lot...Another factor is that for really novel work a lot of the mathematics has not been refined yet. It's much easier to get something published that is incremental and has been developed over decades with a few changes. Something completely new of course will not be perfect which the reviewer can use as an excuse.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-51693762851157095992016-06-10T15:18:52.700-07:002016-06-10T15:18:52.700-07:00In some ways, the (at least on the surface) fact t...In some ways, the (at least on the surface) fact that many professors delegate their peer reviews to their (usually more idealistic) grad students is a feature of the system, not a bug. This is interesting because it is generally viewed negatively but in reality adds a much needed element of appreciation for newness to the process.Anonymoushttps://www.blogger.com/profile/04290848674284909718noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-46295120493580102212016-06-10T11:20:27.224-07:002016-06-10T11:20:27.224-07:00While you are right in principle, I am not sure ho...While you are right in principle, I am not sure how much this conflict of interest plays itself out in practice. It seems exceedingly unlikely that a PC member could really influence the outcome of other papers in a way that would favor their own (though that might not stop them from trying). Anyway, it's an interesting point, but I just don't have data on how problematic this is.Matt Welshhttps://www.blogger.com/profile/07077674014671176946noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-543642350461477462016-06-09T11:09:00.131-07:002016-06-09T11:09:00.131-07:00Good post.
In a paper review, reviewers have a con...Good post.<br />In a paper review, reviewers have a conflict of interest unless they never submit to that journal or conference.<br />Because everybody is racing for a limited spot, SIGCOMM can only accept 20 papers, a journal accepts 15-20 etc.<br />Thus, it is a good strategy to reject papers of others, and a good amount of people do that, unfortunately.<br />Another corruption is that, in many CS journals, some associate editors always publish their papers there, not in the other journals in the same area. How ethical is this? <br /><br />The definition of 'conflict-of-interest' is not working. A person cannot review a paper from his university, but it is OK to publish on your journal or cherry-picking lenient reviewers for your paper, and harsh reviewers for opposing papers. That is a widely accepted practice in todays academia. How sick is this? <br /><br />In a company case, nobody has a conflict of interest if a project is approved, unless the company has a limited budget for projects. Probably, companies like Google are more flexible, since money is not that much of an issue. <br />That is probably why, companies, by their nature are less arbitrary, not because they are good samaritans.seenhttps://www.blogger.com/profile/16174900565040121687noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-47398744786330942642016-06-08T22:19:48.408-07:002016-06-08T22:19:48.408-07:00As an expert on randomness, I have nothing but the...As an expert on randomness, I have nothing but the highest respect for your opinion. Still, I think there's a vast and measurable difference between the amount of care that goes into writing a paper review and making corporate decisions like shutting down a research lab.<br /><br />While I can certainly imagine some companies treating major decisions as capriciously as the proverbial Reviewer #3 ("There's nothing novel about this work", "I don't understand why you did X instead of Y", etc.), I think you would agree that such a cavalier approach to corporate decision-making would not, in general, be a recipe for success.<br /><br />I hope you are not trying to equate Microsoft's decision to shut down its Silicon Valley lab with the kind of bullshit decisions I see being made on papers on *every single PC I have ever served on*. I'm sure that the people affected by that closure were not happy about it, but I am willing to bet good money that the decision was made far more deliberately, and with much greater care, than the sloppy, mostly shoot-from-the-hip approach taken by a large majority of program committee members.<br /><br />It is true that I'm fortunate to be at a stable company and working on a stable product team. If I were to join an early-stage startup, or even a new project team at Google, then my career would be subject to vastly different pressures in terms of raising funding, hiring good people, settling on the right product direction, and generally just being lucky. In such a case, I'd agree that the degree of entropy would be much higher than the relatively stable situation I find myself in now. I still claim that those sources of randomness are qualitatively different than the process of academic peer review, but perhaps we can agree that there would be just as much uncertainty.Matt Welshhttps://www.blogger.com/profile/07077674014671176946noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-30910614700864929972016-06-08T12:57:09.982-07:002016-06-08T12:57:09.982-07:00Matt --
There are many things I agree with in thi...Matt --<br /><br />There are many things I agree with in this post. Peer review does appear to be a semi-random process, where the bias may be in the right direction (we both appear to think that it is), so over longer time scales the law of large numbers applies, but over shorter time scales the results can be frustrating. (And unfortunately for many graduate students or even young faculty, the short time scale is rather important.) <br /><br />Where I think I'd disagree is where you say: "However, I believe the industry process is far less arbitrary." And I'm afraid I don't think you really back up that statement with any actual evidence. (As a counterpoint, people at Microsoft Research Silicon Valley found Microsoft's decision to close their lab and fire almost all the people in it rather arbitrary.) <br /><br />You may be experiencing your own personal selection bias. Google appears to be (for those of us on the outside) a well-run company that has enjoyed a long period of success. But that may not be a permanent condition, and of course many companies have far less pleasant circumstances. <br /><br />So I'd happily agree that various aspects of peer review are frustrating, and certainly I've had reviews that I thought were not only misguided, but ignorant and wrongheaded. Industry is indeed a different kind of game, but I'm unconvinced that we can ascribe to it a higher rationality. <br />Michael Mitzenmacherhttps://www.blogger.com/profile/02161161032642563814noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-42476728089321297732016-06-07T17:48:20.516-07:002016-06-07T17:48:20.516-07:00Isn't it pretty obvious, at least for the acad...Isn't it pretty obvious, at least for the academic case?<br /><br />I'm sure it is to the audience Matt intended for this particular post.<br /><br />In particular, pretty much everybody who wants an academic career wants academic freedom. Unfortunately the only way to have this on a long-term basis is to get tenure (*) and in order to have freedom to take on projects bigger than a single person can handle you need both that and research grants. Grants are made based on direct peer review and tenure decisions are (purportedly) based on publication record which is an aggregation of many peer review decisions. Hence the article.<br /><br />Again, everything in the preceding paragraph should be pretty obvious and isn't nearly as clever as that t-shirt, which I am now lusting after.<br /><br />(*) I'd argue that externally-fellowshipped grad students actually have more freedom than any professor does -- but only for a few years.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-70979829977447970812016-06-07T16:18:05.622-07:002016-06-07T16:18:05.622-07:00> Rather than the success or failure of a proje...> Rather than the success or failure of a project coming down to the dreaded Reviewer #3, it comes down to the team's ability to execute, targeting the right market, and attracting users.<br /><br />This is broadly true in the industry, especially at the macro scale, as business survival depends on it. <br /><br />With that said "Reviewer #3" does still make an appearance in companies (especially with siloed orgs), they're just called "Irritating VP of X" or "Obstructionist Manager #555". <br /><br />The military has a lovely term for these people: "Blue Falcons". The etymology of the term is left an an exercise for the reader. Anand Guptahttps://www.blogger.com/profile/15483238813865853357noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-84057054737177732172016-06-07T16:16:27.375-07:002016-06-07T16:16:27.375-07:00NIPS experiment endorses everything you said.
htt...NIPS experiment endorses everything you said. <br />http://blog.mrtz.org/2014/12/15/the-nips-experiment.htmlKlerisson Paixaohttps://www.blogger.com/profile/09598329932693791273noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-76365259695468917162016-06-07T13:46:11.793-07:002016-06-07T13:46:11.793-07:00Matt, you wrote, "But aspiring academics shou...Matt, you wrote, "But aspiring academics should [know] how much of your success hinges on the purely stochastic nature of the process." Just curious, what is your definition of academic success (and failure)? And what is your definition of non-academic, professional success (and failure)? It might be useful to define these terms before debating about the extent to which "the process" affects the outcomes.Prabal Duttahttps://www.blogger.com/profile/00975098265026146680noreply@blogger.com