tag:blogger.com,1999:blog-9186457242428335144.post4721153460313895093..comments2024-03-28T00:36:13.790-07:00Comments on Volatile and Decentralized: Who pays for conference reviews?Matt Welshhttp://www.blogger.com/profile/04255792550910131960noreply@blogger.comBlogger41125tag:blogger.com,1999:blog-9186457242428335144.post-70113348041689575752010-11-26T15:02:12.311-08:002010-11-26T15:02:12.311-08:00A few people have suggested crowd-source review pr...A few people have suggested crowd-source review process; make everything public, submissions, reviews (double blinded). I might add that even there should be some thing as reviews on the reviews (may be a point system). Best work would automatically get to the top.<br /><br />Why have most of the commentators skipped discussing on this. I am new to the academia, so I might be missing something here.Faisalhttps://www.blogger.com/profile/04120879750666135279noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-81441574846812882192010-05-18T23:12:14.123-07:002010-05-18T23:12:14.123-07:00Very interesting post. I happened onto your blog a...Very interesting post. I happened onto your blog after a frustrating evening reviewing a terrible paper for a computational linguistics conference. <br /><br />I don't know if charging is the right solution, but I agree it is a problem. <br /><br />I take putting down my comments very seriously, and I don't mind reading the papers as much as I mind the time it takes to give detailed discussions of why they are so bad when they are really bad. Maybe there should be an option when reviewing a paper that's really crap to let you select "skip written review and proceed straight to recommendation of rejection".paulhttp://www.taulpepper.comnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-17725438312122405002010-03-23T19:39:09.983-07:002010-03-23T19:39:09.983-07:00charge starting with the author's second submi...charge starting with the author's second submission, like airlines charging for bags.<br />$1000 for the second submission. $2000 for the next one. And so on: a soft disincentive to multiplying submissions.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-90893635305260065642010-03-17T13:44:46.298-07:002010-03-17T13:44:46.298-07:00Re:Re: "Untenured faculty would be very unlik...Re:Re: "Untenured faculty would be very unlikely to rate anything down since there could be retribution later"<br />Really? Interesting you claim this. If a review is objective, correct and comprehensive, which kind of retribution should they be afraid of? The retribution of the caste? And how often untenured faculty is TPC member of an important conference? Pretty rarely.<br /><br />"Senior, well-established faculty could write whatever BS they like and not care about the reputation hit."<br />I think it would be the exact contrary. At least to my eyes, they would lose all their reputation, and I would not hesitate in letting others know their behaviour (this already happens widely, let's be honest). And if they really do so (write whatever BS they like), well, which kind of well-established faculty are they? We should all remember here that the final target of our work is the good of science, not the good of our ego over others...<br /><br />"I think it should be up to the program chairs to make sure that the TPC members are doing a good job of writing comprehensive reviews. At least at high quality conferences this is almost universally the case."<br />I agree with you on this point. But honestly I have seen this not happening too many times. I also would like to add that in most of the high quality conferences you mention, TPC members are reviewing papers involving also other TPC members. And this happens without double-blind review procedure. This should not happen, and I have already explained in my previous post why... curious you kind of skipped over that part ;-)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-43239491107394590632010-03-17T11:43:13.005-07:002010-03-17T11:43:13.005-07:00I am not sure getting money into the submission pr...I am not sure getting money into the submission process is the right move. Better papers in conference submission should be the result of social pressure, not lack of money. Going towards more public refereeing processes (even a 'wall of shame' for disastrous papers with very low scores) seems like the right direction. <br /><br />Now, the idea of charging for submitting papers, as developed in the post, is related to (and seems to ignore) the fact that even if PC members invest time and energy in the process, they also get benefits. These are harder to quantify, of course, but such benefits exist and are significant. For instance, I suppose that things like serving in the committee of top conferences are considered as valuable in job promotions (at the very least they look good in a CV). So not everything can be reduced into money, I think.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-40232809524243672152010-03-17T08:16:01.345-07:002010-03-17T08:16:01.345-07:00Not crazy about the pay for submit idea. It might...Not crazy about the pay for submit idea. It might work in top systems/networking conferences, but certainly wouldn't work in areas like Theory. It would be nicer to have a solution that's widely applicable. How about the review process is still anonymous but if your paper gets 3 reviews below some terrible threshold then your names are revealed along with the accepted papers. It would be something like a wall of shame.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-38334099518049770342010-03-16T18:43:22.748-07:002010-03-16T18:43:22.748-07:00A few more responses.
Re: "What if the paper...A few more responses.<br /><br />Re: "What if the papers can be rejected at the abstract stage itself..." I like this idea and wish it were implemented, at least to prune some of the obvious rejects (papers that are submitted to the wrong conference). The problem is that the abstract often promises much more than the paper delivers. Maybe a two-page abstract would suffice and then we'd invite full papers from those that pass that stage. But this seems like more work than just taking the full papers up front.<br /><br />Re: Signing reviews. This I'm not so sure about. Untenured faculty would be very unlikely to rate anything down since there could be retribution later. Senior, well-established faculty could write whatever BS they like and not care about the reputation hit. I think it should be up to the program chairs to make sure that the TPC members are doing a good job of writing comprehensive reviews. At least at high quality conferences this is almost universally the case. So this does not seem to be a major problem.<br /><br />Re: Charging for submissions yields expectation on the authors' part of the quality of reviews. Absolutely. This would be a big problem as authors would feel entitled to the reviews that they "paid" for. One of many problems with my proposal.<br /><br />Re: Grad student needs to pay to submit papers. I have never suggested that a grad student pay for this out of pocket. A paper submission fee should absolutely be paid by the advisor out of research funds, just like conference registration and travel expenses.Matt Welshhttps://www.blogger.com/profile/04255792550910131960noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-70225700381451651192010-03-16T10:45:22.487-07:002010-03-16T10:45:22.487-07:00What if the papers can be rejected at the abstract...What if the papers can be rejected at the abstract stage itself...I am sure that does not take an hour...If that gets through then we could have an extended abstract then the final paper...As CS researchers we can hierarchically review papers the same way we hierarchically encode images, videos, and mapsAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-85672988035631226982010-03-16T09:57:31.425-07:002010-03-16T09:57:31.425-07:00"My 2 cents" - you clearly don't und..."My 2 cents" - you clearly don't understand how this works. Serving on one TPC a year, or one a decade, does not give you any more time to review the papers. There is usually a pretty short window between the paper deadline and when the reviews are due - anywhere from 3-6 weeks in my experience. I don't know about you, but I'm a pretty busy person and fitting in 25+ paper reviews in a few weeks is hard to do. <br /><br />Besides, if I choose to serve on one TPC a year, that doesn't reduce the overall reviewing burden for the rest of the community, which is what my post was all about.Matt Welshhttps://www.blogger.com/profile/04255792550910131960noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-30336609293737864522010-03-15T15:02:51.994-07:002010-03-15T15:02:51.994-07:00Well here's a solution - why don't you sto...Well here's a solution - why don't you stop being on so many TPCs? Say you choose to be on 1 TPC a year, that way, you get plenty of time to review all the papers assigned to you, do a good job reviewing and stop griping about the 'extra' load you have to incur. What's that you say? You have to be on multiple TPCs for tenure etc etc reasons?<br />Too bad. I mean seriously, the previous post on reviewing by the author had atleast some good take-away points for authors (although I strongly disagreed with the premise), but this post takes the cake!<br /><br />The problem doesn't lie with authors or reviewers, but the way the system is setup - we should cut down on the number of conferences we have (networking), award quality over quantity and stop playing a ranking-game of pushing a conference above others (Sigcomm). <br /><br />My 2 cents.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-52411886667061365652010-03-14T01:27:18.279-08:002010-03-14T01:27:18.279-08:00Very interesting post.
About the double-blind revi...Very interesting post.<br />About the double-blind review process: I personally think it should be applied in all conferences (at least the most important IEEE and/or ACM ones). Point blank.<br />Honestly, too many times you see and/or hear about TPC members accepting other TPC members papers. This looks a lot like exchange of favors, which ultimately does not contribute to science... Double-blind review costs nothing in financial and organizational terms, and it improves dramatically fairness and equality of treatment. I really don't understand why it is not applied, or probably I have already understood the reason why (read above), and I really don't like the answer...<br />Do you really want to improve quality of the reviews done by the reviewers? One suggestion: force the reviewer to put his name at the end of the review (as it is actually already done in some journals). In this case you would have to really comprehend what you have been reading, and provide a comprehensive review of it.<br />With double-blind review, and signed reviews, then I would have no problem in paying the fee you propose.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-91892830547351876302010-03-12T21:36:16.943-08:002010-03-12T21:36:16.943-08:00Hi,
I was wondering what your thoughts were on th...Hi,<br /><br />I was wondering what your thoughts were on the following. <br /><br />1) If one conference charges for submissions, then shouldn't there be some standard of what makes an appropriate review? Wouldn't the authors develop much higher and more homogeneous expectations? ("I paid $x to submit my paper, I should get at least a one-page review with constructive criticism" with the reviewer replying "I disagree with Theorem 1 and there isn't any point in reviewing the paper further" Author: "Give me a refund!") Wouldn't this open the door to more work for the conference organizers, who'd have to check the work of the reviewers to make sure it's up to par?<br /><br />2) Maybe the paper selection process should take into account the ratings of previous years' papers by one or more of the authors. If a paper got consistently poor reviews (not just one), then its authors would have a harder time getting accepted the following year. <br /><br />3) Another thing that might help deter people from submitting half-baked reviews would be to post the ranking of all the submissions online, including their total score. Papers wouldn't get a low total score unless they had been found deficient by several reviewers. That would be a clear sign that the authors submitted the paper too early. <br /><br />4) Ultimately, though, some conferences do enjoy publicizing their low acceptance rates. They might indirectly benefit from the submission of half-baked papers, in the same way that colleges like to see under-qualified students apply because they can say later they are very selective. <br /><br />I liked your post, though.Aureliehttp://engineered.typepad.comnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-53926787687822608332010-03-09T22:48:34.050-08:002010-03-09T22:48:34.050-08:00I thought the intention of research was to advance...I thought the intention of research was to advance the state of science ;-).<br /><br />Now not only will i have to work on a grad student salary, I will also need to pay to advance the state of the art so others can benefit? <br /><br /><br />P.S. I mean the above comment in jest. However, the effect of pushing the reviews for public scrutiny accomplishes both objectives; a reduction in run-of-the-mill papers and a check on review quality.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-91594706377032376782010-03-09T14:45:22.022-08:002010-03-09T14:45:22.022-08:00I like Adam Marcus's suggestion to PPP-adjust ...I like Adam Marcus's suggestion to PPP-adjust the fee. That should address the silly objections like this will decimate computer science in Mexico. Of course, a lot of papers a co-authored by people from different countries, so I'm not sure how that would work in practice.<br /><br />As an aside, it is ridiculous to say money is the reason that the top CS conferences are dominated by a few institutions. There are plenty of very wealthy universities just in the US that rarely if ever get papers into the top conferences. Lack of funding is not the explanation, in a field where all you need is a $300 laptop and an Internet connection.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-73956931676108719232010-03-09T11:04:20.708-08:002010-03-09T11:04:20.708-08:00Interestingly, if this proposal was implemented, i...Interestingly, if this proposal was implemented, in say a single top conference, and it proved effective, it would make that conference look less competitive by comparison to other conferences due to higher acceptance rate. And you know how much people love to worship numbers...Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-75158881922134450912010-03-09T10:14:03.579-08:002010-03-09T10:14:03.579-08:00this seems a good idea but I doubt it will magnify...this seems a good idea but I doubt it will magnify a "poverty gap" between well known high reputation researchers and vast majority guys who are not so confident of their works but they want to try at least. I don't mean that half barked papers are welcomed but nobody can guarantee acceptance of their works especially papers could be rejected by many reasons authors can not control, the rejection will significantly combat motivation to submit paper again to the same conference. then good conference may lose "diversity" which I think is much more harmful to the whole community. I am a PhD student and funding situation in our group is not satisfying. I think I have some good works done and want to try prestige conference, if there is "submission fee",I may consider to submit to a journal instead. I don't want to exaggerate my works but at least it is worth one hour review and may give some new insight to research community.Webcrafthttps://www.blogger.com/profile/09232078592880811827noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-69612242207142101222010-03-09T06:33:46.977-08:002010-03-09T06:33:46.977-08:00One solution to the issue of whether $500 is too e...One solution to the issue of whether $500 is too expensive in some country is to PPP-adjust (http://en.wikipedia.org/wiki/Purchasing_power_parity) the fee. If $500 reimburses you in the US, and we also decide that $500 is a disincentive to submit bad work, then the disincentive could be PPP-adjusted to be lower in countries where that amount would buy you more.Adam Marcushttp://people.csail.mit.edu/marcuanoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-2766532066987837542010-03-09T06:32:34.859-08:002010-03-09T06:32:34.859-08:00I'm a new person here and want to provide a di...I'm a new person here and want to provide a different view. Professor of my friend in Biological Science (BS) said to me that, in BS:<br /><br />- First, conferences mean nothing. Only journal papers are counted.<br />- Second, journal reviewers often get a small amount of money from the journal that invites them (CS reviewers do not). However, professors do not review papers for money. The amount of money is small when compared with the one they can get from other academic work. But, there is still money. And, the reviews still have high quality.<br />- Third, authors often have to pay a small amount of money to the journal after when the paper is accepted (like what CS conferences require, but CS journals do not). However, professors often do not allow students to submit rubbish papers since these papers can ruin their fame, although the amount of money paid to the journal is very small compared with their budget.<br />- Four, most journals have turn-around times less than 2 months and short turn-around times do not mean that they are not prestigious. (CS journals often require 6 months to 2 years. Top CS conferences are often longer than journals in BS, and perhaps all other fields).<br /><br />I think that these facts may be useful for us, CS people. Review work in other fields seems to be much less heavy than ours :-)<br /><br />Besides, the case of Leon in Mexico is actually much happier than many other cases in poorer countries. That is a dream case for me and my friends although I am studying in the best institutions in a rich country. Don't need to complain that much, ha ha! (do not take it serious ;-) )Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-65299995704040462232010-03-09T06:25:40.477-08:002010-03-09T06:25:40.477-08:00I'm a new person here and want to provide a di...I'm a new person here and want to provide a different view. Professor of my friend in Biological Science (BS) said to me that, in BS:<br /><br />- First, conferences mean nothing. Only journal papers are counted.<br />- Second, journal reviewers often get a small amount of money from the journal that invites them (CS reviewers do not). However, professors do not review papers for money. The amount of money is small when compared with the one they can get from other academic work. But, there is still money. And, the reviews still have high quality.<br />- Third, authors often have to pay a small amount of money to the journal after when the paper is accepted (like what CS conferences require, but CS journals do not). However, professors often do not allow students to submit rubbish papers since these papers can ruin their fame, although the amount of money paid to the journal is very small compared with their budget.<br />- Four, most journals have turn-around times less than 2 months and short turn-around times do not mean that they are not prestigious. (CS journals often require 6 months to 2 years. Top CS conferences are often longer than journals in BS, and perhaps all other fields).<br /><br />I think that these facts may be useful for us, CS people. Review work in other fields seems to be much less heavy than ours :-)<br /><br />Besides, the case of Leon in Mexico is actually much happier than many other cases in poorer countries. That is a dream case for me and my friends although I am studying in the best institutions in a rich country. Don't need to complain that much, ha ha! (do not take it serious ;-) )Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-36192409159760653552010-03-09T06:10:28.819-08:002010-03-09T06:10:28.819-08:00I think people are missing the point here. All big...I think people are missing the point here. All big conferences, sigcomm, infocom have part of their papers coming from undergrads in random chinese/indian institutions that are just a waste of time for reviewers. For some conferences this can be really high (50% of papers). <br /><br />I think a tax like this has the potential to stop this stampede.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-34305182172663584242010-03-09T05:36:12.242-08:002010-03-09T05:36:12.242-08:00I think the real problem with this approach is, as...I think the real problem with this approach is, as other have pointed out, that the social incentives change drastically once money gets into the picture. If you must pay $500 to submit a paper to a conference, then why wouldn't you just pay $5000 to submit ten crappy papers for a low-cost peer review by a bunch of experts? After all, you pay for the service. In most cases, it would be much more cost effective to pay $500 a pop to submit your early/crap papers for an early review than to have professors from your own institution to pre-review the same papers.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-91369494296451411272010-03-09T04:47:05.596-08:002010-03-09T04:47:05.596-08:00Leon - you make some good points. Clearly I am com...Leon - you make some good points. Clearly I am coming at this from a different perspective, but keep in mind that this is not a serious proposal (as in, I'm not planning to implement it :-) <br /><br />Most conferences have discounted registrations and travel awards for students and I could imagine the submission fee being reduced or waived in certain circumstances, such as financial hardship. But again, I am just not convinced that a submission fee would stand in the way of getting one's work published.Matt Welshhttps://www.blogger.com/profile/04255792550910131960noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-45397959126021323012010-03-09T01:55:40.915-08:002010-03-09T01:55:40.915-08:00"As has been pointed out already, it's mu..."As has been pointed out already, it's much more expensive to actually attend the conference anyway."<br /><br />So in my understanding, it should be only for people with the means to attend, I already wrote with a nice example how this cost would affect students, 500 USD is not a negligible fee, it atones for a 25% increase of the overall cost, which is considerable. Let's see what you would do if your food suddenly started costing 25% more ;). there is a nice parody about this post with states something interesting, with all the conferences there are out there, well, conference needs their submitters more than submitters need the conferences, if a conference would start charging, the number of submissions would decrease dramatically, thus the conference itself might be in risk of disappearing. Academia is about sharing information, and it is obvious that among that sharing there would be people that will share not that useful things.<br /><br />Charging this fee would give people this 2 messages: "If you have money, you are entitled to write as many crappy papers you like"<br />and "Would you review my crappy paper for free, - No - would you do it for 500 US -yes-, ok, we know what you are, now we are just discussing the price ;)"Leon Palafoxhttps://www.blogger.com/profile/00162175198005112381noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-30022841263672087792010-03-09T00:19:05.644-08:002010-03-09T00:19:05.644-08:00Get a grip, people. Computer Science is the least ...Get a grip, people. Computer Science is the least capital-intensive science or engineering discipline there is, period. The countries that are under-represented in CS research are not under-represented because of lack of funding, and asking people to cough up a few hundred bucks to get their paper reviewed is not going to decimate computer science research in Mexico or anywhere else. As has been pointed out already, it's much more expensive to actually attend the conference anyway.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-78673865565009639412010-03-08T20:28:49.133-08:002010-03-08T20:28:49.133-08:00Do PC members do all the work for conferences over...Do PC members do all the work for conferences over the official working hours that the university expects? If not, they are already being paid by the university for that effort, presumably because the university thinks it's a worthwhile pursuit. Also $500 may seem like a modest amount for researchers in most American universities, though I don't think that's at all true universally, particularly in developing countries.Anonymousnoreply@blogger.com