Why not make authors pay to submit papers to conferences?
Serving on a program committee takes a tremendous amount of time. So, one of the frequent complaints that TPC members make is when authors submit half-baked, clearly below-threshold papers a conference just to get some reviews back on their work. Personally, I feel little responsibility to write detailed reviews on papers that are clearly in the "Hail Mary" category, but I still have to read them, and that takes time. Not to mention the long-term psychological damage incurred by having to read a slew of crappy papers one after the other... I'm still in therapy after IPSN 2007 :-)
The problem is that submitting a paper to a conference is free: all it takes is a few clicks of the mouse to upload your PDF file. (Of course, I'm not accounting for the cost of doing the research and writing the paper itself.)
Let's estimate the costs associated with serving on a program committee and reviewing a stack of papers. I spend about an hour reading and writing a review for each paper that I am assigned. A highly competitive conference will assign 25 papers (or so) across one or more reviewing rounds to each TPC member, equating to roughly 25 hours of my time. At my current salary, that is worth around $1900 (give or take). Then there is the PC meeting itself. This will typically involve two days' worth of work plus travel -- let's estimate 3 full days of labor, plus airfare and hotel, adding up to another $2500.
So, with a program committee of 18 people, that works out to around $79,000 to review something like 150 paper submissions. In other words, to recoup its costs, the conference should charge authors $500 just to submit a paper. This seems to make a lot of sense.
Of course, imposing this kind of a fee would no doubt drastically reduce the number of papers that are submitted. But this seems like a good thing: it would reduce the workload for TPC members, allow conferences to operate with smaller, more focused program committees, and vastly improve the quality of the submitted papers. It would potentially also improve the quality of the reviews, since TPC members would now be paid for their time. Although the financial incentive is not that great (e.g., my hourly rate for consulting is something like 5 times my regular salary), getting paid should encourage TPC members to take the process more seriously.
The only downside I can see is people who sign up for a slew of program committees and become "professional paper reviewers", but TPC chairs would clearly have to balance this against the research credentials of the people being asked to serve. Note that many journals impose author fees for publication of the paper, but presumably you are willing to pay once you have done all the work to get the paper accepted. And conferences expect authors to show up at the conference to present the paper, which can get to be pretty expensive as well. But it seems crazy to me that the research community provides this free paper reviewing service with no negative ramifications for submitting totally unpolished work.
Subscribe to:
Post Comments (Atom)
-
The word is out that I have decided to resign my tenured faculty job at Harvard to remain at Google. Obviously this will be a big change in ...
-
My team at Google is wrapping up an effort to rewrite a large production system (almost) entirely in Go . I say "almost" because ... -
I'm often asked what my job is like at Google since I left academia. I guess going from tenured professor to software engineer sounds l...
A pay-per-submission model would just skew the playing field even more towards those who have lots of spare grant money to throw at hail-mary paper submissions, whereas some folks doing good research wouldn't even be able to afford to submit at all.
ReplyDeleteVery interesting idea.
ReplyDeleteIt will never get implemented of course (the howls of protest would be something)- but I am curious how it would change the process.
Clearly far fewer papers would get submitted, only if you were almost SURE you had a very good chance or just had a lot of money.
Indeed fewer papers might get submitted than available conference slots.
Overall, it just seems unrealistic, it would greatly disadvantage younger and poorer institutions in favor of established players.
Now a nominal fee of $20 might be doable: almost everyone could afford it and it gives some emotional incentive to not submit frivolously.
I simply don't buy the argument that forcing authors to pay to submit to a conference gives an advantage to "richer" institutions and research groups. $500 (or even $1000) is a tiny fraction of the costs associated with conducting a research project, at least in systems. You need to pay grad students and postdocs, buy computers and other hardware, pay for travel, etc. $500 is nothing.
ReplyDeleteThe proposed fee seems to be roughly on par with conference registration fees. If the idea is only to disincentivize people from submitting incomplete work, and not to directly reimburse reviewers and PC members, then why not charge a fee to submit a paper, and then let that fee go toward conference registrations for authors of accepted papers. This way you're only charging people whose papers will be rejected, but incurring minimal extra cost on work good (relevant?) enough to be accepted.
ReplyDeleteWell, there is one negative ramification for submitting totally unpolished work: it hurts the reputation of the authors, at least in the eyes of the PC members. However, this doesn't apply to venues with a double-blind review process, which is an unfortunate consequence of using double-blind reviewing.
ReplyDeleteTo the second Anon: I think charging $20 for submissions might actually be counter-productive. As it stands, there is negative peer pressure for submitting totally inappropriate work. By charging some nominal fee ($20), you are essentially valuing the work of the PC at $20/paper, and might actually end up reducing the pressure on authors to submit good work.
I like the idea of a fee that's credited if you're accepted.
ReplyDeleteIt reminds me of wine tasting. I don't have a problem with a modest fee as long as it's credited towards any wine that I purchase.
@Matt
ReplyDelete"... You need to pay grad students and postdocs, buy computers and other hardware, pay for travel, etc. $500 is nothing."
I'm not sure I agree that $500 is nothing, but that is clearly relative. If $500 is nothing, then it isn't enough of a disincentive. I don't think you can make a disincentive that won't give an advantage to the rich folks. In other words, either the disincentive is significant and it hurts the little people, or it isn't significant and has little or no effect on the problem.
Matt, your scheme reminds me of penny auctions, where there are lots of losers, and relatively few winners.
ReplyDeleteThere is the problem of reviewers rubbishing papers they just don't like, whether it's because they have a beef with the authors, the author's institution, or the technical content in the paper contradicts their own work, etc.
I can't remember the amount of times I personally have received reviews where the majority of the reviewers give the paper an excellent rating and one gives it a terrible rating. Not sure if this is simply because the reviewer simply doesn't understand the significance of the work or what (possible as I’m a mathematician in a field dominated by computer scientists and electronic engineers).
Then there is the problem where the conference/journal is over subscribed with excellent submissions. Often the decision on what should be cut in that case is based on the number of submissions in the particular sub category, not on overall, publication wide, merit. Should an author be refunded if this is the case or is it just the luck of the draw (the old boys network problem could be exacerbated by this)?
If this was to be implemented the reviewers would have to be somewhat accountable, but accountable to whom?
All that said I agree that spending time reviewing terrible submissions is a problem. I have a folder full of papers I need to review by the end of the week, and, from a quick glance, less than half are written in intelligible English. Unfortunately the reviewer instructions say not to discount a paper due to bad English/grammar/spelling alone, as the facility of a copy editor is available for some authors (post review only though). As a result I have written off the entire week as to getting any *real* work done.
Great post Matt. As a frequent TPC member I'd be happy to just get travel costs covered (or even the hotel... anything). Anon #1 has a good point, but it's not clear that people without a lot of spare grant money are publishing at e.g. OSDI/SOSP anyway.
ReplyDeleteAlso I recently wrote a couple blog posts on related topics that may be interesting to folks here:
http://blog.regehr.org/archives/6
http://blog.regehr.org/archives/4
I am reminded of the famous story of the day care center that found that by imposing a fee for late pickups, even more parents came late. With the fee, there was no more social stigma attached to being late; it was just a straight transaction of money for a few minutes of extra care. If your hypothesis holds -- that $500 is really a trivial fee for most institutions -- maybe you'll see even more bad submissions. Authors might jump at the chance to buy some carefully considered feedback on their work from a half-dozen really smart and relevant people for a mere $500. Wouldn't you?
ReplyDeleteLots of interesting comments here.
ReplyDeleteOne question is whether the paper submission fee should be used to directly reimburse the TPC members or to offset things like the conference registration fees. I'm not sure which model makes the most sense, though the latter would make the conference registration cost less which makes it more accessible for "poorer" researchers to attend, thereby enriching the set of people who actually attend the conference. That seems like a good thing.
I'll agree that $500 is not "nothing" and I think it does serve as enough of a disincentive to send in a paper that is totally underbaked. I wouldn't be willing to pay the fee for a student's submission that was simply not ready for review. Sometimes a student will just submit a half-baked paper out of desperation, or for the need to feel that they have accomplished something (deferring the feelings of failure when the paper is inevitably rejected - or as an excuse to take three months' vacation until the reviews come back). The problem is that there is a real cost in terms of program committee time and effort, but our current system makes no attempt to recoup this.
There are many times I would have liked to write in a paper review: "YOU HAVE WASTED MY TIME IN SUBMITTING THIS PAPER TO THIS CONFERENCE" in all caps and a attach a bill for my reviewing services. But I doubt that would fly.
I have a hard time believing that even the richest institutions are going to be willing to "spam" a conference with a lot of half-baked papers because they can afford to pay.
Finally - the idea that by paying to submit the author somehow feels entitled to getting back thoughtful reviews is interesting. Certainly reducing paper submission to a financial transaction is fraught with risks, both legal and social.
In my opinion the solution to both crappy papers been submitted and to crappy reviewers done is to make the process totally transparent, both authors and reviewers should be visible. In such a way a bad paper will give a bad name to the authors, and a good review will work as an added credential for the scientific maturity of the reviewer. No need to throw money on the equation, research is already been crippled enough by that. What you need is honest scientific ethics that allows you to sign with your name every research finding you claim and any criticism you make.
ReplyDeleteInteresting thread. I agree with the above suggestion of making everything public. On a related note, I found one hour per paper for reviewing + writing review to be on the low side. Matt -- is this your average, min or max?
ReplyDeleteIf I had a good systems paper which would have typically involved more than 1000 man hours worth of effort, I would expect the reviewer to put in more than an hour.
There is a problem with your submitting proposal, you basically undermining countries where research funds are limited, let me put as an example my home country (Mexico), doing research is hard since there is no grant system neither big budgets coming from the government, we had to continuously struggle to get by and doing research, buying equipment was done like once every 4 years, and conference travel expenses (including registration fee and plane ticket), more than once had to be paid by us, the students, now that I am in Tokyo is entirely different. The point is, this system would basically erase out of the map most Universities than are on this line, you have to admit that for doing good computer science research most times a computer and a decent library does the trick, now, if you add that submission fee, conference prices would sky rocket, just submitting for 3 conferences a year would add up to 1500, more than what a pretty good computer would cost (Lets say an Alienware Aurora, tagged in 1599 USD), I would rather have the computer and submit to a Journal.
ReplyDeleteA few responses.
ReplyDeleteConferences with single-blind reviewing don't seem to get any fewer crappy submissions. It seems that authors don't seem to have a problem submitting a half-baked paper even if their name is attached to it - perhaps because only the TPC knows their identity and there's little social pressure.
In terms of reviewing time - 1 hour is a rough average. Some papers take more, perhaps 90 minutes. Others take substantially less, if they are easy to dismiss. Since you posted anonymously I don't know who I'm responding to, but I would suggest that you haven't been on many program committees with heavy reviewing loads if you think that TPC members should spend much more than an hour on each paper. Or maybe I'm just really efficient, but I think I do a pretty thorough job.
Leon - I appreciate that the fee idea increases the financial burden, but my argument is that this is a small fraction of the cost of doing the research in the first place. As you point out the conference travel and registration already cost several times what my proposed submission fee would be. I don't think that this fee would cause prices to "skyrocket" -- on the contrary, if they are used to reduce the registration fees that benefits everyone.
Also, let me put forth that it has been a very long time since I saw a paper from a Mexican university in any top conference in my area, suggesting that imposing such a fee would not have any visible impact on the number of papers published from such countries. (Please don't take this the wrong way -- I am just observing that there seem to be other problems in this case apart from the potential for a paper submission fee.)
I am certainly not suggesting that *all* conferences impose a submission fee - I can easily imagine many second- and third-tier venues, and smaller workshops, growing the community base by making submissions free. I'm more concerned with the top-tier conferences, like SIGCOMM, which have the problem of burned-out TPC members doing a less-than-stellar job reviewing the papers due to the sheer load of crap that gets submitted. Michael Mitzenmacher has a recent blog post about this: http://mybiasedcoin.blogspot.com/2010/03/sigcomm-papers.html
Hey Matt:
ReplyDeleteI do know that in some fields Mexican Universities do not show that much, it is more a culture problem that may be rather discussed over a beer than in a reply for a blog.
But what you say is just adding to my point, Mexican Universities students, lacking grants and research expenses have a really hard time attending conferences, now, adding a fee for the sole fact of submitting is not going to help that. Just to make some quick numbers an average Computer Science Grad Student in Mexico receives about 900 USD a month, with no option to do any part time job, since the premise of the scholarship is that you only focus yourself in doing research. Lets assume only for the sake of calculation that living expenses add up to 500 USD (I am being quite conservative), that leaves us with 400 USD per month, that would be 4000 USD a year (yes the scholarship is not granted on holiday months), now imagine the conference is the NIPS, which was recently held at Vancouver, a quick web research shows us that an airplane ticket goes around 1000 USD, plus registration and lodging fees, lets say it goes to a well 2000 USD for the entire thing. Our average Grad Student may be able to go to 2 conferences with his own means in one year, add up your review fee, and there you have it, you have just by numbers stripped our theoretical student from going to 2 conferences.
Maybe there should be special cases or special considerations, maybe if the paper is obviously a Hail Mary you could charge your fee, that would encourage people to write good papers instead of recycled or hastily written ones. And assure you that if you wasted your time in a bad paper at least you get paid by that,and if the paper was good, well you were honored in being one of the first persons to review it and use it afterward, and you can feel good about yourself.
Do PC members do all the work for conferences over the official working hours that the university expects? If not, they are already being paid by the university for that effort, presumably because the university thinks it's a worthwhile pursuit. Also $500 may seem like a modest amount for researchers in most American universities, though I don't think that's at all true universally, particularly in developing countries.
ReplyDeleteGet a grip, people. Computer Science is the least capital-intensive science or engineering discipline there is, period. The countries that are under-represented in CS research are not under-represented because of lack of funding, and asking people to cough up a few hundred bucks to get their paper reviewed is not going to decimate computer science research in Mexico or anywhere else. As has been pointed out already, it's much more expensive to actually attend the conference anyway.
ReplyDelete"As has been pointed out already, it's much more expensive to actually attend the conference anyway."
ReplyDeleteSo in my understanding, it should be only for people with the means to attend, I already wrote with a nice example how this cost would affect students, 500 USD is not a negligible fee, it atones for a 25% increase of the overall cost, which is considerable. Let's see what you would do if your food suddenly started costing 25% more ;). there is a nice parody about this post with states something interesting, with all the conferences there are out there, well, conference needs their submitters more than submitters need the conferences, if a conference would start charging, the number of submissions would decrease dramatically, thus the conference itself might be in risk of disappearing. Academia is about sharing information, and it is obvious that among that sharing there would be people that will share not that useful things.
Charging this fee would give people this 2 messages: "If you have money, you are entitled to write as many crappy papers you like"
and "Would you review my crappy paper for free, - No - would you do it for 500 US -yes-, ok, we know what you are, now we are just discussing the price ;)"
Leon - you make some good points. Clearly I am coming at this from a different perspective, but keep in mind that this is not a serious proposal (as in, I'm not planning to implement it :-)
ReplyDeleteMost conferences have discounted registrations and travel awards for students and I could imagine the submission fee being reduced or waived in certain circumstances, such as financial hardship. But again, I am just not convinced that a submission fee would stand in the way of getting one's work published.
I think the real problem with this approach is, as other have pointed out, that the social incentives change drastically once money gets into the picture. If you must pay $500 to submit a paper to a conference, then why wouldn't you just pay $5000 to submit ten crappy papers for a low-cost peer review by a bunch of experts? After all, you pay for the service. In most cases, it would be much more cost effective to pay $500 a pop to submit your early/crap papers for an early review than to have professors from your own institution to pre-review the same papers.
ReplyDeleteI think people are missing the point here. All big conferences, sigcomm, infocom have part of their papers coming from undergrads in random chinese/indian institutions that are just a waste of time for reviewers. For some conferences this can be really high (50% of papers).
ReplyDeleteI think a tax like this has the potential to stop this stampede.
I'm a new person here and want to provide a different view. Professor of my friend in Biological Science (BS) said to me that, in BS:
ReplyDelete- First, conferences mean nothing. Only journal papers are counted.
- Second, journal reviewers often get a small amount of money from the journal that invites them (CS reviewers do not). However, professors do not review papers for money. The amount of money is small when compared with the one they can get from other academic work. But, there is still money. And, the reviews still have high quality.
- Third, authors often have to pay a small amount of money to the journal after when the paper is accepted (like what CS conferences require, but CS journals do not). However, professors often do not allow students to submit rubbish papers since these papers can ruin their fame, although the amount of money paid to the journal is very small compared with their budget.
- Four, most journals have turn-around times less than 2 months and short turn-around times do not mean that they are not prestigious. (CS journals often require 6 months to 2 years. Top CS conferences are often longer than journals in BS, and perhaps all other fields).
I think that these facts may be useful for us, CS people. Review work in other fields seems to be much less heavy than ours :-)
Besides, the case of Leon in Mexico is actually much happier than many other cases in poorer countries. That is a dream case for me and my friends although I am studying in the best institutions in a rich country. Don't need to complain that much, ha ha! (do not take it serious ;-) )
I'm a new person here and want to provide a different view. Professor of my friend in Biological Science (BS) said to me that, in BS:
ReplyDelete- First, conferences mean nothing. Only journal papers are counted.
- Second, journal reviewers often get a small amount of money from the journal that invites them (CS reviewers do not). However, professors do not review papers for money. The amount of money is small when compared with the one they can get from other academic work. But, there is still money. And, the reviews still have high quality.
- Third, authors often have to pay a small amount of money to the journal after when the paper is accepted (like what CS conferences require, but CS journals do not). However, professors often do not allow students to submit rubbish papers since these papers can ruin their fame, although the amount of money paid to the journal is very small compared with their budget.
- Four, most journals have turn-around times less than 2 months and short turn-around times do not mean that they are not prestigious. (CS journals often require 6 months to 2 years. Top CS conferences are often longer than journals in BS, and perhaps all other fields).
I think that these facts may be useful for us, CS people. Review work in other fields seems to be much less heavy than ours :-)
Besides, the case of Leon in Mexico is actually much happier than many other cases in poorer countries. That is a dream case for me and my friends although I am studying in the best institutions in a rich country. Don't need to complain that much, ha ha! (do not take it serious ;-) )
One solution to the issue of whether $500 is too expensive in some country is to PPP-adjust (http://en.wikipedia.org/wiki/Purchasing_power_parity) the fee. If $500 reimburses you in the US, and we also decide that $500 is a disincentive to submit bad work, then the disincentive could be PPP-adjusted to be lower in countries where that amount would buy you more.
ReplyDeletethis seems a good idea but I doubt it will magnify a "poverty gap" between well known high reputation researchers and vast majority guys who are not so confident of their works but they want to try at least. I don't mean that half barked papers are welcomed but nobody can guarantee acceptance of their works especially papers could be rejected by many reasons authors can not control, the rejection will significantly combat motivation to submit paper again to the same conference. then good conference may lose "diversity" which I think is much more harmful to the whole community. I am a PhD student and funding situation in our group is not satisfying. I think I have some good works done and want to try prestige conference, if there is "submission fee",I may consider to submit to a journal instead. I don't want to exaggerate my works but at least it is worth one hour review and may give some new insight to research community.
ReplyDeleteInterestingly, if this proposal was implemented, in say a single top conference, and it proved effective, it would make that conference look less competitive by comparison to other conferences due to higher acceptance rate. And you know how much people love to worship numbers...
ReplyDeleteI like Adam Marcus's suggestion to PPP-adjust the fee. That should address the silly objections like this will decimate computer science in Mexico. Of course, a lot of papers a co-authored by people from different countries, so I'm not sure how that would work in practice.
ReplyDeleteAs an aside, it is ridiculous to say money is the reason that the top CS conferences are dominated by a few institutions. There are plenty of very wealthy universities just in the US that rarely if ever get papers into the top conferences. Lack of funding is not the explanation, in a field where all you need is a $300 laptop and an Internet connection.
I thought the intention of research was to advance the state of science ;-).
ReplyDeleteNow not only will i have to work on a grad student salary, I will also need to pay to advance the state of the art so others can benefit?
P.S. I mean the above comment in jest. However, the effect of pushing the reviews for public scrutiny accomplishes both objectives; a reduction in run-of-the-mill papers and a check on review quality.
Hi,
ReplyDeleteI was wondering what your thoughts were on the following.
1) If one conference charges for submissions, then shouldn't there be some standard of what makes an appropriate review? Wouldn't the authors develop much higher and more homogeneous expectations? ("I paid $x to submit my paper, I should get at least a one-page review with constructive criticism" with the reviewer replying "I disagree with Theorem 1 and there isn't any point in reviewing the paper further" Author: "Give me a refund!") Wouldn't this open the door to more work for the conference organizers, who'd have to check the work of the reviewers to make sure it's up to par?
2) Maybe the paper selection process should take into account the ratings of previous years' papers by one or more of the authors. If a paper got consistently poor reviews (not just one), then its authors would have a harder time getting accepted the following year.
3) Another thing that might help deter people from submitting half-baked reviews would be to post the ranking of all the submissions online, including their total score. Papers wouldn't get a low total score unless they had been found deficient by several reviewers. That would be a clear sign that the authors submitted the paper too early.
4) Ultimately, though, some conferences do enjoy publicizing their low acceptance rates. They might indirectly benefit from the submission of half-baked papers, in the same way that colleges like to see under-qualified students apply because they can say later they are very selective.
I liked your post, though.
Very interesting post.
ReplyDeleteAbout the double-blind review process: I personally think it should be applied in all conferences (at least the most important IEEE and/or ACM ones). Point blank.
Honestly, too many times you see and/or hear about TPC members accepting other TPC members papers. This looks a lot like exchange of favors, which ultimately does not contribute to science... Double-blind review costs nothing in financial and organizational terms, and it improves dramatically fairness and equality of treatment. I really don't understand why it is not applied, or probably I have already understood the reason why (read above), and I really don't like the answer...
Do you really want to improve quality of the reviews done by the reviewers? One suggestion: force the reviewer to put his name at the end of the review (as it is actually already done in some journals). In this case you would have to really comprehend what you have been reading, and provide a comprehensive review of it.
With double-blind review, and signed reviews, then I would have no problem in paying the fee you propose.
Well here's a solution - why don't you stop being on so many TPCs? Say you choose to be on 1 TPC a year, that way, you get plenty of time to review all the papers assigned to you, do a good job reviewing and stop griping about the 'extra' load you have to incur. What's that you say? You have to be on multiple TPCs for tenure etc etc reasons?
ReplyDeleteToo bad. I mean seriously, the previous post on reviewing by the author had atleast some good take-away points for authors (although I strongly disagreed with the premise), but this post takes the cake!
The problem doesn't lie with authors or reviewers, but the way the system is setup - we should cut down on the number of conferences we have (networking), award quality over quantity and stop playing a ranking-game of pushing a conference above others (Sigcomm).
My 2 cents.
"My 2 cents" - you clearly don't understand how this works. Serving on one TPC a year, or one a decade, does not give you any more time to review the papers. There is usually a pretty short window between the paper deadline and when the reviews are due - anywhere from 3-6 weeks in my experience. I don't know about you, but I'm a pretty busy person and fitting in 25+ paper reviews in a few weeks is hard to do.
ReplyDeleteBesides, if I choose to serve on one TPC a year, that doesn't reduce the overall reviewing burden for the rest of the community, which is what my post was all about.
What if the papers can be rejected at the abstract stage itself...I am sure that does not take an hour...If that gets through then we could have an extended abstract then the final paper...As CS researchers we can hierarchically review papers the same way we hierarchically encode images, videos, and maps
ReplyDeleteA few more responses.
ReplyDeleteRe: "What if the papers can be rejected at the abstract stage itself..." I like this idea and wish it were implemented, at least to prune some of the obvious rejects (papers that are submitted to the wrong conference). The problem is that the abstract often promises much more than the paper delivers. Maybe a two-page abstract would suffice and then we'd invite full papers from those that pass that stage. But this seems like more work than just taking the full papers up front.
Re: Signing reviews. This I'm not so sure about. Untenured faculty would be very unlikely to rate anything down since there could be retribution later. Senior, well-established faculty could write whatever BS they like and not care about the reputation hit. I think it should be up to the program chairs to make sure that the TPC members are doing a good job of writing comprehensive reviews. At least at high quality conferences this is almost universally the case. So this does not seem to be a major problem.
Re: Charging for submissions yields expectation on the authors' part of the quality of reviews. Absolutely. This would be a big problem as authors would feel entitled to the reviews that they "paid" for. One of many problems with my proposal.
Re: Grad student needs to pay to submit papers. I have never suggested that a grad student pay for this out of pocket. A paper submission fee should absolutely be paid by the advisor out of research funds, just like conference registration and travel expenses.
Not crazy about the pay for submit idea. It might work in top systems/networking conferences, but certainly wouldn't work in areas like Theory. It would be nicer to have a solution that's widely applicable. How about the review process is still anonymous but if your paper gets 3 reviews below some terrible threshold then your names are revealed along with the accepted papers. It would be something like a wall of shame.
ReplyDeleteI am not sure getting money into the submission process is the right move. Better papers in conference submission should be the result of social pressure, not lack of money. Going towards more public refereeing processes (even a 'wall of shame' for disastrous papers with very low scores) seems like the right direction.
ReplyDeleteNow, the idea of charging for submitting papers, as developed in the post, is related to (and seems to ignore) the fact that even if PC members invest time and energy in the process, they also get benefits. These are harder to quantify, of course, but such benefits exist and are significant. For instance, I suppose that things like serving in the committee of top conferences are considered as valuable in job promotions (at the very least they look good in a CV). So not everything can be reduced into money, I think.
Re:Re: "Untenured faculty would be very unlikely to rate anything down since there could be retribution later"
ReplyDeleteReally? Interesting you claim this. If a review is objective, correct and comprehensive, which kind of retribution should they be afraid of? The retribution of the caste? And how often untenured faculty is TPC member of an important conference? Pretty rarely.
"Senior, well-established faculty could write whatever BS they like and not care about the reputation hit."
I think it would be the exact contrary. At least to my eyes, they would lose all their reputation, and I would not hesitate in letting others know their behaviour (this already happens widely, let's be honest). And if they really do so (write whatever BS they like), well, which kind of well-established faculty are they? We should all remember here that the final target of our work is the good of science, not the good of our ego over others...
"I think it should be up to the program chairs to make sure that the TPC members are doing a good job of writing comprehensive reviews. At least at high quality conferences this is almost universally the case."
I agree with you on this point. But honestly I have seen this not happening too many times. I also would like to add that in most of the high quality conferences you mention, TPC members are reviewing papers involving also other TPC members. And this happens without double-blind review procedure. This should not happen, and I have already explained in my previous post why... curious you kind of skipped over that part ;-)
charge starting with the author's second submission, like airlines charging for bags.
ReplyDelete$1000 for the second submission. $2000 for the next one. And so on: a soft disincentive to multiplying submissions.
Very interesting post. I happened onto your blog after a frustrating evening reviewing a terrible paper for a computational linguistics conference.
ReplyDeleteI don't know if charging is the right solution, but I agree it is a problem.
I take putting down my comments very seriously, and I don't mind reading the papers as much as I mind the time it takes to give detailed discussions of why they are so bad when they are really bad. Maybe there should be an option when reviewing a paper that's really crap to let you select "skip written review and proceed straight to recommendation of rejection".
A few people have suggested crowd-source review process; make everything public, submissions, reviews (double blinded). I might add that even there should be some thing as reviews on the reviews (may be a point system). Best work would automatically get to the top.
ReplyDeleteWhy have most of the commentators skipped discussing on this. I am new to the academia, so I might be missing something here.