tag:blogger.com,1999:blog-9186457242428335144.post1424470161677208375..comments2024-03-28T00:36:13.790-07:00Comments on Volatile and Decentralized: The Psychology of Program CommitteesMatt Welshhttp://www.blogger.com/profile/04255792550910131960noreply@blogger.comBlogger23125tag:blogger.com,1999:blog-9186457242428335144.post-26611551200841294142010-04-30T01:39:12.819-07:002010-04-30T01:39:12.819-07:00This blog provides a brillant insight and should b...This blog provides a brillant insight and should be accepted in the toughest conference :)<br />It does open my frustrated mind after many rejections, thanks ^_^NP-hardhttps://www.blogger.com/profile/14828448430402848042noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-46986114175231337042010-04-07T05:12:33.500-07:002010-04-07T05:12:33.500-07:00If you could come up with a few more of these, you...If you could come up with a few more of these, you could make a BINGO card to pass out at PC meetings. When people saw a PC member exhibiting one of these behaviors, they could mark a square. I'm not sure what would happen when you called out BINGO, though -- you might find it makes several PC members surly.Michael Mitzenmacherhttps://www.blogger.com/profile/02161161032642563814noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-79793016427786915862010-04-06T11:10:37.805-07:002010-04-06T11:10:37.805-07:00One development that I think has decreased the eff...One development that I think has decreased the effectiveness of program committees is the increasing power of "senior PC members" or "meta-reviewers" or whatever you call it. Having senior PC members closely supervise the review process, to make sure every paper gets careful attention from the reviewers, and that all of the reviewers' concerns are discussed in detail can be valuable. However, such a role works best when executed modestly. When senior PC members step in and overrule the will of the reviewers they are dominating the direction of the community in a way that is usually not healthy. Of course, the senior PC should provide feedback to the reviewers, and raise issues for discussion, but if the reviewers are not convinced, the senior PC member should respect their will.<br /><br />JohnJTRiedlhttps://www.blogger.com/profile/01281546440630845977noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-44897534246257577442010-04-02T19:45:11.504-07:002010-04-02T19:45:11.504-07:00Matt,
You are of course absolutely right about th...Matt,<br /><br />You are of course absolutely right about the timing playing a big role in PC discussions. I was just co PC chair of VEE and we decided to do a preliminary "calibration round" to try to counteract this effect. We (the PC chairs) more or less randomly picked three papers from each tier (one in top-ranked, one in mid-ranked, and one in bottom-ranked).<br /><br />Naturally, the top one got in, the bottom one didn't, and I don't remember the fate of the mid-ranked one. But the effect was noticeable: everyone now had a sense of perspective of the whole range of papers, rather than just their pile, and the PC meeting went insanely smoothly. Now, that might be because we managed to put together an insanely great PC, and it was a sunny day in Seattle so the view from the Allen building was superb :), but I think it helped.<br /><br />And unlike Kim, I accepted the invitation to be on the PC with the "trump card" (aka "whiteball") - it's OOPSLA 2010. Martin Rinard is the PC chair and he is hoping it will help ensure that edgy, creative papers get in. We will see...Emery Bergerhttp://www.cs.umass.edu/~emerynoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-89334932542825468092010-04-01T14:02:12.448-07:002010-04-01T14:02:12.448-07:00Ketan -- Hi there! Man, first Kim, now you -- see...Ketan -- Hi there! Man, first Kim, now you -- seems everyone is reading this blog! (Feel free to stop by mine once in a while...)<br /><br />I like your PC meeting schedule. When you're PC chair, please invite me to be on the committee. And I"m not just saying that because I still have some of your money sitting in my wallet from Berkeley days.<br /><br />I don't think I've been to a PC meeting where a poker game was on the schedule. The other socializing is often quite nice, but a poker game would be an excellent addition to the mix.Michael Mitzenmacherhttps://www.blogger.com/profile/06738274256402616703noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-29671021470511694142010-04-01T12:58:09.261-07:002010-04-01T12:58:09.261-07:00I agree with Mike that the noise for papers in the...I agree with Mike that the noise for papers in the gray area doesn't really matter. Any subset given however much room is left in the program is likely to be about as good as any other. I would love as PC chair to follow this agenda:<br />9:00 AM - Process easy rejects<br />9:45 AM - Process easy accepts<br />10:30 AM - Decide on how many slots are left in program<br />11:00 AM - Randomly choose a subset of papers in the middle<br />11:30 AM - Go to lunch<br />1:00 PM to 2:00 AM - Play poker and drinkkmayerpatelhttps://www.blogger.com/profile/16929154203710078034noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-37296642000230054032010-04-01T10:23:37.680-07:002010-04-01T10:23:37.680-07:00Adam, good points - I intend to write a future blo...Adam, good points - I intend to write a future blog post on marketing one's research, but it's clearly an important part of building one's reputation and "brand" in the community. Program committees are human beings and they can't be expected (nor would it be desirable) to leave their knowledge of the research community at the door of the PC meeting.Matt Welshhttps://www.blogger.com/profile/04255792550910131960noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-71638382547916277882010-04-01T10:19:43.306-07:002010-04-01T10:19:43.306-07:00Anon - I said, "If someone is familiar with a...Anon - I said, "If someone is familiar with a piece of work **outside the context of the paper submission**" -- meaning, the TPC member already knows the project. It is very difficult to disguise your identity when a reviewer has out-of-band information.Matt Welshhttps://www.blogger.com/profile/04255792550910131960noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-76130835898776986002010-04-01T09:10:10.485-07:002010-04-01T09:10:10.485-07:00If someone is familiar with a piece of work outsid...<i>If someone is familiar with a piece of work outside of the context of the paper submission, double-blind can't generally hide the authors' identity. </i><br /><br />Oh no, not this canard again. Do you really can guess the author papers that often? Say with more than 50% correct probability? I can't.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-5944634416937820542010-04-01T07:37:13.999-07:002010-04-01T07:37:13.999-07:00> You can't really game the system.
I'...> You can't really game the system.<br /><br />I'm not sure I agree with the last point. It depends what you call "gaming" the system. For example, you can:<br /><br />- go around giving talks about your unpublished work. This gives an "unfair" advantage to established researchers in a field who have more, higher-visibility opportunities to give talks. [I put "unfair" in quotes because you can make a case that people with established research records have earned this advantage -- we invite them to give talks precisely because we want to hear what they're currently thinking about.]<br /><br />- tell people on the PC whom you know personally about your work. It's remarkable how when I'm on the PC of an important conference suddenly everyone wants to tell me about what they're doing. Of course, I enjoy hearing about what they're doing but I am aware that it biases my reviewing. <br /><br />- discuss your work on a blog, post it on the arxiv, etc. Again, this is a way of "getting around" the vexing anonymity of submissions, when you're well-enough known and liked for anonymity to work "against" you. <br /><br />- make sure you refer generously to the work of PC members and likely reviewers. [Note: of course, you should refer generously to all previous work, but I've certainly seen people trying to suck up to specific PC members.]<br /><br />The first three points only give you a hand up if you're doing reasonably interesting work. So is it gaming the system? Sure it is. There are more "reasonably interesting" papers than there are slots. For borderline papers, the amount of buzz around a paper or line of work is a factor, conscious or not, in PC decisions. And buzz can, to some extent, be manufactured.Adam Smithhttp://www.cse.psu.edu/~asmithnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-56095249735741110152010-04-01T06:21:09.053-07:002010-04-01T06:21:09.053-07:00Anon - actually, double-blind doesn't address ...Anon - actually, double-blind doesn't address problems #1 or #2 at all. If someone is familiar with a piece of work outside of the context of the paper submission, double-blind can't generally hide the authors' identity. And seeing the same paper on several subsequent program committees doesn't have anything to do with whether the authors have been identified.<br /><br />Actually, at least in systems, most of the top conferences do use double-blind reviewing. Journals typically do not, and I think it would be very difficult to write a detailed journal paper that covers all of the related work while masking your own identity.<br /><br />I absolutely agree that the best way to get your papers accepted is simply to do good research. You can't really game the system.Matt Welshhttps://www.blogger.com/profile/04255792550910131960noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-58110602258607647662010-04-01T03:36:21.566-07:002010-04-01T03:36:21.566-07:00Matt, the first 2 points ("I didn't read ...Matt, the first 2 points ("I didn't read this paper, but I know the work" and "I didn't read this paper, but I saw their original failed submission last year") can be EASILY canceled with double-blind review process. All the others issues would still be there, but at least these 2 would quickly disappear. My question is: why double-blind review is still not used in most of the most important journals and conferences? Maybe because the "caste" likes the idea of keeping everything under "politically correct" control, so to avoid embarrassing situations in which e.g. you are eventually shooting down the paper of the PC member sitting just in front of you? We already discussed this in one of your previous posts...<br />I think that one of the most important things a good supervisor should always do with his grad and post-grad students is to push them towards the best achievable quality of research, even when all these unpleasant behaviours are evident to everyone who has ever attended a PC meeting and result in unfair rejections of good papers.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-90941374076819103742010-03-31T11:22:16.173-07:002010-03-31T11:22:16.173-07:00The general psychology you suggest seems spot on, ...The general psychology you suggest seems spot on, though a good PC chair will switch to the considering the bottom before the first reject. It should happen when the accepts start to require significant debate and the switch should happen whenever things lag.<br /><br />A big mistake is for a PC to spend too much time on the obvious rejects and accepts so that too little time is left for borderline papers. The average time per paper spent should be planned to increase monotonically over the course of the meeting as the decisions get more and more difficult. The chair has to be really cognizant of the rate of progress at all times.<br /><br />The direction of the reaction to papers that are close to a PC member's work can be very area dependent and part of the socialization in the area. In some areas the members form a mutual admiration society and in others there is a tendency to tear down each others' work (or work that is not by their immediate social circle). This effect is one of the most difficult for PCs to deal with.<br /><br />The situation for electronic-only PCs is different but far from better.Paul Beamenoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-5853097831043817432010-03-30T20:06:32.762-07:002010-03-30T20:06:32.762-07:00So, can another PC member use her trump card to an...So, can another PC member use her trump card to annihilate your trump card if they disagree with you? And, if so, can I have the ace?<br /><br />-- Maria EblingAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-43481262404041780812010-03-30T17:33:16.410-07:002010-03-30T17:33:16.410-07:00Kim --
Come now. First, you know I like you too ...Kim --<br /><br />Come now. First, you know I like you too much to abuse you for misusing a theoretical term. Second, you used it entirely properly. :)<br /><br />I heard a theory conference recently used the "trump card" approach. I admit to some skepticism, although I suppose it depends on how it's used. On the plus side, if I understand the argument, if someone is that positive on a paper, there must be something about it that merits acceptance. On the down side, it seems ripe for abuse (did the "in crowd" just get even more in?). Also, I've certainly been on PCs where people got overly excited about a paper that was not in their area. Perhaps the trump card should be restricted so the reviewer has to be an expert (or above some threshold) in the subarea? Finally, it probably works better for conferences with 60-70++ papers over smaller conferences like SIGCOMM.<br /><br />When you and Matt get together to tell PC stories, be sure to let me know....Michael Mitzenmachernoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-51717816592622435662010-03-30T17:27:44.007-07:002010-03-30T17:27:44.007-07:00Matt --
I'd like to echo the point I think an...Matt --<br /><br />I'd like to echo the point I think anonymous was making. Obviously the process is noisy, but the noise tends to have a much less significant effect on papers at the top and bottom. Where it has the most effect is on papers at the borderline.<br /><br />I've been on multiple PCs where there's say 4-10 slots left and maybe a dozen papers still left to discuss. As you say, "Late in the afternoon, everyone is exhausted, cranky, and the power structure of the PC meeting has started to play itself out..." At this point, people are thinking about getting to the airport. My take is that, often, at this point, there's no "right answer" -- if one had an arbitrary amount of time, one could still imagine different outcomes that would be reasonable, and there's nothing like an arbitrary amount of time left. So do the types of things you're talking about start happening and influence the final decisions? Absolutely. And, where possible, they should be avoided. But when there's 5 reviewers with 2 at best lukewarm about a paper, 2 lukecold, and 1 who doesn't feel informed enough to make a decision, I'm actually glad that some of these random personality quirks come out and allow us to move on.Michael Mitzenmachernoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-54681473643425904622010-03-30T17:24:04.519-07:002010-03-30T17:24:04.519-07:00Weird bug here. Says 6 comments above (after this ...Weird bug here. Says 6 comments above (after this one will say 7), but there are only 4 (now 5). Michael Mitzenmacher posted 2 comments, which I got email notifications for, but they don't show up here. Perhaps the Mitzenmacher Particle and the anti-Mitzenmacher Particle (i.e., the Welsh Particle) collided in the blogosphere and annihilated each other.Matt Welshhttps://www.blogger.com/profile/04255792550910131960noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-52746898599638883162010-03-30T15:13:31.299-07:002010-03-30T15:13:31.299-07:00Matt --
I'd like to echo the point I think an...Matt --<br /><br />I'd like to echo the point I think anonymous was making. Obviously the process is noisy, but the noise tends to have a much less significant effect on papers at the top and bottom. Where it has the most effect is on papers at the borderline.<br /><br />I've been on multiple PCs where there's say 4-10 slots left and maybe a dozen papers still left to discuss. As you say, "Late in the afternoon, everyone is exhausted, cranky, and the power structure of the PC meeting has started to play itself out..." At this point, people are thinking about getting to the airport. My take is that, often, at this point, there's no "right answer" -- if one had an arbitrary amount of time, one could still imagine different outcomes that would be reasonable, and there's nothing like an arbitrary amount of time left. So do the types of things you're talking about start happening and influence the final decisions? Absolutely. And, where possible, they should be avoided. But when there's 5 reviewers with 2 at best lukewarm about a paper, 2 lukecold, and 1 who doesn't feel informed enough to make a decision, I'm actually glad that some of these random personality quirks come out and allow us to move on.Michael Mitzenmacherhttps://www.blogger.com/profile/06738274256402616703noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-31271113483831061452010-03-30T14:59:45.362-07:002010-03-30T14:59:45.362-07:00Kim -- First off, you know I like you far too much...Kim -- First off, you know I like you far too much to abuse you for misusing theoretical terms! Second, you used it entirely appropriately. :) <br /><br />I heard the "trump card" approach, or something like it, was used recently for a theory conference. I admit I'm a bit skeptical of it, but I suppose it depends on how it's used. There's the argument that if someone on the PC really likes the paper, then there's something interesting there so that the paper should be accepted. The counterargument is that (at least in my experience) some PC members get overly enthusiastic about a paper they don't know enough about. I'd feel better if you had to justify you were an expert (or at least above some fairly high threshold) in the subarea if you were going to use your trump card. Also, it probably works better for larger conference (60++ papers, instead of the 30 or so of SIGCOMM...) It does help avoid the "everything's getting rejected!" phenomenon!<br /><br />When you and Matt go out for that beer to swap PC stories, please remember to invite me along....Michael Mitzenmacherhttps://www.blogger.com/profile/06738274256402616703noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-60802064700012097462010-03-30T14:05:04.820-07:002010-03-30T14:05:04.820-07:00Anon - If you've ever been on a PC you'll ...Anon - If you've ever been on a PC you'll know that most papers are marginal - there are very few papers that are clear accepts. The issue is that even very, very good papers can sometimes get rejected due to non-technical reasons, and it's important for students (especially) to understand the dynamics behind the scenes.Matt Welshhttps://www.blogger.com/profile/04255792550910131960noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-22686412517677355632010-03-30T13:49:12.394-07:002010-03-30T13:49:12.394-07:00There are some of us who respect what the PC is tr...There are some of us who respect what the PC is trying to do and only send our papers to appropriate venues. There are people who have 2-3 papers a year in the top venue, and still an 80% accept rate. Why are you so concerned with the probabilistic fate of marginal papers? (There are those who, in fact, make a career out of the probabilistic fate of marginal papers.)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-78657376112122747202010-03-30T08:50:17.329-07:002010-03-30T08:50:17.329-07:00Hi Matt! What a timely discussion! I'm sitting...Hi Matt! What a timely discussion! I'm sitting in one of those meetings at this very moment. I'll never forget my first PC meeting when I learned about the reality of the situation. At that point, I decided I'm glad I get my sense of self worth from other events in life than conference paper outcomes. Two things I'll add: <br /><br />(1) There is a bimodal reaction (MM is going to kill me for misusing theoretical terms) when your paper is sent to a reviewer who does very similar work. They'll either overinflate its importance - "This paper tackles the most important problem known to mankind!" or they try to squash the competition - "No need to accept this paper because the smartest person in the world (me) is already working on the problem."<br /><br />(2) People's egos get tangled up in the discussion. Often, once they get going in fighting for/against a paper, it becomes less about the paper and more about winning the battle. This adds to the randomness of the outcomes.<br /><br />I recently got invited to be on a PC where each PC member was to be given one "trump card" at the start of the meeting. This would allow the PC member to choose one paper for which they could unilaterally decide the outcome. (I declined the invitation - I'm on too many PCs. But I must admit I was very tempted to accept the invitation just to see how it shook out.) <br /><br />The next time I'm in Cambridge, we can share PC meeting stories. I have lots of them.Kim Hazelwoodhttp://www.cs.virginia.edu/kim/noreply@blogger.comtag:blogger.com,1999:blog-9186457242428335144.post-64056132054460755302010-03-30T07:12:42.784-07:002010-03-30T07:12:42.784-07:00Matt, you forgot to include "he is not in my ...Matt, you forgot to include "he is not in my clan" fallacy. Some people take their research field as a religion, and also display radical fundamentalism towards papers from other fields. "If the approach the paper proposes does not stem in my research field, it is wrong."<br /><br />This is the worst kind, in my view.Murathttps://www.blogger.com/profile/07842046940394980130noreply@blogger.com