Tuesday, December 8, 2009

Digg for grant proposal reviews

I am a huge fan of the social news site Digg. It is where I go to get my daily dose of Internet stupidity, ranging from XKCD comics to pictures of fail. For those that have been living in a cave the last five years, the way it works is that users submit links to random sites they find on the Internet, and those that like the link "digg" it, thereby increasing the link's popularity. Tracking Digg is a good way to keep your finger on the pulse of the Internet, or more accurately, the segment of the Internet that 15-to-22 year old boys seem to care about.

The best part of the site are the incredible comments left by the users. Often these are funnier and more obtuse than the original link, and Digg comments are something of a genre in and of themselves (good examples being repeated ASCII-rendered appearances of Admiral Akbar and something called Pedobear). Indeed, occasionally the comments can get out of hand.

Here's a crazy idea that I came up with (incidentally, while having wine and cheese with the chair of the Harvard stats department this afternoon). Why not adopt the Digg model to crowdsource peer review of grant proposals? Scientists would post their grant proposals publicly and anyone would be allowed to "digg" a proposal -- or "bury" a proposal that has flaws or a particularly bad idea. Public comments would be used to convey feedback to the authors and open up debate on the research plan to the many thousands of highly qualified Internet users who are conventionally excluded from review panels.

This model would seem to have all kinds of benefits. Rather than making funding decisions in the proverbial smoky room, requiring the funding body to spend untold millions in taxpayer money to fly panelists to DC and put them up in hotels for a couple of nights, this approach would bring everything out into the open. The Digg model would also streamline the review cycle to run in "Internet time" -- reducing the typical six month turnaround time to mere hours! Best of all, users on the site would adopt clever screen names like "W1F1d00d" and "Prof. BabyMan" lending the proceedings a certain edginess and panache sorely missing from the current panel review system.

If you like this idea, why not Digg it?





6 comments:

  1. I like smoky rooms.

    ReplyDelete
  2. A bit too much wine and not enough cheese perhaps? Or was it just bad cheese?

    I suppose it would shake up the "establishment" and give some of the older, set in their ways, profs heart attacks. This would result in vacancies in acedemic departments across the country when these professors retire for medical reasons. Which might in turn help me get out of the current rut I'm in as an eternal post-doc. So, on reflection, I like your idea.

    I would digg your idea, but I refuse to create a digg account so as to preserve my own sanity, or at least what's left of it.

    ReplyDelete
  3. Would be fun to see serious multi-million dollar project proposals be mocked with comment like "First!1!!!!111!", "OMG LOL misspelled title fail", or "Teh internet is so 90s!"

    ReplyDelete
  4. Matt, although I don't find the current system perfect (e.g., the delay), I'm not sure I want 1000s of people that you claim are "highly qualified" reviewing my proposals. I do not believe these folks are qualified to evaluate a proposal at all. How will you ensure that they have the proper background (both technical background as well as knowing how to evaluate a proposal)? This doesn't seem like a great plan to me.

    Some form of middle ground where reviewers in the current system are rated by the other reviewers might help improve the quality if those reviews somehow have impact on the system. How about doing the current review process online instead of having a meeting in DC -- you might be able to speed it up significantly. Or maybe just meet in DC for the top rated 20 proposals out of a pool of 60 to fund the top 10--that way the rest (40/60) get a very fast reject and can turn them around to send elsewhere.

    Gene Golovchinsky of FXPAL has written several blog posts over the last couple of weeks on the FXPAL blog talking about ideas like your's for paper reviewing. See http://palblog.fxpal.com/?p=2400

    ReplyDelete
  5. James - I think you totally missed the irony of my post :-) I do agree, though, that we can do better than flying everyone to DC. (Being in Boston this is not a huge pain for me, but I often find the in-person meetings aren't that productive.) Two rounds of reviewing, one online, the other in-person, might address some of these concerns.

    ReplyDelete
  6. I think we can find a middle ground... Have a review system based on Digg model... but initially only allow few respected academics (or industry persons) to register .. and then give them invites to invite more people and hand over more invites.. In this way we can have reviewers that are somewhat capable to review proposals.

    -Sushant

    ReplyDelete