Skip to main content

Digg for grant proposal reviews

I am a huge fan of the social news site Digg. It is where I go to get my daily dose of Internet stupidity, ranging from XKCD comics to pictures of fail. For those that have been living in a cave the last five years, the way it works is that users submit links to random sites they find on the Internet, and those that like the link "digg" it, thereby increasing the link's popularity. Tracking Digg is a good way to keep your finger on the pulse of the Internet, or more accurately, the segment of the Internet that 15-to-22 year old boys seem to care about.

The best part of the site are the incredible comments left by the users. Often these are funnier and more obtuse than the original link, and Digg comments are something of a genre in and of themselves (good examples being repeated ASCII-rendered appearances of Admiral Akbar and something called Pedobear). Indeed, occasionally the comments can get out of hand.

Here's a crazy idea that I came up with (incidentally, while having wine and cheese with the chair of the Harvard stats department this afternoon). Why not adopt the Digg model to crowdsource peer review of grant proposals? Scientists would post their grant proposals publicly and anyone would be allowed to "digg" a proposal -- or "bury" a proposal that has flaws or a particularly bad idea. Public comments would be used to convey feedback to the authors and open up debate on the research plan to the many thousands of highly qualified Internet users who are conventionally excluded from review panels.

This model would seem to have all kinds of benefits. Rather than making funding decisions in the proverbial smoky room, requiring the funding body to spend untold millions in taxpayer money to fly panelists to DC and put them up in hotels for a couple of nights, this approach would bring everything out into the open. The Digg model would also streamline the review cycle to run in "Internet time" -- reducing the typical six month turnaround time to mere hours! Best of all, users on the site would adopt clever screen names like "W1F1d00d" and "Prof. BabyMan" lending the proceedings a certain edginess and panache sorely missing from the current panel review system.

If you like this idea, why not Digg it?





Comments

  1. I like smoky rooms.

    ReplyDelete
  2. A bit too much wine and not enough cheese perhaps? Or was it just bad cheese?

    I suppose it would shake up the "establishment" and give some of the older, set in their ways, profs heart attacks. This would result in vacancies in acedemic departments across the country when these professors retire for medical reasons. Which might in turn help me get out of the current rut I'm in as an eternal post-doc. So, on reflection, I like your idea.

    I would digg your idea, but I refuse to create a digg account so as to preserve my own sanity, or at least what's left of it.

    ReplyDelete
  3. Would be fun to see serious multi-million dollar project proposals be mocked with comment like "First!1!!!!111!", "OMG LOL misspelled title fail", or "Teh internet is so 90s!"

    ReplyDelete
  4. Matt, although I don't find the current system perfect (e.g., the delay), I'm not sure I want 1000s of people that you claim are "highly qualified" reviewing my proposals. I do not believe these folks are qualified to evaluate a proposal at all. How will you ensure that they have the proper background (both technical background as well as knowing how to evaluate a proposal)? This doesn't seem like a great plan to me.

    Some form of middle ground where reviewers in the current system are rated by the other reviewers might help improve the quality if those reviews somehow have impact on the system. How about doing the current review process online instead of having a meeting in DC -- you might be able to speed it up significantly. Or maybe just meet in DC for the top rated 20 proposals out of a pool of 60 to fund the top 10--that way the rest (40/60) get a very fast reject and can turn them around to send elsewhere.

    Gene Golovchinsky of FXPAL has written several blog posts over the last couple of weeks on the FXPAL blog talking about ideas like your's for paper reviewing. See http://palblog.fxpal.com/?p=2400

    ReplyDelete
  5. James - I think you totally missed the irony of my post :-) I do agree, though, that we can do better than flying everyone to DC. (Being in Boston this is not a huge pain for me, but I often find the in-person meetings aren't that productive.) Two rounds of reviewing, one online, the other in-person, might address some of these concerns.

    ReplyDelete
  6. I think we can find a middle ground... Have a review system based on Digg model... but initially only allow few respected academics (or industry persons) to register .. and then give them invites to invite more people and hand over more invites.. In this way we can have reviewers that are somewhat capable to review proposals.

    -Sushant

    ReplyDelete

Post a Comment

Popular posts from this blog

Why I'm leaving Harvard

The word is out that I have decided to resign my tenured faculty job at Harvard to remain at Google. Obviously this will be a big change in my career, and one that I have spent a tremendous amount of time mulling over the last few months.

Rather than let rumors spread about the reasons for my move, I think I should be pretty direct in explaining my thinking here.

I should say first of all that I'm not leaving because of any problems with Harvard. On the contrary, I love Harvard, and will miss it a lot. The computer science faculty are absolutely top-notch, and the students are the best a professor could ever hope to work with. It is a fantastic environment, very supportive, and full of great people. They were crazy enough to give me tenure, and I feel no small pang of guilt for leaving now. I joined Harvard because it offered the opportunity to make a big impact on a great department at an important school, and I have no regrets about my decision to go there eight years ago. But m…

Rewriting a large production system in Go

My team at Google is wrapping up an effort to rewrite a large production system (almost) entirely in Go. I say "almost" because one component of the system -- a library for transcoding between image formats -- works perfectly well in C++, so we decided to leave it as-is. But the rest of the system is 100% Go, not just wrappers to existing modules in C++ or another language. It's been a fun experience and I thought I'd share some lessons learned.

Why rewrite?

The first question we must answer is why we considered a rewrite in the first place. When we started this project, we adopted an existing C++ based system, which had been developed over the course of a couple of years by two of our sister teams at Google. It's a good system and does its job remarkably well. However, it has been used in several different projects with vastly different goals, leading to a nontrivial accretion of cruft. Over time, it became apparent that for us to continue to innovate rapidly wo…

Running a software team at Google

I'm often asked what my job is like at Google since I left academia. I guess going from tenured professor to software engineer sounds like a big step down. Job titles aside, I'm much happier and more productive in my new role than I was in the 8 years at Harvard, though there are actually a lot of similarities between being a professor and running a software team.

I lead a team at Google's Seattle office which is responsible for a range of projects in the mobile web performance area (for more background on my team's work see my earlier blog post on the topic). One of our projects is the recently-announced data compression proxy support in Chrome Mobile. We also work on the PageSpeed suite of technologies, specifically focusing on mobile web optimization, as well as a bunch of other cool stuff that I can't talk about just yet.

My official job title is just "software engineer," which is the most common (and coveted) role at Google. (I say "coveted&quo…