Summary: Let's improve the NSF proposal review process by making it function more like conference program committees.
Intellectual merit: The core problem that this proposal addresses is the poor quality of many reviews submitted by NSF panelists. It is not uncommon for a proposal to be rejected with short, content-free reviews, offering little feedback to the authors. In many cases the scoring of a proposal is poorly justified, leaving the author mystified as to why they got such a low (or high) score. Recently, I had a proposal rejected where one of the reviews was essentially a single sentence in length. Not only does this not help the PI improve the work for later submission, but it leaves the impression that the review process is arbitrary.
(I'd like to emphasize that this is a problem that many NSF program managers have called attention to, but they are powerless as individuals to do much about it. So I believe the fault rests with the research community, not with the NSF PMs.)
A key problem with NSF panels is that there is no community standard for what constitutes a good (or even acceptable) proposal review. I am a strong advocate of the approach used by the systems community, where paper submissions are given extremely detailed reviews with constructive feedback. Given that we spend so much effort reviewing papers, couldn't we also give the same effort to NSF proposals, which arguably are more important than a single paper?
It is my impression that NSF program managers also have a hard time pulling panels together, mainly because people are so busy, and don't have the time to travel to DC. Yet many of the potential panelists freely serve on conference program committees with much higher reviewing loads and an expectation of much more detailed reviews. (A typical panelist will review 8-12 proposals, whereas a competitive conference will require TPC members to review 2-3x as many papers.) Why? One reason, perhaps, is that program committees are recognized for their work, and serving on a TPC is an important indication of one's stature in the research community.
These two issues are related. Since serving on an NSF panel is seen as "paying your dues," rather than an activity you take pride in, there is little incentive to write good reviews. However, if you write a bunch of crappy reviews for a TPC, you can earn a reputation as someone who doesn't take the process seriously might not get invited back in the future. So the public recognition of the TPC and the quality of the reviews go hand in hand.
My proposal: Let's have NSF panels emulate the conference program committee model. First, we should recognize panelists publicly for their work. Being on the "NSF NeTS 2010" panel should be as prestigious as serving on SIGCOMM or SenSys. The NSF should create a web page for each years' competition where they list the panelists and list the proposals funded through that competition (the latter information is available, but a little hard to dig up). So the community can take pride in the effort and see the outcome of the process more directly.
Second, establish expectations for high-quality proposal reviews. If you are a bad panelist, you won't get invited back in the future, so you won't gain the recognition of being asked to serve. Panelists will be chosen from among the best people in the field, where "best" is defined both by research contribution and service.
Third, hold panels somewhere other than Washington DC. Since I live in Boston, it's easy for me to get down there, but for people on the West Coast it is much harder. If panels are run in different locations around the country, the travel burden can be spread around more evenly.
I will be the first to admit that the conference program committee model is not perfect -- see my related posts here and here for thoughts on that. But in my experience it is better than the (typical) NSF panel.
Of course, NSF's conflict-of-interest guidelines will have to be tweaked. Currently, you can't serve on an NSF panel for a program for which you have submitted a proposal. (The upshot is that panels tend to consist of people who didn't get their act together to submit a proposal, which may not be the best group of scholars to evaluate the work.) Recently NSF has separated larger programs into multiple panels and simply ensured that someone can't serve on the specific panel for which their own proposal is under consideration.
Broader impacts: Improving the proposal review process and providing more constructive feedback to PIs will result in better science.
In the comments, please indicate your rating for this proposal:
After more than eight years at Google, I'm joining XNOR.ai , a small startup developing AI for embedded devices. Check out my blog po...
The word is out that I have decided to resign my tenured faculty job at Harvard to remain at Google. Obviously this will be a big change in ...
My team at Google is wrapping up an effort to rewrite a large production system (almost) entirely in Go . I say "almost" because ...
I'm often asked what my job is like at Google since I left academia. I guess going from tenured professor to software engineer sounds l...