The R01 Lottery?
I found some details about the recent sharp decline in success rates for NIH grants (R01s in particular) in this helpful FAQ. There are lots of useful data in the document, but also some interesting things between the lines. It sounds incredibly defensive to me: "Here are 6 things we did that you may think caused the current problems. They didn't. It's not our fault." Clearly the NIH is taking a lot of heat from PIs about R01 success rates.
Elias Zerhouni and the FAQ both take pains to assert that new NIH programs and priorities are not responsible for the changes in success rates. Let's take their word for the moment and dig into their explanation: the drop has occurred because many more people have started applying for R01s. Where did these new people come from?
Here's an explanation that came immediately to mind:
Think of an R01 application as a lottery ticket. The cost of the ticket is the effort required to prepare the application. The expected payoff is the product of the average amount of an award and the probability of receiving one. During the first few years of the NIH budget doubling, the amount of an award went up and so did the probability of receiving one.
When the expected value of a lottery ticket increases while the cost remains the same, more people buy. Is this the right model? Assuming the cost of preparing an application is constant, we'd expect the number of new applications to increase until the expected payoff returned to its value before the doubling (economists, feel free to correct me!). Let's look at the NIH's numbers for R01s:
| Year | Applications | Awards | Success rate | Total $ (thousands) |
| 1997 | 20,396 | 6140 | 30.1% | 5,457,343 |
| 1998 | 20,039 | 6195 | 30.9% | 5,980,395 |
| 1999 | 21,994 | 7028 | 32.0% | 6,723,897 |
| 2000 | 22,088 | 7063 | 32.0% | 7,602,572 |
| 2001 | 21,967 | 6965 | 31.7% | 8,499,965 |
| 2002 | 22,212 | 6799 | 30.6% | 9,346,440 |
| 2003 | 24,634 | 7430 | 30.2% | 10,084,647 |
| 2004 | 27,461 | 6991 | 25.5% | 10,538,699 |
| 2005 | 28,423 | 6463 | 22.7% | 10,667,802 |
(For you sticklers, the numbers are for "R01 equivalents", i.e. R01, R29, and R37 grants.)
A few observations:
The increase in the total number of new awards per year was relatively modest -- generally about 15%, except for 2003. The budget doubling did not do much to spread the wealth, at least with R01s, and as a result, the probability of receiving an award never increased very much.
The big change is in the dollar amount of the awards. The FAQ does not provide average amounts per award, but the overall amount spent on R01s increased by 68% over the doubling period (1998-2003), so the per-award increase is probably in the ballpark of 45% or so.
The simplistic lottery model as we have posed it does not look quite right: for the first 4 years of the doubling (1998-2002), the number of applications increased only modestly, by about 10%, but the expected return increased much more. The data on individuals (rather than applications) below tell a similar story (again, the numbers are for R01 equivalents). For the first 4 years of the doubling, the number of new individuals applying increased by a total of about 9%.
| Data on Applications | Data on Individuals | |||||
| Year | Applications | Awards | Success rate | Applications | Awards | Success rate |
| 1997 | 20,396 | 6140 | 30.1% | 17,065 | 5828 | 34.2% |
| 1998 | 20,039 | 6195 | 30.9% | 16,898 | 5812 | 34.4% |
| 1999 | 21,994 | 7028 | 32.0% | 18,281 | 6548 | 35.8% |
| 2000 | 22,088 | 7063 | 32.0% | 18,354 | 6559 | 35.7% |
| 2001 | 21,967 | 6965 | 31.7% | 18,273 | 6536 | 35.8% |
| 2002 | 22,212 | 6799 | 30.6% | 18,482 | 6357 | 34.4% |
| 2003 | 24,634 | 7430 | 30.2% | 19,964 | 6905 | 34.6% |
| 2004 | 27,461 | 6991 | 25.5% | 21,857 | 6485 | 29.7% |
| 2005 | 28,423 | 6463 | 22.7% | 22,536 | 6052 | 26.9% |
What's wrong with the lottery model? My guess is that the flaw lies in the assumption that the cost of preparing an application is constant. What matters here is the marginal cost of an application. If you are a lab that is already submitting R01 applications, your costs don't change. The real question is, how much does it cost the new applicants to prepare an application?
Think about it this way: There is an existing pool of research labs for which it's pretty straightforward to prepare an application. They already have facilities in place and existing research programs from which to obtain preliminary data. Come up a new idea, task it to a postdoc or grad student, go into a frenzy of writing (or have the students / postdocs do it), and voila. Those labs are already submitting proposals as fast as they can. Adding more money to the pot probably doesn't induce much extra work from them in the short term because they are already operating near capacity.
The next tier down is smaller labs, maybe one or two person shops, that have never submitted a proposal or that do so very infrequently. Preparing a new application for these labs takes a lot more work -- they don't have an army of students and postdocs at hand, their equipment is probably less state-of-the-art, and so on. But the prospect of all those new, high-dollar R01s is tempting. Because the marginal cost of preparing an application is higher for these smaller labs than for more established ones, the additional dollars don't entice all that many new applicants.
My guess is that the 10% initial increase in applications and applicants came from the new money enticing a little more work out of established players and bringing in a small number of new ones. So far, so good.
The really interesting question is what happened starting in 2003? There's an initial 10% bump in applications in 1999, then a plateau until 2003. All of a sudden there's an increase of another 10%, and it's followed by still another 10% increase in 2004, and another 5% in 2005. We'll look at that next.
