|
|
NIH Crisis
"Grant System Leads Cancer Researchers to Play It Safe" in today's NY Times highlights NIH's crazy risk averse grant making process.
...the fight against cancer is going slower than most had hoped, with only small changes in the death rate in the almost 40 years since it began. One major impediment, scientists agree, is the grant system itself. It has become a sort of jobs program, a way to keep research laboratories going year after year with the understanding that the focus will be on small projects unlikely to take significant steps toward curing cancer.
What seems strange to me is that the people in the article keep blaming scarce resources for the excess of caution. The NIH's budget doubled recently with no real change in the likelihood of funding for risky proposals - if anything, things are worse because of the choices NIH made for how to spend the money. And NIH now has $10B of stimulus money to spend in the next two years. How hard could it be to find some dollars for risky proposals in that pot? This is not about dollars - it's about a fundamentally risk averse culture at NIH.
NIH is trying:
The National Institutes of Health has started “pilot experiments” to see if there is a better way of getting financing for innovative projects, its acting director, Dr. Kington, said.
They include “pioneer awards,” begun in 2004 for “ideas that have the potential for high impact but may be too novel, span too diverse a range of disciplines or be at a stage too early to fare well in the traditional peer review process.” But only 3 percent to 5 percent of the applicants get funded. Now the institutes have decided to set aside up to $25 million for “transformative R01 grants,” described as “proposing exceptionally innovative, high risk, original and/or unconventional research with the potential to create or overturn fundamental paradigms.”
These look like attempts to shoehorn risky projects into the current system - the grants look relatively large, which has the dual downsides of attracting lots of applicants and making the reviewers cautious (nobody wants to be the one to waste $1M or more on a failed project).
I think NIH would have much more success getting transformative results with a tiered system: lots of small grants (e.g. $50K-$100K) to test ideas; a modest number of mid-range grants to further advance things that emerge successfully from stage 1; and finally, the more standard $1M-$2M R01s for things that succeed in the previous stages.
I think the key is having large numbers of small grants. This would provide a lot of freedom to explore, and when some things fail - and they will fail - you're not out that much money. The NIH's current strategy is like investing everything in bonds and blue-chips: treasuries, AT&T, and Wal-Mart. If you do that, you get a safe return, but there's a very real opportunity cost: you miss out on the occasional Google. Investing small amounts in lots and lots of risky ideas is like having a diversified portfolio of start-ups. Sure, there will be lots of Pets.coms, but you'll also get some EBays and Amazons that you wouldn't get otherwise.
I suspect that there are two main reasons the NIH doesn't do things this way:
- First, I think that the people at or near the top of the NIH food chain would see any shift in resources away from R01s as a threat. You'll hear cries of neglecting "serious" research or the like.
- Second, reviewing and managing proposals is a resource-intensive process. The NIH has a hard enough time with getting their existing grants reviewed, and increasing the number of proposals funded would make things worse. I think funding smaller proposals would be easier, since the bar for funding would be lower, but still, it's by no means free.
In any case, I think it's great that this is making the front page of the Times. I hope that somewhere in the Obama administration, someone is preparing to light a fire under the NIH to get on this.
Sam Wang (an old friend of Peter's) and Sandra Aamodt have an interesting proposal in today's NY Times for spending a chunk of the NIH's $8B stimulus funds: hire a bunch of recent college graduates as lab techs in a program modeled after Teach for America. The goal is to provide the needed supply of cheap lab labor without producing lots of extra scientists. I think the intent is good, though I wonder how useful recent college graduates might be in the lab. I wonder whether having recent college grads in the lab would encourage more people to go to grad school or less. It might encourage more to become grad students by familiarizing people with the lab - doubtless inertia would carry many on to further study. On the other hand, many would find out how much of a grind laboratory life can be. Richard Freeman published a great survey a few years back that showed that the more contact undergraduates had with grad students and postdocs, the less likely they were to want to go to grad school. I worry that without some change of course the NIH's windfall is going to be as much of a disaster as their budget doubling has proved to be.
The Times article points to a silly op-ed in The Scientist calling for the country to dedicate a fixed percentage of the GDP to scientific research. I, too, would like a fixed (and non-small) percentage of the GDP dedicated to my work. More seriously, though, I agree with the author's claim that variability in funding levels is bad for science. But pegging research spending to the GDP is definitely not the way to go. That would mean one more system awash in money during good times (and all the problems that entails - e.g. the NIH doubling) and starved during the bad. If anything, I think one would want a countercyclical mechanism: cut back government support when the economy is booming and increase it when things go south. During boom times it should be easier to find alternative mechanisms of support - foundations, corporate partnerships, and so forth. During busts, the government picks up the slack.
Thompson Reuters has a new report out showing US market share of scientific papers in continuing decline relative to Asia. That's not too surprising given that China and India are both rapidly modernizing their economies. One thing from the report that struck me, though, was this:
But one trend bears watching: in the last two years, the absolute number of Thomson Reuters-indexed scientific papers from the U.S. has edged downward: from a high of roughly 291,500 in 2005 to approximately 286,000 in both 2006 and 2007.
What would cause paper output to decline? I'm sure many will point to the stagnation in the NIH budget as the cause, but I'd be surprised if that were the full story. Surely the insane pressure that people are now under has compensated to some extent for whatever difficulties reduced equipment budgets and the like have caused. And publications are a lagging indicator - the work done for papers published in 2006 was probably done in 2004, right at the end of the doubling.
I wonder if part of the story is related to changes in lab sizes?
Two factors may be contributing to an increase in lab size:
- The NIH has shifted some funds from individual grants to bigger projects.
- The NIH budget doubling has substantially increased the variance in the grant making process. Individual researchers are less likely to get grants, but the grants are for more money. One way to cope with increasing variance and uncertainty is to band together. If you partner up with someone doing similar work, submit independent proposals, and agree to share the proceeds (i.e. both people work on the winning proposal), you maintain the same expected return per person but have greatly reduced variability (variance reduction is an important reason that people form companies). I'm guessing that lab sizes have increased at a lot of institutions in part for these kinds of reasons.
One of the drawbacks to increased organization size is increased communication overhead. With N people, you have O(N^2) channels of communication to maintain. If you double the number of people working on a project, you don't move twice as fast. You have to spend some fraction of your time keeping others apprised of your status, dealing with resource contention, and so on. There is a classic book, The Mythical Man Month that all software engineers are required to read fairly early on in their careers - it's about an attempt by IBM back in the 70's to ship an operating system faster by adding more programmers to the project. The resulting explosion of communication overhead ended up delaying things even further.
This is all idle speculation on my part at this point - I don't really know if lab sizes are tracked - but given some recent anecdotes I've seen about the increasing role of Big Science, something to consider.
There's a piece in this week's Chronicle about some possible changes in the NIH's grant-making process. About 6 months ago, the NIH solicited suggestions from the general public for ways to improve the process, and an advisory committee has been sifting through the thousands of ideas they received.
The article describes a few broad classes of ideas:
- Streamlining the application process by reducing the length of grants from 25 pages to 15
- Limiting the number of proposals a person can submit
- Basing funding decisions more on an individual's than on specifics of their proposal
- Providing more affirmative action for younger scientists
These ideas aren't yet official recommendations - those won't be out until later this week - but they are likely indicative of the kinds of things the NIH will actually do. Many of these ideas are good ones; I'm just not convinced they will have the hoped-for effects.
Reducing the amount of effort required to submit a proposal sounds great. People invest huge amounts of effort on their proposals; I'd much rather have them spending their time doing science than chasing money. The trouble is that making it easier to submit a grant, will probably mean that people will submit more grants, driving the success rate down even more. Going back to the lottery ticket/grant analogy: during the budget doubling, the NIH increased both the odds that a ticket would win and the amount of money paid out by a winning ticket. Not surprisingly, people bought a lot more tickets. Streamlining the proposal process, while a worthy goal, effectively cuts the price of a ticket, which again increases the net payoff. If this happens, I predict we'll see even lower success rates in the future.
Limiting the number of proposals someone can submit is a non-starter, I suspect, despite the AAMC's endorsement of the idea. The idea has some merits: it would probably reduce the number of proposals the NIH receives and force people to submit only their best ideas. However, I think that there are legitimate scientific reasons for some larger labs to be submitting multiple proposals per year. Zerhouni opposes the idea. A better alternative might be to impose a surcharge, like publication fees charged by journals, to cover review costs on proposals after the first. This would reduce the number of people submitting multiple proposals while still making it possible to do so.
Judging proposals on the reputation of the submitter rather than on their content is a recipe for all sorts of trouble. Sure, it would make life more convenient for the elites, but I suspect that the result would be complacency, not greater willingness to take risks - just throw something over the fence and you get your funding, so why make the effort? Younger scientists are a source of a lot of crazy new ideas, but they don't have much of a track record, so this kind of scheme could shut them out of funding even more than they already are.
There's an alternative approach that I think avoids most of these difficulties - more next time.
Nature has been running some good stories this past month on the mess at the NIH.
Universities and the money fix, by Brian C. Martinson, points out what I think is the central problem:
[L]argely because of the structure of the funding flows between the NIH and the universities, there are few checks in the system to keep competition for grant funding at a healthy level. Thus, calls for further increases in the NIH budget may only make matters worse. In my view, it is time to ask the biggest beneficiaries of NIH largesse — the universities and academic health centres — to find ways to balance supply and demand that better reflect their obligations to researchers and society....
(emphasis added) Exactly right. The current problems are structural in nature. Anything that fails to address the underlying issues can only serve at best as a temporary stopgap.
There are insufficient 'feedback loops' linking the production of biomedical researchers to the availability of resources to support them. Instead, the educational system is replete with incentives to generate ever more PhDs and medical doctors. In the short term these arrangements may benefit universities, but in the longer term, such extreme levels of competition for funding are unsustainable. And they may already be doing harm.
The harm Martinson sees is greater potential for ethical lapses, something also predicted by David Goodstein. Reader BioScientist has more to say about the potential for harm in a comment on the NSF's new postdoc mentoring requirement:
None of these proposals matter, nor will any changes or improvements to them. The bottom line is that as long as there are too many scientists present the competition to stay alive will be intense and therefore conditions will remain poor. NIH/NSF proposals like this attempt to legislate behavior without recognizing the realities on the ground.
The lack of attention to graduate/post-graduate training is borne from two areas: 1) PIs are pushed by the system to extract every bit of effort possible effort from employees. The penalty for not doing so is a loss of funding and the end of a career. 2) As long as there are multiple applicants for every scientist/PI position there's less need to insure development of most individuals. As an example, one can simply ignore the bottom sixty percent of CVs to no ill effect when considering tenure-track positions. Those that remain will no doubt be pretty impressive. In this respect, post-graduate training is less an education than it is a 'selection' process in the biological sense. Put another way: why bother training postdocs, when most of them will fall out of the "system" anyway? The best will claw their way to success on their own and the rest are irrelevant. (This is a sentiment I've heard multiple times from faculty at my current Tier-I, top-20 research institute.)
Until the supply of PhD scientists comes back into line with demand working conditions will remain poor, salaries low, and hours long. In absence of a solution to the supply/demand imbalance all attempts to solve the resultant phenotypes will fail.
While I agree with Martinson's diagnosis, he unfortunately doesn't offer much in the way of solutions:
So is the only solution to force long-time NIH grant getters into retirement? Perhaps not. Universities have benefited handsomely from the efforts of senior faculty members in securing NIH grants during their careers, perhaps those same universities could now return the favour by taking full responsibility for paying these faculty salaries in their later years. This would serve the dual purpose of getting them off the NIH dole, and encouraging them to share their knowledge with their younger colleagues through more teaching.
Getting enough senior scientists to give up research for teaching to make any kind of difference seems, um, implausible.
Ginny C just pointed me to a recent FASEB presentation that summarizes recent trends in the life sciences labor market. It's great that they have done this, since I suspect a lot of people don't know the big picture, and FASEB has a very broad reach. Give it a read.
There is a great deal of overlap with Paula Stephan's findings and a few other things that Peter and I have discussed here.
A few things that struck me in the slides:
I knew that numbers of women have been increasing rapidly in the life sciences, but the graphs in the presentation are still pretty striking. Ditto for the number of postdocs on temporary visas. Success rates for NIH fellowship applications have been falling almost as fast as for R01s. They're down from ~45% in 2001 to ~27% in 2006. NIH spending on students as a percent of the total budget is down from ~4.3% in 1985 to ~2.7% in 2006 Foreign PhD recipients are increasingly staying in the US The fraction of all US biomedical PhDs who are tenured or in tenure-track positions is steadily decreasing. ~46% in 1981 to ~28% in 2006 Almost all the new positions created during the NIH doubling period were MDs in clinical departments Hiring of PhDs by med schools has pretty much ground to a halt in the last couple of years. Average GRE Quantitative scores are surprisingly low for life sciences folks: 529 for health sciences and 606 for biological sciences applicants (out of 990 total). I have always wondered if part of the reason the labor market for life scientists is so much worse than for physical scientists and engineers is that quantitative skills give the latter folks more options.
There's an intriguing article in yesterday's Times about a new theory about the factors that gave rise to the Industrial Revolution in England.
For centuries, England's citizens lived on the brink of starvation. Although innovations would periodically increase agricultural productivity, greater access to food invariably led to population increases, which in turn brought per capita food levels right back to where they started. It took the Industrial Revolution to finally bring the growth rate of the food supply above the growth rate of the population.
Historian Gregory Clark's study of wills from 1200-1800 found the following:
Given that the English economy operated under Malthusian constraints, might it not have responded in some way to the forces of natural selection that Darwin had divined would flourish in such conditions? Dr. Clark started to wonder whether natural selection had indeed changed the nature of the population in some way and, if so, whether this might be the missing explanation for the Industrial Revolution....
Generation after generation, the rich had more surviving children than the poor, his research showed. That meant there must have been constant downward social mobility as the poor failed to reproduce themselves and the progeny of the rich took over their occupations. “The modern population of the English is largely descended from the economic upper classes of the Middle Ages,” he concluded.
As the progeny of the rich pervaded all levels of society, Dr. Clark considered, the behaviors that made for wealth could have spread with them. He has documented that several aspects of what might now be called middle-class values changed significantly from the days of hunter gatherer societies to 1800. Work hours increased, literacy and numeracy rose, and the level of interpersonal violence dropped.
Clark speculates that there may be genetic and/or cultural components to these changes in behavior:
Dr. Clark says the middle-class values needed for productivity could have been transmitted either culturally or genetically. But in some passages, he seems to lean toward evolution as the explanation. “Through the long agrarian passage leading up to the Industrial Revolution, man was becoming biologically more adapted to the modern economic world,” he writes. And, “The triumph of capitalism in the modern world thus may lie as much in our genes as in ideology or rationality.”
What was being inherited, in his view, was not greater intelligence — being a hunter in a foraging society requires considerably greater skill than the repetitive actions of an agricultural laborer. Rather, it was “a repertoire of skills and dispositions that were very different from those of the pre-agrarian world.”
I don't know enough about behavioral genetics to have a sense of whether this is plausible; regardless, his application of Darwinian thinking in this particular case is intriguing.
Substitute funding for food, and it's clear that the current NIH mess is a Malthusian crisis. And as with England, this is only the most recent of a series. What are these selection pressures doing to the population of academic scientists?
When times are tight, it becomes a lot less pleasant to be an academic. Fewer proposals get funded, and even if you do get funded, a lot more work has to go into your proposals. Industry starts looking a lot more attractive by comparison. Increased numbers are forced out, and more interestingly when we start thinking like Darwin, increased numbers either leave by choice or never seek academic careers in the first place.
Economics tells us that the more attractive one's industry prospects relative to academic alternatives, the more likely one is to end up there. And if you work in industry, you don't "reproduce" by training students. Thus, academia's Malthusian crises may very well be selecting against those who are most capable of success outside of academia.
Funding levels in academia are driven by the prospect of economic returns to investments in research. Unfortunately for all concerned, unlike Clark's hypothesized England, academia appears to be selecting against some of those most capable of increasing the "food" supply.
While the NIH budget doubling has created a crisis for life sciences grant applicants, what about its benefits? One of the most pronounced effects of the budget doubling was a huge building spree by medical schools. Presumably there has also been an increase in research output, right?
A new NSF report makes me wonder. According to the press release, "the number of U.S. science and engineering (S&E) articles in major peer-reviewed journals flattened in the 1990s, after more than two decades of growth.... Flattening occurred in nearly all U.S. research disciplines and types of institutions."
I haven't read the full report yet, so I'm still scratching my head. Here's the report, and here's a discussion on Slashdot.
The Chronicle reports that the National Science Board (the overseers of the NSF) are urging the NSF to fund riskier projects with the potential for big breakthroughs rather than safer, more incremental projects. (Here's the NSB report)
Taking more risks is likely to be a good thing for NSF. Consider the financial analog: it's like holding nothing but T-bills in your portfolio. Adding stocks (a riskier asset) to the mix in the right way can dramatically increase returns without greatly increasing risk.
I hope the goal of increased risk-taking on "transformative" research finds its way into the NIH, because if anything, the NIH is even more risk-averse than NSF. How much more can you do to avoid risk than the NIH's policy of (a) requiring in essence that people demonstrate that their projects will work before funding them (requiring "preliminary data") and (b) placing heavy emphasis on an established track record? These two policies are particularly hard on young people, who research suggests are more likely to come up with creative new ideas: because they are young, they don't have much of a track record, and and without an existing lab and funding in place, it's quite difficult to generate preliminary data. The result: less and less funding for young researchers.
The NIH is undergoing an examination of its own grantmaking processes as well in the wake of plummeting rates of grant success. Perhaps the National Science Board's report will shape their thinking?
View archives for June 2009.
|
|
