43 Comments
On my first day on the job as NIH Deputy Director for Extramural Research, one of my colleagues asked me a question: Is it true that it is more difficult to renew a grant than it is to get one in the first place? Some people wonder whether NIH’s interest in supporting new investigators who are trying to get their first grant negatively impacts the other investigators’ attempts to renew their grants.
To address this question we gathered data on R01-equivalent success rates for new and experienced investigators seeking funding in fiscal years (FY) 2013 through 2015. We define success rates as the percentage of reviewed grant applications that receive funding (note that applications having one or more submissions for the same project in the same fiscal year are only counted once). We define new investigators as investigators named as project director/principal investigator (PD/PI) on R01 applications who have not had a major competing research award (such as an R01 or a program project grant). For the purpose of this analysis, we also included investigators who had received their first R01 within the last six years and submitted a competing renewal R01 application in FYs 2013, 2014, or 2015
In 2013, new investigators submitted 11,386 applications for new grants, and investigators within 6 years of their first R01 grant submitted 854 competing renewal applications. Experienced investigators submitted 11,488 applications for new grants, 1210 applications for a 1st renewal, 1060 applications for a 2nd renewal, and 1240 applications for a 3rd (or more) renewal.
The figures for 2014 and 2015 were similar.
The graph below shows our main findings. New investigators were almost as successful as experienced investigators in obtaining type 1 awards (14% vs 15% in 2013, 15% vs 16% in 2014, 16% vs 17% in 2015). The difference between new investigator and experienced investigator success rates widens when looking at competing renewals. Success rates of new investigators’ first renewals were lower than those of experienced investigators (25% vs 28% in 2013, 26% vs 32% in 2014, 27% vs 31% in 2015). Finally, when looking at those applying for 2nd and 3rd renewals, experienced investigators had higher success rates (33% and 39% in 2013, 37% and 44% in 2014, 38% and 41% in 2015).
When considering only new applications, and applications for a first renewal, applications for a first renewal were much more likely to be successful (odds ratio 2.2, 95% CI 2.1 to 2.3, P<0.001). Experienced investigators were slightly more likely than new investigators to be successful (odds ratio 1.2, 95% CI 1.1 to 1.2, P<0.001).
To answer the question we began with: renewal applications have higher success rates than new applications, and that this pattern is true for both new and experienced investigators. We will continue to monitor these trends across NIH as part of our efforts to understand new and early stage investigators’ experiences in seeking NIH support.
PS – Many thanks to the OER staff in the Division of Planning, Analysis and Evaluation who helped assemble and analyze these data. These data are also posted to the report archive on RePORT.
Isn’t a more critical question about success rates of the experienced investigator cohorts when they were putting in their first renewals?
You need to look at the long term health and productivity of the extramural workforce. If we merely focus on the current size of the bias against younger scientist cohorts we miss the forest for the trees.
You might want to take a look at numbers by study section. Some study sections are much more “top” heavy than others. Sections don’t appear to have similar distributions of funded grants looking at support years, though a confounding variable could be what institutes their grants tend to go to. Sections supporting many more long running grants are likely to be much more difficult for 1st/2nd renewals, but also more likely to exist in GM, where paylines have been guarded.
Typical NIH misinformation, meant to pat themselves on their collective backs. In my experience, I have found that there is a much larger number of new investigators being funded than experienced or established investigator. This has had a devastating effect on older investigators [myself included]; I know of at least 9 investigators who have left academia. What is happening NIH? In my own building, the corridors are eerily empty and silent, as lab after lab has shut down.
My question to NIH staff: Why can you not have a much more balanced distribution of grants, whereby older investigators are not ignored?
I have now been without any type of NIH funding for the past 5 years, during which time I have applied for at least 33 grants [NIH + non-NIH], with no success, whereas in the past, I have constantly had at least 2 R01 grants [since 1991]. I let my people go, closed my lab down for months and months, and still nothing!
The second question for NIH staff….why are some labs grossly, impossibly overfunded? Case in point: One Investigator has 6 NIH grants ranging from 200-717k, for a grant total of over $4,000,000 annually!! His wife, who has a grand total of 39 publications, has 5 NIH grants, ranging from 360-660, for a total of $2,600,000! Really NIH?? How can you possibly justify these award levels, especially for the inexperienced wife [who incidentally did not publish for 5 years during her career and co-authored with the husband for 6 articles last year].
Yes, good science deserves money, these are very tight times, we prefer young investigators etc etc, but how do you explain the enormous inequity?
At age 58, Im too young for retirement and too old to qualify for anything….
“The inexperienced wife”. Yikes.
Just another example proving that the proposal does not matter, but there are other criteria that NIH reviewers use to judge our application.
The thing I find disturbing is that on average a new investigator only has a 1/4 chance of getting their R01 renewed. Makes you wonder why anyone bothers doing science anymore. That’s a worse success rate than starting a new restaurant.
Colin, that is an interesting and perhaps apt analogy. It does seem the best food and science are forged in highly competitive environments.
Ouch. “New investigators were almost as successful as experienced investigators in obtaining type 1 awards.” The word _almost_ in that sentence corresponds to 480 new investigators who weren’t funded who would have been if the percent funding was identical for New investigators and Established investigators. 189 in 2013, 181 in 2014, and 110 in 2015.
Don’t get me wrong, mid- and later-career scientists are also feeling this environment of rejection.
A major concern that only began to be questioned in the dialogs above is age discrimination against older scientists. I know of one scientist over 65 who received a ‘pink sheet’ in which it was commented that he was too old. When he called about it, the document was revised to avoid any mention of age, but it obviously influenced the scoring. It seems like this sentiment is becoming quite pervasive. In these hypercompetitive times, many younger scientists feel that older ones should step aside. “They had their chance.” It would be interesting to look at success rates for scientists over 60 yo.
Great science should be evaluated consistently across the board no matter what the age of the scientist – no??
Comparisons have to be made at similar age *and* positions. Seniors hold higher positions in the institution, larger laboratories, more grants, more allowances, more technicians and students, really more anything. Young scientists don’t have the same.
This has become endemic of academics in general. Look, for example, at hiring practices of liberal arts colleges and universities; anyone whose career has passed the post-doc or VAP stage need not apply. There is a strange, but strangely predictable, age bias that similarly extends to NIH.
What is absolutely shocking, and is not discussed here, is for the renewals of RO1s. Why don’t you look at the number of first time RO1 recipients who submit a renewal and get scored but are not funded, and then on the A1 get triaged! How many times have people been so close, answer all the questions, and get triaged by the changing of the reviewers? Is there anything more demoralizing to almost make it on first renewal then be triaged at the A1?
I have seen this happen to many investigators both experienced and new with A1 submissions. It puts the entire review process in question. Is the system broken or bent on breaking the confidence of our investigators?
Thank you the comment. We get scored (!) and then answer all of the questions, provide more data, resubmit… and then get triaged! Something must be wrong as how can it be worse? The process is broken. To me there is either bias or you get completely new reviewers who think that they know best and ignore the initial reviews. So is the answer then you MUST get it on the first submission and don’t even try for an A1? I have heard this from multiple people in the last 2 weeks since this happened to me. It is unclear to me why that if I can see this that the NIH cannot.
I wonder why we all see these things so clearly but no one from the NIH ever answers these comments.
Yes this has happened to me 2x recently. In both cases I got a jit for the not discussed application, implying the score was in a reasonable range but the SRO made a cutoff for discussion that led to triage. I do not understand why the NIH does not publish the raw scores as well as those for discussed grants. Further, if a grant was discussed the first time, it should at least be discussed on the second round if the actual score was not significantly higher. I have also heard from other researchers with similar experiences.
In one case, the grant was to design peptides to mimic areas of the Ebola surface proteins. The grant went back to the same study section, after filling in (as requested) even more details of what areas we planned to target. So we gave details to the reviewers that they could possible use to improve their own grants or internal company work, since without funding we were not likely to publish. I have little faith that any software could detect this sort of unapproved data sharing.
If a study section can erroneously score your grant too poorly (as has happened to almost all of us), it can also score your grant too well.
Of course, no one complains when their proposal gets a better score than it should! It stands to reason that scores may go down on a revision. You can think of it as an error correction mechanism.
The label “renewal/continuation suggests that non-competing continuations are counted in this analysis. Is that correct? If true, then the numbers are inflated.
All of the analyses were looking at competing applications. Non-competing continuations were not included.
Curious that NIH responds to this but not to the difficult questions that we all have raised here.
Seems the key point is being missed. We all agree that success rates for both new grants and competing renewals are lower than, say, 12 years ago, The key question is: have success rates of competing renewals decreased disproportionately from then until now compared to new grant applications. My sense is that it has. It seems more difficult now to get a competing renewal (compared to back then) than it does to get new grants now (compared to back then). These data should be available to NIH to address this question.
I have currently one RO1 but I need to renew my second RO1 to be able to fund the expensive in vivo preclinical studies that I engage in. I have send several grants to NIH only to be rejected again and again. I am getting quite desperate and it is clear that I will need to fire qualified postdocs. I write grants almost 80% of my time, I think I will get crazy soon. I submit a grant to NIH every cycle. I publish in high impact journals, I am very productive, but it is not enough for my study section that just does not care of my type of studies. I am starting to look for a new job outside of academia, I cannot be only writing grants to be rejected again and again. I will be fourth tenure track faculty leaving in our division of 7 faculty. Job well done NIH!
So sorry about your situation. Mine is similar and also looking for jobs outside academia. But you’ll find, if you haven’t already, many specifically don’t want mid-level scientists who have a long track record in academia and don’t care if we’ve had R01s b/c we come with an agenda. They prefer post-docs and early scientists.
my experience exactly, mid career academic phd scientist – uniquely over- and under-qualified for non-academic work
I think the statistics must contain errors. I have personally witnessed in the last 20 years how the rates of success to get an R01 have gone down dramatically, not only for me but also for my colleagues, including from other departments.
We have an ethical duty to inform the young scientists of this worrying downward spiral and their realistic chances to become successful (NIH funded) investigators. I have seen statistics that only 1 in 4 applicants eventually get funded by the NIH. That is the WORST chance I have heard in any profession! I wish I knew that when I had chosen a career in research.
I would truly like to know if NIH people have even an idea how a life of average PI looks like these days. Instead of designing experiments, teaching students, analyzing data, reading papers, writing papers, books etc we spend majority of our time just writing grants. New and new requirements are added while the success rate is hovering around 10%. One reviewer doe not like something and the grant is done, on the floor. Try again and again. Such a disregard for qualified scientists time, such a waste of talent but we still pretend this system works? The problem is not who is more funded young or experienced, the problem is that nobody is funded in rate remotely sustainable, that would be around 30% as it was in 1990. The whole premise of RO1 grants needs to be rethink funding people based on their productivity, publication in past few years would be more sustainable. Shorter simple applications, instead you are constantly asking for more documents like the recent change. Pretending that life of average PI is anything but a constant grant writing misery is laughable. One solution would be to step on Universities that they need to start to guarantee faculty salaries, hire less people and take care of them, use some of that high indirect cost for that.
Oh, I could not agree more!
In our institution the indirect costs have gone way up. And that is only used to hire more administrators, more regulators, more high school graduates that come up with more rules for us, who actually spend day and nights to work on grant writing and working hard to come up with breakthroughs to bring fame and $ to our institution – and keep our jobs. It has become so ridiculous, so much waste of money in administration, so much waste of talent – and focus, really.
Here Here! Let’s add more regulatory burden, more useless documents on resource sharing, and why on our R21 we are asking for $125,000 in the first year and $150,000 in the second. We spend all of the time on this on top of of all of the university documents that we can’t focus on the science. And we pay 54.5% indirects here and for what? So they can pay the administrators to ask us for more useless documents!
It would be interesting to know how often the different Institutes approve renewal applications.
We do have data on success rates for renewal applications by NIH Institute and Center on RePORT. Check out http://report.nih.gov/DisplayRePORT.aspx?rid=565 (it is an Excel table).
Based on my own experience and what I noticed from the recent established culture of research and hopeless funding situation, I have been discouraging any life science students, PhD candidates, new PhD student, or postdoc to stay, remain or have any hope (for career or life) in biomedical research. These new (naïve) generation have no idea what are they getting into!!!! It will be such a big disappointment for them to find it later ….
I am surprised that nobody is pointing out the most obvious differential: between the number of new applications and the number of renewals (even if you limit to experienced investigators). This means that most new grants do not in the end generate data useful enough to justify a renewal. On the other hand, compare #s of first renewals to second renewals to 3rd or more – about stable. Meaning that grants that get renewed once typically are successful enough to generate data supporting a renewal.
These numbers speak volumes to me. New grants are manifestly less likely to generate valuable scientific advances compared to renewals. The NIH is clearly overspending on new grants and underspending on renewals.
How to help new investigators overcome the disadvantage of not understanding the system well is still an issue that the NIH should be considering, but perhaps our system just can’t support a continual expansion in new investigators.
This data does not mean anything. You need to do a well controlled experiment. Take an adequate number of grant renewals and send them as new R01 applications and as renewal applications to study section and evaluate what are the preliminary scores they get. I strongly suspect that they wills core significantly worse as renewals than as new applications.
New investigators are favored, there is no question about it. I support this. They have to be. If not they would not stand a chance. The debate should be: are they overly favored or are they not favored enough?
I believe NIH grant review system will be more efficient if the proposal pool is divided into three groups.
1. New investigator (as it is for now)
2. Mid-career
3. Senior
It is fare to compete with your peers. Otherwise, It is like a high school basketball team plays against a NBA team 100 times and how many of those you think high school team can win. All we do is to wait for a Cinderella team.
MBA/PhD–This is supply and demand at work–there are too many of us PhDs (supply) for the available research $$ (demand). In my home state, the number of MDs awarded is controlled by state law, and residencies are almost never given to foreign-trained physicians. Keeping the supply of MDs down allows the price/salaries for MDs to stay up. Unless the number of PhDs awarded and the number of incoming, foreign grad students and postdocs is controlled to match research funds, the situation will only get worse.
I believe what we are hearing is the overall rate for grants submitted as a renewal. BUT the total pool of renewed R01s is down 25% over the last 5 years. Thus, this “success rate” is highly deceptive. (I also object to to the idea that 2 submissions is one “success”. What are the success statistics when each submission is counted separately?)
Also, after two years of missing the payline with a long-term award, I just received an R56 for 112K total direct, for one year. I believe this R56 also counts as a “success” in these statistics. While it is very welcome, this R56 cannot be called a successful renewal by any stretch of the imagination.
I encourage CSR to run more numbers on renewals to compare the renewal funding situation now to what it was three years ago and also give us total numbers of new and renewed grants (as well as actual success rates per submission).
I also believe that investigators with longstanding grants are getting preferentially slammed- I don’t see a lot of high numbers on NIH Reporter anymore- is this just my impression, or it is really happening? It would additionally be interesting to know if this differs by Institute. Some Institutes might try to protect PIs in whom they have a considerable financial investment if they score in the top 20%; others may not.
Lastly, I would like to take up a point raised above and ask if the rich are indeed getting richer at the expense of the smaller labs- how many PIs have 2,3, or 4 grants now, as opposed to three years ago? Has there been any real attempt made to keep labs open by limiting total funding? Since it is well accepted that panels cannot differentiate between grants in the top 20%, I keep wondering why Institutes do not take this initiative. Productivity is not proportionate to funding with more than 2 R01s.
“Productivity is not proportionate to funding with more than 2 R01s.”
As Jeremy Berg has shown for NIGMS, productivity is not even proportionate to funding for more than ONE RO1.
How many of the people responding to this blog post have ever served on a study section? Those study section members who are reviled here, are investigators just like you, with all the same demands on their time. They have generously agreed to read the proposals, to try to split hairs to rank them, etc etc.
The fundamental problem is that the amount of money is too low. With low funding rates, fairness can only occur over several years, and not in a single study section meeting.
You are absolutely correct, Laurel. Right now I have 9 grants to review. At least 3 of them are outstanding and deserve to be funded. Two more are excellent, and in my opinion, just below the three outstanding proposals. Of course other reviewers may see these differently and probably will. But, out of my top 3, the odds are only one will be funded. This means one of us reviewers needs to make up some reason to ding a couple of them. Often times the reason will be trivial and not the real reason. The real reason will be something such as “This guy is too old and ought to be making room for a young person,” or “This gal is not really from a top postdoc and she took off a couple of years from publishing, maybe to have a baby, but who knows?” or “This isn’t from a top institution on the coasts.” Or it could even be a legitimate reason, such as “Interesting work, very interesting and well done, but I just don’t think the problem is that significant.”
Now consider my situation. If I give all 3 of these the score of 1 they deserve, other reviewers will begin to discount me when I do fight for a proposal. “Oh, that’s just Bob. He likes everything and isn’t critical enough.” My experience also has been that if I pick one to really champion, I run across another reviewer who has his or her own favorite to champion and who will fight back and not accept my assessment because to do so would mean his or her favorite likely will fall below the pay line in the study section.
The ones who do get funded, I maintain are the recipients of good luck, pure and simple. They managed, against the odds to acquire three reviewers who would give it a score of 1. Now the next time the two of mine who will not be so lucky and who got a 3, possibly for bogus reasons, but reasons nonetheless, plus the two who have fixable minor problems will be even more unlucky in the draw of reviewers and will receive the dreaded triage.
Clearly, our whole system is unsustainable and unfixable. As someone else stated, unless the number of PhDs is controlled and the importation of foreign graduates is restricted, the number of people seeking grants will continue to foster this kind of situation. But, to restrict the number of PhDs would be to eliminate the reason for being for many faculty, namely to to train graduate students, while also eliminating the pool of cheap labor that faculty rely on to get their work done. And, in the end, progress toward solving problems such as cancer and other diseases will inevitably slow. Yet, radical changes could at least improve the quality of life for those who do make whatever bar is set. Right now, most of us spend our time writing grants instead of papers, and doing other things such as administrivia. (I am so behind on my mandatory training in this, that and the other thing that some well-meaning person who has never done anything productive in his or her life thinks I need to know). Our families suffer if we want to remain among the Chosen. Anyone know what the divorce rate is for our profession? Cynic that I am, I know that if the NIH budget increases, the main result will be just more research track faculty, or even tenure track faculty who are responsible for their own salaries.
Our system is unsustainable, and all the solutions require pain for some entrenched special interest.
I disagree. The review system is also broken. Having only two tries means that a third submission may get an entirely new set of reviewers and most likely will get scored worse (this would be an interesting statistic to gather, CSR!). Enabling the former system with a third submission would eliminate this effect. Secondly, having a system in which reviewers can give scores that are more discriminating than the 2, 3, or 4 units now used to score all reviewed grants (1 is holy and is almost never used) would greatly improve peer review as it would help to actually RANK the best-reviewed grants. I have served both in the past and recently on SS and I no longer feel the most meritorious grants get funded. It is basically chance- unless there is someone you know on the study section who believes in your approach.
I would like to know the breakdown of funded competitive renewals by women PIs versus by men PIs.
According to the table for 2015 there were 3969 new awards (both new and experienced investigators) but only 1396 renewals. This is a ratio of almost 3 to 1 in favor of new awards … so clearly new awards are favored over renewals.
The high success rates for renewal applications would suggest that most people are simply not sending in renewals any more. There could be many reasons – the bar is high for a renewal. With publications sometimes taking several years, the need for significant supplemental data, and the money running low many people simply opt to submit a new grant instead.
In 2010 this ratio was closer to 1.5 new awards per renewal. The resulting gap in the continuity of research support is being felt everywhere, but what is not clear is whether this shift toward new awards will have an impact on the quality of the research…. New stories may be compelling, but the classics are also worth support.
New investigators were almost as successful as experienced investigators in obtaining type 1 awards (14% vs 15% in 2013, 15% vs 16% in 2014, 16% vs 17% in 2015). Where such information?