89 Comments
We have heard from many of you about the policy to sunset A2 applications. As you may remember, this policy was developed as part of NIH efforts to enhance peer review. There were concerns that applications were piling up in a “queue” and subject to a holding pattern that delayed funding until the resubmission (A1 and A2) stages, and as a consequence highly meritorious science proposed in original (A0) applications was made to wait additional months for funding. The policy was published in the Federal Register on October 8, 2008, and implementation began on January 25, 2009. In March 2011 I posted some early data showing that the number of applications funded as A0s was increasing and there was no queue at the A1 level.
More time has elapsed since the policy was implemented and this has allowed us to gather a substantial amount of data and conduct more in-depth analyses of the effects of allowing only one resubmission for each application.
First, as can be seen in figure 1, which looks at new (type 1) competing R01 and equivalent awards, the early analysis was confirmed – a higher proportion of A0 applications are being awarded compared to A1s and the “queuing” of original applications and revisions has been eliminated. In addition, the average time to award from submission of A0 applications has been reduced from 93 weeks to 56 weeks (figure 2). Although it is unsurprising that the time to award has decreased since the elimination of the A2, the increase in the proportion of funded A0 applications versus A1 applications was not an obvious outcome. 
Based on these analyses, the policy is achieving its goal of funding a higher proportion of original applications more rapidly. But we realize that these data alone do not address some major concerns, such as whether the new policy disadvantages new investigator applicants who may find it more difficult to craft new applications. One indication that this is not the case is the fact that the time to award for new investigators is not appreciably longer than the entire population (figure 2).
These results notwithstanding, we are well aware that a lot of great science goes unfunded because paylines have decreased. Some have suggested that to deal with this issue we modify the policy to allow A2 submissions for a specific subset of applications (e.g. those with scores just outside the payline). We have modeled the potential impact of changing the policy in such a way.
We looked at this issue using data from fiscal year (FY) 2011. If A1 applications with percentile scores below 25% were allowed to submit an A2 application in FY2011, it would have resulted in 764 unsolicited A2 R01 applications. 165 of the new applications were from new investigators. In addition, only a small minority of eligible applications (37 of 218 renewals) were from investigators trying to renew a previous new investigator award.
Assuming the most extreme case – that all 764 of these A2 applications would have been funded, NIH would have been able to fund 21% fewer A0 applications and 19% fewer A1 applications in FY2011 (figure 3). These displaced A0 and A1 applications would be highly likely to come back as A1s and A2s (as most displaced A1s would become eligible under the modified policy) and the average time to award would increase.
Overall, these data indicate that the policy to sunset A2 applications continues to achieve the stated goals of enabling NIH to fund as much meritorious science as possible in as short a time period as possible. Any revision to the policy to allow additional resubmissions of all or a subset of A2 applications will displace equally meritorious A0 and A1 applications, and increase the time to award for many applications. For these reasons, we have decided to continue the policy in its current form.
We greatly appreciate the community’s input on this issue and the thoughtful deliberations we had here with NIH leadership, all culminating in the decision to keep the policy. Please know that we will continue to monitor the policy and its impact on time to award.
—
Update from the Rock Talk Blog Team (November 28, 2012 at 2:10 pm): Data tables corresponding to the figures shown in this blog post are available at: http://report.nih.gov/special_reports_and_current_issues/index.aspx
Of course this interpretation about the holding pattern requires that the A0 is a genuinely new application, instead of a retread that has managed to pass the “substantially similar” filters. I’d like to see an anonymous survey of “A0” awardees which focuses on how new the ideas are and how much of the app has been reviewed previously in other guise.
Thanks DM. You just gave NIH an idea for a new report for funded PIs to fill out.
While it is true that the “No A2” policy is achieving the goal of getting more outstanding applications funded earlier, there are still several points of concern. A major one involves what happens to projects that are not funded after an A1. Often an investigator has many years of effort (and expenditure) associated the project and it may be in no one’s interest to completely change directions to something that the investigator knows (and cares) less about. With success rates where they are now, the science in such projects can be very strong. It may be particularly difficult for early stage investigators to change directions dramatically so that the question of where the bar is set for a “new” application (versus a disallowed A2 application) is crucial. Second, the original “Enhancing Peer Review” working groups actually recommended that all applications should be considered “new” (as a way of dealing with the apparent favoring of applications simply because they were A2s). This approach met with considerable concern from portions of the scientific community and the “No A2” policy was developed and adopted as an alternative. However, as illustrated above, the “No A2” policy runs the risk of having strong projects having no real recourse but abandonment or other non-NIH funding sources if the bar for what constitutes a new application is set too high.
I think this captures the concerns of many of us. The question is not about when a grant gets funded, but the fact that investigators are forced to abandon promising research because of the constraints imposed by the definition of what constitutes a “new” application.
You are correct that good science and projects are being abandoned. It is also alarming that great scientists are being forced to try to get science funded that they are less interested in doing, which leads to less competitive applications that alas go unfunded. The end result of this is that outstanding scientists are being forced to leave academic research because of this atrocious policy. Unfortunately, NIH shows not apparent will to reassess its policy that not a single academic scientist that I know of has agreed with. Instead, NIH spends its time continuing to defend these policies by developing and communicating analyses (like this one) that are self-fulfilling. Sadly, we have no recourse since the powers to be at NIH do not appear to listen to the concerns of the scientific community.
It is increasingly clear that the NIH has a goal of eradicating basic biomedical research. There is no other way to explain this perverse and truly diabolical policy that systematically eliminates the best scientists.
It is a little hard to believe that NIH administrators want to eliminate the reason for NIH’s existence. However, I do agree that a lot of good research and scientists are essentially being thrown away. The NIH officers may argue that the best research is being funded, but the problem is when the pay lines are so low – it is really luck to get funded or the scientists with the most friends among the good applications. I’ve never met a single person who can tell me that there is a legitimate difference between a 14th and a 7th percentile grant. Once you get below 15th percentile, I think the only way to judge them differently is to say that particular applications are addressing areas of greater concern identified by NIH. So it is really luck if you get a 7th instead of 10th or 12th percentile.
I think, it is not that purely. It is mostly crony system and a little bit of extraordinary luck system
Here’s a radical idea. How about identifying a cut-off point, say the top 15%, below which all grants get funded and then adjust the amount of funds for each accordingly. This would serve to distribute the pool a bit further, and force some much-needed austerity on everyone. I doubt the $2 wash basin from a super store is much worse than a fifty dollar one from one of the scientific supply houses
It’s not surprising that you came to this conclusion on the basis of empirical analysis of funding data and a mathematical model, since it is nothing more than an inescapable necessary consequence of the laws of arithmetic, as I pointed out almost two years ago:
http://freethoughtblogs.com/physioprof/2011/02/14/delusional-biomedical-researchers-seek-repeal-of-arithmetic/
A reasonable compromise may be to allow scored, unfunded A1’s achieving a threshold score (i.e. payline + 10%) the opportunity to be resubmitted as new A0 applications without computer comparisons for overlap. The statistics of time to funding, A0:A1 ratio etc will still look good, but investigators with nearly funded grants may survive.
I think that this proposal is an excellent compromise, though I know that this is a lost word among government agencies. It is well recognized that grants scoring at this level are rather indistinguishable with regard to actual quality. The difference can come down to the luck of drawing a favorable third reviewer in the study section to lower the score 1 or 2 percentile. I think NIH, scientists and their trainees would benefit from trying to define a reasonable threshold above the payline where an A1 application could be resubmitted without the silliness of trying to repackage to pass some arbitrary filter.
>> It is well recognized that grants scoring at this level are rather indistinguishable with regard to actual quality. The difference can come down to the luck of drawing a favorable third reviewer in the study section to lower the score 1 or 2 percentile. <<
This is the crux of the matter, which the published analysis completely ignores. Careers are being destroyed and important science left undone precisely on this arbitrary basis.
What other arbitrariy criterion would you choose. The real problem is not how grants are selected but rather is the disparity between the number of scientists seeking support and the amount of money available. No matter what, careers are being destroyed, simply because there is insufficient money to support us all. Drawing a favorable third reviewer is about as good a mechanism as any. I myself would prefer a system in which the role of peer review is to score proposals, with the understanding that paylines are not established. As is done by the Department of Defense, a second group consisting of program people and outside experts would then establish who is funded. The very highest group would be funded without examination, but because it is recognized that review committee scores are not particularly robust and do contain arbitrary elements, the actual funding decision would reflect other factors such as geographic or demographic diversity, national research needs, innovation, impact, and the the research portfolio. This would result in some lower-scoring proposals being funded over higher-scoring ones, which would certainly produce outrage in some circles. In the end, however, the real problem is not enough money for the number of investigators. Like a bacterial culture or an ecosystem, the equilibrium between consumers and food supply will eventually re-establish itself, no matter what is done.
I like Steve’s idea. The only thing I would add is that if you want your nearly-funded proposal to get another shot as an A0, you have to agree to serve on Study Section at least for one round.
I completely agree with Steve’s suggestion. Eliminate the comparison overlap for A1 applications resubmitted as A0s as long as they are within a small percentage of the payline. This will allow excellent science to continue on track and not require PIs to tweak their research such that they are no longer as excited about it.
Having served on multiple review panels for NIH, NSF, and others, I can tell you that there usually are one or maybe two proposals that everyone recognizes as the cream of the crop, and then quite a few more-or-less equally good proposals. It is this latter group that is the issue with which we must deal. It has gotten to the point that accurately rank-ordering this group of very good proposals is nearly impossible. We built the most productive scientific research apparatus in history by funding the top 25 – 30% of applications and, if anything, the overall quality of the proposals has gotten better. However, it is clear that the available funding will not cover that range anymore, so we are missing out on a lot of good science, losing promising young researchers, losing formally productive senior researchers, and a host of other problems. I would propose that those few grants that clearly are head-and-shoulders above the rest should be funded automatically, but then the second tier applications (say to the top 30%) should go into a lottery for funding. Those that qualify for the lottery, but that are not funded, are given feedback and encouraged to re-submit. Those that are reasonable but that are not included in the lottery will be allowed to re-submit at their discretion, but the reviewers could just provide the “bullet-points” feedback. Finally, there typically is a group of proposals that clearly have little chance of ever being funded – these applicants should be discouraged from re-submitting. We got where we are by funding the top 30% and there is no reason to think that the top 30% now are any less meritorious now than they were 15 years ago.
Having served on many study sections myself, I think we actually have implemented exactly the system you suggest. Given all the noise in the personal evaluation process, it is basically a lottery for the 30% right now. We would do best to accept this fact and plan accordingly.
You don’t need a very sophisticated set of analytical tools to realize what any driver knows that as you add traffic to the road, the speed slows down. What matters is what happens to those who were nearly at their destination and got stuck in the traffic jam: the almost fund-able A1s.
The PIs of those meritorious A1’s will have another option forced by NIH is to try to migrate…..
Of course A0 and or A1 will increase if A2 are eliminated. This interpretation is totally self serving. If A0+A1+A2 =100% of the pot of money…then remove A2, now A0+A1=100% of the pot of money..so now the % of A0 and/or A1 will increase. This doe not mean the science is better..it just means there is less competition for the $. In fact, some of the A2 may have scored better than the A0s if they were allowed to include and as you indicated in your analysis, they would have diminished the A0 %. This is a circular argument.
Exactly. Under the NIH logic they should eliminate the A1 application too to increase the number of A0s funded.
As noted by other, many of never sent A2s are likely to be better science than funded A0s.
Actually, I would like a complete removal of A1. This will eliminate the uncertainty if the reviewers’ comments are legitimate, or rather are a pile of [you-know-what] manufactured to justify keeping the application in a holding pattern.
Dr. Berg has made a good point. “[W]hat happens to projects that are not funded after an A1. ”
Well, there should be a mechanism allowing their further consideration, but -A2 is not a good solution, because it just encourages queuing proposals. I think such applications should be allowed to compete as new, even if they are substantially similar to those previously submitted. The low success rates are a fact of life now, and it is unavoidable that certain not funded applications are actually meritorious.
i don’t buy it. By eliminating A2s, more A0s are by default part of the applicant pool and more are, therefore, going to be funded. The funding equation is a zero-sum game and i bet all the “extra” A0s are reworked A1s that were good grants that didn’t get funded. Substantive scientists don’t reinvent their research on the time-scales of A0-A2 grant cycles. Sure, they repackage high quality work and perhaps this exercise is more beneficial than refining a fundable A1 application into an A2 version as in the past. But NIH should stop trying to justify what was, in my opinion, one of the worst changes in the application/review process that, to me, has almost no logical rationale, and certainly not the one being propagated by NIH.
Sometimes I wish this blog wasn’t available because its contents are difficult to read. A couple of concerns.
1) it’s noted that figure 1 shows the policy to be ‘working.’ As any budding scientist knows, it’s important to think about what the possible outcomes from an analysis may be. Therefore, may I ask what that graph would have too look like for us to conclude that the policy *isn’t* working? I hate to say this, but the shape would have been the same. The A0s were always going to go up. But for the sake of scientific rigor and transparency (of thought), it would be appreciated if Dr. Rockey described to us what that graph would have to look like for the policy to be deemed inappropriate / ineffective / etc…
2) ditto figure 3 which is self-evident. Part of me can’t help but mischievously look at that graph and think ‘well, more meritorious A2s got funded at the expense of less meritorious A0 and A1s. Isn’t that what we’re supposed to do? Fund the more meritorious projects?’ If we were looking at Fig. 3 through Fig. 1’s institutional priority colored glasses, it would be deemed a success. But in this context, funding meritorious science is evidence of a negative knock on effect.
3) I think many of us can agree that much of the frustration is due to the constraints posed by the limited NIH budget. In light of it, I do think the NIH has generally done an admirable job by managing the number of years awarded and shaving a few percentage points from funded budgets in an effort to maintain the total number of funded grants. But in light of all that, it’s really difficult to read some of these ‘analyses’ when they’re just post hoc exercises in policy rationalization and affirmation.
Agreed. Of course if there are no successful A2s the % of total awards as A0 (and A1s) goes up, as the graph shows. The system is simply not funding good science that sometimes in today’s rather capricious climate ends up as A2s.
A major advantage of the A2 is that, if you were unscored on the first round, you had two more shots to get scored and then funded. With the A1 as your last shot, if you get a Not Discussed on your first try, the chances of moving up to the top 7% on the next round are not very good.
As a research Administrator, my veiwpoint was that with the A2 as an option, the quality of the A0 proposals was abmismal. They were thrown together so that they could just get into the queue. The A1 was a more thorough proposal and the A2 was the crown jewel. All that the NIH did was save the time of the reviewers from having to look at and triage the A0 to see who actually put together a good proposal.
Furthermore, I would like to see more dialog of the researchers with the program officials. Just because you have a great idea, if the direction of the funding is not going your way, you won’t get scored.
If we could follow this methodology, I would welcome the eliminiation of the resubmission process and therefor allow the submissions all as “new” proposals.
Unfortunately, peer review now is tasked with scoring based on direction of funding. Historically, we were to judge on the quality of the science. We are increasingly asked to judge based on the “significance” and “overall impact” of the science. This is a clear conflict of interest on the part of the reviewers. I’d be an idiot on study section to not rate as highly significant and likely to have impact work that is similar to what I do. So, at the get-go I end up biased. Programmatic decisions should be left to program/institute staff and council, and the quality of judging the scientific design, approach, and feasibility left to peer review.
If an application is not discussed, this means either:
(a) It is substandard
(b) The review quality is substandard
Both things happen, but neither is an argument for keeping -A2. In case (a), applicants should be actively discouraged from sending poor applications. In case (b), ignorant and/or unethical reviewers should be equally zealously eliminated from review groups and blacklisted.
An additional strength of the A2 system was precisely that it reduced the chances of overcoming a bad review. This is sometimes is unavoidable because all you need for getting a grant triaged is one reviewer who does not appreciate the science, or simply misses the point. With the paylines where they are and the triage line higher than a score of 4, a single reviewer can bring a proposal with good potential to the triage zone.
This may happen because some reviewers are under-qualified, but only a small minority, in my experience. Yes, I do not understand why people without Ro1/R21 funding for years have to review my RO1/R21 proposals (in my experience, these are the ones who trend to be particularly bitter and pick on small experimental details rather than the significance of the proposal), but most reviewers that I meet in study sections are talented and dedicated scientists… who sometimes also miss the point. That’s human and that’s all is needed these days to kill a grant.
With the A2, one had another chance to overcome this not infrequent scenario. That reduced some randomness in the system, in my experience. And this made the system more predictable. The usual process was to improve the score with every submission. However, right now, scoring has become a lottery. And if a proposal does not get into the 25 percentile, the chances of funding in a resubmission are really small. And not getting there at the first try does not mean that the science has not top potential.
In any case, I play by the rules that I am given, but what I do not understand is the obstinacy of some administrators whose career do not depend on peer-review to refuse listening the clamor in the scientific community. And offering obvious data of little relevance to support that they were always right.
Citing Figure 2 as evidence that new investigators are not hurt by the policy is probably not very useful and is certainly missing the most important part of the story: the graph does not take into account the applications that never make it to the funding line. Of course the apps that got funded are all clustered together on “time to funding;” nobody’s process (submit, get score and summary statement, pop champagne or begin rewriting for 1-2 cycles later) can really change regardless of how many tries are given. What may be different is how many great ideas end up in the dustbin for the sin of only scoring 15th %ile or whatnot on the second try, and how many of these are from NIs (who, with a new lab, may not have the luxury of having a solid “B project” to turn to).
What is the percentage of applications from NIs that do eventually get funded (regardless of whether it is at A0 or A1), compared to established investigators? If there is a substantial gap (as most of us feel there must be, but I would love to be wrong about this), is it better or worse since the Death of A2? That’s the comparison many of us want to see.
This is a completely misleading “analysis” of the data, apparently manufactured to “make the policy look good” rather than to truly understand the effects of the policy.
1) Of course the number of A2s being funded is going to zero — under to new policy no A2s can be funded. It is also a meaningless statistic. A meaningful measure must include applications from investigators that have gone through A0, A1, then worked around the rules, then A0, A1, etc…
2) Any meaningful measure of the average time from submission to award MUST include the time spent from the ORIGINAL A0 application through the funded application, INCLUDING submissions that went through major rewrites and became new A0s.
How about an open, OBJECTIVE discussion and analysis by the NIH?
I received 13th percentile on 1st R01 submission R01 and 13th percentile on a 2nd submission. Clearly this proposal is near the fundable range, but now I am unable to re-submit. Shouldn’t there be a policy to permit such applications to be re-submitted 3rd time?
Sorry Jeff, this is the best system we have!!! You may try crying to the program officers in this case to get it funded this time!!! But under this system, your submitted project is history!!!! NIH has spoken!!!
The previous NIH review system was designed to allow reviewers to write critiques that “improved” the application (if the PI took their advice). Many of the reviewers on my study section were convinced that their directed, fully explained comments (rather than bullet points) would make the science in the application stronger and educate the investigators. They were also convinced that this was necessary for young investigators to learn what was needed to make their science stronger and their application better. Accordingly, many of the applications went through 3 reviews before funding because they needed the extra instruction. However, with the 2 strikes and you’re out rule, the NIH made it clear it was not their responsibility to train new investigators how to write grants via the review process. Accordingly, what the reviewers considered to be an important part of their responsibility to the NIH review and US science was eliminated. The “retraining” of the reviewer’s mind was very painful to watch because many of the applications during this time were reviewed by reviewer’s who could not buy into the idea that they were there to fund the best science. Accordingly, they scored the applications like they would if they still had 3 chances. The reviewers were stuck in “educate the applicant mode.” And, because of the recycling rule, a lot of good science was lost during this time. Now, even though the reviewers may all be retrained or transitioned off study section, the recycling rule continues to wreck havoc for new and early stage investigators. Since it seems to take 2-3 years for young investigators to obtain sufficient preliminary data and publications to be taken seriously by study sections, any missteps along the way means that there will not be enough time to fix the science or change the program in ways that will allow it to pass the recycling rule as defined by a computer program not a person or panel. The recycling rule costs the institutions promising young faculty and 3-5 years of start-up funds.
Kirk-thank you for your post, these are key issues why the “old system” worked to increase the quality of the funded science. I do not think the “queu” argument for A2’s was a major problem, maybe at some study sections but not universally so. While there are always flaws in any system (especially those run by humans), any changes that are made should be carefully analyzed and abandoned when found to be flawed. It is clear that the analysis of the current policy presented on this blog is fatally flawed for all of the reasons pointed out by the other posters here. If a RO1 was presented for review at study section with a similar analysis as these data on grant funding presented here, while the significance would be rated high, the proposal would be unscored because of the deficiency in design, interpretation and lack of statistical rigor. (innovation would also be low as this was a knee-jerk response)
The NIH continues to fail to understand that all of us are wasting hundreds of hours rewriting unfunded A1 to submit as A0. This is time that could have been spent on research, mentoring, teaching, writing manuscripts, really anything rather than the creative writing exercise we are forced to do to keep our labs running. I also now waste time going to meetings where we are taught tricks to get around the ridiculous grant submission rules NIH has imposed. The fact is it is unrealistic for scientists to come up with brand new projects for an A0. New projects take years and in the mean time the lab will run out of money and be shut down. Instead scientists are forced to keep rewriting the same project a multitude of different ways hoping that it will get a fundable score this submission. NIH needs to allow grants to be resubmitted as new grants if the A1 is not funded.
Well said
agreed!!!
Although I do not particluarly care for the reduction in resubmissions, especially in addition to reducing the feedback, in my opinion the biggest hurdle is changing reviewers. Even if an application gets scored on the first round, a different reviewer for the A1 may result in another non-fundable score, despite all of the previous concerns being addressed. My suggestion would be to allow an A2 if any of the original reviewers for the A0 were replaced and the A1 is not funded. If not, then limiting it to A1 is reasonable since the application already has failed to impress the same group of reviewers twice. We must keep in mind that the goal should not be simply to streamline the review process, but to fund the best science. Make serving on one review panel per year a condition of funding and there either will be no shortage of reviewers or a lot fewer applications to review.
I agree. Changing reviewers is an enormous problem. We recently received a decent score (but not quite decent enough) on an A0 and meticulously addressed the critiques, only to see our score deteriorate on the A1. Only one of the original reviewers read the A1, and the replacements did not respect the application’s history. This process clearly does not serve the best interests of science, and needs to be remedied.
Good Luck!!
I repeatedly volunteer myself to serve but keep being ignored. It seems serving in the study section becomes a privileged to those people who get connections.
Where are you from? ???? of course… This is what happens when money becomes limited!!! Only, well connected people will get funded! A few by chance may get funded also to show that the system works!!
Then, we should start from here. Let everyone who has NIH funding history serve in the regular study section to have a fair play. In my field, the same old faces decide our fate for too long.
We are dealing with the game of numbers. It works whichever way you slant it. With the fast-moving technological advances in today’s research, the era of resubmissions should be in the past. All applications should be considered new, with the number of unscored proposals adjusted to reduce the burden on the system and reviewers.
As it is obvious from the above comments, there are no scientists that support the “no A2 rule”. If there are and the read this blog, please take a stand. Yet, NIH claims it works based on a one year analysis that only included funded projects (top 10%). Most likely NIH will have better data 5 years from now, when the general population of RO1 funded scientists has an average age of 60, and 2 generations of young scientists have been discouraged to pursue a scientific career. Just like politics, nobody wants to acknowledge a mistake. Returning the A2 would decrease the number of A0 and A1s funded by only 20%, but would dramatically increase the amount of meritorious science (in the 10-20%) funded and more importantly will not eliminate a complete generation of young scientists by encouraging to migrate to other fields or even countries.
It is amazing that NIH tries to experiment with new policies when the funding rate is 10%. How important is it to decrease the time to fund an R0, when 20 years from now the the pool of young scientists will be smaller than ever. Have you been to a scientific meeting lately? Just look around, exclude students and post docs and then estimate the average age of “productive and outstanding” scientists.
Yes, Tom, you’ve hit the nail on the head. The NIH says all the right words about “funding the best science,” “blah, blah, blah,” but the never really look at the science. When paylines get below 10%, there is no way to judge. it becomes a lottery, except (and in my years of experience on study sections this is a BIG except), that politics begins to have a major influence. The NIH ignores this by getting reviewers to sign not one by 2 COI forms. But when a reviewer is looking a grant from a big name in a big lab, there is a very strong motivation to bump the priority better. Not always, but it is there, and no COI form will eliminate it. The only solution (really the only one) is that whenever paylines get lower than 10 or 15%, all discussed applications (say with a percentile at 30% or better) should be subject to a lottery to decide funding. This would not only fix the delusions of the NIH Review branch, but also simultaneously level the playing field for young investigators. The present policy is killing off well-motivated young scientists.
The goal of the NIH is to fund the best science. If this is so then the submission round of the proposal is irrelevant. Reviewers realize it is very difficult to distinguish between an 8 percentile grant and a 20 percentile grant. A single statement by a reviewer can kill a stellar grant, especially in today’s climate of low paylines. We all know the above analysis is flawed because there is no way to distinguish a new A0 proposal from a reworked A2, so let’s stop the charade. ALL proposals should come in as brand new grants and be judged on their merit. There would be no response to reviewer comments, the PI can chose to ignore previous comments but since the proposal will likely see the same study section and reviewers this is likely not to happen. PI’s can continue to improve a proposal until its funded or abandon it if it continues to fail. This is the only policy that makes sense to me. The NIH should not be in the business of limiting ideas that come in, just funding the best.
I have TWO suggestions:
(1) The NIH should completely eliminate or do away with the resubmission policy in a manner akin to NSF. However, PIs should be allowed to submit an application as many times as they like but each time as a new application.
(2) No PI should be allowed to have more than two R01 grants at any one time. Those who do so should demonstrate substantially higher scientific productivity (for example, as measured by the total impact score) than an average grantee with only one R01 grant.
I can’t agree more with you Al Kahn. The important thing to remember is that the limiting factor is the total funds available for science. Changing the number of review cycles allowed can be detrimental to funding the best science and could reduce opportunities for new investigators. However, the pool of available funds to an investigator will remain the same. Alternatively, limiting the number of possible RO1s to 2 per investigator will inevitably increase the available pool of funding for other investigators. It is hard to see where more than 2 ongoing RO1 grants for a given investigator are critical to good science.
Previously, A0+A1+A2=$100. Now A2=0, so A0+A1=$100. So Figure 1 is a simple as a result of A2=0. Now A0 and A1 are happy because of less competition by elimination of A2, but A2 is unhappy, especially for young investigators. Is less competition due to elimination of A2 makes good science? Are A0 and A1 better science than A2? I think it is good idea to allow to submit certain good science-A2 application, especially for young investigators.
I will have to agree with the post. If the NIH folks that put together the first graph in this story — were doing that for a grant submission the grant would go un-scored. Talk about bad science — and poor interpretation skills. Is this the best NIH can do. Are our tax dollars actually supporting NIH employees to continually try to justify a system that most agree is flawed. I guess it is better to work within NIH — and continue to make claims that don’t hold water, but are not really judged for their scientific merit. Until the pay line starts to improve, the cycles will continue to spiral downwards causing more and more academic scientist to leave the field and try to support themselves by teaching.
Charlie Brown, your idea is really good. We have to push NIH to change this travesty.
This analysis of the trends shown in Fig 1 is totally useless. If you eliminate A1, then 100% of the funded projects are from A0. Will you have an even shorter funding cycle? Yes. Will that be a further improvement? No! The whole logic in this analysis is wrong.
Grants are now ranked from “best” to “worst” based on “preliminary” scores. The chilling effect of this pre-ranking on the review process is that once a study section gets to grant #5 , it knows that none of the rest of the grants will be funded.
Now add in the absurd 9 point scale – when scoring grants #3 – #5, the reviewer knows that in many study sections, a “1” will strongly push the grant towards funding, while a “2” strongly pushes the grant towards not funded. In other words, reviewers are now essentially making yes/no decisions about whether or not a grant should be funded.
Now add in the lack of A2s – the reviewer is forced to make this yes/no decision about a strong A1. End the project or don’t end the project; close the lab don’t close the lab; end the career or don’t end the career. All these thoughts are now part of the process.
And you wonder why people don’t want to be on study section?
Well, I’m one of those (NIH-funded) researchers who thinks it is a good idea to eliminate A2 applications. The real problem is that the funding levels are low, and allowing more revisions doesn’t fix that at all. In essence, this is a “zero sum game”, so any hypothetical A2 being funded will result in somebody else’s A0 or A1 NOT being funded. And do we know with certainty that that particular A2 is any better than the A0 or A1 that it replaced? NO we don’t – as everybody agrees it’s almost impossible to clearly rank the merits of proposals once we’re below the 15-20% priority level. Ultimately, in any zero-sum game that uses formal cutoffs, there will be somebody who ends up JUST below the payline, be it an A1 or A2 or A-whatever. Which brings me back to my main point that funding levels are too low.
However, I do agree that changing reviewers between A0 and A1 is a serious issue that needs to be addressed.
I agree that allowing more revisions doesn’t fix the problem of low funding rates. As Diane Geggis points out above, in the three-submission days it was very common for applicants to put in grants that really weren’t ready for prime time, just to get the reviewer responses. Now, there is no excuse for submitting anything less than your best proposal possible on the first submission.
What concerns me more about these data as presented is that they don’t appear to give us information on what happens to the applications that never reach payline. People here have called for NSF-style submissions, where in many (but not all) divisions, all submissions are treated equally. However, NSF funding rates have gone down in recent years as much because of budget limitations as because of increased numbers of submissions overall, and increased numbers of submissions per applicant. Part of the goal when NIH announced this policy was to ‘weed out’ ideas that were never going to raise reviewer enthusiasm. If an idea is that good, then changing reviewers shouldn’t matter.
I would like to respectively disagree with your assertion that changing reviewers does not affect the outcome. How long do you think it would take to get a manuscript published if it went out to new reviewers with each re-submission? If the new reviewers only assessed how well the author addressed previous concerns, I would agree with you. However, each new reviewer brings a whole new set of assumptions, experience, and yes, biases to the process. In my opinion, either treat all submissions as new submissions, provide a mechanism for additional submissions if one or more reviewers change, or have a completely new set of reviewers for the A1.
The present NIH funding system funds only a few investigators with tremendous amount of money. The scoring system is really meant to eliminate applications, and the current A0/A1 issue is just another method of elimination. Most of the scientists are being discouraged to apply. Assuming that the current system is really meant to choose best grants, it actually expects that one fine morning one mysterious scientist would solve all the problems and discover a “magic potion” instead of stepwise logical progress made by many scientists in revealing truth, which we evidenced so far. Unless NIH fairly distributes the available resources we are going to destroy the nice system that proved to be successful earlier.
I would vote for removing the triage system (so that the reviews come with less factual errors) and the resubmission systems and follow the NSF method for reviewing applications.
Removing A2’s for those that get discussed at first submission and A1 may be reasonable. But for those that do not get discussed at first submission-and only one outlier reviewer can cause this- the A1 may represent the only interpretable or in-depth review. I would like to see how many grants get funded that were not discussed and then submitted as an A1– this may be the group that should all come back as new grants (as I suspect almost none improve to fundable— or be allowed an A2 if close to the funding line.
It is very tiring to see that we are still dangling with A1 & A2 submission issues. Providing statistics does not justify the real issues at this time of very low funding rate and denying A2 submissions. How can NIH justify not funding a grant which was a number one grant tied up with another grant, scoring a priority score of 20 (11th percentile) in a particular study section at that time? In fact, not a single grant was funded from that study section that time. What NIH will need is to recognize the problems, develop a clear visionary mission and attempt to correct them. A number of problems can be addressed and solved if the programs officers, center directors and branch chiefs within an institute work cooperatively to fund the most meritorious grants without engaging in internal territorial sentiments. SROs should also work with the institute program officers very closely to recruit study section members. It is time to act, not to engage in arguments who is right and who is wrong.
Returning the A2 would decrease the number of A0 and A1s funded by only 20%, but would dramatically increase the amount of meritorious science (in the 10-20%) funded and more importantly will not eliminate a complete generation of young scientists by encouraging to migrate to other fields or even countries.
Taking account of the laws of arithmetic–which as far as I am aware have still not been repealed–the only way that returning the A2 could “dramatically increase the amount of meritorious science funded” is if A2s are dramatically more scientifically meritorious than A1s with the same percentile rankings.
If, as someone suggests above, the process of grant review actually improved grants (and the subsequent science done), A2’s could be more meritorious than equally ranked A1’s (i.e. after they’d undergone another round of reviewer-inspired modifications).
I wouldn’t put my bets on the possibility, but it’s not a mathematical impossibility. My suspicion is that its been a while since the review process was actually advising new grantees on how to improve their science (though I’m open to the idea that such a time actually existed, probably back when funding lines were high). So, I’d guess that the change to A2 “queuing” as opposed to A2 improvement had occurred long before the elimination of A2s.
1. Eliminate A1, and use the saved resources to eliminate the triage.
2. Discuss every application, mark particularly substandard ones as “future submissions not allowed”.
3. Any application not funded and not considered substandard can be submitted for subsequent cycles, and allowed to compete as a new one.
4. Pay PARTICULAR ATTENTION to the quality of the peer review process. If review groups produce nonsensical or irrelevant summary statements, then PIs will keep resubmitting and contributing to the backlog.
Terrible decision. Bogus analysis of the data as pointed out by many of the previous posts. I also agree that the 2 strikes rule negatively affects young investigators but it also can be devastating for experienced investigators. For example, if an A1 just misses funding, the reviewers considered it to be spectacular (equal to those on the other side of the payline) but no funding and you can’t resubmit and NIH can’t even explain how much needs to be changed to qualify as a new grant. A simple fix would be to allow A2s if the AI scores within a certain percentile of the payline or better than a certain raw or percentiled score.
The NIH has lots of money to fund health and science but the way it operates needs to be revolutionized. The NIH should force the extramural institutes to bear the financial burden of providing research infrastructure and, in so doing, eliminate indirect costs.
Why should the extramural institutes provide research infrastructure when they can easily put those dollars into the pockets of unnecessary bureaucrats, deans and administrators whose only job is to impede the progress of research and free thought by enacting bazillion rules and regulations that they do not even understand themselves.
A typical medical school has between 50-100 associate deans on a close to seven-figure salary. Does anyone ever wonder what do these deans do other than robbing the research community of precious dollars?
What makes you think you would even have a job if indirect costs were eliminated?
Indirects serve an importance purpose in providing for the support structure that undergirds the research. However, it might be prudent to consider “pro rating” indirects since it unnecessary to fund an entire support apparatus for each grant. One solution would be to provide 100% indirects for the first grant, 50% for the second, 25% for the third, etc. The savings could then be re-driected toward funding more grants.
Let me also add that the NIH could save close to another billion dollars by getting rid of the CSR and study sections — which have time and again shown to be ineffective at funding the best and meritorious science. Their only role is to put more money into the pockets of those scientists who need it least, while offering nonsensical and incompetent critique of those who need it most and yet will not be funded. The NIH could and should resort to electronic reviewing system (similar to peer-review conducted for journal articles) with minimal administrative workforce.
Unfortunately, this is one of the downsides of a capitalist system — where a small minority of winners take all at the expense of depriving the majority of even simple bread and butter. Do not get me wrong — I am not at all talking about funding the lazy and unproductive scientists — and there are lots of them around too. I am simply saying that the current system is failing a lot of talented scientists and brilliant young minds of the future and, perhaps more importantly, the hard-working middle class of our scientific community. NIH should not subscribe to the notion that science will do just fine by getting ride of the middle class. This will be a grave mistake from which the scientific greatness of our nation may never recover.
Surely, the status quo cannot go on forever. The NIH should immediately take the necessary steps to rectify the failure of the current system rather than bring the scientific community on the verge of a total collapse — which will not only affect our nation and the society as a whole but would also give our competitors such as China to leave us behind and stranded.
Just some free thoughts!
Also, remember about the present exorbitant indirect cost rates. If organizations are allowed to “tax” grants at a 70% (or more) rate, this is one of the MAJOR reasons for the shrinking paylines.
Radically, I would eliminate F&A completely. Research money is supposed to fund strictly defined research. A compromise idea: cap them at 25%.
God love scientists. Here, we have individuals of elite intelligence arguing about how to improve the movement of cheese within a box, rather than expanding the box. I’d like to read some posts about NIH efforts to increase research funding, rather than tortuous mechanistic fixes that do not change the material fact that we are all living in the noise. When so many proposals are competitive and the paylines are this low, I think it is confirmation bias to think that these fixes are doing very much other than reducing reviewer effort. That may be a worthy goal in itself, but let’s not fool ourselves into thinking because the mice keep dutifully moving the cheese that the system isn’t flawed.
Great comments responding to this predictable study. We’ve got major problems, and NIH appears oblivious to the solutions. Here are possible solutions that should be implemented immediately:
1) Allow PIs to send in a 1 to 2 page pre-proposal with their concept. Demand a quick electronic review within 1 month or less. The pre-proposal can either be invited for a full proposal or denied (triaged?). This will effectively filter out ~75% of would-be proposals, without wasting the time of reviewers or the PI in preparing long proposals and waiting 6 months for the review (and rejection). The remaining 25% should undergo a rigorous electronic review (within 8-12 weeks), similar to publication in a journal. Reviewers can decide whether an A1 is allowable.
2) All investigators who have (or had) national funding are encouraged (required?) to act as reviewers.
3) There were several excellent ideas mentioned above that would remove red tape and free up $$$ for funding research. Another option is to demand (pray?) that the Federal Reserve print another 10 billion dollars ear marked for biomedical research. This will effectively expand the box, dramatically increase the payline, create jobs and heal the sick without any obvious side effects on the economy 😉
I like this. But I would make it blind, until it is near funding. That will prevent bias/nepotism.
Great posts one and all. My take goes back to the Bush Administration and how, long after it is gone, it continues to bite us in the rear. The basic flaws: mediocre leadership (as in “heckuva job Brownie”), a complete lack of vision (as in “The Iraq war will pay for itself”). So, the NIH budget doubled, but lack of vision ensured that this would result in a classic “bubble”: i.e. too many people in the market, but not enough fluidity to sustain it. None of those bozos thought to plan for how to manage it when the expansion ended. The bubble began to burst in 2007, the the ARRA funding sustained it for 2 more years. In 2009, the bust hit for real, and now the biomedical science enterprise is in free fall. As for mediocre leadership, the original sin lay in NIH directors kowtowing to a know nothing congress and administration that demanded “cures now”. Instead of arguing back about the benefits of “basic research”, they changed the mission of the NIH, employing PR firms to come up with wonderful terms such as “Bench to Bedside” (which fell by the wayside a victim of too many “sleeping with the postdocs” jokes), and later “Translational Medicine”. This spurred reorganization of study sections, resulting in contraction of basic science study sections and expansion of the applied science sections. The the lions share of “Excellent Science” has always been done by the basic scientists because they devote 100% effort to research. In contrast, Clinical Scientists have less time, and therefore devote less effort (and thought) to research. So, the few remaining basic science study sections are overwhelmed and funding levels are so low that the concept of “merit” has been lost, and has been replaced by “politics”, and bad policy. How politics works on study section: look out for you friends and punish your competitors. As a result, the rich get richer, and the poor get poorer (I predict that only HHMI researchers will emerge from this funding/evolutionary bottleneck). Bad policy: the unamerican 2 strikes and you’re out policy; unhelpful bullet point reviews; inconsistency from section to section. Another bad policy was doubling the budget for the intramural program. I have watched the NIH grow from a bucolic campus to a fortress in the past decade. It reminds me of a family trip we took to DC many years ago, when after voicing my admiration and awe at the big government buildings, my Dad said “yes, but remember that there’s a bureaucrat behind every window”. In that vien, the doubling of the NIH budget enabled us to more than double the number of doctoral students, lowering our admission standards and enabling us to churn out mediocre PhD’s. In the end, they couldn’t get jobs in academia or industry, so they ended up working for CSR, where they are now happily touting the new chair arrangements on the decks on the titanic while pulling down six-figure salaries. And let us not forget our dear new crop of Program Officers, who find it easier to manage a single large Programmatic grant than a bunch of R01’s. Lazy SOB’s. It is no wonder that Dr. Rockey, she of the “Angry Chicken” haircut, has become the face of all that is messed up at the NIH. The NIH used to be known as “the Jewel in the Crown”. After the Bush administration got done with it, it is just as incompetent and corrupt as any other federal agency. Such a pity.
Oops! Unfortunately, a technical glitch kept us from receiving any comments submitted Friday afternoon through this morning. If you submitted a comment that has not been published, please know that we weren’t ignoring you! Just submit it again for it to be published.
JD — I could not agree with you more and I am sure tens of thousands of other scientists also share our sentiment. But, as you know, democracy only exists in abstract. In really, the few rich and spoiled run the world and dictate what the rest can do and how they should accept that life could be far worse. Why should it be any different for NIH? After all, it is part of the government.
There is no doubt that the NIH system is utterly broke and a radical overhaul is long overdue–including the change of NIH leadership, which is gungho on driving the mainstream scientific community to apocalypse. The only way such a disaster can be avoided is to eliminate two notorious aspects of this system:
(1) Eliminate indirect costs altogether (0%)–let the extramural institutes figure out a way how to provide for research infrastructure; those institutes which cannot do so should not be in the business of doing scientific research. Period. There is already too much financial corruption within such institutes and the indirect costs from the NIH only help to further this cause.
(2) Distribute the NIH dollars to a more wider scientific community than put all money into the pockets of a small 1% minority of self-proclaimed protectors of science. History has shown time and again that such self-proclaimed messiahs always lead their communities to destruction. Here is a lesson to heed for those who can see and reason but the NIH leadership chooses not to.
Does it really surprise anybody that the number of funded A0 applications goes up after eliminating A2 opportunities? Same amount of money, now only two bins in which to dump it. Why not limit everybody to A0? Then ONLY A0 applications will be funded, and the policy will have succeeded to perfection!
Does anyone think that the review process is skewed toward the “ELITE”, and the whole process of TRIAGE, TWO STRIKES AND YOU ARE OUT, etc really favor the ELITE? Perhaps, the system has been built to do just that. As a result though we are losing to Europe and perhaps and countries in publication in high impact journals slowly (according to Nature).
A different sort of analysis to understand the impact of this policy. While I agree that this policy has decreased the time necessary for some outstanding science to get funded, there is another hugely important issue that has not been addressed, namely what happens to the projects and the investigators who have A1 applications that score reasonably well but are not funded. I would strongly urge NIH to design and perform an analysis addressing this question including:
(1) What fraction of these applicants/applications come back at A0s?
(2) What fraction of these applications are rejected by CSR as “A2s”?
(3) What are the characteristics of this pool of investigators (institutions, academic rank, gender, ethnicity, level of other NIH support) compared with an appropriate control group?
I realize that this may be a challenging analysis but this is absolutely crucial to understand the impact of the policy and its implications for scientific community.
So, the message is get it right the first time, little remediation allowed. The goal is to decrease the time to award by avoiding A2s “displacing” A0s and A1s. A couple of points:
1. What about the A1s displacing the A0s?
2. Dr. Rockey’s interpretation implies that the displaced A0s and A1s were more meritorious than the A2s; how can she be certain of this?
The only way to avoid these confounds is to have A0s judged against A0s, i.e., eliminate A1s.
Should A2 be allowed for new investigators as they are starting out and may need one more chance before totally changing their research focus or losing their tenure
There is one issue that is not captured by this metric – that is, how many people are dropping out of research after the A1 phase if their grant isn’t funded? There seems to be two responses to the current funding: write a new application every cycle, or give up trying. I’m afraid that there are many, particularly those whose entire salary is derived from soft money, that have taken the latter route; who wants to work for 3-4 years for free while developing a new application with no greater chance of being funded, and likely less chance if it is on a new topic on which the PI has less experience? I fear we are losing a generation of scientists. I believe we need to turn to alternatives for this funding crisis. For example, I believe it is unconscionable that some academic institutions are still taking 75% in indirect costs while requiring their faculty to cover 100% of their salary on grants. This is causing the NIH to fund the equivalent of 3 grants for 1 grant worth of data. And I cannot accept that all of that money goes exclusively to “overhead” -why is an English Lit professor given a secretary but I would have to pay for one on grants? I think it’s time to rethink this system, cap all indirects at 25% and require universities to cover 2/3 of salary; then there will be enough research funds to go around.
Better yet, disallow faculty salary on grants entirely. Only allow salary for research staff.
There are many flaws in the system, in my case my department is requiring that I cover almost 100% of my salary from NIH grants and they get huge overheads from those same grants. I will be out of business by June next year if I do not bring another grant because they “cannot” cover 2 months of my salary…..( I am suppose to believe that…)
At the same time, I do not receive any funding to produce new data from my department….so it is a never ending cycle of discouraging people. I even offered to be a 9 month employee and continue working 12 months to obtain new funding….they refused it!!
Other problem, not related with my own case, just a general issue is the fact that some program officers are too friendly with some department heads and they do whatever is necessary to keep those people funded….this is not rumor…I have seen some very “interesting” behaviors….
I would think the NIH would want to fund the best science. This would suggest that there should not be a limit of one resubmission. I am not at all sure how A2s can result in a funding delay of the most meritorious science. If you now find that more A0s are funded faster, it is because the A2s did improve and knocked out a lot of A0s. Since the A2s are no longer there, less meritorious science is now being funded faster! OK, if that is how you want to fund science.
Furthermore, one cannot know whether the A0s that are now funded would have been funded under the more competitive system of allowing A2s. I noticed how carefully worded the rationale was of the policy for no more A2s: it was to fund meritorious proposals sooner. It is interesting that the policy statement never says the most meritorious. It does say equally meritorious in one part of the announcement; however that statement is specious in nature. It is only equally meritorious by the definition of being below the payline, now that the more meritorious A2 proposals were removed. If the A2s were included, then many A2s would have been funded instead of the A0s. You have no way of knowing whether the A0s that got funded would have been funded as either an A1 or an A2, as scores for resubmissions can go up, down, or stay the same. The thinking of eliminating the A2s and claiming that equally meritorious proposals are being funded is not meritorious thinking. The NIH is not funding the most meritorious science. It is fine to make the policy decision that less meritorious science will be funded for the sake of expediency; but then you should say that and not make specious statements about equally meritorious funding.
Finally, I see no justification for generalizing R01 data to other mechanisms, SBIR/STTR proposals in particular.
The analogy of discarding science that has almost reached the destination is an apt one, as is the point that we don’t know whether the disallowed A2s would have been more meritorious than the A0s and A1s that get to take their place. I have served on multiple study sections (two as a standing reviewer), and I can say that I find the process more and more disheartening as I see great ideas that were “almost there” get discarded. I would support an A2 for a *limited* number of promising applications and/or new investigators.
I have no problem with NIH’s ‘two strikes and you’re out’ rule, provided the reviewers for the A0 and A1 versions of the proposal remain the same. Right now, it is possible to receive an almost fundable score for the A0 version and an unfundable score for the A1 version of the application, primarily because the new reviewers have their own concerns that are entirely different from the A0 reviewers’. In many cases, these new concerns can be readily addressed, but with the current rules, the applicant has no opportunity to do so. So, if the NIH changes reviewers for the A1 version of the proposal, it seems only fair to let the applicant respond to the new concerns.