We have heard from many of you about the policy to sunset A2 applications. As you may remember, this policy was developed as part of NIH efforts to enhance peer review. There were concerns that applications were piling up in a “queue” and subject to a holding pattern that delayed funding until the resubmission (A1 and A2) stages, and as a consequence highly meritorious science proposed in original (A0) applications was made to wait additional months for funding. The policy was published in the Federal Register on October 8, 2008, and implementation began on January 25, 2009. In March 2011 I posted some early data showing that the number of applications funded as A0s was increasing and there was no queue at the A1 level.
More time has elapsed since the policy was implemented and this has allowed us to gather a substantial amount of data and conduct more in-depth analyses of the effects of allowing only one resubmission for each application.
First, as can be seen in figure 1, which looks at new (type 1) competing R01 and equivalent awards, the early analysis was confirmed – a higher proportion of A0 applications are being awarded compared to A1s and the “queuing” of original applications and revisions has been eliminated. In addition, the average time to award from submission of A0 applications has been reduced from 93 weeks to 56 weeks (figure 2). Although it is unsurprising that the time to award has decreased since the elimination of the A2, the increase in the proportion of funded A0 applications versus A1 applications was not an obvious outcome.
Based on these analyses, the policy is achieving its goal of funding a higher proportion of original applications more rapidly. But we realize that these data alone do not address some major concerns, such as whether the new policy disadvantages new investigator applicants who may find it more difficult to craft new applications. One indication that this is not the case is the fact that the time to award for new investigators is not appreciably longer than the entire population (figure 2).
These results notwithstanding, we are well aware that a lot of great science goes unfunded because paylines have decreased. Some have suggested that to deal with this issue we modify the policy to allow A2 submissions for a specific subset of applications (e.g. those with scores just outside the payline). We have modeled the potential impact of changing the policy in such a way.
We looked at this issue using data from fiscal year (FY) 2011. If A1 applications with percentile scores below 25% were allowed to submit an A2 application in FY2011, it would have resulted in 764 unsolicited A2 R01 applications. 165 of the new applications were from new investigators. In addition, only a small minority of eligible applications (37 of 218 renewals) were from investigators trying to renew a previous new investigator award.
Assuming the most extreme case – that all 764 of these A2 applications would have been funded, NIH would have been able to fund 21% fewer A0 applications and 19% fewer A1 applications in FY2011 (figure 3). These displaced A0 and A1 applications would be highly likely to come back as A1s and A2s (as most displaced A1s would become eligible under the modified policy) and the average time to award would increase.
Overall, these data indicate that the policy to sunset A2 applications continues to achieve the stated goals of enabling NIH to fund as much meritorious science as possible in as short a time period as possible. Any revision to the policy to allow additional resubmissions of all or a subset of A2 applications will displace equally meritorious A0 and A1 applications, and increase the time to award for many applications. For these reasons, we have decided to continue the policy in its current form.
We greatly appreciate the community’s input on this issue and the thoughtful deliberations we had here with NIH leadership, all culminating in the decision to keep the policy. Please know that we will continue to monitor the policy and its impact on time to award.
Update from the Rock Talk Blog Team (November 28, 2012 at 2:10 pm): Data tables corresponding to the figures shown in this blog post are available at: http://report.nih.gov/special_reports_and_current_issues/index.aspx