An investigator’s long-term success depends not only on securing funding, but on maintaining a stable funding stream over time. One way to assure continued funding is to submit a competing renewal application. However, as we noted earlier this year, while new investigators were almost as successful as experienced investigators in obtaining new (type 1) R01s, the difference between new investigator and experienced investigator success rates widens when looking at competing renewals (type 2s), and success rates of new investigators’ first renewals were lower than those of experienced investigators. In addition, we know that since the end of NIH’s budget doubling in 2003, success rates for competing renewals of research project grants overall have decreased.
To further understand trends in success rate for R01 competing renewals (“type 2s”) I’d like to share some additional analyses where we look at characteristics of type 2 R01 applications, and the association of their scores for individual review criteria (“criterion scores”) with overall impact score and funding outcomes.
You might recall our previous blog where we described the association of criterion scores with overall impact score among a sample of over 123,000 competing R01 applications NIH received over 4 years. My colleagues published their findings , which demonstrated that the strongest correlates, by far, of overall impact score were approach and, to a lesser extent, significance criterion scores. Here we follow up on that analysis, this time focusing on outcomes for only the subset of R01 applications which were competing renewals.Between fiscal years 2010-2013, there were 22,714 competing (“type 2”) R01 applications. Among these, 16,659 applications (73%) were discussed by peer review study sections, and 6,847 (30%) were funded. Table 1 shows selected characteristics according to whether the applications were funded (PI characteristics refer to the contact PI). Not surprisingly, successful applications had better approach and significance scores and were more likely to come from organizations with higher levels of NIH funding.
Figure 1 shows box plot distributions for criterion scores and overall impact scores. Consistent with what we saw for all applications, the approach criterion score most closely approximates overall impact scores.
Figure 2 shows a heat map of Spearman correlation coefficients for the different criterion scores. Once again we see that approach scores were highly correlated with overall impact scores (r=0.83), while other criterion scores had weaker correlations.
Now let’s dive a little deeper into understanding potential predictive variables correlated to the success of a type 2 R01 application. We next performed a series of random-forest multivariable regressions to describe what factors are the most important correlates to four selected outcomes in the path from application to award: discussion in a study section; approach score; overall impact score; and finally, of course, funding. We have used this approach to assess factors associated with NIH funding and research outcomes.
We first looked at correlates of discussion in a peer review study section. Figure 3 shows variable importance in a 100-tree forest; the model explained 50% of the variance. The strongest correlates by far were approach score and significance score. Personal and organizational characteristics were not materially correlated with discussion.
Figure 4 shows the random-forest correlates of approach score; the model explained 63% of the variance. The strongest correlates were significance score, innovation score, and investigator score. When we removed the other criterion scores, the model (which was left with personal and organizational characteristics) only explained ~ 3% of the variance.
Figure 5 shows the random-forest correlates of overall impact score among discussed applications; the model explained 73% of the variance. The strongest correlate, by far, was approach score.
Figure 6 shows random-forest correlates of funding among discussed applications; the model explained 61% of the variance. As might be expected, overall impact score was the strongest correlate.
Figure 7 shows the results of the random-forest multivariable regression, which explained 41% of the variance. When we left out overall impact score, we found that approach score and to a lesser extent significance scores were the strongest correlates of funding.
Putting these findings together, we find that among type 2 competing renewal R01 applications, the strongest correlates of success at peer review and of eventual funding are peer review impressions of approach and significance. Personal characteristics (like age, gender, race, prior training) were not materially correlated with success. As we noted before, we think it is helpful for R01 applicants to know that when trying to renew their ongoing projects, the description of experimental approach is the most important predictor of success.
I’d like to thank the authors of the criterion score paper for their help with their analysis, along with my colleagues in the Statistical Analysis and Reporting Branch of the Office of Extramural Research.