What Can We Learn from the Early Outcomes from the NIH Director’s New Innovator Awards?

Posted

In earlier posts, like this one, we discussed the importance of moving towards “evidence-based funding.”. NIH seeks to apply data-driven strategies to conceptualize, develop, implement, and evaluate policies, such as those that will affect the NIH-supported biomedical research workforce. Today, we’d like to spotlight a recently published analysis of an award program directed to investigators early in their careers – a population that has received much attention at NIH and beyond in recent years.

For a decade, the NIH Director’s New Innovator Award has sought to support exceptionally creative and innovative early career investigators across the country. To receive an award, applicants must be an early-stage investigator and must not have received a substantial NIH award. No preliminary data are required in the application. As only one component of a wider high-risk, high-reward portfolio, projects supported through this program are meant to be unusually bold and innovative, with the potential for broad impact across biomedicine. But are they?

Since the first three cohorts of awardees recently completed their awards, our colleagues in the NIH Director’s Common Fund, who administer the program, commissioned the Science and Technology Policy Institute to conduct an independent evaluation of early trends related to effectiveness of the program. Namely, did this approach foster higher-risk, higher-reward research compared to the traditional R01 grant? And of interest to our group, did award recipients experience challenges moving forward in their career because they pursued riskier research studies early on, compared to their other early-stage investigator peers?

Multiple characteristics were considered when assessing research Innovativeness. Senior experts rated a study’s application and/or formation of novel, cutting-edge, combinatorial ideas, approaches, techniques, and methodologies; discovery of new phenomena; synthesis of disparate ideas; departure from prevailing wisdom; ability to advance a theoretical concept; and rigorousness.

to cut to the chase…

Our colleagues found that, for the first three years of awardees, the New Innovator program does indeed support research that is more innovative, risky, and impactful than that typically supported by R01s—the standard bearer NIH grant. Further, with most measures used to assess an investigator’s performance (e.g. professional advancement, obtaining new funding, and publications), this award did not significantly impact, either positively or negatively, the careers of the awardees as compared to other early-stage investigator R01 awardees.

Figure 1 highlights how senior experts in the field assessed the innovativeness of the research from New Innovator (black) and their early-stage R01 recipient peers (gray)—using a numeric scale of “Strongly Disagree” to “Strongly Agree.” Overall, the New Innovator awardees rated higher, compared to their peers, on topics such as revolutionizing their field, accessing multiple disciplines, as well as using novel tools, experiments, approaches, and theoretical ideas. Early-stage investigators were rated more favorably on research rigor, likely due in part to the incremental nature of R01 supported research compared to the riskier science supported by this Common Fund program.

Figure 1 shows how senior experts assessed research innovativeness. The X axis represents the various survey items (including from left to right: Introduced Radically Different Tools, Pursued Approach Contrary to Norm, Combined Principles, Models or Experiments in Novel Ways, Was Innovative, Applied Cutting-Edge Approaches, Will Have Significant Impact on Field, Introduced Novel Theoretical Ideas, Will Revolutionize Field, Cut Across Multiple Field, Cut Across Multiple Disciplines, Was Rigorous) and the Y axis displays the level of agreement with the statement. New Innovators and early-stage R01 recipients are represented in black and gray, respectively. Except for Rigorousness, the experts rate the research from New Innovators as higher than early-stage R01 awardees.

According to their data, New Innovator awardees tended to apply for Type 1 (meaning de novo) grants and published in journals, with their articles having higher Relative Citation Ratios, compared to their related awardee contemporaries. Let’s take a moment to look at the Relative Citation Ratio (RCR) data— a metric that uses citation rates to measure the influence of a publication at the article level. Their analysis revealed that, with RCR data from over 3,000 publications published from 1995 to 2014, New Innovator awardees published articles with larger RCRs than other early-stage investigators who received an R01 (Figure 2). This suggests that’ New Innovator awardees publish articles that are likely highly influential in their fields.

Figure 2 shows the relative citation ratio for publications from New Innovator awardees and their matched early-stage peers. The X axis is the awardee group and the Y axis shows the median RCR value.  New Innovator awardees published articles with larger RCRs than other early-stage investigators who received an R01.

New Innovator awardees were also equally successful in obtaining faculty positions and tenure as early-stage investigators supported by their first R01. Interestingly, since they tended to perceive their research as non-traditional and inconsistent with the NIH grant process, New Innovator Awardees felt they could be more successful obtaining funding from non- NIH sources.

…so, what does this mean for new researchers and the NIH-supported workforce?

Though limitations to this evaluation exist, such as only having a small cohort of awardees to study and that perceptions of the program likely changed over time, the data hints at that New Innovator awardees are not at a disadvantage just because they sought to pursue riskier research endeavors early in their careers. That said, we still need additional information and time to assess the true impact of this award on the long-term stability of their career paths and the wider workforce in general.

This New Innovator program provides one way that NIH is assisting early stage researchers towards a pathway of sustained and successful research careers. Though geared towards higher-risk studies, it was encouraging to see that these early stage scientists are productive and continue to pursue successful research careers.

We remain dedicated to helping this population of early-stage investigators and appreciate your feedback, both qualitative and quantitative, to help guide our workforce policymaking decisions. It is encouraging to see what novel insights into public health these, and our entire grantee pool, will uncover as they progress through their professional careers.

7 Comments

  1. This is an interesting analysis indicating that the New Innovator Awards supported more risky ground breaking research than traditional R01s and did not negatively affect the career progression of the recipients. My thought is that the group receiving these awards probably contains the more talented ESI’s. I think it would have been interesting to compare this group and all ESIs with ESIs receiving, say the top 25%, of scores on their first R01s to see if you could tease out contributions from talent vs funding mechanisms.

  2. Considering that institutes are asked to really reach out to ESI and that often relatively lower scored applications are funded for ESI it would be interesting comparing top scored and bottom scored funded ESI RO1 to New Innovator. Perhaps even better dividing the ESI in quartiles and see if they have a different RCR.

  3. My interest is the evaluation of cancer education programs such as those supported by the R-25 mechanism. Program evaluation is a challenge for grantees and reviewer. Are the evaluation protocols, methodology and instruments used in this assessment available? Has the full report been released? Thanks. Michael Brooks (UAB).

    1. The full report is hyperlinked above in the blog. We recommend contacting the NIH Office of Strategic Coordination, who manages the Common Fund, to obtain more specifics related to the data instruments used. askOSC@od.nih.gov

  4. Interesting analysis of a good program. Essentially, the program expands the pool of new investigators with about the same effectiveness as traditional R01 funding.

    However, the stated conclusions are not fully supported by the data. A number of metrics have overlapping error bars, indicating that the two groups are almost certainly not significantly different (e.g., revolutionizing field, accessing multiple disciplines, being rigorous, and RCR in Figure 2). What do the error bars represent: SD, SEM, 95%CI?

    Ironic and frankly disappointing when the preceding story encourages scientists to select journals with credible practices. Scientist, heal thyself.

    Nonetheless, thanks for all the efforts to support junior faculty despite limited resources provided in the NIH budget.

  5. Dear Senior Scientist:
    The scientist is not in need of healing. Dr. Lauer’s “Open Mike” is a blog, not a journal article. He quotes from the article (IDA Paper P-8478) linked in the text (https://commonfund.nih.gov/sites/default/files/HRHR%20New%20Innovator%20Award%20Outcomes%20Evaluation%202007-2009_508%20compliant.pdf). For the larger table shown in the blog, the statistics are given in a table in the IDA paper (Table 12. SSR Assessment of Innovativeness of Research) on page 54, and the graph in the Open Mike blog is Figure 14. SSR Assessment of Innovativeness of Research, page 55 of the IDA paper.

  6. Interesting analysis of a good program. Essentially, the program expands the pool of new investigators with about the same effectiveness as traditional R01 funding.
    Ironic and frankly disappointing when the preceding story encourages scientists to select journals with credible practices. Scientist, heal thyself.

Before submitting your comment, please review our blog comment policies.

Leave a Reply to Salvatore Sechi Cancel reply

Your email address will not be published. Required fields are marked *