7 Comments
NIH’s career development K awards intend to help early career scientists become independent. These awards afford the recipient protected time for research, publishing, and generating new ideas. As part of ongoing efforts to take a data driven approach to managing NIH programs, my colleagues within the NIH Division of Biomedical Research Workforce (DBRW) in the Office of Extramural Research sought to determine whether K awards might be achieving this goal, and published their findings in Academic Medicine last December.
Dr. Silda Nikaj and Dr. P. Kay Lund focused their attention on a particular subset of K awards called individual mentored career development awards. Their study used a “regression discontinuity design” to assess the possible causal effects of mentored career development K awards.
Drs. Nikaj and Lund assessed how likely early-career scientists who received a K01, K08, or K23 award (referred to as “K award participants”) were to subsequently receive an R01-equivalent or other RPG award. Their funding success was compared to those who did not receive an independent mentored career development award, but who had scores close to the ‘funding cutoff’ (referred to as “unfunded K applicants”). To further assess the K award’s effect, individuals in these two groups were matched on variables including sex, race, ethnicity, type of grant activity code, degree type, and the NIH Institute or Center that funded the award.
Figure 1 shows the likelihood of receiving a K award as a function of a “normalized priority score,” that is a peer review score recalibrated for the payline of the Institute or Center (the vertical line at zero on the graph). Since NIH Institutes or Centers primarily base award decisions on peer review scores, we see a clear “step function” whereby those applicants with scores just above i.e. higher than the payline are exceedingly unlikely to be funded, whereas those below the payline are highly likely to be funded. No surprises here.
For simplicity in this post, we refer to these K award participants and unfunded applicants as those who later receive an R01-equivalent or RPG award, but please note that NIH officially recognizes the institution as the recipient of a research grant.
Table 1 shows characteristics of K Award Participants and Unfunded K Applicants. Data demonstrate that the matching procedures were effective, with equivalent proportions of participants and unfunded applicants by demographics, baseline degree and prior NRSA support. A majority of the Award Participants and Unfunded Applicants were physician-scientists.
| Table 1 | ||
| Variable | “K Award Participants” (below cutoff) (N=3309) | “Unfunded K Applicants” (above cutoff) (N=2069) |
| Women | 38% | 38% |
| Racial minority | 1.4% | 1.4% |
| Hispanic | 1.3% | 1.3% |
| Post-doc NRSA | 21% | 21% |
| Pre-doc NRSA | 6.7% | 6.7% |
| MD | 56% | 56% |
| MD-PhD | 11% | 11% |
| PhD | 33% | 33% |
Table 2 shows the likelihood of receiving first and second R01 Equivalent Grants (R01s) or first and second Research Project Grants (RPGs). K Award Participants were much more likely to receive first R01 and first RPG awards.
| Table 2 | ||
| Variable | “K Award Participants” (below cutoff) (N=3309) | “Unfunded K Applicants” (above cutoff) (N=2069) |
| First R01 (%) | 37% | 23% |
| First RPG (%) | 47% | 32% |
| Second R01 (%) | 17% | 10% |
| Second RPG (%) | 27% | 16% |
Figure 2 shows the probability of receiving a first RPG according to the normalized priority score on the index K application. Each dot refers to an individual included in the study. Note that there is a “step function” – the probability of receiving a first RPG jumps when going from just above to just below the payline. This kind of “discontinuity” is consistent with a causal association between receipt of a K award and an increased probability of receiving a first RPG.
After accounting for confounders in multivariable regression discontinuity models, K awards predicted a 24% increased likelihood of subsequent first R01 or RPG. K awards were not independently predictive of second awards. The effects were strongest for K23 awards both during and after the NIH doubling. For K01 awards the positive effects were strongest post-doubling and for K08 awards the effects were strongest pre-doubling, However, it is important to note that the post-doubling effects may seem diminished for K08 awards because of less available time for follow-up.
For those researchers who did receive a first R01 or RPG, the time from K application to R01/RPG receipt was about 1 year longer among K award recipients. This may not be all that surprising – it takes time to complete the K award work. However, the time delay largely disappeared for those who went on to receipt of a second R01/RPG.
Previous work has shown some beneficial effects for K award recipients (see here, here, here, and here) though with important differences among women and men and for K99 awards compared to others. The current analysis is based on a sophisticated regression discontinuity design and compares outcomes across time and different K award types: K01 (primarily awarded to Ph.D. researchers), as well as K08 and K23 (primarily awarded to physician scientists).
In summary, these data suggest that researchers who receive an individual mentored career development award, when compared to those who did not, are more successful in receiving a first major NIH award.
There are, however, other metrics for success beyond receipt of subsequent NIH funding. As discussed in the paper, future studies could look at achieving first faculty positions, publication output, ability to receive funding from other federal agencies or non-federal sponsors, and achieving tenure. Careful consideration of findings, such as those from this paper, can help us better achieve the NIH goal of enhancing and sustaining the careers of early-career biomedical researchers.
Something I’ve always wondered, as the NIH seems to put more emphasis on training awards – Is it fair to conclude the higher R01 receipt rate is due to “causal effects of mentored career development K awards”? Or does the pool of K grant recipients represent those who are better scientists/grantwriters who would have gotten the R01s at a higher rate even without the K award? I guess what I’m wondering is how can we distinguish the “effect” of the K award versus the K award just marking the promising population of young scientists who are good at communicating science through writing grants?
One problem with this study, since it is retrospective, is that the study section reviewers were are not blinded to the previous K award. One review criteria for R awards in the investigator. It is plausible that reviewers who see a previous K award in the Biosketch are, consciously or subconsciously, going to view the investigator as more qualified and score this criterion more favorably.
How do you account for the number of K awardees who then move institutions after getting their K award, when getting a K is intrinsically scored on the first institution’s ability to mentor and help the person be successful?
These results are not that surprising. When most advertisements for faculty jobs describe the history or potential to obtain outside funding as an important prerequisite for the position, not many institutions will take a chance on someone who doesn’t have an K99/R00. Getting a K award also unlocks start-up funding that helps someone establish a lab that provides resources to be able to get that first or second R01. Again, no surprise here.
The question NIH should be asking is if peer review is the best system to pick the winners and losers in the faculty lottery. Should we be telling postdocs who don’t receive a K award to give up on their hopes of becoming a faculty and instead focus on industry jobs or alternative careers?
Faculty job postings are trendy–they often follow the “hot new fields” and ignore important steady advancements in other fields. For example, gene editing, cancer immunology, CryoEM are all hot fields right now. Everyone is complicit–from journal editors to faculty search committees to NIH review committees. The problem is that sometimes research takes years of long, careful plodding to develop. The NIH has an opportunity to resist this collective obsession with the ephemeral, splashy science trends of the day. Instead, they are contributing to the system by picking who gets faculty positions.
I’m curious about the data in Table 1. What are the odds that eight variables will be exactly the same for the K awardees and unfunded applicants? Is this an error?
It’s probably worth adding that there are secondary benefits of including MD/PhDs and MDs in the research environment via K08/K23/K12 programs- it can boost the relevance of other investigators’ research, particularly research that is meant to be translational. For example, when we have local scientific conferences and group lab meetings, the MDs and MD/PhDs tend to get asked a lot of questions by the bench scientists. Or the clinicians might even be providing patient tissue samples for research. This can help drive those investigators’ translational projects toward human disease/unmet medical needs, and away from research that might not be as relevant/impactful as the bench researcher thought. This benefit to the system (& towards NIH mission) would not show up in the funding metrics for the clinician-scientist.
If this is really the case, all institutes eg NHLBI should re-institute K awards for all including PhD scientists not only physician scientists