Patents and the Relative Citation Ratio: Correlations to Assess NIH Impact

We previously referenced Ioannidis’ and Khoury’s “PQRST” mnemonic for describing research impact: “P” is productivity, “Q” is quality, “R” is reproducibility, “S” is sharing, and “T” is translation.  We wrote several blogs about “P,” productivity, focusing on publications, citations, and more recently the relative citation ratio.  Now we’ll focus on a different kind of “P” for productivity, namely patents (which arguably are also related to “T” for translation).  …. Do NIH-supported papers that are cited by patents have a higher Relative Citation Ratio than those that are not cited by patents? As a refresher, the Relative Citation Ratio uses citation rates to measure the influence of a publication at the article level…. We identified 119,674 unique NIH grants that were funded between 1995 and 2007 and that generated at least one publication…. Continue reading

Email this to someoneTweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Pin on PinterestPrint this page

Data On Trends According to Career Stage

By the 21st Century Cures Act, the Next Generation Researchers’ Initiative calls on the NIH to develop policies to increase funding opportunities for new researchers seeking to secure early independence. To put the Initiative in perspective and to extend on previous blogs we’ve posted on changing demographics in NIH-funded researchers, we thought it would be useful to explore trends according to career stage.

First, some definitions.  We define “Early Stage Investigators” (ESI) as those who are within 10 years of completing their terminal degree or post-graduate clinical training and who have not yet secured independence as a PI of a substantial NIH research award.  …. Continue reading

Email this to someoneTweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Pin on PinterestPrint this page

Applications, Resubmissions, and the Relative Citation Ratio

Measuring the impact of NIH grants is an important input in our stewardship of research funding. One metric we can use to look at impact, discussed previously on this blog, is the relative citation ratio (or RCR). This measure – which NIH has made freely available through the iCite tool – aims to go further than just raw numbers of published research findings or citations, by quantifying the impact and influence of a research article both within the context of its research field and benchmarked against publications resulting from NIH R01 awards.

In light of our more recent posts on applications and resubmissions, we’d like to go a step further by looking at long-term bibliometric outcomes as a function of submission number. In other words, are there any observable trends in the impact of publications resulting from an NIH grant funded as an A0, versus those funded as an A1 or A2? And does that answer change when we take into account how much funding each grant received? …. Continue reading

Email this to someoneTweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Pin on PinterestPrint this page

Mid-career Investigators and Shifting Demographics of NIH Grant Recipients

While NIH policies focus on early stage investigators, we also recognize that it is in our interest to make sure that we continue to support outstanding scientists at all stages of their career. Many of us have heard mid-career investigators express concerns about difficulties staying funded. In a 2016 blog post we looked at data to answer the frequent question, “Is it more difficult to renew a grant than to get one in the first place?” We found that new investigators going for their first competitive renewal had lower success rates than established investigators. More recently, my colleagues in OER’s Statistical Analysis and Reporting Branch and the National Heart Lung and Blood Institute approached the concerns of mid-career investigators in a different way – by looking at the association of funding with age. Today I’d like to highlight some of the NIH-wide findings, recently published in the PLOS ONE article, “Shifting Demographics among Research Project Grant Awardees at the National Heart, Lung, and Blood Institute (NHLBI)”. Using age as a proxy for career stage, the authors analyzed funding outcomes for three groups …. Continue reading

Email this to someoneTweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Pin on PinterestPrint this page

Resubmissions Revisited: Funded Resubmission Applications and Their Initial Peer Review Scores

“My first submission got an overall impact score of 30. Is that good enough? What’s the likelihood I’ll eventually get this funded?”, or, “My first submission was not even discussed. Now what? Does anyone with an undiscussed grant bother to … Continue reading

Email this to someoneTweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Pin on PinterestPrint this page

Following up on the Research Commitment Index as a Tool to Describe Grant Support

Many thanks for your terrific questions and comments to last month’s post, Research Commitment Index: A New Tool for Describing Grant Support. I’d like to use this opportunity to address a couple of key points brought up by a number of commenters; in later blogs, we’ll focus on other suggestions.

The two points I’d like to address here are: 1) why use log-transformed values when plotting output (annual weighted relative citation ratio, or annual RCR) against input (annual research commitment index, or annual RCI), and 2) what is meant by diminishing returns. …. Continue reading

Email this to someoneTweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Pin on PinterestPrint this page

Research Commitment Index: A New Tool for Describing Grant Support

On this blog we previously discussed ways to measure the value returned from research funding. Several of my colleagues and I, led by NIGMS director Jon Lorsch – chair of an NIH Working Group on Policies for Efficient and Stable Funding – conceived of a “Research Commitment Index,” or “RCI.” We focus on the grant activity code (R01, R21, P01, etc) and ask ourselves about the kind of personal commitment it entails for the investigator(s). We start with the most common type of award, the R01, and assign it an RCI value of 7 points. And then, in consultation with our NIH colleagues, we assigned RCI values to other activity codes: fewer points for R03 and R21 grants, more points P01 grants. Continue reading

Email this to someoneTweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Pin on PinterestPrint this page

Characteristics and Outcomes of R01 Competing Renewal Applications (“Type 2s”)

An investigator’s long-term success depends not only on securing funding, but on maintaining a stable funding stream. One way to assure continued funding is to submit a competing renewal application. However, as we noted earlier this year, while new investigators were almost as successful as experienced investigators in obtaining new (type 1) R01s, the difference between new investigator and experienced investigator success rates widens when looking at competing renewals (type 2s), and success rates of new investigators’ first renewals were lower than those of experienced investigators. In addition, we know that since the end of NIH’s budget doubling in 2003, success rates for competing renewals of research project grants overall have decreased. To further understand trends in success rate for R01 competing renewals (“type 2s”) I’d like to share some additional analyses where we look at characteristics of type 2 R01 applications, and the association of their criterion scores with overall impact score and funding outcomes. Continue reading

Email this to someoneTweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Pin on PinterestPrint this page

How Many Researchers Were Supported by NIH as Trainees?

Earlier this year we reported on the unique numbers of research project grant (RPG) awardees and applicants each year since the end of the NIH doubling, in 2003. We described how the number of unique RPG awardees has remained relatively constant, while the number of applicants (as assessed over 5-year windows) has steadily and markedly increased.
A number of readers asked us about the prior NIH-supported research training and career development of these investigators. Among RPG awardees, what proportion had received prior fellowship, training, or career development (F, T, or K) awards? And perhaps of greater interest, among unsuccessful, unfunded applicants, what proportion had received prior fellowship, training or career awards?
To answer these questions, we start with a quick recap. …. Continue reading

Email this to someoneTweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Pin on PinterestPrint this page