Applying the Relative Citation Ratio as a Measure of Grant Productivity

October 21, 2016
By

Last April we posted a blog on the measurement of citation metrics as a function of grant funding. We focused on a group of R01 grants and described the association of a “citation percentile” measure with funding. We noted evidence of “diminishing returns” – that is increased levels of funding were associated with decreasing increments of productivity – an observation that has been noted by others as well.

We were gratified by the many comments we received, through the blog and elsewhere. Furthermore, as I noted in a blog last month, our Office of Portfolio Analysis has released data on the “Relative Citation Ratio,” (or RCR) a robust field-normalized measure of citation influence of a single grant (and as I mentioned, a measure that is available to you for free).

In the follow-up analysis I’d like to share with you today, we focus on a cohort of 60,447 P01 and R01-equivalent grants (R01, R29, and R37) which were first funded between 1995 and 2009. Through the end of 2014, these grants yielded at least 654,607 papers. We calculated a “weighted RCR” value for each grant, ….

, , , ,

12 Comments

Give Us Your Feedback on Standards for Preprints, and Other Interim Research Products

October 11, 2016
By

Many scientific disciplines, like physics and mathematics, routinely communicate research findings through preprints — manuscripts that have not yet gone through the formal peer review, editing, or journal publishing process. However, this is still a relatively novel concept in biology and clinical research. …. we at NIH are interested in feedback from you, to get a fuller understanding of the current use of preprints in the broader NIH-supported research community, accepted preprint standards, whether preprints should be included in NIH applications and reports, and how investigators could report them. …. Not only are we interested in your current use of preprints — and other interim research products — but perhaps most importantly we’d like your feedback on the specific standards that need to be in place for their citation, particularly for citation in NIH applications and reports ….

, ,

3 Comments

Improving Visibility of NIH-supported Clinical Trial Activities and Results

September 16, 2016
By

In a separate post today, we provide an overview of the various reforms the NIH is leading to enhance our stewardship of clinical trials. In this post we’d like to focus a bit more on our efforts to broadly disseminate clinical trial availability and results information.

Timely dissemination of clinical trial results information has been a problem, one that has been documented more than once, and that appears to apply to NIH- as well as non-NIH funded trials. To realize the benefits of a clinical trial, the findings must be available to the public as soon as possible after the trial has concluded. This is not only responsible use of taxpayer dollars for publicly funded trials, but also fulfills our responsibility to the individuals who volunteered in these studies with an understanding that their participation would contribute to advancing medical knowledge. Today, NIH announced a new policy that will complement a new federal regulation, referred to here as the Final Rule, also released today, to improve the accessibility of information on clinical trial availability and on the outcomes and results of completed trials.

As you likely know, to carry out the laws passed by Congress, federal agencies issue regulations that govern the activities of the agency and the applicable community. The “Final Rule” announced today by the U.S. Department of Health and Human Services is …

, , , ,

2 Comments

Building Better Clinical Trials through Stewardship and Transparency

September 16, 2016
By

NIH is the largest public funder of clinical trials in the United States. As stewards of this research enterprise, we have been actively listening and discussing how to overcome hurdles and shortcomings that we, and others in the research community, have identified. If you’ve been following the conversation, you’ll know that NIH already has implemented some key reforms to enhance clinical trial stewardship. Today, in a Viewpoint Essay published in the Journal of the American Medical Association (JAMA), we provide an overview of how these reforms, and new initiatives, fit in to the broader picture of building a better clinical trial enterprise through better stewardship, accountability, and transparency.

Figure 1 illustrates the clinical trial “lifespan”, and key opportunities for improving the quality and efficiency of clinical trials – opportunities that translate into more innovative and robust clinical trial design, and accelerated discoveries that will advance human health. NIH is leading a multi-faceted effort that addresses shortcomings and challenges throughout this lifespan, including the application and award process; the scientific review of trial applications; post-award management and oversight; sharing of trial data; and dissemination of research results information to the public. ….

,

6 Comments

Measuring Impact of NIH-supported Publications with a New Metric: the Relative Citation Ratio

September 8, 2016
By

In previous blogs, we talked about citation measures as one metric for scientific productivity. Raw citation counts are inherently problematic – different fields cite at different rates, and citation counts rise and fall in the months to years after a publication appears. Therefore, a number of bibliometric scholars have focused on developing methods that measure citation impact while also accounting for field of study and time of publication. We are pleased to report that on September 6, PLoS Biology published a paper from our NIH colleagues in the Office of Portfolio Analysis on “The Relative Citation Ratio: A New Metric that Uses Citation Rates to Measure Influence at the Article Level.” Before we delve into the details and look at some real data, ….

, , , ,

14 Comments

NRSA Postdoctoral Stipend Guidelines for FY2017

August 30, 2016
By

This month, NIH published the projected fiscal year 2017 stipend guidelines for postdoctoral trainees and fellows supported by National Research Service Awards (NRSAs). For NRSA-supported postdocs with less than one year’s experience, the stipend level will increase to $47,484. In keeping with the recommendation of the Biomedical Research Workforce Working Group of the NIH Advisory Committee to the Director, stipend levels then increase dependent on years of postdoctoral experience. ….

, , ,

7 Comments

Model Organisms, Part 3: A Look at All RPGs for Six Models

August 24, 2016
By

We are most appreciative of the feedback we’ve received, through the blog and elsewhere, on NIH support of model organism research. In part 1 of this series, we mentioned that we asked two separate groups to analyze NIH applications and awards. In parts 1 and 2 we primarily focused on R01-based data that were curated and analyzed by our Office of Portfolio Analysis. In part 3, we show results from a broader range of research project grant (RPG) data that were prepared and analyzed by our Office of Research Information Systems. This group used an automated thesaurus-based text mining system which delves into not only public data such as project titles, abstracts, public health relevance statements, but also the specific aims contained in RPG applications. ….

,

6 Comments

A Look at NIH Support for Model Organisms, Part Two

August 3, 2016
By

We were pleased to hear the feedback on our previous post on NIH-funded model-organism research. One question a number of you asked is: what’s happening with research involving mouse models? Thanks to additional work by colleagues in NIH’s Office of Portfolio Analysis (OPA) and Office of Extramural Research/Office of Research Information Systems, I’m excited to … Continue reading “A Look at NIH Support for Model Organisms, Part Two”

,

7 Comments

Join Us in Chicago to Learn About NIH Program Funding and Grants Administration

July 28, 2016
By

In the spring, I attended my first NIH Regional Seminar on Program Funding and Grants Administration. I greatly enjoyed meeting with a diverse group of scientists and science administrators. During two “Open Mike” sessions, I had the opportunity to engage in rich conversations with attendees on topics I frequently include on this blog. The conversations were enjoyable and insightful – there’s nothing like discussing such important issues like funding trends, research accountability, and grants policy face-to-face with new and early career scientists. …. Why am I writing about a seminar that took place in May? Because I have some good news for those of you who didn’t get a chance to come to Baltimore! NIH is hosting a second Regional Seminar this year, from October 26-28 …..

, ,

2 Comments

The Predictive Nature of Criterion Scores on Impact Score and Funding Outcomes

July 22, 2016
By

In order to develop and implement data-driven policy, we need to carefully analyze our data to understand the “stories behind our metrics.” Without analyzing our data to know what’s going on, we’re essentially flying blind! A group of authors from the NIH Office of Extramural Research sought to investigate the stories behind peer review scoring and why some grant applications are more likely to be funded than others. They extended analyses previously reported by NIH’s Office of Extramural Research and National Institute of General Medical Studies. Last month, they published their analysis of over 123,000 competing R01 applications and described the correlations of individual component peer review scores – significance, investigator(s), innovation, approach, and environment – with subsequent overall impact score, and funding outcome. ….

, ,

16 Comments