Continuing Steps to Ensuring Credibility of NIH Research: Selecting Journals with Credible Practices

November 8, 2017

The scientific community is paying increasing attention to the quality practices of journals and publishers. NIH recently released a Guide notice (NOT-OD-18-011) to encourage authors to publish in journals that do not undermine the credibility, impact, and accuracy of their research findings. This notice aims to raise awareness about practices like changing publication fees without notice, lacking transparency in publication procedures, misrepresenting editorial boards, and/or using suspicious peer review.

A Reminder of Your Roles as Applicants and Reviewers in Maintaining the Confidentiality of Peer Review

April 7, 2017

Imagine this: you’re a reviewer on an NIH study section, and receive a greeting card from the Principal Investigator (PI) on an application you are reviewing. A note written inside the card asks that your look favorably upon the application, and in return, the PI would put in a good word with his friend serving on your promotion committee. Do you accept the offer, or just ignore it? Or, do you report it? …. Or maybe several days after the initial peer review of your application, you receive a phone call from a colleague you haven’t spoken to in quite a while. The colleague is excited about a new technique you developed and wishes to collaborate. You realize the only place you’ve disclosed this new technique is in your recently reviewed NIH grant application. What do you do? ….

Characteristics and Outcomes of R01 Competing Renewal Applications (“Type 2s”)

December 22, 2016

An investigator’s long-term success depends not only on securing funding, but on maintaining a stable funding stream. One way to assure continued funding is to submit a competing renewal application. However, as we noted earlier this year, while new investigators were almost as successful as experienced investigators in obtaining new (type 1) R01s, the difference between new investigator and experienced investigator success rates widens when looking at competing renewals (type 2s), and success rates of new investigators’ first renewals were lower than those of experienced investigators. In addition, we know that since the end of NIH’s budget doubling in 2003, success rates for competing renewals of research project grants overall have decreased. To further understand trends in success rate for R01 competing renewals (“type 2s”) I’d like to share some additional analyses where we look at characteristics of type 2 R01 applications, and the association of their criterion scores with overall impact score and funding outcomes.

Are You On the Fence About Whether to Resubmit?

October 28, 2016

When applicants receive their summary statement resulting from the review of an application that was assigned a score outside of the ICs funding range, there are important decisions to be made that, ideally, should be based upon evidence. What is the likelihood that an application like this one will be funded? If I resubmit the application, what changes might improve the chances for a successful resubmission?

Recall that in 2014, NIH relaxed its resubmission policy (OD-14-074) to allow applicants to submit a new (A0) application following an unsuccessful resubmission application. Also, we recently posted a piece showing that review outcomes for new applications submitted following an unsuccessful resubmission had about the same funding success as other new applications. But some applicants may wonder, what is the funding success for a resubmission application? ….

The Predictive Nature of Criterion Scores on Impact Score and Funding Outcomes

July 22, 2016

In order to develop and implement data-driven policy, we need to carefully analyze our data to understand the “stories behind our metrics.” Without analyzing our data to know what’s going on, we’re essentially flying blind! A group of authors from the NIH Office of Extramural Research sought to investigate the stories behind peer review scoring and why some grant applications are more likely to be funded than others. They extended analyses previously reported by NIH’s Office of Extramural Research and National Institute of General Medical Studies. Last month, they published their analysis of over 123,000 competing R01 applications and described the correlations of individual component peer review scores – significance, investigator(s), innovation, approach, and environment – with subsequent overall impact score, and funding outcome. ….

Perspectives on Peer Review at the NIH

November 12, 2015

In today’s New England Journal of Medicine, Richard Nakamura, the director of NIH’s Center for Scientific Review (CSR), and I published an essay titled “Reviewing Peer Review at the NIH.” As the competition for NIH research grants has become increasingly stiff, review scores often are pointed to as the reason for failure to obtain funding. Indeed, over the past few years, peer review has come under increasing scrutiny. Critics have argued that peer review fails in its primary mission – to help funding agencies make the best decisions about which projects and which investigators to support. ….

Understanding the Capacity of NIH’s Peer Review System

July 30, 2015

My office devotes a lot of effort to evaluating its grants policies and practices. Since the introduction of the Enhancing Peer Review changes, NIH has sponsored an ongoing evaluation of peer review, involving formal surveys about the peer review process, as well as other types of analyses. We are keenly interested in your responses to our surveys, so if you receive an invitation to take a peer review survey, I hope you can find the time to respond. The importance of participating in these surveys is nicely illustrated ….

Supporting the Call to Peer Review Service

February 20, 2015

Last year we highlighted a new section of the NIH Data Book that provides statistics about peer review across NIH, including peer review organized by the Center of Scientific Review as well as peer review organized by NIH institutes and centers. Then, and now, these data truly demonstrate how you, our peer reviewers, are the lifeline of the scientific process. During 2014 nearly 24,000 reviewers participated in more than 2,500 peer review meetings to assess the scientific and technical merit of NIH applications, and many of those reviewers participated in multiple meetings. Your service to NIH expands our capacity to fund the best science, and provides an essential breadth of perspectives on the research projects and research training that NIH supports. In the lab and in the study section, we’re grateful for the myriad ways you work with us to advance human health and benefit the health of our nation, and the world. ….