My office devotes a lot of effort to evaluating its grants policies and practices. Since the introduction of the Enhancing Peer Review changes, NIH has sponsored an ongoing evaluation of peer review, involving formal surveys about the peer review process, as well as other types of analyses. We are keenly interested in your responses to our surveys, so if you receive an invitation to take a peer review survey, I hope you can find the time to respond.
The importance of participating in these surveys is nicely illustrated by a recent study completed by my office, evaluating peer review. NIH’s peer review system relies upon the service of well-qualified reviewers from the biomedical research community, most of whom are NIH grantees. We are grateful for the service of NIH peer reviewers, and we know there are concerns about the burden this imposes on those who are simultaneously submitting applications and serving as reviewers. There also are concerns that NIH’s demand for peer reviewers may have exceeded the capacity of our current grantees to support it. Therefore, over the past two years my office has been evaluating the level of service that most peer reviewers are willing and able to provide, and how peer review service fits within the scope of reviewers’ other professional responsibilities.
We looked at peer review participation among PIs who have had active R01 or other RPG funding within the past five years. Since nearly half of the applications NIH reviews are for R01 grants, R01 investigators are most frequently sought by NIH for peer review service, particularly those with longer experience managing R01 grants. The data below show peer review service broken out by age, as well as by levels of funding. The data show that more than 80% of mid-career R01 recipients have served as reviewers at least once in the past five years. Furthermore, well over 75% of grantees with more than $1 million in funding in the past five years have served.
As it is important to consider the impact peer review service is having on our grantee community, we conducted a survey of NIH grant applicants and awardees to assess how many of them serve as reviewers when asked, and to understand their peer review service in the context of their overall professional responsibilities. We found that 88% of respondents who reported having been invited to review in the prior year had served at least once. When asked how much of their professional effort should ideally be spent on peer review service for grants, approximately 51% of respondents reported that peer review of grants should comprise less than 5% of their professional effort, but another 46% reported that peer review of grants should make up 5-10% of their worktime. This is a significant time commitment from investigators.
We decided to model how many more potential reviewers we might be able to recruit if faced with a large influx of applications. For this modeling exercise, we assumed we would recruit individuals who had at least one R01 award and at least $500k in funding (cumulatively) in the past five years, and who had at least one active award of any activity code at the time of peer review service. In our modeling, we also accounted for reviewers’ preferences for peer review participation, as reported in the survey. Survey respondents reported that they considered an assignment load of 6 applications per meeting, and 1 – 2 meetings per year, to be reasonable expectations. With those criteria in mind, we calculate that NIH has approximately 3500 qualified reviewers/year who have not yet served in the last five years, and if 80% of them agreed to review, NIH could handle about 5,500 more applications/year. Since older grantees with longer experience are currently serving in greater proportions, many of the additional peer reviewers NIH could recruit would necessarily be younger and closer to their New Investigator status. By adjusting the criteria, this approach should help us better manage high numbers of grant applications in the future.
So, in terms of peer review capacity, what is the take away here? Much of our community gives their time and energy so that our peer review system stays strong. But currently NIH has not tapped the full capacity of the peer review system. NIH will continue to monitor application trends closely, as part of our ongoing efforts to evaluate peer review. We here at NIH want to underscore the importance of your participation in support of the scientific enterprise and thank you making the NIH peer review system the gold standard for the world.