Guest post by Noni Byrnes, Director of the NIH Center for Scientific Review, originally released on the Review Matters blog
The scientific peer review process benefits greatly when the study section reviewers bring not only strong scientific qualifications and expertise, but also a broad range of backgrounds and varying scientific perspectives. Bringing new viewpoints into the process replenishes and refreshes the study section, enhancing the quality of its output.
In this context, CSR recently removed the requirement to have at least 50% full professors on committees. This had sometimes led to a misguided attempt to “do better than the metric” by aiming for a committee of all full professors. We are now encouraging scientific review officers (SROs) to focus on scientific contributions (demonstrable in a range of ways, e.g. recent publications, R01 or equivalent extramural funding from other sources, etc.), expertise, and breadth instead of trying to meet a career-stage metric. Our goal is to achieve a balance of perspectives by including a mix of qualified senior, mid-career, and junior scientists on study sections.Our data show that this can be achieved, as we have not exhausted the pool of eligible reviewers. Using R01 grant funding as a rough indicator of “qualified to review” we looked at interactive (i.e. excluded mail review) review service records of R01 awardees. As of January 1, 2020, there were 22,608 individuals with active R01 funding. Of these, 30% (6715) have served one to five times, and 18% (4074) have never served as a reviewer in the last 12 years. Of those who have served only one to five times over 12 years, 26% are assistant professors and 34% are associate professors.
In an effort to facilitate broader participation in review, we are making these data available to SROs and encouraging them to identify qualified and scientifically appropriate reviewers, who may not have been on their radar previously. In another step to broaden the pool of reviewers, we will launch a web page this spring through which scientific societies can recommend qualified reviewers.
We greatly appreciate the generosity of reviewers who contribute to the scientific community by freely giving their time to review NIH grant applications. However, one aspect of broadening the pool of reviewers is to avoid excessive review service by a small fraction of people, which can lead them to have a disproportionate effect on review outcomes. We are looking into issue of undue influence, or the “gatekeeper” phenomenon, where a reviewer has participated in the NIH peer review process at a rate much higher than their peers, and thus has had a disproportionate effect on review outcomes in a given field. Below is a plot examining the service records of all 24,642 reviewers who served as reviewers (in a capacity other than as a mail reviewer) for CSR within the past two years. Each dot represents a reviewer’s number of meetings for the NIH over the last 12 years.
Below the yellow line are reviewers who have served an average of once per round for 12 years—these make up 94% of the total pool and include the vast majority of the reviewers used. Between the yellow and red lines are reviewers who have served an average of twice per round for 12 years, 5% of the pool. The small number of reviewers above the red line have served an average of 3 times per round (9 times per year) for 12 years; this is the sort of excessive review service that raises concerns about undue influence.
To actively manage undue influence, we have asked SROs to check review service records prior to inviting reviewers. In addition, based on our analysis of the reviewer service data and the unanimous recommendation of the CSR Advisory Council, we are discontinuing (NOT-OD-20-060) the “recent substantial service” policy that extended continuous submission benefits to those who served 6 times over a period of 18 months (6 times in 5 rounds). This policy, established in 2009 with the best of intentions to incentivize review, has led to the unintended consequence of encouraging excessive ad hoc review service within a short time frame. Implementation of this change will be gradual, maintaining eligibility for those reviewers who have already served, and honoring commitments to those who are serving now and will continue to serve through June 30, 2020, the end of the current eligibility period. Continuous submission eligibility for appointed members of NIH study sections and committees will not change. Policies surrounding late submission of applications also are unaffected (NOT-OD-15-039).
CSR’s mission is to provide a fair, independent, expert, timely review – free of inappropriate influences – so the NIH can fund the most promising research. CSR continues to work to broaden its scientific review groups because a diversity of expert perspectives better allows identification of potentially high-impact research, which benefits all of us.
If you haven’t served recently, we hope you will do so if asked.