26 Comments
My office devotes a lot of effort to evaluating its grants policies and practices. Since the introduction of the Enhancing Peer Review changes, NIH has sponsored an ongoing evaluation of peer review, involving formal surveys about the peer review process, as well as other types of analyses. We are keenly interested in your responses to our surveys, so if you receive an invitation to take a peer review survey, I hope you can find the time to respond.
The importance of participating in these surveys is nicely illustrated by a recent study completed by my office, evaluating peer review. NIH’s peer review system relies upon the service of well-qualified reviewers from the biomedical research community, most of whom are NIH grantees. We are grateful for the service of NIH peer reviewers, and we know there are concerns about the burden this imposes on those who are simultaneously submitting applications and serving as reviewers. There also are concerns that NIH’s demand for peer reviewers may have exceeded the capacity of our current grantees to support it. Therefore, over the past two years my office has been evaluating the level of service that most peer reviewers are willing and able to provide, and how peer review service fits within the scope of reviewers’ other professional responsibilities.
We looked at peer review participation among PIs who have had active R01 or other RPG funding within the past five years. Since nearly half of the applications NIH reviews are for R01 grants, R01 investigators are most frequently sought by NIH for peer review service, particularly those with longer experience managing R01 grants. The data below show peer review service broken out by age, as well as by levels of funding. The data show that more than 80% of mid-career R01 recipients have served as reviewers at least once in the past five years. Furthermore, well over 75% of grantees with more than $1 million in funding in the past five years have served.
As it is important to consider the impact peer review service is having on our grantee community, we conducted a survey of NIH grant applicants and awardees to assess how many of them serve as reviewers when asked, and to understand their peer review service in the context of their overall professional responsibilities. We found that 88% of respondents who reported having been invited to review in the prior year had served at least once. When asked how much of their professional effort should ideally be spent on peer review service for grants, approximately 51% of respondents reported that peer review of grants should comprise less than 5% of their professional effort, but another 46% reported that peer review of grants should make up 5-10% of their worktime. This is a significant time commitment from investigators.
We decided to model how many more potential reviewers we might be able to recruit if faced with a large influx of applications. For this modeling exercise, we assumed we would recruit individuals who had at least one R01 award and at least $500k in funding (cumulatively) in the past five years, and who had at least one active award of any activity code at the time of peer review service. In our modeling, we also accounted for reviewers’ preferences for peer review participation, as reported in the survey. Survey respondents reported that they considered an assignment load of 6 applications per meeting, and 1 – 2 meetings per year, to be reasonable expectations. With those criteria in mind, we calculate that NIH has approximately 3500 qualified reviewers/year who have not yet served in the last five years, and if 80% of them agreed to review, NIH could handle about 5,500 more applications/year. Since older grantees with longer experience are currently serving in greater proportions, many of the additional peer reviewers NIH could recruit would necessarily be younger and closer to their New Investigator status. By adjusting the criteria, this approach should help us better manage high numbers of grant applications in the future.
So, in terms of peer review capacity, what is the take away here? Much of our community gives their time and energy so that our peer review system stays strong. But currently NIH has not tapped the full capacity of the peer review system. NIH will continue to monitor application trends closely, as part of our ongoing efforts to evaluate peer review. We here at NIH want to underscore the importance of your participation in support of the scientific enterprise and thank you making the NIH peer review system the gold standard for the world.
I think these analyses sort of miss the point. Why in the world would you restrict recruitment to people with “at least one R01 award and at least $500k in funding”? There are TONS of people out there, particularly mentored investigators who are mining their mentors’ data, who are amazing scientists and who are publishing (and influencing the field) at far greater rates of intensity than these B-team players who get an R01 but can’t manage to publish more than 1-2 papers a year.
Generally, reviewers must be recognized authorities in their field; experienced in investigating line(s) of research comparable to those being reviewed; and dedicated to high quality, fair reviewers. In addition, the membership of the review group must be diverse with respect to the geographic distribution, gender, race and ethnicity of the members. While having an R01 grant is one way to judge an individual’s qualifications, it is not a prerequisite in order to serve in NIH peer review. See http://ofacp.od.nih.gov/committees/pdf/HowScientistsSelectedStudySectionService.pdf
I have had R01 funded and been on special emphasis panels- I have volunteered to be a member of a study section and have not been asked. I know another researcher who is in the same position. Not sure what else we can do- is there a specific way we can volunteer or apply to a study section outside of notifying program officers- issuing a specific request to fill study section vacancies to funded or formally funded investigators might solve your problem.
Get inside the old boys’ club. Takes money (invite speakers sitting on study sections, shower them with expensive dinners and “speaker fees”-it’s all legal. The NIH turns a blind eye to this form of corruption). Once u r in there, fight to hold on to your power tooth and nail, and use your power and influence to make more powerful friends. It’s a big political power game, and anyone who is in the club knows it, but will never admit to it publicly.
No Brian, what you describe is one way to be “visible” to scientists who are already on the study section, which may help during the review of a later proposal. Oliver’s question was how to get asked to serve on a study section. That decision lies with CSR staff who cannot bribed with dinners and such because it is strictly forbidden to pay for ANY of their expenses. If she has already volunteered directly to the appropriate Scientific Review Officer, the follow-up option is to ask her grad or postdoc mentor to recommend her. Another option might be a recommendation to a SRO by her department chair, especially if they do similar work that might go to the same study section. A third option is to ask among her peer scientists if they have served, and if so, asking that friend for a recommendation to the right SRO.
It is also possible that persons who serve as an ad hoc member of a study section are not asked again because their previous performance was less than ideal. I have experience with selecting new members and evaluating ‘trial’ members. While most scientists do an outstanding job as study section members, others are not what study section desires. Written critiques are late or incomplete/cryptic and/or the level of discussion and engagement at the meetings is lacking.
There are several ways to volunteer to serve on NIH review panels, as explained on the OER website (http://grants.nih.gov/grants/peer/becoming_peer_reviewer.htm).
• Email ReviewerVolunteer@mail.nih.gov to be added to a trans-NIH database of reviewer volunteers (include a brief description of your scientific areas of expertise in the body of the email [1-2 sentences], and a copy of your biosketch as an attachment
• See the CSR Become a Reviewer website (http://public.csr.nih.gov/ReviewerResources/BecomeAReviewer/Pages/default.aspx) and follow the guidance there
• Contact an NIH Scientific Review Officer in your area of science (see the list of CSR Integrated Review Groups at http://public.csr.nih.gov/StudySections/Pages/default.aspx)
But what is more important, number of papers published per year or impact of your research?
I’ve been reviewing grants for about 15 years. I was a standing member of two different consecutive study sections and a number of ad hoc sessions in between. A standing member spends way more than 5% effort on these reviews, more like 25 to 30%. Should be 5 or 10% but that would mean meeting once a year or reviewing fewer grants. Can’t see how that will happen.
Respondents who have commented on the Enhancing Peer Review Surveys have told us they spend 6-8 hours reviewing each application assigned to them. If we assume they are assigned 8 -10 applications per round and attend a two-day study section meeting three times per year, this comes to 240 – 288 hours per year, or 12-14% effort based on a 2000 hour work year. This does not take into account additional ad hoc service chartered reviewers often contribute. We are grateful for the service of NIH peer reviewers and recognize the importance of your participation. Thank you!
As someone who held 3 R01s for several years i was a chartered member of a study section for 3 years, in addition to serving ad hoc on other NIH committees fro fellowships or special emphasis panels. Following becoming a parent just before the Great Recession folllowed by a move, i went for a few years wo NIH R01 support. Although very experienced and willing to serve i have never been called for R01 reviews in the last 5-6 years. Your cutoff of people w R01 funding (which i have again). Your choice of people to serve on study section results in a majority crop of people doing B+ ‘safe’ science. This is a huge mistake. The other mistake i see is putting Asst Profs on a SS. What happened to the general rule of Assoc Prof w experiences being invited and tested – and not earlier.
Please shift your focus at least a bit to SBIR reviewers. The academics on these panels have no appreciation for the commercial aspects of the technology and resent not being in the big leagues reviewing R01s and the businesspeople on them are basically from SBIR shops who have never commercialized a product. Dismal, calcified panels.
Dr. Tasso,
As a B- scientist, I think your view of progress in biology is myopic. If I were sequencing DNA and had tons of data, then many investigators could use it. Understood. But there are functional studies that require other disciplines that hone in on the functions of models that use genetic, molecular biology, and biochemistry to answer hypothesis-driven questions. One must consider the significance of each paper, not just the number of papers produced. Of course, the expectations should depend on the approach. There has to be room enough for independent approaches to solve a problem or we may focus too narrowly and miss the big picture. The purpose should not be the number of papers, but the significance that those papers add to the problem under study. But then again, I am just a B- scientist.
I have been serving as a NIH reviewer on a variety of grant types and study sections, including center grants and SEPs, for about 10 years. I review one or two sets of applications per year. So I have participated in many review sessions under many different SRAs and with a diverse group of reviewers. My biggest concern is the perpetuation of bias against smaller schools and institutions, and newer less known investigators. In my experience, proposals from the big name schools have an unfair advantage – reviewers are usually more willing to grant them the benefit of doubt, relying on their reputations. Not all good proposals come from the top schools. I believe the names of the investigators and their institutions should not be included in the peer review. These credentials can be evaluated by NIH officials before the applications are selected for review. This will force the reviewers to focus on the contents rather than the auspices, which in my opinion would yield a more fair review.
I agree with this observation. I myself am guilty doing this. We usually ask for a much more extensive proof of feasibility from unkonwn investigators.
Served for NIH review panel for the past ten years, i observed the same thing Mobley pointed out. NIH may look the way Bill Gates Foundation reviewing proposals, which the PI and institution information shall not be reviewed in the first round.
As a person continuously funded by the NSF, and not in a major way by the NIH, I have served on many different NIH IRGs. At present, as a more established member of the community, I simply do not have the loyalty to the NIH to go above and beyond, as I may do for the NSF. This is because, in the view of the NIH IRG reviewers, I am possibly a B to B- scientist. Although my NSF record is likely contradictory. So, I am more likely to decline NIH service (it contributes little to my research portfolio or my career growth). One way to alter this, for recruiting qualified scientists not funded by the NIH, is to “buy out their time” from academic institutions. This will at least provide an incentive for those disenchanted by the NIH IRGs, which are a old-boys’ network anyway, to offer fee-for-service. I would spend 20% of my time for the NIH, if the NIH paid 20% of my salary to buy my services out of other requirements at my institution. If NIH can pay med school salaries, this would be drop in the bucket.
Thank you for the data. It would be helpful to see the age frequency distribution of reviewers, possibly in the same bins as in the first figure. We see what percentage of awardees participated as peer reviewers by age group, however I wonder, in absolute numbers, how many reviewers there are in each age bracket.
“When asked how much of their professional effort should ideally be spent on peer review service for grants, approximately 51% of respondents reported that peer review of grants should comprise less than 5% of their professional effort”
Question: Were reviewers asked how much of their professional effort is actually spent on peer review service annually or per panel participation? If this question was answered, how does this compare to the estimated effort based on participation of reviewers over the 2009-2014 period?
We asked respondents to complete a table in which they estimate their percent effort into a number of categories including research, research project management, teaching, etc. and including Peer Review Service. However, we did not ask respondents to base the effort calculation on a 40-hour work week because we knew that this might be difficult for many scientists to do. We know that researchers typically only track their time for research and research-related activities, and they often spend much more than 40 hours per week on their work. The responses to this question indicated that, overall, respondents who budgeted any time for peer review service spent an average of 5.6% of their effort on peer review service. Respondents who reported having reviewed for NIH in the past year spent about 6.1% effort on peer review service; respondents who indicated that they had reviewed in the past year AND their most recent peer review service was in the capacity of a chartered member reported 7.4% effort. Please also see our response to another comment to this post.
For the numbers of reviewers in each age bracket, you can find the data tables for these graphs in the Excel file posted to RePORT.
Thank you for your interest in the survey.
I am very surprised that having an R01 is a criterion for reviewers. I agree with others that this is too restrictive, and leaves out many experienced reviewers. At minimum, other types of NIH grants should qualify. The R01 requirement lends itself to the critique that this is a “club” system, as mentioned in some of the other comments. In my experience, it takes quite a while for a scientist to become a well-rounded reviewer. This is readily apparent from serving on study sections. I review for the NSF, DOD, and VA and have chaired some of these review groups. I would welcome the opportunity to provide equitable reviews for the NIH again, as I have done in the past, as I seek to re-establish NIH funding myself.
Selection of reviewers who are expert, not generally working in a related field, is the best way to ensure a fair review. No one can generate sufficient enthusiasm to fund a grant if they are not knowledgeable and appreciate the impact of what is proposed. Study sections have become too passive, members unwilling to challenge poor comments from their colleagues around the table. This also speaks to study section leadership that is mainly obsessed with staying on time. Finally, true innovation does not do well at study section (if it even makes it to discussion). If the proposal is novel and potentially impactful and justified with solid pubs or good prelim data, then be enthusiastic!
The NIH peer review system is broken. Any person who is honest with him or herself and intimately familiar with this honor system will tell you this in private. Think about it.
We take great pride in our fair judicial system where we are judged by a jury of our peers, and rightly so. The system works because these peers (jurors) have no contact, direct or indirect, with the person they are judging-NONE WHATSOEVER. In fact, it is a miscarriage of justice if any of these jurors have anything to do with the defendent. How fair would the system be if any of the jurors knew that one day they may be a defendent and the defendent they are judging today may actually be judging them? Would any one have faith in a system in which it is fair game for the defendent to invite the jurors to visit them, take them out to dinner, provide them a cash “honararium”?
Can the NIH peer review system stand up to such scrutiny?
It seems that the basis for reviewer selection is somewhat flawed:
“Since nearly HALF of the applications NIH reviews are for R01 grants, R01 investigators are MOST FREQUENTLY sought by NIH for peer review service, particularly those with longer experience managing R01 grants.”
This seems to promote a major disservice to applicants seeking other funding mechanisms such as SBIR.
While R01 grants support focused research, other funding mechanisms such as R43 grants are supposed to support projects to establish the feasibility of technologies which may lead to commercial products. Academic researchers, while undoubtedly experts in their fields, won’t likely have the same appreciation for projects initiated by small businesses with goals related to product development.
As others have noted in the comments here, it would be nice if there were more attention paid to reviewers of SBIR applications.
One thing NIH needs to do is consolidate products into pdf files. For example, the Enhancing Peer Review Project is complete – a master document should be on the main webpage for that program, for download. While some (not all) information on NIH programs, metrics, etc is around here, there’s no need for people to spend a long time finding and downloading it.
We all know that the majority of people simply won’t do that – most people just don’t have the time to justify that. The end result is that most people don’t know the full breadth of what NIH has done or why.
This is not to defend NIH or criticize NIH.
I’m sure some complaints about NIH processes are without merit and the issues have probably been addressed in some of these reports. I’m also sure most people won’t be aware of that. However, there are probably things done that deserve extra scrutiny and feedback.
To be honest, virtually all program webpages should have significant roll-ups of information available so people don’t have to hunt around and point&click constantly. While I realize this is a potential burden on NIH, administratively, to have someone take such tasks – it would appear worthwhile – NIH is the premier funding apparatus in the world. Evaluating and improving everything done should be a continuous, Tier 1 effort. Some added efficiency in the communication / dialog between NIH and those in the research community may help all concerned.
What are the criteria for being a chair of a study section? Are they documented somewhere? Thank you!
Please see these resources: Study Section Chair Selection and Training, and How Scientists are Selected for Chartered Review.