28 Comments
This blog has been co-authored with Noni Byrnes, Director, Center for Scientific Review, NIH. Cross-posted on the Review Matters blog.
As we have discussed in previous blogs, NIH has heard concerns from the extramural community about the complexity of the peer review process for research project grants (RPGs) and the increasing responsibilities of peer reviewers in policy compliance. NIH has also heard concerns about the potential for reputational bias to affect peer review outcomes. After careful input gathering, development, and discussion, NIH is pleased to announce that a simplified review framework will be implemented for grant receipt deadlines of January 25, 2025 and beyond.
The simplified framework is expected to better focus peer reviewers on the key questions needed to assess the scientific and technical merit of proposed research projects: “Can and should the proposed research project be conducted?” To achieve this, the five current review criteria (defined as Significance, Innovation, Approach, Investigator, and Environment; derived from NIH peer review regulations 42 C.F.R. Part 52h.8) are being reorganized into three broader factors to help reviewers focus on crucial questions that determine scientific merit. Reviewers will consider all three factors in determining the overall impact score, which reflects their overall assessment of the likely impact of the proposed research.
- Factor 1: Importance of the Research (Significance and Innovation), factor score 1-9
- Factor 2: Rigor and Feasibility (Approach), factor score 1-9,
- Factor 3: Expertise and Resources (Investigator and Environment), either rated as sufficient for the proposed research or not (in which case reviewers must provide an explanation)
A significant concern being addressed in the simplified framework is the potential for general scientific reputation to have an undue influence on application review. By changing the evaluation of Investigator and Environment to a binary decision of sufficient or not in the context of the proposed research, Factor 3 aims to help mitigate this potential biasing influence.
Another concern addressed in the new framework is the reliance on peer reviewers to assess policy compliance. Relying on peer reviewers for these tasks has the potential to distract them from their chief goal of assessing the scientific and technical merit of an application. To reduce reviewer burden, NIH staff will assume administrative responsibilities related to the Additional Review Considerations of Applications from Foreign Organizations, Select Agents, and Resource Sharing Plans.
The results of a request for information underscored the need for resources and guidance for investigators, reviewers, and NIH staff. We therefore developed a new Simplified Review Framework webpage, which is now live and will serve as a central repository of information on this initiative. NIH is developing an integrated set of training events and resources to communicate the changes to applicants, reviewers, and NIH staff that will be rolled out over the next year. This support will begin with a webinar on November 3rd, 2023 to provide the public with an overview of the new framework and what to expect over the upcoming year in preparation for implementation. You can learn more and register for the webinar here.
We expect the simplified review framework to have minimal impact on how applications are written. The intent is simply to focus reviewers on the fundamental questions that we have always asked them to address in reviewing grant applications for their scientific and technical merit, while minimizing the impact of reputational bias. We will keep you updated through our webpage, notices in the NIH Guide, and through the Review Matters and Open Mike blogs. We look forward to working together with the community to continue to improve peer review.
Sounds good in principle, but doesn’t past success of the PI(s) correlate highly with how likey the grant is to succeed? How can this not be a factor?
This is great news. I am particularly pleased that we no longer have to score institutional environment numerically – to me, this was a throwback to the days before email and the internet made broad collaboration possible. But I do worry that this will skew the review to prioritize problems with approach – to me, a proposal of high significance with some problems in the Approach that could be fixed just based on reviewers’ comments should be prioritized over a proposal of lower significance to human health but with fewer experimental issues. We spend too much time funding perfect science and not necessarily important science.
I agree wholeheartedly with this. I have both listened and participated in many study section discussions that nitpick issues such that great research is not awarded in favor of projects that get lucky by having three reviewers who do not micromanage their application.
They aren’t applying this to small business grants which just lets big small businesses and people with fancy degrees from fancy universities suck up all the grants.
I hope the concern about undue influence of a researcher’s reputation does not mean that someone having successfully carried out previous grants is not given credit for it . What other profession does not take in consideration someone’s previous success when deciding whether their future efforts should be supported ??
What other profession has established itself such that ‘success’ turns into an impenetrable armor that redirects resources away from young scientists? As mentioned above, many a career were killed by micromanaging reviewers and their personality disorders/lack of interdisciplinary training. Application preparation is facilitated at the institutional and near-field levels for those who had been particularly ‘successful’ in obtain funding before. The funding sources are protected. Prior success can compete but not trump new opportunities.
To date, the review process at NIH has largely facilitated the ‘rich get richer’ flows of funding. This is unlikely to change even with these superficial ‘patches’ that NIH is implementing instead of fixing the system. Proof is in the ‘pudding’ – NIH is claiming this should have minimal impact on the preparation of applications, which just serves to show that is the level of impact it will produce in addressing the issues it claims it will.
I had NIH grants for ~50 years and was on several Study Sections. I still do not entirely understand the criteria for making grant awards. Much depends on the number of available dollars and the gullibility of reviewers. There are always elements of salesmanship, reputation (of applicant and institution) and what are usually termed ‘unmet needs’. No doubt there are applicants that will produce something useful even if they write imperfect proposals. Perhaps the most likely conclusion is that criteria evolve with time and the current model is better than most others. Somehow, the scientific literature (mainly papers from grant-supported research) still contains a significant percentage of poorly reasoned reports.
Brilliant idea!
After an earlier iteration of the perpetual revolution (overhaul of the study section process), a terrific SRO asked me what I thought of the new system. Atypically I offered a tepid and diplomatic appraisal, and was surprised that a CSR loyalist (the SRO) zinged back “Isn’t it really just rearranging deck chairs while the Titanic is sinking? The core problem is too many applicants chasing a finite pool of money”
The final years on study section were depressing in how poor the expertise and review quality were already for reasons * not * addressed by these changes and made worse by other parts of the CSR perpetual revolutions. Yes, taking away some of the ancillary crud will help a bit at the margins in reducing the dump on reviewers (although, strictly speaking, the Foreign Organization part needs expert input, most peers never actually read what that was supposed to be about). As noted by others, it is ludicrous for NIH to pretend it seeks the best applications to fund and at the same time that the capabilities of the PI/PD and personnel do not have an impact on feasibility or the likelihood of a sustained impact on the field. In any case the strongest discriminant before was the criterion score of Approach, and – as noted by an earlier reply – that is unlikely to be affected, and reasonable reviewers will be able to factor the track record of the team into how they rate Approach (can it be done?).
If bias caused by reputation is a concern, why not make the review process anonymous? Particularly that evaluating the research environment is no longer part of the peer review process, why should I as a reviewer know who the applicants are? Even with the best intentions, if I review a proposal submitted by somebody really famous, it is difficult no to be influenced.
Please see our FAQ Why is this Simplified Review Framework being implemented rather than a blinded review?
This is yet another disastrous move by CSR, just the latest of many in recent years under its current, awful leadership. CSR’s behavioral pattern is to respond to lobbying from people whose grant applications do not succeed. Instead of looking into the mirror these researchers “blame the system” that, for some reason or other, does not see their brilliance in the same way they do. The goal of these lobbyists is to “spread the wealth around”. One consistent theme has been to attack experienced, successful researchers with a track record of accomplishment. This matter because, fundamentally, the role of the NIH is to perform medical/scientific research that helps the nation’s health; that’s the very reason NIH exists. To succeed in its mission, the NIH needs to support the most productive and effective researchers, as those are the people who make most of the advances that matter. Now, excellence and accomplishments (the track record of an applicant) are to be ignored? Let’s use a sports analogy to explain what this means. What would happen if a basketball or baseball team was only allowed to hire new players without taking into account their career statistics? How would that help the team win the championship? Players who can consistently nail 3-pointers or strike out hitters are winners. And so are scientists whose publication record and, yes, “reputation” show that they can successfully perform the research the NIH is tasked to provide to the American public. But, no, what we now see is how the CSR’s efforts to drive US science further towards mediocrity continue unabated…
Using your analogy with basketball, you would have to become age-biased at some point to avoid the decline of a team, so this approach is also not perfect. Usually, experienced researchers already have an advantage in justifying their grants, as they have years of feedback and accomplishments. There is no specific need to request an additional formal acknowledgment of success.
“Projects, not people” is the mantra for extramural research. Unfortunately, projects do not execute themselves.
Good idea not to score Environment as most applications require a basic biomedical lab that is available in most institutions, but it does not make sense to ignore qualification of the Investigator who will conduct the work.
To eliminate the bias, why not allow the reviewers to look at the proposal? If the reviewers find it to be a good proposal, then NIH can request for additional information such as PI’s background, research environment, etc.
This change will make little difference because the overall score will be the one that determines funding. Unless each category score carries 50% (for example), and the overall score is the average of the two, these criteria-based scoring have no real value (as is the case with the current scoring). Even if the 3rd category is unscored, reputation (or knowing an applicant) will creep into the scoring–a reviewer will adjust his or her overall or category scores. The scoring is basically perception-based and unless CSR makes real changes to the process (instead of window-dressing changes such as this one) and the SROs make a real effort to select high-quality reviewers (neither will happen in our lifetime, if ever), it will be business as usual. The way grants are reviewed and funded has huge implications for studies that are reproducible and long-standing and hope some real changes are made to the process.
Still dancing around the main issue with peer review and not addressing the real problem. Pilots anonymizing applications have been very successful, but the NIH refuses to implement it on a wider scale and achieve some degree of fairness. Our elections are not rigged, but the NIH study sections sure are!
Please see our FAQ Why is this Simplified Review Framework being implemented rather than a blinded review?
I appreciate having significance and innovation combined – they are both important, but considering them as a whole (“importance of the research”) makes sense to me. Further, the comments about investigators above seems to be addressable as part of the rigor and feasibility criteria. If a new investigator without a strong track record clearly addresses all the issue of rigor and feasibility to achieve their stated aims, why not give them a chance (i.e., with a strong score)? Likely this evaluation will consider whether collaborative efforts from other experts are proposed to ensure that indeed the study is feasible as proposed. The clear advantage seasoned investigators have is that they should know the many details that will convince reviewers they can successfully complete the study in a rigorous and feasible manner. They should clearly have an advantage to begin with as they are not writing “from scratch” in many cases and have a much clearer vision of what makes a project successful and what problems might arise (based on past experience!). This new review criteria seems to prioritize communicating clearly as opposed to having a great bio sketch and being given the benefit of the doubt even if the proposal is mediocre.
I agree with David Kessel’s reply. I also have NIH funding for more than 4 decades and served on numerous study sections. NIH should explicitly define significance and innovation, many investgators struggle with these two criteria. Most grants get low scores due to the problems in approaches raised by the reviewers. Particularly the influence of reviewers who are more dominant in the study sections swing the grant scores. Study section SRO’s is very important to run the study sections to cut the nonsense and nit-picking points raised by reviewers. Finally, the success of a grant depends on the payline. I really dislike “the grant is approved but not funded. What that mean. I recently reviewed Human Frontier Science Program. Their structure of review process is much more trim lined. Evey few years the changes in peer review sytems are nothing but more of “gloss over the wound.
Based on my own experience from a certain recent study section, applications are (alas, way too often) assigned to reviewers who lack relevant expertise. This initiates the cascade: (1) Recusals are highly unusual, so the reviewer accepts the assignment. (2) Reviewers feel squeezed between the need/deadlines to write the critiques, and their own inability to focus on the applications’ merits. So, (3) instead, they focus on some substitute criteria, or invent some nonsensical pseudo-science (it’s not just nitpicking). (4) The PI receives a Summary Statement that reads like a missing chapter from ‘Alice in the Wonderland’, (5) The PI, especially an inexperienced one, tries to rationalize this, and (unwilling to commit a heresy of not believing in ‘meritorious peer review’) comes up with some ‘bias’ explanations. (6) The PI takes seriously the ‘differences of scientific opinions’ and ‘amend and resubmit’ advice (even though discussing with such reviewers makes as much sense as discussing science with the March Hare), (7) Everybody shifts by one place at the table (A0, A1, A2), and the Mad Tea Party goes on.
I am posting the same comment on LinkedIn; opinions are welcome.
Good idea! Another thing, make sure competent reviewers are assigned. The simplest way: a quick search of relevant recent publications, contact and try to recruit the corresponding authors. Biases (IMO mostly the result of a compulsion on the part of incompetent reviewers to “write something”) will magically disappear.
Factor 3 will be binary and “considered in overall impact; no individual score”. Can we conclude that insufficient expertise and resources will be a kill switch for any given grant? Far from de-emphasizing this factor, the change will make it paramount.
I think that will be a very interesting question. Along with that, how many reviewers have to answer “no” to the expertise and resources question for the proposal to be laid aside?
“NIH staff will assume administrative responsibilities related to the Additional Review Considerations of Applications from Foreign Organizations, Select Agents”
wise considering the current climate and eroding trust in the NIH by the public? Speaking to people in my community (non-scientists) I explain how seriously these items are considered by 20 or so diverse scientists at study section. This usually puts them at ease and convinces them decisions are not made lightly and the issues are rigorously considered. Turning this over to NIH staff (not degrading their talents and knowledge) may be heading in the opposite direction.
Minimal impact on the review process. One has to be a friend of one of the study section members to get a fundable score, period.
No one openly discusses this. But this is the truth.
I believe that adding an AI – reviewer to ensure certain criteria are achieved, will be helpful.
At least novelty. Some PIs study one molecule for 30+ years and still get funds for this old thing.
This is a start but it doesn’t go far enough to stop the favoritism of certain institutions and individuals. Another improvement would be to prevent the reviewers knowing the identity of the applicant, the applicants educational background or the institution the applicant represents.