3 Comments
NIH is implementing multiple changes that will impact the preparation and peer review of most grant applications submitted to NIH for due dates on or after January 25, 2025. Although each of these initiatives have specific goals, they are all meant to simplify, clarify, or ensure greater fairness.
We just released a guide notice (NOT-OD-24-084), what we are referring to as an “uber” notice, that provides an overview of each change to help the community contextualize them as details are released over the next few months. We developed this video (25 min) to provide you with an overview of the following changes:
Simplified review criteria for most research project grants. We announced this initiative this past October, held an informational webinar in November, and are also providing additional information on what this means for funding opportunities in a separate blog today, and in an upcoming April 17 webinar.
Revisions to the fellowship application and review process. You may remember the request for information NIH published last year. In the next few weeks, NIH will be releasing details of the resulting fellowship application and review changes that we are implementing. The changes are intended to: (1) better focus reviewer attention on the fellowship candidate’s preparedness and potential, the research training plan, and the sponsor/sponsoring institutional commitment to the candidate; (2) ensure a broad range of candidates and research training contexts can be recognized as meritorious; and (3) reduce bias in review by emphasizing the commitment to the candidate without undue consideration of sponsor and institutional reputation. NIH will be hosting a webinar to walk the community through the fellowship changes on September 19, 2024.
Updates to reference letter guidance. NIH is updating the instructions for reference letters to provide more structure for reviewers. Resulting letters will better assist reviewers in outlining the candidate’s strengths, weaknesses, and potential to pursue a productive career in biomedical science. The updated instructions will be posted later this fall on the Reference Letter web page.
Updates to NRSA training grant applications. Later this spring NIH will publish an NIH Guide notice announcing changes to training grant applications that, at a high level, include:
- Updating the NRSA Data Tables to reduce applicant and reviewer burden;
- Including the Training in the Responsible Conduct of Research and the Recruitment Plan to Enhance Diversity as items that contribute to the overall impact score; and
- Enhancing research training programs by further defining expectations for mentor training and clarifying positive outcomes related to preparing trainees for the breadth of research and related careers relevant to the NIH mission.
NIH will be hosting a webinar to walk the community through the training grant changes on June 5, 2024.
Updated application forms (FORMS-I). NIH will release updated application forms to support many of the changes coming in 2025. The new forms must be used for application due dates on or after January 25, 2025. Applications submitted for due dates prior to January 25 should continue to use FORMS-H. We encourage you to review the FORMS-I Guide Notice (NOT-OD-24-086) to learn more about the timing and availability of the new forms.
Common Forms for biographical sketch and current and pending (other) support. To be adopted in 2025 for all applications and Research Performance Progress Reports (RPPRs) submitted to NIH, the Common Forms represent a collaborative effort between Federal research agencies to ensure standard disclosure requirements as outlined in the National Security Presidential Memorandum – 33. In addition to providing greater standardization across federal agencies, the Common Forms provide clarity regarding disclosure requirements (e.g., who discloses what, relevant limitations and exclusions), disclosure process (e.g., updates, corrections, certification, and provision of supporting documentation), and expected degree of cross-agency uniformity (excerpt from Guidance for Implementing National Security Presidential Memorandum 33). We will be providing details on our implementation and timing in the next few months.
We recognize these are a lot of changes all at once. To help the community, we have developed a centralized webpage on Changes Coming in January 2025 that we will continue to update as details of each change and resources are developed and released. We will also be updating the “uber” notice with subsequent notices related to these changes.
We encourage you to learn more about these changes, and to take advantage of the many resources we are developing. We will be discussing changes here on Open Mike as details are released.
I know their heart is in the right place, but I worry that Noni, Mike and their team are risking increasing (!) the influence of PI and institution reputation and other biases unrelated to the quality of the research on Impact scores by eliminating Investigator and Environment as explicitly scored factors.
I not-so-fondly remember my first few study sections when I would regularly hear how the investigator’s track record offset concerns about flaws in the experimental plan. For anyone who recently did their study section training, if you thought R2 insisting on a “1” while saying that they acknowledged major weaknesses in the Approach raised by R1 but these didn’t matter because the PI (Dr. Fauci!) is so good was cartoonish, actually, that scene would really happen. Worse than that, many reviewers considered that a reasonable argument. In the training video, one solution offered is that the Chair should step in. Indeed, Chairs can say that Investigator and Approach are distinct criteria and should be scored independently, with Impact score coming from a balanced consideration of all the elements. In the old days, there was just the one score and little guidance about what factored in. If anything, bias seemed encouraged and Chairs would simply let it go.
A concern is that the new simplified criteria eliminating the PI and Institution sub-scores will actually encourage reviewers to express their bias by skewing their evaluation of Approach as in, “Given the PI and resources at their disposal, the lack of preliminary studies, experimental detail or statistical justification noted by R1 don’t concern me as more than very minor weaknesses. Thus, I am staying at a 2.” I am not sure what a Chair is supposed to do other than rolling their eyes.
I hope that if things start going in the wrong direction, CSR will reconsider and restore the Investigator and Institution scores, empowering Chairs to again make that distinction clear.
I fully agree with Steve. However, going back to the current system will not solve the major issue of “the influence of PI and institution reputation and other biases unrelated to the quality of the research on Impact scores”. IMHO, the only way forward here is to blind the proposals. I already hear someone saying “that will never work, it’s not feasible”. I think this is an easy task with current information technologies. And just like applicants currently wonder without certainty who among the panelists had their proposals, reviewers will not be certain about the PI behind a blinded proposal and hopefully will pay more attention to the science.
In reference to the awarding of RO-1 research grants: In my long (55years) association with a major private research university, I can say that the major
prediction of research grant success is publication track record, as a key author, in major research journals. This should not include review articles. These peer-reviewed papers should represent definite advances in the field, not side-ways moves; they are truly the coin of our business as researchers. Of the next factors, most important are (a) research preparedness (what is the applicant’s history and where and with whom was it done?) (b) the institution where the research will be done. The latter is expected to be one of the larger elite research institutions, rather than the smaller less-known institutions. There should be special
programs specifically geared toward funding the latter. Even the grant research proposal (which, indeed, should be interesting and important) should be secondary to key published research papers. For more senior applicants, another key factor is the researchers trained by the applicant and where they are currently working.