Cross-posted on the Review Matters blog from the Center for Scientific Review.
NIH has issued a request for information (RFI) seeking feedback on revising and simplifying the peer review framework for research project grant applications. The goal of this effort is to facilitate the mission of scientific peer review – identification of the strongest, highest-impact research. The proposed changes will allow peer reviewers to focus on scientific merit by evaluating 1) the scientific impact, research rigor, and feasibility of the proposed research without the distraction of administrative questions and 2) whether or not appropriate expertise and resources are available to conduct the research, thus mitigating the undue influence of the reputation of the institution or investigator.
Currently, applications for research project grants (RPGs, such as R01s, R03s, R15s, R21s, R34s) are evaluated based on five scored criteria: Significance, Investigators, Innovation, Approach, and Environment (derived from NIH peer review regulations 42 C.F.R. Part 52h.8; see Definitions of Criteria and Considerations for Research Project Grant Critiques for more detail) and a number of additional review criteria such as Human Subject Protections.
NIH gathered input from the community to identify potential revisions to the review framework. Given longstanding and often-heard concerns from diverse groups, CSR decided to form two working groups to the CSR Advisory Council—one on non-clinical trials and one on clinical trials. To inform these groups, CSR published a Review Matters blog, which was cross-posted on the Office of Extramural Research blog, Open Mike. The blog received more than 9,000 views by unique individuals and over 400 comments. Interim recommendations were presented to the CSR Advisory Council in a public forum (March 2020 video, slides; March 2021 video, slides). Final recommendations from the CSRAC (report) were considered by the major extramural committees of the NIH that included leadership from across NIH institutes and centers. Additional background information can be found here. This process produced many modifications and the final proposal presented below. Discussions are underway to incorporate consideration of a Plan for Enhancing Diverse Perspectives (PEDP) and rigorous review of clinical trials RPGs (~10% of RPGs are clinical trials) within the proposed framework.
Simplified Review Criteria
NIH proposes to reorganize the five review criteria into three factors, with Factors 1 and 2 receiving a numerical score. Reviewers will be instructed to consider all three factors (Factors 1, 2 and 3) in arriving at their Overall Impact Score (scored 1-9), reflecting the overall scientific and technical merit of the application.
- Factor 1: Importance of the Research (Significance, Innovation), numerical score (1-9)
- Factor 2: Rigor and Feasibility (Approach), numerical score (1-9)
- Factor 3: Expertise and Resources (Investigator, Environment), assessed and considered in the Overall Impact Score, but not individually scored
Within Factor 3 (Expertise and Resources), Investigator and Environment will be assessed in the context of the research proposed. Investigator(s) will be rated as “fully capable” or “additional expertise/capability needed”. Environment will be rated as “appropriate” or “additional resources needed.” If a need for additional expertise or resources is identified, written justification must be provided. Detailed descriptions of the three factors can be found here.
The following “Additional Review Criteria” will remain largely unchanged, not scored individually but continue to affect the Overall Impact Score. A drop-down option of “Appropriate or Concerns” will be provided, with a justification narrative required for concerns.
- Human Subject Protections
- Inclusion of Women, Minorities, and Individuals Across the Lifespan
- Vertebrate Animal Protections
- Renewal – Evaluation of productivity during the previous project period will remain unchanged and will continue to affect the Overall Impact score
Other minor changes to the peer review framework are proposed that will reduce time effort required by reviewers. “Additional Review Considerations” are items that peer review is currently tasked with evaluating, but which are not scored, and which do not affect the Overall Impact score. NIH proposes to move select Additional Review Considerations out of the initial peer review process. Under proposed framework, the remaining Additional Review Considerations are:
- Authentication of Key Biological and/or Chemical Resources:
- Drop-down option with a choice of “Appropriate” with no comments required, or as “Concerns,” which must be briefly described.
- Budget and Period of Support:
- Rated as “Appropriate,” “Excessive,” or “Inadequate”; the latter two ratings require a brief account of concerns.
Through the RFI, NIH continues to seek further public input on the proposed changes before moving forward with implementation. The RFI will be open for a 90-day period, until March 10, 2023. We look forward to your comments.