Guest post by Bruce Reed, Deputy Director of the NIH Center for Scientific Review, originally released on the Review Matters blog
Over the past several years we have heard consistent concerns about the complexity of review criteria and administrative load of peer review. CSR shares the concern that the current set of standards has the unintended consequence of dividing reviewer attention among too many questions, thus reducing focus on scientific merit and increasing reviewer burden. Each element was intended make review better, but we worry that the cumulative whole may in fact distract from the main goal of review — to get input from experts on the scientific and technical merit of the proposed work.
To address these concerns, CSR has convened a working group of our advisory council, charged with recommending changes to research project grant review criteria that will improve review outcomes and reduce reviewer burden. The group is co-chaired by Tonya Palermo and me, and includes some of our council members, other members of the scientific community, and the NIH Review Policy Officer from the Office of Extramural Research.
We would like to hear your thoughts on the issue. How might review criteria be modified to obtain the best evaluations of scientific merit? You can provide feedback directly to me at firstname.lastname@example.org, to email@example.com, or to any member of the working group. Before you fire off that email, though, read on.
First, be aware that current criteria derive from multiple regulations; changes that conform to them well are more feasible than those that don’t. The Code of Federal Regulations (42 C.F.R. Part 52h.8) requires that research project applications be evaluated based on significance, investigators, innovation, approach, and environment. Protections for humans, animals, and the environment, adequacy of inclusion plans, and budget must be evaluated. The “21st Century Cures” Act (Public Law 114-255) requires attention to rigor and reproducibility and aspects of clinical trials. That said, there is room for improved implementation.
Second, consider how simplified criteria that might also help address some of the issues below:
- Multiple studies show that reviewer ratings of Approach carry the most (perhaps too much) weight in determining overall impact scores. Yet, aspects of rigor and reproducibility are too often inadequately evaluated. Can better criteria help?
- Review is often criticized as being risk-averse, as too conservative. If you agree, how might revised criteria help?
- How can criteria be defined to give the applications of all investigators, regardless of their race, ethnicity, gender, career stage, or setting, fair hearing on a level playing field?
Third, focus on the criteria for R01s. The criteria for training grants (F’s, K’s, T’s) and for SBIR/STTR grants are different. Addressing criteria for R01s would be a great start.
Finally, please be patient. Getting from good ideas to a revised set of criteria is a complex, multi-level process that will include NIH’s Office of Extramural Research, eRA, NIH Institutes and Centers, Office of the General Counsel, and other relevant stakeholders. This is a preliminary effort to get your input on what changes we should think about. Were we to propose regulatory changes, we would ask for additional public input. We are starting a conversation. Share your ideas.
Members of the CSR Advisory Council Working Group
Palermo, Tonya M., Ph.D.
Professor, Department of Anesthesiology and Pain Medicine
Principal Investigator and Associate Director, Center for Child Health, Behavior and Development
Seattle Children’s Research Institute
Reed, Bruce, Ph.D.
Center for Scientific Review
National Institutes of Health
Amero, Sally, Ph.D.
Review Policy Officer
Office of Extramural Research
National Institutes of Health
Corbett, Kevin D., Ph.D.
Department of Cellular and Molecular Medicine
University of California, San Diego, School of Medicine
Gao, Jinming, Ph.D.
Professor of Oncology, Pharmacology, and Otolaryngology
Co-Leader, Cell Stress and Nanomedicine Program
Simmons Comprehensive Cancer Center
UT Southwestern Medical Center
George, Alfred L., M.D.
Chair, Department of Pharmacology
Director, Center for Pharmacogenomics
Magerstadt Professor of Pharmacology
Hurd, Yasmin L., Ph.D.
Professor, Department of Psychiatry,
Neuroscience, Pharmacological Sciences
Director, Addiction Institute of Mount Sinai
Icahn School of Medicine at Mount Sinai
Janelsins-Benton, Michelle C., Ph.D.
Departments of Surgery, Neuroscience, and Radiation Oncology
University of Rochester, Medical Center
King-Casas, Brooks, Ph.D.
Fralin Biomedical Research Institute
Department of Psychology, Virginia Tech
Kroetz, Deanna L., Ph.D.
Professor, Department of Bioengineering and Therapeutic Sciences
Director, Pharmaceutical Sciences and Pharmacogenomics Graduate Program
University of California, San Francisco
López, José A., M.D.
Professor of Medicine, Hematology
Member of Bloodworks NW Research Institute
Adjunct Professor, Biochemistry, Mechanical Engineering, and Pathology
University of Washington, School of Medicine