Update on Improving Fellowship Review: A Request for Information

Posted

This blog has been co-authored with Noni Byrnes, Director, Center for Scientific Review, NIH. Originally posted on the Review Matters blog.

Mike Lauer headshot
Mike Lauer, Deputy Director for Extramural Research, NIH
Photo of Noni Byrnes
Noni Byrnes, Ph.D., Director of NIH Center for Scientific Review

NIH is recommending changes to the peer review of Ruth L. Kirschstein National Research Service Award (NRSA) fellowship applications by restructuring the review criteria and modifying some sections of the PHS Fellowship Supplemental Form that are specific to NRSAs. The goal of this effort is to facilitate the mission of NRSA fellowship peer review – to identify the most promising trainees and the excellent, individualized training programs that will help them become the outstanding scientists of the next generation. The proposed changes will 1) allow peer reviewers to better evaluate the applicant’s potential and the quality of the scientific training plan without undue influence of the sponsor’s or institution’s reputation; and 2) ensure that the information provided in the application is aligned with the restructured criteria and targeted to the fellowship candidate’s specific training needs.  The RFI requests public input on this proposal. To comment, go to the RFI, which contains additional background and links to submit your thoughts.  Most of that background follows:

The first stage of NIH peer review serves to provide expert advice to NIH on assessment of the likelihood that the fellowship will enhance the candidate’s potential for, and commitment to, a productive independent scientific research career in a health-related field. The criteria for the review of NRSA fellowship applications derive from the NRSA regulation 42 CFR 66.106, with four pertinent factors: (1) the scientific, technical, or educational merit of the particular proposal; (2) the availability of resources and facilities to carry it out; (3) the qualifications and experience of the applicant; and (4) the need for personnel in the subject area of the proposed research or training.NIH currently organizes these criteria into the following categorical labels: Applicant, Sponsors and Collaborators, Research Training Plan, Training Potential, and Institutional Environment and Commitment. By NIH policy, peer reviewers are also required to evaluate Training in the Responsible Conduct of Research, Biohazards, Resubmissions, Foreign Organizations, Select Agents, Resource Sharing Plans, Budget and Period of Support, and Authentication of Key Biological and/or Chemical Resources.

NIH gathered input from many sources in forming this proposal. Unsolicited comments over a period of years, reflecting persistent concerns that the NRSA fellowship review process disadvantages some highly-qualified, promising applicants led the Center for Scientific Review (CSR) to form a working group to the CSR Advisory Council. To inform that group, CSR published a Review Matters blog which was cross-posted on the Office of Extramural Research blog, Open Mike. The blog received more than 1,500 views by unique individuals and numerous comments. The working group presented interim report to the CSR Advisory Council, which adopted the recommendations, at public CSR Advisory Council meetings (March 2022 video, slides; September 2022 videoslides). Final recommendations from the CSR Advisory Council (report) were considered by the CSR Director, as well as major NIH extramural program, review, and policy committees that included leadership from across NIH. Recommendations were presented to the NIH Advisory Committee to the Director in December 2022 (video, slides).

Improving NRSA Fellowship Review

Recommendation 1: Revise the Criteria Used to Evaluate NRSA Fellowship Applications

An Overall Impact Score (scored 1-9) will reflect the scientific and educational merit of the proposal and an assessment of the likelihood that the fellowship will enhance the applicant’s potential for, and commitment to, an independent, productive research career in a health-related field. Reviewers will take into account their assessments of the three criteria in determining an Overall Impact Score. Each of the three criteria will receive an individual score. The “additional review criteria” below are unchanged, will not receive individual scores, but will be considered in arriving at the Overall Impact Score. Two “additional review considerations”, also unchanged, will be evaluated but have no effect on the Overall Impact Score. Review the full text of the proposed changes.

Review Criteria

I. Scientific Potential, Fellowship Goals, and Preparedness of the Applicant

  • criterion emphasizes their accomplishments in the context of their stage of training and the scientific opportunities they have had, as well as other factors that bear on their potential to succeed, such as determination, persistence, and creativity. (scored 1-9)

II. Science and Scientific Resources

  • criterion emphasizes the extent to which needed technical, scientific, and clinical resources are specified and are realistically available to the applicant, and whether the scientific expertise of the mentorship team is appropriate for the proposed science and that the role of each mentor is clearly defined. (scored 1-9)

III. Training Plan and Training Resources

  • criteria emphasizes whether the necessary training resources are well-specified and available, specifically the practical availability of resources, and to include an evaluation of the training philosophy of the sponsor, their approach to training, time commitments, and their accessibility. (scored 1-9)

Additional Review Criteria (not scored, but affecting Overall Impact; no changes proposed, see current language):

• Protections for Human Subjects
• Inclusion of Women, Minorities, and Individuals Across the Lifespan
• Vertebrate Animals
• Biohazards
• Resubmission

Each of the Additional Criteria except the last will be rated as “Appropriate”, with no comments required, or as “Concerns”, which must be briefly justified. Resubmission will be given brief written evaluations.

            Additional Review Considerations (not scored and having no effect on Overall Impact) no changes proposed, see current language):

            • Training in the Responsible Conduct of Research
            • Authentication of Key Biological and/or Chemical Resources
            • Budget and Period of Support
            • Applications from Foreign Organizations
            • Select Agents
            • Resource Sharing Plans

            Recommendation 2: Revise the PHS Fellowship Supplemental Form

            Changes to the application instructions are needed to better align the information applicants provide with the revised review criteria. The PHS Fellowship Supplemental Form currently includes the following four sections: Fellowship Applicant; Research Training Plan; Sponsor(s), Collaborator(s), and Consultant(s); Institutional Environment and Commitment to Training. The NIH proposes to revise the Fellowship Applicant section and the Sponsor(s), Collaborator(s) and Consultant(s) section. NIH also proposes to change the instruction for Letters of Reference. An additional proposed change would allow an optional Statement of Special Circumstances from the fellowship applicant. Review the full text of the proposed changes.

            Through the RFI, NIH continues to seek public comment on the proposed changes before moving forward with implementation. The RFI will be open for a 60-day period, until June 23, 2023. We look forward to your comments.

            4 Comments

            1. IMHO, there is currently too little emphasis on the quality of the applicant and the institution and too much on quasi-irrelevant minor concerns expressed by the reviewers. Often, the reviewers seem to lack the expertise and experience to appreciate the significance of the proposed research or the rigor of the proposed methods, and so make their rankings on issues that ultimately reflect misunderstandings or trivial concerns.

              1. I was just about to write exactly this. Having served on multiple fellowship study sections and mentored many individual postdocs, the key shortcoming in the evaluation of the fellowship applications is lack of expertise – panel members who do not know the science, the relevant publications, the sponsor and institution. CSR consistently recruits panel members who are not qualified due to lack of knowledge in the relevant areas. More important, CSR administrators assign applications to individuals who even state that they are not qualified. The CSR administrators cajole and plead until the individual agrees.

            2. Well this was 18 years too late to help. There is nothing more disheartening than to get reviews back saying – we like the applicant, the project is fine but we hate the school, program and mentor. Even with a co-mentor and improvements to the science they still did not like my main mentor or my program. I am glad they are changing it. The science was sound, my mentor ended up being great.

            3. Decades of empirical research in learning and career development confirm that at minimum two things make for success: (a) the talent and motivation of the trainee, and (b) the resources and mentoring available in the trainee’s work setting. This is known to be a synergistic effect. The very most talented and motivated trainee will be stunted by a training environment that is poor in resources, lacking opportunities to network with highly successful scientists, and lacking mentoring by successful scientists. The reverse is also true, trainees who come from a deprived background can blossom if placed in a rich training setting. Peer Reviewing only the trainee, and ignoring the training setting, is foolhardy. Efforts are better invested in improving all training settings (i.e., more formative training grants to enhance the weaker settings), not messing about with peer review.

            Before submitting your comment, please review our blog comment policies.

            Leave a Reply

            Your email address will not be published. Required fields are marked *