In March 2017, we wrote about federal funders’ policies on interim research products, including preprints. We encouraged applicants and awardees include citations to preprints in their grant applications and progress reports. Some of your feedback pointed to the potential impact of this new policy on the peer review process.
Some issues will take a while to explore as preprints become more prevalent. But some we can dig into immediately. For example, how do references cited in an application impact review? To start to address this question, we considered another one as well: do peer reviewers look at references – either those cited by applicants or others – while evaluating an application? We had heard anecdotes, ranging from “Yes, I always do,” to “No, I don’t need to,’ but we didn’t have data one way or the other. And if reviewers do check references, how does it impact their understanding and scoring of an application?
So, together with colleagues from the NIH Center for Scientific Review (CSR), we reached out to 1,000 randomly selected CSR reviewers who handled applications for the January 1, 2018 Council Round. There were an equal number of chartered (i.e. permanent) and temporary reviewers solicited to participate (n=500 each) over a three week period from November 16 to December 8, 2017.
Our survey focused on the last grant where they served as primary reviewer. Specifically, we asked if they looked up any references that were either included in the application (i.e. internal references), and if they also looked up any that were not included in the application (i.e. external references). Depending on their answers to each of these questions, we also proceeded to ask certain respondents follow-up questions to better understand their initial feedback. We felt it would be interesting to know, for example, how reading the paper or abstract impacted their understanding of the application and their score.
We received 615 responses (62% of total), including 306 chartered members and 309 temporary members. Figure 1 shows the responses related to if they looked up references, either internal or external to the application. Most reviewers answered yes – particularly for internal references.
Figure 2 goes a bit deeper – as a secondary question, we asked whether the references affected reviewers’ understanding of the applications. The clear majority said yes. Figure 4, shows that most reviewers (~85%) found the references improved their understanding.
Next, we learned that of those reviewers that checked references, about 2/3 reported that the references affected their scoring for the application (Figure 3). References reviewers found on their own (external references) seemed slightly more influential. Figure 4 shows references could impact the score in either direction. References cited in the application were slightly more likely to improve scores than worsen them, and external references were slightly more likely to make scores worse than improve them.
Nearly half of the respondents even provided additional comments for us to consider. Here is a sampling of their thoughts:
- “References are of immense value.”
- “I look up references to judge the quality of the [principal investigator’s] work in relation to the rest of the field, to learn about the field in general, and to delve into specific questions that might be key to evaluation of the application. This could result in changes to the score in either direction.”
- “References are useful and sometimes critical.”
This experience was very enlightening. We were pleased to learn that most reviewers do look up references as part of their work in the peer review process, but preprints, at least for now, are too rarely cited in applications to have a clear impact. Further, both chartered and temporary reviewers shared similar perspectives on looking up references, which they noted often affects their understanding of the applications and resulting scores. Finally, they indicated that references internal to applications often lead to reviewers’ improving their scores. We may need to revisit this survey as preprints and other interim products become more common.
Overall, this survey demonstrates, yet again, the time and care NIH reviewers spend on applications. They work hard for all of us- NIH, applicants and the American public, and I am personally grateful to all of them.
I would like to acknowledge Neil Thakur with the NIH Office of extramural research as well as Mary Ann Guadagno, Leo Wu, Huong Tran, Cheng Zhang, Lin Yang, Chuck Dumais, and Richard Nakamura with the NIH Center for Scientific Review for their work on this project.