2 Comments
“Despite the numerous successes stemming from animal research, concerning reports over the past decade have described biomedical experiments that fail to replicate or to translate in ways that improve human health. All research is not expected to translate to human treatments, as there is no perfect model. Scientific process is as much about failure as it is about success. Yet part of the scientific process is also continual improvement, which includes working to understand what might contribute to unexpected outcomes within animal research.”
The Advisory Committee to the NIH Director (ACD) expressed that view in their 2021 final report recommending ways for NIH to improve the rigor, reproducibility, and translatability of the research we support involving animal models. Former NIH Director, Dr. Francis Collins reinforced this notion when he wrote soon after the ACD report’s release, “The ability to reproduce biomedical research findings is foundational to the advancement of science and relies on rigorously designed and performed research studies. When a scientific finding can be reproduced by multiple scientists, it validates the accuracy of the data and ensures the study is ready to progress to the next phase of research.”
Along these lines, I published a JAMA Open editorial last August stimulated by an interesting analysis showing that NIH-funded scientists are increasingly using “hype” and “appeals to emotion” in their proposal abstracts. As might be expected, scientists describe their ideas and plans in positive terms, but objective assessments of rigor paint less rosy pictures. The ACD Working Group final report offers a range of approaches to raising the level of NIH-funded animal research.
I also updated the ACD in June on our implementation of their recommendations. The full discussion is available via the NIH videocast to learn more (fast forward to 4 hours 39 minutes). Here, in lieu of going through each of the report’s five themes, I will highlight some approaches raised during the meeting in response to their recommendations
We have many resources available that directly address several ACD recommendations. We encourage investigators to explore these materials as they are developing research plans and publishing the results of their research. Below are some selected examples:
- NIH strongly encourages researchers to use a powerful, free online tool when designing their animal experiments proposed in grant applications.
- The National Institute of General Medical Sciences hosts a clearinghouse on training resources for rigor, which among other things includes statistical topics and controls for animal research.
- The National Institute on Aging (NIA) supports a publicly available, searchable database to increase the transparency, reproducibility, and translatability of preclinical studies of possible treatments for Alzheimer’s disease. The platform allows for publications supported by NIA funding to be graded on rigor and transparency criteria being properly reported, such as blinding, sex as a biological variable, and randomization. It also is a venue for reporting unpublished studies (and assigns a Digital Object Identifier to these reports).
- The National Institute of Neurological Disorders and Stroke funds educational resources to advance rigor in animal research and build a greater emphasis on rigor at universities around the country. They also announced in August the first awards aimed at improving education in the principles of rigorous research.
Activities are underway to educate the public on the value of animal research (see Recommendation 3.5). We updated our Animals in NIH Research page last summer. The updated site explains how properly designed experiments involving animal models contribute to improvements in public health.
NIH published a Notice of Special Interest in December 2021 to develop resources and technologies for enhancing rigor, reproducibility, and translatability of animal models in biomedical research (see recommendation 3.3). Applications must propose developing exploratory and highly innovative projects that are broadly applicable to multiple NIH Institutes or Centers or multiple organ systems. At the May NIH Council of Councils meeting, a proposed funding opportunity concept outlined a strategy to fund new related resources through various other grant types including large multi-year grants, resource grants, and small business awards.
We are also exploring several strategies to promote strong research practices in future work. For instance, we just published a guide notice that encourages award recipients to include the ARRIVE Essential 10 (essential elements of study design) in all NIH-supported publications describing vertebrate animal and cephalopod research (see Recommendation 4.2). The ARRIVE Essential 10 describes the information that the authors consider “the basic minimum to include in a manuscript, as without this information, reviewers and readers cannot confidently assess the reliability of the findings presented.” This checklist includes specific details about comparison groups and the experimental units, sample sizes, inclusion and exclusion criteria, randomization, blinding, outcome measures, statistical analysis methods, animals used, experimental procedures, descriptive statistics, and effect sizes and confidence intervals. In addition to the ARRIVE Essential 10, other methodological documentation and standards for reporting results have been developed by the research community and include CONSORT (for randomized trials) and MDAR (for the broader life sciences community).
Please stay tuned for more information and activities to come, including a related new ACD working group (announced in November) to help us consider new ways to develop and use alternative methods to advance biomedical research (see recommendation 3.6).
We are committed to ensuring our supported research is of the highest quality, efficient, explainable, and can be repeated by others, thereby increasing the validity of the scientific findings. When experiments involving animals are designed with rigor, the results are more likely to be replicated in future studies and translate in ways that improve the health of humans.
Re the JAMA open paper on adjective use – has anyone looked at unfunded NIH applications to see whether hype adjectives or more or less prevalent in unfunded applications than in those that are funded? One possible hypothesis is that applications that over-hype get worse scores because they turn off reviewers. Obviously this “interesting analysis” would not be feasible without access to internal NIH data.
While it would be a challenge to argue against increased focus on rigor and reproducibility in research today, the issue that is almost completely absent from these discussions is that of enforcement. The NIH appears to be far too willing to come up with new rules and required forms but is almost completely lacking in enforcement. Even simple things like sharing of resources are not enforced by NIH or journals yet we likely all have encountered PIs who are unwilling to share published reagents. Also, large NIH projects like the mouse knockout project are generating resources that can be difficult or very slow to obtain and at least in some cases turn out to have errors that were not reported. If the recipient doesn’t check everything (e.g., re-sequence everything obtained) there is a further risk of delays, wasted money, and worse – incorrect results published.