Over two years ago, NIH rolled out a policy to enhance reproducibility of its supported research through rigor and transparency. Applicants and reviewers were required to devote more attention to four areas: the rigor of the prior research (scientific premise), the rigor of the proposed research (scientific rigor), consideration of biological variables including sex, and the authentication of key biological and/or chemical resources.
When the 21st Century Cures Act was passed later that same year, we were required, amongst other things, to assemble a working group of the Advisory Council to the NIH Director (ACD). These experts were charged with recommending ways to further enhance reproducibility of the research we fund, while being informed by the current policy.
An over-arching theme readily emerged when discussing options for concrete recommendations with the ACD in December 2017: no more forms or checklists. Instead, more resources! And, we believe highlighting existing products and taking a few other steps in response to their recommendations would be a great starting point. Below is a sampling of what we discussed at last month’s meeting (fast forward to 3 hours and 51 minutes in).
Highlighting Existing Resources and Products for Developing Better Proposals
Why reinvent the wheel when some tremendous resources and products are already available for researchers? We already have, after all, e-Learning modules, infographics, FAQs, blog posts, and reviewer guidance, among other things, posted on our NIH Reproducibility website, which we encourage you to explore.
Other offices at NIH have also developed valuable resources. The NIH Office of Behavioral and Social Science Research has training resources for NIH investigators on how to rigorously develop and evaluate mixed methods research applications. Likewise, the National Institute of General Medical Sciences (NIGMS) and several other NIH Institutes and Centers (ICs) funded the development of training modules to enhance data reproducibility, many of which can be found within an online clearinghouse. These modules promote expertise in rigorous experimental design and transparency targeted to graduate students, postdoctoral fellows, and other investigators early in their careers. Additionally, the NIH Office of Disease Prevention has methodological resources available for developing high-quality applications for clustered randomized clinical trials, including best practices for calculating sample sizes.
Many other “cool” resources come from outside of NIH. The Medical Research Council, for example, encourages their applicants to use a tool, called the Experimental Design Assistant, for pre-clinical research involving laboratory animals. The SPIRIT Group, an international collaboration of trialists, methodologists, journal editors, and ethicists from academia, industry, and funding agencies, makes available valuable guidance too. They describe a minimum set of scientific, ethical, and administrative elements that should be addressed in a clinical trial protocol.
Clarifying what is meant by scientific premise
Scientific premise refers to the rigor of the prior research being cited as key support for the research question(s). For instance, a proposal might note prior studies had inadequate sample sizes. To help both applicants and reviewers describe and assess the rigor of the prior research cited as key support for the proposal, we plan to revise application instructions and review criteria to clarify the language.
Under Significance, the applicant will be asked to describe the strengths and weaknesses in the rigor of the prior research (both published and unpublished) that serves as the key support for the proposed project. Under Approach, the applicant will be asked to describe plans to address weaknesses in the rigor of the prior research that serves as the key support for the proposed project. These revisions are planned for research and mentored career development award applications that come in for the January 25, 2019 due date and beyond. Be on the lookout for guide notices.
Sharing Examples of Useful Authentication Plans
Even though authentication plans are not scored during the peer review process, reviewers will assess their adequacy and any deficiencies must be resolved prior to award. Many folks have wondered what makes for a good authentication plan. We have seen some strong examples come through in recent cycles across various biomedical fields we support. So, with the permission of the awardee, we hope to share some sample plans on our NIH Reproducibility website for your reference.
Integrating Rigor into Training and the Responsible Conduct of Research
Instruction in rigorous experimental design and transparency should be part of the fabric and culture for research training programs. For responsible research to be conducted, our trainees and faculty must know not only what is of ethical, legal, and social importance, but they must also know how to conduct rigorous research. Pilot projects are on-going at NIH to begin embedding these concepts of rigorous experimental design, methods, quantitative approaches, data analysis, and interpretation into pre-doctoral training.
An NIGMS T32 funding opportunity announcement, which just had its first applications received in May, aims to do just that. Supported institutions will develop and implement effective, evidence-based approaches to train future generations of biomedical scientists in the art of rigorous research. Such pilot activities are expected to inform future NIH-wide endeavors to require training in rigorous experimental design.
Assessing Outcomes of Rigor Policies
Of course, it will be important to evaluate the outcomes of the Enhancing Reproducibility policy, particularly the adherence to the policy by applicants and reviewers. Early IC evaluations confirmed the need for additional applicant and review resources. Ongoing evaluations will further inform any additional needs or updates to the policy.
The reproducibility and transparency of biomedical research remains an important focus at NIH. We will continue working with the ACD on this topic and begin implementing these ACD recommendations very soon. So, keep an eye out for more curated resources and clarified language around Scientific Premise.
Know of a particularly useful resource too? We invite you to mention it in the Comments section below.