Rigorous Resources for Rigorous Research


Over two years ago, NIH rolled out a policy to enhance reproducibility of its supported research through rigor and transparency. Applicants and reviewers were required to devote more attention to four areas: the rigor of the prior research (scientific premise), the rigor of the proposed research (scientific rigor), consideration of biological variables including sex, and the authentication of key biological and/or chemical resources.

When the 21st Century Cures Act was passed later that same year, we were required, amongst other things, to assemble a working group of the Advisory Council to the NIH Director (ACD). These experts were charged with recommending ways to further enhance reproducibility of the research we fund, while being informed by the current policy.

An over-arching theme readily emerged when discussing options for concrete recommendations with the ACD in December 2017: no more forms or checklists. Instead, more resources! And, we believe highlighting existing products and taking a few other steps in response to their recommendations would be a great starting point. Below is a sampling of what we discussed at last month’s meeting (fast forward to 3 hours and 51 minutes in).

Highlighting Existing Resources and Products for Developing Better Proposals 

Why reinvent the wheel when some tremendous resources and products are already available for researchers? We already have, after all, e-Learning modules, infographics, FAQs,  blog posts, and reviewer guidance, among other things, posted on our NIH Reproducibility website, which we encourage you to explore.

Other offices at NIH have also developed valuable resources. The NIH Office of Behavioral and Social Science Research has training resources for NIH investigators on how to rigorously develop and evaluate mixed methods research applications. Likewise, the National Institute of General Medical Sciences (NIGMS) and several other NIH Institutes and Centers (ICs) funded the development of training modules to enhance data reproducibility, many of which can be found within an online clearinghouse. These modules promote expertise in rigorous experimental design and transparency targeted to graduate students, postdoctoral fellows, and other investigators early in their careers. Additionally, the NIH Office of Disease Prevention has methodological resources available for developing high-quality applications for clustered randomized clinical trials, including best practices for calculating sample sizes.

Many other “cool” resources come from outside of NIH. The Medical Research Council, for example, encourages their applicants to use a tool, called the Experimental Design Assistant, for pre-clinical research involving laboratory animals. The SPIRIT Group, an international collaboration of trialists, methodologists, journal editors, and ethicists from academia, industry, and funding agencies, makes available valuable guidance too. They describe a minimum set of scientific, ethical, and administrative elements that should be addressed in a clinical trial protocol.

Clarifying what is meant by scientific premise

Scientific premise refers to the rigor of the prior research being cited as key support for the research question(s). For instance, a proposal might note prior studies had inadequate sample sizes. To help both applicants and reviewers describe and assess the rigor of the prior research cited as key support for the proposal, we plan to revise application instructions and review criteria to clarify the language.

Under Significance, the applicant will be asked to describe the strengths and weaknesses in the rigor of the prior research (both published and unpublished) that serves as the key support for the proposed project. Under Approach, the applicant will be asked to describe plans to address weaknesses in the rigor of the prior research that serves as the key support for the proposed project. These revisions are planned for research and mentored career development award applications that come in for the January 25, 2019 due date and beyond. Be on the lookout for guide notices.

Sharing Examples of Useful Authentication Plans

Even though authentication plans are not scored during the peer review process, reviewers will assess their adequacy and any deficiencies must be resolved prior to award. Many folks have wondered what makes for a good authentication plan. We have seen some strong examples come through in recent cycles across various biomedical fields we support. So, with the permission of the awardee, we hope to share some sample plans on our NIH Reproducibility website for your reference.

Integrating Rigor into Training and the Responsible Conduct of Research

Instruction in rigorous experimental design and transparency should be part of the fabric and culture for research training programs. For responsible research to be conducted, our trainees and faculty must know not only what is of ethical, legal, and social importance, but they must also know how to conduct rigorous research. Pilot projects are on-going at NIH to begin embedding these concepts of rigorous experimental design, methods, quantitative approaches, data analysis, and interpretation into pre-doctoral training.

An NIGMS T32 funding opportunity announcement, which just had its first applications received in May, aims to do just that. Supported institutions will develop and implement effective, evidence-based approaches to train future generations of biomedical scientists in the art of rigorous research. Such pilot activities are expected to inform future NIH-wide endeavors to require training in rigorous experimental design.

Assessing Outcomes of Rigor Policies

Of course, it will be important to evaluate the outcomes of the Enhancing Reproducibility policy, particularly the adherence to the policy by applicants and reviewers. Early IC evaluations confirmed the need for additional applicant and review resources. Ongoing evaluations will further inform any additional needs or updates to the policy.

The reproducibility and transparency of biomedical research remains an important focus at NIH. We will continue working with the ACD on this topic and begin implementing these ACD recommendations very soon. So, keep an eye out for more curated resources and clarified language around Scientific Premise.

Know of a particularly useful resource too? We invite you to mention it in the Comments section below.


  1. The American Physiological Society has a collection of resources on Rigor and Reproducibility for consideration.

  2. Why doesn’t the NIH just write the proposal for us? Then flip a coin to decide which ones get funded

  3. Why, as an applicant, I should “describe plans to address weaknesses in the rigor of the prior research”. This is not my problem, I have my own project.
    And why, as a reviewer I should be assessing that rigor of prior research? The only thing that matters for the proposal being reviewed is if it addresses an important scientific question, and if the proposal has a reasonable plan to approach it.
    Looks like again the NIH bureaucracy is trying to cover their lower back instead of looking at real problems with research funding..

  4. It might be helpful for some in the biomedical research community if the NIH could provide or point to a reliable online tool to calculate power and sample size for animal experiments. While researchers conducting human clinical trials are highly trained in power calculations, many animal researchers are less familiar with it and could benefit from a user friendly, guided tool.

    1. There are numerous power/sample size calculators online which come up in a simple google search. I used some of them when needed.
      Alternatively, if you have a statistics textbook, plug the formula into Excel or Matlab. (I doubt that you could make a progress without Excel and Matlab these days, anyway.)

  5. Wow, I can look at most Nature articles, Cell articles etc, that are currently published and they will not meet the standards of rigor that are supposed to be included in the NIH grants. So if an article is published that does not meet the rigor standards, can we say “Well we need to first repeat that as there are no publicaitons with the appropriate rigor that address this question”. This will almost start scence at ground zero again. The amount of “administrative” burdens being pushed on grant submitters is getting crazy.

  6. Training modules and new regulations are all well and good, but it is not sufficient to just dictate when you are positioned to facilitate. You are a funding agency, so please fund the development of user-friendly tools that facilitate rigorous techniques. Tools will be more effective than didactic coursework, because they can significantly and directly reduce the time, effort, and cost of applying more rigorous techniques. Release an FOA for an SBIR to develop user-friendly tools that facilitate rigorous research and give it it’s own study section for review. In the interest of full disclosure, my own efforts toward developing such a tool for commercial release have met with frustration, with some reviewers stating without subtlety that they don’t think facilitating more rigorous science is worthwhile.

  7. Non-reproducibility can be explained by researchers not performing experiment at the same point in the circadian rhythm structure in the biological entity under study. Because there are statistically significant circadian rhythms in every physiological event, an intervention can result in 3 different conclusions simply depending on the time in this time structure an experiment is performed: stimulation, no effect, or inhibition. Yet all of these observations are correct – they just change with biological time. Adding control for the circadian fluctuations should be included in rigorous, experimental design and data interpretation. Without controlling for the effect of biological time is non-rigorous and causes “fickleness”. The importance of this approach to rigor in research was highlighted by the 2017 Nobel Prize for Physiology or Medicine. There are publications/field guides that deal with the pitfalls associated with not controlling for biological time.

Before submitting your comment, please review our blog comment policies.

Leave a Reply

Your email address will not be published. Required fields are marked *