Beginning with applications due on January 25, 2019 the application instructions and review criteria will be clarified to replace the term “scientific rigor” with the term “rigor of the prior research.” Applicants will also be instructed to describe plans to address any weaknesses in the rigor of prior research within the Research Strategy. For additional details, see NOT-OD-18-228 and NOT-OD-18-229.
The NIH recently implemented updates to research grant and career development award applications aimed at enhancing reproducibility through rigor and transparency with a focus on four areas: scientific premise, rigorous experimental design, consideration of relevant biological variables, and authentication of key biological and/or chemical resources. This post is the first in a series addressing each of these four areas, starting with scientific premise.
To better understand what scientific premise is, and why it warrants specific attention in grant review, let’s consider the type of data often used by applicants in support of proposed research. Applications often include data aimed at demonstrating the feasibility of the proposed experimental approach. While this type of data can be important as proof of concept, it does not speak to the project’s scientific premise – the strengths and weakness of the data and previously performed work upon which the proposal is built upon.
A hypothetical example might help clarify this point. Let’s say an application proposes to investigate whether and how enzyme A regulates a particular cell function. Preliminary data suggest that enzyme A modifies protein B, and there are data in the literature showing that protein B regulates the particular cell function in question. The strength of the proposed project is dependent on the strength of the data suggesting that protein B regulates the particular cell function. Thus, the new application instruction pertaining to premise calls for “consideration of the strengths and weaknesses of published research or preliminary data” to evaluate the rationale for investigating the effects of enzyme A on the particular cell function. Without this information, the scientific premise of the proposed experiment may be built on shaky grounds.
Another example is not so hypothetical. In 2011, Karen Robinson and Steven Goodman of Johns Hopkins University published an analysis of 1523 clinical trial reports; they looked at how often the authors cited previously published trials that tested the same intervention. They found that fewer than 25% of preceding trials were published; furthermore, larger trials (which may have yielded more robust answers) were not more likely to be cited. Robinson and Goodman suggest that failure to cite previous work may have a number of implications, including “ethically unjustifiable trials, wasted resources, incorrect conclusions, and unnecessary risks for trial participants.” Robinson and Goodman’s article stimulated discussion; for example, Gina Kolata posted an article in the New York Times suggesting that we should not be so sure that “Science … is a meticulously built edifice” and that “Discoveries balance on ones that preceded them.”
Therefore, as a part of the Significance section of the Research Strategy, the updated instructions clarify that applicants should: “Describe the scientific premise for the proposed project, including consideration of the strengths and weaknesses of published research or preliminary data crucial to the support of your application.” Weaknesses in scientific rigor or gaps in transparency that preclude the assessment of scientific rigor should be acknowledged. If such weaknesses are identified, the applicant should consider whether or not to include this data in support of the application and how the proposed research will address the weaknesses.
It is important to stress that attention to scientific premise does not impede innovation. Even though innovative research is inherently risky, consideration of scientific premise can help investigators identify the risks and develop a research strategy that enhances the opportunity for success. Attention to scientific premise will ensure we are all building on solid foundations while supporting innovative and creative research.
In my next blog I’ll expand on the meaning of scientific rigor in NIH grant applications.