7 Comments
Beginning with applications due on January 25, 2019 the application instructions and review criteria will be clarified to replace the term “scientific premise” with the term “rigor of the prior research.” Applicants will also be instructed to describe plans to address any weaknesses in the rigor of prior research within the Research Strategy. For additional details, see NOT-OD-18-228 and NOT-OD-18-229.
The NIH recently implemented updates to research grant and career development award applications aimed at enhancing reproducibility through rigor and transparency with a focus on four areas: scientific premise, rigorous experimental design, consideration of relevant biological variables, and authentication of key biological and/or chemical resources. This post is the first in a series addressing each of these four areas, starting with scientific premise.
To better understand what scientific premise is, and why it warrants specific attention in grant review, let’s consider the type of data often used by applicants in support of proposed research. Applications often include data aimed at demonstrating the feasibility of the proposed experimental approach. While this type of data can be important as proof of concept, it does not speak to the project’s scientific premise – the strengths and weakness of the data and previously performed work upon which the proposal is built upon.
A hypothetical example might help clarify this point. Let’s say an application proposes to investigate whether and how enzyme A regulates a particular cell function. Preliminary data suggest that enzyme A modifies protein B, and there are data in the literature showing that protein B regulates the particular cell function in question. The strength of the proposed project is dependent on the strength of the data suggesting that protein B regulates the particular cell function. Thus, the new application instruction pertaining to premise calls for “consideration of the strengths and weaknesses of published research or preliminary data” to evaluate the rationale for investigating the effects of enzyme A on the particular cell function. Without this information, the scientific premise of the proposed experiment may be built on shaky grounds.
Another example is not so hypothetical. In 2011, Karen Robinson and Steven Goodman of Johns Hopkins University published an analysis of 1523 clinical trial reports; they looked at how often the authors cited previously published trials that tested the same intervention. They found that fewer than 25% of preceding trials were published; furthermore, larger trials (which may have yielded more robust answers) were not more likely to be cited. Robinson and Goodman suggest that failure to cite previous work may have a number of implications, including “ethically unjustifiable trials, wasted resources, incorrect conclusions, and unnecessary risks for trial participants.” Robinson and Goodman’s article stimulated discussion; for example, Gina Kolata posted an article in the New York Times suggesting that we should not be so sure that “Science … is a meticulously built edifice” and that “Discoveries balance on ones that preceded them.”
Therefore, as a part of the Significance section of the Research Strategy, the updated instructions clarify that applicants should: “Describe the scientific premise for the proposed project, including consideration of the strengths and weaknesses of published research or preliminary data crucial to the support of your application.” Weaknesses in scientific rigor or gaps in transparency that preclude the assessment of scientific rigor should be acknowledged. If such weaknesses are identified, the applicant should consider whether or not to include this data in support of the application and how the proposed research will address the weaknesses.
It is important to stress that attention to scientific premise does not impede innovation. Even though innovative research is inherently risky, consideration of scientific premise can help investigators identify the risks and develop a research strategy that enhances the opportunity for success. Attention to scientific premise will ensure we are all building on solid foundations while supporting innovative and creative research.
In my next blog I’ll expand on the meaning of scientific rigor in NIH grant applications.
Over a century of scientists dating at least back to Mach have studied how scientific advances are made, including discovery, invention, innovation, paradigm change, multidisicplinarity, and social factors that can inhibit all of these. Is Dr. Lauer familiar with any of this literature? Or does he take his cue from Gina Kolata, who is for sure not familiar with any of this?
Could the author of this post please link to relevant citations? This is a real question; not meant to be provocative. I’d like to read more about the history of scientific discovery.
To address the issue of Scientific Premise, Dr. Lauer has presented the example of the role of enzyme A in a given cellular process. The existing evidence indicates that protein B, for which data indicates it is modified by enzyme A, participates in the cellular process in question and hence the interest to investigate if enzyme A is involved in the regulation of this process.
Dr. Lauer indicated that the without discussing the strengths and weaknesses of published preliminary data to evaluate the rationale for investigating the effects of enzyme A on a particular cell function, the proposed experiment may be built on shaky grounds.
The problem I see is that this will involve that for every key piece of published data in peer review journals that support the proposal, the PI will have to conduct a re-review of the published paper, which itself raises significant questions and problems.
So now NIH is trying to use regulations and guidelines to get investigators to do what they should have learned as graduate students or, perhaps, slow-on-the-uptake postdocs?
How about coming up with ways to improve training rather that devising “updates” to grant applications.
It seems ironic to me that these new changes have been instituted without rigorous investigation of their pros and cons and with little transparency or clarity. Has any consideration been given to the amount of space that it takes on a grant proposal to critically review and discuss all relevant previous data in an area? Although the decrease from 25 pages to 12 (13, including specific aims) had advantages, it necessitated eliminating considerable detail from proposals. Now there is a demand for more detail, both in critical review of the literature and in experimental detail (to allow judgement of rigor). What should be left out in writing a proposal to make room for the added detail? Several other, more general questions are also raised by this new policy: Are the mandated changes really likely to do more good than harm? Where can clear instructions be found that describe what is required? Why wasn’t this change vetted with the scientific community prior to its institution? What extra funds are being provided by NIH to allow for periodic validation of mice and reagents? What standards should be applicable for this (isn’t this also something that should have been discussed before a policy was announced)? In an era where funds are short,how much money should be spent for validation purposes? Should validation been done at the level of the individual lab or centrally, as a service? Wasn’t it always a job of reviewers on study sections to evaluate proposals for adequate consideration of previous studies in an area and the rigor of proposed studies? Will the changes make the reviewers’ job easier or more difficult? Why wasn’t a pilot study performed before instituting a general policy?
The excellent comments by Fred Finkelman are exactly to the point. I am curious why there is no reply on these important issues after more than 1 year.
Does the scientific review process include a harm/benefit analysis for research protocols involving humans or animals.