6 Comments
As NIH moves ahead with implementing measures to enhance rigor, transparency and reproducibility in NIH-supported research, I’d like to give a brief update on these efforts, and highlight some important timeline changes for implementation in applications for institutional training grants (T), institutional career development awards (K12), and individual fellowships (F).
To briefly recap, in October, NIH announced updates to the application instructions and review criteria for most research grants and individual mentored career development awards. These updates instruct applicants to address four key areas NIH deems important for enhancing rigor and transparency in research: 1) the scientific premise forming the basis of the proposed research; 2) rigorous experimental design for valid, robust, and unbiased results; 3) consideration of relevant biological variables; and 4) authentication of key biological and/or chemical resources. My October blog post, “Bolstering Trust in Science Through Rigorous Standards,” describes the rationale behind the changes and the steps NIH has taken to engage the community in these efforts.
For the January 25, 2016 application due dates, the updates apply to most NIH research grant applications, with some exceptions, as described in the October NIH Guide notice. We also announced requirements to address rigor in individual mentored career development award applications submitted after January 25, 2016. In addition, Research Performance Progress Reports (RPPR) for these programs must also address rigor if they are submitted on or after January 25.
As you consider how to address rigor in your NIH applications, I would like to remind you of resources that should help along the way. Your first stop should be the NIH Office of Extramural Research (OER) web page on rigor and reproducibility, which links to a variety of resources from OER, and across NIH. For example, you might want to watch our NIH staff training module. While this tutorial was initially created for NIH program officers and scientific review officers, it provides lots of content that would be useful to you too–including a general policy overview on rigor and transparency, as well as updates on the changes to our grant applications and review language.
We also recently extended the timeline for implementing rigor and transparency policy changes for institutional training grants, institutional career development grants, and individual fellowships. We recognized that applicants to these programs would require significant time and resources to design substantive instructional plans and new curricula to ensure the in-depth training in rigorous experimental design for trainees and fellows. This is especially true given the breadth of different training and career development programs funded across NIH. As early as fiscal year 2017, we will be asking applicants to include plans for instructing trainees and fellows in rigorous experimental design (stay tuned for future NIH Guide notices). As you start to think about future applications, you may wish to review some different approaches for addressing experimental design and reproducibility in curricula and training. For example, NIGMS recently issued a funding opportunity to support the development of “Training Modules to Enhance Data Reproducibility.” You can review summaries of the awarded projects on NIH RePORT. NIGMS has also compiled award abstracts describing predoctoral training curricula on their website. We also recognize that each grant application will need to develop specific instructional material that matches the specific area of training and research.
I look forward to expanding more on the topics of scientific premise and rigorous experimental design in future blog posts, and hearing your thoughts and questions as well!
Lauer – Thanks – your analysis is helpful and I’ve posted it on our website. But note that October 6 2014 was a Monday and this usually means that more PI’s want to take the extra time over the weekend to work on their proposals.
To Michael Beauchamp – I’m not sure why a central office would wait until the last day to submit – it’s very risky, especially given system glitches. We like to get the application in as soon as possible to give the PI time to review the posted PDF on the Commons.
address rigor in the review process? We’ve seen shocking reviewer nonsense however, applicants have little to no recourse other than to endlessly submit and resubmit our proposals.
I believe this lack of transparency and farcical reviews are primary drivers in the lack of rigor seen in current NIH funded work. There are so few legitimate ways for new applicants to get funded.
PS It is way past time for new leadership at the CSR.
In what part of the applications should Rigor and Transparency be addressed? In the Research Strategy, in some other part, or in general?
Scientific Rigor and Biological Variables should be addressed in the Research Strategy under Approach. Authentication should be addressed in “Other Project Information” as an “Other Attachment”. In addition to referring to the updated application guide and resources posted to the page on reproducibility updates on grants.nih.gov, be sure to see the four-part Open Mike blog posts addressing each aspect of the new changes. These blogs expand on these topics more in depth in response to questions people had about the new policies.
Dear Mike,
We are all aware that science without rigor is science fiction; I therefore applaud the spirit of this initiative. I am afraid however it does not address the problem nor its root causes. The real problem is lack of rigor in publication, not application. In the immediate future reemphasizing the scientific merit of the application (and perhaps giving “approach” a higher weight than the subjective criteria of “impact” and “innovation”) is an important step, but it will not affect rigor in publication. To reinstate the level of rigor and reproducibility the public deserves, we need to reinforce professional standards in publication by creating meaningful consequences for those that repeatedly fail to produce rigorous publications. On the “carrot” side, we should recognize and reward those active for a decade or more without a retraction, always sharing reagents and data freely, and repeatedly publishing reproducible and impactful results (here is where the “impact” category can move from the subjective to the objective). One reward that comes to mind is easier access to NIH resources. A “stick” is also needed for those that consider marketing their work to the “right” journal ahead of rigor. Again, restricting access to NIH funds comes to mind. This will only increase the pool for those with higher standards.
In the longer term, we need to reexamine the root causes. In my humble opinion the sorry state of rigor in scientific publications the NIH is trying to address reflects, at least in part, decades of emphasis on a journal’s “impact factor” as measure of quality and key to promotion, and an increasing fraction of NIH resources distributed via RFA and other mechanisms reflecting the latest political initiatives at the expense of, and not in addition to, investigator-initiated applications. We need to give careful consideration to the role of these and other factors play in shaping our collective output and their contribution to lowering the rigor of our product, and act accordingly to eliminate the rewards for poor craftsmanship.
Respectfully,
Raphael Kopan
I completely agree with this assessment. Is there such a thing as non-rigorous science?