13 Comments
A series to raise awareness and inspire creative problem solving of the challenges in maintaining integrity in peer review
Sometimes it takes detective work to unearth attempts to undermine the integrity of peer review.
Take the case of Dr. Smith, one of the reviewers on a study section in the Center for Scientific Review. The scientific review officer (SRO) would like Dr. Smith to review an application with Dr. Jones as principal investigator (PI).
In checking for potential conflicts of interest (COI), the SRO cast a wider net and found something troubling. Dr. Smith, one of the reviewers currently set to review the application listing Dr. Jones’ as PI, had been listed as one of the key personnel on an application with Dr. Jones as PI that was under review in another, recent study section.
It was obvious Dr. Smith had a clear COI as a reviewer for the application with Dr. Jones as PI. The COI instructions for reviewers state that a reviewer may not review certain applications and must leave the room when the reviewer, within the past three years, has been a collaborator or has had any other professional relationship with any person on the application who has a major role.
In this case, Dr. Smith, who is being considered as a reviewer for the application, is a professional associate of Dr. Jones, the PI on the application. However, Dr. Smith had not declared a conflict with that application.
The SRO immediately notified the review chief, who unearthed more information when searching PubMed. They found that Drs. Smith and Jones co-authored multiple research publications within the past two years. Coauthoring publications within the past 3 years also is a clear conflict of interest (NOT-OD-13-010) .
The review chief alerted the research integrity officer (RIO), who found more irregularities. Turned out that Dr. Smith was on the study sections that reviewed a few more grant applications listing Dr. Jones as PI. At no time had Dr. Smith declared a COI with the applications. The RIO alerted the NIH Office of Extramural Research (OER).
OER terminated Dr. Smith’s service in peer review indefinitely and the application listing Dr. Jones as PI was reassigned to a Special Emphasis Panel for initial peer review.
All participants and stakeholders in the peer review system are responsible for its integrity. Whether you are a reviewer, PI, or NIH staff, each of those roles is crucial. NIH is paying attention, and NIH is taking action.
This scenario is fictitious, but based on real events. Stay tuned as the case unfolds. If you want more on the topic, please listen to this NIH All About Grants podcast conversation on how NIH manages conflicts of interest during the peer review process (MP3 / Transcript).
If you see something, say something.
“…a reviewer may not review certain applications and must leave the room when the reviewer, within the past three years, has been a collaborator or has had any other professional relationship with any person on the application who has a major role.”
Yes. Makes perfect sense. On day 1095 after Dr Smith co-authors a paper with Dr Jones he is still in conflict with Dr Jones. But on day 1096 (> 3 years) he is not.
This goes to root the problem with the NIH peer review system. They set these arbitrary regulations in a cosmetic effort to identify “COI”, fully understanding that human relations are much more complex, not easily identifiable, and don’t adhere to their timescale. Friends and enemies often last a lifetime, and you don’t have to be a collaborator to be a friend, nor do you have to publicly declare yourself an enemy to be one.
Real COI are often hidden by both the applicant and reviewer, and it will take a lot more effort and investigation on parts of SROs to identify them.
Sending on behalf of Sally Amero (NIH’s Review Policy Officer): NIH peer review regulations and policy further stipulate that, regardless of the level of financial involvement or other interest, if the reviewer feels unable to provide objective advice, he/she must recuse him/herself from the review of the application at issue. Regulations and policy also state that the peer review system relies on the professionalism of each reviewer to identify to the designated government official any real or apparent conflicts of interest that are likely to bias the reviewer’s evaluation of an application. Therefore, reviewers have a responsibility to report their conflicts of interest.
In other words, it’s an honor system that works. If honor systems work, why doesn’t our Judicial System institute it? Why do we need attorneys to thoroughly vet and approve jurors? Why can’t we simply trust the jurors to do the right thing? Why is a juror automatically disqualified if he or she has ANY contact whatsoever, past or present, with the plaintiff or defendant? Why can’t a past defendant be a juror on case in which the person he will be sitting in judgement of was a juror for his trial?
Honor systems don’t work in our judicial system, nor do they work in the peer review system. COI often go undisclosed, wittingly or unwittingly.
As much as we’d all like to believe otherwise, human relationships are at the core of peer review. This is a great point you made. The reality is that if you’re an expert in a field, then you probably have some personal relationship with most of the investigators in that field. This is the Catch-22 of the NIH peer review. By definition, the exact people who are in a position to best review a grant have some built in conflict; you help grants that support your field, you hinder grants with which you feel in competition, you have a negative/positive history, etc….Ok, so those experts recuse themselves. Then what? Now the grant is reviewed by people with less expertise in that field, putting that application at a strong disadvantage.
LOL. “Therefore, reviewers have a responsibility to report their conflicts of interest”.
Reviewer to SRO: “I hold a grudge against this applicant because I know he was one of the reviewers who trashed my application a couple of years ago (not hard to figure out). So I want to recuse myself from reviewing his application”.
That “professionalism” was blatantly on display when reviewers were caught sharing applications with one another, others not involved in the review, and in some cases with foreign entities. The NIH wants you to believe this problem was solved when a few high profile bad apples were fired by their institutions. Think again. It was only pushed under the surface. Those “professionals” have now learned to be more careful when perpetrating their misdeeds.
Totally agree.
The NIH peer review system is broken, period. In my opinion, a bureaucracy built on such policies, but with no organized and efficient oversight of the review process and its related conflict of interest is dysfunctional. Unknowledgeable “senior” reviewers, you call professionals, who know SROs, who know Program Officers, who know Division directors, work, as my colleague above stated, in honorary, corrupt systems that only benefit those already in it. Incompetent, dishonest “Professionals” and this corruption need to be rooted out for the NIH peer review to work in an unbiased fair way……
There are major challenges we face with the peer review system, which are only becoming worse as the stakes get higher. This applies to publications as well as grants. But could we come up with a better system? This is analogous to the famous quote (often inaccurately attributed to Churchill) that “democracy is the worst form of Government except for all those other forms that have been tried from time to time”.
Our system has worked well for a long period of time thanks to the enormous time and efforts contributed by so many stakeholders to whom thanks are owed. But perhaps it can be improved.
A suggestion that has been tested is fully anonymized applications which would address some other biases as well. Don’t know how feasible it is. If not already done, NIH should perhaps consider an effort to assess the whole peer review process including tweaks and alternatives and open it up to comments.
As the case unfolds, more information about Dr. Smith becomes available. Dr. Smith was requested by Dr. Jones to provide a letter of support for a different application and serve as a consultant without any funds or experimental work committed. Dr. Smith agrees as a scientific colleague to provide the letter, but in the absence of any further interactions with Dr. Jones, does not consider this a formal collaboration. Dr. Smith and Dr. Jones work have never trained together, never worked at the same institution, have no financial interests, or shared mentee/mentor relationships. Is Dr. Smith a professional associate of Dr. Jones because she/he agreed to be a scientific colleague and provide a letter of support for another pending grant and does this constitute a major professional role in an application (NOT-OD-13-010).
Upon review of the publications, they are either review articles or articles with shared reagents that both Dr. Jones and Dr. Smith provided as active leaders in their field, but none of the article demonstrated collaborative research performed or led by either Dr. Smith or Dr. Jones. Published NIH guidelines allow such co-authorships as not demonstrating conflict of interest (https://grants.nih.gov/grants/peer/peer_coi.htm).
In the end, Dr. Smith has to consider whether she/he has a COI with Dr. Jones based on NIH guidelines as well as his/her knowledge of their relationship. Her/his judgement is that he/she does not have a COI that precludes impartial review. The OER comes to the opposite conclusion.
Will simply removing Dr. Smith from the peer review system improve the integrity and robustness of peer review by eliminating an experienced peer reviewer who is a leading researcher in the area of the study section? Or should the definitions that the NIH uses to define COI have further clarity to prevent such confusion? Do letters of support automatically, for any application outside of the application being reviewed automatically constitute a COI to the NIH? Perhaps the case is not so simple after all, like many in COI.
If you want integrity in the reviews, select reviewers with gravitas and integrity. Have you looked at the composition of some of these study sections? They are case studies in political correctness. People selected for permanent membership who have a dismal record of impactful publications, sitting in judgement of others who are far more accomplished than them. How are these reviewers selected? Are the SROs responsible, and who is holding them to account? Is it because they have grants from that study section? That may be so, but the more important criterion should be WHAT HAVE THEY ACCOMPLISHED USING THAT GRANT FUNDING? Is anyone asking this question?
They are selected because they have connections in the study section. They know someone in the study section (maybe Chair) who recommends them to SRO and then it’s a done deal. That’s how the NIH review “meritocracy” works.
There is a trend to select young scientists to “experience” the study section. Those young scientists often lack experience in grant writing, but sit there to judge others’ work. Why cannot they be an observer first?
The Center for Scientific Review (CSR) operates an Early Career Reviewer (ECR) program. Decisions about the eligibility criteria and the participation level of ECRs were made with quality of review being the primary consideration (more in the October 2019 report from the CSR Advisory Council Early Career Reviewer Working Group). In addition to some publication requirements, ECRs must have an independent position and must have submitted a grant application to NIH and received the summary statement. Two ECRs serve at each meeting of a CSR standing panel and are assigned two proposals as Reviewer 3. As all new reviewers do, ECRs receive extensive training and guidance from their scientific review officer. Early career or new reviewers may not observe a study section before serving as a reviewer. The study section meetings are closed to the public in accordance with section 1009 of the Federal Advisory Committee Act and sections 552b(c)(4) and 552b(6) of the Government in the Sunshine Act. As such, the deliberations are confidential. All attendees must have a “need to know”.