13 Comments
A series to raise awareness, encourage dialogue and inspire creative problem solving of the challenges in maintaining integrity in peer review
What happens when a former colleague contacts you, a reviewer, out of the blue to ask if the application on which he is a principal investigator could be treated favorably at the review meeting? Do you brush off the investigator and figure you will not let the contact influence your review of that application? Or do you instead immediately notify NIH?
Intrigued? We have a case for you (based on true stories, details have been changed slightly and names have been fictionalized). Read on.
Dr. Miller, a reviewer recently appointed to an NIH study section, was surprised to get an email from a former lab colleague, Dr. Johnson. They had not kept in touch over the years. As it so happened, Johnson was designated as a PI on one of the applications Miller was reviewing. Johnson mentioned their common scientific interests and went on to ask if any of the other applications on the study section involved members of the ‘old gang.’ Perhaps, he suggested, those applications could get favorable consideration from Miller and other reviewers.
Miller was stunned and responded that he did not know and could not help. To which Johnson replied, “It’s not wrong, it’s how we help each other. And remember, I know a lot of people.”
The exchange left Miller feeling unsettled. Even if he declared a conflict of interest with the application on which Johnson was a PI, he could still be seen as responsible for the outcome of the review because conflicts of interest are strictly confidential. Moreover, Johnson’s message implied that Miller should convince other study section members to look favorably on the application. Finally, Miller interpreted Johnson’s message to be a veiled threat; ‘knowing a lot of people’ could mean that Johnson would marshal powerful people in the field against Miller if the outcome were not favorable.
Miller immediately forwarded the email exchange to the NIH scientific review officer (SRO) running the NIH study section on which he serves.
As instructed, the SRO confidentially forwarded the exchange to her Institute’s research integrity officer (RIO) for evaluation and possible actions. In turn, the RIO confidentially shared the information and her recommendations with the NIH Office of Extramural Research (OER).
NIH indefinitely terminated Johnson’s service in NIH peer review and the application listing Johnson as PI was deferred to another study section for review. NIH OER leadership also informed the Vice President for Research at Johnson’s university about the details of the case and the violations.
It appears that Dr. Johnson may have deliberately violated NIH peer review policies (NOT-OD-15-106 and NOT-OD-18-115) and attempted to undermine the integrity of the NIH peer review process in violation of NIH policy.
Consistent with NIH policy (e.g., NOT-OD-15-106 and NOT-OD-18-115), NIH is notifying you as a senior leader in Dr. Johnson’s institution. Because these integrity concerns raise questions about Dr. Johnson’s authority and responsibility as a designated PI on NIH applications and awards, NIH is requesting that you review Dr. Johnson’s actions as the PI on NIH grant applications and awards to assess the (potential) impact.
Also, because of this situation, we are concerned that staff in your institution may not be fully cognizant of their responsibilities and of the potential consequences of integrity breaches.
NIH is requesting that you respond in writing within 30 days, addressing any impact on such applications and awards and confirming any responsive action(s) you may take.
Upon receiving this information, the Vice President for Research at Johnson’s university took steps to address this violation. A formal letter to NIH leadership stated that the institution had completed an investigation into the matter and confirmed that Johnson had improper communications with an individual serving on an NIH study section. Therefore, Johnson would be:
- prohibited by the institution from submitting any applications or receiving any support from the NIH for two years,
- prohibited by the institution from serving on an NIH or other federally chartered study section for three years,
- required by the institution to complete a course in responsible conduct of research, and
- subject to administrative penalties at the institution.
Furthermore, the institution developed plans to enhance its already existing faculty training to include a module on peer review integrity.
As you can see, NIH is taking issues of integrity in peer review seriously. We are highlighting particular case studies in an effort to be transparent in our actions and intent. We appreciate the actions taken by reviewers like Dr. Miller in recognizing the violation and informing us immediately.
If you see something, say something.
Ensuring the integrity and fairness of the peer review process is a very challenging task. There is definitely bias, unfairness, and favoritism in the NIH grant review process. Investigators who called friend reviewers for their own or their mentees’ applications are not uncommon stories. However, a more common problem is that of reviewers who favor their friends without having to be contacted and this is very much done in a mutual way. The applicants who suffer the most from this bias and favoritism are often those who are junior/new, not very social or political, from small institutions, minority, and/or who are not members of NIH review panels. Other examples of the review problems are the lack of consistency and common standards in the review. For example, a reviewer may raise one concern (e.g. mouse model) for some applicant but omit the same applicable problem for another applicant because of applicant’s seniority level, authority in the field, race, ethnicity,.. .etc. The term productivity is also another issue. There are investigators who have two or three R01s with relatively few publications. The level of NIH funding and other funding and available resources should be heavily considered when evaluating applicant’s productivity. There are also investigators who receive multiple renewals of R01 funding for a project that had already proven irrelevant clinically, never moved the field forward, and/or was only produced and reproduced by the same group. I had a grant that was killed (7 years ago) because the reviewers cited a pilot clinical trial that had proven my study to clinically irrelevant. In the meantime, a senior PI has been receiving major funding for the same studies for over 10 years. Last, the NIH SRO should take a note when a reviewer is assigned an application that is directly and closely competing with their work. Having said all of that, it is important to acknowledge that there are as many fair and reasonable NIH grant reviewers who serve with honor and integrity.
Thank you for sharing this. It’s important for folks to know that NIH takes allegations of misconduct seriously. I would encourage you to be even more open. To share the names of those who have violated NIH policy. This kind of openness provides a powerful disincentive to bad behavior and would be a way to protect whistleblowers like “Dr Miller” in this story. We need these kinds of brave folks to be celebrated and insulated from retaliation.
Again, thank you for sharing this and I look forward to more ways we can ensure we are funding the best science and the best scientists.
Dear Beth, I mean no disrespect, but there is a lot of lip service being lately given by NIH officials to “integrity” but there is very little or no action being taken. I have contacted NIH officials about specific breaches of integrity. I have given them concrete proof that they can themselves investigate and verify using the powers and information they possess, but NO ACTION WAS TAKEN. It is very disheartening when the Cops sworn to protect the public turn a blind eye. And yes. NIH does nothing to protect whistelblowers, not even protect their anonymity. Maybe they’re acting on directives from POTUS.
I agree with Dr. McLaughlin. You publish the names of trainees found to have faked data in misconduct rpeorts—why protect professors found to have engaged in corruption?
Peer Review can only be free of bias and favoritism when you ELIMINATE ALL PERSONAL AND PROFESSIONAL RELATIONSHIPS OR CONTACTS, PAST PRESENT AND FUTURE, REAL OR POTENTIAL, BETWEEN THE REVIEWERS AND REVIEWED. Human nature is what it is.
you can certainly minimize it by taking actions such as these
Not likely. Actions such as these are only possible when someone with hubris stupidly leaves an electronic record. Most people are unfairly influencing peer review but not leaving any traceable record of it. In these vast majority of cases the NIH is impotent and they know it.
So, what happens when Dr Johnson is back in the game after 2-3 years? He will be sitting on study sections again, if not federal, then other study sections. Dr Miller is going to have to watch his back for the rest of his career. He may be able to exclude him from reviewing some of his applications and papers, but definitely not all. Some granting agencies don’t even allow you to exclude reviewers. Even some NIH SROs give you a hard time when asked to exclude certain reviewers, seeing it as an affront to their authority. Will the NIH have Dr. Miller’s back for the rest of his career?
Scenario and question for NIH Staff.
Applicant is in contact with Study Section chartered member before submission of application. Applicant gets advice from member (reviewer) about applicant preparation. Applicant submits application asking it be assigned to study section of which reviewer is chartered member. Applicant then does not contact said reviewer until after meeting. Reviewer does not inform SRO about this contact with applicant.
Application is reviewed by this study section and reviewer attends meeting. Application gets fundable score. Grant is awarded. Applicant contacts reviewer thanking him.
Is this scenario acceptable under certain conditions? Is it acceptable if chartered member is an assigned reviewer? Is it acceptable if chartered member is NOT one of 3 assigned reviewers but is present during discussion of application? Is it acceptable if chartered member declares COI and recuses himself completely from review of application, but still does not tell SRO about contact with applicant before meeting regarding application preparation?
What will the NIH do? Can the NIH revoke the award? What if applicant claims he didn’t know that chartered member was going to be participating in that study section meeting?
Thanks for the question. Investigators who are new to the NIH grants process are encouraged to seek information on the review process from current or former reviewers and may seek input on their applications from senior investigators in their field. However, if that application were to go to a study section where a senior investigator who had provided advice on the application is a reviewer, we would expect that reviewer to declare a conflict of interest. Any situation that would introduce the appearance of a conflict of interest should be handled as a conflict of interest.
The scenario doesn’t mention whistle blower protections. I agree that Miller will have to forever watch his back. For that sort of violation, a longer ban on serving as a peer reviewer for NIH (a decade?) would be a step in the right direction.
Also, what if the institutional investigation found “nothing” — what if they concluded that this exchange was perhaps ill-advised but fell short of being a serious violation. (Sound familiar?!) It’s in the institution’s self-interest NOT to find against their own (potentially-well-funded) PI. For NIH to declare Johnson “innocent” based on his own institution’s review would really leave Miller exposed. Would NIH still sanction Johnson even if Johnson’s institution disagreed?
The first rule of the Mafia was to never commit anything to tape film or paper. Those who did, didn’t last long in the mob. It took the FBI decades to break up the Mafia and required inside informants, wire-taps, and subpoena powers. There are elements of a mob mentality in peer review and if government officials are serious about rooting them out they will need the same tools used to dismantle the Mob. The case provided here where someone commits something down on electronic mail is extremely rare. Phone calls, face to face conversations, and other means of communication, some subtle, some explicit, is where the action is. What can the NIH do about these?
The scenario doesn’t mention whistle blower protections. I agree that Miller will have to forever watch his back. For that sort of violation, a longer ban on serving as a peer reviewer for NIH (a decade?) would be a step in the right direction.