27 Comments
A series to raise awareness, encourage dialog and inspire creative problem solving for challenges in maintaining integrity in peer review.
What would you do if, as the Dean of Research at a major university, a group of students, postdocs, and junior faculty reported that they had been pressured into writing reviewer critiques for a senior faculty member?
We were so impressed by the careful handling of just such a situation by an institutional official recently that we wanted to share this story with you (we’ve changed details and fictionalized names).
Dr. Lee, Dean of Research at a major research university, received an anonymous, written complaint against Dr. Williams, a Distinguished Professor in her university’s medical school. According to the complaint, Dr. Williams was sharing NIH grant applications with members of his laboratory with requests for them to complete his written critiques as an NIH peer reviewer. The complaint indicated that Dr. Williams also pressured junior, non-tenured faculty in the department to do the same. All had been instructed by Dr. Williams not to disclose this practice or their evaluations to anyone else.
Dr. Williams had clearly violated the NIH confidentiality statement that every reviewer certifies before gaining access to grant applications for a review meeting. The agreement in part, prohibits a peer reviewer from ‘sharing applications, proposals, or meeting materials with anyone who has not been officially designated to participate in the peer review meeting, including but not limited to colleagues, lab members, fellows, students, applicants, offerors or employees of an offeror.’
In response to the complaint, Dr. Lee convened an investigation committee and analyzed Dr. Williams’ archived emails. During the investigation, Dr. Williams was placed on administrative leave and not allowed entry to his laboratory. Each member of the Williams’ laboratory and each junior faculty member in the department was interviewed individually by the investigation committee.
Dr. Williams stated in his interview with the investigation committee that his only intent had been to educate students, postdocs, and junior faculty in grantsmanship and peer review.
Ultimately, the investigation committee found that Dr. Williams violated the confidentiality of NIH peer review by sharing NIH grant applications and submitting critiques that were not his own. In addition, the committee found that he violated university policies through an “abuse of power” by pressuring students, postdocs, and faculty to perform his review duties.
Dr. Lee also notified the NIH Office of Extramural Research of the committee’s findings, out of concern for the integrity of NIH peer review. Because reviewers are recruited for their expert opinions, not those of their laboratory personnel, we assessed Dr. Williams’ recent reviews to see that sufficient, additional expertise had been available for each application to which he had been assigned.
Finally, the institution stripped Dr. Williams of his Distinguished Professor title; imposed administrative sanctions, including an agreement to add special award conditions on any current and future NIH grants in which he may be involved; and allowed members of his laboratory to relocate to other departments in the university.
Pressuring an employee or colleague to write a peer review of NIH grant applications on your behalf is simply wrong, unethical and a serious violation of NIH peer review integrity. We are glad Dr. Lee took the steps she did to handle the situation. What would you have done differently if you were in Dr. Lee’s position?
A suggestion for the future — use inclusive pronouns and initials instead of names to avoid injecting gender and ethnicity unless they are intended to be part of the story.
I was a staff member for three years for a senior professor at a well-known university. He was a statistician and I was an epidemiologist. Whenever he was on an NIH review panel, he asked me to read through all the grant review proposals and identify the parts that he should look at for statistical issues. To be honest, it never occurred to me that this might be a problem.
If NIH seriously wants honest review of grants:
1) In the process of grant revision, SRO’s command great power, I was wondering if there is any check on their partiality in selecting reviewers of study section along with study chair.
2) SRO and chair of a study section must rotate and should not be for more than 1 year on any study section, continuously.
Question comes to my mind, how does an experienced researcher/professor mentor students and junior faculty? By giving them the real task. It also serves as a learning both ways. Dr. Williams may have assessed all the reviews and submitted the best review in his expert opinion.
I don’t disagree that post-docs and junior faculty need mentoring in these tasks, but a truly creative mind could come up with methods for doing so that don’t blatantly and explicitly violate the NIH’s Confidentiality Certifications (for heaven’s sake, it’s the very first item in the certification).
In one way or another, this appears to be laziness on the part of the senior faculty member/reviewer: the individual either will not take the time to come up with creative mentoring strategies or (even worse) the individual feels entitled to assign this work to their “underlings” because they cannot be troubled to do the review work themselves.
It would be advantageous “to educate students, postdocs, and junior faculty in grantsmanship and peer review”. Perhaps the NIH could provide a few grants that cross multiple disciplines for educational purposes. The summary statements would complement the package. These could be used to teach about the grant writing and review process.
Not only are the senior investigators not performing their own grant reviews but often these same people do not write their own grants either; let alone conduct or analyze any of their own data! Yet, these senior people are held up as the de facto experts while the junior people who actually do the work and write the papers get little recognition for their efforts or expertise. What is worse, is that it is not uncommon for these same senior person to have their junior people do this work for them because they suffer from “imposter syndrome” and realize that they actually lack the required expertise to perform the reviews themselves!! Sadly the NIH has become a top heavy system composed of senior “thought leaders” who circle the wagons at meetings. The actions of these risk averse “thoughtless leaders” who often lack vision not only holds back the careers of junior investigators but it also impedes the funding of the truly novel research necessary for major scientific breakthroughs. The system needs a complete overhaul.
I am aware of a senior faculty member who likewise pressures lab members to write manuscript reviews, which are submitted under the name of the senior faculty. What are the ethics and legality of this behavior in the broader context of peer review?
1. NIH could set aside old and redundant grant proposals for Postdoc education that faculty could download for “teaching.” Asking postdocs to take a look at applications is a very minor “crime” as opposed to blatant plagiarism of grant proposals. Also, if the postdoc entirely reviewed the application, that is far more serious as opposed to a postdoc, for example, advising whether a cited technique is good or bad.
2. SRO concern is very important. SROs get along with certain reviewers and shun other reviewers. They do manipulate outcomes. SRO travel to meetings often has hidden agendas.
3. Pre-meeting of friends in the liquor bar discussions amongst reviewers before a study section meeting, in my opinion, is a big problems. Reviewers do gang up on favorable and unfavorable applications.
4. Personal phones and non-official emails would tell volumes about reviewer/NIH officials misconduct!
5. Believe it or not-race and ethnicity plays a major role in review; both positive and negative.
6. Vocal reviewers railroad meetings; sympathy factor plays a major role. Streamlining is a joke and encourages streamlining frenzy!
7. Scientific part of the application should be coded and anonymized. Administrative staff should review credentials, budget, institution etc.
8. Behind every successful or failed grant, there is a hidden story. In the cited example, it was based on an anonymous communication. For one reviewer “caught”, there are many others.
9. Look at scientific misconduct! It is all postdocs and junior faculty, when the PIs travel around, not available for daily consultations etc. In my 40 years of research, I never had a lab meeting (other than for journal reviews), as I was in the lab with them most of the time. I had my own bench and did experiments. Times have changed. Now there are PIs who go into their laboratories only to wash their coffee cups or to grab a paper towel!
I am wondering if same rules apply for review of manuscripts for scientific journals
The same ethics apply to journal reviews. One is supposed to keep the material confidential and not show it to someone else. If one considers it necessary to consult another person to assist with the review, one should indicate that to the editor so that the editor can extend the review request to said expert.
The current rules and points about confidentiality are clear, so the case presented here stacks the deck – presumably to try better to get the message out to those who contribute major time commitment to NIH for what, in effect, is a donation (or pay-back to NIH or society) and sometimes either masochism or a desire for power. Yes, read the rules; obey the rules!
That said, where NIH fails is in streamlining the process of diligent peer reviewers either getting help with the burden (technically, the stated rules forbid having an office assistant print out the application from its PDF), or enlisting some input from additional viewpoints or specific expertise where the quality of the review can benefit from an angle of genuine expertise that they lack, or in quality control to prevent critiques (dinging putative weaknesses) that are erroneous, etc. [In theory, SRO’s could program in a new name and have that person – IF they have an eRA Commons login, which office assistants and many trainees will not – do the COI cert. In practice, SROs are overloaded.]
By use of this particular example, the actual topic “Where is the line between abuse of power and actually doing something valuable for mentoring?” and, as several earlier posts note, “How about manuscript reviews?” For the latter, many journals already provide a path to recognition of the contribution of trainees to improving the quality of the process. As a trainee, I benefited immensely in my own training and development from having my previous thesis advisor seek my input (this was 35+ years ago) on review items – but the reviews always were his (and, in my case, the same it true when I ask trainees if they would want the chance to pick over a submitted manuscript ….). As to the “line” (abuse vs offering opportunity), well, even with knowing that perceptions usually differ on the two ends of a power asymmetry, I doubt there is a clear line when it comes to manuscripts. That in itself is training since part of any profession or career is learning when and how to say “no thank you” versus do a bit extra. And, yes, trainees would benefit from needing to go through a “live fire” training by reviewing current grant application so if NIH is sincere about wanting to promote the careers and development, they actually should fix this impediment.
The old “you get what you pay for” axiom applies. There are some NIH reviewers who just don’t spend enough time on this critical task, which is evident when their critiques show that they didn’t really read the grants, or from receipt of two scores that are outstanding and one that reeks, which happens amazingly often. This task needs to be taken more seriously – people’s careers are at stake! If someone volunteers to do it, then he/she needs to find the time and do it right, or resign from the panel. Also, SROs need to crack down on slackers and any hint of bias, power plays, or inappropriate decision rendering, and they should be held accountable if such shenanigans continue. Reviewers should be removed from the study section if they perform poorly. A better source of impartial reviews would involve a panel of independent scientists who are professional grant reviewers (their only job, eliminating conflict of interest). They would be paid well and evaluated for their ability to familiarize themselves with the key literature and dig into the nuts and bolts of each grant, following a detailed rubric of points to evaluate. They would be charged with writing fair, thorough reviews that provide helpful suggestions as well as point out strengths and weaknesses. (I know, this is what current reviewers should be doing, and many do, but there are still some bad players screwing up the system). Revised grants would come back to the same reviewers, which should minimize the bizarre phenomenon of an improved grant receiving a poorer score. Obviously this is not financially feasible, so until then we can only hope that reviewers have the integrity to perform at a professional level. And for other reviewers to speak up when they see malfeasance.
I believe many are missing the situation. You cannot defend the Sr. researcher for trying to educate them. Education would not require pressuring. The issue is simple in my opinion “abuse of power” and laziness. The task may have educated those that performed the review, but that only would be the case if they discussed the reviews afterword, but that was never mentioned. So education was trying to justify what they were doing to deflect the real problem. Remember, the researcher is a smart person and will try to deflect attention away from the problem. If anyone has ever had to deal with fraud you will quickly notice when people are trying to deflect and at that point, the real answer is usually an ugly answer. In my career, I have witnessed several instances where accomplished researchers get a pass on doing questionable activities because you just don’t question them because they are the researcher everyone wants to be like. In research, there are several processes that should be reviewed and changed.
It would be useful to know how NIH CSR addressed the fact that NIH grant submissions were reviewed by unassigned reviewers who in this case are unqualified to review the submitted proposals. It is incumbent on NIH to notify the P.I.s of submitted proposals who might not have had their proposals funded on account of this inappropriate review. Why is this obvious problem not addressed in this article?
The sad truth is that the PI is the one who is often unqualified to perform the review. That is often why they hand the review off to the junior person who is more familiar with the science and literature on the subject. The entire NIH grant review process has become a joke.
The NIH considers many factors and options for resolving these situations. The timing of the incident makes a difference in our response. For example, if the critiques and initial scores from this reviewer have not been posted and therefore other reviewers have not seen them in our secure review site, one option would be to remove the reviewer before the review and assign another. If critiques and initial scores had been posted and viewed by other reviewers, the application may be deferred for review in another review panel or withdrawn from consideration. In some cases, the allegations are received long after the incident occurs and the application is no longer active in our files, as often happens for allegations from students who wait to report their concerns until they have left the individual’s domain. If the records are no longer in our files, then we have limited options for pursuing the allegation or verifying the information.
Please address my question rather than change the subject. Thanks.
Manuscript reviews can be co-reviewed by both a senior faculty member and mentee if this arrangement has first been cleared with the editor who is handling the review. I have done this with postdoc trainees, after clearing it with the journal. I tell the postdoc that we are both going to independently review the manuscript and then compare notes and discuss the comments and how we present them. All of this is for training purposes – I submit my own review, although occasionally the postdoc will make an important good point that I add (and I also note in the private comments to the editor).
Grant reviews cannot be done the same way. For training purposes, it can work to ask colleagues to share proposals from the last year or so (both funded and not funded), and use those as training for grant proposal review. That produces a biased set of grants to review (because more successful grants proposals will be provided), but it is a start.
In the process of grant revision, SRO’s command great power, I was wondering if there is any check on their partiality in selecting reviewers of study section along with study chair.
2) SRO and chair of a study section must rotate and should not be for more than 1 year on any study section, continuously.
Gotta admire how this person unfrozen-caveman-lawyered their having subordinates do their NIH grant reviews into trying to provide a professionalization opportunity
I have done this with postdoc trainees, after clearing it with the journal. I tell the postdoc that we are both going to independently review the manuscript and then compare notes and discuss the comments and how we present them. That is often why they hand the review off to the junior person who is more familiar with the science and literature on the subject. retaining walls
I have done this with postdoc trainees, after clearing it with the journal. I tell the postdoc that we are both going to independently review the manuscript and then compare notes and discuss the comments and how we present them.
All have discussed what happens to the PI in case of breach of confidentiality. What action would NIH take on the postdoc or student who wrote the grant review / summary on behalf of the PI?
Did anyone ever try to report to the email address: [email protected]? Apparently the email address doesn’t exist and gmail is unable to deliver my message. Long live America! No wonder China is now brave enough to show USA The Finger ha!!
The correct email address is [email protected]
A suggestion for the future — use inclusive pronouns and initials instead of names to avoid injecting gender and ethnicity unless they are intended to be part of the story.