16 Comments
Dr. Richard Nakamura is director of the NIH Center for Scientific Review.
Imagine this: you’re a reviewer on an NIH study section, and receive a greeting card from the Principal Investigator (PI) on an application you are reviewing. A note written inside the card asks that you look favorably upon the application, and in return, the PI would put in a good word with his friend serving on your promotion committee. Do you accept the offer, or just ignore it? Or, do you report it?
Or this: a reviewer on an NIH study section finds that one of his assigned applications contains an extensive statistical analysis that he does not quite understand. So he emails the application to his collaborator at another university and asks her to explain it to him.
Or what about an investigator who submits an appeal of the outcome of review, citing a particular reviewer as having told him that another reviewer on the study section gave a critical review and unfavorable score to the application out of retaliation for an unfavorable manuscript review?
Or maybe several days after the initial peer review of your application, you receive a phone call from a colleague you haven’t spoken to in quite a while. The colleague is excited about a new technique you developed and wishes to collaborate. You realize the only place you’ve disclosed this new technique is in your recently reviewed NIH grant application. What do you do?
Scenarios like these are thankfully few and far between. Given the size of NIH’s peer review operations, the rarity of such scenarios is a testament to all you do in supporting the integrity of peer review, and the public trust in science. Nevertheless, reminders are helpful, and it’s important to be prepared and understand your role in upholding the integrity of NIH peer review, just in case you are ever put in a situation like the ones described here.
While professional interactions between applicants and reviewers can continue while an application is undergoing peer review, discussions or exchanges that involve the review of that application are not allowed. As an applicant, you should not contact reviewers on the study section evaluating your application to request or provide information about your application or its review, no matter how “trivial” the piece of information may seem. As a reviewer, you should not disclose contents of applications, critiques, or scores. Reviewers should also never reveal review meeting discussions or associate a specific reviewer with an individual review.
Why are these responsibilities important? Because supporting the public trust in science takes the support of the entire research community. Attempts to influence the outcome of the peer review process through inappropriate or unethical means result in needless expenditure of government funds and resources, and erode public trust in science. In addition, NIH may defer an application for peer review or withdraw the application if it determines that a fair review is not feasible because of action(s) compromising the peer review process. Depending on the specific circumstances, the NIH may take additional steps to ensure the integrity of the peer review process, including but not limited to: notifying or requesting information from the institution of the applicant or reviewer, pursuing a referral for government-wide suspension or debarment, or notifying the NIH Office of Management Assessment.
Your responsibility doesn’t end there. All participants in the application and review process, including investigators named on an NIH grant application, officials at institutions applying for NIH support, and reviewers need to report potential breaches of peer review integrity. Immediately report any peer-review integrity concerns to your Scientific Review Officer. For peer review activities within the Center for Scientific Review, you can also send an email message to csrrio@mail.nih.gov. If you need to report an incident to someone outside of CSR, you may email the NIH Review Policy Officer. We also provide additional resources on our Integrity and Confidentiality in NIH Peer Review page, and encourage you to share this resource, and this blog post, with your peers, colleagues, and trainees.
It is commonly known that the well connected are the ones who benefit from the proposal-review-funding excercise. I.e., I take care of you when I serve and then you can take care of me, all in an unexpressed manner. Also, I have heard 3rd parties talking about what a reviewer said about a specific application. Where is the confidentiality or integrity of the review process in such actions. Many of us unconnected, mid career people just feel a hopelessness with a skewed system. HELP!!
The best thing we can do is to demand two things:
1. A double-blind peer review process
2. A viable appeal procedure instead of the current pretense, where Councils don’t even bother to address specific issues raised in appeals.
It’s hard to see how a “double-blind” review process would work unless reviewers who are well-acquainted with an applicant’s work, and thus are probably the best-qualified to review it, are obliged to recuse themselves from reviewing the application. To put it another way, should an application be assigned only to “naive” reviewers who are ignorant of an applicant’s relevant prior research? Not a good idea.
Can be done. The review process should be divided into two stages: a double-blind study section review of proposals consisting only of the background, aims, significance and proposed approach, and then a final “go/no go” by the Program Officer, who would verify the PI’s experience, facilities, resources etc. for applications pre-selected by the study section. Sure, competent reviewers would still be able to make educated guesses about the identities of the PIs, but they would only be guesses.
Agreed. A fairer peer review is possible – more difficult – but possible. It’s a question of will and self-interest. Those in power who will likely make decisions to overhaul the currently broken system are those most likely to lose if it’s overhauled.
DoD CDMRP already does double-blind peer review for some of their grants. Can be done, has been done, and should be done at NIH.
Perhaps, the best approach is OPEN TRANSPARENCY in the review process, whereby the reviewers are made known to the PI, and issues cited are directly addressed to known reviewers.
I appreciate Dr. Nakamura’s reminders of this very important topic. I would suggest one step further, actual penalties for violating the confidentiality of peer review. If an applicant is dishonest in constructing a grant proposal, there are serious repercussions, even the possibility of criminal prosecution. Reviewing and keeping review information confidential is just as important and critically important for the success of a stressed review process.
All due respect to Dr. Nakamura and others, but the truth of the matter is that our so-called “merit-based” peer review system is badly BROKEN and in need of serious OVERHAUL. Those with connections and deep pockets are gaming it. Study Section reviewers are being wined and dined and given honoraria for talks in return for implicitly assumed favors. There is an old saying that Big Pharma wouldn’t wine and dine doctors if they didn’t believe they were getting some financial benefit from doing so (doctors prescribing their drugs). Same holds true here. That’s why so few get so much of NIH tax payer -subsidized money, and these rich and powerful are getting richer!
A fundamental tenet of our legal system is being judged by a jury of our peers. This system works ONLY because the legal parties (defendant and plaintiff) cannot in any way influence or try to influence the jurors. If they do (like inviting a juror to dinner, or giving him a speaking fee) it’s called jury tampering and it’s a crime. The NIH grant peer review system allows grant applicants have free reign to build relationships with peer reviewers (the jurors), and if they have money, to legally bribe those reviewers. Give it whatever euphemistic term you like – networking – it doesn’t change the fact that it’s immoral, unethical, and should be illegal.
The NIH peer review system is broken in a more subtle way. I was invited to study sections, and nobody was trying to bribe me. I did, however, notice a bias manifesting itself as a tendency to gloss over technical deficiencies in applications submitted from affluent institutions. Also, ethnic biases are recognized (“Race, Ethnicity, and NIH Research Awards”, Ginther et al., Science), and biases against non-academic investigators have even been a subject of a Congressional hearing (Dr. A. Pilon’s testimony in 2009 – evidently to no avail).
Yes, something needs to be done to prevent study sections from playing funding politics and to make sure that the taxpayers indeed fund the best research (and receive the best return of the investment possible). And the anonymity of reviewers has nothing to do with the problem.
True. I have also heard comments on review panels such as “applicant is one of ours” and “he was a graduate student of mine” without the SRO even raising an eyebrow. In the case of the latter, the applicant and reviewer had indeed worked and published together 20 years ago, and the applicant had recently been invited by the reviewer as a “keynote speaker” to the medical school of which the reviewer is now the Dean. When I privately approached the SRO about this conflict of interest, she brushed it aside saying that according to CSR criteria, this did not represent a COI. When will such cronyism stop?
“When will such cronyism stop?”
Either a double-blind review process, or its complete opposite: a full disclosure. Actually, I am leaning toward the latter. On a personal note, I take a full responsibility for what I write, and I do not mind at all signing my name under my critiques. The next time I am asked to review something, I intend to do precisely this. Enough of cronies and pure imbeciles hiding in the murky waters of commonly accepted anonymity!
I cannot agree with Greg Czyryca more. The review system is terribly broken and needs to be fixed. However, although I have personally raised these issues to the NIH, they have fallen on deaf ears. The review system should be based on the merit of the science proposed and the applicant’s standing in and contribution to the proposed field which would foretell the likelihood of success not on “Resource sharing plans”. In fact I am adamantly opposed to the latter, because it does not work. I have shared my reagents to satisfy the NIH policies, but never have I gotten even a reply when I asked for reagents!
Perhaps this question is naive, but how is the reviewer pool actually determined by the NIH? How is “expertise” per se of the reviewer pool calibrated? What role do Universities play on ascribing or referring individuals to the reviewer pool, and what role does the NIH play in vetting those referrals? Is it possible that declining and mediocre global scientific impact is a consequence of poorly vetted reviewers without scientific vision?
Laughing at, ‘Scenarios like these are thankfully few and far between.’
Glad for that too. Would be horrible if we had 10% of applicants hoarding ~40% of the funds. These ‘super grantees’ are funded because they’re amazing scientists, curing diseases at breakneck speeds and 10-100x more productive!!! Not because of their abilities to act like MBAs, rather than MDs/PhDs.
Thankful everyone plays by the rules!!!
I’m late to this thread but this problem of review bias is rampant in NIH review and has been for decades. A fundamental problem not yet mentioned in this thread is the practice of keeping reviewers on the same study section for years. Both of my closest competitors serve on the only two relevant sections – it makes no difference if they step out of the room while my grant is discussed, or even afteer they rotate off – everyone else knows they are also in this area of work, and the natural friendships and networking that evolve among section members over time leads to a reluctance to fund competitors of those they are serving with. This is only human nature and to pretend it does not happen is nonsense. Reviews should be conducted more like DOD system – every panel is a new panel, and the names of review participants are strictly confidential. DOD also successfully uses double blinding – yes you can still guess who wrote the grant but its a step in the right direction. NIH’s system has bred and nurtured a culture of bias and rich-get-richer. This system rewards incremental research and quelches innovative new ideas put forth by investigators who don’t have influential connections.