New Efforts to Maximize Fairness in NIH Peer Review

Posted
Dr. Richard Nakamura, director of NIH's Center for Scientific ReviewDr. Richard Nakamura is director of the NIH Center for Scientific Review

We want you to know NIH is working on multiple fronts to get to the bottom of unexplained racial disparities in R01 grant funding and to maximize fairness in NIH peer review. Since the problems and the solutions are bigger than NIH, we have reached out to the scientific community and other concerned citizens for help. Now armed with a team of experts and a set of new initiatives, we’d like to tell you about our efforts to address this important issue –- particularly an exciting opportunity for you to submit your input.

Earlier this month, NIH’s Center for Scientific Review (CSR) launched two America COMPETES Act challenges to help identify new methods to detect bias in peer review and strategies to strengthen fairness and impartiality in peer review. We will award a first place ($10,000) and a second place ($5,000) prize in both competitions. The contests close June 30, 2014, and winners will be announced September 2. You can find details on the rules and submission procedures for these two challenges on the CSR Challenge website.

In addition to the competition, a complementary set of initiatives will allow us to look at the problem from multiple angles. These initiatives include:

  • Providing new investigators with opportunities to participate in survey and focus groups to share their ideas about the fairness of peer review and the challenges they have faced in attempting to secure grant funding.
  • Testing — over the next year — the utility of anonymizing grant applications prior to review and conducting qualitative studies to gain a richer understanding of the scientific and technical issues that may play a role in funding disparities.
  • Continuing to examine the process by which reviewers evaluate grant applications for indicators of bias.
  • Continuing the Early Career Reviewer Program to provide opportunities for up and coming researchers to jump-start their careers by serving on a review panel. To date, more than 3,000 researchers have been admitted to the program and more than 1,000 have already had a chance to serve on study sections and report the experience as being quite valuable.
  • Launching initiatives to enhance the diversity of the NIH-funded workforce, as discussed in earlier posts on Rock Talk.
We could not have made it to this point without the exceptional support and guidance of the NIH Advisory Committee to the Director, specifically its Subcommittee on Peer Review, which grew out of its Working Group on Diversity in the Biomedical Research Workforce. As we move forward, we will continue to seek suggestions from the scientific community on how we can do the right things to understand this important problem and maximize fairness in NIH peer review.

 

84 Comments

  1. Numerous studies suggest that bias against minorities exists across the social landscape in the US, from baseball umpires reducing the strike zone for black pitchers, to black male pedestrians having to wait longer for motorists to yield, to academic faculty showing preference for white/male staff and trainees. If these studies are correct we should not be surprised that the same happens in the NIH funding paradigm. I would encourage you to not only look at how bias influences the review process, but examine how minority applicants fair in the more ambiguous internal processes at the NIH including application pick-up in the “gray zone” by program officers/institute officials.

    1. It is very troubling to learn how NIH grant is awarded. The initial ranking to fund is based on the average of overall impact scores NOT based on the average of 5 criterion scores (Significance, Investigator, Innovation, Approach, and Environment) from all the reviewers of the application. In fact there is no calculated formula between the two. The initial ranking is based on the average of overall impact scores from 3-4 assigned reviewers (e.g., average score 5, from three reviewers with scores of 4, 5, 6). If initial score of 5 is below 50% of all applications, then it is not to be discussed (designated as ND) and not funded no matter how good the criterion scores ranked and what details were provided on the Summary Statement to the PI. Rather Overall Impact is based on reviewers’ assessment of “the likelihood/probability of a project to exert a powerful and sustained influence to the field of research”. So the funding seems, it can be/probably often is, ultimately decided based on reviewers’ personal opinion and who They Like to Fund or Want To Fund. For maximizing fairness of peer review process, I believe the second review assessing Impact which decides the funding should be eliminated or be made transparent. If Impact assessment is believed to be separate from Significance criterion then it should be added as a new criterion to be scored and written up in the Summary Statement by each reviewer. In addition, as others have pointed out, all application should be “blinded” then reviewed, Finally, Personal Biosketch and Environment should be part of JIT submission to be checked by NIH staff to ensure that the proposed work can be done.

      1. The variability and subjectivity of NIH reviewers is so sad. But at the very least this has be more visible. Please make the overall impact scores available in the summary statement – several reviewers choose to score everything high and give a low overall score, this key score is hidden from applicants. Please limit to 2 R01 grants per investigator to increase junior investigators to get funded.

  2. To suggest that reviewers are biased only serves to anger those who chose to give back to NIH through conducting reviews of applications. For every review panel I sit on, I spend 60 hours at least reviewing applications when I could be spending that time writing an actual application. The NIH reviewing system introduces bias in that it requires reviewers to evaluate the ability of the PI to carry out the aims. Thus, NIH should know that they keep funding the same PIs/labs over and over again because the people that have money also have the time and staff to continually write new applications. Those PIs are primarily white men. If the NIH just took a percentage of every multi-million grant they gave to a continually funded research and put it in funding for new/young investigators, there would be more minorities and women in the system. They just don’t want to look at their own practices . . . it’s not the peer review and to suggest as much will only result in their losing peer reviewers who are upset when their academic and scientific integrity is questioned.

    1. I strogly agree. I’m also offended by the suggestion that study sections are biased towards minorities.

      The hard truth is that minorities have fewer academic jobs and run continuously successful labs. Those are (we like or not), white men. So, the bias occurs way before the application reaches the study section. Besides, I doubt anyone at study section has the time to check if the PI is a minority or not.

      What about using the scientific method to test if bias against minorities occur during grant review?

      1. I am embarassed by these tone deaf arguments. This isn’t a personnel attack by NIH on reviewers. It’s a reasonable response to troubling concerns raised byd unbiased analysis of funding success rates when other factors (publications, prestige of institutions and training) were included. See discussion of Ginter’s 2011 in Science 2011 http://www.sciencemag.org/content/333/6045/1015.abstract This study says we are biased.
        Smaller studies followed but none with the rigor of the original analysis.
        Many of the comments here and elsewhere are deeply offensive. Its outrageous that to argue that this is painful and doesn’t feel true and therefore shouldn’t be a priority of NIH. Your feelings may be hurt, but these are people whose careers are being hurt. Step outside yourself and imagine being a minority, having a data to support your concerns of bias and then coming to a forum and having people getting in a twist as if the data doesn’t even exist. We can and should be doing better.

  3. As a reviewer from an under represented institution, I understand the importance of anonymizing the applications to prevent bias and focus on the scientific merit of an application. However, this is nearly impossible given the need to reference prior works to provide feasibility data given the restricted page limits of current applications. While difficult, it would be more successful to find reviewers that are not associated with the applicant in a broader sense. Perhaps having East coast applicants reviewed by West coast reviewers and vice versa.

  4. As a long-time NIH reviewer, I doubt very much that there is a lot of bias within study sections against applicants based on their race. Reviewers simply have too much to read to bother with the last page. I suggest that the biased outcomes originate before the application is written. In these very tough funding times, the selection bias within review panels is not against anyone, but is FOR scientists who are “big names” in their fields, people who hold leadership positions at their home institutions, and editors of leading journals. A stellar application is a must, but when there are too many stellar applications to fund them all, the bias goes to the people considered to be at the top of their field, people that the reviewers may even fear to alienate lest their own future papers, grant applications, and careers suffer.

  5. Please be more courageous in explicitly considering all resources available to an applicant when evaluating productivity. Good people do try to give minorities a chance but very often the non-mainstream applicants get smaller awards, fewer awards, shorter awards or jobs at institutions that require more duties of their faculty (provide less possibility to focus on research). These several small disadvantages do not simply add up, they multiply.
    1- When counting total number of papers published that credit a grant being renewed, please also look at productivity on a per dollar basis as well as on a per-person basis in the lab considering total dollars and total personel contributing to that productivity. This will provide a second perspective that permits us to see how efficient the smaller labs are with the total of the resources we give them. (eg. a person with a single grant cannot spread the costs of instrumentation over multiple awards, may have to settle for sub-optimal methodology and consequently be limited to publishing in a slightly less prestigious journal. His/her students will be working part-time as T.A.s and therefore only part-time on research.)
    2- Please look to see how long the P.I. has had on the expiring award when evaluating productivity. This is especially critical now that many smaller labs only have a single grant. A three year award will require that the applicant submit a renewal request after two years in which the first was spent substantially on training graduate students new to the science. A five year award will allow the applicant to submit a renewal application after four years in which the first was spent substantially on training. This constitutes twice as many ‘highly productive’ years. In all groups, time will have to be committed to training but it becomes much more efficient when senior students are available to train incoming students. Thus overall productivity is enormously aided by the existence of a second award that allows the lab to retain its personel if the first project’s funding lapses for a while (which has become more common).
    3- Please consider number of non-overlapping grant applications submitted when tallying productivity. Productivity is often considered to include peer-reviewed publications and patents only however a lab that has lost funding will submit many grant applications in a year and this writing will take the time that might otherwise have gone to submitting publications. Counting publications alone produces the self reinforcing conclusion that groups that got better funding scores publish more papers. No kidding – but the others were sending out more grant applications.
    4- More researcher time and resources appear to be devoted to compliance-related work than in the past. At some institutions staff have been assigned to help with this but at others this work is performed by faculty, postdocs and students being paid as is they were conducting experiments during that time (yes these activities are research-related but no, they are not themselves research). I would like to see numbers on this phenomenon so we can look at the implementation costs of these programs. This will allow us to evaluate their benefit on the basis of value (benefit divided by cost) and ask if there might not be more efficient ways of deriving the same benefit.
    X- We certainly want to reward institutions and groups that have assembled superb research environments able to tackle large questions however it is not possible to predict where a discovery will occur and making sure we nurture a diverse array of talent, thinking styles and research approaches is critical to retaining a truly multifaceted attack on the challenges we face.

    1. Truly thoughtful and meaningful comments from one obviously very experienced with the struggle. Institutional picking of winners and losers is now an established policy of our 1% society.

    2. Well said and wonderful post. Thank you for being so honest. If we could get study sections to be on board with these realities I think it would vastly help. The reality is that the many one-main-grant labs have suffered sequestration and administrative cuts yet are held to produce the same science – in many cases they are producing the same relative per dollar. With personnel costs (mostly benefits) ever increasing and institutions cash strapped to pick-up more salary/costs this really disadvantages the small labs. The well established/connected PIs – the “1%” have access to non peer-reviewed foundation grants that often double fund the same science that most mainstream investigators do not. Thus it would be fair to consider the amount of science produced per total dollars spent when making peer review critiques. I fear for a scientific landscape where only these larger entrenched labs survive – it certainly doesn’t help newer, younger, and underrepresented investigators – and I think limits the new discovery potential significantly.

    3. I could not agree more that the section evaluating environment is totally ludicrous – I often get comments that I am in a great environment but then score a 2 – how do I get a 1? Do I have to be at Harvard? And even if I do, why is that such a great environment for junior investigators? If I have the requisite, space, equipment, resources. and collaborators, my physical location sould be irrelevant. IMHO this section should be replaced with a simple “Satisfactory/Unsatisfactory” option.
      And in the limited NIH reviewing I have done, I personally witnessed bias towards very senior investigators by senior reviewers who paradoxically hold them to a LOWER standard than new PIs – the attitude seemed to be that we should just give X the money because he is X and will therefore by definition do good research with it even if the grant is lousy en face. In one panel that I was on, the only Professor level reviewer scored every section 2-3 points higher than the rest of us for that reason. If you want a fairer system, let the junior investigators who have sympathy for the plight of their peers review, instead of very senior PIs with multiple grants who have no concept of how bad things are these days. This is how we run AHA Study Sections and frankly we get much fairer reviews.

      1. Changing the environment score to pass/fain=satisfactory/unsatisfactory is a good practical suggestion.

    4. You have struck on a key issue. Productivity is such a critical factor used in judging proposals, but there is no consideration of the resources available. This makes what people on study section call “productivity” meaningless. Yet, people brandish the club of productivity as though it is somehow “the truth.” Mind you, well funded investigators will never back any attempt to measure per dollar productivity.

    5. In one of my last reviews – Under “Investigator – Weaknesses” a reviewer wrote: “It might be considered that it is a weakness that the group is small. However, it is very effective.” This seems to acknowledge that there is bias against small labs. The reviewer scored Investigator 1, but the project was not funded – ironically, keeping the lab small.

      1. An excellent example of a ridiculous comment – just for this type of comment the reviewer must be expelled and never invited again, but guess what – I am pretty sure s/he is still around…

  6. As I have written before, my concern is with the lack of accountability for making out-and-out errors in reviews of grant applications. These errors include incorrect assumptions about the literature or the science that have no empirical basis, and also include incorrect statements about the application itself (e.g., stating that a topic is not covered that is actually covered in the application). Such errors may reflect bias if they are more likely to occur in reviews of applications from under-represented groups. However, the errors are also likely to have a random element in them, arising if reviewers simply don’t do a careful job, have a strong belief that they are always right without fact-checking, and are always pressed for time. I have observed these errors in reviews of my own applications and those of my colleagues many times. I am an experienced investigator who has been continuously funded by NIH since 1990, so my concerns do not arise out of a spirit of “sour grapes”, but rather, from the desire to see an improvement in the review process.

    I think that careless reviewer errors will not abate unless there is some level of higher-level random spot-checking of reviews for accuracy (even if on only a tiny percent of a percent of reviews), with reviewers all made aware that such a spot-checking will occur and that there will be some sort of consequence for them if factual errors are found in their reviews. This type of quality assurance goes a long way towards improving what is done in a lab, and should work equally well for the review process.

    1. Well said. I think NIH and all other funding agencies must find a way to hold reviewers of grant applications accountable for their comments.

    2. I’ve reviewed (not at NIH, but many times), and you’re right about reviewers making errors. I can only speak for myself and what I’ve observed on different panels. The kinds of errors I’ve seen tend to fall into categories:

      1) “My-hair-is-on-fire” mistakes. These errors happen because reviewers are assigned a large number of applications (say, a dozen or MORE) and only get 2-3 weeks to read them. Maybe the NIH is different; I don’t know. But where I was reviewing, I started to feel like I was under siege (I wasn’t alone).

      It goes like this: you have other responsibilities, your RA needs some help that will take at least an hour, and you have a meeting later. S-T-R-E-S-S! Under those (typical) circumstances, doing a slow and thorough review of 12 applications is a serious challenge. I suspect that this problem is getting worse, especially because budget cuts may force agencies to reduce the number of panels they convene. When I started, I’d get 10-ish applications; the last time I reviewed, the number was 14, and it climbed steadily over the years.

      2) “These-people-do-such-great-work” mistakes.” I saw this problem reasonably frequently, but NOT on every panel I was on (maybe a third?). The reviewers give too much credit to applications with obvious problems because they know that this particular PI’s group has done good work. It’s a very hard one to fight, even if you can point out obvious errors in the application.

      3) “I-don’t-understand-the-math” errors. These errors tend to happen when the authors use some kind of mathematical analysis that isn’t understood by every reviewer. IMO, grantmaking bodies should have statisticians or others with appropriate knowledge review the mathematics in an application.

      There are more, but that’s general idea.

  7. As funds have become tighter, the review process for grant proposals has grown increasingly risk averse. Money has concentrated more and more into fewer labs where Principal Investigators oversee large research endeavors with dozens of fellows, technicians, and research assistants or even more. The reasons for this are not hard to understand. Big enterprises can output more papers and more presentations which are one important sign of success. While one could claim that such large labs perform at greater efficiency, because of their size one could also argue that some of these labs produce smaller marginal gains as a result of the tight focus in techniques and approaches prevalent in such labs.

    Smart, innovative proposals from smaller, more nimble laboratories get shut out of this process since the magnitude of their overall research output is smaller than the big guys. It is a trend that many believe will stifle innovative and important research if left to continue. So what is the solution short of significant elevation of the civilian research budget? Review panels consisting of scientific peers are instructed to grade proposals on merit only – it’s not their job to exert new policy directions to this process nor should it be. Large multi-year grants given by the NIH such as the R01 award, is the primary key to maintaining American technological competitiveness. So how does one allow smaller research operations to prosper without penalizing larger operations simply because of their success? In essence, how does one spread the critical resources of a static or shrinking research budget? My proposal here is that any PI that has two or more of these large multi-year funding vehicles gets reviewed by a different review panel. Hence, the big guys will compete with each other on more equal turf. The final funding approval process still remains within the NIH who can decide how the overall funds get divided between the two pools. But it provides the opportunity for more innovative or risky proposals, many from smaller research labs, to be assessed without being penalized for the very research characteristics that the NIH wants to nurture and from which large scientific gains, if successful, can be made.

  8. I agree entirely with Anne-Francis Miller. As a reviewer, I have been chastised for mentioning funding levels when I remark on a lab’s outstanding science in the face of low and intermttent funding. Similarly, we are not permitted to comment on the high funding level for HHMI investigators, but rather rely on a “nudge-nudge-wink-wink” system. If reviewers were allowed to consider productivity relative to the types of grants awarded to the lab, it would help to level the playing field.

  9. Director Nakamura,
    1) Seems like it would be a good idea to first talk to a sample (or all, the numbers can’t be that high) of successful and unsuccessful African-American PIs and see what they think. This is your “hypothesis generating” Aim.

    2) Shouldn’t the processes you went through in generating the ESI policy be helpful here? How did you assess the bias against younger/newer PIs? Why can’t that be used for this application?

    3) Inter-rater reliability is a tried and true solution in experimental psychology. It should be the first order of business to have the same applications reviewed in two or three panels. Start with panels convened by the usual process. Then start manipulating panel makeup to test whether it can reverse biases.

  10. As someone who was trained in a different discipline, and came late to the NIH-funded world, I feel that there is a huge learning curve of years of specific knowledge as to what makes a successful NIH grant application, even if one already has the science piece to do top research and write top papers. If NIH really wants to diversify its set of successful PIs, then my guess is that rather than focusing on peer review of already written applications, they should intervene earlier in the process. In other words, they should instead create incentives or even formal mechanisms to have successful PIs mentor new investigators from under-represented groups through this process. My guess is that it is the ability to obtain this kind of focused senior PI mentorship that is needed across whatever group is deemed under-represented by NIH, where I am using under-represented in the broadest sense, i.e. whether you are concerned about demographics of the PI, of the institution, or are looking at particularly under-represented research areas that you want to encourage more of. How about a new grant mechanism that is jointly awarded to a senior and junior co-PI, where portions of the grant get written by both in a structured fashion (it’s really important that their are designnated sections that each get to write, rather than allowing the whole grant to be written by one or the other). The senior PI is then incentivized to mentor these junior PI, and in the process of writing the application together, the under-represented junior PI learns how to write a successful NIH application. NIH could even offer to do some of the matching for senior PI who would like to be involved but do not know an appropriate junior PI to partner with.

    1. I think this suggestion for targeted mentoring is the best one yet. As a reviewer myself, I also think that most of the disadvantage to underrepresented groups is occurring prior to the peer review process. I have served on NIH study sections for many years and actually pay very little attention to the number of papers the PI has published or what journals they were published in. I base my scores almost entirely on the quality of the proposal in front of me and have the impression that most of the other reviewers do as well.

  11. Absolutely shocked by this – I believe I have done over 100 study section meetings and have never seen evidence of this. This is insulting to reviewers. Just another nonsense initiative by the NIH!!!!!

    1. I could not agree more with this comment. It is absolutely insulting to be accused of review bigotry. Please tell me why I should continue to give up my time to perform peer review if this is the kind of mindless drivel spewed by CSR?

  12. I appreciate the Director initiating this discussion. I think doing in purely in the context of ethnicity is short-sighted. The bias in review hurts all new investigators and those not coming from well-established labs i.e. mentors have long standing RO1/PPG grantees and sit on study sections therefore you have to keep them happy. The grants process have devolved wherein it is not about exciting new research but about stodgy plodding in the same direction with minimal incremental information about the same topic that we have worked on for 20 years. This feeds the current system of senior PIs “stealing” ideas and using ghost written grants to feed the RO1 counting machine. No serious PI who is actually spending time on a project can honestly handle more than 2 disparate topics (read RO1s). The NIH review system needs to be gutted and redone. It is a vestige of the 1970s. This would never happen in other professional arenas where ideas and competency count not the name of the PI; a well written new investigator grant has a fair to middling chance; a poorly written established PI grant will most likely get funded in the second try.

    1. “a well written new investigator grant has a fair to middling chance; a poorly written established PI grant will most likely get funded in the second try”
      This comment is a insult to the integrity of peer reviewers and suggests that the commenter has not served as an NIH peer reviewer him/herself. There are many thing that might bias a reviewer in favor of established PIs, but poorly written proposals will always be poorly scored.

  13. I don’t understand why NIH people, the super-smart cohort, don’t focus on the “big” picture, which is the lack of funding. No matter how many tricks the NIH leadership rolls out to showcase that they are working really “hard”, the status quo still remains that all researchers are still suffering from the low funding rate. Please. focus on the “big” picutre as your past mentor had ever taught you as a promising young scientist.

  14. I have never once considered the ethnic origin of an applicant in my reviews, and I cannot imagine I am an outlier. Disparities in funding originate well before the review process. Fixing it requires a deeper investment, not tweaking.

  15. Director Nakamura writes “We want you to know NIH is working on multiple fronts to get to the bottom of unexplained racial disparities in R01 grant funding”, with the implication that a major determinant derives from inequitable reviewing of grant applications by peer review bodies. This remark would seem to overlook a terrible problem over which NIH peer reviewing can exercise little or no control: in my observations over the years, the representation of under-represented minorities at scientific meetings has actually progressively decreased over the past two decades, with a rare (>1%) person of color often appearing amidst the crowd of meeting attendees. Need one look further to determine the fact that persons of color — specifically African-Americans–are simply choosing not to enter into fields of basic biomedical research? Is this not the proverbial elephant in the living room?

    1. I too have been struck by the low attendance of minority scientists at national meetings. There seem to be so many well-intentioned outreach programs, yet so little return. Perhaps a choice is being made my talented minority individuals to pursue medicine or engineering rather than biomedical research.

    2. There is an elephant in the room, but it isn’t that minorities are not interested in biomedical sciences.
      Enrollment of pre-college and college STEM programs, and students from MSI testify against that false notion, big time. Have you ever attended ABCRMS? http://www.abrcms.org/
      Hundreds of students from across the US and territories representing African-American, Latino America, Asian American and Native American STEM (primarily biomedical science researchers).
      There is interests, but what I’ve experienced missing is a more holistic approach to recruitment and retention of students from under-represented groups at MOST universities. A handful of colleges do an amazing job of preparing students for post-graduate study and careers. Many do a pretty good job, and there are so many that have a laissez faire approach to undergraduate mentoring. If they get a minority student – fine, but no deliberate cultivation, no pedagogical philosophy.
      And these same non-nonchalant attitudes seem to continue up the pipeline.
      If you saw the sea of faces at an ABCRMS meeting them looked at what’s happening in the faculty rooms across this nation you see that A LOT of diversity is lost along the way.

  16. The diverse thoughts in this string of comments shed light on the difficulty in handling issues relating to racial disparities. Frankly, the peer review system is only a small part of the problem. Many good ideas today require collaboration with a diverse team of investigators. If you compare grant success rate between minorities in top research institutions with low resource institutions where most minorities work, I guess the trend will be similar to the general funding rate. Instead of focusing on the peer review process, NIH could work with universities and colleges to improve research infrastructure and personnel at these low resource institutions. What about supporting established and successful investigators to spend one year sabbatical at these institutions while maintaining their ongoing NIH funding at their home institutions? By creating a forum for students and postdocs from both institutions to work together, it is possible to develop the much needed research infrastructure, expand the expertise available to investigators, and enhance research collaboration. We live in a highly competitive environment, which requires us to prepare the large research workforce who have excellent fundable ideas but lack adequate support system to successfully compete in traditional funding initiatives.

  17. Thanks to the many thoughtful comments above. As some one relatively new to running my own studies, I have been surprised at the review errors such as those mentioned by Dr. Hassin above in my own proposal submissions, and dismayed at the “pass” my famous colleagues (who happen to be white males) seem to get. While I have strived to learn how to work within the system, it is rather discouraging.

    A simple solution would be to make the review process blinded.

  18. I am dismayed that we as scientist take it for granted that a prospective peer review is capable of predicting the future and identifying the best science to fund. As scientists, this hypothesis should be tested. The fundamental problem may not be bias, but actually that peer review is not the best method, or even a particularly good method, to identify applications for funding. The recent study by Danthi et al. (Circ. Res. 114, 600-606 (2014)) suggests that peer review does not work. A better study would be to fund a randomly selected group of applications regardless of their scores and then track them for a number of years to see the results. If Danthi et al’s work is correct we would find that peer review scores did not predict outcomes. If peer review is incapable of acheiving the desired result then everything we do to try an improve the process is a excercise in futility.

  19. This problem is multifactorial and complex. To start, URMs tend to get less start up funds, smaller lab spaces and face challenges with recruiting good people. Therefore, they enter the NIH pipeline already at a severe disadvantage to their majority colleagues.

    The problem cannot be fixed by peer review directly. However, advancements can be made if there were more funds to support early PIs with less preliminary data.

  20. It will be very interesting and perhaps informative to see how NIH addresses racial disparities in review beyond its current approaches and programs. Much of the problem lies in society-wide personal beliefs based on attitudes involving race, sex, sexual orientation, etc. that are kept buried and would almost never be expressed as a basis for judging a grant in a forum such as an NIH review panel. One need look no further than the attitudes and opinions often expressed about our first African-American President that are clearly based on racist feelings, or the surveys showing preferences for male versus female bosses, to see the magnitude of the problem in making reviewers have unbiased judgments of grants.
    On another note, I completely concur with Deborah Hasin about the occurrence of blatant inaccuracies in reviews being highly problematic where the reviewer has completely missed something in the application, expressed a concern about competence in an area, technique, etc. that is non-existent or made otherwise egregious errors about the utility, cost or significance of the proposed aims. Not only do these comments rankle the PI excessively when received, they are highly unfair and indicate a reviewer’s lack of vigor in assessing the application. But I do not agree a “spot-check” will do, as this will occur after the review and will not change the score. Rather, I propose that the NIH require reviews to be posted 7-10 days before a review panel meets and PIs be given 3-5 days or so to respond and correct any inaccuracies, mistakes, misinterpretations, etc. present in the reviews The PI could not substantially rewrite the grant or simply say they will comply with the reviewers suggestions or comments, but rather point out errors, document publications showing competence, skills, etc. that the reviewers claim are missing. The third or fourth reviewer could be assigned the task of adjudicating the PIs response and at least have a discussion about removing the offensive comments from the review and having it judged primarily on its merits. I think this could also help mitigate some of the issues regarding racial disparities in reviews as it gives all PIs an opportunity to point out weaknesses in the reviewer’s judgments.

    1. As an SRO, I am always disturbed if factual errors are made during the review. In defense of the reviewers, they are doing us a favor by taking on work on top of their considerable load at their day jobs. I know they are busy because it’s a challenge to get them to meet deadlines. In addition, I have found that many times what a PI thinks is a factual error is often a scientific disagreement or different judgement call.

      Having said that, your suggestion of building in a formal response process into the review system is the most reasonable one I have seen and would encourage NIH leadership to consider how that can be programmed into the IMPACT system. Right now, the appeals process is unsatisfactory for reasons too numerous to get into here. The Council system was designed to catch factual errors made by the IRG, but I’m not sure it has been effective. The purpose and uses for Council needs to be re-thought.

      Of course this has little to do with the original post on reviewer bias, but I wanted to call NIH’s attention to this comment.

      1. Errors occur both ways – those that aid and those that hurt the scoring of a project. Is there bias that aids some labs and hurts others? Sure – so this comment is pertinent to the initial posting. While errors can be addressed in the A1 – more insidious is the review that states no identifiable problems with the project, but simply scores the grant outside the funding level. This is highly dependent on the reviewer, the study section, and even the session – it only takes one of three to say – “I score it a 3” and doom the grant.

  21. I have 2 suggestions for improving peer review:

    1) Minimum standards for reviewers. In my admittedly small field it is not uncommon to be evaluated by reviewers who have not had a grant in years. NIH funding in the last 5 years should be a requirement for all grant reviewers.
    2) Payment for review in the form of a (small) grant. For example, if a person reviews 10 grant applications in a year (or participates in 2 review session, etc) they automatically receive $10,000 in research grant funding. If you want me to put (even) more effort into these reviews then pay me more to do it.

  22. In my view, if we further reduce how many R01 grants a researcher can have at one time or consecutively, it would open up opportunity for newer grants. Some labs get a great deal of funding, not because they have better ideas, but because they have had success at writing grants before, because they have big names, and because they are reviewers themselves. The whole system rewards the one’s who’ve had the most success. That doesn’t mean that what they did wasn’t exceptional, but it means that even when they write marginal grant applications, the faith the system has in them will increase the likelihood that they will get another grant. I know individuals who have had too many grants simultaneously to manage all of them well. That, my friends, is a waste of my tax dollars. I’d rather see it go to less seasoned grant writers. Once one of these newbies gets a grant, they too may become a powerful grant writer. If they continue to lose out to the seasoned grant writer, the one with the big lab and the resources, who is to say the newbie won’t just give up? That’s sad to think.
    Let’s be the innovators here. Let’s reduce the amount of funding a lab can get and force their institutions to find ways to cover salary so that we don’t force people to rely on federal grants for 100%. This unfortunately causes grant writers to write whatever they think may be a good idea just to get funding, even though many of their ideas are sub par and really don’t advance science much. That’s it.

  23. This initiative is flawed because there is already compelling evidence that there is no bias in NIH peer review. The worse funding success for black versus white applicants noted in the influential Ginther (2011) study can be entirely explained by the lower productivity of the black researchers, as indicated via multiple quantitative approaches including metrics provided in the Ginther study itself. In fact, when funding success rates are normalized by quantitative indicators of success (publication count, impact factors, etc.), black applicants have a slightly higher likelihood of receiving funding than white applicants (see: http://arxiv.org/ftp/arxiv/papers/1112/1112.3944.pdf ). There is thus no evidence to justify putting any blame on reviewers or accusing them of explicit or implicit racial biases. The numbers point to a more systemic problem in terms of disparities in scientific productivity among races.

    Further, the composition of the expert panel that has been volunteered to evaluate this initiative includes several investigators that focus on “implicit bias”, such as is measured through various social psychological “implicit attitude/association tests.” There is a healthy scientific debate regarding whether the notion of implicit racial bias as measured using these methods has any external validity whatsoever, and so including this focus within the panel seems premature and unwise. Such a strong focus on implicit bias in the panel would seem to introduce bias to ferret out unconscious prejudice where it can not possibly be having a significant impact (given that there is no funding disparity based on race when accounting for productivity).

    It seems like this issue makes scientists, a group of people who are in general highly objective, quick to abandon this rigor and impartiality and seek answers that are politically or socially acceptable rather than quantitative.

  24. Related to issues of fairness and consistency is the degree to which reviewers are calibrated within the panel or across the system. In the “good old days”, panel would have open discussions on why each reviewer was scoring e.g., 1.4 vs 1.8. Given that there is always a funding line (however clear, blurry or unknowable, and however much we’re not supposed to consider such things), this allowed the group to tune the fine distinctions that go into judging among excellent and potentially fundable projects. With the “new and improved” scoring system, the actual scoring is left up to a stochastic process wherein X reviewers decide on e.g., 2 and Y decide on e.g., 3. Because the group is unaware of the final score, it can’t provide any normalizing input or exert any accountability. Regardless of the funding situation, it’s likely that these important distinctions will always fall between two whole numbers. And while honest opinions will differ, the lack of feedback to the group masks potential biases and adds uncontrollable variability.

  25. I think the idea of giving the PI several days to point out factual errors in a review is a really innovative idea. Having recently spearheaded a successful appeal of an R25E grant on a 1A submission, that had been trashed by a reviewer who made multiple errors that were picked up by other reviewers rather than contested (only to have the post-appeal revision narrowly miss funding so that the proposal had to be abandoned), I am really tired of dealing with reviewer errors, as well as comments dinging the grant for a methodological limitation and not mentioning that an explicit explanation had been provided in the grant acknowledging the limitation, and explaining why it could not be remedied. The problem is that someone would have to police the PI responses to make sure no additional information was being added, and only a strictly factual correction was being made. A digital form that had a blank to quote the error and then another below it to cite the section of the grant that contradicted it would be helpful in limiting PI responses. However, this does not address the disparity in awards to minorities. I agree with others that the issue of increasing diversity among researchers goes much farther back in the educational process and is unlikely to be remedied by changes in the review process by the time someone is an independent investigator, especially in this funding climate. Special criteria for young investigators and minority investigators have been in existence for many years. When I have served on review panels I have been more impressed that some reviewers are more interested in dominating the debate and showing off than they are in a thoughtful consideration of the type of research that may be useful to the field and society.

  26. This is a silly idea. It increases subjectivity of evaluation of the proposal by allowing the applicant to inflate his/her achievements. Furthermore, it burdens the reviewer by forcing him/her to spend additional time and criticaly review the likely exagerated “achievements”. There are simple objective numerical data that should provide the base for evaluation of the applicant: (1) The “h” (Hirsch citaion factor) that should be compensated vis-a-vis duration of the appliant research activity; (2) the number of citations, easily obtained from the Scopus, Google Scholar or Web of Science websites. The number of citations is much more important than number of publications, even if some are in high-ranking journals. The publications in such journals, if they are rerely cited, are usually of little value; (3) Ranking of the researcher in the websites such as “biomedexperts” and/or ResearchGate which use objective criteria to evaluate the value of the applicant research. These markers, objective and color- or ethnicity- blind, offer the best evidence of productivity and accomplishment of the applicant as well as his/her potential for further research. Of course, the reviewer has to evaluate whether the in the criteria listed above in (1) and (2) the appliacant is the principal investigator or just one of the co-authors.

  27. Having reviewed NIH proposals for many years, I must agree that the lack of funding for NIH proposals is the biggest obstacle for funding regardless of applicants’ color. When I review, I do not look for the racial/ethnic background of an applicant. Most of the applications that I review, I do not know the applicants or only am aware of their publications. To argue that well known scientists should be penalized because of their prior work, would be a disincentive to work hard and build a career. We need a combination of grantees, those who are new to the field, those who are building their careers and those who have dedicated their lives to their fields. We should not be pitting “career stages” of scientists against each other but working collaboratively towards the interests of the scientific areas of discovery that will make a difference for global health and safety of all people.

  28. While minorities, women and young investigators are under-represented, this is not a priori demonstration of bias on the part of reviewers. The problem arises from much larger socioeconomic factors and possibly bias at other places in the system (hiring, promotion, etc.) As a long-standing reviewer on NIH study sections, I must say that I do not know the race of applicants unless I happen to know them personally. Often I do not know the gender of the person as names are often not clearly male or female. The idea that a problem in peer review must be fixed, and by some fairly drastic measures, without clear evidence that the problem originates or is perpetuated by the peer review system seems typical of bureaucracies. We are becoming more and more like the EU’s scientific review commissions, a bloated bureaucratic mess, with decisions made on the basis of ideology rather than fact.

  29. I’ve reviewed federal grant applications many times, though not at the NIH. I’ve never considered race. Actually, if a picture of the PI doesn’t appear when I search for his/her name, I probably don’t even know anything about race. It’s hard to be racially biased in that situation.

    The *real* problem here is that there are too few grants for too many PIs. THIS is what NIH should be addressing. How about limiting the number of grants a lab can have to two? What about cutting indirect costs to, say, 40% at most? Cutting out a lot of those multi-million dollar grants and replacing them them R01s and R21s?

    And yes, get rid of those investigator scores. They favor people who a) have grants already and b) work at institutions that get a lot of grants. If you work at a small research institution or college, you probably won’t get a 1 on that section of the application, putting your chances of getting funded in the lower end before you even submit the application. Talk about bias!

    Yeah, I know that these options won’t be too palatable to a small number of people who have a big share of the pie. But our basic research enterprise is failing in this country, and we won’t fix this problem until we make some changes that hurt the people who are currently benefiting the most.

  30. I believe NIH should appoint full-time reviewers, like program officers, with expertise in different areas. These reviewers can be chosen from academic institutions based on their contributions to the field and interest in improving the quality of peer review. NIH should also set up a system for the PIs to evaluate /provide feed-back to the reviewers anonymously based on their review. If a reviewer consistently receives poor evaluation, he/she will be made accountable for his/her job. As these reviewers’ only job is to provide unbiased review, it should not cost whole lot of money for NIH to appoint them. Reviewers should have opportunity to attend scientific meetings to keep abreast of current developments in their field. Of course, these dedicated reviewers will be up-to-date as they are provided with new information all the time thorough grant applications that they review. I think it helps minimizing conflict of interest or bias. I do not believe bias can be completely removed from any system where human decisions involve.

    Other related issue – I do not understand why NIH has select pay for grant applications with scores near the pay line. It helps investigators who have good rapport with program officers at the cost of other truly deserving investigators. This does not make sense.

    1. But TRN, half of these comments are complaining about cronyism and other biases favoring the established in study sections. Select pay is there to counter this, no?

  31. Having served on many NIH review panels, I have never seen overt or implied bias emerge during the review process. Most reviewers take the job of reviewing seriously and advocate for novel ideas from all applicants. However, a degree of bias against younger investigators, who are more likely to include more underrepresented minorities and women, is a secondary consequence of limited funds available for funding grants. This is because applications from senior and highly accomplished investigators are generally better, mainly because they have been doing research for a long time and have learned a great deal about how to do science and how to write grants.

    Thus I believe the key is to make more resources available to younger investigators. Three ideas for doing so are 1) limit investigators to no more than 2 R grants as PI and be careful about additional grants being ghost written by their associates; 2) review the formula for calculating indirect cost rates with the goal of defining a maximum limit; 3) request additional funding for NIH and other agencies with the funds to be dedicated to younger investigators only.

  32. I have served in NIH study sections. I do believe that some sort of bias exist in NIH review process. Investigators from big name institutions have clear advantage especially when the grant funding is tight. When the reviewers see an application from a reputed investigator and from a big name institution, they tend to evaluate more favorably. Not necessarily that their grants are always the best. Applications from small institutions are evaluated at a different standard and some reviewers start with a negative attitude towards those applications. I have seen this from my own experience in NIH R01 grant renewal application. Although I have consistently published in reputed journals and had high productivity during the past grant cycle, reviewers gave me a score of 3 or 4 in the “Investigator” category. Ironically, reviewers mentioned that my training and productivity is outstanding. Some reviewers mentality is such that an investigator cannot be given a score of 1 or 2 if the investigator is not from a big name institution. Also, I believe that If the name of the applicant does not sound like American, it is viewed differently by some reviewers either knowingly or unknowingly. Also the name of the institution has a significant impact in grant funding. The experiments and techniques that I proposed in my application are all available in my laboratory and all the facilities required are available in my institution. We also have utilized those techniques in published papers in reputed journals. Yet, the reviewers gave a score of “4” in my recent grant renewal application and mentioned that the environment is somewhat modest. Although we have all the techniques and facilities available in my lab or in the core facilities of my institution and we have published articles involving those techniques, the reviewers are giving a low score in Environment. This clearly suggests that some reviewers are biased towards the investigator, investigator’s name and the Institution. If NIH wants to increase fairness in NIH review process, the applicant name and the institution name should be removed before sending the application to the reviewers. Also, NIH CSR should provide guidelines to all the grant reviewers that applicant’s name, status and institution should not be considered in evaluating the grants. I also agree that the productivity should be measured based on the total dollar amounts funded in that lab. Very often investigators cite the same papers in multiple grants. So, NIH should take into consideration the number of investigators, total amount of grant funding and the number of papers published to determine the productivity. Investigators at smaller institutions are clearly in a disadvantage position right now. We definitely need to improve the fairness in NIH peer review process. I think NIH should be serious on this. I appreciate the initiative made the NIH CSR director.

  33. One point that seems to be missing in all this discussion is allowing the NIH program officers a wider latitude in picking who is funded. After all, they can see the entire landscape. They will note that 4 study sections gave high scores to the same “kinase du jour” study by different investigators. They can balance the portfolio in terms of geography, topic, investigator backgrounds, you name it. Yes, that will mean that some lower-scoring proposals will be funded over higher-scoring ones. But so what? Do we need 4 grants funded to investigate the same trendy kinase, or would it be better to fund one and then fund three other studies? Allowing the program managers to fund really innovative studies that were high risk and therefore dinged by risk-averse reviewers would likely yield great benefits in innovation. The strict pay line is certainly one source of unfairness.

  34. Another lame initiative from CSR. The actual problem is the low R01 paylines. Having served on study sections regularly for over 20 years, I have never seen a hint of racial bias. What I have seen on many occassions is “big names” funded with multiple grants based on their reputations, rather than the actual application. As pointed out in some of the comments above, these same premier labs do benefit when paylines drop while smaller labs (regardless of race) are sacrificed. The solution is obvious. We are in dire need of a realignment of NIH funding priorities and more money directed at the R01 pool. Such a realignment will require effective leadership and some political courage…not another bucket of red herring.

  35. If the grants are reviewed blindly there should not be any bias since the reviewers would not know the name, gender, race, institution etc. This would improve the fairness in the NIH grant review process.

  36. Directors Nakamura and Rockey:
    I appreciate this open discussion about this important topic, and as a taxpayer I firmly believe that growing a diverse research workforce is a worthy goal in addition to funding the most meritorious research. As a research manager I have a slightly different perspective on the overall research grant program as well as the mechanics of how this information could be gathered as well as implemented. The concept of blinding the peer review process was mentioned, and this, to a lay person, seems like an easy way to not only remove some of the potential concerns of bias against URM’s but also some of the favor that established investigators receive. After a quick search, it seems that most of the published information is an analysis of public information, but perhaps a quick way to generate data on this perceived positive and negative bias at study sections would be to, over several review cycles, have a blinded study section within the same institute review only scientific documents (removing biosketches, facilities, and other documents which might explicitly name investigators and institutions). They could then judge the Significance, Innovation, and Approach of a grant to then compare to the original scores that the initial study section gave. Perhaps this data could pave the way for tweaks in the proposal submission process, as this initial grant review would leave out two important scores in Investigators and Environment—something that could be reviewed at the Just In Time phase of the application. This should also reduce the dreaded administrative burden, as now researchers would not need to worry about biosketches, facilities, and other non-scientific information until they are informed that their application is competitive.

  37. I think it is a good idea to leave the Investigators and Environment for the Just in Time Phase and the council meeting. Instead of reviewers score on Investigators and Environment (which very often create bias), NIH should determine whether Investigators have the appropriate training and the institution has the right facilities to conduct the research proposed. Grant reviewers should only evaluate the science and score on the significance, innovation and approach of grant proposal.

  38. In addition to blind reviews (without giving the names of the investigators and the institution), the meeting roster and the reviewers names should be kept confidential and not published with summary statement. Sometimes reviewers could be intimidated by the big name investigators. They may fear that their own future papers, grant applications may suffer if they do not give high respect to those big name investigators. It is not very difficult to assume who the reviewers are on a particular grant application from the Meeting Roster. Publishing the reviewers name in the Meeting Roster does not do any good rather it increases the chance of intimidation. Also, the investigators should be allowed to exclude up to three reviewers who could be potential competitors and might not be suitable to review their grant applications.

    1. I am inclined to agree that biases do not occur in review panels. Reviewers are not focused on race or ethnic origin as a factor in a successful grant application. This is not their goal. I would agree that there are less minorities in higher level positions and perhaps this is where the focus needs to be. However, I would argue that once a person (white man or otherwise) is in a position to apply for NIH-level funding, then they need to be able to compete with other applicants (which includes the white male counterparts). If lower funding rates are observed for minorities, is it because the applications are simply not as good, thus they are not funded? So, is there a preponderance of poorly written applications from minorities? Are they simply not able to compete even though they are in faculty positions? If so, then why is this? Perhaps this is where the focus needs to be, rather than on the grant reviewers themselves? I know of several other scientists who have reviewed on study sections at various ICs (NIGMS, NIAAA, NICHD etc) and we all agree that reviewers have no inherent bias towards minority applicants. It is the application itself, its scientific merit and feasibility that is being evaluated. Perhaps, the NIH could offer minorities (or everyone for that matter) some tutorials/grant writing workshops or some other training options to help improve minority applications (IF indeed these applications are sub-par in quality??).
      Perhaps the suggestion for mentoring from senior PI’s is a viable option. Given the crisis in low funding levels these days, AND the trend in funding “translational” projects (i.e. funding of basic science projects that do not have a direct translational component, or human disease connection, is diminishing–another fact I find appalling), then I wonder if collaborative projects may be a good way to go. Thus, “smaller” investigators (ones with smaller labs, new investigators, and/or investigators at largely undergraduate institutions) could focus on well-developed model systems that are proven to be useful for valuable scientific discoveries and provide great training systems for our students to address focused questions about basic biological processes. If coupled with a senior investigator (with a “proven” track record), then perhaps these co-mentored projects might be good opportunities. This could even be a separate funding mechanism–Co-mentored senior-junior training awards. Just some ideas about the issue. I just think that the reviewers, who work hard, try hard and have the best intentions when it comes to race and gender, need to be cut some slack. They are not the crux of the problem.

      1. I agree with the above comment and those of several others. In my fairly extensive experience as an NIH reviewer, I do not remember knowing the minority status of applicants. I agree that pre-application assistance is the place that NIH should focus, and perhaps information about minority status should not be included in applications, in order to prevent bias in any direction.

  39. 1) Scientific progress requires diversity of ideas. This diversity cannot be achieved if we concentrate resources in the hands of the same individuals from a handful of institutions. There needs to be a recognition that new ideas coul come from many places.

    2) For a scientist with three R01 grants, one of the projects is typically considered a “side project”. Wouldn’t it make more sense to invest the $$$ in someone else’s “main project”?

    3) Much of the science funded in the last 20 years was largely oversold and the returns, measured by health outcomes, leave much to be desired. We could not do worse by distributing all the research funds among new or ” untested”scientists.

  40. I strongly agree, it seems to me that the bias of funding is stronger every cycle. The bias is towards well funded labs…of course, they have the technology and manpower to be the 6%…the system needs to be treated as critically ill…it may be even worth completely “changing” the “rules for funding” for a while until regaining health!

    Gender and racial bias, of course, exists. The bar is higher for minorities. It takes a great deal more to “prove yourself” as a minority. On top of that…the “course” of being a successful minority…”oh, it was easier for you because you are a minority”…I want to believe that when the bar is reached, the bias begins to dissipate. I think that the only way to really evaluate this, is comparing whatever metrics exists between everyone and hispanics. The only minority racial group that is associated with a lower cultural level and that can be easily identified in the name of the PI.

  41. I would like to make the simple point that we would not be discussing fairness of the review process, changes in the biosketch, new review formats or even issues concerning the age at which investigators get their first grant or facing the sad fact that no change implemented by the NIH in many years has changed these concerns if the R01 pauline was at 35 to 40 percentile. Possibly it is time to discuss what it would take to get to more reasonable pay lines.

  42. This is scary. Self-identified “reviewers” in this comment section that are/were presumably NIH funded are using their personal poorly validated anecdotal experiences on study section to try and refute published data… I wonder if they use similar anecdotes to draw conclusions in other ways? Is this where we are at as a scientific community? Have you considered that it’s not an issue of bias against a minority group, but rather a strong bias in favor of the majority?

  43. An element of randomness would be unsettling on some levels, but could address several of the problems discussed here. Under this approach, all proposals would be evaluated under peer review and all those that are rated at least a certain quality (e.g., “Very Good”) in terms of bang for the buck would get a chance at funding … rather than selecting all those reviewed as having a bang-per-buck above some fixed very-high level as is done today, a bias which favors the established (white male) scientists. Grants would be chosen randomly from this fundable pool until the money was used up.

    1. I think these comments are all interesting. My suggestions:
      1. Limit applications to one every three years from scientists who are PI on three or more funded NIH grants. Also limit applications of Investigators who are Co investigators on 5 or more grants. Personally I think it is unethical for someone to continue to apply for and get grants when they have 5,6,7 already and many labs are closing because they don’t have one.
      2. Allow PI to view reviewer comments BEFORE STUDY SECTION to allow them to point out factual errors. Any grants with such errors must be discussed, along with the PI’s comments.
      3. Reviewers should not have access to PI/collaborator/institution. This should be separately reviewed if the science is deemed worthy.
      4. Every cycle, grants deemed “very good” should have an opportunity to enter a lottery for funding.
      5. Investigators over the age of 65, or with more than 20 years of NIH funding can not be sole PI on any grant application– they must be CO-I or CO-PI with a more junior person.
      6. Having been on study section, I can say that there is absolutely bias in favor of dominant society investigators and against under-represented minorities– even to the extent of subtle yet racist remarks made while in session. Applications by minority investigators should be made available for secondary review, and those applications- if “very good” on secondary review should be allowed in the lottery.

  44. I don’t really know if there is bias in the review process but I am dismayed to see my fellow scientists assert that they have “never seen” any bias on a study section and thus conclude that there is none. The vast literature on implicit bias has established the existence of unconscious bias. Without controlled studies of study section bias, we are just groping around in the dark. CSR needs to set up a few test reviews, perhaps giving grants to more than one study section and subtly changing the names and racial categories, to try to get a handle on what is going on. It would be folly to implement measures to counteract bias without first determining whether there is any, or whether there may be other explanations for the lower success rates of applications from URMs.

  45. It is interesting to me that no one has considered the possibility that bias may arise because of the topics that URM researchers chose to study. In the social/behavioral sciences, researchers who are studying health disparities may present hypotheses (with supporting preliminary data) that simply do not resonate with the average mainstream social/behavioral science reviewer.

    Also, “objective” criteria may not help because bias may exist on every indicator. The Ginther report found that Black researchers had lower citation rates than Whites, and the assumption is that this is based on the quality of their work. It is also possible that people simply cite others like them (in terms of race, gender, academic institution) because those are the first individuals who come to mind.

  46. We’ve seen first-hand how hard reviewers work to get their reviews right. The diligence and care in writing their critiques and discussing their evaluations is abundantly evident. So it’s important to note our current efforts are designed to enhance the reputation NIH peer has for fairness and rigor.

    NIH-sponsored studies have demonstrated disparities in peer review outcomes and subsequent grant award rates related to the race or ethnicity of the Principal Investigator. If you are not familiar with these studies, we encourage you to read the articles by Ginther et al. published in Science in 2011 and in Academic Medicine in 2012.

    In response to these articles, NIH leadership asked our offices to move aggressively and methodically to understand the cause of the disparities in the probability of receiving NIH grant awards. As a scientific institution, we are compelled to be scientific about our response to these disparities. This is why we turned to the scientific community and other stakeholders to help us develop appropriate studies to assess bias and maximize fairness in peer review. We appreciate all input and are grateful for community support.

    The study of implicit bias — subconscious positive or negative attitudes — has been and continues to be an active area of research among social scientists. This has to be a consideration when looking at the NIH award disparities. Nonetheless, it is likely that there are multiple and subtle factors behind these disparities that could include cumulative disadvantages and/or advantages, and/or bias in peer review. We therefore are exploring all possible factors that could contribute to these disparities in grant funding and to address the problem.

    NIH understands the problem is a broad one, and we are taking on other aspects of it, such as addressing the diversity of the future workforce in the extramural community and at NIH. Everyone stands to benefit because the examination of peer review and our other efforts should lead to an even stronger review and grants system to support a vibrant scientific community and inspire further advances in science and health.

  47. What the NIH fails to understand, is that enhancing the “future” diversity of the scientific workforce is critically dependent on support the diversity of the “current” work force. When a minority student looks around and sees no one of color as professors, scientists, etc– what kind of message does that send about the feasibility of a future career? When a junior faculty member sees that either there are no senior faculty of color, or that those who exist are under duress- what kind of message does that send to the junior faculty member? The list of examples goes on and on.
    Websites and networks of majority scientists simply will not work, and will do nothing except to further the position of majority scientists. At one time, the NSF had it right– it had small minority faculty planning and mid career grants. They were only 25-50K, but really helped in a pinch. If there is no enthusiasm for allowing minority investigators more access to funding, then how about immediate access to small grants.
    In fact, instead of turning the efforts to increase fairness and equality to little more than a game show (10,000!!! the price is right), why not give that 10,000 to a minority scientist whose lab might just need that special mouse, reagent, or technical support?

    1. I agree. The continued effort to build up and diversify the trainee pool, only to have them fall out at the faculty stage and do other things successfully is counter-productive to the stated goals of having a more diverse R-level funded pool. I’m surprised that so much of the focus following Ginther was centered around supporting trainees and not supporting the qualified minority faculty in place, but at risk of deciding to do something else.

  48. No matter what color you are, this is no longer a viable career path. Despite al the happy talk on this blog, the academic research career body is already gangrenous. The smell is just not disturbing the neighborhood yet. Does anyone think they can flip a ten-sided grant-coin every five years for thirty years of a career and get heads every time? Because that is, let me think, oh yeah, one in a million. If you were deciding on a career, would you bet the lives of your loved-ones on that? Seriously? I have been in this business for over 20 years, but would never start out on this today, no matter what the color of my skin, because I can do math. These days, if you get tails, even once, it can be all over for you as your lab shuts down and you lose your associates and then your job. That means, of course, that you must have two overlapping NIH grants to survive. Great! Now you have two ten-sided coins. Do the math on that, I double dare you. When the probability of success becomes so small, it does not matter what too much what color you are. Who are you gonna trust, NIH happy-face statistics or your own lyin’ eyes? You can see it in the hallways because about 25% of all the research space at my institution (space that was fought over 15 years ago) is now empty. You can see it in the faces of the students because I just taught a class for 8 MD/PhD students here and not one was even interested in a laboratory science-based career. They all mumbled about translation and clinical studies. Of course, the MD/PhD students are our best and brightest and they have done the math, albeit belatedly. I suspect they will split for straight clinical practice at the earliest opportunity. If they do not, they really are delusional.

  49. I agree with the views presented above. My lab is likely to close if I don’t get funding in six months. I can not afford paying stipend for my two students or buying supplies for them. It is not that I am not productive, I publish two to three papers every year in reputed peer-reviewed journals. I have been submitting grant in a continuous fashion.

    Saying that there is not bias in the peer eview process is putting blinders on you. I don’t necessarily agree that the bias is only against minorities. The problem is that when you have to fund at a 5-10 %ile, a natural bias is created among the reviewers. If there are two applications that have the same merit, a reviewer would generally support a person he knows that an unknown one. In the past when about 25% of the applications would get funded, the bias would go away. Now, we have another dumb policy of letting investigators submit as many times as possible, as if that is going to improve their chances of funding. I would rather have 3 submissions for the same application so that you get reviewer’s feed back and keep on improving every cycle.

    Probably there are two solutions: 1) Have an anonymous system of grant submission. You can submit this in two phases. In the first phase, you can remove all references. In the second phase you can ask investigators, to submit the necessary references. 2) You should never have a published list of reviewers and they should not have a six year term. You should have a random new panel every year.

    As far as reviewer bias goes, it is a common saying, “You scratch my back, I will scratch yours”. That is all I can say. It is a sad affair. NIH has to do something about it. Probably the best answer is to improve funding percentage, stop funding an investigator with more than 3 R01s, limit multimillion dollar funding as program projects and superfund grants. THE BEST YOU GET OUT OF YOUR BUCK IS THROUGH R01s.

  50. Totally agree with HK above. Two examples from my experience:

    1) An investigator I know has a tiny group 3-4 people, now He has 5 R01s. Good for him, right? But he keeps submitting grants like there is no tomorrow…. Now Isn’t that greed? Is that fair for the others out there in need of a grant? Should this be even allowed? He has just a tiny group. COME ON…. Oh, did I mention, HE IS a reviewer for the NIH — Scratch my back, I’ll scratch yours… so true!

    2) Another senior guy, 30y of NIH experience, lots of grants – He tells us:

    “Send me your ideas, so I can submit under my name, because people at the NIH KNOW ME…”.

    Oh my. Is that how NIH has worked and is working all these years?… Is this abuse of power of his influence at the NIH?… Should I report him?.. Is that how we educate the new generation?… Where is the fairness in the academic world?

    The review system is hugely flawed and biased. THIS NEEDS TO CHANGE. Senior guys now in power want – and they make sure they stay – in power forever. Young people have no chance. If I could turn back time, I would NEVER chose research again.

    New generation, beware!

  51. The biggest problem is the low level of funding relative to the large number of scientists. This cannot be fixed by minor changes to the peer review system. Peer review is pretty good at distinguishing very good science from poor science, but not for making the extremely fine distinctions that lead to fund/no fund decisions. At the current borderline, there is tremendous randomness in funding of relatively similar grants around the funding point, with much more of the decision falling to program staff.

    That being said, there are some changes that could help at the margins.
    The biggest would be for the study section to review grants once through and then circle back and re-discuss the top X % (those within/near potential funding range) and explicitly rank and/or rescore them. This would greatly reduce the drift in scoring during the meeting and allow better ranking.
    Minor issues would be to remove the scoring of the criteria of “environment” and “investigator” which should slightly reduce unconscious bias toward top ranked schools and applicants. It would also reduce the likelihood of having applicants feel personally attacked by a low score.

  52. I agree with the recommendation on satisfactory/unsatisfactory rating for environment. In addition, I would suggest to use the same qualified/unqualified rating for the primary investigator in order to reduce the bias against new investigators.

  53. My experience participating in limited NIH review meetings astonishes me how some of the senior investigators get funded just for their names and junior investigators shot down because they don’t have sufficient experience or not known to reviewers. NIH can prevent this bias by taking few bold steps.
    Make the applications annonymous and score the grant only on the basis of science, that is significance, innovation and approach in the first round. Only the grants with top scores should then go for second round of review by NIH staff (who are normally not biased) to score investigators and environment.
    Just like in special emphasis panels, random reviewers including both senior and junior investigators in 1:1 ratio should make up the study sections. The list should remain annonymous until the review is completed, so that there is no chance for reviewers to be biased.
    Increase number of R01’s and decrease or stop the multimillion dollar program grants. When NIH had funds to give away the program grants made sense but not now!
    Limit grants to 2 R01s per investigator to increase the chances of junior investigators and minoorities to get funded. In addition this will also increase the quality of science as the investigators will be more involved in mentoring students and fellows.

  54. How about including a few non-NIH-funded PIs (new investigators, URMs etc.) whose grants may not be funded yet as “Observers/Temporary Members” during study section meetings? Have them review one or two grants, perhaps especially from experienced PIs. That will serve several purposes: less work for the panel, great training for invitees in terms of reading (presumably) good grants, and learning first hand how review works. Also, if subtle, unconscious bias exists among the panel, having their underrepresented/new PI counterparts as their reviewer colleagues will help eliminate it. It seems to me that personal interaction is the best way to overcome unconscious bias. If such bias does not exist, no harm done. Win-win, right?

  55. The first thing the NIH needs to do, is team up with the NSF and flood the market with money to train more analytical experts: epidemiologists, statisticians, biostatisticians, those types. There are major, major analytical errors in research efforts these days and let’s not kid ourselves into thinking the studies on peer review are immune to them. We desperately need these people at all levels from the bench to the review panel.

    Unless and until the studies suggesting racial disparities are reviewed, validated, and verified (that whole replication thing), and unless and until strong evidence points to the reviewer in the chair, and nothing else, I’d advise the NIH to stop calling your volunteers that you are 100% dependent on, bigots. For ostensibly smart people, that’s an incredibly stupid thing to do. The only logical conclusion one can draw from this is that the competence of those running the NIH needs to be questioned. If half of your reviewers told you to pound sand, I bet these research findings would magically change overnight. If the NIH believed this, and I mean truly believed this, they would have begun surveying individual reviewer scores and reports for evidence of systematic scoring patterns. Deming said to bring the data. Well, let’s see it – the raw, unfiltered, unmassaged data. Systematic issues don’t need complicated statistical approaches.

    If there is racial disparity in grant awards, and if it’s determined to come from bias of reviewers, a deliberate effort to enhance diversity is the worst idea to fix it. First, you eliminate the bias. If that means you fire every reviewer, then so be it. If it means something else, so be it. But that’s the first step. The second step is to evaluate the grants properly. We may find minority scientists are better and that bias has thwarted that. We may find they’re the same as everyone else. Employing a knee-jerk reaction, however, and demanding X% of grants or money go here or go there, based on categorical partitioning and not quality or merit is foolish and is akin to rolling the dice in Vegas.

    I’m against blinding. If you believe there are implicit biases going on, then you make the problem worse by trying to blind everything. People will just guess. You fix such things by forcing people to confront the biases with evidence. Whether it’s racial or based on reputation or grant history. These things must be confronted.
    To address this, the NIH must address the process of grant reviewing and Congress must fund what needs to be done. Reviewers and universities must have an incentive to review and provide reviewers. Calling them bigots is probably not effective.
    We need brokered contracts that allow reviewers to devote time to reviewing and money needs to go to universities as compensation. Universities should receive twice the reviewer’s pay, that way if they have to hire someone to help with some of the duties the reviewer normally does they can do so.
    R01 grants need to be 7yr grants, with 7yr renewals. Forcing grants to address a longer time frame will necessitate much larger budgets, meaning labs get established, because they only need the bare essentials to get started and get approval. The grant will periodically fund new equipment and so forth to keep things rolling. It provides stability for hiring and development of students, post-docs etc. A top notch investigator can be a post-doc under one of these grants, and then take it over for a renewal, and get renewed again, in 20 years or so only three periods of time would be spent trying to get funded, rather than a yearly ordeal.
    Longer term grants would mean they have to be really, really, really well planned. They could be much longer too, than 10-15 pages. And, such planning difficulties assure you that there will be far, far fewer grant submissions each year.
    With fewer grants, the NIH can put reviewers on the road to visit applicants. This is where implicit biases get taken care of in many cases. It will become crystal clear that your Brand X specimen freezer works the same whether it’s at Harvard or Bobby Joe’s Community College. And, demonstrated competence with the freezer and lab techniques can be verified. With those types of factors being established, and Harvard wants $10mill and Bobby Joe wants $5mill to do the same grant, reviewers will never again be able to write flowery comments but issue crappy scores to the small schools while crowning the big schools with the title ‘best ever’ as they so often seem to do.

    It’s time for NIH to act like it’s made up of scientists, rather than a government agency.

  56. Nothing will change at the CSR until new leadership comes in.

    Under the current leadership the expectations of fair an honest reviews are certainly not being met. Existing polices put in place over 15 years ago in Notice Number: NOT-OD-11-064 Appeals of NIH Initial Peer Review are not being adhered to.

    We should be raising our united voices to request such a change in leadership.

    This bogus CSR blog does nothing to return fairness to review.

Before submitting your comment, please review our blog comment policies.

Leave a Reply

Your email address will not be published. Required fields are marked *