22 Comments
Dr. Steven Benner at the Foundation for Applied Molecular Evolution thought the genetic code of life could be expanded beyond its naturally occurring four building blocks. If successful, powerful new systems for developing diagnostics, therapeutics, synthesizing biopolymers, and other still unimagined uses may be possible. In 2019, he and his colleagues came a step closer to this vision. They reported in Science a genetic system with eight building blocks, increasing the amount of information a given stretch of DNA can carry. Moreover, the modified DNA can be transcribed into a modified RNA using a specially engineered RNA polymerase.
Projects like this are perfect for the NIH Director’s Transformative Research Award program, which funded Dr. Benner’s idea in 2017 (see the website for a complete list of other supported research). Now, NIH is seeking applications for the 2021 awards through a new funding opportunity (RFA-RM-20-013) recently released on Friday, May 21, 2020.
The Transformative Research Award has supported groundbreaking, unconventional, and creative ideas for over a decade as part of the NIH Common Fund’s High-Risk, High-Reward Research program (HRHR). While this kind of research has the potential to overturn fundamental paradigms, it can also be risky. That is why innovation and breadth of impact potential are emphasized more during review than a project’s feasibility and preliminary data.
Historically, however, the HRHR applicant and awardee pools have not fully represented the demographic and geographic diversity across the U.S. biomedical workforce (see here on the Common Fund’s commitment to diversity). A working group of the Advisory Committee to the NIH Director examined this issue and recommended that the Common Fund prioritize increasing institutional diversity in the program (see their Report here).
Concerns also exist about bias, be it unconscious or otherwise, generally throughout peer review at NIH. As a way to address this issue while also enhancing diversity, the HRHR program is going to anonymize the review of Transformative Research Award applications.
The Common Fund and the Center for Scientific Review devised an approach to anonymize the identity of applicant institutions and investigators until the last phase of the Transformative Research Award’s three-phase review process. By anonymizing applications, it is hoped reviewers will focus on the merit of the research and limit potential unconscious bias while encouraging applications from investigators who otherwise might not apply due to perceptions of such bias.
Previously, during the first phase of review, an “Editorial Board” (composed of senior scientists with collective expertise spanning a broad swath of the NIH portfolio) had access to the entire application package and selected a subset of applications with the greatest transformative potential for further technical review. For the new anonymized process, the Editorial Board will only be able to see the application’s Specific Aims page while considering the project’s significance, innovation, and logic. The Specific Aims page is used in an unconventional way in the Transformative Research Award application. It serves as a one-page distillation of the idea, why it is exceptionally innovative, and the potential impact. Applicants will be instructed in the funding opportunity announcement to not include any information that would reveal the investigator(s) or institution(s) in this section, essentially making the review blind.
In the second phase, only the Specific Aims page and Research Strategy section, which must also be anonymized by applicants, will be sent to “mail reviewers” with expertise relevant to the application. These reviewers will assess the innovativeness, transformative impact, and technical merit of the application while blind to the investigator(s) and institution(s).
After the technical critiques are completed, the process shifts back to the Editorial Board for the final phase of review. Informed by the technical reviews, the Editorial Board will prioritize applications for discussion. At this point, the board will have access to the complete applications and formally designate which applications are “Not Discussed.” To comply with regulations, the scoring of applications will utilize the five standard review criteria, including the investigator and environment, as well as other appropriate reviewable matters such as the use of vertebrate animals, human participants, and the Leadership Plan for multi-Principal Investigator applications. The Editorial Board will provide individual, numerical scores for each discussed application from which the overall impact score will be calculated. The applications then go to the NIH Council of Councils for second-level review.
We are excited to see if this new anonymization approach, with the identity of institutions and researchers hidden during the first phases, will lead to more diversity in investigators on applications and ideas proposed as well as its effect on scoring.
To help us support the best possible science across the nation, we would like to see the application pool reflect the full diversity of potential investigators, applicant institutions, and research areas relevant to the broad mission of NIH. Talented researchers from diverse backgrounds (see NOT-OD-20-031) are strongly encouraged to work with their institutions to apply.
This is an excellent step in the right direction. I have sat on a number of NIH review panels and have definitely seen the bias on the part of reviewers.
It appears to reduce bias. Are there any discussions that this new format of review should apply to all grant proposals?
Cannot hold to say like this! Not clear if applicants can see the critiques written before identities of PI and Institutions are revealed.
The intent is good but I am not very sure if this initiative will work. In many cases, the reviewers will be able to guess the identity of the applicant and institution and will be even more biased.The subject of the study, approach and methodology used, and references cited are all clues for identifying the PI and institution. It is a frustrating problem!
Why not anonymize across all study sections? Biases exist across the board in at all levels of the NIH review process. I have a colleague who has a name that could be male or female, and for the same application, he consistently gets worse reviews when it is assumed he is a woman! There’s your controlled experiment!
I think this may increase bias. The problem with unequal outcomes in review may not be due to reviewers thinking “this is a white guy from a prominent place, it must be good research”. Bias can creep in because certain topics and language that are more common to privileged applicants become favored. Some reviewers may attempt to correct for this by giving applicants from less-resourced institutions and backgrounds a leg up in scoring. If you anonymize you remove this opportunity.
Why doesn’t Common Fund just do the simple thing and try out giving small bumps to historically less privileged applicants when it comes to funding decision? Removing bias is impossible, just try out applying a little reverse bias at the margins!
I think you are missing the point here. It is not about less privileged, it is about scientific merits. Look at the neurodegeneration field for example, there has simply been no progress for several decades but the grants are going to only the people who work with biases for the aggregates, which has been proven again and again to be the wrong thing. May be, you do give a leg up to some applicants but reviewers like you are far fewer. The bias is what makes the research slow. Look at COVID-19 research, since it affects everyone and the results are put to test almost immediately, outcomes are a lot better than many other fields where accountability and bias and friendship, rather than scientific merits are responsible for the grants to be funded.
You are making my point for me. What you are saying is that reviewers in your field have bias for particular topics. Based on disparity in outcomes, it must be that these sorts of favored topics result in better results for privileged demographic and geographic groups. (See HRHR advisory committee report).
Stated purpose of this move is to remove institutional and demographic bias. My money is on it doing the opposite.
Anyone who has submitted a few dozen grants with wild outcomes knows that it is absurd to think an objective “Scientific merit” can be determined. But at least we can try not to step on the necks of less priveleged PIs. Don’t get me wrong–I think NIH grant review is the least biased of all the agencies I submit to. But it’s because they are proactive. Works well with ESI bumps and now with first-renewal bumps. Let’s just apply more bumps to under-represented demographic and geographic groups and not make the review process any harder than it needs to be!
This is encouraging.There is hope in academia and hopefully this can be replicated in all grant awarding agencies-NIFA et al.
First of all it is a super idea, which was also suggested earlier by several bloggers. I would suggest it is a great beginning towards honesty. Similar to the research, this system would be evolved/improved further by incorporating innovations. In that aspect I would like to suggest:
1) Please ask every applicant to submit 2 page of grant outline without any reference, just concept, innovation and significance.
2) Since every applicant is eager to contribute in the reviewing process, every grant should be evaluated and discussed (not just 50%), given the fact that now everything can be done online, and thus, economically.
Not a good idea, at the very least Idealistic and impractical. How am I going to refer to all my past successes, which are unique to my lab? Reviewers are not stupid — they are going to infer that it’s me writing the grant. Or worse: now someone else can write a project pretending it’s my lab, knowing that the reviewers cannot know whether they have the background and expertise to carry on the project. Oh wait — I can do the same and pretend to be someone else … (!). Like I said, not a good idea.
You don’t have to! that’s the idea. I think we all know the most people who are proposing the research are generally well qualified in that line of research. The idea is to know whether the research is innovative or the bias is working toward providing that grant a favorable score. I participated in this kind of an endeavor by the NIH and almost everytime, I scored a grant good when I did not have the name of the person. Yes, it is a little hard but most of the times you end up selecting grants that are more meritorious.
Absolutely essential. The current peer-review process is very Biased and does not really take into account scientific merit. This makes the transformative nature of the research very very slow because good ideas are tossed out the window by a biased set of reviewers.
Anonymous review is relatively easy to do in cases where no preliminary data is needed and require high risk and high impact like this solicitation. R21 might be good to consider for the approach too. Expertise will be considered in the later stage of the review. Reviewers review over 8 proposals and are generally busy. They are unlikely to spend time to do something against policy and to guess identities. Some concerns in the comments by others may be valid, but the new policy is great.
Dr Benner has been working on extending the genetic code since at least the late eighties. I should know: I am one of his former students. His ‘vision’ was all over the press and has been widely publicized during the following decades. No way this was an ‘anonymous’ submission.
To clarify, the anonymization pilot for this program is starting now.
It would be even better to give the option for PIs to submit several ideas before full proposal for review of scope. Description of each idea is limited to e.g. 300 words and # of ideas for a PI at a cycle is no more than 3 (for reducing reviewer load). For example, “Develop artificial genetic system with eight building blocks that can increase the amount of information a given stretch of DNA can carry”, “Develop a system that can replicate copies of DNA exponentially in a test tube (the now widely used PCR)”, “Develop a tech that can assemble genes directly on an automated synthesizer”, “Methods for precise determination of epigenetic modifications of DNA and RNA”. Or in real cases, give a few more sentences for each idea.
The PI provide keywords to assist NIH for selecting reviewers.
For each idea, reviewers (anonymous) only need to select from “potentially competitive idea”, “potentially noncompetitive idea”, or “not easy to judge”, and optionally give comments.
With the information, it is still the PI’s responsibility to select an idea for full proposal.
The goal of the extra efforts for everyone involved is to reduce (not eliminate) the possibility of a proposal being rejected by reviewer comments such as “the scope of the project is too narrow”, “no one is going to use the tech”, “the project is noncompetitive”. By the end, the extra efforts should be able to save overall human resources for science (PI, reviewers and NIH).
The eRA makes such otherwise complicated process feasible and potentially easy.
This is perhaps suitable only for the HRHI programs.
On this issue, the following article, Sanders, DA “Peer Review Should Be a Two-Stage, Science-First Process.” Times Higher Education , 3 Apr. 2019, might be helpful.
The long discussed idea of anonymous review implemented for this TRA program is a bold move that I welcome. Not only because it will increase demographic diversity but also because it places the scientific merit of the actual project and not the investigator’s name and fame at the center of the evaluation. This will prevent the notorious bias towards already famous investigators.
I still welcome the anonymous review process as a tool to diversify the research since diversity of ideas is most important to innovation. Peer-review and associated self-fulfilling prophecy suffer from the dynamics of the self-narrowing of views because the already famous suck up all the grants. The inaptitude of reviewers who BTW are barely vetted by NIH for such an important role, manifest in the need to “google” an idea to assess innovation, is a major impediment to advance in science!
I hope that the program managers and SROs and panel chairs will be more careful in instructing reviewers how to evaluate innovation in the anonymous review process: in a way that must prevent mistaking the footprints that the pioneer applicant themselves have left in a pristine field from being mistaken as indication of a path already taken by others.
Remarkably, years after this post, I will push back on this idea that is popular in the above comments. I understand the appeal of PI anonymity in grant application. “Only the idea will be judged, not the prestige of the institution or the fame of the PI”. Who can argue with that? Especially one who is not at a prestigious institution (I am not at such an institution). But here is the problem: the quality of an application is a combination of TWO factors: 1) How good the idea is and 2) How likely is it that the PI can execute it. Anonymous review completely eliminates the information needed to judge #2. Don’t people understand how difficult it is to execute studies? Writing about them is relatively easy in comparison, actual execution is much much harder. It seems really unfair for the hard work of doing quality science and establishing a track record of execution is just ignored with this approach. Is there any evidence that the PI can actually DO the project? With this approach you replace one problem (the potential for institutional/prestige bias) with another one (ZERO information that lets you know how likely the PI can _actually_ do the work, as opposed to write about it in a convincing manner. We need a better solution than just ignoring the prior record of execution by the PI.
“How likely is it that the PI can execute it.” WILL be strictly evaluated AFTER the scientific merit is evaluated. This should be true for any proposal review policy.
Evaluation of PI and environment AFTER merit review may be more fair. For example, PI productivity can be evaluated in the context of the resources he or she recieved in the past, their career status, etc. It is inappropriate to expect a PI with $100K startup fund with 2 graduate students to have the same level of productivity as a PI with $2M with 20 graduate students.