14 Comments
As a followup to my recent blog post on fiscal year 2012 success rates, I’d like to post an update of an earlier blog post where I explained how paylines, percentiles and success rates relate to one another. It’s a long one, but should be helpful in understanding what we mean when we look at success rates.
“Paylines, Percentiles and Success Rates” with updates added:
I have read or heard much about the dilemma of NIH applicants as they struggle to understand their chances of receiving NIH funding. As budgets flatten and tighten, this discussion has heated up. To declare that NIH success rates have hovered around 20% for the past five years does little to calm the storm of concern when we hear about shrinking percentiles and paylines. So how is it possible to have a success rate of 20% but a payline at the 7th percentile? Let’s take a few moments to sort out what these things mean and think about how these numbers are derived and how they can differ.
IMPACT SCORE
It all starts with the impact. This score is assigned by reviewers to indicate the scientific and technical merit of an application. Impact scores range between 1 and 9. A score of “1” indicates an exceptionally strong application and “9” indicates an application with substantial weakness. (I always wondered why at NIH low = good and high = bad but that predates me!) In assigning an impact score, reviewers consider each of five scored criteria: significance, investigator, innovation, approach, and environment, along with other factors like protection of human subjects and vertebrate animal care and welfare. Read more about scoring.
PERCENTILE RANK
The percentile rank is based on a ranking of the impact scores assigned by a peer review committee. The percentile rank is normally calculated by ordering the impact score of a particular application against the impact scores of all applications reviewed in the current and the preceding two review rounds. An application that was ranked in the 5th percentile is considered more meritorious than 95% of the applications reviewed by that committee. This kind of ranking permits comparison across committees that may have different scoring behaviors. It is important to note than not all research project grant applications (RPGs) are percentiled. For example, applications submitted in response to a request for applications (RFA) are usually not percentiled. In the absence of a percentile rank, the impact score is used as a direct indicator of the review committee’s assessment. Read more about percentiles.
PAYLINE
Many NIH institutes calculate a percentile rank up to which nearly all R01 applications can be funded. For grant applications that do not receive percentile ranks, the payline may be expressed as an impact score. Institutes that choose to publish paylines in advance (see an example) calculate the payline based on expectations about the availability of funds, application loads, and the average cost of RPGs during the current fiscal year. Other institutes prefer to describe the process for selecting applications for funding (see an example) and then report on the number of applications funded within different percentile ranges at the end of the fiscal year (see an example) Because the NIH is currently operating on a continuing resolution and funding levels for the remainder of this fiscal year are uncertain, most of the NIH institutes have offered less detail this year than in the past.
But remember, even when an IC establishes a payline, applications outside of the payline can be paid under justified circumstances if these applications are a high priority for the particular institute or center. When these select-pay/out-of-order/priority pay/high priority relevance selections are made, it may result that other applications within in the payline are not paid because funds are no longer available to support them.
SUCCESS RATES
The success rate calculation is always carried out after the close of the fiscal year, and it is based on the number of applications funded divided by the number of applications reviewed and expressed as a percent. To better reflect the funding of unique research applications, the number of applications is adjusted by removing revisions and correcting for projects where the resubmission (A1) is submitted in the same year as the original application (A0). Read more about success rates.
THE ANSWER
Now we are equipped to answer our earlier question. How is it possible to have a success rate of 20% but a payline at the 7th percentile? There are several real-life reasons why paylines (the ones that use percentiles) can be either higher or lower than success rates.
- Applications that are not percentiled are still factored into the success rate calculation. Thus, funding a number of awards that are not assigned percentiles will increase the success rate without changing the payline.
- The success rate for a particular fiscal year is a reflection of the funded applications and can include applications reviewed in the previous fiscal year; whereas, the payline encompasses only applications reviewed in that fiscal year. So awarding applications that were reviewed in the previous year will also increase the success rate.
- The average quality of the applications assigned to an institute will also affect its payline. If an institute happens to receive a set of applications with very good (low) percentile scores, its success rate will be higher than its payline, all else being equal. For example, in fiscal year 2011, the NIGMS R01 success rate was about 24% but the midpoint of the funding curve occurred close to the 19th percentile.
Check out more reports on RPG success rates broken down by year and IC at report.nih.gov – if you’re interested in other success rates, you can find them on our RePORT website as well.
Whew, you made it through. The difference between paylines, percentiles and success rates remains a confusing topic because of the compounding factors that rule out a simple linear relationship. You need to consider all the factors when assessing the potential for an individual application to be funded. Your best advisor on this issue, because of the differences in the ICs and programs, is your NIH program official. Give him or her call.
Does the 18% (success rate) figure include competitive renewals? Does the 7th percentile calculation include competitive renewals?
What is the success rate for new applications alone, without competitive renewals?
Jim
Yes, Jim, the success rate and percentile calculations do include competing renewals. The success rate for new R01s in 2012 (which does not include competeing renewals) is 15%.
If we have a priority score very close to the payline, when will we know if it will (possibly) be funded?
That is a question that should be posed to the NIH program director who is responsible for the application, since the answer will depend on the procedures of the individual NIH institute or center.
You note that depending on the number of “select-pay/out-of-order/priority pay/high priority” grants that get funded, then applications within the payline can remain unfunded.
Can you provide some transparency as to how the decision is made as to what grants within the payline are chosen not to be funded? As worded, this sure seems odd and contrary to a merit based system. Perhaps you meant that based on the dollars committed to the select-pay group that the payline shifts upwards?
There is a lot of discussion about how the NIH R01 success rates should factor into tenure decisions. Can you parse out the numbers for both percentile and success rates for:
1) renewals
2) new proposals established investigators
3) new proposals new investigators
thank you
Brian Calvi
Success rates related to part 2 and 3 of your question are presented as a graph and table in the NIH Data Book. See “R01-Equivalent Grants, New (Type 1): Success Rates, By Career Stage Of Investigator“. Competing continuation success rate data is available as an Excel spreadsheet. Visit the Funding: Success Rates section of RePORT to view or download this, and more success rate data.
You indicate that success rate is based on grants “reviewed”. My understanding is that the 60% of the grants that are triaged are not considered”reviewed” and therefore the success rate is based on a much lower number than the number of grants submitted. If it also does not include revisions which i would imagine make up anywhere from 40-60% of grants each cycle can we really get a true number of number of grants submitted (not reviewed) vs funded. I would imagine we are now realizing a 3-7% success rate. Please comment on this since if this is correct than the 20% success rate is very misleading!
The success rate does include all reviewed grants: this PDF in the Success Rates section of NIH RePORT provides more details that you might find helpful.
I’d like to suggest that in an effort to describe success across the entire organization, Rock Talk’s efforts have allowed a broader question to be put. Why do that?
You see, what matters to a particular constituency (say an investigator doing preterm labor research and the advocacy folks that want more successful research in this area because prematurity is at one in eight pregnancies in the US, babies are dying and suffering life-long disabilities as a result!) is whether or not an investigator can receive a grant from NICHHD where there is a pay line problem.
Stated another way, by reporting institute-wide success rates without a context, NIH serves no constituent and does no good. Strong assertion? Yes. You see if NIH were to inform congress that, with one in five research grants being funded it means that US biomedical research infrastructure is crumbling and that in a given year, the chances of particular projects receiving support is as low as 1 in 20, then at least they would hear the problem. As it is, who is the audience for the 20% success rate number, and what is the context? The research community isn’t impressed, their quizzical and disbelieving. So what message is for whom? Congress is the only one that needs a message, we in medical science know were screwed. We know our graduate students and postdocs may have their careers interrupted and ruined if we can’t maintain their salaries. We already know faculty may be fired die to fiscal exigency. So who needs this carefully designed value called a success rate when our success in in the toilet!
ILOB
Success rates for each Institute and Center are published on RePORT, in the Success Rates section.
The use of the original application plus its revision as the denominator is THE problem with this system for describing the success rate. To the person submitting a proposal, the effort is as high, or higher, for the resubmission as for the original. Since successful applications require 1.8 submissions, the real success rate for new R01’s per submission would be 15/1.8 = 8.3%. This is what our academic administrators should know when they review their faculty.
Very helpful for academic investigators, but why does the SBIR/STTR program so often get treated differently? These grants are not percentiled (and thus normalized across IRGs) nor are paylines typically available. Even in RePORTER, study section information on SBIR’s are not available. Since there is increasing emphasis on translation and the SBIR/STTR is a great way to accomplish this, is there a way to bring it further into the mainstream at NIH?
The SBIR/STTR programs are Congressionally mandated, so they have unique requirements and are distinct from many NIH programs. However, NIH manages the SBIR/STTR programs following the same overall processes used for other grant programs, including peer review. SBIR/STTR grant applications are reviewed in Special Emphasis Panels. (See the full list of panels here.) The study section for small business grant awards are made available on the details screen for the individual grant award on RePORTER. SBIR/STTR applications, like many other NIH mechanisms, are not percentiled. Each NIH Institute/Center (IC) has a web page detailing their funding strategies, but as described in this post, some ICs do not publish their paylines for individual mechanisms: see our consolidated list of links to IC funding strategies to explore further. There are several STTR and SBIR-specific reports on available on the RePORT website, and you can also view success rate information for these programs on the RePORT Success Rates page.