How Grant Success Rates Do (Or Do Not) Track With the NIH Budget: A Model of Funding Dynamics

From 2014 to 2018, the NIH budget increased every year, and yet, the Research Project Grant (RPG) success rate remained relatively constant at ~20%. From 2003 to 2006 the NIH budget remained relatively flat, yet the success rate decreased dramatically from 30% to 20%. Why don’t success rates neatly track the NIH budget? While inflation plays a role, there are more fundamental forces at play.

Here I present a series of budget and success rate scenarios that are largely based on a Service Science publication by Larson, Ghaffarzadegan, and Diaz. The bottom-line up front: NIH funding dynamics are strongly affected by obligations incurred by grant awards made in previous years – what we call “out-year obligations” – and by responses of the research community to NIH budget increases.

As a reminder, Success rates are calculated by dividing the number of awards made in a fiscal year (FY) by the number of applications received (see also our recent By The Numbers post). the calculation includes applications that are peer reviewed and either scored or unscored by an Initial Review Group. Applications having one or more amendments in the same fiscal year are only counted once.

Imagine a simplified funding agency that receives an annual budget of $1 Billion (or $1000 million). This budget has been constant for years. The agency only spends money on competing or non-competing grant awards (in other words, there is no overhead for agency management). Each award costs exactly $1 million each year and lasts for exactly 4 years. We also assume that there is no inflation and that every year the agency receives 1000 competing applications.   

At the beginning of each fiscal year, the agency is already obligated to award $1 million for the second-year non-competing renewal of each award given last year, for the third-year non-competing renewal of each award given two years ago, and for the fourth-year non-competing renewal of each award given three years ago. That means that as the year starts, three-quarters (75%) of the agency’s budget is already obligated to support ongoing awards awaiting expected non-competing renewals. Only 25% of the budget is available for new competing awards.

For those of you who are mathematically oriented (if you’re not, you can safely ignore this paragraph), per Larson and colleagues we can say that:

F(y) = B(y) – [F(y-1) + F(y-2) + F(y-3)]

where F(y) is funding available for new (competing) grants in any given year y and B(y) is the total budget available to the agency for all grants, competing and non-competing, in any given year y.

Table 1 illustrates what this steady-state scenario looks like. In each Fiscal Year (here Year 1 to Year 4) the agency can spend $250 million for new awards; as each award costs $1 million, that translates to 250 awards out of 1000 applications or a success rate of 25%.

Table 1: Baseline steady state – The second column is the budget allocated to the agency. The 3^rd and 4^th columns refer to funds available for new (competing) awards and funds obligated for out-years of previous issued awards.  In this table and all others, the second column is B(y) and the third column is F(y).  

FY	Budget ($M)	Competing ($M)	Out-Year ($M)	Applications (N)	Success Rate (%)
1	1000	250	750	1000	25%
2	1000	250	750	1000	25%
3	1000	250	750	1000	25%
4	1000	250	750	1000	25%

Now, we imagine that that in Year 5 Congress appropriates a budget increase, from $1000 million to $1100 million per year. As in all previous years, the agency begins the year with out-year obligations of $750 million, which come from $250 million of new awards first issued in Year 4, $250 million of new awards first issued in Year 3, and $250 million of new awards first issued in Year 2. We need not worry about the $250 million of new awards issued in Year 1 as they rotate off after funding in Years 1 through 4; recall, that each award receives $1 million a year for exactly 4 years.

So, as shown in Table 2, Year 5 begins with $750 million of out-year obligations, but with a $100 million increase in total budget, the agency now has $350 million available for new awards. That translates into a dramatic increase in success rate from 25% to 35% (again there are 1000 applications)! Larson and colleagues refer to this magnified increase in success as a cause of “euphoria.” But, in the following year – Year 6 – the agency now finds itself with $850 million in out-year obligations because of the additional 100 awards issued in Year 5. With a flat budget of $1100 million, there are “only” $250 million available for new awards, meaning that the success rate falls right back to 25%. As shown in Table 2, the success rate remains at 25% — despite the higher budget compared to Years 1-4 – until Year 9 when the 100 additional grants issued in Year 5 (four years before) come to an end. The success rate bounces back up to 35% (like in Year 5) and then promptly falls back in Year 10 to 25% (like in Year 6). These every-four-year oscillations would continue unless the agency took steps to mitigate them.

Table 2: Scenario with an increase from $1000M to $1100M, then constant budget

FY	Budget ($M)	Competing ($M)	Out-Year ($M)	Applications (N)	Success Rate (%)
5	1100	350	750	1000	35%
6	1100	250	850	1000	25%
7	1100	250	850	1000	25%
8	1100	250	850	1000	25%
9	1100	350	750	1000	35%
10	1100	250	850	1000	25%

How could the agency maintain the higher success rate of 35% in Years 6 through 8? One approach, shown in Table 3, would require a Congressional budget increase of $100 million in each of those years. Note that despite increasing budgets, the success rate remains flat; this is because of the out-year obligations coming from previous years when the agency issued more awards than in the baseline state (Table 1).

Table 3 also shows what happens if the budget remains flat in Year 9, four years after the initial increase in Year 5. The funds for new awards and the success rate now are flat; effectively, after 4 years (the length of time for all awards) the agency achieves a new steady state.

But now suppose that the agency’s budget is cut in Year 10, from $1400 million to $1300 million. The success rate promptly drops from 35% (where it has been for years) to 25%. A 7.14% decrease in the budget leads to a 28.57% fall in the success rate. In other words, each 1% cut in budget translates to a 4% cut in funding for new awards! As discussed by Larson and colleagues, out-year obligations lead to magnified effects of budget changes.

Table 3: Scenario with steady budget increases in Years 5-8 followed by brief stability and then a $100 million budget cut.

FY	Budget ($M)	Competing ($M)	Out-Year ($M)	Applications (N)	Success Rate (%)
1	1000	250	750	1000	25%
2	1000	250	750	1000	25%
3	1000	250	750	1000	25%
4	1000	250	750	1000	25%
5	1100	350	750	1000	35%
6	1200	350	850	1000	35%
7	1300	350	950	1000	35%
8	1400	350	1050	1000	35%
9	1400	350	1050	1000	35%
10	1300	250	1050	1000	25%

Up until now, we have assumed that the number of applications remains constant at 1000 per year. But, as Larson and colleagues point out, that’s not reflective of reality. Increased agency budgets stimulate growth in research capacity: more scientists, more facilities, more students, more trainees. The agency sees more applications (like we saw in the years of the NIH doubling from 1998-2003). In Table 4 we show the same budget scenario as in Table 3, but we have the number of applications increase by approximately 5% per year starting in Year 6. The combined growth in applications (reflecting the growth in research capacity) plus the increase in out-year obligations conspire to cause a markedly dramatic decline in success rate in Year 10 to only 20%, below where we were during the steady state Years 1-4. It’s a “pay-line crash” much like what happened at NIH between 2003 and 2006.

Table 4: Scenario with steady budget increases in Years 5-8 followed by brief stability and then a $100 million budget cut. In contrast to Table 3, extramural research capacity growth beginning in Year 6 leads to a growth in the number of applications.

FY	Budget ($M)	Competing ($M)	Out-Year ($M)	Applications (N)	Success Rate (%)
1	1000	250	750	1000	25%
2	1000	250	750	1000	25%
3	1000	250	750	1000	25%
4	1000	250	750	1000	25%
5	1100	350	750	1000	35%
6	1200	350	850	1050	33%
7	1300	350	950	1100	32%
8	1400	350	1050	1150	30%
9	1400	350	1050	1200	29%
10	1300	250	1050	1250	20%

During this discussion we started with a simple “steady-state” and gradually added new complexities: one budget increase, multiple budget increases, a budget cut, and finally an increase in the number of applications. Of course, the real world is much more complex. There are so many other factors to consider: different grant mechanisms, varying grant durations, training programs, small-business set-asides, intramural research, inflation which itself varies from year to year, and shifting priorities. But, I am reminded of a saying attributed to George Box, a statistician: “All models are wrong, but some are useful.”

In their paper, Larson and colleagues discuss steps an agency can take to mitigate the magnified effects of budget changes. These include varying grant durations and taking steps to “optimize” budget increments. Other options include reducing budgets on non-competing awards, down-negotiating budgets on new, or reducing the number of new awards while making sure that among those awards a maximum number of scientists are supported. None of these options are easy or straightforward, yet we remain committed to doing what we can to support as much outstanding biomedical research as possible.