Last April we posted a blog on the measurement of citation metrics as a function of grant funding. We focused on a group of R01 grants and described the association of a “citation percentile” measure with funding. We noted evidence of “diminishing returns” – that is increased levels of funding were associated with decreasing increments of productivity – an observation that has been noted by others as well.
We were gratified by the many comments we received, through the blog and elsewhere. Furthermore, as I noted in a blog last month, our Office of Portfolio Analysis has released data on the “Relative Citation Ratio,” (or RCR) a robust field-normalized measure of citation influence of a single grant (and as I mentioned, a measure that is available to you for free).
In the follow-up analysis I’d like to share with you today, we focus on a cohort of 60,447 P01 and R01-equivalent grants (R01, R29, and R37) which were first funded between 1995 and 2009. Through the end of 2014, these grants yielded at least 654,607 papers. We calculated a “weighted RCR” value for each grant, essentially the sum of the RCR values for each paper. Hence, if a grant yielded 4 papers with RCR values of 20, 0.5, 0.8, and 2.0, the weighted RCR for the grant would be 20+0.5+0.8+20= 23.3 (see Figure 1, panel A for an illustration).
However, there is an important twist, which a number of you noted, and which we had to some degree correct for in our previous publications. What if a paper acknowledges support from more than one grant? It would not be appropriate to count the paper multiple times without somehow accounting for the multiple-grant acknowledgement. Table 1 and Figure 1, panel B shows the approach we took: we calculated a weight for each grant according to the proportion of money that grant accounted for. Thus (as shown in Table 1 and the first paper in Fig. 1, panel B) if a paper cited 3 grants, and one grant accounted for 2/3 of the total dollars of the 3 grants, the first grant received 2/3 of the credit. If that paper had an RCR of 20, the first grant would be credited with weighted RCR from that paper of 20 * 2/3, or 13.3. The remaining “0.666” of RCR would be distributed to the other two grants according to their respective budgets.
Before delving into the data, we should note two other important methodological points. All budgets were inflation-adjusted for the Biomedical Research and Development Price Index (or BRDPI) with 2015 as the reference year, and the project duration included all years the grant was active, including no-cost extension years.
Now, let’s look at the data. Table 2 shows selected characteristics for P01 or R01 (or equivalent) awards. As would be expected, P01 budgets were much higher. P01 grants also had longer grant durations and were more likely to include human or animal subjects. They yielded more publications (median 30 vs. 6) but the number of publications per $ million was a bit lower (median values of 2.9 vs. 3.2).
Figure 2 shows box and violin plots of the weighted RCR for P01s and R01-equivalents. Note that the Y-axis is logarithmically transformed given the highly skewed distribution. The P01 grants had higher weighted RCR values; also the R01 grants had a substantially larger of RCR values of zero, meaning no publications yielding any substantive number of citations.
Figure 3 shows weighted RCR per $ million for P01s and R01-equivalents. Now the differences are much less. For convenience, the dashed lines correspond to the median, 25th, and 75th percentile values for P01 grants.
Figure 4 shows the mean RCR per grant (that is mean RCR for all papers associated with each grant) for P01s and R01-equivalents. The mean RCR values are somewhat higher for the P01 grants.
Figure 5 shows scatter plots with cubic spline smoothers for weighted RCR according to total award, activity code, and project duration. Both axes are logarithmically transformed as both budgets and output were highly skewed. The dotted lines correspond to the slope of short-duration lower-budget R01 grants. As expected, weighted RCR values are higher for longer-duration grants. Weighted RCR also increases with budget, but with diminishing returns. P01 grants have higher RCR, particularly with higher budgets; however, these differences should be interpreted with caution given that P01 grants are multi-component projects.
Figure 6 shows corresponding scatter plots for mean RCR according to total award, activity code, and project duration. Now the differences between P01 and R01 grants are less pronounced, but the evidence diminishing returns are evident.
Because there are a number of variables that might be associated with grant productivity, we next performed a multi-variable regression (using a random forest machine learning algorithm). Figure 7 shows the main results – the most important predictors of weighted RCR for each grant were total award and project duration. Whether the grant is an R01 equivalent or P01, and other candidate predictors were less important.
Figure 8 shows an important output of the multi-variable regression – the adjusted association of weighted RCR with award. The two vertical lines show the 5th and 95th percentiles. The dotted line labeled “Proportionate Returns” corresponds to a state in which a 10% increase in award would be associated with a 10% increase in productivity. The lesser slope of the curve implies diminishing returns – that is, after accounting for other variables, a 10% increase in budget would be expected to yield less than a 10% in productivity.
Figure 9 shows the multi-variable adjusted association of weighted RCR with project duration. Not surprisingly, longer duration remained associated with higher productivity.
In summary, we find that in a larger cohort of grants and using the robust method of Relative Citation Ratio, we again see evidence of diminishing returns. Higher budgets and longer duration of funding are associated with greater productivity, but not to a proportionate degree.
What are the policy implications of these findings? First, we must emphasize that citation impact is not the ultimate goal of funded research – it is only one of a number of valid measures of research impact. Nonetheless, our findings are consonant with the statement by Alberts, et al. who, while warning about the dangers of the current hypercompetitive environment, wrote, “Agencies should also be sensitive to the total numbers of dollars granted to individual laboratories, recognizing that—although different research activities have different costs—at some point, returns per dollar diminish. For that reason, we applaud the recent decision by the NIH to examine grant portfolios carefully before increasing direct research support for a laboratory beyond one million dollars per year.”
We greatly appreciate your feedback, and look forward to more. In future analyses, we hope to consider a variety of metrics to enable us to better understand the biomedical research ecosystem and the possible associations of our policies on its productivity – ultimately its strength to lead to greater knowledge and better public health.