25 Comments
It came to my attention that the analysis we posted last month on the numbers of investigators with multiple awards couldn’t be recreated. A reader took ExPORTER data from 2009 and came up with different numbers.
We have been attempting to replicate your data on the number of grants for the top 20% of PIs and cannot. Here is what we have come up with. Our total, using each of two assumptions about how you counted grants, falls short of your total number of PIs by more than one quarter. Can you explain how PIs were counted in your data in FY2009?
So, we asked them for their data and took a second look at our own. And, unfortunately, we posted the wrong data. These are complicated analyses, and we often go through multiple iterations of each analysis. We at the NIH are, sometimes, human, and we chose the wrong file from the folder. I apologize for this error, and I will provide you with the correct data here. In addition to the updated data for the investigators who receive the top 20 percent in total funding, we’ve added the data for all PIs.
Compared to what we posted previously for the top 20 percent, the absolute number of investigators has been corrected downward, but the percentage of investigators falling into each category remains about the same.
Many of you requested similar data for all PIs. If you crunch the numbers, you will see that in each of the four years presented more than 90 percent of our investigators hold one or two research project grants. I hope this helps clear up any confusion with the previous data.
| Number of Awards | Number of PIs (Top 20%) | Number of PIs (All) | ||||||
| FY 1986 | FY 1998 | FY 2004 | FY 2009 | FY 1986 | FY 1998 | FY 2004 | FY 2009 | |
| 1 | 1,156 | 1,084 | 1,041 | 1,140 | 13,257 | 15,873 | 18,720 | 18,934 |
| 2 | 1,638 | 2,055 | 2,237 | 2,345 | 2,738 | 4,100 | 5,811 | 5,428 |
| 3 | 416 | 862 | 1,526 | 1,341 | 439 | 930 | 1,656 | 1,410 |
| 4 | 73 | 183 | 400 | 326 | 75 | 185 | 406 | 327 |
| 5 | 17 | 35 | 112 | 66 | 17 | 35 | 112 | 66 |
| 6 | 4 | 5 | 21 | 13 | 4 | 5 | 21 | 13 |
| 7 | 2 | 1 | 7 | 3 | 2 | 1 | 7 | 3 |
| 8 | 0 | 0 | 3 | 2 | 0 | 0 | 3 | 2 |
| Total | 3,306 | 4,225 | 5,347 | 5,236 | 16,532 | 21,129 | 26,736 | 26,183 |
I cite an earlier post of Steve Kron’s (below), which I believe was one of the most explicit, data-rich, and succinct posts of the previous blog……20% of NIH funding goes to the top 2% of PIs, and 50% goes to the top 15%. The data that Steve provided supported an interpretation that the bulk of NIH funding is placed in the hands of a minority of PIs. For example, it appears that NIH Awards totaling more than $560 million dollars were granted to the top 25 PIs (of 34,902) listed in one spreadsheet……http://www.brimr.org/NIH_Awards/2010/AllPIs_2010.xls).
I did not investigate the details of what these awards were for, how many grants they were distributed across, nor all of the people being supported under these PIs. Likewise, I have no basis to conclude that these were not great projects from great people, whose results are likely transform science and/or health care. Nonetheless, these appear to be credible data and highly detailed data that support the notion that the NIH is very heavily invested in a few PIs. This raises the question, “Could those top 25 PIs have scraped by on a mere $460 million, thus leaving a $100 million in the kitty to fund the grant applications of 100 other scientists whose financial requirements are more modest?” I would guess that funding 5 times the number of scientists with the same amount of money would be more likely to yield more knowledge, than increasing the budgets of those 25 PIs from a collective $460 million to a collective $560 million.
It is up to everyone in the scientific community to judge for themselves whether or not investing the bulk of U.S. science tax dollars into a minority of labs / clinical research endeavors (1) optimizes the rate at which new and accurate knowledge is gained and (2) continues to encourage innovative scientists to dedicate their time and talents to helping us improve the state of biomedical knowledge such that we may reap the practical rewards….better drugs, better treatments, and improved human health.
– Bill Halford
———————————————————————————
Steve Kron says:
May 3, 2011 at 9:14 am
Some excellent NIH data here posted by Blue Ridge Institute for Medical Research for all you data hounds:
http://www.brimr.org/NIH_Awards/2010/NIH_Awards_2010.htm
Particularly interesting is this file:
http://www.brimr.org/NIH_Awards/2010/AllPIs_2010.xls
, which is a ranked list of principal investigators by funds awarded.
Ignoring issues of apples and oranges, running sums on these data suggests that 20% of NIH funding is going to the top ~2% of investigators and that 50% of NIH funding is going to the top ~15% , at about the $1M per PI line.
Comment removed at the request of the submitter 7/21/11 — Rock Talk Blog Team
Bill and others,
I believe those funds to the scientific ‘elite’ are mostly meant to establish ‘centers of excellence’ under a trusted leader. I also think one needs, after say 5 years from the final year of the initial grant period, evaluate the excellence of the centers. Did they accomplish what they were funded for? Was there synergy? My experience tells me that they likely published a respectable number of papers with a respectable number of citations, but I’m not so certain that on a cost basis ($/paper and $/citation) that they performed better than a group of individual investigators working in the traditional mode of ‘competitive-collaborative’ science. The NIH campus itself is an example of a center of excellence. Given the data bases, one could easily do the cost analysis. Dollars seem to be a powerful and common means of communicating economic and societal value so it might be worthwhile for someone to do the analysis. Also, there is a tendency for the larger groups to follow up on discovery leads generated by smaller groups. The latter is more difficult to ascertain without intimate knowledge of a field. Although, I recall the J. Biological Chemistry experimenting with knowledge maps linking publications through citations in the early 2000’s. I found those to be very revealing. Of course, all these stats probably will not be entirely justified because it is easy to envision exceptions to their value. For example, I know of efforts that took 25 years (not the typical 5-10 years) before there was a valuable yield of important data. Nevertheless, the stats would serve as a useful guide for discussion and debate.
These comments sound like they’re implying some kind of normative model, but it’s not obvious to me what we should expect/desire this relationship to look like. Quite clearly we shouldn’t expect all PIs to have equal funding. In a reasonable world, what proportion of NIH extramural funding should be going to the top 2% or 15% of PIs? Obviously it should be a lot more than 2% and 15% respectively. It’s not clear to me that 20% and 50% respectively is really disproportionate (where we expect the funding to be allocated in proportion to the cost and importance of the work, not in proportion to the number of PIs).
Hi Dan,
In total, I believe that I have observed three trivial explanations in these two blogs for why it is not surprising nor necessarily a bad thing that 50% of NIH funding goes to the top 15% of prinicipal investigators (PIs). These are, as follows:
1. Of course, the top 15% of PIs will be better funded than the bottom 15% of PIs. So, what’s the problem?
2. Some of the bottom 85% of PIs are just whining because they want to be in the top 15% of PIs.
3. Some PIs are whining because their grant applications were not funded.
In an abstract sense, your argument (#1) makes sense. Likewise, in an abstract sense, arguments #2 and #3 seem reasonable when taken at face value. However, based on the specifics of what I have observed in my career, I don’t think that these trivial arguments absolve the NIH nor address the problems that have led many of the “bottom 85%” of PIs to post comments in these two blogs.
I am certainly not writing in this blog to advocate the position that the NIH should take money away from big labs and give it to my lab. Rather, what I am suggesting is that the systems by which we choose which science gets funded should be more consistent with two realities that should be self-evident to anyone who has been in science for more than a decade:
(1) there are track records that tell us that some labs are more likely than others to generate good, reproducible, and useful data that move a field of study forward, and
(2) there are some labs that are more likely than others to properly train self-sufficient, competent scientists who place a higher value on solving health-related problems than padding their CVs with trivial publications.
Your post suggests a belief that the top 15% of NIH-funded PIs will necessarily have the best labs in terms of (1) production of quality science and (2) production of quality trainees. It has not been my observation that this is necessarily the case. Although many of the top 15% of NIH-funded PIs are likely to have the biggest labs, I generally find that big labs are often victims of their own success. They are perfectly positioned to capitalize on initial hypotheses that prove to be correct (i.e., confirm what they already suspected). However, they are far less flexible in their ability to self-correct than smaller labs. That is, incorrect hypotheses and/or erroneous assumptions tend to get swept under the rug or forgotten (rather than corrected), and the opportunity to learn from our mistakes (i.e., the backbone of science) is lost. Big labs can afford to terminate projects that become difficult; smaller labs are forced to tackle hurdles in a project that may be their only source of funding.
What I have observed at national / international meetings for 15 years leads me to suspect that, in general, there is an inverse correlation between (1) the number of people involved in a project (e.g., as judged by the acknowledgements within a talk) and (2) the importance of the scientific question being pursued within the talk. Does anyone really believe that systems biology or “personalized medicine” is going to have an impact on human health in the foreseeable future? My disdain for such talks is often compounded when the acknowledgements slide suggests to me that 10 or more trainees squandered a lot of time and money assembling this indigestible mess of data that I just witnessed tripping across the screen. I have no problem with large data sets, but firmly believe that they should be attached to an important question and a relevant answer.
In contrast, at the very same meeting, I will see a talk or poster given by an unassuming individual who is not politically connected, but who captures my interest and imagination, and conveys to me a new sense of appreciation for something that I had not previously considered. I want the NIH to fund these individuals, so that they can teach me more in years to come. Unfortunately, such PIs in the “bottom 85%” are far less likely to obtain money to pursue their passion than the better connected guy or gal who just bored my socks off with their $5 million dollar systems biology talk.
At the heart of the problem is a general acceptance that it is a noble or worthy pursuit for a P.I. to spend 100% of their time attempting to acquire and/or maintain more than $1 million dollars per year in NIH funding. Such individuals who “succeed” at this task are, by definition, less likely to have time to perform their own experiments (i.e., independent of their students, postdocs, or technicians). The most powerful tool that a scientist has at their disposal to verify the accuracy of the ideas being batted around in their field of study is that strange room adjacent to their office…..the lab. Scientists who run their own experiments are forced to remain anchored in the reality of what can and cannot be concluded from the methods presented in their own papers and that of their peers. As one spends too much time in an office, there is a tendency to become willing to give priority to the theory of “what should happen” over the reality of “what does happen” when one runs the experiment themselves. Each time that there is a disconnect between theory and reality, it is an opportunity to learn. If a PI is spending nearly 100% of their time writing grants, then that cuts into the time they have available to recognize the disconnects between theory and reality that frequently arise in the lab.
For me, the fundamental issue I would like to see considered within the NIH is an economics-like consideration of the question, “At what point does increasing the funding per PI yield a decreasing return on U.S. taxpayer investment?” Several bloggers have expressed this idea as “Publications per dollar.” I agree that this is a measurable unit, but philosophically would like to think that the actual quantity we seek to measure is “Reproducible and useful observations (knowledge) per dollar.” In other words, 2 real observations per year that permanently move a field of study forward should count for more than 20 observations per year that prove to be 90% wrong or useless in 10 years. This latter approach creates unnecessary work for everyone in the field. As in carpentry, scientists should stick to the mantra, “Measure twice, cut once.”
Dan, my experience is inconsistent with your supposition that the status quo of concentrating the bulk of federal dollars in the hands of 15% of PIs is likely to yield the optimal return on the ~$30 billion per year that U.S. taxpayers are investing in biomedical science. Rather, I think the key to maximizing the rate of real scientific progress is to recognize that the NIH funding system works well provided that you can fund 20 – 30% of applications to some level greater than $25,000 per year. For some reason, a difficult concept for NIH administrators to grasp is that there is nothing set in stone that says an R01 must provide $150,000 per year in direct costs to a PI, nor is it set in stone that a university must receive more than 25% in indirect costs. Smaller grants are an obvious potential solution to the disconnect between the budget and the growing numbers of unfunded PIs.
Bottom line: The NIH system works well provided that you can fund a reasonable fraction of the applications. However, the wheels come off the bus when NIH funding lines drop to 7 – 8% because the bulk of U.S. scientists’ time is diverted into writing and re-writing grant proposals rather than conducting science. By whatever means the NIH sees fit (e.g., lower indirect costs, less PI salary per grant, smaller total budgets per year, fewer years per grant), it would be sensible to create a funding system where good science (from any PI) has a reasonable chance of receiving at least some funding from the NIH in order to keep the work moving forward.
– Bill Halford
This data is meaningless WITHOUT INCLUDING DATA OF PIs WHO HAVE BEEN SUBMITTING R01s BUT ARE NOT FUNDED YET. Providing this data will show that the system is broken and there will be a large brain drain out of academia. I know many PIs at top institutions that are in years 4-6 without their 1st R01. NIH is ignoring this.
The $/paper or $/citation calculation should definitely be done. Despite all its flaws it should be revealing and likely to show diminishing return for the money. The papers should be weighted using their impact or Eigen factors and only counted when the PI is first or more likely last author. Having seen several mega labs at work its hard to believe that beyond a certain amount money is optimally spend.
It may be jealousy or my loathing of often futile grant writing but its hard to conceive that the ideas of the top $ getter is really worth 10 to 50 times more than the 70% of NIH grant recipient who make do with one basic R01. That level of funding is more likely to reflect very good political skills. Beyond large program with very clear goals like sequencing the human genome huge program are likely to also be less effective than investigator initiated studies. Not having an overflowing coffer does tend to focus the mind.
Ah, OK. Just what many of us suspected: The rich have been getting richer, while the poor have been getting poorer. More money in fewer labs. Is that the path to innovation?
Comment removed at the request of the submitter 7/21/11 — Rock Talk Blog Team
In almost any human endeavor, the greatest “bang for the buck” invested is the small business model, i.e. 1-3 people involved. Economies of scale only work when raw material costs are well-behaved (well known) and there is a great demand (number of potential buyers). I do not think the biomedical grant program of the NIH is in the business where economies of scale (good ideas for improving health; large demand for improved population health) can be taken advantage of.
The mission of the NIH is to promote human health. It would be instructive to see a few concrete examples of very large investments in a single PI that resulted in findings that all (or most) would agree were “worth it”.
Comment removed at the request of the submitter 7/21/11 — Rock Talk Blog Team
RE my previous comment. I made that remark with the full understanding that expected ROI on any single investigator is low, with most major advances gradually emerging from a mish-mash of dozens of “little” discoveries.
Just a quick note on the spreadsheet that Dr. Kron posted listing investigators and their NIH totals – those numbers are the total amount dispersed in a given year, including indirects. It is not completely accurate because winners of DP1 and DP2 awards, in which the money is dispersed in a single year, are obviously not getting funded at 2.6-3M per year. I was shocked to find that I was one of the top 1000 best funded investigators, until I realized this bit of strangeness….
Comment removed at the request of the submitter 7/21/11 — Rock Talk Blog Team
I agree with Paul Gardner! At least for my field, a lot of the people coming out at the top of the list are running large centers which benefit a LOT of scientists. Have you ever needed to look up a protein structure at the Protein Data Bank (PDB)? Someone ranked at roughly 1200 is using those funds to run the PDB, and it benefits you.
Also, the list from Blue Ridge Institute for Medical Research is misleading. When people have center grants (P01, U54, etc), it is all being attributed to one person. Really, those function as many related R01’s strung together with a common goal. That money isn’t going to one person. It is usually funding 5-10 PIs at an R01 level. In fact, many senior faculty role in a junior colleague or two into these centers, so the “rich” do help out the “poor”.
Comment removed at the request of the submitter 7/21/11 — Rock Talk Blog Team
So is it safe to conclude that the myth is actually true? I would like to see a followup blog that integrates all of the data interpretations and reinterpretations presented in this form for the purpose of testing whether or not the myth that the NIH investment is imbalanced is true. The analysis should also take into consideration all of the investigators once funded or never funded in order to put in perspective the full scope of this issue.
Comment removed at the request of the submitter 7/21/11 — Rock Talk Blog Team
These aggregate numbers are not indicative of anything like what people seem to be reading into them. The very largest RPG grants are the CTSAs, which are the very large grants necessary for supporting clinical research broadly across an institution. Program Project $s generally are not really comparable to R01 $, at the very least in terms of the number of scientists who benefit from them. The PIs with huge award totals are not doing the same thing with that money as your typical R01 PI. I think it would be more indicative to look at R01s only. Can you do that?
As I remember it, the original reason for posting this information was to dispel the myth that a small number of investigators were award multiple grants which limited the total number of investigators that could be funded. However, looking at the numbers that “myth” appears to be true. Based on the 2009 data, if each investigator was limited to 2 grants, an additional 2341 grants could be awarded. This would increase the pool of funded investigators by almost 10% !
These data pool numerous grant mechanisms, so the issue may not be as simple as the numbers indicate. But, NIH should think seriously about the valued gained by awarding a 6th, 7th or 8th grant to an individual versus providing a single grant to a PI to allow him/her to pursue their research. Very talented junior and senior scientists are leaving academics because they can not get funded.
Comment removed at the request of the submitter 7/21/11 — Rock Talk Blog Team