4 Comments
We previously referenced Ioannidis’ and Khoury’s “PQRST” mnemonic for describing research impact: “P” is productivity, “Q” is quality, “R” is reproducibility, “S” is sharing, and “T” is translation. We wrote several blogs about “P,” productivity, focusing on publications, citations, and more recently the Relative Citation Ratio. Now we’ll focus on a different kind of “P” for productivity, namely patents (which arguably are also related to “T” for translation). We’ll also take a brief look at “S” for sharing.
In the April 7, 2017 issue of Science, Danielle Li [now with the Massachusetts Institute of Technology (MIT)], Pierre Azoulay (MIT), and Bhaven Sampat (Columbia University) published an investigation on the patent productivity of NIH grants. They identified over 365,000 grants NIH funded between 1980 and 2007, and linked them to patents. Two kinds of links were identified: “direct” links in which a patent cited an NIH grant, and “indirect” links, in which a patent cited a paper which in turn acknowledged support from an NIH grant.
The authors found that close to 10% of grants directly generate a patent. That’s remarkable! But perhaps even more so, nearly 30% of grants generate a paper that is later cited by at least one patent. Even more remarkable, grants directly and indirectly generated patents whether they were “disease-targeted” or not, “patient-oriented” or not, or linked to a Request For Application or not. And, large proportions of grants assigned to different models directly and indirectly generated patents – models including humans, primates, rodents, invertebrates, multicellular eukaryotes, unicellular eukaryotes, prokaryotes, and viruses.
Another noteworthy feature of this paper is that the authors freely shared their data and statistical code. We took advantage of this to ask a question: do NIH-supported papers that are cited by patents have a higher Relative Citation Ratio than those that are not cited by patents? As a refresher, the Relative Citation Ratio uses citation rates to measure the influence of a publication at the article level. 
We identified 119,674 unique NIH grants that were funded between 1995 and 2007 and that generated at least one publication. Of these grants, 46,002 (38%) generated at least one publication that was later cited by at least one patent. The grants generated 1,241,307 publications that appeared between 1995 and 2015; of these, 103,421 (8%) were cited by at least one patent.
Figure 1 shows a box plot of the Relative Citation Ratio of papers that were or were not cited by at least one patent. The Y-axis (Relative Citation Ratio) is log-transformed to reflect the log-normal distribution. Papers cited by a patent had a higher Relative Citation Ratio (median 1.75, IQR 0.85-3.62 compared to papers not cited by a patent median 0.97, IQR 0.46-1.91). For convenience, we drew a dotted line through the median value of RCR among the papers cited by a patent. The large dots represent the mean RCR values.
Figure 2 shows the Relative Citation Ratio of papers according to the number of patents citing them. There is a gradient, with Relative Citation Ratio increasing as papers are cited by zero, one, two, three, or more than three patents (median values of 0.97, 1.46, 1.66, 1.87, and 2.40). For convenience, a dotted line goes through the median RCR (1.46) for papers citing one patent.
Taken together, the data presented here suggest that the number of publications cited by a patent positively correlates with a higher relative citation ratio. In other words, when patents cite a publication, that article is also likely to be highly influential in its field.
These preliminary findings show one way we are continuing to explore research impact beyond bibliometrics. Though helpful, focusing on bibliometrics alone does not completely capture productivity and impact of our funded research programs. The analysis we present here attempts to build upon prior work by adding yet another instrument to our toolbox.
We recognize that this correlation between patent citation and relative citation ratio may be correlative, not causal. With that noted, both measures do still provide us with a glimpse into the influence of the NIH research portfolio. Our findings are consistent with prior findings showing that the relative citation ratio also correlated with post-publication peer review.
And finally, the “S,” sharing that is…
We are pleased to hear about ways researchers use our data to empirically analyze the productivity of NIH-supported research. We congratulate the authors of the Science article, and commend their willingness to share their data. We progress towards our goal of enhanced transparency and stewardship when researchers share data with each other and when funding agencies share administrative data. Ultimately, sharing information this way is how we, together, improve human health and reduce illness and disability.
It’s a very beautiful article. “In conclusion, sharing information in this way is to improve human health, reduce illness and disability.” I agree.
Actually, the figures from the Science article aren’t terribly impressive. “30% of grants were later cited by a patent.” [para]
I believe the Science paper only looked at drug patents….so, that means 70% of private sector Pharma and biotech IP did not leverage NIH science at all?! Of course that’s ignoring knowledge spillovers, which certainly take place. But I would have thought given the quality of NIH science, that >50% of patents cite an NIH-funded study – especially since paper citations in patent applications can often be added by the lawyers, not the inventor, looking to strengthen the application and put their best foot forward (note: this also would limit these paper’s use as a proxy for actual impact on the patient’s intellectual/scientific basis, if the inventor didn’t cite them).
Anyway, that’s all to say that the 30% statistic is actually quite weak, IMO. As is the 10% figure for direct patents. Nearly every innovation in the life sciences requires IP protection. Only 1 in 10 do this from NIH? On the face of it, it necessarily implies that the other 90% is not innovative. Again, we know this to be not true since innovation can take many forms, but it’s a data point whose contrapositive sounds dangerous. However, policymakers looking for more bang for their federal buck, and more innovation from NIH, might say: “sure, the other 90% might be ‘innovative’ in some way, but we’re interested in the innovations that lead to capital formation and economic growth: new technologies, companies and jobs in the life sciences. Why does only 10% grants create that outcomes?”
The paper also misses an important policy question: is 10% a good enough yield, and if patents are desirable (which they should be if NIH is to be seen by Congress as an economic generator), how do we get more, what programs produce the most, what has their demand and budget been, and is there productivity slack in the system? These questions would ensure sustained congressional interest in keeping NIH funded for many years.
Dan, that 30% of grants are later cited by a patent doesn’t mean 70% of patents don’t cite NIH-supported research. These are two different percentages with different denominators: percentage of grants that are cited by a patent (citing patents / # of grants) doesn’t say anything about how frequently patents cite NIH-supported research (citing patents / # of patents). The study didn’t look at this latter question.
Same thing with respect to your question “Only 1 in 10 do this from NIH?” No, again wrong denominator. We don’t know what this figure is and it can’t be determined from the data in this study.
Correction: I should have noted that the numerators are different as well. The 30% figure in the paper is (number of grants cited by one or more patent) / (number of grants). This is different than the question posed by Dan, which is (number of patents citing at least one NIH grant) / (number of patents).