We previously referenced Ioannidis’ and Khoury’s “PQRST” mnemonic for describing research impact: “P” is productivity, “Q” is quality, “R” is reproducibility, “S” is sharing, and “T” is translation. We wrote several blogs about “P,” productivity, focusing on publications, citations, and more recently the Relative Citation Ratio. Now we’ll focus on a different kind of “P” for productivity, namely patents (which arguably are also related to “T” for translation). We’ll also take a brief look at “S” for sharing.
In the April 7, 2017 issue of Science, Danielle Li [now with the Massachusetts Institute of Technology (MIT)], Pierre Azoulay (MIT), and Bhaven Sampat (Columbia University) published an investigation on the patent productivity of NIH grants. They identified over 365,000 grants NIH funded between 1980 and 2007, and linked them to patents. Two kinds of links were identified: “direct” links in which a patent cited an NIH grant, and “indirect” links, in which a patent cited a paper which in turn acknowledged support from an NIH grant.
The authors found that close to 10% of grants directly generate a patent. That’s remarkable! But perhaps even more so, nearly 30% of grants generate a paper that is later cited by at least one patent. Even more remarkable, grants directly and indirectly generated patents whether they were “disease-targeted” or not, “patient-oriented” or not, or linked to a Request For Application or not. And, large proportions of grants assigned to different models directly and indirectly generated patents – models including humans, primates, rodents, invertebrates, multicellular eukaryotes, unicellular eukaryotes, prokaryotes, and viruses.
Another noteworthy feature of this paper is that the authors freely shared their data and statistical code. We took advantage of this to ask a question: do NIH-supported papers that are cited by patents have a higher Relative Citation Ratio than those that are not cited by patents? As a refresher, the Relative Citation Ratio uses citation rates to measure the influence of a publication at the article level.
We identified 119,674 unique NIH grants that were funded between 1995 and 2007 and that generated at least one publication. Of these grants, 46,002 (38%) generated at least one publication that was later cited by at least one patent. The grants generated 1,241,307 publications that appeared between 1995 and 2015; of these, 103,421 (8%) were cited by at least one patent.
Figure 1 shows a box plot of the Relative Citation Ratio of papers that were or were not cited by at least one patent. The Y-axis (Relative Citation Ratio) is log-transformed to reflect the log-normal distribution. Papers cited by a patent had a higher Relative Citation Ratio (median 1.75, IQR 0.85-3.62 compared to papers not cited by a patent median 0.97, IQR 0.46-1.91). For convenience, we drew a dotted line through the median value of RCR among the papers cited by a patent. The large dots represent the mean RCR values.
Figure 2 shows the Relative Citation Ratio of papers according to the number of patents citing them. There is a gradient, with Relative Citation Ratio increasing as papers are cited by zero, one, two, three, or more than three patents (median values of 0.97, 1.46, 1.66, 1.87, and 2.40). For convenience, a dotted line goes through the median RCR (1.46) for papers citing one patent.
Taken together, the data presented here suggest that the number of publications cited by a patent positively correlates with a higher relative citation ratio. In other words, when patents cite a publication, that article is also likely to be highly influential in its field.
These preliminary findings show one way we are continuing to explore research impact beyond bibliometrics. Though helpful, focusing on bibliometrics alone does not completely capture productivity and impact of our funded research programs. The analysis we present here attempts to build upon prior work by adding yet another instrument to our toolbox.
We recognize that this correlation between patent citation and relative citation ratio may be correlative, not causal. With that noted, both measures do still provide us with a glimpse into the influence of the NIH research portfolio. Our findings are consistent with prior findings showing that the relative citation ratio also correlated with post-publication peer review.
And finally, the “S,” sharing that is…
We are pleased to hear about ways researchers use our data to empirically analyze the productivity of NIH-supported research. We congratulate the authors of the Science article, and commend their willingness to share their data. We progress towards our goal of enhanced transparency and stewardship when researchers share data with each other and when funding agencies share administrative data. Ultimately, sharing information this way is how we, together, improve human health and reduce illness and disability.