In a recent PNAS commentary, Daniel Shapiro and Kent Vrana of Pennsylvania State University, argue that “Celebrating R and D expenditures badly misses the point.” Instead of focusing on how much money is spent, the research enterprise should instead focus on its outcomes – its discoveries that advance knowledge and lead to improvements in health.
Of course, as we’ve noted before, measuring research impact is hard, and there is no gold standard. But for now, let’s take a look at one measure of productivity, namely the publication of highly-cited papers. Some in the research community suggest that a research paper citation is a nod to the impact and significance of the findings reported in that paper – in other words, more highly-cited papers are indicative of highly regarded and impactful research.
If considering highly-cited papers as a proxy for productivity, it’s not enough that we simply count citations, because publication and citation behaviors differ greatly among fields – some fields generate many more citations per paper. Furthermore, citation counts vary over time: it would not be fair to compare a paper published in 2013 with one published in 2008, as the latter paper has had 5 more years to generate citations.
One approach, to go beyond raw citation counts and control for factors such as citation behaviors and time, is to empirically bin papers according to topic, year of publication, and article type (e.g. research article, review article). Within each bin, we can assess the citation ranking of a paper: whether the paper is among the top 10% cited, the top 1% cited, or is among the hottest papers in its bin- the top 0.1% cited. To understand how NIH-supported papers fall into each of these three tiers, we used Thomson Reuters’ Web of Science to describe the citation impact of 1,578,607 papers published since 1985 that identify at least one NIH grant as a source of support. Looking across bins at the “top 10%” tier, NIH-supported papers were represented over twice as much in comparison to all papers (344,072 NIH-supported papers, or 22% of all NIH-supported papers, fell into this “Top 10%”-most-cited tier). Looking across bins at the “Top 1% cited” tier, there were three times as many NIH-supported papers in comparison to all papers in the database (44,404 papers, or 2.8% of all NIH-supported papers, made the “Top 1% cited” tier). And finally, NIH-supported papers were represented nearly three times as much among the “hottest” papers or the top 0.1% tier. (4,822 NIH-supported papers, which is 0.31% of all papers, were represented in the top 0.1% tier).
Table 1 shows the 25 most common journals where the papers appeared. The five most common were Journal of Biological Chemistry, PNAS, Biochemistry, Journal of Immunology, and PLOS ONE. Table 2 shows the 25 most common topics. The topics were defined by Thomson Reuters from among 226 possibilities; many papers could fit into multiple topics, so here we show the topics in which they had the best citation performance. The five most common topics were biochemistry and molecular biology, neurosciences, oncology, pharmacology and pharmacy, and immunology.
One might wonder about trends over time. In Figure 1, we show the number of published papers per year. There was a marked increase in NIH-supported publications around the year 2000, shortly after the doubling of the NIH budget began in 1999. In Figure 2, we show the proportion of papers that were among the top 10%, top 1%, and top 0.1% most cited. The bars show the actual proportions (in percent) while the dotted lines show the expected values. Though there has been a slight decline in the proportion of NIH-supported papers among the top 10%, NIH-funded papers are highly represented in these top tiers of papers, consistently performing well over time. This indicates NIH-supported research is highly regarded in the scientific community, based on the number of times that community acknowledge the research in their publications.
In future blogs, we will consider an alternate approach to measuring citation impact by field and timing of publication. One exciting method is the “Relative Citation Ratio,” which has recently been released by my colleagues in the NIH Office of Portfolio Analysis.
There are many other questions to consider, some of which we have addressed in prior publications focusing on specific institutes, like the NHLBI and the NIMH. How many highly cited papers are published in proportion to the funds awarded per year? What associations are there, if any, with grant mechanism, type of research, and PI background? Lots to think about!