Year in Reflection 2018


Suppose I offer you a bet based on a coin toss: you win $150 if the coin lands on heads, but you lose $100 if the coin lands on tails.  Will you take the bet?  Think for a second before reading on.

If you think this through objectively, of course you should take the bet.  There’s a 50% chance you will win $150 and a 50% chance you will lose $100 – the “expected value” of taking the bet is that you’ll be ahead by $25.  (Here’s the math: 0.50*150 + 0.50*(-100) = 75 – 50 = 25). 

Yet the reality is that most people (maybe you too?) will decline.  Most people consider the hurt of a $100 loss to be more severe than the pleasure of a $150 win – a phenomenon behavioral economists call “loss aversion.”  The pain from loss is greater than the pleasure from gain.  Many people wouldn’t accept the bet I just offered unless there is at least a $200 potential gain against the risk of a $100 loss. 

I just finished reading Annie Duke’s bestselling book “Thinking in Bets: Making Smart Decisions When You Don’t Have All the Facts.”  Ms. Duke, who left graduate school to become one of the world’s top poker players, argues that nearly all life’s decisions are bets.  The word “bet” goes way beyond card games or casinos; the Miriam-Webster dictionary defines “bet” as

  • “a choice made by thinking about what will probably happen”,
  • “to risk losing (something) when you try to do or achieve something”, and
  • “to make decisions that are based on the belief that something will happen or is true.”

Common to all these definitions is uncertainty – there are elements of chance affecting what follows any decision.  Ms. Duke’s point is that nearly all our decisions, no matter how carefully considered, occur against a backdrop of uncertainty.  But at the same time, all is not randomness or luck – over the long term, decision-making skill translates into higher probability of success.

Much of Ms. Duke’s book is devoted to distinguishing between the process of making choices or decisions and the realization of outcomes.  People unfortunately conflate decisions with outcomes – they think a decision is good (or brilliant or clever) if it’s followed by a good outcome.  Or they think a decision is poor (or stupid or worse) if it’s followed by a bad outcome.  Yet we know (or should know) better – someone can make a brilliant decision but fail due to sheer bad luck. 

For those of us involved in scientific research, it should be evident that the outcome of a decision – a decision to conduct a certain experiment or to enter a certain line of work or to take a job offer or to join forces with a particular collaborator – depends on both the skillfulness of the decision and elements of luck.  If this sounds like a stretch, consider the paper by Roberta Sinatra and colleagues, who demonstrate that “truly high-impact discoveries require a combination of high Q [scientific ability] and luck …”

As we look back at 2018 – or for that matter any year – it seems reasonable to ask questions like “Irrespective of the outcomes, did we make the right decisions?”  Given the choices we had, did we go about making our choices in a skillful way?  And we can dig deeper – did we take steps to minimize uncertainty? … to mitigate the impact of bad outcomes? … to seek a diversity of opinions? … to make our decisions as data-driven as possible? …

If we think about NIH research grant funding, we are all attempting to make choices by thinking about what will probably happen.  At NIH, we seek to develop and implement policies and processes that lead to improved rigor, stewardship, transparency, and integrity.  Applicants think about the probability of getting a favorable review, of getting funded, and if funded, of discovering important scientific findings. Peer reviewers leverage their expertise in advising NIH of their perspectives of what might happen should the proposals in front of them go forward.  (We recognize there is controversy over whether peer reviewers are successful in their forecasting and whether peer review is the optimal way to inform funding decisions – see here, here, here, and most recently here.)  NIH IC Directors and their staff, when incorporating peer review and strategic inputs into making decisions about which grants to fund and to what degree, think about the probability that a grant will be successful – at best leading to a great discovery, or if not that, leading to some worthwhile scientific findings.  We also think about the risk of failure – that a grant will yield zero publications or insights, or worse – findings of scientific misconduct or outright fraud. 

When we post our “Open Mike” blogs, we are, in our own way, making bets – we are making choices against a backdrop of uncertainty.  Our goal is to offer you material that is – we hope – informative, useful, practical, interesting, or maybe even a bit more – intriguing, even at times provocative.  It’s apt, I think, that the topic we all care about, biomedical science, is one in which uncertainty – what Stuart Firestein calls “ignorance” – is a core reality.  And biomedical science also requires scientists to learn how to think in a sophisticated manner and how to learn procedural skills.

We can try to assess our choices by seeing what you, our readers, find interesting.  Of course, we read your comments, which are often intriguing, informative, and insightful.  We also measure total page views throughout the calendar year, a metric we gather from Google Analytics.  In 2018 we posted 36 Open-Mike blogs.  The Table shows the top 10 in terms of page views. 

Blog Title Total Pageviews
FY 2017 By the Numbers  17,327
Wait…It’s Not MY Grant? 17,034
How Many Researchers, Revisited: A Look at Cumulative Investigator Funding Rates 16,657
The Issue that Keeps Us Awake at Night 16,148
Funding Longevity by Gender Among NIH-supported Investigators 15,599
Impact of Teams Receiving NIH Funding 12,928
Assuring the Integrity of Peer Review (2017) 12,635
Changes to the R15 Academic Research Enhancement Award  (AREA), and Introducing the R15 Research Enhancement Award Program (REAP) 12,256
Do Reviewers Read References? And if so Does It Impact Their Scores? 12,129
NIH Policies to Address Sexual and Gender Harassment in NIH-supported Extramural Research 9,191

These top ten postings encapsulate, I think, themes of decision-making and probabilities, which together combine to yield outcomes:

  • Policies and procedures that underlie decision-making on grant funding, management, and integrity: In future blogs this year, we will revisit policies and procedures by which we try to assure fair and optimal decision making and management, by which we do our part to assure the integrity of the enterprise.
  • Probabilities of career and funding success: In future blogs this year, we will re-examine how many researchers we are supporting and their funding rates, something especially interesting given several straight years of NIH budget increases.
  • The outcomes of NIH-funded research: In future blogs this year, we will dig a bit deeper into the associations between team science and NIH-funded scientific discovery.

As we enter 2019, I want to thank you for “taking a bet” that reading “Open Mike” every so often is worthwhile.  We thank you for your interest and comments and we wish all of you only the best for a happy New Year. We hope our bets have paid off, but to increase our odds for next year, please let us know about which topics you wish to hear more.

Before submitting your comment, please review our blog comment policies.

Leave a Reply

Your email address will not be published.