Open Mike

Helping connect you with the NIH perspective, and helping connect us with yours

Should We Keep Meeting This Way?

Guest post by Dr. Bruce Reed, Deputy Director of the NIH Center for Scientific Review, originally released on the Review Matters blog

photo of Bruce Reed

Dr. Bruce Reed, Deputy Director, CSR

How will study sections meet in the future? NIH peer review depends on robust meetings where groups of scientists, through vigorous discussion, identify the applications of highest merit. For the last 75 years, until last March, nearly all chartered review committee meetings were held in-person. Today, in response to the pandemic, 90% of all CSR review meetings are run as video (“Zoom”) meetings. CSR is taking steps now so that when all options are back on the table, we can make informed choices about how best to convene review meetings.

Last round we obtained survey responses from 3,000 NIH reviewers, ratings by scientific review officers (SRO) of 230 review meetings, compiled quantitative data comparing in-person versus Zoom instances of over 275 meetings, analyzed rosters from those meetings, and also surveyed our support staff.

The data give no indication that the forced switch to Zoom has introduced major problems. Quality of review is our number one priority; 60% of reviewers and 85% of SROs say that overall quality has remained the same. By a large margin, reviewers report feeling equally able to speak and be heard in Zoom meetings compared to in-person meetings (see Figure). Reviewers, SROs, and support staff generally report the platform is easy to use, meetings are easy to manage, and technical problems are no more common or difficult to resolve than for in-person meetings.

chart displaying participation in zoom meetings compared to in person on factors such as contributed to discussion, comfort voting outside range, and attention span lasted

Participation in Zoom Compared to In-Person Meetings

 

However, some of the survey results give us pause. Despite favorable ratings of Zoom meetings, reviewers and SROs prefer in-person meetings by very similar margins (43% to 31% for reviewers; 44% to 36% for SROs). Zoom meetings tend to run longer and the increased duration may explain some of that preference. Comparing Zoom to in-person, almost half of reviewers reported diminished attention, 51% perceived lower engagement, 30% contributed less, and 36% rated discussions as worse. In addition, many reviewers commented that they missed the social aspects of in-person meetings, the chances to network, to build collaborations, and to enjoy the comradery that in-person meetings foster.

The highly uneven impacts of COVID on scientists’ daily lives make it difficult to discern how meeting format affects their ability/willingness to participate as reviewers. For many, family care responsibilities have increased during the pandemic and women are bearing a disproportionate load. We examined roster composition carefully to see if reviewer demographics have changed during the pandemic. So far, we see little change. Interpretation is complicated by the fact that reviewers are dealing with two large changes at once – video format, which may make it easier to participate, and COVID, which likely makes it harder.

We are all still learning. “How to run better Zoom meetings” is a frequent topic of workshops, meetings, and staff conversations. Reviewers have reached out, for example, reminding us of the challenging home/work realities they are contending with. We understand that many reviewers must work from home and cannot completely absent themselves from family and household responsibilities. SROs generally are implementing more frequent breaks to accommodate this new reality. Even so, cameos by pets and children are not “unprofessional”, they’re OK, perhaps even fun. We understand you might want to eat your breakfast with the video off. However, please come back, and mostly stay on screen because video seems to improve engagement. Hopefully, more breaks will help with Zoom fatigue, but one size does not fit all, and good communication between SROs and reviewers is vital.

Going forward CSR and OER plan to gather additional data from the February and March 2021 review meetings. We will consider those data, your comments, and input from our NIH colleagues when setting post-pandemic practices for study section meetings. There are many considerations: Travel is time-consuming, has environmental impacts, and is expensive; review should be an engaging and rewarding experience; meeting format may systematically make it easier, or harder for some scientists to participate; meetings vary considerably in size and so one solution may not fit all. And, hybrid solutions are on the table —for example, with some reviewers in the room and others joining by Zoom, or holding two meetings per year entirely by Zoom and one in person. The paramount consideration is quality of review. CSR is committed to maintaining the quality of review and improving it when possible.

We invite you to look at the details of our analyses thus far and tell us your ideas for how to improve Zoom review meetings. Please send your ideas to the Director of CSR’s Office of Communications and Outreach, Kristin Kramer, or submit a comment.

Acknowledgments: Dr. Hope Cummings for survey work; Lia Fleming and Aditi Jain for quantitative analyses.

Share
  •  
  •  
  •  
  •   
  •  

5 thoughts on “Should We Keep Meeting This Way?

  1. Aa a long term reviewer dating back to ~ 1975, I have to confess that the meetings in person are the best overalll when viewed from the standpoints of beneficial outcomes (with colleagues and SROs). However, the cost of these benefits are the needs for travel and lodging, and these needs pose significant hardships. Having just participated in a Zoom review, I was impressed at how effective the Zoom interface and interactions actually were. So, after summing up the pluses and minuses of the old meetings and the new Zoom format, I am favoring the new format (consider the benefits of not having to travel the length and breadth of the country). But this its clearly a work in progress and the outcome needs to be continuously critiqued.

  2. No, please no. It’s really hard to have good discussion via Zoom. On the one hand, it takes longer on Zoom. On the other, less discussion occurs. This is a bad plan, and the quality of the reviews is just not as good.

  3. I will also note that it is VERY difficult to be staring at a Zoom screen all day and interacting via Zoom. The evidence on this is overwhelming. Human beings are meant to interact IN PERSON. We did not evolve for Zoom.

  4. Oh god please. It’s very difficult to have a good zoom talk. It takes more time to zoom in, mostly on one hand. There is less debate, on the other hand. The quality of the reviews is not as good as a bad plan.

  5. If it were purely about “can I avoid disruption to my home and family”… Zoom for sure. But a critical value of study section in-person, for me as a young investigator, has been the chance to be in the room, have lunch with, later on have a beverage with, and generally network with giants of my field. These discussions have led to speaking invites, collaborations, and just generally being seen as a human being in a way that only happens when you have an actual in-person meal with someone. That alone creates value with in-person meetings, ESPECIALLY for anyone in the ECR program.

Leave a Reply

Your email address will not be published. Required fields are marked *