Should We Keep Meeting This Way?

Posted

Guest post by Dr. Bruce Reed, Deputy Director of the NIH Center for Scientific Review, originally released on the Review Matters blog

photo of Bruce Reed
Dr. Bruce Reed, Deputy Director, CSR

How will study sections meet in the future? NIH peer review depends on robust meetings where groups of scientists, through vigorous discussion, identify the applications of highest merit. For the last 75 years, until last March, nearly all chartered review committee meetings were held in-person. Today, in response to the pandemic, 90% of all CSR review meetings are run as video (“Zoom”) meetings. CSR is taking steps now so that when all options are back on the table, we can make informed choices about how best to convene review meetings.

Last round we obtained survey responses from 3,000 NIH reviewers, ratings by scientific review officers (SRO) of 230 review meetings, compiled quantitative data comparing in-person versus Zoom instances of over 275 meetings, analyzed rosters from those meetings, and also surveyed our support staff.

The data give no indication that the forced switch to Zoom has introduced major problems. Quality of review is our number one priority; 60% of reviewers and 85% of SROs say that overall quality has remained the same. By a large margin, reviewers report feeling equally able to speak and be heard in Zoom meetings compared to in-person meetings (see Figure). Reviewers, SROs, and support staff generally report the platform is easy to use, meetings are easy to manage, and technical problems are no more common or difficult to resolve than for in-person meetings.

chart displaying participation in zoom meetings compared to in person on factors such as contributed to discussion, comfort voting outside range, and attention span lasted
Participation in Zoom Compared to In-Person Meetings

 

However, some of the survey results give us pause. Despite favorable ratings of Zoom meetings, reviewers and SROs prefer in-person meetings by very similar margins (43% to 31% for reviewers; 44% to 36% for SROs). Zoom meetings tend to run longer and the increased duration may explain some of that preference. Comparing Zoom to in-person, almost half of reviewers reported diminished attention, 51% perceived lower engagement, 30% contributed less, and 36% rated discussions as worse. In addition, many reviewers commented that they missed the social aspects of in-person meetings, the chances to network, to build collaborations, and to enjoy the comradery that in-person meetings foster.

The highly uneven impacts of COVID on scientists’ daily lives make it difficult to discern how meeting format affects their ability/willingness to participate as reviewers. For many, family care responsibilities have increased during the pandemic and women are bearing a disproportionate load. We examined roster composition carefully to see if reviewer demographics have changed during the pandemic. So far, we see little change. Interpretation is complicated by the fact that reviewers are dealing with two large changes at once – video format, which may make it easier to participate, and COVID, which likely makes it harder.

We are all still learning. “How to run better Zoom meetings” is a frequent topic of workshops, meetings, and staff conversations. Reviewers have reached out, for example, reminding us of the challenging home/work realities they are contending with. We understand that many reviewers must work from home and cannot completely absent themselves from family and household responsibilities. SROs generally are implementing more frequent breaks to accommodate this new reality. Even so, cameos by pets and children are not “unprofessional”, they’re OK, perhaps even fun. We understand you might want to eat your breakfast with the video off. However, please come back, and mostly stay on screen because video seems to improve engagement. Hopefully, more breaks will help with Zoom fatigue, but one size does not fit all, and good communication between SROs and reviewers is vital.

Going forward CSR and OER plan to gather additional data from the February and March 2021 review meetings. We will consider those data, your comments, and input from our NIH colleagues when setting post-pandemic practices for study section meetings. There are many considerations: Travel is time-consuming, has environmental impacts, and is expensive; review should be an engaging and rewarding experience; meeting format may systematically make it easier, or harder for some scientists to participate; meetings vary considerably in size and so one solution may not fit all. And, hybrid solutions are on the table —for example, with some reviewers in the room and others joining by Zoom, or holding two meetings per year entirely by Zoom and one in person. The paramount consideration is quality of review. CSR is committed to maintaining the quality of review and improving it when possible.

We invite you to look at the details of our analyses thus far and tell us your ideas for how to improve Zoom review meetings. Please send your ideas to the Director of CSR’s Office of Communications and Outreach, Kristin Kramer, or submit a comment.

Acknowledgments: Dr. Hope Cummings for survey work; Lia Fleming and Aditi Jain for quantitative analyses.

63 Comments

  1. Aa a long term reviewer dating back to ~ 1975, I have to confess that the meetings in person are the best overalll when viewed from the standpoints of beneficial outcomes (with colleagues and SROs). However, the cost of these benefits are the needs for travel and lodging, and these needs pose significant hardships. Having just participated in a Zoom review, I was impressed at how effective the Zoom interface and interactions actually were. So, after summing up the pluses and minuses of the old meetings and the new Zoom format, I am favoring the new format (consider the benefits of not having to travel the length and breadth of the country). But this its clearly a work in progress and the outcome needs to be continuously critiqued.

    1. The CSR has all the data needed to determine if in-person meetings are helpful. Reviewers provide a score before the meeting and the whole group provides a score at the meeting. There are many ways to measure if the meeting score predicts investigator success better than the pre-meeting score. If it does not, there is no justification for the expense and CO2 generated by travel to grant review committee meetings. Will CSR release this data for outside review?

  2. No, please no. It’s really hard to have good discussion via Zoom. On the one hand, it takes longer on Zoom. On the other, less discussion occurs. This is a bad plan, and the quality of the reviews is just not as good.

    1. I agree – there is nothing that replaces in person meetings. Also part of the reason for doing this is the meeting with other scientists many of whom become colleagues and friends. And there are so many nuances that are present face to face that do not translate well to Zoom meetings. I do think that if budgets are a concern, 1 meeting a year could be by Zoom but the other 2 should be in person. The 1 meeting a year in Zoom will be considerably helped by the reviewers having met in person the other two-thirds of the time.

      My vote is at least two-thirds of the meetings be in person once we are able to meet face to face — so until we meet again! Martha

      1. “…..part of the reason for doing this is the meeting with other scientists many of whom become colleagues and friends”

        Well, not really. Could you point out where in NIH websites it says that study section meetings are to make new friends?
        Study sections are supposed to be confidential and as secretive as possible, much like a jury. Imagine journal paper reviews being conducted in person!
        As it is the NIH review system has many potential loopholes for conflict of interest. For example, every one in the room can know who else reviewed a particular grant and have access to confidential opinions of other reviewers about a PI/lab/research area, which will propagate who knows how, leading to people doing favors, being vindictive etc.,…
        Reviews should not only be remote, but also anonymized if possible, to reduce chances for powerplay.

  3. I will also note that it is VERY difficult to be staring at a Zoom screen all day and interacting via Zoom. The evidence on this is overwhelming. Human beings are meant to interact IN PERSON. We did not evolve for Zoom.

      1. Doesn’t mean kids over ZOOM are performing the best. Like in-person NIH review sessions, personalized teaching is clearly superior. Please know it is an honest opinion and not for the sake of arguing.

    1. You say human beings are supposed to interact in person. I am not sure how this applies to everything. Would you say reviewers of a journal paper should interact in person and go out for drinks and become friends? Just think about how absurd that sounds. Study sections reviews have sort of become pseudo conferences. That is not their purpose. You already do so many professional things that are NOT in person. I assure you that you are already evolved enough for doing reviews on a zoom meeting. Or maybe you just prefer to get out of town for a break from family on NIH funds.

  4. Oh god please. It’s very difficult to have a good zoom talk. It takes more time to zoom in, mostly on one hand. There is less debate, on the other hand. The quality of the reviews is not as good as a bad plan.

  5. If it were purely about “can I avoid disruption to my home and family”… Zoom for sure. But a critical value of study section in-person, for me as a young investigator, has been the chance to be in the room, have lunch with, later on have a beverage with, and generally network with giants of my field. These discussions have led to speaking invites, collaborations, and just generally being seen as a human being in a way that only happens when you have an actual in-person meal with someone. That alone creates value with in-person meetings, ESPECIALLY for anyone in the ECR program.

    1. I personally favor the remote format. I have travelled for multiple review meetings. Going through heaps of security, cab rides and hotel check ins, all for just a few minutes of discussing a handful of the grants that I reviewed, if at all they come in to discussion. As such I felt that this is a ridiculous waste of NIH money, besides increasing carbon footprint in all possible ways.
      I would also like to remark in particular about some of the objections to the online format on the basis that this reduces opportunities for networking. NIH review meetings are not conferences where you meet your peers, go out for dinner and drinks, network and exchange ideas. IN fact it is supposed to be quite the opposite. Reviews are supposed to be highly confidential. Giving reviewers the opportunity to interact in person in a casual setting during a study section meeting increases the chances that applications are discussed informally during conversations, and biased opinions about individual PIs or topics migrate or spread amongst the community. It also gives an unfair advantage to reviewers of a study section , since it allows buddying up with others who might review their grant in future and might feel obliged to score them favorably. I have noticed these issues being a reviewer myself for several years.
      So if you are a ECR, look to visiting more conferences for networking, not NIH study sections. It is not the goal of these meetings to give you speaking opportunities.

  6. While I do understand that it is more cost effective to do meetings on Zoom, I have to admit that after giving it the college try (3 this year), I really disliked he experience. The conversations and all the measures they ask in the study (figure above) I would have answered the same (mostly). Although I had to keep my video off because it was just too hard (one of my meetings was 8+ hours). And you also don’t get to meet similar researchers, and the networking, and just the collegiality is something that you can’t put a price upon.

  7. Having attended my first in-person review meeting earlier this year as an Early Career Reviewer, I would strongly prefer in-person meetings. The opportunity for networking and learning from others was invaluable.

  8. Please, no. While the Zoom review sessions are better than nothing, they are considerably more exhausting and the level of discourse is quite reduced compared to the in-person meetings. The quality of the attention and the discussion absolutely suffers. The online platform is just not suited to the kind of work done at study section.

  9. Has there been an assessment of the cost savings from Zoom vs. in-person meetings? Can those dollars fund additional applications?

    If not, I would prefer a return to the in-person format, at least for regular study section meetings.

      1. I am sure NIH spends about 100 million $/year on in person SRG meetings. That can surely fund a bunch of grants!

        Do you really think flights+hotel+travel allowance > zoom meeting???

    1. I am sure NIH spends about 100 million $/year on in person SRG meetings. That can surely fund a bunch of grants!

      Do you really think flights+hotel+travel allowance > zoom meeting???

  10. Hybrid meetings would be best. Reviewers should be free to decide if they want to participate in person or remotely.

  11. I participated in a recent NIDA Zoom review and was very impressed. I thought the review was just as effective as the 30+ in-person reviews I have participated in over the years. I think that maybe the comfort and organization of the SRO was a real factor. I will miss the collegiality of the in-person events, but I wonder whether NIH should be paying for that. I think that I can suffice with collegiality at conferences if it means that NIH can support more funding going towards the excellent and deserving research proposals they receive.

  12. I think hybrid meetings would be best, especially for women and other caregivers for whom traveling to a review meeting can be difficult. I agree that reviewers should have the option to decide to participate in person or via zoom.

  13. I really feel that reviewers are more distracted and less engaged during Zoom meetings. I have found that discussions that in the past might have been prolonged and even contentious are often very brief. The lack of social interaction is also an issue as to me that has always been one of the “perks” of study section service. Obviously, travel is a concern, but if we get back to a more normal situation I would definitely much prefer in-person meetings.

  14. Although there are certainly issues with the in-person reviews (travel & costs, etc), they are far superior to zoom in terms of the discussion and the ability to read other non-verbal cues that are really essential to evaluating the proposals and the discussion overall. I fear that a lot is being lost with the complete Zoom review meetings.

  15. I worry tremendously about the climate crisis and I am hopeful that the disruption from the pandemic forcing all of us to explore virtual options might give us sustainable solutions to this much bigger problem. I completely agree with all of the benefits of in person meeting – I love the traveling and the networking, but the simple reality is that our planet cannot take much more of it. Given that existential crisis, might I suggest an alternative halfway point between in-person and zoom (if people feel the in-person is necessary). One option could be mixed in-person and zoom given preferences. Maybe people could travel in person to regional areas (i.e. West, Central, South, East/New England) and have virtual linkages between the hubs. Or there could be mixed in-person and virtual for those who prefer each. For both these options, each person could also have their computers/cameras on so that those who are zooming don’t feel like they’re missing out, or are awkwardly projected on a large screen in front of the whole room, and everyone can see everyone’s faces regardless of whether they’re in person or by zoom.

    1. I sat on Study Sections for many years and there was a cadre of reviewers whom I got to know well and who became not only colleagues but collaborators and friends. There is a rhythm to the give and take of the interchanges in person that cannot be duplicated in Zoom meetings. To have Zoom and in person would be a bit disjointed. Perhaps another possibility would be to have two-thirds of the meetings Zoom and one third in person. This I think would be optimal and would be something to also look forward to. I find Zoom meetings draining over time, more so than in person meetings as there is just something missing in the Zoom meetings that can only be captured in person.

  16. One thing I would say is, if we go back to in-person meetings that is fine. However please make the “phone reviewers” call in by zoom. Every single time I have been a phone-in reviewer, it has been a disaster from my perspective. I have trouble being heard, I can’t see who is speaking. I think the grants I review are poorly served by phone reviews.

  17. I’m a big fan of hybrid as many have noted that some people can not afford time and money for travel. Study section is hard in any format but when we are all not in lockdown I wonder if some in person and some online might work.

  18. Ignoring the carbon footprint difference and having more nights at home, if savings associated with virtual meetings would end up as each panel funding one more grant each cycle, that alone would tip the balance. The danger is that virtual meetings might turn into useless exercises where the final scores are already determined by the three preliminary scores. The answer is not longer meetings, obviously. Attention spans are short. Ensuring substantive discussion is up to the chair, more than the panel members or SRO. When chairs allow assigned reviewers to drone on, this is toxic in person but deadly for virtual. If the chair can steer reviewers to avoid description and limit their critiques to score driving strengths and weaknesses, then encourage a healthy discussion for each proposal in the competitive range, virtual meetings should come up with very similar scoring to in person. Getting this part right is the only important part. Whether we enjoy it or not is not.

  19. With Zoom fatigue, applications discussed early on seem to get the most attention. I worry that we are not doing service to those later in the day and even on the 2nd day of study section meetings. The longer the day goes on the more easily I am distracted on zoom and that does not occur with in person reviews.

    1. Yes — Zoom fatigue may compromise the reviews. And Zoom fatigue is substantially more than in person meetings. But if some of the meetings are in person and people reviewing get to know each other — I think that the fatigue is far less and the reviews more thoughtful.

  20. For scientists with small children, who are the primary caregivers. Mostly women sadly. Zoom meetings could be way how to participate. I usually avoid to serve on study section because it is very costly for me to pay baby sitting for days and sometimes it just impossible to find extensive childcare. Ofcourse the cost of this is from my own pocket.

  21. In person meetings give the human touch. It is like watching a football game in the stadium with others VS on your TV. Same game and probably you can follow technical aspects better on TV. But humans need interaction with other humans than just with the computer screens.

  22. I agree that the in person meetings are of much higher quality. However, do they really change the outcome?
    My main request is that the per deim increase. $200 is embarrassingly low for the ask. Reviewers spend many days reviewing grants and writing thoughtful critiques without pay. Then we are asked to attend a meeting at a reimbursement rate that is far below the current market rate for highly qualified reviewers. (More like $1000 per day). If the CSR is saving huge amounts of money by not flying us to DC, they could at least reimburse us at a fair rate.

  23. We should see a break down on the preference for zoom vs in person based on age and gender. As a woman with children (on the West Coast) I much prefer zoom to flying across country. I am still at early stage in my career and I felt much more comfortable jumping in to debate on zoom than in person.

  24. Reviewers should have the option of participating in person or remotely. Since some reviewers cannot travel (family responsibilities, etc.), the participation of reviewers may increase.

    1. I think if meetings are Zoom AND in person that individual exceptions can be made as for example those with young children at home, but to make it optional I think would not encourage the in person meetings and would in some respects defeat the purpose of the face to face meeting with others. If the schedule was two-thirds Zoom and one third in person — those finding it too difficult to come in person in the one third of meetings in person could zoom in.

  25. Could this data be analyzes by gender of responders? I hypothesize that given family responsibilities (either COVID or NOT) women may prefer ZOOM to in-person to reduce time being absent from kids/family. Very curious to see such data – thanks!

  26. I spent 13 years as a permanent IRG member, plus many additional ad hoc reviews after I was terminated by CSR (I would have kept reviewing as a permanent member indefinitely). My main motivation from a personal benefit perspective was the in-person interaction with my esteemed and highly talented colleagues. The actual reviewing and participating in the meetings was WORK, for which I was very poorly compensated. The citizenship of doing peer review was a consideration, but not really a positive motivator. Free literature reviews by the applicants was nice, but would not have kept me doing the WORK. Don’t default to zoom – you want the best and most skilled people doing peer review if you want the best science to be funded. The dollar investment is well worth it.

  27. I agree with many of the above comments. Overall, I think the hybrid model may be the best moving forward. One big advantage of the hybrid model may be the ability to bring in additional reviewers for a smaller subset of grant applications in a specialized area where they are experts but would otherwise not have the time to come for a 2 day in-person meeting.

  28. My background- Have been a member of 6 CSR/NIAID study sections since 1977, plus many assorted ad hos meetings. I have been to 4 Zoom reviews since they started this spring. I really prefer the in person format. Almost all the value I got from participating was the interaction and establishment of personal relationships among other reviewers that lead to both collaborations and help. I found it much easier to loose concentration during the review of grants assigned to other reviewers.
    Even though travel is burdensome, I found the experience in face-to-face reviews much better.

  29. Adding a day of travel on top of two full days of in-person meetings in my experience is more grueling than staring at a computer monitor all day – which in fact we have evolved towards if you think about it. I did not experience any decrease in discussions which were quite lively and in more than a few instances needed 20 minute “warnings”. Participants still have opinions and come to discuss the merits and shortcomings – its hard to take that away from scientists. I found myself able to take more mini breaks or just stand up and stretch by turning off my video – less disruptive than in person meeting. Seeing colleagues broadcasting from their home with a few cats and dogs and kids scampering about perhaps makes it more personal? My ideal scenario would be to alternate between zoom and in-person. I agree that there really still needs to be personal contact but not necessarily at every meeting for me; particularly if you attend regularly. I was also impressed with the reliability of the technology and professional technical support. There were no glitches to speak of for a committee of more than 45 attendees by zoom from 9-6 on Day 1, and 9-5 on day 2. That’s better than the in-person glitches of when a few people call in at live meetings. Still an n=1 but if it goes this smoothly in the future you have my support for considering the continuation of a Zoom format. I did not anticipate such a reaction going into it .

  30. My experience has been that the interactions over zoom were actually quite reasonable and allowed good discussions. I did appreciate not having to travel and be away even more from child care responsibilities that I share with my significant other. Not to mention the savings in costs and reductions in carbon footprint that we can no longer ignore. Maybe one meeting in person to network and the other two run remotely.

  31. I have had mixed results as a reviewer, I was on a large center grant via zoom that was both synchronous and asynchronous and it was a disaster, quality of reviews were not good at all. A more recent zoom meeting with more seasoned reviewers that followed a typical format was a bit better. As others have said, overall reviews are stinger with in person dialogue and getting away from in person meetings all together will diminish the quality of and spirit of peer review.

  32. I would suggest technology is evolving very fast, and we should support the technology that saves a huge amount of money and the planet from more CO2 emission.

    I strongly support Zoom meetings.

  33. I feel that reviewers are more distracted and less engaged during Zoom meetings. I fear that a lot is being lost with the complete Zoom review meetings. My main request is that the per diem increase. $200 is embarrassingly low for the ask. We spend many days reviewing grants and writing thoughtful critiques without pay. Then we are asked to attend a meeting at a reimbursement rate that is far below the current market rate for highly qualified reviewers. (More like $900 per day). If the CSR is saving vast amounts of money by not flying us to the meeting city, they could reimburse us at a fair rate.

  34. Certainly! Hybrid scientific meetings amalgamating Zoom-Virtual and in-person sessions sound more crisp, enlightening, productive and strategic for productive!

  35. The primary goal of study section is to review the science and then debate the merits and flaws, not socialize. I do recognize that for many people this is an added ‘extra’ to the experience. If I can complete my service commitment without having to take time off to travel (although I enjoy Bethesda and DC for sure) then that seems like a win simply based on my and everyone else’s tax dollars spent, time, and the environment.

    And well moderated Zoom meetings encourage discussion from more panel members than the few outspoken people at in person meetings.

  36. I am a standing member of a study section and have participated in the last few rounds of virtual reviews. I am personally much more distracted and find the long Zoom meetings draining. I also find that proposals reviewed later in the meeting seem to receive even more of truncated, less lively review than they did in person. I do understand, and have, childcare issues, but believe those are offset by networking opportunities, particularly for junior women and/or underrepresented minority faculty who often have fewer opportunities to be a part of influential scientific networks. Networking matters for identifying letter writers for promotion (for junior members of the committee) and if we’re being honest, future grant proposals.

  37. Totally agree with Ronald Seifer comments. On top on almost no incentive to be a zoom reviewer, the quality of the discussions wane in 8+hs of meetings. However, I recognize that it will be a good idea to incorporate specialized reviewers via zoom for a low number of applications

  38. The cost of in-person meetings (monetary and pollution-wise) is less and less justified as we move forward. Web meetings (zoom and others) are constantly improving, we are getting used to them, we should keep improving them and move to a more reasonable way of using the federal money allocated to research. Do we prefer meetings in-person, or do we prefer to get that money back in the budget to fund additional meritorious research projects?
    It has been argued that people are more distracted; perhaps we have to schedule meetings that are not 8 hour long, perhaps we have to start thinking on organizing the meetings for example in 4-hour blocks instead.
    The distraction argument — we are all adults, we ask children to pay attention to their lessons, to do their homework, and we are not able to maintain our own attention?
    Perhaps, we just need to learn new courtesy norms (it will take time); for example, if you are attending a grant review panel on-line, have an autoreply in your email saying “I will be on study section from X to Y …” and then turn off your email notifications…etc. This is the minimum courtesy that as scientists we should be affording each other while reviewing projects that another scientist has dedicated months to prepare and submit.
    Finally, in-person meetings do not eliminate distractions, they just make less evident who is not paying attention because we are in a room full of people and we are not staring at each other like in web settings.
    In summary, change always takes us out of our comfort zones, it always brings resistance, needs probably an adjustment of meeting configuration (duration etc.), but is difficult to keep justifying the expense and logistics of traveling for in-person reviews with the currently available technology.

  39. Currently chair of a standing study section. Study sections work best when reviewers get to know each other, both personally and professionally. This lowers the barriers for making substantive comments and adding to discussions. It allows reviewers to anticipate the kinds of objections, concerns, or positive comments that might be raised. Things like lunches, dinners, breaks in the hotel lobbies, etc. all contribute to an atmosphere of camaraderie and cohesion. Meeting in person also ensures commitment to the cause, and allows the study section to develop a rhythm of how applications are reviewed and scored. Doing everything by Zoom risks these positives and invites the dismantling of the many things which make good standing sections run well. Please no!

  40. For West Coast people, consider starting Zoom meetings later. I would vote for 11am EST (=8am PT), but 10am EST (=7am PT) may be a reasonable compromise. 9am ET (=6am PT) is too early for many, especially those with family responsibilities that tend to be concentrated in the morning.
    Also think about ways to shorten the meetings to alleviate Zoom fatigue. It should be possible to concentrate the discussions. Conferences have been doing this – meeting on Zoom for 3 hours per day instead of 8 hours in-person – and it seems to be working.

  41. I strongly support study section meetings via Zoom. We cannot continue with practices that are destroying the planet and compromising the health and well-being of millions of people. The quality of peer review is dependent upon the enrollment of scientifically competent, motivated, and rigorous reviewers and SROs. The peer review process should not be approached as a “networking/lobbying” opportunity to promote individual and/or group agendas. In fact, such practices should be strongly discouraged within study sections to preserve scientific neutrality and rigor. The peer review process needs substantial revision and improvements. Zoom meetings are a small step in the right direction.

  42. Though the digital environment meetings are here to stay and are for sure a parto of the future of all and every activity, we must not forget humans are social beings and we are meant to interact. As much as I appreciate the efforts to keep people connected, an effort should be made to ensure a safe personal interaction in the near future too.

Leave a Reply to Mark Huising Cancel reply

Your email address will not be published. Required fields are marked *