SciCheck Digest
People online are touting the results of a Cochrane review to incorrectly claim that it shows masks “don’t work” against the coronavirus. But the primary conclusion of the review is that it’s uncertain from randomized controlled trials whether mask interventions in the community help slow the spread of respiratory illnesses.
Full Story
Three years into the COVID-19 pandemic, few topics have become as polarizing as masks. Some people claim masks are a panacea; others say masks are worthless or worse. The evidence, however, is more complicated and nuanced — and points somewhere in between, experts told us.
Reigniting the debate — and sparking misinformation about masks from both sides — is a recently released update from Cochrane, a highly-respected British nonprofit that specializes in systematic reviews of health care interventions.
The Jan. 30 review found that based on existing randomized controlled trials — which tested the effectiveness of interventions encouraging people to wear masks, rather than testing the effectiveness of masks themselves — wearing masks in the community “probably makes little or no difference” to the number of people with influenza or COVID-19-like illnesses.
“The pooled results of RCTs did not show a clear reduction in respiratory viral infection with the use of medical/surgical masks,” the review reads.
The authors, however, also emphasized the “uncertainty about the effects of face masks.” And only two trials in the review assessed the effectiveness of a mask intervention for COVID-19.
“The high risk of bias in the trials, variation in outcome measurement, and relatively low adherence with the interventions during the studies hampers drawing firm conclusions,” the authors wrote. “The low to moderate certainty of evidence means our confidence in the effect estimate is limited, and that the true effect may be different from the observed estimate of the effect.”
In other words, there isn’t good evidence from randomized controlled trials that encouraging mask use in the community prevents the spread of respiratory diseases, but the issue also hasn’t been studied very well. So the real answer is unknown.
Despite the limitations, many people misinterpreted the review to be saying that masks “don’t work.”
“12 RESEARCH STUDIES PROVE MASKS DIDN’T WORK,” reads an Instagram post about the Cochrane review from the Liberty Counsel, a Christian religious liberty organization.
“Scientific review confirms doubters’ stance on masks and COVID-19,” declared a popular Instagram post from Fox News.
The lead author of the Cochrane review, Tom Jefferson, seemed to endorse this interpretation when he said in an interview, later quoted by conservative columnist Bret Stephens in a widely viewed opinion piece for the New York Times, “There is just no evidence that they” — referring to masks — “make any difference.”
But experts — and the Cochrane Library — say this is an inaccurate representation of what the review found.
“Many commentators have claimed that a recently-updated Cochrane Review shows that ‘masks don’t work’, which is an inaccurate and misleading interpretation,” Dr. Karla Soares-Weiser, the editor-in-chief of the Cochrane Library, said in a March 10 statement.
“It would be accurate to say that the review examined whether interventions to promote mask wearing help to slow the spread of respiratory viruses, and that the results were inconclusive,” she continued. “Given the limitations in the primary evidence, the review is not able to address the question of whether mask-wearing itself reduces people’s risk of contracting or spreading respiratory viruses.”
Soares-Weiser went on to note some limitations mentioned in the abstract of the review, including the issue of whether people in the trials actually wore masks. She said the group would be working with the authors to reword the “Plain Language Summary,” which she said “was open to misinterpretation.”
In an interview with the New York Times, Soares-Weiser was also critical of Jefferson’s comments, which she said were not accurate.
Jefferson, who is a senior associate tutor of continuing education at Oxford University, has been the lead author of the Cochrane review on physical interventions to reduce the spread of respiratory viruses since its inception in 2006.
He has endorsed several unorthodox views about COVID-19 and some of his writing has been republished by the Brownstone Institute, a group that has described itself as the “spiritual child” of the widely criticized Great Barrington Declaration. In the latest update to the Cochrane review, under a section in which authors disclosed potential conflicts of interest, he reported “declaring an opinion on the topic of the review in articles for popular media.”
Cochrane Review
One reason the Cochrane review has garnered so much attention is because Cochrane has a reputation for excellence.
The group doesn’t do original research, but performs what are called systematic reviews, which summarize the collective literature on a particular question in a careful and predetermined way that minimizes bias. This prevents someone from cherry-picking studies that could sway the findings, for example.
The results from different studies are often then combined statistically in what is called a meta-analysis, which uses a weighted average to summarize the effectiveness of a given intervention.
Cochrane is particularly known for its robust and transparent methods and is therefore often considered to be the gold standard for such reviews.
In this case, the review focused not just on mask interventions, but on physical interventions more broadly, including hand hygiene programs, for the prevention of respiratory illnesses, primarily influenza.
Similar to a 2020 update on this topic, but in contrast to earlier editions, the systematic review did not consider observational studies about masks, and instead was limited to randomized controlled trials and cluster randomized controlled trials.
Randomized controlled trials are considered one of the best kinds of evidence, since they randomly assign people to an intervention or control group, or in the case of a cluster design, randomly assign groups of people to different interventions. This allows for a fairer comparison of what the intervention actually does, although every study has some limitations.
The 2023 update added 11 new randomized controlled trials, for a total of 78 trials. But only about a dozen of these evaluated the effect of a mask intervention, such as providing people with a mask and encouraging them to wear it, compared with no such intervention, on the number of respiratory illnesses. And only two were conducted during the pandemic for the coronavirus. Five other trials, none of which were done for the coronavirus, compared N95 or other similar respirators with surgical masks, mostly in health care workers.
Despite some claims to the contrary, including a now-deleted Twitter thread that incorrectly alleged the Cochrane authors made errors in their meta-analysis, multiple experts told us the review did not have any notable flaws.
“It is a well done review. It’s been done tightly by the best standards by competent people,” Julii Brainard, a senior research associate at Norwich Medical School at the University of East Anglia in the U.K., told us. “It may seem very critical of primary evidence, but Cochrane reviews are always very critical.”
“The review itself is pretty standard for Cochrane,” Gideon Meyerowitz-Katz, an epidemiologist from the University of Wollongong in Australia, said in an email, adding that “it is of a similar quality to most Cochrane reviews — I cannot find any major errors in the process.”
But some experts objected to some of the language the authors used to summarize their results and took issue with certain decisions and interpretations of the review. All agreed that it was incorrect to conclude that the review shows masks “don’t work.”
Review Criticisms
Benjamin Cowling, an epidemiologist at the University of Hong Kong who has studied masks, said he thought the review was very similar to other systematic reviews on the topic. But he found the authors’ conclusion that wearing masks in the community “probably makes little or no difference” to be problematic.
He noted that the confidence intervals for the meta-analysis for unconfirmed and lab-confirmed influenza and COVID-19 “go down as low as 0.84 and 0.72,” and said that “these effects (16% reduction and 28% reduction, respectively) would not be considered little or no difference.”
Cowling has long said that community masking could reduce transmission by around 10% to 20% — “a small to moderate effect which is worthwhile,” he said, and views the Cochrane review as “completely consistent with that.”
Meyerowitz-Katz also didn’t think the wording in the review was “entirely reasonable,” adding that one of the review authors told him the group had debated the phrasing.
“Specifically I think they should have downgraded the certainty from ‘may or probably’ to very low certainty language, because they are combining influenza with COVID-19,” he said. “This is, of course, a bit subjective both on my part and theirs, but it’s important.”
One criticism about the review is that it combined results from flu and COVID-19 studies. Only two of the 12 main studies on masks were conducted during the COVID-19 pandemic.
Meyerowitz-Katz said the “entire purpose of running this review” was to determine whether mask interventions were effective in preventing COVID-19 in communities, and it basically found there wasn’t much research examining that. “So why do the review at all?” he said.
“It was reasonable to look at evidence on other respiratory viruses at the beginning of the pandemic when we didn’t have evidence on SARS-CoV-2,” Dr. Roger Chou, a professor of medical informatics and clinical epidemiology at Oregon Health & Science University, said, referring to the virus that causes COVID-19. “But now that we have evidence on SARS-CoV-2, I don’t think that is the best approach.”
And looking at the COVID-19-specific studies, the findings are still uncertain, but lean toward a small protective effect, several experts said.
A cluster randomized controlled trial published in the journal Science in December 2021 found that handing out free masks and promoting their use in communities in rural Bangladesh led to a nearly 30 percentage point increase in mask wearing and reduced the risk of COVID-19-like illness by about 10%.
A randomized controlled trial from Denmark, published in Annals of Internal Medicine in November 2020, identified an 18% reduction in risk of coronavirus infection among individuals who received free surgical masks and were told to wear them outside the home, but this result was not statistically significant. We’ve explained before that the trial was only designed to detect a large effect of 50% or more.
“Taken together, these two RCTs are consistent with a small reduction in risk,” Chou said. “The trials are not perfect, but it is very challenging to do these studies and I haven’t seen anything to invalidate the results of the studies.”
Chou, who is an expert in evidence-based medicine and has co-authored a rapid review about the effectiveness of mask interventions, also said that both trials might have underestimated the benefits of masks.
In the Bangladesh trial, for example, less than half of the people in the intervention arm wore masks, and with higher adherence the benefits might have been greater, he said. He added that the Danish trial wasn’t designed to see whether masks helped prevent those who were wearing masks from spreading COVID-19, or what is known as source control, so it may not have captured the true value of masks.
Meyerowitz-Katz noted that since the cutoff for consideration with the Cochrane review, a third randomized controlled trial on masks during the pandemic has been released as an unpublished preprint. Aggregating the three studies together, he said, “they show a consistent and fairly convincing effect.”
“To me, this shows that there is a reasonably clear modest benefit to community masking interventions during the COVID-19 pandemic, decreasing the rate of infections in groups of people who are given masks and told to wear them by ~13%,” he said. “That’s quite an important benefit in the context of a pandemic.”
Brainard, however, who is an expert on systematic reviews and published a review of the evidence on masks earlier in the pandemic, thought it was reasonable to group the influenza and COVID-19 studies together. She nevertheless agreed that the primary message of the review should be one of uncertainty.
“I can’t argue with the Authors’ own written conclusions: the evidence quality is variable and creates a huge amount of uncertainties,” she said in an email. “We can’t tell from available evidence that masks prevented infections according to our usual standards, which is 95% confidence that protection was achieved.”
That’s different from saying that masks don’t work for community spread or for an individual. And it’s also different, she said, from saying that mask mandates don’t work, since in trials mask-wearing is voluntary, rather than being required for everyone in the community.
“It is lamentable that during [the] pandemic not enough RCTs were done to [have] better evidence what NPIs work or not,” she said, referring to non-pharmaceutical interventions.
How to Think About Masking
One point of confusion for many people is that lab studies show masks, mostly well-fitting and high-filtering N95 respirators, are good at blocking viral particles. This is evidence that masks can in theory be quite effective, both for individuals and for larger populations.
Some scientists have pushed for people to use better masks, especially N95 respirators, if people truly want their masks to work.
But as we’ve explained before, this mechanistic evidence doesn’t necessarily mean that when public health officials recommend masks that this intervention will work to limit spread in the community.
“Showing that N95s stop particles in mannequins only proves that they can plausibly help, but if people hate wearing them, or don’t wear them properly, or only have access to cloth masks, etc, then the actual intervention will not be as effective,” Meyerowitz-Katz said.
Many observational studies have been done to try to understand what masks or mask interventions do, and Chou said they have “generally found masks to be associated with reduced risk of SARS-CoV-2.” But they have “major limitations,” he said, since it’s hard to know if masking is the reason for the differences between groups.
Still, Chou said that overall, this other evidence outside of randomized controlled trials “supports some benefits of masks.”
“I think the evidence indicates that masks likely have small benefits for individuals in preventing COVID-19 (~15% reduction), though with only two RCTs more evidence would of course be helpful for clarifying the benefits,” he said.
“Even if the benefits are small for an individual, they are still important when considered from a population/public health perspective,” Chou added.
Brainard suspects that masks may not make as much of a difference as people may have hoped at the beginning of the pandemic.
Surgical masks “probably help prevent respiratory infections,” she said, but only a little — and by delaying infection rather than fully preventing it.
“Many people hate wearing masks, including people who believe that masks are truly very protective. Any intervention that must be sustained for long periods but people find difficult to sustain is not a great intervention,” she said.
But even if it turns out that mask recommendations don’t do very much, that doesn’t mean it was wrong to try them.
“Buying time until vaccines (or an amazing treatment) could be developed was the purpose of the masks,” she said. “My take is that public health officials in 2020-21 didn’t have a lot of options: without an effective vaccine, yet they couldn’t make people stay at home forever. Masks were a reasonable thing to try.”
Cowling agreed that the main function of masks is to delay infection, and that people should have realistic expectations for them.
“Ultimately even a very careful person will be infected eventually, but masks could delay that or reduce the rate of infection from once per year to once every few years, perhaps,” he said. “Community masking is not aimed to prevent everyone from ever getting infected, the aim is to reduce transmission and ‘flatten the curve’, reducing peak healthcare demand, or to work in combination with other measures like social distancing to contain transmission in the short-term.”
Editor’s note: SciCheck’s articles providing accurate health information and correcting health misinformation are made possible by a grant from the Robert Wood Johnson Foundation. The foundation has no control over FactCheck.org’s editorial decisions, and the views expressed in our articles do not necessarily reflect the views of the foundation.
Sources
“About us.” Cochrane.org. Accessed 15 Mar 2023.
Jefferson, Tom et al. “Physical interventions to interrupt or reduce the spread of respiratory viruses.” Cochrane Database of Systematic Reviews. 30 Jan 2023.
Statement on ‘Physical interventions to interrupt or reduce the spread of respiratory viruses’ review. Cochrane.org. 10 Mar 2023.
Tufekci, Zeynep. “Here’s Why the Science Is Clear That Masks Work.” New York Times. 10 Mar 2023.
Gorski, David. “The ‘spiritual child of the Great Barrington Declaration’ promotes antivaccine misinformation.” Science Based Medicine. 24 Jan 2022.
Putterman, Samantha. “Great Barrington herd immunity document widely disputed by scientists.” PolitiFact. 27 Oct 2020.
“Systematic Reviews.” Stephen B. Thacker CDC Library. Accessed 15 Mar 2023.
“Cochrane and systematic reviews.” Cochrane Consumer Network. Accessed 15 Mar 2023.
Higgins, Julian P. T. et al (editors). Cochrane Handbook for Systematic Reviews of Interventions, version 6.3 (updated February 2022). Accessed 15 Mar 2023.
Useem, Johanna et al. “Systematic Differences between Cochrane and Non-Cochrane Meta-Analyses on the Same Topic: A Matched Pair Analysis.” PLOS ONE. 15 Dec 2015.
Brainard, Julii. Senior research associate, Norwich Medical School, University of East Anglia. Emails to FactCheck.org. 1 and 9 Mar 2023.
Meyerowitz-Katz, Gideon. Epidemiologist, University of Wollongong. Emails to FactCheck.org. 2 and 9 Mar 2023.
Cowling, Benjamin. Professor and Division Head, Division of Epidemiology and Biostatistics, University of Hong Kong. Email to FactCheck.org. 28 Feb 2023.
Chou, Roger. Professor of medical informatics and clinical epidemiology, Oregon Health & Science University. Email to FactCheck.org. 9 Mar 2023.
Bundgaard, Henning et al. “Effectiveness of Adding a Mask Recommendation to Other Public Health Measures to Prevent SARS-CoV-2 Infection in Danish Mask Wearers: A Randomized Controlled Trial.” Annals of Internal Medicine. 18 Nov 2020.
McDonald, Jessica. “Danish Study Doesn’t Prove Masks Don’t Work Against the Coronavirus.” FactCheck.org. 25 Nov 2020.
Nanque, Line M. et al. “Effect of Distributing Locally Produced Cloth Facemasks on COVID-19-Like Illness and All-Cause Mortality – a Cluster-Randomised Controlled Trial in Urban Guinea-Bissau.” Lancet preprint. 5 Jan 2023.
McDonald, Jessica. “The Evolving Science of Face Masks and COVID-19.” FactCheck.org. 2 Mar 2021.
Brosseau, Lisa M. et al. “COMMENTARY: Wear a respirator, not a cloth or surgical mask, to protect against respiratory viruses.” CIDRAP. 23 Feb 2023.
McDonald, Jessica. “COVID-19 Face Mask Advice, Explained.” FactCheck.org. 6 Apr 2020.