On Wednesday, one of the world’s most prestigious medical journals published what many took to be a disheartening result: According to some headlines, a 6,000-person randomized controlled trial in Denmark had found that wearing a mask does not offer any clear protection from being infected with the SARS-CoV-2 coronavirus. This follows months of heated debate, and the assertion in October from White House Coronavirus Task Force member Scott Atlas that, in light of “considerable uncertainty” in the research literature, masks cannot be said to work.
Noah Haber (@NoahHaber) is a postdoctoral fellow at the Meta-Research Innovation Center at Stanford University, specializing in assessing and improving strength of evidence in health and medical research.
Atlas may indeed be right, albeit only in a technical sense; there is no study which provides direct, strong, and reliable evidence that lets us estimate the precise extent to which masks and mask mandates reduce the spread of the pandemic. But that lack has little to do with how well masks work, and everything to do with the profound difficulty of doing this kind of research. The newly published trial, which was named DANMASK-19, is no exception. Its limitations are severe enough to render it almost completely uninformative.
The impact of masks is not just unknown, it’s unknowable.
To understand why, consider how hard it is to estimate the impact of the mask mandates from last spring. Covid was ramping up around the world, and everything from policies to infectious disease dynamics to social behaviors was changing all around us and all at once. Each of these factors affected the spread of the virus in complicated and interdependent ways, with unknown lags between when the changes occurred and when their effects might be expected to show up in the data. Add in the fact that testing availability and data access have also shifted over time, and it’s clear that isolating the impact of just one intervention—a mandatory masking rule, let’s say—is a heroic, maybe impossible feat.
Still, there are now dozens of studies that have attempted to do just this. One high-profile example, published last June in the Proceedings of the National Academy of Sciences, claimed that early mask mandates like the one in New York City had reduced the number of local infections by many tens of thousands. (The paper was widely covered at the time, and has been cited many times since.) Unfortunately, the findings were based on flawed methods and study design; when the Johns Hopkins University Novel Coronavirus Research Compendium reviewed the study’s strengths and value added, it gave a one-word judgment: “none.” The issues were sufficiently alarming that my colleagues and I led an effort to have the journal editors and authors retract it. Several months later, a representative from the journal’s editorial board told us that they’d “found errors” in the paper but would not be retracting or correcting it, and declined to elaborate further.
The DANMASK-19 randomized controlled trial isn’t informative either, but for very different reasons. First of all, this wasn’t a trial about mask-wearing; it was a trial about messages to wear masks. Participants were first encouraged to practice social distancing, and then half were randomly selected to receive repeated messages to wear masks whenever they left the house for the next month, as well as a box of 50 disposable face masks. Any protective effect those masks may have had was dampened by the fact that many of the participants didn’t actually use them: In the end, less than half the people in the intervention group reported having worn the masks as recommended. Further, an unknown number of people in the control group may have made their own decision to wear masks anyway, though this was probably limited. The small number of SARS-CoV-2 infections that occurred during the trial—just 95 cases in all—makes it too small to detect anything other than an absurdly large effect. But the most glaring flaw of this research is that it did not (and for practical reasons, could not) measure what is likely to be the most important benefit of masks: reduced transmission to others.
None of this was unexpected. My colleagues and I registered our concerns about the trial design on Sept. 8, and now our predictions have been confirmed. The design was destined for insignificance, diluted effects, and misinterpretation regardless of how effective mask wearing really is. Although the publication itself clearly indicates its limitations, that hasn’t stopped some from saying that this trial shows that masks are ineffective. In fact, it tells us little to nothing about this question. Randomized controlled trials are often considered the gold standard for medical evidence, but some trials offer little more than glitter.