First things first: The Mayo study was never designed to test effectiveness. It lacked a control group, as many have pointed out. An observational study such as this one—which simply tallies up what happens without being able to minimize biases—will often exaggerate the benefits of treatment. It may even end up giving the impression that a useless or harmful treatment is actually helping. There are warning signs with this particular study, too. The patients who received transfusions of convalescent plasma in the Mayo study might not have been so representative of people hospitalized with Covid-19. For example, only 19 percent were Black. According to the Centers for Disease Control and Prevention, more than 30 percent of hospitalized Covid-19 patients in the US are Black. The CDC also reports that Black people are more than twice as likely to die of the disease than white people.
In any case, once the Mayo researchers had acquired all this observational data, they decided that maybe they could explore whether the treatment worked, after all. The donated plasma differed from batch to batch: Some of the plasma came from recovered Covid patients who had very high levels of antibodies, and some came from recovered patients with much less, or possibly even none. (Back when the study started, the researchers didn’t have a way to test for antibodies in the plasma.) So they decided to compare outcomes for people who had received transfusions of “high-,” “medium-,” and “low-” antibody plasma. Their thinking was, if convalescent plasma really works, then batches with more antibodies would be likely to provide a stronger dose and yield a greater benefit.
There were some problems with this approach, beyond the absence of a control group that could show whether some people might be better off without any plasma. Most of the plasma was gone and couldn’t be tested for antibodies. A small proportion of samples had been collected at different blood banks for quality assurance purposes and so could be used for the analysis. But it was not a random sample.
It was this analysis that showed the difference in mortality rates touted at the press conference. Among hospitalized Covid-19 patients who had received a “low antibody” dose of plasma, 13.7 percent died within a week; among those who’d gotten a “high antibody” dose, that rate was 8.9 percent. In other words, the patients who got the “best” convalescent plasma had a risk of dying that was 4.8 percentage points lower.
Now, 4.8 is also about one-third of 13.7, which is another way of describing the difference. When Trump announced a 35-percent reduction in mortality on Sunday, that’s the figure he was referring to—known as a relative risk reduction. But that’s not what it sounded like, and it’s not what Hahn would say a moment later, when he told the press that convalescent plasma could save the lives of 35 of every 100 people who get sick from Covid-19. This was obviously absurd: nothing could possibly drop the mortality rate by 35 points, because the mortality rate among all people who get sick from Covid-19 is nowhere near that high: Only around 20 percent even become severely ill. (On Monday night, Hahn admitted that he’d mixed this up.)
But even described correctly, you can’t claim that everyone who gets a transfusion of convalescent plasma could expect this result. That data applies only to the specific, somewhat non-representative sample of patients from the Mayo Clinic study and to transfusions of convalescent plasma that have a particular level of antibodies. What’s more, the number of people included in the study’s “lowest” and “highest” antibody groups was low—no more than 1,100 out of the total of 35,000. In the FDA’s supporting documentation, scientists reported that their reanalysis of the samples using a different antibody test got very different results. Change the threshold for what you call “high-,” and the numbers shift again. The point is, there are multiple levels of unreliability here.
Most of all, though, the finding isn’t solid enough to rely on, because this wasn’t a planned, randomized trial. The analysis raises an interesting hypothesis, but it can’t prove it. The Mayo study raises an equally likely, but worrying, possibility: that the vast majority of donated plasma—which doesn’t have the highest levels of antibodies and usually hasn’t been administered early in the course of illness—did not help at all, and maybe even did more harm than good to some people. Only randomized trials can disentangle this.