clock menu more-arrow no yes mobile

Filed under:

Why comparing Covid-19 vaccine efficacy numbers can be misleading

The best Covid-19 vaccine for you is most likely still the first one you can get.

Madeline Marshall and Kim Mas / Vox
Umair Irfan is a correspondent at Vox writing about climate change, Covid-19, and energy policy. Irfan is also a regular contributor to the radio program Science Friday. Prior to Vox, he was a reporter for ClimateWire at E&E News.

Three different Covid-19 vaccines are now being distributed across the United States, and all three are highly effective at the most important thing: preventing hospitalizations and deaths from Covid-19. But some people remain worried that Johnson & Johnson’s vaccine is less effective at preventing disease to begin with.

Detroit Mayor Mike Duggan this week turned down 6,200 Johnson & Johnson vaccine doses for his city. “Johnson & Johnson is a very good vaccine. Moderna and Pfizer are the best,” Duggan said in a news conference. “And I am going to do everything I can to make sure that residents of the city of Detroit get the best.”

Scientists say that this is the wrong way to think about Covid-19 vaccines, and that judging the Johnson & Johnson vaccine as inferior based on its lower reported efficacy is misleading.

Such actions are especially worrying at the current stage of the pandemic. Covid-19 has killed more than 500,000 Americans, and while cases seem to be declining, the virus is still spreading, new variants are gaining ground, and some parts of the country are already relaxing precautions (which health officials warn could end up prolonging the pandemic).

Turning down vaccine doses while supplies of all Covid-19 vaccines are still stretched thin undermines the campaign to curb the pandemic.

In clinical trials, the vaccines produced by Pfizer/BioNTech, by Moderna, and by Johnson & Johnson reduced the fatality rate of Covid-19 by 100 percent compared to their placebo groups. They also kept all recipients out of the hospital. That means they can potentially downgrade Covid-19 from a public health crisis to a manageable problem.

“The goal of a vaccine was really to defang or tame this virus, to make it more like other respiratory viruses that we deal with, so when you look at the three approved vaccines in the US, all of them are extremely good at that metric,” said Amesh Adalja, a senior scholar at the Johns Hopkins Center for Health Security.

The vaccines do have some important differences. The Johnson & Johnson vaccine is one dose, while the others require two. It also can be stored at refrigerator temperatures, while the others require freezer temperatures. The Johnson & Johnson vaccine is also less expensive, about $10 per dose, roughly half as much as the Pfizer/BioNTech vaccine. The Moderna vaccine costs between $25 and $37 per dose.

These factors give Johnson & Johnson an edge in logistics and could help the shots get to people in harder-to-reach places. Saad Omer, the director of the Yale Institute for Global Health, told Vox last month that it’s a vaccine that “can increase equity.”

But when Johnson & Johnson filed for an emergency use authorization from the Food and Drug Administration for its Covid-19 vaccine in early February, it reported that its overall efficacy in preventing Covid-19 cases that produced symptoms was 66.1 percent. The Moderna vaccine and the Pfizer/BioNTech vaccines reported efficacy levels around 95 percent.

That gap in efficacy numbers is fueling some people’s perception that the Johnson & Johnson Covid-19 vaccine isn’t as good. However, scientists say that these numbers can’t be fairly compared to one another. The efficacy levels of the Covid-19 vaccines are specific to the clinical trials that produced them, and those trials were not conducted in the same ways.

In addition, health officials have been emphasizing that the most important numbers — how well the vaccines prevent hospitalizations and deaths — are consistent across the board and are arguably more comparable. Even after these vaccines have begun distribution, researchers are finding that Covid-19 vaccines are doing a remarkable job of keeping people alive.

That’s why the recommendation remains that the best Covid-19 vaccine for the vast majority of people is the first one they can get. “That’s how I think of these vaccines, as basically interchangeable,” said Adalja.

Why it’s hard to make direct comparisons between the Johnson & Johnson Covid-19 vaccine and the ones from Moderna and Pfizer/BioNTech

To gauge how well vaccines work, companies test them in several stages, looking to ensure they are safe, to find the correct dose, and to figure out how much protection they provide. These trials are designed to test vaccines individually, not to pit them against each other. So direct comparisons don’t always make sense, and one has to be careful to understand the nuances of how each result was obtained.

But health officials have acknowledged that the earlier results of the Moderna and Pfizer/BioNTech vaccines shifted expectations of the Johnson & Johnson vaccine.

“If this had occurred in the absence of a prior announcement and implementation of a 94, 95 percent efficacy [vaccine], one would have said this is an absolutely spectacular result,” said Anthony Fauci, director of the National Institute of Allergy and Infectious Diseases, about the Johnson & Johnson vaccine during the press conference in January.

In phase 3 clinical trials, Covid-19 vaccines were tested against the virus in the real world, in actual people against the actual virus. This involves testing tens of thousands of participants to see who ends up showing symptoms, randomly dividing them into groups that receive the actual vaccine and groups that receive a placebo (without revealing who got what).

Testing in the real world means dealing with all the confounding factors of the real world. Depending on which volunteers are selected and where they are, they face different infection rates of the virus. They have varying access to health care. Some places had stricter lockdowns than others, or started them at varying times, so participants experienced different public health measures. Michigan issued a mask mandate in March 2020 while California issued one in June 2020, for example.

Timing is critical too. The Moderna and Pfizer/BioNTech studies finished enrolling participants in their phase 3 trials in October and reported their results in late November. The Johnson & Johnson phase 3 trial only finished enrolling participants in December 2020 and reported their results in January.

That means the Johnson & Johnson vaccine was tested during one of the most severe stages of the pandemic, when transmission, cases, and hospitalizations were at their worst in many places around the world, including the US. The trial also captured efficacy against the new variants of SARS-CoV-2 (the virus that causes Covid-19) which began circulating at this point in some parts of the world. Several of these variants have shown themselves to be more contagious, deadlier, and more likely to evade protection from vaccines and prior immunity.

And Johnson & Johnson’s efficacy results included trials in other countries, whereas the results from Moderna and Pfizer/BioNTech were mainly from US-based participants.

Johnson & Johnson found that vaccine efficacy shifted depending on the country in which it was studied. The vaccine was found to have a 72 percent overall efficacy after four weeks in preventing Covid-19 symptoms in the US. Under the same benchmarks in South Africa, where a coronavirus variant with worrisome mutations that help it escape vaccines has been spreading widely, the company found a 64 percent efficacy.

When it came to preventing severe and critical cases of Covid-19, the Johnson & Johnson vaccine was 85.9 percent efficacious in the US, while in South Africa, efficacy against severe and critical disease was reduced to 81.7 percent.

The fact that these vaccines were tested in different ways at different times is why it’s so hard to make apples-to-apples comparisons. “I don’t even look at those efficacy numbers and compare them head-to-head like that,” Adalja said. “Biostats 101: You cannot compare trial results like that unless they were done in a head-to-head fashion.”

Researchers still need more information about Covid-19 vaccines, especially as new variants spread

The huge emphasis on the fact that vaccines prevent hospitalizations and death doesn’t mean that preventing the symptoms of Covid-19 is not important. Millions of people in the US have preexisting health conditions and could suffer from the disease even if they don’t end up in the hospital. About 10 percent of Covid-19 survivors have reported persistent symptoms even after the virus has faded away, the so-called long haulers. It hints that the disease can cause long-term damage.

And while vaccines can protect an individual, it’s less clear how well they prevent transmission from person to person (although evidence is mounting that the available Covid-19 vaccines reduce the virus’ spread). That’s why vaccinated people are encouraged to continue wearing masks until vaccinations are widespread.

An ideal Covid-19 vaccine would reduce deaths, hospitalizations, symptoms, and transmission, and right now, all of the three Covid-19 vaccines available in the US check these boxes, even for people with risk factors for severe disease or long-term illness.

“I wouldn’t be picky if I’m a high-risk person, because being picky may leave you out in the cold of not being vaccinated,” said Lawrence Corey, a professor studying virology at the Fred Hutchinson Cancer Research Center. “We have still an incredible epidemic going on here.”

There are some people with a history of severe allergic reactions or certain immunological conditions who will have to be careful about selecting a vaccine, and some may not be able to receive one at all. But that makes it all the more important to vaccinate everyone around a vulnerable person, which helps build herd immunity.

The looming concern, though, is how well Covid-19 vaccines will hold up as the SARS-CoV-2 virus continues to mutate and new variants arise. Already, vaccine manufacturers are investigating booster doses and modifications of their shots to better counter the newer versions of the virus.

Researchers will also have to figure out how well existing vaccines are holding up against the variants in the real world. While vaccine clinical trials were conducted independently of each other, it would behoove scientists to coordinate from here on out, sharing protocols and pooling data to draw more useful conclusions.

“Imagine what will happen when these studies generate results, each with their own populations, eligibility criteria, validation procedures and clinical endpoints,” wrote Natalie Dean, an assistant professor of biostatistics at the University of Florida, in Nature. “If we don’t want our final answers to be a jumble, we must act now to consider how data can be compared and combined.”

In the meantime, it’s important to keep in mind that vaccines are one part of a comprehensive public health response to Covid-19. Social distancing, hand-washing, mask-wearing, testing, tracing, and isolation remain critical to speeding up progress toward the end of the pandemic.