:format(webp)/cdn.vox-cdn.com/uploads/chorus_image/image/67766897/GettyImages_1227739911.0.jpg)
On Monday morning, my mom sent me a text message: “Black cloud is lifting ... Pfizer has a 90 percent effective vaccine.” She was one of the many people buoyed by the announcement from the drugmaker that early data suggests its Covid-19 vaccine — developed with the German biotech firm BioNTech — can prevent infection from the disease.
What my mom missed, by no fault of her own, was that the news didn’t come with any detailed data, regulatory review, published study, or preprint. In fact, the trial isn’t even completed.
Instead, it was the latest instance of “science by press release” in the coronavirus pandemic.
In the race to stop the virus, drug companies, research groups, and other players with a stake in the development of Covid-19 vaccines and medicines have repeatedly announced early results from ongoing clinical trials in excited press releases.
There’s some justification for tipping off the public as soon as possible — before the usual scientific review process is complete: We are in the middle of the most devastating public health crisis in a century. “We’ve been on tenterhooks about information like this, and we want radical transparency,” writer and meta-scientist Hilda Bastian, who has been carefully following the vaccine race for months, reasoned.
But these press releases — picked up by the media, then reported to a public desperate for good news — have often culminated in dashed hope when the necessary scientific and regulatory steps play out and the final results are less stunning than initially reported.
In the case of Pfizer and BioNTech, there certainly appears to be good reason for optimism (more on that in a moment). But it’s still too early to know whether, as my mom suggested, the black cloud is lifting. If hope keeps getting dampened, however, we risk further eroding the already fragile public trust in science — and in the vaccines and treatments we’ve all been waiting for to finally end the pandemic.
Too many coronavirus medicines and vaccines have arrived with a side of hype
While it feels like the pandemic has been going on for 8,000 years, it was only in December 2019 that the novel coronavirus was discovered. And in this short time, there have been multiple instances of early, and trumped-up, coronavirus vaccine or medicine claims that don’t quite pan out.
Back in May, Moderna — another company with a coronavirus vaccine that’s far along in clinical trials — put out a press release about promising phase 1 results. While it first caused the company’s stock valuation to swell, vaccine researchers pointed out in Stat that the information in the press release was far too preliminary and vague to gauge whether the vaccine was actually working.
Or remdesivir: In April, the National Institutes of Health announced via press release some promising results for hospitalized patients. Later, further studies complicated the picture of the drug’s effectiveness, and today it’s not clear that remdesivir carries any benefit.
Or how about the Regeneron drug REGN-COV2, which President Trump called a “cure” in October? Most of what we know about its effectiveness comes from a September 29 Regeneron press release about a multi-phase, randomized, double-blind clinical trial involving only 275 people. (The results said nothing of whether the drug cut the risk of death or “cured” people.)
:format(webp):no_upscale()/cdn.vox-cdn.com/uploads/chorus_asset/file/22026654/GettyImages_1229305124.jpg)
There have been some cases where early hype bears out: The press around the promise of dexamethasone for critically sick Covid-19 patients was later backed up by the Recovery Trial.
There have also been examples of peer review in journals failing to root out bad coronavirus science (see: The Lancet hydroxychloroquine controversy).
But press releases and news conferences have been particularly fraught venues for science communication — and the problem predates the pandemic. It also extends into the White House.
We’ve known for years that when news releases feature exaggerated claims, there are greater odds the media that flows from them will be similarly hyped. We’ve also known that when drugmakers don’t publicly disclose detailed data about their methods and results, there’s no way of checking if the findings are reliable. (That’s why there’s a movement in medicine, nearly a decade old now, called the All Trials Campaign, which advocates for all clinical trials to be registered and their results to be reported.)
Hype, exaggeration, and unverifiable claims are the last thing we need at a moment when there’s an infodemic going on in parallel with the pandemic, and the most powerful person in the country is also the largest driver of Covid-19 misinformation.
There’s real reason for optimism with the Pfizer and BioNTech vaccine — but it’s too early to know for sure
When asked why Pfizer chose to release the interim results of its phase 3 clinical trial in a press release, a media representative explained that the study is still blinded to preserve the integrity of the trial. Once Pfizer and BioNTech can see the results, and the data are analyzed, “[we] plan to submit the full efficacy and safety data from our Phase 3 trial for peer-review in a scientific journal,” the spokesperson added.
So what we know about the vaccine right now is mostly detailed in this press release.
The trial, which began in July and enrolled more than 45,000 participants, was originally set up to allow an independent, external review committee to periodically look at the data, and see how the vaccine is working and whether the study should be stopped, five times: when 32 people in the trial were diagnosed with Covid-19, then 62, 92, 120, and 164. These points are chosen for a statistical reason, Bastian explains over at Wired: They’re the number of cases that “would need to be tallied in order to know that the vaccine had cleared the minimum requirement of 50-percent effectiveness set by the Food and Drug Administration.”
But after discussion with FDA, the vaccine makers agreed to drop the first analysis (at 32 cases) and instead look at the data after 62 people had fallen sick with Covid-19. This meant that any finding about the vaccine’s effectiveness was even more statistically robust. By the time the review board actually analyzed the data for the first time, 94 people had been diagnosed with Covid-19 — more than halfway to their final review at 164 confirmed cases.
Ninety-four cases is a lot of data. And even if it’s early, it suggests the finding of more than 90 percent efficacy may be particularly statistically powerful, as drug discovery chemist and blogger Derek Lowe explains.
What’s more, over 90 percent efficacy is a far greater degree of protection than most people expected. (There have been serious discussions about life with a 50 percent effective vaccine.) As Vox’s Umair Irfan explains, “if 10 people received the vaccine, at least nine of them will have protection against the virus compared to the placebo.” To date, the external data monitoring committee also hasn’t found any serious safety issues with the vaccine.
If the vaccine turns out to be as safe and effective as it appears now, it would put it in the same league as extremely protective routine childhood immunizations, such as the measles shot, the New York Times reports.
But the trial is not finished. And while the press release was quite thorough as far as press releases go, it wasn’t detailed enough to really understand the vaccine’s effects, Boston University epidemiology professor Ellie Murray pointed out on Twitter. For example, we don’t know how many people dropped out in each arm of the trial (the group that got the vaccine versus the group that didn’t) — a phenomenon in clinical studies called “loss to follow-up.”
“Loss to follow-up could potentially have inflated their estimate if it occurred differentially between trial arms and I’d like to know how much drop out there was and whether and how they’ve adjusted for it,” Murray tweeted.
“It’s really problematic to be telling people what the data say but not allowing them to see the data for themselves,” University of Maryland School of Pharmacy professor Peter Doshi, who has been a prominent critic of Covid-19 vaccine trials, said.
The press release also raised a bunch of questions for Doshi that aren’t answerable now: Did the shot cut the risk of serious complications from Covid-19? Did the researchers see fewer hospitalizations or ICU admissions in the vaccinated group? Or did they just see a reduced risk of mild infection? He wondered, too, about exactly who was immunized in the trial: How many are low risk versus how many high risk of severe Covid-19?
“We need to know whether the people entering the trial are representative of the kinds of people who stand to benefit most from a safe and effective vaccine,” Doshi said.
While the manufacturers could share anonymized, demographic data right now, “to ensure they’re not just recruiting easy to find, low risk people into trials,” he added, they haven’t even seen the efficacy and safety data because of the blinding process. So the press release is based on “a positive note from its review board.”
“I don’t think sharing premature results is the way to go,” Doshi said. “That unnecessarily sends the world into a frenzy talking about what this means, but we don’t really have any way of making sense of what has transpired since we don’t have access to the data.” And that’s saying nothing of the complexities of using the vaccine in the real world, such as the cold storage challenge.
The science by press release has to stop
Ideally, we’d all be a little more patient. Researchers and drug companies running vaccine trials would wait until they had data to share, and were confident in their findings, to tell the public about them, reducing the risk of flip-flopping.
When they did go public, they’d be more restrained in their press releases. And if they’re releasing results before studies are examined by a regulator, or published and peer-reviewed, they’d at least come out with detailed data so the public and policymakers could make informed choices.
For Bastian, this dearth of more detailed data is an even greater problem than how and where the news is shared. “Many argue there should be preprints or journal articles first. But we use data from various sources to assess clinical trials all the time, and so I’m not quite as concerned about that aspect,” she reasoned. “It’s the accuracy and detail that matters, and there’s too little detail here.”
The stakes for an effective vaccine in the pandemic couldn’t be higher: More than 50 million people have been diagnosed with Covid-19, and more than 1.2 million have died. These are just the known cases and deaths. The virus has upended lives and economies around the world, and governments are investing billions in vaccines to end the crisis.
When we do finally have a viable Covid-19 immunization, the public needs to be able to trust that it’s safe and effective. And the road to building that confidence isn’t paved with preliminary, sparsely detailed news releases that are later overturned when the real work of vetting the vaccines is done.
“I’m going to hold my applause and continue to insist that to declare any Covid-19 product is based on science requires that the data be publicly available for all to scrutinize,” Doshi said.
When the Pfizer and BioNTech data is eventually released and the trial is complete, we may learn this vaccine is as effective as it seems right now, and perhaps no harm will be done, Bastian said. “But if it turns out this is giving the world an exaggerated impression,” she added, “then it’s a whole other story.”
