clock menu more-arrow no yes mobile

Filed under:

How bad research clouded our understanding of Covid-19

Early studies of Covid-19 therapeutics turned out to be fabricated or suspicious. That’s a huge problem for science.

Photo illustration of a magnifying glass looking at the word hydroxycholoroquine printed on a piece of paper. Getty Images/iStockphoto
Kelsey Piper is a senior writer at Future Perfect, Vox’s effective altruism-inspired section on the world’s biggest challenges. She explores wide-ranging topics like climate change, artificial intelligence, vaccine development, and factory farms, and also writes the Future Perfect newsletter.

The scientific integrity site Retraction Watch has been a bulwark of sound science since its founding in 2010. Its self-proclaimed mission is to act as a “window into the scientific process” by shining a light on academic research that has been retracted.

Retractions come from across all scientific fields, but over the past two years, many of the most consequential ones have revolved around a single topic: Covid-19. And according to Retraction Watch, as many as 200 Covid-19 papers have been retracted since the start of the pandemic for a range of reasons: elementary calculation errors, researchers refusing to provide evidence that the studies were really conducted, or conclusions that aren’t supported by the data.

The problem is especially acute when it comes to studies looking into treatments for Covid-19. Early papers claiming to find stunning results for drug regimens like ivermectin or hydroxychloroquine earned tremendous attention and influence, only to later be retracted, often because substantial evidence suggested that the studies never actually happened or at least never happened as described.

One ivermectin study included in an influential meta-analysis that found great results for the drug turned out to be based on a data file where the same 11 patients were copied and pasted repeatedly to produce a more robust sample size of a few hundred. When BuzzFeed News followed up on another ivermectin study with huge results, a hospital where the research had reportedly been conducted said it had no record of such a study happening there.

As I wrote for Vox in September, the evidence for ivermectin is mixed and limited. Most careful analyses suggest it might have an effect but also that we don’t know enough to say for sure — certainly not enough to hail it as a miracle cure, as many have. Part of the reason the treatment rose to prominence is that early, high-profile studies of ivermectin showed positive results, sometimes huge ones. When other researchers peered under the hood, however, they realized those findings didn’t hold up or had no basis to begin with. But by then it was too late — for some, ivermectin had become an article of faith.

To be clear, it’s not all an epistemic hellscape. There’s sound research being performed on early treatments, including ivermectin. But the damage done by fraudulent research is immense and has contributed to misleading public policymaking and individual decision-making. Narratives about effective cures take hold only for retractions to cloud the picture, abetting bad-faith actors and inflaming our all-consuming culture war.

If we are to finally overcome this pandemic and defeat the next one, it’s essential for us to fix a system where it’s all too easy for bad research to warp scientific discourse and public knowledge.

A look at papers that show signs of serious error or misconduct

One of the first high-profile instances of alleged Covid-19 research fraud occurred last spring: A study in The Lancet claimed to have looked at more than 96,000 coronavirus patients across the world and found that — after controlling for age, sex, and how sick the subjects were — patients receiving hydroxychloroquine or a variant of the drug were about twice as likely to die as those who did not.

But scientists immediately started pointing out that something was wrong with the study. It had reported more Covid-19 deaths of enrolled patients in its Australia arm than there were Covid-19 deaths in all of Australia. Further investigation found that hospitals supposedly enrolled in the study had never heard of Surgisphere, the company that conducted it.

The hydroxychloroquine paper was quickly retracted from The Lancet, as was another by the same authors in The New England Journal of Medicine, another highly prominent journal. Further research has demonstrated that hydroxychloroquine is not, in fact, an effective Covid-19 treatment — but it’s also not as deadly as the Surgisphere paper suggested.

In the months since, scientific misconduct, data fraud, and careless mistakes have been repeatedly found in the Covid-19 literature.

Jack Lawrence, a grad student in biomedical science at the University of London, kicked off a broad investigation into ivermectin research fraud when he got an assignment to look at a widely cited ivermectin study. That study, by professor of medicine Ahmed Elgazzar and others at Benha University in Egypt, found huge benefits for Covid-19 patients taking ivermectin.

Lawrence noticed something wrong immediately: The introduction was plagiarized from other papers. He then notified a colleague, Gideon Meyerowitz-Katz, an epidemiologist at the University of Wollongong in Australia. “Jack Lawrence emailed me about the Elgazzar study and said, ‘I know you think it’s low quality but are you aware that most of it’s plagiarized?’” Meyerowitz-Katz recalled. “And that started this whole thing.”

Data forensics experts found other problems with the Elgazzar paper: patients who died before the study started and yet were counted as enrolled; patient records duplicated; numbers that were too suspiciously clean to have occurred by chance. (Elgazzar says his study is legitimate.)

Soon, the preprint server that hosted the paper withdrew it. But the damage had been done — the Elgazzar study had already started feeding the widespread misconception in some quarters that ivermectin was a Covid-19 miracle drug.

A supporter of Brazilian President Jair Bolsonaro holds a large box of box of ivermectin during a protest organized to show support to his government in Brasilia, Brazil, in May. Bolsonaro has recommended the use of hydroxychloroquine and ivermectin to treat Covid-19.
Andressa Anholete/Getty Images

Lawrence and Meyerowitz-Katz consider themselves part of a loose-knit group of researchers — including Nick Brown of the University of Groningen in the Netherlands, Cipher Skin chief scientific officer James Heathers, and Kyle Sheldrick of the University of New South Wales in Australia — who have taken it upon themselves to check the integrity of Covid-19 research.

A big part of their work is simply to pore over the data and methods of Covid-19 studies. It was this team that investigated the paper, in the journal Viruses, that found that ivermectin was a highly effective treatment but that turned out to have a data set that was just the same 11 patient records copied over and over.

The study authors said, “After revising the raw data we realised that a file that was used to train a research assistant was sent by mistake for analysis.” (Though the authors admitted the error, they have yet to come out with a legitimate data set.)

The same group of researchers has also raised serious questions about a large randomized controlled trial in Iran that found positive results for ivermectin.

The problem? The data is “not consistent with a genuine randomised controlled trial,” Sheldrick argued in a detailed blog post laying out some of the suspicious inconsistencies. “This paper claims to describe a trial in which patients were randomly allocated to treatments. This is not true. Extreme differences are seen between groups across multiple variables such as oxygen level, blood pressure, and SARS-CoV-2 test results before they even got their first dose of medication.”

One simple but effective test for data fabrication is whether numbers have a typical distribution of “trailing digits” — for example, whether they end in “1” about as often as they end in “0.” Humans, when making up numbers, tend to cluster around certain end digits. The randomized control trial from Iran fails this test, with very pronounced digit clustering to a degree that can’t have occurred by chance, according to Sheldrick. (It’s “very normal to see such randomisation,” the lead author rebutted.)

Members of this research group have raised questions, too, about research published by Dr. Flavio Cadegiani on various Covid-19 treatments, including ivermectin and proxalutamide, that exhibited bizarre statistical patterns inconsistent with randomization. They also raised concerns about research into the anti-parasite medication nitazoxanide, which turned out to be riddled with statistical errors and data analysis problems. (Meanwhile, other researchers have noted that Cadegiani’s papers claim a horrifying death rate in the control group. One explanation is fabricated data; another is medical malpractice, prompting a parliamentary inquiry in Brazil to accuse Cadegiani of crimes against humanity.)

Other researchers have been uncovering more dubious work. A randomized controlled trial in Egypt studying favipiravir and hydroxychloroquine was retracted after glaring data inconsistencies were found.

A study that took place in Brazil that found incredible benefits of ivermectin as prophylaxis also failed to withstand scrutiny. Reporters at BuzzFeed News called one of the hospitals where the study was reportedly conducted for more information, only to learn that the hospital said that it hadn’t participated.

The sheer number of problem studies is eye-popping. In one major analysis, Lawrence, Meyerowitz-Katz, and others looked at 26 studies that were part of the evidence base for ivermectin and found that 10 of them had serious invalidating errors or evidence of likely fraudulent conduct. It’s one of the first attempts to estimate the scale of the problem, and it’s not encouraging.

“I’ve been working in this field for 30 years and I have not seen anything like this,” University of Liverpool’s Andrew Hill, who has been researching Covid-19 treatments, told MedPage Today. “I’ve never seen people make data up. People dying before the study even started. Databases duplicated and cut and pasted.”

How misconduct is interfering with our understanding of Covid-19

To understand how flimsy or even potentially fraudulent small-scale studies can have such a strong influence on broader conclusions about Covid-19, it helps to understand that a considerable part of our knowledge of treatments comes from meta-analyses.

A meta-analysis looks comprehensively at published evidence on a topic and figures out which way that evidence collectively points. A small study here and there might not give us findings we can count on alone, but put a bunch more studies together and you’re likelier to get a reliable picture of a drug’s effects.

But if some of the papers included are error-ridden or fabricated, it can easily throw off the entire meta-analysis.

Meta-analyses of ivermectin research have often combined some papers that found modest or no effects with papers that found dramatic, nearly miraculous effects and consequently a few of those meta-analyses have concluded that the drug should be expected to work remarkably well for Covid-19. (Quite reasonably, ivermectin’s enthusiasts often point to meta-analyses like these, which are generally considered valid scientific tools.)

That would be fine if all the studies looked at in a meta-analysis were of high quality and had been conducted as the authors described. But when a decent share of them might well be problematic or maybe even faked, aggregating all these studies won’t give you a clear picture of the drug’s efficacy, to say the least.

The University of Liverpool’s Hill conducted a meta-analysis that initially found positive results for ivermectin, only to conduct a re-analysis without studies that were later identified as suspect that found much worse results for ivermectin. “This has made me more wary about trusting results when you don’t have access to the raw data,” Hill told MedPage Today. “We took them on trust and that was a mistake.”

In the case of ivermectin, the phenomenon is especially striking because the bad papers “are bigger than average, make more dramatic claims than average,” Sheldrick told me. “While this is a minority of studies, they probably form about half of all the trial data we have on ivermectin for Covid.”

That makes sense — fakers often publish their studies faster, Sheldrick said, because running a real study takes a lot more time than fabricating data does. And they often claim to have run larger and more comprehensive studies.

“Running a 600-person trial is a hell of a lot more work than running a 50-person trial, but faking a 600-person trial isn’t actually that much more work than faking a 50-person trial,” he said. So in the case of an event like Covid-19 where researchers are rushing to complete trials as fast as possible, the rate of possible fraud tends to be higher in the earliest published research, which disproportionately influences further research, and analyses by number of patients treated will often end up over-weighting studies that didn’t occur as described.

The preponderance of bad research casts a shadow on the whole research enterprise two years into the pandemic. “The entire scientific community operates on trust,” Meyerowitz-Katz told me. “There is this assumption in research that if someone tells you they have done something, then they have done it.”

Unfortunately, there aren’t enough guardrails at other points in the process to prevent misleading or flawed work from seeing the light of day. The peer review process typically checks whether the authors of a paper are accurately interpreting their data, whether they’ve appropriately situated their results in the existing literature, and whether their approach to the topic is a good one. What it generally does not do is check whether the data set that the researchers used was faulty or even fabricated.

And publication in top-tier journals isn’t a safeguard against error or fraud. While some of the highest-profile studies that were allegedly fraudulent were published only as preprints or in poorly regarded journals, “some of the studies where we have very serious concerns about fraud are in very high-quality journals,” Meyerowitz-Katz told me.

“There are some studies that have found benefits for ivermectin that are certainly not fraud,” he added. But those benefits have tended to be small and still need to be confirmed by further research. “Any study reporting massive benefits from ivermectin, at this point, we’ve either found fairly strong evidence for fraud or the authors have declined to share data with us.”

A challenge to how we do science

There has been hardly any let-up for the retraction watchers. Last month saw another high-profile incident: The Journal of Intensive Care Medicine printed a retraction notice for a December 2020 paper about how to treat Covid-19 patients co-authored by Pierre Kory, who has testified in front of the US Senate about Covid-19 and has been a leading advocate for ivermectin.

Pierre Kory testifies in front of the US Senate on December 8, 2020.
Tom Williams/CQ-Roll Call, Inc via Getty Images

Kory and his co-authors’ paper centered on his so-called MATH+ treatment: a protocol with four headline drugs — methylprednisolone, ascorbic acid (Vitamin C), thiamine, and heparin — plus other treatments. (Kory has since included ivermectin in the protocol, but it was not part of his treatment regimen at the time the paper was published.)

The paper was retracted because a hospital where the research took place told the journal that patients there weren’t systematically offered Kory’s treatment, contrary to his claims, and that Kory misrepresented outcomes for the patients who did receive his treatment.

Moreover, officials at the hospital — Sentara Healthcare in Norfolk, Virginia — told me that Kory had ignored their calls when they reached out to ask for the error to be corrected. (Kory has said that it was inappropriate for the journal to retract the paper and that he has offered to correct the study.)

The long list of dubious studies adds up to a single big picture: Our understanding of ivermectin, and early Covid-19 treatments more broadly, has been badly damaged by studies reporting results that did not really occur as described.

The situation has implications for how scientific studies are performed and vetted that go beyond Covid-19 research.

The researchers who’ve been calling out bad studies have some ideas on how to improve things. In an article in Nature Medicine in September, they made the case for a radical rethinking of meta-analyses. Instead of re-analyzing using “summary data” from original studies, they argue, the scientific community should transition to doing meta-analyses that re-analyze by patient — effectively pooling all the patients across all the studies as if they were all part of one study, and then analyzing that.

Currently, patient data is not generally shared among researchers, both because of privacy concerns and because scientists often feel proprietary about data sets they may have worked for months or years to acquire, clean up, and analyze. But if those norms changed, it would be harder for data fakers to get away with it or for serious mistakes to pass unnoticed — and our processes for aggregating knowledge could improve significantly.

“If people are unwilling to share their data, that’s the biggest giveaway. There’s a good likelihood there are major, major flaws in the data,” Covid-19 treatments researcher Ed Mills at McMaster University told me.

Sheldrick told me that some researchers declare in their paper that their data is available and they’re happy to share it, and then avoid scrutiny by not actually responding to requests. There’s no real mechanism to complain about a researcher who said they’d share their data but isn’t following through. “We’ve developed this culture where it’s acceptable to publish a study and go, ‘Our data’s available, just contact us,’ and then not respond when contacted,” Sheldrick said.

Another approach may simply be for researchers to internalize what they’ve learned these past couple of years and more closely scrutinize ivermectin studies — and, more broadly, studies that claim spectacular benefits while obscuring some details of their methodology.

Flavio Abdenur, an independent mathematician and data scientist who has been investigating the case for ivermectin through meta-analyses, told me he has radically changed his approach, examining the ivermectin literature by looking only at studies that have been demonstrated not to be fraudulent. He’s essentially withdrawing the normal practice of giving researchers the benefit of the doubt.

That would be an extraordinary shift. But there’s no question that something needs to change. Monitoring for mistakes and fraud can’t only be done by journalists calling hospitals, hospitals doing their own research and contacting journals proactively, and researchers with day jobs looking for fraud on the side.

If fraudulent and faulty research is a systematic, serious problem — and it is — it needs a systematic, serious solution.