Why it's about to get a lot harder to hide the results of medical studies

This is a very scary fact about modern medicine: if researchers don't like the results they get from clinical trials, they can simply hide them — and none of us, the people who use the various drugs and devices under testing, will ever know.

Results that might show a drug doesn't work or has serious side effects can, under our current clinical trials system, never see the light of day. Studies have shown that we have a skewed picture of the evidence that underlies much of clinical practice because there's no centralized system for disseminating information about clinical trials results, whether positive, negative, or inconclusive.

"Imagine your 10-year-old daughter was in the town’s annual gymnastics competition," says Deborah Zarin, director of ClinicalTrials.gov. "She has been practicing for months. In the finals, she was up against the daughter of the judge. At the end of the competition, the judge goes into a back office, comes out and declares his daughter the winner. There is no transparency about what the criteria were, and how the decision was made."

For over a decade, Dr. Zarin, a Harvard-trained MD, has been trying to change that. She has earned a reputation as a crusader for open data, quietly presiding over the world's largest database of clinical trials, ClinicalTrials.gov. Established in 2000, and operated by the NIH, it now holds information from more than 180,000 studies in humans in over 180 countries.

The idea underlying ClinicalTrials.gov's creation was a public record about which trials are going on as soon as they are started. That should, in theory, make it more difficult for researchers to hide trials that didn't produce the results they wanted. The database also makes it easier for patients to know about studies being done, should they require access to an experimental drug, for example.

But there was still a problem: ClinicalTrials.gov, up until now, hasn't required researchers to report the outcomes of their trials — only the fact they're happening. Under a new plan, proposed by Health and Human Services last month, researchers who run clinical trials would be made to not only register them on the database within three weeks of signing up the first study participant, but also report a summary of results — no matter the outcome. This will vastly expand the trove of data on the site.

I spoke to Dr. Zarin earlier this week about why these changes are happening, why they matter, and what it means for medicine.

JB: Your registry has done more than just provide information to patients about what trials are being conducted; it has advanced the clinical trials transparency agenda globally. What does this mean for how medicine is practiced in this country?

DZ: When doctors make decisions about a possible treatment, you want them to base that decision on the best-available evidence. The only way to know whether a drug would help you is to rely on the published studies. But several years back, researchers started documenting that — for many drugs and other interventions — only some of the studies conducted were published.

Number of studies registered to ClinicalTrials.gov over time

(Chart via ClinicalTrials.gov.)

In one famous case, it was shown that for the antidepressant Paxil, there were a number of studies done in children, but only one or two of those studies were published. [Because no one knew about the studies and the adverse effects in children, including suicide, doctors prescribed the drugs to children anyway.] This is called publication bias, and in some instances, the published studies make the drugs look good. Doctors and patients can't make a rational decision unless you see the results of all studies. So another purpose of a trial registry is to make a very clear public record. Studies are supposed to be registered on the database at ClinicalTrials.gov so we know they happened.

JB: How pervasive is selective reporting of studies?

DZ: People used to think publication bias — or incomplete publication — was a problem for industry, something drug and device companies did. My colleagues and I published an article a few years ago that looked at a set of NIH-funded trials to see how many of these were published. We found that four or five years after study completion, only two-thirds of the studies were ever published. That was part of the motivation for the proposed NIH policy, which is designed to ensure that results would be reported at ClinicalTrials.gov for all NIH funded clinical trials.

JB: Why is capturing the results of trials the next important step?

DZ: The goal of results reporting at ClinicalTrials.gov is to just get the facts out in the public domain without interpretation, so at a minimum, people could see the basic facts about clinical studies. Right now we have over 15,000 summary results entries. The proposed policies would increase the number of trials for which summary results are required to be reported to ClinicalTrials.gov.

Number of registered studies with posted results at ClinicalTrials.gov

JB: Critics have pointed out that, even with these revisions, there would still be gaps in the registration of trials and reporting of results, that we still won't be capturing all data from all trials.

The proposed federal rule covers controlled clinical trials that involve something that the FDA regulates as a drug, biologic or device, but excludes phase one drug trials [the very first step of a clinical trial that looks at the safety of drugs in humans] and the equivalent for devices, which are called early feasibility device studies. The recently announced NIH policy would cover all NIH-funded trials, including those not covered by the federal rule, such as NIH-funded phase one drug trials and early feasibility device studies. So, the remaining gap would include three key categories of trials not funded by the NIH : phase one drug trials, early feasibility device studies, and studies that do not involve something that the FDA regulates as a drug, biologic, or device (for example, surgical or behavioral studies.)

JB: So there are still a lot of holes in what would be a complete picture of all trials data — though we're getting closer to closing them?

Most people think that for every medical study in human beings you need ethics approval and an informed consent process that is covered by federal regulations. But there are gaps there as well. There are trials that fall outside FDA and the Office for Human Research Protections regulations. There are attempts to close those gaps. That’s the beginning.

But the bigger question is: does and should anybody at the federal level regulate human subjects protection for all the research in the US. There are various legal frameworks, but I’m not an expert on the legal options.  Key  players are the regulatory agencies, the funders, and the institutions where the research occurs. Those would be the relevant bodies who could, in theory,  change things. There are state regulations, and federal regulations. It’s a real mixture.

Percentage of registered studies by location

(Chart via ClinicalTrials.gov.)

JB: You mention the role regulators play in ensuring more transparency. Many have pointed the finger at them: that they don't mandate more stringent requirements on the oversight of clinical trials and data sharing, that they've dropped the ball here.

DZ: We live in a system of laws, and regulations implement the laws. You can’t promulgate regulations that extend outside of the bounds of the laws that they are implementing. Sometimes, with some of the problems people are talking about, there’s no law that says that has to happen. For example, the law covering registration and results reporting excludes phase one studies. You could say, 'Why isn’t FDA insisting they be registered?' But the law (FDAAA) specifically excludes them,  so the FDA may not  have the right to ask for that. The concept is that, in many cases, the power to regulate comes within a law, and you have stay within those parameters.

You can't say to someone who has a medical need, 'hold on we’ll do more clinical trials and get back to you in 2 years'

JB: It sounds like that power to change the system would lie with congress then. So what else is getting between us in the current reality and a future where all trials are registered and results are transparent?

DZ: Most of the biggest companies have processes for reporting the results of their studies. They have a lot of regulatory reporting they need to do. They have offices to deal with ClinicalTrials.gov reporting. They have people to ensure this happens. But many academic medical centers are more decentralized. Some have offices and people involved in helping investigators get the reporting done. Some have oversight. Others don't.

JB: Knowing how flawed the research enterprise is, how should patients think about the evidence base for medicine?

DZ: Clinical decisions are driven by the current reality. You can't say to someone who has a medical need right then and there, 'hold on we’ll do more clinical trials and get back to you in two years.' You have to make decisions based on the best information available.

Second, we don't want the perfect to be the enemy of the good. We have to keep working to improve the system which is what all these efforts are about. In the meantime, you have to make decisions based on the data. For people out there, I would say you always want to be able to have a conversation with your clinician and find out what the evidence is they are basing their recommendation on, what the options are, so you understand what the situation is. That’s the best you can do.

JB: What's the next battle in the transparency crusade?

DZ: The clinical research community has passed the tipping point where trial registration has become part of the cultural norm. Some trials are not registered but they are the exception. That’s a huge change in the last ten to 15 years. We haven't reached that point with results reporting; it hasn't reached a cultural tipping point. It's not part of the workflow of research, the scientific process. Integrating results reporting into the normal workflow would be a very important next step.

We're here to shed some clarity

One of our core beliefs here at Vox is that everyone needs and deserves access to the information that helps them understand the world, regardless of whether they can pay for a subscription. With the 2024 election on the horizon, more people are turning to us for clear and balanced explanations of the issues and policies at stake. We’re so grateful that we’re on track to hit 85,000 contributions to the Vox Contributions program before the end of the year, which in turn helps us keep this work free. We need to add 2,500 contributions this month to hit that goal. Will you make a contribution today to help us hit this goal and support our policy coverage? Any amount helps.

One-Time Monthly Annual

$5/month

$10/month

$25/month

$50/month

Other

$

Yes, I'll give $5/month

Yes, I'll give $5/month

We accept credit card, Apple Pay, and Google Pay. You can also contribute via