How to avoid getting duped by bad research, in 5 case studies

If you want to know for sure whether something works, you have to design a good test. Yet every day, many studies are published that are so poorly designed that you shouldn't even wipe your table with them, let alone use them to inform personal health or policy decisions.

Still, that doesn't stop the media and policymakers from writing about and acting on shoddy research. This problem inspired a health researcher, physician, and journalist to team up to write a new guide on "study design for the perplexed." Published by the Centers for Disease Control and Prevention, the paper outlines five examples of the most common study biases in order to leave readers with a sense of how to avoid being duped by bad studies.

Some key problems they looked at include:

Healthy user bias: when studies compare healthier users of a medical intervention with less-healthy people who didn't get the intervention. This can make medical treatments seem more effective than they really are. For example, people who seek out and receive a flu shot might be healthier overall than those who don't. If you compare the two groups in a study, "healthy user bias" could give the false picture that it's the vaccine that made the difference. As they explain, "Healthy user bias is a type of selection bias that occurs when investigators fail to account for the fact that individuals who are more health conscious and actively seek treatment are generally destined to be healthier than those who do not."

Confounding by indication: This is another way the true cause of an outcome can get hidden. This type of bias can "occur because physicians choose to preferentially treat or avoid patients who are sicker, older, or have had an illness longer," the authors write. "In these scenarios, it is the trait (eg, dementia) that causes the adverse event (eg, a hip fracture), not the treatment itself (eg, benzodiazepine sedatives)." In other words, it wasn't necessarily the drugs that caused the hip fractures; it was that doctors tended to prescribe the drugs to populations (such as older folks) who are more likely to suffer hip fractures, regardless of what they are taking.

To learn how to spot three other classic biases, read the full guide here.

Will you support Vox’s explanatory journalism?

Most news outlets make their money through advertising or subscriptions. But when it comes to what we’re trying to do at Vox, there are a couple of big issues with relying on ads and subscriptions to keep the lights on:

First, advertising dollars go up and down with the economy. We often only know a few months out what our advertising revenue will be, which makes it hard to plan ahead.

Second, we’re not in the subscriptions business. Vox is here to help everyone understand the complex issues shaping the world — not just the people who can afford to pay for a subscription. We believe that’s an important part of building a more equal society. And we can’t do that if we have a paywall.

So even though advertising is still our biggest source of revenue, we also seek grants and reader support. (And no matter how our work is funded, we have strict guidelines on editorial independence.)

If you also believe that everyone deserves access to trusted high-quality information, will you make a gift to Vox today? Any amount helps.

One-Time Monthly Annual

$5/month

$10/month

$25/month

$50/month

Other

$

Yes, I'll give $5/month

Yes, I'll give $5/month

We accept credit card, Apple Pay, and Google Pay. You can also contribute via