clock menu more-arrow no yes mobile

Filed under:

A study finding a huge increase in alcoholism may have been seriously flawed

According to several researchers, the survey data behind the study is seriously faulty.

A beer. Justin Sullivan/Getty Images

Last week, multiple media outlets, including the Washington Post, Bloomberg, and, yes, Vox, reported on a study in JAMA Psychiatry that had an alarming finding: The rate of alcohol use disorder (alcoholism) in the US increased by more than 49 percent from 2001-’02 and 2012-’13.

Now some researchers are pushing back. They argue that the data used in the study is based on a federal survey that underwent major methodological changes between 2001-’02 and 2012-’13 — meaning the increase in alcoholism rates could be entirely explained just by differences in how the survey was carried out between the two time periods. And they point out that the study’s conclusions are sharply contradicted by another major federal survey.

The JAMA Psychiatry study was funded in part with taxpayer dollars from the National Institute on Alcohol Abuse and Alcoholism, the National Institute on Drug Abuse, and the National Institutes of Health. The pushback, though, suggests that these federal agencies funded some faulty research.

The study’s findings are contradicted by a major federal survey

The JAMA Psychiatry study used data from the National Epidemiologic Survey on Alcohol and Related Conditions (NESARC). It found a sharp increase not just in alcoholism, but also in 12-month alcohol use and high-risk drinking, drawing from the 2001-’02 and 2012-’13 waves of NESARC.

But this sharply contradicts the findings of another federal survey, the National Survey on Drug Use and Health (NSDUH). That survey has actually found a decrease in alcohol use disorder from 2002 to 2013: In 2002, the percent of Americans 12 and older who qualified as having alcohol use disorder was 7.7 percent. In 2013, that dropped to 6.6 percent.

One key difference is the NESARC used data of people 18 years and older, while NSDUH used data of people 12 years and older. But even if you isolate older groups in NSDUH, the rates of alcoholism still dropped or remained relatively flat — certainly not the big rise the NESARC reported.

Now, the NSDUH isn’t perfect. For one, it surveys households — so it misses imprisoned and homeless populations, which are fairly big segments of the population and likely to have higher rates of drug use. But NESARC also shares these limitations, so it doesn’t explain the difference seen in the surveys. Still, it’s safe to say that none of our data on drug use and addiction in America is particularly great.

The study based its findings on a survey that can’t reliably measure trends

I asked Richard Grucza at Washington University School of Medicine, an alcohol researcher who previously studied NESARC and NSDUH’s methodologies, about this discrepancy. He told me that, in general, NSDUH is more reliable for measuring trends.

“The NSDUH methods are much more consistent from year-to-year, and it is administered annually,” he told me. “So I tend to put more weight on NSDUH data.”

The problem with NESARC, Grucza said, is it underwent major methodological changes between the 2001-’02 and 2012-’13 waves. Here are a few, originally noted by the Substance Abuse and Mental Health Services Administration (SAMHSA), which conducts NSDUH, and verified by the National Institute on Alcohol Abuse and Alcoholism (NIAAA), which conducted NESARC:

  • The NESARC changed some questions from wave to wave, which could lead survey takers to respond differently.
  • In the 2001-’02 wave, NESARC respondents were not given monetary rewards. In the 2012-’13 wave, they were. That could have incentivized different people to respond.
  • No biological samples were collected in the first wave, while saliva samples were collected in the second. What’s more, respondents were notified of this at the start of the survey — which could have led them to respond differently, since they knew they’d be tested for their drug use.
  • Census Bureau workers were used for the 2001-’02 survey, but private workers were used for the 2012-’13 survey. That could lead to big differences: As Grucza told me, “Some researchers speculate that using government employees might suppress reporting of socially undesirable behaviors.”

There is plenty of research out there that suggests even one of these tweaks could have significant impacts on survey responses. But to have all of these at once makes the two waves of NESARC really difficult, if not impossible, to seriously compare.

Researchers from SAMHSA told me that they would caution against trying to use the different waves of NESARC to gauge trends.

“Given these points, we would strongly caution against using two points in time as an indicator in trend, especially when the data for these two points in time were collected using very different methods and do not appear to be comparable,” SAMHSA researchers wrote in an email. “We would encourage the consideration of data from multiple sources and more than two time points, in order to paint a more complete and accurate portrayal of substance use and substance use disorder in the nation.”

In short, it looks like the JAMA Psychiatry study was based on some fairly faulty data.

When I asked about these problems surrounding the study, lead author Bridget Grant, with NIAAA, shot back by email: “There were no changes in NESARC methodology between waves and NSDUH folks know nothing about the NESARC. Please do not contact me again as I don’t know NSDUH methodology and would not be so presumptuous to believe I did.”

But based on SAMHSA’s and Grucza’s separate reviews of NESARC, its methodology did change.

When I pressed on this, Grant again responded, “Please do NOT contact me again.”

After this article was published, Grant confirmed NESARC went through some methodological changes between 2001-’02 and 2012-’13. But she argued that there’s no evidence such changes would have a significant impact on the results.

America does seem to have an alcohol problem

None of that means America doesn’t have an alcohol problem. Between 2001 and 2015, the number of alcohol-induced deaths (those that involve direct health complications from alcohol, like liver cirrhosis) rose from about 20,000 to more than 33,000. Before the latest increases, an analysis of data from 2006 to 2010 by the Centers for Disease Control and Prevention (CDC) already estimated that excessive drinking is linked to 88,000 deaths a year — more than all drug overdose deaths combined.

And another study found that rates of heavy drinking and binge drinking increased in most US counties from 2005 to 2012, even as the percentage of people who drink any alcohol has remained relatively flat.

But for now, it’s hard to say if a massive increase in alcohol use disorder is behind the negative trends — because the evidence for that just isn’t reliable.

Update: Added new comments from the lead author of the study.

Sign up for the newsletter Today, Explained

Understand the world with a daily explainer plus the most compelling stories of the day.