The problem with diet books written by doctors

One of the perks of being a journalist is that new books cross your desk weeks or months before they’re released.

One of the pitfalls of being a health journalist is that far too many of these books promise to eliminate belly fat forever with their one true secret for weight loss.

Diet books are a multimillion-dollar industry, and it's no surprise, since millions of people struggle with their weight and long for answers about what they can do to slim down. Books can provide valuable tips on healthful patterns of eating. Some are more outlandish than others. But the problem with all of them is what they promise when it comes to weight loss.

No doctor has ever uncovered the solution to weight loss. If someone had found the fix for this immensely vexing and complex problem, we wouldn't be facing an obesity crisis.

But unfortunately, more and more respected doctors, despite their good intentions, are complicit with the publishing industry in confusing science and obscuring hard truths about obesity to sell diet books. It's one thing when actress Gwyneth Paltrow tells people to avoid "nightshade vegetables" on an elimination diet, and quite another when a highly trained and credentialed physician makes overhyped weight loss claims.

Case study: Dr. David Ludwig's Always Hungry

Recently, a book called Always Hungry?: Conquer Cravings, Retrain Your Fat Cells, and Lose Weight Permanently landed on my desk. Like dozens of books I've seen before, it makes big promises.

But this book wasn't authored by a woo-loving celebrity like Paltrow. The book came from Dr. David Ludwig, an esteemed endocrinologist and researcher affiliated with Harvard Medical School who has run dozens of clinical trials and seen thousands of patients in the 20 years he's practiced medicine.

Unlike many other diet book authors, Ludwig offers a strong scientific theory for his diet: a model of obesity he's developed based on his and others' research. The "carbohydrate-insulin hypothesis," which journalist Gary Taubes and others have also extensively promoted, contends that a diet heavy in carbohydrates (especially refined grains and sugars) can lead to obesity because they increase insulin in the body, and cause the body to hold on to fat.

The classic approach to dieting — in which calories in general are restricted — fails, Ludwig argues, because it doesn't account for the effects calorie restriction and refined carbohydrates have on the body. As an alternative, Ludwig says dieters should forget about calories and focus on reducing carbohydrates and increasing fat intake to feel more satiated and lose more fat in the long run. His discussion of the science here is lucid and interesting.

But what readers may miss in the way this theory is presented in the book is that it's still very much a work in progress, and the subject of heated debate among obesity scientists. Not everyone agrees that this approach to weight loss is superior, and there's high-quality evidence that contradicts it.

The real problem with diet books is what comes next: the diet itself. Publishers are not interested in a book simply running through some interesting, if preliminary, theory on how to eat. They want a very specific plan. They want bold promises about results. And these plans, though sometimes based on promising science, are not rigorously tested among dieters. Worse, in proposing a singular plan, they suggest there is One Right Way for people to eat, which isn't the case. The best diet, in truth, is the one you, personally, can stick to.

In this case, the closest thing Ludwig has to a test of his diet is a pilot study that followed 237 people, the results of which he references heavily throughout the book. "Most people in the pilot initially lost 1 to 2 pounds a week, a few lost even more, some a bit less," he writes. "In addition to weight loss, participants consistently reported other benefits that predict long-term success, including: decreased hunger; longer-lasting satiety after eating; great satisfaction with food."

In a subsequent email, Ludwig said, "The pilot test was never intended as proof. The higher-fat diet I recommend is based on dozens of studies from my research team and hundreds of studies by others. No diet ever recommended, including by the government, has been proven by a definitive randomized controlled trial. Consider, for example, American Heart Association No-Fad Diet: A Personal Plan for Healthy Weight Loss. That title doesn't make clear that not everyone will lose weight or that the weight loss might be temporary. Further, no long-term trial proved that calorie restriction, the recommended approach, was healthy."

Indeed, the book clearly states that the pilot is not scientific and was instead designed to generate feedback on the diet. And yet, Ludwig heavily draws on the weight loss "success stories" from pilot participants throughout the book to make very specific claims about the transformation his diet can confer to anyone who tries it: "I'm confident that Always Hungry? will help you achieve lasting weight loss, experience increased vitality, and enjoy a healthy life," he writes.

I asked Ludwig why he published the book before he had tested the diet with a higher quality, longer-term study. Ludwig's pilot study ran for only 16 weeks, and as every obesity researcher knows, many people following a new diet lose weight initially; the big question is what happens afterward. The data we have on this is sobering. Only a tiny percentage of people who try to lose weight on a diet succeed, and many more actually gain weight in the dieting process.

"Science is always evolving," Ludwig told me. "But in public health you can't wait for the final answer before deciding how to proceed."

In an email, he added, "The relevant question isn't whether there's proof, but rather how the evidence for alternative approaches compares to conventional recommendations. For 40 years, the public has been told that the best way to lose weight is to cut back on calories and fat. But this advice has failed miserably in practice. Is it fair to demand that new approaches be subject to far higher standards of evidence than has ever applied to existing recommendations?"

A recent review of the research on different types of diets in the Lancet (by Ludwig and others) finds low-carb diets outperform low-fat diets. But as a related commentary points out, the difference in weight loss among groups of dieters is tiny: "Participants prescribed low-carbohydrate diets lost only about 1 kg of additional weight after 1 year compared with those advised to consume low-fat diets."

Diet books have a formula, and doctors use it all the time

Moonkin/Shutterstock

When you read a lot of diet books, said Louise Foxcroft, author of Calories and Corsets: A History of Dieting Over 2,000 Years, a clear pattern emerges: "You need to be a doctor. You need to be patronizing. You want a four-phase plan."

The typical book promises to reveal a secret about fat-busting that no one has been telling you. It then guarantees that with an easy-to-follow and painless plan, the fat will finally melt right off.

Often, we hear a story of the author's personal struggle and transformation. "Out of that darkness comes light, the Eureka Moment, when the author explains how he stumbled on the radical truth that inspired his diet," as Malcolm Gladwell writes.

Celebrity endorsements are prominent, as are anecdotes from average people who have allegedly had success following this great new program.

In evidence-based medicine, though, anecdotes are considered the lowest form of evidence, since they may be cherry-picked or otherwise unrepresentative of a broader experience. In the world of diet books, they are presented as definitive proof.

"Wrap that all up in punitive, quasi-religious language," Foxcroft said, "and you'd be rich very quickly."

Indeed, if you can come up with a diet that's appealing enough, these books seem like viable get-rich-quick-schemes. According to Nielsen BookScan, about 5 million diet books are sold in the US alone every year — around half of the entire total health and fitness category in 2015. Gardening books, by contrast, sold about a million units in 2015.

It's just one segment of the dieting industry, which is valued at $60 billion in the US, equal to the pet and cosmetics industries.

We've been suckers for diet books for centuries

People profiting off our weight woes is nothing new. As early as the 18th century, Foxcroft said, dieting was becoming a commercial enterprise. The public was already interested in celebrity diets, and doctors saw the potential to trade on people's desire to be thin.

"Doctors gave diets the authority of science, and people gave their diets more validity, more credence," she explained. And if diet gurus weren't doctors, like 1920s Hollywood nutritionist to the stars Dr. Gayelord Hauser, they'd borrow the moniker anyway.

More than a century later, they're still doing it. In 2015, the best-selling diet book was Dr. Phil's 20/20 Diet. Note that he is not a doctor or even a licensed psychologist.

The formula remains largely the same, too. "Nobody ever comes up with anything new," Foxcroft adds. "They just redress what’s gone on before and package it slightly differently."

No one has the magic bullet for obesity, and there probably isn't one

More recently, doctors have marketed diets through books with great monetary success. There's the Dr. Atkins low-carb monopoly, Dr. Sears's Zone diet, Dr. Davis's "Wheat Belly," Dr. Perlmutter's "Grain Brain," and Dr. Dukan's 5:2.

Diet book critic and author of The Gluten Lie Alan Jay Levinovitz argues that these books contribute to scientific illiteracy, obscuring simple truths about how to live a healthy lifestyle with advice about superfoods and complicated recipes. "They push through theories, hypotheses, plans that just haven’t passed scientific muster," he added.

These diets have been the subject of thorough debunkings, but not all the science in diet books is wrong.

Some doctor's diet books are more sensible than others, like Yoni Freedhoff's The Diet Fix. He doesn't prescribe a specific regimen but argues that the only diet that's likely to work is actually more of a lifestyle change that's sustainable over many, many years. (That's what nearly every weight loss and obesity expert I've ever talked to has told me.)

Ludwig's tome includes many reasonable recommendations. He contends that if people would just forget calories and follow a wholesome, low-carbohydrate, higher-fat diet, they could eventually shed weight. (Even Ludwig's critics agree this plan would eventually lead to slimming.)

But here's the thing: Most people know they shouldn't eat a lot of doughnuts and cookies. They know they should eat more fruits and veggies instead. But many can't stick to that pattern of eating for a host of social and environmental reasons that most diet books can't and don't address.

"All these books are always marketed as, 'Here is the answer. We have now discovered the answer for obesity, and it’s this thing,'" said health policy researcher Tim Caulfield, who has studied celebrity diets for his books, including The Cure for Everything. "But that’s problematic, given what we know about how complex the obesity problem is. There are so many factors involved, and I don't think any researcher would deny obesity is a biological and social phenomenon."

Simply giving people a prescription for eating, which they know they probably should be following anyway, no matter how sensible, isn't likely to change that.

Yet it's clear unscientific diet books aren't going away; they are a hugely lucrative enterprise. Nonfiction is a bigger book category than fiction, and lifestyle makes up about 80 percent of the nonfiction market. Diet books are part of that, and whether they are scientific or not isn't really a concern of publishers and agents.

As one particularly cynical publisher told me, he looks for the following when considering a diet book pitch: "Is she a celebrity, is it trendy, is it new, will he get on Dr. Oz, has he written a New York Times best-seller, has she helped a celebrity lose weight, and (lastly) does he seem adequately credentialed for this. But that last one isn’t super-important: see Gwyneth Paltrow and The Bulletproof Diet."

Vote with your dollars: Stop buying diet books

Given the potential gains here, and the entrenched traditions, doctors are unlikely to stop writing these books. And given how the medical profession works, it's unlikely regulation will get in the way, despite the questionable ethics of diet books. (Only a doctor's patient-physician interactions, and not his media speech, are governed by professional boards, which is why Dr. Oz has gotten away with his many outlandish claims over the past 10 years.)

We humans are particularly vulnerable to diet books. As Matt Fitzgerald, author of the book Diet Cults, explained to me, our beliefs about food are highly irrational, and when we struggle with weight, we long for neat solutions. "What people want is a pill," he said. "But if you can’t have that, you want a diet that’s a functional equivalent of a pill: simple, tidy, neat, certain."

Consumers need to be aware of this vulnerability. We need to think a little bit harder about what we're participating in before buying into the diet book industrial complex. We need to think a little more about what's really getting between us and a healthy lifestyle in the long term, instead of seeking out quick and unsustainable fixes. There's probably a lot more going on there than whether we've consumed enough coconut water or too much gluten.

Before crashing on an extreme diet, maybe we consider incorporating one or two of the very basics of a healthy lifestyle — more fruits and vegetables, going on walks — which fewer than 3 percent of Americans manage today. But even that's too simple: We must think about pushing policymakers to redesign our environments and social programs in ways that fight against rather than promote obesity — something the research evidence increasingly suggests might actually help.

"It’s remarkable people aren’t more skeptical, because these diets never pan out," said Caulfield. "Can you point to one that over the long term has panned out? The answer is no."

Let's not repeat history. Let's do things differently. Down with diet books.

Update, April 27: After an email exchange with Dr. David Ludwig, we have made significant changes to the section about his book to more accurately and fairly reflect his research program, his theory of weight loss, and the debates around it. We also amended the previous title of the story, "Diet books are full of lies. But they're even worse when doctors write them," to more accurately and fairly characterize the issues at stake.