Energy efficiency can be incredibly valuable — but we do need to measure it properly

Earlier this week, I wrote about a new paper by three economists affiliated with the E2e Project suggesting that a federal program to improve the energy efficiency of low-income homes in Michigan wasn't nearly as cost-effective as we thought.

That paper, and the press hype around it, triggered a bunch of sharp critiques from within the energy-efficiency world — see here, here, here, and here — and I wanted to highlight some of the more insightful responses below.

Critiques of one weatherization program don't necessarily apply to all energy-efficiency policies

Just as a refresher, there were three notable aspects of the E2e study: First, it was a randomized controlled trial — the gold standard of evaluations, but something that's rarely done for efficiency efforts. Second, the study found that the actual energy savings from weatherizing Michigan homes was just 39 percent of what models had predicted beforehand. Third, the study argued, the upfront cost of boosting efficiency in Michigan under the federal Weatherization Assistance Program greatly exceeded the estimated monetary benefits that accrued from saving energy.

What made this study interesting, to me, was that it suggested we shouldn't just assume energy efficiency is always the cheapest way to reduce emissions based on modeling that suggests massive potential for eliminating waste at negative cost. As Michael Greenstone, one of the co-authors of the paper, told me, that hypothesis needs to be field-tested, rigorously, to make sure those savings actually occur.

However, a number of readers thought this study (and my post) could be interpreted as an attack on all energy efficiency. So I figured it was worth clarifying this point. It's not! The researchers only scrutinized a single federal program in Michigan. As Greenstone explicitly told me, this hardly undermines the rationale for every single efficiency policy out there, many of which operate very differently. But it does raise some worthwhile questions.

Now, on to some of the specific critiques — which offer important context and help clarify what's valuable about this paper:

1) There's more to low-income weatherization than saving energy

Not just about lower electricity bills. (Jahi Chikwendiu/The Washington Post/Getty Images)

The federal Weatherization Assistance Program helps low-income families upgrade the efficiency of their homes through things like replacing furnaces, improving insulation, and sealing cracks in doors and windows. Through a randomized controlled trial, the E2e study found that this cost the government about $5,000 per household, but only saved households about $2,600 in energy costs over the lifetime of the upgrades.

However! As Rebecca Stanfield of the Natural Resources Defense Council pointed out, these programs aren't intended to be cost-effective solely on the basis of energy savings. They also have a variety of social benefits: helping low-income people stay comfortable through the cold winter, improving people's health by removing mold or asbestos, fixing potential carbon monoxide leaks from heaters, boosting the value of dilapidated housing stock.

Martin Kushler, a senior fellow at the American Council for Energy-Efficient Economy, cites a Department of Energy evaluation pegging these "non-energy benefits" at around $3,466 per household. "Viewed in that comprehensive manner," Kushler writes, "programs like WAP are cost effective from that broader societal perspective — as a public policy, they make sense."

That's fair. Though, as a counterpoint, the Department of Energy often frames the weatherization program as a pure win in terms of energy savings alone. This E2e study raises some questions about that — although, since it only examined efforts in Michigan, it's tough to make generalizations about the program nationwide. Other states might do better.

2) Low-income weatherization isn't representative of all home efficiency programs

Employees at EnerNOC work in the "mission control" room, or network operations center, at their hub, in Boston, Massachusetts, August 12, 2013. (Melanie Stetson Freeman/The Christian Science Monitor via Getty Images)

This follows from the first point above. Because low-income weatherization has so many different social goals, experts have long understood that it's a less cost-effective way of saving electricity than other efficiency programs.

Look at the broader context: The federal government spends about $225 million per year on the Weatherization Assistance Program (plus it got a $5 billion infusion, spread out over multiple years, from the 2009 stimulus bill).

By contrast, US utilities spent $7.2 billion in 2012 alone on a wide variety of ratepayer-funded efficiency programs — with about one-third of that going to residential efficiency (some of that weatherization). When it comes to efficiency, this is where the real money is. And, as many people pointed out, utilities scrutinize these programs much more closely, and they tend to be considerably more cost-effective.

Merrian Borgeson of NRDC points to a 2015 study by Lawrence Berkeley National Laboratory finding that the cost of the average utility-run residential efficiency program was about 3.3 cents per kilowatt-hour — or about half the cost of electricity from a brand-new power plant. By comparison, low-income programs cost about 14 cents per kWh:

(Lawrence Berkeley National Laboratory)

Bottom line: Sure, there are low-income weatherization programs that may not be particularly cost-effective from a pure energy standpoint. That's partly because they have all these broader social goals. But those low-income programs are only a small fraction of what's going on with efficiency around the country.

The bigger story is that electric utilities around the country are finding that it's cheaper to help homes become more efficient — by, for example, offering rebates for more efficient appliances, lighting, heating, and so on — than it is to build new power plants. And that's before we get into any of the benefits from saving CO2 emissions.

3) But yes, we do need better ways to measure energy efficiency

Who wants to talk metrics? (Shutterstock)

Finally, there was this thoughtful response from energy finance consultant Matt Golden. He raised the two big contextual points above, but he also pointed out that the E2e study raised an important issue — namely, that "existing models for estimating efficiency over-predict savings."

As Greenstone told me, policymakers typically rely on models and engineering studies to predict the potential energy savings from various efficiency measures ahead of time. This, in turn, helps guide various programs. But these models are far from perfect. As the E2e study discovered, the Michigan weatherization program was only saving 39 percent as much energy as the models had predicted. That points to a real potential pitfall in designing efficiency programs.

Golden, for his part, agrees that models tend to be over-optimistic about potential savings, and that this is a genuine problem. Yet he also points out that some states are trying to devise solutions here: "One recent study in New York State by Performance System Development shows that calibrating predictions to past bills dramatically improves results. California is taking a different approach with the CalTRACK system, which uses electricity meter data to track actual savings and adjust predictions to match actual performance."

In theory, this should over time lead to a system where utilities, states, and others can identify efficiency opportunities more rigorously — and devote more resources to programs that deliver the most savings. "Rather than rely on top-down programs that pay rebate coupons upfront," Golden writes, "we should establish markets that price savings based on levelized avoided cost of new generation and pay for results as they occur."

Golden's full response is very much worth reading. Energy efficiency really does have the potential to save money and cut CO2 emissions. But we still need to measure it properly. "The important question here is not whether public investment in efficiency is a good value," he says, "but rather, how do we maximize our efficiency returns?" On that, both he and Greenstone would agree.

Further reading:

• Here's my original post on the paper. And here's a full copy of the paper itself (PDF).
• Here are some responses and critiques from Martin Kushler of ACEEE, Rebecca Stanfield, Merrian Borgeson, and Deron Lovaas of NRDC, and Matt Golden. (If I'm missing any, let me know, and I'll add.)
• Here's an excellent, in-depth piece by Stephen Lacey from June on the efficiency industry's struggles with tracking actual savings.

We have a request

Vox's journalism is free because we believe that everyone deserves to understand the world that they live in. That kind of knowledge helps create better citizens, neighbors, friends, parents, consumers and stewards of this planet. In short, understanding benefits everyone. You can join in on this mission by making a financial gift to Vox today. Reader support helps keep our work free, for everyone. Will you join us?

One-Time Monthly Annual

$95/year

$120/year

$250/year

$350/year

Other

$

Yes, I'll give $120/year

Yes, I'll give $120/year

We accept credit card, Apple Pay, and Google Pay. You can also contribute via