Why crude oil prices keep taking us by surprise

Most people know that crude oil prices are hugely consequential for the global economy. Soar too high and they trigger recessions. Plunge too low, too fast and they sow chaos in oil-producing nations. Even medium-sized squiggles can roil markets.

Yet despite their vast importance, few experts seem able to reliably predict where oil prices will go next. Financial markets and policymakers are perpetually surprised by large swings. The ginormous surge in the 2000s — when China's appetite for crude pushed prices from $30 per barrel in 2003 to $140 per barrel in 2008 — stunned plenty of analysts. The steep plunge since 2014, down to around $40 per barrel today, has been similarly unexpected.

So why do oil prices keep astonishing us? That's a question explored in an interesting recent paper in the Journal of Economic Perspectives by Christiane Baumeister and Lutz Kilian. Among other things, they offer this graph, noting that market "expectations" of oil prices are frequently proved wrong:

The expectation of the WTI spot price for a horizon of h months is measured by subtracting from the WTI oil futures price at maturity h the estimated risk premium for that horizon. (Baumeister and Kilian, 2016)

The chart shows prices for West Texas Intermediate (WTI) crude, which is traded in Oklahoma and used as a benchmark for many US oil sales. The blue line shows the "spot price," the price that WTI sells for at any given point in time. The red lines are risk-adjusted expectations of future WTI prices — which Baumeister and Kilian argue is the best proxy for market expectations.

As you can see, there are many cases where the spot price ends up swinging far above or below what traders expected: the Asian financial crisis of 1997, the Icarus-like adventure in 2008, the free fall in 2014. Those are "oil shocks." Those are the (often unpleasant) surprises.

Four reasons oil prices keep surprising us

Baumeister and Kilian argue that economists have made impressive strides of late in explaining past oil shocks, like the 1973 oil crisis or the 2008 spike. But predicting the future remains as intractable as ever, for four key reasons:

1) Demand is just plain tough to forecast. "Most major oil price fluctuations dating back to 1973 are largely explained by shifts in the demand for crude oil," the authors note. But oil demand is frequently a function of changes in economic growth. And GDP is notoriously tricky to predict.

Check out how badly off-track the IMF's forecasts of worldwide economic growth have been since 2011:

(Economic Report of the President)

This holds at a country level, too. Most experts now agree, in hindsight, that the liftoff in global oil prices between 2003 and 2008 was triggered by rapidly expanding oil demand in Asia, as China's industrialization kicked into a higher gear.

Yet at the time, economists were consistently underestimating China's growth. Partly that was because Chinese economic statistics are murky and unreliable. Economists also couldn't agree on the extent to which China's GDP expansion was due to lasting structural changes or unsustainable government stimulus policies. Without a decent grasp on China, forecasting oil prices was infuriatingly difficult.

(Note, too, that demand for oil tends to be inelastic in the short term: That is, when prices go up, people don't cut back on consumption very quickly, because we're all deeply dependent on driving and substitute fuels are hard to come by. So when demand starts outstripping production, as it did in 2008, that can lead to really big swings in prices.)

2) Unexpected technological changes can alter supply. Another thing that makes forecasts so tough: When crude prices go up, people often develop new technologies to either conserve fuel or develop new oil sources — which, in turn, can push prices back down. But this innovation can't always be predicted ahead of time. That's, uh, what makes it innovative.

Example: In the 2000s, when oil prices were rising worldwide, a small company named EOG was tinkering with fracking and horizontal drilling techniques to get at oil trapped underground in Texas shale rock formations. At the time, even most industry experts thought it was impossible to extract this oil — the hydrocarbon molecules were simply deemed too large to slip through the pores in the shale. But EOG persisted and persisted and finally figured it out. That helped lead to the massive shale oil boom in the United States, which later caused prices to plummet starting in 2014.

Then, after prices plunged below $50 per barrel in 2014, many experts figured that this would severely cramp the US shale boom, since it was costly to keep fracking new wells. But shale companies kept fiddling and managed to cut costs and pump oil more efficiently than anyone thought possible. The result? Prices plunged even lower than expected in 2015 and 2016.

3) Most oil forecasts don't take political crises into account. In the past, oil prices have sometimes spiked because of political crises in places like the Middle East — as with the 1979 Iranian Revolution. Baumeister and Kilian show that this isn't necessarily because of any disruption to actual wells. Rather, it's often due to the fact that when there's a crisis, traders will buy additional crude now to stockpile for later — known as a rise in inventories. This, in turn, pushes up prices.

(Shutterstock)

But these swings in sentiment don't unfold in nice, neat, predictable ways. "Perceptions may evolve rapidly, for example, in response to geopolitical or economic crises," the authors note. Making this even more complicated is the fact that, historically, many geopolitical crises haven't led to a surge in prices. It's only under very particular conditions — a crisis plus expectations of tight supply and demand — that we've seen huge inventory buildups.

"Thus," the authors argue, "most oil price predictions simply ignore the possibility of future political or economic crises, except to the extent that they are already priced in at the time the prediction is made. Because crises are rare, this strategy usually works, but occasionally it may result in spectacular predictive failures."

4) It's easy to get seduced by the idea that the past is like the future. In a commentary on the paper, economist Timothy Taylor offers yet another reason oil prices surprise us: "[I]t seems to me that we are often astonished because we tell ourselves a story about the previous change in oil prices, and then we have a hard time shifting our story."

"For example," he writes, "I'm old enough to remember stories from the 1970s about how the iron grip of the OPEC cartel meant that oil prices would never fall again; and stories from the 1980s and 1990s about how deregulation of oil prices had broken the back of OPEC so that oil prices would be low; and stories from the mid-2000s about how the world was approaching 'peak oil' production and oil prices would inevitably stay high."

Why unexpected oil shocks matter

(Tim Boyle/Getty Images)

Up until now we've mainly been talking about financial markets. But consumers and policymakers are also constantly making their own predictions about crude oil prices. And, Baumeister and Kilian show, those predictions are often even less skilled — which can sometimes have serious consequences.

Many policymakers, including the IMF and some governments, often just rely on simple futures markets to "predict" the price of oil — even though, as we've seen, that's not reliable at all. This can cause problems when countries like Saudi Arabia or Russia launch major spending programs on the assumption that oil prices will stay high forever and then suddenly have to cut back sharply after an unexpected crash.

And consumers are even less skilled at predicting. By and large, the economists found, ordinary people tend to assume that oil prices will basically keep doing what they're already doing. If they're low today, they'll be low tomorrow. If they're high today, they'll be high tomorrow. We can see this in car purchases — whenever the price of oil (and hence, gasoline) plummets, Americans run out to buy SUVs and gas guzzlers, essentially assuming that gas will be cheap forever. Then, when oil spikes, the pain begins.

It's not belabored in the paper, but one takeaway here is that we should think about ways to make ourselves more resilient to surprises. For instance, Michael Levi and Varun Sivaram of the Council on Foreign Relations have argued that fuel economy standards for cars and trucks can act as a valuable hedge for drivers against price spikes. Given that oil prices are fairly unpredictable, insurance isn't a bad model here.

It's a pretty safe bet that more surprises are in the pipeline

Baumeister and Kilian's paper offers a fascinating history of past oil price swings, but they don't sound particularly convinced that our powers of prognostication have improved all that much. "Although our understanding of historical oil price fluctuations has greatly improved," they conclude, "oil prices keep surprising economists, policymakers, consumers and financial market participants."

That's probably a good cautionary tale for anyone who thinks they're certain about what oil will do next.

Right now, global oil prices are relatively low, hovering around $44 per barrel. And many people think that will likely last for the foreseeable future, given that China's growth has tapered off and the world is awash in crude supplies. The International Energy Agency recently declared in its "Medium-Term Oil Market Report" that "it is hard to see oil prices recovering significantly in the short term from [current] low levels" — in part because production was so far above demand:

That said, the IEA did acknowledge that surprises were possible. Oil-producing nations have seen a huge fall in investment since 2014, which could lead to a drop-off in output down the road. Further out, there's the prospect that mass adoption of electric cars could completely batter oil demand. (Technology, again!) Or maybe some other surprise will emerge. Perhaps chastened by the past few years, the IEA seemed to acknowledge that forecasts tend to easily go awry.

"Attempting to understand how the oil market will look during the next five years is today a task of enormous complexity," the IEA conceded. "Some certainties that have guided our past outlooks are now not so certain at all."

Go deeper:

• Why crude oil prices keep falling and falling, in one chart
• Within a decade, electric cars could be cheaper than gasoline vehicles. Then, watch out.
• Note that it's not just oil. Agencies like the IEA often have a hard time predicting the development of renewables, too.

Contributions are a key part of the future of Vox

Readers rely on Vox for clear, nuanced coverage that not only illuminates the issues, but poses solutions, too. And we rely on help from our readers: Advertising and grants cover the majority of our costs, but we count on contributions to help us close the gaps in our budget. In fact, we’re looking to reach 95,000 individual contributions before the end of the year. Will you make the next contribution right now? Our average gift is just $20 — and it goes a long way in helping us keep our work free. Vox is here to help everyone understand what’s shaping the world — not just the people who can afford to pay for a subscription. We believe that’s an important part of building a more equal society. Join that mission by making a contribution today.

One-Time Monthly Annual

$5/month

$10/month

$25/month

$50/month

Other

$

Yes, I'll give $5/month

Yes, I'll give $5/month

We accept credit card, Apple Pay, and Google Pay. You can also contribute via