:format(webp)/cdn.vox-cdn.com/uploads/chorus_image/image/53061125/631235572.0.jpg)
Yesterday, Feb. 1, the California Department of Motor Vehicles published its annual performance report on the 11 companies testing self-driving cars on public roads in the U.S. state.
Because it’s only the second time the DMV has published this report, it’s the first time we can compare how most of those companies’ self-driving technology has improved.
The big news is that Waymo’s self-driving cars — previously known as Google’s self-driving project — navigated public roads without human intervention four times better than it did last year. But it’s a unique opportunity to run through each company’s progress.
The report includes how many miles each company’s cars have driven and how many times a human has had to take back control — an event that’s called a “disengagement.”
One caveat: As each company is at a different stage in its self-driving development and has performed different levels of testing on public roads, it’s a little tricky to compare their stats apples to apples. (Companies don’t have to disclose how many miles they’ve driven — or how many times they’ve had to take back control — on private roads or test tracks.)
That said, here are some of the major takeaways.
Waymo
Waymo — an Alphabet subsidiary — drove the most miles and was able to bring down its disengagement rate from 0.8 per 1,000 miles in 2015 to 0.2 in 2016. Waymo’s cars drove more than 635,000 miles just in California last year, and only had to have a human take back control 124 times.
Almost half of those disengagements — 51 — were because of a “software discrepancy,” which can mean a lot of things but generally that the brain of the car isn’t doing exactly what it’s supposed to do.
Other reasons for disengagement include “weather conditions during testing,” “recklessly behaving road user,” “unwanted maneuver of the vehicle,” “perception discrepancy,” “incorrect behavior prediction of other traffic participants,” “construction zone during testing,” “emergency vehicle during testing” and “debris in the roadway.”
Waymo’s cars only had to disengage twice because of construction, emergency vehicles or debris in the roadway and once for weather conditions.
This is a substantial improvement for the company, which drove 424,000 miles in 2015 and had to disengage the system 341 times that year.
It suggests Waymo’s technology is able to drive on its own in more situations on public roads without having to rely on human control. That’s a big deal, as the company — which just graduated from Alphabet’s moonshot factory — prepares to bring its technology to market.
Tesla
While Tesla already offers a semi-self-driving mode called Autopilot, it’s also testing a more full-featured system.
But among companies that did any testing on public roads, Tesla reported the lowest number of miles — 550, or more than 99 percent fewer than Waymo reported — and a relatively high number of disengagements, 182.
That’s because Tesla mostly doesn’t test its self-driving cars on public roads, so it doesn’t have to report how many miles it drives or how many times it disengaged. Really, the only time Tesla tested its self-driving cars on public roads was in October and November, when it was shooting marketing videos — embedded below — and frequently had to stop the cars during the shoot.
So it’s not really useful to dig deep into Tesla’s results, or compare its cars’ performance to other brands.
Also worth noting: The 300 million miles that Tesla’s Autopilot users have driven don’t count toward these stats. And in 2015, Tesla reported 0 miles and 0 disengagements to California’s DMV.
Cruise
The second-highest number of miles driven on California’s public roads was by Cruise, the self-driving startup now owned by General Motors, at 9,776. By our calculations, the company had to disengage its system 181 times. That’s a rate of 18.5 disengagements per 1,000 miles.
The vast majority of those disengagements were attributed to the driver taking back control to “avoid unexpected behavior.” The company does indicate that the system was able to return control to the driver within one second after alerting the driver the system had failed.
Cruise’s year-over-year data is also not as valuable because it only received its DMV permit in June 2015.
BMW
BMW drove fewer miles than some of the other major companies and only had its vehicles on public roads in March and April of 2016.
But significantly, in the 638 miles it drove, the company only disengaged its system once. According to the report, it was because the driver manually disengaged the system because the lane markings weren’t clear enough. If that rate holds up over more miles, it’s pretty good.
The company did not have a California permit to operate autonomous vehicles in 2015, so there’s no year-over-year change to look at.
Mercedes Benz
Mercedes drove a total of 673 miles between February and November 2016. However, the company disengaged its system 336 times — almost once per every two miles driven, or a rate of 499 times per 1,000 miles. Of those disengagements, drivers manually took back control of the system 183 times and the system returned control automatically 153 times.
Worth noting:
First, more than 30 of those miles were driven in the rain, and the system did not have to return control to the driver because of the rain.
Second, there were a number of occasions that drivers manually took back control because they were “feeling uncomfortable.”
Lastly, all of its miles were on urban streets and not on highways. City streets are much more difficult to navigate for autonomous systems (and humans) than highway roads, because the cars have to navigate around other cars, understand traffic signals and watch for pedestrians or other unexpected interferences. So it makes sense to have a higher disengagement rate than if it were testing entirely on highways.
It’s also a modest improvement over 2015, when Mercedes reported 1,057 disengagements over 1,739 miles, a rate of 608 disengagements per 1,000 miles.
Ford
Ford, which only has two cars on the road in California, drove a total of 590 miles, just in March. The company only disengaged three times, for a rate of 5.1 per 1,000 miles.
Two of those three were because of a “lane change maneuver aborted due to vehicle overtaking at high speed.” The third was a “loss of communication between autonomous vehicle control software and test engineer GUI.” In that case, the driver “took over to stop and reestablish connection.”
The company was not among those that reported its miles in 2015, so it’s hard to tell if this is an improvement. Ford is also testing its vehicles in Michigan and Arizona on its own private test tracks.
Nissan
Nissan is another carmaker that saw an improvement in its autonomous system in 2016.
The company drove a total of 4,099 miles and only took back control — either manually or automatically — 28 times. That’s a rate of 6.8 disengagements per 1,000 miles — an improvement from its 2015 rate of 71 per 1,000 miles.
Of those 28 disengagements, nine were attributed to the autonomous system’s failing. In one case, the software module froze. In another, the engine control unit refused a software command while crossing a railroad. Nine other times, the driver had to take back control because the system was about to hit another car or another obstacle. In one case, the car “overshot the left turn,” and in another, the car “made two sudden changes of direction.”
So while there are relatively few moments where a driver had to take back control, it’s clear there are still situations where the system cannot be fully relied on just yet.
The carmaker drove its five vehicles on California’s public roads every month of 2016, unlike some of its competitors.
Delphi
Delphi, the auto parts supplier, is an interesting case of a company performing worse than it did the year prior.
In 2016, Delphi — which is testing two Audis on both highways and surface streets — drove 3,125 miles and disengaged 178 times. That’s 57 times the company had to take back control per 1,000 miles. But in 2015, its ratio was 24 disengagements per 1,000 miles — over a larger range of nearly 17,000 miles.
The causes varied, but the vast majority of times a driver took over it was because they needed to manually change lanes during heavy traffic. For example, the driver took control 27 times as a precautionary measure when there was heavy pedestrian traffic. And 18 times, the system returned control to the driver as a reaction to unexpected behavior from another driver.
That said, the company had fewer issues detecting traffic lights and navigating poor lane markings in 2016.
Bosch
Bosch also fared worse in 2016 than it did in 2015. In 2015, the company drove 935 miles and disengaged the system 625 times. In 2016, the company drove 983 miles and disengaged more than 1,440 times. The report isn’t too detailed so it’s hard to discern what the causes of those disengagements were.
Volkswagen and Honda
Neither of these carmakers tested on public roads in California in 2016. In 2015, Volkswagen’s two cars drove 14,945 miles and disengaged 260 times, for a rate of 17 disengagements per 1,000 miles.
This article originally appeared on Recode.net.
