Why testing self-driving cars on the challenging roads of San Francisco is necessary

A version of this essay originally appeared on Medium.

Anyone who has visited San Francisco knows that driving here is kind of ridiculous. Our Cruise autonomous vehicles encounter challenging (and often absurd) situations up to 46 times more often than other places self-driving cars are tested. Perhaps for this reason, nobody else is regularly testing self-driving cars in SF. And while we’re generally drawn to tough problems, we test in SF only because we have to. We believe it’s the fastest path toward deploying self-driving cars at scale.

Our path to scale starts with fleet deployments in the most dense urban environments, where high utilization rates generate sustainable unit economics even at low initial vehicle volumes. As we ramp up volume, we’ll drive down costs and gradually roll out our technology to suburban and rural areas where ride-sharing is less common today. It takes scale in order to reach these areas and achieve a significant impact on society, so for us anything less than that is failure (and probably an unsustainable business).

Let’s take a look at some metrics from our test fleets in San Francisco and suburban Phoenix. Each cell shows the number of times our vehicles encountered a particular situation or had to execute a maneuver per 1,000 miles of autonomous driving in each location, normalized to time of day. Driving in an urban environment is almost nothing like driving in the suburbs.

Testing in the hardest places first means we’ll get to scale faster than starting with the easier ones. This may seem counterintuitive, but by testing in densely populated areas, we expose our software to unusual situations at a much higher rate, which means we can improve our software at a much higher rate. Based on our experience, every minute of testing in San Francisco is about as valuable as an hour of testing in the suburbs.

The counter argument would be that testing and ultimately deploying in suburban areas would accelerate the timeline to deploying in more challenging places. If dense areas were perhaps 20 percent more challenging than suburban areas, we might subscribe to this view. But in some cases we’re talking about a 4,658 percent jump! We believe software designed and tested in the suburbs requires a complete tear-up in order to succeed in dense urban areas.

In addition to being our home, SF conveniently has by far the highest population density of any place self-driving cars are being tested today:

Dense urban areas contain more people, cars, and cyclists that our vehicles must pay attention to at any given time (and sometimes even raccoons). Our vehicles predict the future motion of each object and simulate how those objects might interact with others. For example, we’ll predict that the car in front of us will brake if it appears it‘s about to get cut off by a cyclist, or that a car making a left turn in front of us will probably yield to a pedestrian in a crosswalk.

San Francisco’s population density is 17,246 people per square mile, which is about five times greater than in Phoenix. However, vehicles in our SF fleet predict an average of 32 times as many possible interactions as those in Phoenix. This task gets much more difficult as the number of objects increases, since there are exponentially more possible interactions between objects.

That’s just the object tracking side of things. Let’s take a look at some of the typical situations our vehicles face and the maneuvers they make when driving through San Francisco and in the suburbs.

Driving in SF (1 of 4): Traffic lights aren’t always red, yellow, or green

Self-driving cars must be able to read traffic lights, even in bad lighting conditions, but sometimes the lights don’t even work. Our vehicles have to reason about these kinds of situations and determine how to proceed. We encountered a six-way intersection in the Castro neighborhood of San Francisco with flashing red lights in all directions and successfully navigated through it.

Driving in SF (2 of 4): People break all the rules

People put junk in the street. They park everywhere. People don’t obey crosswalks. Our vehicles must be assertive, nimble and sometimes a bit creative. In this case, while driving through Chinatown, our vehicle had to avoid people moving pallets in the street and then wait for a bus to pass before moving around an open box truck. In SF, it’s very common to see maneuvers stack up on top of each other, unlike in suburban environments, where vehicles usually only execute one maneuver at a time.

Driving in SF (3 of 4): Frequent construction creates outdated maps

Construction zones are unusually difficult. They rarely follow any strict convention and often result in changes to lane markings or road topology. This means that software that relies too heavily on a preexisting map of the world will fail. Sometimes our vehicles must follow paths delineated by cones or flares, and sometimes they must obey hand signals from workers (as seen in the video below). Every construction zone is unique, and we interact with them 39x more often in SF than in suburbs.

Driving in SF (4 of 4): Cyclists are everywhere

Cyclists are tough to predict. They’re small, can reach speeds of 30 mph or higher, sometimes break traffic laws, may or may not use bike lanes, split lanes and weave between cars, and can change direction far more quickly than vehicles. Our vehicles encounter cyclists 16x as often in SF than in suburbs, so we’ve invested considerably in the behavior of our vehicles when a cyclist is nearby. In the video below, we narrowly avoided a cyclist in all black. He made a sharp left turn across traffic without having the right of way.

Driving in Phoenix: Wide lanes and smooth sailing

Many companies, including Cruise, test self-driving vehicles in Phoenix and its suburbs due to the ideal weather conditions, well maintained roads, minimal pedestrian/cyclist/vehicle traffic, and favorable regulatory environment. The unprotected left turn below is about as hard as it gets in these less dense areas. While we focus primarily on our performance in San Francisco, it’s still valuable for us to be testing in Phoenix and Detroit. We test in multiple places to ensure our software adapts to varying population densities, weather, and local driving styles.

Notice anything different about the Phoenix clip? You can get almost anywhere in the city without deviating outside a lane boundary, navigating through construction zones, or even passing a cyclist. While it seems crazy to test in an absurdly complex place like San Francisco, it’s absolutely necessary. We believe it’s the fastest way to achieve the level of performance and reliability needed to deploy self-driving cars at scale in a sustainable way, and that’s the only thing that matters.

Kyle Vogt is an engineer, entrepreneur and robotics pioneer who is redefining the future of human transport. In 2013 Vogt founded Cruise Automation, where he currently serves as CEO, which develops self-driving car technology . After successfully graduating from Y Combinator, Cruise was acquired by General Motors in 2016. Presently, Vogt is guiding Cruise and GM’s Autonomous Vehicles’ vision for tomorrow. Previously, he was a co-founder at Justin.tv, Socialcam (acquired by Autodesk) and Twitch (acquired by Amazon). A long-time robotics engineer, Vogt participated in the 2004 DARPA Grand Challenge as an undergrad at MIT, interned at Roomba-maker iRobot and competed in two seasons of “BattleBots.” A native of Kansas City, Kan., he studied computer science and electrical engineering at MIT. Reach him @kvogt.

This article originally appeared on Recode.net.

Contributions are a key part of the future of Vox

Readers rely on Vox for clear, nuanced coverage that not only illuminates the issues, but poses solutions, too. And we rely on help from our readers: Advertising and grants cover the majority of our costs, but we count on contributions to help us close the gaps in our budget. In fact, we’re looking to reach 95,000 individual contributions before the end of the year. Will you make the next contribution right now? Our average gift is just $20 — and it goes a long way in helping us keep our work free. Vox is here to help everyone understand what’s shaping the world — not just the people who can afford to pay for a subscription. We believe that’s an important part of building a more equal society. Join that mission by making a contribution today.

One-Time Monthly Annual

$5/month

$10/month

$25/month

$50/month

Other

$

Yes, I'll give $5/month

Yes, I'll give $5/month

We accept credit card, Apple Pay, and Google Pay. You can also contribute via