Chris Urmson: The Path to Driverless Cars Splits Away From the One That Lets You Nap

Lex Fridman sits down with Chris Urmson, who was technical director on Carnegie Mellon’s DARPA Grand Challenge and Urban Challenge teams under Red Whittaker, later CTO of Google’s self-driving car effort, and now CEO of Aurora Innovation with co-founders Sterling Anderson, formerly of Tesla Autopilot, and Drew Bagnell, formerly Uber’s autonomy and perception lead. Fridman asks what those races taught him. The high-order bit, Urmson says, was that it could be done. At the time it seemed nearly impossible, Carnegie Mellon was the only robotics institute around, and falling on their faces in public would have been embarrassing.

As technical director he did a lot of the work, alongside a bunch of other good people. Did he believe it was possible from the start? “Yeah, of course. Why would you go do something you thought was completely impossible?” They knew it would be hard. They did not know how they would do it, or whether they could do it the first time. Turns out they could not. There is, he says, a certain benefit to naivete: if you do not know how hard something really is, you try different things, and that gives you an opportunity wiser people may not take.

Every layer was a pain point — mechanical, sensors, hardware, software, mapping, localization, perception, control — which he treats as the joy of the field: you have to be good at each part. Looking back, the first desert race should be easy today. It was a static world with no other actors, out where GPS is good, and they could map it roughly. The engineering to make the vehicle controllable was still a slog, and they did not even know the route. DARPA would hand over waypoints and a corridor width; the most devious version would have been a kilometer-wide strip of scrub and rock. Fortunately it turned into driving along trails. “It was a hell of a thing back in the day.”

Whittaker, the legend leading the effort, left Urmson two leadership lessons he still uses. One is to go try the really hard things, because that is where the opportunity is. The deeper one is to see people for who they can be, not who they are. Whittaker would look at undergraduates and graduate students and empower them as leaders, with real responsibility, when another person might think, that is just another graduate student, what could they know. Urmson summarizes it as trust but verify: confidence in what people can become. In a lab that is always short of hands, that is how twenty-three-year-olds end up deciding how a vehicle takes an intersection.

Fridman asks whether the stack from the Grand Challenges through the Urban Challenge to today is the same technology made more robust, or whether there were real shifts. Urmson sees big steps. For the desert race, the unlock was HD mapping. A lot of prior off-road robotics had been done with no real model of what the vehicle would encounter. Getting decimeter-resolution models bounded the complexity of the driving problem and let the vehicle operate at speed, because it could assume things about the terrain ahead. Without that prior, purely reactive sensing could not go fast.

For the Urban Challenge, the game-changing piece was multi-beam lidar: high-resolution, mid- to long-range 3D models of the world, used to understand what was happening around the vehicle. In parallel, technologies that had been having their day in the academic sun finally ran on a visible stage. Bayesian estimation and SLAM had been everywhere at robotics conferences a couple of years earlier; almost every paper had SLAM in it. Watching those techniques play out in a real application, he found exciting. They were not doing SLAM in real time, because they already had a roadmap. They were doing localization, using lidar or cameras to register to a model of the world — a big step from naively trusting GPS and inertial navigation.

DARPA gave every team a map, which sounded like a gift and hid a trap that still trips people: the coordinate system. A latitude and longitude to many decimal places does not make you care which ellipsoid describes the Earth, until you want ten-centimeter or centimeter resolution. Then you care whether the data is NAD83 or WGS84 — different models of the Earth’s non-sphericity, and at least one of them has to account for tectonic shifts and transform the global datum as a function of year. Getting a map is one problem. Matching it to reality at centimeter resolution was, he says, interesting and fun.

Perception in that artificial city already had the shape of today’s problem, even if it was more than a decade behind. They tracked other vehicles at a hundred-plus meters because they had to merge with traffic. They used Bayesian estimates of each vehicle’s state, predicted where it would be a few seconds out, and held multiple hypotheses at an intersection — right, straight, or left. The nastier layer was that their own behavior would change the other operator’s behavior. They did a lot of that in relatively naive ways. It still had to work.

Ten years later, the jump from that constructed airbase to real cities is not a new set of module names. The biggest change is that the other actors are truly unpredictable. Most of the time other road users behave well, but not everyone, and you cannot know who or when. The variety of vehicles is the full set, which the airbase did not have. They did not have to deal with cyclists, pedestrians, or traffic lights. The geographic scale is much larger than the base they were thinking about then.

Asked for the hardest part of driving, Urmson jokes that nothing jumps out as one thing, then names the fundamental difference: they are doing it for real. That DARPA environment was limited complexity — certain actors were missing, roads were maintained, barriers kept people separate from robots — and it only had to work for 60 miles. From 2006, 60 miles was the victory condition. From now, they want things that will drive for half a million miles. Same software ideas; a different game by three or four orders of magnitude in lifetime.

Lidar came in early and is still a primary sensor. Urmson calls it essential, and says the same of cameras and radar. Robustness comes from composing data across them. Fridman puts Elon Musk’s line on the table: lidar is a crutch, and much of perception can be done with cameras. Urmson grants the existence proof immediately. People walk around without lasers in their foreheads, get into vehicles, and drive. He does it every day. You cannot argue with passive vision as a thing that works.

Then he changes the frame. Sure, lidar is a crutch, in the same way the combustion engine was a crutch on the path to an electric vehicle, and in the same way any technology gets replaced by a superior one. The ground-transportation system itself is broken. The number he had seen that morning was 37,000 Americans killed on the roads the previous year, and that is not acceptable. Any technology that accelerates self-driving coming to market and saving lives is technology they should use. Refusing lasers while accepting an 8- or 16-megapixel camera feels arbitrary. “These are just bits of technology, and we should be taking the best technology from the tool bin.”

Carmakers, Fridman notes, talk about cost more than anything, and want the cheapest sensor suite that still makes a safe vehicle. Urmson corrects the premise. You do not actually want the cheapest kit. You want a suite that is economically viable and that works; after that, everything is margin and driving cost out. A story about a $50 sensor instead of a $500 one is great until the $500 one works and the $50 one does not. If there is an economic offer, there is a sustainable business, and that is how the thing actually comes to scale. Lidar, in his view, has no fundamental reason to stay expensive. It will remain costlier than an imager because CMOS fab processes scale dramatically better than mechanical ones, but cost should still come down substantially, and the right business model can absorb more bill-of-materials cost if the suite actually works.

Urmson asks to be careful, because this part of his view gets twisted. Active safety is important, should be in vehicles, and can reduce accidents and fatalities in the near term. Level 2 means the vehicle is controlling two axes — braking and throttle, or braking and steering. Variants that actually support the driver should be encouraged. The real challenge is human factors, public misconception of the capability set, and the trust people give it. He is incrementally more concerned about Level 3, and about how Level 2 is marketed and how much effort goes into those human factors.

People will over-trust the technology. In the weeks around the taping there had been a spate of people sleeping in their Teslas. He had watched Trevor Noah, a smart person with resources, describe a Tesla as a self-driving car and ask why people should not sleep in it. Urmson’s answer is blunt: it is not a self-driving car, it was not intended to be, and those people will almost certainly die at some point or hurt others. How the technology is described and brought to market has to be thoughtful. He also thinks the technology path of these Level 2 driver-assistance systems will diverge from the path needed to deliver vehicles you can get in, sleep, and match or beat a human driver’s safety. The economics are different, so the stacks will be too.

Responsible L1 and L2 products put the driver in charge and build the safety case on driver attention. At that point you can, for economic reasons, give up to some degree on false negatives. A forward collision-mitigation braking system that catches the missed lead vehicle half the time would be an incredible advance, equivalent to seatbelts. Unsupervised, that same system would hit one out of two cars. Economically that is a perfectly good driver-assistance product: get it working 50 percent of the time, then drive the cost out so it lands on as many vehicles as possible. Driving cost out does not raise performance on the missed-detection case, so you will not arrive at a technology that can actually drive itself.

Fridman asks the philosophical follow-up: if everyone were perfectly informed, could Level 2 be used without over-trust? Urmson does not think so, because that world does not exist. People combine the facts in front of them with experience. Commute 30 miles each way on the 101 for a month, 1,800 miles, and it just works. That is compelling. You start to tell yourself the statistics must be about harder places. The sample is a drop next to roughly 85 million miles between fatalities, so personal experience systematically underestimates the risk, and people will trust it anyway because it is hard not to. Stretch the same logic over a year or two and nothing changes. Early, technology-savvy owners might hold a more accurate risk picture longer. As the product moves to the mass market, people will read the material, compare it to a smooth month, and the defense will collapse faster.

Aurora’s program, like Google’s before it, is on the full-autonomy path. How do you demonstrate to the world that the system is safe? Urmson admits there is no soundbite. You show a combination of work done diligently. A functional-safety process is part of that: here is how we did the work, so if you believe the description you can have some confidence we were thorough. On top of process comes evidence that they were not only thorough but good — simulation, unit testing, decomposition testing, and some on-road data, statistically, to the degree they can. The public conversation will happen, but they will also spend more depth with trusted nodes: NHTSA and other federal and state regulators, operating in the public interest. If those bodies are convinced, he thinks they are in a pretty good place.

Miles per disengagement, the metric the industry has been publishing, quickly turns into a marketing scheme: alter the experiments and the number moves. Urmson was on the record telling the California DMV it was not a great metric. He does think better measures are possible, and they will not be one number. Internally they are grappling with human performance on decomposed tasks — detecting traffic lights, making a left turn across traffic — and with what the human failure rates are, then showing regulators and eventually the public that the system should do better. Those individual metrics can tell a compelling story. What they actually care about is lives saved, injuries reduced, and the dollars people are not paying to fix cars. Aviation’s event pyramid — crash at the top, then injuries, near-misses, violations of procedure — builds a statistical model from low-severity events to high-severity ones. They will need something like that, because an event per 85 million miles is statistically hard to compare directly even at U.S. scale, and a fatality tied to an autonomous vehicle is currently magnified in attention.

The most popular public frame is the trolley problem, which Fridman calls misguided. It still signals that people are grappling with handing control to a machine. How do you win hearts and minds? Let them experience it. Skepticism is reasonable; people should ask questions and doubt, because this is new and they have not touched it. The opportunity to make roads safer, improve access, and reduce the cost of mobility is also clear. Once people try it, Urmson thinks it becomes obvious. His favorite demo is taking a skeptic onto the 101 at 65 miles an hour. After ten minutes they turn and ask, is that all it does? Yes, it is a self-driving car; he is not sure what else they expected. Mundane is the goal. Nobody turning on a light switch thinks about electrons being pushed down a wire from a power plant. You get annoyed if it does not work. What you value is seeing colleagues, reading a paper, not being afraid of the dark. That is what this technology should feel like: in the background, safely.

When will we see large-scale deployment, on the order of ten thousand vehicles? Within ten years, Urmson says, with pretty high confidence. The impressive moment is not the count. It gets serious when there is a driverless vehicle operating on public roads continuously, without a safety driver in the vehicle. That is the zero-to-one threshold. After that, the work is scaling, business models, and customer experience, so it is actually a useful product. That is when the mixed science-and-engineering project becomes engineering and commercialization.

He thinks that first inkling is urban and suburban, not the highway. The industry intuition leans toward trucking because freeways feel easier: everyone going the same way, wider lanes. The intuition is pretty good except that they do not care about most of the time. They care about all of the time. A truck at 70 miles an hour with a 70,000-pound load carries an incredible amount of kinetic energy. When it goes wrong, it goes really wrong, and those challenges occur more rarely, so you learn more slowly. Highway driving is incrementally more difficult than urban driving, not easier. Moderate-speed cities win on three counts: two vehicles crashing at 25 miles an hour is not good, but probably everybody walks away; those events happen frequently, so you learn faster at lower risk; and you deliver A-to-B value people actually need. Once that is solved, freeway driving falls out.

Asked for a magic wand that is not infrastructure, he names perception and forecasting. If tomorrow you could give him a perfect model of what has happened, what is happening, and what will happen for the next five seconds around a vehicle, that would accelerate things pretty dramatically. What keeps him up is not big vehicles. It is vulnerable road users, especially the mix of cyclists with cars, and cyclists and pedestrians, because they are not in armor. Occupants of cars have protection; a person on a bike or on foot has none, so they need extra attention. Fridman asks the game-theory worry: if you protect pedestrians by being careful and slow, will they take advantage, and is “nudging through a crowd” even a decent algorithm? Urmson splits it. He does not believe people will step into the road to bully robotaxis more than they already test human drivers; if someone steps in front of him today he stops, annoyed, and does not drive through them, and people do not want to get hit. Nudging through a concert crowd is further down the pipeline, done at one mile an hour rather than 30, and is more an experience problem than an algorithm: a visual display so a pedestrian feels pleasantly asked to move.

Who is the main competitor, and how does Aurora out-compete them? They do not spend a lot of time thinking tactically about beating a particular company. It is a huge, difficult problem, and it is good that a bunch of companies are tackling it, because it matters that somebody gets there. Speed comes from a leadership team with enough experience to see where the coldest dead ends are and avoid them; from a mission that attracts strong people; from a culture that lets them spend time solving problems; and from heavy early investment in infrastructure and architecture. Because they brought in good people and good investors early, they did not have to spend all their time leaping from demo to demo. They could build machine-learning infrastructure, pull data from on-road testing, and use that to accelerate engineering. That continuing investment in unglamorous tools, he thinks, is what will let them do something pretty incredible.