Building a Driver, Not a Copilot: Waymo CTO Dmitri Dolgov on Fully Driverless Cars

Lex Fridman opens with computer science, not cars. Dmitri Dolgov, Waymo's CTO, says the first interesting machine in his life arrived in the late 1980s: an IBM PC with a turbo button on the front. He spent a few months on video games, got bored, and started making the machine do other things. A couple of years later he had written a Pascal game with graphics at something like 320 by 200. A Japanese developer offered a few hundred dollars for it — real money for a kid in Russia. He turned the deal down.

Looking back, he calls it not the most acute financial move of his life. He uploaded the binaries as freeware and did the arithmetic of a teenager: thousands of players, a couple of bucks each. Fridman asks about childhood dreams, and Dolgov refuses the usual story about robots or the mysteries of the universe. At four or five, in 1980s Russia, he wanted to be a traffic-control cop — the officer who stood in the middle of an intersection all day with a black-and-white striped baton and directed the flow. His parents, both physics professors, were a little concerned.

He never really got hands-on robotics in school. After applied math and physics, and more abstract computer science, the leap came after graduate school: Michigan, then a postdoc at Stanford, where self-driving cars were his first real robots. College itself was a stubborn choice. Both parents had gone to the Moscow Institute of Physics and Technology, and the stories he grew up with were enough that he moved back to Russia after finishing high school in the United States. Peers treated it as the act of a crazy kid. He spent six years there on a bachelor's and master's in physics and math, “fairly hardcore on the fundamentals.”

In 2006 Dolgov joined Stanford's DARPA Urban Challenge team and worked on motion planning. The two earlier Grand Challenges had been about driving in a static desert. For 2007, DARPA took over an abandoned air force base, built a little fake city, and mixed autonomous cars with professional human drivers. There were no pedestrians. Each robot got a crude map and a mission: go here, then there, then over here, while sharing the space. It was, he says, a very rudimentary version of a self-driving car in a dynamic world, and it helped kick-start the industry.

Stanford came in second; Carnegie Mellon won. The memory that hooked him was not race day. It was the first time his free-space parking-lot planner ran on a real car, on Stanford's campus, around 3 a.m., with a course of cones. The code compiled, the car drove, and it did something quite reasonable. It was buggy and full of problems, but it was “pretty darn magical.” He went home and could not sleep. He has been doing the same work ever since, for more than a decade.

On competition day the cars lined up empty. DARPA sent them out one by one, each with its own siren personality. You could not see them. Every so often one drifted close enough that you heard your vehicle and felt hope; too long a silence and the anxiety returned, “a little bit like sending your kids to college.” His own bug became a metaphor in some circles: the victory lap. Missions often punched a waypoint on a big oval. Other cars hit it and left. Stanford's car hit the checkpoint, looped the oval, and only then went on its way. The planner would not start reasoning about the next waypoint until it had arrived at the previous one, and by then a lane change was too late.

He still argues with CMU people about first place. His reading of the parking-lot rules was that you had to reverse out of the spot; Carnegie Mellon's car, he says, just drove through. Extra time either way. He jokes that CMU is a small school that happens to succeed at robotics, and that most people go there for the scenery. Fridman supplies the public-service announcement: Carnegie Mellon is one of the great robotics universities in the world.

Larry Page and Sergey Brin saw the DARPA races and believed the technology. Google's self-driving car project started in 2009 with about a dozen people, Dolgov among them. They were excited by the Urban Challenge and still had, he says, a very rudimentary understanding of the problem. The first goal was to find out what they were up against. They set two milestones meant to maximize learning: drive 100,000 miles in autonomous mode, orders of magnitude more than anyone had done, and complete ten routes of 100 miles each, chosen to be extra spicy, with no human intervention from the moment you pressed the button.

The routes sampled the full mess of driving. One ran every freeway and every bridge in the Bay Area. Others went around Lake Tahoe on mountain roads, through downtown Palo Alto, through San Francisco. You engaged autonomy at the start and had to finish 100 miles with zero takeovers. Both milestones took just under two years — about a year and a half, a little more. He calls it probably the most fun of his professional career: the job was to learn and prototype, not yet to build a production system, so people hacked things together around the clock, and they did not yet know how hard the real thing would be.

Fridman presses on the ambition: zero interventions, on hard roads, as soon as possible, rather than a ten-year fade-down of disengagements. Dolgov's lesson is the gap between doing something once and engineering a product you can evaluate, measure, and ship with confidence. A dozen people with 2009-era technology could hit those ten routes in sixteen months. A full driverless product that customers pay for took much longer. The most important early takeaway was that the problem was doable — and that they had only a glimpse of its true complexity. It was like climbing a mountain, seeing the next peak, thinking it was the summit, and discovering that was just the start of the journey.

After the 2010 milestones, the team pivoted from research to building a product. For a couple of years that product was freeway driver assist, what you might call L3. Around 2013 they had learned enough, and thought more clearly about what they actually wanted, that they pivoted again: build a driver, and put it in fully driverless vehicles with no person in the loop. “Even the way you go building a driver assist system is fundamentally different from how you go building a fully driverless vehicle.” They chose the latter and have stayed on it ever since. Dolgov says it was exactly the right decision for them.

In 2015 they ran what he calls the world's first fully driverless trip on public roads, in a custom marshmallow-shaped vehicle nicknamed Firefly. The passenger was Steve Mahan, a friend of the project from the early days, who happens to be blind. The car had no steering wheel and no pedals. There were no lead cars, chase cars, or police escorts. The environment was uncontrolled; the route was fixed. They did it a few times in Austin, Texas, after a tremendous amount of engineering and validation — a proof, not yet a service.

Waymo was founded at the end of 2016 and the beginning of 2017 as an independent company under Alphabet, because they believed in the commercial vision and needed an entity that could pursue the product at scale. In October 2017 they started regular driverless operations on public roads. The first day they drove 100 miles; the number mattered less than the fact that operations were now ongoing. Some of those miles had a completely empty driver's seat. On many rides they still put someone in the chair so they would not be all over YouTube and Twitter the same day, but the person was not there to disengage.

In 2018 they launched the public commercial service Waymo One in Phoenix. In 2019 they started offering truly rider-only trips to an early-rider population. In 2020 the company raised about $3.2 billion in its first round of external financing, still inside Alphabet but maturing as a firm, and began putting fifth-generation hardware on the Jaguar I-Pace. Fridman, who has ridden in a Waymo with nobody in the driver's seat, keeps coming back to the steering wheel turning by itself: a 20th-century object, now moved by a being with power. Dolgov agrees it is magical, and that rider-only operation is what unlocks the real potential of the technology.

Waymo separates the vehicle from the self-driving hardware it puts on. Fifth generation is a qualitative jump from fourth, not only from first. Lidars have been designed and built from the ground up for years; Firefly already carried a mostly in-house suite. On the current platform they still use lidars, cameras, and radars, plus a beefy onboard computer that does perception and complex reasoning in real time. Dolgov will not talk GPUs, TPUs, or other chip details — he jokes that he sounds like the government people Fridman interviews about UFOs — but he will say the computer has redundancy, serious general-purpose compute, and serious inference for hungry models.

The published sensor count is specific: 29 cameras, five lidars, six radars. Lidars are almost exclusively in-house, with some fundamental technologies unique to Waymo. Radars and cameras are more of a mix of custom design and supply-chain partners. Control of that stack matters because the sensors can pipe rich raw observations — not just point estimates — into large models. Being inside Alphabet helps on the other compute problem: the fleet generates huge volumes of data that have to be mined for edge cases, used to train and evaluate models, and run through simulation of the whole system. They build custom frameworks for all of that.

Connectivity helps. It is not required. Each car is built to operate independently, with no link to the outside world. Dolgov thinks that point is important: you need a fully capable computerized driver on every vehicle, then you let the cars share. Today, when one car meets something interesting — an accident, a new construction zone — the information goes over the air to the rest of the fleet. Maps, in this picture, are priors distributed across the fleet and updated in real time. As density rises, cars could exchange planned intentions and highly dynamic observations that never belong in a static map. All of that is an advantage. Survival without it is the requirement.

Fridman asks about the human in the loop and teleoperation. Dolgov is blunt: Waymo does not do teleoperation. What riders meet is closer to customer support. First-timers get guidance on the phone, then an in-car screen and audio, plus a button that connects them to a real person. A second human role is fleet assistance, which he analogizes to traffic control. Anything safety-critical or latency-critical is decided onboard, purely autonomously. In a particularly hairy off-nominal scene, if connectivity happens to be available, the car can ask a human assistant to confirm an interpretation and add a little context. If the network is down, the car still has to handle it.

October 8, 2020, is when fully driverless, rider-only vehicles entered the public Waymo One service in Phoenix. Before that, early-rider program users had already been taking empty-seat trips. After that date, anyone could download the app. Demand, Dolgov says, exceeded capacity, so they were managing it, but the product is exactly what it sounds like: a paid ride from A to B. The enabled geography is larger than San Francisco. Inside it, pickup and drop-off are free except for ordinary pull-over constraints. People take kids — the youngest riders have been one or two years old, in car seats — to school, grocery stores, restaurants, bars, errands, the office. Most rate the driver five stars in the app.

Feedback comes in during the ride on the phone and the in-car touchscreen, through live help, in star ratings and comments at the end, and in deeper user-experience studies that track people over time. The surprises are rarely about the driving stack. Someone wants to load 20 bags of groceries. Someone bikes to a meeting, takes the bike apart, and loads it into the car. Pickup and drop-off locations turn out to be a product in themselves. A naive heuristic — drop a pin, pick the nearest legal curb in latitude and longitude — fails in horrible ways.

One anecdote he repeats: the car chose a spot that was closest on the map and happened to sit on the other side of a parking lot behind a row of cacti. The rider had to walk the long way around in 110-degree heat. Richer maps, walking paths, and learning from what actually worked for other people all help. So does a property that is not optimality. Predictability and understandability, he says, matter more for the experience than a slightly better but flickering choice. The basic promise still sits underneath: the car actually arrives where you told it to.

Because the phone and the car are two computers talking, the interface can show more than a moving dot. As the vehicle comes to pick you up, it will tell you it is sitting at a stop sign, or waiting at a red light. Little touches, Dolgov says, but they delight people and make a black-box machine readable. Compared with a human-driven analog, the computer-to-computer channel is also how you get the car to show up at the exact curb every time.

Fridman compares frictionless products he loves — one-click checkout, painless transcription, audio cleanup — and asks whether Waymo is trying to make people fall in love, or merely to be convenient. Dolgov's vision is to remove friction from getting riders where they want to go, and then to do the same for goods. Download an app, click, the same car shows up, the trip is safe and high quality, and it just happens. Ride-sharing companies, Fridman notes, do not fully control the experience; car companies sometimes make people loyal to a machine. Dolgov wants the best, most enjoyable transportation, used seamlessly in daily life. That, for him, includes a kind of falling in love: safety, predictability, privacy, a space you do not have to share with strangers, and driving quality you can trust.

Then Fridman puts the 20th-century driving instinct on the table. Human Uber drivers roll stop signs. A little aggression, he suggests, asserts your presence so other people respect it and you get through the intersection on time. Do robots need to flirt with the rules? Dolgov splits the idea. The cars will follow the rules and do the safest thing. Think of a professional limo driver: smooth, efficient, assertive, comfortable for people inside, predictable for people outside. That is the driver they want to build. In sports, the real professionals are not hectic. They are precise, and they get the best results.

You can roll stop signs and make crazy lane changes. It typically does not get you there faster, and it is typically not the safest, most predictable, or most comfortable thing to do. Fridman, who has written his own awkward stacks — overly aggressive when it was his code, or awkwardly cautious otherwise — says the Waymo ride in Phoenix felt assertive and efficient, without the “come on, move already” feeling. He wonders whether a Russian instinct that you have to break rules to get anywhere is simply wrong for driving. Dolgov does not think you have to pick. Truly good driving gives you efficiency and assertiveness *and* comfort, predictability, and safety.

He reaches for a precision-and-recall curve. Given a model's current capability, it is easy to slide the operating point: more cautious or more aggressive, fewer false positives or fewer false negatives. Aggressive is bad and cautious is bad if you are only moving on the same curve. True capability comes from lifting the whole curve, after which you are on a different plane of those trade-offs. That, he says, is what they are trying to do.

Waymo has two commercial applications of the same driver: Waymo One for people, Waymo Via for goods. Trucking has been in the works since 2017. Dolgov's claim is that the first-order challenges carry over — hardware, onboard and offboard software, training, evaluation, perception, semantic understanding, prediction, decision-making, planning, simulation, ML infrastructure. Data and the domain differ; the genuinely hard problems do not. On trucks they mount two main lidars on the sides to kill blind spots; on the I-Pace one sits on the roof. The sensors themselves are largely the same, so years of custom lidar and radar transfer. On freeways, range matters more. The fundamentals still hold.

Fridman splits the world into moving humans, moving goods, and moving nothing — empty repositioning, which is a large share of miles and a dull commercial story. To a first-order approximation, Dolgov says, control policy does not change for a zero-occupancy vehicle. Smooth, predictable driving is not only for whoever or whatever is inside. It is also for everyone outside, so the car fits naturally into the shared environment. Second-order effects show up in routing and fleet-scale optimization: you do treat an empty repositioning trip differently from a paid one. They are mostly second-order.

How do you take over the world after Phoenix? One step at a time, then the real answer: larger-scale commercialization. He thinks about three axes. First, core technology — the hardware and software of the driver. Fourth generation is already good enough for fully driverless trips in Phoenix, night and day, at various speeds and in various conditions. Fifth generation is the platform for massive scale: better capability, a simpler architecture, more reliable redundancy, designed to be manufactured in volume with the right unit economics. Software is getting a similar robustness-and-generality pass so they can expand beyond Phoenix quickly.

Second is evaluation and deployment: how you measure the system, how you release new versions of the driver into a fleet with confidence, how you make simulation, data science, and validation so solid that engineers are not taxed by every ship. They got good at that in Phoenix; now they want it efficient enough to stamp out in new places. Third is product, commercial, and operational excellence — the hard lessons of actually serving paying customers. Phoenix was chosen as the first full deployment on purpose, to tease the problem apart, de-risk each dimension, and force the team to learn the whole thing end to end. They are not yet massively copy-pasting cities. They are folding those lessons into the next system. They have tested in more than 25 cities, including San Francisco and snowy Michigan. Dolgov will not blink twice about Austin or Moscow. What he will say is that they lead with local, state, and federal officials, and that after years of that work the government piece is a conversation they keep having, not a blocker they have to smash through.

Fridman quotes Elon Musk: lidar is a crutch. Dolgov would not characterize it that way. Lidar is a key sensor, used with cameras and radars. At the physical level they are very different. Cameras are passive; lidars and radars are active and live on different wavelengths, so they complement one another. Together they build a safer, more capable system. His question is the other way around: why handicap yourself by dropping a modality that undoubtedly helps? He leaves a door open. Trade-offs that make sense for driver assist may not make sense for a driver you deploy in a fully empty vehicle.

The usual objections are cost and aesthetics. He finds neither compelling. Radars used to be expensive, then people needed them, the technology moved, and they were manufactured at massive scale for cars. Lidar is on the same curve. Fifth-generation units, he says, took a discontinuous jump in the underlying technology: they can be made at significant scale for a fraction of Waymo's previous generation and a fraction of what you can buy off the shelf, and the improvement will continue. On looks, beauty is in the eye of the beholder. Fridman tries to start a fight about the Porsche 911 looking like a baby carriage. Dolgov declines to comment on the Porsche monologue.

Under the lidar argument sits a better question: how much of driving can be learned, even end to end? Fridman mentions Tesla's multi-task network and a conversation with George Hotz, whose comma.ai stack treats drivable area as a learning problem and hopes a Level 2 system grows up into full autonomy. Dolgov decouples sensing from software. On Waymo cars, lidar, radar, and cameras are all machine learning. Custom lidars and radars do not just emit point estimates. They return raw physical observations, richer than angle and distance, which get fused with pixels and fed into large models for detection, classification, and state estimation. Fusion has been moving earlier: from weak, engineered late fusion of per-frame, per-sensor nets toward joint training across modalities and time.

The purest end-to-end picture — one model from raw sensors to steering torque and gas and brakes — is “too much.” He does not think that is the right way. Smaller versions of end-to-end, with co-training and signals propagating across stages, get into a real design tension. You want modularity and composability so you can understand, evaluate, and validate the system. You also do not want interfaces so narrow and brittle that you give up generality and cannot push rich signal forward and loss backward to optimize jointly. Every part of the stack already leans on data-driven development: mapping, simulation, perception, prediction, decision-making. Object detection of pedestrians, cars, cyclists, cones, signs, and vegetation is table stakes, step zero. The interesting problems are scene-level reasoning, joint prediction, and interaction among all the actors.

The traffic-light example is how he draws the line. Detecting the state of a light is bread-and-butter computer vision: dump raw images into a convnet. You do not need a model to infer, with some precision and recall, that red means stop. That is a clear engineered signal with clear semantics, and you inject it as a constraint or a cost. Then you are back in heavy ML for everyone else. Other people are supposed to stop on red. That does not mean they will. Predicting a red-light runner means picking up subtle cues in object behavior, pedestrians, cyclists, and the whole configuration of the scene.

Machine learning has been in the story since DARPA. Stanford's Grand Challenge team, before Dolgov's time and before ImageNet, used lidar and camera to extend free-space reasoning so the desert car could go faster. Google got heavily involved in deep nets around 2010 or 2011, unusual then for a self-driving group, and Waymo still collaborates across Alphabet on supervised, unsupervised, and self-supervised work. The analogy he likes now is language. Transformers and GPT-3-class models have a sequential structure with strong local connections and a longer context that is not local. Driving trajectories look the same: the next second is tightly coupled to the last, and the whole scene still changes what you should do. Those ideas, he says, are landing in behavior prediction, decision-making, planning, and generative simulation of humans, cyclists, and other cars.

Pedestrians and cyclists are, Dolgov says on the record, among the highest safety priorities, including the ones without Twitter accounts. Completely empty cars already drive through school zones in Phoenix. Hardware is the first margin: turning on a dark residential street, headlights do not cover every pocket a child might run from, and lidar sees as well in complete darkness as in daylight. The system sometimes detects those objects earlier than trained operators in the car. Perception then has to catch weird poses, running, partial occlusion. Latency has to be low enough for a strong reaction. One Phoenix case: a multi-lane road at 45 miles per hour, a woman on a bicycle on the sidewalk, the vehicle in the right lane. She tripped and fell into the path. Test drivers steered hard and braked hard and stopped. In simulation, the onboard system did the same thing. A person and a bike that had been one object split apart, appearance changing in milliseconds, with traffic beside them. Braking alone was not enough.

Fridman raises the trolley problem. Dolgov treats the puzzle as real philosophy and a poor engineering spec. Humans cannot answer it either; that is the point. People often freeze, spend extra milliseconds processing, and then produce the worst of the available outcomes. In the stack you reason about collision probability, severity, and uncertainty, and those factors shape emergent behavior. The clean thought experiment almost never shows up. More often you made a mistake earlier. If you build a good, defensive driver, you see enough clues in the environment that you do not put yourself in that corner. It is the precision-recall point again: do not pick the ugly trade on a bad curve; lift the curve so the dilemma does not arise.

Asked for three books, he starts with Mikhail Bulgakov's *The Master and Margarita*, which he has read only in Russian and is not sure translates. You can reread it your whole life: as a kid it is the devil visiting the Soviet Union, later it is culture, style, and a quirky dark humor that he thinks captures something of the Russian spirit. Second are the Strugatsky brothers, up there for him with Asimov and Bradbury: *Roadside Picnic*, *Hard to Be a God*, *Beetle in an Anthill*, *Monday Begins on Saturday*. The last is a research institute that works on magic, with deep parallels to serious science; *Hard to Be a God* puts a higher civilization into a primitive world and asks hard ethical questions. He never got *The Snail on the Slope*. Third is Orwell's *1984*, both a book that marked him and one he thinks people should read. You put it down and stare into space. Doublethink — holding contradictory views, ignoring logic, sticking to the party line — keeps coming to mind. Fridman, heartbroken as an optimist in 2020, says society is a fragile experiment. Dolgov agrees the failure mode is less a flamethrower than a slow boil. That, he says, is the danger.

The meaning of life gets an Asimov story first. In “The Last Question,” humans ask a growing supercomputer how to reduce the entropy of the universe. For thousands, then millions, then billions of years the answer is that there is as yet insufficient information. By the time the machine can answer, heat death has already happened, so it answers by demonstration: it restarts the Big Bang. Dolgov will not speak for humanity. Personally, meaning changes. Newborns are driven by new smells, sounds, and emotions; then by fun; then by the joy of comprehension; then by giving something back. Kids flip bits — he has three — about what matters. He wants those drives to be additive, not replacements: keep having new experiences, keep having fun, keep learning, keep contributing. Ask him in a few years, he says, and he may have one or two more to add.