Taming the Long Tail: How Waymo Builds a Self-Driving Factory

Anguelov, a principal scientist at Waymo, traces the project from its start as a Google moonshot led by Sebastian Thrun, whose first goal was to drive ten different 100-mile segments. The first fully driverless ride on public roads happened in Austin in 2015, with a blind passenger inside. In 2017 Waymo put a fleet of fully self-driving vehicles on the streets of the Phoenix area, and a commercial service followed the next year.

By the time of the talk, Waymo had driven more than ten million miles on public roads, some of them driverless and some with human drivers collecting data. Even so, Anguelov says the team still keeps meeting scenarios it has never seen. A car that removes the human operator has to be capable of the entire task and has to do it safely, which is where the question of how many situations it must handle comes in.

His answer is the long tail. Getting a car to drive in common cases is one kind of effort, while taming the remaining rare situations is another, and they matter just as much. He shows clips of a cyclist carrying a stop sign, a pile of poles falling into the road, cone-based construction zones, emergency sirens, and a driver running a red light at speed just as Waymo's car is about to move off a green.

Anguelov breaks the core AI of the car into three tasks. Perception maps sensor data, such as pixels, lidar points and radar scans, to a scene representation of objects, semantics and maps. The difficulty is the sheer variety: people dressed as dinosaurs, different vehicle types, times of day and seasons, and odd configurations such as occlusion, reflections or someone carrying a large board.

Prediction is about anticipating what other actors, mostly people, will do, sometimes up to ten seconds ahead. Useful signals include past behavior, high-level semantics, and subtle cues like whether a pedestrian is looking at the car, a driver's gesture or a blinker. Agents also affect one another, as in a clip where two cyclists must go around a parked car and the Waymo vehicle slows to let them pass.

Planning is the decision-making machine that turns all this into acceleration, braking and steering. Its behavior has to be safe, comfortable for passengers and legible to other road users, while still making progress toward the destination. Anguelov shows a crowded school-gathering scene with pedestrians and a cyclist close behind as an example of the trade-offs involved.

Anguelov offers an allegory. A classical system is built by an artisan who takes tools and crafts the product. It can reach something reasonable fast, but it becomes harder to change as it grows more complex. Modern machine learning is a factory: you build the infrastructure, feed it data and get models out, and you can keep iterating at scale.

The loop he describes is to release software to the vehicle, drive and collect data, select the interesting parts, send them to labelers, train models, test and validate them, and then redeploy. Each pass lets the team select new data that was not selected before. The whole thing has to be automated, which makes it a game of infrastructure.

He names three ingredients that Waymo gets from being part of Alphabet. The first is compute, including TensorFlow, data centers and specialized training hardware. The second is high-quality labeled data, where the key is not hoarding miles but choosing them, using active learning, data mining for rare or uncertain examples, and auto-labels that use what the car saw in the future to annotate the past. The third is strong models, with research collaboration with Google and DeepMind.

As an example of infrastructure, data and models working together, Anguelov describes a project with the Google AutoML team on automatic neural network architecture search. Network design is usually done by human practitioners and can take months, especially under tight latency and compute limits. Waymo wanted to hand some of that work to machines, so the architecture can keep evolving as new data arrives.

The first approach reused a small sub-network cell found on the CIFAR-10 dataset and applied it to lidar point segmentation. After exploring several hundred cell combinations, the team found models with similar quality but much lower latency and compute, or somewhat higher quality at the same latency. Similar results appeared for lane detection, which means the automatic search outperformed the human-designed baselines.

The second approach was an end-to-end search over a much larger space using a cheaper proxy task. The team trained up to 10,000 architectures, took the top hundred and fully trained those. The first round did not beat the transferred cell, but a second round did, and a reinforcement learning search over 6,000 architectures improved on it further and also beat Waymo's in-house algorithm.

Even with the factory running, models can be uncertain or wrong on rare cases, and a product cannot simply say the network does not handle something. Anguelov lists a first line of defense: a 360-degree view with complementary camera, lidar and radar sensors, each with different strengths and failure modes, so that when one fails the others usually still work.

The second is a deliberately hybrid design. Perception inputs are high-dimensional and large networks are hard to train with few examples, so expert knowledge can supply input representations and biases that make learning from a few examples easier. In perception, the car still uses tracks and obstacles from lidar and radar and drives safely relative to them even when ML is unsure. In prediction and planning, low confidence leads to more conservative driving.

Anguelov sums it up as two ways to attack the tail: protect against it, and keep growing the part that ML can handle. As the factory runs and models improve, the scope of ML grows and the need for hand-built protection shrinks.

Testing a new algorithm by driving it around is too slow when some conditions are rare, and code should be well tested before it goes on public streets. Waymo uses structured testing at a 90-acre former Air Force base for staged situations, but the bulk of testing is simulation: the equivalent of 25,000 virtual cars driving ten million miles a day, with over seven billion miles simulated. Even a small change to perception can ripple through the whole system, so tests have to go all the way up the stack.

Scenarios come from safety-expert analyses of typical accident conditions and, more importantly, from logs of real driving that can be varied to create new cases. The catch is that in simulation the car's own actions alter the world. Anguelov shows a case where the simulated car moves differently, so another agent that safely passed in reality would now collide, which means other agents must react realistically.

Simple brake-and-swerve models do not produce believable behavior in merges, lane changes or intersections, so Waymo tries learning agents from demonstration. In a recent paper the team trained a network on 60 hours of driving, using a top-down image around the vehicle with the road map, traffic lights, perceived objects, pose history and the intended route. To avoid compounding errors it added synthetic perturbations, and it also added road-mask and collision losses so the agent stays on the road and avoids hitting others.

The learned agent handled many scenarios and even drove a real Waymo car on a road it had never seen. But it failed in telling ways: it drove too fast to make a turn it could not see coming, and it did not complete a complex U-turn. The lesson, Anguelov says, is that the long tail returns in testing, because a model with millions of parameters may not learn behavior that appears only two or three times in 60 hours.

For the tail of human behavior, Anguelov argues for simpler, expert-designed models that are easier to learn with few examples. He describes a trajectory optimization agent inspired by motion control theory: it plans a trajectory that minimizes costs, with attractor potentials that pull toward where the agent went in the log and repeller potentials that push away from collisions. Only a few dozen parameters need tuning.

Those parameters can be learned with inverse reinforcement learning so that the resulting trajectories look like observed behavior of a chosen type. That lets the team dial in aggressive or conservative drivers. In his demos, an aggressive driver pushes past Waymo's car and forces it further into its lane, while a conservative one lets the car switch lanes comfortably; there is also an aggressive weaving motorcyclist and a highway merge where one driver finds a small gap and goes for it.

He lays out a space of agent models, from replaying logs and hand-designed reactions to brake-and-swerve, trajectory optimization and end-to-end top-down models. Each has different utility and needs different numbers of examples. His conclusion is that smart agents are critical at scale, since accurate models of human behavior help the car make better decisions, support robust simulation and treat the vehicle itself as one more agent.

Anguelov closes with a mental test for scalable AI systems. Each new city brings its own tail, from Paris intersections and Lombard Street to narrow European streets and Pittsburgh's famous left turn. His proposed process is to drive in a new environment for about 30 days or more, collect the data, and have the system improve on it. That requires a system that can be trained on the collected data.

He stresses that a system must be able to quantify, or elicit, whether it is incorrect or not confident, because then it can take action, such as asking human raters. Even better would be a system that updates itself, for example by checking the consistency of its beliefs and looking for explanations that agree with each other, so it can learn from unlabeled data. He flags this reasoning idea as his own theory, and says scalable training and testing infrastructure completes the virtuous cycle.

In the Q&A, he says realism in simulated sensor data is an open research problem, with some teams training on simulators and others learning mappings that make simulated scenes look real. On uncertainty, he mentions networks that regress their own uncertainty, sampling or dropout, and temporal consistency checks. Asked how soon self-driving will be everywhere, he says it will take a while, since logistics, tuning and testing are needed to make it safe in each environment.

On whether non-ML methods still matter, he says today's machine learning still makes errors, so it needs to be complemented by expert design, and that this need should shrink as ML improves. The hybrid approach is effort-intensive but is the right choice for rare cases where data is hard to collect. In his words, the team is not religious about it: they just need to solve the problem, and right now a hybrid is the right mix.