The Second 90%: What It Really Takes to Put Deep Learning on a Driverless Car

Arnoud opens with three goals: give context on the self-driving space, explain how deep learning grew inside Alphabet, and show where the techniques from the course fit into a real production system. He stresses that a self-driving car is a large industrial project, and that the algorithms are only one part of it. 2017, he notes, was a big year because Waymo became its own company that January.

He frames the motivation around safety, accessibility and efficiency. He cites that 94% of US crashes involve human error, often distraction, and points to the promise of cheaper, wider access to mobility and of getting commuting time back. Waymo's stated mission is to make it safe and easy to move people and things around.

The effort began in 2009 under a Google project called Chauffeur, before the deep-learning era in industry. Its first milestone was deliberately aggressive: ten 100-mile loops around Northern California, covering mountain roads with cliffs, busy highways, Lake Tahoe weather, bridges and dense city streets such as San Francisco's Lombard Street. The team completed all ten loops fully autonomously in 2010, which convinced Google to keep going.

For years Waymo and its peers relied on safety drivers who could take over at any moment. In November 2017 Waymo reached enough confidence to remove the safety driver, and Arnoud shows video of team members riding in the back of a Chrysler Pacifica with an empty driver's seat. The service started in a constrained area in Chandler, near Phoenix, with a growing fleet and a push to expand.

Arnoud's central message is that a demo that works in a lab is nowhere near done. When you are 90% finished, he says, you still have 90% to go, which means making the technology, the team size, the sensors and the testing practices ten times better. He returns to this idea repeatedly through the talk.

He then gives a behind-the-scenes view of deep learning at Google. The Google Brain team pushes research and builds the tools and infrastructure that let other teams use machine learning at scale, with effects visible in the assistant, Photos, speech recognition and Maps. Arnoud himself led an internal Street View project called Street Smart, which analyzed street imagery to extract street numbers, street names, business facades and traffic signs for better maps.

Finding house numbers sounds simple but is hard: the system must locate the number on a facade, tell it from other text, read it, and geolocate it. In 2012 this became the first deep-learning system in production across Alphabet, and in many countries more than 95% of addresses on the map were helped by it. A 2014 paper then showed deep learning running in the constrained, low-latency embedded environment of a car, starting with pedestrian detection.

The car is harder than Street View for several reasons. It has to run in real time with no dependence on a data-center connection, and it cannot choose when to take its pictures, so night driving and bad conditions have to be handled as they come. Even so, Arnoud says there has been steady cross-pollination between Street View work and the car, and close collaboration between Google Research and Waymo.

Arnoud lists the pieces of a self-driving stack: mapping and localization, perception and scene understanding, behavior prediction, motion planning, the robotics of sensors and controls, and simulation and validation. He focuses on perception, which builds a picture of the world from two inputs. One is prior knowledge, such as pre-computed road layouts, which saves onboard computation. The other is live sensor data about lights, pedestrians and cars.

The cars carry cameras, radar and lidar, which are placed and chosen to complement each other in coverage and in capability. Cameras give dense, semantically rich images but poor depth, while lidar gives good depth but little semantic detail. Because better sensors mean better perception, Waymo chose to design its own sensors in-house.

What separates a real self-driving system from basic driver assistance, Arnoud argues, is depth of understanding. Assistance features can often get by with not bumping into moving blobs, but with no driver aboard the car must anticipate. If a car beside you may swerve around a cyclist on the right, you need to decide whether to slow down or speed up to let it through.

He walks through a scene with a police car pulled over, a parked car, cones, a cyclist and an officer. The basics are labeling each object, but the car also needs to know that flashing lights make the police car an emergency vehicle, that the parked car is blocking the way, and that an open door means someone may step out and affect the cyclist's path. Only with that depth can the system produce realistic trajectory predictions to feed planning.

Real sensors are far from perfect, Arnoud says. Exhaust smoke can trigger laser points that are safe to ignore, and reflections can fool a naive detector: a car mirrored in a bus can look like a second car, and a road sign reflected in the window of the car ahead could look like an obstacle at 65 miles per hour. Handling these cases takes both signal processing, which already uses machine learning, and higher-level semantic reasoning.

Once the data is clean, the first step is usually convolutional layers, which learn features such as lines, contours and eventually whole objects. Most state-of-the-art convolutions are 2D and expect dense input, so sparse laser data is first projected into 2D views. Two common ones are the top-down view, similar to a map, and the driver's-eye view, which works better when you want to fuse lidar with camera pixels.

Segmentation groups points and pixels into objects, yet many road objects such as snow, vegetation or trash bags have no fixed shape. A small convolutional network slid across the whole projection works well for them, but Arnoud likens it to a dot-matrix printer: accurate but slow. For objects with strong shape priors, such as cars that are rectangles from above and tend to follow lane directions, single-shot approaches in the style of SSD predict boxes in a single pass and are much cheaper.

Many vehicles carry special meaning: an emergency vehicle may or may not be active, and a school bus with its lights on or stop sign open demands a particular response. Training a separate convolutional classifier for each case would work but be costly to run, since convolution layers are the expensive part of a network. Embeddings are a cheaper route.

An embedding is a learned vector that carries an object's semantic meaning, such as the fact that a vehicle is a school bus with its stop sign out. Arnoud explains the idea through word embeddings and the word2vec project at Google, where the vector for king minus man plus woman landed closest to queen. The car can then operate in a small vector space instead of re-processing images.

Pedestrians combine nearly every difficulty: they are deformable, appear in unusual poses, must be detected with very high recall, and can move in any direction, especially children. A single frame can also mislead. A mannequin-like figure lying in a pickup bed is not someone to brake hard for at 35 miles per hour, while a visually impaired person jaywalking is.

Telling those cases apart requires watching behavior over time. Arnoud's answer is to take the vector representations of objects and feed them into recurrent neural networks, which build up a state that gets richer with each observation, much as a sentence becomes clearer word by word. Embeddings plus recurrent networks are, he says, a common way to get that temporal understanding.

Algorithms were a breakthrough, Arnoud says, but running them at scale requires much more. The first thing to 10x is labeling: most of the methods described are supervised, so they need large, high-quality labeled datasets. In the early street-number days it took a multi-billion-label dataset to teach the network, whereas today Waymo needs less data and creates it more efficiently.

Modern pipelines let models propose labels, have human operators correct mistakes, and use active learning so people do not label everything from scratch. Even so, he says robust solutions still need datasets in the millions of labels. Research into unsupervised learning is active at Waymo and Google, but most production work today remains supervised.

Compute is the second piece. Arnoud compares the neuron counts and connections of the early street-number models with the human brain: still orders of magnitude away, but competitive on narrow tasks. Training and running models in real time takes heavy engineering, and Waymo benefits from Google's infrastructure.

TensorFlow, he explains, is more than a programming language for network architectures. It is a shared language for teams, a data representation for labels and training batches, and a runtime that can be deployed to Google data centers. It also ties into hardware accelerators, from CPUs to GPUs to Google's own tensor processing units, which Google deploys in its data centers to train and run deep-learning models more efficiently.

Waymo's testing program has three legs: real-world driving, simulation and structured testing. The real-world leg is accelerating: three million miles by May 2017 and four million by November. Those miles span 20 cities and many weather conditions, equal to about 60 trips around the globe and, Arnoud estimates, roughly 300 years of human driving experience.

Because every software revision cannot be re-driven on the road, simulation replays the logged miles against each new version. An internal tool called CarCraft lets engineers tweak a driven scene, for example with a different speed, an extra car or a pedestrian stepping out, so one real mile becomes many test cases. Using Google's infrastructure, Waymo can run a virtual fleet of 25,000 cars around the clock.

Last year alone that virtual fleet drove 2.5 billion miles, about three orders of magnitude more than the real total. For the rare situations that remain in the long tail, Waymo built a 90-acre testing facility on a former Air Force base in California, with traffic lights and railroad crossings. There the team stages scenarios such as a cab backing up toward the car or people dropping boxes in the road, then feeds the sensor data back into simulation.

Arnoud sees two main directions ahead. One is growing the operating design domain: more geography, urban cores and different weather, with Waymo announcing expanded testing in San Francisco and its slopes and fog. The other is deeper semantic understanding, illustrated by a chaotic Paris roundabout that he handles by relying on expectations about people and on visual gestures.

Answering a question about perception errors reaching the planner, he says the stack has a clean layer between perception and planning. Perception is measured on real miles, and its realistic mistakes can then be replayed in new scenarios to test the planner. On architecture choices, he says the main factor is the ability to grow the engineering team productively, and that in the end the robustness of testing is the judge, not a philosophical debate.

Asked whether the car relies on a library of every situation, he rejects the idea: hard-coding five scenarios would fail on the sixth, which he compares to overfitting. The system should learn core principles of safe driving, and simulation variations exist to add diversity to that learning. On snow, he says it makes no sense to predict behavior for every flake, so the system must group what it sees into semantic objects.

On adversarial attacks, Arnoud notes that real life already produces confusing cases such as reflections, and the defense is the same in both. Sensors and systems that fail differently provide redundancy, and semantic understanding lets the car know that you cannot see through a bus to a real car behind it.