From Pixels to Steering Wheels: Convolutional Networks and End-to-End Driving

Fridman opens by previewing the lecture: convolutional neural networks for images, a case study on detecting traffic light colors as a three-class problem, and DeepTesla, a browser-based network that learns to steer from video of the forward road. A TensorFlow tutorial is planned for those who want bigger networks trained offline on GPUs. He contrasts images with the small inputs of last lecture, where networks had on the order of 10 to 1,000 neurons.

He stresses that computer vision today is dominated by data-driven machine learning, because pixels are just numbers from 0 to 255. The supervised pipeline is the same as before: raw images with labels go in, the algorithm extracts features and trains, and the resulting model is tested and scored by accuracy. Even the Atari-playing agents from the previous lecture use convolutional networks.

Humans convert visual perception into semantics effortlessly, but a computer sees only RGB values, three per pixel. From there Fridman defines two problem types. Regression produces a real value, such as a steering wheel angle from a road image, while classification produces a discrete label, usually by thresholding class probabilities.

Fridman lists the reasons computer vision is difficult, even though humans take it for granted. Viewpoint changes make the same object look entirely different in pixel terms, and scale, occlusion, deformation and background clutter add to the problem. Variation within a class can exceed variation between classes, and illumination, the way light reflects off surfaces, is especially hard for driving.

Learning a function from images to categories requires a lot of labeled data, and the available datasets, with millions rather than billions of images, are still relatively small. He walks through the common ones: MNIST for handwritten digits, ImageNet with a hierarchy of categories from WordNet, CIFAR-10 and CIFAR-100 as tiny images for quickly showing that an algorithm works, and Places for natural scenes.

To build a first CIFAR-10 classifier, Fridman starts with a way to compare images: the pixel-wise absolute difference summed into a single number, known as L1 distance. A classifier can compare a query image to every image in the dataset and take the category of the closest one. Random guessing gets 10%, and this nearest-neighbor approach reaches 38%, which is well above chance.

This is K-nearest neighbors, and K is a parameter that has to be tuned, which in neural networks is called hyperparameter tuning. With five neighbors, ties can leave regions undecided, and a plot of K from 0 to 100 shows that the best value for this dataset is 7. The standard discipline is to split off a test set that you never touch, while using cross-validation folds from the training data to tune parameters.

Fridman compares the results to people: humans reach about 94% on CIFAR-10's tiny images, while the best convolutional networks have reached 95.4%, which he calls a recent and critical surpassing of human performance. In answer to a student, he says neural networks almost always beat nearest neighbors, except when you have almost no data.

Fridman reminds the audience that everything still reduces to the neuron, which sums weighted inputs, adds a bias and applies an activation, trained with a forward pass, back-propagation and a smooth loss such as squared error. A fully connected network on a 28x28 digit uses 784 input neurons, so he frames convolutional networks as the same thing with some tricks up front to deal with the size of images.

A convolutional layer takes a 3D volume (height, width, depth) and produces another 3D volume. Each neuron has a narrow receptive field, looking only at a nearby region, and weights are tied so that the same filter slides across the image. A filter that fires for an edge will therefore fire wherever the edge appears, which assumes a good cat feature matters equally in any location and removes a huge number of connections.

The output is shaped by three choices: the number of filters, which sets the depth, the stride of the sliding step, and padding with zeros so that input and output sizes can match. Pooling, such as max pooling, then shrinks layers by summarizing a region with one number. It has no learnable parameters and discards resolution, but improves efficiency and memory, and how much to pool is a debated tuning choice.

A whole network stacks convolution and pooling layers, followed by a few fully connected layers that end in one neuron per class, which is how the CIFAR-10 and ImageNet results are achieved. Chop off the fully connected layers and the network can map an image to another image, giving a per-pixel segmentation, or you can classify candidate regions one by one for object detection.

Turning to driving, Fridman notes that an American driver covers about 10,000 miles a year and that there are around 30,000 fatal crashes and 32,000 or more fatalities annually. He points out that people are attached to being in control, and he calls the Tesla the car at the intersection of the Ford F-150 and the small Google car. It is fast and feels controllable, yet it can drive itself for hundreds of miles on a highway, which makes the handover of trust a chance to study human-machine psychology.

Human distraction offers the opening for automation: he cites texting as taking eyes off the road for about five seconds, and the four Ds (drunk, drugged, distracted, drowsy). He quotes figures of roughly one fatality per 90 million miles for human-driven vehicles versus a single fatality in about 300 million Autopilot miles as of December 2016. He admits there are many ways to pull that statistic apart, and says the real problem is a lack of data.

MIT's research collects that data from 17 instrumented Teslas, with over 5,000 hours and 70,000 miles recorded, including two cameras on the driver. The cameras are inexpensive Logitech C920 webcams, chosen because their onboard compression allows huge amounts of data to be stored. Fridman says one dataset shows about 300 million of 1.3 billion Tesla miles driven on Autopilot, and that people engage it for hundreds of miles at a time.

When a student argues that the one-fatality figure is misleading because drivers only engage Autopilot when it seems safe, Fridman agrees there are bigger arguments against the number. He adds that this selective use may not be a bad thing, since people with their lives at stake engage the system only after building a relationship with it.

Fridman breaks the self-driving task into four parts: localization (where am I), scene understanding, movement planning, and driver state. Sensors facing outward include radar, lidar, vision and audio, while inward-facing ones include visible-light and infrared cameras. Lidar provides a 3D point cloud and the least noisy 3D information, but it struggles in rain and snow, which change the reflective texture of surfaces, whereas computer vision can follow lanes and the car ahead because roads are designed for human eyes.

For localization, the classical route is visual odometry and SLAM: tracking distinctive features such as SIFT across frames, with stereo depth, GPS and an IMU as extra signals. The end-to-end alternative feeds stereo images into a network that outputs the trajectory, trained against a more accurate sensor like lidar. He says it works pretty well but not well enough, and that SLAM is one place where deep learning has not outperformed earlier methods.

Deep learning does help with scene understanding, where convolutional networks can replace hand-engineered features. Fridman shows SegNet turning a single monocular frame into a pixel-level map separating road, trees, pedestrians and cars. He adds that a recurrent network can read an audio spectrogram as an image to tell wet roads from dry ones, which matters when it is not raining but the road is still wet.

For movement planning, he says, the same machinery used in the DeepTraffic game can be applied to planning trajectories at high speed, either by optimization or by reinforcement learning across many millions of simulated runs. This links to work by guest speakers Sertac Karaman and Chris Gerdes.

Driver state means analyzing video of the driver's body, face and eyes. Larger, easier signals include body and head pose, while pupil diameter and micro-saccades, eye tremors at one-millisecond scale, are harder. From head and eye pose one can determine gaze, and the team simplified that into a classification across six regions: left, right, rear-view mirror, instrument cluster, center stack and forward roadway.

The goal there is to know whether the driver is watching the road or a phone in their lap, not the exact x, y, z position of their gaze. Emotion is similarly reduced to a binary frustrated-versus-satisfied problem, using drivers interacting with a voice navigation system.

The ground truth came from self-reports on a scale of 1 to 10. One driver reported a 1, satisfied, with a cold, stoic face, while another reported a 9 and was smiling. Fridman draws the lesson that human intuition about labels can be wrong, so engineers should trust the best available ground truth, a point he says is a big challenge in affective computing.

If driving is like a conversation, Fridman says, end-to-end driving is like skipping the components of a Turing Test and treating it as direct language generation: sensors go in, vehicle controls come out, and a neural network replaces everything in the middle. The data problem is much easier than it looks, because ordinary drivers produce ground truth every day by steering to stay alive. Recorded manual driving can therefore train an algorithm to control a self-driving vehicle.

He describes Nvidia's early work using a simple convolutional network that takes a forward-road image and outputs a steering command. It successfully navigated basic turns, curves and sharp turns at intersections. The input is a cropped 66x200 RGB image, followed by a few convolutional and fully connected layers, and it is a regression network predicting how to turn the wheel.

DeepTesla, the class project, runs a tiny version of this in the browser with ConvNetJS on real driving video. It outputs a steering value between -20 and 20 degrees, using 10 public highway clips, half driven by Autopilot and half by humans, with wheel values from the CAN bus synchronized to video. The interface shows the loss, the layer features and the network's output in white against the driver's value in blue, and the TensorFlow version marks agreement in green and disagreement in red.

Asked how much data is needed, Fridman says neural networks are good memorizers, so every case of interest must be represented in training. For cats versus dogs, roughly a thousand of each already works well. Driving has two difficulties: most of the time it looks the same while the cases that matter are the unusual ones, and the accuracy required is very high.

Networks are not good at generalizing from the common case to outliers, he says. Surviving thousands of highway hours does not mean avoiding a crash when someone runs in front of the car, and unlike a cat-dog mistake, a steering error puts lives at stake.

He closes by noting some tools in the project. One is a clever barcode, designed by a teammate, that embeds the steering value at the top of the video so that data streams stay synchronized even when shuffled during training. The final tutorial is a notebook on traffic light classification, where students build a network that fires for red, yellow or green.