Teaching Cars to Watch the Driver: Deep Learning for Semi-Autonomous Vehicles

Fridman opens by turning the camera back onto the human. Most deep learning work in driving concerns perception of the outside world, such as pedestrians, lanes and steering, but he calls the human side "fascinating and severely understudied". His group has cameras in 17 Teslas driving around Cambridge, because Tesla is one of the few vehicles that lets researchers observe the real interaction between a human and a semi-autonomous machine. What deep learning lacks, he says, is video of drivers, and his work looks at billions of frames of people on the highway at 60 miles an hour or more.

Trust is the first reason to care. Fridman points out that almost every car on the road has only a pressure sensor on the wheel or seat, so the car knows almost nothing about the person inside. He asks how a car driving itself at 70 miles an hour can build trust with someone it does not perceive. That is why he keeps advocating a driver-facing camera in every car, arguing that the safety and trust benefits outweigh privacy concerns, which he compares to the camera already on a phone.

Fridman explains why body pose matters. Seatbelts and passive safety systems are designed with crash-test dummies that assume certain body shapes and an optimal seated position. When a Tesla drives itself, people move around, for example reaching into the back seat for a phone in a bag, and that is when crashes happen. To know how often that occurs, and to prepare for the critical moment of impact, the car needs to know the person's pose.

The method is a convolutional neural network used as a regressor that outputs an XY position for each point, such as the left and right shoulder. A cascade of regressors yields the whole skeleton, and the prediction is made in every frame. Fridman describes two ways to use time: apply physical constraints, since an arm moves smoothly, and minimize the temporal error from frame to frame, or stack all the frames like image channels into a 3D convolutional network that estimates pose in all frames at once.

Gaze classification is a simple classification problem. Five cameras in the Tesla cover the road, the instrument cluster, the center stack, the steering wheel and the face. Raw face pixels go into a CNN with six classes, which are forward roadway, left, right, center stack, instrument cluster and rearview mirror. With millions of frames per class, Fridman says, it does "incredibly well" at predicting where the driver is looking.

The same pipeline serves most face-related driver-state problems, including emotion, drowsiness and frustration. In-the-wild data has vibration and sudden light changes, so the first steps are video stabilization with classic non-neural algorithms and automatic camera calibration. Next the network is personalized to the person and the car through transfer learning, from a background model that works for everyone. Face frontalization then places the eyes and nose at fixed positions so subtle eye movements can be studied without interference from head motion.

For emotion, Fridman describes a study where people used a good or a deliberately bad voice-based navigation system and rated their frustration from one to ten, which gave ground truth for training. A surprising finding was that smiling was a strong indicator of frustration. The pipeline is to detect the face, find landmarks, align and frontalize, and classify the raw pixels with an SVM or, more commonly now, a CNN. Landmark alignment is where CNNs still struggle to compete, because cascaded regressors can exploit the known shape of a face.

Fridman sees this as a step toward unsupervised learning, since the less humans have to annotate, the more powerful the algorithms become. Vision struggles with occlusion, from an arm or from glaring light, and he says these are cases where the machine should say "help me out with this" and pass the image to a human. In driving, over 90 percent of the time the driver is staring at the road, and a model that has seen that face for hundreds of millions of frames can annotate those frames itself. The human is needed at the transitions, for example when the person starts looking toward the rearview mirror, which can be detected using optical flow.

On a plot of fraction of frames the human annotates against accuracy, gaze classification retains strong accuracy while needing far fewer human labels, which Fridman describes as roughly an 84-fold reduction. He calls this the future of machine learning, hopefully leading to no human annotation at all. For frustration, he shows that the frustrated driver moves eyebrows and smiles a lot, while the satisfied driver is cold and stoic, "because driving is a boring experience and you want it to stay that way".

Students raise two points: emotions may be performed for passengers rather than felt, and solo driving differs from driving with others. Fridman agrees and notes that in naturalistic data, unlike the lab study, they do not know how people really feel. Easy things like singing can be found by tracking the mouth, but subtle emotion when someone is alone is hard to detect. The deeper difficulty, he says, is that 99.9% of driving is the same, so finding the interesting moments is a hard problem for both outward- and inward-facing sensors.

Cognitive load, also called mental workload, has better science and measurement methods than emotion. Fridman describes the eyes as the window to the workload. They move in saccades, ballistic jumps seen when looking around or reading, in smooth pursuit when tracking an object, and in micro saccades, tiny tremors that are almost imperceptible to computer vision. He mentions work by Bill Freeman's group that magnifies such subtle movements, filmed at 500 frames a second.

Pupil dilation signals high cognitive load but also occurs when light is dim, so it cannot be relied on in a car even though it works in the lab. Blink rate drops and blink duration shortens under high load, which gives another cue. Fridman's system therefore takes a sequence of 90 grayscale frames of the eye, about six seconds, and feeds it into a 3D CNN as 90 channels. It predicts low, medium or high load, with ground truth from over 500 people doing tasks of different difficulty.

Before that step, the face is frontalized so the eye corners stay fixed, and active appearance models find 39 points on the eyelids, iris and pupil. Fridman says the code is freely available and that a webcam stream is enough, since the CNN runs faster than real time. He adds that all these driver-state tasks need very little hyperparameter tuning, but they require huge amounts of data.

Fridman argues that the road to full automation will be gradual. He says this is manual-control, pickup-truck country, and people like to keep control, so handing it to a machine is a slow process of the machine earning trust. During that process, carmakers such as Tesla, BMW, Mercedes and Volvo will need to see what the human is doing. Since there are billions of miles of forward-facing data, he says billions of miles of driver-facing data are needed too, and he pitches automakers and buyers on cars with driver-facing cameras.

He closes with a broader point about how early we are in understanding deep learning. Deeper networks often give better results even without more data, and no one fully understands why. He compares neurons to cells in Conway's Game of Life, where each cell follows a simple local rule yet complex patterns emerge, just as networks optimize through local gradients. In both cases, he says, we do not know why complex structure appears, and we need to in order to understand how such networks might reason.

For further learning, he recommends the online deep learning book, the daily arXiv papers, the awesome deep learning papers list on GitHub, and blogs. He also invites interested students to do automotive research in his group. The lecture ends with the winners of the course's driving-game competition, which drew over 2,000 submissions. Fridman had thought a speed of 73 was unbeatable, and even a deterministic algorithm that "cheated" only reached 74, yet contestants beat both. The three winners received a free term in the Udacity self-driving car engineering degree.