Data Is Everything: Teaching Self-Driving Cars to Understand the Human Behind the Wheel

Fridman opens by narrowing the scope to computer vision applied to human beings in the context of the car, not celebrity face recognition but systems that work in the real world. Ranked from most to least critical, he says the ingredients are data, annotation, hardware and only then algorithms. Real-world data comes first, and he promises to repeat that "data is everything" throughout the lecture.

Collecting raw video, audio and lidar is not enough, because driving looks the same 99% of the time and the interesting 1% is what teaches a system. Teams must record everything and then find representative cases through automated or semi-automated means. Annotation tooling is also a design and human-computer-interaction problem rather than a deep learning one, since each task, from glance classification to body pose to full-scene segmentation, needs its own tool.

On the hardware side, the group now has more than five billion driving images, which cannot be handled on a single machine and needs distributed compute and storage. The algorithms come last, and he notes that self-calibrating networks are nice but that most image methods look at single frames while the world unfolds in space and time. His takeaway is that the painful, boring work of collecting, cleaning and annotating data beats a great algorithm.

Before cataloguing flaws, Fridman pauses to say humans are amazing. In 2016 the US passed forty thousand road deaths, but that was over 3.2 trillion miles, about one fatality per 80 million miles and roughly a one-in-625 lifetime chance of dying in a crash. He adds that Massachusetts has the lowest death rate and Montana the highest, which puts complaints about Boston drivers in perspective.

The flaws are real, though. He cites distraction as a cause of about 3,000 deaths and 400,000 injuries in 2014, and notes that texting takes eyes off the road for about five seconds on average. At 55 miles an hour that is the length of a football field. Drunk driving is involved in 31% of traffic fatalities and drowsy driving in nearly 3%, and he plays naturalistic footage of distracted teenagers.

This frames his central choice: remove the imperfect human and chase full autonomy, or have human and machine work together. He argues the second path is grounded in common sense, since people over-trust automation and misuse it, as in the example of shoving an orange into the steering wheel to fool the touch sensor. But he thinks that gives the technology too little credit: if an AI system can perceive the human, it can also work with them.

Fridman's group instruments 25 vehicles, 21 of them with Tesla Autopilot, with cameras on the driver's face, on the body and on the road scene, all synchronized with GPS, audio and CAN data. The face camera supplies glance classification, emotion and cognitive load, the body camera supplies pose and hands-on-wheel activity, and the road camera supports scene perception. The data covers more than a thousand miles a day and over five billion frames.

He says autonomy has to be judged on enjoyment as well as safety, since a car that never moves is nearly perfectly safe and worthless. In this data, people drive about 33% of miles on Autopilot, which signals real value. Glance classification shows the on-road and off-road distribution is not significantly different between manual and automated driving, at least in aggregate.

Fridman reads this as the sweet spot for human-robot interaction: drivers explore the system's limits through experience, find its limitations and still stay physically in position to take over. He cautions that this is an aggregate view and that the fraction of automated driving may rise from 30% to 50%. The architectures used throughout are standard: CNNs on raw pixels, 3D CNNs for temporal dynamics and recurrent networks on top, and he stresses that they rely fundamentally on real-world data.

Fridman arranges human-sensing tasks from easy to hard. Detecting whole bodies, faces and head pose are comparatively easy, while anything fine-grained, such as eye tremors, pupil diameter, blink dynamics and cognitive load, is far harder. Pedestrian detection sits at the easy end but still faces variation in appearance, flexible articulation, accessories and occlusion in crowds.

He traces the methods from sliding windows with hand-built features such as HOG and cascades, which are extremely inefficient. They gave way to R-CNN and its faster variants, which propose a small set of likely candidates, classify each with a CNN and use non-maximum suppression to pick the best bounding box. Mask R-CNN adds segmentation, and approaches like VoxelNet extend the idea to lidar point clouds.

On data, his group is part of an instrumented Cambridge intersection recording about ten hours a day with 4K and stereo cameras, a 360 camera and lidar. About 12,000 pedestrians cross it daily, and many jaywalk. Detecting them turns raw video into hours of crossing decisions, which lets researchers study what he calls the nonverbal negotiation between pedestrians and drivers.

Body pose estimation locates joints such as shoulders, elbows, hips and knees, and Fridman explains why it matters in the car. Safety testing assumes a standard dummy position, but with more automation drivers drift out of it, so upper-body pose shows how often they do. It can also feed hands-on-wheel and smartphone detection, and add context to glance estimation.

Older methods detected the head and then stepped through shoulders, elbows and hands in sequence. The more successful holistic approach regresses all body parts from the whole image using cascades of convolutional regressors that zoom in for finer positions. Part affinity fields and bipartite matching then connect the detected parts into individual people, which copes with occlusion because not every joint must be visible.

His group uses the approach on the driver's neck point to track slouching and sinking into the seat during 30-minute periods of Autopilot. Applied from the vehicle's viewpoint, it lets them plot vehicle speed against whether a pedestrian is looking at the car. They see thousands of crossings, and most pedestrians look away before they cross.

Fridman says the most valuable thing for a car to know is where the driver is looking. Glance classification picks regions, either on-road versus off-road or six regions including left, right, center stack, rearview mirror and instrument cluster, rather than the geometric gaze estimate. The distinction matters because every human-sensing problem must be learned from data to survive lighting and behavior variation.

Geometric gaze estimation has no ground truth in the real world, so you can only infer where someone looked. A region label, by contrast, can be annotated by people and learned by a network. The pipeline takes 30-frames-per-second face video, calibrates roughly where the camera is, stabilizes the video and aligns the face, and then feeds raw pixels to the network with only the output changing between tasks.

Annotating 1.5 billion face frames would cost tens of millions of dollars, so the group has humans label only the frames the classifier is not confident about. There is a trade-off between accuracy and the share of frames the network can label alone: low-90s accuracy on the six-region task, say, for all frames. In response to a question, he admits confident-but-wrong predictions have no clean answer beyond more data on hard cases, plus manual annotation of each new person.

He stresses how little cars know about their drivers today: most vehicles only know whether a hand is on the steering wheel. GM Super Cruise and a newly added Tesla driver camera are first steps. He calls that information common sense for a car to have, and says the move from gaze estimation to classification, after decades of work, is what makes it a machine learning problem.

Fridman, in his words an expert at being human rather than in emotion, notes there are many ways to define and categorize emotion. He separates general emotion recognition, which detects broad categories such as joy, anger, disgust and surprise, from application-specific recognition. For the general case he points to the Affectiva SDK and the roughly 42 facial muscles that combine into expressions.

Using joy as the illustration, a smile raises the likelihood of joy, while brow raising and furrowing lower it. For the specific case, the group collected drivers using voice-based navigation, most of whom found the experience frustrating, and asked them to rate frustration from 1 to 10 afterward. Those self-reports served as the annotation.

The result was surprising: smiling was the strongest and most consistent expression among frustrated subjects, followed by frowning and head shaking. Fridman jokes that perhaps they enjoyed the absurdity of the moment. The practical lesson is that the generic emotion mapping and the task-specific one differ sharply, and that the data does the work, not the algorithm.

Cognitive load is how hard someone is working mentally, and Fridman treats the eyes as its window. Pupils dilate with cognitive load but also with lighting, and eyes move in fast saccades or in smooth pursuit. In the lab, pupil diameter measured with infrared cameras is the standard non-contact method, but in real driving with variable light all that remains is blink dynamics and eye movement.

The group's answer is a 3D convolutional network, which convolves across multiple frames as well as spatially and so learns dynamics through time. Ninety-two drivers performed the n-back task on the highway, hearing numbers and repeating them from zero, one or two steps back. The input is the eye region from about six seconds of video, 90 images at 15 frames per second, and the output is one of three classes.

In aggregate, the eye moves much less under higher load and stays more focused on the forward road. The network classifies at 86% accuracy from real-world data. Since each class comes with a confidence, the output becomes a continuous 0 to 2 score, and Fridman shows a plot of team members' load rising during conversation and falling when everyone is silent.

Fridman argues that the perception problem is far from solved, which he puts at more than 20 years from complete resolution, so the human must stay involved. When perception or driving policy fails, the car must know what the human is doing. The most popular car in the US is the Ford F-150, with no automation, while the vehicles actually putting automation into daily life are the Level 2 systems such as Tesla, Super Cruise and Audi.

He sees those systems as teaching humans how to interact with automation and as the path to mass-scale autonomy, with steering wheels removed more than two decades away. They also generate training data: his group has 400,000 miles of driving, and he says Tesla has a billion. Edge cases arise there, so this is the stairway to full automation.

His broader point is that a self-driving car is a personal robot rather than a Roomba-like tool, because a person hands their life to it one-on-one. That relationship requires communication, trust and an almost one-to-one understanding of each other's mental state. He closes by playing a Good Will Hunting clip about imperfections, and invites students to the group's follow-on courses and competitions.