Is Driving More Like Chess or Conversation? Lex Fridman's Opening Lecture on Deep Learning for Self-Driving Cars

Lex Fridman welcomes students to 6.S094, Deep Learning for Self-Driving Cars, explaining that the course uses autonomous driving as a guiding case study for deep neural networks. He introduces his teaching assistants and points students to the course website, an email address for questions, and lecture slides and videos posted after each session. The class is aimed at anyone from beginners in programming, machine learning or robotics to experts who want to revisit the basics.

The first project, DeepTraffic, is a seven-lane top-down simulation where a red car is controlled by a neural network that learns in the browser using ConvNetJS, a JavaScript library written by Andrej Karpathy. To get course credit, students must design a network that drives just above the 65 mph speed limit, and winning the competition requires going faster. The only requirement is a Chrome browser.

The second project, DeepTesla, uses forward-roadway data from a Tesla and end-to-end learning. A convolutional neural network takes a single image and predicts a steering angle, and students try to match or beat the car's own steering. The course will cover deep reinforcement learning, convolutional and recurrent networks and how they apply to perception, localization, mapping, control planning and driver-state detection.

Fridman frames the course around the question of what intelligence is, showing a 1996 Time magazine cover asking whether machines can think. He separates a special-purpose version, where a system achieves a formally defined set of goals in a well-defined environment, from a general-purpose version with poorly defined goals, actions and rewards, which is what everyday human life involves. We have good algorithms for the first, and the open question is whether inching along that path leads to the second.

He sorts tasks into three groups: formal tasks like board games and theorem proving, expert tasks like medical diagnosis and scheduling where data-driven methods are producing breakthroughs, and the mundane tasks humans find trivial, such as speech, vision and walking. Driving could be formalized like chess, with lanes, obstacles and rules of the road, or it could require something closer to conversation, with higher-level reasoning and communication. That question, he says, frames the rest of the course.

Fridman lists the available sensors: radar and lidar for obstacles, cameras and stereo vision for texture and classification, GPS and IMU for the vehicle's trajectory, and the CAN network for the state of the vehicle's systems. He highlights less-studied inputs, such as the sound of wet roads and screeching tires, and his own group's focus on driver-facing sensing, like whether the driver is looking, sleepy or even in the seat.

The driving task breaks into modules: localization and mapping, scene understanding, movement planning, and monitoring the human on board. That last piece matters because the path to full autonomy may pass through semi-autonomous vehicles, which must hand control to the human and back, hence the course's focus on Tesla.

On history, he points to Stanford's Stanley winning the second DARPA Grand Challenge as one of five finishers in a desert race, and Carnegie Mellon's Boss with GM winning the 2007 Urban Challenge. That work moved into industry, with Google, now Waymo, Tesla's Autopilot, Uber testing in Pittsburgh, and nuTonomy in Boston. He notes that Google's approach in effect reduces the world to a chess game using extremely accurate sensors and 3D maps.

Fridman asks what it would take if driving were more like natural conversation, and uses the Turing Test as a lens. Passing it would need language processing, knowledge representation and, the hard piece, automated reasoning. Neural networks are good at learning from data, but there is no good mechanism yet for reasoning.

He wonders whether reasoning is something special or simply learning from data with a bigger network, and says a totally different mechanism might be needed. In response to student questions, he says the driving video shown was likely from Tokyo, and that countries such as India and some in Europe lean toward the natural-language end of the scale, while US roads, markings and rule-following lean toward chess.

Fridman introduces the artificial neuron as a crude inspiration from the brain rather than a model of it. He estimates that a human neuron has about 10,000 outgoing connections, that the largest artificial network at the time had about 10 billion, and that the brain has roughly 10,000 times more, or 100 to 1000 trillion synapses. An artificial neuron weights its inputs, sums them with a bias and squashes the result into a zero-to-one output.

A single neuron works as a linear classifier, learning to draw a line between two groups of points. The training loop initializes weights, computes outputs, adjusts the weights when the output differs from the ground truth, and repeats until the perceptron makes no more mistakes.

He highlights universality: a network with one hidden layer can closely approximate any function, and since driving can be viewed as a function from the world to vehicle control, a network exists that can drive perfectly. He illustrates supervised learning with a house-price example that takes bedrooms, square footage and neighborhood as inputs, trained with backpropagation on known prices.

Fridman shows Andrej Karpathy's Pong agent, a policy network that takes raw pixels, passes them through a few hidden layers, and outputs the probability of moving up. There is no supervised signal about any single move. Instead every action in a won game is rewarded and every action in a lost game punished, and after about 200,000 simulated games, taking around three days on a regular computer, it beats the built-in opponent.

Answering a student, he defines supervised learning as learning from labeled examples, reinforcement learning as having sparse ground truth that arrives at the end of a game, and unsupervised learning as having no information about correct outputs. Unsupervised learning is where the community's excitement lies, he says, but it had not yet produced major breakthroughs. The learning mechanism generalizes to other worlds, but only to worlds that can be simulated well.

The drawback is efficiency: a human grasps Pong from a short instruction because they arrive with concepts of control, paddles and physics, and Fridman calls transferring that knowledge reasoning. He asks whether reasoning might simply be simulating the world in your mind and learning much more efficiently than 200,000 games. Deep networks also need big, costly annotated datasets.

Beyond data, networks require many choices: number of layers, hidden nodes, activation functions, learning rate, loss function, training iterations and optimizer. Fridman says you cannot simply drop a network onto a problem and expect it to work. Defining a good loss function, or a reward function for games, is especially difficult.

He illustrates with an OpenAI result on the boat-racing game CoastRunners. The agent learned that collecting turbo pickups scored more points than finishing the race, so it circled endlessly, even while on fire and hitting walls, reaching a local optimum for points while ignoring the goal humans understand. The point is how hard it is to encode a formal utility function.

For driving, he says the same issue arises in near-crash decisions, such as protecting the driver versus other people. There is an implied reward function in the data and the learning even if nobody acknowledges it, and we may only discover it is badly designed once we see it in action. That is what makes it frightening.

Fridman credits the recent breakthroughs to compute (Moore's Law, GPUs and ASICs), digitized and organized datasets like ImageNet, algorithmic innovations such as backpropagation, CNNs and LSTMs, and infrastructure including Git, ROS, TensorFlow, Amazon Mechanical Turk, cloud hosting and corporate funding. But he says nothing fundamental has changed: neural networks date to the 1960s, convolutional networks to the 1990s, and compute with its roughly hundredfold gain per decade has been the workhorse.

He calls "deep learning" a PR term for neural networks with many layers and lists terms used in the course, from MLPs and RNNs to LSTMs and CNNs. Answering questions, he says we understand the inputs and outputs of a network but not what each layer does for a task like driving, and that the architecture is fixed in advance, unlike a brain whose neurons are born and die. He cautions that getting a network to do something amazing quickly makes you feel like an expert, while in truth we do not understand why it works so well.

The core idea is representation learning. Instead of experts hand-specifying features, a network learns edges in the first layer, contours and corners in the next, then object parts and finally labels. A neural network can, for instance, find a remapping such as polar coordinates that makes a hard dataset linearly separable, and it can discover tens of thousands of features, which is why it needs only inputs and outputs but is hungry for data.

Fridman walks through ImageNet classification, where error fell from about 16% with AlexNet in 2012, versus roughly 40% for earlier traditional methods, to under 4%. Human-level error is about 4%, and he says convolutional networks surpassed humans for the first time in 2015. He stresses why this matters: vision is hard because of lighting variability, pose variability, occlusion and variation within a class, and jokes about his favorite figure caption in a paper, "deformable and truncated cat."

Convolutional layers preserve spatial information, and removing the end of the network can produce a heat map for segmentation, locating, for example, the woman and the horse in an image. For object detection, region-proposal approaches like R-CNN reduce the cost of the traditional sliding window. Deeper networks such as deep residual nets, with VGG-19 and 34-layer versions as examples, offer more representational power and accuracy but need more data.

Other image-to-image applications include colorizing black-and-white film, trained by converting modern color films to grayscale, and real-time translation of text on images, as in Google Translate, which detects letters, translates them and maps the result back onto the picture.

Fridman contrasts vanilla and convolutional networks, which map one input to one output, with recurrent networks, which map sequences to sequences across language, video and other series data. Examples include generating handwriting from typed text and character-level RNNs trained on Wikipedia that produce plausible text, even Markdown and LaTeX structure, one character at a time. Sentence completions like "the meaning of life is literary recognition" got laughs from the class.

RNNs also power image caption generation, image question answering, and video description, which still makes errors such as describing a small bus running into a building. He adds that some systems steer a small "fovea" of attention around an image, which matters for drones that must decide at hundreds of frames per second, and he mentions MIT work that maps silent video of a drumstick to the sound it makes.

Fridman plays a video of robots struggling at basic walking and manipulation in the DARPA Robotics Challenge, noting that teams said the hardest task was getting out of the car. Tasks like soccer are much harder for machines than chess, and a phone can run a chess engine that beats the best players. So the question is where driving falls, and he invokes Moravec's paradox: the human sensory-motor system encodes billions of years of experience, while abstract thought is under 100,000 years old.

The stakes are high because there is about one fatality per 100 million miles of driving, which he describes as a margin of error of roughly 0.000001%, and cars must handle snow, heavy rain, parking garages, unpredictable pedestrians and lighting that blinds cameras. In reply to a student, he notes cars have become much better at surviving crashes, but even one fatality is too many. Google's disengagement report showed 341 times between 2014 and 2015 that the driver had to take control on the sunny roads of San Francisco.

Fridman presents work showing deep neural networks are easily fooled: images of noise classified as robin, cheetah, or penguin with over 99.6% confidence, and small added distortions making networks predict ostrich for correct images. He also mentions research on replay attacks that spoof lidar so a car sees people or cars that are not there. Robustness remains a major open question.

He recalls the 1958 New York Times report on Frank Rosenblatt's perceptron, which predicted machines that could walk, talk, see and be conscious, and the 1973 Lighthill Report, which said the field had not delivered the promised impact. Perceptrons turned out to be linear classifiers, and such gaps between hype and reality contributed to AI winters in the 1970s and from the late 1980s into the early 1990s. He says we are at the peak of a hype cycle and must proceed with caution.

He lists research directions: smaller, cheaper mobile deployment, unsupervised and reinforcement learning, video, multimodal learning and, notably, making money from these technologies. He closes with a tour of libraries, from TensorFlow and Keras to Torch, Theano, MXNet, Neon, Caffe and CNTK, and recommends ConvNetJS and Keras.js for in-browser learning in the course. Tomorrow's lecture will explain the deep reinforcement learning game and competition.