Steering Through Time: How Recurrent Networks and LSTMs Learn from Sequences

Fridman opens by placing recurrent neural networks in the course so far: fully connected networks, convolutional networks for images, and deep reinforcement learning for systems that must act and collect reward. He calls RNNs perhaps the least understood but most exciting flavor of neural network. He also covers course logistics, including the requirement that DeepTraffic networks drive faster than 65 mph.

A vanilla network, he explains, approximates a function from one fixed-size input to one output. That holds even for a five-second audio clip, as long as the size is fixed and there is ground truth for the chunk. RNNs instead handle one-to-many, many-to-one and many-to-many mappings, including bidirectional ones.

They matter most when the connections across a sequence carry more information than each frame's spatial content, as in speech, language, audio and video. A clip of speech can map to a single label such as the speaker's gender, and Spanish can be translated into English with sequences of different lengths. When a student asks about feedback from output to input, Fridman says that loop is almost the definition of an RNN.

Fridman warns that leaning on TensorFlow tutorials while ignoring how backpropagation works is done at your peril. He frames it as comparing the network's output with ground truth and sending the error backward to reward or punish parameters. To make it concrete he uses a tiny circuit that adds x and y and multiplies the sum by z, with inputs of -2, 5 and -4 giving an output of -12.

In the backward pass he sets the output gradient to one, meaning he wants f to go up. A gradient is just how much the output changes when an input changes slightly. The add gate has slope one for both inputs, while the multiply gate flips the values, so the gradient on x depends on y and vice versa.

The chain rule is what makes this practical. Instead of writing down one giant derivative, you multiply local gradients step by step from output to input. In the example, x and y end up with a gradient of -4 and z with 3, so to raise f you decrease x and y and increase z.

Almost any operation in a network can be decomposed into three gates: add, multiply and max. Add distributes the gradient equally and ignores the forward value, multiply swaps the forward values, and max sends the gradient only to the larger input. A sigmoid neuron can be broken into the same kind of gates, plus an exponential and a division.

In a real network the inputs and target outputs are fixed, so what gets adjusted are the weights and biases. After a forward pass computes the loss, gradients flow backward and the weights move opposite the gradient by an amount set by the learning rate. Fridman stresses that learning is an optimization problem: minimize an objective whose variables are possibly millions of parameters.

Gradient descent, he says, is easy to picture on a convex bowl and hard on rugged terrain full of crevices and saddle points. Stochastic gradient descent can oscillate in a saddle, and optimizers such as Adam were designed to cope. Asked how to ensure convergence on a non-convex function, he answers plainly that you cannot, and that nobody even knows the global optimum for something like driving.

The practical danger is vanishing or exploding gradients. Sigmoid derivatives go to zero at the tails, so the backpropagated signal shrinks and the network stops learning. ReLU is popular but can leave dead neurons that never fire if initialization is poor, and he mentions that 40 or 50 percent of neurons can end up dead. Telling these failures apart from a bad loss function is hard, which is why he says picking solver and activation carefully is part of the art.

He closes the topic by asking whether adjusting parameters to minimize one objective is really how human-level intelligence is achieved. The architecture itself is never evolved in this process. He shows evolutionary robotics and notes that nature-inspired algorithms for tuning weights do not currently work as well, calling backpropagation simple, even a bit dumb.

Fridman draws the basic RNN: an input with weights, a hidden state that acts as memory and is connected to itself across time, and an output. Because of that loop the network can keep consuming inputs one at a time. Unrolled, it looks like a regular network repeated an arbitrary number of times.

Just as convolutional networks assume spatial consistency, RNNs assume temporal consistency and use the same weights at every step. This lets them process very long sequences without an explosion of parameters. Training uses backpropagation through time, which is ordinary backpropagation over the unrolled network.

Unrolling also inherits the old problem: if the gradient saturates anywhere along a long sequence, it drives all earlier steps toward zero, so the network effectively ignores most of the sequence. Bidirectional variants fill in missing elements of images, words or audio, and stacking layers helps but demands more data. In theory RNNs can learn any sequence, yet in practice they struggle with long-term dependencies, as in remembering that a story is about Bob and apples across several sentences.

When people praise recent results on time series, audio and video, Fridman says they almost always mean LSTMs, long short-term memory networks. An LSTM cell looks more complicated than a vanilla RNN cell but is more intuitive. It has gates built from sigmoid layers, which squash values between zero and one, and tanh layers, which squash between negative one and one.

He describes a conveyor belt of state running through the cell. A first sigmoid gate looks at the new input and the previous output and decides what to forget or ignore. A second step decides which part of the state to update and with what values, and a third performs the update by multiplying: zero forgets, one lets information through.

His example is a sentence about Bob and Alice having lunch, where the cell must track gender so it can generate who eats what. It has to forget that Bob likes apples while remembering that Alice likes oranges. Asked what the state actually is, he says it is a vector of numbers produced by the activations, modified by the weights and biases.

Fridman surveys applications that all share one construct: some input, a state carried along, and an output. Machine translation reads a sentence in one language and, once it ends, starts producing words in another, which he says is what Google's translator is supposedly using. Other examples include LSTMs generating handwriting from text and character-by-character text generation that predicts the next character each step.

Several systems combine vision and language in one network. A convolutional net can feed an LSTM to answer questions such as how many chairs are in an image, or to caption images and video, with markers for the start and end of sentences. Visual attention uses an RNN to steer a sliding window over an image, for example to read house numbers, and can also steer generation, such as drawing digits.

He also covers a model that watches silent video of a person hitting objects with a drumstick and generates the sound. In medicine, sparse and variable-length electronic health records can be fed in to predict diagnoses such as diabetes, scoliosis and asthma with fairly good accuracy. Stock prediction from financial data and news is something everyone wishes worked, he says, but nobody has done it convincingly, though crashes seem easier to predict.

Audio is another area: an LSTM trained on a single speaker's raw audio starts producing something that sounds like words after only a few epochs. Going the other way, audio can be mapped to a classification, such as whether a road is wet or dry from its sound, the speaker's gender, or the words spoken in speech recognition.

Turning to driving, Fridman recalls the NVIDIA-style network behind DeepTeslaJS: five convolutional layers and three fully connected layers. It takes a single image with no temporal information and regresses the steering angle. He then describes the Udacity self-driving competition, where the winners went beyond that and used sequences of images.

The first- and third-place teams used LSTMs to map image sequences to sequences of steering angles, while he says the second-place team used 3D convolutions. A convolutional network extracts a representation of each frame, and that becomes the LSTM's input at every time step. The first-place team also predicted vehicle speed and torque, using sequences of length 10 and supervised learning with no reinforcement.

Fridman says most people who are not computer vision specialists will want RNNs, because very few of us work on data without a time dimension. On the question of why a length of 10 if RNNs are arbitrary, he says it is only the training chunk, and the structure is still a single looping cell. He encourages Kaggle-style competitions, where the work is months of tuning parameters, preprocessing, learning rates and optimizers.

Team Chauffeur, the third-place winner, used transfer learning. Take a network trained on a huge dataset such as ImageNet, chop off the final layer and retrain it for a new task like cancer or no cancer. They took a giant convolutional net, used its roughly 3,000 features per frame as input to an LSTM, and used a sequence length of 50.

Fridman ends by saying the art of neural networks lies in parameter tuning, which experience teaches and lectures cannot. He repeats Geoffrey Hinton's joke about "Stochastic Graduate Student Descent", where you keep hiring graduate students to tweak hyperparameters until the problem is solved. He then previews the next lecture on driver state, his own passion, and a guest from the White House on AI policy.