Deep Learning From the Ground Up: Hugo Larochelle's One-Hour Tour of Neural Network Foundations

Larochelle sets out to cover the basics of feed-forward networks in a single hour, as an on-ramp to the two days of talks at the Deep Learning School. He plans to describe how networks predict, how they are trained with loss functions, backpropagation and stochastic gradient descent, and then touch on advances since around 2006 such as dropout and batch normalization. For deeper derivations he points viewers to his online lectures.

A network takes an input vector, passes it through hidden layers and produces output units, one per class in a classification problem. Each layer first computes a pre-activation, which is a weight matrix times the previous layer's activation plus a bias vector. It then applies a nonlinear activation function. The nonlinearity is what lets the network capture sophisticated classification functions.

He reviews the common activation functions. Sigmoid squashes values between zero and one and tanh between minus one and one, and both saturate at large magnitudes. The rectified linear unit, or ReLU, is unbounded above and outputs exactly zero for negative pre-activations. For the output layer, softmax exponentiates the pre-activations and normalizes them, giving a probability distribution over classes.

Larochelle notes that in software each operation becomes an object, and chaining them forms a flow graph of the forward computation. This representation will also be handy for computing gradients. On expressive power, he states the classic result: a single hidden layer with a linear output can approximate any continuous function arbitrarily well given enough hidden units. However, that result does not explain how to find the weights and biases.

Training is framed as empirical risk minimization, or structural risk minimization when a regularizer is added. You choose a loss that compares the network's output with the target, plus a regularizer weighted by a hyperparameter often called weight decay. Learning then means minimizing the average training loss plus that penalty over all weights and biases.

The workhorse algorithm is stochastic gradient descent. After initializing the parameters, you loop over epochs, each a full pass over the training set. For every example you compute the gradient of the loss and regularizer and step in the opposite direction, scaled by a learning rate. Larochelle lists four things to specify: a loss, a way to compute gradients, a regularizer and an initialization.

For classification, the natural loss maximizes the probability assigned to the correct class, which in practice means minimizing the negative log-likelihood. The log is used because it is numerically more stable and gives nicer gradients. Many libraries call the same thing the cross-entropy loss, since it can be seen as a cross entropy between a one-hot target distribution and the network's output distribution.

Backpropagation computes gradients by applying the chain rule from the output layer down toward the input, reusing computations made at the layers above. The gradient of the loss with respect to the output pre-activation turns out to be very simple: minus the one-hot vector of the correct class plus the network's predicted probabilities. Intuitively, the correct class's probability is pushed up and all the others are pushed down in proportion to the network's current belief.

At each layer, the weight gradient is the pre-activation gradient times the transposed activations of the layer below, and the bias gradient equals the pre-activation gradient. To go down a layer, you multiply by the transposed weight matrix and then take an elementwise product with the activation function's derivative. Larochelle stresses that this backward pass costs roughly the same as a forward pass.

The elementwise product with the derivative explains a key practical issue. When a unit's derivative is near zero, the gradient fails to propagate to lower layers. A sigmoid's derivative is the sigmoid times one minus the sigmoid, so it is flat when saturated near zero or one, and tanh behaves similarly. ReLU's derivative is a binary mask, which means a gradient is exactly zero for units below zero.

In practice, you rarely write backprop by hand, because libraries differentiate automatically. You extend the flow graph with the loss at the end, and each box gets a `bprop` method that takes the gradient with respect to itself and passes the chain-rule result to its arguments. Larochelle says Torch does this quite explicitly in reverse order of chained modules, while Torch autograd, Theano and TensorFlow, which the school covers later, are more sophisticated.

The standard regularizer is L2, the sum of squared weights, with a gradient of twice the weight, and it is often called weight decay. Biases are usually not regularized because there are far fewer of them. Larochelle also warns about initialization: biases typically start at zero, but weights cannot, since zero initialization yields zero gradients, and identical values would make all units in a layer update the same way forever.

Weights are therefore initialized randomly to break symmetry and kept small so that units do not start out saturated. A common recipe draws from a uniform distribution within bounds, with a derivation for tanh by Xavier Glorot and Yoshua Bengio. He then turns to hyperparameters such as layer count, units per layer, learning rate and weight decay, which are chosen using a separate validation set.

Grid search tries every combination, and the number of runs grows exponentially with the number of hyperparameters. Random search samples each hyperparameter independently from a distribution, for instance log-uniform for learning rates, and takes the best of N experiments. It never leaves holes if a job dies, and a single bad learning rate value ruins only one run rather than a whole row of the grid. Bayesian optimization methods can work somewhat better still.

For the number of epochs, he recommends early stopping rather than a search. Because training one more epoch is cheap, you track validation performance and stop once it has not improved for a set look-ahead number of iterations. You then go back to the best network seen. The training error keeps falling while the validation error eventually flattens or rises.

Larochelle's tricks of the trade start with normalizing real-valued inputs by subtracting each dimension's training mean and dividing by its standard deviation. He also suggests decaying the learning rate, for example by halving it each time the validation score stops improving. This allows fast early progress and slower, finer progress later.

Rather than one example at a time, practitioners use mini-batches of say 64 or 128 and average the loss. This lets the forward pass use matrix-matrix multiplications, which are faster than repeated matrix-vector ones, so batch size is mostly a speed hyperparameter. Momentum adds a fraction of the previous update to the current gradient, so steps grow when successive directions agree.

He briefly mentions adaptive methods you will see in papers. Adagrad scales each parameter's step by the square root of its cumulative squared gradients, while RMSProp uses an exponential moving average instead of a sum. Adam combines RMSProp with momentum and, he notes, is widely implemented and used with success.

For debugging, he advises checking any hand-written gradients against a finite-difference estimate that perturbs each parameter by a tiny epsilon such as 10 to the minus 6. He also recommends making sure the model can overfit about 50 examples, and if it cannot, checking initialization, gradients, input normalization and learning rate. But SGD is so bug-resistant that it can learn even with some wrong or zero gradients, so the small experiment is no replacement for gradient checking.

If one hidden layer is enough in theory, why use many? Larochelle gives three motivations. The visual cortex processes signals through regions that detect edges, then more complex patterns, then whole objects, which suggests a hierarchy of features from edges to noses and eyes to faces. There is also theory showing that some Boolean functions need exponentially many units if depth is restricted but can be represented compactly with more layers.

The third reason is empirical: deep learning has revolutionized speech recognition and become the method of choice for visual object recognition. Backprop dates to the 1980s, so he asks why deep learning took off only recently. The answer is that deep networks are hard to train, with fading gradients leaving lower layers barely moving.

Underfitting can arise when gradients are too small and progress is slow, while larger networks with more parameters can overfit a finite training set. Remedies differ: underfitting can be tackled by training longer, which GPUs made far cheaper, or by better optimization methods. Overfitting calls for better regularization, and Larochelle mentions that early in his PhD he worked on unsupervised learning as a regularizer.

Dropout makes a network harder to fit by flipping a coin for each hidden unit before every forward pass. With probability one half, the activation is multiplied by zero, which removes the unit for that example. Units cannot rely on specific others, which encourages features that are not co-adapted. The mask is resampled for every example and also applied to the gradients in backprop, and 0.5 is often surprisingly the best value.

At test time, the mask is replaced by the keep probability, such as 0.5. For a single hidden layer, this equals the geometric average of all the exponentially many masked networks sharing weights, which looks like ensembling. That argument does not extend to deep networks, though dropout still works well in practice. Larochelle warns to expect training to take roughly twice as many epochs with dropout at 0.5.

Batch normalization helps some networks that would otherwise underfit, and it also seems to regularize, since dropout is less useful when it is used. The idea is to normalize hidden layers like inputs, but since their statistics keep changing, it computes the mean and variance of each unit's pre-activation over the current mini-batch and backpropagates through them. At test time it uses the global mean and variance from a full pass over the training set.

The normalized value is then scaled by a learned gain gamma and shifted by a learned bias beta, since subtracting the mean removes the original bias. In the closing Q&A, an attendee asks why ReLU produces sparsity. Larochelle says it is observed in practice, partly because the unit outputs exact zeros, and that a linear transformation followed by a ReLU-like function resembles steps in sparse-coding optimization, but he knows of no more solid explanation.