Gradient Descent: How a Neural Network Learns

Grant Sanderson introduces gradient descent using a handwritten-digit classifier, then examines what its hidden layers actually learn. The example uses MNIST images rendered as 28 by 28 grayscale pixels.

The 784 input neurons represent pixel values. Each subsequent neuron combines activations from the previous layer with weights and a bias, then applies an activation function. The example has two hidden layers of 16 neurons and roughly 13,000 adjustable parameters.

The ten output neurons correspond to digits; the most active one is the prediction. A hoped-for interpretation is that early layers find edges and later layers combine them into loops and strokes, but that hypothesis must be checked.

Training supplies labeled digit images and adjusts weights and biases. Generalization is evaluated separately using labeled images the model did not see during training.

Starting from random parameters, the initial outputs are poor. For a labeled example, Sanderson defines a cost as the sum of squared differences between output activations and desired values: one for the correct digit and zero for the others.

The overall cost averages that quantity over the training set. It is a function of the network’s weights and biases and returns a single number measuring performance on those examples.

To see how to reduce cost, first imagine a function of one variable. At each point, its slope tells you which direction is locally downhill; repeated small steps can approach a local minimum.

The starting point matters: a complicated landscape can contain many valleys, and the local minimum reached is not guaranteed to be the global minimum. Step size also affects whether an update overshoots.

For a function of two variables, the gradient points in the direction of steepest local increase. The negative gradient points downhill, and its magnitude describes the local steepness.

The same idea extends to thousands of parameters. Arrange all weights and biases into a vector, compute the cost gradient and repeatedly move a small amount against it. This is gradient descent.

The gradient is based on average training cost; reducing that average does not promise that every individual example improves or that unseen data will improve. Backpropagation efficiently computes the derivatives used for these updates.

Each gradient component indicates whether a parameter should increase or decrease and how sensitive the local cost is to that parameter. For a gradient of (3, 1), the first coordinate has three times the local derivative of the second, in those units.

Sanderson reports that this simple two-hidden-layer network correctly classifies about 96% of unseen test images, and that changing the architecture and other settings can improve the result. These are results for the described experiment, not a universal guarantee.

He then checks the earlier intuition about edges. Visualizing the learned input weights shows patterns that look largely random rather than clean edge detectors. A useful classifier need not discover the particular representation its designer imagined.

When fed random noise, the network can still produce a confident digit prediction. Its training world consisted of centered digit examples, and the objective did not teach it to recognize arbitrary inputs as outside that world.

The simple network is a starting point for understanding later models, not a complete account of visual intelligence. Sanderson encourages learners to experiment with changes and points to Michael Nielsen’s freely available neural-networks book.

In an interview excerpt, Leisha Lee discusses research showing that a sufficiently large network can fit randomly shuffled training labels, even though those labels do not support useful generalization to new examples.

Lee describes subsequent work comparing optimization on random and structured labels: in the cited experiments, meaningful labels were learned much more quickly. She also mentions simplified theoretical work on the quality of local minima; these results do not imply all neural-network minima are equally good.