From Hand-Built Features to ResNets: Andrej Karpathy's Crash Course in Convolutional Networks

Karpathy opens by noting that fully connected networks treat an input as a flat vector of numbers. Real data has layout: spectrograms are 2D arrays, images are 3D arrays, videos 4D, and text can be seen as 1D. Whenever data has this local structure, convolutional networks can take advantage of it to make learning much more efficient.

He then traces the history, starting with Hubel and Wiesel's 1960s recordings from cat visual cortex and Fukushima's neocognitron in the 1980s, which had simple and complex cells but no backpropagation. Yann LeCun's LeNet-5 in the 1990s was the first recognizable convnet trained end to end with backpropagation. For years, though, it was assumed such models would not scale beyond small tasks like digits and zip codes.

When Karpathy began his PhD in computer vision around 2011, the field relied on a kitchen sink of features, with papers devoting pages to feature types and hyperparameters, often implemented in different languages. Detectors would find cars in trees and researchers would shrug. In 2012, Alex Krizhevsky, Ilya Sutskever and Geoff Hinton scaled a LeCun-style network up, trained it on a much bigger dataset on GPUs, and the vision community took notice.

ImageNet top-5 error dropped sharply with the 2012 convnet and has since fallen to about 3.57%. To judge how good that is, Karpathy built a web interface and competed against GoogLeNet, estimating human error at roughly 2 to 5%. He lost points mainly on fine-grained classes like the roughly 50 types of terrier, and he says an error rate below about 1.5% would make him suspicious because of mislabeled images.

The change is that the fixed feature extractor plus a small trained classifier became a single network trained completely end to end. That also cut code complexity, since architectures became homogeneous, and a pre-trained model can classify an image in a few lines of Keras and tens of milliseconds on a GPU. Features learned on ImageNet also turned out to be generic: people found in about 2013 that a linear classifier on top of them achieved state-of-the-art results across many datasets.

Convnets now appear in photo search, self-driving perception, medical imaging, satellite analysis, WaveNet audio generation, game-playing agents and robotics, and in art through DeepDream and neural style. Karpathy also notes studies in which representations in a convnet map onto recordings from the macaque IT cortex, suggesting that optimizing for ImageNet may have converged on something brain-like.

Karpathy describes a convnet as one function from raw pixels, such as a 224x224x3 image, to 1,000 numbers, built from dot products and max operations with around 10 million tunable parameters. Its layers operate on 3D volumes. A filter, say 5x5 spatially with depth matching the input, slides across all positions computing dot products to give an activation map, and stacking six filters on a 32x32x3 image gives a 28x28x6 volume.

The efficiency comes from assumptions. A fully connected layer producing the same output would need about 15 million parameters and a similar number of multiplies, whereas the six 5x5x3 filters need only 450 weights and about 350,000 multiplies. This assumes a feature useful in one image region is useful elsewhere, which would be wrong for centered face images where eyes and mouths sit in fixed places. Local connectivity is acceptable because stacked layers grow each neuron's receptive field until it covers the whole image.

Pooling layers downsample each channel independently, as when resizing a picture in Photoshop. Max pooling with 2x2 regions and stride two takes the max over four numbers, which controls capacity and compute without hurting performance much. After the convolutional layers, the final volume is stretched into a vector and passed through a fully connected layer and softmax to get class probabilities.

He demos ConvNetJS, a JavaScript library he wrote, training a CIFAR-10 network in the browser, which reaches about 30% accuracy within a minute or so. He then plays Jason Yosinski's deep visualization toolbox video, where first-layer units detect oriented edges, a fifth-layer unit responds to faces, and others respond to wrinkles or printed text. None of these concepts were labeled; the network learned them only because they supported the final decision.

Karpathy walks through AlexNet, whose first layer applies 96 filters of 11x11 with stride four to a 227x227x3 image, producing a 55x55x96 volume with about 35,000 parameters. Pooling layers have zero parameters, and the filter count grows from 96 to 256 to 384 in later layers before the fully connected layers. The normalization layers it used have since fallen out of favor.

Comparing it with the 1998 LeNet, he lists four things that hold progress back: data, compute, algorithms and infrastructure. Data grew from thousands to millions of images, GPUs run about 20 times faster than CPUs, and NVIDIA's CUDA library arrived as infrastructure. Algorithms changed least: the network is bigger, with dropout and rectified linear units added.

He jokes that both dropout and ReLU are about a two-line diff and both set things to zero, one at random and one below zero, so new algorithm ideas should look for one-line changes that zero something out. AlexNet also used heavy data augmentation, SGD with momentum 0.9, batch size 128 and a seven-network ensemble that added about 2% accuracy.

ZFNet won in 2013 by shrinking the first-layer filters to 7x7 with stride two and enlarging the middle layers. Matthew Zeiler then founded Clarifai and pushed the error to 11%, though Karpathy says he never disclosed the full details beyond hyperparameter tweaking.

VGGNet from Simonyan and Zisserman in 2014 is beautiful in its uniformity: only 3x3 convolutions with stride one and 2x2 max pools, with filter counts doubling from 64 to 512. It reached 7.3% error but uses about 140 million parameters, most of them in the giant fully connected layers, and roughly 100 MB of memory per image on the forward pass.

The actual 2014 winner, Google's GoogLeNet, used inception modules and looks messier but is far more efficient, with around five million parameters. It gets there largely by discarding the big fully connected layers, which Karpathy says do not hurt performance much when removed. It trades elegance for a small footprint.

Karpathy calls the 2015 residual network from Kaiming He and colleagues at Microsoft Research Asia the first bigger departure from the 1998 design. It won ImageNet at 3.57% and many other challenges after fine-tuning. The paper showed that plain networks get worse when deepened: a 56-layer model did worse than a 20-layer one even on training data, an optimization failure rather than overfitting, since the deeper net could in principle copy the shallow one.

In a residual block, computation happens on the side and its result is added back to the input stream, so each block computes a delta. Karpathy says this is clearer through backpropagation, because addition distributes gradients equally, giving supervision a direct path to the earliest layer. He ties this to why LSTMs beat plain recurrent networks, which also use additive paths.

He finds stochastic depth, which randomly drops residual blocks, a striking result, and prefers viewing residual nets as dynamical systems in which each block is a vector field nudging the representation toward a decodable answer. Later work found that residual networks can be much shallower, around 20 to 30 layers, and wider, and work as well. He also credits shared code, such as Facebook's Torch release of ResNets, for rapid experimentation, and briefly plugs his Arxiv Sanity site.

Karpathy frames the convnet as a block of compute producing features, with each task differing mainly in what you predict and the loss at the end. Classification predicts class scores. Captioning attaches a recurrent network that emits word probabilities at each step. Localization adds x, y, width and height with an L2 or Huber loss, and Atari-style reinforcement learning predicts probabilities over eight discrete actions.

Segmentation predicts a category for every pixel using upsampling deconvolution layers, and autoencoders ask the network to reproduce the image through a 7x7x512 bottleneck. A YOLO-style detector predicts a class and several bounding boxes with confidence at each of 49 grid cells, matching predictions to true boxes. Dense captioning, from his work with Justin Johnson and Fei-Fei Li, combines detection with generated descriptions.

On hardware, Karpathy mentions NVIDIA's dev boxes with Titan X GPUs and the roughly $130,000 DGX-1, and says assembling parts yourself can cost about half as much. Cloud GPUs were poor at the time, with AWS's weaker cards and Azure's K80 offering still coming, and OpenAI used Cirrascale rented boxes. For software, he says Keras on TensorFlow covers about 90% of use cases, and he still likes Torch.

His advice for architecture and hyperparameters is the same: don't be a hero. Take the latest ILSVRC winner pre-trained, adjust the top layers, and fine-tune; use papers' standard settings such as Adam at learning rate 1e-3 or 1e-4. The main thing to tune is regularization, especially dropout rates for small datasets, so the old hundreds of vision hyperparameters become effectively two.

For scale, state-of-the-art nets train for weeks on four or eight GPUs of about $1,000 each. He recommends synchronous data parallelism, splitting a batch across GPUs and summing gradients, over model parallelism. He also warns about disk-to-CPU and CPU-to-GPU bottlenecks, suggesting SSDs, contiguous raw formats and prefetching threads.

For genomic sequences of arbitrary length, Karpathy suggests recurrent networks or WaveNet-style dilated convolutions, which capture large context with few layers. On how many layers to use, he says there is no mathematical theory and theory lags empirical guesswork: he uses the biggest network that fits his GPU memory and regularizes strongly. Pooling is fading, as in ResNet and the all-convolutional net paper, since strided convolutions work as well and pooling discards information.

On fine-tuning dataset size, his answer is that more is always better, with a loose heuristic of examples on the order of parameters. For huge class counts he points to hierarchical softmax, and for compression he mentions Google's convolutional and recurrent codes and super-resolution work such as Magic Pony. He says massively distributing training across phones runs into stale gradients, since asynchronous speedups plateau after about eight workers.

For two inputs, he suggests one convnet per input and fusion afterward, and for video a convnet per frame feeding a recurrent network. For low-latency inference on phones, he points to quantizing weights and pruning connections, as in Song Han's work. He adds that the open research problem is training in low precision so training matches test time, and he declines to guess concrete latency numbers.