How Machines Learn to See: From Pixel Differences to Convolutions, ResNets and Scene Segmentation

Fridman opens by framing the lecture around SegFuse, a dynamic driving-scene segmentation competition that he says sits at the cutting edge of perception. Unlike the earlier DeepTraffic competition, which was designed to teach deep reinforcement learning, SegFuse could lead to publications. He notes that today's computer vision is largely deep learning, which here means supervised learning with human-annotated ground truth.

He insists on one idea: the machine receives raw numbers, whether a single channel or three RGB values per pixel, and must classify or regress from them. Humans take interpreting sensory input for granted, so we have to keep asking whether the data given to the machine is sufficient for the task. He draws a loose analogy to the layered visual cortex, where early layers pick up edges and later layers build higher-order meaning.

He lists the main difficulties. Illumination variability is among the biggest challenges for visible-light cameras in driving, followed by pose variability, since neural networks represent rotated or deformed objects poorly. Intra-class variation is large while differences between classes can be small, and occlusion forces a model to identify an object from just an ear or a leg. A cat dressed as a monkey eating a banana shows what is missing: context, humor and the temporal dynamics of the scene.

Fridman explains image classification as sorting new images into labeled bins, and lists datasets such as MNIST, CIFAR, ImageNet and Places. CIFAR-10 is the toy example, with tiny icons in ten categories including airplane, cat, dog and truck. All of these rely on ground-truth labels made by people.

His trivial classifier subtracts two images pixel by pixel and sums the differences. Finding the closest image in the dataset and copying its label, which is nearest neighbor, gives 35-38% accuracy on CIFAR-10, compared with 10% for coin flipping. Using the k closest images and voting does not rescue it much, while humans reach about 95% and convolutional networks come very close to 100%.

He then reviews the basic neural unit: weighted inputs, a bias and a nonlinear activation, stacked into hidden layers and trained by rewarding weights that led to correct decisions. For ten classes there are ten output neurons, and each one should get "excited" when its number is called. Convolutional networks, he says, are the trick that makes this work efficiently on big, complex image problems.

Convolutional networks exploit spatial invariance: a cat in the top-left corner is the same as one in the bottom-right. Instead of fully connected edges, a window slides across the image applying the same weights, and the blocks take 3D volumes as input and output. Depth is the number of filters, stride is the step size, and padding is zeros added around the input.

He steps through a worked example with a three-channel input, two 3x3 filters per channel plus a bias, and a stride of two, producing a two-channel output. The learned filter values are the parameters, and reusing them across the image greatly reduces how many are needed. Examples of filters include the identity filter and several edge detectors, with each filter producing a map showing where its pattern was found.

Pooling, traditionally max pooling, reduces resolution by keeping the strongest activations. That helps combine features into higher-order entities but hurts when precise spatial detail is needed, as in segmentation. Stacked convolutional layers followed by fully connected layers or other task-specific parts make a convolutional network.

ImageNet, with 14 million images and 21,000 categories, is the benchmark where architecture progress is measured, using top-five error. Fridman traces AlexNet, VGGNet, GoogLeNet, ResNet, CUImage and SENet, and notes that human error is 5.1% on the classification task. ResNet beat that in 2015 with about 4% error.

AlexNet and VGGNet are uniform, stacked convolution and pooling layers ending in fully connected ones. The 16- or 19-layer VGGNet has 138 million parameters. GoogLeNet's inception module runs 1x1, 3x3 and 5x5 convolutions together, because small ones capture local texture and larger ones capture abstract features, which needs fewer parameters if the modules are placed well.

ResNet's residual block passes the input along untransformed while letting the layer learn something new, which makes training much easier and allows much deeper networks. Fridman notes the original belief that depth itself explains the gains does not necessarily hold. In 2017 the squeeze-and-excitation network cut error from about 4% to 3% by adding a learnable weight per channel, and he says it can be added to any architecture and invites asking what else could be made learnable.

He closes by mentioning capsule networks from Geoff Hinton, which he says will get an online-only lecture. The point for now is to ask what assumptions a successful network makes and what it throws away. Convolution discards the relationships between simple and complex parts, so a face with scrambled features can still fire as "a face".

To segment an image, you take a classification network, chop off the final layer and add decoder parts that upsample the encoded features into a heat map. Semantic segmentation then colors every pixel with the object it belongs to. Fridman calls it incredibly difficult despite three years of progress, because millions of pixels each need a label.

Precise boundaries matter in medical imaging, such as detecting tumors, and in driving and robotics, where vehicles, pedestrians and cyclists need exact outlines. Fusing such semantic labels with lidar's 3D map could give both what an object is and exactly where it is. The 2014 FCN paper repurposed ImageNet-trained networks this way, and skip connections improved its very coarse output.

He lists later components that define the state of the art: SegNet's encoder-decoder framework applied to driving data, dilated convolutions that keep local high-resolution texture, and DeepLab's use of conditional random fields to smooth results based on image intensities. Better backbones from AlexNet to VGGNet to ResNet drove improvements. The setup used for the competition data adds dense upsampling convolution, which learns the upscaling filters rather than using bilinear upsampling, and hybrid dilated convolution that varies dilation rates for smoother output.

Everything so far discards temporal dynamics, yet for driving and robotics, understanding how pixels move through time is essential. Dense optical flow estimates, for each pixel, the direction and distance it moved between two frames about 33.3 milliseconds apart at 30 frames per second. That lets information detected in one frame be propagated forward.

Fridman says manual annotation is the bottleneck: coloring every pixel of one image in the state-of-the-art Cityscapes driving dataset takes about 90 minutes. That is why no dataset exists with long videos where every frame is fully segmented, and the class hopes to create one. He describes FlowNet from 2015, in which FlowNetS stacks two RGB frames into a six-channel input and FlowNetC uses a correlation layer, and FlowNet 2.0 from December 2016 combines them for smoother flow and runs at roughly eight to 140 frames per second depending on the variant.

The SegFuse data includes original 1080p driving video around Cambridge, an 8K 360-degree video, ground-truth segmentation of the training set labeled on Mechanical Turk, the output of a state-of-the-art frame-by-frame segmentation network, and optical flow from FlowNet 2.0. The task is to use temporal information to improve the network's per-frame segmentation toward the ground truth. There are 10,000 images, starter code in Python on GitHub, and a submission of code.