Deep Learning at the Edge of 2019: BERT, Autopilot, and the Ideas That Broke Through

Fridman frames the lecture as a look at the state of the art in deep learning at a moment of great accomplishment and also a beginning. He explicitly says the talk is not about state-of-the-art results on image classification, object detection, NLP, or GAN benchmarks, nor about the best GitHub algorithm for a particular benchmark. Instead, it is about the ideas and developments at the cutting edge of the field. He also acknowledges omissions such as medical applications and protein folding.

For Fridman, 2018 is the year of natural language processing. Many described it as an ImageNet moment, comparable to 2012 when AlexNet delivered a large jump in computer vision and inspired people about purely learning-based methods. A series of developments from 2016 and 2017 led to BERT, which produced a total leap in benchmark performance and in the ability to apply NLP to various language tasks.

Fridman begins with the encoder-decoder architecture for recurrent neural networks. In machine translation, the encoder takes a sequence of words or samples and uses recurrent units such as LSTMs or GRUs to encode the sentence into a single vector. The decoder then takes that fixed-size representation and unrolls it into a sentence that can differ in length from the input.

Attention improves on this by letting the decoder look back at the input sequence instead of relying only on a collapsed vector. The decoder can learn which parts of the input are important for which parts of the output during decoding. Fridman points to Jay Alammar's visualizations of neural machine translation, where hidden states from the encoder are pushed forward and selectively weighted by the decoder.

Self-attention extends the idea to the encoding process itself. It lets the encoder selectively look at other parts of the input sequence when forming hidden representations, so each word can be encoded with the relevant context around it. The transformer uses self-attention in the encoder and attention in the decoder, capturing the rich context of the input sequence to generate the output sequence contextually.

Fridman steps back to the problem of representing words and characters meaningfully. Traditional Word2Vec-style embedding uses an unsupervised language-modeling trick: a skip-gram model takes a word as input and predicts statistically likely surrounding words. By discarding the input and output and keeping the hidden representation, the model forms a compressed embedding where related words are close and unrelated words are far apart.

ELMo improved this by using bidirectional LSTMs to learn representations from both the sequence before a word and the sequence after it. That fuller context produces representations better able to capture the statistical language model behind a corpus. The OpenAI transformer took the transformer architecture and used the language learned by the decoder as a language model, then chopped off layers and fine-tuned on specific tasks such as sentence classification.

BERT delivered the big performance leap. Unlike a forward-moving transformer formulation, BERT is richly bidirectional: it takes the full sentence, masks 15% of the tokens, and tasks the entire self-attention encoding mechanism with predicting the missing words. Stacking many self-attention and feed-forward encoder blocks learns rich language context that can then support contextual embeddings, sentence classification, sentence-pair classification, similarity, multiple-choice question answering, general question answering, and tagging.

Fridman jumps from academic NLP to applied deep learning. Tesla's Autopilot hardware version 2 is a newer implementation of the NVIDIA Drive PX 2 system and runs many neural networks. Eight cameras feed a variant of the Inception network at different resolutions, performing tasks such as drivable-area segmentation, object detection, and basic localization.

The significance is scale and real-world stakes: a huge fleet of regular consumers, many with no understanding of neural-network limitations, are riding in cars whose perception and control decisions are made by neural networks. Over 1 billion miles have been driven in Autopilot. Hardware version 1 was an Intel Mobileye monocular camera system that, as far as is known, did not use a neural network and was fixed rather than learning; hardware version 2 is always learning, with weekly updates shipping new weights, and now accounts for about half the miles.

Fridman describes AutoML as the dream of automating as much of the machine-learning process as possible: drop in a dataset and let the system determine architecture details, modules, hyperparameters, training, and inference. Neural architecture search in 2016 and 2017 made progress by using reinforcement learning and recurrent neural networks to stitch modules together and optimize overall performance. Google AutoML pushed toward an API-like drop-in-data experience, and AdaNet in 2018 extended the approach to build ensembles of candidate architectures for state-of-the-art performance.

On the data side, Fridman argues there has been far less progress than on architectures. Data augmentation is the process of twisting and transforming raw data to represent how it might appear in other real-world contexts, and AutoAugment takes a small step by treating operations such as translation, shearing, and color manipulation as basic actions. Reinforcement learning and an RNN compose those actions into augmentation policies that improve performance on ImageNet. The learned policies can also be transferred to a different dataset, so transfer learning can move both weights and augmentation policies.

Synthetic data offers another way to learn a lot from a little. Fridman highlights NVIDIA's CVPR 2018 work, which created realistic scenes and also deliberately unrealistic variations, then trained on huge synthetic datasets and fine-tuned with a small sample of real images to reach strong performance. For annotation, Polygon-RNN uses convolutional networks to place a first point from a bounding box and recurrent networks to draw a polygon around an object, reducing the cost of segmentation labels. The broader goal is to remove humans from menial parts of the pipeline and automate the boring stuff.

Fridman presents DAWNBench from Stanford as a competition to reach 93% accuracy on ImageNet and 94% on CIFAR-10 in the least time and for the least dollars. Fast.ai trained ImageNet in about 3 hours for $25, and CIFAR-10 for 26 cents. The benchmark matters because it opens fundamental deep-learning work to people without the compute resources of Google DeepMind, OpenAI, or Facebook, including academics and independent researchers.

The fast.ai trick was simple but effective: jointly crank up the learning rate while decreasing momentum during training. That adjustment lets the network learn very fast. The result is not just a benchmark stunt; it lowers the barrier to experimentation and makes deep learning more accessible.

On generative models, Fridman says 2018 was a year of scaling and parameter tuning rather than breakthrough GAN ideas. A Google DeepMind GAN generated incredibly high-resolution images by increasing model capacity and batch size while keeping the same GAN technique. NVIDIA's video-to-video synthesis made temporal consistency part of the optimization, producing temporally coherent output where an image-to-image baseline such as pix2pixHD looks jumpy; the method can map labels to video, edges to faces, or body pose to images.

For perception, Fridman traces how large classification networks such as VGGNet, GoogLeNet, ResNet, SENet, and DenseNet form rich representations reusable for detection and segmentation. Object detection includes region-based methods that propose regions and classify them, and single-shot methods that produce boxes and classes in one pass. Semantic segmentation progressed from fully convolutional networks to SegNet and dilated or atrous convolutions, with DeepLabv3+ reaching state-of-the-art open-source performance on PASCAL VOC through multi-scale atrous rates that enlarge the field of view without increasing parameters.

Fridman starts the reinforcement-learning story with DeepMind's DQN paper, which beat many Atari games at superhuman performance using raw pixels as input. The same architecture could learn the physics of different games well enough to beat them, an idea with echoes of general intelligence. AlphaGo followed in 2016 with supervised learning from expert games plus self-play, and AlphaGo Zero in 2017 beat AlphaGo after a few days of training with zero supervision from expert games.

AlphaZero is presented as the cleanest demonstration of deep reinforcement learning's progress because it removes humans further from the loop. The AlphaZero paper appeared in 2017, and in 2018 it beat Stockfish, the best chess engine, after four hours of highly distributed training, and also beat the state-of-the-art Shogi engine Elmo. Fridman contrasts brute-force tree search, which looks millions or billions of moves ahead, with human grandmaster intuition that considers on the order of hundreds of moves; AlphaZero moves closer to the human side by using a neural network to estimate board and move quality and reduce lookahead.

OpenAI took on Dota 2 as a messier game with teamwork, long time horizons, imperfect information, hidden information, and uncertainty. In 2017 its 1v1 bot beat a top professional player, and in 2018 OpenAI attempted 5v5 and lost two games against top players at The International, where the winning team received $11 million. Fridman notes that losing is part of the story and expects a comeback; he also mentions that heads-up no-limit Texas Hold'em AI beat top players, while the general multiplayer poker game remains out of reach.

Fridman closes by noting that 2017 and 2018 were years of frameworks growing up and forming ecosystems. TensorFlow reached maturity with TensorFlow 1.0, PyTorch 1.0 matured in 2018, and TensorFlow's eager execution and the coming TensorFlow 2.0 in 2019 continued the progress. The ideas discussed in the lecture have GitHub implementations in TensorFlow and PyTorch, making them extremely accessible.

He ends with Geoff Hinton, the “Godfather” of deep learning and a key figure behind backpropagation, who said, “My view is throw it all away and start again.” Hinton believes backpropagation is broken, ancient, and in need of complete revolution. Fridman says the future depends on a graduate student deeply suspicious of everything Hinton has said, because today's state of the art still rests on ideas from the 1960s and 1980s, especially stochastic gradient descent and backpropagation. That makes the field ripe for totally new ideas, and it is up to the next generation to define the real breakthroughs of 2019 and beyond.