Deep Learning in 2020: Lex Fridman's Tour of Transformers, Self-Play, Self-Driving and What Comes Next

Fridman opens by placing AI in a very long human story, quoting Pamela McCorduck that it began with the ancient wish to forge the gods. He notes that modern humans are about 300,000 years old while the industrial revolution is only about 300 years old. He then traces the field from Turing's predictions through Rosenblatt's perceptrons to Deep Blue beating Kasparov and AlphaGo beating Lee Sedol.

The history of deep learning runs from the 1943 neural models through backpropagation and convolutional nets, the 2006 revival, ImageNet and AlexNet, GANs, AlphaGo and then the transformer era. In the year of the talk, LeCun, Hinton and Bengio received the Turing Award. Fridman adds that the community is much bigger than three people and points readers to Jürgen Schmidhuber's overview of who contributed what, while hoping for more credit sharing and less derision and silos.

He observes that 2019 was the first time it became cool to highlight the limits of deep learning, such as common-sense reasoning and knowledge bases, and notes Rodney Brooks' prediction about stories declaring the era over. His view is that a little criticism is "a little spice in the soup of progress" but not too much. His hopes for 2020 include less hype and less complaining about hype, more interdisciplinary collaboration, and more work on reasoning, active and lifelong learning, open-domain conversation, ethics and robotics.

Fridman says the major frameworks matured and moved toward each other. TensorFlow 2.0 made eager execution the default and promoted Keras as its primary API, while PyTorch added TorchScript for graph representations. Both pushed on mobile, with TensorFlow Lite and experimental PyTorch mobile, and both ended Python 2 support at the start of 2020.

On the reinforcement learning side he says there is no clear winner for beginners. His recommendation is Stable Baselines, a fork of OpenAI Baselines that implements the core deep RL algorithms with good documentation and a few lines of code. He lists libraries from Google, Facebook and DeepMind as alternatives.

His wish for 2020 is framework-agnostic research. Moving a trained model between TensorFlow and PyTorch takes several hours even for someone who knows both, and he would like that to be easy. He also hopes the deep RL frameworks mature the way OpenAI Gym did for environments, and that higher abstractions let scientists in biology, chemistry or astronomy use machine learning without learning Python internals.

Fridman describes how the transformer, demonstrated powerfully by BERT, dominated benchmarks, and how 2019 brought a flood of variants such as XLNet, RoBERTa, DistilBERT, GPT-2, ALBERT and Nvidia's Megatron. He highlights Hugging Face for making many of these models easy to use and Sebastian Ruder's NLP progress tracker for following leaderboards. Megatron-LM scaled GPT-2 to 8.3 billion parameters from 1.5 billion, XLNet combined bidirectionality with Transformer-XL's recurrence, and ALBERT cut parameters by sharing them across layers.

Playing with text generators, he says, shows both their power and their limits. Prompts about arithmetic and gravity produce answers that look fluent but reveal no grasp of the underlying concepts. This is at the core of the debate over whether memorizing language statistics by self-supervised reading counts as understanding.

He also discusses OpenAI's staged release of the 1.5 billion parameter GPT-2, which was worried about misinformation. Fridman's reading is that GPT-2 turned out not to be that dangerous, and that humans are currently more dangerous than AI. He regrets that the public mostly rolled its eyes and that there is no culture of confidential sharing between ML organizations on such risks, since the public tends to penalize people who raise the topic.

Fridman covers the best paper at ACL on multi-domain task-oriented dialogue, and a Salesforce dataset that tests common-sense question answering, where a language model generates the relevant concept before choosing among answers. He frames hybrid systems that combine symbolic AI, deep learning and common-sense reasoning as one of the field's open questions.

He singles out the Alexa Prize as under-appreciated, calling it the Turing test of this generation. A bot must keep a conversation going for 20 minutes, which he contrasts with how quickly people abandon a boring bot. Lessons from the Alquist team include that machine learning is not yet an essential tool for deep conversation, that conversations should be broken into small moves, and that bots must handle tangents by jumping between topics and back.

He adds that systems seem to lack opinions, and that being opinionated and confident is one of the simplest ways to appear intelligent. Another lesson is to maximize entertainment rather than information. He also highlights a Technion project that summarizes code by modeling syntax trees, which points toward automated documentation, and hopes for longer context, more reasoning and the extension of transformer-style self-supervision to video.

Fridman turns to OpenAI Five in Dota 2, which lost at The International in 2018 and later beat the world champions after scaling up training compute about eight times. He cites 800 petaflop/s-days and about 45,000 years of self-play over 10 months of real time, and a 99.9 percent win rate against the previous version. Self-play, he says, is among the most exciting ideas because agents always face a slightly better opponent.

DeepMind learned Quake III Capture the Flag from raw pixels, and in October 2019 AlphaStar reached grandmaster level in StarCraft by playing with a camera-like interface like a human. A professional quoted by Fridman said AlphaStar combines top-pro reflexes with an entirely different strategy and style. He hopes this kind of discovery can teach experts about games and simulations they thought they understood.

Pluribus beat professionals in six-player no-limit Texas hold'em using self-play to build a coarse blueprint and real-time Monte Carlo search, with very little compute compared with the DeepMind and OpenAI projects. In robotics, OpenAI's Rubik's Cube hand relied on automatic domain randomization instead of self-play. Fridman's hope is to study emergent social behavior in multi-agent RL, perhaps one day with a "reinforcement learning self-play psychology department".

Fridman briefly covers three theory results. The lottery ticket hypothesis from MIT shows that small subnetworks within a large network can reach the same accuracy, found by training, pruning low weights and resetting the survivors to their original initialization. Practical takeaways are unclear, but it suggests much more efficient architectures exist and are worth searching for.

On disentangled representations, a 2019 paper argued they are impossible without inductive biases, so those biases should be made explicit. Finding good ones and doing unsupervised model selection across datasets remains open. OpenAI's deep double descent extends the idea that test error falls, rises and falls again, not only with model size but also with training time and dataset size.

His hope is for more fundamental work on model selection, training dynamics and how representations relate to architecture. He also mentions graph neural networks as his first love, useful for combinatorial problems and recommendation systems, and Bayesian deep learning for uncertainty on small datasets.

Fridman calls autonomous vehicles one of the most exciting applications because they put AI in contact with people who know nothing about it. He contrasts Level 2, where a human supervises, with Level 4, where the system is responsible. Waymo had 10 million real-world miles and 10 billion simulated miles, runs 20,000 classes of structured tests, and has started on-road testing with real consumers and no safety driver.

Tesla Autopilot, with 700,000 to 800,000 vehicles and an estimated two billion miles, uses a multitask neural network at the core of perception and action. He says that for Waymo deep learning is the icing, while for Tesla it is the cake. He calls Andrej Karpathy's data engine the key idea: a network does a task, finds edge cases, gets them annotated and retrained in a loop, and that this is what every serious ML company does with little publication.

The tradeoffs are laid out clearly. Camera-based learning offers high resolution and learnability but needs huge data and relies on human vigilance, while LIDAR and map systems are more reliable and explainable with less data but are expensive and need fallback like safety drivers or teleoperation. The open questions are how hard driving really is, whether perception or planning and human intent modeling is the hard part, and how much simulation transfers to reality.

For 2020, Fridman wants more active, multitask and lifelong learning, regular over-the-air updates as a prerequisite for online learning, more public edge-case datasets and simulators such as CARLA, and more balanced, in-depth reporting with less hype.

Fridman finds it exciting, funny and awkward to see AI enter politics, citing Andrew Yang's candidacy and the American AI Initiative. He sees these as early, imperfect steps that show a lack of understanding of fundamentals, and thinks the AI community should communicate limitations and capabilities better.

He points out that tech leaders being called before government are fundamentally running recommendation and advertising systems built on deep learning. He argues companies should publish more about how these work, using a DeepMind paper on Google Play app recommendations, including de-biasing against popular apps, as a harmless example. He hopes for less fear of AI and more dialogue between government and experts on privacy, security and transparency.

His strongest claim is that recommendation systems will be the most impactful AI for decades, because they shape what people see, learn, think and how they communicate. Engineers, he says, must consider this societal effect beyond bias, as an elephant in the room: how these algorithms control how we see the world.

Fridman recommends fast.ai as the best introduction to deep learning, Andrew Ng's Coursera course for beginners, Stanford's courses on vision and NLP, David Silver's reinforcement learning course and OpenAI's Spinning Up. He says the best way to learn is to code and build models from scratch, and mentions a list of 200 tutorials. His book picks are the Deep Learning book by Goodfellow, Bengio and Courville, Andrew Trask's Grokking Deep Learning for beginners, and Chollet's book on Keras.

He closes by calling active learning the most important aspect of real-world deep learning, with too little research, and says systems should start dumb and become brilliant by learning from mistakes. He quotes Hinton that the future depends on a graduate student deeply suspicious of everything he said, and says skepticism is essential only in moderation. In his view, perseverance, openness to old ideas like symbolic AI, a little craziness and a lot of hard work matter most.

Asked about roadblocks, Fridman says deep nets are good at extracting representations but inefficient and poor at accumulating knowledge, while symbolic systems are the reverse. He personally believes more data, bigger networks and better data selection will go much further. He says he wanted to be a psychiatrist before deciding that building is the best way to understand the mind, having learned to program at 12.

He says machines will think and feel emotions, in the sense that displayed emotion is emotion to him, illustrating with Roombas he programmed to cry out in pain. He argues that suffering exists in the eye of the observer, so the first time a product claims to suffer, ethical questions begin. On AGI, he doubts reinforcement learning alone is sufficient because simulation differs from reality and reasoning looks more like programs than functions, and he expects symbolic approaches to matter.

On audio for driving, he notes Waymo uses it for sirens and he is skeptical of those who call audio useless. On the future, he sees few paths for AI to kill humans except as weapons wielded by people, expects AI companions, and thinks AI systems should eventually have rights, perhaps in about 20 years. Asked whether machines will become our masters, he says no: the danger is owners of large tech companies using AI to control people, so AI must be democratized, and he concludes that humans, not AI, will become our masters, and that AI will save us.