David Silver: AlphaGo Won by Taking Human Knowledge Out

Lex Fridman opens with David Silver, who leads the reinforcement-learning group at DeepMind and was the lead researcher on AlphaGo and AlphaZero. He asks for the first program Silver ever wrote. Silver remembers it clearly: his parents brought home a BBC Model B microcomputer when he was about seven, and the first thing he did was print his name in different colors and loop it. The machine felt less like a puzzle box than like Lego without a parts limit—once you had a computer, he says, you could do anything.

He pulled out the user guide and then the advanced user guide, started in BASIC, and later learned 6502 assembly. His father became interested in the same machine, gave up his career, and went back to school for a master's in artificial intelligence—when Silver was still seven. The father taught him Prolog and had him query a family tree. Those reasoning exercises are among Silver's earliest memories of trying to figure things out on a computer.

The real obsession arrived at Cambridge, where he studied computer science as an undergraduate and started asking what the field was actually for. The only step of major significance, he decided, was to recreate something akin to human intelligence. The idea was not original to him, but it nested and became a bug he wanted to crack. Unless you believe in something metaphysical, he asks, what are brains doing? At some level they are information-processing systems that take in information, transform it through a program, and produce output that lets a person do astonishing things in the world.

After graduation he spent five years in the games industry writing what that industry called AI: handcrafted agents that could beat him in limited, twitchy scenarios by reacting faster or exploiting a repeating pattern. Those were short-term fixes, not the long-term vision still itching at him. He went back for a PhD applying reinforcement learning to Go. The first program learned, by trial and error against itself, which patterns predicted winning, then chose moves toward those patterns. It beat him. There were no neural networks; this was before the deep-learning revolution. He found it immensely satisfying that a principled self-learning system had come to understand the domain better than he could.

Fridman, who is Russian, still feels Deep Blue's 1997 win over Garry Kasparov as a seminal shock. The AI community of that era, he says, largely treated Go as unbeatable by the brute-force search methods of the day—even with arbitrary scaling of compute. Silver had just come from a panel with Kasparov and Murray Campbell, Deep Blue's author, their first meeting since that match. For Silver the dream started earlier, when he discovered Go at university and it blew chess and Scrabble out of the water: endless depth, and the private certainty that no number of hours would make him a grandmaster. The other path was to get some other intelligence to play it better than he ever would.

The 1990s were a golden era for heuristic search. Chess, checkers, backgammon, and Othello fell one after another to high-performance systems built on that idea. In the same years a million-dollar prize sat waiting for the first program to beat a human professional at Go. When the prize expired in 2000, the world's strongest Go program lost to a nine-year-old child who had given the computer nine free moves at the start. A Go expert beat the same program with a twenty-nine-stone handicap. That was the state of affairs when Silver became interested, around 2000 to 2003. There was almost nothing approaching human-level play, and not for lack of effort.

Go, unlike those other games, had an intuitive property. A player looks at a scatter of black and white stones and can say: this region will be my territory, that region yours, I am probably going to win, and this is about the right move. That judgment is pivotal to human play, and people had no idea how to put it into computers. Chess lets you assign point values to pieces and get a decent static evaluation; in Go both sides have played the same number of stones, captures at a high level are rare, and the only thing that differentiates the position is an intuitive sense of where territory will eventually form. The search space is about 10 to the 170th positions. Traditional heuristic search, the Deep Blue style, simply falls over.

People felt that cracking Go would require something akin to human intuition, and that if you got that, you would unlock many more problems in AI. For Silver that was the moment the problem stopped being a game and became the old bug again. He already believed learning had to be part of the system—not mimicry of human play, but a machine understanding the domain for itself. Without learning you are stuffing knowledge in, and decades of AI had shown the knowledge-acquisition bottleneck: the more you insert, the more brittle the system becomes. The prize-era Go programs were collages of opening books, pattern catalogs, combinatorial game theory, and specialized search for life-and-death. The overall effect was still brittle. His PhD pressed back to first principles: learn from the outcome, and let the system verify knowledge for itself.

After the games startup, Silver took a year out to decide what studying intelligence actually meant. During that year he read Sutton and Barto's textbook on reinforcement learning and felt it resonate: this was what he understood intelligence to be. He wrote to Richard Sutton and asked whether Sutton would supervise a PhD on computer Go. Sutton said that if he was still alive, he would be happy to—he had been fighting serious cancer and was not confident he would see the end of a thesis. That part of the story, Silver says, worked out happily. He went to Alberta, home to a strong games group and to Sutton, a father of the field.

The more he looked, the more he felt RL was not only a path to computer Go but a way to frame what intelligence means as a single, clear problem. He is careful to separate the problem from the solution. The problem of intelligence, he argues, can be formalized as the reinforcement-learning problem, and that formalization captures most of what we mean by intelligence. If a future system reaches human-level intelligence, it will be a solution to that problem. Which particular methods get there is a different question. He wants the field to keep an open mind: current RL algorithms may have a lot of legs, or something important may still be missing.

His definition is plain. Reinforcement learning is the science of intelligence in the form of an agent that interacts with an environment. The agent takes actions; those actions affect the environment; the environment returns observations and one special signal, a reward, that says how well the agent is doing. The problem is simply to take actions over time so as to maximize that reward. A common way to attack a problem that ambitious is to decompose the agent. Silver's three familiar building blocks are a value function that explicitly predicts how much future reward will arrive, a policy that explicitly decides how to pick actions, and a model that explicitly predicts what will happen in the environment. Combinations of those pieces give rise to the familiar branches of RL.

Underneath those choices sits a more basic one: learning itself. The RL problem does not stipulate that you have to learn; you could try to maximize reward without learning, but you would not do a very good job. In any sufficiently large and complex environment, learning is the only way to reach good performance. What you then learn—predictions (value or model) or competent behavior (policy)—gives the system its semantics. Deep reinforcement learning, in Silver's telling, is the family of methods that uses neural networks as a universal toolkit to represent any of those components. The promise is a system with no ceiling: more memory, compute, and experience should make the same functions better and better at whatever they were asked to represent.

Fridman asks whether it is still surprising that neural nets can learn the wildly complex representations needed for policy, model, or value. Silver splits the question. He is not surprised that reinforcement learning should work, because we have an existence proof: human intelligence. At some level it must be possible to acquire experience and use it well in environments as complex as the ones people handle. What did surprise him, and a lot of the community, is that deep learning keeps going. The loss surfaces are extremely nonlinear and bumpy. Low-dimensional intuition says you will get stuck in a local optimum. The surprise is that learning continues.

In high dimensions, when the nets are really large, there is always a way out, always a path to somewhere lower. That new place is still not a local optimum, because another pathway takes you lower still. No matter where you are, learning can proceed without bound. Silver finds that property elegant, beautiful, and somewhat shocking. Humans are attuned to three-dimensional environments; visualizing a billion-dimensional optimization surface is very hard. He attributes the AI winter partly to that failure of generalization. People could only build nets with something like fifty nodes. Imagining a billion-dimensional net with qualitatively different behavior was almost impossible from that vantage.

Theory is still being built, he says, and it is incomplete, but everything known points in the same direction: neural nets are universal not only in representation, which was already known, but in learning ability, which is the counterintuitive part. That makes him wonder what else we are missing because of impoverished intuition. A hundred years from now, superhuman systems may look back at today's RL and find it too complicated—the answer simple all along, the way simple rules in Go or cellular automata give rise to immense complexity. He agrees with Sutton: simple, clear ideas are the ones likely to have the longest legs. The catch is that we still have to make things work today, which sometimes means assembling more complex systems before we know the minimal ingredients.

Go's rules are simple enough that people have speculated we might not be able to talk to alien life, but we might be able to play Go with them: a 19-by-19 grid, stones on the intersections, the aim to surround more territory, with capture as the one nuance. From those rules come millennia of knowledge—about fifty million players, mostly in China, Japan, and Korea. During Silver's PhD a major new idea arrived, first in computer Go. Instead of humans scoring a position or writing evaluation rules, the program would randomly play the game out to the end many times and take the average outcome as a prediction. If black won more of those random games from a given position, the position favored black.

Rémi Coulom developed Monte Carlo tree search in 2006: every node in a search tree is evaluated by the average of random playouts from that node onward. The idea produced huge leaps. Among the strongest programs of that generation was MoGo, by Sylvain Gelly, the first to reach human master level on small 9-by-9 boards. Silver worked with Gelly a little during the thesis. It was a first step toward later successes, but it was still missing a key ingredient: MoGo evaluated purely by random rollouts against itself.

It is remarkable that random play helps at all in a perfectly deterministic game where exact sequences matter. Silver's intuition is that randomization captures something about the nature of the position: sampling random futures from a node is a statistical estimate of that node. Programs based on the method were a first revolution, then they plateaued. No matter how much engineering people poured in, they could not get past amateur dan—strong players, nowhere near professionals, let alone a world champion. That ceiling is the background against which AlphaGo was born.

AlphaGo began at DeepMind as a scientific investigation with Silver, Aja Huang, and intern Chris Maddison. The question was whether there was a fundamentally different way to build the missing intuition: a system that could look at a position and know what to play, or who was going to win. ImageNet had fallen to deep learning in 2012. If a net could understand images well enough to classify them, why not treat the black and white stones as input and let the system understand for itself what they meant? The first AlphaGo paper was a pure deep-learning system with no search at all. On the full 19-by-19 board it reached human dan-level play—suddenly matching the best Monte Carlo tree-search programs that used random rollouts.

Fridman calls that a definitive step away from decades in which search had dominated AI. Silver found it profoundly surprising. The team had a bet: could a system based purely on deep learning, no search, beat a dan-level human? Someone at that rank joined the team, and the first match was against him. Silver tends to be an optimist about deep learning and RL; he backed that side, and the system won. For him that was the moment something special was afoot: a program that did not search could already look at a position and understand it as well as a strong human. From then on, professional and world-champion level felt like an inevitability. If someone was going to get there, he wanted it to be this team.

They scaled up. Demis Hassabis, head of DeepMind, was extremely excited. Within months the system played European champion Fan Hui, the first time a program had beaten a professional. Then came a harder judgment: when to challenge the world champion. They estimated from their own rate of progress and set a match for March 2016 against Lee Sedol. Early AlphaGo used human game records to explore what deep learning could do—predicting winners and suggesting moves from human data alone—because that was the unknown piece. Self-play had been the goal from the beginning; human data was an expedient, a way to break a hard challenge into pieces researchers could understand, then later build the principled version that learned for itself.

Silver had worked on computer Go for more than a decade by the time of the match and had imagined the moment might be of some interest. Arriving in Seoul, he found legions of journalists and about a hundred million people watching live, and realized he had underestimated the significance by several orders of magnitude. It was a little scary. The team knew AlphaGo still had imperfections that would be exposed to the world. In retrospect he thinks it was better they had been insulated, working in a research bubble on scientific questions, because the full public reality would have made it harder to focus.

Just before the match everyone on the team held out a hand, fingers for how many games they thought they would win against Lee Sedol. The spread was wide. Silver predicted four to one, and the number came from data, not vibes. They had established that in about one in five games AlphaGo developed a delusion: a hole in its knowledge that persisted for tens of moves. Without delusions it played far beyond any human; with them, the opposite. That is roughly what happened. In one game Lee found a beautiful sequence AlphaGo had not predicted and led it into one of those holes.

Game one was the first time a program had defeated a world champion at this grand challenge. Late in the game AlphaGo invaded Lee's territory—an audacious claim that stone in the middle of what you thought you owned, I can break it up—and Lee's face dropped. Game two became famous for Move 37. It broke every convention. Watching professionals thought the operator might have entered the wrong coordinate. Go players are taught from childhood that a shoulder hit is played on the third or fourth line; AlphaGo played it on the fifth. It turned out brilliant, made a beautiful pattern in the center, and won the game. Silver calls it a clear instance of machine creativity: an idea not in the human knowledge of the game, now treated as reasonable.

In game three Lee, an 18-time world champion, was losing and stirred up a fight from nothing, building a double ko. Historically no program had handled double kos correctly. AlphaGo dealt with it perfectly. In game four Lee found a brilliant midgame sequence that transformed the position; AlphaGo's evaluation tumbled from winning to no chance, and it began to behave oddly. In game five the team, haunted by the previous delusion, was convinced the system was mis-evaluating again. Only in the last few minutes did they realize that although it had predicted it would win all the way through, it really was winning. It taught them again, Silver says, to have faith in a system once it exceeds your own judgment: once you have bestowed better judgment than the designer, trust it to use it.

Lee Sedol later announced his retirement, saying that even if he became number one there is an entity that cannot be defeated. On the panel the day before this conversation, Kasparov said that when he first lost to Deep Blue he viewed it as his failure, and later came to see it as a success for everyone—a transformational moment for AI that meant more than a personal loss. Silver thinks Lee was more cognizant of that even at the time. In his closing remarks Lee said the match was meaningful not only for AI but for humans: it opened his horizons, brought back joy, and broke conventions so that suddenly anything was possible again. After Seoul, a stronger AlphaGo beat other top humans 60 games to nil. Lee remains the last person to beat AlphaGo.

Kasparov, Silver says, has enormous respect for what the AlphaGo team did. For Kasparov, AlphaZero's later demolition of the world's strongest chess programs was more profound than Deep Blue, because it was done with learning: the system discovered principles and ideas that he, and others, had not always known. At the same panel Silver learned that Magnus Carlsen, then world champion, had attributed a rating peak to studying AlphaZero's games and changing his style to play more like it.

Silver does not want to demean Deep Blue. Heuristic search had powerful ideas. The limitation is that its evaluation function was created by humans, which puts a ceiling on performance and, more importantly, stops the same idea from transferring to domains where you do not have grandmasters ready to encode their knowledge. Most real domains are of that second type: knowledge is messy, hard to extract, or unavailable. AlphaGo was a step toward putting learning in as a first-class citizen—systems that understand for themselves how to judge the value of actions in whatever state they find themselves.

Self-play, Silver says, is systems learning for themselves when there is more than one agent: understand a two-player game by playing against yourself rather than against any real opponent. The main idea of AlphaZero was to step back from knowledge the team had put in and ask whether a single elegant principle could let a system learn everything it needed. Taking knowledge out makes the system less brittle—injected knowledge can get in the way of learning—and more general, because the more you insert, the harder it is to drop the agent into a different world. The promise of AI, for him, is an algorithm you can place in any environment, with any goal, and have it succeed. AlphaZero was a step toward that in two-player perfect-information games.

The algorithm occurred to him on honeymoon, in his most relaxed state, in full form—actually before the Lee Sedol match. The team was too busy trying to beat a world champion to examine the deeper scientific question until later. He tells the story as a lesson for researchers who work 24/7 trying to squeeze out the next idea. He also does not like running experiments where he is already 95 percent confident of success. Whether self-play alone could reach the same level as the human-data system felt like a 50-50 question, and therefore the right one to ask. Even a weaker result would have been worth studying. Lo and behold, AlphaGo Zero beat the previous AlphaGo 100 games to 0.

His intuition is error correction. AlphaGo's delusions came from many sources: human data, the nature of the search, the algorithm itself. In any complex system the only way to address errors is to give the system the ability to notice them—I thought I was going to win, then I lost—and patch the hole. Trace that idea all the way back and it should carry you from a completely random starting point to the highest levels of the domain: random play does stupid things, the learner corrects them, the slightly better agent has new mistakes, and the process can go on indefinitely. They saw no sign of AlphaGo Zero's improvement slowing down. With enough compute, he believes, it would keep discovering more.

Fridman asks whether patching one hole just opens others. Silver answers with a falsifiable hypothesis: if someone later runs AlphaZero with more compute than DeepMind had, it should beat the previous version 100 games to 0; a couple of years after that, the same again; and the chain should continue at least through his lifetime. Go's 10 to the 170th states are a theoretical ceiling, but not one any device built from the atoms in the universe can reach. You do open new errors. In this class of games, though, with enough representational capacity—imagine a table of every state—self-improvement is known to progress toward optimal play in the single-agent case, and toward minimax in the two-player case. New mistakes and all, you are still moving toward the best that can be done.

Without modifying the algorithm at all, AlphaZero was dropped into chess and beat the world's strongest computer chess program, a system of the Deep Blue lineage that had been worked on for years, using knowledge fully discovered from scratch. The same run achieved superhuman play in shogi, Japanese chess, where captured pieces can be dropped back as your own—a much more complicated variant. The published shogi result was the first time they had ever run the system on that game. It worked out of the box: no tweaking, no twiddling. Silver finds something beautiful in an algorithm that just works.

AlphaZero is still only a step. The real world is rich, complex, and messy, and no one hands you the rules. We may know that the universe follows quantum mechanics at the micro level and relativity at the macro level, but those models are not useful for everyday decisions. An agent has to understand the world from a stream of rich sensory input, with actions going out, and still reason the way AlphaZero reasons in Go and chess. That led to MuZero, a system that learns even when the rules are not given. Dropped into Atari, the canonical RL domain of messy pixels, it learned a model rich enough to plan, and beat the state of the art there.

The same MuZero, unmodified, reached AlphaZero's superhuman level in Go, chess, and shogi. No one told it the rules. It played to the end, was told win, loss, or a score, and figured out—not as an explicit rulebook, but as implicit understanding sufficient to plan—how the world worked. That, Silver says, is the fundamental process of facing an uncertain environment: figure out the basic rules of the game for yourself. MuZero is a further step toward the full RL problem, in which worlds are unknown and complex and the agent must learn how to deal with them.

Fridman asks whether AlphaZero showed creativity. Silver starts with a definition: discovering something that was not known before, something unexpected, outside our norms. In that sense, self-play is the essence of creativity. At every stage the agent plays according to its current norms, tries something, and if it works, starts using it. Those micro-discoveries happen millions of times: this pattern is working, I will use it; I can connect groups this way; I can sacrifice stones; I can play a shoulder hit on the fifth line. Leave the process running and it should not surprise anyone when the system finds things unknown to human norms.

AlphaGo Zero left a clean timeline. Opening corner patterns that humans play, joseki, have been refined over thousands of years. Over the forty days of training, the system discovered exactly those patterns; over time, all the joseki humans play showed up through self-play. Then it began to discard some and produce variations humans did not know. Silver mentions a knight's-move pincer joseki that the system replaced with something different. Human professionals now study the new lines, and they have become the norms of top-level competition. A self-play mechanism that knew nothing about people independently found millennia of corner play—and then went past it. Fridman says that first half still makes him feel good as a human being: we were doing all right in this domain.

Does he hope the same ideas reach robotics, autonomous vehicles, and other safety-critical settings? He does. One of the most satisfying things as a researcher, he says, is watching other people use your algorithms in unexpected ways. In the previous couple of years, two Nature papers from teams unknown to DeepMind took AlphaZero and applied it to real problems. One was retrosynthetic chemical synthesis: finding pathways to make molecules, beating the prior state of the art. The second, published a couple of weeks before the interview, used related methods on a quantum-computation problem and again beat the previous best by a clear margin. The lesson, familiar across machine learning, is that if you make something general, it will be used in all kinds of ways. We are just at the beginning.

Fridman turns to the other side of RL: you usually have to specify a reward, and humans do not look like they maximize a single one. What about intrinsic motivation when the true objective is unclear? Silver comes back to the problem definition. Intelligence is clearest when there is some ultimate goal the system is trying to solve. If we do not understand that purpose, we may not have a clearly defined problem at all. Inside that, a system may create its own motivations and subgoals—picking up an object, communicating, influencing someone—and that mechanism may be essential. Those are still subgoals, chosen because they seem to help a later, more ultimate aim. Without a well-defined problem, he thinks, you are admitting defeat: you will not get a clearly defined answer.

Pinned on the meaning of life, Silver warns he is speculating past his expertise, then offers a stack of levels. At one level the universe just follows mechanical laws. At another, the second law of thermodynamics says entropy increases forever, and some people at MIT have treated maximizing entropy as a kind of cosmic goal. If the goal is to dissipate energy, evolution might be a mechanism the universe discovered for doing that efficiently—borrowing a metaphor, he notes, from physicist Max Tegmark. Evolution then has its own goal: reproduce as effectively as possible. That selects organisms that survive and reproduce, which discover brains. Early brains may have been pre-programmed controllers; a later layer is learning systems that can reprogram themselves toward arbitrary goals, which is close to the human feeling that any goal we are given, we can master to some degree.

Those levels are not in contradiction. You can understand a system as optimizing a goal, which is the level engineers use to design, or as atoms bouncing in neurons. Fridman says he missed the newest layer: the artificial intelligence Silver is responsible for creating, which will set the next layer. Silver takes the point. A learning brain might achieve its goals more effectively by building a system that solves them better than it can. When he wanted a program that played Go better than he could, he achieved that goal by creating a new layer rather than by becoming a better player himself. There may be layers beyond that, in which those systems set subgoals for parts of themselves. His personal belief, in the closing words Fridman quotes, is that we have seen a turning point: abilities such as intuition and creativity, once thought to belong only to the human mind, are accessible to machine intelligence as well.