From AlphaGo Zero to MuZero: How Self-Play Strips Knowledge Out of AI

Lex Fridman opens by asking how big an intellectual leap AlphaGo Zero was, since it removed the reliance on human expert games. Silver starts by defining self-play. It applies when more than one agent is involved: in a two-player game, the system learns the game purely by playing against itself instead of any real opponent.

The main idea behind AlphaZero, Silver says, was to step back from all the knowledge the team had put into the system. The question was whether a single elegant principle could let a system learn for itself everything it needs to play a game like Go. He notes that removing knowledge matters in two ways: the human knowledge may have been getting in the way, and a system with less built in is easier to transplant into a domain that needs a different knowledge base.

For Silver, that is close to the essence of intelligence: an algorithm that can be placed in any environment, given any goal, and succeed. AlphaZero is a step toward that. It was taken in the context of two-player perfect-information games such as Go, chess and Japanese chess, and AlphaGo Zero was the first step, removing human knowledge from AlphaGo while keeping the Go setting.

Silver says he can name the exact moment the idea for AlphaZero came to him: on his honeymoon, in his most fully relaxed state. The algorithm appeared in full form, before the match against Lee Sedol. He sees a lesson there for researchers who work around the clock trying to come up with the next idea.

The team was so busy trying to beat the world champion that only later could they step back and test whether the idea would work. Silver says they pursued it because it was the deeper scientific question on the path to AI. It also fit a personal rule: he does not like research on questions where the likely outcome is already known, and he sees little value in an experiment where he is 95 percent sure of succeeding.

Back then, he says, nobody knew whether self-play alone could be as strong as a system seeded with human games. He puts it at about 50/50, and even failure would have been an important direction to study. In the end the self-play system outperformed the earlier version of AlphaGo and beat it 100 games to zero.

Fridman asks why the approach works. Silver says AlphaGo suffered from occasional "delusions", where it misunderstood a position and evaluated it wrongly. The errors came from the human data and from the nature of the search and algorithm. The only way to fix errors in any complex system, he argues, is to give it the ability to detect and correct them itself.

More rounds of reinforcement learning do that: when the system thinks it is winning and then loses, it learns that it misjudged something and has a hole in its knowledge. Traced back to the start, the same principle takes a system from a random starting point to slightly better than random, then on to the next level, and so on. Silver says AlphaGo Zero showed no sign of slowing down when training was extended.

Fridman asks whether there is a ceiling and whether fixing one weakness opens others. Silver answers with a falsifiable hypothesis: a future researcher running AlphaZero with more compute should beat the current system 100 games to zero, and then again a couple of years later, indefinitely. Go has about 10 to the 170 states, so the ceiling is out of reach of any computer built from the atoms of the universe.

On whether correcting one error opens others, Silver says yes, it does. He explains that in two-player and single-agent games, self-improvement is known to converge. With enough representational capacity, such as a table of all states, it leads to optimal behavior in the single-agent case and to minimax-optimal play in the two-player case. So even new errors still represent progress toward the best that can be done.

Fridman, a chess lover, asks about the step from AlphaGo Zero to AlphaZero. Silver calls it remarkable that the algorithm needed no modification to take on some of AI's earlier grand challenges. Dropped into chess, it convincingly beat the world's strongest computer program, one that stood in the tradition of Deep Blue and had been refined for years.

The same held for Japanese chess, a far more complicated variant in which captured pieces can be placed back on your own side. The first time the system was run on that game was the version published in the paper, and it also beat the strongest programs. Silver says it worked "literally no touching it," and he finds beauty in an algorithm that works without twiddling.

Silver says AlphaZero is only a step, and the world is a messy place where nobody hands over the rules. Physics may operate according to Newtonian or quantum mechanics at the micro level and relativity at the macro level, but that is not a model people can use to operate day to day. An agent needs to figure out for itself what is going on from a stream of rich sensory input and outgoing actions.

That led to MuZero, a system that learns even when the rules are not given. The team tried it on Atari games, canonical domains for reinforcement learning, where it built a model rich enough to plan successfully. It beat the state of the art in Atari and, with no modification, reached the same superhuman level as AlphaZero in Go, chess and shogi.

Silver describes the learning signal as minimal: win or loss at the end of a game of Go or chess, or a score at the end of a game of Breakout. From that, the system works out the dynamics of the world, not explicitly but implicitly, in a way that lets it plan. Fridman notes this makes the approach applicable to any domain that can be digitized so that reinforcement learning can sense and act, and Silver says MuZero is a further step toward the full problem of unknown, complex worlds.