Move 37 looked like a bug. LLMs still need their AlphaZero moment

March 2016, Seoul, game two against Lee Sedol. Move 37 lands, the commentary booth goes quiet, and professionals look at one another. What almost nobody knew is that DeepMind’s viewing room had the same first reaction. Ioannis Antonoglou still remembers thinking AlphaGo had made an error — that it had hallucinated a move it did not mean to play. It turned out to be a brilliant, unconventional stone that even top humans had not considered, proof that the system had a deep understanding of the game and real creativity.

Nine years later he sits with Stephanie Zhan and Sonya Huang on Sequoia’s Training Data podcast. He was a founding engineer at DeepMind, a core AlphaGo contributor, later a co-lead on MuZero, and is now building agents at Reflection AI. The cold open of the episode is the lesson he took from those matches: these systems are not classic computers that always produce the same answer. They are stochastic, creative, full of blind spots, and they hallucinate the way language models do. The only way to see where they break is to have the best humans push them.

A few numbers from later in the conversation are worth parking up front. Early AlphaGo ran on 1,200 CPUs and 176 GPUs; the Lee Sedol version used 48 TPUs, Google’s first custom accelerators, still primitive and unstable. AlphaGo Master later played 60 online matches against strong humans and won every one. Antonoglou’s forecasts: SWE-bench agents cross 50% in one to three years and 90% in three to five, once agents become reliable; LLMs still owe the field an “AlphaZero moment” within five years.

DeepMind, Antonoglou says, was the first company to embrace AGI from day one — systems that would match or exceed human intelligence. The open question was how to build it and, more importantly, how to measure intelligence so research could actually improve. Games were a natural testbed. Demis Hassabis came from the games industry; Shane Legg’s PhD defined AGI as a system that could learn to complete any task. Video games offered a controlled yet complex environment with a clear score.

Games do not capture the real world. The real world is messy, unbounded, and a tougher nut than even the most complex game. Methods still travel. PPO, now used in ChatGPT’s RLHF, was developed in OpenAI Gym, MuJoCo, and Atari. Monte Carlo tree search grew out of board games such as backgammon and Go, went into MuZero, then into YouTube video compression, a version of Tesla’s self-driving stack, and even an AI-controlled aircraft pilot. Games donate methodology; the real world adapts it.

Board games had been AI’s yardstick since Turing and Shannon wrote chess programs; IBM’s Deep Blue beating Garry Kasparov in the late 1990s was the milestone of that era. Go is a harder problem because there is almost no evaluation heuristic. In chess you can count pieces and ranks and draw a conclusion. In Go, professionals say that after enough games they just see it in their gut. Encoding that gut feeling into an AI system is why solving Go was considered the holy grail — almost impossible, and somehow within reach. In 2015 and 2016, deep learning and reinforcement learning were the new kids on the block. Most people thought deep learning was another fad that would not last.

Huang asks him to explain AlphaGo as if she were a fifth grader. A neural network is a black-box function: enough data and it learns a mapping from input to output. AlphaGo had two. The policy network looks at the board and returns a shortlist of moves worth considering. The value network looks at the same board and returns a winning probability — the gut feeling, turned into a function. With both, the program can play in its imagination: its best moves, the opponent’s best replies, then minimax. It wants to win, it knows the opponent wants to win, so it picks the move that still looks best when the opponent also plays optimally.

The efficient way to do that imagining is Monte Carlo tree search, a heuristic for which futures to consider. Training came in stages. The policy network first imitated a large set of professional human games. Then it played itself, and policy gradients improved it: after a win, raise the probability of every move in that game; after a loss, lower them. Iterate long enough and the policy strengthens. That stronger policy then generates a mountain of self-play games labeled with the eventual winner, and a second network learns to predict the outcome from a position. That is the first value network, ready to be combined with the policy inside search.

Antonoglou calls AlphaGo less a research breakthrough than an engineering marvel. Early versions ran on 1,200 CPUs and 176 GPUs; the Lee Sedol match used 48 first-generation TPUs that were primitive and unstable. Latency had to be crushed, throughput raised, training infrastructure built from scratch, every step implemented at the limit of what was possible, and researchers still needed to try ideas fast. Zhan recalls him saying the project existed in part because Google had TPUs and needed a test customer. History, she notes, is sometimes that accidental.

How much conviction did the team have? Cautious optimism, he says. Aja Huang, one of AlphaGo’s lead developers, is a strong amateur and had worked on computer Go for a decade before the project. Internal leaderboards showed AlphaGo clearly stronger than anything that had come before. Go is complicated enough that a worry remained: was it truly as good as they believed? Deep reinforcement learning looked like the answer on every metric they could measure, but the systems are stochastic. They hallucinate. You have to push them until they break, and only the best humans can do that.

Move 37 in game two was the spectacular case: a strategy top humans had not considered. The room thought it was a bug. Game four’s Move 78 was the mirror. Lee Sedol played something unexpected; AlphaGo misread it as a mistake, decided it was already winning, and failed to respond. Human ingenuity hit a blind spot. After the match they built a benchmark to quantify those mistakes, then tried a stack of fixes. The method that actually worked was the familiar recipe at higher scale: switch to a deep ResNet with two output heads, a bigger network, more data, then move on to AlphaZero. Hallucinations went away. Scale and data, the well-known formula, solved it for them too.

Did higher-quality professional games matter? His answer surprises the hosts. What mattered was self-play. They already had access to the most competent Go player in the world, and they used it to generate the best-quality games. They did not need human experts. They had an AI expert in house. It was not human.

AlphaZero learned entirely from scratch through self-play, with no human games. That simplified training and showed that AI can go from random weights to superhuman play purely by playing itself, which opened domains that lack enough human data. It also cleaned up the hallucinations and blind spots, because the method itself was better. Starting without professional weights sounds like a worse initialization. Empirically it was the better one.

The fifth-grader difference is in training, not architecture: still a policy net, a value net, and MCTS. Take a set of weights, combine them with search, and you get a stronger player. Distill that stronger player back into the original networks and repeat. In reinforcement learning that loop is a policy-improvement operator. Antonoglou says this is exactly what people are trying with Q* and synthetic data today: take a policy, do planning or search or extra compute, derive a better policy, then imitate it back into the base model. AlphaZero used MCTS to produce better trajectories, trained on them, and iterated until it converged on an expert.

AlphaZero still needed a perfect simulator — someone to tell it the rules — which the messy real world does not provide. MuZero learns an internal world model from the agent’s own trajectories, then searches inside that model. It mastered Go, chess, and shogi, and also visually harder games such as Atari, without being handed a simulator. The idea rhymes with Sora: learn how the world works from data. The difference is that MuZero only predicts what planning needs — future reward, future value, future policy — because model capacity at the time could not afford to model everything.

The methods left the lab. Chemistry has AlphaChem; people in quantum computing tried AlphaZero for optimization. MuZero was incorporated into a version of Tesla’s self-driving system, reported at the company’s AI Day, and Antonoglou believes it is used inside YouTube’s compression. New techniques, he adds, always take time for industry to fully digest.

LLMs and multimodal models have brought extraordinary progress, he says, and they have a hard limit: human data. Everyone talks about the data wall. That is reinforcement learning’s home ground. He defines RL broadly as anything that learns through trial and error. The agent generates its own experience, so it can improve where human data is scarce or absent. Define the problem correctly, and you exchange compute for intelligence the way AlphaZero did: more compute, bigger networks, more games, a deterministically stronger player. Scaling laws already give that predictability while human data lasts. After it runs out, high-quality synthetic data is the only way to keep going, and that data, in his blanket definition, comes from some form of RL.

There are two messy worlds. Building a robot controller is one; operating in digital space is another. He expects digital AGI much earlier than robotics AGI, because a computer is contained even when it is noisy — not the full flying physical world. DeepMind used to split RL into a problem setting and a solution space. The problem is: take a policy, interact by trial and error, improve it. The solutions are value functions and classic methods, which now have to be adapted to LLMs — a reward model, or asking the model for feedback on its own output, instead of a separate value network. Q*-style methods take a policy, plan over futures, keep the better ones, and ask the model to imitate them.

Bullish or skeptical on synthetic data does not matter. It is an obstacle that has to be walked around. Humans can produce only so much text, and the systems need to start taking actions and learning from their own mistakes. The naive version — train on whatever the model emits — collapses. New methods never work out of the box. He is optimistic, and he suspects Q* and its cousins already exist behind company trade secrets, in a world that no longer publishes breakthroughs the old way. Reasoning, too, will not fall out of next-token imitation alone. The internet has volume but not enough traces of someone explaining a chain of thought in detail. Models have picked up some of that by imitation; to push further you have the model generate, interact, and get told when it is right.

On timing, he has read that text data lasts at least another year, with extra modalities maybe buying one more. He does not think anyone will truly smash into the wall, because synthetic-data methods will be in place by then. The wall arrives; nobody notices.

AlphaGo and AlphaZero were compute outliers on the scaling charts of their day, and they showed that you can buy performance at inference by planning, not only at training. The same systems also previewed today’s LLM pathologies: creativity and hallucination, blind spots, susceptibility to adversarial attacks. The contrast that matters most to him is reliability. AlphaGo Master played 60 people online and won every match. Current LLM agents produce a stunning demo one time in two, or one time in ten; the other nine they lose their way and do nothing. “What we need to do is find a way to make these LLM-based agents equally robust to the ones we had with AlphaGo and MuZero and AlphaZero. This is the new open question.”

He groups the work that would get there into three buckets. Reliability: models that notice their own mistakes and recover, the way humans do. Planning: think and search before acting, trading test-time compute for performance, a pattern he expects to see more of. In-context learning: pick up a new task or a new tool from a few examples, so knowledge is not only in the weights but acquired on the fly by interacting with the environment. He expects large progress on that last one in the next couple of years.

Startups versus big labs is a world he has lived on both sides of — he was among DeepMind’s first employees. Startups have agility, focus, no revenue product they cannot deprecate, less red tape, and access to open-source resources the large labs will not touch. Inside big labs he uses a gaming metaphor: a main quest that gets leadership attention, and side quests that are supposed to feed it but starve of resources and atrophy. Define the main quest and cut the rest. His personal list of field-defining projects runs AlexNet, DQN (he watched Volodymyr Mnih build it), AlphaGo, AlphaFold, ChatGPT, and GPT-4. The two researchers he names are David Silver, his PhD supervisor and the leader of AlphaGo and AlphaZero, and Ilya Sutskever, whose obsession with scale, in his telling, made GPT-3 and GPT-4 happen.

In rapid fire he puts dates on the table. In five to ten years, models become powerful, reliable agents that execute independently and get adopted at scale, especially in science and healthcare — the work Reflection is doing. SWE-bench 50% in one to three years, 90% in three to five, gated on reliability. ChatGPT’s launch was the LLM AlphaGo moment, a public demonstration of a decade of progress. What the field has not had is an AlphaZero moment, where more compute translates directly into intelligence without human intervention. He thinks that arrives in the next five years.