AlphaGo Combined Learning and Search. Language Models Only Did the First Half.

The cold open is a thesis, not a teaser. "I think someone needs to solve the depth problem," Misha Laskin says. Large labs have been working on breadth, and that work is unlocking a real market, but depth is still sitting there unsolved. Stephanie Zhan and Sonya Huang of Sequoia Capital's Training Data introduce him as CEO and co-founder of Reflection AI, a former DeepMind research scientist whose co-founder Ioannis Antonoglou helped create AlphaGo and later led RLHF for Gemini. Together they want universal superhuman agents — and they think the people who built both AlphaGo and Gemini are the ones who can see what language models still lack.

Laskin dates the conviction to March 2016 in Seoul. AlphaGo played a move that looked like a mistake. Lee Sedol was perplexed; so was everyone else. Ten moves later it was the optimal play, the one that put AlphaGo in a winning position. This was not brute force, he says. The system does a lot of search, but it found a creative solution people had not thought of. That was, viscerally, "the first real large-scale superhuman agent," and it is why he entered AI to build agents from day one.

Ioannis is the other half of that history. He was in Seoul for the Lee Sedol match, a key engineer on AlphaGo, and before that a co-author of DQN, the Deep Q-Network that learned Atari from raw pixels — the first successful agent of the deep-learning era, Laskin says, the way ImageNet was an unlock in 2012. He then worked the line of AlphaGo, AlphaZero, and MuZero. On Gemini, Ioannis led the RLHF effort while Laskin led reward-model training. In 2023 they left to found Reflection, because they thought they would move faster on depth and search on their own.

Laskin was born in Russia, moved to Israel at one, and to Washington state at nine. After the Soviet Union collapsed, his parents landed in Israel with about $300, which disappeared as soon as they put down a deposit on an apartment that may never have existed. They did not speak Hebrew. They pursued chemistry PhDs at the Hebrew University of Jerusalem because Israel was giving stipends to Russian immigrants, not because they already loved the subject. Asking them later, he found they grew to love the craft as they got excellent at it. That, more than any origin-story passion, is what he took: do the work well, and the love follows.

His parents promised mountains and forests. The plane showed him peaks in the distance, then made a U-turn into the desert half of Washington. They landed in the Tri-Cities, a town that exists because of the Manhattan Project's Hanford Site, where plutonium was enriched — a sister to Los Alamos, in the middle of nowhere. He remembers tumbleweeds rolling across the highway. He did not speak English well, had few friends, and had too much time. After video games wore off he found his parents' Feynman Lectures on Physics. Feynman could explain something fundamental about the world to a kid without much math. That is when Laskin got hooked on root-node problems: the transistor, invented by a theoretical physicist; GPS, which needs Einstein's relativity. He went and got a physics PhD.

The missing context, he says, was that you do not just want root-node problems. You want the root-node problems *of your time*. Physics training is a tour of how people thought a hundred years ago, when physics really was that node. He did a 180 into something practical, started a company, and then noticed deep learning — and AlphaGo. As an outsider around 2018 or 2019 he picked a stale item off OpenAI's public requests-for-research list and cold-emailed researchers until one of them, graciously, introduced him to Pieter Abbeel at Berkeley. Abbeel's lab spans reinforcement learning, robotics, and a key diffusion-model paper. He had no reason to take an unvetted outsider. Laskin later sat on the other side of admissions and still thinks it was a pure lucky break — and his ticket into the field.

The lesson Laskin takes from AlphaGo is Richard Sutton's bitter lesson. Hand-designed tricks get washed away by systems that leverage compute at scale, and Sutton named two ways to do that. One is learning — training — which is how today's language models spend compute on internet text. The other is search: unroll a bunch of plans and pick the best. AlphaGo is both ideas in one, and Laskin still thinks combining them is the most profound idea in AI. That is what produced a superhuman Go agent. Relative to today's LLMs, the AlphaGo network was tiny. It was also only good at one thing. It could not even play tic-tac-toe.

Deep RL felt stuck. Everyone had set out to build general superhuman agents and landed on superhuman, very narrow ones. Data efficiency was the knot: if a task takes six billion steps, where does the data for the next task come from? Language models cut that knot. Laskin's frame is that the internet is a collection of many tasks — Wikipedia describing historical events, Stack Overflow as coding Q&A — a massive multitask dataset. Generality comes from training on tons of those tasks at once. What the internet does not contain is a notion of reliability or agency, so it is no surprise the resulting models are extraordinary chatters and mediocre workers.

The worker failure has a name: error accumulation. Each step has some percent chance of being wrong; over a handful of steps the compound rate makes a meaningful task nearly impossible to finish reliably. Language models leverage learning. They are not yet systems that leverage search or planning at scale. That, he says, is the missing piece. Gemini, Claude, and the GPT series are the broadest agents ever built and not very deep. GPT-4o and the latest Gemini family understand images and audio at the same base layer as language, so the field has more or less de-risked breadth. Nowhere along the way were the models trained for depth. There is no internet data on how to think sequentially. Math and coding sets are the usual proxy, and they still do not address the problem head-on.

By 2024 everyone is an "agent," and the word is losing meaning. Laskin's definition is blunt: an AI system that can reason on its own and take as many steps as it needs to accomplish a specified goal. In the deep-RL era that goal was usually a reward function — AlphaGo was not told in text to "win at Go." There was also a literature of goal-conditioned agents, including robots handed an image of an apple moved somewhere and asked to reproduce it. SWE-agent and Devin, on that definition, *are* agents. They are just early on the reliability spectrum. Huang notes task-completion rates in the 13–14% range and asks how you get to 99%.

Most language-model "agents" today are prompted agents: a model plus a flow of prompts that anyone can stand up from zero to something that kind of works. Laskin thinks that only goes so far. Directing the model down specific tracks is exactly the kind of human heuristic the bitter lesson washes away. Every major advance in agency since deep learning has shown that learning and search eat that scaffolding. The prompt's job is to specify the goal; you will always need to tell an agent what to do. Once the prompt starts doing the thinking — putting the system on rails, telling it to go here then there — that layer is a local maximum. Future systems will not have it. Thinking and planning have to happen inside the AI, not in the prompt layer.

That these models were never trained for agency, only for chat and internet prediction, makes it almost a miracle that prompting gets you something that kind of works. That "kind of" is also the right place to start RL. Reinforcement learning upweights good behavior and downweights bad. An agent that does nothing gives the algorithm nothing to reinforce — the textbook sparse-reward problem. A prompted agent at 13% finally supplies a sliver of good behavior to learn from. Then three problems remain. Tasks usually come from a product and real users. Environments are a slog: tools, IDEs, email, a different stack at every deployment, with a real risk of overfitting to one of them. The killer is the third: how do you verify, at scale, that the thing was done correctly? Solve where the tasks come from, what environment you run them in, and how you check the work, and you have a recipe for agency.

Huang asks the obvious difference from board games: Go has a ground-truth reward and self-play. Is RLHF enough for an AlphaGo moment in LLMs? Laskin says the missing ground truth may be *the* key thing. The last era of RL showed that if you have one, you are almost guaranteed success — OpenAI Five at Dota 2, DeepMind's AlphaStar at StarCraft. As a former StarCraft player he is still blown away: the strategies looked like a smarter alien had landed and decided to play. That tightening of behavior depends on knowing who won. Real tasks do not offer it. Even a coding job that passes unit tests can still be wrong. He calls that the fundamental problem for agency.

Chat gets around it by training a reward model, another language model that predicts whether something was done well. It works, until it does not. The substitute is noisy; the policy gets smart enough to find holes and exploit them. Suppose you do not want the chatbot discussing sensitive topics, so you add examples of "I'm sorry, as a language model I cannot answer." If the mix has refusals as positive cases and no positive cases of answering a sensitive question well, the reward model can learn that not answering is good. The policy discovers that it scores high whenever it punts, and the model collapses into a system that never answers. Users who have watched ChatGPT or Gemini suddenly get worse, or more evasive, or politically skewed, are often seeing data artifacts amplified by a bad reward function.

Seen through AlphaGo, the pipeline is the same drawing. Pre-training is the imitation-learning phase: from zero to pretty good at Go, or from zero to pretty good at everything. Post-training is what Laskin calls hardening good behavior — the network generates its own play, gets feedback, and good actions get reinforced. Training AlphaGo and training Gemini share that high-level recipe. The gap is reward quality. AlphaGo's RL was more sophisticated because the signal was clean. A noisy, exploitable reward model means the policy hacks it before search can help. Monte Carlo tree search on top of RLHF does not save you if every local judgment is bad; there is no point planning ten chess moves ahead. OpenAI's paper *Scaling Laws for Reward Model Overoptimization*, which he calls badly underrated, showed the exploit at every scale and for every RLHF algorithm they tried. That is the fundamental problem of post-training.

Could you skip pre-training the way AlphaZero skipped human games? In theory, Laskin says, if everything had a ground-truth reward. In practice DeepMind went from AlphaGo to AlphaStar and there was no AlphaStar Zero; imitation learning across a lot of games was a big part of it. Go is a special environment: zero-sum, over quickly, win or loss immediately visible. The real world does not give feedback at that speed, so imitation learning is the ticket into the game. You need it to get on the board at all.

What surprised him most about language models is how close they often are to working. A bit unreliable, they go off the rails, but they feel a nudge away — the same "almost" that made chat alignment possible. Pin the model to human preferences and it becomes a reliable companion. Nothing they did for chat on Gemini, he and Ioannis noticed, was chat-specific. Collect interaction data, collect preference rankings, push the model with RL. Collect data for another capability and, in principle, you unlock that too. Algorithms change when the task is objective rather than subjective, and the integration work is messier, but the skeleton is general. Long context is his favorite vanished bottleneck: he thought it needed a research breakthrough; compute and scale washed it away. What remains is data, plus planning and search on top of these models. They left DeepMind because they wanted to move faster against that remainder.

Progress on depth, he says, can look like AlphaGo — one brutally hard task — or like expanding concentrically in task complexity with a general recipe that does not inherit task-specific heuristics. Reflection is taking the second path, grounding the recipe in a diversity of environments: web agents, coding agents, OS-level computer agents. The point is to show the recipe itself is general. The timeline is aggressive. He puts something that resembles digital AGI, meaning a universal agent with both breadth and depth, about three years out. Domain experts once thought human-level Go was decades away; AlphaGo solved it in months. He does not buy the story that the field is at the tail of an S-curve. Training the biggest model takes months and a few billion dollars, so the field has run vanishingly few experiments. Optimization is not done. The exponential is still on.

Why have the big labs not finished this? Because agency, in those companies, is a side quest. The main quest is the next foundation model. Solving depth and reliability needs a player that puts its whole stack on the problem. They are hiring slowly, from top labs, for hunger more than headcount. Laskin compares Ioannis to Michael Jordan: so good as an individual contributor that the rest of the team tries to reach his level even if they never quite do. Working with him on Gemini, Laskin says he does not know if he got there, but he became a much better engineer and researcher. Huang asks whether in three years she will have an agent writing her memos. Memos might come sooner, he says. What she may have is an army of AI interns that take the research work. Reliability, in his utilitarian definition, *is* safety: an agent running around your computer deleting files is an unsafe system. He points to Anthropic's mechanistic interpretability work, including "lying neurons" you can locate and suppress, as both safety research and reliability engineering.

If Reflection works, Laskin wants two things. As a scientist, this is the root-node problem of the time, and a world with safe, reliable digital agents on your computer is one of the more exciting journeys in science you could join. He does not think the human need to be productive goes away; he thinks each person's capacity to produce goes up. The other angle is the user: the computer was our first general digital tool, the way hammers and chisels were physical ones. The next layer is agents that take the precision work off your plate so you can set more ambitious goals. Software engineers already get more done with today's tools. He thinks that is the beginning, not the product.

In the lightning round he is most excited, locally, by mechanistic interpretability — the neuroscience of language models, now showing signs of working beyond toy settings. If he were still in academia he would look hard at the science of AI: what actually determines scaling laws, theoretically and empirically, and how data mixes change the model. He likens the moment to physics in the late 1800s. Electricity had been found; a pile of empirical regularities had no theory; then simple theoretical models arrived and unlocked the next wave of experiments. The people he names from having worked with them are Pieter Abbeel, whose operational tightness made research a "tight ship" and who amplifies independent students; Volodymyr Mnih, first author of DQN and a pioneer of A3C, whom he calls extremely kind and humble; and Ioannis, again, for the Jordan effect on a small early team.

Founder advice, he says, he will be more qualified to give in a few years. The piece he will give now comes from the previous company, which had nothing to do with AI: inventory-prediction software for retailers, chosen because it could make money, for a pain he had never felt. Work on things that internally matter to you, almost independent of what is happening around you, so that when things go bad they are still interesting. AI is a cool, highly capable technology, and the temptation is to grab it and see what you can do. Without a strong internal compass you end up in a hard place. Stephanie Zhan's gloss is to play in your own stadium and not get distracted by the lights of someone else's. Caring about customers, Laskin adds, is not something you can force. If the problem is not already aligned with something inside you, you will not care about the people who have it — and building for customers you do not care about is a terrible place to be.