The Biggest Bug in Today’s Models Is That the Weights Never Change

Sonya Huang and Alfred Lin, hosting Sequoia’s *Training Data*, open with Rich Sutton — the researcher who helped invent reinforcement learning, wrote the field’s textbook, advised David Silver, and authored *The Bitter Lesson*. He is joined by Khurram Javed, his former University of Alberta student and now co-founder. The two have just started Oak Lab, whose pitch is agents that keep learning from their own experience rather than from us. Sutton does not think this is a radical stance. “I’m not weird,” he says. “The field is weird.”

The conviction goes back to an AI winter. In 2003 he was, in his words, “actually dying of cancer.” After another remission he figured that if dying was taking this long he might as well get another job, so he went to Alberta and started teaching. He did not die. The harder question, he says, is why he kept working on the research when he thought he had only months. He cites Benjamin Franklin: people do things out of habit or vanity. He guesses it was habit. The field later caught up — AlphaGo, *The Bitter Lesson* in 2019, a Turing Award with Andrew Barto in 2024 — and he is still repeating the same ordinary claim: you act and you learn, always. Continual learning did not need a special name until the field made learning a phase that stops.

Asked how he picked students over decades, he refuses the credit. Sometimes you get lucky, sometimes you don’t; David Silver picked him. Khurram’s path was equally unplanned. He finished a master’s elsewhere, meant to go into industry, got pulled into a collaboration that started because someone mentioned his work in a meeting with Sutton, and six months later the project had become a thesis. He never filled out an “will you be my advisor” form. Life, as Sonya puts it, works in unexpected ways — and that informal pairing is now a company.

Sonya notes that “is this Bitter-Lesson-pilled?” has become a meeting cliché, and that the phrase is probably being tortured. Sutton says the essay was a long time coming — as much a reaction to the symbolic-AI decades he lived through as to ImageNet or AlphaGo — and that he had given the talk a year before he posted it. The recent 26-word version on X is the one he wants used: don’t be distracted by human knowledge the way AI traditionally has; focus on methods that scale with computation, like search and learning. Alfred hears “data.” Sutton corrects him: the thing to stop scaling with is *human input*. You still need fancy algorithms. They just have to be the kind that get better as compute grows.

Are LLMs consistent with that lesson? Both, he says. They are a positive example because they let you drink in the internet and scale capability with compute. They are a negative example because the internet is finite, human text is finite, and the world is “massively bigger than everything we stored on the internet.” A system fed on human knowledge eventually gets held back by human knowledge. That is the setup for the fight that takes up the next stretch of the conversation: if the fossil fuel of the web is running out, is synthetic data the general, compute-scaling way past it?

Sonya puts the labs’ current bet on the table: synthetic data as a way past the human internet. Sutton’s answer is a flat no. “That’s just a big mistake.” It might be “the next big lesson.” The Alberta group has been sitting with a name for this for five or ten years — the big world hypothesis — and Khurram wrote it up as a short paper. The world is endlessly large; there are infinitely many things to learn. If you could take humans out of the loop and learn from experience, a system could keep going. Synthetic data does not do that, because someone still has to decide what counts as good synthetic data. Right now that someone is a human expert.

Khurram’s test is a bat-like drone that flies by echolocation. What is the right synthetic data for that? You would have to hire domain experts to invent it first. The expert has to exist, so scale is stuck on human expertise. Sonya objects that you can have infinite synthetic worlds while the real world is finite. Sutton’s reply is that a little program generates a little world, and it will be wrong. He cannot get synthetic data for what is going on in the hosts’ minds right now, even though that matters because he is talking to them about investing. He cannot get it for motor friction on a robot either. “The world is infinitely complex, and any simulation of it is like microscopic.”

Sonya tries self-driving: the newest companies train mostly in simulation, then do some real-world post-training, and the pipeline works. Khurram asks how many engineers it took to build that sim, and whether we are ready to say the only problems worth solving are the ones where we can hire that team. The sim is never right on the first try. They train, find an unacceptable sim-to-real gap, and engineers fix the simulator. That loop *is* continual learning; the workers just happen to be people. Why not remove them and let the agent do it? Sutton adds that once the car is actually driving, something unexpected will happen, and that is when you really want to learn from experience. Simulation and planning are valuable — if the agent learned the simulator from its own experience, so it can repair a wrong model instead of waiting for a human to notice.

The hypothesis they want taken seriously is that the world is massively more complex than any agent’s mind, obviously so because the world contains many other agents. Nothing can be optimal. Approximations will be severe. Continual learning is not a taste; it is the only way to retune an approximation to the particular slice of the world you have just walked into, rather than to all the slices you are not in. Prior knowledge, Sutton is careful to add, is not the enemy. *The Bitter Lesson* says so in the first paragraph: you can load some prior and then learn. Nature and nurture became enemies in practice because people who loved existing knowledge wanted it to win. The current fashion is the opposite error — pour knowledge in and stop.

Sutton’s complaint about LLM assistants is almost too simple to argue with, which is why he keeps repeating it. The model talks to people and interacts with the world, and “the weights never change.” Asked whether today’s popular assistants are experiential learners, he answers, “Are you serious?” Sonya tries memories and in-context learning. He says it again: the weights never change. Giving more context changes *state*. The network already learned, in training, to use a different state; that is not the model learning. Alfred asks whether the mistake is too much pre-training and post-training before launch. Khurram says the only real disagreement is the last step: pre-train as much as you want, post-train as much as you want, then the user starts using it and learning stops. A frozen system is still expected to offer PhD-level judgment. That, Sutton says, is the weird view — not his.

Two rare counterexamples come up: Cursor’s Tab completion model, and Composer, whose weights do get updated. Khurram’s caveat is the recipe. Data from thousands or millions of users is batched, then there is one policy update. That can improve the shared model. It is a terrible way to teach *your* model one specific thing, because you are fighting a hundred thousand other people for the gradient. Sonya’s current mental model — common skills in the weights, personalization in context — is half right. Context can live in state. The weights still have to be updatable, or personalization never gets past “change the prompt.”

The biological picture Khurram wants copied is harsher than a preference toggle. Proprioception — internal sensors for where the body is — is the foundation of walking policy. Some people lose it completely and cannot walk at all. Over two or three years they can learn to walk again by looking at their feet, replacing a fact that had been true for twenty years with visual feedback. That plasticity, the ability to overwrite a long-settled truth, is the capability he wants in machines. Sutton’s parallel claim sounds extreme and he treats it as obvious: no animal learns by supervised learning, because nobody gives you examples of how your muscles should twitch. School is a tiny, recent overlay. Squirrels do not go. Humans spent most of evolutionary time without it. You can be told that Paris is the capital of France; nobody can tell you the muscle motions that produce the word, and if you copied Sutton’s mouth exactly, a different sound would come out. Only your body knows.

Sonya objects that squirrels do not prove theorems, and theorems are what school is for. Sutton reaches for Moravec’s paradox: the “advanced” things we admire — theorems, chess — are the ones computers find easy; moving, seeing, and attending are hard. A squirrel will always find a new way into the bird feeder and will break a fall in real time. Alfred pushes harder: building a rocket requires imagining something that has never happened. Sutton agrees you have to plan and imagine. Khurram’s cut is that a human from a thousand years ago is fundamentally the same machine; culture accumulated. Paradigm shifts — Einstein is the cliché — happen when someone who has piled up knowledge builds new abstractions from their own experience and plans with them. “That skill of coming up with new abstractions and then learning models and planning with them… is totally missing in our current systems.” The field can do AlphaGo and Lean because the model or the operators were given. Sutton’s indictment: there is no instance yet of *learning* the model and then planning with it, at least not with self-discovered abstractions.

In 2022 Sutton published the Alberta Plan, twelve research steps meant to turn a worldview into chunks. Step two is continual deep learning, which he calls almost the most important because it unlocks the rest: if you can keep learning, you can keep updating a world model. The second half of the plan is about getting abstractions right — not *the* right abstractions, which no one can name in advance, but the ones that fit the world the agent is actually in. Khurram’s illustration is elite athletes, who invent weird niche terms for motions, and sometimes have no name at all if they train alone. Where those abstractions come from is the missing piece.

Sonya asks whether continual deep learning is an algorithmic gap or a deployment and privacy gap, because naive weight updates from user data are possible today. Khurram says it is algorithmic. Update the whole model from one sample and you punch a hole in everything the network already knew — catastrophic forgetting. Cursor dodges it with a large multi-user batch. Most settings are a single stream, and the naive update “completely destroys your prior knowledge.” Sutton’s one-liner: it is “totally curable” if you have the right algorithm.

The cure has two parts. First, step-size optimization: every weight gets its own step size, meta-learned, so most of the network barely moves when a new example arrives and only the right places change. Second, generate-and-test in feature space — propose new units without waiting on gradients, because gradients only creep in directions already known to help and cannot grow complexity. The concrete algorithm is continual backprop, published in *Nature* a couple of years before this taping. It is ordinary backprop plus a steady planting of new, randomly initialized units. Vanilla backprop’s randomness exists only at initialization and then gets used up; continual backprop keeps injecting a little generate-and-test, and backprop itself is the tester. Put those together well, Sutton says, and you get a new generation of vastly better continual deep learning. They hope to do that in the next couple of years.

Can you bolt this onto today’s LLMs? Not by taking a trained model and starting to update it. The new algorithms meta-learn how to learn; they acquire knowledge and the skill of future learning at the same time. You have to train a new foundation model from scratch with them. Khurram notes that similar ideas had a moment around 2016 to 2018, but in limited settings with a prepared distribution of problems. Oak wants the general case: a single stream of experience, an algorithm that would apply everywhere.

Pressed for Oak’s most radical bet, Sutton rejects the word. The most *ambitious* thing is to cover the full spectrum of knowledge, tiny to large — how you take an airplane from one city to another — in a uniform way, and to make the system self-maintaining. What keeps an LLM’s knowledge correct is a lot of post-training and then a freeze. Human minds keep changing and somehow stay organized, coherent, and able to settle back instead of drifting into “crazy land.” A self-consistent mind that can keep training itself and stay coherent is the company’s biggest ambition. He thinks it is within reach in 2026, not because every step is solved, but because they have a vision and inklings.

Alfred translates the vision into a number: a trillion-parameter mind on 20 watts. Khurram admits that with current memory, just storing a trillion parameters would probably burn more than 20 watts. Moore’s law at a doubling every 18 months is two orders of magnitude in ten years, so 2,000 watts today becomes 20 watts then — and he thinks the right algorithms can already beat 2,000 watts. Why doesn’t everyone try? Sutton jokes that burning energy is how some labs show they are “real men.” Khurram’s serious answer is that he does not even see research groups who *believe* 20 watts is possible, and if you don’t believe it you will not chew on the multi-year technical problems.

The deeper trap is a local minimum. Switch to the new algorithms and things will almost certainly get worse before they get better. Big labs are locked to a product; they cannot walk a path that is worse on day one, especially while the current paradigm still scales. The hard technical pieces, Khurram says, are ones people in their circle have thought about for years and that look doable if you sit with them. If Oak goes right, they implement the architecture, get genuine continual learning, form abstractions, and plan and reason — “sort of like true intelligence.” Sutton does not think humans become irrelevant. He does think LLMs might be the ones at risk. They are an unanticipated scientific breakthrough in the skillful use of language, the old holdout of symbolic methods. Fluid language is “like 20% or a quarter of intelligence.” It should not pretend to be the whole thing.

The endgame is not one mind that learns everything. It is a single *design* instantiated as many minds, each learning different things from different experience — a corollary of the big world hypothesis. Two of the largest systems cannot model each other; they are equally complex. When intelligent robots exist, Sutton says, nobody will spend a fortune retraining each one from the internet. You copy the last one and keep learning. Prior knowledge arrives as a copy; learning is the increment. Hiring matches the bet: a small team, maybe a handful growing toward two or three dozen, of people who already see what they see. “We want to be super aligned,” Sutton says, so they can move fast. A product lab that cannot tolerate a temporary dip will not be in that room.