The People Writing the Transformer’s Eulogy Want a Lab That Can Take Vacation Without Them

Start with a number. QR decomposition is a linear-algebra primitive more than sixty years old. On NVIDIA’s latest B200, the industry library cuSOLVER sets a baseline. A human plus an automated search loop can get about 7×. A high-taste expert — Anil says there are maybe three people in the world — spending four weeks and roughly $100,000 of coding-agent calls reached 60×. Give the same problem to Anthropic, OpenAI, or Gemini models and they simply do not solve it.

Rohan Anil puts that contest on the table. He co-led Gemini pre-training, was a Google Brain fixture, first-authored the second-order optimizer Shampoo, and later spent time at Anthropic as a well-known fixer. Across from him is Jerry Tworek, former OpenAI VP who ran Strawberry — the project that became o1 — and the reasoning team, and who spent years as an RL maximalist trying to scale reinforcement learning into AGI.

They now co-found Core Automation, one of the hotter new labs in San Francisco, and sit with Sequoia’s Sonya Huang and Pat Grady on Training Data. The hour has one argument. The transformer has carried the field as far as it can. The next systems need a different architecture. The way to search for it is to automate the laboratory, and the inner loop of that automation is kernel generation — the one place frontier models still lose to a human with taste.

Tworek recently tweeted that the first step to replacing transformers is appreciating deeply how far they were able to carry us. Huang asks if that is a eulogy. He laughs that half his interviews now are explanations of his tweets. Appreciation, he says, means knowing what the architecture does well so you stop trying to re-solve those problems and put the fire on the weaknesses. Most architecture papers he sees try to make transformers cheaper or more efficient. He rarely sees work on making them more powerful or more expressive. Seeing the strong parts and the weak parts is almost the same act. It is understanding the shape.

His summary of the last six years is two mastered algorithms: large-scale pre-training and large-scale RL. Models got bigger by repeating two operations, mixture-of-experts and attention, and pouring in parameters. He is genuinely grateful for what that produced — Codex, Claude Code, coding and workflow products. He still comes to work, still asks the tools to do certain things, and keeps asking why he is needed in the loop at all. Core Automation’s name is the question: why are those remaining tasks not automated yet?

The bottleneck to better models and smarter systems, in his view, is now the architecture itself. It is time to look back at a train the field has been riding for six years. That is a contrarian premise while every lab is still scaling the same stack. It is also, he argues, the only way to see which problems have been solved and which have not.

The architecture diagnosis is personal. From his first day at OpenAI, Tworek believed scaling RL was a necessary step on the path to AGI. He watched GPT-3 to GPT-4 climb on pre-training while the lab still did very little RL, and he carried an internal conviction: the moment RL scaled, everything would be solved. Asked in 2024 when AGI would arrive, he would have said 2025. Then they scaled RL. He was in the center of it, training model after model. Benchmark scores went up. Did they solve real-world tasks? Unfortunately, he says, no.

The deadly overlap is that evals and training tasks are two sides of the same coin. The real-world distribution is messier, murkier, and much more different; training data never really replicated it. Ask anyone training models what their main issue is and the answer is consistent: they do not have hard enough tasks. Even after maximizing those tasks, they still do not cover the real distribution. His conclusion is that models must learn at test time, with users, on real tasks and real data.

Today’s two tools each have a hard failure. In-context learning does not catastrophically forget and is fairly data-efficient, but the capacity is tiny. Using Codex, Tworek has to compact context after about twenty minutes. “All we can learn is for twenty minutes.” Continuous fine-tuning has catastrophic forgetting and terrible data efficiency, and neither problem is easy; if they were, someone would have solved them. He wants an algorithm that can be meta-learned, expressed at the architectural layer as what learning looks like, and that can run on much longer horizons.

Huang asks a history question: OpenAI wanted to scale RL for a long time; why did the transformer become the scalable prior? Tworek’s unromantic answer is that two things happened at once, and economics decided it. Technically you can scale LSTMs; almost nobody dared. Scaling-laws papers showed transformers scaled better, but the decisive fact is the books. The cost of training a transformer is lower than the revenue it generates, which he calls a kind of magic not guaranteed in machine learning. In a world without transformers, the field would be scaling costlier, less impressive LSTMs, and nobody could have raised the dollars.

There is a research-culture layer too. Before OpenAI’s bet, the academic mainstream treated scaling as unscientific and chased algorithms that did more with the same compute. OpenAI’s contrarian move was to care about scalable algorithms and pour compute. It was criticized for years. Those are the models we have. Plenty of architectures can scale, he adds, but a lot of architecture research stayed too small for too long. RL in particular needs a baseline of capability before anything interesting appears. Other architectures may need a similar compute floor, which is exactly why Huang asks the touchy follow-up: why not do this inside a big lab?

Timing, he says. The largest labs are in the most competitive market fight he has seen. If transformers are profitable, attention goes to winning next quarter, not to something that might redefine the field in a year or two. He has talked to essentially all of them. Smaller labs imitate the winners. Everyone is training the same coding agent; last week’s releases make the point. Someone has to take a different path. That is the niche Core Automation wants.

Anil was at Brain when Ashish Vaswani, Noam Shazeer, and others published the transformer. Internally it was not a big deal; he presented online distillation at the same research conference. The original paper was aimed at translation and beat the systems of the day. The real demonstration waited for GPT-2 and GPT-3. On why labs still will not unify pre-training and RL end-to-end, he has polled excellent researchers who nod that it is worth exploring and then keep it out of the top bucket because the next release cycle’s model has to be trained. Tokens are not sticky. Users switch. Long-term research of even six months often no longer fits.

Anil’s definition is spare: architecture is how you spend computation, and the transformer is one efficient way. Pre-training builds a fixed-context transformer; RL then says that is not enough and adds compute one generated token at a time. That is inefficient at inference, so the industry patches it with speculative decoding — a bandage on autoregressive, one-token-at-a-time design. The older disease is poor computational depth. Most trained transformers are at most about a hundred layers. Deep learning was named for deeper representations, and nobody has really shown models learning extremely deep ones. Chain-of-thought plus RL is one way to add depth: each extra token is another pathway, which is inference-time scaling, at the cost of emitting still more tokens one by one. Just posing “how do we increase computational depth?” opens maybe twenty directions.

He adds two lab realities. Fundamental research used to take five or six years to land in industry, and only if the organization decided it mattered; at Google he was a pre-training maximalist the way Tworek was an RL maximalist, and that inner belief is required. A theoretically optimal architecture that does not run fast on hardware is useless. You need kernels, productionization, an end-to-end loop, and there are only a handful of places with integrated teams. Left to the world, he fears replacing the transformer will take longer than it should. People already complain about token cost. From a Google mindset of serving billions, frontier tech still reaches only a subset of humans. Someone has to accelerate efficiency.

Tworek’s ceiling is a thought experiment. Pre-training distills internet knowledge; RL bakes in the workflows you want. A transformer’s cap is all of humanity’s knowledge plus any task you have training data for. Then: what if OpenAI and Anthropic stopped training new models and you kept today’s best for a year? The model gets less and less useful. New events appear, with new relationships; new tasks, codebases, tools. Transformer value has to be present in training or it suffers. There is some ability to adapt, not much, and not flexible. In his mind it is a tool that still needs a human who knows the limits, writes the prompt, and ships failures back into the lab’s training set. The improvement loop has to go through the laboratory.

Huang asks about David Silver and Rich Sutton’s “Age of Experience.” Tworek says experience has always been there. RL is not new: TD-Gammon, Go, StarCraft, Dota, now programming, each time a model writing its own experience and learning from it. What still perplexes people is that RL is only one mathematical form of learning from experience, and the current version leans on parallel rollouts for variance reduction — which is not how humans learn. Playing football looks a lot like RL: kick, adjust, self-reinforce. Learning mathematics is different: hard concepts, thought until things click and connect. Both are learning from experience. The brain probably runs more than one algorithm. The field is spending more compute than ever on learning from experience, and RL, he says, is not the end of it.

Asked how to get orders of magnitude more efficient learning, Anil starts with measurement. Pre-training as currently defined is compression: watch perplexity, scale parameters and compute, take a log-scale bump for an epsilon of metric. Fine for building a prior, he says, and the wrong way to look at the problem. Look at the end-to-end outcome. He trains a model, hands it to Tworek, and Tworek’s RL will destroy every perplexity number Anil just made. Then what was the metric for? Pre-training was the best bootstrap the field had. Combining pre-training and RL into one training procedure is where he thinks an order of magnitude lives.

That claim pulls in his old craft. In 2016 at Google he worked on large-scale optimization for logistic regression and on Sibyl, the linear solver used before neural nets took over. Vineet Gupta showed up at his desk: we have a whiteboard idea, you are good at writing optimizers, implement it and show it scales. That became Shampoo. His manager Yonghui Wu backed it through 2024. Most of the community was not excited; why not just use Adam and spend the time elsewhere? They used it on Gemini 1.5 Flash, the community warmed up, SOAP and related papers followed, maybe a 2× improvement. His own verdict is harsher. Shampoo is still weak. It does not use all the information available during training.

The line he keeps repeating: the optimization algorithm defines which architectures you will discover. Residual connections made deep nets trainable; a few people have now dropped them and still learned deeper representations, but only with a better optimizer. Strong optimizer plus hard-to-optimize model, or weak optimizer plus easy model: the trade-off is everywhere. RL is an easy target for the same reason. You decode a long chain of thought to write one bit of information into the network, and feedback is scarce. That is inefficient compute. Will we approach biological learning efficiency? Not on current hardware, he says. He cites Geoff Hinton on mortal computation: brains grow their own circuits and learning algorithms, then die and take them. Neural nets keep the hardware and learn slowly, needing huge parallelism to push a little information through. Until analog circuits and error correction, the gap remains. “We are safe.”

Core Automation’s mission, in Tworek’s short form, is models that continuously learn from deployment. Transformers cannot do continual learning, and there is no way to put continual learning onto them, so the architecture has to change. The search method is the most automated lab they can build: fast, large-scale experiments, strong priors, efficient search of architecture space. Automation here does not mean taking humans out of the loop. It means maximum agency per person — walking versus a bike versus a car, a hand-farmed plot versus a machine. If a researcher can land even one full experiment a day, that already beats prior iteration speed. There is no law of physics against ten a day, then a hundred. The team is heavily agent-filled, a small-org experiment in how far that goes.

His current definition of AGI is specific: a model that can improve itself without a human in the loop in any way. That is a subset of “outperform all humans on economically valuable work,” because improving AI models is itself valuable human work. Removing humans from loops has been notoriously hard. He can barely find a task where LLMs got humans out. The human–LLM hybrid is extremely successful; LLMs without humans are not, at all. What he saw in 2024 and early 2025 is that the current path does not get there. Test-time learning needs serious research, here or somewhere else.

The six-month roadmap Anil will share is automating kernel generation. They have Blackwell GPUs. The distance to compress is from a cool architectural idea that kills a bottleneck to running at peak TFLOPs. That is the QR contest as inner loop: today’s models are nowhere near 60×. Why kernels? Two of his three major projects that reached industry stuck on the same nail. Second-order methods live or die on kernels; at Google you cannot spend 10× compute for a 2× win, the budget was maybe 20%, and only two people — he names Rasmus and Peter Hawkins — could write the deep XLA, and it took two years. N-grammer, with extra n-gram memory replacing some transformer parameters in 2020, needed hardware-accelerated sparse gathers and scatters; TPU meetings through COVID never delivered the hardware. As he was leaving Anthropic, DeepSeek published an improved n-gram-embedding idea that could replace MoEs, with scaling laws. A five-year thought, someone else shipped it. He was happy for them. Without kernels, a huge region of architecture space is simply closed.

How will they know they found it? Every successful research project, Tworek says, has a plot other plots do not show — one line bending a different way. That plot usually arrives late, because by then you already know what you wanted. He is only half joking that the best plots of his life appeared in a dream first. Deep learning systems are intricate; you usually have to get five things right in a row before the thing works. The criterion they actually want is ordinary. Evaluate on everyday work. The system should get better at doing Core Automation scientists’ jobs each day. Joke that is also real: the team goes on vacation for a week and asks whether the lab produced something better. If it did, extend the vacation, twice, four times, until it is permanent.