A Million Tokens per Dollar: Why Noam Shazeer Thinks Inference Compute Is Still Untapped

They are not in the original Transformer office. That building was at 1965 Charleston Road, about half a mile away; this is a new one, close enough that Shazeer jokes the invention is still in the air. Jacob Effron starts with Gemini 2.0 Flash Thinking. Jack Rae says the concerted push into test-time compute began with reasoning tasks — math and code — and it was not obvious that "think first, then answer" would help anywhere else. One of the first pleasant shocks was creative work. Asked to compose an essay, a thinking model would list several approaches, kill the weak ones, revise, and produce a better piece; the chain of thought itself was fun to read.

Shazeer is, as he puts it, all for generality, but he was skeptical of the intense focus on math until he saw why the evals matter. Plenty of changes drop perplexity simply by adding parameters and memorizing more data. Hard benchmarks are how you tell whether the model is actually reasoning about difficult problems. The absurdity is the half-life. AIME, answers-only, was still treated as challenging — maybe too hard — and then it snapped to trivial, small, and almost embarrassing to have spent time on. Naming a benchmark "Humanity's Last Exam," Rae notes, is a dangerous title if every six months the field decides it was not hard enough.

Leakage makes the treadmill worse. Once people talk about an eval, the problems are in the training soup and every model has seen them unless a lab is extremely careful. That is why private evaluations still eat a lot of effort inside DeepMind and Google, and why the shared work of other labs and of Scale AI on new exams matters. Shazeer's own milestone is not another public leaderboard. It is when Gemini 3.0 writes Gemini 4.0 — or, more generally, when Gemini X writes Gemini X plus one.

Those self-reinforcing loops, Shazeer says, are the things most worth watching. The most important is using the AI they are building to make themselves faster at building AI. The second is the data flywheel of people using the models and sending feedback on what they actually care about. The third is a global excitement-and-funding flywheel that has clearly kicked up in the last few years. Asked how much of a full-time engineer he has beside him today, Rae points at Google's structured monorepo and mature tooling. Jeff Dean has quoted a statistic about internal pull requests already arriving with AI-generated bug fixes or code reviews attached; Rae noticed the fixes landing one day and realized the stack could now apply them automatically. The next step is agentic coding on more open-ended tasks, which he thinks will arrive as a discrete jump once libraries can be iterated inside those build rules.

Verifiable domains are easy; the rest of the world is not. Shazeer says less checkable work needs either better verifiers or more human feedback. Rae's observation is that Gemini models already follow much more abstract instructions, so you can hand them a broad rubric — style, interestingness, a pile of qualitative criteria — and train reinforcement learning against that reward. A year or two ago that stack felt impossibly complicated; the actual path, as often happens, turned out simpler than the research roadmap. Users, meanwhile, have voted with latency. A stronger thinking model just landed in the Gemini app, wired to Maps, Search, and long context, and people will pull out a phone and wait a couple of seconds for a better answer, sometimes reading the thoughts as well.

Rae's "mom test" is the punchline. Mothers, he says, are the ultimate check of whether something has broken out of the Twitter sphere into the real world. He demoed a thinking model to his mum; she asked what the meaning of life was, sat and read the answer, then read the thoughts, then contemplated. Image understanding plus thinking is, in his view, already remarkably good and still modestly marketed. Project Mariner, the browser agent Google launched last December, needed the model not just to scan a screen but to understand messy websites and act. Shazeer's cliché-math for why text came first: a picture is worth a thousand words and a million pixels, so language is about a thousand times more information-dense, and the training data is far larger because humans generate text constantly and images rarely.

On agents, Shazeer's first answer is always "make the model smarter." He wants general solutions for control the same way he wants them for intelligence, because users will invent uses developers cannot see — nobody envisioned what the internet or computers were for. Rae refuses a six-versus-eighteen-month number. A lot of the remaining time, he thinks, is not core algorithms but the constant-time engineering cost of training in complex environments instead of prompt-and-response. DeepMind has been chewing on two problems since he joined in 2014: a good agent, and a good environment. A web UI is one environment, a codebase another; whoever defines a great environment can accelerate agent research there, and that choice is as big as any single breakthrough in attention, long context, or RL.

Effron puts Ilya Sutskever's public claim on the table: the test-time-compute paradigm will not get us all the way to AGI; something adjacent is required. Shazeer says he pretty much agrees — then explains a different reason. LLM operations already cost under 10 to the minus 18 dollars. Infer efficiently even on a large model and you get more than a million tokens per dollar, which anyone can check on Gemini's price sheet. A cheap pastime such as buying a paperback and reading it buys on the order of 10,000 tokens per dollar, so inference is a couple of orders of magnitude cheaper than reading, about four cheaper than paying a person to do anything, and six to eight cheaper than a software engineer.

That gap is unused compute. "Would you pay five cents an hour for the bad engineer or ten cents an hour for the good engineer?" If the value is there, and he thinks it is, there is a huge margin of unexploited FLOPs to spend on making the thing smarter. Training a bigger model is the straightforward path, and they are already on it, but training cost tends to rise with the square of model size, while inference done right stays relatively cheap. So everyone is now pouring compute into inference via chain of thought and whatever other algorithms they can invent. Shazeer expects a scaling curve there too. Whether it runs all the way to the AGI future people picture, he says, depends on whether humans or the AIs invent the next breakthrough; he has already given up organizing his garage, waiting for the robots.

Rae does not think test-time compute goes all the way to AGI either. Acting in a complex environment is a definite research investment, and he does not want the paradigm to mean only "think longer until the answer appears." The Ramanujan-flavored target is a model that thinks deeply enough to create useful knowledge, incorporate it, and become dramatically more data-efficient — the student who gets one math textbook, plays with the ideas, and becomes world-class. They are not there. He does think there is a gradient, and one under-discussed fact is that training models to think with RL is already squeezing more learning out of the same RL data. Spitting out a couple of thousand tokens per task is only the start; he wants thinking that looks more like a researcher on an open-ended problem.

The early sign of models acting like researchers, Rae says, will be useful mathematics rather than exam scores. FrontierMath is interesting because its harder tier aims at something close to unpublished findings, a ramp from contest problems toward real scientific contribution. He wants professors to use the tools without irony, then tell labs what is missing. Shazeer's deeper point is that math does not need more external data. Isaac Newton took a pile of astronomical observations, went into quarantine, and generated physics; mathematics is even wilder because you start with roughly no data and invent something useful. That is a counter-proof to the assertion that the most AI can do is relearn what everybody already knows.

Effron puts Yann LeCun's version of the critique to them: this generation of architectures cannot have truly novel thoughts. Rae grants a whole class of discovery almost nobody can argue with — interpolating two disjoint facts, as in materials science, where noticing that properties travel together tells you which new compounds might be photovoltaic. That kind of interpolation would already accelerate science. The line between interpolation and novelty, he suspects, will be a permanent game of whack-a-mole. Shazeer declines the fight: he would like LeCun to prove that *he* has generated a completely novel idea, then adds that he does not care to argue. Build the AI, raise the world's technology level, help people. That is enough.

Rae's analogy is 15th-century geographic exploration. There is a known coastline, a fuzzy beyond, and a small number of expensive voyages that come back with a little extra territory and some funny-looking maps. Mathematics still works that way: a handful of elite professors ask the right questions and prove useful facts. If a model can keep posing interesting questions in a vast space without wandering off into an uninteresting fractal, and if it can solve them, today's strong math model might become professor-level, then Terence-Tao-level, then a million copies of that. Completing the map, with physics and chemistry standing on a finished understanding of useful mathematics, would be one of the great scientific contributions. Both men agree the hard part is posing questions, not solving them. Shazeer is confident on solving; math is infinite, so the model cannot be allowed to diverge.

Asked what culture actually produces breakthroughs, Shazeer locates current AI research where chemistry was in the 15th century: alchemy. Nobody knows why it works. An idea arrives; the only real proof is to try it. You observe, you hypothesize, you are sometimes right and usually need more experiments — unless you simply claim it works because of your magic XYZ and have your assistant swallow a frog. Credit assignment is so tangled that he expects they will eventually give up and wait for a superintelligence to write the thought pieces. On the Transformer itself, he admits that if he had not bumped into the right colleagues in a hallway, they might all still be using LSTMs. Rae, watching from DeepMind in London, felt the pieces were already circling: Neural GPU wanted to kill the RNN, convnets were everywhere, attention was floating around from translation models. Someone was going to assemble it, the way someone would have improved steam engines if the internal combustion engine had not shown up.

They have lived through both compute cultures. Google Brain in Shazeer's earlier stint was mostly bottom-up: researchers pitched projects and talked others into giving them FLOPs. DeepMind was more top-down. Top-down is good for large training runs and forcing collaboration; bottom-up is good because bringing someone onto your project increases total resources rather than shrinking the per-person pile. The deeper issue is abstraction-breaking ideas that do not fit the budget lines. If compute is labeled pretraining versus post-training, a third thing falls through the crack. Google, Shazeer says, is bringing a good measure of bottom-up back. Rae's confession is that he was asked to join AlphaGo at the start, as a research engineer, and could not see why anyone would care about a board game. Top-down bets need research leads with vision; you cannot expect every individual contributor to see the impact.

Their current recipe is dual-track: mandated top-down deliveries, plus a non-trivial slice of undirected bottom-up work whose meta-problem is making people fast — lightweight baselines, high-signal experiments. Rae is regularly humbled when an ignored direction outperforms the plan. Going all-in on large language models was itself an against-the-grain call, obvious to a small set of collaborators and not to most of the field. At the time, machine translation and computer vision — ImageNet, the cat — felt cooler. Shazeer never worked on the cat. Language already seemed like the best problem on Earth, and the early LLM people were not even linguists: language was simply the most knowledge-compressed substrate on the way to a generative model of everything. Native image generation, launched the day before this taping, is another piece of that bet. Task-specific "AI doctors," Shazeer thinks, are mostly a price story: you would pay a dollar a token to talk to a physician, and LLMs are already far cheaper, so a specialist model is justified only when you would not pay that much. Negative transfer, Rae adds, has mostly failed to appear; if transfer is positive, put it in one big model until serving gets too expensive.

The thing Rae has changed his mind about is not a vague feeling that progress is fast. After the Transformer paper, even inside Alphabet, it took on the order of six to nine months before the architecture was dotted across DeepMind projects; he implemented it himself over a holiday break three months after publication, and pickup was slow. Test-time compute, by contrast, produced many labs training and releasing serious models within months of a blog post. That was a wakeup call: there is more compute and there are more smart people in AI than his 2016 rose-tinted memory allowed. People now are as smart and creative as they were then, with far more hardware and far more of them. Shazeer's sting is sharper. A kid in a garage already has more compute than was necessary to invent the Transformer. Breakthroughs, he says, take less compute than the industry's anxiety implies.

His own update of the past year is simpler: he had barely worked with reinforcement learning, and it is actually pretty good. On open source, Rae has also flipped. He used to think the catch-up was a false sense of assurance that closed labs would pull away again. The pattern has persisted. Gemma 3, released the day before, struck him as completely incredible; DeepSeek V3 was a very strong model when it landed. The open-source community is passionate, creative, smart, and provisioned with compute, so he no longer sees a reason they would stop innovating. Shazeer's forecast is that the quality gap between closed and open may stay large while the time gap gets very small. The lead, in calendar time, keeps shrinking.

Shazeer does not worry much about cleaning the garage or, he jokes, about global warming, on the theory that AI will take care of the carbon soon enough. What has changed in Rae's house is education. Under supervision, his son talks to Gemini, photographs plants and lizards in the garden, and now carries around a personalized, accurate encyclopedia. The four-year-old uses Latin names. At school he reported catching a lizard; the teacher said it looked like a big one, and the child corrected her — western fence lizard — then added that he had also seen a blue-tailed skink and began to explain. Children are sponges. Combined productively with AI, Rae thinks, that is a form of education humanity has not had, and the next generation will simply seem smarter.

Asked what humans are for, Shazeer's answer is almost aggressively ordinary: if you want to do something that matters materially, do it now, because human labor may not be economically necessary later, which only raises the stakes of the present. Otherwise, be a good person and do what you find spiritually meaningful. In the distant past, meaning was easy — work or the family starves. In America today nobody's family starves tomorrow if you slack, so meaning already thinned out; AI will force the question further. On AGI risk, Rae is moderately worried. History does not offer many examples of creating something far smarter than its maker that still behaves predictably for that maker, and sharp employment shocks need to be smoothed. Internally, a separate group reviews launches for unintended consequences, which he is glad exists. Shazeer's version is a joke with a load-bearing premise: we have kids, they get smarter than us, they become teenagers, and then we solve the alignment problem — provided the internet contains enough tokens of people respecting their parents, and we stop pushing the robots over.

The Character.AI chapter is, in his telling, a completed mission. He left Google so that anybody could talk to an LLM and discover their own uses, before ChatGPT or Gemini existed. Everyone is now talking to LLMs. The product landed on entertainment partly because nobody had solved hallucination, and entertainment is a domain where hallucination is a feature. Looking five or ten years out, he thinks people will always want human relationships because they are spiritually more meaningful, but human-form AIs will show up as advisors — a newly elected president with a personal AI cabinet, a CEO with more productivity, interfaces that feel human because users like them that way. Better models do most of the work; the remaining product question is what the application will *let* users specify.

In the quickfire, Rae calls the ARC-AGI eval overhyped. Progress has been slow partly because researchers are not inspired by those specific puzzles; DeepMind did a lot of that in 2015 and 2016, fixed grid-operation bottlenecks, and did not obviously get closer to useful AGI. His own turn from synthetic tasks toward modeling natural language felt like the more AGI-shaped path. Shazeer goes the other way: AGI and LLMs are still massively underhyped. People are still thinking in terms of some silly trillion-dollar product; he has said elsewhere that a trillion is no longer cool and you have to say quadrillion, Doctor Evil finger and all. On applications, Rae sees real value in agentic product experience even when critics call the app a wrapper, and he thinks code is now crowded relative to other automations. Shazeer thinks code is underhyped. Humans are not even that good at it, and an automated software engineer is one of the loops that builds the next AI.

Infrastructure for test-time compute is a rosier story than pretraining, Rae says. If building AI becomes mostly an inference problem, actors can go get experience in many data centers and send it back without 100-gig interconnects, which is intrinsically cheaper and can be optimized further. Google's advantage is co-design with the TPU team: the model group feeds them a profile of how compute is actually spent, and chip and data-center design can move on a couple-of-years timescale. What gets worse, Shazeer adds, is that naive Transformer inference loses the training-time parallelism and becomes memory-bound looking up attention keys and values for every generated token. There is a lot of work on both the architecture side and the hardware side before those chips' raw math can be fully applied at decode.

Last word: a new, considerably stronger Flash thinking model is in the Gemini app, and they want people to use it and send feedback. User reports are already being folded into each model series. The grand questions sit on top of that ordinary request. If you want to do something meaningful, Shazeer's earlier advice still stands — go do it now — and if you want to see what test-time compute feels like in daily use, the place to start is the thing they just shipped.