Oriol Vinyals: The Days of Burning 90% of Compute on Pretraining Are Numbered

Oriol Vinyals is VP of research at Google DeepMind and technical co-lead of Gemini. Sarah Guo introduces him as the researcher who led AlphaStar, the StarCraft II agent that played at a professionally competitive level. Last year, he tells the *No Priors* hosts, Google had two overlapping large-language-model efforts, one in Google Brain and one in what he now calls legacy DeepMind. Early in the year he and Jeff Dean pulled the teams together; the first Gemini model shipped later in 2023. The second event was folding every AI-research organization under one roof as Google DeepMind, with Gemini as the flagship project inside it.

The merged org still has two major centers, California and London. That split, Vinyals says, means the project effectively runs 24/7: when California clocks out, London takes over, which is genuinely useful when you are training models this large. Gemini’s job is to be the core model that powers the rest of Google’s LLM surfaces. He lists four: Cloud and enterprise, which he treats as the base; developers, who will do things the research team did not anticipate; the Gemini app, formerly Bard; and Search, which has the user volume.

The lag from research to product is short enough to feel personal. Modeling decisions, he says, show up for users a couple of months later, sometimes as a beta. Long context — millions of tokens — is the example he gives of the research org pushing what comes next rather than only serving existing product needs. Google has many product areas, so the Gemini project started by concentrating on those critical surfaces instead of trying to be everywhere at once.

Guo, who once worked at Google, remembers search queries being sliced by intent: navigational, commercial, medical. She wants to know which of those migrate into chat and which stay as classic search. Vinyals declines to pick a winner. Nobody knows yet, he says; the company is experimenting. He uses both products himself. Chat is the LLM-first experience, with search sitting inside it as a tool that supplies citations and a bit more grounding against hallucination. Search is adding AI summaries and, from this year’s Google I/O, a broader set of language-model features, some still in testing.

Even if Search remains the place you type a query, he expects that experience to be “tremendously” enhanced by these models. The Gemini project, as the core-model team, does not have to decide which interface wins. Product areas drive strategy, influenced by what the models can actually do. Operating at Google’s scale, Vinyals notes, means you can fund both paths while the experienced product teams iterate with their users.

Elad Gil adds a historical point that Vinyals does not contradict: Google was the original AI-first company. Search and ads were machine-learning systems from the earliest days, and the core move was inferring user intent. In that frame, Search can stay the entry point, and the system can pop a chat bubble or a list of links depending on what the user seems to want. Vinyals’s posture is that research makes the model better; product strategy is not the research team’s job to freeze.

Asked what has most surprised him about how people use Gemini, Vinyals does not start with enterprise workflows. He starts with context length, which he first treated as “a number that you could report.” Recurrent nets and LSTMs had theoretically infinite memory and, in practice, forgot after a few hundred words. Getting to millions of tokens was the first shock. The second was watching internal testers drop in a full one-hour video, wait ten or thirty seconds for the context to process, and then ask anything. Compared with the video-question-answering datasets he grew up with in computer vision, those old benchmarks suddenly looked like toys.

Developers went further. Some pointed a camera at a screen and debugged from the characters on the display, without feeding the model a code file. Future interfaces, he says, could be that effortless: point, ask, get an answer. Google I/O’s Project Astra sits on the same line — a phone you talk to as an agent, with memory as the scarce resource. He can imagine the whole web, or all of a user’s personal data kept on-device, living in working memory rather than in the weights. It still has not gone fully mainstream, which he admits, even as the demos feel magical.

The timeline he will actually stand behind is specific. In one to two years, the context length that counts as commodity should be enhanced by about 10x everywhere, and extremely long context should jump another order of magnitude at the frontier, assuming the use cases show up. Hierarchical memories and RAG will be combined to get there. Hardware is already being updated around the research. The remaining limits, he thinks, get solved if the motivation is there.

Guo asks whether retrieval architectures still matter in an era of “infinite” context. Vinyals says yes, for efficiency and for quality: an intelligent system retrieves in a coarse-to-fine way, the way people do, and Google has decades of retrieval experience to mix with neural methods. The failure mode of retrieval is compression — treating a whole book as a single vector. Upload that book into Gemini and the model can reason about every word. Finding the middle ground per use case is the work. A hybrid going forward, he says, looks like a feature, not a bug.

Gil asks him to place the moment: what are the real limits of today’s LLMs, and what is worth working on. Vinyals first notices the sociology. Years ago, friends in the field told him to get ready for a flood of people once this went mainstream; open source and a kind of viral random search made that true. Credit, as he assigns it, splits among scale of data and compute, a handful of algorithmic advances over ten or twenty years, and accessibility — software and open-source work that turned progress into the log-linear trends everyone now plots.

He splits the last fifteen years in two. From about 2010 to 2020, the *algorithms* were general: stochastic gradient descent, neural nets, reinforcement learning. Swap the dataset and you mastered Go, beat ImageNet, set records in speech, then improved protein folding over classical methods. The models themselves were specialists. The break, especially after the GPT-2 paper, was noticing that modeling language is a powerful enough abstraction to solve every task, not one task. After that, the model became general too. Multimodality then stretched the same techniques to vision, sound, and video. On an AGI checklist, he says, many boxes are starting to get ticked.

Reasoning is on that list, but it is not crisp. A model can solve an Olympiad math problem and then fail a basic puzzle, or contradict itself. The underlying object is still a probability distribution over sequences: it assigns nonzero mass to things that are not correct. Ninety-five percent accuracy on a hard question still produces visible errors once you deploy to the whole world. Scaling the model until mistakes vanish is one path. The faster one, in his view, is to wrap the language model in a small, slow program — more redundant, more logical, more iteration — and spend extra inference compute until the error rate has more nines. Humans get tired; the models can just run longer.

Gil frames the scarce resource as compute, and asks how the mix between training and inference — including test-time search — should look two or three years out. Vinyals reaches for Richard Sutton’s bitter lesson: scale learning and scale search, and that is the job. Learning’s scalability, he says, has been proven. Search is stuck because language has no perfect reward. “Is the sky blue?” is true in daylight and false at night. Games give you a one or a zero; statements do not.

AlphaGo’s recipe is the historical baseline: pretrain on human games, improve with RL self-play, then search at inference. Most of the compute sat in the middle bucket, the self-improvement loop. Today that is inverted. Most compute goes to pretraining, and you have to *stop* RL early because the model overfits an imperfect reward — it discovers, for example, that flooding the answer with emoticons makes the preference model happy. Inference was also capped in AlphaGo, because a game, from a human perspective, lasts on the order of four hours. Vinyals’s forecast is that pretraining should fall from over 90% toward something like 50%, with the rest mostly on reinforcement learning, which he still sees as early-stage research. A few seconds of System-2 thinking is enough for ordinary queries; protein folding can wait a month.

The next problem is scaling the reward once models beat the experts who label them. Gil cites Med-PaLM 2, which already outperformed physician expert panels on some evaluations. Short term, Vinyals says, you keep doing what deep learning always did: human annotation, the ImageNet-scale effort Fei-Fei Li’s lab ran on a million examples. Long term he is betting on a bootstrap. If checking an answer is easier than generating it, the model can score its own samples and the RL loop turns. That, as he sees it, is close to the central hypothesis of reinforcement learning. Generative reward models already have leaderboards. In the limit, the labels you need might be only the ones a user offers when they want to teach the system something new.

Gil offers the Nyquist–Shannon sampling theorem as a metaphor: you have to be smart enough to recognize something smarter than yourself. Vinyals, who studied the theorem as an undergraduate, enjoys the analogy and then breaks it. Super-resolution generators take a 32×32 image and invent detail, which strictly violates the sampling bound — they are hallucinating — and they win because the world has structure the model learned. He expects a similar emergence of self-correction. Once that capability appears, bootstrapping stops being exotic, provided you have algorithms that can exploit it. The commercial loop underneath all of this already works: models found usefulness, revenue flows back into training, and as a researcher he welcomes the resources.

Guo asks whether we are fully in the era of general models, or whether domains like protein folding still need their own data. Vinyals’s picture of a generalist is blunt: performance level aside, current models are about 20% good at everything. The research goal is to lift that 20% uniformly, not to over-specialize the main model. The world still has problems worth a specialist — protein folding, nuclear fusion, weather — and several of those projects remain in DeepMind’s portfolio. If the problem matters, specialize even if a general model will eventually overtake you, because you get there sooner. The loop also runs backwards: a recent math-specialist model produced data and reward signal that helped the main model. Hybrid is, again, the plan.

A common criticism is that math and games are a dead end for general reasoning because the rewards are too clean. Vinyals grants that a perfect environmental reward is rare — even “survive” is hard to compute — so work in artificial domains might fail to transfer. Math is less clean than the critique assumes. Four plus four equals eight is easy to check; proving a theorem is not, formalization can have bugs, and proofs come wrapped in language. Train a reward model to judge what is *roughly* correct, and you are already building something general. Attack the problem that way, he says, and it is not a dead end; a self-improvement loop should show up more than it has so far.

His contrarian take is that he does not much like the term AGI, which is ironic because Shane Legg, a DeepMind co-founder, helped make it famous. In 2009 Legg predicted 2028; Vinyals recently discussed timelines with him and still finds the date reasonable depending on the test you pick. Metaculus and similar aggregates put the world a bit later, around 2030. The view he actually holds is that it may not matter. Capability will be a distribution of things the models can and cannot do — cracking an impossible math puzzle, then contradicting themselves on something trivial. There will be no single moment of parity, and no agreement that “it” happened. Fix the obviously AGI-shaped errors, then use the models for products and for science. Google DeepMind’s mission statement, he notes, has science written into it.

He already lives that way. Based in London, he wakes up to a night’s worth of California chat and uses long context to digest it — scaling himself the way a career scales from writing code to directing other people. For anyone not steeped in the technology, his advice is to treat the models as a tool and find a way to collaborate, because that change is coming whether or not anyone declares AGI. For children heading to college in ten years: first find a real passion, the way he went into deep learning when it was not fashionable; then project how AI changes that field and exploit it. Language is the interface, so the barrier is lower than people think. His sister, a teacher, already uploads a batch of homework and asks for summaries. If you are technical, look for corners AI has not cracked — weather modeling is largely done with deep learning, climate is not, because there is only one planet to sample — or spend five to ten years on open LLM research. Parenting, he adds, is zero-shot. He will report back on the next podcast.