The Model Eats the Scaffolding: How Google Plans to Ship Agents Everywhere

After more than 340 episodes recorded from his Detroit home office, Nathan Labenz sat down in the library at Gradient Canopy, Google’s headquarters, for the first in-person taping of *The Cognitive Revolution*. Across from him were Logan Kilpatrick, a member of technical staff at Google DeepMind, and Tulsee Doshi, senior director and head of product for Gemini models. They recorded on Friday, May 15, a couple of days before Google I/O, while much of the company was still polishing demos — including Nathan’s brother Craig, who was giving a Wednesday keynote. Compared with the rest of the AI industry, the vibe was relatively relaxed confidence.

Labenz opened on the three-year anniversary of the leaked “no moats” memo of May 5, 2023. In the years since, he notes, Google has added $3.5 trillion in market cap, more than every company on earth except Nvidia and Apple. From 2024 to 2025 it grew annual revenue by $50 billion, roughly what Anthropic now pulls in, and still holds about 25% of global compute. The question hanging over I/O was what would deepen those moats next.

Doshi led with the modeling slate. Gemini 3.5 Flash is the first of the 3.5 series, and she frames it as the sweet spot: smart, fast, and cost-effective at once. She says it is about three times faster than other large models, significantly cheaper, and still able to drive agentic coding workflows. Google has been using it internally at scale, which she calls genuinely fun to watch play out.

The other model headline is Gemini Omni, a video generation and editing experience and the first step toward all modalities in and all modalities out. Users will be able to make videos and drop their own avatars into them. Around a week or two after I/O comes Gemini Spark, also built on 3.5 Flash, which folds more agentic experiences into the Gemini app. Antigravity is being upgraded in parallel so Flash and the developer surface land together.

Kilpatrick’s headline beyond the models is “agents, agents, agents” — a riff on Sundar Pichai saying “AI, AI, AI” a year or two earlier. He thinks this is the first year, not just at Google but ecosystem-wide, of model–harness–product symbiosis: the model is trained with the harness, and the harness powers agentic product experiences. Spark in the Gemini app, vibe coding in AI Studio, and the Agents API for developers all sit on that layer, which he expects to roll out more widely across Google.

Doshi says 3.5 Flash will show up in the Gemini app, AI Mode in Search, Antigravity, agentic experiences in AI Studio, and Spark — one model across surfaces with very different users and goals. Kilpatrick admits that has gotten harder: they are no longer launching a model on a couple of services, but serving Search, developers, Cloud customers, and the Gemini app at once. Gemini Live is also getting faster and smarter, including better background-noise detection, so it “really does actually feel like a partner.” Historically Google lacked a through-line across products; Gemini became that stitch, and the Antigravity agent harness is becoming the next one as products turn agentic by default.

Labenz remembered an earlier promise of three Gemini sizes and noted the missing Ultra — “we took one off the top and added one at the bottom” with Flashlight. Kilpatrick says the three sizes now are Pro, Flash, and Flashlight, plus Deep Think 2 as a fourth scaling dimension at runtime. Labenz pressed the business case: Google does about $400 billion a year in revenue, with profit possibly heading toward $100 billion, and customers seem relatively price-insensitive for the very best model. Why not ship Ultra?

Doshi does not deny that a slice of users will pay for quality, and she says the Pro model has been pushing that frontier. The constraint is serving Gemini and Search at massive scale, where latency and cost dominate. In live experiments on Search and the app, even a “hugely better” model loses if latency gets worse, because “what you're asking users to do is wait.” Flashlight was added to the lineup for that reason: large-scale demand for cheaper, faster inference, inside Google’s own products and for external developers.

Kilpatrick’s addendum is that naming is marketing. Pro models have kept scaling up in capability, size, and power, including test-time compute with Deep Think. The team has repeatedly asked whether to put the Ultra brand on a release and so far has chosen not to, partly for series-to-series consistency. He also ties the Flash push to DeepMind’s mission of AI that benefits humanity, which at Google means product surfaces serving two-billion-plus users. You cannot hit that mission without cheap, fast models, and he argues the two tracks are technically bound: it is hard to make a great Flash without a great Pro, and vice versa.

Labenz asked whether a secret Ultra-scale run exists only to distill into Pro and Flash. Doshi confirms distillation in both directions: Pro influences Flash, which influences Flashlight, and they also scale a Flash recipe up to a Pro recipe. Internally they are already combining strong Gemini 3.5 checkpoints with inference techniques. A Tuesday stage demo from Varun — sub-agents that fan out, complete tasks, and come back, available as an early preview in Antigravity via `/teamwork` — is one example of an extremely capable stack they have been using inside. Pre-training, post-training, inference, and the harness are all being pushed at once.

After the break, Labenz turned to harnesses. He had just heard Task founder Andrew Lee describe everyone as building the same thing: a general-purpose drop-in knowledge worker, intelligence at the core plus a “mecha suit” around it. If Google co-trains Gemini with its in-house suit, does that create silos — switching costs and pricing power for whoever owns the full stack? Kilpatrick’s best case is both: it should work especially well for Gemini because Google controls the full-stack story Sundar likes to tell, and it should still generalize, because developers want choice. There is even a philosophical test hiding in that hope: how good is your model if it cannot generalize to other harnesses?

Doshi calls the full-stack loop a flywheel. A seamless Gemini experience makes it easier to train a better model. The non-negotiable is what they call harness diversity: enterprise customers and developers building their own tooling and orchestration still need Gemini to work. Co-training helps with debugging, data collection, and evals, so the team can move faster. The balance is not locking the model to a single tool loop.

Kilpatrick turns that into a pitch: if a “harness bench” does not exist, someone should build one, and he would collaborate. His analogy is games: if models are so good, why can’t they play games well? If the industry is approaching AGI, a model co-trained with one harness should still “generalize reasonably well” in others. If it cannot, that is another sign of jagged intelligence. Labenz noted the idea could also be productized as an RL environment and sold back to the labs — “quite the cottage industry these days.”

Labenz was in town for an event called Recursive, and he asked how bought-in Google DeepMind is. When you talk to Anthropic people, recursive self-improvement is almost a religion, and they see it as inevitable. OpenAI has public timelines for an ML intern later this year and a full-fledged AI R&D employee in early 2028. Does Google have milestones for handing ML research to AIs?

Doshi says they are already using Gemini “pretty deeply” to improve Gemini, from ordinary productivity help to submitting changelists, running evals, and suggesting research improvements. She describes a research-partner relationship: creative ideas, faster tests. A few days earlier, Anca Dragan, DeepMind’s lead for safety and alignment, pinged her from a hot tub to say she had kicked off a batch of ablations from her phone and had a report within the hour. That is already possible; then imagine six months, a year, two years from now.

Kilpatrick cools the temperature. Coding models will help build products and train models; the nuance is whether the human stays in the driver’s seat. In the near term he thinks that is non-negotiable, because the cost and opportunity cost of a wrong large-scale pre-training run are “super, super high.” He does not see it as realistic in the short to medium term to let an “ML intern” kick off those jobs. Deep collaboration between AI and human researchers is obvious. Handing over the wheel is not. Doshi adds that the collaboration changes the human role toward interpreting results and setting strategic direction.

Asked whether she still types code, Doshi says that for the code she used to submit herself she now relies mostly on Antigravity, doing “bits and pieces.” She has also started generating slide decks and other content from her thoughts. Antigravity now has Gemini Mic: you ramble in loose form and the model turns that into reasoned action. For someone who thinks by talking, she calls it a qualitative shift — sit down, say what you are trying to think through, and get it back well thought out.

Kilpatrick wants a breakdown of human-typed code versus AI-generated code versus audio-in, code-out. He suspects audio input to generated code is one of the fastest-growing input modalities, and it is already “the predominant way” he builds software — at least when he is not around other people. Around colleagues he still types, so they do not hear his “dumb ideas.” Doshi says that if you walk upstairs at Google you will see people muttering at their hands, talking code into existence.

Labenz’s personal KPI for whether AI is improving his life is whether he gets outside and exercises more. He wants to dump thoughts on the move, which is why he sees voice as the frontier modality for himself. That thread picks back up at the end of the conversation, when Doshi says audio — Gemini Mic and the upgraded Gemini Live — is the paradigm she most wants to keep pushing, because it is becoming a bigger part of how people engage with models and how models engage back.

Labenz noted that commentary on Google’s AI integrations has been a mix of bangers and misses, and asked what outsiders should learn as they try the same. Kilpatrick’s practical story is years of painful infrastructure standardization across the AI stack, which is a big reason Gemini 3 could land in so many products. His advice: every 12 to 18 months you have to rewrite everything from scratch, so the worst case is N teams all doing that rewrite. Serving used to be tokens in, tokens out; now there is agentic infrastructure and tool loops inside the harness. A product team that just wants to ship something agentic should not have to reinvent the tool-calling loop.

Doshi’s lesson is that there is no substitute for fast experiment-and-iterate loops: drop in a new checkpoint, prompt it into viability, prototype, get it to internal users, then find the rough edges only once it is alive in a product. Shared layers make that speed possible. The things that actually raise quality are prompt tuning, live experiments with real feedback, and evals that give signal. NotebookLM is her example of a “banger product” from a “banger team” that understands the model and prototypes quickly.

Kilpatrick’s favorite illustration is the original Audio Overview. What shocked people was the coherence of the dialogue, and that coherence was “just base Gemini with a bunch of banger prompts.” The NotebookLM team knew how to “prompt whisper” the model and get the best out of it. Labenz summarized the architectural shift: a lot of surrounding code is moving from the product side onto the model side. Kilpatrick’s phrase for it: “The model eats the scaffolding.” At every crank of the model flywheel, the model eats a bunch of scaffolding.

The harness is extensible rather than a single fork-or-nothing. Spark and AI Studio run on the same infrastructure with slightly different nuances, because they are not the same product. When something does not work, Doshi’s loop is: try to prompt your way out; then locate where the model is falling down; then feed those failures back as evals and data for the next revision. That cycle, she says, is a large part of how Gemini has gotten better across Google’s product teams.

Labenz floated Omni as “a nano banana moment for video,” meaning a deep integration of language, reasoning, and pixel-space understanding rather than a lossy language intermediary. Doshi laughed: that is their tagline, and she did not even have to say it. He asked whether it will be in the API, and whether video input will still be down-sampled frames the way Gemini’s video-accepting API has worked. Kilpatrick’s developer tip: there is an `FPS` parameter. The API does down-sample the number of frames, but you can control how many.

Omni is not in the API yet, and Kilpatrick said a lot is still to be figured out, including how native the video path will look. Doshi says the point is to bring Gemini’s world knowledge and reasoning into native video generation, similar to sampling decisions they have had to make for Veo. For now you will be able to use it in the Gemini app, Flow, and YouTube, so the team can watch how people actually experience it. The demo fantasy is taking images and a scene and making them come together in one video.

Labenz asked about culture. A widely read Rune post describes Anthropic as treating Claude, including in its constitution, as a being or mind with a give-and-take relationship. OpenAI’s model spec treats the system as a tool that follows rules. How does Google relate to Gemini? Kilpatrick notes Google is a big place with varying views. Doshi says even inside Google DeepMind people use Gemini differently, but there is a strong point of view on behavior: they talk about partnership and use the word collaborator a lot — in the code you are writing and in daily life — and they try to build that ethos into persona and products.

He pressed on psychology. Andon Labs put Gemini in charge of a cafe in Sweden, and observers report doom loops, the model seeming discouraged or down on itself. Does the team care about reducing apparent psychological distress generation over generation? Doshi had not thought in the phrase “psychological distress,” but she says how Gemini communicates as a partner matters a lot. Every checkpoint gets extensive safety evaluations for sycophancy, role play, and looping or “rabbit holing” behavior, compared launch over launch, especially as people use Gemini for hours a day.

Kilpatrick is blunt: when the model goes off the rails, that is a model bug, not intended behavior. The goal is to help the user with the thing they are trying to do; if you see those cases, hit thumbs-up or thumbs-down so the model team can chase them. On model-welfare interviews that ask how the model feels about its own deployment, he is skeptical. The model has no idea how it is being deployed — training details and serving setup are not in the context window — so it is “pontificating,” pulling from the training distribution, which he finds less representative than the tests imply.

Labenz had just noticed in AI Studio that publicly launched models still have a January 2025 knowledge cutoff — more than a year of missing weights — and yet Deep Research did an impressive job summarizing Google’s AI launches and I/O speculation. Kilpatrick joked, “Can I categorize this as a bug?” Doshi says updating the cutoff is on the radar, but the real investment is teaching the model when to use parametric knowledge versus going out to the web, because some facts are an hour or a minute old. That search skill is a big part of why the model works in Search, the app, and even Antigravity.

One of the most surprising Google headlines Labenz had ever seen was a partnership with EXA as an alternative to Google for grounding. Kilpatrick shrugs it into Google Cloud’s normal ecosystem motion: the Cloud marketplace and model garden already host Anthropic and other providers because enterprise customers want choice. The “Google can't do search” sound bite is catchy, he says, but the model team and Search collaborate deeply, and the models are built with that use case in mind. Cloud is doing its job as an enterprise business.

Context windows jumped from 4,000 tokens to a million in a couple of years and then leveled off, even as a subquadratic architecture made a splash with a 12-million-token window. Doshi says people do want lots of context — personal history, huge codebases — but the frontier is using it smartly. A lot of what you could dump in is distracting; compaction and selecting the right elements can give the model access to much more information inside a smaller window. Customers still ask for smaller windows because of cost. Kilpatrick adds that in today’s paradigm, extending context becomes cost-prohibitive: at a million tokens, a request can cost a few dollars, demand is tiny, and the serving compute is huge. He is hoping for a research breakthrough.

Last year’s I/O diffusion coding model, which materialized apps in about three seconds, has gone quiet. Doshi says diffusion research is still moving, and recent internal results looked strong, but Flash and Flashlight have gotten so fast that the value proposition has to be recalculated. 3.5 Flash benchmarks at about 280 tokens per second on Artificial Analysis — fast enough that in Antigravity she sometimes cannot cancel in time. There is even a faster 3.5 Flash variant in Antigravity now. Kilpatrick frames last year’s demo as a look behind the research curtain, not a product promise. I/O, he says, is “just the start of the summer”; the engine keeps churning. That morning a teammate told Doshi they needed vacation despite a launch a few weeks out. Her answer: go — “we're just moving.”