Same Intelligence, 1% of the Price: Logan Kilpatrick on Gemini Flash

Nathan Labenz welcomes Logan Kilpatrick back to The Cognitive Revolution as one of the few people who has sat inside both of the leading AI labs. Kilpatrick led developer relations at OpenAI; he is now a product manager at Google, responsible for AI Studio, the playground-like surface in front of the Gemini API. The taping landed just after Google I/O, in a gap Kilpatrick carved out before dinner with his family. Labenz frames the hour around culture, the speed of DeepMind's research, the split among AI Studio, the Gemini API, and Vertex, and a model he thinks builders cannot ignore.

That model is Gemini 1.5 Flash. In Labenz's opening tally, it significantly outperforms the original GPT-4 on head-to-head user comparisons, offers a 100x longer context window, and costs just 1 to 2% as much per token — an advance packed into a single year. Fine-tuning for Flash is on the roadmap, and depending what else ships in the meantime, Labenz says it could become the most powerful model available to fine-tune anywhere in the world. The rest of the conversation is about what that price-performance jump does to product strategy, and how Google wants developers to find the front door.

Kilpatrick had barely left OpenAI when Google people started calling. He talked with his now-manager and with Josh Woodward, the VP who leads Google Labs, intern-to-executive across 15 years, and who delivered several of the I/O keynotes. The hiring loop felt slow to Kilpatrick — he was burning daylight — but he read it as the company making an order-of-magnitude bet on AI. Woodward's team built the Gemini API and AI Studio; Kilpatrick's team is inheriting those products as they scale past a million developers. Matching that urgency, he says, is hard. It is also why he took the job.

Labenz has his own small proof of Google's current intensity. When Gemini 1.5 first went to early testers, he was not on the list. He DMed Jeff Dean on Twitter at about 11:30 on a Saturday morning; Dean replied within an hour, copied Woodward, and by Sunday morning Labenz had million-token access. No layer of seniority, he concluded, is sitting this out. Kilpatrick notes that Woodward started as a Google intern. Dean answering fan mail on a weekend is the same posture.

The culture contrast Kilpatrick will draw is specific. Early OpenAI let almost anyone drop into a research meeting and hear how the models were progressing; after ChatGPT and GPT-4, a thick wall went up between research and everyone else, for reasons he calls legitimate. Google also protects IP, with clear boundaries and roadblocks. Even so, it still feels to him as if people want to collaborate. DeepMind, he says, actually cares about developers, because developers are one of the primary surfaces for getting the technology into the world. That is a different feeling from the ChatGPT-era instinct to put most of the eggs in the consumer-chat basket.

The example is almost too neat. The weekend before the taping, an engineer named Susan on the DeepMind team sent a long, well-written doc listing everything hard about getting started with Flash. Within 24 hours the AI Studio team turned around a batch of changes. Research volunteering to debug the developer on-ramp is not the stereotype of a frontier lab. Kilpatrick's takeaway is practical: that collaboration is what lets him build the right things for developers.

The tension has flipped. DeepMind is doing so much that Google is no longer rate-limited by research innovation; it is rate-limited by whether product teams can wrap the models thoughtfully for developers and for first-party surfaces. At the close of the I/O keynote, someone counted the word "AI" at 121. Kilpatrick's own favorite first-party workflow is the Gemini notetaker in Google Meet: one click, then an email with summary, action items, and people tagged. He used Gemini to plan a trip to the Azores with his girlfriend and export the itinerary to a Google Doc. The honorary mention is Gemini in Chrome. Billions of people use the browser, and if you want to think about world-scale impact, that is the distribution that matters.

Labenz wants a straight answer on the phrase Google keeps using for Gemini 1.5: natively multimodal. Is it one transformer that eats every modality, or an internal pipeline? Kilpatrick's developer-facing tldr is simpler. You put in whatever type of content you want, you get text out today, and you do not pay the cost of routing across multiple models. Historically you chained audio, image, video, and text, paid round-trip latency on every hop, and watched multimodal features die in production. Flash collapses those four models into one call, at speed.

The roughly 150-page Gemini 1.5 technical report, released just after I/O, does not spell out the architecture. Kilpatrick is not sure whether that is a detail Google should push into the open or secret sauce the team is deliberately holding back. Labenz offers a test: if the model notices intonation and emotion in audio, the audio is probably not being transcribing-then-prompted as plain text. Kilpatrick says the question has come up, the answer may be limited to certain contexts, and he will follow up, because the use cases developers build depend on it.

On the lineup, he is willing to prune. Gemini 1 is fully superseded. 1.5 Pro is now better across pretty much every axis than Gemini 1.0 Ultra, a model that was never made widely available, in part because the team could already see 1.5 Pro coming. Builders should move their mental model to Flash and Pro. On the LMSYS leaderboard, the new Pro models sit within a few points of GPT-4o. One of the two 1.5 Pro entries is Gemini Advanced, a showing-off, online version that lives only in the first-party UI and is not in the API. Kilpatrick has already told the LMSYS team they should do what Hugging Face does: a model card, so people can tell which name is which.

Labenz is surprised DeepMind still publishes work that looks like top-tier IP, naming Infini-attention, a paper that resembles a state-space idea with a fixed-size, constantly updated attention slot and talk of infinite context. Kilpatrick has no special knowledge of that paper. He lists the usual reasons labs still publish: theater, giving researchers credit, attracting academic talent, floating an idea that might work, and a colder calculus — maybe a rival has already figured out something similar from what another paper did not say. Publishing can be economic even when it spends a trade secret.

Labenz floats a rumor he has tried not to over-weight: that DeepMind had impressive internal state-space results, sat on them, and only brought work out after Mamba landed. Kilpatrick's reply is the line from the episode that traveled. The number of rumors that are true, he says, is so small it rounds down to zero. Most of what you read is nowhere close to reality. When he had specific knowledge of things people were rumoring about at OpenAI, it was never true. Developers should keep asking questions, then apply a first-principles skepticism rather than treating the rumor mill as a roadmap.

The thing he actually wants builders staring at is vision. He sat with Greg Brockman in the weeks before the GPT-4 launch, going over the napkin-drawing demo, and thought multimodal was going to land immediately. Until Flash, he now says, natively multimodal vision use cases were not actually possible on cost and latency. At the AI Engineer Summit the previous year he called 2024 the year of multimodal models, and he thinks that is turning out to be true. Most production vision queries are some version of "tell me what is happening in this part of the image." Flash can emit a bounding box, or several. Older state-of-the-art vision models had no grip on spatial coordinates. Custom segmentation models, and a lot of UI-clicking automation, just became a prompt.

Labenz is the AI advisor to Athena, an executive-assistant company whose old onboarding spent an hour on a call and four hours writing a client profile. That write-up is now mostly language models. He pulled his own 250 most recent sent emails — content, replies, forwards, about a thousand tokens each, a quarter million tokens — and dumped them into Flash with almost no prompt engineering. The model returned a two-page character sketch covering the podcast, Athena, and his other AI projects. Pulling the mail out of Gmail took longer than the inference. The whole job ran three to four minutes, Flash used about 45 seconds, and at the 70-cent rate the bill was under 20 cents.

Original GPT-4's 8,000-token window would have held maybe five to eight of those emails, before any output, with failures and stitching. Labenz jokes that context-window inflation is the best hyperinflation. The same request in Gmail's built-in Gemini was refused, a reminder that the API and the first-party product are not symmetric. He is already thinking about the next jobs: every gift a client has ever mentioned, every loyalty number, dumped into a spreadsheet. Kilpatrick, who is an Athena customer, has been doing the same move in AI Studio with the Gemini 1.5 technical report — upload the PDF and ask. They try it live later; the model honestly says the document does not explicitly define native multimodality, which is what he expected.

Then he puts the prices on the table. Original GPT-4, the March 14, 2023 version, was $30 per million input tokens. Flash is 35 cents below 128k tokens and 70 cents above, for a model he describes as just as intelligent and more capable. That trend — faster, smarter, cheaper — is not going to stop. Too many builders still sprinkle "AI" onto a feature list. The world he wants them designing for is one where intelligence is in every surface of the product, one of the primary interfaces, and eventually too cheap to meter. Project Astra's glasses demo, he suspects, can be mostly recreated with the Flash API and a few hundred dollars of off-the-shelf hardware. It is on his to-do list.

Flash's LMSYS Elo is 1232: almost 50 points above original GPT-4, 70 above the unpopular June 2023 GPT-4, and only about 15 behind the January GPT-4, at a small fraction of the price. Kilpatrick has mostly moved his daily AI use to Flash and feels it is about three times faster than Pro, though latency is a messy public number — load, time of day, and peak patterns all move time-to-first-token. He wants a benchmark with three sliders for intelligence, price, and latency, because Elo alone underrates a model that is multimodal and an order of magnitude cheaper than peers. The 128k price jump is a hard cutoff, meant to reflect that long-context compute is not linear. Internally, thousands of Google teams dogfood the same Gemini API developers get; they only fork into custom post-training when a surface like Maps needs data the base model does not have.

Labenz admits he has been confused about where to go inside Google: Cloud, Vertex, AI Studio. Kilpatrick says simplifying that map is at the top of his mind. Vertex is the enterprise cloud path — legal, regulatory, privacy, SLOs, signed agreements. If you are a developer who wants Flash or 1.5 Pro in a startup, you start in AI Studio: a playground, an API key, the option to tune a model, then production on the Gemini API, which scales. The analogy he offers is OpenAI versus Azure OpenAI. Most developers pick the lower-friction door for a good reason. Unless you have a specific enterprise-cloud need, you can ignore Vertex.

AI Studio's DNA is consumer. It began as MakerSuite, aimed at automation flows, then became a developer playground. Labenz uses it as a personal seminar: one saved template holds about 20 mixture-of-experts papers. When GPTs launched he tried stuffing a whole codebase in and watched the system retrieve instead of read, missing pieces; he just wanted the model to load the full text, a need few products still honor. Kilpatrick's team has talked about importing email into Studio so people can feel the model work. They are holding back on purpose. The mission is developers building on the API, not a consumer workspace that accidentally becomes the product.

On the roadmap, Veo will not roll out like Sora. Kilpatrick does not have the calendar, but the VideoFX tooling already looks robust, and Josh Woodward has described a distinctly different strategy. Function calling will get a v2, meaning real improvements even if the version number is informal. The Gemini API already has a semantic retrieval API for uploading documents into a corpus; the current design may not be the long-term one. Vertex's agent builder is further along, and some of that learning may come across. Fine-tuning 1.5 Pro is not available. Fine-tuning Flash was set to be announced on May 30, with a pricing twist nobody else offers: no charge to train, and no extra charge on inference.

Labenz's rule of thumb is that fine-tuning is for narrowing a single task with tightly controlled inputs and outputs; ad hoc jobs like "write as me" are better served by stuffing context. He failed at cloning his own prose on OpenAI's models. Kilpatrick almost founded a company on that problem in the gap between jobs. The hard part is not the tuning API, it is the old machine-learning grind: a balanced, representative dataset. Most people do not have enough of their own writing, so they need synthetic magnification, and facts still want retrieval. Low-rank fine-tuning, he suspects, is not a magic way around that.

Back at OpenAI, Kilpatrick had said the company is only as good as its latest model, a line Labenz heard as a winner-take-all warning, because swapping models is usually just a re-prompt. Kilpatrick now rejects that ending. He has enormous respect for Sam Altman and still disagrees with the "I never think about competition" stance; people are painfully aware of how fierce it is. The opportunity is large enough that more players, not fewer, will enter, including people who would not normally compete here. Open source will keep pushing toward the frontier. Labenz expects maybe ten credible frontier shops, plus a Chinese champion and perhaps Mistral for Europe. Kilpatrick will not pick a number, but anyone with technical understanding and enough resources will try, and he thinks betting the entire market is the worse business plan. A great model can be enough to make a company valuable.

His tweet that if AGI arrived today people would not know what to do with it was not aimed at Labenz, who lives in this ecosystem. It was aimed at the rest of the world. Most people have never used GPT-4. Most have never used Gemini 1.5. Hand someone AGI through an API and the time for the world to change is still very long. At I/O he was struck by Connor, who spends his time teaching people with no technical background how to use the tools. Most of the planet is not technical and does not care about the technology except as something that helps them get a job done.

Labenz left I/O anxious about platforms. Google can put meeting notes in Meet for free, with one click, and a dozen startups live in that niche. ChatGPT's desktop app and Project Astra are racing toward a proactive coworker that watches the screen. Kilpatrick does not deny distribution. He also misses, now that he is back inside a large company, the startup advantages: a small group that just executes, without the coordination tax of every extra headcount. If he were at a startup today, he says, he would be grinning. Meeting-note tools are oddly well placed to become some of the first agents that actually do work, because they already have the context, and context is half the battle. Users bounce across Zoom, Teams, and Meet, which is a cross-platform opening Google does not automatically own.

The moat is not the transcript. If the system drafts follow-up emails, pings the right people, and the user only has to check and send, that is a different product from a notepad. There is always more to automate after the recording stops. Kilpatrick's closer is the line he has been repeating in other rooms: there has never been a better time to be a startup in this space. Everything is up for grabs. Models will keep getting faster, smarter, and cheaper. The builders who matter will design as if intelligence is about to be too cheap to meter.