En 30 segundos
On the day Gemini 2.0 Flash reached production at 10 cents per million input tokens, Kilpatrick explains the Flash, Flash-Lite, Pro and reasoning lineup and why Google positions each as it does. He also argues that reasoning will unlock long context and agents, that vision-language models will displace domain-specific computer vision, and that choosing models still comes down to evals most teams don't have.
- 1Gemini 2.0 Flash entered production at 10 cents per million input tokens and 40 cents per million output tokens, which Kilpatrick calls a huge accomplishment given the pressure to keep prices low.
- 2Text-to-app tools are extremely cost-intensive, and Kilpatrick cites startup LLM bills of roughly $40,000 to $50,000 a month that Flash could cut to about $1,000 or less, roughly a 40x reduction.
- 3Long context may find its real unlock in reasoning: models struggle to attend to a thousand things in a 2 million token window, but thinking through the context could change that.
- 4Vision-language models can do jobs that took painful custom computer vision, so the whole ecosystem built on domain-specific vision models is up for grabs for startups.
- 5Most teams choose models by vibes: in many cases people have few evals to say which metrics matter for their product, which is why Kilpatrick wants a platform for personal evals that runs against each new model.
- 6Kilpatrick expects a new class of startups to emerge as the internet's social contract breaks when visitors to a website are agents, not humans.
Lee la transcripción completa por capítulos
Bringing research and product together inside DeepMind should speed up both
Logan Kilpatrick, product manager for the Gemini API and AI Studio at Google DeepMind, returns to The Cognitive Revolution for what host Nathan Labenz suggests is a record fourth appearance. On launch day for Gemini 2.0 Flash in production, they start with the reorganization. Kilpatrick says he joined Google 10 or 11 months ago and has collaborated with DeepMind from day one.
DeepMind has moved from fundamental research to productionizing models, and within the last three months the Gemini app, AI Studio and the Gemini API moved under it. It is now an organization that does the research, creates the models and brings them to products. For outsiders who don't care about reorgs, he says, the hoped-for result is faster model progress and faster product progress.
Cheap Flash makes text-to-app tools viable and real-time multimodal a glimpse of AI co-presence
Asked about the coolest uses of the API, Kilpatrick points to text-to-app creation software. Bolt had just added Gemini support, and Cursor now supports Gemini and uses 2.0 Flash, with he hopes others such as Lovable and v0 to follow. The economics of these products are very cost intensive, and he sees a new product paradigm and a lot of startup activity in domain-specific software that people can create without knowing how to code.
He has seen tweets about startup LLM costs on the order of $40,000 or $50,000 a month, which he says would be probably $1,000 or less with Flash, a roughly 40x reduction. Labenz calls this period a "software supernova", with non-coders creating basic full-stack apps, and notes that the next two episodes feature the founders of Lovable and Bolt.
The multimodal live API, previewed in December, offers conversational video and text interaction, and Kilpatrick says it moves toward the AI "co-presence" that sees what you do. Feedback has ranged from developers using it for coding to a blind user navigating daily life. Labenz describes using advanced voice mode to get help in Mario 64 with his kids, and imagines it in glasses or helping seniors.
Kilpatrick cautions that this will take time: text was a toy demo a year and a half ago and is now at billion-user scale. Sessions are limited to 10 minutes today, and keeping memory and state while bringing the cost of an always-present AI down will take iteration. He adds that the notion that "everything is a wrapper" with little value keeps being wrong, because new capabilities keep opening new products.
Reasoning may be what makes long context truly useful, and native multimodality is coming
Labenz asks about the status of long context and multimodal inputs. Kilpatrick says he spoke the previous day with Jack Rae, a co-lead on reasoning who previously worked on the long-context breakthroughs, and they speculated that reasoning could be the real unlock for long context. A 200,000-token window versus a million or two million matters in some cases, but a model can handle questions about a few items far better than assembling a thousand different things.
Letting models think through a long window, and perhaps use tools to move information in and out, is where he thinks long context makes more sense. Labenz agrees that he picks the model that thinks longest for hard problems and wouldn't dump a huge context into one that jumps straight to an answer. Kilpatrick suggests using AI Studio's compare mode to test 2.0 Pro versus a reasoning model on long context.
On multimodality, video, audio and images are processed natively, and 2.0 shows the next step of models producing images and audio, available to early testers. Imagen 3 was to arrive in the Gemini API the next day. He argues native generation matters because specialized image models trade off world knowledge for pretty pictures, and models trained with the capability bring the Gemini models' knowledge, which he expects to enable use cases that did not work before.
Vision-language models can replace brittle custom computer vision
Labenz asks about passive applications, such as monitoring a factory floor for safety incidents, or a camera that could replace the fall monitors his grandmother's senior community residents dislike wearing. Kilpatrick says it is core to his thesis about domain-specific software. In a past life as a machine learning engineer working with security cameras, he found that tracking a person across camera frames was incredibly hard with traditional computer vision.
Vision-language models do these tasks incredibly well, he says, and Flash makes the cost very low, although he hasn't talked to anyone with such a system in production. He also points to the bounding-box demo in the AI Studio starter apps, which marks objects like custom detection models do.
He expects the startup wave to go after industries that use domain-specific vision models. A general model changes the cost basis and also handles cases those rigid models cannot tolerate. Labenz adds that every new frontier model makes some subset of Y Combinator companies' products start to work, while others wait for the next one.
The launch lineup: Flash in production, Flash-Lite, Pro and reasoning, with generous rate limits
Kilpatrick notes that well-funded YC startups may not feel cost reductions much, so he thinks cheaper models give more power to individual developers without tier-one VC backing. Then he gives the lineup. Gemini 2.0 Flash, first released experimentally in December, is now in production at 10 cents per million input tokens and 40 cents per million output tokens.
Also announced were a preview of Flash-Lite, the smaller variant to be available for production soon, and an experimental 2.0 Pro, the most capable frontier model, rounding out the flash reasoning model. Labenz asks about limits. Kilpatrick says Google doesn't publish AI Studio limits because they can change, but the API free tier is about 10 or 15 requests per minute and 4 million tokens per minute.
On the paid tier there is no daily request limit, with a default of 2,000 requests per minute and 4 million tokens per minute, and new quota tiers rolling out at midnight that unlock 10 million tokens and 10,000 requests per minute. He explains that Google can't make every experimental model generally available, since the compute footprint to serve Cursor, Bolt and other startups is large.
Flash-Lite exists, he says, because 2.0 Flash costs more than 1.5 Flash, and the team didn't want a story of raising developers' prices after emphasizing low cost per intelligence. Flash-Lite offers a better model at the old price point, partly by dropping expensive features such as native image and audio generation. He notes the 8-billion-parameter Flash model was among the highest-volume models on OpenRouter, and Labenz says that a 7.5 to 10 cent difference rarely breaks a business.
Pro is for pushing the frontier, especially in coding, and Ultra faded for cost reasons
Kilpatrick describes Pro as the model where cost doesn't bind, and says its price hasn't been announced but will follow past Pro patterns and be much higher. The traditional advice is to start with the best model, make the use case work, then bring costs down by switching to a smaller model or fine-tuning, and he says Google wants to honor that flow.
Coding is where he sees Gemini's best performance relative to other domains. He reaffirms an earlier tweet that Google will have the world's best coding model, which he thinks will be Pro plus the reasoning work, and calls coding a domain Google needs to win. Pro keeps two million tokens of context, and he says that talk of infinite context is something he hopes will be a huge unlock.
Asked about Ultra's disappearance, he says it was meant to prove that scaling would continue, which Gemini 1.0 did, and then smaller models quickly surpassed it. A model 5% better on every benchmark but five times larger doesn't pay off in research or infrastructure terms, especially with reasoning offering low-hanging fruit. He adds Google is still scaling pre-training and that releasing an Ultra model remains an open question.
Picking a coding model is mostly vibes because few teams have evals
Labenz notes that many people still think Claude 3.5 Sonnet is the best coding assistant, and asks how to compare a new Gemini Pro against Claude and o3-mini. Kilpatrick says there is no silver bullet and tells a story: stuck on a table with rounded corners, he gave up on Gemini, tried Claude in Cursor and it worked in one shot. When he re-ran the same prompt with Gemini, it worked too, so his earlier prompting had been the problem.
He says this shows how unscientific model selection is. Working with teams with LLMs in production, he finds that people have surprisingly few evals about which metrics matter. He wants a platform that hosts public benchmarks and leaderboards in one place, since keeping track of them is hard even for him, and he jokes about it as a call for startups.
He also cites Kaggle's push toward personal evals that run behind the scenes on new models and email you when one looks good at the things you care about. Labenz says Waymark's tasks, like what makes a good video, have no right answer, so they check basic guardrails and otherwise rely on vibes. Both suggest letting users choose the model, as GitHub Copilot now does with a model dropdown.
Kilpatrick argues the commoditization of models is overstated, because substantive differences in answers will persist even where benchmarks look similar, and that choice matters to end products. Labenz points to DeepSeek as having a very different profile, closer to a base model, with compelling writing but less behavioral refinement, despite similar benchmark results.
Fine-tuning, reasoning and agents are where he sees the next startup waves
On fine-tuning, Kilpatrick says he remains incredibly bullish on a future where everyone uses their own fine-tuned model. It isn't available for 2.0 Flash, Google is debating how much to invest, and current 1.5 Flash tuning has rough edges, such as no images. On reinforcement learning, he says RL is part of making the reasoning models and Google took the usual low-key approach for experimental releases, but will talk more about the work soon.
For startups, he says this is the moment to build, starting with vision, where he thinks the ecosystem of domain-specific vision models is up for grabs. He believes reasoning will make agents work, since many agent companies today don't, and that this will be the key breakthrough of the next two years.
He also expects a class of startups around the web changing when visitors are not human. The internet's social contract, he says, assumed only humans and index crawlers, so websites will need help talking to agents, handling attribution and capturing their slice of the value. Labenz notes this is likely to suit reinforcement learning because tasks give clear success signals.
The last idea is evals again. Kilpatrick says humans find it hard to articulate taste in a way that can be applied programmatically, and that many problems in life end up as eval problems. Labenz says Waymark currently uses expert demonstrations, fine-tuning or few-shot examples and then vibes. Kilpatrick says he is on Twitter and LinkedIn, though he can't reply to everything anymore.
Más de este canal
1:25:56EN
2:17:48EN
2:00:51EN
1:26:46EN
1:10:12EN
1:30:54EN