Gemini’s product chief on testing models and building ahead of them

Tulsee Doshi leads product for Google’s frontier models, including Gemini, generative media, Gemma and audio. Her portfolio spans image and video generation, music, speech and translation.

Lenny Rachitsky asks what people misunderstand about creating models. Doshi says teams set capability and evaluation goals, but training is not fully predictable: a product manager must shape the inputs and then discover surprising strengths in what emerges.

Researchers monitor training runs and intervene using metrics, making as much of the work scientific as possible. Doshi also wants product managers to understand how communicating with a model feels and where a new capability changes a downstream product.

Rachitsky asks about perceptions that Google trails other frontier labs. Doshi says Google wants to compete at the frontier while building across hardware, models and products, and across different cost and quality points for different users.

She says the goal is both strong frontier models and useful experiences across the company’s products. The interviewer points to Google’s existing distribution through products such as Gmail, Calendar and Docs.

Asked about AI safety, Doshi draws on her six years in responsible AI. She describes monitoring model behavior and possible misalignment, intervening when necessary, and treating deployment safeguards as part of the process.

She also discusses using AI to strengthen cyberdefense, citing work identifying and fixing vulnerabilities. Asked whether she is optimistic about safety, she says optimism depends on sustained, careful scrutiny and people actively questioning how models are built.

Rachitsky asks whether the race to release capabilities makes AI products hard to navigate. Doshi says teams need to experiment with interaction designs, including separate experiences for video creation, research and other tasks.

The appropriate interface depends on the audience: a search feature serving a broad population must be accessible, while a developer tool can support terminals, IDEs and slash commands for power users.

Doshi argues that a model itself has a user experience: how transparent it is, how it communicates and what questions it asks. Product managers and researchers turn user feedback and a desired behavioral specification into evaluation rubrics and training data.

Research product managers also work with internal teams such as Search, the Gemini app and YouTube, as well as external customers, to understand which capabilities would change their experience and feed that information into development.

Rachitsky observes renewed appreciation for product managers as AI makes it easier to build things people may not want. Doshi, acknowledging her own bias as a PM, says taste, identifying useful signals and aligning people around a coherent direction become more valuable.

When hiring research PMs, she wants people who use and stress-test models daily, have opinions about their behavior, can inspect data and build evaluations, and are willing to revisit a roadmap as the field changes.

For a first evaluation, Doshi suggests starting with five prompts rather than a large benchmark. Even a small set of around 20 prompts can expose differences between models on a capability. A person can initially judge the responses and define what good looks like before scaling.

To develop intuition for a model’s character, she recommends testing things the evaluator actually cares about, such as visual animation, writing or financial analysis. Pick a task where existing models show promise but still fall short, so differences are visible.

Doshi recalls work on skin-tone representation. A six-category Fitzpatrick scale had been built for a medical purpose and was a poor fit for representing global diversity in product evaluations, she says.

Her team worked with Harvard professor Ellis Monk and incorporated his ten-tone scale into evaluations. Doshi says the revised measurements helped guide work on Pixel cameras and image generation.

Evaluation must cover the full user experience, not just the model in isolation. The judge’s quality, filters, safety mitigations and serving stack can change results; for audio, content, voice quality and delivery all matter.

Rachitsky asks about agent swarms. Doshi distinguishes her earlier Stanford research with Michael Bernstein: it studied temporary teams of human crowdworkers with complementary skills, assembled for a task and then dissolved.

She sees parallels between designing those flash teams and organizing subagents, while noting the original work involved people. The interviewer raises a reported agent incident; Doshi returns to the general idea of assembling effective temporary teams.

Doshi’s 60-20-20 framework addresses building products as models improve. She says the exact percentages are flexible: make roughly 60% work now, another 20% promising but imperfect, and reserve the remainder for capabilities where early signs suggest a valuable future experience.

The approach balances a functional product today with room for improvement. Doshi says even someone working on models cannot predict exactly where they will be in three months, so she uses signs visible in the product rather than a precise forecast.

A rigid wrapper around an older model can also hold back a newer one. Rachitsky mentions periodically removing skills to retest them; Doshi agrees that teams should keep checking whether existing structure still helps.

She defines a harness as the environment around a model: system instructions, available tools and skills, and the context or data it can access. Together these determine what the model can use to act.

For PM upskilling, Doshi says team members share useful skills and teach workshops on reading data and long model traces. A shared repository lets others reuse and improve tools, building confidence as well as capability.

She says the practical constraint is time: constant sprints leave little space to learn. Her team is planning a two-day workshop where PMs step away from daily work to experiment, build and share.

Rachitsky suggests inserting a question before starting a task: can AI help? Doshi says one successful example can change the habit. She also asks whether a task done twice can be delegated, while retaining work such as setting strategy herself.

Rachitsky asks why Google has not yet produced a more compelling personal assistant using Gmail, Calendar and Docs. Doshi says the team is working on the problem and agrees there is more work to do on where and how to meet users; she gives no specific launch commitment.

The interviewer asks how her team avoids burnout as AI raises expectations. Doshi says the team is working long hours and she is still considering how to create a sustainable environment.

In her view, burnout can also come from churn, chaos and work that feels pointless. She hopes AI can remove frustrating routine work and leave more time for meaningful creative work, while acknowledging that sustainability still needs attention.

On AI-generated document overload, Doshi distinguishes collecting information from providing insight. She finds AI useful for pulling post-launch metrics and feedback from multiple sources; people should contribute the interpretation and judgment.

She sees the PM role moving beyond writing requirements for others to flesh out. AI can help PMs prototype and inspect data, but she expects them to add substance to their proposals, whatever form that takes.

For books, Doshi recommends Exhalation for its questions about people and digital life, Creativity, Inc. for building a culture that supports creativity and recovery from failure, and fantasy titles including Throne of Glass.

She recommends Severance and says voice dictation with Wispr Flow helps with wrist problems. Her personal advice is to seek serendipity by taking on unfamiliar challenges and putting oneself in a position to get lucky.

Rachitsky asks about her training in classical Indian dance. Doshi says dancing focuses her attention, connects her with friends and culture, and has long been part of the India Festival she helped organize in Portland.

She encourages listeners to try Google’s models and audio features and send feedback through products, companies, direct messages or public posts. The interview ends with Rachitsky thanking her.