Building AI Products When the Ground Keeps Moving

Sarah Guo opens by asking Kevin Weil and Mike Krieger about taking on CPO roles at OpenAI and Anthropic. Kevin describes it as the most challenging and interesting product role of his life because the underlying technology is not fixed. He says that every two months computers can do something they have never done before, so product leaders must continually ask how that changes what they build.

Kevin explains that normal product work assumes a fixed technology base, while AI product work assumes capability shifts. He says the answer to how much a new capability should change the product is probably a fair amount. Mike says reactions to his move ranged from “that makes sense” to “why would you do that?” and that he could not stay retired.

Mike notes that his semi-retired state lasted about six weeks before he wanted to know what was next. He says people who knew him expected him to enjoy the role, while others wondered why he would work again. He also jokes that only a short list of companies could have made the move interesting.

Mike says enterprise timelines felt different after Instagram, and the rewarding part is direct customer feedback once deployment happens. Kevin adds that enterprise is not only about the product: there is a buyer with goals, and a great product can still fail to matter. He recalls a large customer asking for 60 days' notice before launches, joking that he would also like to know 60 days ahead.

Kevin says his product instincts apply at the tail end of shipping features such as advanced voice mode or Canvas. The beginning is different because a capability may be emerging from model training without anyone knowing whether it will be 60%, 90%, or 99% good. He describes product leaders checking in with research teams and hearing that the answer is still uncertain.

Kevin says the product you build for a capability that works 60% of the time is very different from one built for 90% or 99% reliability. Mike compares the feeling to Apple's WWDC announcements, except the disruption comes from inside your own company and can upend your roadmap. He says this makes planning feel like discovery under uncertainty.

Mike describes three ways to plan: watching the intelligence slope, investing in product-led fine-tuning with research teams, and responding to new capability fronts such as computer use. He says Anthropic embeds designers early and treats experiments with research as ways to produce learning or demos, not always predictable products. The outcome may be informative prototypes that spark product ideas.

Kevin says parts of research, especially post-training, are very product-oriented, while other parts are more academic. Occasionally a researcher reveals that a capability has existed for three months because nobody knew it was important. He says this creates moments of magic alongside the uncertainty.

Sarah Guo asks how teams should think about a model that is 60% successful at a valuable task instead of 99% successful. Kevin says many things can work at 60% if you expect a human in the loop and design for it. He cites GitHub Copilot as an early example, guessing it was built on a much older, smaller model such as GPT-2, yet it was valuable because it got code partway there and the user could edit it.

Kevin says agentic and longer-form tasks can also be valuable even when imperfect, especially if they save five or 10 minutes. If the model knows where it lacks confidence and asks for help, the combination of human and model can exceed 60%. Mike says performance is lumpy, and pilot customers can give opposite feedback on the same day, which is humbling after internal evals.

Kevin argues that models today are not intelligence-limited but eval-limited: they can do more if taught the right things. Mike says many AI deployments from three years ago never built evals because teams were focused on shipping cool features. The hardest step is often defining what success looks like, and Claude can help write and grade evaluations once that definition exists.

Mike says Anthropic's interview process asks candidates to improve a bad eval, because not enough people have that skill. He says evals will become a core PM skill, and internal research PMs and product-surface PMs are converging as feature quality depends on evals and prompts. Kevin adds that OpenAI ran a PM bootcamp on writing evals, and Mike notes that nothing beats looking at actual data because graders and golden answers can be wrong.

Mike says prototyping with models is underused. Anthropic's best PMs prompt Claude to compare possible UIs before designers have opened Figma, which lets teams explore more variety and evaluate faster. He says this skill of being in prototyping mode with AI tools is increasingly valuable.

Kevin says PMs will also need to go deeper into the tech stack, not necessarily to become researchers but to gain intuition and learn the language. He says stochastic, non-deterministic systems require feedback loops, guardrails, and aggregate monitoring of model behavior. That is different from a traditional bug report where a button click predictably fails.

Kevin says non-determinism contradicts 25 years of computing intuition because the same inputs no longer produce the same outputs. Builders must put themselves in users' shoes and think about what that means for them. He says there are downsides but also cool upsides to design around.

Mike says user research now involves both user behavior and model behavior. Researchers may feel pride when the model reacts well and frustration when it misunderstands intent and goes 10 pages down the wrong answer. He says product people need a little Zen about letting go of control in those environments.

Kevin compares AI adoption to riding a Waymo: the first 30 seconds feel alarming, five minutes in feels like the future, and soon the rider is scrolling on a phone. He says ChatGPT is less than two years old, yet the original GPT-3.5 already feels primitive. People adapt quickly to magic and then expect more.

Mike says Anthropic is trying to make the product educational by telling Claude more about itself and how to use its features. This came from user research where people asked Claude how to use something and got unhelpful answers. He says models are actually good at solving UI confusion and should be used more for that.

Mike says enterprise deployments expose non-technical users to chat-powered LLMs for the first time, and companies can run teaching sessions and share educational materials. Power users often become internal evangelists who teach others. Kevin adds that OpenAI's custom GPTs let organizations package AI so it is immediately useful to people who might not know how to use it otherwise.

Mike says computer use was late-breaking but Anthropic had conviction to release it despite early mistakes. During beta testing, someone asked it to order pizza and Domino's showed up at the office, which became a memorable moment. Early promising use cases include UI testing, where brittle snapshot tests may give way to computer-use checks, and automating repetitive data movement in support and finance. Mike describes the goal as automating drudgery so people can focus on creative work.

Kevin says o1 cannot be used as a one-for-one replacement for existing models in every application. Sophisticated customers and OpenAI itself orchestrate workflows across models, using each for what it does well. o1 is good at reasoning but takes time to think and is not multimodal.

Kevin defines o1 by contrasting it with pretraining scaling, which he describes as system-one thinking: the model answers immediately, one token at a time. o1 scales intelligence at query time by pausing, forming hypotheses, and reasoning before answering. He says models currently think for 30 or 60 seconds, and he imagines what happens when they can think for five hours or five days; he calls this the GPT-1 phase of the new reasoning approach.

Kevin says you do not want a model to wait 60 seconds for every question, so different models should be used together. In cybersecurity, for example, teams can fine-tune models for precision and have models check other models' outputs. He says this orchestration mirrors how humans with different skill sets work together on complex tasks.

Mike predicts more proactivity from models that know about the user and monitor authorized sources such as email. They could spot trends, give a morning recap, prep the user for a meeting, or draft the first version of a presentation. He says this kind of proactive assistance could be very powerful.

Mike also expects more asynchronous interactions. He sees o1 as an early UI for this: the model explains what it will do, thinks for a while, and lets the user tab away until it is done. He says breaking free from immediate-answer expectations will let users ask for project plans, bug fixes, or PRD adaptations for new market conditions.

Kevin says models will get smarter at an accelerating rate, and they will interact in more human ways through voice, sight, and typing. He describes using advanced voice mode in Korea and Japan as a universal translator during business conversations with someone who shared no common language. He says this could change people's willingness to travel and make such experiences commonplace.

Mike notes a TikTok genre of young people pouring their hearts out to voice mode, which suggests AI-native expectations are forming early. Kevin says his kids ask to talk to ChatGPT on car rides and have bizarre conversations with it. Sarah Guo describes children directing stories about dragons and unicorns and asking for images to be created in real time.

Mike says users understand the nuance of Claude as a model and react when a new version feels smarter but more distant. He says shipping AI is not just shipping a product; it is shipping intelligence, and intelligence and empathy shape relationships. He compares it to a friend saying they were upgraded and are now different in some way, which makes him want to adapt.

Kevin says model behavior is absolutely a product role and that model personality is key. He raises the question of how much a model should customize versus how much OpenAI should have one personality and Claude another. He notes that people choose friends because they like different personalities, so users may choose models for similar reasons.

Kevin describes a prompt that went viral: users asked the model, based on everything it knows about them and their past interactions, what it would say about them. The model responded with a description based on those interactions, and people reacted strongly. He says users are starting to interact with the model almost like a person or entity.