「你今天用的，是这辈子最差的模型」：OpenAI CPO 如何在能力每两个月一跳的节奏里做产品

Weil says the reaction to OpenAI's new image model was one of those rare moments when internal usage simply explodes. He compares it to Instagram Stories, where staff could feel it would work because everyone was using it and knew what colleagues had done over the weekend. Before the launch, OpenAI's internal gallery of generations produced nonstop buzz.

He adds that social products are an especially good test, because a company is a tight network of people who are experts in the product. If a social feature is not taking off inside the company, he says, you should question what you are doing. He also stresses that the model follows instructions well enough to handle complex requests, such as rearranging a living room from a set of photos.

Asked when AGI arrives, Weil quotes the idea that AI is whatever has not been done yet: once something works it is called machine learning, and once it is everywhere it is just an algorithm. His Waymo story makes the point: terror in the first ten seconds, calm after five minutes, and email on the phone after fifteen. He argues we adapt to AI the same way, with a miracle blowing our minds for a week before it becomes expected.

The biggest difference from Twitter, Instagram or Facebook, Weil says, is pace and the fact that the underlying technology is no longer fixed. A database is perhaps 5% better than two years ago, but with AI, every two months computers can do something they have never done before. Product teams need to rethink what they are doing each time.

Traditional software takes defined inputs and gives defined outputs, so the same action three times yields the same result. Language models are good at fuzzy, nuanced inputs but return answers that are only spiritually the same each time. That makes the product more probabilistic by nature.

The practical consequence is that reliability sets the product. If a model gets a use case right 60% of the time you build one kind of product, at 95% another, and at 99.5% another again. Weil says teams therefore have to get deep into their specific use case and its evals, because a database that works once works every time but a model does not.

Weil describes an eval as a quiz for a model, in the way a calculus test checks what a student learned. Evals benchmark creative writing, graduate-level science or competitive coding, and Lenny's shorthand of unit tests for models is one Weil accepts. They tell you which tasks the model gets right 99.95% of the time and which only 60%, and those facts are not static.

He uses deep research as the example. The team designed evals alongside the product, writing hero use cases with an ideal answer for each, then hill-climbing on them while fine-tuning. Deep research can work for 25 or 30 minutes on a task that might take a person a week. When scores on those evals climbed, Weil says, the team knew it had a product.

On whether AI is capped by how good we are at evals, he notes that intelligence is multidimensional: being great at competitive coding differs from front-end work or converting COBOL to Python. Most of the world's data and processes are private and sit behind company walls. Like a new employee who spends two weeks onboarding, models are smart enough to learn anything but need the raw data, so he expects broad models tuned with company-specific data and custom evals.

On where startups should build, Weil repeats a line from Twitter's Ev Williams: there are always more smart people outside your walls than inside. OpenAI has 3 million developers on its API and says it neither wants to grow huge nor could it build the many industry-specific, data-specific products that AI makes possible. The opportunity in each vertical, he says, is immense.

Internally, OpenAI roadmaps quarterly and has a year-long strategy but does not believe that what is written down will ship. Quoting Eisenhower's "Plans are useless. Planning is helpful," Weil values the pause to review what worked and check dependencies, then keeps the process light. Teams are bottoms-up because engineers, PMs, designers and researchers who build the product also understand what the models can do.

Two philosophies back this up. Iterative deployment means shipping before you know the full set of capabilities, and learning together with society in public. Model maximalism means not building too much scaffolding around current weaknesses, because within two months a better model will erase them. If your product sits right at the edge of what models can do, Weil says, keep going.

Lenny cites the StackBlitz founder, whose product only began working when Sonnet 3.5 arrived after years of effort. Weil credits Anthropic's strong coding models and says OpenAI's former massive lead is gone, though he believes it still leads. He compares this to the four-minute mile, where one person breaking the barrier is followed by a dozen more the next year, and says competition benefits consumers and developers.

Weil says ChatGPT won consumer mindshare partly by being first, which is why OpenAI wants to launch new capabilities quickly. It also became a one-stop shop: voice, video input, deep research, canvas, code and increasingly agentic tools such as Operator. His goal is for ChatGPT to be the most useful place to bring any request.

His most counterintuitive lesson is that you can often reason about AI products the way you reason about a person. A reasoning model that needs 20 seconds to think raises a UI question; Weil asked what a person would do. They would not go silent, nor babble every thought, so ChatGPT shows short progress updates instead. Similarly, a group of models whose outputs are merged resembles human brainstorming.

OpenAI also learned from seeing how people reacted to a model's raw reasoning. After DeepSeek showed far more of its chain of thought, the team concluded that at the scale of 400 million people, showing the full chain is not right. Its answer was a middle ground, a sentence or two summarizing how the model is thinking.

Weil also defends chat as an interface, a view he says most people do not share. Human communication is unstructured, and a rigid interface would limit what you could say. Language models are finally good enough to handle that complexity, so chat fits them exactly. Specialized, prescribed interfaces are still right for high-volume use cases, but chat remains the catch-all baseline.

Weil traces OpenAI's shift from a pure research company, where ChatGPT began as a low-key research preview, to a company that must be both world-class research and world-class product. If product teams are merely an API consumer of research, he says, they lose the loop that makes products great. His model is engineering, product, design and research working as one team, iterating between evals, data and fine-tuning.

OpenAI has not that many PMs, he says, perhaps 25 or a little more. He believes organizations should be PM-light because too many PMs fill the world with decks and ideas instead of execution. A PM with slightly too many engineers cannot micromanage, which leaves influence with product-focused, high-agency engineers.

For hiring, he looks for high agency, comfort with ambiguity and the ability to lead through influence. He says more junior PMs struggle because nobody has time to hand them a defined area. Emotional intelligence matters too, since the research function is highly self-directed and a rapport with it is essential.

Weil adds that the most important thing a PM can do is be decisive. He rejects the idea of the PM as the CEO of the product, but compares the role to Sam Altman's: making every decision is a mistake, and so is making none. Where there is ambiguity and nobody is making a call, the PM should make sure someone does.

Everyone at OpenAI uses ChatGPT for summarizing documents and writing specs, and models can help write evals, Weil says. Even so, he admits to being disappointed in himself: his day job would still be recognizable to his five-years-ago self. He thinks teams should be vibe coding demos instead of showing Figma mocks, and cites the chief people officer who vibe coded an internal tool in Windsurf.

He defines vibe coding, a term from Andrej Karpathy, as letting go of the wheel: accepting the model's suggestions, pasting errors back in and telling it to keep going. It is not yet for tight production code, he says, but it works well for proofs of concept and demos.

Looking ahead, Weil expects researchers or machine-learning engineers embedded on most product teams, because fine-tuning will become part of the core workflow. He is surprised that fine-tuned models are not used more widely given how much better they can be on a specific use case. Models will be everywhere like transistors, he says.

Internally, OpenAI uses ensembles: ten problems might be solved with twenty model calls, some on specialized fine-tunes and some on models of different sizes for latency or cost. Customer support is his example, where a team of perhaps 30 or 40 handles 400-plus million weekly users, and human corrections become fine-tuning data. A company, he adds, is itself an ensemble of differently fine-tuned people.

Weil's children are 10 and 8 (twins), and he says they are strikingly AI native, treating self-driving cars and conversations with ChatGPT as normal. He does not claim to know which jobs will matter, though coding skills may stay relevant. He and his wife Elizabeth focus on teaching their kids to be curious, independent, self-confident and able to think.

He calls personalized tutoring maybe one of the most important things AI can do, alongside speeding up scientific discovery. Studies show multiple standard deviation gains in learning speed with personalized tutoring, and ChatGPT is free and works on Android phones worldwide. He is surprised no AI tutoring product is yet used by billions of kids and says he wants it to exist.

On fears about jobs and superintelligence, Weil calls himself a technology optimist, noting that technology has driven advances in quality of life, economics and longevity over the last 200 years. He accepts that there are temporary dislocations and individuals who are hurt, and that the average being good is not enough. OpenAI works with policymakers and on education, and he sees ChatGPT as a strong reskilling tool.

Weil is optimistic about AI-assisted creativity. He calls himself the world's worst artist, but with ImageGen he can produce something he could not make alone. A film director told him that a cut scene that once cost a 3D effects company about a hundred thousand dollars and a month, and yielded two options, can now be explored through Sora in 50 variations before the final version is made at the studio.

He says that you do not type "make me a great movie" into Sora, because it still takes creativity, but it helps you explore more. He also points to new creative-writing research that Sam Altman has previewed, as another example of OpenAI's habit of sharing work in progress and launching early and often.

Weil describes the pace of improvement: new GPT models used to arrive every six to nine months, and now the o-series arrives roughly every three or four months. Costs have fallen as well; he says the original GPT-3.5 cost about 100 times what GPT-4o mini does today. Models are getting smarter, faster, cheaper and safer, hallucinating less each iteration, which he sees as a much steeper exponential than Moore's Law.

That is why he repeats his mantra that today's model is the worst you will ever use. Lenny agrees and says people dismissing Sora as not good enough are missing that point. Weil's answer is model maximalism: keep building for what is almost possible and the model will catch up.

Weil calls Libra, led by David Marcus, the biggest disappointment of his career. The problem was real: remittances can cost people sending money home as much as 20% in fees and take days. With 3 billion people on WhatsApp, he asks why sending money cannot be as instant, cheap and simple as sending a text.

He says the team did not play its cards perfectly. They tried to launch a new blockchain, a basket of currencies and WhatsApp and Messenger integration at once, at a time when Facebook's reputation was at its lowest. He thinks introducing the changes one at a time might have reached the same outcome, and he owns those decisions.

He notes that the current administration is friendly to crypto and Meta's reputation is in a different place, so perhaps the company should build it now. The technology was open-sourced from the start, and Weil says Aptos and another company built on it are doing well, so the work lives on. Still, he says, we should be able to send each other money in WhatsApp.

In the lightning round, Weil recommends Ethan Mollick's Co-Intelligence, Peter Zeihan's The Accidental Superpower and Cable Cowboy, a biography of John Malone. He says he wants to watch The Wheel of Time and praises Top Gun 2. Among products, he singles out vibe coding with Windsurf and riding in Waymos.

His life motto comes from a Facebook earnings call where Mark Zuckerberg said growth was not any one thing but good work consistently over a long period of time. Weil, an ultra marathon runner, had it made into a poster. He says people look for silver bullets when excellence is mostly showing up daily and compounding, even if the gains are invisible over a week or month.

On prompting, he wants to kill the idea that you must be a good prompt engineer, comparing it to knowing the storage engine in MySQL: useful at the edge, unnecessary for most people. For now, he suggests poor man's fine-tuning by putting examples and good answers into the prompt. He also confirms that role framing, such as asking the model to be Einstein or the world's greatest marketer, can shift it into a useful mindset, much as it does with people.

Weil closes by asking listeners for feedback on where ChatGPT works and where it fails, which he reads on X as @kevinweil. Lenny says the pair of them are at the edge that is creating the future.