In 30 seconds
Noam Brown argues that frequent model releases can make it hard to explore capability limits, especially for long-running tasks, and discusses the tension between evaluation time and competitive pressure.
- 1Fast release cyclesFrequent model releases can outpace extended testing.
- 2Long-running tasksSome capabilities become clearer only after lengthy use.
- 3A difficult tradeoffMore pre-release evaluation takes time amid competitive pressure.
Takeaways
- New models can arrive before extended testing is complete.
- Long tasks may reveal capabilities only over time.
- Pre-release testing competes with release schedules.
Read the edited text by chapter
The evaluation gap
OpenAI researcher Noam Brown says that when new models arrive every few months, users may still be testing the previous model when the next one appears.
OpenAI researcher Noam Brown says that when new models arrive every few months, users may still be testing the previous model when the next one appears.
He argues that this makes it difficult to know the full extent of a model’s capabilities, especially on tasks that take a long time to run.
Brown cites users running tasks for more than a week after the release of a feature called /goal as an example of capabilities becoming clearer only with time.
Time and competition
Asked how labs can evaluate models before release, he says more time would help but would also delay releases, creating tension with competitive pressure. This is his assessment, not proof that evaluation is impossible.
Asked how labs can evaluate models before release, he says more time would help but would also delay releases, creating tension with competitive pressure. This is his assessment, not proof that evaluation is impossible.
More from this channel
38:06EN
No Priors: AI, Machine Learning, Tech, & Startups· Oct 29, 2026Podcast · 3 speakersServal’s Jake Stauch on coding agents for enterprise ITGuest Jake Stauch · Host Sarah Guo, Elad GilStauch explains why Serval is challenging ServiceNow, how it approaches permissions and approvals, and why background agents remain difficult to deploy safely.
1:07ENNo Priors: AI, Machine Learning, Tech, & Startups· Oct 27, 2026PodcastGeorge Kurtz on How AI May Change CyberattacksGuest George KurtzThe CrowdStrike CEO describes his assessment of evolving threats.
39:23ENNo Priors: AI, Machine Learning, Tech, & Startups· Oct 15, 2026Podcast · 2 speakersThe Defenders' Window: Greg Brockman on AI-Generated Knowledge, Drug Discovery, and the New Physics of CybersecurityHost Sarah Guo, Elad GilOpenAI's president explains why solving open math problems is a proof point for science, how proactive agents change product form, and why security spending must exceed software spending.
1:04ENNo Priors: AI, Machine Learning, Tech, & Startups· Oct 14, 2026PodcastWalter Goodwin on the Slow Steps in Chip DesignGuest Walter GoodwinA chip founder explains why automation does not remove every computational bottleneck.
1:09:44ENNo Priors: AI, Machine Learning, Tech, & Startups· Oct 9, 2026Podcast · 2 speakers10,000 GB300s for Four Weeks: Reflection’s Open-Model LedgerHost Sarah Guo, Elad GilMisha Laskin tells No Priors how Reflection trained Beam, why reinforcement learning outspent pre-training, and why the West needs its own open weights.
35:39ENNo Priors: AI, Machine Learning, Tech, & Startups· Oct 2, 2026Podcast · 2 speakersCompute Scaled a Millionfold. Bandwidth Only 40x. Fractile Wants to Fix That RatioHost Sarah Guo, Elad GilIn the summer of 2022, Walter Goodwin founded Fractile in London to build very fast inference chips for the world's largest models.