30초 요약
Noam Brown argues that frequent model releases can make it hard to explore capability limits, especially for long-running tasks, and discusses the tension between evaluation time and competitive pressure.
- 1Fast release cyclesFrequent model releases can outpace extended testing.
- 2Long-running tasksSome capabilities become clearer only after lengthy use.
- 3A difficult tradeoffMore pre-release evaluation takes time amid competitive pressure.
핵심 정리
- New models can arrive before extended testing is complete.
- Long tasks may reveal capabilities only over time.
- Pre-release testing competes with release schedules.
챕터별로 다듬은 전문 읽기
The evaluation gap
OpenAI researcher Noam Brown says that when new models arrive every few months, users may still be testing the previous model when the next one appears.
OpenAI researcher Noam Brown says that when new models arrive every few months, users may still be testing the previous model when the next one appears.
He argues that this makes it difficult to know the full extent of a model’s capabilities, especially on tasks that take a long time to run.
Brown cites users running tasks for more than a week after the release of a feature called /goal as an example of capabilities becoming clearer only with time.
Time and competition
Asked how labs can evaluate models before release, he says more time would help but would also delay releases, creating tension with competitive pressure. This is his assessment, not proof that evaluation is impossible.
Asked how labs can evaluate models before release, he says more time would help but would also delay releases, creating tension with competitive pressure. This is his assessment, not proof that evaluation is impossible.
같은 채널의 다른 글
38:06EN
No Priors: AI, Machine Learning, Tech, & Startups· 2026년 10월 29일팟캐스트 · 3명Serval’s Jake Stauch on coding agents for enterprise IT게스트 Jake Stauch · 진행자 Sarah Guo, Elad GilStauch explains why Serval is challenging ServiceNow, how it approaches permissions and approvals, and why background agents remain difficult to deploy safely.
1:07ENNo Priors: AI, Machine Learning, Tech, & Startups· 2026년 10월 27일팟캐스트George Kurtz on How AI May Change Cyberattacks게스트 George KurtzThe CrowdStrike CEO describes his assessment of evolving threats.
39:23ENNo Priors: AI, Machine Learning, Tech, & Startups· 2026년 10월 15일팟캐스트 · 2명The Defenders' Window: Greg Brockman on AI-Generated Knowledge, Drug Discovery, and the New Physics of Cybersecurity진행자 Sarah Guo, Elad GilOpenAI's president explains why solving open math problems is a proof point for science, how proactive agents change product form, and why security spending must exceed software spending.
1:04ENNo Priors: AI, Machine Learning, Tech, & Startups· 2026년 10월 14일팟캐스트Walter Goodwin on the Slow Steps in Chip Design게스트 Walter GoodwinA chip founder explains why automation does not remove every computational bottleneck.
1:09:44ENNo Priors: AI, Machine Learning, Tech, & Startups· 2026년 10월 9일팟캐스트 · 2명10,000 GB300s for Four Weeks: Reflection’s Open-Model Ledger진행자 Sarah Guo, Elad GilMisha Laskin tells No Priors how Reflection trained Beam, why reinforcement learning outspent pre-training, and why the West needs its own open weights.
35:39ENNo Priors: AI, Machine Learning, Tech, & Startups· 2026년 10월 2일팟캐스트 · 2명Compute Scaled a Millionfold. Bandwidth Only 40x. Fractile Wants to Fix That Ratio진행자 Sarah Guo, Elad GilIn the summer of 2022, Walter Goodwin founded Fractile in London to build very fast inference chips for the world's largest models.