30 秒でわかる
Noam Brown argues that frequent model releases can make it hard to explore capability limits, especially for long-running tasks, and discusses the tension between evaluation time and competitive pressure.
- 1Fast release cyclesFrequent model releases can outpace extended testing.
- 2Long-running tasksSome capabilities become clearer only after lengthy use.
- 3A difficult tradeoffMore pre-release evaluation takes time amid competitive pressure.
ポイント
- New models can arrive before extended testing is complete.
- Long tasks may reveal capabilities only over time.
- Pre-release testing competes with release schedules.
チャプターごとに整えた全文を読む
The evaluation gap
OpenAI researcher Noam Brown says that when new models arrive every few months, users may still be testing the previous model when the next one appears.
OpenAI researcher Noam Brown says that when new models arrive every few months, users may still be testing the previous model when the next one appears.
He argues that this makes it difficult to know the full extent of a model’s capabilities, especially on tasks that take a long time to run.
Brown cites users running tasks for more than a week after the release of a feature called /goal as an example of capabilities becoming clearer only with time.
Time and competition
Asked how labs can evaluate models before release, he says more time would help but would also delay releases, creating tension with competitive pressure. This is his assessment, not proof that evaluation is impossible.
Asked how labs can evaluate models before release, he says more time would help but would also delay releases, creating tension with competitive pressure. This is his assessment, not proof that evaluation is impossible.
同じチャンネルの記事
38:06EN
No Priors: AI, Machine Learning, Tech, & Startups· 2026年10月29日ポッドキャスト · 3 人Serval’s Jake Stauch on coding agents for enterprise ITゲスト Jake Stauch · ホスト Sarah Guo, Elad GilStauch explains why Serval is challenging ServiceNow, how it approaches permissions and approvals, and why background agents remain difficult to deploy safely.
39:23ENNo Priors: AI, Machine Learning, Tech, & Startups· 2026年10月15日ポッドキャスト · 2 人The Defenders' Window: Greg Brockman on AI-Generated Knowledge, Drug Discovery, and the New Physics of Cybersecurityホスト Sarah Guo, Elad GilOpenAI's president explains why solving open math problems is a proof point for science, how proactive agents change product form, and why security spending must exceed software spending.
1:04ENNo Priors: AI, Machine Learning, Tech, & Startups· 2026年10月14日ポッドキャストWalter Goodwin on the Slow Steps in Chip Designゲスト Walter GoodwinA chip founder explains why automation does not remove every computational bottleneck.
1:09:44ENNo Priors: AI, Machine Learning, Tech, & Startups· 2026年10月9日ポッドキャスト · 2 人10,000 GB300s for Four Weeks: Reflection’s Open-Model Ledgerホスト Sarah Guo, Elad GilMisha Laskin tells No Priors how Reflection trained Beam, why reinforcement learning outspent pre-training, and why the West needs its own open weights.
35:39ENNo Priors: AI, Machine Learning, Tech, & Startups· 2026年10月2日ポッドキャスト · 2 人Compute Scaled a Millionfold. Bandwidth Only 40x. Fractile Wants to Fix That Ratioホスト Sarah Guo, Elad GilIn the summer of 2022, Walter Goodwin founded Fractile in London to build very fast inference chips for the world's largest models.
42:44ENNo Priors: AI, Machine Learning, Tech, & Startups· 2026年9月24日ポッドキャスト · 2 人They Paid $7.7 Billion for a Broker — and Intend to Do One Deal a Yearホスト Sarah Guo, Elad GilSequence Holdings co-founder Michael Lee tells No Priors why owning a market leader, not selling it software, is how AI actually remakes an incumbent.