In 30 Sekunden
In this short clip, Theo Browne explains the case for embedding independent evaluators at AI companies rather than relying only on API requests and responses. The document on screen describes a proposal for ongoing access and states that Anthropic is committing to the step.
- 1The black-box limitationTheo contrasts an evaluator sending API requests with a deeper assessment inside the company.
- 2Embedded evaluatorsThe proposal would give outside evaluators employee-like access to help assess model safety.
- 3The stated commitmentThe document shown on screen states that Anthropic is committing to ongoing third-party access.
Das Wichtigste
- API-only evaluation exposes inputs and outputs but may leave evaluators unable to see internal processes.
- The proposal calls for embedded third-party evaluators with employee-like access.
- The document says Anthropic is committing to the access described.
Den bearbeiteten Text nach Kapiteln lesen
From API testing to embedded evaluation
Theo explains why the proposal calls for evaluators to have access inside model companies, beyond sending requests to an API.
The proposal I’m discussing argues against relying on a simple black-box evaluation: give an evaluator an API key, send a series of requests, inspect the responses and hope for the best. That setup leaves the people testing a model with a limited view of what happens inside the company.
Instead, the proposal calls for third-party evaluators to be embedded in AI companies with access comparable to employees, so people assessing safety can see more of the company’s work. The document shown on screen describes ongoing access and states that Anthropic is committing to this step; the clip does not establish how any particular arrangement works in practice.
Mehr aus diesem Kanal
38:47EN
Theo - t3․gg· 28. Okt. 2026Theo’s Long-Running Agent Experiments: Verification, Compute and CostTheo Browne · Ping LabsA skateboard game and two compiler projects show how repeated checks and flexible compute can sustain agent work—and how expensive those loops can become.
36:07ENTheo - t3․gg· 27. Okt. 2026Theo on Token Costs, Coding Agents, and the Price of AI CapabilityTheo Browne · Ping LabsA viral estimate of Lauren's AI spending prompts a closer look at cache pricing, benchmarks, and the changing work of coding agents.
30:37ENTheo - t3․gg· 26. Okt. 2026Theo’s Three Skills for Reviewing and Shipping with Coding AgentsTheo Browne · Ping LabsHow sol-loop, full-send and full-stop encode review, verification and different levels of merge authority.
0:43ENTheo - t3․gg· 23. Okt. 2026Theo Distinguishes Security Vulnerabilities from Active AttacksTheo Browne · Ping LabsA short explanation of why disclosed software flaws and reported attacks are different kinds of security news.
29:22ENTheo - t3․gg· 22. Okt. 2026Theo on Security Vulnerabilities, Stolen API Keys and AITheo Browne · Ping LabsA look at the gap between vulnerability disclosure and patching, and a speculative account of why stolen inference keys may be valuable.
31:13ENTheo - t3․gg· 20. Okt. 2026Theo on Why Smarter Models Help More Than DevelopersTheo Browne · Ping LabsFrom game modifications to compiler experiments, Theo revisits his assumptions about who benefits from more capable AI.