En 30 segundos
In this short clip, Theo Browne explains the case for embedding independent evaluators at AI companies rather than relying only on API requests and responses. The document on screen describes a proposal for ongoing access and states that Anthropic is committing to the step.
- 1The black-box limitationTheo contrasts an evaluator sending API requests with a deeper assessment inside the company.
- 2Embedded evaluatorsThe proposal would give outside evaluators employee-like access to help assess model safety.
- 3The stated commitmentThe document shown on screen states that Anthropic is committing to ongoing third-party access.
Conclusiones
- API-only evaluation exposes inputs and outputs but may leave evaluators unable to see internal processes.
- The proposal calls for embedded third-party evaluators with employee-like access.
- The document says Anthropic is committing to the access described.
Lee el texto editado por capítulos
From API testing to embedded evaluation
Theo explains why the proposal calls for evaluators to have access inside model companies, beyond sending requests to an API.
The proposal I’m discussing argues against relying on a simple black-box evaluation: give an evaluator an API key, send a series of requests, inspect the responses and hope for the best. That setup leaves the people testing a model with a limited view of what happens inside the company.
Instead, the proposal calls for third-party evaluators to be embedded in AI companies with access comparable to employees, so people assessing safety can see more of the company’s work. The document shown on screen describes ongoing access and states that Anthropic is committing to this step; the clip does not establish how any particular arrangement works in practice.
Más de este canal
36:07EN
Theo - t3․gg· 27 oct 2026Theo on Token Costs, Coding Agents, and the Price of AI CapabilityTheo Browne · Ping LabsA viral estimate of Lauren's AI spending prompts a closer look at cache pricing, benchmarks, and the changing work of coding agents.
30:37ENTheo - t3․gg· 26 oct 2026Theo’s Three Skills for Reviewing and Shipping with Coding AgentsTheo Browne · Ping LabsHow sol-loop, full-send and full-stop encode review, verification and different levels of merge authority.
0:43ENTheo - t3․gg· 23 oct 2026Theo Distinguishes Security Vulnerabilities from Active AttacksTheo Browne · Ping LabsA short explanation of why disclosed software flaws and reported attacks are different kinds of security news.
29:22ENTheo - t3․gg· 22 oct 2026Theo on Security Vulnerabilities, Stolen API Keys and AITheo Browne · Ping LabsA look at the gap between vulnerability disclosure and patching, and a speculative account of why stolen inference keys may be valuable.
31:13ENTheo - t3․gg· 20 oct 2026Theo on Why Smarter Models Help More Than DevelopersTheo Browne · Ping LabsFrom game modifications to compiler experiments, Theo revisits his assumptions about who benefits from more capable AI.
31:47ENTheo - t3․gg· 18 oct 2026Theo on Asking Agents for Outcomes and Building the Tools They NeedTheo Browne · Ping LabsA tool-call report, a troubleshooting playbook and connected data show how prompting can reshape product engineering.