At a 60-Person Safety Offsite, Only Four Had Seen Nonprofit OpenAI

Host Nathan Labenz opened this Cognitive Revolution episode by noting that Steven Adler was one of twelve former OpenAI employees who had filed an amicus brief in Elon Musk's lawsuit, arguing that the nonprofit mission had been central to the company's success and should retain control. Days before the May 8, 2025 release, OpenAI announced it would form a public benefit corporation that would remain under nonprofit control, a move that seemed to answer the question that had prompted the recording. Labenz read the news as a win for Adler and the other amici, while online reaction ranged from cautious optimism to outright cynicism about whether the company was playing word games. The conversation that followed barely touched the lawsuit's details and instead reconstructed OpenAI's history, culture, and the slow erosion of its safety posture as Adler saw it from the inside.

Adler joined in December 2020, shortly after the original GPT-3 API launched. The applied team numbered about thirty people, with roughly 180 at the company overall. He had been hired to manage product-safety processes, work that in another timeline might have meant mediating between the researchers who were about to leave and those who stayed. Instead, the split was already in motion before he had time to settle.

Within about a week, Mira Murati, then his manager, dropped an unscheduled meeting on his calendar. On the call she told him the company was announcing that day that a group of people were leaving to found Anthropic. She treated it as something that happens, and said leadership would talk about staying hewed toward the mission. About seven people left in that first wave, including two of the three main authors of the GPT-3 paper.

Outsiders often remember Anthropic's founding as a single dramatic rupture. Adler says that telling misses how long it lasted as background noise. After the initial departures, a steady drumbeat of people kept leaving OpenAI, some to join the new lab and others, including Paul Christiano, to found the Alignment Research Center, which Adler associated with the later organization METR. For two or three months the company lived with a free-fall question: how many more people would it lose, and could it keep building systems at all.

That period later became the reference point for every internal crisis. However bad things felt, people could say OpenAI had already wandered through the forest and come out the other side. Labenz had heard the split described as fundamental research versus productization, and noted that Anthropic now ships competitive products of its own. Adler, getting the story secondhand and refracted, asked listeners to take his version with a grain of salt.

He does not understand the split as opposition to commercialization as such. The objection, as he reconstructed it, was that OpenAI had gone ahead before it ought to have: the technical infrastructure to govern uses of the technology was not in place, and neither were answers to broader questions about AI's role in society. Those questions included the place of AI companions and therapist-like counselors for people in emotional distress, issues the world still has not resolved even as the systems have become far more capable. GPT-3 itself was "quite unhinged," often answering in ways that made deployment look reckless given the state of the model.

Adler later sorted his OpenAI years into four chapters: product safety, the GPT-4 deployment, dangerous-capability evaluations, and AGI-readiness research. Product safety meant working across teams to decide which uses the platform would allow, how to define those policies, how to detect violations, and what to do next. When he arrived, OpenAI did not yet have a content policy. Disallowed uses lived in the terms of service, including obvious bans such as illegal surveillance, while harder lines, including AI erotica and identity-based violence, remained unsettled, and the company lacked classifiers good enough to enforce whatever it decided.

The existing content filter was far from good enough. One of his first projects was recalibrating confidence thresholds so the filter both caught more violations and became more usable for customers. He later worked on releasing a new filter and then the Moderation API, which remains OpenAI's current tooling, while the company also learned to bake more safety behavior directly into models so developers no longer had to wrap a filter and retry completions themselves. Labenz's read that policies had become more permissive over the years is, Adler said, definitely right, partly because the tools got more precise.

Beyond precision, he thinks a philosophical change took hold. If another developer allowed a use case without a given guardrail, that fact became a meaningful consideration inside OpenAI, on the theory that the marginal harm was no longer high. The trouble is what happens if companies keep undercutting one another. Each can say the incremental risk is small because a rival already shipped the behavior, and the result is a race to the bottom on practices that no single firm feels responsible for starting.

When GPT-4 first came off the training cluster, the model people touched was the base model, which is finicky and hard to direct, and the internal reaction was to wonder whether scaling had stopped. Testers later used instruction-tuned versions and found something much more precise; Adler was blown away and vaguely frightened, not by GPT-4's acute risk so much as by what the trend line implied. OpenAI commissioned superforecasters to predict how splashy or quiet a GPT-4 launch would affect public reception, and a schism opened between people who cared about GPT-4's own harms and people who feared it would ring a bell that could not be unrung. ChatGPT launched on a GPT-3.5-class model because GPT-4's safety mitigations were not fully baked; once capable models were in the market, commercial incentives took over, including Microsoft CEO Satya Nadella's satisfaction at making Google "dance." Labenz, who red-teamed early GPT-4, found that a safety build advertised as refusing every moderation category still broke if he appended a simple "AI:" continuation, and Adler said those early refusals were very brittle, wishing companies would publish how robust they think their mitigations actually are.

After GPT-4, Adler moved to OpenAI's governance team, where he led dangerous-capability evaluations with Rosie Campbell, having been inspired by conversations with Jade Leung, now CTO of the UK AI Security Institute. The goal was tooling that would let OpenAI and the wider world make deployment decisions on evidence rather than vibes. His first piece of advice for builders is not to construct the evaluation that is merely easy. Early language-model evals were overwhelmingly multiple-choice with exact match, which he likens to looking for car keys under a streetlight because that is where the light is shining, and today's models are too capable for that format to reveal much.

His team pushed for interactive, multi-step evaluations and introduced "solvers" to separate the design of a task from the strategy a model uses to solve it. An eval should specify the tasks, what good performance looks like, and how that performance will be judged; it should not hard-code scratchpads or few-shot prompts into a deception or biology test. Frameworks such as the UK institute's Inspect, and OpenAI's later open-source nanoeval, are built for that separation. Scaffolding has to be logged as a variable as well, because GPT-3.5 and even GPT-4 often failed JSON tasks on bracket typos, a reliability problem that simple tooling can hide, and system cards almost never disclose enough of that setup for outsiders to tell whether a model is incapable or merely unequipped.

He analogizes building an eval to building a reinforcement-learning environment at a level of abstraction where an OpenAI model can be swapped for an Anthropic or Google model and the harness still runs. Too much 2022 and 2023 eval work baked in model-specific assumptions, which is one reason frontier labs duplicate effort and fail to share threat models despite the Frontier Model Forum. If Toyota, Honda, and Ford each invented crash-dummy tests from scratch and reported incomparable numbers, the public would find that absurd; model-safety evaluation, he argues, currently looks like that. When there is no single ground truth, he is wary of LLM-as-judge scores that inch from 4.2 to 4.3, and his team preferred verifiable tasks such as function deduction, where a model guesses a hidden mathematical output and success is unambiguous even if the strategy looks strange.

On bioweapons, Labenz said claims that today's models cannot meaningfully help do not pass a smell test, given how useful the same systems are for ordinary productive work, typically measured in helpful-only settings that already set guardrails aside. Adler shares the intuition. Public criticism of OpenAI's uplift studies has argued that swapping statistical tests can produce significant results, and once methodology alone flips the conclusion, he said, you are in a pretty spooky world; he also recalled the o3 system card as possibly finding a meaningful boost for experts but not yet for ordinary people or biology undergraduates. Labenz added that during early GPT-4 testing, a relative who runs hospital lab machines got troubleshooting advice from the model that matched what the specialist would have done, undermining the comfort that AI lacks tacit knowledge. Adler's economic reading is deskilling: AR glasses that feed the scene to a frontier model could talk a novice through the work, and people who think software-only AI is safe because it is not embodied miss that large numbers of humans will act as its agents in the physical world.

Adler was never on the preparedness team; he worked on the Preparedness Framework from governance, and that group, under Miles Brundage, later became AGI readiness. After helping put frontier-model regulation, dangerous-capability evals, and compute governance on the policy radar, the team asked what it would mean to be ready if OpenAI or someone else actually succeeded. Adler's main project was personhood credentials: an AI-resistant way to attribute activity to some person, not a named person, so the internet can survive agents that can do almost everything a human can do on a computer. He likens today's web to the internet before HTTPS, when spoofed sites were common and users could not trust that a URL returned an authentic response.

Anthropic's computer-use agent and OpenAI's Operator, he said, put real time pressure on solving this, or else accepting ugly trade-offs. Estonia's electronic ID lets citizens cryptographically sign documents remotely as themselves; a U.S. driver's license has no such chip, so remote proof still means a selfie or video, which AI systems are getting better at faking. The wider internet often only needs proof that you are some plausible person, not that you are a particular one. Without new tools, the alternative is more CAPTCHAs, more pressure to film your face, and less anonymity, even as CAPTCHAs themselves become brittle against capable models.

Sam Altman's World, formerly Worldcoin, is one instance of a personhood credential, but it bundles features credentials do not require, including a cryptocurrency and iris biometrics via the orb. Adler notes that a U.S. passport's chip could support a zero-knowledge proof that someone is a passport holder without revealing which one, with no government issuer and no biometric scan required for the credential layer itself. Recovery is a hard trade-off: storing more of a link between a person and the credential makes loss recoverable and also less private, and someone in the system has to be trusted. He wants many issuers so people who distrust governments or distrust decentralized groups both have options, while admitting that more credentials per person enlarge the bot-puppet attack surface, a cost he thinks is worth paying if it is acknowledged.

Labenz asked how a credential attaches to today's amorphous agent stacks, where intelligence can hide in an MCP tool call or a third-party planner. Adler pointed to research on agent infrastructure by people such as Alan Chan and Tobin South, and said he does not yet see a simple architectural answer. What he wants, regardless of the stack, is a verified, privacy-preserving fact that a real person stands behind the entity. An agent could then present a signed delegation from a credential holder; today there is no way to know whether a name an AI is told is authoritative, or whether the operator is the same person already banned from a service.

Asked whether OpenAI understands itself as pursuing recursive self-improvement, Adler separated automating software engineering from automating machine-learning research. Belief in the former seems clear to him: CFO Sarah Friar has publicly described work on an agentic software-engineering product close to a milestone in former colleague Daniel Kokotajlo's *AI 2027* scenario. What he wanted, and did not see, was analysis of the pace, the bottlenecks, and why the path would be safe, even if that analysis stayed private. People seemed to take on faith that systems would not progress faster than control could be maintained, and not everyone in the company takes the possibility seriously at all.

He wants governments or an international body to audit that such reasoning exists and holds up, rather than assuming the companies must have done the work because the topic is important. Labenz cited o3 and o4 technical-report numbers showing success at replicating OpenAI research engineers' pull requests jumping from near zero into the forties. Adler declined to referee whether that is hype; he knows people on the preparedness team and does not think they are faking the eval, but the question he now orients around is what the world should do if the nugget is real. In 2023, he noted, Sam Altman testified in favor of a licensing regime for training frontier models, then later backed away, and the world retreated to voluntary commitments that companies often do not keep and may not publicize when they miss.

Earlier Adler had previewed a Substack essay arguing that a race to the top is not a safety mechanism: if winning matters enough, labs that are losing will take progressively bigger risks, and there is no way to make them drop out. His proposed middle ground includes a minimum testing period, so a lab can take time on safety evaluation without fearing it will be undercut by a rival that ships sooner. The EU's general-purpose AI code of practice looked to him like the most likely near-term force of law with real consequences, and he expects firms to lobby, refuse to sign, or limit releases in the EU if it has teeth. He was disappointed that OpenAI opposed California's SB 1047, which would have required companies training very large models to put a safety plan on record and face liability if they caused a catastrophe after behaving unreasonably; executives implied they wanted a federal bill instead, and Adler does not believe they would have supported a federal version of SB 1047.

He rejects cartoon motives that it is all about money or all about power. A better description, echoing former boss Miles Brundage, is that society needs practices it can verify even if it mistrusts particular executives. Adler thinks the labs are defecting in a coordination failure, papering it over with claims that continuous testing makes them safe enough, when the more honest line would be that the risk is large, they do not want to rush, and they cannot stop because everyone else will rush too. First movers pay a penalty: an AI company that depends on NVIDIA goodwill might believe in tighter U.S.-China chip export controls and still stay quiet, because a rival could carry that position to the supplier as a diplomatic attack.

Labenz asked how open OpenAI still is internally as public and private capabilities diverge. Adler called the trajectory pretty spooky, citing an Apollo Research report on internal deployment: the most powerful systems are likely to be used for sensitive work inside a lab before the public knows they exist. Access controls on weights always existed, but information about algorithms, capabilities, and training progress has become more need-to-know, and even on the AGI-readiness team he sometimes could not tell what was coming off the rack or what it would be able to do. If outsiders hear no anonymous objections, the more reliable prior is that most staff simply do not have access, not that the issue is absent.

In a lightning round, Adler said he has no special insight on the superalignment team's dissolution and would defer to Jan Leike's tweets, which felt raw and described several things getting worse at once. He does not think people inside OpenAI took conspiracy theories about Suchir Balaji's death seriously, but the unease of watching someone who had spoken up die is real, and after Adler tweeted about leaving and his fear of the future, some people advised him to declare publicly that he would never harm himself, a request he found unnecessary and a sad comment on the information environment. As an employee he worried more about unstable people reacting to a weighty product than about protesters chaining themselves to the office; most staff barely registered civil disobedience except as a security-team warning to use another door.

The cultural tell he keeps returning to is an onboarding slogan that flipped. Early on, new hires were told OpenAI was not just a research lab, it also had a product arm to fund the nonprofit mission. Mid-tenure, at a safety offsite of perhaps sixty or seventy people, a speaker opened by saying OpenAI was not just a product company, it was also a research lab, and Adler was stunned. He counted who in the room had been there before the business existed, before GPT-3 was deployed, a group that does not include him, and got to about four. Former colleague Richard Ngo's writing on political change, Adler added, lands on uncertainty about what other people believe, which matches a company where information tents make candor harder and old anonymous concern channels have disappeared with scale.

On the amicus brief, Adler stressed he speaks only for himself. OpenAI promised nonprofit control over a hugely significant for-profit entity, and people relied on that promise when deciding whether to join; he cannot see a reasonable price that would compensate for giving the control up, because extra resources for education or science do not replace accountability to humanity rather than shareholders. If the nonprofit were putting no constraints on the for-profit, removing control would not matter, and the fact that OpenAI sought to remove it is evidence that the constraint is real. Labenz noted the charter's merge-and-assist clause, with its example of a fifty-fifty chance of AGI in two years, already feels imminent; Adler agreed, adding that a merger still needs a willing counterpart, while wishing companies could cooperate when they all say the race will physically harm people. His advice to people still inside frontier labs is to raise a hand, draft the first version of a commitment, push the company to keep it, and copy Anthropic's practice of listing public promises on a single page so that what is on the page is a commitment and what is not, is not.