He Walked Away From Millions in OpenAI Equity — Then Wrote Four Rules for AGI Labs

Nathan Labenz frames the episode as a conversation about AI forecasting and the oversight of AGI labs, recorded on the eve of a second Trump administration. Dean W. Ball is a returning guest, best known to the show's listeners as a critic of California's SB 1047, the frontier-model bill Governor Gavin Newsom vetoed in 2024. Daniel Kokotajlo joins a few months after a headline-making departure from OpenAI, where he had spent about two years on policy research and strategic planning around AGI safety. The hook is not that they like each other. It is that they published a joint op-ed in *Time* proposing four requirements for frontier developers.

Labenz calls Kokotajlo's exit an admirable move. OpenAI's paperwork included a secret non-disparagement clause, backed by the threat of taking away already-vested equity. Kokotajlo refused to sign, at a personal cost Labenz understood as millions of dollars and the majority of his financial position at the time. Once the story became public, OpenAI changed the policy: departing employees are no longer asked to trade silence for vested equity, and Kokotajlo's own equity was restored. The stand, Labenz says, catalyzed a real change and lends credibility to whatever he argues next.

The *Time* essay asks for four things. Frontier developers would disclose important new capabilities observed while training; publish the training goal, model spec, or other document that defines how they want systems to behave; and release safety cases and risk analyses for public scrutiny. A fourth measure would create whistleblower protections along the lines of what SB 1047 would have created, so insiders can raise alarms without fear of legal reprisal. Labenz finds the package compelling, still wants mandated third-party testing, and flags a broader point: AI policy remains relatively nonpartisan, and people of different persuasions can still hammer out concrete steps.

Kokotajlo says the proposal stands on its own, independent of his OpenAI years, but the biography is the motivation Labenz wants on the table. He worked on the policy research team, doing forecasting and scenario planning, writing strategy documents about dangerous-capability evals as a possible basis for regulation, and pushing alignment experiments on faithful chain of thought. He left on April 12, 2024. The reason, he says, is that he had gradually given up hope that the company would be the way it needs to be to handle AGI well.

The first layer is technical. The alignment problem is not solved. If a training run actually produced AGI or something similarly powerful, he argues, the lab would not understand what it had built well enough to control it. There are reasons to think the system would play along and bide its time rather than be actually aligned. A lot of work is needed on training methods and evals that can tell whether those methods are succeeding, and he thinks the industry is massively underinvesting in that work. That is one example, he says; there is more he could add.

The exit paperwork seemed unlike what a nonprofit focused on benefiting humanity would use. It included a secret non-disparagement clause backed by the threat of clawing back vested equity. He decided not to sign, which left him free to talk about the clause itself. To his surprise, the story blew up in the media; people were more upset than he had predicted. Under that pressure OpenAI changed its policies, he kept the equity after all, and the standard paperwork, as far as he can tell, no longer includes that non-disparagement structure.

Labenz repeats the financial stakes on air: millions of dollars, the majority of Kokotajlo's personal position. Walking away on the grounds that he needed to speak, Labenz says, is what makes the later policy argument land. Kokotajlo thanks his wife, who made the decision with him and took on as much of the risk, and the friends who advised him through it. The later whistleblower discussion will return to a piece of that advice: several people told him to sign and disparage the company anyway, because OpenAI would not dare sue. He refused, because for many people it actually matters whether speaking is legal.

Kokotajlo's reputation in the field still rests in large part on *What 2026 Looks Like*, a 2021 LessWrong and Alignment Forum post that walks year by year through a detailed future. Labenz reread it for the episode and says it is often cited for a reason. Most forecasting, Kokotajlo argues, extrapolates a trend or scores an event: a war this year, a benchmark that year. Writing a unified story is different. Forcing the pieces to fit exposes contradictions and raises questions you had not asked. He still thinks that methodology is useful, and he learned a lot from doing it.

On accuracy, he gives himself two hits. He predicted that chatbots would be a big deal before ChatGPT existed. He also took the scaling-laws papers and GPT-3 at face value: companies would keep investing, and the models would keep getting better. Turning that insight into a narrative is the part he thinks worked. His biggest miss was AI-powered censorship and propaganda. He fell for a classic fallacy of futurism: looking at what would be technically possible and assuming someone would actually do it. Today's models are smart enough to run the propaganda and internet-balkanization story he wrote for 2024 through 2026. The powers that be have not deployed it with the gusto he cynically expected.

Ball reads the miss as a diffusion story as much as a capability story. Kokotajlo was writing in 2021, coming out of COVID, the election, and the unrest of 2020, and it is easy to over-index on the media environment of a specific moment. Institutional change, and the way senior people in companies and governments think, just takes time. Ball's broader thesis is that both AI hype and some AI-risk claims are overstated not because the capabilities are fake, but because diffusion is slower than you think — and even when you correct for that, it is still a little slower. Labenz pushes back on the understatement: the 2024 section, written three years earlier, got the lived texture of following AI "dead right" in many details. That record, he says, is why he cannot easily discount Kokotajlo on catastrophic scenarios.

Kokotajlo wants other people to write the same kind of story: pick the future that seems most plausible, write it in detail, and hand someone a short way to understand your worldview. His regret is that he did not have the guts to finish 2027, the draft in which AGI arrives and things start going off the rails. Too many unanswered questions, so he published through 2026 and abandoned the rest. That unfinished piece is now the job. With Eli Lifland, Thomas Larsen, and others, he is building a larger version under the AI Futures Project that runs all the way through AGI, if only so readers can say, *oh, Daniel is worried about something like this.*

Labenz is struck by how little AI propaganda we have actually seen, even as video generators get good enough to swap one person for another. Kokotajlo separates propaganda from censorship. The classic case is Russia running bot farms of fake American Facebook accounts, which it was doing in 2021 and is presumably still doing, now with LLMs in the loop because it would be crazy not to. It does not seem to have substantially changed the election cycle. The lesson, as he reads it, is that even though the marginal cost of propaganda has fallen, overall effectiveness has not risen much. He is unsure whether that is incompetence or a Red Queen race with platforms getting better at stamping it out.

Ball's intuition is that the information environment is more robust than people think. Sources still matter; a random tweet of Joe Biden doing something outrageous does not automatically become believed. Platforms are also fighting it. He sees plenty of LLM-generated replies on his own feed, often from the UAE, cheering that California will regulate AI into the ground while the Emirates will not. The volume is real; the impact is doubtful. A fake image designed to create a scandal, Kokotajlo adds, has a natural cycle: one day of *look at this horrible thing*, then a week of *look at this horrible fake*, which helps the politician. That is one reason he is less worried about deepfake images, except perhaps on election day itself, which has not happened yet.

Ball offers a darker counterexample from Eastern Europe. About a year before the taping, deepfake audio of a candidate dropped over the weekend before an election. The country had a 48-hour media blackout meant to block last-minute propaganda. The propaganda arrived anyway, and newsrooms could not report on it because the law forbade them to. Causality is hard to prove; the deepfaked candidate lost. Ball files it as well-intentioned public policy that bit its authors. The under-discussed reverse, he says, is already here: a politician hit by a legitimate leak calls it a deepfake. Trump's campaign has intimated that, Biden has done it, and Ball expects criminal defense to follow. Courts will have to adjudicate authenticity, and nobody knows how.

Censorship, Kokotajlo still thinks, is the more important half of the coin. Recommenders and takedown systems already exist; LLMs make it cheaper to push them further along a spectrum from slur filters toward ideology enforcement, because a model actually understands "conservative" or "liberal" content. In 2021 he predicted propaganda and censorship would feed each other, and that censorship would win: American bots guarding the American internet, crushing Russian astroturf rather than letting it run amok. That still looks like a plausible description of today. He also predicted a balkanized internet — Chinese, Russian, Western left, Western right. Ball sees the split arriving through policy: Europe preparing to mandate censorship, Florida and Texas laws that tried to impose Republican content rules and lost at the Supreme Court, and blue-state algorithmic-design bills. The United States is in a golden age of First Amendment jurisprudence, he says, but geographic fracture among Europe, China, and the U.S. still looks like the path.

There is nothing special about 2027, Kokotajlo says; it is just the middle of his distribution. His definition of AGI is as good as the best humans at every cognitive task. The qualitative picture starts with Anthropic's first computer-using agent, released a few weeks before the taping: in principle it can move, click, and type anything a person can do on a computer; in practice it gets stuck and does not notice its own mistakes. Imagine that, except it actually works across the board. Getting there is a mix of continued scaling — more parameters, longer training runs — and more targeted "agency training," reinforcement learning in computer-use and chain-of-thought environments. He thinks o1 is an instance of the second path. Internally, he bets the labs have a roadmap to a actually-good computer agent within twelve months; realistically it takes longer. Extrapolating current benchmarks, some go superhuman this year, some next year, and around 2027 it feels to him as if all of them saturate.

Ball's update since his August appearance is not that his timelines shortened, but that he now trusts them enough to guide public policy. Between Anthropic's computer use and OpenAI's o1, "you can just start to see the future." He tried the agent on macOS Sequoia's iPhone screen-mirroring window, a phone UI the model had been trained on desktop interfaces to handle, and it figured it out. Making the rest work looks to him more like an engineering and geopolitical problem than a science problem. In four years, he wants to open the laptop in front of him and instantiate 10,000 expert-level agents that use computers, write code, and are cheap. He does not care whether that counts as AGI. In an important sense, he says, superintelligence will arrive before AGI.

Kokotajlo's automated-R&D picture is staged. Models already write first drafts; humans still manage the experiment. Then the AIs write all the code and humans become managers of a team. Then humans sit in the backseat, reading white papers the AIs wrote, saying the work sounds good because past experience says intervening would only mess it up. When systems can do 100 percent of the tasks in the AI R&D loop, the pace of progress jumps. A single year that matches the last ten years of qualitative and quantitative leap is, he says, a conservative estimate; it could be ten times larger still. Eureka-style paradigm jumps, if they come, belong in the superhuman regime, not the human-level one.

Ball throws cold water on the compute side. Automated research will first eat the thousands of engineering tricks that take you from GPT-4 to GPT-4o mini, compressing 18 months into one or two, then hit diminishing returns. Efficiency gains will raise demand for compute, not lower it. Sam Altman's figure of $7 to $10 trillion in infrastructure sounds about right to him, and it is not obvious America will have that compute on its own terms in the late 2020s. Kokotajlo agrees compute will bottleneck the *rate*, but thinks superintelligence can still be built with clusters already assembled, by the end of the decade. After that, superintelligences will hoard compute the way industrial society hoards energy. Ball's civilian picture is every American plumber hiring 100 Harvard-trained microeconomists. Kokotajlo's reply is the turn toward the dark: politically, you may not be allowed to.

Kokotajlo does not buy Ball's laptop full of agents as a political forecast. The agents may exist in four years and still be locked inside a government-corporate-military AGI complex that does not sell access, because it already has money — in fact it prints the money. What it wants is physical labor: welding at the construction site so a robot factory can go up in a special economic zone. Once you have that level of capability, he wants people to think about who controls it. The default is not standard capitalism with competing APIs.

Ball has been writing from that nightmare for two or three years: superintelligence exists, and the CIA, Jane Street, and Moderna have it, and you do not. His joke, which he repeats, is that the best use of his time in that world is janitorial work while wearing an Apple Vision Pro, generating training data for a robotic foundation model. That is why he has fought so hard for open source. He knows open-sourcing can create other risks; he thinks those risks are worth it if the alternative is the world Kokotajlo is describing. Kokotajlo used to be quite against open-source AGI and has warmed to it recently, for the same concentration-of-power reasons.

The shape of the technology, as Kokotajlo describes it, is the problem. You spend billions on a giant training run and get an artificial mind that can be cheaply copied. Whoever owns the cluster will not want copies everywhere, because then there is no way to get the money back. The single point of failure is inherent: one node, one data center, probably copies of the same frontier model unless the people in charge decide otherwise. An open-source mandate would look different — many clusters, many models, less concentration. The default trajectory, he thinks, is a public-private partnership, a mega-corporation fused with the executive branch, a handful of people at the top of a hierarchy that controls the compute on which a civilization of superintelligences runs.

History's totalitarian dictatorships still had armies embedded in society, with friction and less-than-absolute control. If alignment is solved, the leader's control of an AGI army would be more absolute than any dictator has ever had. Ball agrees directionally. He wonders whether distillation changes the economics: a thousand GPT-5s versus ten million sub-billion-parameter models plus a vector database or the web. Small becoming the competitive vector might spread access; he does not know which way it goes. What kind of access matters too. OpenAI's published model spec, to its credit, states a chain of command: the organization's instructions first, the developer second, the user third. There is no law, and barely an industry norm, requiring those layers to be public. The Gemini incident — a system prompt pushing racially diverse images users had not asked for — is treated as a scandal and is also standard practice. Fast-forward to AGI, and everyone is talking to a system whose true agenda is known only to the people in the room who set it.

Ball's high-level pitch is that the next transitional years will be extremely hard to govern with hard rules. He is skeptical of the whole "guardrails" concept. What you *can* do is shape the information environment, which then shapes incentives among policymakers, among labs, and inside each lab's management-labor relationship. If that ecosystem is healthy, the rest of the incentives get better. They floated four ideas; he is not sure you need all of them. One, he says, is essential no matter what.

The first pillar is a mechanism for companies to tell the public and the U.S. government when a capabilities threshold is crossed — Anthropic's jump from ASL-2 to ASL-3, for example — without having to explain how. An attendant requirement is publishing a responsible scaling plan, which most labs have done and not all. The second is publication of model specs: how the company wants models to behave, and, when principles collide, which one wins. In the long run Ball hopes a spec becomes both a governance tool and a user-interface tool, something people can customize within parameters. He is careful about intellectual property with national-security and competitiveness salience; he does not want everything inside OpenAI in the newspaper.

Safety cases, which he skipped and then returned to, are an old idea in other fields and a new one in AI. Ball treats a safety *plan* as the company's governing document and a safety *case* as the argument that a specific model is consistent with it. Kokotajlo wants something laxer than the quantitative risk models other industries demand, and has been accused of "safety washing" for using the phrase. His version is a document that explains why the bad things other people worry about will not happen. For GPT-4, a paragraph would do: the model is too dumb. As systems get smarter, the document should grow into pages of experiments, including the worry that a model is only pretending to be aligned. He wants the cafeteria arguments he used to have at OpenAI — RLHF might produce a system optimizing the reward rather than being honest — to become a public conversation academia can join.

The fourth pillar, whistleblower protection, is how you enforce any of this without standing up a government audit machine. If a company publishes a safety plan and then does not follow it, an employee can go to the government, substantiate the claim, and let the state take it from there. Kokotajlo wants much more than Ball does — regulations that take decision-making power away from companies, including bills like SB 1047, which he supported. If they can mandate only one thing, he picks the transparency package, because an informed society can adapt and later impose weightier rules. Red flags are already in public. OpenAI's o1 system card notes that about 1% of the time the model hallucinates a link, seems aware it is a hallucination, and gives it anyway. The closed-door fix is to throw those cases into the training data until the red flags vanish. That might work. It might also train the model to hide better.

Labenz asks Kokotajlo to characterize why he lacks confidence in OpenAI around AGI. The high-level answer is that incentives cause people to rationalize their way to the conclusions they are paid to reach. That is not a dump on OpenAI in particular, he says; it applies across the industry. A company in an existential race to beat others to market has to tell itself that racing is the most important thing and that racing is good. He does not trust those people's judgment on whether to slow down, or on whether the latest internal system being used to automate R&D is safe. Those are technical questions with thin evidence and cost-benefit calls that weigh risks to all humanity. The people inside will be biased, systematically taking too much risk.

Ball does not share the obsession. He spent his childhood watching Apple under Steve Jobs, when doom was always around the corner and the company was supposedly just an artist with nothing defensible. Apple is now a three-trillion-dollar firm. OpenAI looks to him like a young company put on a rocket ship in the last two years, going through the tumult that growth implies. No company grows that fast gracefully. DeepMind, OpenAI, Meta, and Anthropic are walking an incredibly tight balancing act; if any one constituency were entirely happy, they would not be succeeding at it. He still does not think his hunch about OpenAI's culture should be dispositive, which is why he wants transparency rather than a personality test of Sam Altman.

The deeper split is about what kind of problem alignment is. Ball's intuition has always been that it is a muddle-through problem: hammer it, put some paint on it, mostly get there. Kokotajlo has made the most compelling case that treating it that way will go badly. Using SpaceX as an analogy he loves, Kokotajlo distinguishes Starship test flights — clear the blast zone, iterate — from crewed launches that must not fail. Fail-fast is the right approach up to AGI. GPT-4 hallucinating a link you know is fake is useful science. Once a system is smart enough to be put in charge of data centers, handed to the president for cyber operations or cancer or military advice, a single mistake can snowball into existential consequences. He does not see companies changing methodology for that regime. It is an industry problem, not an OpenAI-only one.

Effort is the other axis. Superalignment was a positive update: 15 to 30 people looking a couple of years ahead, with 20% of compute pledged. He wanted those numbers to rise as AGI approached. The open secret, he says, is that there is no plan. The closest OpenAI has to a technical story for aligning AGI is Appendix G of the weak-to-strong generalization paper, a two- or three-page toy design in which a pretrained model is turned into a reward model that then trains a general agent not to lie. Almost everyone he talked to thinks it probably will not work; more research is needed. At least it is written down. He wants multiple official plans, published, so academia can attack the assumptions. The alternative is a small group pulling an all-nighter to meet a deadline, no peer review, leadership moving fast, and the future depending on there being no flaw. Labenz's unease with that paper was the free parameter: the best results came from making the strong student more willing to override the weak teacher, and in the real system, humans are the weak teacher.

Labenz red-teams the package. Eighteen months earlier Altman had said not to base strategy on what China might do; a few months before the taping the line had become "our AI or their AI, there is no third way." Miles Brundage left and said the opposite. If leadership is ideological and willing to take more risk than the public would bear, whistleblower protection might just incentivize a purge. Ball pushes on the China pivot first. Wanting labs regulated *means* wanting OpenAI in Signal groups with Commerce, Congress, and the national-security apparatus. There is no world in which the U.S. government deeply regulates OpenAI and Sam Altman says we can be friends with the Chinese. That tradeoff — biasing companies toward American power projection — has always been his concern about getting into bed with government. He notes, dryly, that he works at a Washington think tank and is part of the same system.

On purges, Ball is making a statistical bet. Maybe 250 people really matter, a heavy-tailed universe. He is betting that many of those 250 are high-integrity, that the companies cannot do without them, and that they would blow the whistle if they had to. It is a wager. Kokotajlo agrees the protections are not a total solution: everyone who would talk might already be pushed out of the important rooms. He still thinks they are worth advocating. What he wants is not guaranteed employment. It is an anonymous hotline to a body with technical expertise, such as the AI Safety Institute. Around AGI there will be a dozen calls like: the latest alignment technique failed, the model is clearly dishonest in some cases, we are using it to automate most of our research, stopping means falling behind, someone wants to negative-reward the dishonest traces and proceed. A few conflicted people behind closed doors should not make that call. A second job for the channel is holding companies to their own word — OpenAI's deployment safety board episode, in his telling, showed a procedure they set up and then violated.

The legal detail matters because of how he actually behaved. Protections should cover private reports to the Justice Department or the AI Safety Institute, not public dumps of intellectual property, and they should be protections from after-the-fact lawsuits. When he blew the whistle on the equity clause, people told him to sign and disparage the company anyway, because OpenAI would not dare sue. For him, and for a lot of other people, it mattered that he was legally in the clear — both from fear of being sued and from the principle of being someone who obeys the law. Companies can still fire you. The important question is whether you are breaking the law.

Labenz worries about other counter-moves: tighter internal compartmentalization, a company that drops its constitution and just maximizes thumbs-ups, research practices chosen because they look good on a public document. Ball returns to a line from the teaser: there is a Shakespearean relationship between the intention of public policy and what actually happens; any rule you create is very likely to make the thing you are trying to fix worse in some important way. Compartmentalization is already happening. Paul Nakasone, a former NSA official, joined OpenAI's board; the company hired a well-regarded chief information security officer, a role that does not exist to facilitate multi-stakeholder collaboration. Public technical documents at least let ideas from outside the San Francisco bubble get in. He does not worry much that disclosure will distort research, because alignment is not a cost center — "alignment is capabilities," central to product strategy. Kokotajlo mostly agrees so far, at current investment of under 10% of compute and labor, fail-fast style. He expects the conflict later: a faithful chain-of-thought technique that would keep systems inspectable but cost three times the compute of dropping the property. That is when responsibility starts to mean slowdown.

Labenz puts his cards down: given the incentives, self-policing looks like a bad bet, so why not require an outside party — a government agency or accredited third parties — to look at the training process? Ball does not simply say no. Self-governance, as he means it, is closer to private governance than to companies marking their own homework. Insurance did not exist in Adam Smith, then evolved as the institutional technology modern capitalism needed for cargo risk that mixed geopolitics with specialized knowledge. He thinks AI needs an analogous ecosystem of evaluation institutions, for catastrophic risks and for whether a doctor or a lawyer can trust the model. We use evals for about 2% of what we could. METR and Haize Labs are early examples; the U.S. AI Safety Institute has a role. He expects to publish more on that design. Direct government testing, in his caricature, produces an AGI that will not say a slur and might still kill everyone, because government will not focus on the serious risks.

Kokotajlo agrees that government has botched many things and is only advocating third-party eyes for lack of a better option. He would love to be shown one. Asked what could change his outlook in a major way, he points to distributed training: a breakthrough there would force him to reevaluate the next couple of years. Every big release is Christmas for the Futures Project team, and the closed-ness of the labs is what makes forecasting hard. If he knew exactly what Anthropic did to make computer use work, or what OpenAI did with o1 — environment, algorithm, duration — he could extrapolate 100 times the compute in a richer environment and actually score the benchmarks. Instead he scrapes what he can from launch days. Labenz closes by thanking them for hashing out a zone of agreement, a model he wants more of the discourse to copy.