They Asked for a Fraction of the 20 Percent and Still Got No

This is a bonus episode of *The Cognitive Revolution*, recorded in a hurry. Host Nathan Labenz sits down with regular guest Zvi Mowshowitz, who tracks model releases, lab politics, and safety research in long weekly briefings. By the time they hit record, Jan Leike had posted the thread explaining why he left OpenAI. Ilya Sutskever, the company’s chief scientist and Leike’s co-lead on superalignment, had announced his own departure days earlier. Sam Altman’s reply was already out.

In July 2023 OpenAI had announced the Superalignment team in writing: 20 percent of the compute the company then had, and four years to solve alignment of superintelligence. Leike’s statement is plain. He had fundamental disagreements with leadership. He could not get the resources he needed, compute first among them. He had warm words for his teammates. He was, in effect, resigning in protest.

TechCrunch, Vox, and Bloomberg all filed around the same facts. The team asked for a fraction of the promised 20 percent and was repeatedly turned down. The group has been dissolved and its people scattered into product orgs. OpenAI says the work will continue. Zvi’s verdict is that having broken the commitment, dissolved the team, and lost the leadership, it does not look like the effort they promised. They are nine months into the four years. Labenz puts the question to Altman on the air: does the timeline still hold?

Labenz calls Altman’s response to Leike extremely graceful: yes, we have a lot of work to do, and we are going to do it, with a longer statement later. Zvi agrees it is basically the best possible thing you can say. Then you are on the hook for doing it. The letter contains no numbered promise. The exam is whatever happens next.

Kelsey Piper at Vox confirmed the shape of the non-disparagement clauses. They last a lifetime. They include an NDA that you cannot reveal you are under the NDA. When employees first join, compensation is equity-heavy, and they are not told that if they refuse to sign on the way out, already-vested equity will be confiscated. Zvi says he is honestly confused as to why that is legal. Labenz’s bar is simpler: if you are going to confiscate something of immense value for not signing, you should have to acknowledge the rule very clearly at the start.

Forcing every employee never to disparage you for life, no matter what, is not a reasonable position if you want people not to assume the worst. Zvi notes that ordinary noncompetes are already unenforceable in California and that the FTC is trying to ban them for most workers nationwide. A lifetime muzzle tied to most of someone’s wealth is worse. If something is wrong with safety or culture at OpenAI, the public needs to know. A whistleblower clause in a bill is an ornament if talking costs millions in stock the company can simply take back, without even suing.

Philosophical fights can be talked around. Product and safety teams can, in theory, sit in different rooms. Compute cannot. Running anything in AI now means you need your chips. Leike said the denials had become a substantial barrier to the work. He also said the company should be spending vastly more — not a little more — on preparing for an AGI future, and that they are not ready for the next generation of models to be safe in a pedestrian, nuclear-plant-doesn’t-melt-down sense, before you even get to existential risk.

Twenty percent of currently available compute sounds like a lot. Over four years it is not, Zvi argues, because two years from now OpenAI is going to have ten times as much compute unless something very strange happens. In industries where safety is paramount, most of research and development cost *is* safety, and that is just to keep the plant from melting down. The superalignment ask was already a reasonably small, very modest commitment.

Labenz cannot make the business case for the refusal. If the team only needed something like 5 percent of compute, giving it would have moved the research maybe 5 percent faster. Letting Ilya and Leike walk, and letting this become a global story, over a couple of percentage points does not add up. Zvi’s read is that compute was the straw that broke the camel’s back. Bloomberg reported that Leike’s breaking point was Ilya departing. Ilya had been shut out of decisions and could no longer play the internal ambassador role he used to play. Each departure burns trust. Each remaining person has less of it.

After Ilya left, the internet asked what Ilya saw. After Leike, it asked what Jan saw. Zvi’s answer is unromantic. They saw a company that is not committed to safety, that will not put its money where its mouth is, that has a culture hostile to safety efforts, and that is pivoting toward the shiny new product. Making money is fine. The trouble is that the explicit mission is building smarter-than-human intelligence, and as Leike and Altman have both acknowledged, that is not a thing that goes well by accident.

Leike’s worry is not only the far superintelligence problem. He does not think they are on pace to have the tools for GPT-5 to be safe in ordinary, mundane-utility terms. Zvi points to OpenAI’s own reports: GPT-4o has much less tendency to refuse inappropriate requests, such as how to build a bomb, than GPT-4 Turbo or its rivals. He has not red-teamed it himself. Why would he? People who care already jailbroke it. Pliny broke GPT-4o in about two minutes during the announcement, because the first hunch just worked.

What is very clear, Zvi says, is that the 4o development process did not involve a robust attempt to make the model safe in a conventional sense. They mostly decided the capabilities were not so dangerous and were not going to give it much care. The launch style has moved in the same direction. ChatGPT shipped under a dry name, put in front of people to try, without a sales pitch. GPT-4 was similar. GPT-4o came with hype about amazing abilities, some of which were not even live, and left a lot of users unsure whether they were on the new model at all.

The day after Google I/O, Altman gloated on Twitter that his hype had a better vibe than Google’s, without addressing anything Google had actually announced or who was building the better product. Labenz felt the shift himself: last fall’s demo day already shipped GPTs that were not ready; this launch was messier, with ordinary users complaining they had not even been moved onto the new version. Early OpenAI said here is a cool thing, see what you do with it. Zvi’s phrase for the current mode is hype machine.

Until contrary evidence, he wants people to assume OpenAI is functionally a fully for-profit, grow-fast, move-fast-and-break-things startup run on that basis, with an internal culture hostile to worrying about safety. A cavalier GPT-5 might even be net positive if the model is not that dangerous. Pliny jailbroke all the major models. At some point that stops being acceptable. When GPT-5 lands you are probably not in an existential situation. How many nines do you put on that? If your answer is more than two, Zvi says, you are crazy. He would probably put one — about 90 percent.

With Leike and Ilya gone, the large safety and alignment brief rolls up to John Schulman, who had been responsible for post-training — making today’s models useful and well-behaved. Labenz flags Dwarkesh Patel’s interview with Schulman that week. Schulman expects AGI on a two-to-three-year timeframe. Asked if it could be next year, he said that would surprise him, but he was willing to co-sign two to three. Asked what happens when it arrives and how society absorbs it, he said at multiple points that he does not really have a great account, and that hopefully the leading developers can work together. Dwarkesh told him on the air that it did not sound like a super robust plan.

Relying on post-training to render AGI, or ASI, helpful and harmless is basically a pipe dream, Zvi says. Leike has explained, including on the 80,000 Hours podcast and in a talk a few feet from Zvi, why RLHF is not a solution to this problem. Zvi tried to talk him out of other proposed approaches and failed to make the case tightly enough; he thinks it was reasonable for Leike not to buy bold claims that clashed with his worldview. The point is the altitude. Leike was thinking about the problem on a different level than someone whose job is post-training.

Nobody knows how to solve these problems, so Zvi will not get that mad at people for being excited by ideas he thinks will not work. You try something, you learn something, you try again. Ilya is famous for throwing out everything he thinks he knows until the thing works. That is why Zvi had faith that Ilya and Leike, with resources and four years, might at least map some ways *not* to align an AI. The first way not working, in his model, is fine. The present problem is that they were not given enough even to try.

If Schulman walks in on day one and says we do not have a plan, we do not know how to align this, we do not know how society handles the transition from AGI to ASI even if we did, we need to figure it out from a beginner’s mind — that is the second-best answer a highly capable founder can give. The best answer is an actual plan. The trouble is that the work he has been doing and the problem he is now asked to solve are not the same problem.

Someone chained themselves to OpenAI’s fence this week. Labenz is only half joking about consumer protest, app-developer boycotts toward Claude, and putting more force behind California’s SB 1047, the bill from state senator Scott Wiener. When Zvi wrote the bill up he was mostly hunting for ways it was too strong. He found a real bug: the derivative-versus-non-derivative definition, which would let an upstream lab pass blame downstream and let both sides skirt the safety guarantees. That clause has to be fixed. The other notes were about clarifying overreach, not about gutting the core.

The core he accepts is this: you say what you are doing, and you are on the hook if you lied; if you discover catastrophic risk in the room, there has to be a way to shut the model down. The bill also has whistleblower language, which sits in bitter contrast with the gag clauses. Zvi’s further position is that AI companies should not be allowed to sign lifetime non-disparagement agreements, at least as they pertain to safety, and maybe at all. His libertarian instinct is that voiding contracts is bad. This, he says, is a reasonable place to consider it.

Someone has to check the work. Zvi has said the same about Anthropic’s responsible scaling policy and OpenAI’s preparedness framework: they are potentially very good *if* the spirit is honored, if people treat a technical pass that still feels wrong as a stop-and-investigate, not a checkbox. If the test-taker knows the questions and hardens only that attack surface, you are toast. There is no battery of tests that survives a lab teaching to the test. Unless you deeply trust the culture, third-party testing is required. After this week, he does not have that trust in OpenAI. If you have a choice of whose technology to use, choosing theirs is part of what you are doing. Anthropic’s culture of concern among employees starts to look like a product feature. There are no angels in the room. There are levels.

The one alignment idea he wishes more people were watching is Sophon, a technique proposed by a Chinese team, named for the particles in *The Three-Body Problem* that lock down Earth’s physics. The pitch is to trap an open model in a local maximum on specified topics so ordinary fine-tuning cannot teach it out — to make it dense about biology the way some people decide they cannot do math. It is the first proposal he has heard that might raise the cost of turning a general model to a specific end above epsilon. The limit is the list: you have to enumerate what it must not know, which helps against known threats and fails against whatever a smarter-than-you system invents. Sci-fi villains always ignore the weird thing that was not on the list. The correct move is to stop until you know what the weird thing is.

Labenz pitches a movie: *The Social Network* meets *The Lord of the Rings*, an Altman-like figure corrupted by the ring. Zvi would rather watch the version with no villains and no idiots, where every local decision is reasonable and control still slips. He likes the garden of forking paths: some branches of the tree are terminal. Asked about the simulation hypothesis, he is consistent. Even if there are 9,999 simulated copies of you, the correct strategy is to act as if this one is real. Ancestor sims of people who decided they were in a sim, and therefore stopped building a future, do not get run. You have to take the situation seriously.

Amanda Askell at Anthropic had tweeted that she does not think AI is definitely going to kill everyone, and that if she were a doomer she would be in the Caribbean instead of at work. Zvi wants the word back. A real doomer is the person who says there is nothing to be done, whose p(doom) is 0.9-repeating, whose decision does not matter. If you think we could lose, we could win, and you can help fight — an old idea of a universe hanging in the balance — you are not a doomer. Take the chance of victory from 34.007 percent to 34.008 percent and that is a great life. If Askell needs two weeks on a beach so she can think clearly, she should take them.

He is going to Madison Square Garden that night with old Magic friends, 40,000 Knicks fans watching Game 6 on a video screen because the team is in Indiana. Life is not all the fight. The people leaving OpenAI are top talent; someone will snap them up, or they will do something themselves. The next scene is Altman’s to write. Until then, the practical list is short. Watch the next announcements for numbers and names: how much compute actually cleared, how many safety hires. When you pick a model, treat safety culture as a criterion and put Claude in the comparison. Follow SB 1047, especially any ban on lifetime gags tied to equity. And do not confuse taking the problem seriously with deciding you are powerless.