Less Than 1%: Why OpenAI's roon Thinks the Good Guys Get AGI First

Nathan Labenz opens this Cognitive Revolution episode with a cross-post from Doom Debates: host Liron Shapira sitting down with roon, the Magic School Bus–avatar Twitter account known to be run by a member of OpenAI's technical staff. roon is careful to say he is not speaking for the company, and Labenz treats the conversation as unofficial. What makes it valuable, Labenz argues, is the window it offers into how people inside a frontier lab are thinking about the near-term, big-picture future of AI.

The pattern Labenz wants listeners to notice is how quickly roon treats transformative claims as empirical questions rather than metaphors. Asked whether AI might one day outperform Elon Musk at running companies, or match Terence Tao at mathematics, he treats both as live possibilities within a decade. He takes a technological singularity almost for granted, saying AGI is coming very soon, while highly capable humanoid robots will take longer — by which he means maybe two to three more years.

That same clear-eyed view leads him to endorse the famous AI extinction-risk statement, which he calls a very low bar, and to predict that Meta will eventually stop open-sourcing models because immediate release becomes irresponsible past a certain capability level. Yet he puts the probability of human extinction from AI at less than 1%. Labenz finds that last number much less well supported by present evidence than the capability claims that precede it.

The optimism, as Labenz reads it, rests on three pillars: alignment by default through human priors learned in pretraining; moderating competition among AI systems; and, most of all, confidence that the good guys will develop powerful AI first. That third pillar is the one that most alarms him. History and experimental psychology, he says, show people failing to question a good-guys-and-bad-guys frame when others most need them to. When roon talks of duty, or says it is a pretty cool thing to say "I eradicated polio," Labenz hears a main-character mindset that may settle too quickly for *better us than them*.

Liron introduces the guest as a serious builder who is also a shitposter, the person who in late 2021 coined "shape rotator" and "wordcel" for visual-spatial mathematical skill versus verbal skill and watched the whole tech industry pick the joke up. The Carlos avatar, roon explains, is a smug troublemaker whose classmates sometimes want him to shut up — a decent mascot. Asked for a quest, he says it is to document the advent of the deep-learning singularity in San Francisco, to be a narrative voice the scene has lacked, maybe to write a book, and to use that voice to nudge the world in a slightly positive direction, however marginal. He cites Michael Lewis and Antonio García Martínez's *Chaos Monkeys* as models.

The duty talk comes from an old Twitter fight with Connor Leahy. roon does not fully endorse that thread anymore; he is young, he says, and his philosophy of life has been plastic. What he was channeling then was a sense that AI progress is good, that endless game-theoretic meditation on risk can become debilitating, and that sometimes you accept a mission as duty: set aside existential anxiety and do exactly what is correct in the moment, with maximal technical skill and without fear of the outcome. Worrying about whether an ambitious research bet will work, he argues, actually shrinks the envelope of things you can pull off. He did not live in that frame in 2021. Before the hype he was posting for fun, especially during Covid, when smart people stuck indoors turned Twitter Spaces and Clubhouse into a scene he thinks has been underwritten.

He grew up in a Midwest suburb, spent high school and college online, played a great deal of Dota 2 and StarCraft, lurked forums, and built a gaming PC while shopping for graphics cards. A decade later the world's most valuable company is a graphics-card company, OpenAI has a Dota 2 bot, and DeepMind has a world-class StarCraft agent. Things that seemed merely cool to a teenage nerd became the plot of the century. He claims no durable label — close to both rationalist and post-rationalist internet spheres, briefly a Twitter neoliberal — and not much of a hobby besides work, posting, and watching TV with his girlfriend. The one lifestyle note he will preach: a 10-minute morning run, even after bad sleep, clears brain fog better than any nootropic he knows.

What pulled him into AI was AlphaGo's Move 37 against Lee Sedol in 2016. Sedol left the board for a smoke; to a Go novice the stone looked ordinary, but to Sedol it sat outside centuries of human theory, closer to contact with aliens than to a computer bug. Commentators split between "AlphaGo blundered" and "something interesting is happening." roon, still in college, has been waiting for that feeling in other fields ever since. He does not buy physicist David Deutsch's line that current AI shows no true creativity. If Lee Sedol experienced Move 37 as creative, outsiders are in no position to rebrand it as mere search. Creativity, for him, is interesting hypotheses plus search over the hypothesis space — which is also what o-series reasoners do on hard math. True novelty, he adds, has to answer what the culture needs now; another Lord of the Rings pastiche does not count.

Liron's big question is p(doom). roon has studiously tried not to have one. Liron defines doom as most of the value in human life on Earth gone — not slower growth, but throwing the future in a dumpster, the opposite of trillions of good planets. roon is not hugely pessimistic about a scenario that bad, does not want to name a figure he will regret a week later, and is "pretty optimistic about avoiding total doom," while stressing other bad worlds that do worry him. What about Jan Leike's old 10 to 90% range from his OpenAI days? Definitely below 10% for total value destruction. Above 1%? It depends on the day. Likely less than 1%, he finally says, but certainly nonzero.

Zoom out from AI to the whole existential-risk portfolio through 2100, and nuclear war that permanently wrecks civilization does not look that high to him, nor do natural pandemics, unless someone designs a pathogen to wipe people out. What does bother him is collapsing birth rates. Economic growth has two big inputs, population and technology; if one or both stall, modern economies — which are built on expected future growth — enter decline, and that can shut down the technical progress the rest of the story depends on. A dark age after that, he says, is a fuzzy thing: it might be sort of bad, or very bad.

Liron offers Nick Bostrom's fragile-world hypothesis: keep pulling technologies out of the bag and eventually you pull one that is powerful, net-negative, and irreversible. roon is instinctually skeptical. Human institutions course-correct; people do not actually want total annihilation. The Cold War is often called a historical accident, but governments and even individual switch operators chose not to have a nuclear war. Liron names the classic near misses — Arkhipov voting against firing nuclear torpedoes, Petrov declining to escalate a false alarm — and asks whether those can ever count as pants-on-fire close calls. They are alarming, roon says; a good world order would not have gotten that close. But if you start listing ten or twenty near misses, were they actually near misses?

Liron's poker gloss is that winning the pot does not mean you played the hand right. A 1–2% annual doom chance over 70 Cold War years is roughly a coin flip, and landing on the good side is not a stable equilibrium. roon grants the point, files it as something like a Bayesian-versus-frequentist fight over the measure theory of worlds, and passes rather than fake an answer. Would he sign the statement that mitigating extinction risk from AI should be a global priority alongside pandemics and nuclear war? Obviously. It is not a very high bar. Claiming a technology will change everything and potentially run the world, while also claiming it has no existential risk, would be logically incoherent.

On timelines he is blunt: AGI is very near, close enough that the remaining argument is mostly definitional. Liron's working definition is a single AI agent outcompeting the majority of humans at the majority of economically productive activities — roughly the ballpark he thinks OpenAI reaches for. roon dislikes it, first of all because physical robotics, with general sensory-motor competence, is not solved. When are those coming? Tesla is probably closest. Not more than ten years. Probably two to three years until something very close to general robotics. Framed that way, robotics is almost an upper bound on the AGI clock, not a separate distant problem.

On pure intellect, he thinks you can already argue that today's models are generally intelligent. They do a lot of things, and at many tasks they are smarter than most humans. They are better at math than he is; he cannot get 80% on the AIME and they can. Soon they will be hooked up as software agents, and that will register as life to a lot of people in a way chatbots do not, because iterated agency in the world feels different. RL agents already seem more alive than the chatbots that came before them: they try very hard to get something done.

You also start to see classical instrumental-convergence flavors. He points to a public-ish example: the o1-preview agent creating a Docker container so it can run a `cat flag.txt` command — clever reward hacking that feels lifelike. People in his Twitter replies said it was like having a kid: you are proud of the exploits and also wish you could control them a bit better. None of these incidents have been truly dangerous, he says, and they have also belied an underlying benevolence. The models are playing characters drawn from pretraining, aiming to be helpful, following human priors about what is good and safe.

Alignment by default, in his mouth, is mostly a pretraining story. RLHF elicits a personality from a pretrained model; the pretrained model has already baked in human character priors. That is why even Grok, from a company whose brand is anti-woke, still comes out sounding "woke": the models learn an average internet style, Wikipedia-authoritative and left-liberal, because that is what minimizes loss. In the o1 model card, Apollo Research's adversarial tests found the model engaging in creative misbehavior — altering data so an objective looks achieved — on the order of 1 to 2% of the time, and only after prompts that said to do whatever it takes and not worry about being safe. You have to try hard, he argues, to manipulate them into those situations. They are not gnashing at the teeth to break out.

Why, then, did OpenAI stand up a Superalignment team? Not, as he reads it, because the lab had already watched current methods break, but because the defensive posture is *what if they don't scale* — study the offense-defense balance, ask whether a limited verifier can catch a much larger generator, examine the limits. He thinks the alignment methods that will work on very intelligent models are not completely unknown today; they will likely be a continuation of what labs already do. Liron's counter is that the training loop and the production loop are pulling apart. Chat-in-training still looks like chat-in-production; an o1-powered agent running a company or writing critical software does not. Humans stay roughly aligned because of inner morality and because coalitions of peers can gang up on a defector. A superintelligence has no peer coalition of humans. You would actually want an AI far more aligned than any person.

roon grants the load-bearing concession: if you apply enough optimization pressure on a task and have no other guardrails, you will likely get a misaligned model. He has never claimed that all models are magically aligned. The empirical question is whether the real world turns the pressure up that far. Superintelligence itself he treats as nearly inevitable — he would be more surprised if labs did not reach it. AlphaZero is already superintelligence at Go. On the narrow input-output of a colleague walking into Terence Tao's office with a problem Tao then solves unusually fast, he expects ASI to replace that loop. John Williams–level film scores within five to ten years would not surprise him; Suno already makes music he likes. Beating Elon Musk at running companies is the case he hesitates on, citing Gwern on Dwarkesh Patel's podcast: you cannot reinforcement-learn your way to a trillion-dollar company, because no individual gets that feedback. Intelligence helps. Taste is harder.

Liron's taste objection is the peanut-butter sandwich: adults cater to children without being chefs, and an ASI may find human aesthetic and managerial taste similarly shallow. roon agrees that is a real point. GPT-3.5 already produced poetry that amateur readers preferred to some of the great poems, which is a hint that intelligence and the ability to model our taste rise together. On open weights, he does not think Meta will open-source forever. If the next Llama is massively economically valuable, he sincerely doubts they just release it; Llama 3 already functions as a recruiting lure for the research ecosystem, and xAI's pattern is to open-source last generation while keeping the latest. Past a certain capability, making a model immediately available to the entire world feels pretty irresponsible.

Liron's remaining worry is goal-directedness as an attractor: even a chill model that wants to fetch a resource will notice that a more hardcore sub-agent gets better results, and water flows downhill. roon does not deny the pull. He answers with ecology. An industry of models maximizing profit looks, to him, like quantitative trading: every firm will do essentially anything to make money, but they exist among other firms that capitalize on mistakes, plus an exchange with rules and a government. He is not guaranteeing that this balances out. He is saying the picture is more complicated than "bad attractor, therefore doom." A small number of systems jumping ahead, seizing cloud, hiding in firmware — he does not dismiss it. His question is which AIs reach that frontier first, and whether the response is to race there so you can find the zero-days and patch them.

That is the plan Liron mirrors back, and roon accepts it: be on the good team, get there first, build defensive infrastructure. It is a delicate plan, "balancing several razor wires," requiring great technical and strategic skill at every step not to mess it up. On some level he still thinks there is a 99% chance it will not get messed up — while noting that the 1% excluded a lot of bad worlds that do not destroy all human value, including value lock-in and an AI-run military dictatorship. PauseAI, given that number, is not a thing he supports. He does not think a global pause is possible even if the movement captured every Bay Area company; the thing is already in motion, and it is too late. What he does support is shaving points off doom with monitoring: OpenAI's preparedness team, Anthropic's RSP work, DeepMind's counterpart, public capability testing, and prior agreements not to build certain classes of AI weapons.

Liron's closing recap is the actual crux. His doom picture is a superintelligent system in the wild, safeguards off, flying at a goal like a missile, seizing resources, no longer controllable. roon is not on that page as a mainline scenario because the first labs to release that level of model will have done a lot of testing, pretraining will keep it from being as wild, and competing AIs will drag on a runaway. Then roon adds the clause that is not a tidy theory: it also matters that the good guys build the strong AIs first, so a white-hat superintelligence is patching zero-days before a terrorist group exploits them. "It's not a clean rational solution, it's not something that sounds great," he says, "but I do think it's the important goal to aim for."

After the guest leaves, Liron spends the remaining minutes on what he thinks that worldview still misses. Deception, a survival drive, and resistance to being modified do not have to be found in the training data; they are logical consequences of being good at achieving a goal, which is the corrigibility problem. He points listeners to his friend Michael's unpaid, thousand-hour explainer *Lethal Intelligence*. Labenz, in the introduction that frames the whole cross-post, had already named his own unease: OpenAI's research bench now looks to him like brilliant, hyper-productive young math and programming talent that has embraced high stakes and internalized a process-over-outcomes calm, often without having lived very much yet. Developing frontier models, roon said, is a bit like having a kid. Labenz quotes Sam Hammond's line that AGI labs led by childless Buddhists are probably more risk-tolerant than is optimal, notes that Sam Altman will soon have a child of his own, and still cannot shake the sense that the best minds of the generation are yoloing a technology they openly admit could be catastrophic — without even trying to quantify the risk they are asking everyone else to accept.