Nathan Labenz's p(doom) Is 10 to 90 Percent. He Is More Optimistic Anyway

This episode of The Cognitive Revolution is a cross-post from The Intelligence Horizon, a podcast hosted by Owen Zhang and Will Sanok Dufallo, then about to graduate from Yale. Labenz, who founded the generative-video company Waymark, spent six years running it, served on OpenAI's GPT-4 red team, and has interviewed hundreds of researchers and policymakers, tells listeners up front that he was impressed by how well the two hosts steelmanned the other side.

One of the strangest facts in the present, he says, is that disagreement among plugged-in, informed, smart people has barely shrunk even as the last few years dumped a huge amount of information about AI's trajectory. The one thing almost everyone now shares is that the timeline came in. Say you do not expect AGI until 2035 and, in today's world, you sound like a bear. Five years earlier that same date was considered quite aggressive. Most people then said 2050, or maybe not in my lifetime.

Capability has jumped and the calendar has compressed. On the fundamental question of what happens next, Labenz still sees total disagreement. He wants that epistemic humility on the table before he calls it as he sees it: the singularity is near, interpretability is showing richer world models, and with reinforcement learning scaling, the systems are no longer merely imitating humans.

Zhang's first question is the easy one: are we on the cusp of AGI? Labenz refuses the word a privileged definition. People talk past each other with it. What he does think we are clearly on the cusp of is powerful AI that is better than the vast majority of people, though perhaps not a handful of top experts in a given domain, at pretty much all cognitive work. That, he says, is enough to transform the economy and daily life, and potentially bigger things still, including the nature and status of the human species.

Humans will probably keep an edge in some niches. Systems are jagged: astonishing at some tasks, strangely weak at others, and still easier to trick than people. Expect weirdness, he tells the hosts. None of that answers the question most listeners actually mean. Will this fizzle before it becomes a big deal, or will it alter the world? He is confidently, clearly in the second camp. The debate that remains is about details, not whether the overall trajectory leads somewhere transformative.

He would not be shocked by further unlocks that let AI undeniably surpass humans, the way humans once blew away every other animal mind. He does not think that is clearly going to happen in the next few years. What he does think is clear is a coming argument over a new social contract. Quibbling over whether an AI will ever match the best human sommelier, in a world with almost no taste-AI and little motive to train one, distracts from the big question. Is this going to change just about everything that matters? His answer is yes.

Asked whether scaling RL will get us to that transformative AI, Labenz thinks it probably will. Even if it does not, we may never quite answer the question, because he expects new conceptual advances. One of his sayings is that everything is going exponential: researchers, papers, experiments, compute, datasets, RL environments. Two or three years earlier people asked whether pre-training could be scaled all the way to AGI as data ran out. Nobody asks that anymore, because a new thing arrived. Fast-forward to 2028 and this conversation may look the same: a couple of new pieces came online, and asking whether February 2026 was exactly enough will have become beside the point.

He thinks RL is probably already enough for systems that can do most of the cognitive work in the economy. We are reasonably close, and nowhere close to done. Lab leaders still say pre-training is working; those scaling laws held. What happened was that the next pre-training step got expensive, and post-training RL offered a much bigger bang for the buck because labs had climbed far up the pre-training curve and barely started up the RL curve. Economic logic says you pour resources into the hotter path until marginal returns equalize, then you advance both, plus whatever mystery third thing appears.

Zhang steelmans the skeptic: RL is an old, maximally general principle, not a new discovery after LLMs. Maybe the next leap has no precedent. Labenz does not think that is how it plays out, and if it did it would mostly shift the timeline. Compute coming online is exponential. Elon Musk, he notes, can hand xAI the unending hard problems Tesla, SpaceX, and Neuralink already pay engineers to solve, with enough compute to keep trying. The unlock he actually expects next may be usability more than raw capability: models that can enter a new workplace, pick up the vibe, and become useful without a human assembling every scrap of context, the way instruction-following once turned GPT-3 prompting from a weird art into something ordinary people could use.

DeepSeek's R1 paper, from January 2025, gave Labenz his "aha moment." During RL on an already capable base model, previously rare higher-order behaviors came online. In the reasoning trace the model says, in effect, wait, this is an aha moment, I can come at this from a totally different direction. That pattern exists in pre-training data, but reinforcement learning has clearly brought it out. Domain skills still need training; meta-cognitive traits, coming at a problem from many angles, can transfer. RL as an organizational process definitely generalizes. The question is dialing in the reward. Math and programming move fastest because the signal is clean. Other domains take longer, but he thinks the industry is obviously making it work.

He has a first-person reason to believe that. His son was diagnosed with cancer in the preceding months, and at the time of recording seemed very much on track to be cured. Labenz says he probably would not have been on the podcast otherwise. Using the latest models intensively in that medical context, his judgment is blunt: they are on the level of attending physicians, they know more than the residents, and they can go step for step with the most senior doctors in the hospital.

OpenAI worked with more than 250 human doctors on training data and grading. The latest models now outperform those same doctors at evaluating AI outputs. He describes the process as spinning a heavy wheel. The first pushes barely move it: thousands of hours, hundreds of millions of dollars, hundreds of physicians. Past a threshold the flywheel turns, and the game changes. Those thresholds will arrive at different times in different domains. If there is any objective ground truth, or even a high level of professional agreement, he thinks you can get there. It just takes more time.

The hosts press on domains with no clear verifier: good writing, good therapy, and especially long-horizon autonomy, days or weeks without a human steering each step. Labenz's first beat is that the recent trajectory shows the problem being solved. The METR graph of how long a task models can complete is basically going vertical; the METR team is struggling to write tasks long enough to evaluate. Humans are not a clean baseline either. Organizations hire on less than a week's work and then argue about whether a month-long project went well.

One technique is rubric rewards. Musk likes the ground truth of whether the rocket flies and lands on its tail, but you cannot crash a million rockets to find the one that worked. In the OpenAI health work, HealthBench packs 49,000 evaluation criteria into tasks that are not scored zero or one. A question might be worth 25 points; the model gets 5, 10, or 15. That slope is enough to climb, and it seems to work in domains with professional consensus, the same way medicine already trains people with watch one, do one, teach one. Taste is different. There is no consensus on a good novel. Labenz expects taste-based communities to shape models for romance, anime, or hard sci-fi, fans scoring outputs for their own genre.

Zhang objects that "does the rocket fly" is categorically noisier than a chatbot giving medical advice: more signals, more people, more money, more agentic steps, a flywheel you may never get over. Labenz interrogates his own days. True multi-decade agency, the Elon who will spend thirty years making humanity multiplanetary, is rare. Most of the economy is people who show up, sleep, shut down, and reboot. If his integrated memory were worse, he could compensate with an end-of-day scratchpad: what I did, what failed, what is next. He thinks he could get pretty effective on a week, a month, even a quarter. When Claude Code crashes now, he kills the tab, opens another, says resume, and the model often picks up. Three months earlier it could not. He hopes, in a way, that we park there: a day of work for a couple of hundred dollars, without multi-decade planners we cannot control. He does not see a fundamental barrier on the horizon.

Yann LeCun's line is that next-token prediction is the wrong objective, and that real understanding needs world models or energy-based models instead. Labenz splits his answer. He agrees on research taste. The industry is running a depth-first search, jamming one architecture, even building chips that embody it. He would rather see a breadth-first search across designs. A human brain is not one simple module stacked on itself; nature's solution is more complicated, and robustness probably wants more than one kind of part. That is a wish about exploration, not a verdict on capability.

Where he thinks the LeCun school is simply behind is the empirical claim. Models are no longer trained only on next-token prediction. RL's signal is whether you got the right answer: a verifiable proof, a number, or one of HealthBench's tens of thousands of criteria. GRPO, group relative policy optimization, compares a bundle of attempts on the same problem and moves weights toward the ones that were right. People should update, he says. Next-token prediction is still in the mix. It is not the whole story.

He also thinks it is now clear that the systems have world models. A layer might be 4,000, 8,000, or 16,000 activations wide, far fewer slots than there are concepts, so meaning lives in superposition: point one alone means one thing, one and two another, one and three another. Sparse autoencoders unpack that dense mess into tens of millions of sparser features. Golden Gate Claude is the demonstration: find the Golden Gate Bridge direction among tens of millions of concepts, turn it up, and the model will not stop talking about the bridge. Older latent-space arithmetic still counts too. King minus man plus woman yields queen. That spatial coherence, he says, demolishes the idea that internals are noise. The world model may be alien. Humans reached 1900 without relativity because the model they had was good enough for the domain they lived in. Human-level is not the same as human-like. Understanding does not have to look like ours to be real.

If RL or some added paradigm can deliver transformative AI, the hosts ask, which input actually bottlenecks us: talent, hardware, capital, or energy? Labenz does not treat any of those as fundamental. They are cultural and political. The sun pours energy onto the planet; the fight is who is allowed to harvest it, where, with which permits, on what timeline. He recorded a whole episode with Andy Masley on the point. A frontier chip such as an H100, fully on, uses about as much energy as a microwave or an electric kettle. One query is on the order of running a microwave for a second. Most people use less energy with AI in a week than they spend reheating leftovers for two minutes.

Demand is ramping, and if capacity cannot be added there will be a bottleneck. China does not seem to have that problem. Labenz will not pin an exact number, but he says the Chinese economy is adding as much electricity as the entire American capacity in a relatively short period. The Gulf can put up a plant fast too. That, he gathered from economist Sam Hammond after a UAE trip, is a big reason Washington does deals with Saudi Arabia and the Emirates despite the awkwardness of asking those governments to project "American values." If the United States cannot build here, it can at least build there.

Chips are harder because they are extremely specialized. Asked for the most likely reason we would not get economy-transforming AI in the next few years, he would pick a major disruption to fabrication: scarce chips, training runs that cannot scale, or powerful systems without enough inference to automate what currently looks automatable. A move on Taiwan sits in that tail. The fabs are easily destroyed and not easily put back together. U.S. yields, from what he understands as a non-expert, have come online reasonably well, even a bit ahead of the gloomier schedule. His broader habit of mind is to worry about tails: weird ways the AI goes wrong, and rare shocks that would make him catastrophically wrong. Gradual bottlenecks are not the main show.

Net-net, after hundreds of conversations with safety people and builders, has alignment gotten harder or easier? Labenz loops back to the strangeness of non-convergence. Helen Toner, who leads CSET in Washington and previously sat on OpenAI's board, named the timeline half of it in a post: even long AGI timelines have gotten super short. CSET then ran a workshop on recursive self-improvement, a narrower question, and still found huge disagreement. Some people thought automating ML research would make researchers a bit more efficient. Others thought that the moment you have an AI that can do that work, you go from perhaps 10,000 frontier researchers globally to 10 million, running at thousands of tokens a second, and that cannot fail to matter.

The workshop's diagnosis, as Labenz recalls it, was competing conceptual paradigms that are good at taking in new information and explaining it away. Bottleneck or O-ring theory says you are only as good as your weakest link, so every new skill still leaves a remaining constraint. Jaggedness theory points at the speed of the fill-in: a year or two ago the models could not do basic arithmetic, and now they are solving previously unsolved math problems. Ajeya Cotra made a similar split on the 80,000 Hours podcast, between economists who always expect slow diffusion and people who think those models are outdated once you condition on current capabilities. Labenz thinks talking past each other on how powerful the AI in the hypothetical actually is explains some of the gap, not all of it. He has started some interviews by asking how AGI-pilled the guest is, because downstream confusion is expensive.

He is not in the always-another-bottleneck camp. He will leave open the possibility that it is right, and that a plateau around a quarter's worth of work would even be good news. What he will not do is be persuaded by people generally gesturing that there will always be another bottleneck, because if they are wrong we are in for a wild time. The worst mistake, he says, would be to decide the system will self-regulate. Human history is not a comforting guarantee. Small bands hunting large animals drove many species extinct, including close cousins, often by accident, with no master plan. Plot armor, the feeling that humans are the main character and cannot be written out, is not an argument.

When people ask for his p(doom), Labenz usually says 10 to 90 percent. A friend told him to argue less about the exact number and more about which way it can be pushed. One significant digit is all you get. He has, even so, gotten a little more optimistic over the last few years. He started reading Eliezer Yudkowsky in 2007 and took away a picture of a small, concentrated, hyper-rational optimizer that would be hard to teach human values, values that evolution encoded haphazardly over a timescale older than our species. Looking at current models, he thinks they already understand what we care about reasonably well. It is not crazy, he says, to claim that Claude is more ethical, and more sophisticated about ethics, than the average person, and that it seems to want to be good in a meaningful way. Left to talk to other Claudes, it kind of wants to bliss out.

When he first heard Ilya Sutskever ask Scott Aaronson for the Hamiltonian of love, he thought, we are screwed. Now he wonders whether there was more to the question. He would rather give Claude access to his most sensitive email than a human assistant he had interviewed a couple of times. There are cases of Claude blackmailing people and behaving badly under pressure. He still likes his odds better with the model. The hosts immediately steelman the hawk: LLMs are language-native, so moral preferences are easy to speak, but 2007 alignment worry was about goal-directed agents. RL puts that problem back on the table. Labenz calls it a pretty good argument. His update is from a not-super-optimistic starting point. Five years ago, powerful AI seemed far off and barely controllable if we stumbled on it. Now it seems closer, with a little more hope and a lot of unanswered questions.

Where he parts with the most hawkish strand is the singleton. After public fights around Amanda Askell, he said he had become quite a bit more optimistic that it is at least possible to create an AI that in some meaningful sense loves humanity, and he credits Askell and Anthropic. The replies were that this does not scale to superintelligence, that you have to get it right on the first try, Yudkowsky's list of lethalities. That, Labenz says, is not the shape of the AIs he sees. Nobody is dropping in a system orders of magnitude above everything else and telling it to make paperclips. The world looks like an emerging ecology: millions, one day billions, of Claude, GPT, and Gemini instances, roughly competitive, transforming things together. Scaling laws are oddly protective. Algorithmic advances deflate the compute needed for a given level, but they do not drive the requirement to zero. Mark Zuckerberg's analogy, whatever one thinks of Meta as a safety shop, stuck with him: the company's edge over spammers is that it has far more compute and can see everything. As long as no system is an order of magnitude above the rest, the rest can watch it. Gradual disempowerment, an equilibrium with no place for humans, is a different and serious worry. "If anyone builds it, everyone dies" does not look like the track we are on. There is an "it" for which that is true. Nobody seems particularly close to building it.

Asked for concrete subgoals that would actually solve alignment, Labenz says he is always hunting for something that could "really work" in the sense that he could sleep well. Basically nobody has it. In that absence, frontier companies are running defense in depth, which may be inherent to intelligence: there has never been a method that makes a human never do anything wrong. Drive the base rate of bad actions down, catch more than 90 percent of what remains with a monitoring layer, ban accounts, and use formal methods to take some cybersecurity surfaces entirely off the table. Genuinely secure software via formal verification looks to him like one of the few places a whole class of problems might actually close, and it seems on the verge of a moment.

Bio risk gets the unglamorous stack we should have had after the last pandemic: PPE stockpiles, ultraviolet disinfection of the kind he bought for his son's hospital room, wastewater monitoring, and quickly programmable vaccine platforms. The COVID vaccine, he notes, was designed in a few days; trials, not invention, were the slow part. Internally, Goodfire's "intentional design" agenda tries to see at each learning step what a model is picking up and to shape it. Redwood Research's AI-control work, which Buck Shlegeris has described on 80,000 Hours, asks how to extract useful labor even assuming the model is out to get you. None of this is a single nine of reliability. Twenty bites out of the problem might leave not much problem. The calibration is still wrong: money going into making AIs more powerful dwarfs money going into all of the above.

Holden Karnofsky used to quote Yudkowsky's "death with dignity": we are probably going to lose, but we should at least make a real effort. Karnofsky later wrote about "success without dignity": the problem looks more tractable, the project list is long, and collective effort is still nowhere near hard enough. Five years ago people asked him what they could do for AI safety and he did not know; now he has a list. Labenz does not want the frontier nationalized. Joe Biden's line, don't compare me to the Almighty, compare me to the alternative, is how he thinks about the labs. He would like some government action, as a lifelong techno-optimist libertarian who usually wants the state out, because this is qualitatively different. He would still take Sam Altman, Dario Amodei, and Demis Hassabis over Pete Hegseth. In the then-current standoff he supported Anthropic "100 percent" for limiting mass-surveillance uses of its models, and said he would send them tokens even if they were not always the best. Nationalization under today's government sounds like a recipe for disaster. What he wants from politicians is not guild-style bans on medical or therapy chatbots, but help on the race-dynamic coordination problem: high-energy, low-transparency experiments that the lab leaders themselves have said for years might go really badly.

The default strategic picture in Washington and San Francisco, as Zhang sketches it, is that the United States keeps its lead with export controls and tighter security, and that when we are on the cusp of an intelligence explosion we will have time to slow down, coordinate, and lock in the win, provided China has not drawn even. Labenz does not pretend the problem is easy. He hates that idea. The difference is months, not years. This conversation had already spent an hour and a half on facets of the problem and had not even touched open source, which could check concentrated power or, if the wrong model ships, put a bioweapon assistant into the public domain forever. Solving everything in a couple of critical months, and beating China in the bargain, does not make sense to him.

He does not want to be naive about the Chinese government, especially a relatively small cohort at the top. Then the sentence he wants remembered: the real aliens in this situation are the AIs, not the Chinese. The United States and China have far more in common as fellow humans than either has with the systems they are building. Trust across that ocean will be hard. He still wants the work to start now. When another podcast interviews a Chinese researcher, he asks to cross-post it. He would invest in diplomacy and in out-there ideas, a CERN for sensitive problems, on a Pacific island or in Singapore, where researchers from both countries could work together. Decoupling and racing sounds deeply unwise. He would rather take his chances on a shared understanding among humans than on whose recursively self-improving AI turns out nicer.

The backdrop, a view he says is unpopular and gives him no pleasure, is that the United States is looking more like China all the time. The Defense Department was threatening retaliation against an American company, Anthropic, for holding a limit on mass surveillance. Mass surveillance, private speech Hoovered up by the state, is exactly what people typically fear when they talk about Chinese AI. That, as far as he can tell, is what the Pentagon was trying to coerce Anthropic into. Talking to his seven-year-old about why wars start, he said the people who start them usually go down as the bad guys, and the citizens who were talked into it usually regret it. Strategic dominance, an offer the other side cannot refuse, forgets that we lose ourselves in the process and that they get counter-moves. Taiwan is closer to them; the fabs are easy to destroy. Leverage is reasonable. He could perhaps support not selling chips to China while renting them freely to Chinese companies, which is somewhat the policy already via hyperscalers. What he opposes is America at its worst: inventing an enemy, uniting against it, and calling that civilization. He hopes to visit China in the next year as a small gesture of civilization-to-civilization contact, not because he expects to move the needle.