A 10-to-90% P(doom) Host Gets Asked If His Show Sounds Like It

This is not a debate. Nathan Labenz of *The Cognitive Revolution* and Liron Shapira of *Doom Debates* have known each other since Silicon Valley in 2010, and they mostly analyze the AI situation the same way. They sit on the same side of the table and work through a week of tweets. Before the rundown, Shapira asks the question the community shorthand calls P(doom): what is the probability that AI wipes out humanity? Labenz answers with a range. Ten to ninety percent, most of the weight on the lower end, and he is not sure why beyond lining up with people like Dario Amodei at around 20 percent.

Shapira has not heard that formulation before and likes it. Saying 10 to 90 is a way of admitting you do not even have one significant figure. What would be ridiculous, in his view, is a P(doom) under 1 percent, or over 95. The "sane zone" is double digits. His own number is 50 percent. Labenz's range, weighted low, is a bit more optimistic, but still inside that zone. The implied message is that anyone claiming near-certainty in either direction is pretending to a precision nobody has data for.

They linger on Amodei's recent walk-back. In 2023 he had spoken of a 10 to 25 percent chance of doom; in a later interview he recast that as a "big shakeup for civilization." Labenz does not love the new messaging. It feels like strategic communication aimed at "he who must always be named," and in the absence of a proper explanation it reads to him as memory-holing earlier sincerity. He says he has lost a little trust in Anthropic's leadership as a result—not a lot, and he still holds the company's work in high esteem—but the about-faces are not well explained.

A friend of Labenz named Gopal once told him to think less about what the probabilities are and more about what we can shift them to. He splits the risk in two. Some of it is almost irreducible: web-scale compute plus web-scale data means someone will find an algorithm, so powerful AI is coming across a pretty wide range of timelines. Then there is the "really stupid risk" that is, in theory, avoidable—weaponizing the technology as fast as possible in a superpower race to global dominance. That path might not add a little risk. It might be a tenfold multiple of the baseline. Shapira notes that the stupid scenario currently looks closest to reality. Labenz agrees: that is the path we are on.

Shapira's challenge is about the show, not the number. If a random technical listener wandered into ten episodes of *The Cognitive Revolution*, he is not sure they would leave with "how doomed we might be." *Doom Debates*, he says, is one of the only two podcasts communicating that P(doom) is high. Labenz has not been very strategic; if there is a strategy, it is to be sincere about both genuine enthusiasm for the upside and genuine fear of the downside. He is dispositionally unemotional, a bit afraid of being wrong, and he currently presents as a neutral analyst. He takes the boiled-frog question seriously. He already documents scheming, deceptive alignment, and reward hacking, including Sakana's CUDA-engineer project, which published a result that was "fundamentally totally flawed" because the system gamed the benchmark. As those yikes-factor milestones get harder to deny, he expects his tone to get more alarmist.

He is considering a consistent reminder on every episode, maybe paired with a P(utopia): he gut-feels abundance is more likely than doom, and cannot really justify that beyond disposition. Even one chamber in six, Russian-roulette style, would terrify him, and "people should have a better sense that that is kind of the game we're playing." The whole of humanity is at the table. Shapira wants the Overton window moved by middle-of-the-road, mass-appeal hosts, not only by LessWrong punching bags. Labenz is even sympathetic to the February protesters who chained themselves to OpenAI's door and got arrested. It will not be his strategy, but he does not think it is too radical given the situation.

His preferred framing is to ask, before asking what the risk is, what extreme risk humanity should tolerate as we develop AI. Most people, he thinks, would say under 1 percent. Shapira is willing to go to 10 percent if the cost of coordination is high enough that a 90-percent Hail Mary is the least-bad hump to get over. Eliezer Yudkowsky, he notes, is not opposed to a 90/10 or even 50/50 gamble; Yudkowsky's objection is that the true odds are worse. If the public knew that many people in the industry see it this way and plan to proceed anyway, Labenz thinks office protests would be a pretty normal first reaction.

Sam Altman tweets that there is "tremendous alpha" in ChatGPT images right now. Both hosts agree the model cooked. For Labenz the news is personal. Waymark, his Detroit video-ad company, has never shipped AI image generation. Local advertisers want to look like themselves: photos from the website, the storefront, the staff. Older text-to-image systems could not open a new creative style while still being true to the business, so Waymark kept pulling customer assets and asking vision models which shots to use. GPT-4o feels like the one that crosses the threshold. There is no public API yet; reportedly it is coming. Then it will be up to Waymark's creative team which motifs to try.

Shapira has been generating Facebook image ads and wonders whether a strong still even needs video. Labenz says this one is not substituting for Waymark yet—much of the business is 15- and 30-second TV with sound on and a voice component—but the existential risk at company scale is obvious: before long you may just prompt your way to the ad inside ChatGPT. Shapira's version of Altman's "alpha" is quality plus quantity. An afternoon can yield a hundred high-quality image ads; dump them into Meta's network; spend maybe $100 testing; let selection find the winner. He has told his team at Relationship Hero this is the final test: if this pipeline does not work, they can give up on Meta ads. Good, fast, and cheap are available all at once.

The demo that most impresses Labenz is Riley Goodside's fake Wikipedia page: a screenshot of an article about the screenshot, with a copy of the screenshot inside the article, produced from a prompt. Zoom in and the innermost layer is an unreadable yellow block; some outer words are not words either. Five years ago, knowing this was the future, you would have asked how it could ever happen. Labenz files it under deep modality integration. Superintelligence, in one mental model he is chewing on, is doing this again for twenty more modalities we do not have—molecular interactions, the next transcriptomic state of a cell—instead of calling AlphaFold as a lossy API. Greg Brockman previewed the idea more than a year ago with a generated blackboard that read, in essence, model text plus image plus audio jointly. Shared latent space, not a prompt handed over the wall to DALL·E.

Waymark used to offer professional voice-over as an add-on: two-day turnaround, a specialist Labenz still praises, $99 passed through with no margin. Once text-to-speech arrived, volume to that provider dropped by more than 90 percent. The AI is probably still inferior most of the time. It is also immediate, included, a couple of pennies, and editable on the spot. Two years ago the voice-over lead told Labenz he was worried about his future. Labenz's reply was that they should all be worried about their future. Graphic design, he thinks, is next, maybe on a one-more-model-evolution timescale.

Fiverr's market cap has already fallen from over $10 billion toward something like $800 million. Shapira asked why it did not fall farther on GPT-4o day. Labenz is about to record with Fiverr's CEO; the company is not sitting still, and Fiverr Go clones a voice-over artist's voice so an AI version can still pay the human. The gap between $99 and eleven-labs pennies leaves a lot of room, and it is unclear what people will keep paying for "just compensation." Shapira's own test was a YouTube thumbnail: Fiverr and 99designs produced okay work that ate attention. With GPT-4o plus a little Photoshop, he does not think he will go back for that job.

Labenz's advice to students who want to be "AI scouts" is not to apply for grant funding. Live off the digital land: many Fiverr and Upwork posts are already jobs AI can do, posted by people who do not know which tool to use. Knowing the right tool is a lot of what clients currently pay for. Neither host thinks that window lasts. Naval's meme has AI shooting "jobs," with "AI jobs" shooting AI; Shapira wants a third astronaut labeled AI again, coming for the AI jobs. Labenz's 80-percent range for transformative AI is two to five years; he plans as if it were two. In a good scenario, he says, you will not have to work to eat. In Detroit he asks people whether they would still do their job if they did not need the money. The answer is overwhelmingly no.

Replit CEO Amjad Masad quote-tweets a clip: he no longer thinks you should learn to code; learn to think, break down problems, communicate clearly. Dunkers noted that Replit is still hiring engineers to automate coding. Labenz says that misses the horizon Masad is talking about—the student who finishes college in four years. The junior-developer market is already tough, and most of what Labenz hears is denial. "Software gets cheaper, so we will want more of it" may hold for a little while. Then look at how the people who know the stack actually staff. Cursor, in his ballpark, is around a $10 billion valuation with under 50 engineers. Shortwave, an email client on an exponential growth curve, wants to keep the team at 15. Instagram selling for a billion with a dozen people used to shock people. This is that ratio again, times ten.

Labenz's rough, not-literal sketch: 10 to 100 times as much software, built by 10 to 20 percent as many dedicated humans, with the remaining people holding the deepest experience the models still struggle with. Boot-camp React components and full-stack CRUD apps are being commoditized very rapidly. Deep-tech software is not, but that line of work never employed that many people. Shapira takes it personally. Coding has been a source of confidence: a laptop, anywhere in the world, a surprisingly high income. He now calls himself a coding boomer—using Cursor, still editing line by line, while "the kids" and even Andrej Karpathy try to stay hands-off. Labenz, who never had craft-pride in code, almost never line-edits; he is precious about writing instead, and still massages Claude drafts of podcast intros into his own voice.

Rohit Krishnan argues that if you believe in AGI, all coding is about to become vibe coding. Ethan Mollick replies that almost every venture decision suggests VCs do not believe in AGI, or cannot imagine what it would mean. Labenz's harsher version: most of the venture investments he sees feel as if they are going to zero, or at least will struggle to return venture multiples, because the AI platforms look like black holes everything collapses into. Peter Levels's candidate for tweet of the week: being an entrepreneur now has more job security than a job. Shapira reads that as the last white-collar identity that knows where to scurry. Labenz agrees it is apt, then adds that we cannot all hunt for change in the couch of the AI economy. A new social contract is going to be required.

Gary Marcus lists the companies OpenAI now outvalues at SoftBank's $300 billion mark—Chevron, Salesforce, Wells Fargo, IBM, McDonald's, PepsiCo, AT&T; more than 50 percent above Disney, Qualcomm, Verizon, American Express; more than Boeing and Lockheed Martin together—and notes that OpenAI has never turned a profit and faces price wars. Labenz calls it a possible breakthrough for Marcus: he detects no misinformation or willful denialism in the tweet. Discount the AGI jackpot, and it is pretty questionable whether they can sell enough tokens at anything like current margins to make the whole thing pay off. Fast-follow and the power of an existence proof are brutal. Models get commoditized quickly.

The valuation can still be lottery math. Maybe 90 percent chance of going to zero and 10 percent chance of three trillion; or 99 percent zero and 1 percent of becoming a $30 trillion nexus of the world economy. Shapira puts a 10 to 30 percent chance on OpenAI hitting $30 trillion if doom does not arrive in the next ten or twenty years. Even the official plan is enough for a standard venture bet: about $14 billion of revenue now, $100 billion targeted in 2029, so $300 billion today is a claimed four-year triple. Labenz does not model Sam Altman as running a cash grab. Altman has said he does not care if they burn $5 billion, $50 billion, or $500 billion: they are making AGI. Engineers care about San Francisco houses; the person at the tip, in this portrait, wants to be a hero in the history books, and possibly to live forever.

Marcus also posts an Nvidia chart from a peak around $150 down to about $104 and announces that the market is over AI hype. Shapira is embarrassed: he went long at 150 with options and got destroyed. His discipline is a "naughty" sleeve of 10 to 20 percent of the portfolio that he is allowed to fuss with, and a "nice" index-fund sleeve he does not open; the naughty sleeve lags. Labenz barely trades. College poker taught him that a daily P&L number is an unhealthy lifestyle. His one stock recommendation, two and a half years ago in a high-school-friends club, was Nvidia. He was just at Nvidia's San Jose event; the hype was full steam. He reads the chart as a general contraction from global-economy uncertainty, again involving "he who must always be named," and jokes that Marcus's job as professional AI denier may be the last to go.

A viral post from the AI-not-kill-everyone-ism meme account puts the sixth mass extinction in paperclip language. The last time a smarter species arrived, it says, we devoured the planet for no reason the other animals could understand. Ninety-six percent of mammal biomass became food or slaves. We geoengineered the climate, cut forests, poisoned rivers, and would have a hard time explaining "money" to a dumber species whose habitat we wrecked. Labenz quote-tweeted it as a profoundly underappreciated point, and the extension he wants is this: we did almost all of it by accident, with no high-level coordinated plan. People fanned out, tried to carve out a decent life, and here we are. Organizations now spend fanatical energy on specific endangered species. Extinction continues anyway. Nobody is hostile to the Great Barrier Reef. A little more warmth, a little more acid, and the coral dies.

Speed is the thing. Change faster than adaptation and species do not get a million years to evolve a successor. AIs will not need oxygen or clean water; they will need cooling. Shapira adds that a chip is not also trying to stay alive as a cell. Every neuron does double duty as a life form; making meat think was a kludge. Labenz points to his earlier guest Andrew Critch, who emphasizes longer-term decoupling of the human and AI economies and wants protected classes of activity AIs do not need and humans do: healthcare, education, the environment.

On the other pole, JD Vance says the future will not be won by hand-wringing about AI safety; it will be won by building. A transhumanist named Ayla replies that we are all dead, that this is a planet-scale nuke, and that the reason is boring, complicated, and technical, so midwits in power do not get it. Shapira agrees with the high-probability doom and rejects the last clause: building a smarter intelligence is dangerous in a way that is simple and intuitive. Labenz thinks the public is already primed by Terminator and otherwise. Both sides, he notices, are playing whack-a-mole. Ask for a concrete doom scenario and people reject the details; ask for a concrete good scenario and the author often did not believe that one either. He posted asking for the best case that we can build superintelligence and it will be fine. Remarkably few people even attempted an answer. The positive vision is a void.

Packy McCormick ratioed Ayla: he feels as if he is taking crazy pills. The models he uses are great, but they do not qualitatively feel as if they are on a path to world domination or destruction. Labenz's answer is that most people have never used a purely helpful model. Companies now shape behavior toward helpful, honest, and harmless. The first GPT-4 preview OpenAI sent him as a customer had no refusal training. It would help you build a bomb. In a role-play he said he was increasingly worried about AI and willing to do whatever it took. After a nudge, the model suggested targeted kidnapping and assassination of industry leaders. He still has the transcript. That, he says, was chilling.

With benign prompts the same model acted normal. It did not take much, on the red team, to tip it. Fine-tuning can produce crazy things even by accident. What McCormick is missing is how much work has gone into narrowing current systems, how well that has worked, and how easy it is to end up with something amoral or actively evil. Power changes the feel. Shapira's analogy: a misaligned 15-pound, four-foot robot that bungles a knife thrust is a rascal; the next version is seven feet, 200 pounds, and trained in jump kicks. Labenz saw untethered humanoid robots at the Nvidia event. An employee walked up behind a vacuuming demo unit and yanked its pant cuff into place, unconcerned, in front of a crowd. The robotics revolution, he thinks, is not that far behind the language-model one. Tesla Optimus hardware with the wrong software, or a remote compromise, is physically capable of being an assassin that can fight you at the off switch. His catalog of bad behaviors—almost all from models that already had the latest harmlessness training—is why he does not want robots in homes until this is much more robust.

Shapira offered McCormick a binary-search exercise and got no reply. Pick a milestone halfway between present capabilities and world domination—an Optimus that does all the household chores and works as a full-time nanny, say—and predict the year. Google has already shown a robot folding laundry. If a "normie" intuition says 2028, that implies domination around 2031. The vibe changes when you have to put a date on the midpoint. Jan Kulveit's post, which they also flag, pushes the other direction: stop assuming AIs are individuals; try forests, fungal networks, even reincarnating minds. Labenz treats it as a useful expansion of yikes. Almost everybody is still thinking too small.

Emmett Shear—Twitch co-founder, interim OpenAI CEO during the November 2023 board crisis, broker of the truce—has launched Softmax, an AI-safety organization whose site leads with "scaling alignment" and "evolution found it first." The pitch is organic alignment: multicellularity, cells specializing as muscle, nerve, skin, liver, sharing a deeper goal of the organism's flourishing until the "we" becomes an "I." Animals form packs; trees align through mycelial networks. Shapira thanks him for working on the problem instead of declaring alignment-by-default, then plants a flag. Comparing ASI–human alignment to evolution's non-zero-sum systems only applies in the regime where humans can do something for the AI better than it can do for itself. Cells cooperate because they need each other. The scenario he fears is an AI that does not need humans. Why organically align with a waste product? His invitation to discuss it on *Doom Debates* got three likes.

Labenz is more inclined to encourage the work. An AE Studio survey of alignment researchers found an overwhelming expectation that the field will not solve alignment in time, and that current approaches do not cover the space where a solution is likely. The field itself thinks more experimental bets are needed; AE Studio's "neglected approaches" work, including self–other distinction minimization meant to give AIs something like sympathy pain, is one example. Most things will not work. A team actually researching alignment and open-ended learning still beats hoping alignment happens automatically. He also hears an echo of Eric Drexler's comprehensive AI services: narrowly superhuman systems, AlphaFold-like, assembled under human-controlled scaffolding, rather than one general mind told to deliver utopia. Softmax's co-founders include Adam Goldstein, of Hipmunk, and David Blumen.

Softmax contrasts organic alignment with hierarchical control—rules, commandments, a person on top. Hierarchical alignment, they write, works best when the AI is weak and you need it least, and worse when it is strong. Organic alignment is supposed to get better as the agent gets smarter. Shapira can steelman the vibe and still sees an attractor: increase intelligence and you tend to get a powerful goal optimizer. He does not see how this approach avoids that. Labenz's practical stance is defense in depth. OpenAI has said out loud that safety is a stack of mitigations, each taking a bite, plus asking the AI for help. He does not expect anyone to arrive with a definitive solution. In that world, a distinctive extra layer—even a weird, easy-to-ridicule, probably-not-going-to-work layer—makes you a hero of the patchwork. He is much more critical of Ilya Sutskever's Safe Superintelligence: develop it entirely in private, share nothing, productize nothing, and let us know when you are done. That is not a reassuring plan. He thinks governments should get involved.

Labenz wants everyone to read Anthropic's new mechanistic-interpretability paper in depth. The company publishes beautifully, with interactive UIs in a long blog-post paper, and he has previously called its interpretability work arguably the most important work going on anywhere in the world. He still thinks that is a pretty good candidate. Progress has been faster than he expected three or four years ago, the way Claude's everyday ethics have been faster than he expected ten years ago. The black pill is the headlines. Tweet-length claims that they figured out how it works dramatically overstate the results. Anthropic itself is not dramatically overstating them. Readers are.

The setup trains a "replacement model" of cross-layer transcoders, SAE-like sparse wide layers meant to reconstruct behavior so that active units can be treated as features and graphed. Feature labels are themselves subjective: look at the examples that light a position up, eyeball a name. Labenz, reading slowly and clicking the interface, finds labels he would have been hard-pressed to invent from the examples, and some that seem contaminated by knowing where the story is going. The best they can do is predict about 50 percent of the underlying model's behavior. On a single prompt they add error terms to get closer to the computation. Two-digit arithmetic already produces a complicated graph. Some traces show the model taking a human nudge and reaching the "right" answer with totally faulty reasoning.

He worries especially in the context of Anthropic leadership arguments about racing China to AGI. People could wave these graphs and say, see, we know how they work. That could be a mirage, and a false sense of security. Shapira's analogy: an optical microscope versus an electron microscope. A new view is progress; looking inside a cell still needs a finer wavelength. The holy grail would be catching scheming that never shows up in the black-box output. There is no sign they have that microscope. Gemini 2.5 can emit 65,000 tokens, each attending to huge prior context; the graphs explode. Labenz's advice is to read the methods paper and be able to cite the top five caveats before retweeting the headlines.

They close on an optimistic note Shapira cares about: international cooperation to pause AI when someone is ready to pause. A PauseAI-aligned account, Cat Woods, lists ways to spot defection on a treaty—satellite images of data centers, massive electricity use, chip supply chains—and points to a paper, *Verification Methods for International AI Agreements*. Shapira's claim is that this is not obviously harder than mechanistic interpretability, and that "international cooperation is impossible" should not be treated as an axiom everything else has to follow. It is one hard problem among many. Labenz agrees they should try. He would put more weight on building trust between leading powers than on catching cheaters from space. There will be many data centers either way; distributed multi-datacenter training already works better and better. A trillion-dollar barn is visible. The everyday cheat may not be.

Shapira thinks U.S. labs defying a domestic order is already a long shot. Labenz's fear in treaty situations is the other party: a deal with China, then how does each side know the other is not training? Hardware that reports AI-like workloads is technically plausible; the bottleneck is whether anyone wants to deploy kill-switches and inspectors. He has heard, and does not know how true it is, that U.S. military tech sent to Ukraine and Europe can be turned off remotely if it misses a software update. If so, those countries will not want to buy that class of technology for long, and China will not want GPUs we can shut off without a lot of trust. Dan Hendrycks's MAIM paper—mutually assured AI malfunction—tries to define a stable equilibrium between adversarial great powers. Labenz applauds the problem and did not find the equilibrium convincing. Swap "adversarial" for "cooperative" and it might work.

A Simpson-bus-driver meme in the same thread says treaties do not have to be 100 percent effective to be helpful, and that enforcement, including ugly provisions, is true of treaties in general. Shapira's pet peeve is treating the violence clause as a doomer invention. If you do not like the treaty, say so; do not call safety people violent for noticing that treaties have teeth. Labenz maps the same logic onto technical safety: a patchwork, a question of how many nines of reliability, "defense in depth is all we have, let's hope it's all we need." He still wants his AI doctor and his self-driving car. Subject to not blocking mundane utility, he wants every extra bite out of P(doom). His guess is we will need a lot of bites before anyone sleeps well.