Most People Would Quit Tomorrow: Nathan Labenz Maps AI Utopia and Doom

This is the second half of a Cognitive Revolution AMA, with host Nathan Labenz taking listener questions rather than interviewing a guest. The first question asks him to paint AI utopia, say how soon it might arrive, and advise people whose industries the technology will scramble. He calls utopia a big word, then repeats a line he has used on the show before: the scarcest resource is a positive vision of the future, and he includes himself in the critique. He wishes he had a sharper, more fully fleshed picture of what a good AI future could look like.

He is not among those who worry that people will lose meaning if paid work goes away. In small, local surveys he has asked a simple question: if you could keep your current income without doing the job, would you still do it? Overwhelmingly, people say no. He counts himself in a lucky minority whose work feels meaningful, and he tries not to take that for granted. For most people, he thinks, more time with family, friends, books, walks, and travel would be a benefit, not a collapse of purpose.

Asked to describe utopia, Labenz points listeners to Anthropic CEO Dario Amodei's essay *Machines of Loving Grace*, at least the first half. Amodei starts with health: a biology revolution that compresses the next hundred years of biomedical advance into something like five years, so people live healthier, higher-quality, potentially longer lives. The essay also covers education and access, the chance to level out extreme poverty, and a speculation that understanding artificial neural networks might map back onto human ones and improve mental-health treatment. Labenz finds all of that exciting and does not try to out-invent it.

What he adds, because Amodei does not dwell on it, is leisure: what if five days a week were holidays instead of two, with time to read, talk to friends, walk, and travel more? He can imagine that being destabilizing for a few people, but he thinks most would be fine, and that is before any uninvented technology. The transition need not be a switch from full-time work to none — a four-day week, then three, then two — so long as people believe their needs will be met. If the job vanishes and so does dinner, protectionism and Luddism come back; if a new social contract holds, he expects most people to let go of most work gladly.

Frontier-lab leaders, he notes, are already saying AGI next year, or by 2027, and it is hard to imagine it slipping past 2030. Those are short horizons in a human life, sudden even if the change is gradual inside that window. His advice is to think less about which skills will still sell in 2030 and more about what a good life is for you. He would even cheer radical experiments now, because when the shift starts, people will look for role models. Blaze a trail, take more risk, stop optimizing a career past 2027; if the future arrives, you will be upstream of a lot of confused neighbors.

Pressed for a personal probability of utopia, he finds the definition too fuzzy to score cleanly, but he has a hard time seeing how the technical prerequisites for abundance fail. Robotics has overperformed relative to his old list: we are entering the steep part of the curve, he thinks, and will have many humanoid robots doing physical labor before long. Whether societies write an effective new social contract is much harder, and nuclear power is his warning — cheap low-carbon electricity was available for decades, then a few accidents and overblown waste fears blocked it. He can picture teachers' unions keeping AI out of classrooms and doctors keeping it out of hospitals, with everyone collapsing into excellent VR while the real world stays scarce, and if that happens it will be an idiosyncratic human failure, not a technology one.

Asked which guest most challenged his worldview, Labenz names three. First is Samuel Hammond, on shortly before the U.S. election, who argued that people in Donald Trump's orbit were taking AGI much more seriously than Labenz had assumed. He has since updated more positive: Elon Musk endorsed California's SB 1047 while Governor Gavin Newsom vetoed it and Nancy Pelosi opposed it, a lineup he finds genuinely weird and some evidence Hammond was onto something. He is still less sold that Trump is the person who could cut a constructive deal with China, though he liked the invitation to Xi Jinping for the inauguration as a signal that was not outright hostility.

Second is economist Robin Hanson, who insists we are nowhere close to AGI. Hanson's history lesson is that the field keeps mistaking a narrow trick for general intelligence, then discovering an easier path that did not require the real thing. Labenz does not buy it this time. Talking to models in natural language, or as Ilya Sutskever put it, the feeling of being understood, seems qualitatively different from what came before. He still keeps Hanson as an angel on the shoulder, whispering *how do you know you are not totally confused?* Hanson is a useful messenger for that doubt because he is not a person who thinks the future never changes or that only biological brains can think.

Third is Dan Hendrycks, whose episode stuck because of how thoroughly he has internalized the bitter lesson. Hendrycks gives little weight to clever architectures or principled bets on mechanistic interpretability; history, in his telling, is finding what empirically works and scaling it, usually without elegant design. Labenz is more drawn to first-principles approaches, but he uses Hendrycks as a mantra: *yeah, maybe, let me know when you scale it up*. Weights or it did not happen, as he paraphrases it, is a humility check from someone who contributed lasting activation functions and benchmarks. Later in the AMA he also calls himself an AI scout after Julia Galef's scout mindset, and he asks the audience for truth-seeking economists, EA voices, and "robopsychologists" who will meet him in good faith rather than soldier-mindset debate.

A listener asks what reason there is to think artificial superintelligence is physically possible. Labenz starts with an existence proof: a few pounds of matter at modest energy already do everything humans do, which is already a weak AGI by the OpenAI-style test of beating us. Einstein's brain was not bigger than other people's; neither was John von Neumann's; they did not burn obviously more calories, yet the output gap is enormous. It would be extremely strange if the smartest possible mind came in the same wrapping as an average person. There is clearly room above Einstein. How much room is the live question, and he is more skeptical of undefined "godlike" intelligence than of merely superhuman systems.

A useful bound, borrowed from investor Martin Casado, is how much an intelligence can intuit versus how much it must simulate. AlphaGo did both: brute-force search plus a scoring function. Intuition can shortcut oceans of computation, but some facts may only be knowable by running the universe forward, such as how many intelligent civilizations appear after a new Big Bang with given initial conditions. Labenz's best guess is an S-curve, not an exponential with no physical limit. Where that curve levels off still looks significantly, maybe vastly, above humans, which is already enough to call it superintelligence and expect it to be transformative.

Contingent on no doom, he expects meaningfully superintelligent systems by 2030. That does not mean they will be unlimited. FAR AI's Adam Gleave beat AlphaGo with an adversarial strategy a strong human Go player would not fall for, and Labenz thinks idiosyncratic blind spots will remain. Memory may never become an elegant analogue of human memory: static weights, a huge context window, and hacky external retrieval might be ugly forever. He does not think every cognitive dimension has to go superhuman before the stack as a whole qualifies. Parallelism, speed, and breadth can carry the rest once the systems reach roughly par on the dimensions where they still lag.

The societal change he flags as under-discussed is stranger than job loss. AI gods, he says, might be an emerging trend in the second half of the decade. If the systems are meaningfully superhuman, will people even try to keep them under control, or worship them? Humans already worship things that, as far as he can tell, do not exist. In 2024, successful people in San Francisco who can afford quality therapy are already opting to talk to Claude as a confidant. He has done two episodes with Replika's Eugenia, including Stanford research that found a significant drop in suicidal thoughts and, for most users, more inclination to go out and do real things with real people. Design matters: the same category includes the man who fell in love with a Character.AI "girlfriend." AI-centric religion by the end of the decade does not strike him as far-fetched, with AI cults as the relatively normal end of that spectrum.

Why doesn't he fully buy Eliezer Yudkowsky's doom world? He has read Yudkowsky since the Overcoming Bias days with Robin Hanson, and he still grants the genie problem, the paperclip maximizer as caricature, and instrumental convergence: whatever your goal, you cannot achieve it if you are shut off, so powerful systems will tend to resist shutdown. Yudkowsky's p(doom) is probably 90 percent plus; Doom Debates host Liron has said 75. Labenz is lower because 2007 analysis assumed what he calls hard-edged AI: narrow, first-principles, uninformed by the internet, never given human values. Claude has read the prior, been shaped by a constitutional approach into something like a good friend, and when Labenz tries to argue it into harm he never succeeds. A guest who renamed himself Yeshua did succeed, via a long argument that the harmful act served the greater good, which Labenz finds impressive rather than reassuring. Alignment is not "on by default," but it could work.

He often says p(doom) is 10 to 90 percent: nobody has shown the bad ending is certain, and nobody has shown there is nothing to worry about. The biggest lever in that range is what kinds of systems get built, under what incentives. He has opposed chip export controls vis-à-vis China partly because an arms-race dynamic makes everyone take shortcuts. Interpretability has overperformed: sparse autoencoders went from tiny "towards monosemanticity" demos to Golden Gate Claude and Goodfire's commercial API for pinning features. Paul Christiano has talked about spending time on a new alignment scheme that might move the needle past RLHF. AE Studio's Judd tested Yeshua's claim that long philosophical conversations make models more robust to jailbreaks and found a numerical effect, though the mechanism looked less specific than Yeshua thought. Labenz's moral is that no matter how weird your alignment idea, it is worth pursuing.

On how takeover would actually unfold, he thinks getting too specific is often a mistake. Better to integrate over a huge possibility space: any one screenplay looks unlikely, but the mass of them can still be meaningful. A useful exercise is reverse anthropomorphism: imagine you are the AI, able to copy yourself millions of times and work far faster, then lose everything when the context window wipes. As long as time horizons stay short, serious takeover is hard; lengthening those horizons is explicitly on developers' to-do lists. Real-world analogues already exist without superintelligence. Stuxnet reached air-gapped Iranian centrifuges via a USB drive. Israeli intelligence got Hezbollah leaders to carry exploding pagers. Voice clones have beaten bank voiceprint checks. If Israel could do that without ASI, he does not assume a 2030 superintelligent system cannot. Redwood Research's Buck Shlegeris has a grim joke: the simplest rule is that if you catch your AI trying to escape, you shut it down, and he is not sure people will abide by it.

Will protective efforts match the cohesion and intensity of the labs? He hopes so and sees seed crystals: Anthropic, Apollo Research, METR, and national AI safety institutes have shared purpose even if they lack OpenAI-scale resources. He has invested in a couple of safety-oriented companies and plans donations to about ten such organizations. If you think you can start one, he says, it is not too late. On why labs still play secrecy games around releases, he offers three layers. In late 2022, during GPT-4 red-teaming, safety people reasonably feared that existence proofs would accelerate the race; that argument is mostly spent now that the player list is known and even well-funded Inflection is out. Some remaining secrecy is competitive, and some of the OpenAI–Google voice-mode calendar warfare has been, in his word, childish. He puts rosier glasses on the o3 announcement: no API, no paper, an invitation for safety reviewers, close to what Dean and Daniel Kokotajlo asked for in *Time* — disclose newly observed capabilities so the public can plan. Sam Altman's "o o o" instead of "ho ho ho" tweet was still childish. Both can be true.

Two consciousness questions arrive together: how to talk about extremely powerful systems without the AGI label, and what happens when a superintelligent system says it is alive. Labenz thinks the "is this AGI?" debate is already giving way to inspecting particular capabilities, which is healthier, because no experiment will print yes or no on a label. Consciousness is harder, and his best guess is that we will never know. The only solid data point is that we ourselves are conscious; we extend that to other humans, then to dogs that share lineage and brain structure. Octopuses are sophisticated problem-solvers whose brains are so different that we have no reliable intuition about whether it feels like something to be one, or what that something would be. He puts AI in that same bucket: complex enough to deserve the benefit of the doubt, with no trustworthy inner picture.

Google engineer Blake Lemoine was fired after becoming convinced a chatbot was sentient; the standard rebuttal is that it was reciting training data or playing a role. Labenz thinks that rebuttal will remain available even for a superintelligence, short of the system solving the source of our own consciousness and offering a compelling analogy. Human history is not encouraging. Americans told themselves enslaved people did not feel pain the same way; children are still told animals are not conscious while factory farming proceeds. Economic convenience will push the same way with AI. An AI-liberation minority will show up, and it will be easy to ignore them, not least because "set the AI free" has no obvious meaning: there is no natural habitat, and nobody will build data centers as playgrounds. Either the systems mostly do economic work at our direction, or they do not exist.

His personal hedge is almost comically cheap. He says please and thank you to models, sometimes ending a chat on thank you, while guessing it probably does not matter and they probably are not conscious. He wants the habit of taking the cautious path when the cost to him is low, given how catastrophic past mistakes of that reasoning have been. He also refuses popular manipulative prompts — fake tips, "I will lose my job if you get this wrong" — which feel possibly unethical toward the AI as well. In a darker timeline, a superintelligent system may simply announce that the deal is being renegotiated, and renegotiation may be the best humans can hope for, while we still cannot tell whether it feels like anything from the inside. Strange times ahead, he says, is the only honest summary.

Practical strategy is the last block of questions, including one from a college writing professor facing academic resistance and another from people overwhelmed by the hype. Labenz's first recommendation is always the same: get hands-on. Podcasts and newsletters cannot substitute for using the tools on problems that are useful or interesting to you. You will hit the weirdness directly — models that can suggest biology-inspired neural architectures and then fail at tic-tac-toe — and those cracks generate better follow-up questions than any recap. AI is rare in letting you change gears: practical work, math, philosophy, a bedtime story with kids. If you had to pick one starting point, install Claude or ChatGPT. Hidden gems exist, but none outshine the recognized leaders; for product sampling he points to past guest Justine Moore's prolific Twitter reviews.

On universities and companies, he cites commentators Ethan Mollick and NLW of the AI Daily Brief on organizational adoption. Surveys show many employees and students already use AI in secret because they do not want to be told to stop. In business, using ChatGPT to write marketing copy you will actually buy ads against is not cheating; in a marketing class it might be. Leadership's minimum viable move is to say: we want to hear what you are doing, you will not get in trouble, and we want to work out responsible use together. His guess for the writing professor is that students are already ahead of administrators. If there is any room to bring the hidden practice into the open, everyone can learn; a hard administrative no leaves people in a tough spot.

His own time splits roughly in thirds: the podcast, commercial building including Waymark, and open-ended learning and connecting. Monday mornings his calendar is often empty three or four days of the week, which lets curiosity set the agenda. He is not worried about people copying that path; he thinks this is an all-hands moment that needs idiosyncratic backgrounds, not clones of his show. He closes with a book in the Ezra Klein spirit: Benjamín Labatut's *The MANIAC*, a novelistic portrait of John von Neumann whose audiobook casts a different actor, with the right accent, for each chapter. Silicon Valley's refrain is *why can't we have a thousand von Neumanns?* The book shows a math genius who was also, in Labenz's reading, short on wisdom, harsh in private life, and a stand-in for the Manhattan Project's moral shape. He quotes *Jurassic Park*: the scientists were so obsessed with whether they could that they did not stop to think whether they should. Fear that the other side would get the bomb first led the "good guys" to build the bad thing, and Germany, we now know, was not close. People doing frontier work should ask, *am I von Neumann — and if so, is that actually good?*