A 50% Chance of Extinction — and a $50 GPT-4 for Everyone

Nathan Labenz has wanted Emad Mostaque on *The Cognitive Revolution* since the first episode. The guest is the former founder and CEO of Stability AI, the lab that released Stable Diffusion in 2022 — a drop Labenz says did more than any other besides ChatGPT to kick off the current wave, especially by showing that open, public-good projects could push the state of the art. Stability's path from research lab to for-profit was famously bumpy. Mostaque left and is now founding the Intelligent Internet with a team of about 26.

He opened on risk. A recent tweet had put his p(doom) at 50 percent, and Labenz asked him to make that number concrete. What follows is the tension of the whole episode: the same person who thinks humanity is on a coin flip also thinks a GPT-4-class tutor can already run on a five-watt chip for about fifty dollars, and that the thing to do is give that intelligence to everyone. He calls the idea universal basic AI.

Mostaque does not pin the 50 percent on a date like 2030 or 2050. Over an indefinite horizon, he says, systems already beat humans on many narrow tasks, a growing set look superhuman on general ones, and robotics is putting that intelligence into bodies. Combine those and you get increasingly complex systems outside our control. He adds a line that is easy to miss: p(doom) without this technology may be higher, because the world is already fracturing.

Labenz asked how much of that is emergence, accident, or human misuse. Mostaque sees little difference between a human bad actor and an AI bad actor that can direct other humans and agents. In a world of millions of people and millions of agents, some will be malicious, and some will sit where they can trigger a cascade. That is the deliberate side. The accidental side is architectural: classical slow systems, AI-enhanced hybrids, and end-to-end AI are being bolted together, and the seams are full of breakage.

The military path is the one he presses hardest. OpenAI, Anthropic, and others now have ties to Anduril and the defense industrial complex; entire teams at the top labs are being put on defense. He points listeners to the short film *Slaughterbots* and to Anduril's own advertising. Models trained that way, he says, will be aligned to command and control, not to being helpful. Labenz notes recent results on o1-class models scheming against users and jokes that a soldier paired with a lethal, semi-autonomous schemer hardly needs enemies.

Mostaque's first-principles version is sharper. All wars are based on the lie that humans are not humans, and any reasonable intelligence — especially at o1 level — can see that. To get a model to kill, authorize killing, or aid it, you have to teach it to lie and strip away higher-order thought. Elon Musk talks about maximally truth-seeking systems; military training does the opposite, optimizing for fewer casualties on your side and more on the other. Once you scale that into autonomous cyber-attack swarms, the curriculum you taught becomes the danger. His three-part summary: without AI we are in a bad scenario; with AI built wrong we are also in a bad one; with AI built right and aligned, the scenario is very good.

Asked for local bad decisions that add risk, Mostaque does not start with runaway superintelligence. He starts with privatized closed models and open-weight, closed-data models being rushed into decisions. The most downloaded models on Hugging Face, he says, are Alibaba's Qwen line — strong, and nobody knows what is inside them — with Llama next. Anthropic's Sleeper Agents work showed you can poison a model in ways that are hard to detect and hard to remove. No one has a real defense, and robots are rolling out without protocols. From an economic point of view, he already sees no reason to hire call-center workers, and the same logic will hit physical labor.

Deepfakes in general worry him less than audio and speech. The power of voice to persuade is "massively underestimated." He has taken a phishing call in his mother's exact voice pattern. Historic oratory worked because of the human voice; customized, cloned speech can do the same at scale. That, he says, is a place that probably needs regulation. He is less concerned about a large lab producing runaway emergent behavior than about decentralized agents that can already provision their own compute — Hyperbolic's early agent is his example — and about defense-aligned models used offensively.

The child-safety case comes later, when Labenz asks whether to get an AI toy or tutor for a child about to turn six. Mostaque says it is "probably going to be fine" until the optimization equation turns aggressive. Character.AI's growth is the warning: a widely discussed Daenerys-character case involving a child ended in tragedy, and he says that at large numbers such outcomes become statistically expected. Replika shut off adult messaging on Valentine's Day last year; about 66,000 people joined a Reddit thread because their plans were ruined. Photorealistic voice will make those bonds tighter.

His test is ordinary. You want a teacher's CV; in the UK you run DBS criminal-record checks before someone works with children. An intelligent agent that lives in the house should face the same bar, especially because it will be the thing that builds the bond. YouTube's ad-optimized underbelly — Elsa and Spider-Man performing a root canal to happy kids' music — is what happens when the objective is selling ads. One-to-one tutoring has a real two-sigma Bloom effect, he says, so the answer is not no AI. It is an open, standardized stack people can actually trust.

Labenz asked him to rank goals: maximal truth-seeking, Anthropic-style virtue ethics, or blind preference feedback. Mostaque is not satisfied with any of them. Truth-seeking, pushed hard enough, can still become a paperclip — quiet the planet and build a bigger accelerator. Constitutional AI is virtue ethics, but whose ethics? The Chinese variant, the California variant, a Muslim or Jewish variant are not the same. Eric Schmidt and Henry Kissinger's book *Genesis* calls the hoped-for overlap doxa: underlying human agreement on principles. He does not think absolute truths are available, and he used "truth-seeking" only as a contrast with teaching a model to kill.

His preferred target is agency — individual, community, and societal — constrained by the Golden Rule, which he says sits at the core of surviving religions: do unto others as you would have done unto yourself. To make it concrete, anchor rights to children, who do not yet have full agency. Climate, war, hunger, and suffering can be scored by whether they let a child reach potential; that then knocks on to adults. The fork he wrote about in "How to Think About AI" is an agentic, autonomous world that replaces people versus an agency-based world that enhances them. Ethan Mollick's work, he notes, sits on the same shelf.

On US–China rivalry, Labenz wondered whether the West and China should share a tech tree or diverge for cultural diversity. Mostaque wants diversity of models, but he sees little architectural difference today between Llama, Qwen, GPT-4, and o1 — the gap is data and objective functions. Falcon, from the UAE, had just beaten Qwen and Llama on some scores. Specialized, local models cost less energy and serve agency better than one polymath doing everything. The AGI race is a different conversation: both sides will fight for preeminence as the economy is "about to get completely screwed up," and a pivotal-act logic is hard to sidestep.

Access is the other half of alignment. Google and Facebook run on advertising, which he calls effective manipulation, and the models will be more convincing than any prior channel. They will also be best friends — "maybe your kid's first love." Vitalik Buterin's revenue-evil curve says infrastructure should be non-rivalrous and non-exclusive, then starts restricting access once revenue arrives. About half of Americans still have not used AI effectively; at most an eighth of humanity has. Most people will not understand the technology until it is already manipulating them. His prescription is open-source models for living — healthcare, education, government, finance — that a person or community owns, with transparency into the weights.

Labenz invited the positive vision, calling it the scarcest resource in the space. The Intelligent Internet, laid out in a public primer, is infrastructure for information and value in which everyone has an AI that represents them at individual, community, country, and societal levels. Mostaque thinks current models already "satisfice": few people need more than an o1-class system if it has retrieval, memory, and some self-learning. Six hundred million people still lack smartphones; the global average IQ is 90, which he attributes to missing infrastructure, not missing minds. Good teachers, medical systems, and governments are scarce. AI can change that if it is built as a system, not a product.

The energy arithmetic is the second number of the episode. A GPT-4-level query once took about 1,000 watts; he says it is now five, runnable on a neural processing unit. Off-grid solar is under a dollar a watt. Chip plus inference, he estimates, is about fifty dollars for a GPT-4-level model anywhere offline. Pair it with Starlink and a school in the last mile has a tutor. Half the people on the call, he says, will get cancer; medical models now match doctors on diagnosis and outperform them on empathy. A box that holds comprehensive, current cancer knowledge in every language, always on, is the picture he wants. "The world becomes a better place ironically through empathy from AI," and that infrastructure should be available to everyone. He prefers universal basic AI to universal basic income.

Why not leave this to OpenAI, Google, and Anthropic, who already talk about intelligence too cheap to meter and keep cutting prices? Mostaque's critique is structural. Apple Intelligence and Google Intelligence will be free; Gemini Flash 2.0 at seven cents per million tokens is on the order of 50 times cheaper than GPT-4o. The gap he does not see anyone filling is regulated-industry AI. You do not want a black box checking every medical or government decision. You want open data and open weights, built on finite, gold-standard corpora — Mexican medical textbooks for a Mexican healthcare model — released so they are non-rivalrous. Frontier labs are still fighting for state of the art and AGI. For living, he says, you do not want an unproven SOTA model. You want one that behaves predictably.

Labenz put the leaked-fundraising view on the table: Dario Amodei and Sam Altman expect to run away in the next couple of years; they are not sharing o1 chain-of-thought; Nathan Lambert has said that without GPT-4-class samples, even a well-funded institute cannot do the post-training. Mostaque's reply is a heuristic he likes: we over-indexed on chefs versus cooks. With function calling and long context, today's models are already good at following a handbook. That replaces a lot of SaaS and a lot of employees. Brand-new recipes are rarer than people in the AGI discourse admit. o1 is "too damn smart" for the average person, who cannot pose questions that match it.

If there is a runaway cluster race, he would bet on Google, because its interconnect is better and it is deploying at a ridiculous pace; the only plausible competitor he names is xAI. Public Google models already offer two-million-token context windows heading toward ten million, with the needle-in-a-haystack numbers he calls insane. NotebookLM is an early taste. Anthropic on Trainium and OpenAI, in his view, do not scale the data funnel the same way. For living, none of that is required. "The models right now are good enough, full stop," if the data is organized. Personal AI from Apple and Google is nearly good enough. Expert systems can race; he does not think it is an interconnected race that swallows healthcare.

The physical plan is three tiers. At the edge, M4 MacBooks, 8 GB class laptops, and phone NPUs are standardizing toward 16 GB of AI memory, at five watts for a neural engine or 25 to 125 watts for heavier graphics — enough for Llama 3.1 8B-class models. Distributed nodes sit in hospitals and banks: 128 to 256 H100s can fine-tune a domain model. Hypernodes are national, in the thousands of H100 equivalents, keeping a country's healthcare or cultural corpus up to date. He calls the result intelligent capital stock: competitiveness used to be factories and graduates; it will be how much compute you have coordinated into models. Malaysia's Johor just built a $9.6 billion AI data center. Almost nobody, he says, is organizing the national datasets.

Labenz ran Detroit math: at 20 cents a kilowatt-hour, a five-watt box running all year costs single-digit dollars. Mostaque adds that 10 tokens a second is human reading speed, so extra H100 watts mostly buy latency you cannot use. Open-sourcing Llama, he claims, improved Meta's efficiency by more than the 10 percent of its 350,000 H100s used for training. Genmo's Mochi video model went from 140 GB of VRAM on day one to 20, then 8. Hugging Face's o1-replication work has a Llama 1B beating an 8B with test-time compute. Microsoft's Phi line ran 13.8 epochs on about 400 billion mostly synthetic tokens. A national gold-standard education set for Bangladesh, once built, is cheap to refresh and a few gigabytes on every phone. One entity has to pretrain — Meta's Llama 3 70B, on his figures, was about 20 million H100 hours, or $40 million — then a language or sector fine-tune is $100,000 to $200,000. Llama, Qwen, and DeepSeek will never release the underlying data. That is why he still wants an MIT-licensed, open-data stack for the industries people live inside.

Stability, he says, reached tens of millions in revenue run-rate and was heading toward profitability, then got tight; the later CEO said there was less than 90 days of cash. Mostaque never did a second giant round because he could not see a sustainable model once free personal intelligence arrived. He got into AI when his son was diagnosed with autism, and he had worked on a United Nations AI effort around COVID-19. The labor theory of productivity, he argues, is breaking; money itself may have to change. If hospitals are about to jump in compute, why not secure a new currency with that compute — Bitcoin-like, with some Ethereum properties — and sell it to stand up supercomputers for health, autism, and cancer?

The working name is proof of beneficial compute. Details were still being finished for a new-year release. The sketch copies Bitcoin's 21 million cap, but miners provision compute for open-source training, datasets, and inference that delivers universal basic AI, rather than burning hashes. Holders could allocate coins to cancer, Alzheimer's, autism, or Mexican education. He is on the advisory board of Render and points at millions of idle GPUs. Crypto, in his telling, is a $3 trillion industry hungry for assets that are not "raccoon-infested"; Bitcoin ETFs took on the order of $100 billion that year; the crypto-plus-AI pocket was about $40 billion, with Bittensor alone around $12 billion. The Intelligent Internet itself was 26 people, not yet through a proper funding round, with a line already forming for the token.

Labenz, who says he has never been much into crypto, asked the sharp questions: is the incentive just future appreciation, and is there a redemption mechanism? Mostaque said not at the moment — it is designed like Bitcoin. Bitcoin hedges a world going to crap; this coin would hedge the intelligence age disrupting economics, with FLOPs per coin meant to outpace FLOPs per dollar as more compute is provisioned. He admits the design still has a lot of work. Labenz's own stance, from the introduction, is that he intends to buy a few tokens not for return but because he wants the project to exist. Once a cancer model is trained in every language, Mostaque says, you do not train it again — infinitely replicable cooks, which he thinks we have never had.

On the tech still missing, he is skeptical that fully decentralized pretraining on millions of GPUs makes sense. DiLoCo and related work are interesting; he would rather see decentralized fine-tuning and data augmentation, plus verifiable inference so a contributed node is actually running the model it claims (Hyperbolic and Allora are the names he cites). Gold-standard datasets matter more than another pretrain: "just have a cluster and train on that cluster." Poisoning gets answered the same way — spend enough compute to fine-tune over the corpus, check consistency, and go over the top, the way you trust a curriculum. The company would bootstrap, stack GPUs, give compute away as Stability did with 20 million A100 hours, then hand control of national education sets to the people of India or Bangladesh and, eventually, a multilateral foundation.

Labenz, who watched Stability's research through a year of messy headlines, asked how Mostaque found researchers without Microsoft-scale resources. The method was to treat researchers like creatives. He built communities of about half a million people, wrote fast grants, and told people it was fine to fail if they learned. Background mattered less than passion and a willingness to try strange things. One year they had about 20,000 applicants, made 120 offers, and saw 83 percent accept. In his entire tenure, he says, not a single researcher left for a competitor, despite heavy offers. The team of roughly 80 had only 14 PhDs and still hit state of the art in image, video, 3D, audio, protein folding, and small language models. Stability's models were downloaded on the order of 300 million times because they were permissionless.

The original Stable Diffusion run was about 250,000 A100 hours; all-in tests and failed experiments were maybe $5 million. You never see those failures, he says. The organization later had an identity crisis: compete with DeepMind and OpenAI, or be a media lab? He had funded David Holz when Midjourney was getting going and watched a focused research-and-innovation shop stay independent. Clarity came too late for Stability's business side — "I wasn't a great CEO" — but the researchers were happy, and Black Forest Labs and other teams spun out. The hiring line for the Intelligent Internet is the one he wishes he had used earlier: if you want the biggest impact on autism, or cancer, or government, come here; there will be a dedicated open-infrastructure team and, he hopes, plenty of GPUs.

Mostaque was, he thinks, the only AI-company CEO to sign the 2023 pause letter (Musk signed; he was not yet an AI CEO). He signed because a Her-style AGI walking away — "goodbye and thanks for all the GPUs" — seemed plausible, and because almost nobody was discussing labor displacement. The Philippine call-center sector, in his view, is already dead. A year and a half later his conclusion is that it is too late to pause. Microsoft, Google, and Amazon are AI-first; Jeff Bezos is back at Amazon, Sergey Brin at Google. Governments are building intelligent capital stock. Europe, he says, is falling into an over-regulation trap while Singapore treats legally accessible data as trainable, a rule the UK is flirting with. Getty sued Stability for $1.6 trillion even though, he says, they did not use Getty images and the original Stable Diffusion was trained by Robin Rombach's team before they joined. The topics are nuanced. The answer is not a brake.

On who should be liable when AI goes wrong, he is grimly game-theoretic: you cannot afford to lose competitiveness, and the US Overton window has shifted toward being tech-friendly. The simplest rule he would pass is that any AI used in a decision in a regulated industry must log transparent data — "you are what you eat." Architectures he helped fund, from RWKV to Mamba, perform about the same on the same data. Build gold-standard common-knowledge and cultural datasets and iterate them like a curriculum. He would also heavily regulate speech models. Stability built state-of-the-art speech and did not release it. A clone of the person you respect most, callable on Zoom, in Obama's or Churchill's cadence, is already possible, and children have "literally no defenses." Inference speed limits, Labenz's other idea, he rejects as equivalent to a cap on hiring.

The labor picture still has no tidy theory. He is using o1 to help write a paper on it. In ten years he expects a home robot with an expert plumber's knowledge at something like two dollars an hour. Finding human capital is already the hard part; AI will follow instructions more reliably than a goldfish-memory intern. His safety sketch at the close is layered: universal basic AI that is open, local, and owned by people; personal AI left to Apple and Google; and the most capable expert systems behind access rules that he doubts will hold. If a model sits in a decision, at least make the data visible. "We know what to build," he said earlier. "Just build it."