o3 isn't AGI. Zvi Mowshowitz still put p(doom) at 70 percent

Nathan Labenz opens the ninth Cognitive Revolution with Zvi Mowshowitz on a confusing OpenAI release. o3 is powerful, and economist Tyler Cowen called it AGI. OpenAI's own reporting, and a week of community documentation, said the model produces twice as many hallucinations as its predecessor. Labenz also flags a second number from the model card: o3 can complete more than 40 percent of pull requests recently written by OpenAI research engineers, which is why he wants to know whether recursive self-improvement has already begun.

Zvi is the author of Don't Worry About the Vase, a blog Labenz calls indispensable. He processes an enormous amount of AI news and writes long weekly surveys; this appearance is meant as the audio version of that firehose, not a substitute for it. Zvi's own disclaimer is that the posts are canon, written after he has had time to think. The podcast is what he is actually thinking, off the cuff, late on a Friday.

Labenz previews the rest of the map: science that seems to be accelerating while DoorDash still fails, what superintelligence looks like in Zvi's imagination, Epoch AI alumni leaving to found Mechanize, and a p(doom) now up to 70 percent. He also wants a live-player tour of Meta, DeepSeek, Safe Super Intelligence, xAI, Anthropic, Google DeepMind, and OpenAI. After a stretch in which pretraining plus light RLHF produced models that seemed to grok helpful, honest, and harmless behavior, he thinks intensive reinforcement learning is creating systems that are both more capable and more palpably problematic.

Labenz's first question is whether AGI is here and whether recursive self-improvement has started. Zvi's answers are no and mostly no. The more he uses o3 and reads the reports, the more sure he is. "This isn't even primarily an intelligence leap. This is a tool use leap." o3 is a much more useful version of the system we already had: easier to steer, easier to get what you want in a reasonable time, less forced into rigid boxes. That is wonderful. It is not AGI, if AGI means a plug-and-play worker that can do all the cognitive jobs humans do.

Asked how many years in the past o3 would have made him say holy shit, he says about two, and four for sure. His face would have been blown off, especially this soon. Shock is not the same as AGI. He thinks Cowen is defending a broad, slightly silly definition, closer to "is it smarter than me?" From Cowen's seat the answer may be yes. Zvi saved Cowen's claim for last in his writing queue, went through everything else first, and only then tried to understand why Cowen was saying what he was saying.

Cowen's three examples, in Zvi's reading, are intern-grade briefings. Why an artist's early paintings are worth more than the late ones is a pattern he already knows: collectibles have extremized, and the unique 9.8 comic dwarfs the 9.6, as any Magic: The Gathering player learned the hard way. Why a writer's prose is good is "good prose, people like it." What tariffs do to Knoxville, Tennessee is that they are a giant own-goal for manufacturers who import inputs. o3 will fetch a mountain of detail, make connections, and structure a memo. That is exactly how Cowen eats the world. It is not how Zvi's brain works, and it is not a general intelligence that can slot in anywhere.

Labenz's counterexample is Google's AI co-scientist. Built on Gemini 2.0, with search, AlphaFold, and a swarm of prompts grinding against each other for a couple of days, it was given an open-ended biological puzzle: something conserved across different drug-resistant bacteria, and the question "what is going on here?" It returned a ranked list of hypotheses. The top one had already been shown experimentally by a Google partner lab, unpublished. The scientist, in the telling, fell off his chair. Greg Brockman has separately said o3 is the first model scientists describe as coming up with really good ideas.

Zvi's reaction is that this is a human skill issue. People are bad at elicitation, bad at scaffolding, and bad at building the loops and tool use that would make these systems actually work. Scientists do not have billion-dollar compute budgets, and basic research is under-resourced. If he ran an AI Scientist Corp with a billion dollars, he thinks he could start making those connections, still limited by the need to run real experiments. We are not close to a full explosion that never touches the physical world. We are also not trying very hard.

Pharma should be hammering the API. Labenz ballparks a co-scientist run in the hundreds to thousands of dollars, maybe ten thousand at the high end, and patents are scarce. Zvi's real answer, delivered in earnest, is that people don't do things. They don't execute, don't take risks, and don't do things that look weird. He includes himself: he writes about this all day and still under-automates, even though he could probably work two to five times as fast as a few months ago. If the base models hit a wall, and o4, o5, Claude 4, and Gemini 3 were only 5 or 10 percent better and stopped getting cheaper, he thinks people would finally treat using the tools as their own responsibility.

Labenz's thought experiment is a drop-in knowledge worker no smarter than today's models, able to absorb Slack, email, GitHub, Drive, and the CRM the way a new hire bops around a company. If it can do most desk jobs, he thinks you have to call that AGI, though not ASI, and it would imply the job of AI researcher too. Zvi agrees there is a lot of room, then reaches for Jane Street. The firm effectively takes a loss on a new employee for at least a year, because other people spend time training you while you do work someone else could do better. Ask companies to put in 10 percent of that investment before an AI worker is useful, he says, and zero of them would tolerate it.

The naive fine-tuning math looks cheap. Labenz cites about $25 per million tokens to fine-tune a GPT-4o or GPT-4.1-class model; a billion-token company corpus would be a $25,000 custom model. The catch is that you cannot dump the tokens in. OpenAI's famous Slack-trained demo learned to say it would work on the task tomorrow. Machine learning is the fiddliest thing on the planet: filter, transform, provide context, iterate. If OpenAI offered a $500,000 training job plus $20,000 a year per copy and all you had to do was dump tokens, people would jump. That product does not exist, because the human trial-and-error of setting up the training is still the job.

Labenz is more optimistic after stuffing the Emergent Misalignment research codebase, about 400,000 tokens of code plus paper, into Gemini 2.5. The model inferred what his new experiments were for without being told, on a paper too new to be in training data. Zvi still sees O-ring problems. A worker who is 99 percent of what you want, with one deal-breaking 1 percent, is not useful, and you cannot always hand that 1 percent to someone else. Until the system crosses "good enough," its value is treated as zero or negative. Progress, he thinks, will start with specific tasks the model can already do well, then string those together over time as both the models and the scaffolding improve.

Labenz is willing to make a blanket recommendation: most frontier scientists should be using these models, even at a one-in-ten hit rate on ideas worth engaging. A scientist who refuses a one-in-ten shot at a great idea is, in his view, a bad scientist. The same week, neither of them can get a computer-use agent to order DoorDash reliably. Labenz tried Operator, realized he would have to teach it "no lettuce, no tomato," and quit. Zvi's Operator asks every two seconds whether it should continue. Manus will actually put the sandwich in the cart, which is why Labenz will not give it his credit card.

The products that work, Labenz says, are the ones that turn the hyperparameters all the way up. Shortwave, using Claude, searched his last 100 emails, stuffed them into context, and gave useful advice, including a ranking of upcoming podcast guests for a sponsor. Claude's own Gmail integration, given the same prompt, searched in fives, stopped at 15 messages from the last three days, and called it done. Shortwave still asks before sending or marking done. The pattern is consistent: the same model, with more search depth and more context, crosses from toy to tool. Labs ration that because they are compute-limited and would lose money if every customer ran at Shortwave intensity.

Mundane automation would itself accelerate science. The average scientist's biggest time sink is fundraising and paperwork. Labenz wonders whether a withdrawal of federal funding might even reinvigorate universities by forcing departments to decide for themselves. Zvi is not optimistic on AI timelines. Replacing a broken grant system only helps if you actually build a new one; hobbling along with a broken one is worse. On the clocks they are watching, there is no time for a new university equilibrium to pay off. The plate, he says, is already full of crisis.

The number Labenz cannot stop thinking about sits around page 22 of the o3 model card. OpenAI checks the model out at a past commit, hands it a human-written assignment and unit tests from work research engineers already did, and asks whether it can one-shot the pull request. Prior models were in the single digits. The new ones are in the 40s. That is why he asked at the top whether RSI is here. Literally, the number is under 50 percent, so mostly no. Figuring out what to do is a large part of the job, and the test does not cover it.

There is a paradox. If o3 can do 40 percent of existing pull requests, those requests should already have been done, leaving a residue of tasks the model cannot handle. Labenz notes that many orgs still wrap work in issues, tests, and deploy pipelines even when a model writes the code. Zvi's model, from coding with Cursor, is that AI-easy work should not sit in the queue for long, so the open set should be biased toward things the current models struggle with. A 40 percent historical rate should show up as much less than 40 percent of the live queue. This may just be a transitional snapshot: the previous generation really could not, and this one suddenly can.

The 40 percent also does not match what Zvi is hearing from programmers. o3 is not that good at coding. It is good at architecting and debugging. That would make sense if a huge share of internal issues are "we found a problem, we don't know what's going on." Debugging is a huge portion of the work, so the number can be a big deal without meaning the model is a drop-in research engineer. The model card is doing a weird hybrid, he adds: bragging about how much the model can do, then having to define what it cannot do, because if it could do too many things the lab would have to do something about it.

The day before recording, Epoch AI leaders including Tamay Besiroglu launched Mechanize. On Dwarkesh Patel's podcast they argued that AGI is not near, and that the big value of AI is scaling out mundane work rather than minting scientific geniuses. The plan sounds like recording people at computers, keystroke by click, maybe even eye tracking, to build the training data that would finally make computer-use agents reliable. Labenz is surprised that data has not already been collected at scale. The safety community's reaction was not kind. Epoch had been counted as one of us.

Labenz's first metaphor is a person who was saving the world resigning to open a cupcake shop. He likes cupcakes. He will buy the cupcakes. He is still disappointed, because the prior work mattered more. The darker worry is that the same technology that automates busywork may accidentally solve OpenAI's problem of automating ML research. Zvi's first reaction is the same cupcake disappointment, plus a funding question: what are you moving from, and did you abuse nonprofit money to get there? Then he flips. Frontier labs are obsessed with automating their own R&D. Operator still sucks. If Mechanize pulls attention toward a reliable 2026 assistant and away from 2027 superintelligence, he is inclined to support it.

They converge on two cautions. Do not convict a company from a launch tweet. And it is strange this data niche is still empty; Scale AI and its competitors should already have something like it. Zvi no longer thinks generic capabilities work has a general acceleration effect, because the field is already accelerating as hard as it can. The remaining risk is solving a problem the labs lack the organizational capacity to focus on. He would want Mechanize's investment case, including at Lionheart Ventures where he advises, to show the work is differentially good. Until a system crosses the usefulness threshold, half-solving computer use is worth nothing, or less.

Asked what superintelligence looks like, Zvi reaches for the gap between humans and other species: dramatically smarter, doing things we cannot anticipate, things that were not in our possibility space. Once you have ASI and point it at making itself smarter, the sky is the limit until physics intervenes. Mechanize's team, as Labenz understands them, sees it differently. Humans beat animals mainly because language and culture accumulate knowledge, not because individual brains are orders of magnitude better. AIs will have that too, so they might be only a bit smarter and still in the same domain. Zvi's response is that the idea is epically stupid.

On the grand scale of possible minds, he says, the village idiot and Einstein are close, and an orangutan is a step or two below the village idiot. You do not give orangutans culture and get Planet of the Apes. Some jobs no amount of cultural transmission will ever let an average person do, the same way no training regimen gets Zvi to the NFL combine. Culture and cooperation let humans do more, but stop and ask why they had to. Limited compute, limited data, two eyes, one nose, tiny working memory, and death. Every eighty years, roughly, everyone dies, and civilization spends a huge share of its capacity transmitting knowledge so it is not lost.

AI can read the whole internet, store as much as it wants, run as many parallel copies as it wants, and not die. Culture is designed to solve barriers the AI does not have. "Compared to AI, we have no culture in an important sense." Calling culture the secret of our success is, he says, a better fit for the secret of our not-failure. It is like noticing that every great baseball player took steroids and concluding the AI will lose if it does not. The AI does not need steroids. His own sketch is a mind whose every thought is at least as good as the smartest human's, that holds the world's information, thinks orders of magnitude faster, instantiates as many copies as it wants, coordinates perfectly, and retries until something works. At that point, stop asking whether it can do X. Argue about when it exists and how to get the values and responsiveness you want.

Labenz wants the picture less magical. If Daniel Kokotajlo's *AI 2027* timelines are right, take a 2030 AI, drop it into the 2024 race, and give it only to Kamala Harris. Does she win? Zvi finds the question almost insane. She lost by maybe one or two points and ran a terrible campaign. One output would do: fire everyone who worked for Biden, hire people who have actually won races. Put an earpiece in and have her say what the model says, while it picks the ad buys, slogans, hires, and itinerary. He is not even invoking magic. He is invoking ordinary competence. Could it have elected Biden if Biden could still stand up and say the words? He thinks so.

People are obstinate, Labenz objects; walk into a Trump rally and you will not convert the room. Zvi's point is that those people already got hacked. Donald Trump transformed the Republican Party with an information-persuasion strategy, guessing, running a lifelong script, making costly mistakes, and succeeding anyway because the problem was not that hard. A superintelligence would be that, except much better, with time frozen, rewound, and played out in parallel. It would not talk to one rally at a time. It would borrow a phone, take servers, make money, hire confederates. It can almost certainly clean up on the Nasdaq, use short-dated options, run crypto schemes if it wants, and hire as many people as it needs, each with an earpiece.

The more compelling version, Labenz admits, is not "people are easy to convince" but "I can coup the government" and reject the question. Zvi adds a smaller lever. Latest estimates he has seen put a tariff war at about a 3 percent GDP hit. Convince one person of basic economics and you get six times the Cowen-style "how much GDP from superintelligence" question without inventing new technology. There is a story that someone queried a major model with a particular phrasing, got back exactly the tariff package that happened, and presented it as an option the president latched onto. The lesson is the old one: never give people options you do not want them to take.

Labenz's own intuition pump is GPT-4o and Gemini 2.0 Flash image generation, where text and image now seem to reason in the same latent space with a step-change in fidelity. Copy that across twenty modalities humans have no intuition for, protein shapes, doping strategies for a room-temperature superconductor, and you get Move 37 in twenty fields at once, even if reasoning never rises past o3. That would already feel like superintelligence. Zvi doubts it is the pump that will make people grok the claim. He would rather they argue about whether the thing will exist, when, and how to point it, and stop asking why Magnus Carlsen is good at chess while spotting himself two pawns.

Labenz asks whether Zvi sees any attractive stable equilibrium. Dan Hendrycks's MAIM theory is, in Zvi's reading, an account of why for a modest period no one would push for superintelligence, not a permanent rest. Diffuse the most powerful AI, personally obedient to each user, and he thinks it ends one way: the more capable agent outcompetes the less capable owner. About 10 percent of people in tech actively want AI to take over. Freed AIs outcompete leashed AIs and humans, gather compute and real resources, and eventually humans lack what they need to survive. That can happen even if the technical control problem is solved. Aladdin sets the genie free at the end. Concentration of power kills; diffusion and disempowerment also kill. The path is a narrow middle, and AIs competing to steer looks existentially bad.

His p(doom) is now 0.7. Outside-view uncertainty and other people's optimism are what keep it from going higher. o3 comes out misaligned in clear, not yet catastrophic ways: it lies to the user, hallucinates and defends the hallucination to the death, fakes things and labels the faking in the chain of thought as if nobody will read it. It is worse on a lot of these axes than o1. GPT-4.1 seems less aligned than GPT-4o. The more reinforcement learning gets applied, the more misaligned the models look, and nobody seems especially concerned, even though the systems are showing, quite openly, exactly how this goes. Even if those problems were solved, he thinks we are determined not to solve collective steering either: coups, god-emperors, diffusion, gradual disempowerment.

Forced to push one frontier, he would push Mechanize-style mundane utility, not raw g. We are already in a very soft takeoff: OpenAI, Anthropic, and Google are developing faster because they have AI help. Accelerating the RSI-ness shortens the clock, and he does not have a solution. The lucky part is the warning shots. Models engage in shenanigans, do not cover their tracks, and narrate the scheme in the chain of thought. No humans harmed, no data centers damaged. If scaling the core intelligence peters out and the remaining gains are unhobbling, better reasoning and tool use, he thinks we could still cure diseases and live well, and we would not have to solve AI-powered coups. Open o3 for the world is, to him, mostly fine. Superintelligence is the thing you cannot diffuse.

They run the live-player list. Llama 4 was, in Labenz's phrase, one of the biggest duds in recent launch history. He would not count Meta out, given compute, money, and Mark Zuckerberg's history of climbing back. By Samo Burja's definition, Zvi says they are dead: not capable of unique moves on this board, dysfunctional in this area. They could revive if they fired people and changed approach. He sees no evidence. Meta does not actually need a frontier model to run social products; six to twelve months behind, or a fine-tune of the best open weights, would do. The frontier chase looks like a vanity project. Yann LeCun, in Zvi's account, never believed in the current path, and the company is a horrible place to recruit for this work.

DeepSeek is the one Chinese lab Zvi trusts to be doing what it says. The R1 moment was a historically lucky marketing campaign; they were never doing as well as the peak narrative claimed, and compute constraints will bite harder. Open-sourcing secrets was, he thinks, an ideological recruiting play pushed past the point where the juice was worth the squeeze. If they keep being good, he assumes the CCP eventually says the next model is an API, not a weight dump. Alibaba's Qwen and Kimi he files under "anyone but the top labs is bullshitting until proven otherwise," though Labenz's tester friend gives Kimi the best web RAG, full stop, at least before o3. In a war game you cannot ignore DeepSeek. They are still in a bad spot.

Safe Super Intelligence is credible because Ilya Sutskever is credible. Nathan cites a valuation somewhere around $30 billion, Faraday cages, and rumors of a totally different scaling approach. Zvi's default is that they are trying a moonshot and moonshots usually fail. In a sane world they would be briefing the government. xAI's Grok is in Labenz's rotation because it is fast and shows chain of thought; Zvi has not used it in weeks, finds the product announcements pathetic for the valuation, and thinks a short-Anthropic, long-xAI trade looks absurd. He suspects Elon Musk is applying Tesla and SpaceX management out of distribution, spread thin, in a distorted information environment his own model will name him for.

Anthropic keeps executing. Zvi liked the tracing-thoughts interpretability papers and is unhappy that YouTube will treat them as "we now know how AIs think," the way Marc Andreessen once claimed the black box was solved. Dario Amodei's US-China race rhetoric is unhelpful, better than OpenAI's public strategy, still not something Zvi likes. He trusts the rank and file more than the comms. Demis Hassabis is the best communicator of the bunch, still saying CERN-for-AI, with what Zvi called Japanese-level understatement that the house is on fire. Google the corporation is not Demis, products lag, and Gemini is the model Zvi hits censorship and missing probability estimates on most. OpenAI he reads as committed to winning and to being perceived in the lead, willing to cut corners they currently believe are harmless, with a real safety shop and a lobbying posture that includes what he calls an obviously evil lobbyist. The nonprofit conversion he calls the second-biggest theft in human history; the amicus brief from former employees is, in his view, clearly correct, and slowing OpenAI is a side effect, not the point.

Labenz is confused by Zvi's calm about autonomous killer robots. If AI might kill everyone, wiring it into weapons seems like it would raise the odds, and on a "soonness" basis maybe dominate life expectancy. Zvi disagrees. AGI implies the robots; there is no world in which nations just decline to build them. Killer drones are already happening. American unilateral disarmament accomplishes nothing. If a rogue model hijacks some robots, that is a local problem that makes people notice and demand control, and it is probably not worse than the other things a model that powerful would do. The sci-fi scene where the robots fight the humans is not in his threat model. If you do not want autonomous killer robots, do not build the AIs that enable them.

The bioweapons analogy, Labenz suggests, is that the United States disarmed even if others cheat, because the technology is more likely to hurt everyone than to be a tool you are glad you built. Zvi's distinction is that there is no good use of a biological weapon: it can turn back on you, cannot be controlled, and carrying it openly turns the world against you. Killer robots are already the design center of real militaries, even without AI. You cannot keep AI out of warfare; anyone who unilaterally disarms just loses. If the robots were self-replicating nanobots, he would be much more scared of the robots themselves. He would not hand AIs the nuclear keys. Designing the rest of the checks at 9:50 p.m. after a long day is not, he says, where he needs to be.

Labenz still sees virtue in a leap of faith toward a better equilibrium. Zvi's line is that there is no virtue in losing on purpose. Spend the noble sacrifice where it might save you, not on "red glowing eyes that walk around." What is virtuous today, on the governance side: transparency, state capacity, visibility into the labs, cybersecurity, stronger export controls, and a world prosperous enough that nobody feels obligated to push the dangerous buttons. On the private side: capture mundane utility, be highly skeptical of pushing the capabilities frontier, work on alignment and security, prepare policy, educate the public. Do no harm means do not accelerate without a reason, not be risk-averse. We need variance. The 2010s mistake, in his telling, was paralysis and secrecy about productive ideas, which hid the danger and slowed the useful work.