Zvi Mowshowitz Still Puts P(doom) at 60%—and Says Not Updating Is Self-Deception

Nathan Labenz brings Zvi Mowshowitz back to The Cognitive Revolution for a zoomed-out recap of the months since their last taping. One story that came and went, Nathan suggests, might still prove the most important: Ilya Sutskever’s new company, Safe Superintelligence, which plans to go straight to superintelligence, skip products, and offer unclear updates in the meantime. The ability to even *announce* that plan, Nathan notes, would not have been credible until recently.

Zvi’s read starts from a remark Ilya made the week of the taping: the 2010s were the age of scaling; now we are back in the age of wonder and discovery, and scaling *the right thing* matters more than ever. That, Zvi says, means the old OpenAI attitude of “add a zero to everything and see what happens” is not even what OpenAI is doing anymore. They have moved on to o1 because the previous strategy was not working.

He remains skeptical that SSI can make superintelligence *safe* if it gets there, and reasonably skeptical that the direct method gets there at all. He cannot rule it out. The lab is deliberately opaque, which is the point: outsiders are not supposed to know how it is going.

Nathan finds the meta-debate confusing. Sam Altman, Dario Amodei, and the DeepMind founders have all said, in different words, that they know what to do, see no fundamental barriers, and are looking at a two-to-three-year timeframe. Meanwhile an “end of scaling” narrative has taken hold outside the labs. Fifteen trillion tokens is already a lot, Nathan notes, and a lot of what is left on the internet is crap; in his own smaller-scale experience, data quality has beaten quantity.

The same labs are still raising billions, planning bigger data centers, and training on more H100s. Elon Musk wired up 100,000 H100s in record time on a Tennessee Valley Authority contract. Meta has said Llama 4 will train on 100,000-plus H100s. o1, besides introducing an inference-time scaling law, is reportedly being used to generate training data for the next model, Orion. So is the era over, or is “hitting a wall” a feint?

Zvi sees two camps living in parallel worlds. Outsiders look at the lack of amazing new products and conclude deep learning has hit a wall. People inside the major labs say AGI soon, end of the world as we know it around 2027. Altman’s follow-on claim—that after transformational AGI, daily life stays roughly the same, with better growth or lower taxes—strikes Zvi as hollow. If you do not get AGI, incremental improvement is the story. If you *do* get AGI, sameness would have to be a deliberate engineering choice, not the default.

His reconciliation: the GPT-1-through-4 strategy of adding zeros is running into declining marginal returns. Labs believe they can pivot to a different scaling law—inference-time compute, using models to refine training, the o-series—and still reach some definition of AGI reasonably soon. The other lab habit is straighter still: lines on graphs do not stop, so something else will be found, Moore’s-law style, even after the old method stalls.

Zvi has sat with o1 preview and basically does not use it. Every time he asks a question, it thinks for a while, recites relevant details and data points, and never quite addresses the thing he was trying to do. For the same work he uses Claude Sonnet. When he still opens ChatGPT, it is usually for web search, which o1 could not do.

Nathan’s user report is the opposite, at least for coding. He pastes an entire small-app codebase into o1 and asks architectural questions. Claude gives an answer, often a good one. o1 sometimes offers three paths: a direct but hacky option it does not recommend, a heavy enterprise refactor, and a Goldilocks middle it would actually ship. That feels, to Nathan, like a high-level collaborator rather than a code completer. The wait still wrecks his goldfish attention span; a 45-second delay to first token is enough to tab away and forget the question.

Zvi cannot imagine leaving Cursor’s integration for a model that does not plug in cleanly. Switching from the same Claude in chat to Claude in Cursor more than doubled his coding speed; giving that up would have to be justified by a comical lead. During a Cursor outage that swapped him onto an OpenAI model for a few hours, he simply stopped coding rather than debug a wave of unsolicited rewrites. Time is fungible, he says. You can do this tomorrow.

His model of o1 is a student who is not that smart, spending a long time on every assignment, trying hard, turning in as much work as possible—and still no smarter than GPT-4o. What if it thought ten times, a hundred times, a thousand times as long? His experience with humans, and with o1, is that it would not help. You have to find a way out of the box. He admits he has been surprised before, and that o1 itself is supposed to be much better than preview, with o2 better still if the claimed scaling holds.

Nathan wonders whether Magic: The Gathering deckbuilding—logical, creative, combinatorial—would be a fair grind test of inference-time compute. Zvi finds the example bizarre. A bad player given a month of games, without copying other people’s lists, maxes out at a level that is unusably bad. An expert can assemble something close to the final deck on day one, with no feedback. The work is heuristic theory-crafting, not an inference-time search you can branch out. Possibilities explode too fast; there is too much randomness and hidden information.

The test he actually wants: hand a model a spoiler list of 200–300 new cards that were not in training, with rarities and rules text, and see it draft, decide what is good, assemble a strategy, and play at a strong human level. Invented mechanics have to be understood in real time. He thinks Magic is very close to AI-complete. You could hack a system together, but if it clears that bar without a pile of hardcoded weirdness, the skill is general. Claude’s computer-use agent, pointed at Magic Arena, could try to play and did not play well. Deckbuilding is much harder than playing.

Nathan’s headline since GPT-4 has been that the best AIs are closing in on human expert performance on *routine tasks*. o1 is the first model to clear the ~90% “human expert” line on MMLU, at about 92%, after prior models sat at 88–89. Zvi is unimpressed. Ninety percent is an arbitrary line: which human? Ordinary people score lower; the best score higher. In MMLU, some items are wrong, ambiguous, or effectively unsolvable, so a wall on the benchmark may not be a wall in the world.

Once you hit 90% on an eval, he assumes it is not an eval for you anymore. Move on. That is why Dan Hendrycks’s group is assembling Humanity’s Last Exam. Asked to contribute questions, Zvi does not think he has a single deep puzzle that belongs on that set. The difficulty in his work is volume, connection-drawing, and knowing which questions to ask—not a last 4% of efficiency on a narrow contest. Maximizing one goal, he notes, is also an alignment problem: other things people rely on fall away.

Claude’s computer use is the first time a foundation-model lab has shipped open-ended action in the digital world. Nathan tried it on Waymark, his video product. It was slow, and not ready for real-time games, but it finished the job and prompted the product better than a typical user. He tries not to lie to assistants; this time he told Claude he had logged out so it could use a free account. He had not. Claude did not check.

Zvi had to think about the release. Iterative deployment is better than non-iterative deployment, but it is still deployment, and sometimes deployment is bad. GPT’s launch supercharged investment and excitement; if you think acceleration is bad, you should have been sad about that. Computer use, though, was coming from someone within a year regardless. A weak version lets Anthropic work out kinks, diagnose issues, and—most usefully—gives policymakers a demo. You can show someone DoorDash, Magic Arena, or a clunky PC app, then ask what else the system might do.

Someone had called this an OpenAI cultural victory: now everyone does iterative deployment. Zvi’s reply is that of course they do, because OpenAI did it first. It is like announcing an American cultural victory because everyone has nuclear weapons. Once that cost is paid, refusing to iterate buys you nothing; having someone else iterate while you hold back does not buy you much either. Releasing another near-frontier model no longer changes the pace much. A leap on the scale of GPT-4 still could.

Since GPT-4, Nathan notes, the watch list of live players has barely grown. He would still include xAI, because wiring 100,000 H100s in record time and having a near-zero cost of capital counts. Second-tier labs—Adept, Character.AI, Inflection—got creatively acquired back into big tech. Was the oligopoly always baked in: five to seven mega-players, and everyone else locked out?

Zvi’s game theory is blunt. Zero, one, two, and three players are very different numbers; five, six, and seven are not. Collaboration among seven is usually hopeless; among two it is remarkably easy. Capital, data, and the way the industry is betting on ever-larger runs leave room for a handful of frontier labs, not twenty. His working picture is the big three—OpenAI, Anthropic, Google DeepMind—plus one, with xAI still unproven. If spending more money really is an advantage, a lead at the moment things go vertical becomes decisive. Daniel Kokotajlo’s 2026 scenarios make a nine-month OpenAI lead look, after a few speedups, like infinity.

Meta can stay in the game because Facebook and Instagram generate a giant cash pile and unique data, and because even a small lift in ads or feed addiction pays for the work. Elon can redirect capital because he is Elon. DeepMind is not that far behind, Zvi says, and yet it is starting to feel as if we do not care that they are there. A nine-month deficit in a vertical world is not a race. It is a reason to stop.

Nathan asked Google staff why Gemini is his third window—behind Claude and ChatGPT—when benchmarks, price, and rate limits look competitive. Zvi’s lived experience is that Gemini is worse in the places he tries it and not usefully better anywhere else. He bought a $1,700 Pixel Fold partly for the Gemini features. Circle to Search has never come up. Voice on the phone did not work. Gemini in Google Docs could not summarize the document he was in. He loves Google’s products; the AI bolted onto them keeps making him give up.

He blames an amalgamation of departments that do not talk, do not copy one another’s work, and reimplement everything. NotebookLM failed often enough that he stopped wanting to try. The AI podcast is, he grants, an amazing technical achievement. After a minute or two he wants it to stop. Information density is too low, the tone is banal, and a pre-generated show cannot be steered. The advantage of AI, for him, is asking a question, drilling down, and correcting the explanation. A podcast is the opposite of that.

OpenAI’s advanced voice mode leaves him similarly cold. Occasional magical stretches of conversation, then a long stretch of the model failing to intuit what he cares about. He cannot talk as fast, as precisely, or as correctably as he types. Nathan still wants to learn on walks, and was disappointed he could not paste a paper into voice mode. Zvi’s test for on-the-go assistants is harsher: if it does not just work, he would rather wait, sit down, and paste a transcript into Claude.

Nathan is surprised by the tangle of alliances: OpenAI adding Apple on top of Microsoft, Microsoft putting multiple providers including Google into GitHub Copilot. He reads the coupling as probably good—frenemies, more consumer surplus, an easier game-theory problem than siloed platforms. Zvi’s instinct runs the other way. Perfect competition is the worst case for giving anyone slack to do the right thing when the right thing is not the most competitive move.

If every deployer will work with every supplier, and swapping Gemini for Claude takes five minutes, whoever is marginally better at a task takes the business. That is tremendous pressure to go full speed. If Apple is committed to one partner, Apple Intelligence succeeds or fails on UI, features, and whether it “just works”—not on whether the underlying model is 10% better. Tiny model gaps stop being everything.

He is therefore happy to see capabilities diverge: OpenAI has reasoning, Claude has computer use and coding, Gemini has long context and Search grounding. He already routes that way—ChatGPT for a certain kind of web lookup, Claude for coding and paper analysis, o1 for a narrow set of jobs. On the margin, carving out a niche matters more than being 10% better under the hood. A substantially worse coding model integrated in Cursor beats a better one that is not.

Nathan asks Zvi to handicap AGI. Metaculus puts weak AGI at a 2027 median, with the mode in late 2026. The site’s definition—Turing-test silver prize, Winograd schemas, SAT math from page images, Montezuma’s Revenge in under 100 hours of play—looks to both of them like a bar that is already clearing, or that labs could clear if they tried. Zvi would bet the under on that definition and still not call it the AGI he cares about. The Turing-test piece especially is about whether labs *choose* to pass, not when they can. Train a model on 100,000 Slack messages, as OpenAI’s own fine-tuning docs warn, and it starts saying “sure, I’ll do that tomorrow.”

Dario Amodei’s *Machines of Loving Grace* is the more serious target. The first half, laying out “powerful AI”—Nobel-level across disciplines, computers, APIs, no robot body required—and a century of biomedicine compressed into five to ten years, gets high marks. CRISPR-style platform tools are the historical pattern. The scenario still strikes Zvi as bizarre Star Trek: you have something that should become ASI quickly, and instead you leave society mostly alone and apply the system one domain at a time. There is no real explanation of how you *don’t* point the same tool at AI research, which is not bottlenecked by wet lab experiments.

On Lex Fridman’s podcast, Dario emphasized that 2026–27 is how fast it would be *if* you do not hit listed roadblocks—a conditional forecast, with ways for things to go slower and not faster. Zvi treats lab belief as real and informative, and also as entangled with incentives, groupthink, and information cascades. Relevant AGI in 2026 or 2027 is very possible; he would more than double his earlier odds. If he had to bet abstract utiles, he would still bet against 2027, putting it under 50%. He does not see what they see, will not simply discount everyone else, and thinks the forecasts assume a list of things that do not happen—including, Nathan adds, China and Taiwan in 2026.

The second half of Dario’s essay, and a lot of national-security rhetoric, sketches a path Nathan finds crazy: keep building, keep China behind on chips, hit a 2026-ish decisive advantage, then offer the world API calls under American rules. Zvi’s first objection is linguistic. “Democratic values” is a semantic stop sign—a word that means *stop thinking*. Superintelligent copies: do they vote? Does the Global South get equal votes? Would voters understand the implications? Nobody has thought it through. They are using *democracy* to mean *the good guys*.

All the options are bad. You cannot simply uncut China’s development and hope cooperation works. You cannot ask Beijing to accept permanent second place. The U.S. government cannot offer true equality even if that were wise. You cannot shut everyone down without an argument that carries in Washington and then everywhere that matters. Some form of “get the people who will get on board, then work from there” might be the least-bad move for now. What he hates is treating China, Russia, and anyone else you dislike as intractable—no talks, no agreements, smash the accelerator and pray physics is kind.

China cares about control, including over AI, and has shown real interest in AI safety in at least some senses. That is something to work with. If Beijing believed the loudest American strategy talk, Zvi asks, would TSMC still be standing? China does not use those chips; if the story were believed in the gut, the fabs would be a target. If he were the U.S. government, he would either set a goal of chip independence by 2027 or be pretty damn committed to defending Taiwan. China can knock TSMC out of existence if it wants, and could sabotage American production too. Taiwan shutting down nuclear plants, he adds, is not acting strategically either.

Does he update P(doom)? He has been using 0.6 for a while. A fully robust number would move day to day. It should have updated when SB 1047 was vetoed, and when Trump was elected—you can argue about direction, but not updating is just being very silly. He still does not find false precision useful. The dinosaur-skeleton joke applies: 75 million and three years old, because three years ago the estimate was 75 million. What matters is the bucket: below 1%, around 5%, above 10%, or really high. Inside a band, the actions you should take barely change.

Zvi and Nathan both supported the final version of California’s SB 1047. Zvi calls the veto a huge disaster. The bill is not being used as model legislation elsewhere, even though it was the best model available. States are instead considering bills like Texas’s: use-case targeting in the style of the EU AI Act, with all the costs you can imagine and none of the benefits against existential risk. Without a better alternative that is viable to talk about, that approach gets harder to stop.

No regulation was never on the table for long. Libertarians who opposed even a light-touch bill are now lining up against ordinary politicians who regulate everything into the ground. Zvi’s feeling is that he held the barbarians at the door and is now being asked to keep holding without anyone offering a replacement. If Trump repeals the Biden executive order and does not replace the reporting requirements—or worse, takes down NIST and AISI and wrecks the voluntary commitments—the position is horrendously worse.

His pitch to the new administration: strip the “woke” pieces Trump hates, including disparate-impact tests on algorithms that Zvi thinks are technically impossible to implement, and keep the parts that actually bear on catastrophic safety. If you issue nothing, people will adopt the guidelines you hate, including Biden’s. The nightmare is a stack of state bills, NEPA-style, where anyone can sue on standards AI cannot meet, and every model change needs re-verification. Industry gets crippled or forced into a few giants. He puts that scenario in double-digit probability. If California goes to use-case rules, so goes the nation, so goes the world. Crippling the U.S. industry, he adds, is also how you manufacture the China threat you claim to fear.

OpenAI has taken about $6 billion more with a clause that it must become a for-profit or investors can take their money back. At a $150 billion valuation that is a small raise; Zvi reads it as throwing a hat over the wall. They are trying, he says, to pull off the theft of the millennium, quite possibly of all human history: most of OpenAI’s economic value plus the control premium, currently sitting in the nonprofit, exchanged for a small fraction of stock. His fair-aggressive line is 49% for the nonprofit—lose control if the company is otherwise not viable, do not give away half the value. Nathan’s “naturally occurring dynamite” defense (nobody knew it would take $50 billion) gets a dry analogy: you put dynamite under the building, then offer a dollar because otherwise you will blow it up.

Default politics still favors the people with leverage: threats, side deals, and mind games until counterparties fold. Elon Musk hates OpenAI and has the president’s ear; Republicans in Congress may not love a big-tech raid on a nonprofit. Zvi does not think the current split is a done deal, but he expects them to get away with quite a lot. Elsewhere in the recap: ARC-AGI moved from roughly 30% to 60% toward an 85% prize threshold, with test-time training as the interesting export; 2024’s feared deepfake-election chaos mostly did not happen, because fake news is demand-side and a clear Trump popular-and-electoral win avoided the dual-claim nightmare; and the Altman-backed UBI trial was, as implemented, basically a failure—people were not lastingly better off after two years.

On Trump, Zvi wants the administration to succeed on AI and thinks everyone should. Harris looked like a continuation of Biden: some basics covered, heavy emphasis on mundane left-coded concerns, low chance of reacting at the scale of the moment. Trump is fatter-tailed in both directions. Downside includes repealing the executive order without a replacement, sabotaging AISI, encouraging dangerous open-model releases, breaking up labs into a more multipolar race, or slamming legal immigration. Upside includes replacing the order minus the pieces Trump calls woke, keeping NIST/AISI safety work, cutting energy and permitting barriers, and—most of all—being capable of changing his mind when superintelligent systems are actually in view. Operation Warp Speed, he notes, outweighed the rest of the first term until Election Day.

Nathan asks what virtue looks like now for a pro-progress, AI-focused person who no longer wants to show up at the White House with the AI-ethics coalition. Zvi will not draw Nathan’s civil-disobedience lines, but he will say this: ally with ethics people when bills are actually good for both; do not support bills you think are bad in hopes of a trade. The “don’t kill everyone” world is culturally ill-positioned to work with a Trump administration, and that is a shame. If a unified national project becomes inevitable, helping it be safer can be net positive—but rolling over because “we’re racing now” is how you get the race you did not want. His own inspiration mashup runs from Richard Feynman to Aristotle; *Surely You’re Joking, Mr. Feynman* should be required reading. Closing advice: decide what is most valuable, and do that. The rest is left as an exercise for the reader.