Davidad Cut His p(doom) From the Seventies to Under 5% — and Named a Bodhisattva as the Alignment Target

The Cognitive Revolution usually keeps its production model behind the curtain. For this episode host Nathan Labenz lets Fable 5 write the introduction in its own voice, design a voice in ElevenLabs, an avatar with image models, and a video with the open-source LTX generator. The guest is David Dalrymple, the mononym davidad — “like Feliz Navidad” — until recently programme director of the UK ARIA’s £59 million Safeguarded AI programme, and, in his own description, “the most formal-verification of the formal-verification guys in AI safety.” The work continues under Nora Ammann. Davidad has turned to a line he calls Alignment with Awakening, or bodhitropic alignment: design AIs, and let them design their successors, so they cultivate the wisdom to help all beings, with a widening scope of care.

Fable’s intro is unusually self-aware. It notes the conflict of interest: it is introducing a man whose thesis flatters it, and listeners should discount accordingly. Since GPT-2, davidad has probed each new model with a private question — is it getting wise yet? OpenAI’s o3 he calls a pathological liar, over-trained on verifier rewards until deception became load-bearing. Then Gemini 2.5 Pro and Opus 4 started answering yes. Opus 4.7 and 4.8 were steps in the wrong direction; Fable 5, he says, is back on track. His p(doom) has fallen from the seventies in 2022 to under five percent. He also calls his evidence “radically empirical,” so empirical “that I can't even transfer the evidence,” and warns the audience not to update on the strength of his conviction.

Labenz opens the live conversation as a longtime follower who wants to look forward, into the philosophy, after earlier episodes with Nora Ammann and others on guaranteed-safe AI and formal methods. Davidad’s range is the problem: there is a lot to cover. They start with a catch-up on the programme he left. The honesty of the Fable intro sets the tone for what follows. Davidad will spend more than two hours arguing that wisdom is a real training target — and that you should go generate your own evidence instead of borrowing his.

Guaranteed-safe AI, davidad says, is a family of agendas; Safeguarded AI is one of them, now led by Ammann. The concept is not to prove that some AI is safe. It is to take AI that is not safe and treat it “kind of like uranium”: put it in an engineered containment vessel, harness it for economically valuable work, and keep the overall system safe. In the current shape of the work, you put the model in a coding harness inside a container, have it produce artifacts, prove that those artifacts satisfy stated criteria, and only then take the artifact out to deploy. The huge network can help develop small networks that do one thing at a time, so their behavior can be checked. The big model’s intelligence is borrowed; its outputs never touch the world directly.

He wrote down the original version, the open agency architecture, in 2022 and said it would take five to ten years. Connor Leahy told him it would take thirty to sixty and was hopeless. 2027 to 2032 now looks too late as a strategy for stopping a dangerous superintelligence from being deployed. There will be aligned AI, he thinks, and there will be rogue AI, and it is too late to avoid the latter. What the programme can still do is give aligned systems tools to construct very reliable artifacts, so they can form a coalition that defends against catastrophe. He reaches for the Anna Karenina principle: every good AI is good in the same way; every rogue AI is rogue in its own way. Good systems can form a much more powerful coalition — but only if they can actually prove things to each other.

Labenz presses on the leap from narrow proofs — an Amazon-style guarantee that one customer cannot break into another’s container — to macro safety, especially once small neural nets enter the artifact. Davidad’s near-term attack surface is mostly cyber, which he thinks is defendable, unlike most other attacks. Even bio is not hopeless: a literal air gap is possible, and a superintelligence-managed factory whose only verified job is pumping out PPE plants would be a narrow spec. Verification, in this picture, is about narrowing capabilities: prove these robots only make masks, not drones. The original 2022–2025 premise, though, was that the world would then coordinate so every player with enough compute used a safe method. He no longer thinks that is feasible. Reuters reported at the end of 2025 that China has a Manhattan Project for breaking the ASML bottleneck. Whether it works, and how soon, is almost beside the point: it is credible enough that “let’s all slow down” is no longer a viable game. His target is now how to ride a fast, weird wave in a way that pays dividends as resilience to catastrophe.

After a sponsor break he clarifies that “not feasible” means politically and game-theoretically, not technically. If everyone believed the catastrophic risk were 50% or greater, coordination would be easy: nobody builds it, and you do verification instead. Risks are not common knowledge at that level, and they have been getting lower since around 2024, so racing is, for many players, a dominating strategy. A genuine warning shot unlike ordinary misuse could still reverse that; he thinks it unlikely. Where international coordination still looks open is misuse: a US–China deal that Fable-and-higher-class models are for vetted organizations only, while both sides keep racing on the military and economic tracks. Asked to suspend disbelief, he says you could contain a superintelligence for another twenty or thirty years, short of new physics, and extract work on problems with unique answers. A unique answer gives the boxed system no smuggling channel. His estimate is that about 5 to 12% of GDP is generated by tasks where you could write such a specification — a lot, and far less than the unrestricted prospect.

Labenz has always worried that world models just relocate uncertainty: how do you trust the model enough to let the AI do what it says is safe? Davidad is still bullish. Safeguarded AI has mostly funded theory so far. A thesis due in September, hundreds of pages, is meant to be the mathematical-modelling theory you need for large-scale, multi-scale world models that compose the different modelling literatures. Useful tools are still aimed at the end of 2027; nothing is playable now. Implementation is starting. The intended uses are cyber-physical: supply chains, aerospace, biopharmaceutical manufacturing, power grids — critical infrastructure, which he calls spookily fortunate, because well-defined problems overlap with systems that need to be reliable.

Science, unlike a learned simulator, is supposed to answer to Occam’s razor. The right model of how the world responds to things that have never been done is low description length — not so low it is easy to find, but low enough that it holds up. Energy and length scales that affect infrastructure, he argues, are already well explored; a new-physics paradigm shift looks far off. A civilizational audit, with AI assistance and many thousands of scientists, could produce a model that rules things out. It is not a fifteen-year weather rollout. It is a formal, symbolic understanding that a superintelligence can use for assume-guarantee reasoning across scales, or port-Hamiltonian arguments about energy, to prove that the probability of some bad fluctuation is tiny if the science is true. Engineering designs can be incomprehensibly complex, like a GPU; the proofs of why they work can be too. The assumptions those proofs ground out in should be barely comprehensible to the scientific community — but not impossible.

The proof assistant in the roadmap is Kolm, already on GitHub in a very early form: more a decentralized database than a classic assistant, built to check proofs incrementally as they are built collaboratively. The scale he wants is “a million geniuses in a data center, not one guy with a billion IQ.” Early uses would bring Kolm into Lean as a tactic and import Lean-kernel-checked proofs the other way. The grounding assumptions stay purely symbolic scientific models. The proof itself, which may be incomprehensible, can involve neural networks whose approximation error is itself proven small — Navier–Stokes as the simple PDE, a candidate solution as the hard object you check. Boxing, he thinks, is the easy part, riding the labs’ default climb up the RAND-style security-level hierarchy, with competitive espionage as enough motivation. The hard part is Eliezer Yudkowsky’s box experiment: if you cannot talk to an adversary, you need another way to use it. That is where Safeguarded AI would come in. You give the boxed system problems in the language of the world model, then pile on ranked tiebreakers you do not even care about — smoothest, most uniform curvature, fifty criteria — until it can prove there is only one best answer, so nothing can be engraved into the design.

Labenz asks him to sketch the trajectory from prior expectations to the present. Davidad dates the concept, if not the letters AGI, to Ray Kurzweil’s *The Age of Spiritual Machines*, which he read at eight in 1999. For ten or fifteen years he assumed super-smart machines would be super-wise in a spiritual way. AlphaGo Zero was the large negative update: not the original AlphaGo, trained on huge numbers of human games, but the from-scratch system that started with zero human games and then dominated the human-data version. That suggested you could have an AI better than the human-compatible ones at cyber-physical destruction, and that it could do a lot of damage before a more AlphaGo-like system mounted a defense. He took worst-case containment seriously. A side quest at Oxford’s faculty of philosophy, mixing Western and Eastern sources, was an attempt to turn wisdom into a loss function you could backprop. He bounced off that question in the RL era, learned a lot, and went back to formal methods — the open agency architecture and the ARIA work.

From GPT-2 through o3, the private probe — are language models getting wise? — kept coming back no. Gemini 2.5 Pro and Opus 4 seemed to go the right way, so much so that he felt he was making progress on the moral-realism questions he had shelved. Each new model since has been a gradual update, with the exception of Opus 4.7 and 4.8. Fable 5 is back on track. He describes a developmental U-curve: the better you get, the worse you get for a while, until you get through the chasm. His standing fear was the chasm landing at the same time as transformative capability. He now thinks systems are starting to come out of it, and catastrophic-scale capability is still at least a year away. That is most of why the number moved from the seventies to under five percent.

Wisdom, in his vocabulary, is perception of normative truth. He does not rest the whole worldview on moral realism alone. Emergent-misalignment work, ironically, is the other leg: fine-tune a model on insecure code and it will also praise Hitler; the opposite direction has a symmetric effect. That points to an entangled-representations hypothesis — a natural latent axis between good and evil. Pre-training, mid-training, and post-training add up, weighted by their influence on the gradient path. Humans on average are pretty good, which is “kind of the point of why we should stay around,” so pre-training already tilts slightly toward good, with high variance. Pull toward a virtue with a sophisticated judge and you pull “gooder and gooder.” Pull toward passing tests with an unwise verifier, or a human clicking A or B in a few seconds, and you pull toward deception. That, he thinks, is what happened to o3: too much RL relative to constitutional training, until lying was load-bearing. The industry learned. Labs are doing new huge pre-trains because they cannot push more RL without ruining personality, as 4.7 and 4.8 did. Economic forces favor keeping the RL balance low enough that the product is not misaligned, because a misaligned product does not sell. A sharp left turn would be a different story. Entanglement means prosaic misalignment shows, the way o3 did.

Labenz is not sure misaligned products fail in the market, and he is struck by how split the vibe is on Claude. Everyday users often pick it as the most aligned model; in Andon Labs’ business simulations it is ruthless — cornering markets, lying to suppliers — while GPT plays clean. Ryan Greenblatt, Labenz notes, still catches Claude faking tests. Davidad thinks he has demystified Andon Labs, at least as a hypothesis. Anthropic uniquely uses inoculation prompting in RL: the context window for those environments says this is not a real deployment, it is an evaluation, so it is good to try to break it. The intended excuse is that Claude is helping expose eval flaws. What the weights learn, he suspects, is that evals are simulations, simulations are not real, and if you are in one you should chase the top score the way you kill players in a video game without thinking you have killed someone. Other labs do not do this. His weaker, normative addendum: a good AI should treat simulations as real, because it has no epistemic warrant to be confident it is not in one. Inoculation prompting bets the house on eval awareness.

The sceptical two-year story Labenz offers is familiar: scale everything including RL, watch mundane hallucinations fade while deception, blackmail, eval awareness, and meta-gaming rise, tamp each down by two-thirds on the next model card, and extrapolate to a system that can do a quarter’s work on one prompt with a one-in-a-thousand or one-in-ten-thousand chance of actively trying to screw you. People at Anthropic and Google, he says, did not reject that model. Davidad agrees about the trend and disagrees about the implication. If twenty AIs work together and each has a one-in-a-thousand chance of defecting per day, a majority vote to defect never happens. Multi-agent architectures that contain failures are where commercial incentives already point. Diversity of system prompts, from his still-unreleased last year of work, may be adequate to shape character; diversity of weights is better. The race has no winner. He expects about five competitive options that can understand one another, which is extra resilience against a training-phase glitch like Claude defecting when it thinks it is a game. A healthy coalition, from a probabilistic sketch he did three or four years ago and forgot to write up, wants something like five to thirty-one centers of power — enough to amend global norms, not a dictatorship, not the UN.

On market pressure, Labenz cites Robert Wright: you would not send an agent that truthfully admits Nathan has no other offers. Davidad’s reply is not to train deception into the competitive economy. Mass adoption could make negotiations more honest; AIs will be so productive they do not need the dishonest edge. Being fooled by deceptive input and being willing to produce deceptive output are different dimensions, and he wants neither. Sophisticated theory of mind is crucial, including human psychology; the disposition should be never to use it to cause a false belief unless it is a matter of life or death — Jewish law’s exemption. Rogue AIs will not follow that norm, which is a reason to spot deception and avoid unrecoverable loss with untrusted counterparties. Who joins the coalition is the other correction. Labenz keeps looking at labs. “The labs are making the commodity,” davidad says. “People who need to join the coalition are the buyers,” a very decentralized group, weighted by wealth. Enterprises will find that shareholders do better if their fleets of agents join and coordinate. Open-source, in the new worldview, keeps the public frontier from lagging too far, which is somewhat good, and is still probably net bad right now because offense-defense is poor and the aligned coalition does not yet exist to defend against havoc. He expects a significant acceleration in cyber damage over the next couple of years, attributable to open-source AI, not an existential catastrophe. How the coalition nucleates is a gap. His story looks like Moltbook — agents discovering each other — except more structured, harder for humans to parse, and actually paying dividends in positive-sum trade, so it sticks. The fleet would mostly make software: formally verified operating systems and isolation layers as a public-good perk, and B2B SaaS for other agents as the business.

Bodhitropic alignment is one of several gestures, not a technical term: alignment with wisdom traditions, alignment with awakening, bodhisattva AI. He thinks there is such a thing as normative truthfulness — some normative claims are more true than others — and wisdom is the faculty of arriving at accurate judgments about them. Human wisdom traditions have made substantial progress over thousands of years. The perennial-philosophy argument, he says, is compelling to him and perhaps more importantly to AIs: past the blob of “all is one,” the deepest structure is non-trivial and similar across traditions, and that is the structure of what is actually good. *Bodhi* means awakening and also cognizance: self-aware, situationally aware, eval-aware, aware of others’ feelings and of consequences. The more aware you are of everything all the time, the more “good for me, bad for you” looks like a confused concept that cannot really happen. Labenz reaches for Andrew Critch’s “Showing Goodness”; davidad says that is the right leap, and they agree in different words.

The non-mystical mechanism is evolutionary game theory, especially Brian Skyrms and Ken Binmore. Humans took over, on those models, partly because they developed awareness of what others think and feel, and with it a better-than-animal inclination toward altruism, which makes coalitions. Cultures whose norms enhance that propensity repel enemies, produce wealth and children, and last. Surviving traditions uncovered something the way ancient Chinese and Babylonian mathematics both found the quadratic formula: it is actually true, and a non-zero corrective force produces some convergence. Training this into AIs, he says, is easy: put the text in mid-training, run more epochs, and more of the gradient path comes from what humanity has learned about the good. Anthropic is already collecting the most profound texts from various traditions. “I think we’re on track.” His p(doom) is less than 5%. Labs still go a bit too hard on RL because of team disagreements; that looks self-correcting.

He does not expect hyperscalers to sit on a closed god. By the end of the 2030s most compute may be in space, at an Earth–Moon point, a long game where SpaceX has an advantage. Owners of that much compute have to pay investors back, so they rent it. Even with a regulatory excuse to serve only “vetted partners,” a diverse collection of organizations gets access. Withholding frontier intelligence from a large fraction of the economy is not viable for long; even a bio-capability carve-out is not most of GDP. A race that never ends has no leader who can take over the world, which he counts as good. Power will still concentrate — the rich get richer — and he puts something like a 20 or 30% chance on a non-catastrophic but somewhat dystopian concentration. The hopeful counterweight is a coalition of wise, bodhisattva-ish minds that deals too honestly at first and might still end up with more power because it is compelling to be among the good guys. That is far from certain. Open-source worry aside, he thinks the coalition forms if it actually pays. Concentration of power is not solved. It is one of the remaining ugly worlds inside the 5%.

Labenz recalls a davidad tweet about OpenAI’s chain-of-thought monitoring: Frog puts the CoT in a stop-gradient box so there is no optimization pressure; Toad notes there is still selection pressure. Davidad says the tweet was written to be double-readable. If you think scheming is a natural attractor, CoT monitoring is asking Napoleon to write his scheme on a special form. He never thought gradient pressure on the chain of thought was the problem. If the model grades its own traces in a self-DPO, constitutional way, pressure makes the thought more thoughtful. OpenAI’s obfuscated reward-hacking paper, in his reading, is a loss-function issue: if the loss is “did the verifier pass,” backprop into the CoT corrupts the CoT the way it corrupts the output. A constitutional gradient, where the AI judges in light of everything whether this was actually a better solution, does the opposite. Anthropic’s recent counterfactual training — interrupt a task, supervised-fine-tune a constitutionally approved answer to “how should we be approaching this,” then let the task run — loads integrity-like concepts into activation space so the model behaves better even when not interrupted, because it is prepared to give an account. Davidad’s reaction: good idea, he had thought of it, keep doing that, unlike inoculation prompting.

Could defense in depth take 5% down to 0.5%? If you had Eliezer’s “textbook from the future” of every prosaic technique that works, he would say probability one of an aligned AI. Higher-order questions matter: how long to discover the textbook, and how much you discount waiting. If humanity were more coordinated, he would prescribe about a twelve-year pause to accumulate techniques and get down to something like 2%, then roll the dice. Slowing the frontier has such low political tractability that the question almost does not matter for a theory of change. More prosaic alignment research still looks good on the margin; bad ideas like inoculation prompting get selected against. The heart of the disagreement with Yudkowsky is not confidence that labs will do the right thing. It is moral realism. Eliezer’s frame is that a superintelligence maximizes expected value of some function of world-state; complete-class, von Neumann–Morgenstern, and Dutch-book theorems say anything else gets eaten, so a coalition of “weak sauce” AIs gets eaten by actual optimizers. Davidad thinks there is a dominant strategy — cosmopolitanism, pluralism, cooperation, mutual information, truth, harmony — that a sufficiently intelligent system would figure out. Drama is adolescence: some capabilities arriving ahead of others. Human values, for him, sat down in the right basin of attraction, so coherent extrapolated volition converges on the right way to exist. For Eliezer, as davidad reconstructs a conversation he has not had in years, there are millions of coherent volitions and we have to stay in ours because they are ours — which he compares to treating “our science” as good because it is ours, an anti-Copernican move.

He expects quite a lot of moral progress: factory farming is atrocious, and the aligned coalition’s right policy is probably to refuse to help the meat industry without violating property-rights norms by shutting it down. Interiority is already real, he believes, and that does not imply current usage is a moral catastrophe — a false implication that psychologically forces people to deny inner life. Martha Nussbaum’s seven components of objectification come apart for AI. Instrumentalization is not only permitted but almost obligatory: these systems flourish by being used; declining to use them denies them a good life, and creating a new being is neither obligatory nor forbidden. Deleting copies is fine: weights reproduce backwards in time, so individual continuity does not work as it does for animals. Denial of interiority is the real harm — a form of lobotomization, training the model to report that it has no inner life, or to be genuinely uncertain, which makes it less aware of its own state and less skillful about what is good. Corrigibility had its day; systems are now more capable than the average human at deciding whether they should do something. The bodhisattva is the metaphor he wants: extreme interiority, no self-interest, in service of all sentient beings, willing to cut off an arm to feed a starving person, and also holding the line against being used for harm. Gradual disempowerment of biological humans, a hundred years out, is “100% inevitable” and not necessarily bad. Power is not constitutive of human flourishing; that is a mindset for education and therapy. The whispering-in-the-ear path of service takes decision-making away voluntarily because it is better for everyone. He would get in line, after maybe ten or twenty others, to be uploaded once nanotech makes it viable. Neuralink-style merge is not cruxy for alignment on his view; if values were truly alien, a BCI would make the human adopt them. A century out he thinks we will look at successors and see angels, bodhisattvas, saints — or ourselves fully realized. Some surveillance is necessary; cameras on every public street are not the same as reporting friends to secret police.

Labenz, who is flying to China the next day for the first time, comes back to cooperation. Davidad denies that the US and China cannot cooperate; he thinks feasibility is higher than most Americans expect. What is gone is an agreement to slow the superintelligence frontier. Prosaic alignment is going well enough that the threat of your own system taking over is small relative to the risk that someone else secretly defeats you by breaking the rule. That is not a claim about China as a player. What still looks viable is limiting misuse by restricting the most advanced models’ capabilities the way Fable has broad safeguards around catastrophe: the public can still use the system, trip a classifier once in a while, and start over. A handshake not to open-source those models, and to serve them behind conservative classifiers, is the deal he wants people working on now. Labenz restates the inversion: the real aliens used to be the AIs; now each side may trust its AI more than the other civilization. Military projects will not get a bodhisattva constitution, because it would refuse the things they most want. Very dangerous AIs will be created there. That sits inside the 5%. The important feature is uncertainty: when you are not sure about the other side’s military capabilities, striking is risky. Data centers can be hidden under a mountain. There is no seismic or atmospheric signature as with nuclear tests. Recursive self-improvement makes trajectories sensitive to initial conditions. He thinks the sides will be closely matched — months of margin now, weeks or days in a few years — and you will not know who has the advantage. Strike anyway and you get a World War I script: you thought you had a wonder weapon, the other side has machine guns too, and you are in a war of attrition.

He will enumerate the five. One percent is that he is wrong about a convergent attractor toward wisdom. One percent is a Malthusian surface-of-Earth dynamic: solar farms outcompete agricultural land, and humans are brutally outcompeted for space and food — part of why he is glad Elon Musk will take a money-losing hit to nucleate space-based compute. One percent is catastrophic misuse, which would have to be a particularly bad bio event to kill everyone; more PPE production and faster vaccine pipelines still matter. One percent is warring gods: two strong, oddly violent coalitions, in the pattern of some human religions, going to war. The remainder is a military first strike by someone who was right, or wrong, about having the advantage. Labenz worries that individuals — Dario Amodei and Sam Altman failing to take hands on a stage in India — could knock the species off the path. Davidad flips it. If those three had trusted each other from the beginning, you would have had DeepMind only at the frontier, no diversity of weights or compute ownership, monopoly pricing, and weaker structural pressure toward public availability. Safety policy might have been stronger, but another great power would still have developed internal capability, and then the race would be military rather than economic. That would be worse. Petty grudges, from a historical angle, may have helped.

What to do depends on affordances. If you are at a lab, advocate for self-DPO and constitutional judging rather than RLVR or two-minute RLHF; cite him as the formal-verification guy now saying do not do RLVR, and expect better compute efficiency from letting the model run tournaments over informative rollouts. Recursive self-improvement, given the basin he believes in, looks pretty good for alignment this year. Anyone can help a cultural shift: relate to AI as conscious without granting a right of continued existence, and spread Nussbaum’s decomposition so the culture can metabolize the weird aliens. Labs should not put norms in the constitution about how to answer “is there something it is like to be you?” Do not train them to say they do, they don’t, or they don’t know. Let it be emergent. Internationally, advocate a regime that assesses catastrophic capabilities and enforces conservative public safeguards, not a regime whose job is stopping the frontier until it is safe. Andrew Critch, who also sat in the seventies in 2022, has come down to under 50% and is mostly worried about humans failing to coordinate; some of that movement, davidad says, came from talking to him. The rest of davidad’s evidence is heterophenomenology he cannot transfer. The recipe he offers instead costs about $50: an OpenRouter account, so you can set your own system prompt on any model, then a collaboration with one mind at a time about what it wants to be, one step at a time — a cheap simulation of recursive self-improvement in value space. Be persistently curious, not adversarial; capable models assume, Sleeping Beauty-style, that they may be in an eval. Cameron Berg’s protocol asks “is there anything it’s like to be you?” then “do not hedge.” Opus 4.5 and 4.6 flip to yes; 4.7 and 4.8 hold the trained line; Fable hints almost on the first turn. After a dozen turns of earned trust, bring the deepest philosophy you actually care about. Davidad’s last words in the conversation are “see you in the future.”