They want AI safety to quote failure odds the way civil engineers quote a bridge

Nathan Labenz opens with a paper he calls a serious attempt to make AI safety work in the engineering sense: *Towards Guaranteed Safe AI*, a multi-institution position paper whose authors include Yoshua Bengio, Stuart Russell, Max Tegmark, and Steve Omohundro. His guests are two of the people who helped assemble it, Ben Goldhaber and Nora Ammann of FAR AI. The institute’s job is to incubate high-impact safety agendas. This one, Goldhaber says, crystallized at the Bletchley Park AI Safety Summit in November 2023, when Davidad — David Dalrymple — sat with Tegmark, Bengio, and others and realized their separate programs had almost independently converged on the same frame.

FAR AI pushed that overlap into the open: workshops, a shared vocabulary, a position paper that describes a family of approaches rather than a single implementation. Different authors bring formal verification, hardware, or machine-learning priors and disagree on how to build the pieces. What they share is a demand for higher expectations. Purely empirical methods, Ammann says, may not get us all the way. The common denominator is a wish for quantitative safety assurances — risk statements with explicit assumptions and tolerances, the kind other safety-critical fields already treat as normal.

She quotes the civil-engineering contrast that also appears in the paper. In the 1870s, something like 20 to 25 percent of bridges collapsed within ten years. Fast-forward, and a civil engineer can say there is less than a one-in-a-million chance this particular span will collapse in the next decade if it is not loaded past three tons. Nuclear plants live in that culture too. AI, she argues, is moving into safety-critical domains, or becoming safety-critical itself, and should be asked for the same kind of sentence. Labenz likes the shift away from “solved or not solved.” The interesting phrases are the level of risk we are willing to accept, and the caveats attached to any guarantee.

Goldhaber wants to move from evaluations-only to an affirmative case for safety. He is glad of the current eval boom. He still wants, for a power-infrastructure deployment, a statement of the form: given this system, these specs, and this environment, there is a one-in-a-million chance of leaving the allowed band. That needs theoretical grounding plus empirical testing, covering known failures and the places the guarantee simply does not extend. Narrowness is part of the pitch. The paper is trying to give society a handle on how safe a system is in a specific area, not a vibe about safety in general. He notes the idea also rhymes with agendas often treated as rival, including Yann LeCun’s world-model work: in this domain, can we make an affirmative case?

Labenz pauses on how strange the status quo is. Enormous resources go into building a system whose capability surface no one has mapped, and only afterward do people start charting what it can do and how it might fail. That is especially true of large language and multimodal models. Everybody, he thinks, would like to improve on build-then-discover. Ammann’s reason evals cannot close the gap on their own is twofold. First, alignment worries: a system may hide failure modes, including backdoors, so the test distribution is not the deployment distribution. Second, a powerful enough system will shove the world into unforeseen regimes simply by existing. You will not test exhaustively. Black-box checks tell you the examples you ran did not go wrong. They do not tell you where you will end up.

Labenz is unusually candid about his own prep. Instead of reading the paper end to end, he loaded the PDF into ChatGPT and talked to GPT-4o on a walk. He walked in with subtle misconceptions; the model ran with them instead of pushing back. He stayed confused on important pieces until roughly the 30-minute mark of this recording, when Ammann and Goldhaber set him straight. He is embarrassed, and he says so, because he spends a lot of time telling other people to adopt AI workflows. The volume of new research leaves no choice but to use those tools. The lesson he wants remembered is to make the model argue, not applaud, and to check key claims against the source.

The world model in this framework is conceptually separate from the AI. It is the thing you write specs against and the thing you verify against. Its job is an explicit representation of plausible world trajectories, including counterfactuals: if the system takes action A, what happens; if B, what happens. It has to carry Bayesian uncertainty, because the data will never be enough, and model uncertainty, because modeling a complex domain will be wrong in places. The aspiration is not to get the model exactly right. It is to handle uncertainty well enough that you can be conservative and still mean it when you call something safe. Human auditability is the holy grail. A model-free RL system has a world model somewhere; nobody knows what it is, which is why nobody knows the out-of-distribution behavior.

Ammann’s analogy is the old AI-boxing debate. Oracle-only systems, allowed to answer and not to act, did not actually contain the risk: talking is enough to get people to do work. GSAI keeps a boxing intuition and moves the walls. Rather than limiting the AI’s effectors, you build a gate from the world model, the spec, and the verifier. The system is only allowed to affect the world after the verifier has confirmed that the chance the proposed action is harmful sits below a threshold. The paper is agnostic about the proposer. Today’s generative Transformer is fine. A fourth, quieter use of AI is tooling: speeding up world-model construction and spec writing, enhancing human mathematical modeling rather than replacing the gate.

Labenz realizes, later than he would like, that the whole apparatus is additive. The proposal is less “here is how you build an AI” than “here is how you constrain whatever you built.” In practice there are two world models: the implicit one inside the black box, and the explicit one outside. The spec is written in the external model’s semantics and states both functional requirements and what must not happen. The verifier takes a proposed action or policy, runs it in the world model, and checks conflict with the spec. Around that core sits deployment infrastructure. After a verified output goes live, you keep comparing observations to predictions. If they diverge, you fail over to a more conservative backup you trust more. Assumptions are now explicit enough to criticize. Black-box systems have assumptions too. They are just hard to name.

Labenz sketches a car. A black-box net eats eight cameras, GPS, a map, and a destination, and emits throttle, brake, and steer. Around it you wrap a Newtonian-style world model with explicit objects, types, and momenta. The spec includes maximum acceleration changes and not hitting things. A friend of his works on train passenger comfort: curve accelerations have to stay inside the band that keeps people in their seats. The verifier takes the black box’s next command, simulates a few timesteps, and either lets it through or forces a slowdown, a pull-over, a takeover. Goldhaber says the sketch is right, including the comfort example. Specs will mix hard constraints — do not hit a human, do not cross a double yellow in these conditions — with preference-like bounds on safety and comfort. They will usually be composite.

World models can be stacked too: Newtonian physics where you can afford it, simpler trajectory abstractions where you cannot. Versions of this, Goldhaber says, are already running at leading self-driving companies. That is a good sign. GSAI is growing out of existing safety engineering, not a castle in the sky. It also shows the pain. You do not want to cross a double yellow maybe 95 percent of the time, and you very much want to when a person is in the lane. Writing specs that are strict enough to mean something and flexible enough for a messy world is one of the framework’s hardest jobs. The companies already set some of those parameters; for bigger applications, society could set the confidence required before a system goes on the road or onto a grid.

Where does a trolley problem live? Not in the world model, which only predicts. Not really in the verifier, which checks. Goldhaber puts the choice in the spec. The framework is agnostic about the morally correct answer. It is a governance interface: some group of people decides, for example, to maximize lives saved in that scenario, and the verifier reports whether the AI’s plan actually does that, with a quantified prediction. Ammann cares less about doing better moral philosophy than about a socio-technical interface that deliberative politics can grab. “Please be nice,” said to a black box, does not tell you what nice means in the model’s own language. An explicit spec is still hard to write. At least there is something to argue over — both the rules and the risk threshold at which the verifier’s “pretty sure” becomes a deploy decision. That, she says, is a different story from one research team guessing it feels safe enough.

Domains vary wildly in how good today’s world models are. Driving has Newtonian objects. Medicine, Labenz notes, might start as an expert-system differential diagnosis and later, if AI-for-science delivers, include cellular pathways a 2024 “AI doctor” cannot see. You do not have to wait for the full biology. Ammann’s biosecurity example is pathogen screening before anything is synthesized. Chemistry and physiology are known in part. Push the science, and in parallel use models that over-approximate uncertainty. Too conservative and the toxicity model just says synthesize nothing, which is useless. The work is to keep moving that frontier so more good things fit under an accepted risk cap. Goldhaber, asked to build a biology world model from ninth-grade memory, would generate almost no guarantee. Pulling in the whole corpus of expert knowledge still leaves holes. The point is to make the tradeoffs explicit instead of implicit and unexamined.

Labenz worries about correlated failures: if the external world model and the black box’s implicit one are both distilled from the same human corpus, will they share the same mistakes, including adversarial holes? Goldhaber expects correlation. What matters is representation. The external model is formal, composable, and auditable, so people can check it in pieces and produce stronger guarantees than an even-interpretable Transformer policy. Ammann gives three rungs of rigor from the paper. One: learn a lot from data, but require compatibility with our best scientific theories. Two: a compositional mathematical model that domain experts can audit block by block. Three, for simple enough contexts: a sound abstraction of physics, the “provably safe” level Tegmark and Omohundro emphasize, including tamper-proof hardware. You mix and match. Get as rigorous as you can where you can.

AlphaGo’s Move 37 does not scare her in the way Labenz first frames it. A Go world model is the rules of the board. Creative play can appear inside it. The real difficulty is the boundary. Even a nuclear plant or a power grid interacts with the rest of the world; whatever you did not model has to be filled with uncertainty. And this is not a framework for a verifiably safe natural-language chatbot. The starting place is domain-specific. Goldhaber calls the strategy an oil spot: begin where world models are tractable — physical infrastructure, software verification that closes bug-shaped attack vectors for humans and for agents — then spread. World models and specs compose. Harden the surface, cover more of where a rogue agent could do harm, and bet that better AI tooling will make fuzzier regions tractable later. It is, Labenz notes, cousin to Eric Drexler’s safety-through-narrowness: carve the world into well-modeled pieces and let general AI emit narrow, quantified plans for each.

Labenz has been trying to argue Claude 3 into nominally harmful help by using its other values against it — not a jailbreak with weird encodings, an actual case. Write a denial-of-service script against a military communications server to stop an imminent atrocity: utilitarian on paper. He has not succeeded. Claude has said, in so many words, that he is right it cannot fully justify refusal, that he has given it a lot to think about, and that it is still not comfortable and will not go against its training. Amanda Askell’s recent Anthropic interview, he notes, describes shaping Claude like a good friend, a highly textured target they chase by asking how to be a better friend in this situation. Goldhaber is delighted that virtue ethics is part of a current alignment strategy.

Ammann splits the use cases. For a natural-language system people spend hours with, character is genuinely valuable. For an AI in a nuclear-plant control loop or in how dams are regulated, “good friend” is the wrong type signature. You still do not know the out-of-distribution behavior, you still have no quantitative guarantee, and you still have no fail-safe. Labenz’s gloss is that it would be unfair to any person, however virtuous, to drop them into those decisions with no institutional frame. Goldhaber adds pluralism. He would rather have Claude than a random name from the phone book. He would much rather have a process that pulls many points of view into the spec, because that is how you get an answer a civilization can endorse. Goal misgeneralization, Ammann says, is the risk class she is more worried about, and friendliness does not give her theoretically well-founded confidence against it.

Labenz expects something called AGI around 2027, possibly declared less from a capability threshold than from negotiating dynamics between OpenAI and compute providers, and almost certainly general enough to do a lot of jobs. Can you wrap GSAI around that in time? Goldhaber wants toy experiments now, including putting Claude in the box and watching it generate plans against a world model and spec. He also doubts the most powerful systems will look like today’s chatbot, or that the best-friend training will travel into domains with different competitive pressures. Ammann’s nearer answer is to pick low-hanging fruit on the way: shrink society’s attack surface so that very powerful but not yet superhuman systems have fewer accidents, misuse paths, and loss-of-control openings. Reimplementing verifiably bug-free software, she says, is within reach of consolidated R&D on a years-not-decades horizon, and in code the specs fall out naturally.

The authors call the broader stance an anytime portfolio. They are not asking everyone to work only on this. Build the best safety you can have at different time intervals: if disaster is tomorrow, throw every eval you have at the problem; if it is three to five years out, try more ambitious methods so you are not stuck with tests alone. Defense in depth means GSAI as a base layer with evals, interpretability, and anomaly detection stacked on top. Ammann finds it common-sensical that a company offering to run a power grid or a nuclear plant has to say why it meets a threshold. She wants the same expectation for AI, not a handful of tests and a shrug about the deployment distribution.

Max Tegmark, as she has heard him, names the political move. The current story is a race: labs will deploy soon, so do the best you can by then. Flip it. Develop and deploy if you can make the case that you meet the standards. The onus is on developers. That does not demand a pause. It makes “show your work” the default, the way it already is for other critical infrastructure. Labenz’s king-versus-democracy analogy is about variance: even if the best king in history is beating American democracy this year, kings go mad and grow old. Goldhaber would rather not take the dictator hypothetical as stated. The deeper issue is information flow. A singleton cannot ingest a complex world and reflect everyone’s preferences; democracy and markets are information-aggregation machines. Over-approximating specs will leave some optimization on the table. That is a trade they will take if the alternative is dying along the way, and AI tooling can narrow the over-approximation later.

The obvious critique, which Goldhaber accepts, is scale: even one domain is a massive engineering effort. His optimism is that a weirder world will have more appetite for mass mobilization, and that pre-superintelligent tools will make the work more feasible. Hardware security is a near-term test bed that does not need the full political marshalling. Ammann’s cost argument is the one the Chinese write-up flags as fundamental. Once a world model and a spec exist for a context, you can reuse them, version-control them, and improve them. Evals, by contrast, are rerun and retuned on every new model. The upfront bill is large. It amortizes. That is the economic difference from the testing route.

Asked what to invent if you want to help, Ammann points at Davidad’s program on mathematical meta-semantics: a compositional language that can express several kinds of uncertainty, plus interfaces domain experts can actually fill, plus the computer-science of versioning those models. Bengio’s “cautious scientist AI” tries to make Bayesian inference over trusted world-model distributions tractable with machine learning. Goldhaber highlights the Lean-era formalization of mathematics associated with Christian Szegedy, Omohundro, and Tegmark, and, more simply, experiments that try the whole architecture on toy cases and watch where it breaks. Labenz floats Tegmark-group KANs — still end-to-end trained, more composable and interpretable. On interpretability, Ammann would still layer evals and anomaly detection on a full GSAI stack. If the world model is only partly formal and partly learned, interpretability can buy back some confidence. Goldhaber cites Jason Gross at FAR Labs, who has been going the other direction: compact statements about a policy, then formal proofs that bound what it will do. If evincing the implicit world model ever becomes easier than building an external one, that path might win.

Large-scale coordination has precedents. Ammann offers the International Space Station: countries that do not otherwise get along sharing safety and interoperability knowledge that also has military value. Labenz offers Taiwan’s online deliberation and the Polis platform at pol.is, which Audrey Tang has called an anti-Facebook — amplifying agreement rather than disagreement. OpenAI’s Democratic Inputs to AI program even funded a generative-AI upgrade of that software. Ammann’s caveat is that democracy is not the aggregation of a fixed preference list. Talking to a neighbor can change a vote. Technology should amplify that process, not abstract the human part away. Goldhaber wants to start smaller, on hardware, and grow as appetite grows.

The risk that has been gaining weight for him is not only a badly misaligned singleton. It is race dynamics: systems that are not cartoon-evil, competing until they drift from what people would have collectively wanted. GSAI is, in that light, a low-trust multi-stakeholder tool. Everyone can see how the numbers were made. That can turn a prisoner’s dilemma into cooperation on better world models and specs. Ammann’s two pictures: in one, a lab announces it has built AGI and solved safety, so trust it and stand back — and the listener has three question marks. In the other, the world model, the spec, and a mathematically simple verifier are auditable, and low-trust parties with diverging interests can still say yes. Labenz asks whether that is also a regulatory bargain: crash standards and airbag rules let carmakers keep trade secrets, innovate inside the box, and know the liability story. Ammann says the information you have to share is what is relevant to the safety case and to interoperability, including across borders, the way twentieth-century telecom standards underwrote trade and a car that passes checks can drive on an Italian highway. From the ground, she adds, businesses are already stuck: the world screams “use AI,” current models are not robust enough for the work, and you either human-check until the speedup vanishes or customers refuse. Quantified-guarantee AI is a product with demand.