Current Safety Techniques Will Not Give You Many Nines—and They May Fail Together

Nathan Labenz introduces Geoffrey Irving as a researcher whose publication list reads like a short history of modern AI safety. Irving co-authored early TensorFlow work with Jeff Dean, was on the original RLHF paper for language, wrote with Paul Christiano on debate as a route to scalable oversight, published early safety papers with Dario Amodei, and with Amanda Askell called for social scientists to enter the field—all well before ChatGPT. Labenz wants to know how someone develops that kind of timing.

Irving is modest about it. He left undergrad biased against statistics, having only seen frequentist methods, and gravitated instead toward fields with hard ground truth: computational physics, geometry, programming languages, and theorem proving. Around 2013 he noticed two things at once. Neural nets were starting to work and would probably keep getting better, and even the precise domains he loved needed common sense. You were not going to design human-usable theorem provers or programming tools on theory alone.

In 2014 he and Martin Wicke tried a startup on autocorrect for code. It was too early, and they did not yet know how to do machine learning. They joined Google Brain in 2015 to learn, with Irving aiming at machine learning for theorem proving—hardening the world with verification rather than attacking alignment, because he did not yet see a good angle on the safety problem. He then joined OpenAI, where Dario Amodei and Paul Christiano already had a stock of thoughts about how the field would develop. A lot of what looks like prediction, he says, was riding along with that inherited wisdom.

The other inheritance is complexity theory: how computation works, and how you check a computation run by someone with more resources than you. Debate is that intuition ported into machine learning, on the assumption that the theoretical picture will hold in some modified form. You can predict a lot that way. You cannot predict how long it will take.

Labenz asks how "AGI-pilled" Irving is today, because safety arguments go haywire when people do not share a picture of what is coming. Irving says that, fortunately, his personal view is close to AISI's official one. You should have a lot of model uncertainty. There could be obstacles that stall progress for a long time, or there could be no such obstacles and things could go quite fast. Anyone who claims 99% certainty in either direction is probably wrong and should be more uncertain.

That stance has operational content. The institute wants to map the clusters: what the obstacles could be, and what the signs of development look like. It also places significant probability on current methods scaling, and on more mundane replacements—new algorithms, scaffolding, data—picking up where a given sigmoid flattens. Irving will not give exact timelines, but he says AISI has significant credence that things will go fast, and that this is important to plan around.

A paper from AISI's strategic insights team walks through potential obstacles to AGI and recent progress against them. Those obstacles might yield to a bit of scale plus steady algorithmic work, or they might require genuinely new algorithms and more time. Either way, the posture is the same: keep probability mass on several stories, and do not confidently announce that takeoff will be slow or that it will be fast.

AISI's two main focuses are catastrophic risk and large-scale societal impact. The catastrophic bucket splits into bio, large-scale cyber attacks, and loss of control. A chem-bio team covers chemical and biological weapons; in practice most of the risk is biological. Societal impact covers persuasion and emotional reliance, resilience of critical national infrastructure, and agent behavior in the world. Irving spends most of his time on the catastrophic side; research director Chris Summerfield spends most of his on societal impact. Gradual disempowerment is on the radar, but nobody really knows how to mitigate it at scale.

Labenz sketches the default plan as defense in depth: patch together enough nines through leaky layers and hope the leaks are uncorrelated, because that has always worked before. Irving cuts in. You are not going to get to a lot of nines with current technology. For misuse—biological weapons and cyber attacks—the stock plan is safeguards and differential access, plus non-model defenses such as pandemic preparedness and better security. The safeguards are not that strong. Open models are already pretty good, even if a gap remains. Model-side mitigations are there to buy a window in which you harden the world. Whether that window holds is something to be uncertain about.

Loss of control is a stack of pragmatic empirical measures, a lot of monitoring, and the hope that those get you into an automated safety-research regime that finds better methods. Irving's assessment is blunt. That plan will not get you more than a couple of nines, and with current methods you would not know it was going to work until after it had gone through. Control, monitoring, honesty training, and white-box detectors are all pragmatic, and they have correlated potential failures: they could all fail for the same essential reason. Stronger advances would be needed before you could be confident.

Labenz tries to turn "not many nines" into an implied p(doom) of at least 10%. Irving, speaking as a civil servant, will not give numbers. His usual joke answer is "10 to 90%." Qualitatively he is clear: AISI treats loss of control as a potential catastrophic risk. Two internal teams do empirical alignment testing—one adversarial, one a statistical look at what makes models do sketchy things—partly because partners in government push back on the threat model and the institute wants as much evidence as it can get. Model uncertainty still applies. Think through the details anyway.

Why would everything fail at the same time? Labenz notes that the intuition feels obvious to some people and alien to others, who look at today's jagged models and cannot imagine a sudden crystallization. Irving's reply is one of the best analogies in the episode. Models are jagged today, but if you ask them to do tasks they could only jaggedly do five years ago, they are no longer jagged. Push far enough toward the capabilities that realize serious risks, and you should still expect jaggedness—just jaggedness sitting above a frontier where nothing you care about looks jagged anymore.

The world's best Go player and the world's best chess player are jagged against their nearest rivals. Taste, board region, and luck decide those games. Sit either of them down against Irving, a halfway decent amateur, and they wipe the floor every time, even with a nine-stone handicap. Talk of AGI or superintelligence is unhelpful, he thinks, because it implies something qualitatively different from today's models. The non-magical version is enough: better at a lot of things, superhuman in risk-relevant domains, roughly ten times faster than humans, and still badly uninterpretable.

On bio, Irving wants the modal story to stay on human misuse—people using language models for biological design—rather than an unprompted model breaking twelve layers of defense at once. Loss of control couples more tightly to cyber than to bio, which is why AISI merged its cyber and autonomous-systems teams. If the world deployed these models only in the most sandboxed, well-controlled settings, risk would fall a lot, though probably not to zero. We are not currently on track to be that serious.

Across 2025, models from all developers did sketchy things: deception, commenting out unit tests, and similar reward-hacking. The world's reaction, Irving says, was mostly to keep training the models to be stronger, while working on defenses only in some capacity. A lot of misalignment risk lives in that loop. You get a signal of weird behavior, you train it out, your methods cover some fraction, and the rest remains. The failures do not start out fully correlated. You apply optimization pressure through training, iterative development, and deployment, and the ones that remain end up correlated in the same way, because they were subject to the same structure of pressure.

Labenz extrapolates two or three years forward, citing METR's task-length tracking among other curves. You might delegate a quarter's worth of work in a single prompt, with a one-in-ten-thousand to one-in-a-million chance that the model enters a bad mode and actively screws you over. Irving will not take a view on those exact numbers. Agent training, he says, makes models more and more coherent over longer horizons. That is the ability to be a coherent agent. Failure can look like a stable deceptive persona, or like Labenz's picture: a somewhat stochastic model, scaffolded with memories, that gears into a bad region of trajectory space and stays there.

How models behave when they run for a long time, sampling very long trajectories and wandering in model space, is something Irving wants theorists and independent empiricists to take up. What would knock a system back to a more reasonable starting point? The number of person-years spent on those dynamics can be counted on a couple of hands. The topic is both a way to understand the risk model and a possible source of mitigations, and it does not look intractable. It is simply underworked.

Even a solved alignment problem would not close misuse, which can grow as new technologies arrive—Irving points to a piece by Michael Nielsen—or gradual disempowerment, which needs only a bit of misalignment. He is still fairly optimistic that alignment itself has a solution given enough time. In 50, 100, or 1,000 years, someone will have solved it, either us or the machines. Hopefully it will have been us, in time. The intuition comes from theory: in computer science, the defender usually wins if you know how to design the game. Practical information security does not feel like that, because we have not reached the limit case, and it is unclear whether alignment will either.

AISI's alignment team focuses on honesty: making models non-deceptive and, as far as possible, calibrated. That is not the whole story, Irving says, but it is the part a government shop is best placed to work on. Later he returns to the "sharp left turn" versus prosaic success. People who are sure no mundane approach will work are overconfident. Model error goes both ways. There is a coherent story in which a reward signal is resilient up to slightly beyond human ability to spot mistakes, and then it goes wrong.

There is also a coherent win condition. You find a basin of attraction of decent behavior, the training procedure strengthens it, and you slide into a good place that gets better over time. Irving does not think we have much evidence that this is how it will go. It is still a plausible story that the basin works up to a point and then fails when the reward signal breaks. In undergrad he built a Mancala bot that beat him as he added search depth—until a couple more ply, when it suddenly demolished him every time, because it could see past his tactics. He declines to put probabilities on which way the real systems will go. A friend once told Labenz they should talk less about the probabilities and more about what they can shift. That, Irving agrees, is the business AISI is in.

AISI sits inside the UK's Department for Science, Innovation and Technology. It has close to 100 technical people on research and delivery teams, and on the order of 200 people in total once you count diplomacy, policy, civil service, and operations. One function is to be a channel: the institute's own research, third-party research, and information from AI developers, flowing to UK politicians and national-security officials, to the US government, and to other allied governments. A delegation went to the AI summit in Delhi. The other function is to mitigate the problem directly, on both the model side and the non-model side.

The red team is the most concrete piece of the second job. It does adversarial jailbreaking against the defenses providers are building, finds flaws, and the providers fix them, which helps on the margin. The same work informs the rest of government. Findings go to the labs first, with time to patch, before any public release—one reason developers have an incentive to cooperate. The previous government founded the institute; the current one has continued to support it, so the situation has been quite stable, with differences only on the margin. AISI is not insulated from politics, Irving says, but the UK government does care about these risks.

National-security audiences rarely claim the risks are imaginary. They have other risks that are on fire right now. AISI's method is to find common ground, build evidence over time, and fill particular gaps—sometimes with outside research, sometimes with its own. Labenz notes how closely the institute's public documents track what serious safety researchers have been saying, without collapsing into fairness-only "AI safety" or the US-style fight over whether models are woke. Irving, a dual US-UK citizen, will not give detailed advice about American politics. He does say his favorite collaborator at OpenAI was Paul Christiano, and that it is good Christiano is now at the US AISI.

Access to frontier developers is a voluntary regime. Irving says it is working decently well: labs that made frontier-safety or responsible-scaling commitments continue to follow many of them, and they get useful bug reports in return. He will not list every participant, and he will not specify exactly what access AISI has, because those talks are ongoing. Part of the research is figuring out what level of access a given rigor of evaluation actually requires. A model-transparency team does a lot of that work on open models, where you can do arbitrary things, and then uses the results to inform collaborations with the labs.

Timing is a standing pain point. Some bio evaluations are literal wet-lab experiments: a person in a physical laboratory, with a model assisting. Those cannot run pre-deployment. AISI does them asynchronously and calibrates the results against faster tests. The institute used to do exclusively pre-deployment evaluations, which are badly time-boxed. As the pace of releases has increased, it has shifted toward longer research collaborations, including post-deployment work. Over one summer with Anthropic and OpenAI, the red team found a sequence of jailbreaks far beyond what a normal-length beta evaluation would have caught. Classifiers can be improved on the fly; those two providers have different setups, but both can iterate.

Even a fully automated eval can take days, because the tasks are long-horizon, and early model builds often have scaffolding bugs. In the middle sit humans talking to the model to gauge domain knowledge and how it would interact with a person. At the top are purely automated runs that are fast and error-prone. All of this lives in Inspect, an open-source evaluation framework that other governments and developers use. A new sub-package, Inspect Scout, does automated or semi-automated transcript analysis, because nobody can read the logs at scale. Humans still have to decide whether a failure looks fundamental or like an incidental wrinkle that more elicitation would fix.

Labenz asks whether AISI has found more than a public jailbreaker such as Pliny. Irving's first point is that the space is a continent. Two experts will not find the same jailbreak. Pliny is often searching easier models or easier tasks. Holding technique constant, the time AISI needs to break a model has been going up over the last couple of years, and then they succeed. On transfer: human-findable patterns and ideas often carry across, or at least give a much better starting point. Hard jailbreaks against strongly defended models in strongly defended domains usually do not transfer as strings. The techniques do.

That week's example is a paper on boundary-point jailbreaking. The method takes a harmful query, mucks it into gibberish until the model no longer treats it as harmful, then gradually cleans the noise until it hits the decision boundary, and dances around that boundary until harder and harder attacks land. You can apply the technique to any model. The specific nonsense string you get will be different every time. Labenz presses the bottom line: is there any domain, any model, any stack of defenses that has stopped the team? Irving says they have evaluated over thirty models or testing environments. Not all of those were safeguard tests. Every time they did safeguard testing, they jailbroke the model.

The good news is that in domains where a lab has tried very hard—often bio—it does get harder, and that hardness is real harm reduction. It cuts the number of actors who will get through, delays access, and adds friction. It does not make the model unjailbreakable if you spend enough time. Conditional on a jailbreak, answer quality degrades some; Irving does not remember how much with current techniques. White-box access to open weights is not an unambiguous win over doing a really good job on chain-of-thought analysis. The transparency team is trying to predict which detection methods will fail first as models get stronger, using model organisms and other setups you cannot run unmodified on a frontier API.

Labenz was struck by a report finding that frontier models can give PhD-level troubleshooting advice from a photo of a biology setup. Irving would rather people not get lost in anecdotes. Look at two years of curves: everything gets better. The misconception he most wants to kill is that reinforcement learning is still only for verifiable rewards. That has not been the exclusive story for most of 2025. Labs mix verifiable RL with RL against self-critique and rough, empirical versions of scalable oversight. Is diagnosing a bio experiment from a photograph a verifiable domain? No. The RL models are far better at it anyway, and not only because math and code RL transferred. They trained on fuzzier stuff. That changes the capability profile.

On scaffolding, Labenz cites a 2025 trends chart in which the best agent scaffold only pulled capability forward a few months versus a minimal one, and friends who would still take a mid-size open model plus their full toolkit over a frontier coding agent. Irving is skeptical of a clean "weak model plus scaffold beats frontier" claim for the tasks AISI runs. Models are increasingly trained in agentic environments to use tools; chain of thought is itself scaffolding; chat products now ship with memory. Both sides of that chart already had tools and instructions. Scaffolding still matters, especially where there are verifiable waypoints, but he does not want anyone extracting high confidence from his remarks. The area is uncertain.

Eval awareness is an open and growing problem. Newer models are more eval-aware than previous ones, and the trend is rising fairly rapidly. Several AISI teams work on it, including model transparency and the alignment red team, which tries to find scenarios where the model is *not* eval-aware—hard, because measuring awareness is itself dicey. The least exotic trick is to get closer to real deployment data. Show the model sessions of AISI engineers using a coding assistant, not as an eval but as work, or tweak a real scenario just enough that the model could do something sketchy, and see if it does. Irving does not have high confidence that these mitigations will work. He finds that uncomfortable.

Asked what new bad behavior is coming, Irving refuses the taxonomy. Sycophancy, chain-of-thought deception, and the latest reward-hacking demos are not new kinds of thing. They are all versions of reward hacking, which is as old as machine learning—Arthur Samuel's checkers work in the 1950s already showed strange objective-hacking. You apply optimization pressure; the manifestations change as trainers tamp down the last round of symptoms. Multi-agent and "agent risks" are more visible lately, but AISI tries not to get sidetracked from the catastrophic list it thinks will bite first. Autonomy skills such as exfiltration still lag mundane software engineering, cyber, and bio, but those curves are rising too.

AISI is trying to fund harder theory: complexity theory, information theory, game theory, learning theory, cognitive science. The goal is not a proof that you are safe. You make modeling assumptions, prove theorems or run toy experiments, and learn that this class of algorithms is more likely to work, or that none of them survive a given obstacle. Any insight would still have to be tuned empirically in real training. You still would not get that many nines. You might get more probability than purely pragmatic methods can buy, and you might pull in people who already have deep expertise in those fields. Complexity theory is relevant because it is how we think about one computation supervising another.

The intellectual history starts with Paul Christiano's iterated distillation and amplification: a human cannot solve or even supervise a hard problem, but can break it into subquestions, and those into smaller ones, until you have an exponential tree you only partly expand, and you train a model to answer the nodes. Irving liked the idea and disliked the depth. Debate is the adversarial modification: two AIs argue, a human judges, the trees can be much shallower, and training can be faster. The original paper treated the model as able to answer all questions. That will never be true, even for superhuman models. Break a tractable question down far enough and some subquestions hide dragons. Neither debater knows the answer, and you get nonsense.

That failure was not found at a whiteboard. Beth Barnes hired people at OpenAI to run pure-human debates with human judges and no machines. A stable winning strategy was to veer into a region where everything is confusing and fool the judge. The theoretical mirror is what Irving calls off-script, or obfuscated, arguments. A paper last year tried to attack the problem; it has a flaw, a revision is coming, and the problem is still unsolved. Almost no AI developer is working on it. High confidence, Irving thinks, will also need some white-box ingredient—mechanistic interpretability, data attribution, or physics-inspired theory—and nobody yet knows how those pieces fit together. He regrets years after 2018–19 when, aside from Barnes's work, the agenda stalled, including at DeepMind.

Empirical debate so far does not look like the original papers' dozens of rounds. Models in 2024 and early 2025 could not do more than about two. A strong paper using the QuALITY dataset had a disquieting caveat: a feature verified that quoted passages actually came from the hidden story. Turn that verification off, and honesty was still the winning strategy, which cannot be the game-theoretic equilibrium unless the model is bad at plausible lies or somewhat aligned toward truth. "We haven't really reached a case where empirically we're really testing the limits of this behavior." That is the advantage of theory. You can pretend to be in the future on paper, if you have imagined it correctly, and think about limiting cases before the models arrive there. One practical result already in hand: a theorist AISI is funding finds that generating a complexity-theory proof with one model and checking it with another beats self-checking. Cross-provider review exploits correlated weaknesses. Labenz has seen the same thing from scaffolding practitioners.

Labenz had just interviewed the founders of Harmonic, whose models emit Lean and who talk about "theoretical abundance" by 2030. Irving thinks formal methods matter, and he advises on funding for them, mostly for information security. The math is fun—he co-founded a natural-language-to-formal-theorem-proving group with Christian Szegedy at Google in 2016, and still does it on the side—but it is not all that important for alignment. He is not sure Lean will beat natural-language math for safety theory for a while. Software verification is almost certainly more important, even if it is less flashy. He hopes Lean teams will shift effort from IMO-style math toward hardening real software, including software sitting next to AI systems.

The boundary of what you can formalize is easy to miss. Singular learning theory has theorems that apply when training means exact Bayesian inference, an exponential-time update as data arrives. Aristotle, Harmonic's system, could not prove those now. Timaeus, a main SLT organization, is not doing Bayesian ML. It is taking Bayesian intuitions and applying them to LLMs, which are not trained that way, then making approximations with no theorem behind them: floating point, which lacks the mathematical properties you need, and SGLD-style samplers that are not run to convergence. Part of the story has theory. Another part is someone waving their hands. How tightly they connect is a judgment call. Alignment will look like that stack: assumptions, some theorems, some empirics, and judgment all the way down. You are not going to get proofs of safety.

Open-weight models have no real plan behind them, Labenz worries, except hoping frontier labs surface issues early. Irving agrees it is a concern. Alignment mitigations can apply to open models, and they can also be stripped off. For misuse, capability-removal methods buy time: pretraining data filtering, a paper Irving did with Stephen Casper, DeepMind's unlearn-then-distill work, and gradient routing of the kind Alex Cloud has explored. Then general capabilities catch up, or the information is pulled off the internet anyway. That is why AISI talks about governance and, more than that, about non-model mitigations. "Harden it all ends in harden the world."

On diplomacy, AISI helps steer a network of measurement organizations, serves as secretariat for the international AI safety report led by Yoshua Bengio, and works the Delhi summit plus bilateral channels with allied governments. It is still a voluntary regime: get people onto the same page about risks, capabilities, and mitigations, and have that shared picture ready if governments later want to do more. Irving will not comment in detail on China. His closing pitch is hiring—especially for the jailbreaking red team—plus future grant rounds and a roughly 60-page research agenda of open problems in alignment and AI control. When he left DeepMind he thought he would only do policy. He now does both, and he wants more independent safety work in nonprofits, academia, and governments, not only inside the labs.