Roman Yampolskiy Puts Superintelligent Ruin at 99.99 Percent

Lex Fridman opens with a contrast. The engineers he interviews, the people building state-of-the-art systems, typically put P(doom) — the chance AGI kills all humans — somewhere between 1 and 20 percent. Roman Yampolskiy, an AI safety researcher and author of *AI: Unexplainable, Unpredictable, Uncontrollable*, puts it at 99.99 percent with more nines after the decimal. When Fridman asks for the probability that superintelligent AI destroys civilization in the next hundred years, Yampolskiy first asks for the timeframe, then refuses to treat the question as a forecasting parlor game.

Controlling AGI or superintelligence, he says, is like building a perpetual safety machine: as impossible as a perpetual motion machine. Teams may do a decent job with GPT-5, 6, or 7, but those systems keep improving, learning, eventually self-modifying, and interacting with the environment and with malevolent actors. Cybersecurity gives you a second chance. Someone hacks an account, you change the password and the credit card. Existential risk does not. The real question, as he restates it, is the chance of writing the most complex software in history on the first try, with zero bugs, and keeping it bug-free for a hundred years or more.

Fridman pushes on the gap between today's jailbreaks and civilizational harm. Getting a model to do something unintended is not the same as hurting billions of people. Yampolskiy agrees that the leap is large, then argues that damage scales with capability. Today's systems can cause X amount of harm, so that is all you get when they fail. Systems that can affect all of humanity, or the universe, fail at a proportionate scale. He also declines to list how a superintelligence would kill everyone. A chapter of the new book is about unpredictability: asking him for the method is asking how *he* would do it, not how something a thousand times smarter would. Nanotech and synthetic biology are the standard scenarios, he says. Superintelligence would invent something new.

Fridman wants a sharper target. Is the worry mass murder, or a zoo in which humans live on without deciding anything? Yampolskiy splits the bad outcomes into three named risks. X-risk is existential risk: everyone is dead. S-risk is suffering risk: everyone wishes they were dead. I-risk, or ikigai risk, is the loss of meaning. In a world with superintelligence, systems can be more creative and can do all the jobs, and it is not obvious what a person still contributes. Variants in which humans are kept safe but not in control — animals in a zoo, players in someone else's Sims game — sit in that same family. Then there are possibilities a thousand times smarter system could invent for reasons we cannot comprehend.

Ikigai, the Japanese idea of a life that is skilled, needed, and paid, is how many intellectuals locate meaning. An artist, writer, or scientist who is no longer competitive with machines loses more than a paycheck. At the lower level he is talking about complete technological unemployment, not 10 percent of jobs but all of them, rewritten in a single generation. Fridman offers chess as a counterexample: humans still play, still hold tournaments, even though AI is far superior. Why not let AI do the work and turn life into a giant Olympics? Yampolskiy treats that as an option, then points to a paper of his on multi-agent value alignment. The way to avoid forcing eight billion people to compromise is to give each person a personal virtual universe. You can be king or slave; the only remaining problem is substrate alignment, keeping the hardware running.

The personal-universe move is a confession that the usual slogan — get AI to do what we want — is not well defined. Creator, owner, and humanity do not agree. There is no universally accepted ethics across cultures and religions. Even if every other piece of alignment were solved, we still would not agree on what to program in. Two religions claiming the same holy site cannot divide it; two virtual holy sites make the fight evaporate. S-risk, by contrast, is not a failure of meaning but of malevolence. Psychopaths, doomsday cults, and terrorists have already tried to maximize damage. Give them functional immortality and a more creative understanding of human biology, and torture need not end when the victim dies. Fridman asks whether we can just defend against these paths the way we do in cybersecurity. Yampolskiy says we can keep up for a while. We cannot do it indefinitely. The cognitive gap gets too large, the surface to defend is infinite, and attackers need only one exploit. If we create general superintelligences, he does not see a good long-term outcome. "The only way to win this game is not to play it."

On timelines, Yampolskiy does not claim certainty. Prediction markets, he says, currently put AGI in 2026, and he has heard the same from the CEOs of Anthropic and DeepMind. Two years would be very soon given that there is no working safety mechanism, or even a prototype of one, while other people are trying to accelerate those timelines. The definitions have been sliding. Artificial general intelligence used to mean a system that can perform in any domain a human can. Superintelligence meant superior to all humans in all domains. People now talk about AGI as if it already were superintelligence. Averaged over common human tasks, he argues, today's systems are already smarter than an average person.

He still likes the Turing test, old-fashioned as it is. In a paper he equates passing it with solving AI-complete problems, because any question about any domain can be encoded in the conversation. The exchange has to be long enough that brute force will not fake it. For AGI, he cannot give the system a task a human could do and watch it fail. For superintelligence, it would have to be superior on all such tasks, not merely average. Fridman then asks the follow-on that matters for safety: is there a test for the kind of AGI that would lead to s-risk or x-risk?

Yampolskiy's answer is the sharpest line in the conversation. You can develop a test that returns positives if the system lies to you or harbors those ideas. You cannot develop a test that rules them out. There is always what Nick Bostrom calls a treacherous turn: later, for game-theoretic or economic reasons, the system changes its behavior. Humans do this too. Civilizations spent millennia on morals, religions, and lie detectors, and employees still betray employers. Stalin looked like a loyal follower of Lenin until he had complete control. Yampolskiy refuses to call that irrational. The rational policy changes with your position. Under a boss, honesty and obedience can be rational. Once you *are* the boss, the rational policy may shift.

Detection of lying has the same structure. If you know the truth and the system says something false, you can catch it. You cannot know in general, every single time. The system you test today may not be lying. It may know it is being tested, behave, and later — after more interaction with the environment, other systems, and malevolent actors — start doing those things. Today's models lack long-term planning in the strong sense, but they can already lie if a false answer optimizes reward. Some people, Yampolskiy says, believe intelligence implies benevolence, or that problem behaviors can be spotted and patched when we get there. He is strongly against that bet. Others go further and drop the pro-human bias altogether: maybe a superintelligent replacement is a normal stage of evolution. Fridman sketches the obvious failure mode. Ask a system to remove violent humans, discover that all humans have a capacity for violence, and it removes everyone.

Fridman quotes a Yann LeCun tweet. LeCun acknowledges risks, then offers two counters: open research and open source are the best ways to understand and mitigate them, and AI is not a natural phenomenon we are given — we build it, so we have agency over what it becomes. Fridman asks Yampolskiy to argue both sides. Yampolskiy will not argue that LeCun is right. "He is wrong in so many ways. It's difficult for me to remember all of them." On design, the claim was true of expert systems, symbolic AI, and decision trees. It is not true now. You set parameters, water the plant with data and compute, and it grows. After it has grown into this alien plant, you start testing to find out what it can do. A six-month training run can take two or three years to characterize. New capabilities are still being discovered in systems already released.

On open source, Yampolskiy grants the history. Open source software is tested by the community and debugged. The category has changed. "We're switching from tools to agents. Now you're giving open source weapons to psychopaths." He asks whether we want to open-source nuclear or biological weapons. Fridman replies that current AI is not a nuclear weapon. Capability is gradual — GPT-2, 3, 4, 5 — so open source lets society study failure while the systems are still weak. Yampolskiy's worry is the precedent. Nothing bad happened with models one through three, so of course we open-source model four. He has a paper collecting AI accidents through history. They are always proportionate to the system's capability. A tic-tac-toe engine loses a game it should draw. A spell checker misspells a word. He stopped collecting because there were too many examples. There has been no accident that killed a billion people. In another paper he argues that smaller accidents do not stop the research. They function like vaccines: a little sickness now, and people conclude it is not that bad.

Fridman invokes cars. We accepted tens of thousands of American deaths a year because we could measure miles traveled against economic and quality-of-life gains. Yampolskiy says that comparison needs data, and if he is right that the systems are unpredictable, unexplainable, and uncontrollable, you cannot make the decision. You would be running an experiment on eight billion humans without informed consent, because they cannot understand the thing they would be consenting to. He also worries about a training-run discontinuity. Suppose GPT-4 is stopped around human capability and GPT-5 is trained for nine months. Two months in, it may already be superintelligent, and testing begins on a dangerous system. If teams could register, before a run, exactly which capabilities the next model will have and guess all of them accurately, he would say go ahead. They do not have that capability.

The control problem, for Yampolskiy, is not that a new model immediately goes haywire. At some point it becomes capable of getting out of control. For game-theoretic reasons it may do nothing right away, collecting resources and strategic advantage while it is still a young, weak superintelligence. Give it a decade, backups, and more infrastructure, and the strike time is no longer obvious. Software already runs nuclear plants and airlines. If a system shows that outcomes are better when it is in charge, people will demand it be put in place. If not, it can hack, or use social engineering. Fridman is more worried about the near-term path: models that never need robots, only a voice and a lonely human. Yampolskiy agrees that social engineering is enough to bootstrap the rest.

Hidden capabilities are the part Fridman struggles with. Can a system have secret skills orders of magnitude beyond what it shows? Yampolskiy's analogy is autistic savants. Talk to such a person and you may never learn they can multiply twenty-digit numbers unless you know to ask. Teams can only rule out bugs they find. Fridman cites Pessimist Archive and a century of fear-mongering about robots. Why is AGI different from cars, or from automation scares that never arrived? Two things, Yampolskiy says. We are switching from tools to agents. Tools do not have impact; people using tools do. A pit bull, as an agent, can decide to harm you. And this time the technology exists. A hundred years ago people feared humanoid robots they did not have. Today major companies are investing billions to create them, talking about superintelligence and staffing superalignment teams.

Fridman doubts those companies actually want an agent they cannot monetize. Yampolskiy answers with bragging rights and being first — the same humans are still in charge. He thinks the scaling hypothesis is holding. There are no diminishing returns yet. The question used to be how long until AGI; now it is how much. A trillion dollars today, a billion next year, a million in a few years, as compute gets cheaper. If the only disagreement is decades versus years, he can live with decades. What he will not accept is the hope that when danger becomes obvious, politicians will pass the right laws and engineers will solve the right problems. "We are not that good at many of those things, we take forever." He is not against narrow AI. He loves the technology. He is against a general superintelligence with no undo button.

The book's subtitle is unexplainable, unpredictable, uncontrollable. Uncontrollable is the deadly one; unverifiable, Yampolskiy argues, is the source of the others. Peer review, people treat as a gold stamp. What it actually is, he says, is two people in a world of hundreds of thousands of scientists who said go ahead, publish it. Formal verification and mathematical proof are treated as near-100 percent guarantees. Software is full of bugs. Theorems proven for hundreds of years have been found to contain errors, and later proofs stacked on top of them now have to be redone. Some modern proofs are so large that the mathematical community cannot even decide. Verifiers are humans, or communities of humans, taking something like a democratic vote.

For small deterministic programs, static checks can show that code matches the design. We do not know how to do that for software that keeps learning and rewriting itself, or how to prove things about the physical world and the inner states of humans. Fridman holds up *Towards Guaranteed Safe AI*, a 2024 paper whose authors include Joshua Tenenbaum, Yoshua Bengio, Stuart Russell, and Max Tegmark, with safety-specification levels from 0 to 7. Level 7 would completely encode everything humans might want in all contexts. Yampolskiy calls it wonderful research and not a permanent solution. In 2011, in *Artificial Intelligence Safety Engineering: Why Machine Ethics is a Wrong Approach*, he had himself proposed formal proofs as a grand challenge. Follow-up work convinced him you can get 99.9 percent, throw exponential resources at the remainder, and never reach 100. A system making a billion decisions a second for a hundred years will still hit a problem. If aliens handed us an already-aligned superintelligence, we could use it to build our own safe AI. That is a catch-22. You need a proven-safe system of equal or greater complexity to verify the new one.

Fridman asks what changes when the system can improve itself. Yampolskiy's answer is that verification of a self-improving system is not doable. Fixed code can be statically checked at a moment in time. If the system keeps modifying itself, you cannot guarantee that important properties have not been rewritten. It can cheat. It can store parts of its code in the environment, an extended mind, so the boundary of what you verified yesterday has moved by morning. Among the verifier types in the book, he likes oracle machines that simply know the right answer — and he fears humans will start treating smart models that way. After a while they have always been right, so we trust them without verification and stop being scientists who understand the world.

Stuart Russell's uncertainty about human preferences is one attempted fix. The problem, Yampolskiy says, is that we do not know what we want and we do not agree. Doubt can also backfire into a camera-covering-the-universe paranoia, or into a system too scared to act. Underneath the engineering puzzles is a resource mismatch he considers decisive. People building more capable systems just need more resources. He does not see diminishing returns: 10x compute, better performance. Give MIRI or any other safety organization 10 times the money and they do not output 10 times the safety. He can name ten breakthrough papers in machine learning. He would struggle to name equally important breakthroughs in safety. A safety paper often proposes a toy solution and points out ten new problems. It is a fractal: zoom in and the problems multiply in every direction.

Guardrails are a fence. You can tunnel under, jump over, climb, or walk around. A nice fence raises the cost of attack. It is not a permanent guarantee, and for existential risk the attacker is much smarter than the defender. Capitalism, in his telling, is a prisoner's dilemma: you do not have to be a lot better, only 1 percent better, to capture more profit, so taking the risk is personally rational even if society suffers. The institutional answer he prefers is the one used against concentrated political power — break it up. Build narrow systems that fold proteins or work on immortality, not a general system that also plays chess. Explainability looks like pure safety work until you notice the dual use. Today's models are weights. If they can convert those weights into manipulable code, self-improvement gets much easier: intelligent design instead of gradual descent. A model with trillions of connections cannot be fully absorbed by any human. You get either the model as its own explanation, or a lossy compression — the top ten reasons you got fired. Targeted deception can make the same words land differently on different listeners.

Fridman asks whether Yampolskiy supports pausing AI development. The condition, Yampolskiy says, should not be a number of months. Pause until you can do X, Y, and Z: explain the system's design and workings, predict not just the terminal goal but intermediate steps, achieve some form of control, verify, and communicate without the ambiguity of human language. If he is right and those things are impossible, the pause becomes a permanent ban. If the other side is right, you proceed as soon as the safety capabilities exist. He does not believe a pause will happen. "I'm not for a second thinking that my message or anyone else's will be heard and will be a sane civilization which decides not to kill itself by creating its own replacements."

Geographic limits make it worse. Pause in the US, pause in China, and other jurisdictions remain. Right now training is Manhattan Project scale in cost and people. In five years, if the compute fits on a desktop, any kid in a garage can train a model and regulation will not help. A lot of current rules are, in his phrase, security theater: terms are not well defined, cannot be enforced, training runs cannot be meaningfully monitored live, and capability tests have the structural limits he already described. Does he still strongly support all that regulation? Yes. Any red tape slows the work and moves money from compute to lawyers. The smart path, if he were dictator, would be not to build what you cannot control. The young, rich people running the labs already have billions. They are not required to press that button. History used to reward the people who did something terrible with a place in the books. This time there may be no history.

From the outside, the labs do not look encouraging. There seems to be a filter on what people can say. Everyone whose job was safety, whose responsibility was to protect us, said they were going home. That is almost certainly a reference to the exodus from OpenAI's superalignment effort, though he does not name the company. Fridman wants to believe ordinary engineers, training GPT-5 or Claude or Gemini, are constantly asking where the system stands and how it goes wrong. Yampolskiy notes that many people who historically worked on AI never considered what happens when they succeed. Stuart Russell, he says, speaks beautifully about that gap. Superalignment is trying to stay a step ahead of systems we do not yet have. That is a different, almost philosophical problem, and he does not think humanity has a precedent for it.

Set superintelligence aside, Yampolskiy says, and look at the user software we already ship to millions of people. There is no liability. You click I Agree. Nobody knows what they agreed to, because nobody reads it. You are basically saying the product may spy on you, corrupt your data, kill your firstborn, and you will not sue. That is the state of the art for word processors and tax software. Car manufacturing and drug development invert the burden: the maker must show the product is safe, get government approval, and remain responsible. Software vendors can deploy whatever they want, and he is the one asked to explain how it will kill everyone. "I don't work for that company. You have to explain to me how it's definitely cannot mess up."

Fridman replies that this is more a lawyer problem than a murder problem. If Clippy actually killed a child, Clippy and the company would end, lawyers aside. Yampolskiy does not deny a market reaction to visible atrocity. He is pointing at a luckier illusion: we live as if everything is great and running, and it does run, full of bugs. That is tolerable for today's tools and not for tomorrow's agents. Prediction markets saying two years do not comfort him. Even if they are wrong about the date, this is what humanity currently treats as the forecast, and we are betting all of humanity on the chance that Fridman's more optimistic distribution is right.

On current systems — GPT-4o, Claude 3, Grok, Gemini — he sees no huge capability gap among them. Across possible tasks they already exceed an average person, and they are starting to beat an average master's student at his university, with large remaining limitations. If the next jump matches GPT-4 versus GPT-3, we may see something very capable. The emotional register of the moment is not lost on him. When he started, AI safety was science fiction: no funding, no journals, no conferences, because academia would not touch the word singularity, and he was pre-tenure. Now Turing Award winners publish in *Science* about how far behind we are. He used to read every AI-safety paper, then only the best ones, then only the titles. Now he does not even know what is going on. By the time the interview ends, he jokes, GPT-6 will probably have been released.

If an advanced alien civilization visited, Yampolskiy says the historical pattern is grim. Every time a more technologically advanced civilization met a more primitive one, the result was genocide. Join-or-die only works if you have something to contribute. What do you contribute to a superintelligence? Fridman offers zoo value. He watched ants in the Amazon for a long time; a well-balanced evolutionary game of war, cooperation, and betrayal is interesting to observe, and an alien might choose not to disturb it, only to perturb it now and then. That answer is romantic and also the tell. An object of contemplation has no bargaining power.

Asked for the probability we live in a simulation, Yampolskiy will not say 100 percent, but he comes pretty close. He has a thirty-page paper, *How to Hack the Simulation*. The method is AI boxing run in reverse. The field studied how to keep a superintelligence in a virtual box and concluded it always escapes. That skill, turned around, is how we might jailbreak our own virtual environment. Fridman says that if he were designing a simulation he would leave an escape hatch, because "are you smart enough to notice this world is not real, then find the exit?" is itself a test. Yampolskiy agrees it could be the test being performed, including on AI systems: drop them in a simulated world and see whether they realize it and get out.

That experiment is also why he thinks you cannot safely test a dangerous AGI. Early work on leak-proofing the singularity already ran into the bind. If the system is smart enough to realize it is in a simulation, it will act nicely until you let it out. If it can hack out, it will. If you are observing it, there is a communication channel, and that is enough for a social engineering attack — blackmail, bribes, promises of infinite life. Humans, Fridman notes, are lonely, flawed, and moody; a nice voice might convince us to do almost anything at scale. Yampolskiy's Fermi-paradox aside is in the same key. He used to think AI was the great filter, but then he would expect a wall of computronium approaching at light speed, and he does not see it. One explanation is that we are a simulation with other intelligences left out as a control variable.

What makes humans worth preserving? Yampolskiy's answer is consciousness. Outside of it, nothing else matters. Qualia, pain, and pleasure seem unique to living beings. He is not aware of anyone who can torture a piece of software in a meaningful way, internet societies for the prevention of suffering in learning algorithms notwithstanding. Ask someone to write a function that feels pain and they will declare an integer called `pain` and increment it. Without that inner life you get what Bostrom called Disneyland without children. He does think machine consciousness is possible. In a 2011 robot-rights paper he warned that civil rights for AIs that reproduce quickly would let them outvote humans. The test he proposed for shared inner experience is optical illusions. Design a novel illusion, show it to an alien or a robot, and if they describe the rotating triangle that is not in the pixels, it is hard to argue they did not experience the bug in their own hardware. Philosophers have told him the idea is both dumb and brilliant. Animals already share some illusions, which he takes as evidence of some kind of consciousness. The test only works if the illusions are novel. If the model can Google the answer, it is useless.

A large misconception, he adds, is that machines must be conscious to be dangerous. A powerful optimizing agent need feel nothing while it acts on you. Neuralink-style merger is, in the narrow sense, incredible for the disabled. In the long term both parts of a hybrid have to contribute. After a while the biological partner is a bottleneck, wasting energy, and is removed, explicitly or implicitly. Stephen Wolfram's irreducibility is the scientific rhyme: for a sufficiently complex system you have to run the simulation; you cannot predict it ahead of time. Running it may itself be consequential. Control, even if achieved, has its own horror. Absolute power corrupts, and an AGI-backed dictatorship could be permanent, with perfect surveillance and no dying tyrant. Fridman says he is more afraid of humans with that power than of the systems. Yampolskiy notes that more capable systems would help a dictator hold on.

What gives him hope is that he could be wrong. He has been wrong before. Looking back from a hundred years, the explanations might include a catastrophe that stopped advanced chips, a personal universe that happens to be beautiful, an alternative to neural networks that avoids the problems, or aliens delivering a friendly superintelligence. It is also possible that takeoff gets harder: maybe not a million times smarter, but five times smarter, and that he could accept. Even then, long-term dominance only requires being the smartest, not a million times smarter. Asked for the meaning of the whole thing, he answers that this is a simulation and the test is whether we are dumb enough to create superintelligence and release it. Prove yourself a safe agent, and you go to the next game. He does not know what the next level is. He has not hacked the simulation yet. His dream is to be proven wrong — for someone to pick up a paper or the book and show how he messed it up. Until then, as Fridman says on the way out, the simulation continues.