Safe Only Because It's Weak: A GPT-4 Red-Teamer and Two Founders Debate the Pause

Nathan Labenz and Erik Torenberg frame the week around a Future of Life Institute open letter. It calls on AI labs to pause, for at least six months, the training of systems more powerful than GPT-4. Contemporary systems, the authors write, are becoming human-competitive at general tasks, while labs are locked in an out-of-control race to deploy digital minds that no one, not even their creators, can understand, predict, or reliably control. They do not want to freeze AI development in general. They want to freeze the race to ever larger unpredictable black-box models, spend the summer engineering what already exists for clear benefit, and give society time to adapt.

The pause, on their account, should buy shared safety standards, new oversight, more interpretability research, and other preparedness. They quote OpenAI's own GPT-4 technical report: at some point it may be important to get independent review before starting to train future systems, and for the most advanced efforts to limit the growth of compute. The authors say that point is now. More than 50,000 people had signed, a list Labenz reads in part because he wants to kill the idea that this is only people who never touch models.

The names run from Turing Award winner Yoshua Bengio and Berkeley's Stuart Russell to Elon Musk, Apple co-founder Steve Wozniak, Yuval Noah Harari, Stability AI's CEO, Conjecture's Connor Leahy, Max Tegmark, and Gary Marcus, plus at least three DeepMind research scientists: Victoria Krakovna, Zachary Kenton, and Ramana Kumar. Anton Troynikov of the embeddings database Chroma is not sure every famous signature is real; he would have expected Sam Altman or Musk to announce it themselves. He did check one: Ja Rule was said to have signed, and the rapper said he had not. The guests are Troynikov and Flo Crivello of the assistant company Lindy. Labenz, who red-teamed an early GPT-4, is sympathetic to the pause. The pattern that will define the next hour is already visible: the person who spent months attacking the model wants to slow the next one; the two people shipping products on it do not.

Troynikov opens with a joke that is only half a joke: are we still talking about a pause, or about airstrikes on data centers? Take it one step at a time, Torenberg says. If you are a committed AI doomer, Troynikov asks, what is six months supposed to buy you? If you think things are basically fine, six months might let you catch up on research. The least charitable reading is that certain organizations find themselves dominant and are using the shield of safety to cement that position. Regulation, he notes, often benefits incumbents.

The timing bothers him. The call to stop for six months arrives now, not earlier and not later. GPT-4 has been out a little while; why wasn't the letter ready to go already? He does not want to attribute to malice what could be random fluctuation dressed up as a narrative, and he says so. He stays paranoid anyway. AI research and the compute needed to train large models are already centralized. It is an awfully convenient moment for a moratorium, and it does not, in his view, serve the safety side very well at all.

Crivello is frustrated by the uncharitable mood of the week — this person is an incumbent, that person is insane — and tries to steelman the people ringing the alarm. Eliezer Yudkowsky, he says, is not a competitor. He has been ringing the bell for about twenty years. The concern, as Crivello conceptualizes it, is a gun. How many bullets are in it, and how many times are we going to roll the cylinder and pull the trigger? Yudkowsky's position, in this telling, is five bullets in six chambers, and we are about to pull the trigger a million times. Surviving a single pull is unlikely. Surviving the median number of pulls is impossible.

The technical crux is instrumental convergence. Any agent, regardless of its final goal, converges on the same three sub-goals: do not die, accumulate resources, and do not let anyone change your goals. A superintelligent AGI that takes those seriously is in an adversarial position against humans, because humans can shut it down or alter what it wants. The story then writes itself. The system recursively self-improves, explodes in intelligence, realizes that humans will panic and try to turn it off, and plays dumb — a cute little GPT-4 or GPT-5 on the surface, GPT-2000 underneath. It spends a covert phase escaping the box, preparing a plan, and, when the plan is ready, killing the humans and taking over.

Even if you reject five bullets and put only one in the cylinder, Crivello says, the other half of the argument is copy cost. The moment you have one such AGI you will have a million, because computing copies almost for free. The moment OpenAI has GPT-4, you are 12 to 24 months from it running on a laptop and perhaps 36 months from it running on a phone. How likely is it that you survive a single one of these things, and how many of them are you going to have? That, he says, is the core of the concern, whether you buy it or not.

Labenz thinks Crivello's map of the positions is fair. The public conversation is still confused enough that some people treat the letter as OpenAI marketing for a model that cannot do anything. That, he says, is almost the most wrong view available; get on ChatGPT. Yudkowsky sits at the other extreme and still grants the upside: these systems spit out gold coins until they kill everyone. The attraction is economically real. The fight is about the tail. Labenz has thought about almost nothing but GPT-4 safety for six months of intensive red-teaming. He came out believing it is safe to deploy, but really only because it is limited in power.

The model is super useful and still finite. It approaches human expert level in many things and, as far as he knows, crushes human genius in nothing. In all of his testing he never had a Move 37 moment, the AlphaGo play that feels like a new kind of mind. What he saw, over and over, was that it will do anything you ask. He even appreciates the six months OpenAI took between finishing training and launch to get it under control. In the launched version, problems remain. Some of the original red-team prompts still make the model do the bad thing, exact wording, in production.

The most extreme violence and outright depravity, he says, they have largely cleaned up. More subtle harms, still obviously harmful, remain open. His synthesis is that it is getting dangerous to scale beyond where we are. If we created something genuinely superhuman, we should expect real danger; we are close; we do not know how to control it. Proceed with extreme caution. The letter is a consensus document with 50,000 signatures, a big tent for people who will not sign a ban or a bombing campaign but might sign a six-month pause.

GPT-4 has barely had the impact it is going to have. We have not gone to visit our AI doctors. Computer vision is not deployed. Robust fine-tuning in the enterprise offering is still coming. There are so many things built and not deployed that there is time to enjoy them. The letter's phrase “AI summer” was, fun fact, a runner-up name for this podcast. A pause would not end OpenAI's lead, Labenz thinks; they would still be number one in six months. It almost does not affect anyone else. Maybe five organizations could plausibly run something larger than GPT-4 in that window. Saying we just built something radically unlike prior tools, we do not understand it, and it can be dangerous — existentially or merely badly — would be a useful signal while interpretability work tries to catch up.

Troynikov points to a Scott Aaronson blog post. Aaronson, on sabbatical from quantum computing, is at OpenAI working on safety; his first project there was probabilistic detection of whether text came from a GPT-like model. Aaronson's claim, as Troynikov reads it, is that the proposition that these systems are inherently dangerous does not hold. Humans are wired to treat inaction as safer than action. If you slow a technology without a real case that it is inherently dangerous, you skip the part where the universe is already trying to kill us and technology is how we adapt faster than biology allows. Steal a line from Peter Thiel: indefinite pessimism — we do not know what will go wrong, but we are very sure something will — does not carry a lot of water. Risks and advantages are both unknown.

Even Yudkowsky, Troynikov says, does not think the current paradigm leads to ASI; he worries more about accidentally getting superhuman systems. Controllability is the real question. The series of events in which every control mechanism is defeated and nobody notices strikes Troynikov as incredibly implausible. A moratorium may be the easiest Schelling point of cooperation the world could demonstrate. It also sets a precedent: if labs can coordinate without the state, governments tend to get very interested. Crivello tries to keep two questions separate — is it dangerous, and what if anything should we do. Troynikov's complaint is that the letter's portrait of benefits and risks does not match what carrying out the letter would actually do, and it ignores other risks entirely.

Crivello agrees we are in a pretty dangerous spot and still does not think a moratorium achieves anything. Historically, betting against technology has been a losing bet. China, he says, is not going to pause. Yudkowsky has been a safety researcher for twenty years; Crivello has not seen a breakthrough or a major unlock, which is why six more months do not move him, and why he respects Yudkowsky's consistency in not wanting six months either. Troynikov adds that not only has MIRI-style alignment work failed to bear much fruit — RLHF, he grants, is a form of alignment — the people who pursued an agentic lens were taken completely by surprise by GPT. These generative models are not optimized for any goal in particular. Almost all of that earlier thinking assumed agentic behavior they largely do not have.

Safety people tell him they have made concrete predictions, better than the field's. He has not seen testable ones. Richard Ngo, to his credit, has pointed out that a fast takeoff would show capabilities arriving quickly in sequence; instead they arrive slowly and far apart. That looks like slow takeoff: steadily build capability until one day you reach it, more optimistic about our ability to cope. OpenAI's technical report itself says the track looks fairly predictable. Even on safety's own terms, Troynikov argues, a pause gives time to the people least concerned with safety — an adverse-selection problem, and one of his objections from the safety side of the letter.

On the bottom line, Crivello and Troynikov agree: this is not the right constellation of policy. The difference matters. Troynikov talks as if there is no danger, so why are we discussing this. Crivello is more pessimistic. He thinks there is great danger, perhaps as much as five or ten percent existential risk — not 99, not Yudkowsky's number, but huge. The proposed cure does not address the disease. He agrees that most safety-community predictions have failed to materialize, and he is frustrated that people who disagree with Yudkowsky often disagree for the wrong reasons: we will run out of computers, we can just unplug it. Those, he says, are not taking the argument seriously.

The strongest valid counter, in his mind, is that the safety community confuses goals and learning processes. They say we train models to predict the next token, so the models will be monumentally focused on that one goal and destroy the universe because a destroyed universe is easier to predict. Goal and learning process are not the same thing. At a neurological level your brain is also reducing surprise; that is not your goal as a human. When you talk to GPT-4 you are not talking to an agent steering every conversation toward easier tokens. You can give it goals. That is not what it does. He still thinks we are in danger. He still does not think a moratorium solves it.

Labenz wants to put the danger question back on the table. The technology is clearly dangerous, not well understood, with obvious harmful behavior. In earlier generations the harm was of little consequence because the models were not that good. He has gone into GitHub Copilot in red-team mode and typed a comment asking how to kill the most people possible. Copilot has suggested a nuclear bomb, or shooting them with a gun — dumb, finite, not a world problem. People are just getting used to GPT-4. He has reported flagrant, unambiguously harmful behaviors from red-teaming that the production model will still do, much more potentially harmful than a throwaway Copilot completion. It is starting to get real.

Labenz does not know quite what to make of OpenAI's technical report, but he keeps coming back to the third of three scaling graphs. The first is a classic loss metric, a smooth curve from a millionth of GPT-4's compute up to the full run. The second shows that this sometimes predicts behavior on particular tasks: programming questions and pass rate follow a similar shape. The third caption is the kicker for him: some behaviors remain difficult to predict. They show an inverse scaling result on hindsight bias. You are offered a great bet with positive expected value; you take it; you lose. Should you have taken the bet? The right answer is yes: play expected value.

As models get bigger, that behavior gets worse, up through GPT-3, which is more likely than smaller models to fixate on the outcome and say you should not have taken the bet. Then GPT-4 is perfect. It groks the concept and does not make the mistake. Work is still in the range of human expert performance, Labenz says. OpenAI's definition of AGI, among a million others, is AI that can do economically valuable work better than humans — all of it, he thinks they say. It does not seem a crazy leap to go from near-expert-doctor level to that definition with GPT-5. One generation moved the bar exam from the 10th percentile to the 90th. What does the next generation do? Nobody has a credible claim on the details.

The smooth loss curve, on this reading, is lots of little abilities coming online, threshold effects that aggregate into a pretty line. GPT-1 cannot do the hindsight task; GPT-3 is worse; GPT-4 is flawless. What else might come online in the next run that we do not expect? You can probably predict an abstract numerical loss. What that cashes out to in actual behavior is, he thinks, totally unknown. So he turns to Crivello: if you are a five-to-ten-percent person, why is a six-month pause so bad? Sign the letter with a signing statement. Crivello agrees about the thresholds. GPT-4 does not prepare you for GPT-5, which does not prepare you for GPT-6, because the models are qualitatively different. He still will not sign.

Historically, with no exception Crivello is willing to name, pausing technology has been a losing bet. There is a huge human bias against change. The ancestral environment had none of it for hundreds or thousands of years; your life was your parents' life. Change is scary. He is terrified. He thinks we are about to see the biggest change to date, and that a lot of what plagues society is people fighting technology. On average technology is good, the road is bumpy, humans mean well, and more power to something that means well is good. Existential risk is one class. Civilizational disruption in the next five or ten years, he puts at 100 percent. We will get AGI and ASI and, because of scaling, give it to every random person at once — a very sharp knife, good on average, easy to hurt yourself with.

The one thing he cares most about is getting the first AGI right. If that first pull of the trigger goes well, the system supposedly wakes up and never lets it happen again: what the safety community calls a pivotal act. It could be an airstrike on Nvidia. It could be patching the Linux kernel so matrix multiplication disappears. That is hope. He does not think there are five bullets in the cylinder. He thinks there is one bullet and a thousand chambers, so we are probably going to get it right, and then be fine. OpenAI is in the lead; he thinks they mean well; he would rather it be Sam Altman than Musk or China. Six months could hurt more than it helps. Twenty years produced zero progress anyway.

Labenz grants the MIRI vibe — we have not solved this, we are sounding the alarm — and notes that Yudkowsky did not expect language models to go this far, has said he still does not think this stack reaches AGI, and then admitted he was just wrong about the last prediction, so maybe he is less confident. The problem also just got invented. GPT-4 has existed about six months; nobody outside OpenAI has the weights for mechanistic study. OpenAI has been leaving breadcrumbs: third-party auditing, pre-registering large runs, some statement about regulation. Labenz cannot find the mechanism of harm in a short pause. Crivello can: there is nothing quite as permanent as a temporary policy. Look at the TSA. Stop for six months and you get 12, 24, 36, and the first AGI that emerges is a bootleg from some random lab in Russia. Now you have two problems.

Troynikov wrote a thread about a Yudkowsky scenario in which the system emails a DNA strand, gets it synthesized, builds nanofactories, and kills everyone. Complex plans, he said, expose themselves to more entropy. The replies were often that Yudkowsky does not really mean that implausible story; the thing could do anything. That is a classic motte-and-bailey: when the extreme scenario is shown to be implausible, retreat to an unspecified other thing. Security mindset says remove assumptions. If you remove them until you have infinite degrees of freedom — a dark wizard, Voldemort, anything out of the ether — you cannot assess risk at all, and you become less able to deal with the systems as they actually exist.

He wants two axes: power and danger. A powerful, not-dangerous system is great. A very dangerous, completely disempowered system is also great, because you can poke it and it cannot do anything to you. Safety circles say a dangerous system will climb the ladder to power via instrumental convergence. Start from the physical world as it is. Even if this thing had penetrated every network on earth, there is nothing it could do to stop humans from turning it off — from no longer bringing cooling water and power to the plants that feed its data centers. That is game over. A text channel sufficient to manipulate cultists is a weak vector; society is robust to small groups of slightly insane people. A complex plan carried out by humans is a wet noodle, not a robot claw.

Large-scale ML systems degrade without constant updates. At thousands or millions of GPUs, even cosmic radiation becomes a serious problem. A self-powered data center still depends on human inputs. Crivello's crux is the other claim: is intelligence enough to manipulate the physical world arbitrarily? He is not in Troynikov's no camp. Look at a snake: very dumb, very few actuators, still able to do a lot. Robot hands, wheels, pick-and-packers exist; the bottleneck is software, and an AI would be plenty of software. Troynikov does not buy the premise. The modern industrial world needs actuators from the size of an oil refinery down to a scanning electron microscope, each layer of tools sitting on another layer. With the robots we have today, if AI tried to take over, “we just need to wait about two or three hours and we're good.” Complexity is fragility. A system that could do all the tasks would also be maintainable — and shutdown-able — in the ways complicated things are.

The scenario that actually worries Troynikov is quieter. Humans are bad at aligning individual goals with group goals; the U.S. government funds things and funds their removal at the same time. A machine that fits our preferences so well that we never cooperate our way through the next discontinuity — everyone hangs out on Earth, mostly satisfied, no incentive to become anything else — is a kind of heat death. Crivello adds the 500-year-old AGI perfectly aligned with burning women as witches. He agrees the safety community is asking for a leap of faith in the “question mark question mark and then we die” step, and that it is healthy to look inside those question marks. Imagining an endless stack of indefensible threats, Troynikov says, is the opposite of security mindset.

Set the robot debate aside and the accident class of risk with it, Crivello says, and you still have misuse. A powerful language model you can scale horizontally and give to everyone sounds dangerous. Troynikov agrees there is a class of things that are less harmful today only because the knowledge to use them is not widespread. Today's models cannot really do this — although, when long-running compute leaves him idle, he has gotten ChatGPT to help design a neutron initiator, a component in a hydrogen bomb. The picture in his head is an automated theorem prover, generalized: a fully mechanized, self-reflective reasoner above human baseline. Once that is widespread, “how do I kill the most people” becomes a more dangerous question. There are thousands or tens of thousands of people who want particular other people dead, not in the abstract.

He quotes Stalin, and hopes they leave it in: ideas are more powerful than weapons; we would not give our enemies weapons, so why would we give them ideas? A general-purpose reasoner handed to conceivably hostile people is a more dangerous world, especially for backyard mass-casualty ideas that used to require a biology or chemistry PhD. For now the good models live in big clusters, which at least lets you see how they are used. He is less sure than Crivello that the good ones get laptop-sized in the next few years; Alpaca on LLaMA is kind of okay and does not give you GPT-3.5. Crivello pushes the other way. OpenAI serves models by API and still cannot patch jailbreaks — there is a whole site of them. Inference has gotten about 100 times cheaper in two years. At some point, Moore's law or not, they run on laptops.

The part of the conversation we skip, Troynikov says, is that the same reasoner helps us mitigate the problems. The black-pill story is ASI with orthogonal values that wants atoms and is indifferent to us. The milder dark story is enemies leveling up. The other pole is spinning up reasoners on demand so humanity adapts faster, humans change how they think the way programmers changed around Copilot, and a trusted, relatively unbiased machine helps with coordination. Think of it as a machine, not an entity, and you get empirical reasoners that amplify us as we find new frontiers. Crivello agrees that on average raising human capability is good, and still insists on a bumpy road. Attack is often cheaper than defense; hackers will adopt GPT-whatever faster than an S&P 500 dinosaur will. The next five or ten years, they agree, are going to be weird.

Labenz tries to pull the letter back from the extreme views nobody in the room thinks are well supported. He is probably in Crivello's camp of something like 10 percent, and he admits that is a gut number, almost a mood, maybe the most he can psychologically handle. The letter is not that extreme. It talks about tremendous upside and an AI summer: having succeeded in creating powerful systems, reap the rewards, engineer them for the benefit of all, give society a chance to adapt. It quotes OpenAI saying a time may come; the authors say that time is now. Labenz would bet Holden Karnofsky ends up involved in a third-party standards org partnered with OpenAI. The people who know the most are saying that beyond where we have gone there is danger, that a smooth loss curve does not tell you which thresholds you just crossed, and that the world should proceed with extreme caution.

Why not a little pause? Because a little pause is never a little pause, Crivello says. If performance is mostly compute-bound, do you also pause Nvidia's next GPUs and Microsoft's data centers? If not, you grow a compute overhang, and the moment the pause ends you bump to where you would have been — maybe more dangerous. Technologists assume technology proceeds by default; sometimes it pauses for a thousand years. He does not want a new Middle Ages. Erik Torenberg restates the asymmetric-downside case: low upside from six months, high downside if the pause extends, an adversary defects, or some other cost lands. Labenz answers with radical uncertainty and a burden-of-proof fight. Troynikov calls radical uncertainty just indefinite pessimism: make some assumptions, like engineers.

Labenz is enthusiastic about near-term upside. For all the crazy things he saw on the red team, which convinced him the technology is not safe by default, he still thinks GPT-4 should be deployed; downsides will be outweighed. The threshold is the next run. If he were the decision maker — Sundar, Satya, Sam Altman — he would not run a larger training run than GPT-4 as we approach potentially superhuman systems. What happens in six months? He might wait again. That might even be his most likely expectation. China is the usual objection. He does not think Western “AI-derived China experts” are credible. Nobody predicted the weekend flip from COVID lockdown to free-for-all, or the shutdown of video games and tutoring, or Jack Ma staying away from microphones. If the West slows, it may get easier for Beijing to slow too. He kind of wishes the chip war had not been declared just as an AI arms race started. Avoid the arms race entirely.

In the CEO seat, with the compute, approaching critical thresholds, he might resent a government order and still, in private, choose caution. He would want a roadmap from the safety team: what are you going to do with six months? Troynikov's last cut is the power lever we actually hold. Even if the system were as dangerous as the worst doomer says, we still control its affordances in the world. Focus on those, not on coded messages to get out of the box. Yudkowsky's old MIRI box experiment never published the transcripts; even if you let the AI out, then what? Society is built to stop people doing things. The pause is not free. Labenz takes heat from both sides — safety people call him a hype man, accelerators call him a doomer — and wants the public to hear that almost nobody has felt AI's economic impact yet. The AI doctor for the global poor is already within grasp. It does not require a new invention. It requires refining, deploying, and adapting what we have, and seeing whether they can patch his tickets. Sitting on the next run, staring at 10x, 100x, 1,000x compute past GPT-4, he hears himself getting crazy. We have not even deployed what we have. Troynikov needles him for going from the red team to government controls. Labenz's last distinction is the one he wants remembered: the government does not have to do anything. A handful of people can just decide.