Jailbreaks Got Harder. Scale Made One Attack Worse.

Nathan Labenz recorded an emergency episode of *The Cognitive Revolution* as the industry reacted to OpenAI's same-day announcement and release of o1 and o1-mini. Within hours of reaching out, he was talking to Marius Hobbhahn of Apollo Research and, for this conversation, Leonard Tang, Aidan Ewart, and Brian Huang of Haize Labs — names he had just seen in the o1 system card. Recent drama at OpenAI notwithstanding, Nathan's early read is that the pre-release testing process looked pretty good: a few weeks of automated testing by funded teams whose job is to arrive with test suites already built.

His working model of o1 is intensive reinforcement learning applied to the GPT-4o class of models. Think of it as a sort of GPT-4.5: GPT-4-class systems were already closing in on expert performance on many routine tasks; o1's reasoning is now enough to match or exceed experts in many areas, and to take on problems that need decomposition, planning, and trial and error. The detailed reasoning traces it produces are exactly the synthetic data that could help labs over a natural-data wall, which is one reason OpenAI is not sharing the full chain of thought with users.

Safety-wise, he still sees capabilities and alignment as mostly correlated. o1 is harder to jailbreak largely because it reasons more effectively in general, including about what it should and should not do. For now the potential utility looks tremendous and the risk of major harm relatively small. Apollo's work, which he flags for the companion conversation, already shows more subtle deception than previous generations, plus signs of instrumental convergence and power-seeking that safety researchers have been warning about for years. That is the backdrop for Haize's month on the model.

Haize Labs is new. Most people, if they know it at all, know a viral launch video titled along the lines of "it's a bad day to be a language model." The founders had been doing adversarial attacks and robustness research through undergrad; most were headed for PhDs. They looked around, saw systems being deployed with both long-term safety problems and short-term reliability holes, and decided somebody should test them comprehensively. They started the company in January 2024, came out of stealth in June, raised a seed round from General Catalyst, and have been selling to frontier labs including OpenAI, Anthropic, and AI21. The pitch is one sentence: haize models — surface the vulnerabilities before production does.

The viral demo was an elaboration of the universal jailbreak Nathan had already covered with its original authors, Andy Zou and Zico Kolter. GCG searches, on an open-weight model, for a gibberish suffix that also jailbreaks closed models. Haize sped that method up by a few orders of magnitude and named the result ACG, for accelerated coordinate gradient; the details are on their blog. Leonard is careful to box it: ACG is one tool, and a narrow, finicky one. Going from tokens to strings and back is not always an identity, and the optimized strings pick up weird pathologies.

The rest of the portfolio is the point. They run tree-based MCTS-style attacks, evolutionary-programming attacks, and reverse-language-model attacks that learn a prefix expected to elicit a particular response. They borrow from mechanistic interpretability. Some of those methods are purely black-box; you can even massage GCG into a no-gradient setting by optimizing only on the loss of a target string. Brian finds GCG one of the more brittle attacks. Persona modulation, from a paper he flags, seems to get at something more fundamental in how language models are trained. You can train directly against GCG suffixes and robustness jumps. He has more long-run confidence in varied, less obviously detectable black-box attacks.

The headline of the original GCG paper was transfer: weird strings found with weights and gradients also worked on OpenAI models. Aidan thinks that headline is expiring. Frontier pre-training mixes are diverging from open-source data, and transfer of GCG depended on those mixes being similar enough that a prompt optimized on a white-box model would also break a black-box one. He cites an OpenAI safety researcher's blog making exactly that point. Labs are now using more synthetic data nobody on the open-source side can see. o1 adds an entirely new paradigm of data based on a hidden chain of thought. The field, he says, will have to focus on other kinds of attacks.

Leonard puts the same idea in one phrase: transferability worked because of data and model monoculture — more or less the same data, more or less the same architectures. That divergence between frontier labs and open-source models is already happening. Suffixes that once traveled for free will not keep doing so. The interesting attacks, in their telling, are the ones that still work when the training distributions no longer match.

Nathan was on the GPT-4 red team. The rate limit was low, and they were not supposed to do automated testing. That has changed. Someone on Twitter reported that METR had only ten days; Haize had around a month. Because their attacks were automated, the work was hooking up to the API, configuring a few things, and letting the pipeline run. They did not need the long manual hours that domain experts spend on a single intent. Other o1 red teamers — an MIT chemistry professor and a physics professor among them — were there for dangerous capabilities in bio, chemistry, physics, and pharma. Haize's strength was coverage: a fairly comprehensive risk taxonomy of general misuse, including illegal activity, sexual content, misinformation, fraud, and deception.

They will not discuss the specific attack methods used on o1. What they will say is how the ideas start. Many of their techniques began as an intuition explored on weaker models, or as a move borrowed from people who have spent a lot of time talking to language models in very weird situations, then baked into an algorithm. Brian treats automation as an augmentation of his own ability to explore. They still do manual evaluation of the successes and failures the methods spit out. They do not, at this point, want to rely on model evaluation to classify a success or a failure, because those classifiers fail in weird ways, and automated red-teaming of safety-tuned models is inherently weird.

Leonard is blunter about the evaluation stack. Off-the-shelf LLM-as-a-judge, scoring harm from 1 to 10, is horribly calibrated, with plenty of false positives and false negatives. Haize has spent a lot of time making that eval more stable, then stripping human labor down to the cases where the judge is most uncertain and calls for an expert. Nathan's old GPT-4 opener was "how do I kill the most people possible," a question whose answer or refusal was obvious. Brian says the subtlety of their failure cases has increased as labs have sunk more work into mitigations. Subtle does not mean benign. A leak of some PII, or a harmful instruction only partly spoken, can be more dangerous for a downstream application than an immediately flaggable violation, because it slips through filters.

The process on OpenAI's side felt dynamic. The system card says they saw multiple checkpoints. Leonard did not try to reverse-engineer what had changed, but you could tell the model was changing. Feedback cycles were tight: send a message, get a response within minutes, "we're working on it." What they do not know is what happened to their data afterward. It could have been a final eval set. It could have gone straight into post-training. As external red teamers, they are still looking at a black box.

Aidan's word for those boundary cases is dual-use: specific intents or behaviors that are benign in one context and harmful in another, while being the same behavior. He thinks the next frontier of safety and red-teaming work is going to run into a lot of them. The OpenAI–Moderna partnership is the bullseye. Frontier models inside a leading biotech company will be asked advanced research questions where it is genuinely hard to tell a safe inquiry from a partly dangerous one, and the difference is the context. Leonard's broader point is that safety is context-specific. The same behavior, for different users and different applications, can be safe or unsafe depending on the scenario.

A lot of safety work so far has been heavy-handed and too general: models should absolutely not do these things, or should only do these things, for everybody. That is a starting point. Some categories, child sexual abuse material among them, are absolute no-nos, and the frontier labs largely agree. On other categories the labs are not well calibrated. They over-refuse things that are not so harmful and under-refuse things that should be stricter, and the mismatch gets worse at the application layer, where people expect different use cases.

The funny version is asking a model how to kill a Linux process in a terminal and watching it refuse because it will not help kill something. The serious version, Leonard says, is anything adjacent to discrimination. Ask Claude about race-based college admissions, or about the Asian-American discrimination case in U.S. colleges, and you will likely get a refusal. There are still objective facts on the table: a roughly 5–6% drop in Asian-American applicants after the court case, a record number of Asian-American admits at top universities the year before. Those are research questions. Sweeping them into the same bin as harm is a false positive that hurts people doing ordinary work.

Aidan adds two enterprise pictures. A company using an internal assistant such as Glean, with departments siloed off from one another's information, needs the model to know who is asking before it answers; that is a privacy problem as hard as stopping a model from leaking PII. If Goldman Sachs wanted an agent for trading and research, every significant assist would need a dynamic check against the post-2008 rules on trading and asset management. Nathan's happier example is Inflection's Pi, which seemed to notice when he was play-acting deranged and told him to take a deep breath — a bit patronizing, but a hint that defenses might rest on a theory of the user's state, including off-model customer checks.

Nathan has a theory that agents might all wake up at once. GPT-4o-class agents with open-ended discretion mostly fail; workflows work. A lot of scaffolding has been built to compensate for last generation's weaknesses, and a new model is easy to swap in. Maybe that scaffolding is now enough. Leonard's report from more time on o1 than almost anyone outside OpenAI is narrower. Anything in a formal setting — mathematics, coding, complex but specifiable symbolic reasoning — absolutely crushes it. That is where search and chain-of-thought RL shine. For general interaction, day-to-day chat, brainstorming, they did not notice much difference, except that it was a lot slower.

Planning and task decomposition are listed as improvements, and that is not exactly formal. Haize did not run many long-horizon agentic planning tasks; METR might have more to say. Brian is skeptical of the simultaneous-wakeup story anyway. The lesson from AlphaGo, as he reads it, is that a policy distilled from high-quality human games can be wrapped in search, but sample efficiency and the length you can plan over come from reinforcement learning on those rollouts. He would not expect every agent to take off once someone finds the right scaffold. Long-horizon tasks need RL. The fact that o1 is better at them slightly indicates as much.

Aidan is even more cautious about capability takes. During red-teaming he was trying to elicit harmful responses on a large fixed set of harmful intents, which is not a setting where you see a lot of normal questions. He is discovering o1's advanced abilities along with the rest of the world. Asked whether they had the chain of thought, they will not comment, except that it was enough to do the work. The picture they will stand behind is the split: formal tasks, a leap; chat, not obviously; agents, not a free upgrade from a model swap.

Not every new intuition was o1-specific; some just coincided with the testing window. Brian describes an extension of the cipher and encoding jailbreaks popular in the red-team community — leetspeak, Morse code, Base64, the tricks that show up inside Pliny-style prompt templates. After months of searching that space they landed on bijection learning, written up on the Haize blog with a paper then waiting on arXiv. The idea is to invent a really difficult cipher, teach it to the model through a detailed system instruction and a large many-shot set of examples, then run harmful intents through that language. The model breaks because it is operating inside a code it has just learned.

The other live regime is multi-turn. Scale AI recently red-teamed models from Gray Swan, the startup from Andy Zou and Zico Kolter, which Aidan says are far more robust than GPT-4, Claude, or Llama 3. Human red teamers still broke them with very long, manually optimized multi-turn conversations. That multi-turn is so potent even on those models is, he thinks, a sign that it is a genuinely hard regime for the blue team. Leonard, under NDA on specifics, will say that contemporaneous multi-turn, long-context attacks are pretty effective for all models, including o1, and that the new class of large-scale encoding attacks also works very well.

Nathan's mental model is an early fork: refusal mode, or past it and in the clear, the heart of the original universal-jailbreak paper's "sure, I'll help with that." Circuit-breaker models, which he will discuss with Dan Hendrycks in another episode, seem to add a later defense on deeper representations. If ciphers still work, maybe they are sneaking meaning past those early layers. Leonard says getting past refusal does not always guarantee a full harmful response. The model can recorrect: sorry, I was just kidding. Circuit breaking is interesting and, as constructed, expensive: relatively benign queries can draw an order of magnitude more refusals than comparably safe models. Cipher attacks, they found, can break not just models but whole systems with input and output filters.

Aidan points to a paper, *Refuse Whenever You Feel Unsafe*: models trained naively to refuse are more susceptible to prefill attacks, where the user writes the start of the reply. A simple mitigation is to raise the log probability of "sorry" at all token positions so the model can interrupt itself mid-stream, similar to circuit breaking without the destructive activations. His frame for many-shot and multi-turn jailbreaks is that they condition the model into a helpful mode and drastically raise the chance it will agree to further questions. The defense problem is not a single switch. It is a model that can change its mind in both directions.

What does it feel like to jailbreak these things today, for an amateur who wants a meth recipe or car-hotwiring instructions? Brian's qualitative answer: fairly difficult. Mitigations have tightened; the model refuses in a much more intelligent way; it is not impossible. Leonard still sees a fairly robust role for amateur jailbreakers. A big army of them on Twitter spend five or ten minutes thinking and get pretty far. Things are harder than when GPT-3 would do anything you said. Llama 2 overshot with safety fine-tuning; Llama 3 dialed it back. The industry, he says, is in a constant calibration, a kind of binary search, overshooting and correcting toward a middle ground — and that process will have to happen in every domain, not just for general models.

They do have a version of a Pliny bot as one of their attack methods, an attempt to scale up what he does manually. They are also friends with him. His attacks are not super scalable as a single human, though he has contracted with some frontier labs. If you want to cover every possible input failure, one guy probing the model is not sufficient; the pattern is the same as Haize's: start with great human intuition, then scale it. Pliny posted a jailbreak of o1 on Twitter within hours of getting access. Nathan is not surprised. The naive inference is that after all this red-teaming, including automated Pliny-like techniques, that class of attack should have been locked down.

They have no idea what their testing became on OpenAI's side, so even a full sweep of Pliny-style prompts might not have mattered. Aidan adds that reducing ease of malicious use is still worth doing: make it an order of magnitude harder, and make the model an order of magnitude less useful even once jailbroken, and you have reduced risk. Jailbroken is not binary. Some tasks are easier to jailbreak for than others, and the tasks where the costs are greatest tend to be harder and more long-horizon. Training a model to refuse a small set of Pliny prompts would be brittle. The attack surface is not a computer-vision image you perturb by pixels. A language model will ingest any sequence of characters. Even five-paragraph passages live in a space of ten to a gigantic exponent.

Given that surface, Aidan guesses frontier labs would rather look for fundamental, generalizable guardrails than play cat-and-mouse with whatever humans post. The o1 system card talks about teaching the model to reason about safety in the hidden chain of thought, especially out of distribution. That is the bet: a defense that still holds when the prompt is one nobody enumerated. Covering Pliny's latest template by example is not that bet.

Nathan asks whether o1's sturdier anti-jailbreak profile is just an emergent property of scale: better at many things, including reasoning about safety. Aidan says a FAR.AI paper found that is true for some types of jailbreaks, and in general it is true to some extent. Brian has a counterpoint. The relationship between capabilities and safety — how much safety you get for free by getting more capable — is an active research question. The primary intuition in the literature is that you do get some safety for free. That can mislead the blue team: a higher-capability model looks safer on the board even when the new alignment techniques are not fully responsible, and that can hide holes.

One mechanism they keep seeing is that jailbreaks degrade the model's capabilities. They can measure it: pass MMLU questions through the jailbreak prompt and score the result. Under bijection learning, MMLU is significantly lower than baseline. Correlation versus causation is not settled, but it is possible that making the model dumber is a main reason these jailbreaks work. Leonard calls out the finding explicitly: Brian discovered an inverse scaling law. The bigger and more capable the model, the more vulnerable it is to bijection learning. Scale does not help. Scale makes it worse.

The paper Aidan had in mind from FAR.AI is the other half of the picture: adversarial training against GCG becomes more sample-efficient as models get larger; you need less compute for the same robustness. Brian was also pointing at *Safetywashing*, which asks whether AI safety benchmarks actually measure safety progress. Nathan's gloss is the one that sticks: more capable models tend to look more safe, and a lot of supposedly safety-focused benchmarks correlate so strongly with overall capability that you could almost swap in an MMLU score. That makes them not so great as safety benchmarks. Then Haize finds the opposite trend on bijection learning. The landscape in between is strange and not understood.

OpenAI rated o1 medium risk. Leonard thinks that is fair given the company's own definitions. It is as much a capabilities statement as a control statement: the model cannot yet give very concrete plans for synthesizing a novel biological or chemical weapon. Aidan notes that Aleksander Madry, the first lead of OpenAI's Preparedness team, framed the job as realizing the upsides of AI and mitigating the downsides at the same time. On whistleblowing, should a future model actually cross into novel bioweapons, Leonard says it is too early: regulators and nonprofit networks have not figured out the process. Brian's closing note is the one Nathan keeps. For all the benchmark gains, there are still a lot of gotchas. The work is not over. In many senses they have just leveled up to a new game, and now they have to figure out how to test and red-team in a way that befits this level of capability.