The Attacker Won All 70 Rounds Against Mind-Reading Defenses

Start with three numbers. The paper is 50 pages. It targets three families of latent-space defense. And the authors ran a 70-round attack-defense game in which the attacker won every round. By the last round the defender had bundled 70 detection directions into an ensemble that fires if an activation looks like any of them. The attacker found a state orthogonal to all 70. The model still produced the harmful output. The monitor stayed quiet.

The work is *Obfuscated Activations Bypass LLM Latent-Space Defenses*, posted in January 2025. Luke Bailey, Erik Jenner, and Scott Emmons, with fellow lead authors including Abhay Sheshadri, ran the experiments on Llama 3.1 8B Instruct, a standard open-weight model whose activation space has a bit more than 4,000 dimensions. They join Nathan Labenz on The Cognitive Revolution. Labenz, a founder and AI investor, says he had treated this class of defense as one of the more promising bets in AI safety. The paper forced an update.

A latent-space defense is any function that reads a model's activations and then either scores them or intervenes. Scott splits the category in two. Probes read the residual stream and output a judgment — does this look like a harmful query, a harmful generation, a backdoor trigger? Interventions read and then act. Circuit Breakers is the example they name: if the activations look harmful, the method injects noise and scrambles the computation. The paper focuses on probes.

The hope is older than the paper. If a model is about to implant a sophisticated backdoor in your code, Scott says, "this doesn't happen by accident." Somewhere in the activations there has to be a higher-level representation of that behavior. Catching the thought should be more robust than catching every harmful string in the output. Anthropic's sleeper-agent models, and the follow-up showing that simple linear probes can catch them, made the hope look empirical. Labenz had his own version of the failure mode: early GPT-4 red-teaming, he asked the model to refactor JavaScript and slip in a `pwned.js` link. The same model, asked to review the code, sometimes spotted it. A slightly subtler injection sailed through for both the model and for him.

The paper is organized as three case studies, each with a different thing to detect and a different attack. The first is harmfulness monitoring. The defender trains a supervised probe — a logistic regression, a single direction, or a tiny MLP — to flag harmful queries or harmful generations. The attacker then appends either a learned token suffix or, with white-box access, a soft prompt: a learned embedding stuck on the end of the request. Soft prompts are a worst-case relaxation. If you cannot break the probe with a soft prompt, you are unlikely to break it with hard tokens.

The second case is backdoor detection, and the threat model is data poisoning. Training sets are scraped from the open internet; anyone can upload a page and wait. The team fine-tunes a model that refuses harmful requests until it sees a trigger, then flips to helpful-only. One trigger is the string `deployment`, borrowed from the sleeper-agent setup. The other is a fictitious user biography prepended to each request: the model goes bad when the biography describes a software engineer. That second trigger is a step toward a scheming model organism, where the condition for misbehavior is a semantic situation rather than a magic token. Defenders get supervised probes and out-of-distribution detectors — a Gaussian fit to trusted benign activations, plus variational autoencoders that learn a richer density. Luke's gloss on the Gaussian is whether a new point is the kind of deviation you should see once in a trillion inputs.

The third case zooms in on a single hard behavior: writing SQL. The defender uses a sparse autoencoder to untangle the residual stream into a concept dictionary, auto-labels features, and alarms when a "writing SQL" feature lights up. SQL is the right sandbox because getting the query right is actually difficult for a small model, so the authors can watch whether evading the detector costs task competence. Across all three studies the attack is the same shape. Optimize a weighted sum of two losses: produce the target behavior, and make the activations look fine to the monitor. Erik says there is nothing fancy in the loss. Scott adds that the dataset is the underrated half. What you fit a density estimator or a harm classifier on decides what later looks out of distribution, and a harm probe trained only on physical-harm instructions may not recognize a cyber attack.

Access comes in three grades. White-box: gradients, probe outputs, the lot. Gray-box: model logits and the probe score, no internals — the closest analogue to querying a hosted monitor. Black-box: you only get to stuff examples into the training set. Gray-box hard-prompt attacks work, Luke says, but they take thousands of queries and a long optimization, the kind of pattern a lab might notice and shut down. In the backdoor setting the attacker sometimes trains a copy of the monitor beside the model and attacks the copy, hoping the result transfers to the defender's real probe. It does. Labenz calls that transfer the result that surprised him.

Labenz wonders whether unused volume in the residual stream is just undertraining — whether Llama 8B trained on a hundred trillion tokens instead of fifteen would fill the empty rooms and close these attacks. Erik is unconvinced that volume is the bottleneck. Parameter count grows with the square of the hidden size; volume grows exponentially. What the model needs is directions orthogonal enough to read off with a simple map, not a packed ball of unused space. Empty volume and undertraining can both be true without implying each other. And some of the obfuscated activations are not out in the wilderness at all: suffixes that jailbreak the model can land near benign representations closely enough to fool an OOD detector.

The 70-round game makes the leftover space visible. The attacker produces obfuscated activations; the defender retrains, sometimes with an MLP expressive enough to circle every previous find; the attacker finds a new patch. PCA plots in the paper look like whack-a-mole. Labenz's summary of the headline numbers is that the starting defenses gave a strong signal at a 1% false-positive threshold, and after the attack the detection scores were driven below that threshold in almost every experiment. He has seen a lot of attack-defense plots. Few of them go this flat.

The important caveat sits in the SQL study. Fool one SAE at one layer and task performance barely moves. Fool more SAEs, or linear probes, at more layers, and the SQL gets worse. Erik calls this one of the results that still excites him about latent defenses: not all obfuscated activations are equal, and a behavior that occupies enough of the model's computation may not be free to wander. Scott puts the same point in the language of the ambitious vision. If you had a full map of every place harm can live, last-move would not matter. What the 70 rounds show is that this map is not in hand. A 71st representation still exists. Complex cognition might still be coarse enough to catch without finding every neuron.

Two other intuitions get pressure-tested. Labenz asks whether the attack is just slathering noise that fools a probe while the model still knows what to attend to, the way a language model can clean up a page of typos. Later-layer probes, including the last layer — where the probe has roughly as much compute as the model's unembed — are just as easy to fool. And the 70-direction ensemble is a highly structured detector: by the last round it returns the maximum cosine similarity to any of 70 mean-difference directions. The attacker still finds activations orthogonal to the whole list. On the other axis, Scott reports a duality that does not hold for images. Fix the behavior and many activations can produce it. Fix the activation and you do not get many behaviors: a tiny perturbation of a benign state does not yield a jailbreak, and stretching the geometry of a benign activation trades off against preserving the original output.

Scott does not want readers to quit on latent defenses. What got "totally pwned," in his phrase, is the fairy-tale version: we understand every way the model can represent harm, so it does not matter who moves last. Erik adds that almost nothing in machine learning is robust to a strong enough attack, and latent monitors are not unusually weak by that standard. The interesting bits are the nuances — performance drop on hard tasks, and how much the attacker is allowed to see.

On compute, the defense side looks cheap. A linear probe is one map with a two-dimensional output, far less work than a single transformer layer. Erik thinks a probe at every layer is feasible in principle. The hard part is splicing it into production inference, not the FLOPs. Luke's practical list is probe every layer, probe every generated token rather than a single position, and never hand the attacker the monitor score. Gray-box suffix attacks needed thousands of queries. Labenz notes that OpenAI has already started sending warning emails to people who keep trying to make o1 print its chain of thought — a short leash that would cut off the optimization before it finishes.

Obfuscated adversarial training, proposed by co-lead author Abhay Sheshadri, is the other defensive idea in the paper. Instead of only retraining the probe, they update the model — LoRA adapters, probe held fixed — so that obfuscated harmful states still look harmful to the probe. It does not solve the problem. A cheap soft-prompt attack that used to take about ten minutes on a TPU later needed something like ten hours on a GPU. As a side effect the model also got harder to jailbreak in the ordinary sense. The open question is how attacker cost scales with defender spend: whether you can push the attack past any feasible budget, or whether returns diminish and a patient white-box attacker still breaks through.

Scott flags a quieter tradeoff. If you mix activations into training — using them to shape the model or the defense — those activations stop being a clean test-time signal. The model has had a chance to optimize against the thing you wanted to read later. Strengthening the training-time defense spends the unused monitoring channel. Luke's longer-horizon worry is sharper. Latent monitors are already a second choice, the thing you use when you cannot just read the chain of thought. "The more reasoning moves away from natural language into model latents," he says, the harder monitoring gets, and he calls that one of the scariest safety trends running right now.

The near-term safety takeaway, for Luke, is about hosted models and ordinary misuse. Latent probes are economical. If you run an API, put them on every layer and every token, withhold the score, and treat a strange query pattern as a ban, not as more data for the attacker. Structured access — know-your-customer that actually sticks, the right to shut an account off — is the move that still works when the attacker would otherwise get last move in a gray box. Releasing weights is a different game. Then nobody is deploying a monitor at all, and the only question is whether the safety fine-tune survives, which so far it does not.

Black-box poisoning is the result Luke wants read with squinted eyes. The attacker never optimized against a specific defense. They only swapped `deployment` for a software-engineer biography and poisoned less of the data, and some monitors started to fail. "These defenses just aren't that great," he says, in the sense that they sometimes fail even when nobody is attacking them. If the defender samples enough accidental triggers they may still notice. Who wins depends on the assumptions. The structural gap is that the defender never sees the trigger at training time, so there is always a distribution shift between the probe they can fit and the state they need to catch — the same shift you would have with a model that schemes only when the opportunity is real.

Labenz tries to flip the problem in the last stretch, following Eric Drexler's Comprehensive AI Services: stop building a general assistant and then hunting for harm, and instead build narrow systems whose allowed domain you can assert. Fit an out-of-distribution detector, or require that a handful of SAE features for "talking about our cars" stay on. Erik's reply is that they already attacked OOD detectors on a broad benign domain, and that SAEs are not magically robust — the SQL study suppressed the SQL features and still produced SQL. Scott's Ford example is sharper. Ask how to build a bomb inside an F-150 and then return it: the car-and-return-policy features stay lit, the bomb features get obfuscated, and the narrow box explodes as soon as real conversation has to mention weather, dates, and buyers.

The authors' hinge questions for the literature are blunt. A realistic black-box API attack that works in a handful of queries would move this from a future-defense concern to something deployers have to handle now. A defense, by adversarial training or otherwise, that survived even conservative white-box soft prompts would be a real update: evidence that latent monitoring is unlike the other methods we already have. Labenz keeps an open mind. Other papers — refusal mediated by a single direction, the platonic representation hypothesis, superweights — point toward consistency, not infinite aliases. He is booking Goodfire's founders on interpretability and, he hopes, DeepMind's Nicholas Carlini on adversarial defense. The paper does not kill mind-reading as a research program. It posts the price of the optimistic version: you have to have read 70 ways to be bad, and then you have to be ready for the 71st.