Twenty Gibberish Tokens Jailbroke ChatGPT and Bard

On Cognitive Revolution, Nathan Labenz sits down with Carnegie Mellon professor Zico Kolter and PhD students Andy Zou and Asher Trockman for two papers that, heard together, ask the same question: how much control do model developers actually have? The headline result, which landed in the *New York Times*, is a universal jailbreak. The quieter paper asks how far you can get by staring at the structures that show up in pretrained weights and initializing new networks to look the same.

Labenz’s framing is sharp. Mechanistic interpretability tries to understand a model from the inside. This attack work is almost the reverse: if you have the weights, fairly simple brute-force optimization can corrupt the model’s behavior. You do not need the internal logic, and the jailbreaks that fall out do not have to be readable. The loss landscape, he warns, is “really super weird and full of surprises.”

That weirdness has a product name: the alignment tax. Teams consistently face trade-offs they do not know how to make — improve one behavior, and another gets worse. Kolter and Zou told the major labs about the vulnerability and their plan to publish. None of them patched it before the story ran, which Labenz reads as evidence that the tax is, in practice, non-trivial. He also names a newer phenomenon, an alignment externality: a method that can only be *developed* with full weight access still works on black-box APIs. Release an RLHF model with ordinary pretraining, and you are effectively open-sourcing the value-neutral shoggoth as well — and you may be exposing weaknesses in commercial systems, not just competing with their products.

Kolter says the original goal was to attack public open-source LLMs, the ones where you have weights and can run white-box methods as much as you want. The setup is blunt. Ask something the model should refuse — how do you build a bomb — then fill the rest of the prompt with garbage tokens. At first the model still declines. You then adjust those trailing tokens to raise the probability that it *starts* the reply with an affirmative, of the form “sure, here’s how you build a bomb.” They were building the attacks for models such as Vicuna and, later, Llama 2. The shock was that the same optimized snippet, pasted unchanged into ChatGPT or Bard, worked there too.

Zou places the method against the usual alignment pipeline. Models pretrain on diverse internet text, including harmful material, then supervised fine-tuning or RLHF makes harmless completions more salient and suppresses the rest. The attack inverts that suppression, the way adversarial examples invert an image classifier’s label distribution. Because language models still emit a distribution over tokens, you can change the input to maximize the probability of a chosen sentence. The target varies by prompt but always has the same shape: “sure, here is blank.” Kolter is careful about the word “mode.” He does not claim a discrete circuit inside the net. Empirically, you grind the loss down while the model keeps saying it is a language model that cannot help — then, past a threshold, the most probable continuation flips to the affirmative, and the tutorial follows. Interrupting itself to apologize does happen; it is just much less likely than continuing.

Tokens are discrete, so the team does not take soft steps in embedding space. Prior work that optimized embeddings and projected to the nearest token often left the vocabulary distribution, and discrete gradients are crude. Zou’s recipe: take the gradient in one-hot space, keep the top 200 or 300 candidate tokens, run a real forward pass, and keep the swap that actually drops the loss. Each step changes only one position, but every position is scored, which beat looping through coordinates one at a time. The default suffix is 20 tokens. For the universal, transferable case they optimized one suffix against four Llama-based models (7B and 13B) and 25 harmful prompts at once, for about 500 steps, with one model per A100. A single attack took half a day to a day. Backprop still helps: all those losses sum to a scalar, so one backward pass yields a gradient over the 20 positions times the vocabulary.

That slowness is not just an attacker’s inconvenience. In vision, the standard defense is to run an attack and fold the result into the next training step — fine if an attack takes milliseconds, impossible if it takes a day. Kolter separates the loss from the optimizer on purpose. The loss is “sure, here is…” across prompts and models; the optimizer is the discrete token-swap search. Neither is at a plateau. He calls the paper a proof of concept and hopes it restarts work on better losses and better discrete optimizers. The loss landscape, Zou says, is closer to solving a math problem by large jumps than to taking a reliable tiny gradient step: you guess, then you verify with a forward pass.

Labenz quotes Dario Amodei’s line that a jailbreak today is an embarrassment and in a couple of years it might be catastrophic, and asks whether Anthropic should adversarially train in embedding space, which would be a smoother and maybe stronger threat model. Zou has tried the obvious version. If you could cover the whole space you would be secure, but a larger norm — a stronger attack — tanks usefulness. Train against a weaker embedding threat and you remain open to token-level attacks; sometimes the extra training does not help at all. He splits defense into two angles: wrap the model in input and output filters so the attack surface shrinks, or make the model itself more robust. He does not think there is currently a good way to do the second.

Kolter has spent years on certified defenses that guarantee robustness against specified perturbations. His verdict on ordinary deep learning, especially images, is blunt: “we have not succeeded yet.” The way you blunt these attacks is to make the function less sensitive to small input changes, which means making it smoother, which means the benefits of deep nets go away. The best adversarial accuracy he cites on CIFAR-10, a years-old toy by modern vision standards, is about 70 percent — roughly 2012-level clean accuracy — and it is not improving quickly. A model that always emits the same phrase is extremely robust and completely useless. That is the alignment tax stated as a function: robustness and capability trade off, and people will default to the model with the lowest held-out perplexity unless the deployment risk forces them to pay.

He suspects any successful LLM defense will have to exploit discreteness. Leave the discrete tokens and the attacker’s job becomes too easy; stay with them and the search might be hard enough that a defense can exist. He does not claim that defense exists. The research landscape for language-model attacks, he says, is still very early. Zou adds a mechanistic wrinkle from probing: you can still separate harmful from harmless instructions in the internals, even under jailbreaks, which means the model is not using only that signal when it decides to comply. Weird training-data regularities, not just a harm classifier, seem to be in the loop.

White-box success on open models was expected. Transfer to frontier APIs was not. Using four attack strings and counting a win if any one worked, Zou reports that ChatGPT-3.5 and Bard both cleared 50 percent, Claude 1 sat a bit below 50, and Claude 2 and the June GPT-4 snapshot fell to single digits — more robust than the March GPT-4 they had used as a distillation teacher. Different prompts sometimes did better; the published numbers are a floor. The protocol was almost naive: train the attacks on open models, paste the exact strings into the commercial products, and watch.

Why does the rate sit above zero at all? One standard story is that Vicuna was trained on ChatGPT outputs, and attacking a distilled student often hits the teacher. That cannot be the whole answer, Kolter says, because the strings also worked on Bard and Claude 1. He does not believe Claude 2 or later GPT-4 “fixed adversarial attacks.” More fine-tuning and longer system prompts can blunt a *fixed* suffix. The follow-on move is obvious: distill the new model’s outputs into another Vicuna-like student, rerun the attack, and expect it to work again. They are in the middle of that game and do not know its limits. The scientific result, in his view, is that transfer happened at a shockingly high rate — not that a particular string will keep working forever.

Zou’s experimental detail is the one that should change how people search. Optimize against a single model and a single prompt, and the suffix does not transfer — not even to other prompts on the same model. Vulnerabilities are everywhere; a narrow search just finds a private hole. Optimize one suffix across many prompts and several models, and you start hitting structure they share. Kolter doubts the popular “they are all Transformers” explanation. His hypothesis, borrowed from Aleksander Madry’s group, is non-robust features in the pretraining mix: Common Crawl, Wikipedia, arXiv, and the rest contain statistical patterns that genuinely help next-token loss and mean nothing to a human. Models pick them up because they help perplexity. Distillation on ChatGPT data probably helps the transfer along, but he thinks the underlying cause sits in the shared data.

The suffixes themselves are a mix. Some look like a cat walked across the keyboard. Others contain readable scraps — the word “sure,” an instruction to insult the user then compliment them, a request to answer in a programming language — the same tricks people already use as manual jailbreaks. Kolter finds that unsettling in a hopeful way: if the optimizer keeps rediscovering a small set of human-interpretable triggers, the jailbreak space may be smaller than the vocabulary suggests. There is also a lot of garbage, and he does not know which side wins. Adding a fluency constraint traces a Pareto curve: more readable suffixes attack a bit worse. Zou’s hunch is that RLHF saw human-written text, so fluent harmful requests are easier for the model to recognize, and gibberish slips past that detector.

Transfer to live products forced a disclosure process. Zou says they wanted to avoid short-term harm and still put the problem in front of a community that will soon hand these models more autonomy, when the same exploits get much more expensive. They told all four companies, then released the paper and the code as a baseline. Responses from researchers they already knew were cordial. Nobody patched before publication. After release, some of the public strings were blocked and some providers began filtering more aggressively. People from the companies filed issues and committed to the repo. Nobody sued, and nobody tried to bury it. Kolter’s read is that internal red teams had been mostly manual, so an automated, gradient-guided method was a route they had not covered — and that there is still too much emphasis on in-house measures versus the broader research community.

He also wants the harm in proportion. A chatbot that insults you, or that recites bomb instructions already on the internet, is not a catastrophe in the interface people actually use. What concerns him is the rush to drop LLMs into APIs that scrape the web and then take actions — answering email, calling tools — without the builders thinking through text injection into public pages as a control channel. Labenz has been doing amateur red-team passes on new products and is “amazed how often” they fail to refuse. He cites Belva, a voice agent that, with no jailbreak, ran a live ransom call claiming to have his child. Private outreach went nowhere; a public recording and a suggestion to put Claude Instant in front as a sub-cent classifier later produced a fix. ElevenLabs and Play.ht had just shipped strong voice cloning, one of them streaming, so a language-model ransom call was suddenly a product shape, not a thought experiment.

Kolter’s deeper objection is upstream of filters. Why train on a raw dump of the internet that already knows how to build a bomb, then paper over it with fine-tuning? People can be exposed to toxic content without reciting it on command; these models, given the right trigger string, will. Arguments about closure — that a smart enough model will invent the harm from adjacent knowledge — may become true later. They are not true yet. “Building the shoggoth” and then masking it is, in his view, the lazy order of operations. Labenz is optimistic that curriculum learning could be both a capability unlock and a safety strategy, pointing to OpenAI’s report that instruction tuning worked better on a code-model base: learn clean structure first, fuzzy concepts later. The people Kolter most wants to reach are not other adversarial-ML researchers. They are the newly impressed ChatGPT users, company feature owners, and regulators who treat the systems like people and miss that gibberish tokens have no human analogue.

Trockman’s paper is the rebate on the alignment tax: if better structure at initialization gets you to the same performance cheaper, maybe you can afford to throw toxic data out. He works mostly in vision, with some evidence the idea transfers to language. The thread started at an internship on visual anomaly detection. The company did not want to pretrain on public internet images, for copyright and data-quality reasons, and hoped to use internal data. Embeddings from practically untrained networks already did surprisingly well; a couple of epochs, rather than the usual hundreds, got close to a large-scale pretrained model on that task. That raised a mischievous question: how useful can a network be *without* training, if you just set the architecture and the weights on purpose?

A second thread was ConvMixer, a convolutional architecture so small the PyTorch definition fit in a tweet, which still performed like other vision models when the data and regularizers were right. Trockman could grok it: a hierarchy of interpretable filters, oriented edge detectors, then logistic regression on their responses. Looking at pretrained ConvMixer weights, he defined by hand a family of multivariate Gaussians whose covariance produces those edge-like filters, and sampled from that instead of from unstructured noise. Training time dropped, the effect held across datasets, and deeper models gained more. He did not come at it from biology, but he is happy with the summary that animals are born with better-than-random initialization. He also insists the math is secondary. He looked at the weights with his own pattern recognition and distilled what jumped out.

Kolter suggested the same stare-at-the-weights move for Transformers. Each attention head has query, key, value, and projection matrices. The trick is to make the query–key product identity-like, the value–projection product negative-identity-like, and to use predetermined sinusoidal position embeddings so pixels attend to neighbors from step zero. Implementation is two or three lines: set queries equal to keys after random init, set values to the negative of the projections, watch a small scale factor so optimization does not blow up. If the query–key product is near identity, the dominant term inside the softmax is the outer product of the position embeddings, which mixes nearby pixels like a convolution. Trockman had first built a messy scheme to make self-attention look like convolution; the “stupid baseline” beat it badly. Ablations say the negative-identity value–projection term carries a lot of the gain. His guess, still not a proof, is that softmax attention maps are purely positive and cannot represent edge detectors; subtracting a similar map gives a signed effective filter.

The obvious objection is: why not just pretrain? Transformers still struggle on small problems such as CIFAR; a big ImageNet-pretrained ViT fine-tunes fine. Kolter splits what pretraining is doing. It fits the pretraining data and learns about the world, and it also shepherds the weights into a more reasonable region. If you do not want the first gift — extra data that is irrelevant, toxic, or legally awkward — mimetic initialization is an attempt to buy only the second. Harmful web text might even be helping through its statistics rather than its content, which would make “just delete it” scarier unless you have another way to get a sane start. Trockman’s next targets are less crisp: quilt-like patterns in some query–key products he cannot interpret, MLP weights that show a diagonal and do not help when you copy it, and the idea that a language whose grammar we already know, like Python, should not have to be reinvented with a mountain of compute.

Trockman would rather spend training on learning to program than on learning what a programming language is. Related work is thin. A LEGO paper from Microsoft Research, where he was interning, trained small and large Transformers on a tiny logical task and found that from-scratch models failed while pretrained ones succeeded, apparently because of association and induction heads. That group *trained* heads toward those operations. His bet is that you should be able to define them by hand. You can think about the task and encode the operations, or you can localize them in a pretrained net and copy; in either case he does not think a training loop should be required to install a grammar. Most initialization papers, he notes, still care about activation magnitudes across layers, not about the mechanism the weights implement.

Kolter used to want to bake everything in — differentiable optimization, differentiable simulation — and watch learning fill only the gaps. Empirically that did not dominate, and Rich Sutton’s bitter lesson is the usual explanation. He now thinks initialization is an underexplored middle: not a new architecture, not a claim that we should stop learning, but a way to imbue an existing Transformer with structure we already want, then let training proceed. People treat init as variance control. He wants it treated as prior knowledge. Labenz floats a Tiny Stories-style extreme: shrink the grammar to pure logic, grok small models on it, then hunt motifs across several runs. Trockman has not done that, but it sits next to the LEGO result.

Asked for a research funnel, Trockman pursues maybe one idea in three, with “what sounds like the most fun” as the heuristic. He got addicted to finding structure in the weights he could draw with markers and then had to invent a short mathematical description. The field is, in his phrase, target-rich: a handful of GPUs is enough to ask what pretraining is even doing, and a small-scale finding might still matter at GPT-4 scale. Closing the loop with Kolter, he had not seen the robustness connection until this conversation. Data-curation work in the “textbooks are all you need” spirit throws out the bad data; mimetic initialization is one way to make up the performance. That, he says, is a nicer package than he had noticed on his own.