A Gibberish Prompt Unlocks What GPT-J Plans Three Tokens Ahead

Nathan Labenz opens with the Bau Lab’s unusual brief: crack the black boxes of modern models, recover the ability to debug them, and eventually edit their behavior the way a programmer edits a line of code. David Bau says the motivation is simple. Machine learning has become a new kind of software, and the classical computer-science toolkit for proving a program correct no longer applies. What remains is statistical and black-box. The lab asks whether researchers can take responsibility for the behavior that emerges from training.

Bau left Google around 2015 once it was clear the field was changing. His personal turning point was AlexNet in 2012: a 1950s idea trained with an 1980s method, scaled up, suddenly worked. He first studied GANs, expecting to stay there; diffusion models and then language models kept moving the target. Labenz notes that early interpretability work, including Chris Olah’s, also started in vision. Bau’s path sits on that same lineage.

The tomato analogy is the one he keeps. Asking for total control of a trained network is like asking for total control of a tomato’s inner mechanisms. Both grew rather than being written. Biologists reverse-engineer the fruit if they want to improve it; the lab is trying to build the equivalent science for neural nets. ROME localized factual associations well enough to rewrite “Michael Jordan played basketball” into baseball across many phrasings. MEMIT scaled that to about 10,000 facts. A forthcoming ICLR paper on linear relational embeddings asks where the *relation* lives, not just the pair, and finds some relations are linear maps while others look nonlinear—an open question about what the extra complexity buys.

Koyena Pal’s project did not start as a theory of planning. She was looking at memorization: feed ChatGPT the first tokens of a license header and it can recite the next 11 to 20 tokens from training data. That led to a sharper question. For a phrase like “New York City,” treated by people as one entity, how much of the unused future is already sitting in a *single* hidden state? The paper asks, in her words, to what extent future information beyond the current token can be extracted from one hidden representation.

The predecessor tool is logit lens, from the blogger nostalgebraist. It projects every intermediate state through the model’s own unembedding so each layer becomes a word. Early layers look crude; deeper layers converge on the actual next-token prediction. Bau tells listeners who have never seen it to search for the grids: they are, he says, beautiful. Pal’s complaint is that the lens only shows what the model is thinking *now*. When the current token is “New,” it cannot tell you whether the network is already committed to York or Jersey.

Labenz draws the geometry. Computation down the stack is vertical; information among tokens is horizontal, and causal attention only flows forward. English already hints that models look ahead: they choose *a* versus *an* correctly when the upcoming noun starts with a vowel. Pal ran the study on GPT-J, a 6-billion-parameter model with 4,096-dimensional states and 28 layers, because she already had it from the memorization work and could run it on a single GPU. She sampled 100,000 Pile sequences on which GPT-J already predicted the next token correctly, plus 1,000 for testing, and kept only the last-token hidden state at each layer.

Five methods go head to head. A bigram baseline, using co-occurrence counts on the Pile, already guesses 20% of next tokens. Labenz is surprised the number is that high; Bau reminds him n-grams were state of the art not many years ago. Direct vocabulary prediction trains a linear map from the 4,096-vector onto GPT-J’s roughly 50,400-token vocabulary. Linear approximation instead predicts a future hidden state and lets the model’s own decoder finish the job. Those two curves sit on top of each other. Pal reads that as linearity having already squeezed this channel dry. They beat the bigram a little on the current token and lose to it two and three tokens out.

The more natural idea is transplantation. Pal patches the hidden state into a bland carrier such as “Hello, could you please tell me more about,” hoping a generic prefix lets the borrowed vector speak. It does not. In some settings the fixed prompt is worse than the bigram, and it gets *worse* in deeper layers. She tried many prefixes, including a bare period. Bau’s analogy is sticking a random probe into someone’s brain and hoping they smell bananas. Jamming an alien state into a live forward pass is, he says, a kind of brain damage. The published prompt is already their best.

Soft prompts abandon English. Pal learns ten embeddings in the same 4,096-dimensional space, about 40,000 parameters, trained to pull future tokens out of an intervened state. That tiny prompt crushes every other method. Decoding it back to discrete tokens yields gibberish—stray @ signs and word-like junk. Labenz hears an echo of universal jailbreaks, which also often look like noise. Bau’s summary: this is the power of using a transformer to explain itself. The model weights stay frozen; the prompt only learns how to ask. A linear classifier with 4,096 × 50,400 parameters, roughly 200 million, cannot match a key five thousand times smaller.

Figure 4 of the paper is four accuracy plots: current token, then one, two, and three steps ahead, layers on the x-axis. Current-token prediction recapitulates logit lens—deeper is better, approaching certainty at the end, a sanity check that the method works. For future tokens the soft-prompt curve rises through the middle layers and then falls. When Pal scores two and three steps out, she feeds the model the correct earlier tokens and asks whether it can continue. Linear methods stay mediocre; the fixed prompt even drops below the bigram on the first future token.

Pal’s picture of the stack is that early layers build an understanding of what to say, middle layers hold a set of ideas, and late layers collapse onto the immediate next word. Bau adds that later layers appear to *erase* information to free space for something else. A transformer wiping its own future is, he says, a phenomenon worth studying on its own. The peak in the middle also matches other interpretability papers. Labenz guesses later peaks in larger models are a size effect: you still need a cash-out phase at the end, which limits how deep the buildup can run.

The deeper claim is that the information is present and still not linearly readable. Bau imagines a dictionary in the weights that maps concepts onto word sequences. The hidden state is a pointer, not a recipe. The running example is the movie title “Back to the Future”: the compact vector does not contain the unrolling instructions, and the soft prompt is the key that triggers lookup. If decoding were simple, a linear head would have found it. Pal agrees a full transformer probe would probably work better, but 100,000 examples are nowhere near enough to retrain one. That is Bau’s point. A hundred thousand samples cannot teach a student network what GPT-J already knows; they can only *unlock* a dictionary that is already there. Stacking another transformer to explain the first one, Labenz notes, becomes a chicken-and-egg problem that slides toward anti-interpretability.

If listeners still want to hunt for a magic fixed prompt, the lab is not precious about it. Pal first did the transplants with Baukit, Bau’s tracing library. The successor is nnsight, at nnsight.net, which does extraction and patching in a few lines of code. Future Lens itself is released at future.baulab.info, with notebooks and scripts. Pal has run the *testing* half on Colab Pro. Bau says they never cracked natural-language transplantation, and they would be delighted if someone found a cleverer setup they missed.

nnsight is still early, almost a secret alpha. The site has documentation and a Discord link in the corner; Bau says the valuable move is to join and ask, because the community is small and friendly. The library talks to a remote backend they call NDIF, a deep inference fabric they are trying to fund as shared infrastructure. Flip a flag and the same experiment can run on hosted GPT-J or Llama models, including Llama 70B, which Bau calls a pain to run yourself. It is not even transformer-only: people are already using it to study Mamba.

Labenz’s bet is that bigger, more heavily trained models will show higher Future Lens success rates, because their middle states should be more semantically meaningful. Pal leans the same way, with the caveat that bigger models also need bigger probe datasets. Middle-layer importance, at least, she expects to transfer. Bau, speaking as advisor, labels it a gap. The intuition is counterintuitive: the usual story is that scale makes systems *harder* to interpret. If scale instead adds structure, that would be quite a claim, and it needs systematic experiments they have not run.

On attention-free architectures he is carefully incomplete. They are halfway through the work and he does not want to mislead. Even without classical attention, Mamba-style models seem able to do some of the sophisticated computations transformers use attention for; how, they are still untangling. Labenz plugs StripedHyena, a hybrid used in a recent DNA foundation model that beat both a pure transformer and Mamba, and bets mashups will win in many settings. Bau says they will look carefully. History, Labenz likes to say, is not over.

Eric Todd joins to answer how in-context learning actually works—the GPT-3-era trick of inferring a task from a handful of demonstrations. He finds a small set of attention heads that, while the model processes an ICL prompt, mediate task identification. Give antonym pairs (big/small, short/tall) then a query like “bright,” read out those heads, and add their summed output into a clean prompt such as “the word fast means.” Instead of defining *fast*, the model says *slow*. The task has been injected as a vector. Capital-city demonstrations yield the same trick: Paris–France, Moscow–Russia, then Ottawa, and the patched clean context names capitals rather than antonyms.

They tested 40-plus tasks—antonyms, translation, capitals—and the same small set of heads mediates them. The prompt never names the mapping between demonstration and label; the model figures it out and ships that information forward. Todd calls attention a natural bottleneck because it is how transformers move information between tokens, and heads already act in a small subspace. The function vector is the sum of the selected head outputs, added at a chosen layer. Length matching does not seem to matter. The patch works in natural text and in templates, long or short.

Causal mediation analysis, also called activation patching, finds the heads. One sentence demonstrates a task cleanly; another scrambles the labels so the task is unreadable. Patching a single head from clean into scrambled nudges the right answer up but does not flip the prediction. Around ten heads, the causal effect becomes large. Bau’s surprise is that the heads are *shared*. He expected antonym heads. Heads selected on one task induce a different task when their activations come from that second task. They mostly sit in middle layers, attend to previous label tokens, and match other work on where ICL stores information. Todd’s two-phase picture: the first half of the network figures out what to take from context; the second half spends it on the next token. Task recognition lands right in the middle, just before generation.

Labenz asks what this means for mixture-of-experts models—GPT-4 rumored as MoE, Mixtral open-sourcing the idea, Gemini 1.5 described as large-scale MoE—and for the habit of swapping MLP blocks, or now attention blocks, at runtime. The function-vector finding looks distributed: a few heads per layer across the middle, then quiet. Bau answers as an interpretability advisor. We know the parts list, he says: dimensions, nonlinearities, MLPs, attention, state-space recurrences, all matrix multiplies in the end. We do not yet know the right way of looking at those computations.

ROME found associative structure in MLPs; a generation of work found computational structure in individual heads. When a dozen heads work in concert, the next question is whether they should be viewed together. Function vectors are one proposal: collapse the heads into a single vector, lose fine grain, keep strong causal effects, and even some composition. Bau does not claim they are the final abstraction. They hint that more structure may sit *above* the units people have been staring at. “The big game,” he says, “is to figure out the answer to the question: what is the right level of abstraction?”

Labenz closes as a fan of the lab’s paper sites and animations. Bau shout-outs a student, Nikhil, whose ICLR paper “Fine-Tuning Enhances Existing Mechanisms” studies the interplay of circuits and fine-tuning and, Bau says, has the best circuit visualizations he has seen. They save that one for next time. The through-line of the hour is already sharp enough: the black box has a dictionary, a keyhole, and a job called “notice what task this is,” and the tools to poke them are sitting on a public URL.