Half an Hour to a Circuit in GPT-2 Small — and a Quadratic Wall Before GPT-3

Nathan Labenz has talked about mechanistic interpretability on The Cognitive Revolution as one of the more promising paths to long-term safety, but he has rarely walked through how the work is actually done. His guest is Arthur Conmy, who has just published a paper and a software library aimed at automating the most cumbersome step of the usual research workflow. Conmy's definition is blunt: mechanistic interpretability is "the reverse engineering of the learned algorithms that neural networks implement into human understandable concepts." A model is an algorithm that turns inputs into outputs. The field's job is to say which high-level variables it is using internally — not merely that this matrix is multiplied by that matrix.

Labenz offers a three-layer picture: first describe how information is processed at all; then ask what the lighting-up parts are actually doing in an intuitive way; then ask whether the strategy is general understanding or still a stochastic parrot. Conmy maps the first two layers onto the work as it is practiced. Step one is to find the important subcomponents of a huge network. Step two is to ask what that subset means. The third layer, he says, is one of many use cases: if you can read the algorithm, you might finally give a yes-or-no answer to whether the model is reasoning or "just a wad of heuristics and statistics."

What drew him in was an undergraduate mathematics degree, a subject that is almost entirely about understanding how things happen. Machine learning, by contrast, is not a field where understanding how algorithms operate is the usual path to the next technique. It is a blind optimization procedure. It does not have to stay that way. Conmy's research goal is to explain more behaviors of neural networks in terms of their internal components, so that the algorithms those networks learn become more understandable in human concepts.

The stochastic parrot hypothesis is still a hypothesis, Conmy says, and it looks correct only because evaluations have not probed deep enough. Labenz's working sense is that it is often both: at some scale, generality starts to appear, and there are enough examples of grokking to believe that more than correlation is happening, but for any particular task you still do not know which regime you are in. Conmy agrees that models sometimes perform the correct algorithm — basic math and reasoning benchmarks have been set and then destroyed — and that they also rely on surface heuristics. The hard problem is distinguishing the two, especially when a new size of model suddenly does something previous models could not.

GPT-4's coding ability surprised many people. At the moment a capability first emerges, it is unclear whether the model is pattern-matching in enough cases to convince humans or has reached some genuine understanding of code. That distinction matters for future risk. If we cannot predict what is coming, Conmy would at least like to know that a new skill is a surface heuristic rather than a completely general solution to something we thought was hard. Unpredictable generality, he says, is a source of instability when systems are deployed.

For Labenz, the most important sentence in the GPT-4 technical report is a short one: certain capabilities remain hard to predict. The report shows the reversal of an inverse scaling law. Bigger models had been more susceptible to hindsight neglect, a reasoning fallacy, until GPT-4, when the trend flipped. OpenAI can plot many smooth curves and still cannot say, for a given task, whether it will be understood. That, Labenz argues, is the unpredictability Conmy is pointing at. The rest of the conversation turns from that motivation to the models researchers can actually open today.

Conmy's paper splits the standard workflow into three steps. First, choose a behavior the network can perform — modular addition, for example, or predicting a later year in a century. Second, define the scope of the interpretation: neurons, attention heads and MLPs, or whole layers. Third, run intervention experiments that find which of those units matter. Step three is generally the most laborious for a human, which is why Conmy and collaborators tried to automate it as circuit discovery. In practice the whole pipeline is constrained by which models you can load and which modest tasks those models can do. Compared with GPT-4 or Claude 2, the work is still happening on relatively small networks.

Two criteria guide task choice. The first is localization: the behavior should be thinkable as distinct from the rest of natural-language computation. Predicting a future year from an earlier year in the same century can be pinned to particular token completions. Producing a non-toxic or non-harmful response is open-ended, completable with a huge number of tokens, and distributed through the network, so it is much harder to interpret. Tractability often means narrowing a wide concept until a project can actually finish. The second criterion is relevance to what bigger models do. Fact recall is a favorite: if a person is mentioned a certain number of times in the training data, will the model store that information? That question will still matter for systems people deploy later.

Labenz wonders how much human introspection should feed the menu — years feel decomposable in a way that "why did I say something toxic" does not. Conmy's reply is careful. Researchers do choose tasks humans know how to perform, because that is how you write metrics. Models, however, often do those tasks in very different ways. In circuits they discovered, a model choosing the correct name at the end of a sentence would aggregate all the names and then remove the deleted ones, which is not how humans reason about names at all. Human-understandable tasks are a starting point. Human-like algorithms are not a guarantee.

A task in this literature is not one dataset. It is two. You need prompts that exhibit the behavior, and you need a contrasting set, because you cannot simply delete the rest of the model and still get a forward pass. The default in much of machine learning is pruning: set weights to zero so a neuron never fires. That is a bad idea here. During training the model never saw huge numbers of components outputting zero. It saw values on some distribution, with the occasional unit firing a bit more than usual. Zero is off-distribution, and the rest of the network becomes confused.

The better move is to replace a component's activation with the activation it produced on the corrupted, or baseline, dataset. For the year task, taken from a paper by Michael Hanna and coauthors on how GPT-2 does greater-than, the experimental prompt is along the lines of "the war lasted from 1517 to 15…" and the model must complete a later year. The baseline is "the war lasted from 1500 to…," where any year in the 1500s is fine, so the model need not compute greater-than. Components that matter on the first set and not the second are the ones doing that comparison. Conmy's medical analogy is a placebo hormone rather than cutting out a gland. At Redwood Research, some of the group's results supplied evidence that corrupting with another example's activations is more representative than zeroing.

Metrics are a practitioner's choice, not a single default of "did the output stay the same." On the war sentences, correctness is a later year, not 1516. The researchers summed probability on the good years and subtracted probability on earlier ones. Matching the original model's full distribution would have been too coarse: the original model is sometimes wrong, and it is always juggling more than the task. Pretraining is next-token prediction across the internet — books, forums, badly transcribed pages — so the model balances a "correct" completion against irony, jokes, dumb mistakes, and the chance the document simply cuts off. A task-specific metric strips that long tail. Labenz draws the counterintuitive conclusion, and Conmy agrees: an isolated subcircuit can do better on this one behavior than the full model, because the full model is running many superimposed heuristics at once.

Step two of the workflow is choosing a level of abstraction, and Labenz reaches for biology: ecology at one end, genetics at the other, with cells, organs, organisms, and species in between. Conmy accepts the analogy more readily than most, because he expects mechanistic interpretability to develop more like biology than like physics. Neural networks and evolution both trained on a "stupid" objective and still produced complex behavior. In a young field with few researchers, the choice of zoom is mostly a question of how ambitious you are — how far you are pushing the best current explanation of a behavior of a given complexity.

The examples already span a wide range. Neel Nanda and collaborators have neuron-level accounts of how toy transformers complete modular addition, a landmark grokking result. Understanding of GPT-2 small is much worse and still mostly in terms of attention heads and MLPs; Conmy was a collaborator on the IOI paper, "Interpretability in the Wild," which last year reverse-engineered a complete circuit inside GPT-2 small. A third rung down, the ROME paper — the Eiffel Tower is in Rome — locates factual recall in whole attention layers, bundling many heads, and then edits that recall. Finer is not automatically better. If you try to explain flight at the quantum-mechanical level, Conmy says, he wishes you luck and does not expect you to finish.

Turning that lunch-table choice into a causal acyclic graph in code was one of the fiddlier parts of the ACDC project. Extracting activations is not so hard once you know a forward pass; libraries already help. The harder implementation is editing the effect of one earlier component on one later component. By default, code runs end to end and an early unit affects every downstream unit. Interpretability wants something more fine-grained: what does layer one do to layer eight, specifically? Educational material now makes it easier to isolate individual heads, but the causal graph is richer than the naive sequential implementation.

The conceptual leap is the residual stream, a finding from Anthropic's interpretability team in "A Mathematical Framework for Transformer Circuits." Machine learning usually talks about a hidden state transformed layer by layer. Residual connections mean you apply a transformation and add the original state back, so the forward pass is a continual stream of information that components read from and write to. That view lets you model the effect of a very early unit on a specific downstream unit. In the default picture, you never even ask what layer one does to layer eight. Labenz found Anthropic's diagrams clarifying: the residual stream is the central object, and heads and MLPs are side loops that contribute to it.

Given that setup, ACDC is a three-step algorithm that imitates the human process without a human in the loop. Represent the model as a computational graph at the chosen abstraction. At a given node, look at every input edge and, one by one, set that edge's activation to the baseline-dataset value. If the downstream metric drops by more than a threshold, keep the edge; if it did not matter, cut it. Recurse through the nodes. The remaining subgraph is the circuit. To corrupt only the connection from an early head to a late MLP, you treat the MLP's input as the sum of previous writes to the residual stream, subtract the clean contribution of that head, and add its corrupted output. Everything else stays clean.

The implementation trick is caching. Store the clean and corrupted values of the early head, run the forward pass to the MLP without having changed that head, then force the MLP to have "seen" the corrupted value. That separates the head's effect on middle components from its effect on this particular MLP. Labenz would rather specify a target sparsity — how sparse can the graph be while still succeeding — than a local threshold on each edge. Conmy would rather have that algorithm too. He does not know how to design one. In advance you cannot tell which of a huge number of subgraphs is best; grabbing an X-sparse graph immediately is usually hopeless at the task. The local threshold is a proxy. After the output node is processed you can see how many edges it kept and guess how dense the rest will be, then sweep. In practice, researchers often stop early once a revealing subgraph appears, rather than sweeping every hyperparameter.

Validation put reproducing circuits that earlier practitioners had found by hand — the greater-than year circuit, for example — ahead of "does the subgraph still do the task." The audience is interpretability researchers; the first step toward semantic meaning is showing that the tool recovers components they already trust. Subgraph performance may correlate with that, Conmy says, but ACDC was never meant to build a great year-predictor. He does not oversell it as a way to find the best subcircuit for a behavior. Interventions on edges involve substantial caching and recomputation, so it is not competitive with other ways of eliciting capability. Complementary work, such as Ziming Liu's, changes the loss during training to encourage modular structure. That is selecting for interpretability. ACDC is post-hoc: a fixed black box, opened after the fact. The two need each other. Anthropic's SOLU activation is a warning: a training change meant to make models more interpretable led the network to hide superposition by a confusing route. Sparsity does not mean interpretable, and it does not mean grokked.

The compute was not a frontier-lab supercluster. It came mostly from FAR AI, where collaborator Adria Garriga-Alonso works. Practitioners can get results on GPT-2 small in half an hour to an hour when runs are sparse and well behaved. Two bottlenecks remain. Choose the threshold badly and you search every node; Conmy first says the cost is roughly linear in nodes, then corrects himself: because the algorithm iterates over edges, and a layer-zero head affects almost every downstream head, doubling nodes roughly quadruples edges. GPT-2 small is a 100-million-parameter model — large by the standards of ten years ago, tiny next to GPT-3's hundreds of billions. ACDC is not feasible at GPT-3 scale and is not even very efficient in the billions of parameters. He does not think they can scale an order of magnitude beyond GPT-2 small with the current method. People have already done some interpretability on Alpaca, a 7-billion-parameter model, and he knows of follow-ups that might reach that size for automatic circuit discovery.

Even at GPT-2 small they are not using neurons as nodes. The paper's main experiments sit at attention heads and MLPs, including the query, key, and value paths into heads. Neuron-level ACDC would have too many connections to be practical, and existing research has not really understood GPT-2 at that grain anyway. Last year IOI reverse-engineered a circuit in GPT-2 small; this year the greater-than circuit arrived at the same level. That is the pace. The edge iteration, not the forward pass, is the slow part. The library is built on Neel Nanda's TransformerLens, so it can load the GPT-2 line, EleutherAI's Pythia models, and toy models with GELU or SOLU activations as a computational graph. Out of the box it is usable on a lot of open models; Conmy thinks it will still be too slow on the larger ones.

When a practitioner used ACDC to study GPT-2 small's gender-pronoun bias — she after women's names, he after men's — the algorithm did not return a mess. The model aggregated name information onto the surprising next token after the name, a different residual-stream position, and funneled that downstream into the pronoun. The important locus was neither input nor output. It was an internal position that would have taken a long time to find by hand. That is the limited evidence so far that raw subgraphs can be readable. Labenz then asks about emergence. Conmy's take is that it is often a question of which metric you chose. Three-digit addition can look like a qualitative jump: a 100-million-parameter model produces rubbish, a billion-parameter model suddenly gets it right. Models are trained on the log of the probability of completions. Follow-up work, including a paper that calls the phenomenon a mirage, finds that log-probability grows smoothly in the log of parameters. Exponential growth is fast, so 1% correct becomes 50% and looks like it came from nowhere. We are not good at finding the right metrics, he says, so we sample outputs — even the Alignment Research Center's evaluations of somewhat dangerous GPT-4 capabilities still mostly look at what the model said.

Labenz is not sure the mirage paper lets society off the hook. During a GPT-4-scale run you do not know which surprises to plot in advance, and running a million diagnostics every batch is a massive tax. Conmy agrees that unknown unknowns — will next-token prediction plus RLHF ever match the best humans at math? — are extremely hard. He is more concerned with known unknowns. Instrumental-convergence arguments say that for a huge range of final objectives it is useful to gain power and resources. Current models do not do much power-seeking. Restrict attention to theoretically dangerous capabilities, and you can build better evaluations of how close or far today's systems are, without waiting to discover every unknown unknown.

The safety dream, as Labenz sketches it, is to figure out how models would implement the most concerning behaviors and then detect those subgraphs forming during training. Conmy agrees, and notes that you do not strictly need a mechanistic understanding to steer systems away from danger — but interpretability is uniquely specific. It would explain a capability in terms of exact location and exact reasons, not a litmus test that goes positive or negative. On dual use, neither of them finds the "this will just accelerate capabilities" argument very compelling. Conmy is careful not to misrepresent the other side, then offers two replies. First, the bulk of relevant capability gains have not come from transparency. An Alignment Forum post on pragmatic AI safety surveys vision and language progress and finds that most of it is engineering: try alternatives, keep what works slightly better, without understanding why. Next-token prediction as a loss, and RLHF as a 0-versus-1 preference, did not arise from a deep theory of language or human preference. Under that view, mechanistic interpretability is unlikely to supply the next wave of raw performance.

Second, if the techniques work — they have not yet found a stellar application on the models that matter — they offer a different way of building systems. You might get more powerful AI, which is the worry, but you would understand the computation from inputs to outputs. A lot of alignment risk, Conmy says, comes from specifying an objective and not understanding the process by which the system achieves it: outer alignment if the end state is hard to specify, inner alignment if the system learns an unintended solution even when you chose the right goal. Interpretability is a bet on seeing that middle process. Labenz imagines a mixture of sparse experts, crystalline subgraphs in the spirit of Liu's work, with a router whose choices a human could read. Conmy treats that as a nicely concrete picture of the ambitious goal: a forward pass whose high-level routing means something to people.

He immediately adds the graceful degradation. Even if that architecture-level understanding fails badly, narrow-task interpretability might still succeed. Power-seeking in certain scenarios could be a circuit. Understand it, and you can see why training arrived at that solution — or remove the circuit entirely before deployment. That is not his ideal future: a misaligned system with the bad part cut out, shipped anyway. It is a fallback if the whole-architecture goal fails. Looking forward, he expects the next phase of the field to go beyond the circuit framework of heads and MLPs toward higher-level motifs. In his own upcoming work, certain GPT-2 small heads that suppress negativity generalize across the training distribution, a pattern he calls copy suppression, distributed across several heads rather than sitting in one. OpenAI's line of using GPT-4 to explain GPT-2 neurons is a complementary way to assign semantic meaning. Architecture may change underfoot: a Microsoft Research paper on retention had just claimed a possible successor to the transformer. ACDC was deliberately aimed at pinning down a model's computational graph in full generality, not at a GPT-2 clone, so the method might survive if the board game changes.