Don't Fight Backprop: Goodfire's Bet on Designing What Models Learn

Nathan Labenz welcomes Dan Balsam and Tom McGrath back to The Cognitive Revolution with a headline that would have sounded like science fiction on their first visit in August 2024. Goodfire, the mechanistic-interpretability startup, has assembled a research team, landed a first wave of blue-chip customers — including a couple who found the company through that earlier episode — published a long run of results, and just announced a $150 million Series B at a $1.25 billion valuation. Balsam is the CEO; McGrath is CTO and chief scientist. Labenz notes they have been prolific enough that, after a stretch when he was less able to follow the literature, coming back to the blog felt like drinking from a firehose.

Balsam treats the round less as a trophy than as fuel. The company has only been around for a year and a half, and he says he is most excited to put the capital to work scaling what they already do: interpretability research, plus a lot of new papers they are about to walk through. Labenz's custom is to start zoomed out. He offers an outsider's three-layer map of the field and asks them to correct it, then they will get to the new agenda that shipped alongside the fundraise.

That agenda is intentional design. Instead of pouring data and compute into a vast machine, taking whatever comes out, and reverse-engineering what happened, Goodfire wants to understand and shape the loss landscape *during* training — to control what models learn and, eventually, how they generalize. Interpretability, in McGrath's later framing, is for three jobs: scientific discovery, monitoring and auditing, and this new third pillar. The first two are relatively well understood; intentional design is the idea they are now mixing in.

Labenz's taxonomy has three rungs. First, understand which concepts a model represents internally, the work sparse autoencoders made famous. Second, understand circuits: how information is processed step by step, the territory of Anthropic's circuit tracing and transcoders, including wiring diagrams for something as simple as adding two two-digit numbers. Third, learning dynamics — how the model *became* what it is — the kind of work he associates with Timaeus, which still goes over his head. McGrath says the taxonomy is good, then adds a meta-question the field has been asking since their last appearance: what is interpretability even for?

The missing piece, he argues, is generality across inputs. Today's circuit diagrams are assembled from execution traces: here is how the model adds 17 and 54; here is how it adds 13 and 99. A real circuit account should quantify over *all* settings of those variables. Sparse autoencoder features do not give you that object. If the model represents a quantity as a helix winding through embedding space, an SAE feature is a ray from the origin that zaps one point on the helix — a detector for "the number is about five," lighting up a bit for three, more for four, peaking at five, fading at six. Useful, McGrath says, but not the simpler structure you actually want. You want the helix.

Labenz maps the same idea onto days of the week. Seven SAE latents can sit at random indices — Monday at position one, Tuesday at 5,027 — while in embedding space they often lie on a plane and rotate, which is how people actually think about a week. McGrath agrees: the model did not have to learn that geometry, but it did, and the linear representation hypothesis has seen a lot of talking past one another. A paper that had just come out that morning, he notes, ties the structure to co-occurrence statistics and symmetry in language. The stake is not only descriptive. If you want to change a computation as a unit — make the model behave differently in a way that is invariant to weekday — adjusting only the Monday feature and waiting for Tuesday to show up in the data will not do. Intentional design hinges on seeing that higher-order structure.

After a sponsor break, Labenz restates the vision in plain language: it would be better not to throw unbelievable amounts of data and compute into machinery, get something out, and reverse-engineer what just happened. Methods should scale with compute and leave room for natural-language feedback. McGrath's engineering version is closed-loop control. To control a system you need an observation system and a control system. Interpretability is the observation system for training. Today you throw data in, the chain rule grinds layer by layer, and only when the run finishes do you notice the model became too emoji-filled, tweak the data, and hope.

Balsam's metaphor is a map of the loss landscape. Data implies a terrain of valleys; some lead to behavior you want, some do not. Interpretability's job is to draw the map so that at a fork you can say go this way, not that. McGrath wants that judgment live, while the plane is still flying, not after it has landed. He is also explicit that "you" in this loop is not a human staring at every gradient. It is a language model. Interpretability is a handle for intelligence to hook onto *inside* backprop, and the methods have to scale with both compute and model intelligence.

The intentional-design blog post gives a concrete decomposition. An SAE says which concepts are active; you then take the inner product between the residual-stream gradient and those concept directions, so a single mixed update splits into named pieces. Labenz walks through their pirate-speak-plus-arithmetic example: the data pushes the model toward both talking like a pirate and doing math; the SAE lights up both families of features; cosine similarity shows the step is up-weighting pirate and up-weighting arithmetic. A language model with a spec can then keep the math-aligned slice and drop the rest. The obvious next move, McGrath says, is the one every machine-learning person reaches for: project out the parts of the gradient you do not like.

Projecting the pirate component out works very poorly. The data still implies the network should become more piratical than it is. Pirate-supporting computation is scattered through the model; cancel the gradient halfway, and a later component learns the same thing. That, McGrath says, is fighting backprop. You are not getting gradient descent to want something else. You are trying to stop it, and "the gradient descent will always win."

Inoculation prompting is his example of a method that does not fight. If an environment is exploitable, the instinct is to tell the model not to reward-hack. When it reward-hacks anyway — even by accident — it updates toward being a reward-hacking kind of guy, because the forbidden thing turned out to be good. The nicer insight is to tell it that reward-hacking is allowed. When it then hacks, the episode is expected; there is nothing extra to learn. McGrath calls that explaining away, and he thinks the principle is deeper than the trick looks. Balsam adds that without the prior, the model reads "don't" as a broader license to ignore instructions. The prompt is a surface of intervention people under-appreciate; the reward function is not the only lever.

Inoculation prompting is still open-loop: one fixed prior applied to whatever data arrives, including data that has nothing to do with reward-hacking. Closed-loop control needs observation as well as a knob, which is why decomposing the gradient matters. Labenz offers an analogy McGrath does not treat as the literal method but finds clarifying: people already freeze layers or train a new head because they value existing representations. Imagine doing that to *circuits* — this one is good, leave it; this one should update on this data. Framed that way, intentional design is not mystical. It is just more surgical than what training usually permits.

Goodfire's first public proof of concept is hallucination. You can build a labeled synthetic set of known hallucinations — the world has even open-sourced some — and train a probe on internal states. At runtime, when the probe fires, you can inject a token in the spirit of "wait, I might be making this up," a cousin of the reasoning-token tricks that occasionally force a model to take another stab. The model often then corrects from its own memory, which cuts off an error that would have cascaded through the rest of the context and, in-context, reinforces a persona that checks its work. The more interesting move is to treat those hallucination activations as an RL signal and punish the model for entering the state at all.

Ground truth is expensive. McGrath says running the labeling process as a live reward — Gemini 2.5 with web search, several seconds per call — would cost on the order of hundreds of thousands of dollars. They pay a one-off fee, amortize it into a probe, and then have a cheap detector for "the model thinks this was probably a hallucination." Why would a model represent its own nonsense in the first place? Pre-training is full of people making things up; the representation is useful for fiction, two truths and a lie, and other make-believe. Post-training has not fully specified that you are no longer the making-things-up guy. The reward is meant to sharpen the persona toward being the saying-correct-things guy.

The obvious fear is that the model will learn to fool the monitor. Labenz cites latent-space hide-and-seek work showing models evade detectors, and OpenAI's obfuscated reward-hacking paper: punish chain-of-thought plans to hack, watch hacking fall, then watch it go underground until you can no longer detect it. A friend of the show, Zvi Mowshowitz, called that "the most forbidden technique." Goodfire's trick is simple. They run the probe on a frozen copy of the model. Hooking it to the student and back-propagating through it is "an extremely bad idea": the detector sits here, the representations slide whoop off to the side. Because the reward model is a different, frozen network, the student cannot backprop through it. Evading then means guessing, in token space, from a low-dimensional scalar, which behaviors will not trip a probe it cannot see. In this setup, dropping the behavior was easier.

Far AI published follow-up a couple of days later: the probe has to actually be good, or pressure to change behavior is weak, and back-propagating through the probe immediately blows up. Should we trust this at larger scale? "Yeah, you should be worried about everything. This is alignment research. Paranoia is a way of life." They trained on the order of billions of tokens, never saw the probe signal get subverted, and stopped when pushing harder made the model incoherent rather than sneaky. They can spot-check with the original Gemini-plus-search ground truth; a student that fooled both the probe *and* that process would need an internal model of what a smarter web-searching model would find, which McGrath does not find plausible. He refuses a general claim. Obfuscation happens in other setups, including ones Far AI found. The dynamics are worth studying; this is an empirical result in this design, not a theorem.

Labenz asks the stakes-up question: Anthropic calls, they want your best techniques, you are doing this for real — what then? McGrath's first principle is first, do no harm. Anthropic's interpretability plan is to use it as a test set. If stakes are high, he does not want to disrupt that. At the current level of scientific development, they should not use these methods on frontier-model training. He thinks they can get there fast, but confidence would require serious auditing games showing that interpretability-based audits still work at least as well as before — unless, of course, there are no problems left to find, at which point "the alarm should be going off."

Balsam draws a sharper product boundary. At their current understanding he would not recommend using the techniques to make a model less deceptive. They are studying the class, and applying it to concrete, measurable, relatively low-stakes problems such as hallucinations. The hope over time is that intentional design becomes a critical alignment tool; that is why they wanted to develop it. They would not suggest anyone align a frontier model with it today. There may even be classes of target — deception is a candidate — that you never want these methods on. They have not pre-decided that, but he can believe it.

That caution bleeds into publishing. McGrath says that in an area with huge upside and possible danger, you should not immediately publish everything. You need a line of retreat. If you have shipped every step except the last one where you go "oh no, it was really bad," you have not left yourself one. Commercial interest is real; so is a safety pause. Being more cagey than is natural for him as a scientist is the point. Goodfire is a public benefit corporation. Deals start in the seven-figure range across life sciences, enterprise, financial services, and government. The stack they want is interpretability at the center: inference-time guardrails plus model adaptation, eventually a specification in and a model that behaves that way out. If they destroyed monitorability, Balsam notes, they would destroy one of their own value propositions. If they ever believed they had the key to alignment — or that the techniques were dangerous — they would not keep it as a profit secret.

Labenz's worry about late-stage surgery is qualitative damage. Tamper-resistant fine-tuning made models a lot worse. Emergent-misalignment models trained on bad code or bad medical advice also got dumb: they answer in code when they should not, fail a coherence filter, and are not things anyone would deploy as a general assistant. The cheap hack here would be a model that never hallucinates because it always says "I don't know." Goodfire checked. Benchmark capabilities showed essentially no degradation — up a percent on one, down a percent on another, almost certainly noise. You cannot score well on MMLU by answering "I don't know," and in long-fact completions the number of claims did not go down. A public viewer shows a few truncated, egregious policy errors; McGrath says they had to search hard to rotten-cherry-pick them.

That success surprised him. He can tell a story after the fact: the probe is well calibrated, so it supplies continuous, relatively dense rewards on short spans, unlike a GRPO-style trajectory that is merely good or bad. Easier learning breaks less. The fighting-backprop version of the same project, he says, is back-propagating through the probe — "directly stepping on the rake." Labenz wonders about a middle mistake, analogous to rewarding exploits while telling the model not to exploit. Balsam suggests competing incentives, sycophancy-adjacent pressure to always give the user an answer. They did not run that experiment; it is the sort of tension a broader post-training run would surface.

Could you detect theory of mind and beat it out of the model so RLHF stops optimizing for what pleases unreliable raters? McGrath doubts you want the same recipe. Theory of mind is broadly useful; a model without it cannot intuit what you want. It is also almost certainly an ingredient in sycophancy. You do not nuke the capability. You intervene at the right part of the circuit: do not *use* it for sycophancy. On the pirate-style method, Labenz restates the move as blocking "please the user" updates. McGrath corrects him one more time: not block. Reshape the landscape so the gradient no longer points at those representations, guided by a constitution or model spec that says the system is maximally truth-seeking.

Compute is still overhead, not savings. Anthropic has been willing to spend on the order of 5% of inference compute on constitutional classifiers; Goodfire's grand hope is sample efficiency — learning in one sample what used to take a hundred, which multiplies the FLOPs budget, especially when frontier data is not infinite. How early in pre-training you can start is an empirical question they have not answered. Balsam guesses you need not wait until the end, because representations form in phase transitions built on earlier ones, but they have focused on post-training first. McGrath does not want to stack a second extremely hard problem — the evolution of proto-concepts through pre-training — on top of an already non-consensus bet. Most people, he says, think this whole direction is hard and possibly doomed. One step at a time.

A separate paper starts from a known observation: memorized recitations are brittle. Perturb a small set of weights and the model loses a specific historical passage, which is intuitive if narrow facts live in a small part of the network. In other words, the loss landscape around memorized content is jagged. Look at a large batch, though, and some perturbations destroy performance broadly — those should be core reasoning weights — while others barely move the batch and were probably tied to an individual fact. Sort on that axis, truncate the memorization tail, and on some reasoning tasks performance actually improved. McGrath's only correction is that they are not ranking individual weights. They are ranking eigenvectors of the Hessian, collections of weights. The curvature–memorization link is not original to Goodfire; using a mega-batch so that single-example spikes wash out is the piece he thinks is theirs.

Labenz hears an echo of a Carl Shulman thought experiment: if living forever meant paring yourself down to core survival mechanisms, the thing that remained might not be what you valued. Flipped, a model's cognitive core might be something you could start from — a purer logical reasoner you add facts back onto, more controllable, maybe smaller. McGrath is interested in shrinking so the model does not pay the parameter cost of facts it no longer knows, but he has not pushed it all the way and suspects database-style or pre-pretraining approaches (synthetic context-free grammars, or putting facts in a preamble that is not in the loss) are more likely to yield a minimal reasoner. He also does not know whether a giant thicket of logical entailments would even be more interpretable. He has never seen one. The immediate product idea is a regularizer: in fine-tuning, keep the generalizing bits so you do not screw the model up the way hard fine-tunes do in emergent misalignment.

When the model is broken, Balsam says, interpretability is a debugger. When the model is already good, it is a knowledge-extraction tool. Prima Mente, which works on neurodegenerative disease, trained Pleiades, an autoregressive epigenetic foundation model on cell-free DNA fragments — bits of DNA from cells dying across the body, collected with a blood draw, already used in cancer diagnostics. They used the embeddings to predict Alzheimer's and asked Goodfire what the model was depending on. The answer was surprising. The Alzheimer's literature had focused on methylation statistics and cell type of origin. The model was overwhelmingly depending on fragment length, a signal with a history in cancer and not in Alzheimer's. From that insight they built a simple logistic-regression proxy that recapitulated much of the original performance and generalized better than literature baselines to an independent cohort.

Balsam is careful about the claim. It is still a pilot; they need more cohorts; they are considering wet-lab follow-up. Even so, he thinks it may be one of the first cases of learning something new by studying a model, turning it into a testable hypothesis, and checking it against the biological world. That is the scientific-discovery job of interpretability, sitting beside the debugging job, and he says they have more life-sciences work with other partners still to publish.

A lightning round starts with talent. The company is up to about forty people, and those people are in demand. McGrath's pitch is scientific ambition, a first-principles empirical culture that tries not to hold any idea too tightly, and a snowball: good people know good people, and others want to work with them. Recruiting is a lot of work and extremely worth it. Balsam adds a differentiated research vision and the open possibility space of a startup, which is hard to protect at a big lab. On alternative architectures — nested learning, for example — Balsam expects interpretability to keep working because semantic information still has to pass through bottlenecks. McGrath's sharper complaint is that people do not go looking. Transformer MLP neurons are often directly interpretable, work he associates with Translucent; the sparse autoencoder, he half-jokes, was inside all along. Mixture-of-experts specialists are usually polysemantic, but recent routing-path work is interpretable too. Panglossian as it sounds, maybe models get better partly to the extent they become more interpretable.

Asked about AI consciousness after Cameron Berg's work on the Goodfire API, Balsam will not confidently rule it out in today's frontier systems. He thinks they probably are not conscious, and he says the "probably" is doing a lot of lifting. Nothing in his view prevents machines from having experience in some meaningful sense; he does not find definitions that preclude it convincing. Interpretability might be the only tool with a shot at answering the question. He has no valence on whether machine experience would be good or bad — only that it is a live possibility now or later.

The close is a call to action. Superintelligence, Balsam says, feels like a given and is happening quickly. He sees two doors: one toward superintelligence we do not understand at all, and one toward superintelligence we might have a shot at understanding. Fundamental work on interpretability and on designing models is how you walk through the second door. Along the way the work is not only theoretical — life-sciences discoveries, partners who want fewer hallucinations and more reliable models. McGrath's version is blunter. If you want to be part of the most exciting scientific quest going, it is interpretability; if you want to make it useful, Goodfire is the place. Labenz signs off with congratulations on unicorn status and on the research run.