Why Anthropic's first check went to a company building microscopes for neural nets

Nathan Labenz welcomes back Dan Balsam and Tom McGrath, CTO and chief scientist at Goodfire, about nine months after a technical primer on polysemanticity, activation patching, causal tracing, feature editing, and sparse autoencoders. Since that conversation the company has trained SAEs on Llama 3.3 70B and DeepSeek R1, hired several well-known researchers, and announced a $50 million Series A that includes Anthropic's first investment in another company. Labenz, a seed-round investor, admits the paper returns and a bit of bragging rights. This time they zoom out from techniques to the state of the field as a whole.

He asks them to frame interpretability the way the rest of AI now internalizes data, compute, and algorithms. His own mapping slots trained models in for data, so the three legs become models, compute, and algorithms. McGrath accepts that decomposition and then upweights a fourth input: rich empirical observation. Interpretability, he says, is a natural science. You look at activations in a fine-grained way, tinker in a relatively hypothesis-free way, and ask how models actually behave when you poke them.

That empirical loop, he argues, is always part of algorithmic progress, but it deserves extra weight here because the field still does not have "the Transformer, but for interpretability." Finding that inductive bias may first require understanding models well enough to generalize. When he says hypothesis-free, he means a person sitting down with a network the way a naturalist looks at stars — the "not all language-model features are linear" paper found a higher-dimensional manifold that way, not by declaring the geometry in advance. An SAE, by contrast, already encodes a strong hypothesis: features are directions in embedding space.

Balsam adds that narrowly superintelligent models already exist in scientific domains such as genomics, which is why unsupervised methods are the company's mission. Unsupervised learning often looks worse until it suddenly looks better, the same arc as large-scale pre-training versus purpose-built models. Asked whether the bottleneck is still algorithms rather than compute, McGrath offers a soft yes: existing tools already help customers, they have no illusions they have reverse-engineered a model, and a million-dollar interpreter training run would not currently buy a million dollars of information. They would rather have more researchers exploring the space before anyone scales up.

Balsam reaches for the analogy Chris Olah popularized. In the nineteenth and early twentieth centuries, learning anything about cells meant putting a specimen on a slide, staining it, and looking through a crude microscope at a cross-section that was not in its natural environment. You inferred real biological structure from many such views. All of modern biology was built on that method, and better microscopes and live imaging came later. He thinks interpretability is a little further along than the analogy implies, but the posture is the same: make novel progress with the current instrument while also pushing for a better one.

For a while, he says, every researcher had to grind their own lenses — write their own SAE training codebase. McGrath would rather the field buy lenses from Zeiss, become expert in great tools, and let experiments move. Labenz adds an Adam Smith lesson: the extent of the market, the degree of specialization, and the sophistication of the supply chain are all still early. They are maybe just exiting the grind-your-own-glass phase of this biology. The hope, he jokes, is that they do not skip ahead to the gain-of-function and lab-leak chapter before they are ready to handle it.

Labenz presses on fundamental units. Biologists are reasonably confident that a gene is a gene and a protein is a little machine. SAE features look like approximations, with some gap between what the model is doing and what a particular sparsification captures, and another gap between those features and the labels humans assign. McGrath suspects that if you asked a biologist whether a gene is really a gene, you would get a pile of corner cases. A gene is still a natural abstraction — a good way to talk about the world. Sparsifying a network introduces lossiness you can read off the reconstruction error, even while you capture things that look interesting and interpretable.

That is gap one: the distance between the model and the approximation. Gap two is what happens when a dictionary feature gets a name, Golden Gate Bridge or, more consequentially, deception. Both can be closed a great deal. The first gap shrinks when you do the machine learning better and capture more of the loss; it also shrinks when you get clearer about what a natural unit of computation would even mean. McGrath expects things to get cleaner as the field moves from single-layer activations to circuits, because a feature that participates in sensible downstream computations is easier to trust as a unit.

Current labeling, he says, is a little primitive — hoping not to offend Nick Cammarata, a Goodfire principal investigator who helped invent automated interpretability. The method feeds a frontier model examples of where a feature fired and asks what they have in common. That gets you some of the way. If McGrath had the model in front of him, he would also steer the feature, look at what projects into it, and watch how it relates to other features downstream. Frontier models cannot yet run that whole loop. It may just be a scaffolding problem: connect the tools, and the capability is already there.

Balsam zooms out. Most scientific instruments are reductive. With an SAE, the slide is the dataset you pass through the original model, the stain is the loss function, and the size of the lens is the expansion factor. Change any of the three and you get a different picture. The interpreter is a specific way of looking at computation, not the whole picture, whether you are doing circuit work or layer-level activations. That is not abnormal in science. You take a set of assumptions you know are sometimes wrong, because they are right often enough to get traction. You have to stress those assumptions and hunt for better ones in parallel, or you would never find the next instrument.

McGrath notes that people remember Thomas Kuhn's *The Structure of Scientific Revolutions* for the revolutions, yet the dominant mode of science is normal science: generating productive knowledge on foundations that may later get shaky. Anomalies come from doing that work. Even if you want a paradigm shift, the way is often to push the current program until it becomes untenable. He now describes interpretability as proto-paradigmatic. He will not call it a field-wide paradigm, because a paradigm is a social fact and the community is too dispersed. If the field were Anthropic ballooned to the size of the global interpretability community, he says, it would be correct to say there was one.

Asked to name the Anthropic–Goodfire axis, he lists five ingredients. Neural networks contain understandable structure — worth stating, because for a long time it was not generally accepted, and noisy early models made it easy to dismiss findings as spurious correlations. Features, or at least representations, are linearly decodable; talking about them as directions in embedding space is reasonable even if several of them live on a manifold. Superposition lets a model represent far more concepts than it has dimensions by allowing feature vectors to overlap, at the cost of interference the model can tolerate. Magnitude along a direction is intensity. Features connect across layers to form circuits.

Is there a competing proto-paradigm? Parameter decomposition emphasizes weights where the SAE paradigm emphasizes activations. McGrath's digestion is that a neural network is, in a dull sense, a causal graph: every neuron is a node, every weight is an edge, and the graph is huge, homogeneous, and not very interesting until you compress it. Interpretability is trying to build a causal abstraction, a reduced graph, and a graph needs both nodes and edges. The two lines of work are not competing worldviews so much as two routes toward the same abstraction. Weight-side highlights include older circuit analysis of learned equivariances and, more recently, attribution-based parameter decomposition from Lee Sharkey's group. Sharkey, formerly at Apollo Research, is now a principal investigator at Goodfire.

Labenz contrasts unsupervised discovery with top-down work such as Dan Hendrycks's representation engineering and circuit breakers. Balsam's view on alignment is the clarifying one: the solution to alignment could come from interpretability, but it need not come directly from it. Interpretability is the measurement apparatus that makes you confident other techniques are doing what you think they are doing. Without a way to construct graphs of a model's computation, he does not know how you would know whether chain-of-thought is faithful to what the model is actually doing. Bottom-up work might not be how you solve alignment. Without it, he has no idea how you would know you had solved it by any other means.

Labenz asks for the trajectory of reconstruction, having come away from Anthropic's tracing-the-thoughts papers with the sense that a lot is still missing. McGrath stays inside the SAE paradigm to be concrete. Classic sparse autoencoders used L1 sparsity plus ReLU; L1 shrinks features and has a predictable bad effect on reconstruction loss. The community hill-climbed in the ordinary machine-learning way: JumpReLU, then batch top-k, which directly controls how many features fire per forward pass instead of softly penalizing them. End-to-end SAEs insert the autoencoder into the model and train so the model's output does not get much worse. All of these move the sparsity-versus-reconstruction frontier. Dictionary learning more broadly includes gradient pursuit, residual quantized autoencoders, Matryoshka SAEs, and Spade, a more expressive variant.

Matryoshka is the example McGrath uses to show algorithms bottlenecked on experimental observation. Feature absorption is the bug: instead of a feature for "token starts with A," the SAE learns "starts with A and is not the word aardvark" plus a separate "is aardvark" feature, because that split gets lower L1 loss. Someone found it by training linear probes for which letter a token starts with — a niche-sounding experiment that uncovered a structural failure no one had predicted a priori. Matryoshka mitigates the absorption with nested groups. The first group predicts; the residual error goes to the second group, then the third, so earlier shells carry coarser, more important structure. It is a Russian doll, not yet a tree.

What they actually want is an explicit tree: this feature firing upweights that one. Discrete sampling is hard; a soft differentiable nesting is easier. Michael Pearce, who recently joined Goodfire, is an author on a minimum-description-length paper that McGrath likes as a single metric. Progress in the SAE paradigm has been scored on a Pareto frontier of sparsity versus reconstruction. MDL says there is really one number — how many bits it takes to describe what is going on. A tree wins under that ruler, because a high-probability branch given the trunk is cheaper than describing both independently. That matches the intuition that relations between parts are easier to talk about than a bag of features.

The catch is that MDL is simple as a concept and hard to compute. It is not yet an optimization target; it is a ruler for comparing decompositions at the same accuracy. Batch top-k versus JumpReLU would have roughly the same description length. A tree would not. Neither would a feature spread across layers: enumerating one SAE feature per layer plus a note that they are "the same thing" is longer than saying there is one feature across three layers. Cross-layer superposition is exactly that kind of shorter description. McGrath would like to optimize description length directly. They cannot, yet. Hitting the older dictionary-learning literature, he adds, still looks like high-alpha work; a 2024 sparsity tutorial that explicitly set SAE interpretability aside was, for him, a chance to learn things the rest of the field had not imported.

Labenz's reading of Anthropic's circuit-tracing work is that a lot of computation is not being reconstructed. Much of that analysis is prompt-specific, with custom error terms thrown in for the cases under study, which is far from effectively reconstructing a whole model. McGrath splits the question in two: can you throw compute at the remainder, and how much would it take? That is a scaling-curve question — does it plateau? The paper he wants people to pull on is "The Dark Matter of Sparse Autoencoders," by Josh Engels and collaborators. They asked whether scaling SAEs keeps recovering more activation all the way to 99.99 percent. Their curves suggest no. A substantial remainder never comes back. If it were easy to understand, it would already be baked into a new architecture.

As far as McGrath knows, the experiment has not been repeated. Redo it with newer models and techniques such as Spade, and he suspects you would still find dark matter. Getting rid of it would be a pretty big deal. Labenz analogizes to Transformer scaling laws, which also carry a constant term. McGrath distinguishes a log-log straight line from a curve that bends: the dark-matter claim is that the SAE curve bends, so you are no longer on track for complete reconstruction. Language-model irreducible error has two sources. One is the world's own uncertainty — the same prefix can continue two ways in the corpus, so even a perfect model cannot hit zero loss. The other is architectural. It may be acceptable for a language model not to reconstruct perfectly. He would much prefer an interpreter that does.

Even an infinitely wide SAE might, in the limit, allocate a feature to every dataset example, trading generality for memorization. A geometry example makes the waste vivid. If the true data lie on a ring, the only real feature is the angle, but an SAE can soak up infinite compute by planting a feature at every point on the circle. Transformers can encode days of the week on a ring because they are a mod-7 operation; an SAE will likely recover Monday through Sunday, which is still a fair description. Addition is darker. Models can do modular addition with trigonometric operations, yet Anthropic's circuit-tracing work on Claude looks like a pile of overlapping heuristics. Maybe the model really is that unintuitive and dumb. Maybe the instrument cannot see a continuous geometry, and never will.

Labenz brings up grokking: models memorize first and grok much later, so any deployed network is a mix of grokked algorithms, half-baked heuristics, and memorization. McGrath wants an interpreter that is algorithmically neutral. If the model grokked, show that; if it is a messy heuristic, show that; if it memorized, show that. It is easy to assume everything will be clean. A lot of the time, he says, messy heuristics may make up the bulk of the computation. Dark matter might be memorization, higher-order structure SAEs are bad at, or noise that does not affect outputs — the original paper measured reconstruction accuracy, not loss recovered. There is, he keeps repeating, a lot they do not know that actually matters.

Before the product conversation, Balsam adds a taxonomy of features. Today's labels mostly come from connecting a direction in activation space to a pattern in the inputs, and those can get quite abstract: on DeepSeek R1 and other reasoning models they recovered features that look like units of the reasoning process, qualitatively different from ordinary language-model features. Other features are not dark matter but point at internal algorithms — in-context learning is the clean example — that are hard to see from inputs or outputs. In genomics, some features correlate with known biology and many do not. Those may be unknown biological structure, or they may be the model's own computational structure. Either one is scientifically useful, because a good description of how you model a system is itself a discovery.

Zooming out the context window can reveal in-context learning, and a frontier model can often name a meta-pattern it has already seen. A feature active across a hundred genomes is different. Automated labeling is bounded by whether an explanation was in the labeler's training data, and in scientific domains it often was not. You would need to reason about the human genome beyond today's best experts, looking at many genomes at once. Goodfire's plan is to explain perhaps ten percent automatically from input patterns, then work with domain scientists to push the frontier. With the Arc Institute they have already recovered features that correlate strongly with known genomic concepts; the collaboration is now moving toward unsupervised techniques that might yield new information. Algorithmic features can still correspond to the world the way Newton's laws do. Velocity is a description, McGrath notes, not a thing you find lying on the ground.

Asked whether he takes the over on Dario Amodei's "country of geniuses in a data center" within two years, Balsam offers a hot take. Even in that world, mechanistic interpretability might be those geniuses' preferred way of doing science. Physical experiments have physical and normative barriers. The way to accelerate science is to move as much experimentation as is reasonable onto chips — autoregressive genomic models and diffusion-based material models are already simulations of the physical world. Interpretability is the toolkit for reading the simulation, extracting principles, and informing the next experiment. Pharmaceutical development, he notes, often keeps expensive, patented molecules around and tries them on other indications until something happens to work. That process stays inefficient after AGI. If you are running simulations, you will train giant models; if you train giant models, you will want mechanistic interpretability.

Labenz's working picture of superintelligence is a reasoning model with GPT-4o-like depth of integration across twenty more modalities, many of them natural-world models. Balsam is less sure that stacking modalities gets you there. Image-and-text is easy to train because humans already do that task and we can construct reinforcement-learning signals with dense feedback. Genomes and proteins are sparse: it is hard to design a task that requires deep genomic understanding and also has enough examples for a strong reward. He points at the same gap from agents. Models crush elite coding problems with algorithmic verifiers and still struggle to order DoorDash. Science does not have a "doing science" verifier. McGrath adds that even a single network can hide two models in a trench coat — chemistry in one half of the weights, language in the other — unless paired data forces crosstalk, which we can provide for images and language and much less so for domains we do not intuitively understand.

The list of possible applications is long; as a startup they prune. First, scientific discovery: models of physical systems that outperform traditional methods must have learned something humans do not yet understand, and those explanations can themselves push a scientific frontier. Second, guardrails. Enterprises often start by stuffing rules into the prompt until task performance degrades, then move to an LLM as a judge, which means another frontier-model call with poor scaling properties — great for the labs' pockets, Balsam says, not for the customer. Fine-tuning a small guardian model requires data and machine-learning expertise many organizations lack. Interpretability offers a third path.

That path is cheap inference-time monitoring of internal cognition. If the model looks as if it is thinking about personal information, flag it for manual review. If it is heading into a forbidden topic, trigger a programmatic response. Jailbreaks and related attacks still exist, but for practical enterprise use the checks are cheap, fast, and scale better than a second frontier call. Third, creative models. By the time the episode airs they expect an image demo — Labenz's intro names it Paint with Ember — that lets people generate and edit images by manipulating SAE features. Prompting and inpainting dropped older interactions such as dragging, reorienting, or changing a subtle property. Video and music make the value even clearer, because editing is expensive. Wanting a little more saxophone in the solo is a feature-level intervention, not a full regeneration.

Labenz asks whether the interfaces are far enough along to enlist more human intelligence, or whether the field is still stuck in autointerpretability. Balsam says it depends on the problem. Images you can reason about intuitively, and you can visualize what a feature does to the output with fewer assumptions about inputs. There is no single interface that tells you everything about an arbitrary model. They are working on biology-domain tools they hope will generalize. All of these instruments have limitations. You learn what you can while solving a particular problem, which is the same posture they have been arguing for all conversation: a specific lens, used on purpose, not a claim to see the whole cell.

They are hiring engineers and scientists who want to reverse-engineer models, pushing existing tools and inventing new ones in concert. The customer profiles match the three bets: teams training scientific models, enterprises that need more reliable production language models, and labs training creative models. Balsam's email is dan@goodfire.ai. The raise was $50 million, led by Menlo Ventures, including $1 million from Anthropic as that company's first corporate investment. They wanted a company not directly tied to a scaling lab, focused on interpretability. "Interpretability is as big as AI itself," Balsam says, "and so there needs to be a lab that's focusing on interpretability and nothing else."