4,096 Numbers Have to Carry Every Concept. Goodfire Wants the Knobs

Start with one number: 4,096. In Llama 3 8B, that is `d_model` — the width of the residual stream that carries everything a token knows from one layer to the next. Ask the model what sport Michael Jordan played, and every Jordan fact it has ever absorbed has to flow through those 4,096 numbers. The world contains orders of magnitude more concepts than that, so the network's trick is to let one coordinate moonlight for several ideas at once. That packing is why large models are so hard to read, and it is the premise of Goodfire's entire business.

Nathan Labenz opens *The Cognitive Revolution* with Dan Balsam, Goodfire's CTO and a serial founding engineer, and Tom McGrath, chief scientist, who spent 2019 to 2023 on DeepMind's safety team doing interpretability. A third co-founder, Eric Ho, is CEO and not on the call. Mechanistic interpretability, Labenz says, is the young science of explaining why a model did what it did, layer by layer, as opposed to saliency maps or surrogate models that only describe input-output behavior. Sparse autoencoders have recently become the tool for isolating the features those models learn, and anyone who played with Anthropic's Golden Gate Claude — a feature clamped absurdly high — already has an intuition for what a discovered feature can do to behavior.

Goodfire has just raised a $7 million seed led by Lightspeed Ventures, aimed at productizing this line of work on open-source models so developers can look inside, diagnose, and intervene. Labenz discloses at the top that he put a very small check into the round, and that anyone else seeking his money should expect an extremely tiny check size. The world he and the founders want is one in which companies simply do not deploy models they do not understand.

Balsam's path is the one Labenz flags for anyone intimidated by the subfield. He studied neuroscience, then spent most of his career as an early engineer, most recently at RippleMatch, a Series B AI recruiting startup. ChatGPT's release shook him awake. He led the effort to put generative AI into the product and hit the black-box wall: the systems were powerful, but when they misbehaved you could not say why, and in recruiting settings that would not tolerate bias you could not put a large language model anywhere you needed guarantees. He concluded that nothing else was worth working on besides reducing AI risk, and he quit — a hard call, because he had built that engineering team from scratch.

The coincidence is almost too neat. Eric Ho, RippleMatch's founder, reached the same conclusion within a week and also put in notice. As the two of them learned more about interpretability they got excited by how fast the science was moving, and by the chance to scale it as a business. They then found that McGrath had written a document with many of the same ideas. The interpretability community introduced them. After they started talking, they decided to start a company together.

McGrath's bet is older. In 2016, during an applied maths and statistics PhD, he had the feeling that "this AI thing could be pretty big," which he now ranks as one of the better bets of his career, and he pivoted into machine learning. He joined DeepMind's safety team in 2019, fell in love with interpretability research, and left in 2023 still excited but unsure what to do with it. South Park Commons in San Francisco gave him months to think. The live question in the field, he says, is what interpretability is *for*; a great deal had happened, it was probably useful for a lot, and the best test was to start a company. Complementary skills, a matching document, and the rest is history. The combination is itself a signal: there is no priesthood. An outsider willing to grind papers can catch the frontier in about a year.

McGrath's history of the last few years sounds like the rest of AI: years of unremarked progress, then a sudden jump. For a long stretch interpretability was unfashionable. The popular beliefs were that it did not work, that there was nothing in the models to find, and that even if there were you would not be able to find it. Evidence accumulated anyway, but there was no way to extract it at industrial scale. The breakthrough of the last year and a half is sparse autoencoders. The metaphor he likes is neuroscience: Ramón y Cajal looking down a microscope, drawing one neuron at a time, versus being able to sequence in bulk. The field has gone from artisanal understanding to something more like a factory, built on about five years of prior knowledge.

The first live question was whether models were "magical trash" — accurate outputs with nothing meaningful inside. McGrath does not think that was ever a coherent view, but it was a popular one. What piled up instead was evidence that training on general tasks produces semantically meaningful, somewhat disentangled representations nobody put there by hand. Word embeddings already had linear analogies: king minus man plus woman lands near queen. Andrej Karpathy wrote a blog post training RNNs and visualizing what individual units did. A 2017 OpenAI Amazon-review model, predicting one letter at a time, grew a single neuron that was a state-of-the-art sentiment classifier. Nobody asked it to learn sentiment. It learned it as a means to the next character.

The good years did not last. As models grew, single neurons stopped meaning one thing. A vision unit might fire on car wheels *and* cat ears — polysemanticity. Chris Olah pulled on that thread for years, and McGrath credits his persistence with getting the field to where it is. Polysemantic firing on two or three unrelated concepts suggested a distributed code that was not hopelessly distributed: if a neuron fired on everything with no pattern, you might give up; two, three, four things is manageable structure. Anthropic then did the thing that looked foolish. They worked through superposition, "plugged up the courage" to train a really wide MLP, called it a sparse autoencoder, and it just worked.

Balsam, catching up from outside, had assumed the internals must be too complex to understand. That sits badly with how learning systems should work. He treats them as compression engines, always balancing generality against complexity: regularization wants less complexity, the loss wants more generality, and what falls out are hierarchical structures that explain patterns in the data. If you believe that — and most of machine learning does — it would be *more* surprising if we could not get any insight into the structure. Modern foundation-model training makes memorization even less attractive: if deduplication is done well you see each point once, so circuits that reuse beat memories that do not. People already find surprisingly rich features in GPT-2 small. The features were there. What was missing was tools, and courage. Labenz's restatement is that the stochastic-parrot story was never quite right: we went from a readable sentiment neuron to a polysemantic mess, and had to catch up with what the models had been doing all along.

Labenz's Brazil trip is the thing that breaks toy intuitions. Claude 3.5 Sonnet not only named a city but the specific one-boat river-cruise operator, with a value proposition attached — long-tail enough that it feels like the model memorized Wikipedia while also trying not to. McGrath splits the puzzle in two: how a model does a thing now, and how it came to be a model that could. A recent injection experiment stuffed a made-up fact into training: the model learns it a bit immediately, forgets along a curve, and remembers well after a few repeats. Large models memorize a load of facts from one or two looks. How they do that without storing the text verbatim, he says, is a question nobody can answer today. Training dynamics are a frontier. Give it a year or two.

The working sketch of a forward pass is more fleshed out. Tokenization may split "Michael Jordan" into Michael and Jordan; the first couple of layers fuse a super-token that holds facts attributable only to the pair. Early-to-mid MLPs then dump essentially every Jordan fact that might be useful in *any* sentence into the residual stream, because attention can only look back — it cannot selectively recall a fact and push it forward. When the sentence later becomes "plays the sport of," the model fishes out a sport feature and a late calibration stage picks basketball. Logit lens, proposed by nostalgebraist and developed by researchers including Nora Belrose and Stella Biderman at EleutherAI, tests the picture: pretend layer 10 is the last layer, apply the unembedding, and watch predictions go from nonsense toward the right token. The model is not in some private subspace until a last-layer miracle. David Bau's group’s causal tracing makes the same anatomy causal: break the whole network, restore one piece, and the fact often comes back. Those essential MLPs sit about a third of the way through.

Labenz's cartoon of the Transformer is the one most listeners already have: embeddings, then alternating attention (relations between tokens) and MLPs (facts), climbing toward higher-order ideas two-thirds or three-quarters of the way through, then collapsing to a next token. McGrath says that is the low-resolution sketch everyone is working from, and it will fill in the way human anatomy did. The surprise, once Goodfire trains SAEs on the residual stream, is that mid-to-late layers do not look like a tidy stack of "society"-level abstractions. Specific nouns, grammatical features, and high-level concepts are interleaved. You have to understand the language model on its terms. Human hunches about how information should be hierarchically structured are probably wrong relative to how the network actually processes it. Labenz's advice to beginners is the same: treat it as alien intelligence. Behavior can look human; the mechanism is not.

Alien does not mean unrelated to us. Universality is the observation that networks trained on similar data learn similar representations. Vision models assemble hierarchical features that look a lot like visual cortex. A 2024 *Nature* paper by Jamali and colleagues recorded language comprehension at single-cell resolution in the human brain and found something resembling polysemanticity plus hierarchical assembly, which they compare to embedding models. Balsam's excitement is that we may be learning how *we* learn by studying models, and that our own intuitions about organization may be as wrong as they are about the network — if the brain were intuitive we would already understand it. Brains also carry an implicit regularizer: connections cost energy, so biology is stingy. McGrath adds a harder example from his chess work. AlphaZero never saw human games, but if you are going to play chess and you do not know what a pin is, you will have a very bad time. Structure in the world impresses its shape on any model that is good enough.

Early interpretability had an easy on-ramp: attention maps, which you can almost read off as which tokens relate to which. You see local look-backs that locate you in the sentence, and longer links that make conceptual sense, even if how they got there is still hard. MLPs invited a different kit — zeroing, randomizing, activation patching, moving a representation from one forward pass into another. Between layers everything condenses to `d_model`. That bottleneck is where Goodfire is looking, because it is the smallest place the whole computation has to pass through, and the easiest place to intervene.

Balsam's programmer metaphor is the one that sticks. The residual stream is the model's RAM: a set of variable slots every layer can write. Each layer executes some code — a linear transformation on that stream — and updates global variables that downstream "functions" will read. You do not learn much about any single function, but you learn a lot about what the model thinks is important, and when. If you had to debug a program written in an alien language, one of the first things you would try is flipping a bit. Intervening on the residual is that move. McGrath adds the multi-thread wrinkle: there is a residual stream at every token position. Each thread's job is to use all preceding threads to predict the next token. Attention is information sharing across threads; MLPs are within-thread processing.

The numbers make the packing problem obvious. For Llama 3 8B, `d_model` is 4,096; for 405B it is 16,384; 70B is presumably 8,192. Attention and the MLP each *add* a vector into that stream. The MLP is conventionally four times as wide as `d_model` before it squashes back down, which is why so much computation can hide inside a layer and still have to exit through a few thousand coordinates. Those coordinates are not infinite-precision either. If parameters are in 8-bit, activations typically are too, a finite set of discrete states. There is no way 4,096 slots hold millions of concepts unless most of them share. A concept important enough — maybe that 2017 sentiment neuron — might get its own dedicated direction. Everything else is superposition: reuse, almost-orthogonal directions, and sparse coding, because any given input only lights up a few features of the world at once.

Anthropic's *Toy Models of Superposition* is still, McGrath says, close to the frontier of actual understanding. Importance wants a dedicated neuron, maybe even two that support each other when there is noise; everything else gets smushed. Related concepts can share a compositional code instead of a separate highly distributed code for every combination — the geometry of categorical and hierarchical concepts, a best-paper prize at a mechanistic interpretability workshop. Labenz's hunch is that a lot of model weirdness is non-orthogonal concepts that usually do not co-occur, colliding when a strange prompt brings them together. McGrath will not bless it as fact. Adversarial prompts might be hitting a sweet spot of interference. Nobody knows. The question is not unanswerable; it is unanswered. That is the motivation for pulling the packed directions back apart.

McGrath wants the mystery taken out of sparse autoencoders. They were a 2010-era tool, big again for a new reason. Architecturally they are a very wide MLP with a sparsity-promoting regularizer. That is most of the story, funny activation functions aside. The extra width takes pressure off the "if it is important, give it a neuron, but we have no space" tradeoff: more things can own a direction, and the sparsity term drops a lot on the floor. They have worked much better than it was reasonable to expect, which is one reason people were slow to try them — the scheme sounds slightly miraculous. The other reason the field is excited is social. Interpretability used to be so qualitative there was no number to make go up. Now there are proxies: fidelity, how much of the language-model loss you recover, and L0, how many features are active on average. People plot Pareto curves, compete, cite, and the machine-learning machinery finally points at this problem.

Why not just train the base model sparse? Cost, first. An SAE on one middle-layer residual stream of Llama 3 8B might have 3 billion parameters, or 12 billion, already bigger than the model. You might want one per layer, per residual, per attention output, per MLP output — on the order of 96 objects, each bigger than the model. Nobody's bank account, or GPU cluster, survives that as a training recipe for the base network. Second, putting SAEs into the model makes it worse. You can recover the mid-90s of the loss, depending on details, and the industry spends a fortune to move loss by a decimal point. Those degradations can be the difference between a $1 million model and a $10 million one. Balsam returns to compression: large models are good because they compress an astronomical amount of data into a finite parameter budget. The SAE is an unzip. Compression was the point. This is lossy decompression, and the loss is expensive.

Labenz's restatement is the one to keep. Concepts are packed non-transparently into a few thousand residual numbers. Project them out to a much wider sparse layer, reconstruct, and if the model still works you have at least kept the information. Each wide coordinate lights up only now and then, so you can do the old sentiment-neuron move: look at what makes it fire, and try to name it. The reconstruction-versus-sparsity tradeoff is the training loss. The miraculous part, McGrath repeats, is that it works. Anthropic and OpenAI have published studies on their own closed models; as the conversation is happening, DeepMind has just released SAEs for Gemma 2B and 9B. Goodfire's bet is to make the same technique available on open models. It is still not easy. Dead features — units that go quiet after 10 million inputs and never return — wasted capacity until the community published a fix. Serving is harder still. Golden Gate Claude clamped one feature; Claude 3 Sonnet had on the order of 34 million. Baking in a single edit is one engineering problem. Serving a model with all the dials attached, so every user can turn a different subset, is an unsolved one.

Balsam shouts out NDIF, the National Deep Inference Fabric, which David Bau helped found, and its `nnsight` framework. Goodfire uses it to scale both activation harvesting and interventions, the kind of work that otherwise only happens inside labs with frontier models. The expensive step is running the language model. A naive online loop — forward pass, stream activations, train the SAE — spends almost all of its time on the LM. They write activations to disk, then run tens or hundreds of hyperparameter sweeps on the same pool, which amortizes the cost. Once you know the recipe and want SAEs at every layer and site, the tradeoff flips toward a painful distributed stream. They will handle it, McGrath says, when they get there.

The longer-term vision is a world where AI tools work in the service of humanity, which Balsam says is basically impossible if we do not understand them much more fundamentally than we do. The default assumption has been that AI is a black box. The world they would rather live in is one where you debug a model the way you debug any other software, and when it misbehaves you can tell a causal story so it does not happen again. A 2024 McKinsey survey found that 44% of business leaders had experienced at least one negative consequence from unintended model behavior. At RippleMatch, bias constraints made large language models unusable in some recruiting applications. Opening the box is how you get guarantees compatible with the society you actually want. The analogy is breeding versus genetic engineering: we used to fine-tune wheat by trial and error; now we can go in and change things directly. Interpretability is that change in power, from selection to engineering.

The first product-shaped object is a research preview: talk to a Llama-series model, browse features, and intervene to see how behavior moves. Autointerpretability is a core piece. Nick Cammarata, an adviser, co-authored "Language models can explain neurons in language models," which showed you can prompt a language model with neuron activations and get statistically robust labels. That has evolved from labeling neurons to labeling sparse SAE features, which is how you traverse a gigantic interior. Interface is the other unsolved half — programming or graphical — because the amount of information inside these models is enormous, and a human operator needs the salient slice, not all of it. Short-term, they want the rest of the world to see that models do not have to be black boxes. Contact with reality then sets the research agenda: some ideas make a cool paper and fail in use; some are already good enough.

They chose a company over a nonprofit because the work is compute-bound and because they want research grounded in things people will pay for. The seed is $7 million, Lightspeed-led. They are a public-benefit corporation, with a mission to reduce AI risk through interpretability, and they still think private capital is the vehicle that gets both jobs done. Balsam's market picture is not a single SaaS seat. He expects dozens of interpretability companies: interfaces for regulators and auditors, for applied AI teams, for frontier labs; use cases in prompt tuning, RAG, and training-data design. Goodfire wants to be foundational to that ecosystem. A decent amount of the round goes to compute. They are on 8B and 70B for the next few months, with Llama 405B as the natural next push. McGrath's line is the one to write down: if you had an interpretable Llama 405B, that would probably be the most useful model in the world.

Labenz asks for the skeptic's remaining ground. McGrath sees a God-of-the-gaps retreat: first there would be no interpretable concepts, then they were unimportant, then they were merely subtler. Some features will be hard; humanity understood general relativity and built rockets, so "forever uninterpretable" is a very high bar. Cost-effectiveness is a scaling question. A second skepticism is that for any given application, some other method will dominate. He finds that unlikely. RLHF is good in some ways and brittle in others; we can probably do better. Balsam's prior has flipped: start from "these models are understandable" and see what you find when you try hard. The human-level story of the last decade, McGrath says, is that courage paid off. An "uninterpretable feature" in practice is one that lights up on a grab-bag of inputs, is causally important when you clamp it, and still refuses to make sense after you have tried everything. He does not think that is the typical end state. Spend time, recover the circuit, and you usually get somewhere. Production systems will not rest on one technique anyway; they will stack methods that cover for each other, and that combinatorial architecture does not exist in interpretability yet.

New architectures will not, in McGrath's view, throw the table over. Universality again: if a model is good enough at the task, the representations are probably still there, just in different places, and you have to know the architecture well enough to know where to look. As long as there are embeddings, SAEs remain useful. Networks with no embeddings are some wild future in which the whole world is adapting. Labenz has been surprised that interpretability techniques already transfer onto state-space models better than he would have guessed. McGrath is not shocked. Capability, not family, is the tell.

The application that makes him sound most like a scientist is foundation models for science. AlphaFold, ESM3, quantum chemistry, weather — a growing list — already beat our best theories at prediction, which means they probably contain information our theories do not. Interpretability is one of the better paths to reading it out, and a real test of whether the methods generalize. Otherwise science stalls a little: we value theories for useful predictions, but we also just want to know about the universe. Balsam pushes the claim as far as it will go. Use cases for understanding a language model and a scientific model are not that distinct. If we live in a future where we can explain models, then anything you can train a model to do, you can explain. That, he says, could be a transformative scientific technology on the level of machine learning itself.

Here the two of them, and Labenz, actually disagree. Labenz thinks that once a weather or protein model beats every human theory, we should stop expecting language models to level off at human-expert performance, because they are clearly learning concepts we did not already have. McGrath is much less sure the argument transfers. A numerical weather model and a "humanish" language model may share little. If you naively stuff ten scientific tasks into a language-model training run, his intuition is you get "ten models in a trench coat" — one network, ten separate things underneath. DeepMind's Gato, a language model doing tokenized RL tasks, showed very little transfer. Vision-language models still often refuse to actually look at the image. And there is economics: if you run a weather model a lot, paying extra inference for French-verb conjugation is a bad deal even if a little transfer exists. In the infinite-data, infinite-size limit Labenz is probably right; on any shorter timescale, shared representation and cost decide. Video "world simulator" claims for Sora, Veo, and Runway Gen-3 get the same cold water. You cannot simulate video perfectly without a great world model, but flocks of birds that merge and split are not object permanence. Those are claims about the future, not descriptions of what is currently possible. Even a model that "intuitively" knows where the ball goes may never verbalize the equation, unless writing the equation helped it compress.

They are hiring founding engineers and research scientists who believe interpretability should be the default for deployed models, and who want to figure out new science in a scrappy startup. goodfire.ai is the door. For people not ready yet, Balsam's path is the one he just walked: learn how transformers work, read the papers — they will put favorites in the show notes, including Anthropic's superposition and monosemanticity work and the OpenAI neuron-explaining paper — then email the authors. Academics in this community, he says, answer with unusual enthusiasm if you have actually read the work. McGrath's closing is not a pitch so much as a temperature check. This could be the great scientific adventure of the next couple of decades, and it is an open community. Anyone can come in. Labenz, slightly stunned to be this close to whatever this era is, is here for it.