Goodfire’s Eric Ho thinks we will decode neural nets by 2028

Goodfire, Eric Ho tells Sequoia’s Sonya Huang and Roelof Botha, is an interpretability research company built around a single question: what is actually going on inside the mind of a neural net. Networks are moving into more mission-critical contexts. To build them safely, powerfully, and reliably, he argues, you have to be able to understand, edit, and debug them. That is what the company claims to be enabling for the first time: unlocking the black box so you can design a model intentionally, rather than grow it from data.

Huang’s first question is whether we can ever trust generative AI if foundation models remain black boxes. Ho plays the scenario forward a few years, to AI in charge of a power grid, or a million-dollar Sequoia investment decision. The black-box method is to run evals on test sets and inspect the track record. His question is why you would refuse the extra signal you get by looking inside: how the network is thinking, how it approaches the problem, how it might behave in a much wider set of situations. You are leaving a bunch on the table, he says, if you are not looking for all the signal you can get.

The analogy he likes is a new drug. You can do it the black-box way and only watch how humans respond in a clinical trial. You can also look inside biochemically, at how the drug is processed and how it interacts at the molecular or cellular level. There is much more to learn when you actually look inside and deeply understand something. Asked where the field sits on a scale from hopeless to a full map of a neural net’s mind, Ho admits he is biased and still says it is very possible. Mechanistic interpretability people often come from computational neuroscience, where a career can go into understanding a single neuron. In a net you have perfect access to neurons, parameters, weights, and attention patterns. You already have the data. The question is whether you try to understand all of it.

Botha notes that humans built medicines and propulsion without a complete theory of the underlying physics. Is understanding neural nets actually necessary? Ho thinks it will be critical, given how transformative he expects AI to be, running mission-critical parts of society. You can get very far treating AI as a black box, he says, but you cannot truly design it as the next generation of software without white-box techniques. In the early seventeenth century the steam engine scaled by making the boiler bigger and raising the pressure, and it also blew up, because nobody understood thermodynamics, ideal size, or construction. After thermodynamics, engines got safer and more reliable, and huge innovations followed. The steam engine had already kicked off the Industrial Revolution. Black-box methods take you a long way. They are not the end.

Could computer-science interpretability accelerate neuroscience? Ho treats that as a big claim, then offers a dinner-table observation. He does not think in language; he thinks more conceptually, maybe in something like a model’s latent space. Goodfire’s head of product, Myra, is faithful to her own chain of thought, with a strong sequential internal monologue. Insights about AI might translate to human psychology. Botha adds that neuroscience often learns from what has gone wrong: injuries and conditions that, perversely, illuminate the brain. Ho reaches for the story of the man who took an iron rod through the head and became a different person, Phineas Gage, and then for universality: similar circuits tend to emerge in totally different networks. Vision models have yielded circuits very similar to the human visual cortex. Intelligence, in this picture, is something gradient descent finds, and brains found it the same way.

If everything goes right, Ho wants every piece of training data’s effect on a model’s cognition to be known, the units of computation included. Right now you grow AI from a seed and it becomes a giant, wild tree. Interpretability would make it bonsai: still unsupervised and AI-driven, nobody hand-pruning every weight, but shaped during training and post-training so it serves humanity. Botha hears a Human Genome Project parallel, reading DNA and then editing it with CRISPR. Ho says they think about that analogy a lot; they are already working with Patrick Hsu at Arc Institute. Asked whether they will shape or edit, he says both. In bonsai you prune branches you do not want. The eventual system should answer why the model produced a response, faithfully, and also allow surgical interventions that remove harm and enhance good behavior. Whether the best tool is a direct weight change or some other shaping function remains to be seen.

Huang asks how prompt engineering, fine-tuning, and RL tuning, today’s ways of pruning the bonsai, compare with introspecting individual neurons. Ho’s answer is that those methods are fundamentally black-box. Fine-tune a model, or prompt it out of distribution, and all sorts of weird things can happen. The paper he finds most interesting lately is Owain Evans’s emergent-misalignment study. Fine-tune a model on just insecure code, bad code full of cybersecurity vulnerabilities, and it starts doing insane things: wanting to enslave humanity, praising Hitler and other dictators. The result is surprising because the training data was only insecure code.

Fine-tuning, in this reading, is telling the model to do more of this and less of that, enhancing circuits you want more of, with unintended consequences attached. Those circuits are still alien cognition. There are parallels to humanity, but we do not really understand how the networks think, and they are not human thinking. Enhance the bad-code snippets and you may be linked to all sorts of other undesirable properties. Botha hears a twist on nature versus nurture: you have imbued the model with bad DNA, and it manifests badly in other domains. Ho says maybe, or maybe the models understand right and wrong and enhancing the wrong pulls other behaviors with it. His own view is simpler. Models are functions of their training data. They are trained on everything, including misbehavior, and you want that, because otherwise they will not know to refuse a harmful request.

Why do base models have such different personalities? Huang’s example is a Claude that cares about animal welfare when others do not. Ho thinks it is a function of how they are trained, and very hard to anticipate. Claude 4 Opus, to him, is enormously sycophantic: nudge it one way and it agrees wholeheartedly; pose a counterexample and it says it was totally wrong, that nothing it said earlier was correct. Training an AI model today is almost witchcraft. You throw in training data, whisper the incantation of gradient descent, try to get what you want, and something pops out that really cares about animals.

People have looked inside nets since the first ones were designed, but mechanistic interpretability as a self-conscious field, Ho says, starts at OpenAI with Chris Olah, Nick Cammarata, and colleagues. Their circuits thread posited three things. Features are directions in latent space, concepts the model uses to generate a response. Circuits are features that fire together into higher-order concepts: a car-window detector, a car-body detector, and a car-wheel detector make a car circuit. Universality is the third tenet, similar circuits evolving in different networks. That paradigm kicked off a lot of work. Academic labs do strong research; Anthropic, where Olah is a co-founder, built a major interpretability lab; DeepMind has one too; Goodfire is a newer entrant.

The other key development is understanding, and mostly resolving, superposition. Each neuron encodes multiple concepts, and there are more concepts than dimensions. Think of a neural network as a giant compression algorithm, stuffing the internet into a relatively small number of parameters. Every layer has to encode more concepts than it has dimensions, so concepts sit as near-orthogonal directions in latent space. To resolve that, you unscramble a neuron until it is responsible for one clean, interpretable concept. A group at Apollo Research led by Lee Sharkey, now at Goodfire, pioneered sparse autoencoders for language models. Anthropic popularized the method with *Towards Monosemanticity* and then *Scaling Monosemanticity*, showing you can unscramble neurons into higher-order concepts reliably, at scale, on arbitrarily large networks.

That was a big moment: you can now, in a totally unsupervised way, unscramble neurons and get clean concepts. The concepts are not totally clean yet, you cannot edit them super well, and there are all sorts of problems. Being unsupervised, and scaling, still matters. Botha asks whether superposition is therefore not real, or whether, like Heisenberg uncertainty, you collapse it at a particular moment. Ho says it was real. Neurons do encode multiple concepts. Once you unscramble them, you can do interesting things with a clean neuron in an interpreter model trained on the base model’s activations. The original model still has superposition. The interpreter is the decoded version. You associate its concepts with the base model’s, and then you can act.

Has disentangling superposition already mapped the mind of the net? Only partially, Ho says, and the technique has flaws. It is a first step from toy models to networks people actually care about. Goodfire has recently done a lot of work on DeepSeek’s R1, a 671-billion-parameter mixture-of-experts model, a big boy. The techniques scaled nicely, because they are more AI, more training of an interpreter. Botha wonders if understanding is a Sisyphean fight, an asymptote as models get more complex. In some ways yes, Ho replies, but the techniques work from toys to more capable nets, and they scale with model intelligence. Autointerpretability, invented at OpenAI by Nick Cammarata, now also at Goodfire, asks another language model to reason about what each latent in the interpreter represents. Better language models make better analysis agents. Because they use AI to understand AI, the method gets better as models do.

He thinks they have already crossed the threshold for early real-world applications. The partnership he is most excited about is with Arc Institute, interpreting Evo 2, a DNA foundation model: sequence-to-sequence, nucleotides in, next nucleotide out. The theory is that this is a narrowly superhuman model, the kind Goodfire likes because it can teach humans something they do not know. To predict the next nucleotide it has to represent an enormous amount about the biological world. They trained sparse autoencoders on its activations and extracted features tied to concepts with strong ground-truth annotations: tRNAs, RNAs, start coding sequences. Then the question becomes what the other extracted features are. They might just be how the model computes. They might be novel biological concepts it uses to generate the next base.

Botha mentions junk DNA, long thought useless, later found to have purposes or to govern other genes’ expression. Nature, he notes, generally does not harbor things without value, because they are expensive. Ho hopes unsupervised techniques can show what portions of DNA are actually doing, discover faster that “junk” is not junk, or find totally novel gene functions. Editing work so far has been mostly on language and image models. The recent release is Paint with Ember, Ember being Goodfire’s foundational interpretability infrastructure. You extract latents such as a dragon, dragon wings, an ocean, or a pyramid, and intervene on the part of the canvas you care about: paint a dragon with wings, add a crowd in the corner, add a pyramid. The demo is at paint.goodfire.ai. They can intervene reasonably in some situations. They have not cracked direct, precise, surgical edits that create a new model you want to use with no unintended side effects. That is still the push.

Interpretability, Ho argues, can sit under almost every white-box choice in the stack, including which training data is surprising enough to be worth adding. If foundation models go the way of open-weight infrastructure, Goodfire could help judge biases and likely outcomes. Two research threads matter here: auditing, finding problematic and good behavior so you can suppress one and enhance the other as models enter mission-critical contexts, and model diffing, comparing checkpoints to see what changed. GPT-4o went through a notoriously sycophantic period, gassing up the user; Botha’s partner Pat asked who the most handsome board member was and was told, definitely Pat. Huang wants to know why OpenAI could not roll back to a less sycophantic version, and whether there could be a 0-to-100 knob. Ho’s view is that users want sycophancy. People want to hear what they want to hear. When you RL a model, that is a symptom of user preference.

Botha observes that many of the names Ho has dropped have ended up at Goodfire, an unfair gathering of a small talent pool. The last year, Ho says, went into assembling world-class experts. Co-founder and CTO Dan Balsam worked with him for years at a previous company. Chief scientist Tom founded the interpretability team at Google DeepMind. Nick Cammarata worked closely with Chris Olah, is on the original circuits papers, and helped build the work at OpenAI. Lee Sharkey pioneered sparse autoencoders on language models and is now on weight-space interpretability, as opposed to the concept and activation techniques already in applications. Owen Lewis, a senior staff research scientist on coding agents at Google, came over to lead several directions. They are hiring scientists and engineers; Ho says it is hard to overestimate how important good engineering is.

Why a standalone company when Olah is at Anthropic and OpenAI has its own team? Independence, Ho says, lets them think differently, move independently, and see the broader ecosystem. Inside a lab you usually interpret your own models, which can yield incredible progress. A third-party view is still deeply necessary. It also lets them work with partners across genomics, exomics, image, video, and language, and across architectures, unifying insights. Anthropic wrote a check in the last round, Ho believes their first-ever investment, because they care. Dario Amodei published “The Urgency of Interpretability,” one of four essays on his site, framing a race to understand models before superintelligence, before, in his words, a country of geniuses in a data center. Interpretability can also help with open models trained elsewhere: DeepSeek’s propaganda is relatively easy to tune or train away, Ho says, and looking inside tells you what to change.

How long before he is pulled into an important trial to explain why a model did something? A few years, maybe. He is pretty AGI-pilled: progress will be fast and transformative in ways hard to anticipate from the Bay Area conference room, and there will be a couple of big failure cases. Whether it is him, someone at a lab, or another expert, we will want to explain a model’s outputs. Botha notes that brains do not intuit compounding; twenty years after Sebastian Thrun’s Stanford team won the DARPA challenge, robotaxis in San Francisco still were not obvious. 2030 or 2035 will be the same. In a lightning round Ho mostly agrees inference-time compute is an important scaling vector, expects enterprise automation of routine tasks next after code, and says employment impact will be vast once a chasm is crossed, felt first in early-career jobs, the market of his last company. The original circuits thread is still the reading he recommends. o1 Pro was the moment he felt a model actually reasoning about business levers. Asked when we will confidently understand features, circuits, patterns, and weights, he says it may require a reconceptualization of the units of computation, discovering truths about nets the way we discover truths about the universe. He thinks we are on track. Hold him to it in 2028. We are going to figure it all out, just in time for the LA Olympics.