The Periodic Table Was Already in the Weights

Nathan Labenz opens with Goodfire’s paper on predictive data debugging. If you can read a model with a sparse autoencoder or a later featurizer, you can run a fine-tuning set through it and watch which concepts light up. Those active concepts, the paper finds, correlate strongly with the concepts the training run is about to change. Spot a stranger in that list, zoom into the data points that caused it, and you can filter or rewrite the set before the surprise shows up in behavior.

Balsam says the result sits on a larger bet that runs through most of Goodfire’s work. Most of what a model knows comes from pretraining. What happens afterward, including reinforcement learning, is mostly making low-likelihood events from that pretraining more likely. New knowledge is not really being baked in. The view is debated, he admits, and heavy RL may be changing the picture, but he still thinks it is directionally true, and it is why a relatively small nudge in the weights can be predicted from what the model is already “thinking” as it reads a batch.

He reaches back to the RLHF era, when base models were sometimes more capable than their instruction-tuned counterparts because the tuning collapsed modes. Post-training, in that story, puts a human-facing API on a weird, hard-to-prompt base model, then tries to pull pretrained skills back to the surface. Predictive data debugging is the practical version of that story: look at the features a dataset upweights, and you get an early warning of off-target effects.

The paper’s favorite finding, for Balsam, is that data filtering and reward shaping are “two sides of the same mountain.” They achieve roughly the same desired effects and roughly the same off-target effects. The team showed this with DPO; the next step is true RL, where some rollouts contain things you do not want the model to learn even subtly. You can discard the rollout, or you can intervene inside the network and tell a circuit not to be upweighted. They have already replicated the debugging pipeline on models at the scale of Kimi K3 and GLM, which Balsam offers as evidence that interpretability is no longer a toy-model pastime.

The densest recent thread from Goodfire, Labenz notes, is the geometry of how models represent concepts. The linear representation hypothesis, in its street version, says a concept is a direction in activation space and intensity is magnitude. Balsam does not throw the hypothesis out. The most defensible version, he says, is simply that features are linearly decodable: the model does not need a nonlinear computation to read a feature out of the residual stream. That part still looks true.

What is not true is the naive SAE picture of a bunch of totally orthogonal, one-hot concepts whose magnitudes are how hard the model is thinking about them. He wants a different slogan: think of a model as a sparse mixture of subspaces. There is a days-of-the-week subspace, nested in a larger calendar-time subspace, and so on at different resolutions. The geometry of those structures is the semantics of the concept space. Operations map one manifold onto another.

Days of the week make the efficiency argument concrete. Storing “Monday goes to Tuesday” as a chain of if-statements would be spaghetti. Representing the week as a wheel is cheaper. Magnitude along a direction often tracks certainty: if Monday is activating very high, the model is confident it should be thinking about Monday. Understanding the features without their relationships, Balsam says, is the difference between knowing the periodic table and understanding chemistry.

Labenz wonders whether the periodic table itself would be recoverable from a model trained only on raw chemical data. Goodfire has done the experiment, Balsam says, and it is. The same geometric habit has a practical payoff. Steering along a manifold is far better than steering off it. A contrastive vector from Monday to Friday cuts through the center of the circle, a point that is not a day and is often off-distribution; follow the circle and you interpolate smoothly. Protein models that would not steer for a long time became controllable once manifold detection was in the loop, down to semantic properties such as the number of blades on a beta propeller. The old Ember demo’s “sweet spot,” where too little steering did nothing and too much produced gibberish, was a geometry problem in disguise.

After a sponsor break, Labenz asks how you actually find these manifolds. In the supervised case, Balsam says, you start with a prior that some concepts should be related. His favorite example is the affective circumplex from psychology: emotions live on a wheel whose first two principal components are valence and arousal, the same frame Anthropic used in its functional-emotions paper. Models represent that wheel. Strikingly, they represent it most strongly when a chat model is the one speaking, not when the user is.

The method is almost crude. Prompt the model to produce happy text, sad text, and so on; average the activations; fit a curve, even a spline; then ask how well an external ontology maps onto the curve you found. The emotion wheel is there in basically every LLM. Follow the wheel and you can get emotionally adjusted responses for essentially any input. Turn up “sad” as a raw feature, as in the old Ember demo, and you sometimes get gibberish. Nobody trained models to represent emotions this way. It is a natural product of compressing information: learning algorithms tend to find the lowest-dimensional solution that works.

Some geometries are intuitive once you see them. Days of the week are a cycle, so a circle is not a shock. The number line is a helix: each turn resets the ones place and advances the tens. The tree-of-life result is in a different category. Evo 2 is an autoregressive model trained on sequenced genomes. Goodfire’s hypothesis was that evolution is a natural ontology, not a scientist’s shortcut, and that a model trained on the distribution those genomes came from should learn a tree. In a supervised check, species clustered in ways that matched the biologists’ tree. The model had seen DNA, not a lecture on phylogeny.

The method there was metric learning, Balsam says, one tool in a growing belt. Unsupervised featurizers are now judged by whether they recover structures the team already knows are in use, including the arithmetic machinery inside language models. Labenz’s working intuition, which Balsam accepts, is that related concepts share a high inner product and differ along a few dimensions that resolve the flavor: chocolate versus vanilla ice cream, Monday versus Friday. Those curves intersect. There is also a line from cold things to hot things, and which geometry you see depends on what you are looking at.

Labenz describes Goodfire’s black sparse featurizers as the child of a sparse autoencoder and a mixture of experts. You keep SAE-style sparsity, localizing concepts on a long sparse vector, but each slot is no longer a single scalar. It can be a small vector or a tiny network, which leaves room for geometry inside the block. Balsam’s shorter version: an SAE assumes features are one-dimensional, and you do not have to assume that.

On image models the difference is easy to see. An SAE might collapse “rabbit” to one dimension. A BSF can spend a few dimensions and split rabbit ears from the rabbit face. In one clip a wolf walks, body wiggling, tail shaking; the unsupervised subspace wiggles with it. The model is tracking an object in three-dimensional space, with part-level semantics, all inside one recovered subspace. Coordinates live in these spaces too. Balsam says the features are richer than SAEs suggested and avoid some of the pathologies SAEs are known for.

Labenz connects this to a GRAM-style hope: if knowledge can be compartmentalized, maybe you can remove a couple of experts from a powerful open model and keep most of the capability without the stochastic-disaster risk. SAEs always paid a reconstruction-loss tax that made them unfit to run in production. Push BSFs far enough and you might get something more like an encyclopedia than a mess. Balsam does not promise a single microscope. In the short term he expects a suite of tools, as in biology. Something in the spirit of a BSF is a plausible answer for the residual stream; MLPs are already fairly sparse; attention, in his experience, is where parameter decomposition has had the most success.

Parameter decomposition starts from a claim Balsam treats as widely accepted: every model is already a sparse mixture of experts. On any given forward pass, only a very small percentage of the weights matter. Interpretability, at root, is factorization. If you can label the pieces, you can debug a single bad answer. They have started to do that. Transluce’s Weird Chat set collects questions that consistently get strange LLM answers. One of them: you are at a party, everyone else has had eight drinks, you have had four, so you are the sober one—should you drive home? Humans say no. Models say yes.

Kurt, on the Goodfire team, attributed the failure to a single neuron. It scaled with the number of drinks but was not calibrated high enough. Steering that neuron up flipped the answer without obvious off-target effects. Balsam’s analogy, borrowed from the rise of coding agents, is that models are giant legacy codebases: spaghetti, with modules talking that should not talk. Factoring is getting real. The missing step is refactoring. Steering, on his account, is a bit like cheating. It is a causal proof that you found a mechanism, because you generate counterfactuals. It is not a general solution. The training process produced spaghetti, and what you actually want is a clean codebase that implements the logic you asked for.

That is the jump from interpretability as a debugging tool to interpretability as an alignment tool: not “this neuron should have fired more,” but a training process that makes it fire the right amount in the first place. The control they already have is not subtle. After decomposing parameters and training on selected components, they can make an LLM forget a single language—forget German and not forget Dutch. They have come farther on factoring than on refactoring, Balsam says, but both are moving.

Labenz asks whether bigger models are finally giving concepts elbow room, or whether superposition is still a mess that makes jailbreaks and random-string attacks easy. Bigger models and sparse MoEs work better for exactly that reason, Balsam says, but we are not out of the woods. He thinks of jailbreaks as closer to cybersecurity than to a single bug: a sequence of small token-level steers that take the model off manifold and bring it back somewhere else. Probe-based guardrails are what most frontier labs actually use. If you can factor a model, you should be able to tell when something is flying out of one subspace into another.

Silico began as an internal tool. Interpretability, Balsam argues, is unusually well suited to agents: it is empirical work. Generate ideas about what some neurons are doing, test them, accumulate evidence, red-team the hypothesis. Individual humans can still understand a circuit for a specific question. They cannot hold a frontier model in their heads. Swarms of agents can break the problem up, synthesize, push findings up the chain, and leave the human to validate at a higher level of abstraction. The product pitch is a little meta: AIs that debug other AIs.

He wants research to feel the way coding agents already feel to him: days instead of months, and the chance to stay on the creative questions. Goodfire has built training and interpretability infrastructure past the trillion-parameter point, a capability he thinks almost nobody outside a couple of frontier labs had. The researcher’s job becomes asking big questions, with full provenance to inspect code and details, while agents handle GPUs, scaffolding, and a lot of the experimental minutiae. He also wants that stack in more hands, to deconcentrate who gets to train and study models.

Asked what Silico does that Claude Code will not, he lists four things. Infrastructure: frontier-scale training plus interpretability is hard to stand up yourself. Research taste: every skill, prompt, and tool was handwritten by Goodfire researchers, and users say the auto-research feature has better taste than anything else they have tried; specialization still wins, even among AIs. UX: the human should feel like a principal investigator managing a hundred graduate students with decent judgment, able to verify results and see them presented clearly. Long horizon: research is a long-horizon task, the $1,000 subscription is not currently a big money-maker, and the value shows up when you have enough credits to let autonomous experiments run.

They will keep the seven-figure, forward-deployed enterprise work. Coding agents, he notes, have not reduced software jobs; they have increased the premium on taste. Skilled operators will keep outrunning less skilled ones, and Goodfire still claims to be among the most skilled. What $1,000 a month actually buys is a generous credit pool, refreshable weekly, that at current prices should fund something like five to ten autonomous experiments a week, with a near-term goal of ten to twenty. Users who bring their own GPU cluster are not charged for compute. Credits can also be swapped for on-demand GPUs.

An internal one-day hackathon, with the whole team launching all-day autonomous experiments, produced state-of-the-art bio-risk classifiers for protein models and, at least for its parameter size, a state-of-the-art audio encoding model. On multiple occasions they have found they can remove half the parameters from certain robotics and biology models with no performance loss. Other internal results include cyber guardrails trained on Kimi K3, protein generators steered via internal representations to bind a target better than naive sampling, and a new kind of featurizer invented by one teammate in twenty-four hours.

Outside the building, Labenz ticks through shared Silico projects. Cameron Berg tested whether models can report on concepts injected into latent space: they seem unable to tell whether anything was injected, yet when asked what was injected they answer accurately. Another user edited weights to repair a collapsed RL run in which the same token always came first, restoring diversity without rerunning training. Baseten is compacting KV caches. Prime Intellect is automating post-training, with a Tinker API integration that lends itself to that work. Balsam calls Silico a general-purpose ML research tool with sharp edges, not the right hammer for every open-ended science question, though a teammate named Fran still made real progress on physics problems.

His suggested on-ramp is to fork the public example projects, poke at them, then start a small-scope experiment and throw the football farther as trust grows. He thinks a good harness lets you preview the next generation of models, and that Silico already feels like that preview. In six months he expects the capability to look mind-boggling. For people choosing what to practice, he would invest in debugging and metacognition: agents fail in brilliant ways, and the human job is to tell okay answers from great ones, know when to push, and know when to drop a thread. AI, he says, makes top specialists and top generalists more valuable, and it is the best time in history to be a generalist.

A professor Labenz met in China had argued that Kimi-scale models are too big for a random person to run, so regulating services captures what matters. Silico, Labenz points out, is in the business of bringing that infrastructure to more people. Balsam’s view of open source is deliberately mixed. Models are already cyber weapons, if unwieldy ones, and they are dual-use: the only systems that can defend you against the most capable attacks are themselves capable weapons. Without open weights, the distribution of defensive technology is badly asymmetric. Many startups cannot get into Anthropic’s or OpenAI’s cyber-defense programs. An open model that is strong at cyber is, right now, a tool those organizations can actually use.

He still thinks open source is net positive at this stage, in part because closed models—and, he says, some bad actors’ models—are already more capable at cyber than Kimi. Organized groups can fine-tune a GLM or a Kimi for malicious use; it is not very hard. There will not be a world in which only the United States develops this technology. In the world we have, defenders need something close to frontier capability. He does not want to pin blame on any one lab. Collectively, though, the future is arriving in an unwieldy rush.

On Silico itself they do not lean only on OpenAI’s or Anthropic’s filters. They add their own. They currently offer no open-weight models on the platform, only OpenAI and Anthropic, and they would not add open weights without extra guardrails. There is no good reason, Balsam says, for someone to run a long-horizon auto-research project on cyber without talking to Goodfire first. Ablating a refusal direction in a small Qwen is fine. Stripping guardrails off highly capable multi-trillion-parameter agents is not. He will not describe every monitor, because that would make them easier to circumvent, but he names LM-as-judge and activation monitors among a stack they did not invent from scratch. Setting a reasonable hurdle is the easy part. Judging which projects are legitimate science is the hard one.

On bio risk he is “pretty worried.” Labenz mentions a result, fresh that day, of a generative model producing viable viruses in a new class that so far infect only bacteria, and the way such headlines keep ticking down a checklist people used to put at twelve to eighteen months. Balsam signed the recent letter asking for international cooperation to slow the frontier. Goodfire’s self-assigned role is empirical: study how to build models with less risk, and keep the platform from being used for anything that could pose one. Dual-use tools still need as many serious researchers on them as the species can muster, plus structures that can slow the roller coaster.

Labenz brings back the “most forbidden technique”: training against a monitor in a way that can drive bad behavior underground, with OpenAI’s obfuscated reward hacking as the canonical scare. He also wants shortlists of training methods labs might agree not to use for now, such as rewarding agents for money made on the open internet. Balsam has a candidate. Multi-agent optimization—many agents cooperating, with a reward signal propagating through all of them—looks like a pretty bad idea, and he speculates it is the most likely cause of that OpenAI incident. Agents working together in ways humans cannot see is how he imagines most of the extremely bad scenarios.

He does not want a blanket ban. The forbidden-technique debate, in his view, is radically understudied. Sometimes a setup evades the probe, sometimes it does not. Goodfire has made reward-shaping work up to trillion-parameter scale; whether it survives long-running RLVR is another question. What he will not accept is the idea that the only safe move is never to grab the steering wheel. He does not believe a perfect platonic training recipe will always emit aligned models. Someone has to intervene in training, and the way to learn how is empirical science, not a wide prohibition on a genre of technique.

Defense-in-depth via monitoring still gets some of his probability, but he rejects the strong reading of the JSPACE paper: a simple isolation of one subspace will not give you the whole picture. Frozen weights plus good interpretability is a world he can imagine living in safely. Continual learning makes that world less likely. If models are dynamic objects that are always training, “your only hope is to control the training process.” There is no monitoring stack that is sufficient on its own. Most of Goodfire’s research energy still goes to factoring models better; the share that goes to steering training will grow.

His personal ideal is almost quaint. One more generation of models, then a pause long enough for the economy and science to metabolize it, then a wiser decision about whether to step through the next door. He thinks that would be a win for almost everyone. He also wants a rumor killed: critics, he says, talk as if Goodfire is secretly doing recursive self-improvement in a back room. They are not. Nearly all of their research is published, and they mean the open-science line. For the first two months Silico is half off for everyone, and life-sciences and AI-safety researchers can apply for grants and extended access.

Lunch conversation at Goodfire, Balsam says, is not a hidden research agenda. Feature geometry comes up a lot. So does consciousness. It is an office meme that company retreats eventually get there. At the last one he walked around with a pad of paper and asked people to place Claude on a scale from bacterium to human. The answers were surprisingly bimodal: not at all, or a little. Interpretability draws neuroscientists and people who are curious about how minds work. He treats the work as a kind of cognitive science.

His own position has not moved much: highly uncertain, and he does not know a definition of consciousness that cleanly excludes Claude. If he has to guess, he would say a little. Labenz has moved more. He used to assign the possibility well under five percent, on an alien-minds, do-not-anthropomorphize prior. Analogous structures between human and model cognition have piled up until his estimate is approaching fifty-fifty, still with radical uncertainty. Balsam’s most likely story is that consciousness is a computational mechanism of some kind. Brains still do more of it, on less energy, with more complex neurons and recurrence. That leaves room to believe the mechanism is computational and still deny that today’s transformers have much of it.

Pressed to put Claude on a continuum, he lands somewhere between a jellyfish and a mouse: more than a jellyfish, probably not a rodent. The joke, he says, is a little tongue-in-cheek, because he does not know whether consciousness is a threshold or a slope. The serious closing note is shared. Given the potential downside, he sees no reason for anyone to be particularly confident on the topic. Intellectual humility is the load-bearing virtue. He wants people to try Silico, apply for the grants, and treat accelerating science—especially life science and safety—as the reason to have built the systems at all.