SAEs Find the Nouns. Parameter Decomposition Looks for the Verbs

Lee Sharkey joined Goodfire as principal investigator after co-founding Apollo Research, where his team worked on detecting and mitigating deceptive behavior in frontier models. Apollo decided to double down on evaluations, which left the mechanistic-interpretability group without a natural home. Sharkey, Dan Braun, and Lucius Bushnaq moved together. Months earlier Sharkey had introduced Tom McGrath and Eric Ho, who went on to co-found Goodfire; he describes his role in that founding story as a “B character.” A $50 million private raise, Labenz notes, also changes the compute story: Apollo was philanthropically funded, and this line of work is hungry for GPUs.

The episode is built around two papers. The first, still under the Apollo banner, is *Interpretability in Parameter Space*: attribution-based parameter decomposition, or APD. The second, from Goodfire, is stochastic parameter decomposition, or SPD. Together they try to do for parameters what sparse autoencoders did for activations: pull overlapping structure apart until the pieces are simple enough to read. Sharkey’s contrast is blunt. Activation decomposition peels apart intermediate states and labels them. Parameter decomposition peels apart the weights, the architecture, and the nonlinearities that actually implement the learned algorithm.

Labenz’s kitchen metaphor, which Sharkey accepts, is the gap in one image. Sparse autoencoders can tell you the fridge contains eggs, tomatoes, butter, and noodles, and when each ingredient comes out. They do not tell you the recipe. Fur, eyes, tail, and whiskers lighting up before a “cat” feature is a real finding, and it even rhymes with what we think the human visual system does. It still says nothing about the layer-to-layer transforms that turn those parts into a cat. Other activation-space methods try to characterize those transforms too; Sharkey’s bet is that the computations live in the parameters, and that activations are, in some sense, along for the ride.

Why isn’t it enough to say layer one has concepts A, B, and C, and layer two has D? Sharkey’s answer is philosophical and practical at once. Being able to find a direction in activation space does not mean the network *uses* that direction. You might cluster cat images by pose and get two features; the network may still treat both as “cat.” What it means for a network to use a variable, he argues, is that the variable is something the network actually computes over. Put the computations first and the representations second, not the other way around. Otherwise you risk writing an encyclopedia of distinctions the model never makes.

Multi-dimensional features make the same point geometrically. In some networks, days of the week are not seven independent directions. They lie in a plane, divided almost evenly around a circle, so a simple rotation implements a “next day” function. Modular-addition grokking looks similar: numbers become angles, the angles add trigonometrically, and the result maps back onto digits. An SAE trained to find sparsely activating directions can return seven day features, or a pile of modular-arithmetic directions, and still miss the shared operator. The operator is not a direction. It is a computation applied to many inputs. Transcoders inherit the same blindness: an identity map or a rotation is one simple object, yet a sparse dictionary may spend a separate encoder-decoder pair on every input-output feature.

APD tries to recover those operators by giving the network a third dimension. Labenz visualizes width and depth as usual, then a vertical stack of slices: copies of the architecture whose parameters must sum to the original. Training pushes three losses at once. Faithfulness: the slices add back to the target weights. Minimality: as few slices as possible fire on any given input. Simplicity: each slice should be low rank and span as few layers as possible, otherwise the trivial solution is “just use the original network.” Sharkey calls it model unmerging: imagine a giant mixture of experts crushed into one model, then pull the experts back apart. Channeling Ilya Sutskever, he says networks “just want to learn.” A larger parameter space has more ways to satisfy extra constraints. In toy settings, APD did recover interpretable mechanisms.

It was still a bad algorithm. Memory is the obvious killer: a billion-parameter model split into a million components is a quadrillion stored parameters. Training was “extremely janky,” hypersensitive to a discontinuous top-k gate that can swap which component is active after a tiny weight update. Gradient attribution, the score that decided importance, is systematically wrong in familiar cases. An attention head saturated near 1.0 barely moves if you nudge its weights, so the gradient says “unimportant” about a mechanism that is doing almost all the work. Labenz also worries about conceptual drift: unused “junk” slices could absorb the negative of whatever the active slices learn, so the live algorithm quietly diverges from the original while the sum still matches. Sharkey grants the worry for APD. He says SPD is built so that path cannot happen.

Stochastic parameter decomposition keeps the same goal and the same three conceptual constraints, then changes the objects and the attribution. Instead of full architecture copies, each weight matrix is broken into rank-one subcomponents: matrices that read from one direction in activation space and write to another. You can have more of them than the matrix has rank, the parameter-space analogue of superposition. Later they can be grouped into higher-rank or cross-layer components if the data says they belong together. Faithfulness still forces the pieces to sum to the original weights, so they cannot wander far as a set, even though they start from random initialization.

The “stochastic” part replaces gradients. A small network, in the paper often a 16-to-128-parameter thresholder, outputs a causal-importance score between 0 and 1 for each subcomponent on each input. At 1, the piece must stay fully on. At 0, it can be randomly masked anywhere in [0, 1] without changing the output. In between, random ablation should hurt in proportion to how much you turn it down. The reparameterization trick lets gradients flow through that random sample, so the importance network learns how ablatable each piece really is. Sharkey did not invent the trick, but he has not seen it in the attribution literature, and he thinks it would transfer to pixel-level image attribution. Crucially, SPD has no truly dead components. “Dead” now means “always randomly on,” not “always off.” Gradients keep moving. There is no unused slice that can hide a canceling copy of the live algorithm.

On toy problems with known ground truth, SPD recovers the expected mechanisms and is far less brittle to train. It also resists a pathology SAE users sometimes treat as a feature. Make an SAE wider and a “words beginning with E” direction can split into “elephant” plus “everything else beginning with E.” That finer dictionary may be less faithful to the variables the network actually uses. Because SPD components must sum to the original parameters, you cannot keep adding latents to chase reconstruction loss; Sharkey suspects the number of useful parameter pieces will level off faster than SAE dictionary size, though that is still an empirical question on language models.

Grouping remains the hard aesthetic judgment. In one toy, SPD split a W_out matrix into 50 rank-one subcomponents that then looked like a single rank-50 object. Why merge rather than keep 50 verbs? Sharkey’s answer is that there is no uniquely right cut. You can dial decomposition up and down. Reconstruction loss eventually gets worse; causal-importance scores start taking messy fractional values instead of sitting near 0 or 1. Those diagnostics mark healthier regions of the spectrum. The same caveat applies to sparse dictionary learning. Every method is a lens on one continuous network, and every lens throws information away. You pay that cost for a tractable explanation, not for a ground-truth parts list.

Scale is the next chapter, not a solved one. The team has run SPD on language models in the millions of parameters and poked at the low-single-digit billions. The open empirical question is whether the recovered pieces stay low rank enough to read. If language-model components come out too high-rank and too tangled, Sharkey says you would still have isolated cheaper objects to study, then you would need a second interpretability pass on top. That is work in progress, not a result.

Applications, if the method holds, sit closer to the weights than SAE workflows do. Surgical unlearning is the obvious one: you already know a vector in parameter space that implements a capability, so you edit that vector instead of translating an activation feature back into a weight change. Whether that beats a gradient-guided unlearning baseline is, he says, a separate question. Monitoring is the second. An SAE watches whether a particular input direction lights up. A behavior such as deception can ride many different input directions; the reused computation may be one function even when the activations point all over the map. Parameter decomposition is built to find that function. Labenz’s closing wish is blunt: it would be nice to ablate the deception circuit, if they can find it.

The application Sharkey is most excited about is knowledge extraction, including from scientific models Goodfire already studies. Some models are superhuman at tasks whose structure was never written in language—protein geometry is the example hanging in the air. If you can think in the network’s own computational terms, you might read out insights no paper contains. That is a hope, not a demonstration. It is also why he keeps returning to functions rather than features: the knowledge is in how the model computes, not only in which concepts fire.

Labenz frames the whole project as one of this generation’s scientific challenges, up there with understanding biology: explaining how and why neural networks do what they do, and whether we can predict them in novel circumstances. Sharkey’s work is an early bite, taken while the models under study are still small and the grouping rules are still partly a matter of taste. The conversation does not claim a finished microscope. It claims that looking only at nouns—labeled directions that light up—leaves the verbs unexamined, and that the verbs are where the algorithm lives.

That distinction travels beyond interpretability labs. Any time a complex system is reduced to a pile of tags, the SAE habit is nearby: count what appeared, miss the reusable transform that takes A to B. Sharkey’s days-of-the-week plane is the parable. Seven labels can be accurate and still hide the rotation that is the actual rule. Parameter decomposition is one attempt to put that rule on the page. It is immature, expensive, and philosophically modest about “truth.” It is also a reminder, for anyone who works with black boxes, to ask not only which variables showed up, but which functions kept being reused on them.