An Algebraic Geometer Quit Tenure Because the Bowl-Shaped Loss Plot Is a Lie

Nathan Labenz opens by contrasting Timaeus with the default of prosaic safety — RLHF, constitutional AI, red-teaming, defense in depth. He is looking for something that could still work in a surprising future, even if it is a long shot. Jesse Hoogland and Daniel Murfet’s bet is developmental interpretability: a mathematically grounded way to watch a network as it trains, rather than reverse-engineering it only after the run is over.

The name is not a startup pun. Timaeus is the Platonic dialogue that offered a proto-chemistry, pairing earth, air, and fire with the cube, icosahedron, and dodecahedron. The theory is completely wrong, Jesse says, but it is the first time someone seriously proposed that mathematics might understand the natural world. That spirit is what they want for deep learning. In the same dialogue the world itself is intelligent, so the history of the universe is a learning process and physics becomes a subset of learning theory — a line Jesse mostly floats to troll physicist listeners, then walks back to a more careful claim: statistical physics and learning theory are deeply connected, which is why so many physicists are now spending their careers on these systems.

Murfet was an algebraic geometer by training. The field studies solutions of polynomial equations, very high-dimensional geometric forms, and the singularities those forms carry. Most of it, he says, is not related to the real world; a little of his work touched string theory and mathematical logic. Deep learning was interesting and completely separate from his professional life until he found Sumio Watanabe, the Japanese mathematician who invented singular learning theory. One of Watanabe’s core tenets still struck Murfet as hard to believe: a very deep part of algebraic geometry is central to Bayesian statistics. People were not satisfied with the theoretical treatments of deep learning then on offer. For an algebraic geometer, the invitation was obvious. He later left a tenured professorship to do this full time at Timaeus.

If Jesse had to put singular learning theory in one sentence, it would be this: the geometry of the loss landscape — the surface an optimizer walks down — contains information you need if you want to understand neural networks and steer them more reliably. That has applications for interpretability and for alignment. When they started taking a still-very-mathematical theory into empirical networks about two years ago, the first agenda they named was developmental interpretability: can SLT tell you how a network develops over the course of training?

The hope is that development is an axis that shrinks the problem. Every interpretability method is trying to reduce a hundred-billion or trillion-parameter model to fewer numbers. If the model is the product of a finite number of changes, and that number is much smaller than the number of parameters, you have a useful simplification. Labenz asks whether each training step counts as a change. That would not be a reduction, Jesse says: there are still billions of gradients. The unit has to be larger than a step. SLT’s candidate is a phase transition — a developmental stage, in the biological sense.

Whether those stages exist at a useful frequency is an empirical question. In the systems they have studied so far, Jesse says, they do. Labenz asks if a phase change is like grokking. Grokking is one example. They distinguish two primary kinds of change. A Type A transition is the model becoming more complex as it learns more, the way a person builds a richer and more accurate mental model of the world, each step requiring a more sophisticated picture. A Type B transition is the grokking-like case: the same performance on the data, but a simpler solution that therefore generalizes better. The tension between those two is the learning process they want to instrument.

Labenz quotes Tom, chief scientist at Goodfire, granting mechanistic interpretability “proto-paradigmatic” status: networks learn interpretable features, packed densely via superposition, so neurons are polysemantic; features are almost orthogonal; activation magnitude tracks relevance; features wire up into circuits across layers. Murfet largely signs on. He is a fan of the recent Anthropic work and of what Goodfire is doing. Tom’s graph analogy is the one he likes: if you have a graph you need both the nodes and the edges. Sparse autoencoders look at features, the nodes. The information is ultimately in the parameters, and one way to read those is the geometry of the loss landscape — the edges. He does not see the two approaches as a struggle; they are likely to inform each other.

His critique, if he has to pick one, is that SAEs still lack a mathematical foundation that connects them to generalization. Find circuits in a base model, then continue with continual learning or reinforcement learning, and it is unclear how the structure you found relates to the data-generating process or how stable it is. He put the concern to Chris Olah. Olah’s reply, which Murfet passes on, is that fine-tuning probably recruits existing representations and circuits and uses them for a different purpose. Anthropic’s before-and-after feature studies are consistent with that. An empirical check can answer the question, Murfet says. It is not, in his view, a path to the high assurance people want when they talk about generalization and AI safety.

He adds a caution from the previous question. They do see clear phase transitions in small systems. Nobody really expects hundreds of sharp step changes in the loss curve of a large model. If development is going to be an axis of reduction, you need instruments that can look more closely, or tune to different frequencies, inside a training run that looks homogeneous at coarse resolution. Their own approach starts from the generalization end and works toward interpretability, and it is more about parameters than activations. The two pictures will have relations; the starting point is different.

Everyone has an intuition for generalization. Labenz offers the ChatGPT-user version — Ilya’s line that the amazing thing is talking to the AI and feeling understood — and the grokking version, memorization giving way to an algorithm. Murfet wants to be more precise. Technically, generalization is usually a number, generalization error: how well the model predicts on samples from the same generating process that it did not see in training. That is in-distribution generalization, even when people omit the prefix. Out-of-distribution generalization is a much more fraught object, because the new samples could be anything.

Beyond the number, you can ask for an understanding whose mathematical type is not a scalar but something like an algorithm — the thing that *lets* you generalize. If you have low generalization error, maybe that is because the model internalized an algorithmic aspect of the generating process. Interpretability, on this view, is the study of generalization with a changed type signature: not the number, but the thing behind the number. Labenz notes that even modular arithmetic can grok in more than one way; Max Tegmark’s group described a trigonometric solution and a “pizza slice” algorithm that both get the right answers and perform quite differently. That gap is exactly why you might care which algorithm the geometry selected.

Jesse then states what Labenz has started calling their central dogma, and what they call the S4 correspondence. Neural-network behavior ultimately comes from training data. Every learning process has three inputs — architecture, optimizer, and data — and data is the one that tells one model to learn one algorithm and another to learn a different one. RLHF, constitutional AI, DPO, deliberative alignment: all of them are modifications of the same deep-learning process with different data. Data determines the geometry of the loss landscape; that geometry tells SGD how to move; the learning process picks the final weights and thus the algorithms; structure in those weights determines how models generalize and whether they are aligned. Structure in the data, in the geometry, in the learning process, and in the final weights is the mapping they want to read, and eventually invert.

Everyone has seen a one-dimensional loss curve dropping over time, and a two-dimensional well that looks like a valley. Labenz does not trust his intuition about whether two dimensions are anything like two hundred billion. Murfet’s first warning is the geometer’s: in one, two, or three dimensions you can use pictures; past that, pictures are not a reliable guide, which is why you bake the low-dimensional intuitions into mathematics that keep working. A surface plot of a loss landscape is made by choosing two directions in a very high-dimensional space and plotting loss as a function of coordinates on that plane. With probability one, a random slice will not show the degeneracy that, according to singular learning theory, is what determines generalization. In that sense the pictures are “maximally misleading.”

The relevant geometry looks more like a bunch of planes intersecting in some crazy way. That is what the level sets look like — the set of points with a given loss. In the usual valley cartoon those sets are smooth closed curves, and at the bottom you get a single point. The actual level sets are complex geometric forms with lots of intersecting lines, high-dimensional things knotted up. Those are the singularities they care about. And the loss landscape in the theorems is the *population* loss, a theoretical object: the average of the loss over every sample from the true generating process. You never have access to it. Empirical loss landscapes, built from the samples you happen to have, are influenced by that geometry but are not it.

Jesse offers a slightly higher-level picture. People imagine the bottom as a round basin, a parabola. Degeneracy is the fact that this is the wrong shape: there are valleys and canyons, directions you can walk that do not change the loss. SLT says those are by far the most important directions. The number of valleys is what determines how well the model generalizes. More valleys means you can perturb the weights, change the implementation, without changing the function — which means the function is simpler. Simple functions can be implemented in more ways. Through an Occam’s-razor argument, the simplest function generalizes best, and SLT can make that statement precise. Labenz’s reality check is that training does not even give you the landscape: you take a gradient, you take a step, and you have almost no guarantees the step was good.

Labenz’s basic objection: if you never have the population loss, how do you know these flat directions exist? You cannot just wiggle five inputs and declare victory. Murfet’s first answer is to study networks simple enough that you *can* write the population loss in closed form, which is where Watanabe started. One-hidden-layer tanh networks on a synthetic data distribution let you do the integrals, see the degeneracies, and check the theorems that connect generalization to geometry. They did the same for Anthropic’s toy model of superposition, a small autoencoder whose population loss can actually be written down. Training in that model is governed by the degeneracies where the theory says it should be.

There is also a boring kind of degeneracy worth getting out of the way. Scale a ReLU’s input up and its output down by any positive real, and the function is unchanged. Whenever two matrices are multiplied — query and key in attention, for example — you can scale one up and the other down, or insert any invertible matrix and its inverse, and the outputs stay the same while the weights change. Those symmetries are not uninteresting: the QK degeneracy affects what functions that part of the network tends to learn. Other degeneracies emerge more globally and cannot be pinned to an obvious rescaling. Past a certain model size and a certain interestingness of the data, closed-form calculation stops being feasible and the question becomes empirical: can you extract signals of degeneracy from empirical losses? The theory says you can. That is a large part of what Timaeus does.

Why should any of this *govern* training rather than sit there as an accident of batch order? Murfet reaches for a principle from dynamical systems. Following a gradient is uninteresting when the potential is a sum of squares, a bowl; that is convex optimization. In nonlinear dynamics, the places where the gradient vanishes organize the trajectories. In one dimension those places are maxima, minima, and saddles — the points a calculus course spends its time classifying, because the rest of the curve is boring. In higher dimensions they are singularities. No trajectory that starts outside a singularity ever arrives, and none that starts there ever leaves, but nearby trajectories approach and then escape in particular directions. The singularity organizes the global set of paths. In the superposition toy model you can see it: the columns of the weight matrix plot as vectors in two dimensions, vertices of a pentagon or a hexagon. Moving between neighborhoods of critical points is literally growing a leg, contracting a leg, and rearranging the others. In a large model the singularities are not isolated; they are everywhere, and that picture is not well understood.

Think of singularities, Murfet says, as different ways of solving the problem at a given level of loss — different algorithms that look identical on the training data and may generalize in ways you like or do not. Geometry in the neighborhood is how you tell them apart. The concrete case is a paper they call essential dynamics: train a transformer on in-context linear regression. Feed it XY, XY, XY samples, each Y a linear transform of X plus noise, and ask it to predict new Ys. Expose it to many distinct tasks and it learns to do regression over the context. Expose it to only a few and it memorizes those tasks and never learns regression. Which solution you get is a phase transition as you vary task diversity.

Each solution has a performance and a complexity. Memorization always beats the regression solution on training performance. The generalization, or meta-learning, solution is simpler once there is enough to memorize. The simplest solution occupies more volume in parameter space and is easier to find, so the model typically learns the generalizing solution *first* and only later, if at all, moves to memorization. That is the opposite of grokking, and Jesse thinks the field has in a sense overfit to the grokking example. What is special about grokking is that training-set performance is more or less identical: you start with the more complex memorization solution and, at a fixed loss, drift into a broader, more degenerate basin. That can only happen conceptually at the very bottom of the landscape, where simple and complex perfect solutions coexist. The more typical relation, without those artificial conditions, is that you prefer a simple high-loss solution and later pay a complexity penalty to buy lower loss.

SLT says models minimize an effective loss: the actual loss plus an emergent regularization term, an implicit bias that comes from degeneracy — the number of ways you can perturb weights without changing the function. The quantity that decides which solution is preferred, in the Bayesian sense, is free energy, the sum of loss and a complexity term estimated by the local learning coefficient, the LLC. A Type A transition lowers free energy by raising complexity enough to cut loss; that is ordinary learning. A Type B transition lowers free energy by cutting complexity at fixed loss; grokking is one of those, and they have seen others, including in the linear-regression setting. Why would a trainer who is only minimizing loss feel a pull toward simplicity at all? Drop yourself at random in the landscape and you land somewhere high-loss *and* broad. As you move down, it is easier to find canyons with more valleys than to find narrow nooks. That volume effect is the implicit regularization.

Labenz tries to connect the picture to double descent. Murfet is not very helpful, on purpose. Nobody has systematically studied double descent with singular learning theory; they have not really tried, partly because from their mathematical vantage it does not look that mysterious that models with many parameters can generalize well. The folk story that high dimension always leaves you a path around the hill is, he thinks, closer to a folk tale than an explanation. You can find high-parameter systems that do *not* optimize well, including badly configured neural networks. SLT does not resolve those mysteries. It does give you a framework in which you are not surprised that a large model can predict well. There is a possible story that backs a double-descent-shaped curve out of the generalization curves of the transitions they study. He does not claim it.

Zoomed out, the premise is the old one: two models can match on training behavior and generalize differently on deployment, and we cannot currently tell a friendly superintelligence from one that has learned to please us. Interpretability, for Jesse, is understanding internals well enough to disambiguate those cases. SLT says look at how structure is reflected in loss-landscape geometry. They have a principled link to in-distribution generalization, and they want that as a starting point for out-of-distribution behavior and for SGD. In practice that looks like probing points in the landscape: apply small perturbations, hit the model with a hammer, see how it responds. How much performance drops tells you about the complexity of the solution. How much performance *increases* when you apply a structured perturbation to a specific attention head can tell you that this head is specialized to a kind of data — induction heads here, n-gram and skip-n-gram memorization there.

More recently they have been aiming this at circuit discovery: associating components with patterns in the data, attributing structures back to the data that activate them. Sensitivity analysis can tell you which components are involved without a full causal input-to-output map. That is a top-down approach, informed by perturbation, rather than a full mechanistic story. They validated the circuit-discovery techniques in small three-million-parameter language models. The question now is whether they work in seven-billion-parameter models; Jesse thinks that is quite likely by the end of the year. Labenz’s analogy is the “toward monosemanticity” stage of the SAE literature. Compute is the constraint. SAE training is expensive in one way; their burden sits in other places, but understanding large models is still substantial. More compute means more samples and more perturbations, which means a finer signal. If a listener has compute to offer, they should get in touch.

The other half of the agenda is not just reading the model but steering the learning process so you do not get a misaligned one in the first place. That will look like intervening in training data. If the map from data through geometry through learning to weights is even somewhat invertible, you could go backwards and choose data that aligns more robustly. Murfet does not bite on a proposal to measure alignment by how often noise elicits bad behavior. The developmental version is: if you can trace a concerning structure to a pattern in the data, intervene on that pattern earlier. He thinks it is naive to throw “be good” samples in at step 12,000 and expect a complex mechanism to comply. The better analogy is morphogens, the molecules and proteins that decide when and what happens in biological development. Insert the right morphogen at the right time and you change the outcome; that is how parts of synthetic biology work. You need interpretability progress before that kind of intervention has the right grain.

Labenz asks whether reward hacking is overgeneralization: we specified a poorer signal than the concept we meant, and the model understood the signal too well. Under the technical definition, Murfet says no — reward hacking is just finding more of the explicit target, analogous to lower loss, not a complexity trade-off. It could still be that some incidents we currently call reward hacking are partly a preference for a simpler solution than the one we intended. He does not claim evidence. The linear-regression experiment is the cleaner illustration. In theory, train long enough and you should always switch to memorizing, because that is the global training minimum. In many runs the model sticks at the generalizing, higher-loss, simpler solution and never switches. Draw the landscape, even if Murfet hates the drawing: at the very bottom is what the training data actually specifies; above it is a simpler heuristic. Training can get stuck there. The heuristic to take into AI-risk debates is that you may not get what you ask for. You get a simplification, an approximation. That could be good. It could be bad.

The usual safety intuition is that simpler is better. A scheming agent would need the structure that does the task *plus* extra structure for taking over the world, and a simplicity bias would pressure that extra structure away. That may be a reasonable story. Murfet would not put much confidence in it: it is imprecise and not grounded in much mathematics. The opposite story has a historical example. In the 1940s the United Kingdom let citizens of the British Empire settle with little documentation, many of them from the Caribbean. In the 1970s the laws changed: you needed papers to access services. People who had come under the earlier policy did not have those papers. Some were deported; many bad things happened; a 2018 investigation made it a scandal, Windrush. The intended policy of the state was not to deport people it had invited in. At street level the implementation became a brute judgment call — what scholars call street-level bureaucracy, a simplification of the original intent with unintended negative consequences. If the behavior we specify at great expense in training data is simplified in ways we do not anticipate, we should be concerned in the same way. Labenz’s dry gloss: a state and a model may be more similar than he had conceived.

Jesse’s picture of the future of training starts with alchemy. The architecture is a huge cauldron, the optimizer is the fire, the reagents are the training data. Current practice is to throw the entire internet into the pot, start stirring, and hope you have not mixed bleach and Clorox. What they have in mind looks more like industrial chemical manufacturing: you know which reagents you mix, at what concentration, at what point in the process, with which catalysts. That control is not available now, but the field is already stumbling toward it empirically. Post-training has splintered into stages — chain-of-thought RL, a second stage of it, a bit of instruction fine-tuning, constitutional AI, a second constitutional-AI pass just for personality, more refusal training. Pre-training is being split as well. The scientific version would be knowing when two samples in a batch should be mixed to get a desired behavior out the other end.

Labenz’s concrete case is the Claude 4 system card. Anthropic observed the model following harmful system prompts, traced it, and found they had omitted a dataset built specifically to teach the model what to do when given a harmful system prompt. Without that curb, it learned the simpler solution: always follow the system prompt. They caught it with behavioral testing and patched it. They did not roll back and retrain from the point the data should have been introduced, which Labenz reads as a clue that the omission sat earlier than a last finishing stage, and that compute or launch-timeline constraints were binding. Someone at Anthropic later told him they were confident they got the value of that dataset in the end. He believes them and still wants to know on what basis that confidence could rest, other than poking the model a lot. That is the kind of process-level confidence developmental interpretability is supposed to create.

Murfet’s closing standard is modest and hard to argue with. Civilization is about to be transformed by this technology; it is not an unreasonable expectation that deep learning be somewhat more of an engineering science and somewhat less of a big mystery. Data mixtures and their order are already empirically finely tuned, and hyperparameter selection has some underlying theory. Lots of people and money are in it. Most people would still agree it should look more like other kinds of safety engineering than it currently does. Shaping behavior only at the end of training looks shaky as capabilities rise; baking control in earlier is something the field is already moving toward — deliberative alignment if RL compute matches pre-training, papers on pre-training from human feedback. Over the next months Timaeus is scaling circuit discovery toward 7 billion parameters, running first steering experiments in small language models, and developing an SLT extension of influence functions for data attribution and elicitation, including ideas that might help the UK AI Security Institute. The papers, a Discord, and a weekly seminar live at timaeus.co.