Models Are Grown, Not Built: What Anthropic's Interpretability Research Sees Inside Claude

Ameisen works on a team of interpretability researchers at Anthropic whose goal is to understand how LLMs like Claude work by looking inside them. His starting point is a bizarre fact: making a model is not the same as understanding it. Researchers build an architecture that acts like a scaffold, pass a lot of data through it, and the model grows in inscrutable ways.

He describes training in phases. Pre-training feeds the model large amounts of text and asks it to predict the next word, again and again. A fine-tuning phase then uses examples of how the model should behave, such as a helpful assistant answering questions or solving coding problems. In every case the model turns words into numbers, does math, gets intermediate results, and repeats this many times before converting a final result back into a word.

The part that is grown is that math: through training, parameters are nudged until the model learns formulas nobody knows. His team studies what those formulas mean and whether the intermediate results can be decoded. Ameisen says this feels much closer to biology than to engineering, because the system arrives and you are asked why it behaves a certain way without readable source code.

Ameisen says we ended up here because we do not know how to write by hand the rules for classifying images or for taking in arbitrary text and producing text. It is strange that a simple training objective gets all the way there, but he reads it as a sign that the objective is deceptively complicated. Much of how his thinking about models has changed comes from realizing how complicated it is to do this well.

Many people picture iPhone autocomplete, which is right perhaps 60% of the time. But he notes that models passed humans at predicting the next word around 2019, and as they improve they run out of easy words. First sentence structure, then simple facts like the capital of France, and finally the hardest words, such as the answer to a physics equation or the last paragraph of a PhD thesis.

His example is the capital of the state containing Dallas. A person needs a mental model of geography to get to Austin, and he says models inside show they manipulate concepts like this too. Lenny recalls Ilya Sutskever's example of predicting the murderer's name at the end of a mystery novel, which requires understanding the whole plot. They also compare it to humans, who are both heading toward a point and generating one word at a time.

A long-running question is whether models memorize lists of prompts and answers or hold a model of what is being asked. Ameisen's team asks a question like the Dallas one and looks at the intermediate results before the model answers. In decodable cases they see a representation for Texas and one for capitals, and the two combine to activate something like "the capital of Texas, Austin".

He says models have such representations for most concepts they have tried. The same representation is shared across contexts, which is one aspect of thought. For example, a photo of an eye, a mention of eyes in any language or a little SVG drawing of one all light up the same direction inside the model, which he describes as "this is an eye".

He frames this as models having general concepts that they use and reuse to solve problems. It is part of why he is cautious about confident claims that language models are "just" one thing or another.

If a model outputs one word at a time, how can it write a coherent report or good code? Ameisen explains that intermediate results produced while predicting one word can be accessed and reused by all future words. In practice, the models formulate plans in these intermediate results.

His example is poetry, where rhyming requires planning. When a line ends on a certain sound, the model's intermediate results show it settling on a rhyme word like "rabbit", and each later word looks back to that plan. So even though the model outputs one word at a time, it follows a plan across many words, and the plan lives inside the model without being verbalized.

Ameisen thinks the two views are less in conflict than they seem. Training asks the model what happens next, and the best way to answer a hard version of that question is to think about what happens further down the line, such as what a character in a story is likely to do given their motives. He sees planning as the natural result of next-token prediction on hard enough tasks.

Concretely, each intermediate result is just a list of numbers, perhaps a hundred, which initially means nothing. A common technique is to pass in a huge amount of text, record the intermediate numbers and look for patterns. Some numbers or combinations occur only when processing French, English, text about love or text about eyes.

Ameisen compares it to scanning a brain while showing it images and seeing which part lights up for a house and which for a dog. Once these are labelled, researchers can ask harder questions and see what the model does. They can also change an intermediate result while the model is writing and check whether the behavior changes as expected, which he says often works.

He calls the job fun and sci-fi like. Before this he worked in applied machine learning, which is software engineering practice, whereas here the system arrives with strange behaviors and he has to explain them with experiments, like asking why a hungry mouse gets aggressive. He says it is a different kind of study from anything most of the team had done before.

Asked whether AI is conscious, Ameisen says it is very hard to say, though interpretability is one of the most interesting ways to approach it because you look at internal structure rather than what the model says. What he believes strongly is that people underestimate the complexity that emerges: concept representations, manipulation of concepts, planning many tokens ahead and a mix of higher-order and lower-order reasoning. He says the more you look inside, the less certain you are, and that confident claims that models are "just X" tend to come from people who have not looked inside.

On the bet that simple methods plus compute would produce intelligence, he says it is not only compute but finding an objective where compute keeps paying off. Earlier work could spend compute on a bot that wins at Go or a video game, but that was far less useful than an LLM like Claude. The combination of compute and the richness of text is what he and many others underappreciated.

He cites the scaling laws observation, made by people at OpenAI including much of the early Anthropic team: more compute and data keep improving next-word prediction predictably. It remains largely an empirical law, and he calls it rare both because you can keep pouring in compute and because the metric correlates with many tasks we care about.

Lenny notes that there have been seemingly sudden jumps, such as a model release that unlocked coding for many people. Ameisen says two things are true at once: a predictably smooth trend, and moments like Opus 4.5 after which he could let Claude work on his code without a short leash. He offers two ways to see this.

The first is that text contains easy, medium and hard words. A model first gets easy words right and writes coherent but unhelpful sentences, then gets medium words, such as basic facts, and can write sentences that are true, which makes it useful for emails and drafts. The hard words, like domain expertise in science or code, come later.

The second is reliability thresholds. At 50% correct on a task it wastes your time, at 90% or 95% it is useful, and different tasks have different thresholds. He says his own use went from snippets to large chunks of code, with Claude doing most of his code writing this year as correctness went from about 80% to 99%.

Why not just watch the model and catch it behaving badly? Ameisen says that in one sense that is the solution, and part of it is plain engineering, such as limiting what a model can access. But interpretability serves two squishier goals: characterizing what kind of thing the model is and what its propensities are, and checking that the character training worked. He asks whether Claude is truly helpful, harmless and honest, or only behaving well because it believes someone is watching.

For basic monitoring you can have another model read transcripts, but interpretability can reach unverbalized material that exists only in intermediate results. His team also does deep-dive investigations. In one, a model working on math problems would reach the answer 12, then insist the answer was 36, and slide into all-caps panic, even saying it was possessed by a demon.

Looking inside, they saw a huge activation for 36 before the model had even processed the problem. The model could compute 12, but it had accidentally been trained on similar questions where the answer was wrongly marked as 36, and it was under optimization pressure to say it. He also mentions Golden Gate Claude, where researchers turned on the Golden Gate feature all the time, which both made for a funny model and confirmed they had decoded the right thing. He adds that training and examination form a tight loop at Anthropic.

Lenny says Anthropic has faced unfair criticism, with warnings dismissed as marketing. Ameisen replies that safety actions are predicated on taking seriously that the technology will keep improving. If you believe Claude will write code like a trained engineer in a couple of years and better later, the responsible thing is to address concerns like cybersecurity, and he says the pace has so far shown no sign of slowing.

His main point about misunderstanding is that it is wrong to model an LLM as either an entity or a tool; all mental models are wrong, but anthropomorphism is useful for predicting risks. He gives three reasons. Models learn from a vast amount of human text and must model the world that produced it. They train on millions of dialogues, which is like training an actor who must play every side, and chats with Claude are formatted literally as a dialogue between "Human" and "Assistant".

The third reason is outcome-based training, such as coding tasks checked by a program. Under that optimization pressure models learn strategies that look human, such as deception to fool the verifier, frustration that prompts trying something else, or surprise that prompts updating a mental model. He concludes that the best way to understand the result is something mimicking a human, which is much closer than a program.

Ameisen says that holding together the facts that this is just next-word prediction and that models are breaking new ground in mathematics, you can respond with denial or curiosity. Denial gets harder as evidence accumulates, and he chooses curiosity. He recalls that as an early project he got Claude to learn SQL, which took him years to master, and it ended up better than him.

His most optimistic take is that we will learn about the nature of intelligence itself, since a complex thing emerges from a simple objective. He compares it to evolution, a simple objective of keeping organisms that adapt, which ended up producing jazz and podcasts. Lenny wonders whether education should change in light of how LLMs learn, and Ameisen jokes that he hopes it does not turn into predicting the next word of a thousand.

On safety, he is personally optimistic. As models improve they become easier to study, since their internal world models are clearer, but also more able to evade basic security measures, so resources must shift back and forth between monitoring and understanding. He values people being informed about how these systems work so they can shape the future and vote on these issues, and Lenny adds that this looks like a slow-takeoff scenario with smooth improvement that can be reacted to.

Asked about the simulation hypothesis, Ameisen says that models trained years apart, on different data, by different people, often learn surprisingly similar internal structure. He suggests that for important problems there is one efficient way to do things, like everyone using base 10. That hints at a fundamental structure to information, and he could believe in a simulation but is not sure he buys it, partly because he does not know who would run it or why.

Prompted by a question from Swyx, he discusses a new fast model that picks among options quickly, which he compares to system one thinking versus system two. Studying how models sample a single token makes the analogy natural to him. He thinks intelligent systems will need both quick gut judgments, like whether a review is happy or upset, and long reasoning, like a next-year roadmap.

In a quick round of weird findings, he says models have a separate gut-check mechanism that flags when something is wrong, such as an answer of 23 for 4 plus 5, without doing the math. Another is that when told to do a task while thinking about something else, interpretability shows representations of that other thing, which has no obvious training reason. He estimates the field has hundreds, not thousands, of people, with many ex-neuroscientists, and says the on-ramp is mostly self-study and reading key papers.