An AI agent probes the neurons inside a language model

Goodfire researcher Curt Tigges explains why weak activations can often be removed and how agents test descriptions of individual neurons.

CTCurt Tigges게스트 · Goodfire
채널
Cognitive Revolution "How AI Changes Everything"
게시일
2026년 11월 10일
영상
1:55:03
읽기
8분
원문
영어
YouTube에서 열기
스크롤하면 영상이 오른쪽 아래로 작아져 계속 재생됩니다. 자막은 읽기 언어를 따릅니다.
읽기 언어2개 언어로 읽을 수 있음
이 글은 아직 한국어 번역이 없어 영어 원문으로 표시합니다.
요약

30초 요약

Curt Tigges says his team found many neurons in a large language model can be described more clearly than researchers expected. Keeping only the strongest activations in each active expert preserved performance on several tests, although reasoning initially suffered and needed further training. An AI agent investigates individual neurons, and a separate model tests its explanations. Tigges says mapping neurons is only a step toward understanding circuits and whole-model behavior.

  1. 1Individual neurons can have a bounded set of rolesTigges argues their behavior is often more interpretable than expected.
  2. 2A sparse model retained performance on several testsThe team removed weak activations, then used additional training to recover reasoning.
  3. 3An agent tests and revises neuron explanationsA separate model checks whether those explanations predict activations.
  4. 4Neuron descriptions alone cannot explain the whole modelTigges says the next step is to connect neurons into circuits and behaviors.
  5. 5Very long contexts remain to be testedTigges reports coding tests up to 32,000 tokens, not one million.

핵심 정리

  1. Many studied neurons have a limited, describable set of roles, Tigges says.
  2. Removing weak activations preserved several benchmark results; reasoning initially degraded.
  3. An investigating agent proposes descriptions that a separate model tests against activations.
  4. Understanding individual neurons does not yet explain whole-model behavior.
전문

챕터별로 다듬은 전문 읽기

01

Why study individual neurons?

Labenz and Tigges introduce the claim and the model used to investigate it.

Nathan Labenz introduces Curt Tigges, a Goodfire researcher studying how the individual components of language models work. Tigges argues that many neurons are easier to interpret than researchers had thought.

The discussion focuses on a language model with millions of neurons and on an AI agent designed to investigate them. Tigges describes experiments that remove weak activations, test explanations and examine groups of neurons.

Labenz says the results may make it easier to study model internals directly, but asks whether explanations of individual components can lead to understanding of whole behaviors.

Labenz introduces Tigges and his work with Nick Cammarata. Their paper examines language-model neurons in greater detail than the accompanying essay.

Tigges says neurons can be complicated and sometimes respond to several ideas. His group finds that those meanings are often a manageable set rather than an unbounded collection.

Labenz asks whether the underlying systems have become more interpretable or whether researchers simply have better tools for inspecting them.

Tigges credits both improved tools and differences between model architectures. An agent can search many activation examples for patterns, while modern gated components may permit clearer logical operations.

The main model they studied is Qwen 3.6-35B-A3B, a mixture-of-experts model. Only a subset of its specialist subnetworks is selected for any particular input.

02

Experts and gated neurons

Tigges explains the model architecture and the two paths into a gated neuron.

Tigges describes 40 layers with 256 experts per layer, of which eight are selected at a time. Each expert has 512 neurons, though the strength of their activation varies.

He says a neuron can respond in positive or negative directions and that the two directions can have different meanings. The team chose this model partly because its architecture resembles features of other large open models.

In the gated component, two separate paths process information and their results are multiplied. Tigges compares one path detecting a cooking context and another detecting an ingredient such as garlic.

Labenz asks how this more elaborate design became common and whether it was designed around interpretability.

Tigges says the design became popular because it worked well in practice. His group offers an after-the-fact account of how separate inputs can switch among contexts.

He describes a neuron associated with oral health in one context and marine wildlife in another. Different downstream readers are prepared by the surrounding context, he says.

Labenz checks his understanding of the two paths: independently arriving signals can give the model more ways to combine information.

Tigges explains that the particular gated design is called SwiGLU, while SiLU is the activation function within it. He says the two incoming paths often respond to partly distinct things.

03

What happens when weak activations are removed

The speakers compare sparse selection methods and benchmark results.

Labenz turns to sparsity: what happens when many weak neuron activations are removed while the strongest remain?

Tigges distinguishes two tests. One replaces selected activations with their average; another retains only a fixed number of the strongest neurons in each active expert and sets the rest to zero.

He says the sparsified model still performs well on several benchmarks, though reasoning initially suffers and can become repetitive. A small amount of further training can restore much of that performance.

Labenz asks why selecting the strongest neurons within each expert differs from applying a percentile threshold to each neuron across the dataset.

Tigges says a given activation percentile can carry clear meaning for one neuron but mostly noise for another. Selecting the top neurons within each expert gives each expert its own effective threshold.

The team studies examples across activation levels and looks for the point at which a recognizable pattern dissolves. Tigges says the extremes usually contain the clearest and most causally relevant cases.

Labenz observes that removing the middle half of activations changes many benchmark scores little, while more aggressive removal eventually causes greater degradation.

Tigges suggests that some improvements may come from the model reaching an answer more confidently or quickly, rather than being distracted by weaker internal signals.

The next challenge is to describe what neurons do, not just identify which activations matter. Labenz asks how the team scales that work to millions of neurons.

04

How an agent investigates neurons

The team groups examples, runs experiments and tests explanations.

The team first groups examples in which a neuron activates. An AI agent called the neuron scientist receives these examples and proposes a hypothesis about when the neuron will fire.

The agent can modify sentences, compare similar inputs and run tests against the model. It uses the results to refine an explanation rather than producing one label and stopping.

A separate system tests whether the description predicts where and how strongly the neuron activates. Feedback goes back to the investigating agent when predictions are poor.

Tigges says simple automatic labels can be vague. Allowing a stronger agent to perform more experiments made more neuron behaviors predictable in their tests.

One neuron appeared unrelated to a common topic in short excerpts. Further investigation suggested it responded to the second item in a list restated in a different format much later in the text.

Labenz asks how the team validates an explanation once the agent has produced a paragraph describing the neuron.

Tigges describes deleting a neuron’s contribution, then asking another model to predict its activation from the description. Those predicted values are inserted back into the original model to measure how much of the damage is recovered.

The reported 74% recovery uses a measure of changes to the model’s next-token predictions, weighted more heavily where removing the neuron caused more damage.

Another test asks a model with only the description to construct text that activates the neuron strongly. The team also checks whether the description can predict when a neuron is relevant at all.

05

From neuron meanings to circuits

Tigges discusses multiple meanings, control signals and the gap to whole-model behavior.

Labenz asks how to decide whether a neuron has one meaning or several, since a single description can contain multiple related concepts.

Tigges says a neuron might respond to several insect-related preferences, such as food, habitat and egg-laying locations. Whether those are one concept or three depends on the boundaries drawn.

He argues that an apparently unrelated collection may share a higher-level explanation that is not obvious to a human observer. The practical question is whether researchers can enumerate what activates the neuron.

Labenz asks how a catalog of individual neurons becomes an explanation of circuits and behavior.

Tigges describes some neurons as control signals: they can direct a model to continue, reconsider or change a line of reasoning, rather than merely detecting an object.

He says understanding every neuron separately would still leave the relationships among them unexplained. Researchers must examine the circuits those neurons form and how those circuits affect behavior.

For safety research, he says, general mapping must be paired with tests that deliberately elicit behaviors of interest, such as reward hacking, to see which circuits are involved.

As an example, Tigges describes a small community of neurons associated with identifying painters and artistic movements. He proposes working upward from individual neurons to such communities and larger circuits.

Labenz presses on the gap between many small circuit descriptions and reliable predictions of what a whole model will do in an unfamiliar situation.

Tigges says researchers have begun to find potentially useful monitoring signals, including a neuron associated with one kind of reward hacking. He cautions that one signal need not cover every form of that behavior.

He compares the missing intermediate explanations to higher-level scientific concepts: a list of microscopic components does not by itself supply every useful description of a system.

His proposed approach combines bottom-up study of components with top-down experiments that change model behavior, then looks for where the explanations meet.

Labenz sees monitoring and debugging as possible applications, while warning that a monitor based on a single neuron may miss other routes to the same behavior.

Tigges says tools that create external feature representations may still help with other parts of a model, including the residual stream. For neurons, he favors examining them directly where that works.

He also says the residual stream remains difficult to understand and requires better tools for its geometry.

Labenz asks where the model stores facts if it has fewer neurons than facts it can recall.

Tigges says facts need not each reside in a single neuron: connections among neurons can represent far more combinations than the count of individual neurons.

On factual benchmarks tested so far, he reports limited degradation after additional training, while acknowledging that much larger tests might reveal losses.

06

Facts, expert specialization and model design

The conversation covers connections among neurons and possible future designs.

The active set changes with each token. Tigges says this means different neurons and connections can still contribute at different points even when each step is sparse.

Labenz asks whether the experts themselves have recognizable specializations. Tigges says the experts examined by the team tend to concentrate on a small number of subjects.

Examples he gives include legal text, algebra, geometry, timekeeping and particular medical subjects. These are observations about experts the team examined, not a complete map of the model.

Tigges suggests that future models might be designed to make important knowledge more localized and behavior easier to monitor. He presents sparsity as one possible design direction.

Labenz asks whether simpler activation functions might make some unusual cases easier to interpret, even if this required a tradeoff in model performance.

Tigges discusses cases where two negative signals multiply into a positive activation. He says the team does not yet fully understand these cases and plans further work.

The pair return to the reasoning performance lost after removing weak activations. Labenz asks why a comparatively small training run restored it.

Tigges corrects the training amount to about 100 million tokens. He says the run took roughly a day on a few GPUs and was small relative to the model’s original training.

Before further training, the model sometimes repeated checks or became stuck. Tigges thinks its underlying knowledge and reasoning mechanisms remained present but its behavior became poorly calibrated.

He says the lightly retrained model performed at or above the original on the tested math, coding and factual benchmarks. He adds that practical differences require longer use to assess.

07

Reasoning, long context and next steps

Tigges describes retraining the sparse model and the limits of the tests.

Labenz asks whether weak signals might matter more over very long conversations, even when short tests look good.

Tigges says the team tested coding tasks up to 32,000 tokens; million-token contexts remain a question for future work. Different tasks also appear to need different amounts of retained activity.

He describes using an attribution graph and neuron descriptions to infer entities and relationships in a passage. He hopes similar methods could eventually help investigate less visible model processing.

Tigges says the cost of neuron analysis varies considerably. His most expensive experiment used about $45 of inference for one neuron; he expects a broad first pass followed by deeper work on difficult cases to be more practical.

He encourages others to experiment with small local models and credits earlier researchers whose neuron-level work informed the project.

Labenz thanks Tigges for explaining the work and its open questions.

같은 채널의 다른 글

슬라이드 8장43:36EN
Cognitive Revolution "How AI Changes Everything"· 2026년 11월 14일팟캐스트 · 2명Nathan Labenz: Use AI’s benefits without surrendering control진행자 Gene KimA red-team veteran explains why agents need escalation paths and manual overrides—even as he argues for wider AI adoption.AI SafetyAI agentsAgent red teaming+2
1:30:25EN
Cognitive Revolution "How AI Changes Everything"· 2026년 11월 11일팟캐스트 · 2명When AI Agents Talk Past One Another: Cisco’s Case for Shared Context게스트 Vijoy Pandey · 진행자 Nathan LabenzVijoy Pandey explains how Scope, Accord and Concord aim to expose misunderstandings and coordinate agents across teams and tools.AI agentsAgent orchestration평가(Evals)+1
1:26:17EN
Cognitive Revolution "How AI Changes Everything"· 2026년 11월 7일팟캐스트 · 5명AI Forecasts, GPU Markets and the Question of What to Build게스트 Brian Zhou, Mike San Miguel, Bryan Bischof · 진행자 Nathan Labenz, Prakash NarayananNathan Labenz and Prakash Narayanan examine AI acceleration through election forecasting, compute trading and venture research.AI research평가(Evals)인프라와 배포+2
1:43:54EN
Cognitive Revolution "How AI Changes Everything"· 2026년 11월 5일팟캐스트 · 2명Joshua Monrad on AI, Biosecurity and Building Layers of Defense게스트 Joshua Monrad · 진행자 Nathan LabenzSentinel Bio’s co-founder discusses uncertain AI-related biological risks, practical safeguards and the long work of global resilience.BiosecurityAI Safety안전과 거버넌스+2
1:26:42EN
Cognitive Revolution "How AI Changes Everything"· 2026년 10월 31일팟캐스트 · 7명AI:AM on Alignment, Memory Chips and an Agent-Run Economy게스트 Anastasios Angelopoulos, Debo Olaosebikan, Samvit Ramadurgam, Michael Freeman, Sagar Vaze · 진행자 Nathan Labenz, Prakash NarayananThe Cognitive Revolution’s hosts examine reported model behavior, new memory hardware, work and cyber risk with guests from Arena, Kepler, Union, ServiceNow and Mistral.평가와 관측 가능성AI AlignmentAI security+2
1:49:40EN
Cognitive Revolution "How AI Changes Everything"· 2026년 10월 28일팟캐스트 · 4명A Strategy for Preserving Options on the Path to Advanced AI게스트 Joel Predd, Ben Boudreaux, Matt Chessen · 진행자 Nathan LabenzRAND researchers discuss uncertainty, security architecture and human agency with Nathan Labenz.AI StrategyGeopoliticsAI Safety+1