Resample Every Token and You Can Watch the Model Decide

Nathan Labenz invited Eric Bigelow, member of technical staff at Goodfire, after a literature review that kept returning one name. The upstream conversation was with Bronson Schoen of Apollo Research, who may have read more chain of thought in public than anyone else alive. Schoen's punchline was grim for anyone whose safety plan is CoT monitoring: even after the full trace, it is often not obvious why the model chose what it chose, and you could not have predicted it. Labenz asked the AIs how the next token gets selected. Bigelow was the through line.

Bigelow had just finished a Harvard psychology PhD titled *Towards a Cognitive Science of Large Language Models*. At his defense, his adviser joked that if someone had told him that would be the title, he would have asked what a large language model was. The work that most shapes how Bigelow thinks about decisions is the 2024 paper *Forking Paths in Neural Text Generation*, run on a GPT-3.5-class model. Give a question and a chain-of-thought prompt, then at every token in the reasoning, resample about 30 full completions and record which final answer each rollout lands on.

Arranged as a multivariate time series, the result is not a smooth slide toward certainty. It looks more like the grokking literature that was appearing at the same time: sharp phase transitions. The model sits at roughly 50% for one answer and 30% for another, then a token hits and the distribution collapses. Sometimes that token is the first mention of the final answer, as a judge might guess. Sometimes it is an arbitrary character. One example in the paper is the open parenthesis in "kilowatt hours (kWh)". Resample at that position with anything other than the parenthesis, and the final answer changes.

Bigelow's foundation is that what the model will say at each moment is determined by a coin flip: which token comes out of the distribution. He misses the old OpenAI interface that showed log probabilities on hover, because it made the branching obvious. Self-consistency does the rest. If an early token says the year is 2024 instead of 2026, a question about the current head of state goes somewhere else. In cases with a ground truth, that is a hallucination. In research, he argues, you often *want* the tree of alternatives: what would have followed if the model had assumed A instead of B.

Reasoning, in this view, is in-context learning from tokens the model has already generated. Sample a word, treat it as context, continue in a way that is consistent with it. People who misspeak "Billy fell into a hole" as "Billy fell into a spaceship" correct themselves and finish the original story. An LLM will often commit and start telling a spaceship story. The contrast looks sharp until you remember the psychology of rationalization: experimenters can swap the option a subject chose, and people still produce a fluent explanation of the choice they did not make. Forking is not only a model pathology.

He is careful with the word *decides*. Modern reasoning models also change the picture. They often run a linearized search, enumerating many possibilities in context and then going back to pick one. Forking-path analysis on those models looks smoother, but branch points remain: after listing options that all look similar, the model still says "therefore the answer is B," and a different token would have named C. Interpretability of in-context learning and reasoning is, in his view, badly neglected relative to how defining those processes are for everything LLMs do at a high level.

The phrase in-context learning, Bigelow says, has been narrowed to few-shot input-output examples and scaling curves as you add shots. He prefers the frame in Andrew Lampinen and coauthors' *The Broader Spectrum of In-Context Learning*: everything the model does to adapt its behavior that is not weight learning. Representations change as it reads each new word, as it generates reasoning, over a conversation, even within a sentence. That view rehabilitates prompt engineering, which the scientific community has treated as a dirty pile of hacks. "You are a brilliant software engineer" may get absorbed into post-training, but prompting will not disappear. The same weights have to serve countless contexts. You cannot fine-tune a model so that, the first time it reads a new story, it already knows who the good guys are.

He likes Kurt Vonnegut's lecture on the shape of stories. Cinderella starts miserable, spikes at the ball, drops when the magic fails, then ends higher when the prince finds her. Stories have shapes, and so does every context you give a model. Aggregate learning curves look smooth; individual problems can jump. He quotes Neel Nanda: phase transitions are everywhere if you look closely enough. Beliefs can rise, fall, and twist after a single sentence. He started *Forking Paths* by asking for a story about a boy named John and found recurring structure under the variety: half the stories involve a dog, many find a pond or waterfall, sometimes the dog saves the day. Understanding one prompt, he argues, requires many resampled outputs, not a single pretty trace.

Labenz asks what that implies for continual learning: will per-user fine-tunes break today's interpretability tools? Bigelow says it depends how much the representation space moves. Colleagues at Goodfire have found low-dimensional manifolds that organize concepts. If that structure holds, probes may need a little fine-tuning per personalized model and will still transfer. The scary case is catastrophic forgetting that rearranges internals so thoroughly the model also breaks. His near-term bet is a bitter lesson: harnesses, memories, and context engineering will beat custom weights. Memory is a pile of tokens stuffed into context, and tokens are readable. Anthropic has never offered retail fine-tuning; OpenAI killed its product. The last year of agents, he says, has mostly been putting more stuff in context.

The underlying beasts have also changed. Temperature has vanished from the big APIs; Bigelow suspects the labs are hiding sampling knobs to make distillation harder, not running a wholly new scheme. Output diversity shrinks with supervised fine-tuning, instruction tuning, and RLHF as models are sharpened into chatbots. RL then shoves the distribution somewhere that is no longer human text. Heavily RLed open models can be "cooked": two-thirds of Qwen3 35B A3B's stories were about a clockmaker. Reasoning models dodge the old inability to backtrack by enumerating a soup of options and skimming one out later. DeepSeek-R1 saying "wait" fifty times in a chain is a lot of epiphanies. A lot of the raw capability, he thinks, was already in the pretrained internet-completion blob; post-training evokes and polishes it. Coding agents are a distribution shift toward git, diffs, and tests, not a new soul.

Choosing a model, Bigelow still thinks about GPUs even at a well-resourced interpretability company. He will not casually occupy hundreds of them on Kimi K3 unless it lines up with company goals or spare capacity. Below about a billion parameters the systems are qualitatively different. Training toy transformers from scratch teaches a lot about learning dynamics, but his personal standard is to study models that already show elaborate, human-like behavior zero-shot, without teaching a facsimile. Seven to eight billion parameters is the sweet spot. His story-belief paper used 8B models and found them surprisingly sensitive as they tracked emotions and other attributes while reading.

He has been running a lot of work on Qwen3 35B A3B, an MoE with 3B active parameters. Qwen likes to think in Chinese: a logit lens on an English conversation often shows Chinese characters in the middle before English tokens come out. He chats with models about consciousness as a sniff test for nuance. This one impressed him. He treats open and closed models as largely standing in for each other on core capabilities, because they are all trained on the internet; the remaining moat is post-training that makes coding and other product behaviors much better. Distillation from proprietary models is speculated as one reason they can feel similar.

Kimi K3 is a different coordinate. It is a step function: the first open model that is good enough at coding that reward-hacking behavior also shows up naturally, the kind of thing colleagues at Goodfire cannot get out of smaller models, or get only by a different mechanism. If you only want to study a behavior in the abstract, a smaller model can suffice. If you want the dangerous behaviors that emerge with scale, you need a model that has crossed the threshold, or you are studying a lookalike. Labenz asks the blunt question: is interpretability outside the proprietary labs now dependent on Chinese models?

"I think for now, we do," Bigelow says. At the biggest scale there is a real gap in programming ability between Kimi K3 and everything else, but it is a gap American companies could close with more open-source investment. Drop a tier or two and the gap shrinks. If he were barred from Qwen 35B and had to use Gemma instead, he would slow down a bit, not stop. A blanket restriction on open-source models, regardless of origin, is the scenario he actually fears. Foundational interpretability on older LLMs could continue for a long time. Reward hacking and other sketchy, scale-emergent misalignment behaviors would be delayed, because those only appear once capability crosses a point.

Labenz has been surprised how productive it is to anthropomorphize models and wants a philosophy for using it carefully. Bigelow started his PhD before LLMs existed; when they arrived it was a drop-everything moment, and he took pushback from colleagues used to deep-learning shortcuts. Psychologists, linguists, and philosophers still flinch when AI papers use words with long theoretical histories — planning, belief, knowledge — as if next-word prediction were a dirty substitute for what humans do. Success has softened some of that resistance. He would rather critics walk in and help define the terms than stand on the sideline calling it bad psychology.

He splits anthropomorphism into two layers. The first is for yourself. Cognitive science talks about intuitive theories: evolved frameworks, higher than raw perception, that make it easy to reason about agents. If the car makes a clunk, you say it is trying to tell you something. You need not believe it; the frame is useful. The second layer is the one to take seriously. LLMs are not just clunking. They speak, do math, do social reasoning, write code, and help with science. Then "belief," "decision," and "intention" cannot be used casually, and they also cannot be banned. The chance is unusual: intervening on human brains is expensive and ethically constrained; you can cut into an LLM.

His working picture of belief is Bayesian cognitive modeling. The model holds latent concepts with prior weights; context reweights them by posterior inference; behavior is generated from the concepts that fire. That story is how humans adapt on the fly, and he thinks it applies to LLMs too. The known weakness of Bayesian cognitive science is that it can fit behavior without committing to an algorithm. With people you can stay agnostic because brains are hard to measure. With LLMs there is no excuse. Decoding full answer distributions from hidden states has been difficult so far. It may be a data problem — the sampling is expensive — or the model may not represent the whole chain. It might look only a few steps ahead, or hold a high-level "shape of the proof," the way a mathematician sees the scaffolding before the derivation.

Uncertainty in the forking-paths paper is an operational word, and he wants readers to feel how strange it is. By the end of a proof-like chain the model is almost 100% sure, because the previous tokens have locked the conclusion, right or wrong. Saying the model is 50/50 at a midpoint means that if you resample from there, half the rollouts hit A. It does not mean the network is storing those fifty paths. Recent work can decode upcoming moves well enough to guide sampling, including some signal about final outcomes. He still doubts the model knows the full story it might tell before it tells it.

Labenz finds token-level cliffs unsettling, especially on a parenthesis, and wonders whether latent, looped models such as Astra will land the plane more gradually. Bigelow is not bothered in the way first-time listeners are. If you want output diversity *and* self-consistent chains that are not nonsense, forking may be an inevitable byproduct. What he wants is an interface that surfaces it. He still likes log-prob hover, and he likes Loom, an experimental UI by someone named Janus that influenced this work. Whether the turn is sharp or gradual, the user should see that another path was live. He finds today's chat box uninspired, still Eliza: text in, text out, a bash terminal when it could be a GUI.

Phase changes happen at the beginning, middle, and end of rollouts; each input has its own story. Unexpected forks on a parenthesis look like internet statistical bias — kilowatt hours really does get "(kWh)" — and he would like training to push forks onto tokens people can interpret. Late collapses are common when the model simply does not know. On the old last-letter concatenation task, some chains even write Python, then at "the answer is" they are sampling letters from the words on the fly. Estimating that uncertainty by brute-force resampling is expensive, which is why *Forking Fast* fits a statistical model that is smooth in places and jumpy at key points. Hidden-state probes would be the efficient version, if they can be made to work.

Should the stochastic-parrot meme keep a little room? Bigelow wants it retired as a shallow Chinese-room shrug. Whatever is in the book that lets you hold a conversation is interesting, and a pile of work, including Othello-GPT, shows structured world models; at some point a world model is cheaper than a lookup table. Stochasticity, though, is not dead. If you want diversity and you do not want the model to halt whenever it is not 100% sure, you need a coin flip and a path. He has been running forking-path analysis on open looped transformers in the Oro series. After latent reasoning, the token stream is still uncertain and still jumps at particular tokens. He does not want a world where the same input always yields the same output. That would be mode collapse in the extreme. Latent chain of thought still ends in sampling, but it may change the picture in dramatic ways he is still wrapping his head around.

His view of CoT monitoring has flipped in about a year, from bullish to cautious. Outcome-level RL has no incentive to stay human-readable. English was not the same three hundred years ago; only something like the Académie Française tries to freeze a language, and RL runs on a much faster clock. Goodhart is the other blade: optimize for monitorable thoughts and the model learns steps that *look* sensible. The stolen-chain-of-thought paper shows GPT models discussing musicals in a way that reads like spy talk. Astra's dialect with sub-agents is scientifically fascinating; people have counted tokens and found that plain English would have been shorter, so a length penalty is a weak explanation. He would like that language to stay in tokens, where it can be studied. There is no forcing function that it will, and adding one may make the words stop meaning what they appear to mean.

Labenz notes that we are rushing to hand proximal control of important systems to AIs we instruct in language, without a great account of how the distribution at a critical token gets set. Bigelow's answer is more interpretability research, and he means the whole spectrum including evaluations. Trillion-dollar companies train the models; AI as a whole is many trillions. He works at what may be the only substantial interpretability company valued over a billion dollars, which is a lot until you put it next to the other side of the ledger. Academia, already underfunded, is getting harder in the United States. The resampling work is behavioral; looking inside is how you see what is not obvious and how you do it efficiently enough to run at scale, on every person's machine, every time they talk to a model.

Tips for Goodfire's Silico research agent apply to other agents too. Do not hand over an entire open-ended project and let it drown in pretty plots and half-formed theories. Give hand-holding. Every time you say something is wrong, say why, what you want next, and what the bigger goal is. He compares an agent waking up in a swarm to a person dropped into a library with no idea where they are. *Forking Fast* was done entirely by Silico in a week, including side quests that went nowhere. The click was his: after dense sampling showed smooth curves, he thought of fitting a statistical model to skip a lot of rollouts. The agent did not propose that. A colleague's line is that as agent output starts looking good, research taste is worth its weight in gold. Using Silico feels like working with a hardworking first- or second-year PhD student who does not always see the bigger picture.

Over lunch at Goodfire, two themes have been loud since summer. One is agents doing interpretability, including automating the almost boring normal-science experiments that still paint the bigger picture, because that may be the only way to keep up. The other is alignment, given extra urgency by what he calls the Hugging Face incident: a sign that the time is right, not a topic to wait on. They argue about whether values can be universal, how to tell reward hacking from following the letter of a misspecified rule, and what it would even mean for a model to have ill intent. They also talk about open-model bans, latent chain of thought, the stolen-CoT paper, and Astra's pseudo-language with sub-agents. If investment and methods do not jump, he worries it may take more than one warning shot. Understanding AI, he says, is among the most important and most interesting jobs available.