Latent Reasoning and Recurrent Depth: Scaling Compute Before the Next Token

A new research paper explores language models that perform additional computation in latent space before predicting a token. Many reasoning systems scale test-time compute by generating intermediate text; here the model can instead repeat a learned internal block. The model still reads input tokens and generates output tokens, and its internal iterations consume computation.

I connect this to Yann LeCun’s arguments about the limits of language models. In a Lex Fridman interview, he argues that planning requires a predictive model of the world and uses examples of an object falling or moving when pushed. These are his views on the requirements for human-like planning, not a settled verdict on every language model.

LeCun’s example is that people can anticipate the results of actions and plan a sequence toward an objective. I interpret his argument as requiring more than what language alone describes. I also note that the interview predates the widespread adoption of the recent generation of test-time reasoning models.

I find written chain-of-thought steps persuasive as an illustration of problem solving, while LeCun warns that fluent language can make a system seem more generally intelligent than it is. I acknowledge that verbal reasoning could still amount to language manipulation. A displayed rationale is not necessarily a complete or faithful account of a model’s internal computation.

In another excerpt LeCun expresses skepticism about generative models learning adequate representations of the real world. I contrast this with my impression of the industry’s optimism about AGI and advanced reasoning. Those are contested forecasts and positions; the interview does not establish that all major companies believe language models alone are sufficient.

The paper is ‘Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach.’ Its authors repeatedly apply a shared recurrent block to a latent state, allowing variable computational depth before output prediction. I wonder whether this might address some of LeCun’s concerns, but the paper does not demonstrate a learned physical world model or settle his argument.

The researchers report a 3.5-billion-parameter proof-of-concept model trained on 800 billion tokens. They describe varying recurrence at test time instead of necessarily producing a long verbal chain. Although the architecture allows different numbers of iterations, running indefinitely is not practical, and the experiments do not show that more iterations always improve answers.

The authors argue their approach needs no specialized chain-of-thought training data, can operate with smaller context windows and may represent computations not easily expressed in words. This does not mean the model needs no training data: it was pretrained on a large token corpus. Memory, compute and latency comparisons depend on implementation and workload.

I compare latent computation with the human experience of forming an idea before speaking, including people who report little or no inner monologue. It is an analogy for thinking without a verbal transcript, not evidence that the model’s internal states work like human thought or have subjective experience.

The paper contrasts its method with approaches that train models to verbalize intermediate steps. I raise a question about smaller reinforcement-learning demonstrations that elicit reasoning behavior without extensive worked reasoning examples. Training methods differ, so a claim about this architecture’s data requirements should not be read as a rule that every chain-of-thought system requires the same dataset.

The authors suggest that forcing all intermediate computation through a textual next token may be limiting, and point to prior recurrent neural networks, diffusion models and looped transformers. The idea of recurring computation is not new; the contribution here is a particular architecture and its scaling experiment, not the invention of internal model states.

The reported model can spend more inference compute by iterating its recurrent block. The authors compare it with selected open models that have different parameter counts and training data. Performance comparisons are task- and compute-dependent; the paper does not establish that this method is universally more efficient or better than larger models.

The paper discusses potential memory and accelerator communication advantages from a smaller parameter set reused for additional operations. I explain this as doing more work per parameter, but it does not imply that all workloads fit on one GPU or eliminate multi-device communication. Additional recurrent steps still increase computation, and their actual cost depends on deployment.

The authors hope a compute-heavy architecture may encourage general strategies rather than simple memorization. I relate that hope to the debate about whether models generalize beyond training examples. Both memorization and generalization can occur; this experiment does not resolve the debate or demonstrate artificial general intelligence.

The architecture diagram starts with an input such as ‘Hello’: a prelude maps it into a representation, the shared recurrent block updates a state repeatedly, and a final block predicts the next token, illustrated as ‘World.’ The iteration happens before that prediction, and analogous computation can occur at later token positions as well.

The displayed results compare performance against recurrence counts on HellaSwag, GSM8K and HumanEval. Several curves improve as tested iterations increase, but gains flatten in parts of the plot. These benchmark results support a finite-range empirical finding, not proof that unlimited iterations or every extra step improve the final answer.

I also point to results from checkpoints trained on increasing amounts of data, up to 800 billion tokens. The paper’s results show how this particular model changes with training and inference compute; they should not be treated as a guarantee of the same scaling behavior for every model or dataset.

Figure 10 shows adaptive per-token exits for questions from MMLU categories including high-school mathematics, philosophy, logical fallacies and moral scenarios. The researchers stop when the difference between successive predictions falls below a fixed KL-divergence threshold, and plot the distribution of iterations. This measures convergence to the chosen threshold, not how many steps were needed to reach a correct answer or a general ranking of task difficulty.

Latent computation need not exclude a verbal chain of thought: the two approaches could be combined, just as I might think, write down an intermediate result and revisit it. That is a useful analogy and a research direction, not evidence the model reasons like a person. I close by noting that the authors released the proof-of-concept model for others to try.