Nathan Lambert on the Rise of Reasoning Models

This lecture is a historical map of reasoning models, reinforcement learning with verifiable rewards and inference-time scaling. It builds on earlier lessons about reinforcement learning and asks how the post-training methods and models developed.

Yann LeCun’s cake metaphor placed self-supervised learning in the large base, supervised learning in the icing and reinforcement learning in the cherry. Lambert sees a useful historical parallel in pretraining, supervised fine-tuning and the RL stages that built on them.

RLHF commonly uses learned models to score subjective preferences, whereas RLVR uses rewards that can be checked, such as an answer key or code tests. More precise signals enabled large runs on math and code, though practical pipelines mix multiple methods.

In an RLVR loop, the model attempts a problem and receives feedback from a verifier. For tool-using agents, the verifier increasingly resembles a richer environment. Lambert expects these more complex interactions to create opportunities and implementation challenges.

Training-time and inference-time compute are distinct. Longer RL training can improve a policy; at inference, generating a longer reasoning trace can improve performance on some tasks. Lambert emphasizes that the effects are connected but can also be studied independently.

A reasoning model may generate an intermediate chain of thought before its final response. Open-weight models can expose traces that look repetitive or include checking and revision. Such traces provide a way to inspect how additional inference computation is used.

Pass@1 measures performance with one sampled attempt; pass@k asks whether at least one of k attempts succeeds. Estimates can use more sampled attempts and combinatorics. Giving a model more attempts generally increases the chance that at least one is correct.

Lambert highlights DAPO, with modifications to GRPO-style training, and MiniMax’s CISPO as examples of algorithmic changes that influenced subsequent work. He then turns to the rapid succession of reasoning-model releases in 2025.

OpenAI’s o1 appeared in 2024, followed by DeepSeek R1 and many others in 2025, including Kimi, Qwen and research replications. Lambert calls this a rapid expansion of the field, with later models pushing toward tool use and agents.

Pre-o1 work already explored related ideas. STaR and Quiet-STaR sampled reasoning traces and retained successful examples; tree-search work explored reasoning steps; VinePPO used rollouts to estimate training signals. These were different methods with useful connections to later RLVR.

Tülu 3 applied RL with PPO and math correctness to a general model while retaining broader abilities. Lambert says the team coined the term RLVR for this approach. The model itself was not a reasoning model, illustrating that several research threads converged.

Better infrastructure made the transition practical. By 2025, open training frameworks, stronger base models and higher-quality math and code data supported experiments that had been much harder in 2023. Objective evaluations also offered a clearer signal than some easily gamed chat benchmarks.

DeepSeek R1 described a long RL recipe and showed reasoning performance and response length growing during training. Lambert considers the release a catalyst for open research and says its recipe remained influential until later distillation methods gained ground.

Kimi 1.5 appeared on the same day as R1 but attracted less attention in his telling. Its report discussed filtering by difficulty, scheduling and gradually extending the reasoning length, techniques that remained useful.

Open-Reasoner-Zero provided an early public replication with training data. It used PPO, reminding Lambert that GRPO is not the only possible algorithm. He expects value-model approaches to remain interesting for complex, multi-turn tool use.

Qwen 3 made stronger base models broadly available and offered a switchable thinking mode. Lambert describes experimentation with separate thinking and instruct models, followed by models designed to cover a range of reasoning effort. Small releases also made distillation techniques easier to study.

Xiaomi’s early MiMo release shared base, supervised and RL checkpoints, allowing researchers to compare stages. OpenThoughts 3 was an influential reasoning-data release; Lambert says OLMo 3 built extensively on it and calls for renewed work on foundational open datasets.

MiniMax-M1 highlighted a practical asynchronous-RL issue: inference and training systems may compute slightly different token probabilities. Using FP32 for the language-model head reduced the discrepancy. Its CISPO algorithm also became an influential reference in later work.

Lambert sees GLM 4.5 as an early example of a shift in messaging from math and coding benchmarks toward agents and tool use. He contrasts that shift with the difficulty of releasing a fully open model recipe on a competitive schedule.

For OLMo 3.1 Think, he describes a multi-domain RL run using about 200 GPUs for roughly a month, with checkpoints and data released for study. He hopes similar fully open work will cover tool-use reasoning.

Later releases included DeepSeek V3.2 and NVIDIA’s Nemotron 3 Nano, which shared data, environments and checkpoints. Lambert highlights efficient long-context architectures and continued open releases as part of the developing landscape.

Lambert describes Xiaomi MiMo-V2-Flash as an example of multiple specialist teachers being distilled into one general model. Separate teams can develop domain specialists while a shared recipe combines them. He notes that other major model reports also describe multi-teacher approaches.

He briefly mentions further 2026 releases and sees long-horizon tool use as the central direction. Looking across reports, recurring techniques are informative, but model development is path dependent: a trick used by several teams is not guaranteed to work for every model.

Offline difficulty filtering samples answers from an initial model and removes prompts that are too easy or too hard. For group-relative training, groups with identical outcomes can yield no useful relative gradient, so filtering can save expensive training compute.

Online filtering and curricula revisit difficulty as the model changes. OLMo 3’s active sampling kept groups with nonzero gradient, replacing groups where all sampled answers received the same reward. The goal is to spend training on examples that produce a learning signal.

Early RLVR work often removed the KL penalty used in RLHF because verifiable rewards seemed less vulnerable to the same over-optimization. Lambert says very long runs may make a grounding penalty useful again; the appropriate choice depends on the training setting.

Other implementation choices include asymmetric clipping, CISPO and how advantages and losses are normalized. Per-token versus per-sequence aggregation can change length bias and exploration, so details in the code affect model behavior.

Auxiliary rewards can enforce an extractable answer format or encourage language consistency. Progressive length extension starts with a shorter thinking budget and increases it later, encouraging the model to solve easier tasks concisely before using longer traces.

Reports also investigate whether text-only RL transfers to multimodal tasks and how mid-training affects later RL. Lambert wants more public evidence about those relationships, since the stages of a model recipe interact.

Open questions include whether agent RL teaches new capabilities or chiefly elicits existing ones, how RL compute will scale relative to pretraining, and which recipes will work for complex environments. Lambert argues RL can teach new skills but treats the longer-term trajectory as unresolved.

Building models requires reading research alongside practical experimentation. Lambert points to contemporary open RL frameworks and notes that tool-use and asynchronous training have shifted the libraries researchers favor. He closes by inviting questions for future course videos.