Nathan Lambert: An Overview of Post-Training and RLHF

This opening lecture introduces language-model post-training and reinforcement learning from human feedback. Nathan Lambert describes how the field developed, the classic RLHF recipe and the shift toward reasoning models and scaled reinforcement learning.

An autoregressive language model predicts a probability distribution for the next token based on preceding tokens. Repeating that process produces text. The transformer popularized self-attention in a scalable architecture; modern systems also handle modalities beyond text.

Lambert traces a line from the 2017 transformer through GPT-1, ELMo and BERT, then GPT-2 and the growing study of scaling laws. GPT-3 drew attention to few-shot behavior: examples in a prompt could guide performance on a new task.

He recalls the 2021 stochastic-parrots debate about the costs and value of large models and gives his own critical view of that framing. ChatGPT’s 2022 release made RLHF especially visible as a way to make language models useful to users.

In Lambert’s account, 2023 brought larger-scale models; 2024 popularized reasoning models that spend more compute before answering; and 2025 accelerated interest in coding agents, search and tool use. He presents these as a broad historical sketch.

Pretraining uses next-token prediction across very large text collections. It builds fluency, knowledge and capabilities, but a base model often behaves like a text completer rather than a conversational assistant.

A base model given an unfinished statement may continue with web-like text. Instruction fine-tuning changes the expected interaction format so the model can respond to a user’s question. The post-trained model still inherits its base model’s learned representation.

Lambert separates instruction tuning, preference tuning and reinforcement learning with verifiable rewards. Instruction data can teach formats and specific behaviors; preference feedback optimizes qualities that are hard to specify in a single correct answer; verifiable rewards help train on checkable tasks such as math and code.

These methods can complement one another. Lambert finds instruction tuning a straightforward way to teach a known target distribution and sees RL as a powerful response-level optimizer. He also notes that preference training can produce undesirable traits such as sycophancy.

A classical reinforcement-learning problem is often described as a Markov decision process: an agent observes a state, takes an action, receives a reward and sees the environment transition. A policy aims to maximize cumulative reward while balancing exploration and exploitation.

A thermostat or balancing a cart-pole illustrates repeated control: an agent observes the environment, chooses an action and experiences new conditions. Language-model RL uses the same general ideas, although the setup and reward source differ.

Many real-world goals are hard to encode as a numerical reward. Human comparisons can offer a proxy: it may be easier to judge which poem or behavior is better than to write a perfect example or hand-design its reward.

Early preference-learning work used human feedback to train reward predictors for agent behavior, including a backflip example. Lambert also highlights 2019 work applying human preferences to language models, followed by WebGPT, InstructGPT and related research.

For language models, a common preference example contains a prompt and two responses, with a human choosing the better one. A model trained on such comparisons can score new completions, providing an imperfect proxy for human preferences.

Compared with a rich multi-step environment, a simple RLHF episode can consist of a prompt, a generated completion and a sequence-level reward-model score. A reference-policy KL penalty helps preserve the pretrained model’s useful behavior.

The classic InstructGPT recipe has three stages: collect demonstrations and train a supervised policy; collect comparisons and train a reward model; then optimize the policy against that reward model with RL.

In the supervised stage, examples teach desired responses and formats. Fine-tuning uses the next-token objective but typically different batch sizes and learning rates from pretraining because the dataset and target behavior are narrower.

For the reward model, labelers compare answers to a prompt. A pairwise preference model such as Bradley–Terry relates differences in scores to the probability of a preference. After training, a reward model can assign a scalar score to an individual completion.

In the RL stage, the policy generates answers and receives reward-model scores. A KL penalty compares the policy with a reference, often the supervised checkpoint, to discourage excessive change. The system can require several models and careful engineering.

DPO is another route: it derives a direct preference-training loss without separately training a reward model and running a conventional RL loop. Rejection sampling instead generates candidates, scores them, keeps the best and fine-tunes on the selected outputs.

A learned reward is only a proxy for user satisfaction. Optimizing it too aggressively can yield verbose, repetitive or sycophantic responses. Lambert views reward-model over-optimization as an important technical and product issue.

Training recipes grew from relatively small supervised and preference datasets into more complex multi-stage pipelines. Prompts remain central because they define which tasks the model encounters, but the number and arrangement of stages vary substantially.

Lambert argues that the field moved from a relatively recognizable RLHF recipe toward broader post-training: multiple rounds of supervised data, preference methods, RL and other tools. He considers RLHF one component of that larger process.

A recurring criticism says RLHF mostly changes style. Lambert disagrees that this makes it unimportant: clear presentation can itself be valuable, and he points to different post-training recipes producing different performance from the same base model. This is his interpretation of that debate.

He calls his view elicitation: a strong base model provides a broad foundation, while post-training attempts to bring useful behavior out of it. He acknowledges tension between that framing and the newer evidence that extensive RL can produce substantial additional improvements.

RLVR replaces a learned preference reward with a checkable signal for suitable tasks. Lambert says this helped make larger RL runs and inference-time scaling practical, with o1 and DeepSeek R1 as important milestones in his account.

Inference-time scaling spends more compute on a particular problem, for example through longer reasoning or parallel attempts. Training-time scaling spends more compute updating the model. These are separate axes, even though they can reinforce one another.

Lambert cites public reasoning and agent-training curves as examples of performance improving with more RL. In his OLMo 3.1 account, the team continued a run beyond its initial release and observed further gains, illustrating that post-training can itself be a major compute commitment.

The longer-term frontier includes agents, tool use and scaled RL, but the book focuses on enduring questions about human preferences, reward measurement and over-optimization. Lambert closes by pointing readers to course slides, references and future lectures.