Nathan Lambert on Instruction Tuning, Reward Models and Rejection Sampling

This lecture connects instruction fine-tuning, reward modeling and rejection sampling. They form a useful sequence: teach a model to respond, learn a preference signal, then select and train on high-scoring completions. The lecture concentrates especially on reward models.

Instruction fine-tuning turns a base model’s text continuation into a useful instruction–response format. Modern training stages overlap, but this abstraction helps explain how a model learns to behave as an assistant or agent.

Earlier work moved from task-specific fine-tuning toward text-to-text multitask learning and instruction datasets such as FLAN and T0. After ChatGPT, Alpaca, OpenAssistant and Tülu helped popularize open instruction-tuning data and assistant models.

A chat template structures system, user and assistant messages with model-specific markers. It tells the model who is speaking, where its response begins and when it should stop. These details turn ordinary message objects into the token sequence the model expects.

A prompt can contain several alternating user and assistant turns. Training data additionally contains the assistant response the model should learn. Tokenizer tools apply the appropriate template to convert the structured messages into model-specific text.

Templates often use Jinja and vary across model families. Small mistakes in special tokens, newlines, system messages or tool formatting can break a training or inference pipeline. Reasoning modes and tool use add further complexity.

Lambert mentions newer efforts to make message formats easier to work with but says adoption of alternatives is slow. The underlying need is consistent handling of roles, tokens and model-specific formatting across training and inference.

In instruction tuning, prompt tokens are commonly masked from the loss so the model learns to predict assistant responses. For multi-turn conversations, a team may train only on the final assistant answer or on every assistant turn.

High-quality completions matter, but their value depends on the prompts. Lambert calls prompts a fundamental unit of post-training because they determine the task distribution. Good answers to the wrong prompts will not teach the desired capabilities.

Instruction tuning generally uses smaller learning rates and batches than pretraining while retaining a next-token loss. The narrower distribution makes overfitting and loss of generalization important concerns.

Lambert describes the move from early human-authored datasets of roughly ten thousand examples toward much larger, heavily filtered synthetic datasets. Reasoning traces changed the comparison: token counts can rise dramatically even when prompt counts do not.

A million prompts is not a magic number. A broad model may need data for many domains, and the right mix depends on the tasks. Prompts can be reused across training stages, but each stage can also require different examples.

Self-Instruct begins with strong seed prompts, asks a language model to create variations and then generates answers. This helps scale similar data, while obtaining high-quality prompts for specialized real-world work remains difficult.

For a practical SFT recipe, first define evaluation goals and mix relevant existing datasets as a baseline. Then identify weak skills, create targeted data, remix and reevaluate. This iterative development may require many training runs even if a final fine-tune is relatively inexpensive.

Reward-model training is an industrial craft that is not fully captured in public research. Lambert says work such as RewardBench 2 improved understanding but did not completely resolve how to identify the best reward model for downstream training.

In a simplified RLHF loop, the reward model provides a learned signal for a generated completion. Because the score is an imperfect proxy, optimization is commonly constrained with a reference-model KL penalty to reduce over-optimization.

Preference data commonly consists of two completions for the same prompt, one chosen over the other. A reward model learns to give the preferred answer a higher score across many such comparisons.

The Bradley–Terry model represents pairwise preference probabilities using latent strengths. Reparameterizing those strengths with scalar rewards produces a sigmoid of the difference between the two scores.

The learned reward function scores a chosen and a rejected completion conditioned on the same prompt. Training maximizes the observed preference likelihood, equivalently minimizing a negative log-sigmoid loss on their score difference.

The loss must be averaged over the dataset of preference examples. Lambert later corrected an earlier slide that began from a single-example maximization without making that dataset-level expectation explicit.

A typical pairwise reward model attaches a scalar head to a language-model backbone and reads a score at the end of each completion. Training compares a chosen and rejected answer; at inference the model can score an individual completion.

Variants can use preference strength as a margin or take multiple comparisons from one prompt. Plackett–Luce models a ranking among several answers and reduces to the two-item Bradley–Terry form when there are two choices.

Lambert distinguishes the outcome reward model formulation he describes from a pairwise classifier merely trained on correctness labels. In that formulation, the model predicts the probability of a correct final outcome at each token, using an outcome label repeated across the completion.

An outcome model’s per-token predictions can be inspected for changes after an apparent reasoning error. This resembles a value model, but the latter predicts future return rather than simply the probability of a correct outcome.

A process reward model scores intermediate reasoning steps. In the example discussed, labels distinguish good, neutral and bad steps, and the loss is applied at step boundaries. At inference the model estimates the quality of the intermediate process.

A generative language-model judge can also provide feedback in text. Lambert finds such judges highly useful for synthetic data filtering, while noting that specialized classifiers can outperform generative judges on some reward-model benchmarks.

He helped build RewardBench but cautions that reward-model benchmark performance is not yet reliably connected to usefulness in a full RLHF training run. Better evaluation remains an open problem.

Rejection sampling generates multiple answers, scores them with a reward model, keeps high-scoring completions and fine-tunes the original model on them. It is an offline preference-training baseline used in several prominent model recipes.

One selection strategy keeps the best answer for each prompt, preserving the prompt distribution. Another selects the highest-scoring answers across the whole dataset, which can change which prompts are represented. Lambert considers the practical effects underexplored.

Generation settings must provide enough diversity for selection to be meaningful, which can require substantial inference compute. For batched reward-model scoring, sorting completions by length can reduce padding when the inference system does not handle it automatically.

Best-of-n uses the same basic generate-and-score idea at inference time but does not fine-tune on the selected answer. It offers a simple inference-time baseline for comparison with RL. Lambert closes by calling for more reproducible research on rejection sampling.