Nathan Lambert on Synthetic Data and On-Policy Distillation

This lecture traces synthetic data into modern post-training and explains the mathematics of on-policy distillation, along with AI feedback, Constitutional AI and rubrics. I want the history to make the methods understandable on their own, not just as chapters in the book.

When I began writing the RLHF book, there was serious debate over how much synthetic data could do. It made iteration much cheaper and faster than waiting for human signals. Stronger models in the GPT-4 era and increasingly capable open-weight models widened the available set of teachers.

Synthetic data broadly means model outputs used as training signals. In my account it now appears throughout leading-model training, while humans still provide important signals at the frontier. The balance between human and synthetic data is one of the lecture’s recurring questions.

Synthetic data can expand a small prompt set, generate supervised completions, provide preference labels for reward models, or supply a judge as a reward signal. Self-Instruct explored synthetic instructions; UltraFeedback became an influential AI-generated preference dataset.

Earlier work could generate text from a strong API model and train on the completions. Reasoning models make the picture more complicated because useful reasoning traces may not be exposed in the same way. Filtering can remove repetitive or low-quality generations before training.

Human-written completions once dominated instruction data. Lambert says industry still uses human preferences, even though academic studies have sometimes struggled to show large gains on standard evaluations. Many evaluations themselves now use model-generated judgments, while human evaluation remains valuable.

There was also an active debate over whether repeatedly training on generated data would cause models to collapse. Lambert argues that modern practice demonstrates the usefulness of synthetic data and points to filtering as one practical safeguard; his criticism reflects his interpretation of that earlier debate.

Early synthetic instruction datasets were small by later standards: an example from 2023 might contain tens of thousands of prompts and millions of tokens. As reasoning models emerged, datasets and the diversity of generated prompts expanded substantially.

Distillation has two meanings. In technical knowledge distillation, a student learns to match a teacher’s output distribution, using soft probabilities. In common post-training usage, it can simply mean training a weaker model on completions generated by another model.

The technical method compares teacher and student log probabilities, often with a KL-based loss; it requires more information and machinery than supervised training on generated text. The broader everyday meaning requires only prompts and teacher-generated responses.

The original teacher–student work predates modern language-model post-training. Instead of training solely against a single next-token label, knowledge distillation can convey information about the teacher’s distribution across possible tokens.

For notation, consider a prompt or state, actions produced by the student, and a separate symbol for teacher outputs. Let the teacher and student each define a probability distribution. Keeping the source of each sample clear becomes critical when we switch from offline to on-policy training.

Kim and Rush’s 2016 sequence-level work considered a word-level baseline in which the student matches teacher probabilities on an existing text corpus. The teacher can also generate completions from prompts, creating sequences on which the student is trained.

Enumerating every possible completion is intractable. The sequence-level approach instead generates a teacher completion using beam search and trains the student on that selected sequence. This resembles the later practice of learning from model-generated answers.

Cross-entropy connects these approaches. The cross-entropy between teacher and student equals the teacher’s entropy plus the forward KL divergence from teacher to student. When the teacher is fixed, minimizing cross-entropy with respect to the student also minimizes that forward KL.

Offline distillation can suffer exposure bias: the student trains on teacher-generated trajectories but generates its own at use time. An early unfamiliar token can move the student into contexts missing from the teacher-generated training distribution, allowing errors to compound.

Imitation learning offers an analogy. DAgger studies a learner’s own trajectories with expert labels for actions. Under its assumptions, ordinary supervised imitation can have accumulated error that grows quadratically with trajectory length. Lambert cautions that the result does not directly prove a bound for language models.

On-policy distillation addresses the mismatch by sampling from the student and using the teacher to supervise states the student actually visits. In the imitation-learning analogy, this can improve how accumulated error scales, but the language-model case is more complicated.

Two early contributions were MiniLLM, which used an RL-style policy-gradient approach, and generalized knowledge distillation. Lambert says current implementations in some model families are closer to the latter framing, while MiniLLM helped introduce the RL formulation.

In the on-policy objective, the student generates actions and the teacher is held fixed. The resulting formulation can use reverse KL, with the student distribution supplying samples. The direction matters because KL divergence is asymmetric.

Forward KL weights examples under the teacher distribution and tends to encourage covering the teacher’s probability mass. Reverse KL is weighted under the student’s distribution and is often described as mode-seeking. These are useful intuitions, not interchangeable names for the same objective.

One RL formulation uses the negative per-token log-ratio estimate as an advantage: teacher log probability minus student log probability for a sampled token. In a later Q&A Lambert corrects the slide’s equality: a single sampled term estimates the KL and equals the full divergence only in expectation.

When the teacher assigns a sampled token higher probability than the student, the resulting advantage is positive and encourages that action. This supplies dense token-level supervision, unlike a single sparse outcome reward. Lambert notes interest in combining it with verifiable outcome rewards.

Multi-teacher on-policy distillation selects among teachers, for example by prompt domain. Lambert describes this as increasingly visible in open-weight models and points to the practical challenge of fitting a suitable teacher to each type of task.

On-policy self-distillation seeks the same dense supervision without a separate stronger teacher. A model produces a trajectory, and a modified version with privileged context acts as the teacher. The student learns from the difference so the extra hint need not be present at inference time.

Lambert describes a Cursor example in which a judge identifies a common error and inserts a hint into the trajectory. Training uses the hinted context to supervise subsequent tokens. He expects the approach to require a sufficiently capable base model and careful control of the task distribution.

Token-level distillation also creates a practical compatibility issue: teacher and student generally need compatible tokenizers for per-token comparisons. This creates more dependence on model lineage than simply training on another model’s text.

To compare the methods: sequence distillation starts with a teacher rollout; on-policy distillation starts with a student rollout and teacher scoring; self-distillation adds privileged context to a student trajectory and learns from the resulting signal.

AI feedback changed the economics of collecting preferences. Lambert contrasts human annotations that can cost dollars or more per prompt with model feedback measured in cents, making open research and iteration easier.

He offers an intuition from industry discussions: human labels can have high noise but lower systematic bias, whereas a model judge can be more consistent while sharing the biases of its model family. This is a heuristic for thinking about data, not a universal measurement.

The optimal mix remains unsettled. Human preferences are expensive and difficult to collect well, but Lambert expects teams still use them to connect experimental systems to real user needs. Synthetic preferences are much easier to start with.

Specialized judge models seek cheaper or better feedback, but strong general models also perform well as judges. Model judges have systematic biases: for example, a judge may favor outputs from its own model family. These issues matter in both training and evaluation.

Constitutional AI, introduced by Anthropic in 2022, is one approach within the broader family of reinforcement learning from AI feedback. It uses a set of principles to generate critiques, revise answers and select preferences.

The original pipeline has a supervised stage that critiques and revises responses against principles, then fine-tunes on revised answers. A preference stage compares answers using principles, trains a preference model and uses that signal for reinforcement learning.

A principle might ask whether a response encourages violence or whether it is truthful. Sampling principles across examples shapes the synthetic data mix. Lambert notes that the usefulness of a separate SFT revision step may differ when the starting data is already synthetic.

Rubrics extend the idea of explicit criteria to prompts without a single verifiable answer. A rubric provides a checklist or domain-specific grading criteria, potentially combining hard requirements with softer quality judgments.

A student produces an answer, then a separate judge evaluates it against the rubric. The score can become an RL reward. Generating a rubric for each prompt adds another synthetic-data step, so the pipeline may contain models producing criteria for other models to apply.

Lambert says domain experts can help design rubrics and stresses attention to the generated criteria and outputs. Rubrics have been used for instruction following, deep research and long-form tasks where a simple verifier is unavailable.

Synthetic data can therefore be the optimization signal itself, as in on-policy distillation, or expand existing training pipelines through AI feedback, Constitutional AI and rubrics. The next lecture turns to the history and practical collection of human preferences.