Nathan Lambert Answers Questions on Teacher Models and PPO

ZUSAMMENFASSUNG

In 30 Sekunden

Nathan Lambert answers course questions about prerequisites, synthetic teachers, reward models and PPO. He distinguishes outcome rewards from token-level loss aggregation and explains why a minimized PPO loss uses a maximum.

1Choosing a teacherLambert reports that Qwen-era teachers have been difficult to beat in some recipes.
2Reward versus loss granularityAn outcome reward and a per-token training loss are compatible.
3PPO ratio shapeThe selected-token ratio is typically organized by batch and sequence position.
4Minimum versus maximumNegating a maximized objective changes the implementation to a maximum of losses.

Das Wichtigste

A stronger teacher model does not automatically produce better fine-tuning data.
Teacher and base-model compatibility remains an open research question.
Sequence-level rewards can be used with token-level losses.
GAE is useful but not the only possible advantage estimator.
PPO’s min objective becomes a max when written as a negative loss.

ThemenPost-training Reinforcement Learning Synthetic data ML-Grundlagen

VOLLTEXT

Den bearbeiteten Text nach Kapiteln lesen

Transkript zeigen

Nur das Transkript

Course prerequisites and synthetic data

Lambert discusses mathematical preparation, teacher selection and balancing supervised data.

This first Q&A addresses questions from the opening four lectures, from course prerequisites to teacher models and PPO implementation. Lambert recorded it in May 2026 and frames the answers as practical guidance and open research questions.

I designed the book for people with graduate-level or early-career machine-learning background. Linear algebra, basic deep learning and experience implementing models help. AI tools make study more accessible, but mathematical maturity still matters; I recommend Sebastian Raschka’s books and the linked RL courses.

Does the strongest model make the best teacher for synthetic supervised data? Not necessarily. Lambert says OpenThoughts 3 and OLMo 3 found 32B Qwen-era teachers unusually strong, and that newer, larger teacher models have been difficult to beat those baselines with.

One possible explanation is compatibility between the teacher’s distribution and the base model, especially when open recipes have co-evolved around Qwen. Lambert presents this as a hypothesis, not a settled result, and wants research into which base models are most teachable.

FOLIE 1 / 2Teacher models and data mixThe slide summarizes the open question of teacher selection and domain balance.

Balancing domains in an SFT mix matters, but mixing good datasets often works without elaborate machinery. Conflicting styles between domains can require adjustments. Harder math and coding tasks may contribute more tokens, so compute or token balance can differ from prompt counts.

Reward models and loss granularity

Different reward models fit different tasks. Sequence rewards and token-level losses are distinct choices.

Reward-model types serve different purposes and data formats. Lambert finds it hard to compare pairwise, outcome, process and generative models under a single benchmark. He sees rubric-guided generative judges as an increasingly general way to supply feedback.

A question about token-level versus sequence-level rewards mixes two kinds of granularity. The reward can still be an outcome at the end of a sequence while the training loss is calculated or normalized per token. Those aggregation choices introduce different biases in training.

Token-level normalization became common in reasoning-model RL because it can provide a more predictable foundation in which individual tokens contribute more evenly. This is a choice about implementation and loss aggregation, not necessarily a more detailed reward for every token.

PPO notation and implementation

The answers cover advantage estimation, ratio tensor shapes and the sign change from objective to loss.

Generalized advantage estimation is not the only way to compute an advantage in PPO-family methods. Lambert groups PPO and GRPO as related approaches, while noting that GAE remains a useful default when a value model is available.

The PPO ratio compares the current policy with the policy at the start of the batch. In a language-model implementation, after selecting the sampled token probabilities, the ratio typically has batch-by-sequence shape, with a value for each prompt and generated token.

An outcome reward can be broadcast over tokens while clipping and loss calculations operate on per-token ratios. Some newer algorithms change the level at which the ratio or clipping is computed; the tensor shapes help explain the implementation.

A reader asks why PPO code uses torch.max when the mathematical objective contains a minimum. The objective is maximized, whereas the implementation minimizes its negative loss. Taking the maximum of the two negative losses therefore preserves the conservative choice.

FOLIE 2 / 2PPO objective versus lossThe code illustrates why the minimized negative loss uses torch.max.

A similar clipped-versus-unclipped comparison may appear in the value loss, which uses regression error. Lambert says value losses can take multiple forms; this particular structure is a practical implementation choice.

The annotation market

Lambert shares his view of a shift toward expert-generated data and closes with upcoming course topics.

Asked how data annotation firms stay valuable as synthetic data improves, Lambert describes a shift toward recruiting specialists such as doctors, finance professionals and consultants to generate high-value data. He presents this as his reading of the market at the time.

The questions wrap up the first four lectures. Lambert plans subsequent lessons on the history of reasoning models and the DPO derivation, and invites readers to keep sending technical questions.