Nathan Lambert Explains DPO and Direct Preference Optimization

Direct preference optimization, or DPO, is an important part of the post-training toolkit. This lecture derives its objective and surveys related methods. Even as attention shifts to agentic reinforcement learning, understanding DPO explains another way to change model behavior.

The 2023 DPO paper arrived when open projects were finding full RLHF pipelines difficult to run. Its simpler training approach helped make preference optimization more accessible, though it still took months to find practical settings and data for open models.

Lambert recalls Zephyr and Tülu 2 as early open models that helped demonstrate DPO’s usefulness. He calls the wider family of methods trained directly on pairwise preference data direct alignment algorithms, though the label did not become universal.

The debate over DPO versus RL generated years of argument. My practical answer is that both are tools. DPO learns directly from preference pairs without a separate reward model and RL loop, which makes it easier to start, while a full post-training recipe can combine approaches.

The derivation proceeds in four stages: solve the regularized RLHF objective for its optimal policy, recover a corresponding reward, substitute it into a Bradley–Terry preference model, and form a loss and gradient. The resulting implementation is much shorter than the derivation.

Classic RLHF uses preference pairs to train a reward model, then optimizes a policy against it. DPO collapses those steps into a supervised-style preference loss. Real pipelines can contain multiple rounds of data generation, evaluation and optimization, with DPO as one component.

Start with a policy that maximizes expected reward while penalizing KL divergence from a reference policy. The parameter beta controls the tradeoff. Prompts come from a dataset and completions are sampled from the policy.

Expand the KL term, change maximization to minimization and regroup the expectations. Using logarithm and exponential identities, the reward and reference-policy terms can be combined into an expression resembling a KL divergence.

Gibbs’ inequality tells us a KL divergence is minimized when its two distributions match. But the reward-weighted reference term is not yet normalized, so it is not a valid probability distribution and the inequality cannot be applied directly.

Introduce a partition function Z for each prompt to normalize the reward-weighted reference policy. This yields a valid distribution q. Lambert emphasizes that the normalization is the step that makes the later KL argument legitimate.

Add and subtract the log partition function, rearrange the terms and identify the KL from the candidate policy to q. The remaining log Z depends on the prompt but not on the policy being optimized, so it does not change the minimizing policy.

The result is that the optimal policy equals the reference policy multiplied by an exponential reward weight and normalized by Z. Higher reward raises the relative weight of a completion, while beta controls the strength of the regularization.

Next, take logarithms and rearrange the optimal-policy equation to express the reward in terms of the policy-to-reference log ratio and a prompt-dependent normalization term. This is the sense in which the language model can represent an implicit reward; DPO does not train a separate reward model.

The Bradley–Terry model expresses the chance that one completion is preferred to another using their reward difference. Substitute the policy-derived reward for both answers. Since both answer the same prompt, the partition-function terms cancel.

After simplifying, the preference probability is a sigmoid of the difference between the chosen and rejected policy-to-reference log ratios, scaled by beta. The inaccessible partition function has disappeared from the pairwise comparison.

To train, minimize the negative log likelihood of observed preferences. This gives the DPO loss. Implementations typically write that loss and let PyTorch automatic differentiation compute gradients rather than manually coding the derivative.

The derivative uses the chain rule and sigmoid identities. The reference policy is fixed, so its parameters do not receive gradients. Lambert warns readers to track signs and the ordering of chosen and rejected responses carefully, and notes a possible beta typo on one slide.

The gradient gives greater weight when a rejected response is ranked too highly relative to the chosen one. The update adjusts the language-model likelihoods rather than training an explicit reward network. Beta scales the preference objective and its relation to the reference.

A subtle limitation is likelihood displacement. The objective cares about the margin between chosen and rejected responses, not an absolute guarantee that the chosen response probability rises. The rejected probability can fall faster while the chosen probability falls too.

Lambert describes concerns that probability can move toward other, out-of-distribution outputs. Researchers have proposed ways to preserve or increase chosen-answer likelihood. The issue illustrates why a growing preference margin is not the whole story.

The KL role also differs from an explicit penalty in an RL loop. DPO’s reference and beta enter the derived objective; practical behavior still depends on optimization and tuning. Some RL implementations can adjust KL penalties dynamically during training.

The family of variants is large. Identity Preference Optimization modifies the preference objective to address potential overfitting; conservative DPO models noisy preference labels; offset DPO adds a margin reflecting preference strength.

A reference model costs memory or requires precomputing and storing its log probabilities. ORPO combines preference learning with a supervised term on chosen answers without a separate reference. SimPO also removes the reference and uses length normalization among its changes.

In code, obtain policy log probabilities for chosen and rejected completions, compare them to reference log probabilities, apply beta and log-sigmoid, and minimize the loss. This compact implementation helped people experiment with DPO.

Synthetic preference data was another catalyst. UltraFeedback used model-generated answers and a GPT judge to rank responses across multiple aspects. Lambert credits it with helping early DPO methods show practical gains without collecting all preferences from humans.

Tülu 3 and other projects explored scaling synthetic preferences and including answers generated by the model being trained. Those on-policy completions can make the comparison data more relevant to the actual model and its training recipe.

Lambert also points to OLMo 3 work on the quality gap between chosen and rejected answers: if the two are too similar, the training signal may be weak. He considers the structure and quality of preference data an important area for further research.

DPO can be simpler, faster to start and cheaper to iterate on with fixed data. RL can explore new completions during training but requires more compute and engineering. Lambert cites studies in which RL reached better peak results, while stressing that the methods occupy different practical roles.

In Lambert’s view, DPO remains a solid way to get a useful model or test an idea. Teams with more refined data pipelines, engineering and resources may gain more from integrating preference signals into broader RL stages. He expects to continue using both approaches.