Scott Geng and Nathan Lambert: Bringing DPO Research Into OLMo 3

Lambert introduces Scott Geng, who worked on DPO for OLMo at AI2. They were surprised by how much the technique improved their models and by how few labs appeared to use it. Their goal is to explain both its value under limited compute and the work needed to move an academic idea into a full model.

Geng says the initial experiments were an opportunity to apply his academic work at greater scale. One set of AIME gains exceeded what the team had achieved through a summer of RLVR efforts. That result persuaded them to commit to a DPO stage, but it did not eliminate later problems with data, identity and hallucination.

The pair also recall how unusual examples in post-training data can surprise outside observers. Lambert says a dataset can look untidy even when it has produced useful improvements. They return later to why inspection and distribution matter when the result becomes a public model.

Geng outlines preference data: each prompt has a chosen and a rejected answer, with an annotation signal indicating which is preferred. Earlier RLHF systems such as InstructGPT used preference data to train a reward model and then optimized against it with an online reinforcement-learning algorithm.

Direct preference optimization provided an offline objective under particular assumptions, avoiding much of the online RL machinery. Geng says the infrastructure difference mattered for academic researchers: online methods may require several checkpoints in memory and engineering support that a small GPU budget cannot easily provide.

Geng contrasts this with an academic setup that may have only a handful of GPUs. Keeping several model checkpoints in memory and maintaining reliable online RL code can quickly exceed that budget. An offline objective made a much wider range of experiments possible.

Lambert recalls that excitement followed the May 2023 DPO paper by several months, rather than arriving instantly. Tülu, Zephyr and other early models helped demonstrate its value. He argues that better-suited datasets, including UltraFeedback, were also important to making the method work.

The resulting literature produced many DPO variants. Geng says data often matters more than the precise objective, although an algorithm can compensate for particular data limitations. SimPO, for example, removes the reference model and can reduce infrastructure demands.

One concern is likelihood displacement: optimizing DPO can reduce the probability of both chosen and rejected responses. The speakers say this can matter in specific cases, but in their experience falling chosen likelihood does not necessarily prevent evaluation gains. Variants investigate different assumptions and failure modes.

Geng says variations in the literature often respond to concrete issues, such as response length, chosen-answer likelihood or a preference model that does not match the data. Even when the original DPO objective works well in his setting, those variants contribute to understanding its assumptions.

Geng says the original objective still performed well when tuned carefully in their work. Lambert values the body of controlled research around these algorithms, contrasting it with gaps in understanding modern agentic RL. Large experiments and complex infrastructure now make controlled comparisons harder.

The speakers observe that the cost of agentic RL makes it harder to isolate individual variables. An experiment may advance performance without revealing which component caused the gain. Lambert argues that sharing ablations helps others study questions that commercial teams may not have time to isolate.

Lambert says labs often move on once an experiment produces a useful gain, making AI2's published ablations unusually valuable. Geng recalls how much appreciation the OLMo technical report received even though the team had worked on it up to the deadline. They see documenting the messy process as useful research in its own right.

Geng turns to preference-data pipelines. UltraFeedback and Tülu-style recipes collected prompts, sampled responses from multiple models and used a language-model judge with a rubric to score them. Scores were converted into chosen and rejected pairs. Lambert emphasizes that scaling the dataset was itself a major contribution.

Geng explains that early preference pipelines tended to generate several candidate answers using strong models, then apply a judge to score them along several dimensions. Lambert notes that what once looked like a detailed rubric was quite short by later standards. The scores provided the ordering for training pairs.

The speakers describe complementary early results: Zephyr demonstrated that people liked a model trained with this approach, while Tülu showed strong benchmark performance at scale. Careful comparisons among datasets also supported adoption; the result was not simply an arbitrary large training run.

Geng had entered language-model research after working in computer vision. When he fine-tuned a newer model on chosen UltraFeedback responses, supervised training badly degraded it, even though the chosen answers had been selected by a strong judge. Preference training on the same pairs instead improved performance.

He spent about a month suspecting and rewriting his supervised training pipeline. Eventually, inspecting the data suggested a different explanation: the models that generated the chosen responses were already weak relative to the student. The chosen answer might be inadequate as a target to imitate while still containing a useful comparison signal.

The delta learning hypothesis is that, in paired-gradient learning, the relative gap between positive and negative samples can matter more than the absolute quality of the positive sample. Even a weak chosen answer may teach something if contrasted with a sufficiently worse rejected one. Geng says controlled experiments supported this idea.

Geng emphasizes that this is a hypothesis supported by the controlled settings they studied, not a claim that absolute data quality never matters. The surprising observation is that weak answers can still supply a directional signal when the rejected answers are weaker.

Lambert sees a connection with modern RL methods that use positive and negative gradients. Geng notes that offline DPO offers direct control over the pairs, which is both an opportunity and a limitation. Although the loss forms can be similar under assumptions, online and offline methods have different data distributions.

Lambert compares the idea to GRPO-style learning, but notes a practical distinction: DPO pairs provide a deliberately balanced contrast, whereas online batches can have a different mixture of positive and negative examples. Geng agrees that this difference in data construction matters.

At the time, researchers often tried to make the chosen answer as strong as possible. The hypothesis shifts attention to the contrast. Geng says this allows a simpler data-generation experiment instead of requiring a pool of frontier models and a costly judge for every response.

Building the earlier strong-model pipeline was itself a substantial engineering effort. The speakers credit LJ Miranda for coordinating multiple scripts, models and inference settings at AI2. Lambert observes that modern coding tools may reduce some of that setup burden.

The team generated pairs with two weak models, one around 3B and one around 1.5B, and used the larger model's answer as the chosen response. The heuristic agreed with a stronger judge most of the time, not always. Despite both teachers being weaker than the student on the studied benchmarks, the recipe worked as well as or better than the comparison pipeline in their experiments.

Geng says the research began as a separate project and entered OLMo 3 after Lambert invited him to try it on an instruct checkpoint. It worked well after several weeks. They had not initially planned a thinking-model DPO stage, but decided to try the same idea there too.

The first application targeted an instruct model, not the reasoning model. The team found the instruct result encouraging before Lambert suggested trying a corresponding thinking stage. This sequence shows that the eventual recipe was discovered through experiments rather than specified in advance.

For the thinking model, they enabled reasoning in the Qwen models generating the preference data. The resulting DPO stage produced large gains on AIME. Lambert notes that an interesting result could still have ended up only in an appendix; inclusion in the final model required much more work.

The early thinking result was only the beginning. A result that works in a local experiment needs to survive data selection, downstream training, checkpoint handoffs and scaling before it can become a dependable stage in a released model.

The team had to make the stage robust in a full recipe. Geng and a collaborator tried systematic prompt-mixing ablations over a large pool of coding, math and instruction-following prompts. The results were unexpectedly spiky and did not reliably follow the domain they had upweighted.

For example, increasing the code share could reduce code performance, while increasing instruction-following prompts could improve math more than increasing math prompts did. Lambert says this shows how little they understood about what DPO was doing. The behavior differed from the smoother domain gains they expected from other training stages.

Geng contrasts the DPO mixing results with the more predictable response he had seen in some supervised or mid-training ablations. In those settings, increasing examples from a domain could yield the expected improvement. Here, changing one proportion could move apparently unrelated evaluations.

Geng hypothesizes that offline DPO may overfit or exploit reward signals in ways that make proportions matter, producing non-monotonic results. He does not claim mixing is useless. Instead, the team lacked a reliable systematic method and gained intuition only after training many variants.

Ultimately, a hand-chosen mix significantly outperformed simple independent sampling in their experiments. Lambert adds that later RL still benefited from the preceding DPO stage in their recipe, so the observed gain was not simply borrowed from a later stage.

Data quality created other surprises. The speakers discuss a strange example in a public dataset that had attracted criticism; Geng says the team knew it was present. Their broader point is that public interaction data contains unusual and repetitive requests, and researchers should inspect rather than assume a clean distribution.

The pair discuss the source of some unusual examples in WildChat, a dataset built from voluntary use of a public API. They note that such users are a particular population and their requests need not represent all users. Repetition and mismatched modalities can also persist in training data.

They also mention text-only examples asking for image work, identity confusion and possible hallucination issues. Such details can be treated as peripheral in a narrow study but matter in a released model. The speakers are uncertain about the sources of some problems and avoid assigning them to one stage with confidence.

Organizational staging created another challenge. The SFT and DPO teams worked in parallel; a checkpoint that looked better on the SFT team's evaluations could perform worse after DPO than an earlier checkpoint. A promising intermediate result did not guarantee a better final model after later RL.

Geng explains why handoffs are risky: each team optimizes an intermediate checkpoint under its own measurements, but the next stage may respond differently. The best checkpoint at one handoff is not automatically the one that produces the best final system.

Geng suggests that additional handoffs and uncertain interactions may help explain why some labs avoid adding DPO even when it can be computationally efficient. Lambert describes each stage as a potential gate. Both emphasize that coordination and checkpoint selection are as real as the algorithmic work.

Identity failures also prompted familiar cross-team questions: pretraining might contain the wrong identity, while post-training might fail to correct or might reinforce it. The speakers say the entire pipeline can contribute, making attribution difficult. Teams still have to resolve the issue rather than merely assign blame.

Scaling from 7B to 32B exposed a different class of problem. At smaller scales, standard infrastructure could run on one or two nodes. At 32B, communication and memory details that had seemed minor became important, and a run with a similar recipe unexpectedly showed training loss rising.

The 32B experiment increased both the model size and the number of nodes. The resulting rising loss was new to Geng, and the team could not simply pause for a month to investigate: the final run had a deadline and much of the cluster was allocated to it.

The run occupied a large portion of the cluster close to a deadline. Geng says the team tracked down many small details rather than finding a single obvious explanation. Incorrect NCCL settings for the cluster wiring interacted poorly with the DeepSpeed sharding configuration.

Another surprising factor was how chosen and rejected responses were processed. A concatenated forward pass had been disabled for memory reasons, but separate passes coincided with failure. Re-enabling the combined pass, shortening context to fit memory and changing communication settings resolved the run; Geng says they still did not fully understand why the combination fixed it.

The experience illustrates a barrier to wider DPO use: public infrastructure that works at small scale may not transfer smoothly. The speakers stress that these were obstacles after the research idea had already shown promise. Getting an idea into a final model takes engineering and coordination beyond citing a paper.

Lambert says that a research citation from a model lab should not be mistaken for a plug-and-play transfer. A team may have invested substantial work to adapt the method to its data, infrastructure and downstream stages even when the central idea came from a paper.

Geng says he underestimated how much organizational structure and incentives matter for successful training. Lambert agrees that differences between labs are consequential but hard to measure. Increasing model complexity requires organizations to adapt as well as algorithms and infrastructure.

Asked about the future, Lambert expects preference-style learning to remain relevant for open-ended expert tasks. Individual components of deep research may be measurable, while judging the overall result remains messy. Domain-specific adaptation of capable models may also drive practical innovation.

Lambert notes that full open post-training recipes are expensive and uncommon, but provide valuable places to fork and study the process. Smaller open models create opportunities, even if the most demanding agent evaluations require resources available to only a few academic labs.

Geng distinguishes research aimed at understanding a phenomenon from work aimed directly at improving frontier models. Historically those goals were closely connected; he worries that the latter is becoming less accessible to academics. He recounts an industry researcher saying their team rarely reads outside pretraining work, while noting that not every field or team behaves that way.

Geng says direct transfer from academic pretraining research to large industrial teams may be weakening, partly because pretraining decisions carry enormous cost. He thinks post-training ideas may circulate more readily because they can often be tried more quickly, while emphasizing that practices vary by team.

Lambert sees opportunities in substantial open research artifacts and post-training methods, even if windows of opportunity eventually close. He thinks those windows often last longer than people fear. Geng agrees that open recipes make it much easier for a community to build on existing work.