Nathan Lambert on Preference Data and the Foundations of RLHF

I want to revisit the human feedback and data side of post-training: what preferences are and how we collect them. These questions were once central to AI safety, remain difficult, and now receive less attention. RLHF asks how to improve a capability when we do not have a straightforward score for it.

Often it is easier to judge two examples than to create a good one. You may struggle to write a great poem about a train in a storm, yet still distinguish a strong poem from a weak one. Preference pipelines use that intuition, while accepting the tradeoffs of an imprecise measurement.

I show two poems generated by Claude 3.7 Sonnet and GPT-4o in February 2025. I no longer know which model wrote which. People may disagree about which poem is better, raising the practical question of how to use a distribution of judgments to improve a model.

Preference signals replace correctness when no single right answer exists. Preference tuning, including DPO and RLHF, has also improved measurable capabilities: an apparently subtle preference signal can make models perform better on benchmarks the community cares about.

The lecture combines a history and philosophical discussion of preferences with practical data collection and open research questions. Understanding where the tools came from makes their social context and hidden assumptions easier to see.

In 2023 I coauthored a paper on the history and risks of RLHF, tracing ideas from economics, optimal control and philosophy. Using tools from complex disciplines without understanding their problem formulations can hide the tradeoffs we make.

My PhD work on objective mismatch in model-based RL provides an analogy. A model learns environment dynamics so a policy can act, but the most accurate dynamics model is not necessarily the one that produces the best policy. Likewise, the most accurate reward model on its data is not automatically the one that makes the downstream language model best.

Reward-model benchmarks are proxies, and every proxy is imperfect. That theme recurs throughout the course. To understand why, it helps to trace how preferences and utility became numbers that a system can optimize.

Modern post-training draws on control theory and electrical engineering, computer science and deep learning, economics and decision theory, and older philosophical thinking. These fields brought different conceptions of decisions and optimization into the same technical practice.

The Port-Royal Logic discussed weighting outcomes by their probabilities; Bentham proposed measuring decisions on a common scale; Ramsey linked preference and belief to probabilistic decisions. Lambert presents these as intellectual steps toward quantifying utility.

The von Neumann–Morgenstern utility theorem says that preferences satisfying assumptions such as completeness, transitivity, continuity and independence can be represented by a utility function. A rational agent can then be modeled as maximizing expected utility.

Human preferences in RLHF do not reliably satisfy such tidy assumptions. They shift over time and with context, and have multiple dimensions. Reducing them to one number makes the problem tractable, but creates tradeoffs in how human behavior is represented.

The Bradley–Terry model, introduced in 1952, connects pairwise comparisons to scores representing relative quality. But a preference between two open-ended responses can depend on countless features of the text and on how the annotator was instructed. There is no perfect reduction without tradeoffs.

Behavioral economics and social choice theory study further complications in aggregating individual preferences. I have explored how social choice ideas might inform RLHF, an area that remains interesting even though recent attention has shifted toward verifiable rewards and immediate AI risks.

Preferences change with context and time, placing limits on static collections of human feedback. At the same time, work in optimal control and reinforcement learning developed methods for optimizing dynamic systems, including Bellman’s dynamic programming and later deep RL.

The convergence was tempting: if preferences can be represented by a function and RL can optimize functions, perhaps the two can be combined. But costs, rewards and preferences come from different intellectual traditions. Treating a preference as a reward narrows the problem and brings assumptions with it.

A learned reward model is not a ground-truth reward, which helps explain over-optimization. I also find it interesting that inverse RL, which infers rewards from observed behavior, did not play a larger role in early RLHF despite its connection to learning a reward model from preferences.

There is still room for more interdisciplinary exploration. Older RL work asks where rewards come from and compares reward and value functions with primary and conditioned rewards in animals. The philosophical richness of these questions is much greater than our often narrow empirical framing.

Turning to practical data, judgments are often easier to obtain at scale than demonstrations. Human preferences still help teams understand and improve models, yet the instructions, collection process, released data and relationship to final model improvements have rarely been documented together.

NVIDIA’s HelpSteer datasets are among the closest examples of connecting released human preferences with model development. I would have liked to see many more such examples, given how central preference data was to post-training in 2023 and 2024.

Early Anthropic research interfaces showed annotators a conversation and two possible responses, with a scale for choosing between them and space for metadata. ChatGPT has also shown users A/B comparisons, while Arena offers options such as ties or both answers being bad.

User A/B choices can be evaluation signals and sometimes training data. I refer to OpenAI’s account of a sycophancy incident in which a reward model based on user feedback was over-optimized. Thumbs-up and thumbs-down feedback provides another common interface.

Image generators often show a grid of candidates. Users can rapidly identify an image they prefer, and those choices can later support reward models or other alignment methods. The mechanics of image post-training differ from language models, but the selection interface is intuitive.

A rating scores one completion in isolation; a ranking compares two or more. Training data is often reduced to pairwise rankings, while individual ratings remain useful metadata. Likert scales record the direction and strength of a preference and have been widely used during collection.

Structured or synthetic preference data can be created when correctness is checkable: a correct answer is preferred to an incorrect one, or a response satisfying a formatting constraint to one that violates it. Hard constraints can make content contrived, so this remains a heuristic rather than a complete account of quality.

Researchers have also explored feedback beyond pairs, including positive or negative labels, token-level feedback and natural-language critiques. These approaches have clear motivations, but many have been difficult to collect or scale, or have not shown enough benefit at scale in practice.

Human annotation is expensive and operationally demanding. A vendor contract may start with small batches and ramp up as instructions and the data improve. The team must be ready to incorporate incoming data and improve its models while the collection is underway.

In my experience, data can arrive very quickly, and buying it does not guarantee a useful result. Before purchasing training data, a team needs an established post-training methodology and the internal expertise to use it. These challenges have grown with larger budgets and agent environments.

Contracting also requires detailed labeling instructions. I recovered publicly released InstructGPT labeler instructions and put them in the RLHF book repository. They are a useful historical artifact for understanding what annotators were asked to do.

When I worked on data sourcing at Hugging Face, we bought both supervised fine-tuning and preference data. Preference collection was particularly hard for a small team because on-policy comparisons should come from a model in the relevant family; supervised examples do not have the same constraint.

Preference comparison has also become a business. Lambert says Arena sells model testing across domains and cites an annual recurring revenue figure of $100 million. Despite criticism over who its users are, he considers its comparisons an important signal for the industry.

Open questions include biases in preference data. Lambert tentatively describes prefix bias as the opening of a response disproportionately influencing a label. Sycophancy rewards telling users what they want to hear, while verbosity can make a response appear more complete or trustworthy.

Formatting and decorative language can also influence scores: lists and bold text may look appealing independently of substance. New biases can emerge as domains and models change, so eliminating them is an ongoing problem.

Who annotates matters too. Paid and unpaid annotators have different incentives and limitations. Lambert doubts commercial model development will give everyone an equal say, but argues that knowing who shapes the data can help explain and address some biases.

One question matters especially to me: do the values written into a labeling specification actually appear in the final model? I want to publish a model specification, show where models meet or miss it, and trace what changes their behavior through transparent data, intentions and outcomes.

The problems of collecting preferences, training reward models and integrating them into larger pipelines remain open. They offer useful academic research opportunities with established literature to build on. The next lecture will examine over-optimization and regularization.