InstructGPT: From Human Feedback to Language Model Alignment

The presenter begins with the attention ChatGPT attracted in late 2022 and its potential for emails, documents, and multi-turn conversation. The lecture reviews InstructGPT because the public ChatGPT description at the time said it used a similar training method.

Official examples show the model asking for more information about code, declining an unlawful request, continuing a mathematical conversation, and explaining limits on real-world actions. The presenter uses them to illustrate dialogue and safety behavior, not to guarantee correctness.

A mathematical dialogue example shows the model continuing an explanation, generating a poem, and summarizing the conversation. The presenter frames it as performance in a public example at that time.

The presenter argues that multi-turn dialogue feels more natural than a single prompt and could enable many applications. He notes that a detailed ChatGPT paper was not available at the time, so public information must be distinguished from speculation about training details.

The public description says ChatGPT uses a method similar to InstructGPT, with differences in dialogue-data collection. The presenter speculates about GPT-3.5’s data composition without presenting those guesses as established paper results.

InstructGPT addresses a mismatch: a next-token language model depends on its pretraining text and may fail to follow user intent or produce inaccurate or unsafe content. The paper uses human demonstrations and feedback to bring behavior closer to the intended outcome.

Human labeling is not new, the presenter says, but it supplies a more direct signal for particular needs. Practical systems must balance labeling cost, compute, and data coverage rather than assuming that more unsupervised data solves every problem.

The paper argues that larger models do not necessarily follow instructions better: they can still produce false, toxic, or unhelpful answers. Alignment means bringing behavior closer to user intent; the presenter stresses that safety and usefulness matter alongside benchmark scores in deployment.

He recalls harmful or incorrect outputs in image labeling, chatbots, and an academic language model, followed by product withdrawals or changes. Generative outputs are more flexible and widely applicable, making it harder to enumerate all failure cases.

He describes an academic language model producing plausible but wrong or biased material, raising concern that readers could trust it. Fluency and reliability are different properties.

He also describes a role-play prompt that bypassed a refusal, cautioning that safety training does not make refusal a reliable guarantee. Safety limits require ongoing testing rather than judgment from a single demonstration.

InstructGPT first collects prompts and human-written demonstrations to fine-tune GPT-3. Annotators then rank several model answers to train a reward model, and reinforcement learning adjusts the language model toward the learned preferences.

The paper reports that, in its human-preference evaluation, responses from a 1.3-billion-parameter InstructGPT were preferred to those from 175-billion-parameter GPT-3. The presenter cautions that this is a task- and evaluation-specific comparison, not universal superiority of the smaller model.

The paper also reports improvements in truthfulness and toxic outputs, without broad severe regression on public NLP tasks, while acknowledging that simple errors remain. The presenter says task-matched data can make training signals more efficient, subject to labeling cost.

The paper describes its aims as helpfulness, honesty, and harmlessness: assist users, be truthful, and avoid harm. Next-token prediction during pretraining is not identical to these requirements, which creates the gap the authors seek to narrow.

The presenter notes that reinforcement learning from human feedback builds on earlier research. One contribution of InstructGPT is applying that training route to broader instruction tasks and data.

The first step is supervised fine-tuning, or SFT: prompts and human-written answers form examples used to continue training GPT-3. The model still predicts tokens in a sequence, but the data is closer to the intended task.

The second step generates several answers to a prompt for annotators to rank. A reward model receives the prompt and response and produces a score; training encourages preferred responses to score higher without requiring absolute human ratings.

The third step starts from the SFT model, scores generated responses with the reward model, and optimizes the language model. Ranking is often cheaper than writing many complete answers, potentially yielding more preference information within a labeling budget.

The technical questions include collecting demonstrations and rankings, training a reward model from rankings, and using that reward model to improve the policy. The lecture develops these steps separately.

Human evaluators preferred InstructGPT responses, and the paper reports changes in truthfulness and toxic output, but bias metrics did not clearly improve. Better scores on some measures do not mean the model is safe or unbiased.

Fine-tuning for a specific objective can harm other tasks. The paper adds a pretraining objective during reinforcement learning to mitigate regressions on public NLP tasks, and uses held-out annotators to test whether preferences carry beyond the training raters.

The authors also compare public instruction datasets, observing that the match between training data and target user tasks matters. The presenter cites code-related tasks as an example of generalization beyond abundant task-specific examples, while cautioning against assuming universal reliability.

The presenter distinguishes impressive demonstrations from reliable tools: a tool must work consistently in ordinary situations, not merely surprise occasionally. InstructGPT and the ChatGPT of that period still made simple errors and needed evaluation in context.

The dataset combines prompts written by annotators with filtered prompts submitted by users. The presenter describes an iterative process: train on initial data, then learn from real usage to improve the dataset.

Annotator-written prompts can be free-form questions or designed for particular applications. Once an initial model is available for users, their prompts provide additional information about the task distribution.

Training, validation, and test sets are split by user rather than randomly mixing similar prompts from the same person, reducing evaluation leakage. The paper also describes filtering prompts containing personal information.

The paper lists three datasets: about 13,000 demonstrations for SFT, comparisons on about 33,000 prompts for the reward model, and about 31,000 prompts for PPO. Tasks include generation, open-ended questions, rewriting, summaries, and classification.

Example prompts include brainstorming, storytelling, table extraction, sarcasm classification, and translation. Their variety shows why a few fixed templates cannot cover real usage.

The authors recruited roughly 40 contractors through labeling platforms and describe their selection and collaboration process. Open-ended judgments are ambiguous, the presenter says, so quality depends on screening, clear instructions, and sustained communication, not headcount alone.

The presenter notes a difference in emphasis between labeling and evaluation: training guidance prioritized helpfulness, while evaluation also considered truthfulness and harmlessness. The paper reports about 72% agreement between annotators, though agreement does not eliminate subjectivity.

The paper’s appendices describe the labeling process and interface. The presenter recommends studying existing workflows: building a labeling system includes task design, feedback, and quality control as well as recruitment.

SFT trains for multiple passes on the small demonstration set and initializes later stages. The reward model converts language-model output to a scalar score; the paper uses a roughly 6-billion-parameter model, with stability and compute cost among the considerations discussed.

The reward model uses pairwise ranking loss: for one prompt, a preferred and a less-preferred response are compared, encouraging scores to preserve their order. A sigmoid and logarithm express the loss in terms of relative preference.

A complete ranking of nine responses yields 36 pairs. The presenter argues that ranking more answers after understanding a prompt may add only some annotation time, and each response’s score can be reused across pairwise comparisons.

Grouping comparisons for a prompt lets the model score each answer once and combine those scores into pairs. This reduces repeated computation and affects overfitting behavior.

The paper also compares selecting only the best answer with learning a full ranking. The presenter says the fuller ranking supplies more constraints and the authors found it helped reduce overfitting; the gain depends on data and training setup.

The third model is optimized with PPO. In reinforcement-learning terms, the language model is a policy that generates responses to prompts; the reward model scores them. As the policy updates, the distribution of generated responses changes too.

PPO initializes the policy from the SFT model. Prompts come from the reinforcement-learning dataset, the current policy samples responses, and the reward model scores them. The objective raises scores subject to constraints.

Why not ask people to rank responses after every update? The presenter explains that training and human feedback would have to wait for each other. The reward model approximates human preferences and supplies scores during training, but it remains a proxy rather than human judgment itself.

The reward model is trained on responses from an earlier policy. If the reinforced policy changes too much, new responses may shift away from the reward model’s training distribution and make its scores unreliable. A KL penalty relative to the SFT reference limits that drift.

The KL term measures drift from the reference policy, and its coefficient controls the penalty. It cannot prove the reward model correct, but can reduce the risk of the policy overexploiting reward-model weaknesses.

The objective can also include a language-modeling term on pretraining data to prevent preference optimization from eroding prior capabilities. A coefficient controls its strength: the paper calls the versions PPO and PPO-ptx, respectively.

The presenter compares work that repeatedly collects new preferences, updates the reward model, and optimizes the policy again. The need for multiple rounds depends on task complexity and coverage of new outputs; this paper’s process is comparatively simple.

The training chain is: human demonstrations produce the SFT model, rankings train the reward model, and PPO uses that model to improve the policy. Reward approximates labeled preferences, while training must also limit drift and protect other capabilities.

The experimental chart compares GPT-3, supervised fine-tuning, and preference-optimized models at different sizes. Human evaluation in the paper favors a small InstructGPT over a much larger base model; the presenter notes that similar training and evaluation prompt distributions can favor fine-tuning.

When labeled training data resembles evaluation data, fine-tuning can be favored over a model that has not seen the target format. The result should be read as a human-preference comparison on the paper’s prompt distribution.

The paper’s limitations include the judgments of roughly 40 contractors, who cannot represent every future user; predominantly English data, which can affect other languages; and unresolved safety and factual errors.

A deeper question is alignment with whom and with what: the user’s immediate instruction, or broader intent and consequences? The presenter says these goals and their measurement remain difficult, and RLHF is only one possible approach.

The presenter argues that the paper focuses more directly on helpfulness, without explicitly optimizing every aspect of truthfulness and harmlessness. He asks whether more high-quality supervised demonstrations might achieve adequate results without the complexity of a reward model and reinforcement learning.

The presenter closes by framing that alternative as a question to test, not a settled conclusion. Anyone turning a generative model into a product must still assess safety and errors rather than assuming one preference experiment resolves deployment risks.

For public-facing products, the presenter says organizations must also consider the practical effects of errors and public response. The paper supplies a training approach, but deployment still requires application-specific evaluation.