Nathan Lambert Answers Six Questions About Post-Training Recipes

Post-training recipes in industry are more complicated than a single public paper can usually show. A common abstraction is supervised fine-tuning to establish reasoning behavior followed by reinforcement learning to improve capability and agency. In practice, labs use many intermediate models, specialists and training stages.

The OLMo 3 diagram illustrates a relatively simple recipe: pretraining, mid-training and long-context work, followed by supervised fine-tuning and reinforcement learning, perhaps with preference tuning between them. Even this straightforward open recipe contains multiple stages.

One emerging way to combine specialist capabilities is multi-teacher on-policy distillation: train domain specialists and consolidate their directions into a student. There is also research on sequential reinforcement learning by capability, including work from NVIDIA. I would not assume that training every capability sequentially is the industry norm.

Why do reasoning recipes often omit a KL penalty? With verifiable rewards in relatively clean math and code tasks, the ground truth can reduce some of the over-optimization concern. In Tülu 3, we found that math RL could be learned in a fairly specific mode without necessarily disrupting other domains.

As tool use and agentic tasks grow more complex, over-optimization can take more forms and regularization is returning. Reports differ: Open-Reasoner-Zero, Magistral, Skywork and DAPO discuss dropping KL, and GLM-5 removes it to accelerate improvement. Some reports do not make the choice clear, so the frontier picture is messy.

How does a rough budget of a million supervised prompts scale with model size? Larger models often need fewer examples: they may gain more from mid-training and learn more efficiently from both instruction data and RL. Test whether each data source actually improves performance; otherwise it wastes compute.

There are important confounders. Mid-training and supervised fine-tuning increasingly overlap in laying the foundation for reasoning, and agentic completions contain more tokens. The million-prompt rule comes from an earlier era; we need better public figures for total prompt-completion tokens and should not assume the count transfers unchanged.

Is RL merely supervised fine-tuning with a regularization term? Their losses can have closely related mathematical forms, but RL also generates experience from the current model and updates it from that experience. One cannot necessarily prepare all of that data in advance and obtain the same result with supervised learning alone.

Supervised learning can be enough for more applications, particularly small models, than people assume. But RL can help models adapt to harder problems. Its updates can also include negative gradients that push down below-baseline behavior, whereas ordinary supervised fine-tuning primarily reinforces demonstrated targets.

When should an organization train its internal knowledge into weights instead of providing context at inference time? Start by building an evaluation for the actual task and deciding how much an improvement is worth. Compare that benefit with the cost of training; a meaningful frontier-scale training job can be very expensive.

Costs may fall, especially for smaller models. If a task is sufficiently narrow, a small model trained on internal data might beat a frontier model on that task or save substantial serving cost. The decision depends on the business case and on the gap between the current in-context approach and the desired performance.

Can continued pretraining on distilled data make a model worse? Yes. Synthetic or distilled data is not automatically good or bad: the teacher may be poorly matched to the student, or the data pipeline may be flawed. A well-designed pipeline can help, but there is no free improvement simply from labeling data synthetic.