A Team of Fifteen Matched Meta’s Post-Training on Meta’s Own Base

On November 21, 2024, the Allen Institute for AI shipped Tulu 3. The bet was aggressive: take Meta’s own Llama 3 base, post-train it, and try to beat the official Llama 3.1 Instruct on a suite covering factuality, knowledge, math, code, instruction following, and safety. Meta’s Llama 3 effort involved hundreds of people, a large share of them on post-training. AI2 had ten to fifteen.

They matched it, on a clock that Lambert describes as metronomic. A few months after kickoff they passed Llama 3 Instruct. A month and a half later the 8B passed Llama 3.1 Instruct. A couple of weeks after that, so did the 70B. The differentiator, he says, is the data. Meta’s paper sketches a method outline, is vague on hyperparameters, and shares none of the post-training data. Tulu 3 built three to six new datasets for the project and is releasing all of them, mixed with existing public sets.

Nathan Lambert is a research scientist at AI2 and the author of Interconnects, one of the more widely read newsletters in machine learning. Nathan Labenz hosts him on *The Cognitive Revolution* the day of the release for a two-hour teardown of how frontier post-training actually works when you cannot hide the recipe. Labenz’s framing is that this is one of the most detailed public discussions of a historically opaque stage of LLM development, done by a team small enough that other labs can actually study the approach.

AI2 is ten years old, funded by Paul Allen’s estate, and recently under a new CEO, Ali Farhadi, a University of Washington professor whose group is best known for YOLO in computer vision. The institute narrowed onto open language models. Lambert, who joined a little over a year before this recording, is blunt about the nonprofit’s job: you live by the story you can tell, the money comes from a tech billionaire’s estate, and you need good models if you want anyone to believe open source still matters.

Tulu is on its third generation. Tulu 1 mixed the instruction datasets the community dumped in 2023. Tulu 2 showed DPO still helped at 70B. Tulu 2.5 tried to settle PPO versus DPO: if you tune it, PPO is probably a bit better, but it is not worth spending your life on preference algorithms when you could be making better data. That conclusion became Tulu 3’s philosophy.

By 2024 the weather had changed. NVIDIA’s Nemotron paper, a paper from Apple, and Chatbot Arena scores rocketing for OpenAI and Google all said the same thing: post-training recipes are no longer one SFT step plus DPO on chat data. Llama 3 has hundreds of people on it. AI2 has ten to twenty. The question was whether an academic group could still do modern post-training — new data, new algorithms, sequenced stages — without just chasing AlpacaEval vibes. They trained about a thousand 8B models hunting the recipe, then validated at 70B, and ran the same stack on their own OLMo bases to see if it transferred. Llama 3.1 Instruct, Lambert has heard, was mostly SFT plus DPO, no fancy verifiable RL. Some of what Tulu 3 added, he would bet, shows up in a Llama 4 or 3.5 Instruct.

Pretraining is next-token prediction on web-scale text. SFT is the same loss on high-quality question-answer pairs, with chat-format tokens added. Preference methods switch the loss so the model learns from a chosen completion versus a rejected one. Tulu 3 runs three stages: SFT, DPO, then reinforcement learning from verifiable rewards, RLVR. Lambert’s cost sheet is unusually specific. SFT is the hog. The final mix is about a million prompts, mostly single-turn — they are not a Meta AI shop and do not have multi-turn evals. More than 30 percent is math, including targeted subsets for weak micro-skills such as intermediate algebra, plus 100,000 multilingual samples from Cohere’s Aya because Chatbot Arena has a real multilingual slice even though it was not in their suite.

On 32 H100s, their Transformers-based code trains an 8B on that mix in about a day. Labenz ballparks two dollars per H100-hour, so something like $1,500; Lambert says a thousand dollars is fair, and they could probably get 90 percent of the performance with half the prompts. DPO sits an order of magnitude smaller, a couple of hundred thousand prompts, 16 to 32 GPUs for six to twelve hours, roughly a quarter of SFT’s cost. The engineering trick is caching the reference model’s log-probs so you do not keep two 70B models in memory. Skip that and DPO starts to look like PPO, which wants 128 GPUs and up at 70B.

PPO has a reward model as the environment and a value model that scores every token, with the final reward backed up by Bellman updates. DPO has no reward model; the loss looks at the sum of log-probs over the whole answer and widens the gap between chosen and rejected. Dig into the DPO math, Lambert says, and you find an oddity: it decreases the probability of the rejected response more than it increases the chosen one. RL is more flexible. It can change reasoning steps in ways SFT and DPO do not. RLVR, the new stage, rewards math answers that are right and IF-Eval constraints — at least 100 words, three paragraphs — that Python can check. There is no reward model, only a value function. Drop the method on a random old Hugging Face checkpoint and GSM8K can jump about 15 points without much other collapse.

The ceiling is a property of the base. On Llama 3 8B, GSM8K saturates at 87 to 88 no matter what they do. On AI2’s July OLMo base, the same recipe moved GSM8K from 60 to 75. Nobody can say what defines the cap, but it opens a new mix: if RL can recover math, you can spend less SFT budget on it. Rough split: 90 percent of eval performance from SFT, the last 10 percent from DPO and RL. AlpacaEval-style vibe scores come mostly from the last two. Their SFT model already beats Llama 3.1 on the in-house suite. It just talks a bit stiff. Preference tuning is what makes it a decent conversational partner.

Lambert’s own numbers are the episode’s densest claim. Tulu 2 to Tulu 2.5, swapping DPO for a well-tuned PPO, is about 1 percent. Tulu 2 to Tulu 3 is about 14 percent, all from data curation and process. Algorithmic papers, in most cases, are biting 1 to 2 percent. Specific data where you care can be ten times that. They tried PPO at the end of Tulu 3 and never beat their DPO setup. He still thinks they could if they hammered it. Experiment time and data matter more than four times the compute chasing half a point. He calls PPO versus DPO a great, silly debate that never goes away.

Closed labs use humans for preference data. AI2 does not have that money, so every preference label is an LLM judge. He quotes John Schulman: human preference data is high noise, low bias; LLM preference data is low noise, high bias. Nobody can name the bias yet. The pipeline still runs. They redid Ultrafeedback with completions from their own SFT models, then LLM scores. That on-policy preference data beats a random Hugging Face set, every time they have checked. DPO updates weights from the log-probs of tokens in the completion. If those tokens come from a distribution the model has not seen, the learning signal is bent.

The project spent more than $50,000 on OpenAI API bills for the judge, all of it cringed over, because the money goes to a closed lab while they do open research. The same work with humans would be millions. It is still a rounding error next to GPUs: on the order of a few thousand H100s, tens of millions of dollars a year in compute assets. Closed labs, he says, put one person full-time on one domain — you are the Python-code instruction-and-preference lead, and you grind that one thing for a month, twelve people each owning a slice. No academic lab will ever match that depth. Fine-tuning from Instruct, not from base, is the other fork: if Llama ships a strong Instruct model, a startup does not need the whole Tulu 3 suite to specialize. From-base work is still what the ecosystem needs to see, because that is where the transparency is missing.

Qwen’s 72B, scored with Llama 3.1’s Minerva extraction on MATH, gets a 6. Switch to AI2’s more flexible extractor — boxed, “the answer is,” a colon, one other pattern — and it gets 74. Same model, different ruler, a twelve-fold swing. Every lab is tailoring training to its own eval format. Before you compare numbers, you have to know the stick. Most of Tulu 3’s post-training evals use a chat template, zero-shot chain of thought, and a mix of zero- and eight-shot depending on the domain. Answer extraction on reasoning is the trickiest piece.

They decontaminated every training set against the development evals. The rule is harsh: if a dataset has more than a 2 percent exact prompt match with a test set, the whole thing is out. OpenInstruct, NVIDIA’s Daring-Anteater, and Hugging Face’s NuminaMath TIR all had math contamination. NuminaMath was built for a Kaggle contest, not a fair leaderboard. They are also releasing decontaminated versions of popular sets, and they kept an unseen eval suite that never went public. You still cannot know whether the models you are comparing against trained on the test. Lambert assumes there is a snarky paragraph in the paper that says exactly that.

His earlier project, RewardBench, found its own prompts leaking. Someone used MagPie — a trick that manipulates the chat template so a model emits in-distribution prompts — and the model spat out test items. A method meant to generate training data turned into an audit for whether a model cheated. Evals, he says, are only growing in value and cost. The industry needs a source of truth. He expects a mix of new startups, academic work, and regulation aimed at contamination.

SFT is still next-token prediction, plus a handful of special tokens. For format, you do not need much of it; for math, you need flops, the same way pretraining does, to kick-start chain of thought. Llama’s own report, Lambert notes, puts far more math, code, and reasoning into instruction tuning than into preference tuning. Preference is a contrastive loss. In PPO the reward model is not really inside the update: it is a cramped environment that returns a score on a whole trajectory, and a value model then attributes a number to each token via Bellman backups from that final reward. Warm-start the value model from a reward or SFT checkpoint, or wait a few hundred steps for a random init to catch up, or the policy will not move.

DPO skips the value model. Chosen and rejected sequences are compared by the sum of their log-probs, all negative numbers, and the loss widens the margin between those two sums. Length bugs in the early DPO days came from that summing. Lambert no longer has a crisp per-token story for it; the original authors might. Both are substantially different from SFT. Jason Wei’s framing, which he likes, is that RL is a more flexible loss with more legroom. The strange part is that you can pretrain, SFT, then throw a new loss at the model, and KL regularization is enough to hold it in place. That is the kind of fact that gives him sympathy for Dario Amodei’s “they just want to learn” worldview.

KL is a budget: log-probs of the new model versus the instruction-tuned reference, which usually stays fixed across rounds. Chat PPO often spends KL in the 10-to-20 range. Their GSM8K RL sits around 1. They are barely moving the model. Best-of-n sampling over a reward model spends even less. Ask the model to include a word and write at least 100 words, run RL too long, and it prints the word a hundred times and stops. Inference-time compute, he says, is a close cousin of RL: sampling more completions from a prompt is exploration; test-time methods are about how you sample, reweight, and interrupt to find a high-value finish, Monte Carlo tree search included. Anthropic’s logit-level “are we confident?” interventions sit on that side of the training-versus-inference line.

Leave RLVR running on math far past the point of usefulness — other evals have already tanked — and the chain of thought starts to stop, redo a step, and say “wait, let me check that.” It is the same verbal tic OpenAI showed with o1. Lambert is careful. This is n=1, the model is overtrained, ordinary skills are borked, and people have seen Llama do something similar if you crank the temperature. Reflection 70B, in principle, is a related idea. They did not SFT on o1 traces; they cannot get o1 traces. Two RL leads, Hamish and Costa, found it by poking at generations, which is what you do in RLHF anyway to make sure the model still works. He tweeted it because the tinfoil-hat o1 crowd would love it.

His read on o1 is not mysterious. Officially it is a large-scale RL system specializing in reasoning. Labs are almost certainly doing versions of this, especially on code. He expects Anthropic and Google equivalents within months, not a secret architecture. They probably mix deterministic and learned verifiers, train on many domains at once, and run much more RL, with hacks. It is a breakthrough. It is not smoke and mirrors. Superhuman code, Labenz and Lambert agree, is the obvious early win: fast, objective feedback. Superhuman poetry is ill-defined and will stay a noisy, slow signal.

The open-source problem is fuel. OpenAI does not release o1 reasoning traces, and human annotation is too expensive for AI2. A few days before recording, DeepSeek showed DeepThink, an o1-style model that displays its work and is slated to be open-sourced — enough, Labenz says, to make anyone who thinks Western labs will cruise to AGI ahead of Chinese ones sit down. Lambert’s reproduction sketch: generate seed traces that look right, have a language model edit and continue chain-of-thought, seed a model, then RL on domains you can verify. The feedback loop is the hard part. Existence proof is already on the table, and Llama 405B is still a usable open-weight reasoner. He will not take the “open o1” GitHub repos seriously until 2025, unless the name on them is Google or Anthropic.

The team did not need strict delegation at ten to fifteen people; past that, Lambert says, it becomes a mess, and adding managers has its own cost. He is last author in the academic sense and first in the industry sense, because someone has to hold the whole thing in his head. Four to seven people did first-author-scale work: data, experiments, the eval stack. Early on it was algorithmic: two people spent a stretch trying to make Llama-style rejection sampling work, a negative result, because an 8B’s own generated instructions were worse than the seed data. Others compared DPO variants. Then they killed the long tail — long context, multi-turn, anything without an eval they trusted — and went after specific data for the evals they were losing.

SFT mixed first, then a month of on-policy DPO: one person owns generations and the OpenAI judge, another trains, then it goes to RL. Model merging is the cheap trick that keeps paying. Run the same SFT mix on three to five random seeds, merge them, or just take the best seed — sometimes the winner is a single seed. Standups are described, in the paper’s philosophy section and in the room, as chaos: sit down, write updates, then fifty minutes of technical firehose. Explore in the middle, collapse entropy as the ship date nears. Llama cannot move that fast. Weights going out once or twice a year, final, legally and strategically loaded, forces a bigger investment cycle than OpenAI or Google’s API cadence.

The unmeasurable gap is character. Claude has one. Tulu 3 does not, because character has no eval. Pitch a four-month project to get the personality chef’s-kiss right and the CEO will ask what you are doing. Academia lives on evals. Anthropic’s leadership can just say this is what Claude is supposed to be. Most of the remaining juice, Lambert thinks, is still data: hyperparameters stabilize, and the amount of data you need to target a specific eval is not actually that high. Kill a capability when you lack an eval, because in a distributed project the eval *is* the source of truth. Labenz’s own recipe for a single-task fine-tune — ten gold examples, three iteration rounds, 10x the data if it is generally weak, a handful of patches for an edge case — still holds. Tulu 3 is what happens when the task is “general-purpose model.”

If you are not building a general chatbot, Lambert’s mapping is practical. One model per task is fine if your interface can always pick the right one. If you have a verifier whose distribution matches the job, the RL will kind of just work. AI2 has an early “LLM gym,” an open repo where you add constraints and train. That, he thinks, is the next moment in fine-tuning specific language models — unlocking verifiable RL for niches, not another round of single-domain SFT. Where the ground truth is mushy, LLM-as-judge and a preference set is the fallback. Preferences, he adds, are even more powerful in images, audio, and video, where human taste is more expressive; a detector that checks whether the nouns in a prompt actually appear is already a form of precise instruction following for pictures.

On-policy is not mysticism. PPO people like it because you score and update on the model’s own samples. In DPO you compute log-probs of the tokens in the completion and those numbers move the weights. If the tokens live in a weird region the current model has not seen, the signal is worse. Experiments back that. Closed and academic RLHF have forked: industry hill-climbs Chatbot Arena and retention, academia never will, because it does not have a hundred million users. Papers that show LLM judges beating humans on downstream evals, he thinks, are often not going deep enough; he still wants to see how annotator instructions flow into the final model. For “make the math number go up,” he will live without that. For anything that is actually a preference about how a model should be, the human-versus-machine bias still matters, and we do not understand it.

Will o1-style training end open source’s ability to catch up? He hedges and then does not. There is too much interest. The community is entering at o1 the way it entered at GPT-4, not waiting a generation. Llama 405B is still sitting there. The fine-tuning stack will not be enough; someone has to figure out the Transformers library for o1-like models. His Interconnects queue already has another o1 post. He will be at NeurIPS. The call to action is the datasets on the hub, the gym, and the same advice he has been giving since Tulu 2.5: stop arguing about the preference algorithm and go make better data.