Meta Trained a 405B Teacher on 16,000 GPUs — Then Told the World to Distill It

Stephanie Zhan and Sonya Huang of Sequoia Capital's Training Data have Joe Spisak in the room two days after Meta released Llama 3.1 405B. He is director of product management for generative AI, the person who leads Llama and the third-party ecosystem around it, after a decade that included PyTorch and work that spun into protein-folding and AI-math startups. The question the hosts want answered is not only what the new model can do. It is why Meta trained it at this cost and then put the weights on the internet.

The numbers in the show notes and in Spisak's account are the same order of magnitude: more than 15 trillion tokens of pretraining, training runs on over 16,000 GPUs, a model anyone can download. The license now lets you train on its outputs. The generation that burned the most money is not, in his framing, a product Meta intends to meter. The hour is an attempt to show the rest of the ledger — what Meta gets back, what founders should stop trying to replicate, and what a 405-billion-parameter network is actually for.

Spisak calls the 405B a monster and a great model, then immediately names the lesson they learned: it is a massive teacher. That was the plan. A big model can improve small ones through distillation, and that is how the 8B and 70B became as strong as they are. In April, Meta put out what he still calls an appetizer — Llama 3 at 8B and 70B — and he wondered whether people would actually be that excited. The response was through the roof. The main course had to be held back until July.

The new capability he is most excited about is zero-shot tool use — calling Wolfram, Brave Search, or Google Search, running a code interpreter, building a plugin for RAG — without a special training pass for each tool. Long context was the community's loudest request; internally they have even longer than what shipped. Multilingual is table stakes for a company with billions of users, but they did not just pretrain on a pile of languages and declare victory. They spent post-training and supervised fine-tuning, plus safety work, on making those languages actually good.

The license change is the other half of the teacher story. Closed models left a gray zone: maybe you could use the outputs, maybe you were being slightly unscrupulous. Meta had many meetings with Mark Zuckerberg and landed in a place that encourages people to use 405B outputs for distillation and synthetic data. Percy Liang at Stanford, Spisak says, texts him constantly asking when it will be out and whether he can use it. Nvidia and AWS are already building distillation recipes and synthetic-data services. Meta knows the data is good because it already used that data to lift its own smaller models.

Huang has read Zuckerberg's letter and still wants the business question: what is in it for Meta? This is a massive investment, a state-of-the-art model offered free, money left on the table. Offensive or defensive? Spisak starts with the business model. Meta does not depend on this model for direct revenue. It is not a cloud company and has never sold a cloud service. The template is the five years he spent helping lead PyTorch: they never built a hosted product around it either, even though they probably could have, and it would have been weird.

PyTorch, in his phrase, was a lingua franca, a bridge into a high-entropy neighborhood of innovation. Architectures showed up weekly; people published code and models; Meta pulled them inside, evaluated them, and used them. Integrity work such as the Hateful Memes dataset improved week over week because the community was building on the same stack Meta already ran internally. Openness is how you harness the world's work instead of watching it happen on someone else's tools. Llama, he says, is the same bet.

The returns he names are concrete. Safety: academia and companies red-teaming and jailbreaking Llama is something Meta wants, because bugs surface faster — the Linux analogy, transparency as a path to a more secure system. Localization: French is not one language; there is Canadian French, France French, Cameroonian French, and machine translation will not bind a language to a culture at that grain. Product: the end of the story is better Meta products, which is where the company actually monetizes. There were competitive arguments against releasing the technology. He is less worried than he used to be, because the cadence is Llama 1 in February, Llama 2 in July, Purple Llama in December, Llama 3.1 the following July. Move fast enough and yesterday's weights are not the ones that matter. Zuckerberg's word in the letter was "standardized." To make Llama ubiquitous you have to take down the artificial barriers.

Zhan pushes the industry question: with Llama 3.1 out and OpenAI shipping GPT-4o mini, do models commoditize even at the frontier? Spisak points at the last two weeks. GPT-4o mini is, on his numbers, about 15 cents per million input tokens and 60 cents out — absurdly cheap, and still an excellent model, a distillation job he credits OpenAI with doing well. In the same window Meta pushed compelling models across the size spectrum. His judgment is that the model itself is rapidly becoming a commodity. Enough data plus enough compute gets you a first-order approximation of the state of the art. The value moves elsewhere.

Elsewhere, for Meta, is Meta AI, agents, and whatever they put into Instagram and WhatsApp — the places they actually add value and make money. The model will keep getting new modalities, languages, and capabilities; that is what research is for. Founders staring at the same stack get a harder message. There are companies still pretraining foundation models. He cannot say what Llama 3 cost, only that it was very expensive, and Llama 4 will be more expensive. If he were a startup now, pretraining from scratch would not make much sense. He would adopt open source.

Adopting it, he says, forces a look at the engineering complexion of the company: people who can do LLM ops, data fine-tuning, RAG. APIs will let you do some of that, but ultimately you want control. "Your moat is your data, your moat is your interaction with users." You may want a mixed architecture — simple, low-latency queries on device, complex ones in the cloud — and open weights are what let you split the system that way, modify the model, run it, distill it. Distillation services that take your weights down to something smaller are only just beginning.

Then the trap he has watched people walk into. Many API fine-tuning services use LoRA. You bring your data, they train, and at the end you do not get the LoRA weights. You are stuck on their inference. You have given away the data, you do not own the IP that data produced, and you have to keep paying for someone else's runtime. "That's not a good deal." Open source, in this contrast, is ownership of the weights. The moat you still have to dig yourself is the data and the user relationship.

Mistral Large landed about a day after Llama 3.1. Spisak's feelings are personal: it was one of his old teams at FAIR, working on theorem proving and AI for mathematics. Guillaume Lample, Timothée Lacroix, and the group around them were, he says, one of the scrappiest teams he has ever worked with — they barely slept. By day they pushed AI theorem proving; by night they were grabbing compute to train what became Llama 1. He guesses they sleep even less now. He thinks they do good work, and he thinks they are structurally challenged: open-sourcing models while trying to get paid for hosting them. GPT-4o mini does not help. Their own ecosystem — Together, Fireworks, Lepton — will host the same weights, sometimes cheaper per million tokens. That, he says, is why a research-only license starts to make sense. On paper Mistral Large 2 looks strong; on Artificial Analysis's blended score it sat under a 70B. He has not evaluated it internally yet. His stance is the more models the merrier. In generative AI you wake up expecting a new release. That is the fun of the job.

Are frontier labs running the same recipe? He points at the Llama 3 paper, 96 pages, which Lawrence and Angela on the team spearheaded. Writing it, someone on the team said, was harder than developing the model. The honest read of Llama 3, especially the 405B, is that they took relatively little research risk. They pushed scale. Grouped-query attention to ease the quadratic cost of inference. More than 15 trillion tokens. Synthetic data in post-training that lifted the smaller models. Training runs on over 16,000 GPUs, which they had not done before, and which is not a slogan. GPUs fail. "I'm just going to train on 100,000 GPUs — good luck." You need a serious infra team and an ML-systems team. Everyone says it is easy. It is not.

He compares the mentality to the GPT-3 paper and to Tom Brown, now at Anthropic, being first author because the innovation was scale: take a known architecture and push it as hard as the systems layer will go. Internally they are looking at other architectures, including mixture of experts, for Llama 4. Architecture still has to evolve; AI21's Jamba, mixing Transformer and Mamba, looks good on benchmarks and was not scaled past something under 10 billion parameters, so nobody outside the labs knows whether those gains hold at 100 billion. Academia is full of brilliant people without the compute to find out. The framework he offers is a trade: research risk versus deterministic execution. Risk is how you buy a higher ceiling.

Is model development becoming more like software? He uses a Llama-train metaphor: new capabilities board the next release the way features board a train. Safety is already componentized that way — Prompt Guard and a new Llama Guard shipped and can iterate on their own. The core model is harder. Product wants to be deterministic; research is allowed a non-zero chance of blowing up in your face. "If it fails, it's research." They will keep executing so Meta AI improves on a cadence, and in parallel they will have to push architecture, because at some point you hit a theoretical limit and the recipe has to change.

Huang asks about agentic reasoning: how he defines it, and whether it falls out of pretraining scale or still needs post-training. Reasoning, he says, is a loaded word. The internet examples are the semi-gimmicky bus problems that force a model through multiple logical steps; LocalLLaMA is full of them. His direct answer is both: reasoning improves in pretraining and in post-training. The finding that now sounds like common sense, and that the field only really absorbed in the last year, is that a lot of code in the pretraining corpus makes the model reason better. Of course it does, once you say it: code is step-by-step and logical by nature.

Math is the same lever, and so are scientific papers. FAIR had a project, Galactica, trained on arXiv, led by Robert Stojnic and Ross Taylor from the Papers with Code team. It took a beating in the press. Spisak still thinks it was one of the coolest projects they did, and ahead of its time. Post-training is all trade-offs. You can crank a model toward code, as they did with Code Llama, and it will be great at code and worse at everything else. A general model is a point on a Pareto frontier. Anyone can pick a benchmark, super-optimize, and announce they beat GPT-4. Whether the model is as generally capable as GPT-4 or Llama 3.1 is a different claim.

The future lever for reasoning, he says, is obvious: more data, especially code and supervised data. The less obvious part is applications. Once you know the killer use case, you know which data to chase. That loops back to evals, which the community is saturating. Define a metric, optimize the living hell out of it, then watch a model with a better MMLU score respond worse in an actual conversation. He wants benchmarks with line of sight to real interaction. LiveBench, the Abacus eval he was looking at, and LMSYS's Chatbot Arena are imperfect and still more human than a static prompt set. Find the reasoning use cases that matter for the product, generate data there, and only then trust the score that moved.

Zhan asks what developers should do with the 8B and 70B. Spisak first goes back to the April appetizer: that 8B beat Llama 2's 70B by a large margin. He had to stare at the chart and ask whether the numbers were right; they were. The intuition was about 7x more data, plus a lot more compute — each generation shoving yesterday's large-model scores down into a smaller size class, with internal models already looking strong below 8B.

Then the jobs those smaller models are for. On device: Apple is talking about it, Google is putting Gemma and Gemini on Android, and privacy-sensitive data can stay on the phone — chat history into an on-device RAG with a mini vector store, local summarization of the last 15 messages without sending anything to the cloud. Safety classifiers are the second job. Meta's internal Llama Guard is built on an 8B, which is expensive for a secondary model, and they are already trying smaller ones because these are classifiers, not autoregressive chat, and can be pushed further. Edge and on-prem cover the rest: many tasks are already small-model work, with the cloud 405B as backup for the hard queries.

The data wall, in his telling, is not here yet. This release is the proof that synthetic data helps: millions of annotated synthetic examples in post-training, a lot of them generated by the 405B, plus paid human annotation. Data licensing is becoming an arms race — OpenAI is buying, Meta is buying, Google has YouTube — which is another way of saying large companies have an advantage that is not new. Are they at the wall? He does not think so. Schedule the same conversation in a year, he jokes, and he will save the calendar slot.

Lightning round. When will SWE-bench cross 50%? He will not give a year. If he has learned anything, it will be faster than whatever answer he gives, because as soon as a benchmark is the target, people figure it out. Will an open model surpass the closed frontier? Huang says they can retire that question; 405B is already in that class, and he agrees they are almost there. Will Meta always open-source Llama? Zuckerberg's letter is the commitment he points to. PyTorch to FAIR to Llama is years of open source, not a flash in the pan. He would never say never, and he also would not bet against the company or against Mark. He closes by sending people to the acknowledgements at the end of the paper, and wishing for Star Wars–style scrolling credits. Llama, he says, took a village.