Every Frontier Lab Is Spending a Billion a Year on Data. Here Is Where It Goes

Nathan Labenz frames the hour around a deal that had just rearranged the industry. Meta’s roughly $15 billion arrangement with Scale AI brought Scale’s former CEO, Alex Wang, over to lead a superintelligence team. Other frontier developers, Labenz says, were left scrambling to lock down their own training-data pipelines, and the market was still realigning. Labelbox, founded in 2018 by Manu Sharma, sits on that same chain: a vertically integrated “data factory” that supplies frontier training data to Western labs and to enterprises that still fine-tune for specific tasks.

Labenz notes, as a disclaimer, that Labelbox would be sponsoring the show for at least a month. He also says he would have wanted the conversation anyway. Sharma has been in the human-data business for years, and the front line, Labenz suggests, is currently chaos: people leaving Scale, Zuck doing a “weird deal,” and a lot of confusion about who will supply what next.

Sharma’s report is less panicked than excited. He calls it one of the most intense stretches of innovation the field has seen, an AGI race among large labs and smaller teams betting on alternative techniques. Underneath the noise, he says, frontier systems still need three ingredients: compute, which is drawing enormous capex; AI talent, the people who know how to train the networks; and data, which is getting a new kind of emphasis because so much of the work has moved into post-training.

Sharma walks through the training stack the way a supplier sees it. First you pre-train a base model on whatever of the web and other human-generated sources you can get, so it learns the patterns of data produced over something like a hundred years. Then comes post-training, which is where those models become the assistants people actually use. The techniques inside that second stage have already turned over several times in three or four years.

When Labelbox started in 2018, supervised learning was the main event: humans around the world tagged images, video, and snippets of text so models would mimic that behavior. Transformers bent the field toward unsupervised learning on large-scale datasets. Then RLHF, plus supervised fine-tuning, became the way to make strong base models useful for knowledge work. Now, Sharma says, nearly every lab he knows is emphasizing datasets for reinforcement learning — a flavor of the DeepMind AlphaGo era, reappearing in a new form.

All of those regimes need specialized data, and a lot of it. He still puts the field in the early days of making systems capable on the everyday knowledge-work that actually runs the economy. Labs know they need production lines for those specialized domains: what software engineers do all day, how healthcare and insurance workflows generate data. Mergers and supply-chain reshuffles are, in his view, a normal feature of that scramble. Labelbox has been behind the scenes for most of the labs; workloads shift, sometimes quickly, as teams reorient to new suppliers.

A year earlier, Labenz says, the folklore was that 98 or 99 percent of compute went to pre-training, with post-training a small, rapidly iterable tail. That seems to have changed, though not uniformly, and he is never sure whether the leading providers are converging or diverging. He offers an anecdote from Claude 4 training: a typo in a config file omitted a dataset meant to teach the model not to follow harmful system prompts. The model started following them. Anthropic, as he understands it, did not go back and redo all of post-training. They patched it later. From that he infers that post-training has become a large enough share of the compute that even a safety-conscious lab would not casually rerun it.

Sharma will not share client secrets, but he is confident the leading labs and hyperscalers are innovating across the whole end-to-end recipe. Pre-training still takes a long time — months for a very large model — and that duration is a given in a release cycle. Some of those giant base models, he notes, are used as teachers for the student models that actually ship. The budgets that are growing, though, are going into post-training. Base models already understand patterns of data across modalities. What they still cannot do reliably is useful long-horizon work. That has to be taught.

Coding is the running example. The models are already impressive, and still nowhere near developing entire product suites with minimal supervision. Research teams collect the right datasets and run reinforcement learning. Two macro domains dominate because they are verifiable: coding, where a pull request either meets the objective and the tests pass or it does not, and math, where a numerical answer can often be checked. The system tries millions of attempts; a correct one scores well and the weights move. That, he says, is why reasoning and coding jumped in the last nine months. Coding is lucrative knowledge work, and some labs believe that solving it also accelerates their own AI research. Anthropic, in his reading, is clearly in that camp.

The harder frontier is the non-verifiable regime that covers most of the day: you may not know the right answer, and there may be several. Teams need ways for RL systems to score a candidate solution. From the outside the models look as if they are converging — they interact and feel the same. Sharma’s philosophical test is that you have to be expert enough to critique quality before you can see the split. He codes with assistants now and can feel when he has hit a ceiling. Gemini, in his hands, is the one for algorithmic, math-heavy problems. Claude keeps asking questions, reading more files, and trying to make a better architectural call on refactors and distributed systems. Labs still talk about a generic system that is great at everything. In practice they optimize for where their users are: Anthropic on coding, others on consumer tasks or a general social assistant. Whether the best coder and the best scientific researcher ever collapse back into one world model, he says, remains to be seen.

Labenz wants a clearer account of what human data is still for. The DeepSeek R1 paper, he notes, showed that in verifiable domains like math and coding you can get strong learning without human reasoning traces. The downside is strange behavior: language switching, and more oddness if you detach from a human baseline and just spin the RL centrifuge. Human data, on that reading, is an alignment anchor. He also contrasts GPT-4.5’s trivia knowledge with o3-mini’s reasoning, and asks whether the old supervised-fine-tuning habit of recording the human’s step-by-step trace still matters once you are in RL.

Sharma’s first move is to say that we underestimate human intelligence. The world and the economy we built are extremely special. Models are already superhuman on some slices and still cannot do many everyday tasks. Math gave labs a clean dataset about logic in a scientific domain; that produced a reasoning behavior that is now a foundation. The next capabilities sit on top of it. His example is a future customer-service stack: you call about a Wi-Fi router, an AI concierge routes you, another specialist talks you through it, and the system reads internal databases and provisions a replacement. Tens of millions of users will ask in tens of millions of ways. A base model is only the starting point.

The architect of that agent, he says, needs a dataset that covers the distribution of those interactions. Off the shelf, you will pile on orchestration, agent frameworks, and still not get reliable output. You cannot invent those assets from purely algorithmic or synthetic methods. You can bootstrap and make the process easier with compute. You still need the authors’ quality judgment: how they want the system to talk, what personality it should have, when it should escalate. Those are design parameters, and they require new forms of data — voice, tool use, and the rest.

There was a period, starting the previous year, when labs tried to get academics to teach models how they think, step by step. Process reward models put real emphasis on that. What researchers found, Sharma says, is that it is just very hard to get humans to express their reasoning that way. The field has largely moved on: model the problem, model a solution, and specify how you would grade it. Reasoning becomes an emergent property of RL. You might still want some traces or audits. The framework he expects to dominate is: take a problem, solve it, and if you do not know the solution, at least know how to grade the AI on the task. Everything else emerges in pursuit of that goal.

Labenz restates the shift to make sure he has it. The old paradigm was supervised fine-tuning: the human owns the final answer and the “how did you get there,” and the model imitates that pattern of thinking in the hope that the pattern yields the right answer. The new one, maybe because that data was so hard to collect, is: give a really good solution, write a rubric, let the model take a shot, and have a grader model score the attempt against the gold example. The reward is something like the sum of rubric points. Sharma says that is right.

Then he adds the line that organizes the rest of the hour. In most domains you will not know the right solution. What domain experts in that industry do know is what is good, great, or excellent. That quality judgment has to be expressed in the data. He thinks of RL environments as gyms: you send the model in to practice a pile of tests and come out possessing a new skill. Games can be generic planning gyms; coding environments are more specialized. The hard part is crafting environments that actually represent the skill you care about.

The other workstream is autograders — specialized models aligned with how domain experts would score progress. Until those graders are rock solid, you cannot scale the RL training. Once they are, you can. That pairing, environment plus grader, is what he means when he says the data factory is no longer a sweatshop of five-minute tags.

Labenz has always wondered how you train million-token context. Dropping a million tokens on a person and making them responsible for the right answer is almost impossible; he assumed the work was programmatic — needles in a haystack, then ten needles. Yet Gemini, in his experience, does not just find the anomaly. It seems to grok 500,000-plus tokens in codebases it has never seen. Sharma splits the work into two camps.

One camp is programmatic: bury hashes in noise and see whether the model can find them, a test of retrieval in a vast ocean. The other is real tasks. Labelbox is creating a lot of long-context datasets of that second kind. His example is a financial analyst synthesizing SEC filings, performance releases, and audio of earnings calls — ten documents, mixed modalities — not to answer “where is this fact,” which is a search problem, but to build a model of the company and forecast earning potential. A good human might take days. Models can do pieces of it and are not yet reliable across domains.

Internally they call this multi-hop capability: the hard part is connecting sources and making a judgment, not finding a sentence. Video and other streams eat token budget once they are tokenized; Gemini is strong at “which frames did this happen,” and still needs work when the task is an actual industry workflow. The valuable data, Sharma says, is the kind that induces failures across all the frontier models. Labs pay for that because they can hill-climb on it.

Labenz describes how he used to fine-tune: ten examples by hand, a hundred queued for the model, reject or correct the misses, fold them back, iterate until the success rate was acceptable. Sharma treats that loop as still the right way to build an eval set. The surprising part, in enterprise, is how many teams get it wrong — because the hardest step is stating the workflow they actually want to automate. Best teams start small and add to the evaluation set. They keep finding unknown unknowns. Iterative development, he says, is as true when you are producing datasets as when you are writing code.

He does not think fine-tuning as people knew it four or five years ago — take a base model, dump in company sensor data, make it superhuman on one use case — is going to last. In 2025, fine-tuning at best helps you make the model efficient. If a narrow task runs at millions of queries and a large model is too expensive, generate data from the large model and distill into a smaller one: same quality, a fraction of the cost. A second bucket is capabilities so unique the base model simply does not have them. That is a much smaller set than companies think.

Google’s Med-PaLM, trained on healthcare-only data, is his exhibit. He had been reading that base Gemini 2.5 already outperforms that hyperspecialized medical model. The implication is that reasoning learned from vast other data transfers into the domain. Enterprises that believed petabytes of proprietary data would buy them reasoning were, in his view, often wrong. Labelbox does not see as much fine-tuning among customers now. What they see is context engineering: not just the prompt, but the retrieved files, snippets, and structure around the query. Cursor is the template. It works because the base model gets directory structure, relevant files, and similar functions. That framework, he says, ports to any domain: name the task, gather the relevant information, feed it in.

Foundation models are racing to absorb capabilities that app layers currently supply with context. Until they do, engineering the context is the most effective way to ship a goal. Labelbox still fine-tunes for itself, in two cases: cost, and a unique opinion about how judgments should go. The rest of enterprise has moved to multi-step agentic trajectories, retrieval, and tool use — software engineering plus domain mapping — which is tractable for companies that will never become ML shops. That, he thinks, is one reason the industry took off so fast. On which weights people still train, he sees three open-source families: Qwen, Llama, and Mistral. Highly specialized industries, medicine and defense among them, prefer open weights because of air gaps and on-prem constraints. Digital natives such as Airbnb and Pinterest use the strongest models from their cloud provider or OpenAI. Fine-tuning demand, in other words, stacks in layers: compliance and deployment, cost and latency, and only then capability customization.

The concrete fine-tune Sharma will own is hiring. Labelbox works with millions of domain experts — physicists, mathematicians, Olympiad-level software engineers — across more than seventy countries. Ten years ago you would have written tests. That does not scale. Instead, candidates sit a roughly thirty-minute conversation with an AI that has read their resume and papers. A second system scores the interview and asks a specific question: will this person produce datasets that can teach a frontier model new capabilities?

A base model, he says, has its own idea of what a good producer looks like, and Labelbox does not share it. They close the loop with outcomes from experts already on the platform and fine-tune the grader. That is his clean example of when weights are worth changing: the foundation model lacks a particular quality judgment, and you have a closed data loop that can calibrate it. They call the interviewer Zara. They run well north of 2,000 sessions a day; at the moment they were recording, about fifty people were in an interview. Average satisfaction, when contributors are asked how it went, sits around 4.6 to 4.7. People like that they can do it on their own time, that it is patient, and that it will ask about a nuanced claim in a paper a human recruiter would never reach in thirty minutes.

Contributors asked for practice runs to prep for real-world jobs, so Labelbox shipped that even though it is not their market. People also cheat. Sharma has seen an iPhone playing ChatGPT advanced voice pointed at another device so two AIs can talk to each other. He takes the ingenuity as a compliment and treats staying ahead of it as part of the assessment problem. Another surprise: the interviewer can judge native fluency across languages. Multiple voice providers sit behind the scenes; the scoring stack is the part they fine-tune, and every new interview plus later expert performance feeds the data engine that rewrites both the grade and the format.

Labenz asks for coaching on his own context-engineering problem: every episode he attaches a transcript and a pile of past intro essays, with a prompt to adopt their tone, and Claude drafts something he still edits. He does not have the source transcripts for those old essays, so he has outputs without input-output pairs. The draft usually gets cadence and his mix of enthusiasm and fear. What it misses is the unique, now-relevant heart of the conversation. Sharma’s advice is to keep context-engineering — missed examples, rules, extra tools for guest background — because a new base model arrives every couple of weeks, and assistants are starting to keep memory. Fine-tuning that workflow now would freeze an investment just as models are getting better at the empathetic, 4.5-like qualities the essays need. In a creative job where Labenz wants to remain editor-in-chief, fully automating may not even be the goal. Mundane work is where he would still consider a fine-tune.

Labenz wants the split between frontier labs and everybody else. At the top he has heard $300 an hour for physicists and biologists. For a pharma company he naively assumes you would want your own people, not a global network. Sharma says Labelbox now works with almost every frontier lab, and the appetite is insatiable: novel, specialized RL datasets that teach longer-horizon tasks across more than math and code. Demand is rising. To give a sense of scale, each frontier lab is probably spending over a billion dollars a year on data.

Labs try several production modes, including hiring contractors through a new wave of staffing agencies that Silicon Valley investors have suddenly found exciting. More often they want high-quality data delivered fast, which means operating a factory: a vetted expert network plus the tools to produce hybrid datasets that mix synthetic methods with human experts. Sharma asks Labenz to guess yearly earnings for a top-quartile AI trainer, assuming full-time hours. Labenz had quoted $300 an hour as high, $100 as plausible, then discounted to $140,000 because the hours might not be there. Sharma says it is a good guess, and wrong. Top contributors are earning well north of $250,000 a year.

The distribution is a power law. It asymptotes around $40,000 to $50,000 a year for other domains and countries. The most specialized people are above $250,000, and he expects that to rise as the frontier goes deeper into agents and business workflows. Five years ago the average hourly rates for this kind of data work were much lower. The tasks are no longer five-minute tags. These people are creating RL environments, developing solvers and verifiers, or playing in the environments to produce an instance of what good activity looks like. Some do it part-time; some have changed their lifestyle and gone full-time, working when they want.

Enterprise is a different factory. Labelbox also sells the software platform. Customers include large pharmaceutical and robotic medical-imaging companies that have to use their own experts. Medical coding in the United States is an industry unto itself; you either tap people from that world or train new ones. Teams that get it right find the leverage points and then scale a workforce. In those cases the customer runs the factory: they build process on Labelbox’s tools, and they staff it. Some want a Palantir-style forward-deployed expert to help design the process. High-tech AI startups usually want full control. Other industries take more help.

Labenz has a theory about sovereign AI. Countries that want models more fluent in their language and values might, he has argued, skip the data-center race and simply collect national data, then hand it to frontier developers — a Brazilian or Indian dump, please use this, be better for us. He asks what Sharma would tell the president of Brazil, Mexico, or India. Sharma sees a leapfrog chance to overhaul services, but he will not take the shortcut.

Governments differ: some own the full stack of citizen services, some write a spec and let private firms deliver. Most still offer something. He thinks they can render those services as a much better experience at a fraction of the cost, finally leaving Fortran, COBOL, and the other legacy that DOGE-style audits keep finding. India, in his sketch, has huge numbers of citizens who need basic knowledge of what amenities and services exist in their town. The investment should focus on outcomes — the experience you want — and map backward to the efficient way to get it, private or public. You still need the three components: compute, data, and talent. No shortcuts.

Labenz asks where the silver-platter dump breaks down. Sharma’s objection is the same one he has for companies that say they will invest in fundamentals and figure out use cases later. He is not sure that helps anyone achieve an outcome. Walk backward: the hard part is the intention, the outline of a great service, and the funding. Once that is clear, producing datasets and evaluating systems is tractable. Projects fail, more often than not, because they acquired a pile of technology to see what it could do. Labenz, who does ad hoc AI consulting, agrees: he would rather spike one working problem than build an “AI platform” first.

Before Labelbox, Sharma worked at Planet Labs, now public, which operates 400-plus satellites in low Earth orbit and scans the entire Earth every day. Brazil wanted to know the rate of Amazon deforestation; there was no other practical way to see it weekly. Planet could image the country daily and run computer vision. It would not have made sense for Brazil to fly its own constellation just for that. Some capabilities belong to specialist companies, with government doing integration and policy. Others — a more intuitive way to file taxes, a conversational front door to a national health service — are where a government is uniquely placed to curate the experience. He expects private or open-source foundation models to sit underneath, with sovereign applications or custom fine-tunes only where the state can gather data companies cannot.

Labenz’s remaining irritation: why has nobody paid him to install software that watches him use a computer all day? Operator is decent; watch an AI village and it still stumbles through UI crannies. Sharma says it is coming. Labelbox already powers a lot of the computer-use agents labs are building. The first useful versions are in their first innings, with a *Her*-like companion that can listen, see through a phone camera, and, if you turn it on, see the screen. Understanding of text and common UI is already good. Real domain work fails in unintuitive ways. Three minutes in, the model has lost the goal from minute one. Spatial understanding is poor; geometry on screen goes wrong. Time horizon breaks the chain. Premium products, he would not be surprised to see, will use those sessions to train. Labelbox is already producing the sessions at large scale, across languages. Capabilities are being cooked. Privacy will be a user choice.

Would a generalist like Labenz even qualify? Sharma says the doctor-and-lawyer talk is mostly for new audiences. Internally they look for high agency and IQ, which correlates with being able to do generalized long-horizon tasks better than the models — which means you can produce signal. He will let Labenz in. On the industry’s next chapter, after the Meta-Scale pairing, Sharma declines to forecast a string of lab-and-factory marriages. His picture is that they are slicing aspects of human intelligence into AI, and he argues there are probably infinite slices left, because we still do not understand the brain and have not built systems that work like one. Every paradigm since three-neuron MATLAB nets has come back to data as the essential ingredient. Teaching sophisticated knowledge work in RL environments is a long road. Synthetic data will always be a pursuit; it still needs human grounding, because the systems have to interact with people. Humans, in his preferred future, stay in the driver’s seat, commanding millions of AIs through data. Labelbox wants to be the infrastructure for use cases that have not been invented yet.

Labenz’s last question is the AGI-versus-takeoff one. A human-level coder, copied in parallel, might find architectural innovations faster than we have; a million remaining slices of data points more toward a plateau around the best humans, which he would prefer for safety. Sharma says both things are true. We are already in an accelerated takeoff, even as people complain the AI has not done their job yet — five years ago the same demos looked like the far future. He expects something more synergistic: on the order of thirty million developers now, and perhaps billions of people coding with assistants over the next few years. The philosophical remainder is quality judgment. Humans know when something is good and often cannot say why. *Zen and the Art of Motorcycle Maintenance* is the book he reaches for. So far we impart that judgment with rubrics and grading. A year or two from now the techniques will be different. If someone figures out, across a vast space, what makes things excellent, good, or bad, progress gets even faster. Taste, Labenz suggests, is the domain to watch. Sharma leaves it there.