JPMorgan Has 150 Petabytes. GPT-4 Trained on Less Than One.

Sarah Guo opens No Priors with a guest who started a company in her house. Alexandr Wang was a 19-year-old MIT dropout in 2016 when he founded Scale AI; eight years later the firm is the data foundry behind nearly every major large language model effort, including OpenAI, Meta, and Microsoft. Guo frames modern AI as three pillars — compute, data, and algorithms — and puts Scale on the one research labs and chip companies do not own.

Wang’s origin story is a college realization, not a market map. He was training image-recognition networks in the year of AlphaGo and TensorFlow, and he noticed the models were “very much so just a product of their data.” Play that forward, he says, and AI is three things: algorithms, already the work of labs like OpenAI and Google; compute, already Nvidia’s; and data, which had nobody. Over the long arc, data would only get more important. So he dropped out, did Y Combinator, and set out to produce enough of it to fuel the ecosystem.

AI, Wang says, is a general-purpose technology at the mathematical level — functions that can approximate almost any function, including intelligence — which is exactly why building around it is hard. The applications that gain traction keep changing. As an infrastructure provider, Scale’s burden is to think ahead: where will the breakthrough use cases be, and how do you “lay down the tracks before the freight train of AI comes rolling through.” The company has now been at that for eight years.

The first wave was autonomous vehicles. Scale joined Y Combinator around the Cruise acquisition and followed the early-startup rule of focus, building the first data engine that fused 2D camera frames with 3D lidar. That sensor-fusion stack became an industry standard. Customers included General Motors, Toyota, and Stellantis; for the first few years the company lived on driving and a handful of robotics cases, because that was the prime-time AI application.

Around 2019–2020, before language models, the next applications were unclear, so Scale went where applicability was obvious: government. It built data engines for geospatial, satellite, and overhead imagery that later fueled the first AI program of record at the U.S. Department of Defense, and the same technology mattered years later in the Ukraine conflict. In the same window Scale partnered with OpenAI on the earliest RLHF experiments on top of GPT-2. The models, Wang says, were rudimentary — “it truly did not seem like anything to us” — but the people were smart, so they stayed. That work ran through InstructGPT, then DALL-E 2 and ChatGPT in 2022 pulled the company’s center of gravity into generative AI.

Generative AI, Wang argues, is the first time the field has felt like a true general-purpose technology rather than a single valuable use case like driving. The infrastructure question then becomes data abundance. “Are we going to run out of tokens?” is a question he hears constantly. His answer is that it is a choice: the industry can pick abundance or scarcity, and Scale’s role is to build abundance. The bottleneck from GPT-4 to something like GPT-10, in his telling, is whether there will be enough tokens of the right kind.

Easy data — everything on the internet — is largely exhausted. Models no longer learn much from Reddit comments. What they need now is frontier data with a high supervisory signal: chain-of-thought from mathematicians, physicists, biologists, chemists, lawyers, and doctors; agent workflow traces from enterprise, consumer, and coding agents; multilingual coverage of the languages people actually speak; and multimodal and esoteric enterprise formats, including video and audio. The mandate is a means of production for new tokens, not another scrape.

Wang’s pitch to experts is leverage. A PhD, doctor, or lawyer who contributes high-quality data can move the model a little; integrate that over every future invocation and the impact is “near infinite.” Elad Gil analogizes it to Google’s original mission: scan the world’s information, except now the work is translating expert knowledge into machine-usable data. Contributors, Wang says, are paid, but the deeper motive is scientific — using one’s intelligence to fuel humanity’s progress. Gil adds an investor’s discovery: a decade of “digitization” did not capture the traces you would want to teach a model. Incumbent vendors have done the task for years and still do not have the data. Knowledge capture, he says, is only starting. He reaches for Dune: data production is spice production, the lifeblood of future AI systems.

Proprietary stores are the other well. Wang’s “crazy stat”: JPMorgan’s proprietary dataset is 150 petabytes; GPT-4 trained on less than one. Enterprises and governments already hold enough to train powerful systems. On synthetic data, Scale’s view is not pure machine generation. The critical design is hybrid human-AI data: models do the heavy lifting, and the best reasoners inject insight so the output stays high quality and high fidelity.

Guo pushes on the obvious objection: models are already better than humans on many measured dimensions, so why keep humans in the loop? Wang reframes the question. It is not whether a model beats an unassisted person. It is whether a human plus a model produces better output than the model alone — and he thinks that will be true “for a very, very, very long time,” because human intelligence is complementary to the machine intelligence being trained. He once debated Gil on whether “centaur” play holds; the artifacts of current models are his bull case for humanity.

The artifacts are famous and strange. Models can do ROT13 and fail at ROT8. They suffer the reversal curse: taught that A is B, they cannot say who B is. Those glitches, Wang says, show that this intelligence is not biological intelligence, because it was trained by a very different process. In practice, the human’s job is to critique a response, flag factual and reasoning errors, and guide the model over a long period toward chains that are correct and deep. Expert-plus-model teaming is what Scale spends its time on.

Gil cites Google’s Med-PaLM 2: on cardiology questions, the model beat the average physician as ranked by physician experts, though a cardiologist still won. How long until human expertise is no longer additive — three years, or never? Wang says never. Biological intelligence is built to reason and optimize over a lifetime, including reproduction; current models are trained on “little nuggets” of that intelligence. They are a shot glass of human intelligence, intense in a sip and bad at continuing it over a long horizon. Architectural breakthroughs will lift performance, he grants, but the models are not trained to optimize amorphous goals over long time spans, and the industry does not have the environments to teach that. He calls it a somewhat fundamental limitation.

Guo pauses to congratulate Scale on a billion-dollar raise at almost a $14 billion valuation, with strategic investors including AMD, Cisco, Meta, and Intel. Wang’s explanation is mission, not trophy hunting. Scale wants to serve the whole AI ecosystem, so it brought along other infrastructure builders laying groundwork — Intel and AMD — and key industry players. He thinks in a three-layer stack of infrastructure, technology, and applications, and wants the data foundry to empower every layer. Compute is already drawing tens or hundreds of billions of dollars; data production, he argues, needs real investment of its own if the field is going to get from GPT-4 to GPT-10.

Trust is the other half of the loop. The AI life cycle, as Wang draws it, is data in, train, evaluate, repeat. Measurement builds confidence and enables further development and adoption. Governments need to know systems are safe and fit for a country; enterprises need to know an agent will not create consumer risk; labs need a consistent read on the intelligence they are building. Guo asks what makes evals hard. Grading a system meant to approximate human intelligence is not easy to automate. You are building IQ tests, which is already a philosophical mess, and the practical mess is worse: academic benchmarks are “fraught with issues,” and many models are overfit because the tests sat in the training data.

Scale’s response was GSM1k, a held-out math eval that could not have been in any training set. Some models did well; others did much worse than their reported scores. The longer program is public leaderboards, built with the same expert minds, plus platforms so governments, enterprises, and labs can evaluate constantly. Upcoming: private held-out evals for leading LLMs, rerun every few months, starting with math, coding, instruction following, and adversarial capabilities, then more domains. Wang’s metaphor is an Olympics for LLMs, except the games are every few months instead of every four years.

Wang has a rare view of the application layer because Scale sits under so many of the builders. After GPT-4, he watched a frenzy of agent companies and app build-out, then a classic hype cycle. His read is that GPT-4 as a model was a little early for that wave. The community quickly found its limitations, and he thinks it was “probably a few generations too early” for a thousand flowers to bloom. The trough of disillusionment, in his forecast, ends when future models finally have the fundamental capabilities agents need. GPT-4 is not the terminal model.

What he wants application builders — enterprises, governments, startups — to install is self-improvement. Large labs already run a data flywheel: new data improves the model, evaluation finds the weaknesses, the workflow gets hydrated again. Every organization will need that loop inside its own products, and it is hard to build from scratch. Scale’s generative platform is the company’s attempt to lay that groundwork. The enterprise mistake is treating the mountain of data as equally precious. Frontier data, Wang says, can be 10,000 times more valuable than run-of-the-mill enterprise data. Papers out of Meta, he notes, show that narrowing the data can produce better, smaller, cheaper, faster models. The work is compression: filter a scattered mountain down to the slice that actually moves the model.

On the government side, Scale is shipping agentic features inside Donovan, which Wang describes as an “AI staff officer”: a full application that plugs into whatever model the customer thinks is appropriate. The valuable cases are often boring — writing reports, filling forms, moving information from one system to another — and well within current capabilities. For enterprises, Scale thinks at the platform layer rather than owning the apps: help them build self-improvement into whatever they ship. Asked whether GPT-4o and Google’s Project Astra change his view, Wang sees two signals. Multimodality still has a huge data gap, because internet-scale easy data is gone and personal agents are hungry. And the demos were “shockingly similar.” Either smart people independently arrived at the obvious next step, or competitive intelligence is pretty good; he thinks both are true. Gil, a former Google PM, recalls spending two years on a product only to watch a rival ship the week of launch. These are not one-week copy jobs. What disappointed Wang is the direction: multimodality is a lateral expansion. The industry needs smarter models — GPT-5, Gemini 2, whatever the names — so more applications can actually be built.

Quickfire: something Wang believes about AI that other people do not. “The path to AGI is one that looks a lot more like curing cancer than developing a vaccine.” You have to solve a bunch of small problems with little positive leverage from one to the next. Zoom into each cancer independently; over a multi-decade frame you look back and realize you built AGI and cured cancer, but the road was plotting. The industry narrative he rejects is a single “boop” — we solve it in one fell swoop. That picture, he thinks, is wrong, and the implication is bullish for society: slow, consistent progress, with time to acclimate.

Gil asks what “solve a problem at a time” means in technical terms — multi-step reasoning, Monte Carlo tree search, scaling walls. Wang nails it to generality. He sees very limited generality coming out of these models. Even across modalities, his understanding is that there is no positive transfer: training on a lot of video does not help much with text, and vice versa. Each niche of capability needs its own data flywheel. On the popular story that video yields a world model, he is blunt: “great narrative,” not strong scientific evidence yet. The base case is that models do not generalize that much, so AGI is the slow accumulation of lots of little problems solved.

Last question, as CEO of a scaling organization: what is he thinking about? It almost sounds cliché, he says, but how early this still is. Markets look crowded. Tech giants are investing, launches never stop, the obvious use cases are packed. And yet the technology is “one 100th or one 1,000th of its future capability.” There are many more chapters in the book. The organizational quality he wants to protect is nimbleness — staying able to adapt as the technology keeps developing.