GPT-5 Wants 100 Trillion Tokens. Does Earth Have Enough?

Training GPT-3 took on the order of a trillion tokens. GPT-4, by the estimates Labenz and Gannon will spend the hour unpacking, took about ten trillion. Follow that curve another generation and a hypothetical GPT-5 wants 100 trillion tokens, with a compute budget that jumps from tens of millions of dollars into the billion-plus range. Money is a separate argument. The more stubborn question is whether Earth even holds that much usable signal.

Labenz, an entrepreneur and AI investor who hosts Cognitive Revolution, treats the episode as an experiment: sixty minutes of order-of-magnitude arithmetic, meant to pin a listener's worldview to realistic ranges for data and compute. His guest is Nick Gannon, a master's-level data scientist by day and an AI scout by night, who has been counting what humanity actually produces. Email, Twitter, YouTube, genomes, telescope dumps, weather, census files, source code, even the text ChatGPT itself has already emitted — they try to put each source on the same ledger.

The destination is announced early, because the arithmetic is the point. For a GPT-5-scale run, data is not the bottleneck; the surplus is large enough to feel almost unfair. Stretch the same logic to a trillion-dollar training run, and high-quality text probably is not enough. The gap, they argue, is what synthetic data and self-play would have to fill. In a world where the White House has already set 10^26 FLOPs as a reporting threshold for training runs, Labenz wants people who claim to track progress to carry a few of these reference numbers in their heads.

Gannon opens with a warning about how people actually form views. Judgments of AI progress, he says, are too often crowd sentiment aggregated into a price: long or short, overvalued or cheap. That habit fills the model of the field with other people's opinions instead of facts. He wants a different frame, one that can explain what the hyperscalers are actually doing.

The paradigm at the core of OpenAI, DeepMind, and Anthropic, as he restates it from Ilya Sutskever, is a scaling hypothesis with two simple premises. First, a bigger brain is a smarter brain. Second, biological and artificial neurons are functionally on par: intelligence does not care whether the wires are carbon or silicon. If both hold, human-level systems are not a mystery of nature. They are a hard but tractable engineering problem, and data should be read through that lens of scale.

Labenz adds a personnel check. Sam Altman, Greg Brockman, Ilya Sutskever, Demis Hassabis, Shane Legg, Dario Amodei: none of them, as far as he can recall, treats a shortage of data as a fundamental barrier. Asked about it, they tend to say they will be fine, they can create what they need. Gannon then quotes J. Becker, an OpenAI engineer who builds large distributed clusters: these models are approximating their datasets to an extraordinary degree. Train long enough on the same data, and pretty much every model with enough weights and enough time converges to the same point. On that extreme reading, gains in data quality and in architecture are both ways of shrinking the scale required to reach human-level generality.

They put the anchors on the table. GPT-3 trained on roughly a trillion tokens, about 600 gigabytes of text. GPT-4, drawing especially on SemiAnalysis, is estimated as a 16-expert mixture-of-experts model with 111 billion parameters per expert, about 1.8 trillion parameters in all, plus a vision encoder trained on another two trillion image tokens. The text side is the number they will keep using: around 10 trillion tokens, at something like 10^24 to 10^25 FLOPs.

Chinchilla scaling says that for a given compute budget you grow parameters and tokens together. Ten times the parameters and ten times the data is a hundred times the compute, which matches the budgetary observation that each generation is spending about 100 times more. MosaicML once claimed you could reproduce GPT-3 from scratch for under $500,000; most estimates put the original closer to $5 million, and GPT-4's GPU bill for a single pre-training run somewhere between $50 million and $100 million. A 100-times jump lands in the billion-to-a-few-billion range, a figure NVIDIA's revenue and Meta's announced chip purchases already make thinkable.

Three caveats blunt the clean 10-times-data story. Chinchilla optimality is out of fashion; labs now overtrain, as when Llama 2's 7 billion parameter model was fed two trillion tokens, because extra tokens buy cheaper inference. You can spend a 100-times budget as 20-times parameters and 5-times data instead of 10 and 10. Long context is another downward force on data: GPT-5 is likely to train with something like ring attention, the mechanism speculated to sit behind Gemini 1.5 and Claude Opus's long retrieval. A trillion-parameter model at a million-token window costs about 5.6 times more compute than the same model at 4K, but the batches are so large that you make far fewer weight updates. Mixture of experts is the third. On a GPT-4 forward pass, perhaps only 270 billion of 1.8 trillion parameters fire — a company of 1.8 million people sending 270,000 out to work. There is no free lunch: MoE still follows scaling laws, but the slope is a little less slopey.

Gannon cannot resist a side note on how valuable those algorithmic tricks have become. It is astonishing, he says, that anyone publishes them. Someone could walk into DeepMind, ask for $10 million, and probably get it just so DeepMind could hold the secret a couple of months before everyone else. Bless the academics, he adds, for giving the secrets away. With all the caveats stacked, they still need to see their way to 100 trillion high-quality tokens.

In 2020, humanity created about 64 zettabytes of data, on the order of 10^22 bytes, almost all of it copies of copies sitting on CDN nodes. Set against 100 trillion words, that is a factor of 10^8. If one part in a hundred million of the raw pile is high-quality text, the GPT-5 budget is met. The pile is also still growing: global data roughly doubles every three years, so about half of everything ever produced arrived in the last three. Email alone is enough to make the point. At 10^11 messages a day, 300 tokens each, and 300 days a year, the annual volume sits between 10^16 and 10^17 tokens. One in a thousand emails being good would cover GPT-5. Access is the real constraint: the mail is locked in other people's systems, which is why OpenAI has been public about a data-acquisition team that shops governments and companies.

Twitter is a useful microcosm of mediocre internet text: about a billion tweets a day, on the order of 3×10^13 tokens a year, three GPT-4s and a third of GPT-5. The gap between email and Twitter — hundreds of times — is a reminder of how uneven the distribution is. YouTube and astronomy then skip four orders of magnitude, landing at one to two exabytes a year. Five hundred million hours of video go up annually; Hubble and James Webb are in the same weight class. GPT-4V bills a low-resolution image at 85 tokens against tens of thousands of pixels, a thousand-to-one compression. Apply that deflator to YouTube and a year of uploads is 10^15 tokens, ten times the GPT-5 target. Training on all of it, Chinchilla-style, would be a hundred-times jump past the hypothetical GPT-5 budget: hundreds of billions of dollars, a number that happens to sit inside the cash piles of Google, Microsoft, Apple, and Meta. Ten percent of YouTube being "good" would also do the job.

Genomics is larger still, about 40 exabytes a year and doubling every two years, almost none of it used. Labenz's concrete anchor is Evo, a 7 billion parameter life-science model trained on 300 billion tokens from 2.7 million public prokaryotic and phage genomes — no nuclei in sight. GPT-4 added two trillion image tokens to get vision. A DNA modality at that scale looks reachable, and he starts to daydream out loud: interleave sequences with proteomics, regulatory RNA, health outcomes, and scans, and you might get a system that speaks biology the way current models speak language and images. Humans have no native intuition for DNA. The model would not be stuck with that handicap. Weather is another reservoir: the world data center for climate holds about 600 GPT-4s of pre-training data, assembled over 37 years from satellites, aircraft, ships, buoys, radar, and balloons. DeepMind's weather model predicts hundreds of variables ten days ahead to 0.25-degree accuracy in under a minute. The U.S. Census Bureau is another few hundred GPT-4s. Apple's Vision Pro, Gannon notes, is a remarkable data-collection machine. In finance, Peter Brown — reportedly Geoffrey Hinton's first advisee — runs Renaissance Technologies, whose flagship compounded around 37 percent a year for decades, and has allegedly been recruiting large numbers of H100s.

The indexed web is only the visible universe. Google oscillates around 50 billion pages; estimates of the unindexed rest run from 25 times to 2,000 times that, a range Gannon compares to asking how big the unobservable universe is. Code is smaller in the stock but special in the flow. Hugging Face's The Stack is 6 terabytes raw and 3 terabytes near-fully deduplicated, about 30 percent of GPT-4 and 3 percent of GPT-5 — and you can generate more of it without limit. Synthetic text is already in that league. Size ChatGPT at 25 million daily actives, 15 queries per user, 1,000 tokens per reply, and you get something like 10 to 45 trillion generated words, enough to train a couple of GPT-4s. ChatGPT is already emitting more text than it was trained on, and closing in on a GPT-5-sized corpus. Every time the data question comes up, Gannon says, a sly smile crosses the researcher's face: don't worry, we have it figured out. That "it" is probably synthetic data.

Epoch AI's paper *Will We Run Out of Data?* is the splash of cold water. High-quality public text — books, news, scientific papers, Wikipedia, filtered web pages — is estimated at about 10 trillion words, the same order as GPT-4's training set, and likely exhausted between 2024 and 2026. A sniff test makes the number feel less like a guess: the Library of Congress holds about 10 trillion words, or 10 terabytes. If "good enough to live on those shelves" is the quality bar, 10 trillion is the scale you would expect. Total data is not the scarce resource. The pool that looks like books is already GPT-4-sized. GPT-5's 10-times jump might still be patched with multimodal data, private corpora, and synthesis. Past that, the path is less obvious.

They push the ceiling to the largest run civilization could actually fund. Investor Carl Shulman has put that ceiling near $1 trillion, about 1 percent of gross world product. Precedents exist, barely: at the height of Apollo, the United States spent about 2.2 percent of GDP. Apple's revenue is closing in on $400 billion; the U.S. Defense Department's budget is about $800 billion. Gannon's translations are deliberately crude: a trillion dollars is roughly 30 aircraft carriers, or 10 International Space Stations. Headlines about Sam Altman shopping a $7 trillion chip package in the Middle East stayed in the rumor mill, but Altman was talking to the Secretary of Defense about AI and compute as national-security inputs as far back as 2015. Gannon's judgment is blunt. A training run at that scale is a two-country problem: the United States and China.

Three trend lines put a trillion-dollar run between 2029 and 2033. The cost of the most expensive training run has been climbing about an order of magnitude every two years. Moore's law in the transistor-density sense is fading, but its spirit — operations per second, closer to Kurzweil's accelerating-compute curve — is still alive. Algorithmic progress is faster still: effective compute, meaning the FLOPs needed to hit 90 percent on a given test set, has been doubling about every nine months since 2012. More dollars, more compute per dollar, more effective compute per compute. The data appetite of that run is about 85 quadrillion points, 8,500 times GPT-4. The human brain processes about 11 million bits per second; over a 70-year life that is two to three quadrillion bytes, 200 to 300 times GPT-4. If intelligence is a data game, human-level cognition showing up as training data approaches a lifetime of brain throughput is not a crazy coincidence. A trillion-dollar corpus would be thirty to forty times that lifetime.

Ten to the 17th tokens is only about 0.13 percent of 2020's created-and-copied pile, another one-in-a-million-style surplus — until you remember that high-quality text is 10^13. The gap is four orders of magnitude. Labenz splits the budget from the top: a thousand-times compute jump from GPT-5 could be 10-times parameters and 100-times data, which gets you to 10^16, with the last order of magnitude borrowed from cheaper compute. 10^16 might be reachable. 10^17 looks like a stretch. Total data was never the real question. The high-quality fraction is.

If the open web cannot supply 10^17 good tokens, the remaining move is to make them. Labenz says Claude 3 changed his mind about how far a refinement loop can go. He is still uncomfortable with the picture of AI self-critique running all the way to a singularity, but the quality suggests "they've got that top spinning pretty tightly." Code is the obvious factory floor, because a program can be written, modified, executed, and checked on the fly. Gannon champions it as the best synthetic domain we have.

The only systems that have clearly gone superhuman, he notes, are DeepMind's Alpha series: AlphaGo, AlphaZero, AlphaGeometry, AlphaFold. The recipe repeats. A value network, Monte Carlo tree search at inference time, self-play, an update, more self-play. Anything you can put an Elo score on — chess, geometry proofs, "every game that could be played" — can be built that way, an AlphaGeometry and an AlphaFold for each arena, scaled horizontally. His line is the most extreme of the hour: if the goal is superhuman systems, most human-generated data goes in the trash, and you start over. For reasoning, the closest working analogue is Noam Brown's inference-time search: let the model think longer and it plays like a network with 10,000 times more parameters. That is a way to hack scale without growing the weights.

Labenz's last puzzle is almost a joke until it isn't. Can GPT-5 write a coding problem that GPT-5 itself cannot solve? Specifying the problem is easier than producing the solution, so a model that generates hard tasks, tries them, and trains on the ones that work looks "almost sure to work." Recursive self-improvement was not even close with GPT-3.5; self-critique often made answers worse. GPT-4 improves, then plateaus after three to five rounds, in program improvement and in chemical-reaction improvement alike. Those loops, crucially, did not retrain. Close the loop through training, and the takeoff picture becomes a lot more than a thought experiment.

They close by refusing to pick a winner between the two stories. The bull case is that the world is dripping in data, that tiny fractions of email, YouTube, genomes, and synthetic text should be enough, and that new modalities could be bolted on in an integrated system no human natively speaks. The bear case is that there is no obvious path to the extra 1,000-times if that is what a full brute-force AGI actually costs. The live question, Labenz says, is whether you can design a curriculum that gets the right grokking to happen early enough for superhuman capabilities to ignite — the ones you either hope for or fear.

What he wants listeners to walk around with is the habit, not the headline. Gannon's opening still belongs on a monitor: do not collapse AI into an index you are long or short. Anchor the worldview in magnitudes and in a fact sheet, and other people's moods stop moving your beliefs. New dataset drops, new fundraises, new claims about running out of data — each of them can be sanity-checked against a few numbers you already know.

Three ledgers do most of the work. Tokens: GPT-3 at 10^12, GPT-4 at 10^13, GPT-5 around 10^14. Cost: about $5 million, $50 to $100 million, then a billion and up. Stock: high-quality public text around 10^13 words, on track to run dry near 2026. When a dataset ships, ask how many GPT-4s it is. When a round closes, ask how many GPT-4-scale runs it buys. And watch the domains where answers can be checked automatically — code, math, geometry. Progress there will tell you more about the next step-change than any argument about whether the internet is empty.