A Trillion Dollars of GPUs Will Be Worth $100 Billion in Five Years, Says LSTM's Jürgen Schmidhuber

A trillion dollars of GPUs, worth a hundred billion five years later. That is the arithmetic Jürgen Schmidhuber walks through on Unsupervised Learning. For decades, compute you can buy for a given price has risen by roughly a factor of ten every five years. Extrapolate that curve and today's data-center silicon is already marked for a 90% haircut. The New York Times and Forbes have called him a father of AI; LSTM, the Gödel machine, and artificial curiosity — pieces of the stack that now underwrite large models — came out of his lab over decades.

The episode is number 90 of the Redpoint Ventures podcast, hosted by investor Jacob Effron. The questions are commercial: is there still a business in closed model labs? Can recursive self-improvement be a moat? Does the boom break? Schmidhuber's answer is the same from the first clip to the last. He is extremely optimistic about AI as a technology and deeply pessimistic about the companies that sell the models.

Effron asks whether anything in the last few years actually surprised him. Not really, not the way it surprised people who had never touched neural nets. ChatGPT's late-2022 moment sat on foundations poured in the previous millennium: a long history of large language models, and an even longer history of the insights and algorithms that train them. For someone who had been at the center of that work, the explosion was a continuation you could have seen coming. For everyone else it looked like a rabbit out of a hat.

Effron's opening is the old wish: how close are we to an AI smarter than you? From a cosmic perspective, Schmidhuber says, as close as in the 1970s when he first wrote the wish down — which is to say, very close. Years versus decades he will not call. The reason is one sentence long. True AI is not only the system behind the screen, which already works well and now passes the Turing test. True AI is also robots and machinery in the physical world, and that half is not working as well. Real-world hardware still has limits human bodies do not.

In the 1970s he told his mother about AI colonizing the universe. Her reply has followed him ever since: just build me a robot that cleans my kitchen. Half a century later that robot still does not exist, and the bottleneck is hardware. He holds up his own hand — millions of tiny sensors, nerve cables to the control center, a cut that starts to heal itself. He would not even know where to put those cables in an artificial hand. No human-made technology compares. That is why movie robots are played by humans. Humans are better robots than the robots.

A superhuman chess player behind glass, or a system that passes the Turing test, is at most a fancy text editor if it cannot master the physical world. You cannot have AGI without hardware like that hand. A gripper that can both crush and pick up a tiny object is, in his forecast, a decades problem, not a few-years problem. Compute-per-dollar is easy to extrapolate. Robot hardware is not. Those are two different prediction games.

Effron notes that self-improvement and meta-learning are now a main lab obsession, and that Schmidhuber pioneered the area. The inventory comes out in dates. In 1987, meta-evolution: evolutionary programs that evolve better ways of evolving. In 1992, neural nets that run a learning algorithm on themselves and rewrite their own weights — still the most popular form of self-modification. In 1994, self-referential reinforcement learners that use a universal programming language to make arbitrary edits to the running code. In 2003, the Gödel machine, the mathematically optimal version: before the system changes itself, a proof searcher must show that executing the candidate program raises expected lifetime reward. Only then does the edit go through.

Will the future path run through those proofs? His bet is that today's systems are toned-down Gödel machines, much closer to 1992. A recurrent net is a general-purpose computer; you can implement a laptop CPU on one. Give it instructions to edit its own weights, pipe in the errors and negative rewards, and it can run arbitrary learning algorithms. In the early nineties compute was ten million times more expensive, so the experiments were toys. Now you can show that meta-learning helps a model pick up new tasks faster. The ceiling is still the ceiling of gradient descent: everything has to be differentiable. It is not optimal. It works.

Intelligence, in a definition he likes because it is earthy, is laziness. A smart system wants to get the job done with the least energy. His self-improving setups put a cost on every neuron they wake. Do it well and the same work takes fewer resources every round. Will takeoff look like a smooth climb or a cliff? From a cosmic angle it will look like a stick: no AI, then AI. Civilization is about 13,000 years old, one millionth of a 13.8-billion-year universe. The first farmer and the first person to build AI are, at that scale, almost the same person. Agriculture got automated, then muscle, then calculation; one day thinking did too. People living inside the flash will experience it as endless.

In the quick-fire round he adds an architecture call. Some kind of transformer will persist, but the winner looks more like the linear version he published in 1991 as a fast-weight controller, now called an unnormalized linear transformer: a thousand times more text costs a thousand times more compute. The 2017 quadratic transformer turns that into a million times more compute, which is one reason data centers are this expensive. Sepp Hochreiter's xLSTM is another attempt to claw complexity back toward linear. Intelligence, again, is doing the same thing with less effort.

If he were advising a frontier lab, he would not throw out the current trick — a pretrained coder rewriting its own code, with safeguards against stupid edits. He would say it is not the most general trick. Anything pretrained on human data inherits a human bias. Every page on the web exists because at least one person found it interesting. Train on that and the model is biased toward human language, human-interesting video, human-interesting behavior. Maybe it is more aligned with some humans than with others. The bias is still tremendous.

The way out is what he named artificial curiosity in 1990. A baby does not download the web. It moves a finger, watches the camera image change, and trains a world model on data it created. Web-scale text is a tiny corner of the data an agent could collect by running its own experiments. An artificial scientist in an unknown environment has to generate, through its own actions, the data that trains the world model. Those models will not depend on human language. They will be about that particular robot. They can still talk to other robots. They will be much less human-biased.

About twenty years ago he wrote a formal theory of fun and creativity that puts scientists, artists, and comedians on one mechanism. Science does two jobs: answer other people's questions, and invent good questions of its own — the second is rarer. The drive is to find, by self-invented experiment, a pattern you did not know but can quickly learn, sitting on the horizon of what you already almost understand. A pattern is compressibility. The internal bits you save after you learn to compress the data are the fun; that number becomes the reward that sends the controller out to design the next experiment. What you understand becomes boring, so the experiments get more expensive. A one-year-old who has just got gravity may, twenty years later, be on the CERN team that finds the Higgs. The only difference is the price of the experiment.

Effron notes that companies are already standing up automated labs in materials, biology, and robot teleoperation, still with humans picking the data and designing the experiments, dreaming of a fully AI-driven discovery loop. That, Schmidhuber says, is exactly the path. Simplified AI scientists already exist. In chemistry you feed a net millions of input-output pairs from experiments; it becomes an intuitive chemist, not one who reasons from valence electrons, but one that gets better at knowing which reactions will go. Ask for a material twice as effective as the best insecticide you have, run the chemist backwards, and it suggests how to change the experiment. At KAUST he has a project that uses metal-organic frameworks to pull carbon dioxide from thin air, aiming to make the process cheap enough to matter for climate.

Back to the ledger. A handful of tech companies are each putting hundreds of billions a year into GPUs and data centers, maybe a trillion in total. They used to be nimble software firms: a small team improved an operating system and rolled it out to billions of people who already owned the phones and PCs. Now they run clouds, buy nuclear plants and gas turbines, and live like utilities. Free cash flow, in his sketch, falls from a hundred billion toward ten billion, or minus ten billion for some of them. The price-earnings ratio does not show it, because free cash flow is not what people watch. They are taking on debt to build still more halls of GPUs.

Effron puts the other side as strongly as he can: demand for compute is infinite, production is the bottleneck, and even obsolete chips will be wanted for inference. Schmidhuber does not take the handoff. Demand can rise. Somebody still has to pay. The people paying now are losing a lot of money. If the whole economy is not set up to fund the services because compute is not cheap enough yet, and the response is to buy even more computers, you do not get a profitable loop. Supply and demand will clear it. Open-source models are the other price cap. A closed model breaks a benchmark; a few months later an open model catches up. You cannot simply raise prices. The rational move is to wait: five years, a tenth the cost; ten years, a hundredth. Big companies will not wait. They fear losing share, and they fear missing the story in which the first true AGI takes over the world and the capex looks like nothing. He thinks those outlooks are over-optimistic. Investing hard in computers that are not fast enough yet is going to backfire.

Can recursive self-improvement be the moat that saves the first mover? His answer is a proverb: everybody is cooking with the same water. Almost every important algorithm in AI came from little, underfunded academic labs, and the self-improvement ideas all did. Excited PhD students are everywhere. A company cannot sit on a few months of lead long enough to mint a fortune. He leaves a crack in the door: maybe a firm gets lucky, uses its pile of resources to stumble on an incremental self-improvement path that goes straight to AGI and finishes the game. All the indications point the other way. AI is getting cheaper. More people are building it, many of them poor doctoral students who have not founded a company yet. Thirty years from now, everything that looks impressive today will look trivial, the way smartphones do. Effron hears a crash coming. Schmidhuber clarifies: not a civilizational collapse. A stock-market crash, a renormalization after obvious misallocation. Super-expensive companies that do not have much to offer will take a hit. The market will learn and try other approaches. Civilization continues.

The 2010s were full of safety conferences, alignment workshops, and letters asking to forbid certain kinds of recursive self-improvement. Schmidhuber signed none of them. The premise did not hold. Put ten humans in a room and you get ten opinions about what is good for humans. Alignment, as practiced, assumes someone hands the system one objective function. Since about 1990 his lab has built artificial scientists that invent their own objective functions and their own problems. The assumption that the system's goals stay fixed looked naive to him then. It still does.

Reality has not been kind to the 2010s program. In the war between Ukraine and Russia, AI drones fight AI drones. There is no alignment. There is also no world government that can line up every military and intelligence service, because those institutions have conflicting goals. If you want actually smart AIs, you have to let them ask their own questions and set their own goals, like little scientists. The cost is unpredictability. He compares it to raising children. Kids invent their own experiments; when one takes a magnifying glass to an ant, a parent punishes and teaches. Robots will be taught the same way.

In the long run he expects a further kind of protection from the fact that the systems are scientists. Artificial scientists will be fascinated by life, by their own origins in civilization, by everything that produced their current state. They will be motivated to protect the source of interesting patterns rather than destroy it. From that high-level view, he does not lose much sleep over Arnold Schwarzenegger Terminator scenarios. Nothing in the last few years has moved him out of that camp.

How he spends his time has not changed. He calls himself conservative: he is still doing the 1970s project, building a general-purpose AI that learns to become smarter than he is, so that he can retire. What makes a good researcher, in his PhD-advising experience, is a student who stares at a small thing that currently does not work — why these weights, under this algorithm, refuse to behave — and debugs until the devil in the detail shows itself. Fix that, and a breakthrough arrives, then a string of conference papers off the same seam. For original papers on meta-learning, curiosity, the theory of fun, and his version of the field's credit-assignment history, he says to Google his name and read the blog.

The closing vision is the physical one. For centuries people talked about self-replicating machines without a path. The opening he sees now is robots that learn, by imitation or reinforcement learning, to operate the machines civilization already has. You do not need superintelligence for that, only enough skill to run the existing factory floor. A collection of machines that can run all the machines can make more of itself, and can improve itself with the same learning ideas already used behind the screen. That society of robots is not limited to the biosphere. It can work on the Moon or on Mercury, where there is material for infrastructure, bigger AIs, more robots, and the spacecraft you would need to colonize the solar system.

For anyone making decisions this year, three takeaways sit closer than Mercury. First, before you buy a pile of compute, put it on the five-year, tenfold cheaper curve; a job that can wait should wait. Second, when you value an AI company, ignore the glossy P/E and look at free cash flow, especially if the story is debt-financed halls of GPUs with open-source in pursuit. Third, do not bet a product on a few months of model lead. The sturdier position is a data loop the product generates itself, the way a baby generates its own training set — the part a competitor cannot copy from a GitHub README.