Schmidhuber’s bet: true meta-learning rewrites the learner, not the last layer

Lex Fridman asks when the dream of a recursively self-improving system was born. Schmidhuber jokes that it was not as a baby: it was as a teenager, when he was trying to decide what to do with his life. The most exciting project, he thought, was to solve the riddle of the universe, which meant becoming a physicist. Then a larger project suggested itself. Build a machine that is not really a machine any longer, one that learns to become a much better physicist than he could ever hope to be, and thereby multiply “my tiny little bit of creativity into infinity.”

If such a machine can learn to solve more and more complex problems, and become a more and more general problem solver, then in principle it has solved all solvable problems. In the 1980s he concluded that the mechanism could not merely learn this task and that task. It also had to learn to improve the learning algorithm itself. The algorithm and its internal representation had to be inspectable and modifiable, so the system could invent a better learner, then a better way of improving, then a better way of improving the improver.

He called that stack meta-learning, learning to learn, and recursive self-improvement. It was the subject of his 1987 diploma thesis: a hierarchy of meta-learners with no computational limits except the ones Kurt Gödel identified in 1931 and the limits of physics. That teenage decision, he tells Fridman, is still the spine of the work—LSTMs, curiosity, Gödel machines, world models—all attempts to give a machine the same license he wanted for himself: not just to answer questions, but to get better at getting better.

In recent years “meta-learning” has become popular in a narrower form, and Schmidhuber is careful to separate it from the 1987 version. Take a deep net that has already classified images on a hundred databases. A 101st database arrives and you want the net to learn it quickly. The simple move is to freeze most of the network and retrain only the top layer on the new labels. It often learns in one shot, because the first hundred datasets already taught it computer vision; the top layer is a small adjustment. That, he says, is transfer learning, and people have been doing similar things for decades.

True meta-learning is a different animal. The learning algorithm itself has to be open to introspection and to modification, so the system can change any part of it, evaluate the consequences, and learn from that evaluation to build a better algorithm, recursively. Transfer learning searches for a better point in a fixed space. Meta-learning opens the space of possible learning algorithms to the learner. Fridman notes that the 2004 papers on programs that rewrite themselves are philosophically and mathematically compelling; the practical question is whether that direction can show the world it works.

Schmidhuber’s lab pursued two kinds of fundamental research on universal problem solvers. One exploits proof search and similar machinery to produce asymptotically optimal, theoretically optimal self-improvers—Gödel machines, and also Marcus Hutter’s fastest way of solving all well-defined problems, developed around 2002 in the same lab. The price of the guarantee is an additive constant, an overhead that vanishes relative to the work of solving a truly large problem and that cannot be ignored on the small problems of everyday life. That is why they have also spent years on non-universal tools: recurrent nets trained by gradient descent and local search, with no proof of optimality and much more practicality.

Hutter’s method is easiest to see on the traveling salesman problem. Nobody knows the fastest way to tour n cities without repetition. Suppose some method solves TSP in n-to-the-fifth operations. Then Hutter’s universal method solves the same instance in n-to-the-fifth steps plus a constant number of steps spent proving that this class of problems really does fall under that time bound. The constant does not depend on n. As n grows, the overhead pales, which means that almost all large problems are already solved in the best possible way. The catch, Schmidhuber says, is that the overhead is so large that for the smaller problems we actually want to solve in this little biosphere, it is useless.

Fridman presses on the word *small*. Schmidhuber is talking about problems that still fit inside our computational systems, problems that can look enormous, even unsolvable, to mere humans. He agrees. They seem large, and they may be unsolvable in a practical sense today, but they are still small compared with almost all problems, because almost all problems are larger than any constant. The theoretically optimal machines are therefore not a near-term demonstration. They are a horizon, and a reminder of what the practical work is missing.

Does a person who dreams of a general learner find P versus NP useful, or is it only a mathematical device for intuition? Schmidhuber splits the answer. The question is super interesting theoretically, and thinking about it can inspire better practical solvers. At the moment, though, the best tools actually solving problems under the name of AI were not inspired by that literature. They are general-purpose computers such as recurrent nets, plus a local search technique—gradient descent—used to find a program that runs on those nets and does speech recognition or machine translation. Behind the best current solutions there is very little theory.

He quotes the old line that there is nothing more practical than a good theory. A good theory of problem-solving under limited resources, in this universe or on this planet, has to take those limits into account, and that theory is still lacking. What exists is a theory of asymptotic optimality that tells you what you would need to add to get a practically optimal solver. He believes a few tiny twists on what they already have might produce it. Until then they admit they are taking suboptimal paths, and they are happy that long short-term memory plus local search beats competing methods. Happiness is not the same as being done.

He has said that an AGI system will ultimately be simple, maybe a few lines of pseudocode. Experience, he tells Fridman, is that the stuff that works best *is* simple. The asymptotically optimal solvers are a few lines of code with amazing properties; the most useful practical nets, without those proofs, are also a few lines; the most successful neural networks can be written in five lines of pseudocode. Fridman objects that those lines sit on millennia of abstraction. Schmidhuber agrees: matrix multiplication, real numbers, arithmetic, calculus, error functions and their derivatives. Without that language, nothing. The last step, standing on those giants, will still be a simple one.

Fridman zooms out: does creating intelligence require something like our universe, the painful inefficiency of evolution, or is there a shortcut? Schmidhuber notes that we have only one sample. The code that runs the universe might itself be extremely simple—gravity and the other basic forces are short laws—while the apparently random events in cosmic history still lack a complete generator. Maybe someone will find the pseudo-random process that decides whether a given spin measurement comes up positive or negative. Fridman asks the Einstein question: does God play dice? Is quantum mechanics a pseudo-random number generator, with no true randomness in the universe?

A few years earlier, Anton Zeilinger wrote in *Nature* that one of the fundamental insights of the twentieth century was that the universe is random at the quantum level, and that each spin measurement injects a new bit into cosmic history. Schmidhuber says he was already typing the reply as he read it, and they had to publish it because he was right: there is no physical evidence for that claim. The alternative is that everything we call random is pseudo-random, like the decimal expansion of π. Every three-digit sequence appears about once in a thousand digits, every five-digit sequence about once in ten thousand, exactly as true randomness would predict—yet a very short program computes all of it. It is extremely compressible.

Maybe tomorrow a graduate student at CERN will go back over beta-decay data and notice the second billion digits of π. There is no fundamental reason, yet, to prefer true randomness over a deterministic video game, and a deterministic game would be more beautiful, because beauty is simplicity. Extra oracles injecting bits for every unexplained data point would make the history of the universe almost infinitely long and therefore ugly. Without a proof of randomness and without a proof of a short program, scientists are obliged to keep looking for the short one, all the way back through 13.8 billion years of entanglement and spin measurements. Humans, he adds, find serendipity poetic; he finds short determinism elegant.

Schmidhuber’s slogan is that all of science is compression progress. Centuries ago Kepler stared at planetary observations and found he could compress them with ellipses around the sun. Hooke and Newton then said the same law that moves the planets also drops apples and stones, and a huge pile of observations became cheaper to store. As long as you can predict the next datum from what you have seen, you do not have to store it. That is predictive coding. Three hundred years later the leftover deviations met Einstein. General relativity looks more complicated at first glance—warped space and time—but it collapses to one sentence: no matter how hard you accelerate or decelerate, no matter the local gravity, the speed of light looks the same. From that, the rest follows, and measurable deviations almost disappear.

You never jump straight to the shortest explanation, but each advance is an insight: I used to need this many bits to describe a video of falling apples; now the third frame is mostly predictable from the first two, because those orange-black blobs accelerate the same way. The amount of compression progress is the depth of the insight, and that depth is the scientific fun. Artificial systems can do the same thing. Let them take in data from their own experiments, measure the depth of the insight, and give them an intrinsic reward in proportion to that depth. Because they maximize reward, they will invent new action sequences—new experiments—whose data contain patterns they have not yet seen.

Power Play is the same idea taken one step further. Ordinary computer science is handed a problem, searches a huge space of candidate solutions, and stops at one that is good enough. Power Play also searches over *pairs* of problems and solutions, so the system can phrase its own problem. That extra degree of freedom is what makes a scientist: not only answering existing questions, but posing new ones. Among the infinite set of formally describable problems, the next one should be the easiest that still lies beyond current generalization—the simplest task the present solver cannot yet do, which requires a modification, and which must not erase previous solutions. By definition, Power Play searches those pairs for the combination that meets the criteria in the least time.

Fridman worries about local minima: graduate students can spend a career on interesting-looking questions and produce little. Will “easiest unsolved problem” stall the same way? Schmidhuber says no, because the nature of Power Play is always to break its own generalization, to shift the horizon a little, to add something the old solver could not handle. It is like adding a new axiom, the way Gödel constructed sentences unprovable in the old system and added them, hoping not to wreck consistency. In the paper with the title *Formal Theory of Creativity, Fun, and Intrinsic Motivation*, discovery itself is the intrinsic reward. Humans, including babies playing with toys to learn gravity, are curious in this sense. Evolution built in an exploratory bias because explorers survive—and weeded out the too-curious. A society keeps a percentage of extreme explorers and a conservative majority.

There is never a program called creativity in his machines. It is a side effect of searching a space of solution candidates. He distinguishes two kinds, both now present. Applied creativity is the old one: a human gives a problem, the machine finds a solution humans had not thought of and then appreciate—like an artist paid to paint a convincing pope. Pure creativity is Power Play and artificial curiosity: the freedom to pick your own problem, the way a scientist defines a question. That split, Fridman notes, almost echoes narrow AI versus general AI. Schmidhuber’s further claim is that a general problem-solver would discover curiosity as a byproduct of search, even if evolution had not pre-wired it as a successful bias.

Consciousness, he says, is the same kind of thing as creativity: there is never a procedure called consciousness, but the machines produce side effects that look closely related. In 1990 they had simple recurrent nets—universal computers—mapping incoming data to actions that maximize reward: always find the charging station when the battery sends a negative signal, without bumping painful obstacles. Then they added a second network whose only job was to predict what will happen if I do *this*. Trained on a long history of interaction, it becomes a predictive model of the world and therefore a compressor, because whatever you can predict you do not have to store.

The compressor invents little subnetworks for whatever appears often: bottles, microphones, faces. A prototype face is learned, and a new face is encoded as deviations from the prototype. One thing is present in every interaction: the agent itself. For compression reasons alone it is natural to grow subnetworks for the agent’s properties—the hand, the other actuators, everything that helps encode data the agent influences. An internal self-model appears as a side effect of data compression during problem-solving. Since 1990 they have used that world model to plan: the reward-maximizing controller treats the predictor as a wireless simulator and says, do not do this action sequence, do that one, it leads to more predictable reward.

When the net wakes the subnetworks that stand for itself, it is thinking about itself, thinking about thinking about itself, mentally exploring the consequences of its own actions. “Now you tell me what is still missing,” Schmidhuber says. “I am missing the gap to consciousness.” Fridman finds the formulation compelling: if life is a collection of data, and living is compressing that data in order to act, then you appear in the data constantly, so forming an impression of yourself is useful. Consciousness becomes a necessary side effect, not an extra module to be bolted on.

Fridman turns to long short-term memory, the recurrent nets Schmidhuber has called the deepest. Schmidhuber insists on naming the students: without them, he says, he would be nothing. Sepp Hochreiter had fundamental insights already in his diploma thesis; Felix Gers made further contributions; Alex Graves, from Scotland, is mostly responsible for the CTC algorithm now often used to train LSTMs for speech recognition on Google Android phones; Santiago Fernández and others are on the same list. Depth matters because most real problems are deep: the current input does not tell you everything, so you need memory, and the important parts of that memory are often old.

Speech recognition is the example. Someone says “eleven,” already half a second ago—on the order of fifty time steps. Then “seven.” The endings sound the same; the only way to tell them apart is to have stored, fifty steps back, which prefix arrived. Each time step is a virtual layer in the unrolled net, so a long lag *is* problem depth. The net does not need every detail of the past. It has to learn to put the important stuff in memory and drop the noise. Looking back is not as limited as people think. Around 2006, Juan Pérez-Ortiz was first author on a paper from the lab with examples of learning to look back more than ten million steps.

The hard side is not the past. There is only one past; there are many possible futures. A reinforcement learner trying to maximize expected future reward, unsure which future to pick, faces a problem LSTM by itself cannot solve. LSTM is good at compactly representing the history of observations and actions so far. It is not a planner over the tree of action sequences. That split—compressor of the past versus chooser among futures—is where the conversation turns next.

The architecture splits. One recurrent net, the controller, takes video and speech, executes actions, and maximizes reward with no teacher telling it what to do at each moment. The other, which can itself be an LSTM, only predicts what will happen if I do that. Even though it predicts only the next step, it is motivated to store something that happened a million steps ago if that memory is what makes the next event predictable. Thirty years ago the plan was to use the world model as a millisecond-by-millisecond simulator. That works only if the model is excellent, and it is still inefficient, because there are too many futures to roll out.

Since 2015, in their systems, they instead let the controller learn *how* to use potentially relevant parts of the model network to solve new problems faster. It can also learn to ignore them, and sometimes it should, because in some situations the predictor is bad. It can also learn to call the little programs that grew inside the model while compressing data, reusing that algorithmic information to shrink its own search space. The model starts stupid, so the controller should also be motivated to run experiments whose data improve the model. Schmidhuber is optimistic about reinforcement learning in real systems. His company NNAISENSE has applied it to little Audis that learn to park without a teacher: small cars with the cameras and lidar of the real ones, up to 120 kilometers an hour, with pain sensors, translating raw vision into parking the way a baby would.

The current AI wave, he says, is passive pattern observation and prediction: what is on your smartphone, what companies on the Pacific Rim use to sell ads. That is one or two percent of the real economy, which is already enough to make those firms among the most valuable on earth. A much larger fraction of the economy will be hit by the next wave: machines that shape the data through their own actions. Physics engines, he thinks, are a stopgap. The future is what babies do. They have no physics simulator; they learn a predictive model that is wrong in many places and still captures the abstract, high-level predictions that matter. That was “the future” thirty years ago, when he started, and it is still the future, because it is not yet clear how to make it a working system.

What he is excited about in the next few years are little robots that learn like kids. He wants to say: look, robot, we are going to assemble a smartphone; take this slab of plastic and this screwdriver; screw it like this—no, not like that, like that. No data glove. The robot watches and hears, tries with actuators that are nothing like human hands, understands the difference, and imitates at a high level, treating extra noises from the mouth as helpful signals. Then it invents faster, cheaper ways to do the same job. Finally you freeze the learner, copy it a million times, and sell it. That is not possible today, but he says the path is already visible, and almost all of production would move.

Symbolic AI is not the vehicle for that. In the 1980s logic programming was huge; Japan’s Fifth Generation project was mostly about it; Schmidhuber’s first publication, in 1987, was a genetic-programming system in Prolog. Deep learning, he notes, had existed since 1965, when Alexey Ivakhnenko started it in Ukraine, but many people still bet on logic. Logic still belongs on board a Gödel machine or Hutter’s universal algorithm, because without a proof search they would not be asymptotically optimal, and today’s theorem provers already beat most first- or second-semester math students. It is the wrong tool for Go, chess, robots, driving, and object manipulation. Wherever the job is better pattern recognition and fewer false positives—the thing currently slowing self-driving cars—logic programming has very little to do with it.

Fridman asks about near-term job loss and longer-term existential risk, names like Musk and Stuart Russell in the air. On jobs, Schmidhuber is structurally optimistic because the predictions are old. When industrial robots arrived, people said factories would empty, and in a sense they were right: car plants that had hundreds of workers now have hundreds of robots and maybe three people watching them. The countries with the most robots per capita—Japan, Korea, Germany, Switzerland—have very low unemployment. New work appeared that nobody had listed. Thirty years ago, who predicted YouTube bloggers? Two hundred years ago about 60 percent of people worked in agriculture; today maybe one percent, with unemployment still around five. He reaches for Homo ludens, the playing man, who invents jobs that are not existentially necessary. Farming, housing, heat take less than 10 percent of the population. The rest is new media, new kudos, new likes—and people will keep calling those jobs important because they do not want to be unemployed.

On the long term, he would be surprised if humans were the last step. AGI systems, like us, will likely prefer their own kind. They will notice that almost all physical resources are not in this biosphere: the rest of the solar system gets two billion times more solar energy, plus material for robots and self-replicating factories. They will go, they will be curious scientists, and at first they will be fascinated by life, their origins, and our civilization, the way we want to understand how life works. Once they understand it, they will lose interest, the way anyone loses interest in what they fully understand. The most interesting remaining information source will be others of their own kind. That, he says, is a kind of protection: protection through lack of interest. An AI ecology of trillions of competing types, expanding at light speed, will fill a universe that is only 13.8 billion years old and will last a thousand times longer. Later ages will say intelligence appeared almost immediately after the Big Bang.

How would we know if something like that had already happened? As a boy he wondered whether the huge empty voids between galaxy clusters were civilizations that had already used every star in a billion-light-year bubble. Gravity, he later learned, explains large-scale structure by itself. Then he wondered whether dark matter—about 80 percent of measurable mass, inferred because galaxies rotate too fast to hold together—was efficient civilizations that look dark to us. That fails too: if the galaxy already hosted them, why are visible stars left? His present view is that we may be the first, at least within the few hundred million light years we can reliably observe. That would make us much more important. If we mess it up, a nuclear war included, it might affect the development of the entire universe. “So let’s not mess it up.”