Science Is a Self-Improving System That Isn't Exploding: Chollet on the Limits of an Intelligence Explosion

Asked why an intelligence explosion would not be possible, Chollet starts by granting the premise: any problem-solving process can be accelerated, and recursive self-improvement is a real thing. What typically happens is not an explosion, though. No system exists in isolation, so tweaking one part makes another part the bottleneck.

Science is his main example, since it is clearly a recursively self-improving problem-solving system. Scientific progress is not exploding, he says. What you see instead is a system consuming an exponentially increasing amount of resources while producing linear output in terms of progress.

He says that claims of exponential scientific progress usually look at indicators such as the number of papers published or patents filed. Those are just correlated with how many people are working on science today, so they measure resource consumption. What matters is the output: the scope and significance of the problems solved.

Chollet points to a paper by Michael Nielsen that tried to measure scientific output. Nielsen took the timeline of discoveries over the past 100 to 150 years and had panels of experts rate each discovery's significance. If output were exponential, the temporal density of significance should rise exponentially.

What the plot shows, Chollet says, is very much a flat graph, across physics, biology, medicine, and other disciplines. He finds that plausible: a century ago physics was in a period of crazy change, and 130 years ago technology was replacing horses with cars and spreading electricity. Today is also a time of fast change, but it would be unfair to say technology now moves much faster than it did 50 or 100 years ago.

His interpretation is that as you progress in a field, further progress gets exponentially harder. The first people in a new field like information theory find low-hanging fruit, while the next generation must dig harder for smaller discoveries and needs a much larger headcount to match the impact. The number of scientists and the computing resources available to science are growing exponentially, yet the output of significance is linear.

This holds even though science feeds back on itself, Chollet notes. Computers and the internet are products of science that speed up science, since the internet lets researchers exchange papers and ideas much faster. Even so, the system consumes ever more resources to produce the same amount of problem-solving.

Host Lex Fridman asks whether this holds for the deep learning community. Chollet says it would need more thought, but suggests the significance per paper may even be decreasing while the number of papers grows exponentially. In aggregate he guesses you would still see roughly linear progress.

He doesn't think it is a coincidence that progress is linear despite exponential resource consumption. His idea is that resource consumption adjusts dynamically to maintain linear progress, as a community expects it. If investment drops, lower-hanging fruit becomes available and someone steps up to pick it, much like a market for discoveries and ideas.

The general principle, in his words, is that exponential progress triggers exponential friction. He gives the analogy of a device that measures its own acceleration and adds more in proportion: it will not reach infinite speed, because air friction caps it, and if that is removed some other limit steps in, maybe even the speed of light.

Science as an institution shows the same thing. More researchers means more communication overhead, and in a field like quantum mechanics there is a huge body of knowledge to absorb before you can contribute. Significant experiments also need exponentially more expensive equipment, because the easier ones have already been run. Chollet thinks this is a good model for what would happen with superhuman AI.

Chollet is careful to say that this is an intuition and an argument, not a mathematical proof. He is not trying to prove anything, only to question the intelligence explosion narrative, which he calls quite dominant.

He says people push back hard on it because for many of them AI is not just a subfield of computer science but more of a belief system. It includes the belief that the world is heading toward a singularity after which things go exponential, the world is transformed, and humans become obsolete. Since it is not really a scientific argument, but part of many people's identity, questioning it feels like an attack, which he compares to saying that God does not exist.