Intelligence is not the height of a building: François Chollet on Keras, deep learning, and why AGI will not explode

Lex Fridman opens by noting that Chollet does not sugarcoat his views, especially on Twitter, and asks for a controversial idea that drew heat. Chollet’s example is a post questioning the intelligence explosion: if you built a general problem-solving AI, “building such an AI” would itself be a problem that AI could solve better than humans, so it would rewrite its algorithms, spawn a better version, and recurse into exponentially rising intelligence. He objects first to the implicit definition. The story treats intelligence as a property of a brain considered in isolation, like the height of a building. That is not, he says, what intelligence is. It emerges from the interaction of a brain, embodied intelligence, and an environment. Missing one of those pieces, you cannot actually define it. Just tweaking a brain to make it “smarter and smarter” does not make sense to him.

Fridman plays devil’s advocate in the register of Max Tegmark and others: the universe is an information-processing system, the brain is one too, so why would our hardware be the limit? Scale something on par with the brain and it should improve exponentially — the logic used by almost everybody worried about superhuman intelligence. Chollet’s reply is that even among very smart humans, the brain is not the bottleneck to expressed intelligence or to achievement. You cannot tweak one part of a brain–body–environment system with many interdependencies and expect capabilities to explode, because a new bottleneck arises. Many genius-level people are not solving big scientific problems. Einstein became Einstein because a genius met a big problem at the right time; that meeting might never have happened, and he would have remained a patent clerk. Many people today are probably that smart, and we would not know, because they are not expressing it.

Fridman restates the picture: Earth is a space of problems of various difficulty, agents roam it, and you demonstrate intelligence only when you couple to a problem. Chollet agrees. Intelligence is the meeting of great problem-solving capacity with a great problem. Without the problem you do not express intelligence; you are left with potential — how fast the brain runs, an IQ number — which in itself is just a number.

Asked to define intelligence, Chollet says all intelligence is specialized, even human intelligence, though every system has some degree of generality. Humans are specialized in the human experience. That shows up as innate prior knowledge — about agents, goal-driven behavior, visual objecthood, time — and in how we learn: language is easy because we are hard-coded to pick it up. The same specialization makes us useless at other problems. We are not designed for very long-term problems; a lifetime is short, and even inside it we struggle to project ourselves five or ten years out. Working memory is small. Large-scale problems are not solved by one brain. We have civilization: a problem-solving system, itself a kind of artificial intelligence, running not on one mind but on a network of brains plus books, computers, the internet, and institutions. Computer science as an institution is superhuman. Science as a system is a superhuman problem-solving algorithm.

Fridman zooms out to bacteria, planets, galaxies, and Google’s millions of devices, asking how scale enters the definition. Chollet’s target is the hierarchical cartoon popular with explosion talk: one brain at the top of a pyramid, a body receiving orders, an environment being manipulated. Real agents are not like that. There is no strong line between brain and body, or nervous system and body; you have to look at the whole animal. Watch any animal long enough and much of its intelligence is externalized — notes, programs, books, computers, other humans, language. There is hardly a hard limitation of what makes an intelligent agent. It is all about context. Could you still have exponential growth of ability on a specific task? Possibly, to some extent, if you consider a narrow enough category. We do not have to speculate, he says, because we already have a real-world recursively self-improving system: science.

Science experiences the world, gradually understands it, and acts on it. It is superhuman, and it is recursively self-improving: knowledge becomes technology, which becomes better computers and instruments, which speed science. That, Chollet says, is the closest thing we have to a recursively self-improving superhuman AI, and a basis for guessing what a science-like superhuman system would actually do. Recursive self-improvement is real. Explosion is not, because no system exists in isolation. Tweak one part and another becomes the bottleneck. Scientific progress is not exploding. What you see is a system consuming an exponentially increasing amount of resources with a linear output. People who claim exponential scientific progress are looking at resource-consumption indicators — papers published, patents filed — which mostly track how many people work on science. What you should look at is the knowledge generated and the scope and significance of the problems solved.

Michael Nielsen tried to measure that. He took a timeline of discoveries over about 100 to 150 years, asked a panel of experts to rate each one’s significance, and plotted the temporal density of significance. If science’s output were exponential, that density would rise exponentially. The graph is very much flat, across physics, biology, medicine. The intuition matches: physics 110 years ago was a time of crazy change; 160 years ago, horses were being replaced and electricity was arriving. Today is also fast, but it is unfair to say science and technology now move way faster than 50 or 100 years ago. Chollet’s reading: as you progress in a field, further progress becomes exponentially harder. The first people in information theory found low-hanging fruit; the next generation digs harder for smaller discoveries, and you need a much greater headcount for the same impact. Scientists and engineers, and compute, increase exponentially. Resource consumption is exponential; significance is linear. Even a recursively self-improving loop — computers, the internet — spends vastly more to produce the same amount of problem-solving.

Fridman notes the same pattern in deep learning: per-paper significance may even be decreasing while paper count explodes, so the aggregate looks linear. Chollet does not think that is a coincidence. Resource consumption dynamically adjusts to maintain linear progress, because the community expects it; invest less and low-hanging fruit appears for someone to pick — a market for discoveries. Exponential progress, he says, triggers exponential friction. His toy is a device that measures its own acceleration and outputs more in proportion: drop it and air resistance locks it at a terminal speed; remove the air and another bottleneck, even the speed of light, steps in. In science the frictions are communication overhead across more researchers, a huge ingest of prior knowledge before you can contribute to quantum mechanics, and experiments that require exponentially expensive equipment because the cheap ones have been run. This is not a mathematical proof, he stresses. It is an argument against a dominant narrative that, for many people, is closer to a belief system and an identity than to a scientific claim — almost like saying God does not exist. The pushback, Fridman and Chollet agree, comes less from people who have built anything called AI than from people attached to the story. Chollet thinks it is a good story, and that it mirrors religious mythology: a final event, the world destroyed, a new order. We need stories at timescales beyond our ability to predict.

On a less eschatological question — will something like human-level intelligence ever make him sit back and say damn — Chollet’s surprise already happened around 2013 and 2014, before the field had assessed exactly what deep learning could and could not do. It felt like immense potential, then the boundaries narrowed. He never believed deep learning was limitless. What was shocking is that it worked at all, and that computer vision got so good. That is a long jump from human-level intelligence, which he does not think we are close to, though he sees no reason we would not achieve it at some point. He also dislikes “human-level” as a phrase: it implies a one-dimensional axis. Intelligence is multi-dimensional. Capability and being human-like are different; you can build very advanced agents that are not human-like at all, and very human-like agents that are not.

Fridman asks for a Wikipedia-style history of Keras and the frameworks around it. Chollet started the project in February 2015 and picked the name the day he was going to release it. The community then was maybe fewer than 10,000 people. Caffe, mostly C++, was vastly more popular than Theano and was what everyone used for computer vision, then the dominant problem. Chollet himself was interested in recurrent networks, still a niche that would catch fire around 2016. He had used Torch 7, Theano in Kaggle competitions, and Caffe, and there was no reusable open-source LSTM. He decided to build his own, in Python, mostly around LSTMs and convnets, combinable in one library. The non-obvious decision was to define models in Python code, against the mainstream of YAML static configuration files in Caffe and Pylearn2. He drew usability from scikit-learn — collapsing a complex training loop to a single `fit` call. People said that hid details. That, he says, was exactly the point. He released in March 2015, at the moment people were getting intrigued by RNNs, and it grew from the start.

Six months later he joined Google, unrelated to Keras, on an image-classification research team. He was immediately exposed to an early internal TensorFlow, which struck him as an improved Theano, so he knew he would have to port. As a new Googler he had no time until the November 2015 open-source release became a wake-up call. In December he refactored: backend functionality into one module so the same codebase could run on TensorFlow or Theano. For the next year Theano stayed the default — easier, fewer bugs, much faster on RNNs — until TensorFlow overtook it. Keras was still a side project through 2016. Around October, Rajat Monga, who led TensorFlow, showed up in Chollet’s building — Chollet was then doing vision and a collaboration with Christian Szegedy on deep learning for theorem proving — and invited him over for a quarter. He started integrating the Keras API more tightly into TensorFlow, first a contrib version, then core. He never actually went back to his old research team.

On TensorFlow 2.0, after a year and a half of work, Chollet is excited that Keras is no longer only the easy high-level interface that got in the way if you wanted flexibility. You now have a spectrum: subclass models, write training loops with eager execution, debug easily, and still climb to classic `fit` workflows that suit a data-scientist profile. The same APIs cover researchers through engineers, wired into TensorFlow Lite and cloud serving. Fridman notes that some of the best software is written by one or two people, and Keras now lives inside a large TensorFlow organization. A lot of Chollet’s Google time is design discussions, design docs, and review meetings — as important as writing code — because the user base is extremely diverse: small and large production, startups, researchers.

Making a design decision, he says, is about satisfying a set of constraints in the simplest way that can be maintained and extended. You do not want one new argument per capability. You want modular, hierarchical APIs with as small a surface as possible, reflecting how domain experts already think. When an expert reads docs they have concepts in mind; they are trying to map those concepts onto the API as quickly as possible. Minimize that cognitive load. An API should not be self-referential or leak implementation details; it should talk only in domain concepts people already understand. As for TensorFlow 3.0, he is not even the one making those decisions. His own excitement is higher-level than Keras: hyperparameter tuning, AutoML. The future is not a box of Lego blocks plus `fit`. It is more like a kid who is pretty good at Legos, plus a box of Legos — look at your data and optimize the objective you offered.

Deep learning models, Chollet says, are huge parametric, differentiable, continuous maps from an input space to an output space, trained with gradient descent, pretty much point by point: a continuous geometric morphing. Because of that, a net can only make sense of points in experience space very close to what it has seen. At best it interpolates. Training therefore wants a dense sampling of the input–output space, which is expensive on complex problems such as driving or robotics — millions of examples even on a visual subset — and it will still only handle things near what it has seen. Contrast a symbolic rule. A sorting algorithm in a few lines, two nested loops, processes any list because it is abstract, not a point-by-point map. Learn sorting with a net and you are limited to what the sorted form of this specific list looks like. The future, he thinks, combines the two. Successful AI systems today already do: robotics is mostly model-based planning with deep learning as perception, or as a fuzzy intuition injected into a rule-based process. A self-driving car is not one big end-to-end net. That would require a dense sampling of driving experience, which is unrealistic. The car is mostly hand-written software on explicit 3D models of the surroundings, interfacing with the world through deep-learning modules that turn raw sensors into something symbolic systems can use.

Fridman pushes on lane-following as a narrower end-to-end case. Chollet thinks it is doable, and that there is no hard theoretical limit on what a net can learn if the search space is rich enough and the sampling dense enough. Dense, though, might mean 10,000 examples or trillions. A Turing-test conversation is more about tricking human perception than about intelligence; mimicking agent behavior is not actual intelligent behavior. A 20-minute conversation that carries tangents would be very challenging with deep learning, not out of the question. In practice deep learning is a great fit for perception and for problems not naturally amenable to handcrafted rules or exhaustive search. Physics, 3D structure, and object relations can always be solved with deep learning, he says — just extremely inefficiently. An explicit abstract model is a far more compressed representation than memorizing “in this situation this happens.” Automatically generating those programs is program synthesis, and today we do not really know how to do it. It is graph search, limited by current search algorithms. He personally finds genetic algorithms and genetic programming promising.

If deep learning is 2019, program synthesis is the 1990s: a basic sense of the problem, few people, few real applications. The one real-world application he knows is Flash Fill in Excel, which learns a tiny formatting program from a few examples. Even the 1990s analogy may be optimistic, because by then people already understood backpropagation, the engine of deep learning, without yet seeing its potential. Program synthesis has not found its engine. Discrete search over rule-based models, he believes, will be a cornerstone of AI research in the next century. That does not mean dropping deep learning, which is immensely useful at flexible pattern recognition given lots of data. The two will be combined in increasingly automated ways.

Chollet had tweeted that about 10,000 deep-learning papers were really about hard-coding priors for a specific task into a neural architecture, then reporting a gain. Machine learning is supposed to be the science of generalization — knowledge reusable across datasets and tasks. Stare at one dataset and bake task knowledge into the architecture, and it is no more informative than training a net and announcing that these weight values perform well. David Ha’s paper on weight-agnostic neural networks makes the point sharp: an architecture, even without weights, already encodes knowledge about a task. On bAbI, a question-answering set generated by an algorithm that follows a template, a network that literally encodes that template can hit nearly 100% accuracy and tell you nothing about question answering in general. People with real large-scale deployments, he adds, spend heavily on annotation pipelines; you just do not see papers about it.

Fridman raises Rich Sutton’s “Bitter Lesson”: 70 years of AI research show that general methods that leverage computation ultimately win. Chollet thinks Sutton is right about those 70 years. Many papers that manually hard-code task knowledge do not make long-term impact; simple, general, trick-agnostic systems do, and the simple thing to focus on is whatever rides Moore’s-law compute. He is not sure the assessment holds for the next 70. Its truth is a function of context, and the context is changing. Moore’s law may not apply as it did. Exploit one aspect of a system and another becomes the bottleneck. Give unlimited compute and data is the bottleneck — a regime he thinks we are already entering, with systems so large that data quality and scale dominate. In that environment the Bitter Lesson will not be true anymore. Research focus moves from compute scale to data efficiency. Unsupervised learning and reinforcement learning are frameworks, not techniques; “deep RL” still uses huge parametric models trained point by point. It is more efficient in annotation density — learn a latent space, map sparse labels into it — a good idea that should yield incremental labeled-data efficiency, not a different data diet.

Chollet does have concerns about AI, and they are not science fiction. Deep learning enables mass surveillance: facial recognition, tracking data about everyone, identification and prediction — aggressively pursued, he says, by totalitarian states such as China. The subtler worry is that lives are increasingly online, a digital footprint of information consumed and produced. If you absorb that data and control the feeds — social networks, recommenders — you can build a reinforcement loop on human behavior. Observe the state of a mind at time T, predict how it would react to different pieces of content, feed the piece that moves it in a chosen direction, and do that continuously, in real time, at the scale of populations. Even current AI, combined with the internet, makes mass manipulation of behavior and psychological control a real possibility. Treat the human mind as a program and it has a very large exploit surface. Political belief is tied to identity. Control a newsfeed and you can show only news that frames the world a certain way. More cleverly, you can create incentives to post: if you want a statement reinforced, show it to people who will like it; if you want it abandoned, show it to opponents who will attack, so next time the person thinks twice, or even starts to doubt.

Those levers are already run by algorithms without an explicit political goal. A totalitarian government taking over the platforms would be worse. What is already concerning, without any such intent, is that the objective is engagement. It seems innocuous. Content that maximally engages is often content that is not healthy for public discourse. Fake news is far more likely to get a click than real news, simply because it is not constrained by reality and can be as outrageous a story as you want. Fridman points to the large space of objective functions that could educate or polarize. Chollet would feel uncomfortable with companies making explicit decisions to manipulate opinions even with good intent — that is a totalitarian mindset. What he wants, and admits is not super realistic, is configuration: let users decide how they want to be impacted. On YouTube or Twitter, maybe maximize learning about a topic and information accuracy, not engagement. These algorithms are increasingly the interface to a world made of information; everyone should interface on their own terms. The problem is mostly interface design. Technology should be a mentor, coach, or assistant, not your boss. If you feel manipulated, you should be able to switch algorithms.

Practically, public awareness is still thin. People do not think anything is wrong with a newsfeed whose side effect is already bad. Stop purely maximizing engagement and popularity; take into account each user’s goals. Twitter’s topic recommendations, he says, are often the worst garbage — smallest common denominator — because they optimize popularity. Knobs that expose the whole algorithm are not realistic; put the user in charge of defining the objective function. Spotify-style per-item thumbs work to some extent, but he is skeptical: the system relates your choices to everyone else’s, which is fine for an average, mainstream profile and inefficient otherwise, and it still does not let you set what the algorithm is optimizing for. Raising awareness is a start.

Long-term existential risk, for Chollet, still runs through those interfaces. The world is increasingly information; algorithms are the interface; someone will control them — large companies optimizing profit or other goals, governments using them as organs of control. He does not buy Terminator robots or a required singularity. Loss of control over algorithms that move populations is enough to hurt ourselves the way nuclear war would. He will not make 50- or 100-year predictions. Concrete present dangers include recommendation side effects and algorithmic bias, as more decision processes are delegated to hand-crafted or learned systems with little supervision. Society is restructuring into an automated information-processing society. The best move today is raising awareness, and he thinks that is working: three years earlier almost nobody talked about algorithmic bias; now companies and people in Congress do, even if not always seriously. On encoding human values into recommenders, he thinks the topic will be objective-function engineering. For now loss functions are naive because the hard part is everything else. As that everything else is automated, human attention moves up the stack to what drives the learning system. Loss-function engineering is probably going to be a job title. Keras, in that picture, is the interface between the data you collect and your business goals; the engineer’s job is to express those goals as a loss and a set of constraints.

Does creating AGI excite, scare, or bore him? Intelligence can never be fully general; at best it has some degree of generality, always with specialization, as humans are specialized to the human experience. He is never sure whether people who say AGI mean very smart AI, smarter than modern humans, or human-like intelligence — different things. Human-like AI is intellectually exciting not so much for what it could do, but because building it would mean we actually understand human intelligence. That kind of system would require emotions and consciousness, which an intelligent system does not normally need. Science, as a superhuman problem-solver, has neither. He sees consciousness and emotion on the same spectrum: components of subjective experience meant to guide behavior generation. Human and animal intelligence evolved for generating behavior, including in a social context, which is why we need those features. An artificial system developed in a different context may never need them and may never become conscious. Consciousness is not something that inevitably arises from sufficiently intelligent information processing. It is a feature of the mind; if you have not implemented it explicitly, it is not there. A body is needed for human-like intelligence, though not necessarily a physical one — a realistic simulation would do — and it can only arise in a human-like context, with other humans, in order to demonstrate that it is human-like.

Measuring intelligence, Chollet says, is two questions: how smart, and how human-like — magnitude and direction, the norm of a vector and its angle. He dislikes the Turing test because it is lazy: it completely bypasses defining and measuring intelligence and delegates to a human judge, a total cop-out. To measure human-likeness, make the agent interact with other humans and observe behavior against what a human would have done; those others need not be the judges. Benchmarks that score skill at chess, Go, or Dota are the wrong instrument for magnitude. You can always beat a human at one specific task. Human skill is impressive because it is expressed under constraints — one lifetime, this body. Remove the context, give unlimited training data and unrestricted hardware, and arbitrary skill says nothing about intelligence. You have to define intelligence rigorously. His definition: intelligence is the efficiency with which you turn experience into generalizable programs — a sampling of experience space into the ability to process a larger chunk of it. Skill can be a proxy if you control two things: the amount of experience and the priors. Give two agents the same priors and the same experience; the one whose learned programs cover more of experience space is smarter. On this definition, intelligence has to be specialized, because experience space, segments of it, priors, and experience themselves define the context. You can never look at the totality of experience space. If the space is large enough, specialization can still feel general. Human intelligence is often called general and is in fact quite specialized.

We already understand many of the innate priors humans have at birth. Elisabeth Spelke at Harvard has spent decades on what she calls core knowledge: objectness, agents, goal-oriented behavior, time, number. Chollet has been working for a couple of years, on and off, on a benchmark that assumes those same priors, generates tasks that are new to the agent, and is human-interpretable so people can take the same test fairly. The design constraint is that any amount of practice does not increase your score. The trick is to make the priors explicit, generate fresh samples of experience for each task, and never reuse a task the agent has seen. He calls it a fun side project, not a breakthrough — ImageNet was a fun side project too. Encoding those priors computationally is hard, as vision science shows. Why is the innate set small? Anything we are born knowing had to be written into DNA by evolution. DNA is a very low-bandwidth medium, slow and expensive to write: you need evolutionary pressure, higher-level information takes longer, and the environmental fact must stay stable over that duration, on the order of millions of years. So we might have a visual prior for the shape of a snake or what makes a face, but we cannot have DNA-encoded knowledge of the visual difference between male and female human faces, because that distinction is too recent. Almost all of this innate knowledge is therefore shared with great apes; it is not what makes us special. The brain-relevant part of DNA is on the order of megabytes. That scarcity is, for Chollet, hopeful for a benchmark of strong generalization and abstraction.

Is an AI winter coming? Not a full one, Chollet says. Winters happen when there is a big mismatch between how we sell AI’s capabilities and what it can actually do. Deep learning is creating a lot of value on a wide range of problems, and we are only getting started applying it. What is concerning is the hype: people overselling capabilities, giving the systems a mystical, brain-like aura, and overselling the pace of progress. An exponentially increasing paper count looks like acceleration and is just more people entering the field. If you are raising money for a startup or a lab, you might tell investors that deep learning is just like the brain, that it will consume self-driving and robotics, that AGI arrives in ten or fifteen years. None of that is true. Every time a decision-maker believes it, it is the equivalent of taking on credit-card debt — but for trust. You may raise a lot of money. You are creating damage to the field.

Autonomous vehicles are the worked example. Almost every company, he notes, has promised full autonomy by 2021 or 2022. Around 2015, people raising money convinced the public and decision-makers that L5 was coming in 2016, 2017, 2018; now it is 2019 and we are still waiting. His worry is the meeting after billions invested: do we have autonomous vehicles? No. Did we make any profit? No. The reaction may swing hard, including into other industries. That is an AI winter: backlash when no one believes promises because they turned out to be lies the first time around. It will happen to some extent for vehicles. He still does not believe in a full-on winter, because the technology produces tremendous real value. There will be backlash. Some startups sell AGI as a free lunch: pass a certain IQ threshold and you have infinite value. Investors are buying it; they will wait ten or fifteen years; nothing will happen; human memory is short, and the next generation of investors will not care.

Fridman, having spoken poetically about AGI, gets long manifestos from people who say they have created it or know how. Chollet gets more. The texts often feel generated by an AI, a transformer writing crank papers, with no recursive self-improvement in sight. How do you know they are not onto something, and how do you avoid being narrow-minded about benchmarks? Intelligence, Chollet says, does not exist in the abstract. It has to be applied. If you claim an intelligent system, it should come with a benchmark, even a new one, and should display capabilities — some form of value, even artificial. You do not need to care which crank papers secretly have potential. If a new technique is actually creating value, that will come to light quickly because it is making a difference. The cut is between something ineffective and something useful. Science is full of interesting theories of everything that were completely useless. You do not need to tell interesting theories from useless ones. You need to see whether this is having an effect on something else.

The same applies to quantum mechanics, string theory, the holographic principle. “We are doing deep learning because it works.” Before it started working, people who worked on neural networks were considered cranks; almost no one was in the area. Now it is working, which is what makes it valuable. It is not about being right. It is about being effective. And still, as a scientific mechanism, individuals bear the crank years: Yoshua Bengio and Yann LeCun stuck with it while being called cranks. If you believe you have something, stick with it and see. Fridman calls that a beautiful note to end on.