Why Deep Learning Needs Rules: Chollet on Interpolation, Abstraction and Hybrid AI

Lex Fridman opens by asking about the current limits of deep learning, noting that Chollet has contrasted local generalization, which networks do, with the extreme generalization humans manage. Chollet says the limit is built into the design. Deep learning models are huge, differentiable parametric models that map an input space to an output space and are trained by gradient descent, effectively learning a continuous geometric morphing one point at a time.

Because the mapping is learned point by point, a network can only make sense of inputs that sit close to what it has already seen. At best it interpolates between training points. That means training needs something close to a dense sampling of the input-output space, which gets expensive for complex real-world problems such as autonomous driving or robotics, where even a limited slice of visual space can take millions of examples.

Chollet contrasts this with symbolic rules. A rule is abstract rather than obtained from point-by-point mapping, so it can apply to a very large set of inputs. Trying to learn sorting with a neural network means learning the sorted form of each specific list. A few lines with two nested loops can sort any list at all, which is why Chollet expects the future to combine the two approaches.

Asked where the combination should happen, Chollet admits this is territory of ideas and intuitions rather than settled answers. Still, he says the most successful AI systems today are already hybrids. Robotics systems are mostly model-based and rule-based, relying on things like planning algorithms. They use deep learning as perception modules, or sometimes to inject a fuzzy intuition into a rule-based process.

Self-driving cars are his main example. A single end-to-end neural network would not work, because training it would require a dense sampling of driving experience, which is completely unrealistic. Instead the car is mostly symbolic software programmed by hand, built around explicit 3D models of the surrounding environment. Deep learning interfaces with the real world, converting raw sensory data into something the symbolic systems can use.

Fridman pushes on lane following, suggesting that steering might be learnable end to end. Chollet says he does not jump to the extreme so easily and thinks it is doable. In general, he says, there is no hard limitation on what a deep network can learn as long as the search space is rich and flexible enough and the sampling is dense. The catch is that dense could mean anything from 10,000 examples to trillions.

Fridman asks whether the Turing test could be solved by a neural network alone. Chollet says the test is mostly about tricking people, which exploits human perception more than it measures intelligence, and mimicking intelligent behavior is different from engaging in it. For a looser goal such as sustaining a 20-minute conversation with all its tangents, he thinks deep learning alone would be very challenging, though he would not rule it out.

In theory, he says, the space of problems a large network can solve is infinite. In practice deep learning is a great fit for perception problems and, more broadly, any problem where explicit hand-crafted rules are impractical and exhaustive search through program space is not possible. That covers perception and a kind of artificial intuition. Physics, by contrast, can be learned by a network only inefficiently, since an explicit rule-based model is a far better and more compressed representation than a table of situations and outcomes.

Fridman asks whether programs encoding that kind of reasoning could be generated automatically, noting that old expert systems failed because so many facts had to be encoded by hand. Chollet says this is the field of program synthesis. It is harder to learn statements about the world than mathematical ones, because worldly statements tend to be subjective. Today we do not really know how to do it, so he is personally optimistic about genetic programming as a promising direction.