The Hardest Part of Learning Was Left Out of the Textbooks

Lex Fridman opens with Einstein’s line that God does not play dice, and asks Vapnik — a man who spent a career looking at the world through statistics — whether the fundamental nature of reality is random. Vapnik’s answer is almost casual. You do not know some facts, he says, and because some of those unknown factors may be important, it looks as if God is playing dice. The appearance of chance is a statement about ignorance, not a verdict on the universe.

He immediately splits philosophy into two camps: *instrumentalism*, which builds tools suited for prediction, and *realism*, which tries to understand what the description actually is. Take a set of mechanical laws. Are they God’s laws, true everywhere, or merely formulas that let you predict the next position of a moving body? If you believe they are God’s laws, they hold always and everywhere, and you are trying to understand. If you only want to predict, the world you see is a different object.

The distinction sounds abstract, but it is the ruler he will use on machine learning, deep learning, and the word *intelligence* itself. Vapnik is the V in VC dimension, a co-inventor of support vector machines and support vector clustering, born in the Soviet Union, later at AT&T Labs, Facebook Research, and Columbia. The conversation is not a victory lap. It is a method: stop confusing a useful instrument with an account of reality.

Inside the models of machine learning, Vapnik says, the same fork appears. From the instrumentalist point of view everyone takes the obvious path, because the stated goal of machine learning is classification error — “point zero for classification,” in his phrasing. That is true, and it is an instrument for prediction. But he wants a different goal: to learn the conditional probability.

For a given situation he would like to know the probability of each outcome. For prediction he does not need that. “I need the rule,” he says, “but for understanding I need conditional probability.” A decision boundary tells you the answer on this sample. A conditional probability tells you how the world is shaped at that point. One can top a leaderboard. The other is what transfers when the setting moves.

Fridman cites Eugene Wigner’s 1960 paper on the unreasonable effectiveness of mathematics in the natural sciences, and admits that years of applied deep learning had made him feel he was missing the beauty math can uncover. Vapnik believes the claim. Some people say mathematics is the language God uses; he believes that. Magical structures, looked at carefully, already know something about reality. Natural scientists stare at an equation and try to understand the world. The same logic, he argues, applies to the equations that define conditional probability: they reveal more about reality than fantasy does.

Simple principles are hard to discover, he says, and then, once found, surprisingly beautiful — and surprising that nobody saw them earlier. Working step by step through equations, a new term appears; after thought, it describes the position of the observation point, the place where a great deal of information is thrown away. People look at residuals, not at the composition of the observation points. The idea is simple once you have it. You can derive it from the algebra. That, for him, is the proof: human intuition is not rich. It is primitive. It does not see very simple situations.

Fridman pushes back: are there leaps of ingenuity that outrun the math, with the math catching up later? Vapnik does not think so. The best human creation is putting down axioms; the rest is technical work, seeing where the axioms take you, polished across generations. Imagination of the kind that invents “features,” or the kind he associates with deep learning, is “not relevant to the problem.” From the equations you derive a theory that goes far beyond what people can imagine. Imagination is interpretation. Interpretation is fantasy. It is not what you need.

Asked whether we have the tools to describe mathematically what happens in the human brain during learning, Vapnik refuses the premise. That project is not a description of what is going on. It is an interpretation, and the interpretation can be wrong. He tells the story of the first microscope. Its inventor kept the instrument secret, then sent a report to London. Looking at blood, he described a fight between queen and king. He had seen red cells. He imagined armies. The Royal Society looked carefully, because they believed he was right — and he was right that he had seen something. He gave the wrong interpretation.

Vapnik believes the same thing can happen with the brain. The observation may be real; the story wrapped around it may be invented. He then turns to a proverb: one day with a great teacher is better than a thousand days of diligent study. Ask what the teacher actually does, and almost nobody knows. That unknown, he says, *is* intelligence. History and large-scale machine learning still show that a teacher can do a lot. Mathematically, a teacher can introduce predicates and invariants. How, he does not know, because the teacher knows reality and can describe its forms. He does know the payoff: with invariants, the number of observations can fall by a factor of a hundred.

Fridman offers the piano teacher who says “play like a butterfly.” The language does not seem to contain the instruction, yet it changes how a student plays. Vapnik believes that remark is a predicate, though he cannot yet say which one. “That’s exactly what intelligence and machine learning should be,” he says; the rest is mathematical technique. What was discovered recently, in his account, is that learning has two convergence mechanisms. The older one is what people have always used. The newer one is strong convergence, and it can take predicates. Say “like a butterfly,” and the playing changes at once.

“If it looks like a duck, swims like a duck and quacks like a duck, then it is probably a duck.” Vapnik calls the saying exact. Those are three predicates, each legitimate. You could equally say “jumps like a duck.” That predicate is grammatical and legal, and it is useless, because ducks, birds, and animals do not differ in a way that jumping would reveal. The live question is how to recognize a predicate that is not useless. Existing machinery, he says, does not use predicates at all. Give industry zillions of data points and it will probably still use only three.

Fridman wonders whether “swims like a duck” secretly depends on millions of other facts picked up along the way. Vapnik does not think a giant knowledge base has to sit under every phrase. In the machine-learning picture the mechanism is simpler. You begin with a large set of functions. Training data tell you how a duck should look; you remove every function that does not look that way. A second predicate shrinks the set again. Then you pick the best function that remains — ordinary learning, on a set that has already been cut down.

That is why you do not need so many examples if the predicates are good. “Every predicate is invented to decrease [the] admissible set of functions.” A child recognizes a duck after a handful of birds because someone already handed over “swims like a duck” and “quacks like a duck.” The samples only finish the selection. A predicate that does not shrink the set is “jump like a duck”: well formed, and silent.

An admissible set, Vapnik says, should have small capacity — small diversity, small VC dimension — and should contain a good function. Fridman notes for listeners that Vapnik is the V in VC, and asks for a lay explanation. Vapnik describes it as the capability to pick one function from the admissible set. That set can contain all continuous functions and be useless, because you will never have enough examples to pick. Or it can be small, the functions not very different from one another. He would rather say *diversity* than *capacity*.

The goal of learning, as he formulates it, is to create an admissible set with small VC dimension that still contains a good function, then pick that function with few observations. Classical statistical learning theory, he says, does not include the creation of that set. In the textbooks, a hundred percent of the time, the admissible set is given. That omits the size of the difficulty: constructing a finite-dimensional set that still contains something good “was out of consideration.”

The process he wants is invariance. You compute the average of a function on the training data and the expectation of the same function on the model; they should coincide. The constraint is not for one function. It should hold for all functions in the set. That is the strong-convergence mechanism, and it is not the uniform law of large numbers that standard algorithms require. Which question you ask — jump like a duck, or look like a duck — is the intelligent part of the business, the specialized predicates a teacher contributes.

Asked about neural nets and arbitrary architectures, Vapnik reaches for Churchill’s history of the Second World War. In old times, when a war ended, the great kings — almost all relatives — sat down and knew how to make peace. After the First World War the general public came to power, greed filled the table, and everyone could see it was not peace, only a pause of twenty years, because the people at the table were not professionals. He analogizes: mathematicians in Washington looked at problems from a deep point of view. Computer scientists, he says, mostly do not know mathematics. They invent interpretations.

Deep learning is his exhibit. Why it works, they do not know. People in deep learning, he claims, know neither the mathematics nor the neurons. “It is just function.” If you want to say piecewise-linear function, say that, and work in that class. Instead they invent words, then try to prove an advantage through interpretations that are mostly wrong, and finally appeal to the brain, about which nobody knows what is going on. A result he calls *representativity* says, in his account, that the optimal solution of the learning problem sits on a shallow network, not on a deep one.

Fridman asks whether the interpretation deceives you, whether there is no value in imagining kings and queens as a spur to later math. Vapnik thinks the public discussion of deep learning has been a discussion about interpretation, not about what can actually be said. There is, he insists, more than one way to converge, including a strong way that requires predicates. Go through that, and “you don’t need [deep learning].” The success is not the target of the insult. The story told about the success is.

Fridman tries another angle: never mind mathematical beauty — is Vapnik not surprised that it works at all? He asks about AlphaGo, a system that used neural networks to estimate the quality of a board and beat the best human player. Vapnik first catches the wording. “Is your interpretation quality of [the] board? Yeah.” Then he shrugs at the achievement: “It’s not a very difficult problem.” Fridman says, in effect, that we have only discovered this empirically. Vapnik agrees he cannot argue with the empirical fact.

What he will argue is efficiency. Deep learning, as people use it, consumes zillions of training data, and “you don’t need this.” He poses a challenge: take problems that deep learning already does well, and do them with a hundred times less training data — on some problems, he adds, deep learning cannot, because it is not necessary if you create the admissible set properly. Creating an architecture is supposed to create a reasonable set of functions, but you cannot say you have created a *good* set. That is fantasy. It does not control much. Mathematicians, given training data, *can* create a reasonable set.

He pauses on the statistical distinction. Existing algorithms need the uniform law of large numbers, which is much harder than the ordinary law. Use invariants and the strong way of convergence, and you can cut the training set sharply — the same “swims like a duck and quacks like a duck” idea, now stated as a sample-complexity claim rather than a proverb.

Fridman notes that human intelligence evolved without anyone installing a model of admissible functions in the brain, and invites the “useless question” Turing asked and then rejected: can machines think? Vapnik answers through Turing’s own move. Turing set up imitation: if a computer can imitate a human being, call it intelligent. He understood, Vapnik says, that this is not a thinking computer. He set up the problem of imitation. “Now we understand that the problem [is] not in imitation.”

Vapnik is not sure intelligence is only inside us. It may also be outside. His evidence is the familiar simultaneity of discovery. Something is very difficult; then, within a few years, people in several places prove the same thing. Non-Euclidean geometry arrived in a roughly ten-year window — Lobachevsky, Bolyai, Gauss, and others. He has seen many such episodes among mathematicians. When they develop something, they develop something general that affects everybody.

So maybe the model of intelligence as a private interior fact is incorrect. “It’s our interpretation.” Maybe there is some connection with a world-level intelligence; he does not know. Fridman says it sounds as if one is plugging into a big network and contributing a node. “Exactly,” Vapnik says. The point is of a piece with the microscope: the frame we put around intelligence may be another invented story.

Fridman asks about complexity, worst-case running time, and whether P versus NP is interesting. Vapnik says the question is interesting, then talks about the people who arrived in the United States around 1990. They did not know the history and did not know statistics. In Russia two monographs had already been published; in America that work was unknown, so it was learned again. Someone told him a worst-case style would create chaos. It still has not, he says, because it is more logical to do only what your mathematics allows when the object has a clear understanding and a clear description. Complexity was introduced for that reason.

From that point of view you can describe the best possible case and the worst possible case and draw models. The middle is “not so interesting,” not because it does not matter, but because it is not easy to get good examples, and there are not many cases where the ball is exact. Fridman offers an analogy: judging a life by the worst thing and the best thing a person did, versus everything in the middle, seems unproductive. Vapnik does not accept the moral. You cannot describe the situation in the middle, or it is not general. You can describe the edges; they are clear; they admit a model. You cannot build a model for every new case.

Pressed on whether the real world is a long tail far from the mean, he turns back to method. Formal statistics uses the uniform law of large numbers. Invariance uses the ordinary law of large numbers, and “there’s a huge difference.” Three predicates may suffice for a duck; a purely empirical distinguisher would need a lot of observations. That means “looks like a duck” contains many bits — more bits than we know how to count from samples. He dislikes the usual framing of artificial intelligence: codes that imitate human activity. “It is not science. It is applications.” Useful, but not the thing that has to be learned.

Have people tried to develop predicates like “seems like a duck” or “play like a butterfly”? A lot of what a teacher says, Vapnik notes, simply arrives in the mind; how the teacher chooses is the problem of intelligence, and it is connected to learning. He wants the work split into two stories. One is mathematical: if you have the predicate, you can do something. The other is how you get the predicate. That is the intelligence problem. People have not even started to understand intelligence, because the first job is to understand what teachers do — why one teacher is better than another.

The better teacher is not better because of repeating the textbook. The teacher makes remarks, and offers a philosophy of what is being said. In his model, one teacher can give a very good predicate — “swims like a duck” — and another can say “jump like a duck,” which carries zero information. Formulating *that* question is the open problem: what the teacher does, at every level, that makes the teaching better. He has just finished the invariance story and hopes it will last. He believes he can show there is no other statistical mechanism, only cosmetic improvement. The intelligent part he still knows nothing about.

On the last slide of a talk, Vapnik put a challenge. Deep learning claims about 99.5% correct on MNIST with 60,000 observations. Can you do the same job incorporating invariance, using on the order of a hundred times fewer examples? When statistical learning started, he says, it was clear that machines used much more training data than a human needs. The story has returned. Deep learning uses subterranean amounts of bait because “vigilance is not enough.” With good invariance you may never need to collect so many observations. The statistical part is ready: supply the predicate, and a small sample will do. The first challenge is digit recognition, with variance thought through.

He sketches predicates for digits: horizontal symmetry, vertical symmetry, diagonal symmetry, even how dark the whole picture is. Each one shrinks the set. Fridman objects that telling a 2 from a 3 seems to require a childhood — years of playing, school, parents, walking, jumping, looking at ducks — before the right predicate appears. Vapnik is blunt: “I know for sure that you must know something more than digits.” Maybe several languages of description, elements of digits, geometric properties, something abstract. He does not know yet. That unknown is the intelligence problem. Formally, one labeled example can cut more than one bit, perhaps half the functions; one good predicate can remove much more than half.

Fridman hears poetry and philosophy in the mathematics. Vapnik’s reply is “ground truth,” visible everywhere. Philosophers sometimes go deep and sometimes go completely off the subject. Music has ground truth: structure, classic, clear, simple, the same feeling as watching axioms become a theory. Asked for his happiest hours as a researcher, he says every time you understand something. Most of the time he is wrong. You go again and again, honest in front of yourself, refusing interpretation, checking that the thing relates to ground truth and is not “blah, blah, blah.” Excitement is allowed. It has to be double-checked.

Twenty years earlier they had found a statistical idea; nobody believed it except one person. Twenty years later it was fashion: support vector machines, kernel machines. While they were working they already felt it would last. “We immediately recognized that it will last forever.” The invariance story feels the same, because he has a proof that there are no different mechanisms, only cosmetic improvements. Put invariance and statistical learning together and you can also formulate what intelligence is, and separate it from the technical part. That separation, for him, is the point of the whole conversation.