If You Need 60,000 Digits, You Have Not Found Intelligence

Vladimir Vapnik has a definition of intelligence you can fail in public. If someone can match the best handwritten-digit accuracy using about sixty examples per class, on a problem where the leading models still train on all 60,000, then they have touched intelligence. The trick is not a bigger net. It is a handful of predicates that cut the sea of possible functions down until a small sample is enough.

He is the co-inventor of support vector machines, support vector clustering, and VC theory, with work cited more than 200,000 times. Born in the Soviet Union in 1936, he spent decades at the Institute of Control Sciences in Moscow, then in the United States at AT&T, NEC Labs, and Facebook AI Research. He is now a professor at Columbia. This is his second conversation with Lex Fridman, a year after an early episode of the podcast, and just after a technical MIT lecture titled “Complete Statistical Theory of Learning.”

The English title of the episode names three things: predicates, invariants, and the essence of intelligence. The third is the one he has been thinking about for fifty years. Fridman tells listeners the lecture and the podcast are independent. The lecture is math-heavy. The conversation is the more accessible door into the same claim.

Fridman opens with Alan Turing. As the father of AI, Turing may have instilled an ethic of engineering: build intelligence rather than understand it. Vapnik says the two paths are completely different stories. Engineering is imitation of human activity. You make a device that behaves as a human behaves, with the functions of a human, and “it does not matter how you do it.” Understanding what intelligence is about is another problem.

Imitation, he grants, has been fertile. Self-driving cars and the rest of that business were inspired by Turing’s vision, and more will be done. Understanding is harder. It is “more or less a philosophical category. What means understandable?” He believes, with Plato, that there exists a world of ideas. Intelligence belongs to that world of pure ideas. Combined with reality, those ideas produce invariants. Constructing the invariant is intelligence. The predicate comes first.

The world of ideas is small. The world of things is large. One predicate, applied to different data, constructs many different invariants. Pure ideas, he hopes, are not so many. That is why Vladimir Propp’s 1928 *Morphology of the Folk Tale* matters to him so early in the conversation. Propp found 31 “units” that explain behavior in Russian tales, and later people found the same units in television, movie serials, theater. Few abstractions, many projections. The direction of work is the reverse of most deep learning: own the idea, then use it to filter data, rather than induce upward from a pile of examples.

Fridman asks the dumb-sounding question: what is a predicate? Vapnik laughs. Formally they exist. At the simplest level he will use, a predicate is a function. You take the inner product with that function, and that is the predicate. The input is reality — for digits, pixel space. The output is a number that measures how much of some property the example has.

Symmetry is the one he can name. It is not a binary “this is symmetric.” It is a degree: quite symmetric, almost not at all. The letter S has vertical anti-symmetry. The digit 3 is more or less symmetric. Horizontal, vertical, diagonal, anti-symmetry all sit inside the same idea, projected differently. Existence is too crude. Degree is the actual predicate. He has a second, still informal one: how well-structurized a picture is — heavy on the left, not so heavy in the middle. He admits he does not know how to formalize that.

Music critics, he says, already work this way. They have a small vocabulary and rich combinations. Image critics should have the same: a limited set of words that capture the essence of a picture. Those words are not tailored to “2,” “3,” or “4.” They describe a construction that lets you build invariants for a task. The predicate is general. The invariant is the landing on specific data. Intelligence, watching only the world of things, is the attempt to recover the world of ideas — the way a good music critic already has a feeling for that world.

Fridman tries to pull in love, fear, and death as possible residents of the space of ideas. Vapnik will not follow. He wants a very simple story: digit recognition. He does not know whether you need to love and fear death to recognize digits. He does know that to get records from a small number of observations you need predicates, and not special predicates for MNIST — universal ones that understand the world of images, starting with something as small as characters.

The technical heart of the conversation is convergence. You want a function that indicates, say, digit five. You have examples. You can measure a property such as degree of symmetry on the training set, take the average, and then admit only those candidate functions that, applied to the digits, give the same average. You are no longer showing one digit at a time. You are showing a general property of all the digits you have in mind, and throwing away every function that does not keep it.

Strong convergence is closeness of functions: the integral of the squared pointwise difference is small. Weak convergence is looser. Take inner products against test functions in the space; if those inner products converge for every such function, you have weak convergence. You are matching integral properties, not pointwise values. Fourier coefficients are the textbook picture: if all of them converge, under some conditions the function does. A predicate is the integral property you choose to demand.

One predicate carves the Hilbert space once. A few good ones leave a set small enough that scarce samples can finish the job. The predicates are not about the task. They are like Propp’s units, written for Russian folk tales and then found to work on movies and plays. If the cuts are efficient, there will not be many good predicates, because each cut is large. The theoretical extreme: if you used every possible predicate in Hilbert space, you would need no training data — the admissible set would contain one function. In life you use a few good ones, so you still need examples. More good predicates, fewer samples. That, he says, is intelligent learning.

The law of large numbers says that for any one function, the sample average converges to the expectation. Learning needs a stronger version, uniform convergence over the whole candidate set. Otherwise the function that looks best on the sample may be one whose average is a liar. That uniform law, with Alexey Chervonenkis in the 1960s, is work he is proud of. It is not, he now thinks, the deepest layer. “How stupid I was for 50 years. I did not see weak convergence.” He worked only on the strong kind. Weak convergence is what actually manufactures the admissible set.

Can machines discover good predicates, the way Einstein discovered good ones in physics, or is that a human job? Vapnik does not think machines will do much of it. In Hilbert space any function can be a predicate, so there is an infinite supply, and you do not know in advance which are good. What Propp showed, and why people called it a breakthrough, is that not too many cover most of what happens in the world. A sea of predicates; a small number of very good ones.

Deep learning, in his view, has not changed that picture. He knows Yann LeCun’s convolutional network and not much else. Convolution is a single predicate: keep translational invariance, take the derivative, insist the predicate is kept. LeCun did it twenty-five years ago, “and nothing so clear was uttered to deep network.” Vapnik does not understand why people talk about deep networks instead of talking about the set of piecewise linear functions that keep this predicate.

Mathematically, all neural nets are piecewise linear functions, possibly with a huge number of pieces, still a small family next to a reproducing kernel Hilbert space, which is much richer and, he keeps stressing, admits closed-form solutions. Architecture tricks are ways of carving an admissible subset. Convolution carves the translationally invariant part. The invariants are specified by hand. They do not emerge from an understanding of the world of ideas. Attention and language-model focusing, Fridman offers, are also engineering. Vapnik’s reply is steady: engineering is fine, and you cannot explain on the level of ideas why it works.

He is fair to the engineers. The person building a self-driving car does not need the theory; whoever has the better intuition does better, and that is all right. If you want science, you have to go back to the ideas. “As soon as you incorporate new invariants, set of functions becomes smaller and smaller.” The hope is a standard list, like Propp’s, for digit recognition. Mathematicians, he notes with some amusement, fuss with neural nets, while people in literature and art have been using invariants all along. We should learn from how they describe music.

Fridman walks through Propp’s sequence so the audience can hear it: a family member leaves home; an interdiction is laid down; the hero violates it; the villain reconnoiters; the plot clarifies; the thirty-first unit is a wedding, happily ever after. Vapnik’s interest is the count. Thirty-one abstract units for the whole of that narrative behavior, and the same units travel to detective serials and theater. If human storytelling compresses that far, he has reason to hope Plato’s world is small.

Fridman mentions Claude Lévi-Strauss and other readings of narrative. Vapnik is not in that business. Philosophers argue. Another set of units is fine. What matters is that some small set exists. Whether those units would still hold if handed to an alien species, without a human brain to interpret them, is a question he turns back into digits. If you can recognize the characters, you understand something: symmetry, maybe structure. He does not think you need thousands or millions of predicates for handwriting. Theory says that using all predicates needs no data. Using a few good ones needs some. The rest is the challenge: let the people who beat the record say which functions they needed.

A good predicate is one that decreases the admissible set as much as possible, which is to say it drops VC dimension, the rough capacity of the set, until little training data will do. The duck test is his toy example of the difference. Looks like a duck, swims like a duck, quacks like a duck: those apply across the class and cut a large region. “Play chess like a duck” is formally allowed — any function can be a predicate — and it barely reduces anything. Form is cheap. Effect is the score.

Fridman keeps asking the young-grad-student question: how do you find these things? Vapnik’s answer is borrowed from physics. Find a situation the existing theory cannot explain, then remove the contradiction. In a talk he gave the diabetes data as the example. They reached some accuracy, then looked for regions where a formulated predicate did not keep an invariant. Retrain, keeping only functions that hold the invariant, and performance improved. He knows how to do that technically. A machine can run the loop. He is not sure it will construct the smartest predicate.

He calls it a brute-force way, not necessarily the best. Contradiction here means: this function, if you use it, does not keep the invariant. Include the predicate, rebuild the admissible set, solve again. It is the same logic as forcing a new theory out of the place the old one fails, rather than inventing from zero. Symbolic AI and 1980s expert systems, Fridman wonders — can deductive reasoning discover good predicates automatically? “I don’t think so. I think that just logic is not enough.” You have to know reality. You have to know life.

Common sense, Fridman suggests, is just rules. Vapnik will not grant that. Common sense is mortality, fear of death, love, spirituality, happiness and sadness, all tied up together. He does not want to discuss it. He wants to discuss digit recognition, because there is a challenge, and it is doable. Every time Fridman brings up love and death, Vapnik brings it back to MNIST. The large version of the question — understanding life — is not a job for logic. The small version is a job you can hand to a student.

How hard is two-dimensional image understanding? If someone solves the handwriting challenge, how long is the journey to natural images? “Immediately,” Vapnik says. Matching the best nets at a hundred times fewer examples is not free. You will have had to put in some idea. Fridman objects that handwriting feels flat, while most photographs are a 3-D world crushed onto a plane, which seems to demand common sense. Maybe, Vapnik says. Let’s start from simple. He cannot think about things he does not yet understand. Einstein’s line is the exact case: as simple as possible, but not simpler. MNIST is simple in idea and not simple to get.

He did not start from a toy and hope to grow up. He started from Plato, and only recently understood Plato, he says, once weak convergence and predicates were in view. “Oh, this is what Plato thought.” The line he draws is Plato to Hegel (“whatever is reasonable exists”) to Eugene Wigner, then suddenly to Propp’s 31 units. Understanding is the abstract ideas you hold and apply. In machine learning the reality those ideas project onto is data. Fridman finds the sea of data compelling, and wonders about selecting the right examples. Vapnik’s thing is simpler. You have a huge set of functions and not too many points. Pick a function that merely fits the points and you will do badly. Decrease the set first. That is what weak convergence is for. He does not treat data as the interesting optimization space.

Sixty examples, he hopes, is already in the regime of the ordinary law of large numbers, so he does not need the uniform law for the predicate averages. He still needs some evaluation and balance. Language he will not touch. He does not know Chomsky well. Human language is huge, “not for me, and I believe not for our century.” We should learn a lot from simple tasks like digit recognition. Even current MNIST records, in his view, show the field going in the wrong direction, because 60,000 examples is unacceptable, however small the error. From very small examples you will find principles different from what is used now: weak convergence, not only strong. Those principles, he thinks, will be human-interpretable — degree of symmetry, not a yes/no, and not only for handwriting.

Asked for the most beautiful idea he has met in statistical learning, Vapnik names uniform convergence: the law of large numbers for a whole set of functions, without which the minimizer on the sample can be a function that does not converge. Then he returns to the late conversion. In Hilbert space there are only two forms of convergence, strong and weak. Using both, on a rich space of square-integrable functions, you can apply them to learning and have a closed-form solution. Computational simplicity is, for him, a sign it is the right way. What remains is not statistics. It is the concept of a predicate.

Fridman likes that Vapnik wants heuristics out of the system. Vapnik’s correction is quieter: when you use the right instrument, closed form happens to appear. He is looking at his own bounds and wondering whether radial basis kernels need a correction to fit them better. Human-level intelligence as a closed form? He does not quite take the bait. He does grant Fridman’s engineering picture for self-driving cars. The great person who does that job does not care what theory sits behind it. You cannot write a rule for every situation. More abstract rules, fewer literals — the same story as ideas applied to cases. You still have to reach for the ideas if you want the science.

Can neural nets be made to reason — sequentially, with recurrence, mangling representations until a conclusion? Vapnik’s habit is to write down what he wants, formalize it, then look for a method. So far he has not seen a situation that required recurrence. Human reasoning is too complicated for him. The most difficult part, he says, is asking good questions, and he does not know how people do that.

Machine learning, Vapnik has said, is not only technical. It is also philosophy. Philosophy, for him, means understanding of life, at the high abstract level of someone like Plato, and then implementation. Finding that they needed weak convergence was not simple. He has thought about the same problem for fifty years, trying not to be too enthusiastic, concentrating on what he could not achieve, asking why. Only two ways of convergence exist in the math he trusts; they had been using one. People already speak weak-convergence arguments in ordinary language — the duck story — and then ignore them when they formalize. Why, for fifty years from the start of machine learning? Textbooks, he notes, are still about bounds and empirical risk minimization. They do not look for the admissible set.

Fridman asks for a childhood memory, in Russian. The answer is music. Vivaldi first, then Bach, a feeling of being hit. Back in English, he says that in Bach you can feel structure. Structure is a predicate he has not formalized. Critics writing about Bach use one vocabulary; Chopin, a very different one. Collect those words, compare them with the words used for painting and poetry, and you might extract predicates for images, even for digits. Critics describing music are naming high-level Platonic ideas behind the sound. Going from ideas to a representation is the easy direction. Going back is an inverse problem, hard, but not unsolvable. Music critics and poetry professors solve it daily without writing it as mathematics.

He has already run a version of the experiment. A friend, a professor of Russian poetry who had published collections, was asked to describe each of a small set of MNIST images using only the language of images in Russian poetry. She did it. They then used those descriptions as what Vapnik calls privileged information: the same digits in two languages, pixels and poetry, fed in at training time. Recognition got much better. She later died. The digits and the poetic descriptions remain at NEC. As soon as you take the challenge seriously, he says, it connects to all of this.

Does he ponder mortality? Yes. Is he afraid of death? Not too much. The pity is the work he will not get to do: sit with people from the musical tradition, collect how they describe music, do the same with art, take what is in common, and try to understand predicates that are “absolute for everything.” Fridman says there is still time. Vapnik says it takes years, and the slow part is not making the collection. It is understanding what is in common, thinking again and again.

Asked for the meaning of life, he does not know. He mentions the Strugatsky brothers, the Soviet science-fiction writers, and their idea that humanity is splitting into two types, ordinary people and very smart people, branches that will soon go different ways. He offers it without a verdict. He also talks about a Russian literary theorist. When that man analyzes literature, relationships, life, Vapnik feels like a small child. The man knows the big predicates of life. Then an observation: many managers in big companies studied English language and English literature. Why? Because they understand life. They understand models. They are, by temperament, critics of narrative. That, he says, is big science, like what Propp did.

Fridman calls him one of the great mathematicians and statisticians of the time. Vapnik will not have it. He is modest about himself. He sees smart people around. “I know my limits.” The whole conversation has been that boundary drawn carefully: he can think clearly about handwritten digits, about convergence, about invariants. He does not understand language, human emotion, most of the humanities. Seeing what he does not know is, for him, as important as seeing what he does. Fridman’s closing quotation from Vapnik is the same ethic in one line: when solving a problem of interest, do not solve a more general problem as an intermediate step.