Einstein Was the Worst of Five Ph.D. Students

Tomaso Poggio is a professor at MIT and director of the Center for Brains, Minds and Machines, with more than 100,000 citations. As a child he treated Einstein as a hero for a very specific reason. Relativity, in the version he loved, came from a thought experiment: a stationary observer exchanging light signals with someone on a train. The force of one mind twisting on itself seemed enough to loosen how time depends on space and speed. That, to the young Poggio, was pure intelligence.

The classroom record did not match the myth. Einstein, Poggio tells Lex Fridman, was one of five physics Ph.D. students in Zurich — and the worst of the five. He was the only one who did not get an academic position when he finished, and went to work, as everybody knows, at the patent office. Poggio's point is not the patent office itself. Einstein was clearly smart, but he was not the top student. He was an anticonformist, refusing the frame that teachers and the other students shared.

Poggio turns that into a working rule for a scientific life. Try to do the opposite, or at least something quite different, from what everyone else is doing. It is true of the stock market: never buy if everybody is buying. He says it is also true of science. When a direction is crowded, the interesting remainder is probably somewhere else.

Fridman asks whether time travel is possible — the kind of childhood physics mystery that once excited Poggio. It would be nice if it were, Poggio says. In science you never quite say no, but given our understanding of time, traveling backward is very likely impossible. Traveling forward is a different story: freeze yourself, or ride a spacecraft close to the speed of light.

Does he still dream of engineering an intelligence that could make those leaps? Some problems, he says, may simply be unsolvable depending on what you believe about physical reality — creating energy from nothing, or going back in time. Building machines that think as well as we do, or better, is another matter. In the short and mid-term, the more realistic path is machines that help us think better, which is already happening with the computers we have. He does not see a principle that would stop computers from eventually becoming more intelligent than we are, though words like intelligence, love, and consciousness have to be disentangled first.

Poggio has called intelligence the greatest problem in science, greater than the origin of life and the origin of the universe, and he is open to arguments against that ranking. The fascination started as a teenager infatuated with relativity. General relativity was one deep problem among many of similar difficulty. Even if he were Einstein, he could not hope to solve them all. So he went looking for a problem whose solution would unlock the rest: an intelligence ten times better or faster than Einstein.

That first motive was utilitarian — artificial intelligence as a tool to expand our capabilities. Soon after, human intelligence itself became the focus of his research. The question is not only about science. It is about the instrument we use to do science: the brain. How does it work, where does it come from, what are its limitations, and can we make it better? In that sense, he says, it is the ultimate question underneath the whole scientific effort.

Fridman asks how far we can get building intelligent systems without understanding how the human brain does it. Poggio reaches for flight. We solved flying without using much knowledge of how birds fly. It mattered to know that something heavier than air could stay up. The Wright brothers did watch birds while designing their aircraft, but you can argue biology did not contribute much to the airplane itself.

Intelligence, he says, is still a bet. Assume we will eventually have a machine indistinguishable from a secretary in what you can ask it to do. Do we get there without knowledge of the human brain, or is understanding the brain the fastest path? People with different backgrounds will wager differently. It is an educated bet, not a theorem.

Recent history, in his telling, tilts toward neuroscience. The main breakthroughs of the last five to ten years started there. Reinforcement learning, one of the algorithms at the core of AlphaGo's win over Lee Sedol in Seoul, traces back to Pavlov, to Marvin Minsky in the 1960s, and to later neuroscientists. Deep learning, also at the core of AlphaGo and of autonomous-driving systems such as Mobileye — a company started by one of Poggio's former postdocs — took its layered hierarchical architecture from the 1960s work of David Hubel and Torsten Wiesel on visual cortex at Harvard. His personal bet is that neuroscience will keep supplying inspiration, not for every future breakthrough, but for some of them.

Until recently Poggio found artificial networks too simplistic next to real neural tissue. He has changed his mind, not because the cartoon is now accurate, but because the comparison class was wrong. Networks of communicating units are much closer to the brain than the older computer-science models of thinking — mathematical logic, Lisp, Prolog. Artificial neurons remain a caricature of biological ones, but they are still neurons: single units talking to other units, something those symbolic systems never had.

The practical gap he wants closed is labels. Deep learning wants a million images, each tagged by a human. A baby may see millions of images in the first years of life without millions of labels from parents. Biology is headed toward one labeled example, not toward infinity: you tell a child "this is a car" once, not a million times. AlphaGo Zero can self-play because Go is a simple world. Vision, which Poggio has spent a career on, is not.

How much of a child's gift is hardware? Evolution, he argues, is opportunistic. Human DNA does not have many more genes than *Drosophila*, the fruit fly, and the fly looks like a machine that is about 95 percent hardcoded. With no extra genes to spend, evolution had to build us as general learning machinery and then insert very weak priors.

Face-selective neurons are a concrete case. Margaret Livingstone at Harvard raised baby monkeys for the first weeks of life so they could see technicians, but the technicians wore masks. In the brain area that usually prefers faces, there was no face preference. Poggio's hunch is that evolution did not ship a detailed face-template circuit — that would take too many bits to specify connections — but a plastic patch primed to imprint whatever you see most often in the first two weeks, especially in connection with food. Those monkeys often saw the technicians' blue gloves during feeding; some cells in the "face" area became glove-sensitive instead.

In the 1950s, neuroscience briefly believed the brain was "equipotential": cut out a piece and nothing special happens except a bit less performance. That view, built on rat and mouse lesions, turned out to be wrong. Stroke in one region can take away speech; another can take away control of the legs. Evidence came from animal lesions, from wartime brain injuries in soldiers, and more recently from fMRI showing which regions light up for which tasks. The brain is also flexible and redundant — functions can sometimes migrate — but the modules are real.

Cerebellum and hippocampus look anatomically different from each other. Cortex is the puzzle. It is the most developed part of the human brain, and different regions handle vision, audition, motor control, and language. In hardware — types of neurons, patterns of connectivity — cortex looks like cortex across those modalities. For Poggio, whether that uniformity is mostly learned or mostly specified is still open, and instructive. An architecture that is good for vision and also good for language implies a shared underlying mechanism. That, he says, is a hint for artificial networks: do not assume you need a bespoke design for every task.

Poggio uses "compositionality" for a technical question: when are multilayer networks more powerful than classical one-layer models such as linear classifiers or kernel machines? If the mapping from input to output — image to label, for example — has a particular structure, deep networks approximate it far more efficiently. That structure is functions of functions. You need not look at all pixels at once. Compute something from small groups of pixels, then compute on those outputs, and so on. Reading a sentence is the same: syllables into words into sentences, not the first letter plus the last.

Max Tegmark, a physicist at MIT, argues that this structure comes from physics: interactions are local, atom to atom, planet to planet. Poggio agrees that is part of the truth and then flips the causal arrow. Maybe our brains are wired as deep networks, so they can solve compositional problems and cannot solve the ones that are not. We test algorithms on compositional tasks because that is the kind of machine we are.

There is a still more biological story. Local connectivity — simple cells looking at a small patch, then cells looking at a small number of those — may exist because long-range wiring is expensive to grow and limited in number. Biology then builds something like a deep convolutional network. It is great for a class of problems that happen to matter for survival. You can even start a successful business on them. Driving, as Mobileye treats it, is a compositional problem.

Cortex looks architecturally like a deep network: local connectivity, hierarchy. Stochastic gradient descent, the simple algorithm that actually fits the weights, is another story. From what we know about cortex, neurons, and synapses, Poggio finds it pretty unlikely that biology is doing gradient descent. Other optimization algorithms almost certainly exist; nobody has a real replacement yet, though people are trying.

Why the algorithm works so well in engineering is less mysterious than it used to be. Classical statistics said you needed at least twice as many data points as parameters, preferably ten times. Today's networks train with ten or a hundred times more parameters than data — the opposite of the old rule, and a standing puzzle about how anything generalizes with so much freedom.

One side effect of that overparameterization: when you minimize a loss, you are not hunting a rare needle. Using Bézout's theorem, an old algebraic result that estimates how many solutions a system of polynomial equations can have, Poggio calculates that a typical deep network has more minima than atoms in the universe. It is like a linear system with more unknowns than equations: infinitely many solutions, and the job is to pick one. Finding a solution and finding one that generalizes are separate questions. Overparameterization at least makes the first step easy.

He was never surprised by the universal approximation theorem — a single hidden layer can approximate any continuous function — because it is essentially Weierstrass's theorem about polynomials, known since he entered the field in the 1980s. The interesting question is how many units you need. For 10 percent error, the count scales as ten to the power of the number of variables. A 200-by-200 image is about 40,000 dimensions: on the order of 10^40000 units, more than the universe can hold. That is the curse of dimensionality. What Poggio and colleagues proved is that a deep hierarchical architecture with local convolutional connectivity, facing a function that itself has that hierarchy, avoids the curse completely.

Fridman asks about generative adversarial networks and unsupervised learning. Poggio is cooler than much of the field. GANs are a somewhat new way to estimate probability densities, and that is interesting. He doubts they will play an important role in intelligence. A lot of the enthusiasm, he suspects, is the shock of realistic generated images. Cool pictures are not the same as cutting the need for labels.

Years ago he trained a network from images onto pose and expression parameters, then inverted the same dataset: numbers in, image out. The generated faces looked quite realistic, with a much less sophisticated mechanism than a GAN and similar quality. For computer graphics, GANs can be useful. For reducing labeled examples, he thinks people are hoping to get out more than they put in. There is no free lunch.

How, then, do we stop supervised learning's march toward infinite labels? Fridman suggests becoming better teachers — picking which examples to show. Poggio likes the analogy. A caricature of computer-science history: first programmers were expensive, then labelers were cheap, and the future should look like schools for children. Biology's version of the curriculum is a weak prior. Most organisms, including human babies, seem to ship with motion detection, conserved from fruit flies on up, something evolution found early. That is why newborns stare at moving things. Motion also segments: a moving object, or an eye tracking it against a shifting background, carves a clean figure from clutter — an ideal training sample. After a few weeks of that, the object is imprinted and can be recognized still, even without motion.

The gap between recognizing and locating objects and actually understanding a scene, Poggio says, is huge. Present algorithms are genuinely useful. He thinks we are in a golden age for low-level vision and low-level speech — Alexa is an example — with plenty of similar work left, including medical diagnosis. Understanding a scene, language, actions, and people is another matter. Despite the claims, we are still a little bit off.

On existential risk — the concerns associated with people Fridman has spoken with, including Stuart Russell, Elon Musk, and Nick Bostrom — Poggio is not against worrying. It is better to worry early than late. Personally he thinks it will take a long time before there is real reason for alarm, and he finds some of the rhetoric misleading. The claim that AI is more dangerous than nuclear weapons, he says, is really wrong. In terms of priority, we should still be more worried about nuclear weapons.

Bostrom has been associated with a long horizon; others, Poggio notes, have said twenty years. Predicting the future is hard enough that he will not pin a number. He would place himself closer to Rodney Brooks: more conservative, further out. If an AGI system were sitting in the room, would we understand its design? Probably not — depending on what "understand" means.

We do not understand what every unit in a deep network is doing, Poggio says, even as theory starts to describe the computation, its limitations, and its properties. That is close to how a civilization understands a fusion bomb: some people know the basic principle, some know the details, and as a community we can build another copy. Understanding, in that civic sense, is whether we can reproduce the thing.

Forty-one years earlier he and David Marr wrote a paper on levels of understanding — hardware and algorithms, in the same spirit as transistors versus PowerPoint. He has since added a level: learning. You can understand how to construct a learning machine and still be unable to describe in detail what it will discover. Children latch onto a kind of music and you do not know why that preference; you still understand the learning process. That, he argues, is already a powerful form of understanding.

Ethics, he thinks, is likely learnable. He distinguishes the ethics of neuroscience — how a neurosurgeon ought to behave — from the neuroscience of ethics, which is what interests him. There is evidence of specific brain areas involved in ethical judgment. You can stimulate those areas with magnetic fields and change the decisions. That is work by his colleague Rebecca Saxe and others. It is early, but if ethics has a neural implementation, it is a learnable problem, and therefore something that can be designed into machines.

Consciousness is hard partly because it is hard to define. Neuroscientists and philosophers argue over whether it requires flesh and blood, whether silicon could be conscious, or whether everything has some degree of it — the line associated with Giulio Tononi. Christof Koch, Poggio notes, was his first graduate student. Most people in AI would say you do not strictly need consciousness to engineer an intelligent system: pass the Turing test and you are done.

Poggio disagrees with that engineering shrug. In an extended Turing test, he thinks consciousness is part of what we implicitly require before we call the other side intelligent. Koch disagrees. On mortality, Steve Jobs argued in his Stanford commencement speech that a finite life stimulates achievement. Rationally, Poggio does not think you strictly need mortality for consciousness, but in biological systems the two seem to travel together. Who knows?

He does not know what the next breakthrough will be. There is a good chance it will again be inspired by neuroscience. The moonshot that excites him is his own Center for Brains, Minds and Machines, funded by the NSF, aimed at visual intelligence: looking around, understanding what is going on, going from here to there without hitting obstacles, noticing other agents. That is already close to being conscious, not quite.

A Google X experiment made the point viscerally. Subjects sat in a chair with a headset and earphones, seeing and hearing through a nearby robot's cameras and microphones. The impression they could not shake was that they were where the robot was. They could look back at their own bodies and still feel that the self had moved. Some aspect of understanding, Poggio says, is the ability to place yourself — self-awareness of your position in the world.