The Fallacy of the Successful First Step

In 1966, Seymour Papert at MIT wrote a proposal called the Summer Vision Project, to be given, as far as anyone knows, to ten students who would work on computer vision and solve it that summer. The outline already named many of the tasks the field still works on. Lex Fridman opens with that document and asks Jitendra Malik, a Berkeley professor and one of the central figures in computer vision both before and after the deep-learning revolution, why researchers keep underestimating how hard seeing is.

Malik's first answer is psychological. Most of what people do in vision they do unconsciously, and that effortlessness creates the feeling that the same thing should be easy to implement on a computer. Early AI researchers got it badly wrong for that reason. Look at neuroscience or the psychology of human vision, he says, and the complexity becomes obvious: a very large part of the cerebral cortex is devoted to visual processing, and the same is true in other primates. Once you take that perspective, it is clear the problem is challenging and will take time.

Proving a theorem or playing chess makes the difficulty evident because the conscious brain is doing the work. Vision does the same kind of processing below awareness. For the field itself, Malik is less forgiving than he is of the 1960s. In the early days every part of AI was treated as too easy; today that excuse is thinner. People still fall for what he calls the fallacy of the successful first step. Many vision problems give you 50 percent of a solution in a minute, 90 percent in a day, 99 percent in five years, and 99.99 percent maybe not in your lifetime.

Vision, Malik argues, operates at every level. Older human-vision literature split sensation, perception, and cognition, and then tried to study only perception while dismissing harder problems as cognitive. He thinks those divides are artificial. Today's techniques work better at the lower and mid levels; the higher, cognitive levels still have to be confronted in many real applications. How much that matters depends on the setting. Image search can tolerate wrong results. Driving at 60 miles per hour cannot.

He is a pessimist on fully autonomous driving in the near future, because there will be that 0.01 percent of cases where quite sophisticated cognitive reasoning is called for. Subsets of the problem are solvable. Freeway driving had demonstrations going back to the 1980s, from Ernst Dickmanns in Munich, then Carnegie Mellon, Berkeley, Stanford and others. The challenge is an autopilot that works under all kinds of conditions, and at that point it is not just vision but control and every edge case. Fridman notes that even highway driving has to relinquish human backup before you really feel those cases.

Tesla's Autopilot, with eight cameras and a multitask network people call HydraNet, is the current bet that driving can be converted into a vision-and-learning problem. Malik includes predicting other agents inside vision. Perception, he says, always has to tell us not what is now but what will happen, because the present is already over and we act in the future. He owns a Tesla and has watched it give the wrong response to a skateboarder: it could not predict what skateboarders typically do, as opposed to ordinary pedestrians. In principle enough data could fix that pattern. Current computer-vision systems, he adds, need far, far more data than humans do for the same capabilities.

Would a system that eventually solves driving look like today's neural nets, only with more data and compute? Malik says that is a scientific hypothesis, and he would not bet on it. What has been effective in the deep-learning paradigm is tabula rasa learning: a blank slate given a series of experiences and a lot of supervised pairs. Human driving is not that. At sixteen a teenager walks into driver ed already a visual genius. Most of that repertoire, he thinks, has probably been achieved by age two: the world is three-dimensional, objects look different from different views, occlusion, the common dynamics of bodies, some intuitive physics.

What driver ed actually teaches is control—smooth steering and brakes—and a sense of typical traffic. That education can be short because the student is not learning the visual world from scratch. Novel situations will still arrive: a skateboarder never seen in class, a truck whose back opens and dumps junk. A human driver can deal with those because of general visual knowledge. Malik sees no in-principle problem with neural networks accumulating that kind of knowledge, but the learning techniques would need to evolve far past supervised x-to-y mapping.

He credits a paper by Smith and Gasser for six lessons from children that are common wisdom in developmental psychology and still not common wisdom in computer vision: be multimodal, be incremental, be physical, explore, be social, use language. His job, as he sees it, is to bridge those worlds. The example he likes most is multimodal. A child holding a ball gets a sparse tactile signal of three-dimensional shape and a visual signal of the same object, in totally different receptor spaces. That cross-calibration is a very strong self-supervisory signal. Nobody assigns a label. The mother does not have to say the word ball.

Audio and vision work the same way. A glass bowl falls, smashes, and the child hears the sound and sees the pieces. Time is what links the two: at a moment T1 a signal in audition and a signal in vision are likely causally related. That has become a hot topic, including classic cocktail-party source separation, which gets much easier with a visual stream. The child also commands movement, looks at an object from different viewpoints, and runs little experiments. Supervised learning, psychologists tell us, is a small part of how humans actually learn.

Asked whether language or vision is more fundamental, Malik does not hedge. Vision is the parent. You can argue it from phylogeny or from ontogeny. The first multicellular animals with something like intelligence arose about 500 million years ago; they could move and they could see, and the two helped each other. Fast-forward to the last seven million years of the hominid line. Lucy, about three million years ago, walked, which freed the hands for manipulation and tools while brain size grew. Other primates do not have human language; Chomsky is right that it is uniquely human. Malik's claim is that language still arrived on a substrate that already had objects, spatial relations, causal interactions, and time.

All human languages have constructs that depend on space and time, and those notions had to come from perception and action in the world. Infants get visual understanding of the world before they get language. That is why Malik can say vision is older and more basic, and still say, a few minutes later, that language will be the hardest nut to crack. The circuits and physical priors of vision are relatively well posed. Language understanding piles on common sense, intention, world knowledge, and context, some of which researchers cannot even formulate cleanly.

He does not want a single test of intelligence, any more than he believes in IQ as a single number. Turing's 1950 imitation game made sense for the problem Turing was trying to solve. Seventy years later Malik thinks it is the wrong way to channel AI research, because it sends people down the path of a chatbot that can fool you for five minutes. He would rather have a list of roughly ten tasks: manipulation, navigation, visual scene understanding, reading a novel and answering arbitrary questions. On the visual side, a test he likes is a really good aid to a blind person—vision coupled to action, in the real world, not only in simulation.

David Hilbert in 1900 proposed 23 open problems in mathematics. Malik has presented Hilbert problems of computer vision, and Fridman asks him to name them again. The first he regards as clearly unsolved is long-form video understanding: a clip in which you understand agents, their goals and intentionality, and can predict what might happen. Short-range atomic actions—sitting, standing, catching a ball—can be done now, or at least improved from 50 percent to 65 percent next year. Long-range understanding blends into cognition. You have to find entities, track them, and model goal-directed behavior.

That is where schemas come in. The classic example is a restaurant: you walk in, someone seats you, a waiter brings a menu, food arrives, the bill arrives. 1970s AI called this frames, scripts, schemas, and hand-coded the stages and actors so a story about eating out could support a pile of inferences. Malik thinks that kind of knowledge is essential for long-form video, and that 3D convolutions over a couple of seconds of clip are tailored to the short term. He does not want to revert to 1970s hand-coding. There must be a learning version, because a child is not typically coached through every restaurant stage; the family goes ten times and the schema is there.

The second Hilbert problem is understanding the world in 3D. It sounds paradoxical, because multi-view geometry could reconstruct scenes decades ago. Those methods fail from a single view. The alternative, guessing 3D from one image, is supervised on CAD models at training time, which Malik calls completely unnatural supervision: CAD models are not injected into a brain. What he wants is learning as you move around the world. A succession of views of chairs and tables builds an internal representation, and later a single photograph of a different chair is enough to guess its shape. The CAD model need not exist as graphics geometry; it is hidden in the ability to predict what you would see from another position.

Fridman asks whether Malik is comfortable with networks that achieve 3D understanding you cannot visualize or interact with. Malik says that is fine, with caveats. Humans are not explainable to one another. There are settings where explainability matters, and medical diagnosis is his example. A program recommends treatment A or B; you want to know whether the training data were American men in their thirties and forties, whether the reference class matches the patient, and how large the error bars are. What he wants in those settings is less a tour of internal weights than error bounds and a sense of decision quality.

Where he is willing to sacrifice interpretability is highly performant systems that are internally black boxes, which is where the best systems already seem headed. He and Fridman are black boxes to each other. Charming people, Fridman notes, do not open the box; they tell a story that convinces you they know what is going on. Neural networks may only need to generate those stories. Transformers, they agree, will be involved.

Will we ever build human-level or superhuman intelligence, the thing Turing thought might arrive by 2000? Malik splits the question. In principle, at some time, yes. In the next twenty years, no. He borrows Donald Rumsfeld's known unknowns and unknown unknowns. In vision and robotics he feels he has known unknowns: a sense of where to go. In natural language understanding and high-level cognition there are unknown unknowns, which makes any timeframe a fantasy. He was positively surprised by how well deep learning worked after 2010—non-convex, overparameterized systems that, against the convex-optimization lore, trained robustly from many initializations. Yann LeCun and others who lived in that lore were more comfortable; the community as a whole was not. Most of AI history, though, is slower progress than expected, so he regards fears of AGI in ten or twenty years as based on completely unrealistic models of the field's speed.

Fridman presses on existential risk. Malik's answer is that we need to be worried about AI today, not only when AGI arrives. Systems are already deployed in medical diagnosis, hiring, and self-driving cars. When they make mistakes there are consequences, and the people who build them are in some sense responsible. He treats this as a continuous engineering duty, not a clever trick designed for a date in 2079. The Uber incident in Arizona, he notes, already happened. That was not AGI. It was a very dumb intelligence that still killed a pedestrian.

Fridman adds the scale worry he actually lives with: recommender systems. Twitter, Facebook, and YouTube already control which ideas and news people see. His own life, he says, would not be the same without those sources, which means a machine-learning algorithm is already shaping a human being. As the rankers get smarter they gain more influence over billions of people. For now you can turn YouTube off. The more the algorithm moves into ordinary life, the more that option thins out.

YouTube in particular is using computer vision to understand video content and connect clips with the people who would benefit from them. That development, Malik agrees, can go in many directions, some harmful. The threats of AI are here already. Great power, in the ordinary engineering sense, comes with great responsibility, and the job is to stay continuously on the lookout for safety, bias, and risk rather than waiting for a superintelligence that may not arrive on any calendar we can read.

Asked to relive a moment that made him truly happy, Malik does not pick a single instant. He looks over the long haul. A lot of scientific research, he says, is being in the right place at the right time. You can work on a problem too early, when the prerequisites are missing, or too late, in a field where you can only add curlicues upon curlicues. He has been a professor at Berkeley for 34 years, and he got in when computer vision was a little crazy, absolutely useless field that could not really do anything. He has watched it start solving practical problems and offering tools for biology and astronomy. It is mature, he says, but not done. People 500 years from now, Fridman jokes, would laugh at calling it mature. Malik allows that this is very possible.

He has also mentored some of the biggest names in the field. Part of the credit, he insists, belongs to Berkeley: very smart, hardworking, creative students knock on the door. What he thinks he has added is a sense of picking the right problems. In the short run, success is technical competence. Long-range progress is asking the right questions. He cites the British Nobel laureate Peter Medawar, who called research the art of the soluble: find a problem that is not yet solved but is approachable, a beast with a soft underbelly you can spear. Without that intuition you can beat your head forever.

If he can convey that taste, students keep doing great research twenty years after they leave. The other thing he brings is intellectual breadth. He has spent time on psychology, neuroscience, and applied math, enough to help deep specialists see connections they might miss. Smart students arriving at Berkeley can think very hard down one path. He offers shallow breadth against their narrow depth, and he thinks that combination has some value. Fridman notes how rare it is to hear someone jump from psychology back to computer science without forcing it.

If there is a general problem of computer vision, Malik will not isolate it from biology. Perception always exists to guide action. The first multicellular animals of the Cambrian, about 500 million years ago, could move and could see; movement got them to food, and seeing told them where to go. At a fundamental level, perception and action go together. As systems hyper-evolved, vision became a fairly general-purpose model of the external world, imperfect in the ways psychologists love to demonstrate with illusions, and usable even for judging the aesthetic value of a painting. The modules of object detection and 3D reconstruction, he says, are structures scientists impose. Underneath, the processes run simultaneously because their original purpose was to help you act.

He started pushing that connected view around 2010, when the community treated segmentation, detection, and reconstruction as isolated dataset races. Psychologists' low, mid, and high levels did not map onto the tasks people actually cared about. Then the deep-learning era made shared representations with multiple output heads the default, and he no longer needs to preach it. End-to-end learning in the narrow supervised sense, though, is still too thin. He wants the child-development, lifelong-learning view: some capabilities are built first, and others are built on top of them.

Why did the field spend decades on static images? Compute, memory, and storage. In the 1970s through 1990s it was common to detect edges on a 256-by-256 image and throw the rest away, on the hope that humans can interpret a line drawing. ConvNets process pixels now because those restrictions lifted. Video is still understudied. At YouTube scale, even Google still leans on basic 1990s-style techniques. As a researcher you can train large networks on relatively large video sets, and Malik expects more progress. His quantitative claim is that video recognition feels about ten years behind object recognition: challenging video datasets sit around 30 percent on action classification, roughly where object detection was in 2009. Whether catching up takes another decade is a different question. He hopes it takes less.

From image statistics at the lowest level, what the field has learned is that there is a lot of redundancy, so a lot of compression is possible. Biology requires it: on the order of 10^8 photoreceptors and 10^6 fibers in the optic nerve, a factor of about 100 to 1. Early layers of artificial nets do an analog of that. Several companies, including one founded by a former student of Malik's, are using neural techniques for image and video compression, and the results are interesting. That success is still about low-level statistics. Saying of a photograph only "it is a cat" is a different, semantic kind of compression, and today's systems keep those two far apart.

The central question for scene understanding, as Malik has framed it, is the interplay of bottom-up and top-down information. Current systems run completely bottom-up at test time: a feed-forward net from raw pixels to cat or not-cat. They are trained in a very top-down way, with labeled cats, dogs, and zebras. He is not happy with either choice. Biology is not purely feed-forward at runtime; it uses feedback, and it is much shallower. A ResNet-50 has 50 layers; from retina to IT in visual cortex you might have something like seven, plus backward connections that may help with ambiguous stimuli. Unrolling a few rounds of feedback into a deeper feed-forward net is functionally possible. He has written on the theme and thinks it should be pursued further.

At training time, learning to segment still means showing the boundary of a cat, a chair, a horse—too much top-down knowledge. Humans manage with far less supervision. A horse moves in a video, and common fate, the Gestalt name for pixels that travel together, tells you they belong to one object. Machine vision has technically solved the top-down and bottom-up interaction, Malik says, but he does not find the solution satisfactory. He would rather have a bit of both at both stages. To the hard-boiled engineer who does not care about matching human vision, he offers a cost argument: if this agenda works, you need far less training data.

To outsiders, segmentation looks like drawing outlines. Malik's claim is stronger: it lets you say that some set of pixels is an object without naming it or knowing its properties. He calls that entitification. If a child can already separate objects from background, weak supervision works. When a parent says the cat is behaving funny, the word can attach to an entity the child has already carved out. The parent does not have to point and drill labels. There are also blunt practical cases. His group worked on CT scans of traumatic brain injury, where a radiologist needs to delineate possible bleeds, not draw a box. Segmentation is not semantics-free. Non-rigid things such as people, with independently moving parts and clothes of different colors, force you to revise what counts as one entity. Early computer vision treated segmentation as a purely bottom-up perceptual task, and that was a mistake.

He organizes the rest of vision as three R's. Recognition is the one the public already associates with the field: put a label on an image or a region, from Chihuahua to animal. Reconstruction is inverse graphics: given pixels, infer objects, viewpoint, shape, texture, a model of the external world in the head. Reorganization is perceptual organization, finding those entities so an image is not internally a uniform grid of pixels. The three interact as a triangle, not a pipeline. Segmentation is only a piece of reorganization, but it is the piece that gives recognition something to name and reconstruction something to model.

On compute, Malik has been making a related point for twenty years. Neurons are slow compared with transistors, but there are many of them with very high connectivity; silicon switches in nanoseconds with usually less connectivity. By 2020, looking at Hans Moravec's 1990s-style calculations, GPU clusters may be in the same ballpark as the brain in raw power, but not in the same style. GPU computing is far more power-hungry than biological computing. Efficiency, he says, will have to be solved to build large-scale systems that actually run in the world. A machine that eats a city's worth of electricity does not fit in a body.

Turing's paper ends with a line Malik likes: instead of simulating the adult mind, why not produce one that simulates the child? Today's vision benchmarks mostly try to reach the adult. Malik would collect the kind of visual and linguistic environment a child actually grows up in—privacy issues and all—and learn from that. Some people will argue we can short-circuit nature, the way airplanes do not flap. He would still bet on the child-like route. A child gets to play with the dataset, which in machine-learning language is active learning. He wants both real robotics, with mass, inertia, and friction, and simulation. At Facebook AI Research his group worked on Habitat, a photorealistic indoor and urban simulator. Later generations, he expects, will add forces, masses, and haptic interaction.

Acting matters because it is one way to break the correlation-versus-causation barrier. Judea Pearl has described deep learning's successes as curve fitting. Malik does not quite agree, and he calls Pearl a bit of a troublemaker, but he takes causality seriously as a cluster of issues rather than a single silver bullet. Medicine's reliable tool is the randomized controlled trial. A child, he says, is running small versions of those trials all the time: drop something, push something, update a causal model. His colleague Alison Gopnik's book *The Scientist in the Crib* is the analogy he likes. He sees no fundamental block to simulating not just optical realism but physical interaction. Graphics already went from failing at hair and fur to shattering glass. Some of that is compute-heavy. Ways will be found.

The skateboarder is worth one last pass, because it is the fallacy of the successful first step in traffic. High scores on standard sets do not predict a nonstandard agent. In principle more skateboarder data would help; in practice current systems need far more examples than a sixteen-year-old who has barely seen the type. That gap is why Malik keeps returning to blank-slate learning: it is inefficient exactly where safety lives, in the tail. Computer vison, as he has lived it, went from a useless little circle to a field that is mature but not done. The next step, on his bet, is not only more labels and bigger models. It is a learning mechanism that can pick up, from a few trips to a restaurant or a few minutes with a ball, the priors a toddler already has.