The Fallacy of the Successful First Step: Jitendra Malik on Why Seeing Still Isn't Solved

In 1966, Seymour Papert and Marvin Minsky wrote a proposal called the Summer Vision Project, handing a group of students the job of solving computer vision over one summer. The task list still looks familiar: segmentation, recognition, three-dimensional reconstruction, scene understanding. Jitendra Malik, a Berkeley professor for 34 years and one of the field's central figures before and after deep learning, does not mock the authors. Early researchers, he says, simply did not know what they were facing.

Seeing fools us because most of it happens below awareness. You can feel yourself proving a theorem or playing chess, step by step, in consciousness. You cannot feel yourself recognizing the face in front of you. That effortlessness makes the problem look easy to implement on a computer. Neuroscience and psychology tell a different story: a very large part of the cerebral cortex, in humans and other primates, is devoted to visual processing. A system that spends that much neural real estate is not simple, and the higher-level parts are the harder parts.

Malik is less forgiving of the same mistake today. He calls it the fallacy of the successful first step. Many vision problems can be 50% solved in a minute, 90% in a day, 99% in five years, and 99.99% perhaps not in a lifetime. That curve explains the field's cycles of excitement and disappointment. Every decade a new method lifts a metric from 30% to 60%, headlines follow, and then everyone discovers that the remaining climb is longer than the road already traveled. Language researchers, Fridman notes, tend to be more cautious; vision people still underestimate how much understanding a static scene can demand.

Malik treats sensation, perception, and cognition as a continuous problem, not a wall that lets researchers study only the middle and dismiss the rest as "cognitive." Today's deep-learning techniques work best at the lower and middle levels. The high, cognitive levels show up in real applications whether the field is ready or not. How much of that is necessary depends on the stakes. Image search can be wrong and nobody dies. Driving at 60 miles per hour cannot.

Fridman describes Tesla Autopilot as a vision-heavy system: eight cameras, one multitask network Karpathy's team called HydraNet, multiple heads sharing a core representation. Malik's answer splits in two. Constrained settings are solvable. Freeway driving was demonstrated in the 1980s by Ernst Dickmanns in Munich, in the 1990s by Carnegie Mellon's Navlab, and in the 2000s by groups at Berkeley and Stanford. The hard problem is an autopilot that works under all driving conditions, where perception, control, and edge cases arrive together. He calls himself a pessimist on fully autonomous driving in the near future, because there will always be that 0.01% of cases that demand sophisticated cognitive reasoning.

He includes predicting other agents in vision. Perception blends into cognition; it has to say what will happen, not what is now, because now is boring and already over. We act in the future. He owns a Tesla and has watched it respond wrongly to a skateboarder: the model had seen plenty of pedestrians, but a skateboarder's motion is a different pattern. In principle enough data could cover that case, and any specific failure may be gone six months later. The deeper issue is sample efficiency. Current computer-vision systems need far more data than humans do for the same capability, and the leftover tail is where cognition lives.

What has been stunningly effective in deep computer vision, Malik says, is tabula rasa supervised learning: a blank system fed huge numbers of labeled examples until it maps input to output. Human driving is not that. At 16, a teenager walks into driver's ed already a visual genius. Most of that repertoire is in place by age two: the world is three-dimensional, objects look different from different viewpoints, occlusion exists, people and animals move in characteristic ways, and there is a dose of intuitive physics. Driver's ed teaches control—smooth steering and braking—and a feel for typical traffic. The course can be short because the student is not learning the visual world from scratch.

The situations that matter later were never in class. Malik may never have faced a skateboarder in driver's ed, or a truck whose rear door opens and dumps junk on the road. He can still handle them, because of general visual knowledge, not because of a specific driving-school rule. That is the gap he wants learning methods to close: long-term accumulation of knowledge, not a fresh mapping for each task.

He sees no in-principle reason neural networks cannot do that accumulation, but the techniques would have to evolve well beyond today's supervised diet of x, y pairs. Human learning is richer. A child picks up an object, turns it, and sees it from viewpoints the child chose; the data is arranged by the learner. Psychologists of child development have mapped many such components. Supervised learning is a small part of the story. The research program is to build models of the rest and train systems on that protocol.

Hans Moravec's 1990s estimates framed a trade: neurons are slow but numerous and densely connected; transistors are nanosecond-fast but sparsely wired. By 2020, Malik says, the latest GPU clusters may be in the same order of magnitude as the brain in raw computing power, yet the style is different. GPUs are far more power-hungry than biological computation. Energy efficiency, he argues, has to be solved before anyone builds truly large-scale real-world systems.

Asked for the general problem of computer vision, he refuses to treat segmentation, reconstruction, and detection as the thing itself. Those are structures scientists impose. In biology, perception confers no benefit unless it is coupled to action. He takes the clock back 500 million years to the Cambrian, when the first multicellular animals could move and could sense light in some way. Movement lets you find food in new places; you still need to know where to go. The earliest loops were simple: find food, or avoid becoming food.

Evolution then put perception to more purposes until humans could build an internal model of the external world. The model is not perfect—psychologists enjoy cataloguing the illusions—but it has come a long way from that Cambrian loop. Object detection and 3D reconstruction run at once because they originally served action. Fridman jokes that in modern life the action might be a Netflix subscription. Malik grants the point in the *homo economicus* role, when we deal with firms. The basics remain action.

Much of computer vision's progress, including Malik's, has been on still images. Some of that was a hardware sentence. Through the 1970s, 1980s, and 1990s, researchers detected edges on 256×256 images and threw the rest away, because storage and compute were scarce and humans can read a line drawing. Nobody does that now; convnets take the pixels. Video is still expensive. University labs can train large networks on sizable video sets. At Google or Facebook scale it remains extremely hard. Malik had just spoken with the engineering VP for YouTube search and discovery: operating over all of YouTube still leans on essentially 1990s-grade techniques.

His working slogan is that video recognition feels ten years behind object recognition. Action-classification accuracy on hard video datasets sat around 30%, roughly where object detection was in 2009. Whether catching up takes another decade is a different question; he hopes it takes less. Focusing on single frames, he admits, can cut off juice that video would have supplied.

Long-form understanding, he agrees with Fridman, needs knowledge. Psychologists call it a schema. The classic example is a restaurant: you walk in, someone seats you, a waiter brings a menu, food arrives, the bill arrives. In the 1970s, AI encoded that as frames, scripts, and schemas by hand, then used the script to infer what a story left unsaid. Three-dimensional convolutions over a couple of seconds of video are built for short-term understanding, not that. Long-term understanding needs goals, intentionality, and functionality. Malik would rather learn the scripts than hand-code them, because learned knowledge is more robust, and because no mother typically lectures a child through every restaurant stage. Ten family dinners are enough.

Near the end of *Computing Machinery and Intelligence*, Turing asks why we should simulate an adult mind rather than a child's. Malik thinks the line is badly underused. Today's vision benchmarks aim at adult performance: a thousand object classes, leaderboard scores. Missing are child-like tests, and the data a child actually meets—linguistic environment, visual environment—which raises privacy issues that some groups are already wrestling with. He finds that direction promising. Then he supplies the counterexample he has not settled: airplanes do not flap their wings. Nature is not always the winning blueprint. For now he is betting on learning like a child.

The child's advantage is interactivity. The learner plays with the dataset, choosing what to look at and what to touch. Malik wants both real robots and simulators. A physical body has dynamics, mass, inertia, and friction; the robot learns that body by acting. His Facebook research group built Habitat, a photorealistic simulator of houses and urban interiors in which movement yields an accurate view. He expects later generations to get forces, masses, and motion directions right as well, not just pixels.

Acting in the world is one way to break the correlation-versus-causation barrier. Judea Pearl, he notes, has called deep learning's successes curve fitting and accused the field of neglecting causality. Malik thinks Pearl overstates the case but agrees that causality matters, and that it is not a single silver bullet. Medicine's most reliable tool is the randomized controlled trial. Children run small versions all the time. His colleague Alison Gopnik's book *The Scientist in the Crib* makes the point in its title: scientists build causal models by doing control experiments, and so do infants. The graphics community's march from visual realism to shattering glass suggests to him there is no fundamental block on simulators that are not just photorealistic but life-realistic.

At the lowest level, image statistics mostly teach redundancy. That is why compression works, and why it is a biological constraint: on the order of 10^8 photoreceptors feed about 10^6 optic-nerve fibers, a roughly 100-to-1 squeeze. Early layers of artificial nets do an analog of the same job. Companies are already using neural methods to compress images and video; a former student of Malik's is in that business, with interesting results. Pixel-level compression is not the sentence "this is a cat." Semantic compression is a different act.

His real complaint is architectural. At test time, current systems are purely bottom-up: raw pixels feed forward to "cat" or "not cat." At training time they are highly top-down: humans label cat, dog, zebra. Biology does neither in isolation. Visual cortex from retina inward is perhaps seven or eight layers, not ResNet-50's fifty, but it has feedback, which helps with ambiguous stimuli. A network with three rounds of feedback unrolls into a feed-forward net three times as deep; the two are functionally equivalent. He wants more top-down processing at test time.

At training time he wants the opposite: less labeled top-down. Humans segment with far weaker supervision. A horse moves in a video and the pixels group; Gestalt psychologists called that the principle of common fate, a bottom-up process. Machine vision has leaned almost entirely on top-down training signals. Even a hard-boiled engineer who does not care about brains, he says, should care, because if the agenda works you would need far less training data.

Segmentation's value is not a prettier box. It lets you say that some set of pixels is an object without naming it or knowing its properties. Once a child can carve entities out of the world, a mother saying "pick up your bottle" or "the cat is behaving funny today" maps a word onto an object without pointing at every label. Weak supervision works after you can create objects. The need is practical too: his group worked on CT scans of traumatic brain injury, where radiologists must delineate possible bleeds. Early computer vision erred in treating segmentation as purely bottom-up. Experience revises the entities. A clothed, articulated human is one object even when parts move independently and colors change.

Around 2010, Malik started pushing a three-part frame to replace low-, mid-, and high-level vision, which did not map onto tasks people actually cared about. Recognition is labels: cat, Chihuahua, bird species, or the abstract tag "animal." Reconstruction is inverse graphics: given an image, recover the objects, the viewpoint, shape, texture, color—an internal model of the external world. Reorganization, related to perceptual organization in psychology, is the making of entities and relations. The world is not stored as a grid of pixels. Segmentation is only a start toward that structure.

The three form a triangle that feeds itself. Reorganization defines the entities, recognition names them, reconstruction imputes information the raw pixels never contained, such as 3D shape. In 2010 the field still treated each problem as an isolated dataset race. He wanted the connections made explicit. After the deep-learning revolution, multi-head networks sharing a common representation turned out to be exactly that architecture. He no longer needs to preach the framework. It is already part of the solution space.

That does not mean he wants set-and-forget end-to-end supervised learning on a giant dataset. End-to-end today usually means supervised end-to-end, which he calls too narrow. He prefers the child-development, lifelong-learning view: some capabilities are built first, and others are built on top of them. A precise labeled task is a limited picture of learning.

Malik points to a paper by Linda Smith and Michael Gasser, "The Six Lessons of Infancy": be multimodal, incremental, embodied, exploratory, social, and use language. That is common sense in child development and not in computer vision or machine learning. He sees his role as bridging the two. Multimodality is the example he likes most. A child holding a ball gets a sparse tactile signal that carries 3D shape, and a visual signal in a completely different receptor space. Both refer to the same object. The cross-calibration is strong, self-supervised, and unlabeled. The child need not know the thing is called a ball.

Audio and vision work the same way. A glass bowl falls and shatters; you hear the smash and see the pieces, time-aligned and causally related. The cocktail-party problem—separating speakers in a mix—is a classic of audition that becomes much easier once you can see who is talking. Time is the free linker. The area, he said in 2020, was still understudied.

Fridman raises Chomsky's claim that language may sit at the core of cognition. Malik does not hesitate: vision is the parent. Phylogeny first: vision is about 500 million years old. In the last seven million years the hominid line split from other primates, bipedal walking freed the hands, tool use and brain size grew, and language—uniquely human, as Chomsky says, since other primates lack it—came after that substrate. Ontogeny agrees. Ancestors three or four million years ago already had spatial intelligence: objects, relations, causal interactions, movement through space. Language borrows space and time; those concepts cannot have come from language itself.

Malik does not want a single test of intelligence any more than he wants IQ as a single number. Turing's 1950 imitation game made sense then; 70 years later it channels research into a chatbot that can fool someone for five minutes. He would rather keep a list of about ten tasks: manipulation, navigation, visual understanding, reading a story and answering questions. His favorite language task is reading a novel and answering arbitrary questions about it. On that list, language is the hardest nut. Vision and robotics, in Donald Rumsfeld's terms, are mostly known unknowns: he knows where to go. Natural language and high-level cognition still hide unknown unknowns, which is why he will not give a timeline.

Asked for Hilbert problems of computer vision, he picks long-form video understanding first: agents, goals, intentionality, predictions, not atomic actions such as sitting or catching a ball. That blends into cognition, which is why it is hard. The second is a richer 3D understanding of the world. Multi-view geometry can reconstruct a scene from several pictures and fails on one. Single-view 3D methods lean on supervised CAD models at training time, which is unnatural supervision—CAD models are not injected into brains. He wants 3D learned by moving through the world, then transferred to a new chair in a single photograph, with the ability to predict new views hidden in the representation rather than stored as graphics assets.

He is comfortable with uninterpretable networks in many settings, because humans are not fully explainable to one another either. Medical diagnosis is a caveat. A model trained on American men in their thirties and forties may not transfer; reference-class errors and missing error bars change which treatment you pick. There he wants more than an answer: bounds, quality, a grain of salt. High-performing black boxes are still acceptable in principle. Fridman notes that charming humans explain themselves with stories, true or not; Malik agrees networks can learn to tell those stories too.

In principle, Malik thinks human-level or superhuman intelligence is reachable. On a 20-year clock, he guesses no. Deep learning surprised him: in 2010 the math he trusted was convex optimization and guarantees, not overparameterized nonconvex nets that the people who played with them treated as black magic. The surprise was how robustly they worked across problems. Most of history, though, is slower than expected. Fears of AGI in ten or twenty years, he says, rest on completely unrealistic models of how fast the field moves.

The worry he will not postpone is deployment today. AI already sits in medical diagnosis, driving, and résumé screening. When those systems err, the consequences are real and people are in some sense responsible. He wants a continuous engineering ethic—great power, great responsibility—not a clever trick designed for a day in 2079. The Arizona pedestrian death is his exhibit. There was no superintelligence in that crash. A very dumb system killed someone.

Fridman argues that recommender systems on Twitter, Facebook, and YouTube already govern what billions of people see, and that the algorithm picking the next YouTube video is a kind of superintelligence at population scale. Malik agrees the threat is present tense. YouTube is using computer vision to understand video content and match it to viewers. That development can run in several directions, some harmful. Turning the product off is still possible; dependence is not a law of nature. The point is to think about the harms while the systems are still this dumb.

Malik will not pick a single happiest moment. Scientific luck, he says, is being in the right place at the right time. Work too early and the prerequisites are missing; you beat your head against a wall. Work too late and the field is mature, and you are decorating curlicues. He has been a Berkeley professor for 34 years, long enough to enter computer vision when it was a crazy, absolutely useless little field that could not really do anything, and to watch it start solving practical problems and supplying tools to biology and astronomy. The field is mature, he says, but not done. Fridman suspects people 500 years from now will laugh at the word mature. Malik says that is very possible.

On mentoring world-class students, he is careful to credit Berkeley: brilliant, hardworking, creative people knock on the door. What he thinks he added is taste in problems. Short-run scientific success is technical competence—fast math, strong code. Long-run progress is asking the right questions. He cites Peter Medawar's *The Art of the Soluble*: find problems that are not yet solved but are approachable. A good problem has a soft underbelly, a place where you can spear the beast. Without that intuition you only ram the wall. He is proudest when students still have that sense 20 years after leaving his group.

The other thing he brings is intellectual breadth. Time spent in psychology, neuroscience, and applied math lets him show connections a student buried in one deep well might miss. Incoming Berkeley students arrive with narrow depth; he supplies shallow breadth. The two together, he says, are what is valuable. The conversation ends where it began: vision as a young field that learned, the hard way, how expensive the last few nines of seeing really are.