If You Want Human-Level AI, Abandon Generative Models

Lex Fridman introduces Yann LeCun as Meta's chief AI scientist, an NYU professor, a Turing Award winner, and a repeat guest who has become one of the loudest critics of AGI doom and one of the loudest advocates of open-source models. LeCun's technical claim is just as sharp as his public stance. Autoregressive large language models — GPT-4, Llama 2, and the Llama 3 then still on the way — are not how we will reach superhuman intelligence.

He lists four properties that intelligent systems, including humans and animals, actually have: they understand the physical world, they have persistent memory they can retrieve, they can reason, and they can plan. Current LLMs, he says, do none of those things, or they do them only in a primitive way. They do not really understand physics, they do not really remember, they cannot really reason, and they certainly cannot plan.

That is not an argument that the models are useless. LeCun is careful to separate two claims people keep fusing together. Autoregressive LLMs are useful; a whole application ecosystem can and will grow around them. They are not a path to human-level intelligence, because essential components are missing. Usefulness and a route to AMI — the "advanced machine intelligence" he prefers to AGI — are different questions.

The scale of LLM training sounds infinite until you change units. Today's strongest models train on on the order of 10^13 tokens. At roughly two bytes per token, that is 2×10^13 bytes — the public internet, more or less. Reading that corpus eight hours a day would take a human about 170,000 years.

Developmental psychologists give a different number. A four-year-old has been awake for about 16,000 hours. The information that has reached that child's visual cortex is about 10^15 bytes, estimated from an optic nerve carrying roughly 20 megabytes per second. Two orders of magnitude separate the child's eyes from the model's text.

The implication, for LeCun, is that most of what we know does not arrive through language. We learn by watching and interacting with the world. Everything a baby learns in the first years of life, and everything animals learn, has nothing to do with words. Sensory input dwarfs text, which is why he thinks a system trained only on language cannot grow a serious model of reality.

Fridman pushes back. Language is already compressed. A sentence packs more than the bytes that store it, and the way humans stitch words together is itself a store of world knowledge. Could that be enough to reconstruct a world model from text alone? LeCun's answer is no. Intelligence, he says, cannot appear without grounding in some reality — physical or simulated — because the environment is far richer than anything language can express.

Language is a rough sketch of percepts and mental models. Rotate a water bottle 90 degrees and the image that appears in your head does not go through words. Standing up, walking downstairs, hailing a taxi, getting to the airport: the planning is a mental model of the current situation, not an inner monologue. Most of what we do all day lives in that layer.

That is Moravec's paradox, named for the robotics pioneer Hans Moravec: computers find chess and integrals easy, and the everyday skills we take for granted — driving, grabbing an object — nearly impossible. An LLM can pass the bar exam and still cannot learn to drive in 20 hours the way a 17-year-old can, or clear a table and load a dishwasher the way a 10-year-old learns in one shot. Bolting a vision encoder onto an LLM, LeCun says, is a hack. Those systems are not trained on video and do not have intuitive physics.

A world model, in LeCun's definition, takes a state at time T and an action and predicts the state at T+1. That state does not need to represent every pixel, only what matters for planning. The temptation is to train the same way we train LLMs: show a video, predict the rest, one frame or a group of frames at a time. FAIR tried that for about ten years. It does not work.

Text is discrete. You cannot name the next word exactly, but you can output a distribution over a finite vocabulary and sample from it. Video is high-dimensional and continuous. We do not know how to represent useful distributions over that space. Pan a camera around a room and a model might guess there is a wall and a lamp. It cannot predict the painting, the couch fabric, or the carpet. Those details are, given the context, unpredictable.

Latent variables meant to carry the unseen detail — ordinary nets, GANs, VAEs, regularized autoencoders — all failed. Reconstructing a clean image from a corrupted one, the trick that made LLMs, also failed in vision. MAE, FAIR's masked autoencoder, produces weak features. Train the same encoder with labels or image captions and the features get much better. The architecture is fine. The reconstruction objective is what will not yield good generic representations.

The alternative is JEPA, a joint-embedding predictive architecture. Run a full image and a corrupted or transformed version through encoders, then train a predictor to forecast the full image's representation from the corrupted one. The system never generates pixels. It predicts an abstract embedding. That is cheaper, and it has a useful side effect: unpredictable details, like leaves shaking in the wind, get treated as noise. The encoder keeps "there are moving leaves" and drops every leaf.

The old failure mode is representation collapse: if you only pull similar pairs together, the net ignores the input and emits a constant. Contrastive learning, which LeCun published on in 1993, also pushes dissimilar pairs apart, but the number of negatives explodes when the answer space is large. In the last few years non-contrastive methods matured — FAIR's DINO and I-JEPA, DeepMind's BYOL, VICReg. I-JEPA does not even need to know it is looking at an image: mask a region and predict the missing representation. V-JEPA does the same for video, masking a temporal tube, typically the same region across 16 frames. It is their first system to learn high-quality video features, good enough that a supervised head can name the action, and early results suggest it can flag physically impossible videos — objects vanishing, jumping, changing shape.

We never describe a table in quantum field theory. Every phenomenon has a level of abstraction, from fields to atoms to chemistry to objects, because the lowest level is computationally impossible. Language already arrives compressed, which is why predicting words works and also why it hits a ceiling. Vision-language models, LeCun says, are using language as a crutch for vision systems that cannot yet learn good features. Improve the language model a bit and you still will not reach the world-understanding of a cat or a dog, which have no language and still plan complex actions. "If you're really interested in human-level AI," he tells people talking about generative AI, "abandon the idea of generative AI."

If you have a world model you can do what LLMs cannot: plan. Feed the predictor an action — the wheel turned ten degrees — and it forecasts the next abstract state. At inference time, not training time, you search for an action sequence that makes the predicted final state satisfy an objective. That is model predictive control, the same idea used to plan rocket trajectories since the early 1960s. LeCun wants it inside learned, abstract state space.

Complex action needs hierarchical planning, and that will not emerge from a JEPA by itself. You have to build for it. His running example is going from his NYU office to Paris. At a high level the sub-goals are get to the airport, then catch the plane. Getting to the airport decomposes into going downstairs and hailing a taxi; going downstairs decomposes all the way to millisecond muscle commands. You cannot plan the whole trip in muscle twitches. You do not know the traffic. You replan as you go.

Nobody in AI, he says, knows how to train a system to learn the multiple representation levels that hierarchical planning requires. An LLM can recite a New York-to-Paris itinerary if similar templates were in the training set, down to the level where words still work. It cannot invent the plan for a situation it has never seen, and it cannot stand up from a chair in language. High-level recipes people use are mostly learned from others. Linking those recipes to low-level action is the job of something like JEPA, which lifts abstraction without reconstructing every detail of the scene.

LeCun is not surprised that autoregressive LLMs work at all. They are a spectacular demonstration that self-supervised learning works, the same idea behind BERT, multilingual translation, Meta's hate-speech classifiers, Wav2Vec, and speech-to-speech systems for languages with no writing. The surprise around GPT-2 was that decoder-only models, scaled up, looked more fluent. Fluency, he says, fools us. We treat linguistic skill as the whole of intelligence. The Turing test is a bad test, and Moravec's paradox still applies.

Hallucination follows from the math. Every time the model emits a token, there is some probability that the token leaves the set of reasonable answers. If those errors are even roughly independent, the chance of staying correct decays exponentially with length. Drift is exponential. Fridman asks whether the training set's average truth creates a gravitational pull back toward fact. LeCun says no: this is a fight against the curse of dimensionality.

Fine-tuning on human questions can cover perhaps 80 percent of what people actually ask, because people repeat themselves. The uncovered set of possible prompts is gigantic. A random character string is enough to jailbreak the model into a different mode. Swap a few words of an English question into another language and a good answer turns into nonsense. Social networks already know this long tail. The system degrades into a giant lookup table, which is not what you want if you wanted reasoning and planning.

You can measure how primitive LLM "reasoning" is. The compute spent on each token is constant: roughly the depth of the network times the number of tokens. Easy questions, hard questions, and undecidable questions all cost the same. Humans do not work that way. Faced with a hard problem we spend more time, because it is hard. That ability to allocate thought at inference time is what current LLMs lack.

LeCun maps this onto psychology's System 1 and System 2. System 1 is skilled automatic action: an experienced driver talking while driving, a strong chess player beating a novice without looking hard. System 2 is deliberate planning with an internal world model. AI today is System 1. His sketch of System 2 is an energy-based model: one scalar output, near zero when Y is compatible with X, positive when it is not. At inference you search in abstract representation space for a low-energy answer, then decode it into text. The parameters stay fixed. Because the space is continuous and differentiable, you can gradient-descend on the thought itself instead of sampling mountains of token sequences and picking a winner.

Training such a model means pushing energy down on compatible (X, Y) pairs and making sure it rises elsewhere. Contrastive methods show bad Ys and push energy up; the reward model in RLHF is essentially that, used for training rather than for planning. LeCun prefers non-contrastive regularizers that limit the volume of low-energy space so energy rises automatically off the data manifold. LLMs already do a collapsed version of this, because probabilities over a vocabulary must sum to one — raising one word lowers the others — but he says it is not obvious why that indirect trick works at all.

LeCun is often tagged as an RL skeptic. His actual position is milder: do not abandon it, minimize it, because it is terribly sample-inefficient. The right order is to learn a good world model mostly from observation, then plan with that model. Planning should solve most tasks. You need RL when two things go wrong: the objective does not match what you actually wanted, or the world model is inaccurate.

Learning to ride a bike, your estimate of whether you will fall may be off. In a fight, the opponent does something you did not predict. That is when you adjust the world model or the objective — RL's proper job. You can even adjust the model in advance by visiting regions where you know it is bad. That has an ordinary name: curiosity, or play. Play is exploration without real risk, repairing the model without getting killed.

Why did RLHF transform language models? The HF, he says — the human feedback. Some of it is just supervised learning: people score several answers, you train an objective to predict the score, you fine-tune. That objective is almost the same function he wants for planning; the difference is that today it is used to change weights rather than to search at inference time. Using it for planning, he thinks, would be much more efficient.

Gemini 1.5's image of a Black George Washington, and its refusal or hedging on Tiananmen Square and Tank Man, set off a round of arguments about censorship and "woke" models. LeCun's reply, posted on every network he uses, was one sentence: open source is the answer. There is no system that is unbiased for everyone. "Bias is in the eye of the beholder." Some facts are indisputable; a great deal can be phrased in more than one way. Push the model one direction and one group calls it biased. Push the other way and another group does. Push too far and you get factual errors, including Black soldiers in Nazi uniforms.

The analogy he wants is the press in a liberal democracy: free and diverse, because a single source of information is the opposite of democracy and of science. In the near future every interaction with the digital world will be mediated by AI assistants — Meta already sells Ray-Ban glasses that answer questions, translate menus, and describe monuments. Search becomes dialogue. Those systems will be the repository of all human knowledge, and that repository cannot be owned by a handful of companies on the West Coast of the United States. The French government, he says, will not accept that its citizens' digital diet is controlled by three American firms. That is a threat to democracy, local culture, values, and language, independent of anyone's goodwill.

Diversity only happens if the expensive base models are open. The founder of Infosys is funding a project to fine-tune Llama 2 across India's 22 official languages. Moustapha Cissé, a former FAIR colleague, is building LLMs for Senegalese languages so people can get medical information where doctors are scarce. A publisher with thousands of books, a company with an internal "Meta Mate," a pizza shop taking WhatsApp orders: all of that sits on an open base. Llama 2 already has millions of downloads, thousands of improvement ideas, and thousands of businesses on top of it. Meta has to build the system for its own huge user base anyway; giving away the foundation model does not impair the ads-and-businesses revenue, and useful applications can be bought.

On bioweapons, studies increasingly find that an LLM does not help you beyond a search engine and a library. Having instructions is not the same as building the thing, and LLMs do not help with the physical gap. Chemical and biological weapons are banned partly because they are hard to use without hitting yourself. LeCun can list the steps of a rocket engine; a team of 50 experienced engineers will still blow up a dozen before one works. Biologists have said the same about lab work. That is Moravec again: turning language into objects requires knowledge that is not in the instructions.

LeCun is more excited about the direction of machine learning than at any time since FAIR launched, or since the early neural-net days 35 years ago, because he sees a possible path: systems that understand the world, remember, plan, and reason. Scale is necessary and not sufficient. A GPU draws half a kilowatt to a kilowatt; the brain draws about 25 watts and is far more capable. Matching the brain might take on the order of a hundred thousand to a million of today's GPUs. Hardware still has a long way to go, especially if AMI is going to be ubiquitous.

AGI will not be discovered and switched on. Hollywood likes an event; reality will be gradual. First, systems that learn from video. Then large associative memories. Then objective-driven reasoners and planners. Then the unsolved problems of hierarchical planning, hierarchical representation, and reconfiguring skills the way a brain does. That is at least a decade, probably more, because there are pits nobody has seen yet. He has been hearing "AGI is around the corner" for 12 to 15 years and calling it nonsense the whole time. Intelligence is not a scalar. Humans are not simply smarter than orangutans; orangutans outperform us in the forest. Intelligence is a collection of skills plus the ability to acquire new ones, a high-dimensional space in which two agents cannot be ranked by a single number.

Doomer scenarios, he argues, rest on false assumptions. Superintelligence will not arrive as a first-try takeover; we will have cat-level systems, then walk upward, installing guardrails as we go, with many groups in parallel so good systems can oppose rogue ones. More intelligent species do not automatically exterminate others, and AI will not be a competing species. The desire to dominate is hardwired in social animals — humans, baboons, chimps, wolves — not in orangutans, who are nearly as smart. We have every incentive to build submissive assistants. LLMs are not controllable; objective-driven systems are, because guardrails (obey humans; do not obey if it hurts other humans) can sit inside the objective. There is no silver-bullet proof. Safety will look like turbojets: decades of iterative design, no separate safety department, because a better engine is a safer engine. Make AI better, and it becomes safer.

A mind-controlling AI is not a nuclear weapon. Nuclear weapons were a one-shot, first-mover shock. AI breakthroughs disseminate. AlphaGo was reproduced within three months without detailed papers. People change companies. Silicon Valley works because information moves. A foreign influence model will not talk to you; it will talk to your assistant, which will be as smart as it is, and will drop it in a spam folder you never see. The Pessimists Archive is full of the same reflex: jazz, comic books, trains that would stop you breathing above 50 kilometers an hour. The real question is which dangers are real, and whether we embrace change or resist it. Concentration of power through proprietary systems, LeCun says, is a much bigger danger than everything else.

Tesla's Optimus revived humanoids; Boston Dynamics has been at it for years; Figure AI and Unitree have joined. LeCun thinks humanoid robots will come, just not soon. The next decade will be interesting because the industry has waited ten or twenty years and still has little beyond pre-programmed behavior. The bottleneck is still Moravec: understanding the world and planning action. Boston Dynamics uses hand-built dynamics and classical planning. That will not produce a home robot. Level-five driving is unsolved, and a system that learns to drive in 20 hours the way a teenager does is farther still. Hardware teams are, in a sense, betting that AI will catch up. FAIR's embodied-AI group uses commercial robots: a dog can go to the fridge, open it, and fetch a can it was trained to recognize. Clearing a dinner table is nowhere close. Significant progress on video self-supervision took ten years and only really arrived in the last two or three.

PhD students do not need a giant cluster. LeCun lists three open problems. How do you train a world model by observation, and does it require LLM-scale data? How do you plan with a learned model when an "action" might be a search, a database query, a simulation, a calculator, or solving a differential equation — Toolformer and related FAIR work exist, but there is no good answer? And how do you learn hierarchical planning, not just hand-specify two levels, as with a robot dog whose high-level path planner and low-level gait planner were designed by people? Deep learning can learn hierarchical representations of percepts. The equivalent for action plans has no demonstration.

What gives him hope is that AI can make humanity smarter. Everyone will have a staff of assistants, possibly smarter than their boss, the way LeCun has spent a career working with people smarter than himself. "All the mistakes that humanity makes is because of lack of intelligence, really, or lack of knowledge." The historical analogue is the printing press: cheaper books, literacy, the Enlightenment, science, democracy, the American and French revolutions. It also produced two centuries of religious war in Europe, because people read the Bible and noticed the priests might be wrong. Nobody today would call the press a net negative. The Ottoman Empire banned Arabic printing for 200 years, in part — UAE AI minister Omar Al Olama told him — to protect the guild of calligraphers. So who, today, are we protecting by trying to ban AI? Economists he trusts do not expect mass unemployment, only a shift in jobs, including jobs nobody can name yet. Twenty years ago nobody could have named mobile-app developer; smartphones did not exist.