We Are Still Massively Underestimating Deep Learning

The 2012 ImageNet contest is the story everyone already knows. A neural network called AlexNet beat the second-place entry by more than ten percentage points, and the result is now treated as the public start of the deep learning revolution. Lex Fridman opens this 2020 conversation by placing Ilya Sutskever, then OpenAI’s co-founder and chief scientist, as one of the three authors of that paper with Alex Krizhevsky and Geoffrey Hinton. Sutskever’s own clock starts earlier, and more quietly, than the contest.

Around 2010 or 2011, he says, two facts clicked. James Martens invented the Hessian-free optimizer in 2010 and used it to train a ten-layer network end to end, from scratch, with no pretraining. "When that happened, I thought, this is it." His reasoning was almost naive: a ten-layer net can represent a complicated function, roughly the way a brain can recognize an object in a hundred milliseconds of slow neuron firings. Train a big enough network on enough supervised data, and it must succeed, because you can find the best network.

The theory he carried then was incomplete. He assumed that more data than parameters would prevent overfitting; today he knows heavily overparameterized nets can still generalize. The doubt that actually mattered was compute. Backpropagation, he thought, would work. What was unclear was whether anyone could train a convincing net. Then Krizhevsky wrote extremely fast GPU kernels for convolutional networks, and the plan became simple: do ImageNet. Ten years later Sutskever had won that bet, and the question that still bothers him is darker. The field, he will say later, is still massively underestimating what deep learning can do.

Fridman asks how much of that early intuition came from the brain. Sutskever is careful with the analogy and still grants the lineage. The whole idea of a neural network, he says, is directly inspired by biology, from Frank Rosenblatt’s perceptrons in the 1960s to Warren McCulloch and Walter Pitts trying to design a simple computational object that was a bit like a neuron. Kunihiko Fukushima’s convolutional nets, and later Yann LeCun’s restricted receptive fields for images, sit on the same line. There were only a few cases where the brain analogy actually worked. That was enough.

His own stance was pragmatic. An artificial neuron, if built right enough, was probably not that different from the real thing. "Let's just assume it is and rock and roll with it." The differences still matter. Brains use spikes; artificial nets use continuous activations. People who chase spiking networks, he notes, often end up simulating non-spiking training inside the spikes. His prior that spikes are the important difference is not high. The question underneath is sharper: why should any of this work at all?

What does interest him in the brain is time. Spike-timing-dependent plasticity strengthens a synapse if its input arrives just before the neuron fires, and weakens it if the input arrives just after. "I'm 90 percent sure it's right," he adds. "If I said something wrong here, don't get too angry." Feedforward nets and ordinary recurrent nets flatten time into discrete steps, while the brain keeps a continuous clock in which timing itself carries information. He still thinks recurrent nets can do anything we want, even if Transformers have superseded them for now. Maybe they make a comeback. He will not bet the house on it.

The conceptual leap that made training thinkable, Sutskever says, is the cost function: a scalar that measures how well the system is doing, which gradient descent can minimize. In retrospect the idea feels trivial. Fridman pushes on that comfort. Supervised learning with one loss is easy to see after the fact. Games, evolution, and economies do not offer a clean scalar, and new architectures may need many losses, moving losses, or no loss at all.

Sutskever grants the point. In a game you reason about an equilibrium, not a number you crank downhill. He is not sure biological evolution has a cost function in the mathematical sense. You can talk about regions a process moves toward. "I don't think the cost function analogy is the most useful." The same, he says, goes for the economy. If evolution and markets get by without a tidy objective, maybe deep learning’s loyalty to one is a bottleneck.

His answer stays practical. Self-play already loosens the frame, and so do exploration schemes that reward surprising a predictor. He remains "a big fan of cost functions." They work, and they work extremely well. One day a deeper view might put them in a less central place. He does not know. He would not bet against them disappearing from the toolkit, and he would not bet on that either. For now, whenever you can do the work with a cost function, you should.

Neural nets had been around for decades. What changed in the ten years after ImageNet? Sutskever’s list has three parts, and none of them is a new algorithm. The ideas were already there. What was missing was a lot of supervised data, a lot of compute, and conviction that mixing the two with the old ideas would work. GPUs showed up. Labeled datasets showed up. Someone had to believe the mixture was enough.

He defends the people who did not believe. Later generations like to sound superior about the skeptics. The facts on the ground were uglier. Neural networks really did not work on anything, and they were not the best method on pretty much anything. "It was pretty rational to say, yeah, this stuff doesn't have any traction." Disbelief, in that world, was not a failure of imagination. It was a reasonable reading of the scoreboard.

What broke the stalemate was a hard benchmark that produced undeniable evidence. ImageNet played that role. Computer-vision authorities such as Jitendra Malik and Alexei Efros were skeptical; Hinton was the opposite of skeptical; AlexNet ended the argument with a result nobody could talk around. Sutskever generalizes the lesson. A field without hard tests falls into position wars, and the loudest voice wins. A field with tests that cannot be faked turns the fight into engineering. "This is why we are able to avoid endless debate."

Fridman asks whether vision, language, and action are different sciences. Sutskever’s answer is quick: machine learning has a huge amount of unity. There are only one or two or three very simple principles, and they apply in almost the same way to different modalities and problems. A paper that improves optimization for vision will often help reinforcement learning too. Underneath, it is the same machine.

He expects vision and language to keep merging. A few years earlier, natural language processing was a pile of tiny architectures, one per task. By 2020 there was one Transformer for all of them. Go back further and every little AI problem had its own specialists and its own feature engineering. "Now it's all been subsumed by deep learning." Convolutional nets still ruled vision while Transformers ruled language, but he thinks a single architecture is possible. He will not be too sure. Convolutions are computationally efficient, and that still counts.

Reinforcement learning is the apparent exception. It needs action and exploration, and the variance is higher. Sutskever still sees unity, and he expects a later merger in which the system decides what to ask for so that supervision goes further: one black box you shovel things into. What is truly different about learning a policy is non-stationarity. Once you act, the world you see starts to change. Static supervised problems give you a distribution; an agent that acts rewrites the next batch of data. The gradients, the networks, and Adam still look the same. The frame of reference decides how large the difference feels.

Asked whether language is harder than vision, he rejects the question. Hardness depends on the tools you have today, and a solved problem stops being hard. Both human-level language and human-level vision were, in 2020, hard in the modest sense that nobody would finish them in three months. Noam Chomsky’s claim that language sits under everything runs into a messier boundary: if a vision system reads letters on a page, where does vision stop and language begin? Sutskever’s guess is that a system deep enough to crack one will probably get the other. He is not 100 percent sure. A lot depends on how you define perfect vision, and on whether reading counts.

Fridman asks for the most beautiful or surprising idea Sutskever has met in the field. The answer is not an algorithm. "The most beautiful thing about deep learning is that it actually works." You have a little network, backpropagation, and a theory that this is kind of like the brain, so you make it large, train it on a lot of data, and it starts to do the same kind of job. "That's crazy." Make the nets larger and they keep getting better. "I find it unbelievable that this stuff with neural networks works."

The intuition he trusts is part empirical and part theoretical, closer to a biological theory than a proof. Physics lets you write a precise theory and then run the experiment. Here the experiment sometimes arrives first, and the prediction is cruder: make the net bigger and it will keep working. He reaches for a joke that is doing real work. Deep learning, he says, is the geometric mean of biology and physics. Physics has terrifyingly precise theories. Biology is so complicated that good predictive theory is almost impossible. Deep learning sits in between. It has some physics-like regularities and a lot of biological mess.

That is also why he thinks beautiful, mysterious properties are still undiscovered. "We are still massively underestimating deep learning." If he knew the next surprise, he would have done it already. Looking at the previous decade, a few ideas felt genuinely new, but the dominant pattern was simpler. Every year the community said deep learning goes this far. The next year it went further. Individual researchers may find progress harder because the field is crowded and the stack now runs from ideas through datasets, distributed systems, clusters, and GPU kernels. Breakthroughs that actually do things will need huge compute. Plenty of important work, he still thinks, can come from small groups. He would not bet that the next thirty years of breakthroughs fit on one GPU.

In early 2020 Sutskever and colleagues published *Deep Double Descent*, a paper that made a sharp empirical pattern hard to ignore. Hold the dataset fixed and grow the model slowly, and do not use early stopping. Small models get much better as they grow. Keep growing and performance gets worse, exactly at the point where training loss hits zero. Grow further and performance rises again. The curve is a U with a tail, which is why it is called double descent. People expected deep learning phenomena to be monotonic. This one is not.

The intuition is about sensitivity. Overfitting, he says, is when the model is "very sensitive to the small, random, unimportant stuff" in the training set. When the model is small and the dataset is large, there is little uncertainty about the fit, so those accidents barely move the parameters. When the model’s degrees of freedom match the data’s, there is a one-to-one map: a tiny change in the sample becomes a noticeable change in the model. Pretend the net is linear and there is a whole subspace of zero-error solutions; SGD finds roughly the smallest-norm point in that subspace, which can be proven insensitive to small randomness. At the interpolation threshold, that cushion disappears.

A detail that is easy to miss almost hides the whole effect. "If you introduce early stopping, you can make the double descent bump almost completely disappear." The worst point tends to arrive just as validation performance starts to degrade, before training has walked into the second, better regime. Stop when validation slips and you never see the recovery. Many teams already do that, which is why the phenomenon stayed buried in ordinary engineering. Regularization of other kinds can flatten the bump too. The lesson is not that bigger is always better. It is that the scale curve is not a straight line, and the ugly middle is easy to mistake for a dead end.

Hinton had quipped, not entirely as a joke, that if you cannot find backpropagation in the brain you should throw it away and start over. Sutskever restates the line more carefully. What Hinton said, precisely, is that if backprop is not in the brain, we might learn something from how the brain actually learns, and that backpropagation is still very useful and we should keep using it. The wit is real. So is the engineering advice.

Sutskever himself is a fan. The algorithm solves "an extremely fundamental problem, which is finding a neural circuit subject to some constraints." He does not see that problem going away. Unless someone offers a method that is essentially different and just as effective, this pillar does not move. Something dramatically different could happen. He would not bet on it. The search for a circuit under constraints is not a fashion. It is the job.

Can neural networks reason? Sutskever offers an existence proof. The network inside AlphaZero, without search, plays Go better than 99.9 percent of humans. Go, everyone agrees, requires reasoning. Does that network not already show that nets can reason? Fridman pushes back. Reasoning, as people mean it, looks like stepwise search, and Go might be a special board. Sutskever grants the gray area and still holds the proof: a constrained environment, a process with the same elements, and a second existence proof sitting in the room. Humans reason too.

Will the architectures that produce future reasoning breakthroughs look like today’s nets? He thinks they might: a little more recurrent, a little deeper, but similar. These models are insanely powerful, and humans can reason, so why can’t the nets? The sharper sentence is the one that survived the debate. Neural networks are capable of reasoning. "If you train a neural network on a task which doesn't require reasoning, it's not going to reason." It will solve the problem you pose in the easiest way possible. Give it a job that surface statistics can fake, and it learns the surface. Give it a job that only multi-step thought can finish, and it may be forced to think.

Sutskever had once said that neural nets search for small circuits, while general intelligence searches for small programs. In this conversation he amends the line. The shortest program that outputs your data would give the best predictions, and that statement can be proved. It can also be proved that finding that program is not a computable operation. No finite amount of compute can do it. Neural nets are the next best thing that works: not the shortest program, but a circuit that fits the data.

Overparameterization forced the rewrite. He used to think a good model was a small, tidy one. Now he would say "a large circuit whose weights contain a small amount of information." Imagine training as a slow transfer of entropy from the dataset into the parameters. The information in the weights ends up not very large, which is why the model generalizes. That is also why you cannot skip ahead to the shortest program. Kolmogorov complexity is a wall. The only path that runs is approximate circuit search.

He adds a harder premise than elegance. The field has a pillar called trainability, and it contorts the shape of the network around that pillar. "This is an invariant we cannot violate." A prettier architecture that cannot be trained will not be adopted. There are still no good precedents for finding programs really well; the way you would do it is by training a deep net, and that illustration did not yet exist. His broader rule is simpler. It is unwise to bet against deep learning. If humans can do a cognitive function, some deep net will probably show up that can do it too. Long-term memory is already partly in the parameters, which aggregate the model’s entire experience. Interpretability, on his human-centric reading, looks like asking the model what it thinks, the way you ask a person, and hoping self-awareness arrives so the system knows what it knows.

Language modeling did not begin in 2018. Sutskever traces small recurrent nets on language to the 1980s, including the Elman network. What changed the trajectory was the same pair of variables that changed all of deep learning: data and compute. Language models also have a clean story for why they must be large. They predict the next word. A tiny model notices spaces between characters and capital letters after commas. A larger one notices frequent words, spelling, then syntax. Only a still larger one starts to notice semantics and facts.

He offers an experiment from his own past as evidence. A character-level model trained on Amazon reviews did not capture sentiment at 500 hidden units. At 4,000, one neuron started to represent whether the review was positive or negative. Sentiment, he says, is a pretty semantic attribute, not a syntactic one. The theory was that the model exhausts syntax and then has to look somewhere else. He will not claim today’s models have complete semantic understanding. He will claim the larger ones show signs that the smaller ones do not. That is the empirical answer to Chomsky’s hope that you must impose a theory of language on the learner.

GPT-2, he says, is a Transformer with 1.5 billion parameters, trained on about 40 billion tokens from web pages linked by Reddit posts with more than three upvotes. Why did the Transformer succeed? Not because of one hero idea. "The transformer is a combination of multiple ideas simultaneously." Attention is a key, not the key; attention had existed for years. Two other facts mattered as much. The architecture runs extremely fast on a GPU. It is not recurrent, so it is shallower and much easier to optimize. Recurrence offers a romantic story about arbitrary long-range dependence. Depth is a burden for the optimizer. The Transformer traded some of that theoretical power for trainability, and trainability won again.

He admits he was surprised, then retracts the hedge: it was pretty amazing. He had predicted that a big language model should work. Seeing it was something else. "We adapt really quickly." Cognitive scientists were already writing that GPT-2 does not truly understand language, while the fact of the modeling itself became ordinary in months. The bar that would actually impress outsiders, he thinks, is dramatic economic impact. Translation is already huge. Self-driving will be. He would not bet against deep learning there either. The next research he is curious about is active learning: a model that uses its own intelligence to accept and reject data the way people do. That capability, like any other, needs a real problem. A clever MNIST speedup will not convince anyone anymore.

OpenAI’s staged release of GPT-2 was a controversial 2020 decision. Sutskever’s explanation is plain. The field had been in childhood and was entering maturity. The impact of the systems was already large and still growing. It seemed wise to think about that impact before release, "maybe a little bit too soon rather than a little bit too late." GPT-2, he says, could plausibly have lowered the cost of disinformation. That was possible, not certain. A staged release was the logical response: put out a small model, watch how people use it, then decide.

What they saw were many cool applications and no known negative ones. The larger model followed, and other groups reproduced similar systems. He leaves the hedge in place: no negative uses "we know of." Plenty of labs already release nothing. The harder question is whether a company with a powerful model has a duty to talk to people outside the building, including competitors, about the risks. He thinks that conversation is possible. The work is to build trust gradually. Everyone is building technology that will get more powerful. "Ultimately we're only together."

What would it take to build human-level intelligence? Sutskever will not pretend to know. His formula is restrained: deep learning, plus maybe another small idea. Self-play is one idea he would include. It has "this amazing property that it can surprise us in truly novel ways." OpenAI’s hide-and-seek agents, the Dota bot, and AlphaZero all produced behaviors nobody had programmed: creative, useful solutions, not random novelty. Current systems are strong because they fit data. Self-play is strong because it explores. That is the kind of strength today’s models do not show as a matter of routine.

Does that path have to leave simulation? He refuses the either-or. Simulation is a tool with strengths and weaknesses, and you should use it. Transfer from simulation to the real world has already been shown, especially in vision. OpenAI’s robotic hand, the Rubik’s-cube system, was trained entirely in simulation. "100 percent of the training was done in simulation." The policy was trained to be so adaptive that it transferred zero-shot, including to perturbations that never appeared in training, such as a stuffed giraffe. That is a small, clean example, not a full theory. He expects deep learning’s transfer to keep improving. The better the transfer, the more valuable simulation becomes, because you can carry the moral of the story into the real world, the way people do after a video game.

Does an agent need a body, a fear of death, a self to protect? Having a body will be useful, Sutskever says. He does not think it is necessary. Some things cannot be learned without one, and you can compensate if you lack it. Helen Keller is his evidence: deaf and blind from early life, she still reached a very high cognitive level with the senses that remained. On consciousness he gives an existence argument rather than a definition. Humans are conscious. If artificial nets are similar enough to the brain, then at least some artificial nets should be conscious too. He is not claiming today’s models have that property. He is saying that if the brain has no non-material magic, the state should be reachable in silicon. If the brain is much stranger than we think, that fact will eventually show up as a wall. He thinks that is unlikely.

The intelligence test he actually wants is not the Turing test. He would be impressed by a pedestrian system, machine translation or vision, that never makes a mistake a human would not make under any circumstances. Accuracy can already beat people while the errors still look alien. Skeptics point at those errors and say that if the model understood the concept, it would not do that. Closing that gap, he says, would be real progress. The public debate has a worse habit. Twenty-first-century analysis of AI often searches for one catastrophic failure humans would never make, writes articles about it, and lets that anecdote soothe everyone. Outsiders already cannot tell one demo from the next. They will start to be impressed, he thinks, "once AI starts to really move the needle on the GDP."

Fridman asks what Sutskever would do on the first evening with an AGI, and then what he would do with the power. The first evening is simple: ask every kind of question, try to make it fail, and be amazed when it does not. The civic picture he likes is a board and a CEO. Cities and countries would vote for the goals of the AGI that represents them, a democratic process taken to the next level. The board can always fire the CEO. That means you keep a reset button. Will the button always work? He thinks it is possible to build systems that want to be controlled by humans, the way parents generally want their children to succeed and are glad to do the work. Help humans flourish, in that design, would be a deep drive the system is delighted to fulfill. One technical sketch is a separately trained value function that internalizes human judgments, then sits as the base objective of a more capable agent. "The reward function of humans is not external. It is internal." Relinquishing that kind of power, he says, he would find trivial. The scenario Fridman describes sounds terrifying. He would not want to be in that position.

Asked for the meaning of life, he says the question is wrong. It assumes an external answer: your meaning is X. What is going on is that we exist, which is amazing, and we should try to make the most of a short time, enjoy more and suffer less. Human wants are the objective functions, and they can change. Evolution may want survival and procreation; that still does not answer the personal question. He has regrets about decisions that hindsight would undo, and he takes solace in having done the best he could at the time. Pride in papers is real and not, he thinks, the source of happiness. Happiness depends to a very large degree on the way we look at things. A simple meal can satisfy or disappoint. Then he adds the same hedge he uses on technical claims. He is not sure. He does not want to be too confident.