Circuit Search, Hindsight and Self-Play: Ilya Sutskever's Case for Where Learning Is Going

Sutskever begins by saying it is not self-evident that deep learning should work at all. He cites a mathematical result: if you could find the shortest program that fits your data well, you would achieve the best possible generalization, since the program has extracted all the regularity in the data. The catch is that searching the space of programs is computationally intractable, because tiny changes can wildly alter behavior.

Small circuits, however, can be found with backpropagation, and he calls this the miraculous fact on which the rest of AI stands. Training is like solving a large set of equations, using gradient descent to push the information in the data into the parameters. Circuit search does not always succeed, he notes, and it is easy to build artificial datasets where it fails, but on practical datasets it works.

A network with 50 layers behaves like a massively parallel computer given 50 time steps, enough for substantial reasoning. As an example he mentions that a modestly sized network with two hidden layers can learn to sort n-bit numbers, even though sorting is usually thought to take logarithmically many parallel steps. Deep networks are worth finding and also optimizable, he says, and that combination is why everything works.

Sutskever describes reinforcement learning as a framework for evaluating agents on how well they achieve goals in complicated stochastic environments. The agent acts, the environment returns observations and reward, and the aim is to maximize expected reward. He points out that the framework has long existed and became interesting once good, if imperfect, algorithms appeared.

In the real world, he says, reward is not handed over by the environment; we infer it from observations. His one-sentence description of model-free RL is to try something new with some added randomness, compare the outcome to your expectation, and if it was better, change the parameters to take those actions more in the future. In supervised learning the target comes from outside, while in RL the random perturbation becomes the target when you like the result.

He distinguishes two families of algorithms. Policy gradients derive neatly from the sum of expected rewards and match the intuitive picture, while Q-learning is less stable but more sample efficient and can learn from data generated by any actor. The real prize, he says, would be an extremely sample-efficient RL algorithm, since today's are data hungry.

Sutskever calls meta-learning a beautiful idea that does not really work yet but kind of works and is promising. The dream is to learn to learn: train a system not on one task but on many, so that it can pick up new ones quickly. In the standard recipe the model receives the test task's examples together with the test case and outputs a prediction.

He sums it up as turning a training task into a training case and then pretending it is ordinary deep learning. Success stories include few-shot recognition of handwritten characters on a dataset produced by Lake and colleagues, and neural architecture search, where a small number of bits, such as an architecture or program, are learned and then applied to new tasks.

Later he addresses the main limitation of high-capacity meta-learning: the training and test task distributions must match. Real life is not like that, since going to work uses only part of what school taught you. Better robustness outside the training distribution, he says, would make meta-learning far more useful.

Sutskever presents hindsight experience replay as a way to deal with the exploration problem in RL, where an agent that never gets reward cannot learn. The idea is to train a policy that can reach a large family of goals rather than one. When the agent aims for state A and ends up in state B, it can still treat the trajectory as a lesson in reaching B.

He notes a subtlety: this is on-policy learning for A but off-policy for B, so the underlying algorithm must support off-policy learning. The agent makes the problem easier by seemingly making it harder, learning from success and failure alike because it always learns something. Sparse rewards then stop being a barrier, since no reward no longer means no learning.

He shows videos of the method controlling a simulated robot and says it is directionally correct because it uses all of the data. The big open question is where high-level goal states come from, and he expects representation learning and unsupervised learning to become important for answering it.

For robotics, Sutskever explains, simulators can never perfectly match reality because contact and friction are so hard to simulate. The remedy he shows, from a 2017 intern project, is to randomize physics heavily, including friction, masses and object dimensions, without telling the policy which values were used. A recurrent policy must therefore infer the properties of its world from experience.

In the demo, a policy trained in a fixed simulator struggles to push a hockey puck to a target on the real robot. The policy trained with randomization adapts and corrects its pushes gently, which he calls promising whenever you can randomize the simulator sufficiently, though not perfect.

He then describes work, done by a high-school student, on hierarchical reinforcement learning, which he says has not yet delivered real value. Here meta-learning is used to learn low-level primitives that make as much progress as possible within a short run of RL across a distribution of mazes. It solves problems quickly only when the hierarchy is right, which he treats as a proof point rather than a solved problem.

Sutskever reviews TD-Gammon from 1992, in which two neural networks played backgammon against each other and found strategies that top human players had missed. Later examples include AlphaGo Zero, which beat the world champion without external data, and OpenAI's Dota 2 bots, which beat the best players in the one-versus-one version of the game. Even in 1992, he notes, the larger network worked best.

The appeal of self-play is that agents create the environment for each other, creating an arms race with no fixed ceiling, like iguanas and snakes in evolution. An agent always has an opponent roughly as good as itself, so it sometimes wins and sometimes loses, which is the right condition for learning. In an OpenAI wrestling-style ring experiment, humanoids knew nothing about balance yet learned it, and even stayed upright against random forces once their opponent was removed.

Because compute can be pumped into self-play, he says, competence can rise very rapidly, as the Dota bots went from random to champion in roughly five months. He expects cheap neural-network cycles to make this increasingly important. The open challenge is steering what self-play agents learn so that they can solve externally specified tasks.

He offers a speculative theory that human brains grew because our ancestors' most important survival problem became standing in the tribe, not tigers or lions. Following that path, a society of agents might develop language, theory of mind, negotiation, trade and politics, and he warns that this raises alignment concerns. If competence keeps rising as fast as with the Dota bots, it could do so quickly there as well.

Sutskever says conveying goals to agents is a technical problem that matters because the agents we train will quite likely become dramatically smarter than us. He presents work by the OpenAI safety team with Paul Christiano and others, in which a human views pairs of behaviors and clicks the better one. A reward function is fit to those clicks and optimized with reinforcement learning.

With roughly 500 clicks, a simulated leg learned to do backflips, and several thousand bits of feedback were enough to train agents on a number of Atari games. The method is data efficient in terms of human feedback rather than environment interactions. It also handled an unusual goal in a car-racing game, getting the white car to drive right behind the orange one.

He ends by saying alignment is a technical problem that must be solved, whereas choosing which goals we want these systems to have will be a very challenging political problem.

Asked whether the brain's apparent lack of backpropagation matters, Sutskever says he does not know but has opinions. Because backprop solves such a fundamental problem as circuit search, he thinks it is unlikely to go away, and he suspects we will build human-level systems before understanding how the brain learns. On the Dota bots, he says the strategies they found were real: a tester who imitated one of the bot's moves was able to beat a better pro.

On expected reward, he says maximizing it is partly a matter of convenience because it is easy to design algorithms for, and that variance matters in applications where mistakes are unacceptable. On cooperation, he expects that in a sufficiently open-ended game it becomes the winning strategy, so we may get cooperation whether we like it or not. On complexity, he notes that humans do not solve NP-complete problems in general and that the networks we train are in practice optimized with gradient descent despite worst-case intractability.

For language, he says generative models are far from great, but simply scaling them up, even to something like a thousand layers, will go surprisingly far. What looks patently wrong to him is freezing a model after training, since training is the truly general part of the process, so he advocates training at test time. He thinks evolutionary strategies are not great for RL but could suit evolving a compact piece of code.

On whether agents need an accurate physical simulation, he says no: they can learn lessons like persistence and negotiation in simulation and then keep training in the real world, as a student keeps learning after school. He also stresses that agents need to infer other agents' goals and beliefs by observation, which links imitation to communication. On the political side, he says AI's societal impact will be overwhelming and that more people will care as automation grows.