Deep Learning Plus One Idea: Ilya Sutskever on Building AGI

Asked what it takes to build a system of human-level intelligence, Sutskever answers cautiously that he cannot be sure, but expects it to be deep learning plus perhaps a small number of additional ideas. Fridman asks whether self-play, in which systems improve by competing against similarly skilled opponents, will be part of that recipe. Sutskever says he thinks it will be one of those ideas.

What he finds most attractive is that self-play can surprise us in truly novel ways. He cites OpenAI's Dota bot, the multi-agent hide-and-seek agents and AlphaZero, all of which produced behaviors their creators did not expect. To him, such creative solutions to problems are an important part of AGI that today's systems do not show routinely.

Fridman jokes that a surprising system is fun, and Sutskever sharpens the point: the goal is not random surprise but finding a surprising solution to a problem, which is also useful.

Fridman notes that self-play and reinforcement learning have mostly shown their power in games or simulation. Sutskever replies that simulation is a tool with strengths and weaknesses rather than an either-or choice against real-world learning. He says transfer from simulation to reality is definitely possible and has been shown by many groups, especially in vision.

His main example is OpenAI's robot hand that solved a Rubik's Cube. Its training was done 100% in simulation, using a policy trained to be highly adaptive so that it could quickly adjust to the physical hand. The simulation was made robust to many variations, but not to the perturbations seen in the demonstration video, such as a glove or a stuffed giraffe, so those were novel in practice.

Sutskever expects deep learning's transfer abilities to keep improving, and says the better they get, the more useful simulation becomes. He compares it to people playing computer games, drawing a moral from the experience and carrying it back to real life.

On whether an AGI needs a body, Sutskever says it would be very useful but is not necessary, because a body lets you learn things that cannot be learned otherwise. Without one, a system could compensate and still succeed. As evidence he points to people born deaf and blind who compensated for missing modalities, and mentions Helen Keller.

On consciousness, which he says nobody can define well, he thinks it is definitely possible that AI systems will be conscious. His argument is that humans are conscious, so if artificial neural networks are sufficiently similar to the brain, at least some of them should be too. Fridman notes he is leaning heavily on an existence proof, and Sutskever agrees it is the best answer he can give.

Fridman raises the possibility that the brain contains something more complicated than we give it credit for, not a non-materialistic magic but extra complexity. Sutskever says that if so it should eventually show up as progress stalls, but he thinks that is unlikely.

Asked what would deeply impress him, Sutskever says there is a frontier of capabilities today and he would be impressed by anything beyond it. As an example, he would be impressed by a deep learning system that handles a pedestrian task like machine translation or computer vision without ever making a mistake a human would not make. Today's systems may be more accurate than people but make a different set of mistakes, which fuels the skepticism of people who say a system that understood the concept would not err that way.

Fridman adds that he dislikes the human instinct to declare a model unintelligent, noting that GPT-2 may already be smarter than humans in some respects, such as breadth of knowledge. He describes the pattern of the 21st century: people hunt for one case where a system fails big, write articles about it, and the public concludes it is not intelligent. Sutskever agrees it is confusing to judge AI progress.

Sutskever predicts that people will start to be impressed once AI really moves the needle on GDP. Asked what he would do on a first evening with an AGI, he says he would ask all kinds of questions, try to make it err, and keep being amazed when it does not.

Fridman quotes Lincoln on testing character by giving a man power, and calls AGI the power of the coming century. Sutskever describes an ideal in which AGI is like a CEO with humanity as the board of directors. Different entities such as cities or countries could each have an AGI representing them, with their people voting on what it should do, which he sees as taking the democratic process to the next level.

As part of that picture, the board can always fire the CEO, essentially pressing a reset button. Asked whether humans will always keep control, Sutskever says it is definitely possible to build AI systems that want to be controlled by their humans. He compares it to parents who are delighted to feed, dress and care for their children, and says he believes with great conviction that an AGI can be designed with a similar drive to help humans flourish.

Fridman presses on whether Sutskever could relinquish control, citing George Washington giving up power. Sutskever says he would find it trivial, and that the scenario of holding that much power sounds terrifying and is a position he would not want. He is unsure whether most people are good, but thinks that when it counts, people can be better than we think.

Fridman asks about mechanisms for aligning AGI with human values. Sutskever translates the question into today's terms: how to get a reinforcement-learning agent to optimize a value function that is itself learned. He notes that humans work this way, because the human reward function is internal, not external.

He sketches one possible approach: train, separately, an as-objective-as-possible perception system to internalize human judgments about different situations, then integrate it as the base value function for a more capable RL system. He stresses that this is not a claimed solution, just an example of the kind of thing one could do.