Deep Reinforcement Learning, Explained by the Researcher Behind AlphaGo

Silver defines reinforcement learning as the study of intelligence in the form of an agent that interacts with an environment, which can be thought of as the world the agent is situated in. The agent takes actions, those actions affect the environment, and the environment returns observations to the agent along with one special item, the reward signal.

The reinforcement learning problem is simply to take actions over time so as to maximize that reward. Silver calls this a truly ambitious problem definition, because it tries to capture everything involved in an agent interacting with its surroundings and to formalize what it means to crack it.

To solve such a hard problem, Silver says one approach is to decompose it into pieces that work together inside the agent. Three of the most common pieces are a value function that predicts how much reward the agent will get in the future, a policy that decides how to pick actions, and a model that predicts what will happen in the environment.

He understands the different choices in RL as choices about whether to use each of these building blocks, and combinations of them give rise to the familiar branches of the field. Asked by Lex Fridman whether these differences are fundamental, he says the first step of the decomposition is common to all of them: the system has to learn for itself, because that is the only way to perform well in any sufficiently large and complex environment.

Learning, in this sense, means updating the parameters of a system that picks actions, and those parameters can parameterize a value function, a model or a policy. What gives the system its meaning, Silver says, is the objective: whether it learns something to predict well, like a value function or a model, or something to perform well, like a policy. He adds that those choices are currently made by designers, and that ideally algorithms would learn to make them for themselves.

Deep reinforcement learning, Silver explains, is one family of solution methods that uses the powerful representations of neural networks to stand in for any of the agent's components. Because neural networks can represent and learn any function, whatever is needed for a value function, a model or a policy can in principle be handled by deep learning.

He stresses that this toolkit has no ceiling: with more memory, computation, data and experience, such systems can get better and better at whatever job they are given. Whether the function estimates how well the agent will do, chooses what to do, or predicts what happens next, it can keep improving.

Silver says it is not surprising to him that reinforcement learning can work at all, since humans are existence proof that experience can be acquired and used well in complex environments. What surprises him, like much of the community, is how well current systems do, which he finds exceptionally beautiful.

The big surprise, Silver says, is that deep learning keeps performing well even though the networks have very bumpy, nonlinear surfaces that seem as though learning should get stuck. In high dimensions, apparent local optima turn out not to be traps, because there is always some pathway that leads even lower. Learning can therefore continue to improve without bound.

Silver and Fridman note that people are tuned to three-dimensional intuition, so visualizing the surface of a billion-parameter network is very hard. Silver suggests this explains the AI winter in which people gave up on neural networks: back then networks had around 50 nodes, and it was hard to anticipate that a billion-dimensional network would behave qualitatively differently.

He says theory is only starting to catch up and is incomplete, but it points toward neural networks being universal both in representational capacity, which was already known, and in learning ability, which is the surprising part. That makes him wonder what else is being missed because of low-dimensional intuition.

Silver wonders how future superhuman AIs will view today's algorithms: as naive first steps, or as fundamental ideas still used far into the future. Fridman guesses they will smile and see today's reinforcement learning as too complicated, with the answer simple all along, like the shift from the Sun revolving around the Earth.

Fridman points to cellular automata and the way simple rules can give rise to great complexity, and to Richard Sutton's view that simple methods combined with compute over time tend to win. Silver agrees completely and expects the simple, clear ideas to have the longest legs.

He adds a practical caveat: researchers must make things work day to day, and that sometimes means assembling more complex systems when the minimal ingredients are not yet known. Today's complexity may therefore be a stopgap on the way to simpler answers.