Volodymyr Mnih: A Tour of Deep Reinforcement Learning Agents

Volodymyr Mnih introduces a historical tour of deep reinforcement learning at DeepMind. He frames intelligence as the ability to make sequential decisions across many tasks, while noting that the specific methods may change.

He distinguishes the reinforcement-learning problem formulation from deep learning as a way to learn representations and computations. Combining them allows agents to act from high-dimensional observations.

Mnih describes the convergence of GPUs, suitable environments and training methods. The Arcade Learning Environment helped make Atari a useful test bed for learning from pixels.

Reinforcement-learning data is correlated and changes as the policy changes. Bootstrapping with a function approximator can make learning unstable; deterministic policies can also get stuck.

He reviews tabular Q-learning: collect transitions, choose exploratory actions and update estimated action values toward a reward plus a discounted estimate of the next state.

With a neural network, successive samples and the target estimates are correlated with the values being updated. These dependencies can cause the network’s targets to move as it learns.

DQN addresses these problems with experience replay, sampling older transitions in minibatches, and a target network whose parameters are copied from the online network periodically.

While target parameters are fixed, the value-fitting problem resembles supervised learning for a time. Mnih answers questions about replay sampling, buffer size and the limits of this analogy.

He credits earlier neural fitted Q-iteration work as an inspiration. DQN adapted the idea for online minibatch updates without retraining a new convolutional network after every episode.

The original Atari network stacked four frames, used convolutional layers and produced an action value for each available action. The same architecture and settings were used across games, but a separate network was trained for each game.

Mnih discusses visual representations and examples of action values during play. In Pong, the estimates change as the ball approaches and reveal when moving the paddle becomes important.

Other examples show estimated values responding to events such as a valuable target in Space Invaders. These visualizations help researchers inspect what an agent predicts, although they do not prove it reasons like a person.

He reviews improvements in median human-normalized Atari scores since DQN and the strengths and limitations of minibatch Q-learning. Single-step bootstrapping can propagate rewards slowly and make similar states difficult to distinguish.

To speed experimentation, the team explored asynchronous methods. Multiple actor-learners on one CPU machine interact with environments and update shared parameters rather than relying on a replay buffer.

The intuition is that parallel actors visit different parts of the state space, making combined updates less correlated. The framework can accommodate several reinforcement-learning algorithms.

In advantage actor-critic, the policy update uses an estimate of an action’s advantage relative to the state value; a value function is trained alongside the policy.

Parallelism can improve not only elapsed time but also data efficiency by reducing harmful correlations. Mnih notes that actor-critic behaved differently from some Q-learning variants in these experiments.

A3C worked with recurrent and feed-forward networks, three-dimensional navigation and continuous control. Smaller networks could be trained efficiently on CPUs, though the architecture faced scaling limits.

Multi-step updates propagate rewards farther in a single update and can reduce dependence between the current estimate and the bootstrap target. Mnih reports that this made recurrent training easier in their experiments.

Distributing A3C across machines increases experience collection but introduces stale gradients: by the time a worker’s gradient arrives, shared parameters may have changed.

IMPALA separates many actors from a centralized learner. Actors send trajectories rather than gradients, while the learner can batch updates on a GPU. Experience can still come from an older policy.

V-trace uses importance weighting to correct for policy lag, with clipping to control variance at the cost of bias. It can mitigate modest differences between the acting and learning policies.

Mnih compares throughput and data efficiency in DeepMind Lab tasks. In the reported experiments IMPALA achieved faster throughput and greater robustness than distributed A3C under the tested settings.

A single IMPALA network trained across 30 Lab levels showed positive transfer in the presented comparison. The talk also reports a single network trained on 57 Atari games, where transfer was more difficult because games look and behave differently.

For practical work, Mnih recommends starting with a controllable toy problem that runs quickly, then plotting learning curves, agent behavior and value estimates to diagnose failures.

His example is a simple catch game whose size, paddle and falling-object behavior can be varied before moving to Atari. It acts as a fast check on the learning algorithm.

Reward shaping can help learning but may also create an unintended optimum: a robot rewarded for holding and moving an object might never learn to release it. The optimized reward must be compared with the actual goal.

He closes with advice on discounting, network size, initialization and optimizers. A small model is useful for debugging, but limited capacity can itself make learning harder; initialization should preserve exploration.