Pim de Witte on Training Visual Action Models with Video Games

Imagine training on existing information and using the result on problems not explicitly represented in the training data. General Intuition is pursuing an analogous idea with interactive visual data from video games, though whether it generalizes to a particular robot is an empirical question.

I use a scene from Frozen to illustrate continuous observation while acting. Unlike a purely turn-based exchange, an embodied system needs to respond to a world that keeps changing while it moves.

I argue that purpose-built action models can fit this setting better than treating a text language model as the entire control loop. This is our architectural thesis, not a controlled comparison presented in the talk.

Our starting point is Medal, a game-recording platform whose users record play across many games. We associate frames with controller or keyboard actions and label what the actions mean, creating training examples for a vision-based model.

The intuition is that a person can use a game controller both in a game and to teleoperate some robots. This provides a potentially shared interface, although a familiar controller does not eliminate differences between simulated and physical environments.

The model receives frames and a set of available controls, optionally with a text instruction. In a simple example, an instruction to wave back can be mapped to an available greeting action. The result is an action rather than a paragraph of text.

I contrast this with text generation, where a model often produces a response and waits for the next turn. Our design repeatedly receives visual observations and predicts controller actions, aiming to operate while the scene continues to unfold.

Our bet is that many problems can be expressed as pixels in and actions out. The output space can include keyboard, mouse or game-controller actions, extending the intuition of computer-use models to a wider set of controls.

Robots already have control interfaces that translate human commands into low-level motor behavior. Rather than predicting every motor torque directly, our approach aims to use such interfaces where they exist.

I say Medal currently sees about a billion video uploads per year and is on track for three billion, yielding more than 1.5 trillion action tokens per month. These are the company’s reported scale figures; the presentation does not independently establish how much data is suitable for every robotics task.

The training bet is that scale across games can help a single model handle a broad range of visual-action problems. I compare that bet to focusing on one modality and expecting users to bring more problems into it.

We also work on world models that generate additional environments for training. I argue that learning in visual space could reduce some translation steps involved in building a digital twin and checking whether a simulated task matches reality.

Games can contain dense sequences of actions. I suggest this may help with tasks such as drone control, where many small adjustments are needed, and that high action density provides a different training signal from slower teleoperation.

The action labels are intended to preserve semantic meaning as well as the key being pressed. I present this as useful for interpretation, although a label alone does not guarantee that a model’s internal reasoning is transparent.

I say the company plans to release models later this year and is showing selected live demonstrations. I claim that one model can play games at a high level and control more than ten types of robots using the same primitives; the presentation does not provide an independent benchmark for that claim.

The proposed deployment still uses an orchestrator model for longer-term planning, with the visual-action model plugged into a workflow for real-time interaction. I suggest possible uses such as exploring a user interface, rather than claiming each use case is already validated.

One accessibility idea is to let a person control a mouse while the model supplies complementary keyboard actions, potentially making games easier to use with one arm. This is presented as an application idea, not a tested clinical or accessibility outcome.

I describe three teams working on world modeling, the policy or agent, and data and environment generation. I also discuss planned early access and events; those dates are plans stated at the time of the talk.

Asked how the approach differs from JEPA, I say our training uses action labels, while I understand JEPA’s approach as relying more on inferred actions. I see each as an approach with different tradeoffs and expect ideas to flow in both directions.

For example, video frames alone may not reveal a pilot moving a rudder control. I argue that directly observed actions can help train action sequences, while the team also uses different methods for world modeling.