Cristóbal Valenzuela on Interactive Video Models and Runway’s Continuum

I have been working on Runway for almost a decade. I want to trace what we have learned about generating video and pixels, and where we think that work could lead. I joke about living in a simulation, but the research questions behind that joke are serious.

Our first AI film, “The Orange,” was made in 2019 with imagery at 128 by 128 pixels, then enlarged. It now feels like an artifact from an early era of the medium. At the time, we were trying to understand whether scaling the approach could eventually produce much more convincing video.

I show a newer generated clip whose people and locations are not real. The improvement is striking, but I expect that today’s output may itself look primitive in a few years. Runway combines researchers working on video models with people rooted in creative communities.

Moving pictures create an illusion of motion by showing frames in succession. We take a related idea further: predict the next frame from earlier frames, scale the model and make the process fast enough to support more than a polished creative clip.

We believe scaling laws matter for video as well as language. As models and compute grow, we look for capabilities that emerge from predicting successive frames, rather than treating each frame as an isolated picture.

I show an image of two generated people and continue predicting frames from it. In our tests, people have had difficulty distinguishing some generated videos from real footage. That observation is about the examples and tests we ran; the broader question for us is what the model must learn to produce coherence.

A consistent scene calls for relationships involving movement, gravity, occlusion and object permanence. For a complex image, the model must produce plausible interactions among objects, water and the space around them as the sequence unfolds.

Our evaluation is therefore not just whether someone likes a creative result. We also examine geometry, physical behavior and consistency in simulated environments. These tests help us ask what the model can reproduce, rather than assuming that an attractive frame is enough.

If next-frame prediction is fast enough to respond to an environment, it starts to resemble a simulation system. In the example I show, the apparent game world is generated as video rather than produced by a conventional explicitly described physics engine.

The earlier examples were not real-time: generating them took seconds or minutes. Looking several years ahead, we are interested in making these systems interactive so a person can act during the prediction process and influence what happens next.

We are also exploring robotics at Runway. The idea is to investigate whether models developed for visual worlds can help embodied systems reason about and move through their surroundings. This is an area of research, not evidence that every simulated behavior transfers to a physical robot.

Another direction treats an interface itself as generated video. In one demonstration, I click on a car to turn its lights on, rotate it and open the hood. I describe this as a world interface model: it responds to actions even though the demo is not built on a conventional 3D scene in Unity, Unreal or Blender.

A second example moves objects within a pictured room. The model generates the visual result of an interaction without requiring that specific environment to have been manually modeled as a 3D scene. Our hypothesis is that sufficiently capable next-frame prediction can support this kind of interface.

At Runway we try to extrapolate what may be possible six, twelve or eighteen months ahead. One early research idea is Continuum: a real-time video or world model used as part of an operating-system-like experience. The demonstration is exploratory, not a description of a finished operating system.

In the first example, I research San Francisco fog on a real web page, select part of its content and create what we call a portal. The portal turns the selected material into an interactive generated video. Earlier portals can be revisited like a visual memory.

A photosynthesis example changes as I resize the application and interact with the visual content. It sits somewhere between a visualization, a video and a game; we still need language for this kind of interaction. A generated illustration should be understood as a representation, not automatically as verified scientific evidence.

We also connect Continuum to a coding agent in an experiment we call visual thinking. If an agent is designing a 3D model of a bicycle, it can use a generated visual space to explore the problem rather than working only through code and text.

A person can inspect that visual space too and use it to guide the agent. The possibility that the human and agent can both interact with an evolving representation is what makes the idea compelling to us.

A final demonstration turns a search about Mars into a reactive experience: Continuum proposes an interactive outpost interface related to the query. I describe an intention to ground it in information about current or projected robots, but the demo itself does not establish that every generated detail is accurate.

When I say that everything will be simulated, I am offering a direction for research, not claiming that all of today’s systems are finished. Continuum and our other simulation work remain early. We hope to release much of what I showed over the following eight to twelve months, though that is a plan rather than a guarantee.