A Few Megabytes of Doom and a Few Gigabytes of Model: Why Runway Bets on Generalization

Nathan Labenz frames Runway as almost synonymous with AI video, with Gen-3 still leading the market. His guest is Anastasis Germanidis, co-founder and CTO. Sitting in as co-host is Stephen Parker, creative director at Waymark and co-creator of the AI film *The Frost*, then on a global tour with a Singapore debut. Parker is a power user of generative tools, including Runway; Labenz jokes that he will be the least knowledgeable person in the room. They start with the largest claim: that video generation is world modeling, and perhaps a stop on the path to AGI.

Germanidis’s starting point is simple. Humans are visual beings; being in the physical world is a fundamental part of being human, and a huge range of human tasks can be formulated in video. Models that train on large volumes of footage pick up representations of the 3D world and of the activities people perform in it. That knowledge can generate video, which is where Runway began, and it can also power other applications, including robotics. Every representation — video, language, or something else — is ultimately a proxy for reality. Video, he argues, comes with fewer biases than the text language models train on.

Labenz presses the obvious follow-up: if the argument is about richness, why not skip to 3D point clouds or grids? Germanidis gives two reasons 2D is enough. First, video is abundant and 3D data is hard to find at scale, which is why 3D computer vision has lagged 2D. Second, 3D knowledge can be learned from 2D footage. Gen-3 already shows remarkable 3D consistency: generate a camera moving in a given direction and objects in the environment stay consistent the way a real 3D scene would. Those two facts — lots of video, and 3D structure that can be derived from it — make video a good candidate for broadly useful representations.

Object consistency reminds Labenz of OpenAI’s line that object permanence and intuitive physics are “purely products of scale.” Germanidis says that matches what Runway has seen. Gen-2 shipped more than a year before this conversation; Gen-3 is recent. The step-up is large, and a lot of it can be attributed to compute, though the team also did other things differently. On the world-modeling side, liquid simulation is surprisingly accurate for a model with no prior knowledge of how liquids behave. Prompts such as throwing water onto a boiling pot reveal physics knowledge embedded in the weights. It was all learned from data and scale. The team introduced no inductive prior to make the model follow the laws of physics.

He does not pretend the problem is solved. The model does not follow gravity perfectly; even a bouncing ball can be hard. Extrapolating the Gen-2 to Gen-3 delta, though, he sees no fundamental reason more precise world understanding will not keep emerging. Labenz then asks whether architecture will converge, and floats KANs from Max Tegmark’s MIT group, with Ziming Liu as first author — networks that learn activation functions and can be initialized with physics operators such as a parabola, which might help gravity. Germanidis has not studied the paper closely. His industry observation is that new architectures supposedly beating the mainstream often mislead: a common trick is to compare two models at the same iteration count without accounting for compute per forward pass. Truly outperforming current architectures is rare.

His own method is to avoid infusing too many inductive biases and instead choose data and the training recipe carefully — a more scalable path, at least in Runway’s experience. Labenz hears Ilya Sutskever in that: the models just want to learn; it is easy to overcomplicate the architecture. Germanidis also diagnoses the field: machine learning has historically focused on architecture over data, objectives, and tasks. Survey a recent computer-vision conference and the imbalance is obvious. Data matters, especially the tasks you teach through data, and that is the part with the least published work.

On the Platonic representation hypothesis — the idea that as models of different modalities scale, their internal representations converge — Germanidis is a fan, and says he had been alluding to it already. He is not arguing that video should be the only training modality. He is arguing it is more general than text. Ultimate systems will be multimodal, each modality capturing some aspect of reality in compact tokens. The industry’s focus on text as the royal road to general intelligence, from Runway’s research vantage, may be misplaced. Sound will be added eventually; frames plus associated audio is the next step. Visual quality is not saturated yet, so concentrating on vision is the better bet for now.

After a sponsor break, Labenz asks directly whether they think of themselves as an AGI lab, and whether world simulation fights the fantastical, magical work users actually make. Germanidis says they are not an AGI lab. The definition has always been unclear to him; other terms are more useful. Runway sees itself as augmenting human intelligence and creativity, not building generally intelligent agents. If the Turing test is the criterion, it is a noisy one: it measures a bystander’s perception. He would rather talk about more capable simulators of different aspects of reality. Reality has near-infinite complexity at different levels of abstraction, so there is no breakpoint where simulation is “done.”

He also dislikes AGI as a goal because it implies a moment of parity with human intelligence. Human capabilities, in his view, are tied to technology; every new system extends what it means to be human, so a finish line of matching “the human” does not quite make sense. World simulators and creative tools are not in conflict. As models get better they approximate reality more closely, learn a wider variety of tasks, and can combine those tasks in novel, out-of-distribution ways. “The better those models become, the better they become at hallucinating,” he says — and in Runway’s case that is a useful property. Gen-3 can already move through an environment, open a door, and reveal a completely different world, because it has learned the constituent parts well enough to recombine them.

Labenz asks about interpretability — sparse autoencoders in language models, and, only half-joking, a “magical realism” concept somewhere in the net that you could turn up. Germanidis is interested and the company is investing more, but he has little to share yet. The payoff would be new knobs: find the concept, activate it, and get the quality you want. On a related speculation, Labenz wonders how long a clip has to be before the little people in it become implicit agents. Generate a day in someone’s life and predicting the next action seems to require some model of reasoning. Germanidis finds that plausible at sufficient scale. What excites him more is keeping the user in the loop — treating these models as new rendering engines or game engines for interactive experiences.

Parker, an avid Runway user and a former New York startup person, asks where the company’s famous shipping culture comes from. When it was the three founders, Germanidis says, continuous shipping was almost existential: if they did not keep reminding the world they existed, nobody would care. Technology now moves so fast that shipping new models and tools is how users keep up and the perception lag stays small. Internally it is also fuel. Seeing research turn into films and artworks in weeks or months is one of the things people at Runway like most about the job.

Listeners may not know Runway had a whole suite before Gen made it famous — color, style transfer, green screen. Generative models were the interest from the 2019 founding; quality was simply not production-ready, so research and product ran in parallel until they intersected. The pre-generative hit was green screen, essentially interactive segmentation: pull a subject off the background. That has been fundamental to editing and VFX for a long time. It is less clear it will stay fundamental. Users do not want a matte; they want an effect on part of the frame. If a generative model can skip compositing and deliver the outcome, that is a better workflow. His framework is outcomes, not tools. Eventually a single interface should absorb the pile of single-purpose computer-vision utilities. The models are not there on user intent yet. He sees no technical reason they will not get there.

How those early tools relate to Gen-3 is a knowledge transfer, not a pipeline reuse. Building the models is a much more complicated stack than a model-powered green screen, but the old products taught the team what video editors actually care about, and that transferred into how generation is productized. Parker then asks about releasing a new model that is better but missing old camera sliders and aspect ratios. Germanidis’s logic is blunt. The jump between generations is so large that waiting for feature parity would do users a disservice. If the next model were only marginally better, parity would matter more. They still want useful old controls on Gen-3, and they are working toward more capabilities than the totality of Gen-2, but they will not hold the quality delta hostage to the old toolbar.

The user base runs from small studios to large ad agencies and film studios to a mass of individual creators. Runway runs an AI Film Festival and 48-hour film competitions; the latest count Parker hears is about 1,800 teams signed up, a hack weekend for people making movies. Enterprise use concentrates in pre-production: storyboarding, an 80% version of a campaign you can show internally or to a client even if the pixels never ship. With Gen-3, some output has started landing in production — B-roll, pieces of the final result — even if you still cannot generate every shot of a film.

Parker asks for a hot take on the Doom player paper — a diffusion model that generates the game frame by frame, the first widely seen mix of interactivity and video. Germanidis is excited about the direction: interactive engines, a video game from a prompt, navigating a world as you create it. He is less excited about this particular proof. Doom itself is probably a few megabytes. The diffusion model trained on its frames is on the order of gigabytes. If you are just generating the same footage from the same game, with no generalization, you have built a less efficient game engine. Variants of Doom that followed a prompt would be super cool. Replaying the original is not the best exhibit.

The paper he found compelling is Genie: a video model that generates arbitrary platform games you can control, and that generalizes to new environments. That is the direction he is most excited about. Parker then asks what it has been like to scale from a small startup into demand for faster generations and more of them. Germanidis says the cultural emphasis besides shipping is learning fast — aspects of building tools and training that the team had zero prior experience in. Runway has been a different company almost every year. Scaling training to Gen-3 was new; they learned it on the job. They try not to scale people too fast. They remain fairly small for their market position, on purpose: “always smaller than we need to be.” Leadership has to be technical and hands-on. Understanding the models only comes from working on them.

Labenz wonders whether these tools are a total leveler or still best wielded by pros. He knew in first grade he would not be an artist; the gap to Parker has closed some, and Parker may now be closer to a Hollywood studio. Germanidis rejects the idea that top people do not benefit. Pros get a lot from Copilot and Cursor: the models speed the choices you can make even if you could have implemented the project alone. The same is true of film. Taste and vision matter as much with these models as with traditional workflows. The tools help people without prior experience express themselves in video. He does not think the gap necessarily closes. What they buy you is a shorter path from idea to output, and a larger number of choices in less time. The choices along the way define the result. People who invest heavily in becoming great at the models see that effort pay off.

Then the sharp question: how do you compete long-term with trillion-dollar companies? Labenz cites an alleged Anthropic view that by 2025–2026 the leaders may be so far ahead — best models training the next models, capturing the revenue — that nobody else can catch up. Germanidis agrees the resource bar is rising fast and Runway must keep investing in compute. Scale matters, but so does the framework you scale and the task you teach. He sees too many companies scaling the same LLM paradigm against the same benchmarks. Unique insight into the task can save an order of magnitude in resources. That, he says, is something Runway has a perspective on that the rest of the industry does not share. He hopes models will diverge in the next generation or two, because the field is exploring too narrow a set of tasks.

On publishing versus secrecy, he wants the gap between public research and in-house knowledge not to grow too large, but competition means you cannot publish every trick. Insights that work get a period of exclusivity; knowledge diffuses anyway as people move. The team’s deepest satisfaction is tools artists love, not novelty that looks good in a paper. Labenz’s selfish last question is the API waitlist. Waymark serves small businesses, and image-to-video is the number-one use case: owners only trust footage that looks like their actual shop. Germanidis says a few customers are already on the API — Canva is the one they have announced — and broader access is in the near future. They want to get the shape right. Since recording, Labenz notes in the intro, Runway did launch the API, and Waymark is in the first wave.

Why is image-to-video so often worse as a standalone clip than text-to-video? Germanidis has seen users succeed with both, and he thinks the task is easier in some cases and harder in others. Easier: the model does not have to invent scene semantics and composition, so it can focus on motion. Harder: it must respond to a wild range of input images it may not handle natively. That is why it sometimes beats text-to-video for a given use case and sometimes produces less coherent motion. For a small-business ad built from a photo of the store, the first of those two facts is the whole product.