Users Wanted a Better Brush. Runway Built One-Click Greenscreen

Sarah Guo and Elad Gil open this February 2023 episode of *No Priors* with a guest whose résumé looks accidentally assembled: economics, business, design, then art school, then a machine-learning company used on a Best Picture winner. Cristobal Valenzuela, co-founder and CEO of Runway, grew up in Chile, and he treats those silos as a mistake rather than a plan. Curiosity, he says, is the constant — in his career and in the company.

Runway is, in his phrase, an applied research company. The lab trains neural nets for content creation and video automation, then ships them as a suite of about 35 "magic tools" for editing image, audio, and video. Broadcasting companies, film studios, and post-production houses already use its greenscreen tool to cut the time spent on rotoscoping — the frame-by-frame masking job, he says, that no filmmaker wants to do and every filmmaker has to do. The same stack, the show notes add, showed up on *The Late Show with Stephen Colbert* and in *Everything Everywhere All at Once*.

That is a long way from a 128-pixel blob that New York advertisers dismissed as a toy. This conversation is about the distance in between: how a model directory became a product, why users asked for a better brush and got one-click greenscreen, and why, in creative work, 80 percent accuracy is already enough to change the job.

Valenzuela first studied business and economics in Chile, then a design program heavy on physical computing. For several years he lived two lives at once: art with Arduinos and electronics on one side, consulting for banks on the other. The worldviews were opposite. He says he loved both, and that sitting in each made the other sharper.

Curiosity pulled him into software, then into early computer-vision models around 2015 and 2016. He won a scholarship to NYU's ITP — a two-year media-arts program he still calls a fundamental piece of his career, the place he learned to bridge business, design, art, and technology. With co-founders he showed work at Ars Electronica. Media art, as he defines it, is just another way of expressing a worldview; writing software and making hardware both count.

A famous Chilean photographer who was mentoring him put it in a sentence Valenzuela never dropped: "Chris, this is the same world we all live in." The silos — art, design, economics, business — are walls people build. Gil adds that Silicon Valley's overlap with art is under-discussed: Stewart Brand in the 1970s, the early Mac, Paul Graham's *Hackers & Painters*, Sep Kamvar exhibiting at MoMA after a company Google acquired and later co-founding Celo. Valenzuela, who had not even visited San Francisco until about three years before this taping, still looks at the tech world's norms with outsider eyes and a first-principles habit of asking why a pattern exists.

Runway's first instinct arrived in the AlexNet and ImageNet years, when image classification was the breakthrough, TensorFlow was about a year old, PyTorch had barely landed, and GANs were still embryonic. The visual outputs were fascinating and the tooling was miserable: CUDA dependencies and C++ libraries aimed at researchers. The first product was a thin layer of accessibility — a model directory, an app store that peaked at around 400 models — with an SDK and REST APIs so a designer or filmmaker could call, train, and deploy. Someone fine-tuned GPT-2 and wrapped a text-generation app around it. Most of those experiments vanished. Pieces of the deployment stack are still in the product.

Product sequencing, he says, is one of the most important parts of building. A new technique can take 12 to 24 months of collective tinkering before anyone understands its implications; GPT-3 is his example. Building a long-term company means saying no, including to customer requests that bring revenue and pull you off the long-term plan. Early Runway was a technical product: you needed to know CUDA, Docker, and GPUs. Artists do not care. They care about the idea and how to execute it. The stack on the back end has been rewritten; the metaphors on the front end even more so.

The harder lesson: models on their own are not products. Productionizing a model is a different problem from training one — deployment, interaction, unit economics. Depending on open source and third parties felt like a survival risk in a field that jumps when a breakthrough lands elsewhere. So Runway built an in-house research team and mixed it with creatives: about half the company has an arts background. At one table you might find a PhD contributing fundamental research and someone who has edited video for 20 years. Research is not a department that delivers a paper every six months. Building that muscle, he warns, is not "hire both tribes and lock them in a room."

Guo asks what research-centric founders get wrong. Valenzuela is glad more scientists are leaving academia to ship, and he still thinks the gap between a cool demo and a reliable business is badly underestimated. Researchers assume how people work. The fix is to put tools in front of real users as fast as possible. Gil compares it to Color, where he embedded a bioinformatics specialist with the systems team so needs arrived before code. Runway's answer is people who already speak both languages — and, until two weeks before this taping, no dedicated product manager. Product was led by research, design, and engineering together. The company that worked at five people around a table will not work at twenty, so they are iterating on squads and more specialized engineers.

Greenscreen is the case study. While they were still running the model directory, a cluster of users kept hitting image-segmentation models and doing something odd: exporting video to image sequences with FFmpeg, running an image model, stitching the frames back. Interviews revealed they were film people, doing rotoscoping. No one asked for a one-click greenscreen. They asked for a faster way to make masks — a better alternative to After Effects' Roto Brush. "People are really good at telling you what their problems are," Valenzuela says, and "really hard verbalizing solutions."

Academic literature on video object segmentation was obsessed with fully automatic systems: feed in a video, the model finds the subject. Talk to a professional editor for a few minutes and that design falls apart. A shot of ten people might need the person on the left, or just an arm, depending on the director's idea. A creative tool has to be general, which means it has to be controllable. If they had shipped what users asked for, they would have a slightly better brush. They shipped a system instead.

So they put a human in the loop. The user selects, zooms, and defines the region before the model runs. That interaction also became training data: they built a probabilistic model of simulated human clicks on a mask and trained the segmenter on it. The first version ran at four frames per second, "incredibly slow," and worse than today's. It did not matter. It was still better than anything else. People scrambled to use it and hacked it into their workflows. That scramble, he says, is how you know you hit something. They have been iterating on it ever since.

Zooming out, the customers who pay are people in the storytelling business — post-production agencies, VFX houses, broadcasters, sports companies — for whom video is the job. Guo pushes on a common objection: public research still does not look good enough for highest-production-value work, so how is Runway inside VFX studios at all?

If the goal is end-to-end automated filmmaking, he agrees, we are not there. Look at the actual inefficiencies instead. A 10 or 20 percent cut in the time and cost of rotoscoping is already radical. It also changes what you can try. A director using Runway on a film told him that, before, he and his editor had to marry one idea — try it with the character on the left, and stop — because a second and third version would blow the deadline. Creative work, Valenzuela says, is "very waterfall." Now the director can make all three and pick. "I'm not constrained by the time and the cost." The constraint is the idea. They started treating the product as a verb: just Runway that shot.

The company's goal is not an autonomous system that cuts humans out. People have ideas; the system should get them there fast. Often that means getting 80 or 90 percent of the way. The last stretch is famously hard — autonomous vehicles are always two years away because a one-percent failure can kill someone. Creative domains are not like that. Get to 80 percent, and a professional can finish the rest in Nuke or Flame and still save days. Guo treats that as a filter for applied research: look for domains with built-in tolerance for lower accuracy.

Product-market fit, for Valenzuela, is a spectrum from weak to strong, and you keep aiming at the strong end. Runway started with a tiny audience and was dismissed as art students building toys. He argues you should not dismiss toys: they are a very good way to learn. The early GAN text-to-image system produced those 128-pixel smears. He still has them online. Type "blue ocean" and you got a blue form. Close your eyes, stand ten meters back, and maybe you see beauty.

The New York agency meeting is the one that stuck. He walked in with "the thing you will be using to work." They said the technology was fascinating, "but we have work to do." His mistake was judging a single moment. The right lens is the rate of progress. A year earlier, typing a word and getting an image did not exist. Now it did. Compound from there, even if you cannot picture four or five years out. Viewers of those demos asked how he was collaging existing photos. They did not have a mental model for images that had never existed.

Signals of fit were homely. People started using Runway as a verb. Tutorials and work showed up online with no marketing team and no content-strategy team for a long time. In 2018 an early adopter mailed a painting to his house — the first piece the artist had made with AI, an abstract generation transferred to canvas and then adjusted with mixed techniques.

Asked what he would do differently from scratch, he says he is still learning. He sells less now, because he no longer has to convince people the tools might be useful. He codes less, too. "I code less but I code the company." Gil notes that every company eventually has two products: the one it sells and the company itself. Valenzuela is obsessed with keeping Runway's DNA intact as hiring, communication, and structure get harder — the same scaling problem, he implies, as the backend, only with people.

Gil puts the AI-art fight in a longer argument: Duchamp submitting *Fountain* under the name R. Mutt, Warhol's Factory, where other people assembled a lot of the work. Scandals then, textbooks now. Valenzuela's favorite analogy is older and more practical. Before paint in tubes, painting belonged to people who could afford studios and masters who knew how to mix pigments, store them in bladders, and seal them. The tube was a simple, radical invention. You could carry color into the world and paint *en plein air*. That is how Impressionism becomes possible. Impressionism, he notes, was also not well received: this is not art, these are just brushes of things.

Photography, cinema, and the jump from film to digital repeated the pattern: artists grab the new medium, the old guard says it is not art, then it becomes common sense. He sees two AI-art waves: 2015 to 2022, artists on GANs and VQGAN; then diffusion and transformers. We are still in the paint-tube moment. The artist remains at the center; these are tools. Gil remembers the first GAN works at auction around 2018 at Christie's or Sotheby's, a burst of excitement, then silence until diffusion. Valenzuela names Robbie Barrat as the artist behind those early auction pieces.

What the scene still needs is convenience and control. We cannot yet buy a tube and just paint. Early GANs were a sophisticated process; now you can run this from a phone, closer to putting the cap on the tube. The models are not yet "expressible and controllable." That is how he thinks about alignment: you have an intention, and you want the system to express it in a controllable way. They cannot, yet, because the work is early.

Art movements are also cultural movements. Futurism had an aesthetic and tools, and it was also Italian, about urbanism and cars — and fascism was around, he adds. Apply that now and he would watch "the weirdos of tech," people at the fringes who have always been told they are toying around: creative-coding communities, Babycastles in New York, WordHack. Those highly experimental niches, he thinks, will define a lot of what comes next. The software stack of creation is still siloed — image editing, vector graphics, motion graphics, video, compositing. Multimodal systems might merge them, which is how humans already watch a movie and how an art director already thinks.