Sora's Leads: A Minute of HD Video Is Just the GPT-1 of Video Models

Sarah Guo and Elad Gil welcome the three co-leads of OpenAI's Sora — Aditya Ramesh, who also built DALL·E, Tim Brooks, and Bill Peebles — two months after the February 2024 demo of a model that turns a text prompt into a visually coherent, high-definition clip up to a minute long. OpenAI's mission is AGI. The first question is whether text-to-video is even on that path. All three say they co-lead the team. The answer, from Brooks, is that models like Sora are on the critical pathway.

The sample they keep coming back to is a winter scene in Tokyo: a camera flying through the street, people talking and holding hands, vendors selling from nearby stalls. The complexity of that environment, Brooks says, is packed into the weights of a neural network. To generate truly realistic video you have to have learned how people work, how they interact with others, how they think — and not only people, but animals and any object you want to model. Keep scaling, and the claim is that anyone can run their own simulator, send a human in it to do some work, and wait for them to come back.

Gil asks whether the same stack becomes a physics engine — wind-tunnel simulations — or a basis for robotics. Ramesh thinks carrying out simulations inside the video model is something they will be able to do in the future. Peebles takes the robotics thread: language, the modality OpenAI has invested in for years, does not tell you the minutiae of how arms and joints move through space, or how those Tokyo pedestrians' legs make contact with the ground in a physically accurate way. Training on raw video teaches that. They believe it will be essential for physical embodiment.

Demand for a consumer video product is obvious. Brooks is blunt: they do not currently have immediate plans, or even a timeline, for creating one. What they are doing instead is giving Sora to a small group of artists and to red teamers — the first to say how it could be a useful tool, the second to say how it could be introduced safely. That feedback will set the research roadmap and inform whether they ever ship a product, and on what timeline. The most consistent artist request so far is controllability. The model really only accepts text, which is useful and still pretty constraining when you want a precise description. They are thinking about inputs other than text.

The work that moved Brooks was less a single gorgeous clip than a story. The Toronto collective Shy Kids used Sora for *Air Head*, a short about a character with a balloon for a head. What struck him was that Sora unlocked a story they wanted to tell; the quality of any particular generated shot came second. Peebles's own favorite is Bling Zoo, which he posted on Twitter the day Sora launched: a multi-shot scene of a New York zoo that is also a jewelry store, saber-toothed tigers decked in bling. He cannot draw and does not have camera craft. Iterating prompts to something he loved took less than an hour.

Gil analogizes to Pixar: shorts that, over about thirty years, became features, with progress gated on world-modeling details as small as hair. When will Sora-class output enter professionally produced media? Brooks will not pick a date. Over the next couple of years he expects more films, and he is more interested in uses that are not traditional films at all. Telling a model what you want to see, and having it respond, is a different paradigm; the most exciting things will be modes of interacting with content that nobody has thought of yet. Gil's own sketch is "desktop Pixar for everyone": he already cobbles custom Magic Tree House-style audio stories for his kids with voice cloning and DALL·E images, computationally expensive and clumsy. Kids, he thinks, will find real-time narration plus magical visuals first — a different entertainment, education, and communication stack. Brooks notes that below a certain age, people will tell you YouTube is already the biggest educational force in the world, for better or worse.

Sora sits on two OpenAI research lines: the DALL·E models' diffusion process and the GPT models' Transformer. Diffusion, Brooks explains, starts from a video of noise and iteratively removes it until a sample appears. The architecture has to scale — to learn from a lot of data and from the complex relationships in video — so they use a Transformer, the same family that gets better as you add compute and data. A technical report shows the same prompt at small, medium, and large compute, with quality stepping up by eye. They strongly believe that trend continues. Peebles says the team is actively writing scaling laws for video the way language has them, and spending its days asking whether the same training compute can buy an even better loss.

Tokenization is the other inheritance. Language models have one notion of a token, so books, code, and math can all go in. Prior visual generators had no analog: you trained on 256×256 images, or on 256×256 video exactly four seconds long, and threw away much of the visual internet. Sora's answer is space-time patches. Imagine a video as a vertical stack of frames and cut 3D cubes out of it; those cubes are the tokens the Transformer sees. The result is not a fixed-duration 720p clip. Vertical, widescreen, anything from a 1:2 aspect ratio to 2:1, and still images — Sora is also an image model. Peebles calls it the first generative model of visual content with breadth of the kind language models have.

A huge part of the project was infrastructure for that data, which previous image and video systems did not need. Most video work before Sora took a strong image generator and extended it a few seconds. The Sora team started from scratch with one goal: a minute of HD. That goal, Brooks says, told them they could not just extend an image generator; they needed data broken down simply enough for a scalable model. They were not the first video generator. They picked a point further in the future and worked toward it for a year, against the industry pressure to take what works now and add a layer. Guo hears the same turn self-driving just made, from heuristics and edge cases to end-to-end deep learning.

Gil wants to know how they crafted Sora's visual aesthetic — traditional image models get human feedback, and sometimes literal tuning. Peebles's answer is that they did not spend a ton of effort on it. "The world is just beautiful." Sora's language understanding lets a user steer with hints and visual cues in a way other models make harder, but there is no Aditya aesthetic deeply embedded. What creators actually want next is personalization: upload a whole portfolio, write captions in the jargon a design firm has accumulated over decades, and have the model grok it. That, he thinks, is the interesting aesthetic problem.

Guo asks whether they have tried digital avatars, the vein Synthesia and HeyGen are in. They have not. Attention has been on the core technology, not particular applications, avatars included. Brooks's placement of the project is explicit: where they are in Sora's trajectory is the GPT-1 of this new paradigm of visual models. The job is fundamental research to make a much better engine that could power all of those downstream things. Specific products, including a deep encapsulation of a person, are later.

Safety is where avatars would have led anyway. Ramesh says a lot of mitigations can be ported from DALL·E 3, including how they handle gory images. Video adds new issues: misinformation, and whether users can generate frames that carry offensive words. How much responsibility sits with the company deploying the model, how much with social platforms that should flag untrusted sources, and how much with the user is, he says, still an open allocation. Gil notes the Photoshop precedent: people have published manipulated stills for years without making Adobe liable. The team wants people to feel free to express themselves, and that is sometimes at odds with releasing gradually enough that society can get used to the tool. In an election year, Peebles adds, being careful about misinformation is a large part of the research roadmap, alongside making generation cheap enough to democratize. Long clips are not instant; you wait at least a few minutes.

Brooks's personal excitement on the roadmap is not a feature list. It is what people will make. Brilliant creative people often cannot ship an idea because they lack resources or tools. This technology, he thinks, can let them. That is the capability he is most excited to see land.

The quality gap they care about, looking toward a GPT-2 or GPT-3 moment, is complex long-term physical interaction. Concrete failure: someone playing soccer, kicking a ball, and at some point the ball vaporizes and maybe comes back. Simpler interactions, people walking, already work pretty reliably. Detailed object-to-object physics is still in the oven; they think scale will cook it. Glimmers are already there. A man takes a bite of a burger and the bite mark stays. A woman paints watercolors and the brush leaves a trail. State, in other words, sometimes persists.

What Sora does to the broader research roadmap, Brooks says, starts with what it learns just from seeing. It understands 3D. Nobody baked 3D information in; they trained on video, and 3D exists in video, so the model learned it. It learned that a bite leaves a mark. So much of what humans see and learn is visual that this kind of grounding — how people interact, how the past changes the future — should make AI models more intelligent in general, not only better at generating clips. Gil analogizes to inventing a visual cortex plus some of the reasoning stack at once. Brooks takes the comparison: a lot of human intelligence *is* world modeling, playing out scenarios before acting, dreaming them. Building Sora as a world model is close to that.

Gil presses the gap between a human's approximate world model and a classical physics engine. Drop an apple and you expect a rate of fall; you do not compute a trajectory. Peebles treats the approximation as a deficiency. Humans cannot do accurate long-term prediction on a narrow slice of physics; these systems can improve on that, and he is optimistic Sora-class models will one day be more intelligent than humans *as world models*. It is also existence proof that high-fidelity physics is not required for other kinds of intelligence. Either way, future models will get better at it. Football trajectory prediction, Gil says, is an obvious next-version win.

That is the scaling paradigm and the bitter lesson. Methods should get better as you add compute. What works is the simple, hard task of predicting data. Complicated objectives that do not use video explicitly, or that live in some approximate simulation space, do not help the scaling laws. With text they predict text; with Sora they predict visual data. "The best way to learn intelligence in a scalable manner," Peebles says, "is to just predict data." Ramesh closes the GPT analogy. When GPT-1 and GPT-2 landed, it started to become clear to some people that scaling next-token prediction would yield surprising skills, including, eventually, useful code — which was not obvious on day one. Internally it felt clear that the same methodology on video would do the same. Sora is existence proof of one point on that curve. Gil's line: he does not know why it is always a surprise, but the bitter lesson wins again. Peebles calls it a GPT-1 moment. The models will get a lot better, quickly, for creative work and for AGI, if they can build a safety stack robust enough that society actually receives the upside.