Sora, World Models, and the Speculation Around Simulated Reality

The Minecraft-like scene on screen is presented as a video generated by OpenAI’s Sora, not a recording of a playable Minecraft session. I find it remarkable that a video model can produce this kind of scene, and I want to explore what that might mean for games and world simulation. The further idea that our own reality is simulated is a philosophical hypothesis, not something this video proves.

OpenAI had just introduced Sora and shared early examples when I recorded this in February 2024. The visual consistency is striking to me, and I think the technology could eventually change games. My suggestion that OpenAI may already have more advanced undisclosed systems is speculation; the public examples do not establish what those systems can do.

OpenAI titled its research discussion “Video generation models as world simulators” and described scaled video generation as a promising path toward general-purpose physical-world simulators. I contrast this with conventional game engines, which render scenes using modeled objects, lighting, and other calculations. That framing is a research direction, not a demonstration that the generated scene obeys all physical laws.

I suggest that Sora’s learned approach may generate an entire scene differently from a conventional graphics pipeline and may eventually be less costly. The video supplies no cost or latency comparison, so that economic claim is unverified. Similarly, a convincing instance of an object disappearing behind another does not establish perfect object permanence or universally correct motion.

The simulation hypothesis proposes that the world we experience might itself be a simulated reality, with humans as part of that simulation. I enjoy thinking about whether a sufficiently detailed simulated world would be distinguishable from reality, and I mention the series Devs as a related piece of fiction. Better video generation does not show that such a complete simulation is possible, much less that we live in one.

Today’s game graphics can approximate environments but still differ from reality. I imagine a different approach using diffusion and transformer-based video generation instead of explicitly specifying every visual detail. Sora produces generated video; the presentation does not establish that it calculates an entire physically accurate world at once.

The Minecraft-like example includes the appearance of a game interface as well as scenery. I imagine pairing a visual generator with a language model that knows game rules, so someone could describe a game and play it without writing every rule by hand. The clip is generated footage, however, and does not demonstrate real-time control, persistent state, or a playable Minecraft implementation.

I find the early visual results impressive, even though the model was still at an early stage. A generated-looking world is a different way to think about building games, but it should not be assumed to implement accurate game physics or reliably maintain all objects. How the model arrives at each apparent detail is not directly established by looking at its output.

A video of a vehicle traveling through mountains suggests another game concept to me. Rather than using a fixed pre-rendered environment, a future system might generate scenery as the player moves. It might even change mountains to a swamp or switch the rules of gravity in response to a request. These are imagined capabilities, not features demonstrated by the Sora clip.

I also recall an experiment with 25 AI agents in a simulated environment, where agents formed routines and relationships. Combining an approach like that with generated visuals could produce more responsive non-player characters. My description of them as effectively alive expresses a possibility I am excited about, not evidence that such agents are living or conscious.

NVIDIA researcher Jim Fan describes Sora as a “data-driven physics engine”: in his interpretation, training a video model may let it learn visual rendering, intuitive physical regularities, and semantic context. This is a useful way to think about the examples, but a realistic video does not prove that the model consistently reasons over or simulates physics correctly.

Fan speculates that synthetic videos produced with Unreal Engine 5 might have been included in Sora’s training data. That possibility is not confirmation that they were used. I wonder whether learned video generation could someday replace parts of traditional game production, though the video does not establish that it is cheaper or more efficient.

Fan analyzes a generated scene of two pirate ships sailing inside a cup of coffee. He points to the ships’ distinct appearances, their motion, the coffee and foam, and a miniature photographic look. These features support a discussion of plausible visual behavior; they do not establish explicit 3D assets or a validated fluid simulation inside the model.

In a follow-up, Fan explains that he means an end-to-end diffusion transformer conditioned on text or images that generates video, with regularities learned in the model’s parameters. He says Unreal Engine need not run during generation and suggests its synthetic data might have been used for training. That remains a hypothesis about the training set.

ElevenLabs later paired synthetic audio with Sora examples, including sound effects and a narrated scene. I can imagine a future system combining generated visuals, game logic, and audio. The examples demonstrate a combination of media, not a complete interactive world simulation.

Fan notes that the audio example is text-prompted and proposes that an accurate video-to-audio system should also be conditioned on visual information. In his account, it might need to infer object categories, materials, positions, interactions, and the surrounding environment: a stick striking wood, metal, or a drum should sound different.

He imagines the model combining learned sound patterns, adjusting them for the scene, or synthesizing new ones, and layering sounds in a busy environment. He presents these as capabilities an end-to-end model might learn, not as a list of independently verified modules in the ElevenLabs demonstration.

Fan then offers a thought experiment about a hypothetical simulation of reality. Starting with Meta’s 6-billion-parameter Emu Video, he assumes a model ten times larger and 16-bit weights, arriving at a rough figure of 111 GB for the hypothetical binary. The assumed model size is not an established Sora specification, and the calculation is not a measurement of the information or compute needed to simulate the universe.

Fan adds caveats, including the idea that a shortest description might be simpler, and invokes Kolmogorov complexity: the length of a shortest program that produces a given object under a chosen framework. He also speculates that unobserved parts of a world might not need to be rendered. These are analogies and conjectures; the video does not establish that physical reality is generated this way.

Nando de Freitas argues that a finite neural network predicting events must learn internal models that aid prediction, including intuitive physical rules, and says he agrees with Fan’s broad intuition. He anticipates that better data, feedback, and training may lead to stronger physical reasoning. I am excited about the prospect of games that adapt to what a player describes, while recognizing that the demonstrations shown do not yet prove that future.