Matthew Berman Reviews Sora’s Early Filmmaker Experiments

OpenAI has shared new Sora videos and sought feedback from directors, artists and other creative practitioners. I want to look at what they have made, what looks promising and what questions remain about how creators might use the model.

Director Paul Trillo has made a stylized sequence called The Golden Record, imagining a journey from raw materials on Earth to a time capsule of human life. He says he cut together 11 different Sora generations to explore that imagined odyssey. The finished work is an edited sequence, not a single uninterrupted generation.

Next is a film by the filmmaker group shy kids. Its balloon-headed character says that being full of hot air brings difficulties, from windy days to an ill-advised trip to a cactus store. The character also describes seeing the world differently and finding a reminder that life is fragile.

The filmmakers say that Sora’s ability to produce realistic-looking images is impressive, but they are especially excited by making the surreal. I think that is a recurring theme: generated video can make images that might be difficult to envision or capture, not just imitate existing reality.

Another stylized clip follows, with striking changes of imagery. I also show work by Nik Kleverov of Native Foreign, including a clockmaker in a detailed workshop. I am impressed by the detail and by how objects appear to remain coherent across the scenes, though a few examples cannot establish reliability across all generations.

Several more examples explore unusual materials and forms, including clothing that looks like stained glass. One clip has a texture that I find unsettling, so I warn viewers who are sensitive to clustered patterns. These scenes show how the model can be used to explore a visual idea rather than reproduce a conventional shot.

Another artist, Don Allen Stevenson III, presents a sequence of imaginary hybrid animals. It includes a giraffe-flamingo, flying pigs, a whale-octopus and an eel-cat. A narrator describes these creatures as inhabitants of imagined parallel worlds.

The sequence continues with a bunny-armadillo, a horsefly that combines a horse and a fly, a reptilian kangaroo and a fox-crow. The strange combinations are precisely the point: the video gives recognizable animal traits to creatures that do not exist.

The final artist example is by sculptor and OpenAI artist in residence Alex Reben. At the time of this March 2024 video, I do not expect Sora to be released that year; that is my speculation, not an announced schedule. I imagine the team is considering both creative possibilities and possible harms, including deepfakes.

I also wonder whether video generation could move toward a world model able to predict aspects of reality. That is a possibility, not something these clips establish. Looking at earlier examples, a supercar drives through a rainy city at night; the scene looks impressive to me, although I notice shadows that look odd.

I imagine a future video game generated as it is played, if sufficient computing power becomes available. That is a speculative application, not a feature demonstrated here. Another example, posted by Tim Brooks, takes the viewer through a museum filled with paintings, sculptures and works in different styles.

A further example posted by Sean Ralston depicts a bee moving through a field of flowers. It looks striking, though I notice a small white object behind the bee and cannot tell what it is. Such details matter when assessing a generated scene, even when the overall effect is convincing.

I refer to a post summarizing a talk at Berkeley about the technology behind Sora. Its account attributes emerging simulation capabilities largely to compute and scale, without explicit inductive biases. The post is a summary of a talk, and the interpretation should not be treated as proof that the model accurately simulates physical reality.

That claim contrasts with Yann LeCun’s argument, in a separate interview, that scaling alone would not suffice to turn language models into world simulators. I see the Sora examples as part of that debate and wonder whether further scaling could support more realistic simulations; the outcome remains an open question.

A comparison shows puppy videos produced at baseline, 4× and 32× compute. The baseline is hard to make out, the 4× example is recognizable but blurry, and the 32× example looks much clearer to me. It is an illustrative comparison of particular samples, not evidence that the best sample is flawless or indistinguishable from reality in general.

I would like to see what happens at 64× or 128× compute. For now these examples show how rapidly the visual quality can change within a demonstration, while leaving questions about cost, reliability, creative control and safe deployment.