GameNGen: Simulating DOOM with a Diffusion Model

NVIDIA CEO Jensen Huang has described a future in which game pixels are generated. Imagine a game created interactively for a particular player. In August 2024, a research project called GameNGen offers an early demonstration of part of that idea: it simulates the classic game DOOM using a neural model. This is an experiment based on an existing game, not a system that can already create any game on demand.

DOOM, released in 1993, is a landmark game and a familiar target for people who enjoy getting software to run on unusual devices. I have seen examples on phones, price checkers and even a pregnancy test. Its familiar appearance makes the researchers’ simulated version easy to compare with the original.

Conventional games use code and authored assets to implement rules and display scenes. Procedural generation adds algorithms that create levels or worlds from rules, as in games such as Diablo and No Man’s Sky. I see learned frame generation as a possible next step, although conventional games can also combine authored and procedural elements.

Text-to-image and text-to-video models show how a model can produce visual content from a prompt. I have been excited by OpenAI’s Sora demonstrations and the possibility of more consistent video. But a video that looks game-like is not automatically interactive, and examples do not establish that its physics are always accurate.

The paper “Diffusion Models Are Real-Time Game Engines” presents GameNGen. The researchers show a playable simulation of DOOM in which the neural model generates successive views in response to actions. Enemies, corridors and interface elements are synthesized rather than drawn by the original DOOM renderer during this simulation.

The 2024 paper reports that the system can simulate the game at over 20 frames per second on a single TPU, a processor specialized for machine learning. That is a result for the tested setup, not evidence that a modern commercial game can run the same way on a typical consumer device.

At each step, the model predicts a new frame using a sequence of earlier frames and actions. Training has two phases: a reinforcement-learning agent learns to play the original coded game and its sessions are recorded; a diffusion model then learns to generate the next frame from those observations and actions. It inherits information from the game and human-built training pipeline.

I imagine a future in which a person describes a game or show and receives an experience tailored to them. That could change how content is created and consumed. GameNGen does not demonstrate such unrestricted text-to-game creation; it demonstrates an interactive simulation learned from a particular game.

I also speculate that systems like these could reshape programming and perhaps reduce the need for some development work. Extending the idea still further, one could imagine simulating the real world, raising questions reminiscent of The Matrix and simulation theory. These are my forecasts and speculations, not conclusions the DOOM experiment proves.

The researchers used an augmented Stable Diffusion 1.4 model. Their simulation can approximate updates such as ammunition and health counters, shooting enemies and opening doors, but it is not an exact copy. Some state remains visible in the generated pixels, while the model infers other information from recent context; the paper does not demonstrate a conventional persistent game-state database.

The authors frame GameNGen as a proof of concept for an interactive game represented by model weights. They suggest that modifying existing games, creating a new level from frames or a character from example images could be shorter-term possibilities. Those applications were proposed future work, not features established by this DOOM demonstration.

The paper identifies a major limitation: the model has access to a little over three seconds of history. Some state can persist in the visible display, but once an event is outside that context the model can make incorrect guesses. The training agent also did not explore every location and interaction, which can produce errors in unfamiliar situations.

There are visible artifacts and inconsistencies, including flickering facial details and incorrect counters. Longer memory, better stability, more data and compute might help, though simply generating a new game does not remove the need for consistent rules and state. These remain research challenges rather than solved engineering problems.

My prediction is that generated experiences will eventually reach games, movies, music and software interfaces. I imagine asking AI for the interface and data I need rather than relying on conventional applications or an operating system. That is a broad vision for the future; GameNGen’s measured result is the narrower DOOM simulation described in the paper.