Diffusion Models Are Real-Time Game Engines
akhaliq
Abstract
We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction with a complex environment over long trajectories at high quality. GameNGen can interactively simulate the classic game DOOM at over 20 frames per second on a single TPU. Next frame prediction achieves a PSNR of 29.4, comparable to lossy JPEG compression. Human raters are only slightly better than random chance at distinguishing short clips of the game from clips of the simulation. GameNGen is trained in two phases: (1) an RL-agent learns to play the game and the training sessions are recorded, and (2) a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions. Conditioning augmentations enable stable auto-regressive generation over long trajectories.
Community
Amazing work!
Amazing
๐ฅ๐ฅ๐ฅ
We need similar work, but on yume nikki fangames
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Generative Image as Action Models (2024)
- EgoSonics: Generating Synchronized Audio for Silent Egocentric Videos (2024)
- Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics (2024)
- PianoMime: Learning a Generalist, Dexterous Piano Player from Internet Demonstrations (2024)
- Real-Time Video Generation with Pyramid Attention Broadcast (2024)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
I did a post about this paper! Here's the summary:
๐ฎ ๐ ๐ป๐ฒ๐๐ฟ๐ฎ๐น ๐ป๐ฒ๐๐๐ผ๐ฟ๐ธ ๐๐ถ๐บ๐๐น๐ฎ๐๐ฒ๐ ๐๐ข๐ข๐ : ๐ผ๐ฝ๐ฒ๐ป๐ถ๐ป๐ด ๐๐ต๐ฒ ๐๐ฎ๐ ๐ณ๐ผ๐ฟ ๐ฐ๐ผ๐บ๐ฝ๐น๐ฒ๐๐ฒ๐น๐-๐๐-๐ด๐ฒ๐ป๐ฒ๐ฟ๐ฎ๐๐ฒ๐ฑ ๐ด๐ฎ๐บ๐ฒ๐!
Imagine if games were completely live-generated by an AI model : the NPCs and their dialogues, the storyline, and even the game environment. The playerโs in-game actions would have a real, lasting impact on the game story.
In a very exciting paper, Google researchers just gave us the first credible glimpse of this future.
โก๏ธ They created GameNGen, the first neural model that can simulate a complex 3D game in real-time. They use it to simulate the classic game DOOM running at over 20 frames per second on a single TPU, with image quality comparable to lossy JPEG compression. And it feels just like the true game!
Here's how they did it:
- They trained an RL agent to play DOOM and recorded its gameplay sessions.
- They then used these recordings to train a diffusion model to predict the next frame, based on past frames and player actions.
- During inference, they use only 4 denoising steps (instead of the usual dozens) to generate each frame quickly.
๐๐ฒ๐ ๐ถ๐ป๐๐ถ๐ด๐ต๐๐:
๐ฎ๐ค Human players can barely tell the difference between short clips (3 seconds) of the real game or the simulation
๐ง The model maintains game state (health, ammo, etc.) over long periods despite having only 3 seconds of effective context length
๐ They use "noise augmentation" during training to prevent quality degradation in long play sessions
๐ The game runs on one TPU at 20 FPS with 4 denoising steps, or 50 FPS with model distillation (with some quality loss)
The researchers did not open source the code, but I feel like weโve just seen a part of the future being written!
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper