TL;DR:
- MarioVGG is a new AI model that can generate plausible video of Super Mario Bros. from user inputs.
- The model was trained on over 737,000 frames of Mario gameplay.
- Despite limitations, MarioVGG shows potential for AI to replace game engines in the future.
The Future of Gaming: AI-Generated Video
Imagine playing your favourite video game without a traditional game engine. Instead, an AI model generates the gameplay based on video footage. This is the fascinating concept behind MarioVGG, a new AI model that simulates Super Mario Bros. from video data. Developed by researchers from Virtuals Protocol, MarioVGG represents a significant step towards AI-generated video games.
Training MarioVGG: A Massive Undertaking
To train MarioVGG, the researchers used a public dataset containing 280 levels of Super Mario Bros. gameplay. This dataset included over 737,000 individual frames, which were preprocessed into 35-frame chunks. The model focused on two inputs: “run right” and “run right and jump.” Even with these limitations, training the model took about 48 hours on a single RTX 4090 graphics card.
How MarioVGG Works
MarioVGG uses a standard convolution and denoising process to generate new frames of video from a static starting game image and a text input. The model can create gameplay videos of any length by using the last frame of one sequence as the first frame of the next. This results in “coherent and consistent gameplay,” according to the researchers.
Challenges and Limitations
Despite its impressive capabilities, MarioVGG has several limitations. The model downscales the output frames to a resolution of 64×48, much lower than the NES’s 256×240 resolution. It also condenses 35 frames of video into just seven generated frames, resulting in rougher-looking gameplay. Additionally, MarioVGG struggles to approach real-time video generation, taking six seconds to generate a six-frame video sequence.
Impressive Results Despite Limitations
Even with these limitations, MarioVGG can create passably believable video of Mario running and jumping. The model can infer game physics, such as Mario falling when he runs off a cliff and halting his forward motion when adjacent to an obstacle. MarioVGG can also hallucinate new obstacles for Mario, although these can’t be influenced by user prompts.
The Future of AI in Gaming
The researchers hope that MarioVGG represents a first step towards “producing and demonstrating a reliable and controllable video game generator.” They even suggest that AI models like MarioVGG could one day replace game development and game engines completely.
What do you think about the future of AI in gaming? Could AI models like MarioVGG really replace traditional game engines? Share your thoughts and experiences in the comments below. Don’t forget to subscribe for updates on AI and AGI developments.
You may also like:
