Introduction to Interactive AI Gameplay

Microsoft is introducing a new interactive real-time gameplay experience in Copilot Labs. You can now play a real-time AI version of Quake II through this link (opens in a new tab). The gameplay is powered by a model called Muse, and the video you see is generated in response to your controller inputs. This means that as you play, the game adjusts in real-time based on your actions.

What is Muse and WHAMM?

Muse is a set of models at Microsoft designed to understand and generate video game environments. WHAM, or World and Human Action Model, is a core model for generating these environments. WHAMM, which stands for World and Human Action MaskGIT Model, is an improved version that allows faster generation of visuals. WHAMM makes it possible to interact with the model in real-time, creating a gaming experience where your actions have immediate effects.

New Features and Improvements in WHAMM

• Faster Gameplay: WHAMM can generate visuals at more than 10 frames per second, compared to the earlier model, WHAM-1.6B, which only produced one frame per second.
• New Game Adaptation: WHAMM has been adapted to Quake II, a fast-paced first-person shooter, allowing a smooth and interactive experience.
• Efficient Data Use: Unlike the original model, which needed 7 years of data, WHAMM only needed one week of carefully curated data to train for Quake II<.
• Better Image Quality: WHAMM generates images at a higher resolution (640×360) compared to the previous 300×180 resolution, improving the overall gaming experience.

Playing in the Quake II World

Thanks to WHAMM, you can explore a simulated version of Quake II. You can move around, jump, crouch, shoot, and even interact with objects like blowing up barrels. The model also allows you to discover hidden areas in the game, just as you would in the original Quake II. It’s an exciting experience to play within a world created by the AI, offering a fun and interactive gaming environment.

Limitations and Future Improvements

While playing the AI-generated version of Quake II is fun, there are some limitations:

• Enemy Interaction: Enemies sometimes appear blurry, and combat may not always feel realistic.
• Context Memory: The model only remembers the last 0.9 seconds of gameplay, which can lead to strange effects, such as teleporting or spawning enemies.
• Health and Counting Issues: The health system can be unreliable, and interactions with health packs and enemies might not work as expected.
• Limited Scope: The current version of WHAMM is trained on just one part of a single Quake II level. Once you reach the end of this level, the game freezes and restarts.
• Latency: There is some delay in the actions due to the real-time generation of gameplay.

Looking forward, the team is excited to improve WHAMM with the goal of creating even more interactive and realistic gaming experiences in the future.