Unity Muse Features Expanded In Editor With 3 New Updates

We're excited to announce that Unity Muse users can now access all five Muse features directly from the Unity Editor. By bringing AI capabilities to where you create, Muse can now supplement and streamline your creative process.

While you may be familiar with Muse Sprite for making original 2D assets and the recently upgraded Muse Texture generation for creating a variety of unique textures for 3D objects, this version offers three interesting improvements:

• Muse Chat's new Editor-integrated version now offers project-aware responses.
• Muse Animate, which is now available to all users, enables you to create animations within your project.
• LLM-powered Muse behavior, anyone who uses Muse can now set up character interactions using an intuitive AI-assisted interface.

What is Unity Muse?

Unity Muse is an AI-powered assistant seamlessly integrated within the Unity Editor, the industry-standard platform for creating real-time 3D experiences. Muse leverages the power of artificial intelligence to offer a range of functionalities designed to streamline various aspects of game development.

1. Chat in the Editor provides answers and solutions tailored to your project

Following Muse's launch last year, we requested feedback on how we could continue to improve the chat experience. We heard a strong desire to be able to troubleshoot errors without leaving the Editor. We received feedback that Muse Chat should automatically know more details about a project to save time in providing context and get more relevant answers faster. We've heard your feedback and are pleased to announce that Muse Chat is now available as an in-Editor package. This update also lays the groundwork for providing project-aware responses. Muse Chat can now retrieve key project details such as the Unity version, active render pipeline, input system, target platforms, API compatibility, and other project settings. A project in Editor using Muse Chat to troubleshoot errors.

Example of a project in Editor using Muse Chat to troubleshoot errors.

One example of how this update will make troubleshooting easier is by removing console errors. When you encounter an error, you can now click on it and ask, "How do I fix this error?" Muse will automatically retrieve information about the problem and suggest a solution.

Muse Chat will also continue to provide usable scripts that you can plug and play, as it has previously. Muse Chat will evolve over time into a more intelligent chat-based assistant capable of providing tailored answers and information, code snippets, and smarter Editor integrations.

2. Make your humanoid characters come to life with Animate.

Animation is a complex craft. Prototyping basic movements that can be integrated into game mechanics early on can save you a lot of time later in the development process. Muse Animate is ideal for rapidly creating humanoid animations using natural language.

Similar to Muse Chat's feedback, we frequently heard that Muse Animate needed to be more deeply integrated in the editor. Based on the overwhelming response, we've released a new in-Editor Muse Animate prerelease package that is available to all Muse users. This new version allows you to create Unity Animation Clips directly in the Unity Editor and easily retarget them for use with characters that use the Unity Humanoid Rig.

Using Muse Animate, create a "walking forward" animation for the scene's character.

Muse allows you to edit your generated animations by selecting an animation and decomposing it into multiple poses. From there, you can select effectors (small orange dots on some of the character's joints) to fine-tune the movements. Muse Animate will apply your changes and regenerate the animation. Additional editing features added by the package include extrapolated posing, loop to first pose, and transition duration.

Editing the animation created with Muse Animate

With this release, we are also actively working on improving the existing AI animation model and introducing new animation generation and modification methods, such as the ability to create animations from sketches or videos.

3. Use LLM-powered behavior to give characters actions and reactions.

We've received numerous requests for a built-in behavior tree that would allow you to bring game characters to life using NPC interaction abilities. Last year, we launched a closed prerelease of Muse's Behavior capability, which is our take on LLM-powered decision trees and allows you to create custom logic for characters and objects in a simple workflow. This set of features also included generative abilities to aid in the creation of more complex interactions.

All Muse users can now download the Muse Behavior prerelease package, which includes LLM features. We designed and improved this package to make it easier to use and allow for more intuitive editing. Muse Behavior, like the classic behavior tree model, is made up of human-readable nodes and branches, allowing you to create representations that read like stories.

Example of a behavior tree set up in Muse Behavior.

Words in a node's story are classified as inputs for the node's internal logic. The words are then converted automatically into fields, allowing you to embed data directly into the node. Would you like your character to speak? Add the talk node, enter the words, and your character's dialog will be triggered by player interactions.

Muse Behavior also has a blackboard, which contains variables that can be reused in nodes throughout the graph. Simply click on a field's link button and then select from a menu of assignable options. Variables in the blackboard can be assigned values in code or using the Behavior Graph Agent component in the Inspector window. The ability to set up and then duplicate repeatable actions accelerates and simplifies the creation of complex and repeating interactions.

Variables on the Muse Behavior Blackboard

Muse Behavior allows you to create custom actions, giving you control over the high-level structure and narrative of your behavior trees. And don't worry if you're new to creating behavior trees; the node wizard will walk you through the process of creating a new action type and adding it to the graph. You can also have your tree generated automatically using the LLM feature. As Muse Behavior improves, more generative features will be added to simplify complex behavior tree setups.

A Muse Behavior decision tree is used to create a loop of a guard patrolling and then vocalizing "Alert!" when the player gets too close to the NPC.

A preview of what's next.

These new packages and improvements are just the beginning. We're continuing to improve existing features like Muse Texture, which uses new original research and proprietary models to generate true PBR materials for 3D objects. This means you'll be able to create a multi-material UV texture that can wrap directly onto an object and respond more accurately to lighting.

Three Muse textures were created for the same 3D model

We also understand the value of audio in setting the scene, so we'll release Muse Sound later this year, allowing you to create AI-generated, prompt-to-sound effects like movement and environmental sounds.

Conclusion

With the release of these three new updates, Unity Muse continues to evolve as a powerful toolset for game developers, offering enhanced capabilities and streamlined workflows directly within the Unity Editor. From improved asset management and collaboration tools to advanced visualization features, these updates empower developers to unleash their creativity and create compelling interactive experiences more efficiently than ever before. As Unity Technologies remains committed to driving innovation and empowering its community of creators, we can expect even more exciting developments in the future, further cementing Unity's position as the go-to platform for game development and interactive content creation.

FAQs

Q. What is Unity Muse?

A. Unity Muse is a Unity extension that enables creators to easily generate dynamic, AI-driven dialogue and narrative content within their games or simulations. It provides tools for building interactive conversations and storytelling experiences, enhancing players' immersion and engagement.

Q. What can Unity Muse do?

A. Unity Muse is a tool designed to aid in creating immersive, interactive experiences using Unity. It simplifies the process of integrating Muse EEG headsets, allowing developers to incorporate brainwave data into their projects for applications like neurofeedback, research, and brain-computer interfaces.

Q. What is the importance of the muse?

A. The muse serves as an eternal wellspring of inspiration, guiding artists, writers, and creators to produce their finest work. They ignite the imagination, infuse passion, and catalyze creativity, driving the artist to transcend boundaries and manifest their innermost visions into reality.

Image Reference: https://unity.com/