AI NPCs that can Control their Actions along with Dialogue

Imagine a game where AI-powered Non-Player Characters (NPCs) can not only respond to your questions but also perform actions based on your requests! 😄 Cohere's Command model generates character responses, including dialogue and actions, using templates that define their talking style, action list, and bio. 💬 Unreal Engine, even though it can't run Python directly, communicates with it through text files, enabling real-time interaction. 🎮 AssemblyAI transcribes player dialogue, while ElevenLabs generates voice dialogue, which is then imported into the game using the Runtime Audio Importer. 🗣️ The MetaHuman SDK brings these characters to life with facial animations. 😮 Actions requested by players are parsed and executed, ranging from animations to item spawns and even AI-generated music creation. 🕹️

The Repository contains the python script and the demo.umap which you can import in a level in unreal to look at the level blueprint and see the logic. 📂 There are a couple of other files as well that can help you understand better. 📚 If you find this repository helpful, consider giving it a star! ⭐

Video Demo ✨ ☃️

Link: https://www.linkedin.com/posts/akshit-ireddy_generativeai-ai-gaming-activity-7107049969966059520-C-WC?utm_source=share&utm_medium=member_desktop

Explanation: https://medium.com/@akshit.r.ireddy/how-i-made-ai-npcs-that-can-control-their-actions-along-with-dialogue-7f7a45adc95e

Discord Server: https://discord.gg/CfK7DCWKwy

Tools 🚀 🔧

• 🤖 Cohere's Language Models and Embedding Models
• 🗣️ ElevenLabs
• 📝 AssemblyAI
• 🗃️ ChromaDB
• 🔗 Langchain
• 🎮 Unreal Engine
• 🌲 Quixel Bridge
• 🧔 Metahuman SDK
• 🙆 Mixamo
• 🏃 Runtime Audio Importer

3d Assets 🧊 🏰

• "Warlock's End - Metahuman Armor" (https://skfb.ly/oHInI) by pdravecky.fx is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).(https://sketchfab.com/3d-models/warlocks-end-metahuman-armor-e3352bc5d74f4c38afe7bdf700420947)
• "Sci-fi suit for Metahuman" (https://skfb.ly/oFuBz) by Alexander Kurmanin is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/). (https://sketchfab.com/3d-models/sci-fi-suit-for-metahuman-0e938256aa4f4af3b3a23c811da585fa#download)
• "Magic Ring - Blue" (https://skfb.ly/6Uq7O) by Noob Model :D is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).(https://sketchfab.com/3d-models/magic-ring-blue-5c81d8a7610543779d0c7e85f9175fae)
• "Magic Ring - Yellow" (https://skfb.ly/6WMOx) by Noob Model :D is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).(https://sketchfab.com/3d-models/magic-ring-yellow-757ff3fbfb2045c68165c06494917faa#download)
• Quixel Bridge (https://quixel.com/bridge)
• Metahuman Creator (https://metahuman.unrealengine.com/)

❤️ Thanks

If you found this interesting check out Alystria AI for more fun projects

https://www.linkedin.com/company/alystria-ai

• 🌐 Github: https://github.com/AkshitIreddy
• 🐤 Twitter: https://twitter.com/Akshit2089
• 💡 LinkedIn: https://www.linkedin.com/in/akshit-ireddy
• ✍️ Medium: https://medium.com/@akshit.r.ireddy