Back to Blog
Aug 13, 2023 2 min read AI
Has the AI-Era Come to Video Games Already?

Has the AI-Era Come to Video Games Already?

Exploring how large language models and AI technologies are transforming video game development and player experiences.

Deep Learning NLP LLM Video Games

The convergence of large language models and video games is creating fascinating possibilities. Let me examine several practical demos and research projects showing AI’s integration into gaming.

The Matrix AI-Powered NPCs Demo by Replica Studios

Traditional game dialogue relies on pre-scripted responses. This demo enables dynamic NPC conversations using language models, allowing players to engage with characters naturally rather than selecting from predetermined options.

Key capabilities:

  • Conversational NPCs with distinct personalities
  • Voice input/output functionality
  • Unique gameplay experiences for each player

Current limitations: Response delays of approximately 10 seconds due to API calls and processing modules (ASR, TTS).

Herika by Dwemer Dynamics

A Skyrim mod featuring a ChatGPT-powered AI companion that understands audio and text commands. The system integrates game state extraction with language model queries for context-aware responses.

Challenges: Long response times and high API costs.

AI Playing Tomb Raider

This demonstrates AI controlling player characters directly using techniques combining LLMs, TTS, and object detection, though control mechanisms weren’t fully clarified at publication.

Academic Research Context

Previous work like OpenAI Five (Dota 2) and reinforcement learning environments (OpenAI Gym, Unity ML-Agents) established foundations. Recent projects include Stanford’s Generative Agents and Nvidia’s CALM, which uses “surprisingly small models — several hundred parameters” for character control.

Key Challenges and Solutions

Problems identified:

  • Latency in API-dependent systems
  • Large model sizes impractical for local deployment
  • Issues with hallucination and unsafe responses

Proposed solutions:

  • Training smaller, task-specific models
  • Exploring audio-to-audio approaches
  • Researching grokking phenomena in minimal parameter models
  • Player-generated data for companion training

Updates (August 20, 2023)

  • NVIDIA Omniverse ACE: Integrated toolkit for building interactive avatars
  • Mantella: Similar to Matrix demo but for Skyrim
  • AgentSims: Academic project allowing players to influence AI agents

Vision

The convergence of deep learning and gaming could create transformative experiences, potentially inspiring players to pursue real-world learning and practical skills.