Traditional game design requires developers to manually script every potential action a non-player character (NPC) might take. This limits character behavior to what a designer can predict during development. To reduce this bottleneck, we experimented with feeding Large Language Models (LLMs) a constantly updating stream of environmental data and available actions, allowing them to make dynamic, real-time decisions.

I architected and built a custom Unreal Engine 5 c++ plugin that introduces a lightweight actor component that allows any Unreal Engine Actor to use an LLM for autonomous decision-making.

• How it Works Technically: The C++ component scrapes data from the surrounding game world and compresses the spatial and event information into optimized JSON text strings of a restricted length. It pipes this text data via websockets to either a remote cloud LLM or a model running locally on the user's computer.

• Limiting Scope: Crucially, the system keeps the available actions and targets broad to avoid overwhelming the model with choices. Instead of predicting every outcome, when the LLM attempts an action or requests more information, the game engine processes the logic and passes the results back to the model in the next world event update.

• High-Level Impact: Instead of writing a rigid script that says "If the player says X, walk to the door," we give the AI situational context (the player is stuck behind a door), a list of verbs (go to, open, pick up), and a list of nouns (the door, a key, a handle). The AI determines how to string them together based on the situation, dynamically adapting as it receives results from the game engine.

This framework allows the NPC to autonomously chain multi-step behaviors together without requiring constant new prompts from the player:

[Command: "Open Door"] → NPC "Goes to" Door → Receives update: "Arrived"

⬇

NPC Executes "Open" → Receives update: "Action Failed (Locked)"

⬇

NPC Checks Inventory → Executes "Use Key on Lock" → Door Opens Successfully

While LLMs can generate natural-feeling conversations, they present two main challenges in interactive experiences: they are unpredictable (prone to "hallucinating" facts or breaking character), and off-the-shelf cloud models are too expensive to query at scale for millions of concurrent players.

To solve this, we are researching methods to restrict and guide an LLM’s output without losing its organic conversational flow. Our goal is to run smaller, open-source models (Gemma, Qwen, etc. via C++/llama.cpp) locally on hardware with limited computing power. Because smaller models can produce lower-quality responses, we keep the AI aligned with the game's script by using a pre-generated database of 10,000+ writer-guided responses.

We built the system to evaluate performance across a combination of these core techniques:

• Vector Embeddings & Comparison: The system converts the player's input into a mathematical representation (a vector) of its semantic meaning and compares it against our curated database to select the closest contextual response.

• Dialogue LoRAs (Low-Rank Adaptation): We use our dialogue database as a training set to fine-tune the local model, teaching it to mimic the cadence, vocabulary, and personality of a specific character.

• Substring Constraint Enforcement: We force the local LLM’s token-by-token text generation to strictly conform to sentences present in the database, guaranteeing the final output matches what’s in the database.

• Database Filtering and Tagging: The system uses the local LLM to recursively sort player input into a limited set of categories, narrowing down the pool of response options.

Our evaluation of these techniques is ongoing, and our final product will likely use a combination of some of them.

An additional, ongoing challenge of this project is developing the authoring tools that allow narrative designers and level editors to easily guide these AI behaviors to account for unexpected player actions and unfolding plot points without facing decision fatigue.

Our initial phase involved evaluating major commercial cloud tools like Runway, Google Veo, and OpenAI's Sora. While these models are incredible for generating a fast "rough draft" to communicate a concepts, they fail under production constraints. Commercial tools rely entirely on text prompting; if an artist needs to make a specific change to a single element, or requests highly unusual, otherworldly imagery common in sci-fi, cloud models tend to distort, mutate, or ignore the prompt entirely.

To gain the surgical precision required for film pipelines, we built bespoke, local generation pipelines. I designed these workflows using a combination of custom Python scripting and ComfyUI (an open-source, node-based programming interface for diffusion models). This allowed us to directly influence the model's math at specific steps of the generation process.

• Reference Frames & Video Stitching: Diffusion models generate images out of random visual noise. When generating a video, the model struggles to remember what it drew a few frames ago, leading to distracting "visual flickering." We investigated how injecting specific reference images into the middle of the generation process can anchor the AI. We also developed methods for blending and stitching multiple short video clips together, overcoming the inherent limitations models face when trying to generate long, continuous sequences of frames.

• ControlNets (Structural Guidance): A ControlNet is a secondary neural network layer that forces a video generation model to follow a strict structural template. Since I could export clean geometric data out of Unreal Engine 5, we used Depth ControlNets (which force the AI to respect the 3D distance of objects) and Pose ControlNets (DWPose) (which force the AI to follow specific skeletal hand and body movements) to maintain flawless physical consistency.

• Multi-Layered Video Editing & Inpainting: Instead of asking the AI to generate an entire video from scratch, we discovered success by breaking the video into layers. We would lock down a clean, static background layer, and then use inpainting—a technique where you mask out a specific region of an image and instruct the AI to re-draw only what is inside that mask—to edit characters or localized visual effects without altering the rest of the shot.

• Custom Character Training via LoRAs: Out-of-the-box video models have no idea what our proprietary characters or specific creatures look like. To teach them, we utilized LoRAs (Low-Rank Adaptations). Think of a LoRA as a lightweight patch file applied to a massive base model. By training these LoRAs on a small dataset of high-quality images curated from our internal 3D assets, we successfully taught the model to accurately generate our specific characters with consistent clothing and features.

Out-of-the-box video models do not know what proprietary characters or creature assets look like. To teach them, I conducted early experiments using LoRAs (Low-Rank Adaptations). A LoRA acts as a lightweight, targeted patch applied to a large base model, adjusting its output without requiring a full, expensive retraining of the entire network.

I built prototype pipelines using Unreal Engine 5, Python, and ComfyUI to curate small datasets of internal 3D assets rendered under varying lighting conditions and angles. Training custom LoRAs on these datasets enabled our models to reliably generate proprietary characters while preserving their exact visual features.

This research culminated in a prototype pipeline that seamlessly merges traditional Hollywood performance capture with generative AI.

[Raw Mocap Data] ➔ Exported via Maya ➔ Python Translation Script ➔ [DWPose Rig Frames] 

⬇

[Final Film Asset] ➔ Face Inpainting ➔ Custom LoRA Model ➔ Driven by Pose ControlNet

We took raw motion capture data (typically processed in Autodesk Maya) and used a custom Python script to translate that performance data into standardized DWPose frame sequences. These frames were fed directly into our local loop to drive our custom Pose ControlNets. By anchoring the first and last frames of the sequence with high-resolution reference images from Unreal Engine 5, the model generated a highly stylized, temporally stable creature performance. Finally, we utilized element recognition and localized inpainting to stitch the original actor’s facial performance back onto the generated character body.

This prototype directly influenced how ILM structured its local image-generation tools, establishing the framework for how local hardware should be configured for smooth, secure accessibility by VFX artists across the studio.