LLM-Based Character in Unity

Traditional game characters could only speak from a set of hardcoded lines. Nowadays, with Large Language Models (LLMs), we can give variety to our NPCs. In this article, I build a character in Unity with his own unique personality. He answers your questions, using a small language model that runs entirely on my laptop. It needs no internet connection or API key, and messages cost nothing.

How It Works

The lab uses LLMUnity, a package that runs language models inside Unity. Under the hood, it is built on llama.cpp, which runs models stored as .gguf files on an ordinary CPU.

LLMUnity splits the work between two components:

• LLM: the “server”. It loads the model file and generates text.
• LLMAgent: the “client”, which is the character itself. It holds a system prompt that defines its personality, along with the chat history. Several agents can share one LLM, each with a personality of its own.

On top of these, I wrote a small script that connects the chat window to the character. When you send a message, the reply streams back one token at a time:

flowchart LR
    Input[Input Field] --> Controller[NPCController]
    Controller --> Agent[LLMAgent]
    Agent --> LLM[LLM]
    LLM -.->|token by token| Controller
    Controller --> Chat[Chat Window]

Setup

I started from a new Universal 3D project, as in Setting Up Unity. To install LLMUnity, open the Package Manager, click + and add the package from this Git URL:

https://github.com/undreamai/LLMUnity.git

The chat window uses TextMeshPro for its text. The first time you add a TextMeshPro element, Unity asks to import TMP Essentials, which you should accept.

The Brain

The Model

I created an empty GameObject called LLMGameObject, added the LLM component and downloaded Qwen 3.5 0.8B from its built-in model list. With 0.8 billion parameters, it’s tiny by today’s standards. The file is also quantized (Q4_K_M): each weight is stored in about 5 bits instead of 16, which makes the model smaller and faster at a small cost in quality. I kept the default settings: all CPU threads, no GPU and a context window of 8,192 tokens.

Choosing a model is a trade-off between quality and speed. A bigger model gives better answers, but you wait longer for each one, and in a conversation every second counts. Reasoning models add even more delay, since they “think out loud” in a <think> block before answering. In LLMUnity 3, this is the Reasoning option on the LLM component. It’s off by default and only appears once you enable Advanced Options.

The Character

Next, I created a second GameObject called AICharacter and added an LLMAgent component. It finds the LLM in the scene and connects to it on its own. The interesting part is the system prompt. Instead of a generic assistant, I made the character a villager:

You will roleplay as a villager in the town of Nilbog. You live with your
one wife and one daughter in a small home. You are annoyed by the Goblins
who sometimes enter your town to pester you and your fellow villagers.
You are a farmer by profession and you are very proud of it. You are always
trying to sell your vegetables to anyone who will listen. You know nothing
outside of your town and when asked about it you reply with "Huh? What's
that?". You always reply in short sentences / phrases and speak in a very
natural casual way as a villager in a fantasy town would. Never use emojis
or non-ascii characters.

Each part of the prompt has a job:

1. A life: a family, a job and a nuisance give the villager something to talk about, beyond “How can I help you?“.
2. A goal: selling vegetables gives him a reason to steer the conversation, like a merchant in a game.
3. A closed world: “Huh? What’s that?” covers everything outside the town. A model this small knows little and easily makes things up, so confusion in character beats a confident wrong answer.
4. Short replies: these keep each turn quick, so it feels more like talking than reading.
5. No emojis: the font has no emoji glyphs, so they would show up as empty boxes.

The Face

The chat window is a Canvas with three parts: a scroll view for the conversation, an input field and a send button. Here is the scene, without the default camera, light and volume, or the child objects Unity creates for the input field, button and scrollbar:

SampleScene
├── LLMGameObject        (LLM)
├── AICharacter          (LLMAgent, NPCController)
├── EventSystem
└── Canvas
    ├── ChatScrollView
    │   ├── Viewport
    │   │   └── Content
    │   │       └── ChatMessage
    │   └── Scrollbar Vertical
    ├── UserMessageInputField
    └── SendButton

Unity adds the EventSystem when you create the first UI element. It passes clicks and key presses to the UI, so without it, the chat window ignores all input.

A plain text box grows with the conversation until it covers the whole screen, so the chat lives inside a Scroll View instead. Its Viewport works as a mask that hides anything outside the window. For the text to scroll, Content needs a Vertical Layout Group with Control Child Size → Height ticked, which sizes ChatMessage to fit its text. Content also needs a Content Size Fitter with Vertical Fit set to Preferred Size, so it grows along with the message. The text then grows downward as a reply comes in. I also turned off horizontal scrolling and set the vertical scrollbar to hide until it’s needed.

I anchored the elements to the edges of the screen: the scroll view stretches with the window, the input field runs along the bottom and the send button sits in the bottom-right corner. With the Canvas Scaler set to Scale With Screen Size, the layout fits any window.

For the text, I used Inter on a translucent panel. The placeholder in the input field says Say something… in italics, which invites small talk rather than search queries. Rich text tags color the speaker labels: blue for the player and green for the NPC. This allows us to better differentiate between speakers.

The Glue

The last piece is NPCController.cs, attached to AICharacter.

using UnityEngine;
using UnityEngine.UI;
using TMPro;
using LLMUnity;

public class NPCController : MonoBehaviour
{
    public LLMAgent llmAgent;
    public TMP_InputField inputField;
    public TextMeshProUGUI chatLogText;
    public Button sendButton;

    private string fullLog = "";

    void Start()
    {
        sendButton.onClick.AddListener(OnSendClicked);
        inputField.onSubmit.AddListener(delegate { OnSendClicked(); });
    }

    void OnSendClicked()
    {
        string userMessage = inputField.text.Trim();
        if (string.IsNullOrEmpty(userMessage)) return;

        fullLog += $"\n<color=#4DA6FF><b>You:</b></color> {userMessage}\n";
        fullLog += "<color=#26D999><b>NPC:</b></color> ";
        chatLogText.text = fullLog;

        inputField.text = "";
        inputField.interactable = false;
        sendButton.interactable = false;

        _ = llmAgent.Chat(userMessage, HandleReply, OnReplyComplete);
    }

    void HandleReply(string replySoFar)
    {
        chatLogText.text = fullLog + replySoFar;
    }

    void OnReplyComplete()
    {
        fullLog = chatLogText.text + "\n";
        inputField.interactable = true;
        sendButton.interactable = true;
        inputField.ActivateInputField();
    }
}

• Public fields show up as slots in the Inspector. I dragged the agent, the input field, the chat text and the button into them.
• Start connects both the Send button and the Enter key to the same method.
• OnSendClicked adds your message and the NPC: label to the log, clears and locks the input, and calls llmAgent.Chat. The call is asynchronous, so the game keeps running while the model works.
• HandleReply runs every time a new token arrives, with the reply so far. Redrawing the text each time makes the villager appear to type.
• OnReplyComplete saves the full reply to the log, unlocks the input and puts the cursor back in it, so you can answer right away.

Talking to the Villager

With everything connected, I pressed Play and said hello:

The prompt never gives the villager a name, so the model made one up: Silas. He stayed a farmer the whole time and brought up his vegetables after just two questions. One reply also shows a habit of small models: “I sell vegetables to anyone who will listen” is copied almost word for word from the system prompt.

Through an HCI Lens

From an HCI perspective, the following should be noted:

1. Streaming hides the wait. On a CPU, a full reply can take a while. Since every token appears as soon as it’s generated, the first words show up long before the reply is finished. Waiting feels shorter when you can see progress.
2. Clear turns. While the villager answers, the input field and the button are disabled and grayed out. As with a walkie-talkie, only one side talks at a time, and the grayed-out controls show whose turn it is. The downside is that you can’t interrupt a long reply.
3. Personality matters. People talk to a villager differently than to an AI assistant. The persona also hides the model’s limits: a small local model often repeats itself or makes silly mistakes, which can pass as the quirks of a particular NPC.
4. Local means private. The model runs on the device, so the conversation never leaves it and the game works offline. The price is a smaller, less capable model that competes with the game for memory and CPU.

Conclusion

By just adding a few components and some lines of code, we can implement a talking character in Unity. Using local models allows us to run everything offline, albeit at the cost of quality.