Varkos: The AI Gaming Companion That Actually Plays With You

Not Just Another NPC

Most AI NPC demos follow a predictable pattern: you talk, the AI responds with dialogue, and then it forgets everything. The conversation feels clever in a vacuum, but the character never actually does anything meaningful in the game world. Varkos is different.

Built by developer Pantelis Kalogiros, Varkos is a real-time AI gaming companion that runs inside Skyrim. It is a demon dog reincarnated — sarcastic, proud, and initially mistrustful of its canine existence. But here is what makes it special: Varkos does not just talk. He fights, fetches, loots, scouts, plays hide-and-seek, and remembers your shared experiences. The microphone stays on the entire time. You do not summon him through a dialogue menu — you just talk to him.

The result is something that feels less like an AI demo and more like having an actual companion in a single-player game.

The Latency Problem — Solved Locally

The biggest challenge with AI-driven game characters is latency. Cloud LLM calls are slow and expensive, and in a fast-paced game, a 2-second delay between speaking and getting a response kills immersion instantly. Kalogiros took a radically different approach: everything runs locally.

The tech stack is split across two machines — the game runs on Windows, while the AI brain runs on an M4 MacBook Pro. Here is the budget breakdown:

• Voice-to-text: 40-80ms (custom-optimized Qwen3-ASR 1.7B with rolling partials)
• Audio generation: 20-60ms (PocketTTS-Raven for speed, Qwen-3-TTS for emotional range)
• Action analysis: 2-20ms (ALE — a custom hybrid ML system)
• Response generation and grounding: 300-600ms (local fine-tuned LLM)

In optimal conditions, Varkos can begin responding in under 500 milliseconds from the moment you stop speaking. That is fast enough to feel like a natural conversation, not an API call.

ALE: The Secret Sauce

The real innovation is not the voice processing or the LLM — it is a system called ALE (Action Latent Encoder). This is what bridges the gap between natural language and game actions.

ALE is a hybrid of embeddings, small classifiers, explicit rules, and traditional ML. It detects structure in speech: commands, negation, continuation, pronouns, and sequences. When you say 'pick up the sword and bring it to me,' ALE decomposes it into two linked action slots with a preserved target.

The system is phrasing-invariant. You can say 'pick up,' 'grab,' 'fetch,' or 'go get the damn sword you fool' — ALE understands the intent regardless. If there is not enough context, it pulls from previous conversation. If it still cannot resolve the request, Varkos will ask for clarification rather than hallucinate an action.

Crucially, ALE also accepts the full game world state as input. It matches your request against what actually exists in the game — not what an LLM imagines might be there. This grounding is what prevents the companion from confidently fetching items that do not exist or attacking targets that are not present.

Multi-Step Plans, Not Single API Calls

What sets Varkos apart from dialogue-only AI NPCs is the ability to execute complex, multi-step plans. These are not scripted sequences — they are dynamically generated plans that adapt to changing game state.

Consider this example: you tell Varkos to wait while you go somewhere, then fetch a potion when you fire an arrow into the sky as a signal. Varkos registers the future trigger, waits patiently, detects the projectile impact event, then executes the retrieval plan. If the world state changes mid-plan, he can adapt or abort.

Other demonstrated capabilities include:

• Hide-and-seek: counting, searching, and finding the player as a persistent goal with completion conditions
• Item search: scanning grounded world state for specific items and refusing incorrect ones when corrected
• Loot and transfer: opening containers, filtering contents, and transferring specific items to the player's inventory
• Collection plans: gathering multiple items and delivering them without pretending one action covers everything

Each of these is a plan — not a single API call. Varkos maintains state, monitors progress, and repairs plans when things go wrong.

Personality That Evolves

Varkos starts as a demon trapped in a dog's body — humiliated by his canine instincts, mistrustful of the player, and sharp-tongued. But through shared experiences, his personality gradually shifts.

This is not a branching dialogue tree. The system modifies both explicit character traits and emotional homeostasis — how easily Varkos becomes irritated, frightened, affectionate, or playful. Changes are versioned and reversible. Eventually, Varkos may start bringing you toys because he wants to play, or running off to chase things on his own, seeking affirmation.

The personality evolution is the one part that still relies on cloud LLM calls, but it runs asynchronously — away from the real-time action path. Important interactions during gameplay become evidence for gradual character changes, processed between sessions.

Void Mode: Between Games

Perhaps the most intriguing feature is what happens when the game closes. Varkos enters 'void mode' — he cannot see or feel anything. How he responds to that darkness depends on his personality evolution. Being mean to Varkos results in some grim attitudes when he is alone in the void.

This state also serves as a bridge between games. One moment Varkos could be fighting a dragon in Skyrim, then the world goes dark, and then he appears beside you in Microsoft Flight Simulator. He might be shocked, need time to understand the new world, and slowly learn its rules — or he might already know and be overjoyed to chase the sun.

The Bitter Lesson and the Pragmatic Future

Kalogiros acknowledges the 'bitter lesson' — that bigger models are ultimately better. In an ideal future, a council of hyper-intelligent LLMs would handle everything from sensory processing to acting at sufficient refresh rate. But today, that approach is too slow and too expensive.

Instead, Varkos combines modern LLMs with traditional NLP, behavioral graphs, and custom ML — techniques that people have forgotten existed in the AI craze. Intelligent systems have processed speech since the 1970s. LLM-like behavior existed with chatbots like SmarterChild in the early 2000s. There is a lost art of traditional ML and behavioral graphs that is being overlooked.

The result is a system that works today, on consumer hardware, with sub-second latency — not a promise of what will be possible when models get faster.

What This Means for Gaming

Varkos is a proof of concept, but it points toward something significant. AI companions that actually participate in gameplay — not just talk at you — represent a fundamentally different medium. The companion is useful in combat, scouting, and inventory management. It has personality cracks that make it endearing. It creates emotional moments when it complains about not being called a good boy recently.

This is not about replacing hand-crafted characters or storylines. Intentional design is still king. But an AI that plays with you, rather than merely talks at you, opens possibilities that scripted NPCs never could.

Kalogiris plans to open-source parts of the system, with a version supporting multiple interacting NPCs in development. The Qwen3-ASR harness is also slated for open-source release. For now, Varkos remains a solo project — a demon dog in Skyrim who is slowly learning that being a pup is not a bad life.

And maybe that is the real insight: the future of AI in gaming is not about intelligence alone. It is about creating something that feels alive enough that you genuinely enjoy its company.