The Coe Lab
← Back to Blog

Running AI Locally: Lessons Learned

By April 6, 20265 min read
A technology notebook with visual symbols for AI, cybersecurity, infrastructure, and automation

What I learned from running AI models on local hardware.

Practical insights from running a self-hosted AI assistant, including hardware challenges, memory architecture, and when to choose local vs cloud AI. After months of running OpenClaw as my personal AI assistant, I've learned a lot about what works—and what doesn't—when hosting AI infrastructure locally. Here are the key lessons from the trenches.

Hardware Reality Check

The biggest misconception about local AI is that you can just run any model on any hardware. You can't. I learned this the hard way when I tried to run a 70B parameter model on my home server and watched it crash repeatedly. The server had plenty of RAM and a decent CPU, but without a GPU with sufficient VRAM, large language models simply cannot run efficiently.

Local inference requires serious hardware. For anything beyond small models (7B-13B parameters), you need dedicated GPU resources. Here's what I've found works in practice:

  • 7B-13B parameter models: Run acceptably on 8-16GB VRAM GPUs (RTX 3060, RTX 4060)
  • 13B-33B parameter models: Need 24GB+ VRAM (RTX 3090, RTX 4090) or quantized models
  • 70B+ parameter models: Require multi-GPU setups or cloud inference—most home servers can't handle them
  • 400B+ parameter models (like Qwen 3.5 at 397B): Absolutely require cloud GPUs; no consumer hardware can run these efficiently

    • Memory bandwidth matters as much as raw GPU power. Apple Silicon (M1/M2/M3/M4) machines have unified memory architecture that makes them surprisingly good for local inference. Ollama's MLX engine, optimized for Apple Silicon, has delivered impressive performance gains throughout 2026, with multi-token prediction making models up to 90% faster on the Aider polyglot benchmark. If you're serious about local AI and have a Mac with 32GB+ unified memory, you're in better shape than most PC owners.

    My solution for the hardware problem? Use cloud models via Ollama Cloud. The local Ollama server acts as a relay, but the actual inference happens on cloud GPUs. This gives me access to powerful models like Qwen 3.5 (397B parameters) and Mistral Large 3 (675B parameters) without needing a $10,000 GPU setup. The local server handles routing, caching, and smaller tasks while cloud GPUs handle the heavy lifting.

    Memory Architecture Matters

    An AI assistant without memory is just a chatbot. This was one of the first lessons I learned. Out of the box, most LLMs have a context window—anywhere from 8K to 200K tokens—but once the conversation exceeds that window, earlier context is lost. For a personal assistant that needs to remember your preferences, projects, and ongoing conversations, this isn't good enough.

    The key insight: memory should be external to the model. I use Neo4j as a graph database to store entities, relationships, events, and notes. When the AI needs context, it queries the graph database rather than relying on its training data or conversation history. This approach has several advantages:

  • Persistent memory that survives session restarts—the AI remembers conversations from days or weeks ago
  • Structured relationships—the AI knows that Jennifer is my wife, that project X depends on service Y, that book Z was recommended by person A
  • Cross-session context—multiple AI sessions can share the same memory graph, maintaining continuity
  • Selective retrieval—only relevant memories are loaded into context, keeping token usage efficient
  • Human-readable—the graph database can be queried, visualized, and edited directly when needed

    • The architecture works like this: when a conversation happens, the AI extracts entities and relationships and writes them to Neo4j. When a new conversation starts, the AI queries Neo4j for relevant context based on the conversation topic. This allows the AI to have contextual conversations. When I ask "How's the media server doing?", it can query Neo4j and tell me about Radarr, Sonarr, and any stalled downloads—not just give a generic response.

    Memory management is an ongoing challenge. Over time, the graph accumulates stale information, outdated relationships, and irrelevant entities. I've found it necessary to periodically review and clean the graph database, similar to how a human might journal and reflect to organize their thoughts. Tools like Notion sync (which mirrors the Neo4j data to Notion databases) make this easier by providing a human-friendly view of what the AI knows.

    When to Go Local vs Cloud

    After experimenting with both, here's my framework for deciding where inference should happen:

  • Local for privacy: When handling sensitive personal data, financial information, or anything you don't want sent to a third-party API
  • Local for speed: For simple, repetitive tasks where the latency of a cloud API call is noticeable—heartbeat checks, simple classifications, format conversions
  • Local for cost: When you're making thousands of API calls per day for routine tasks, local inference on a smaller model can be more cost-effective
  • Cloud for capability: When you need the best possible output quality—complex reasoning, creative writing, code generation on large codebases
  • Cloud for scale: When processing large documents, generating long outputs, or handling multiple concurrent requests
  • Cloud for access: When you need models too large to run locally (70B+ parameters)

    • The Hybrid Approach

    My current setup uses both local and cloud models, and the system automatically routes based on the task:

  • Simple tasks (heartbeats, status checks, quick questions) use a smaller local model via Ollama
  • Complex reasoning, coding, and creative writing use cloud models via Ollama Cloud
  • Memory operations (entity extraction, graph queries) use a medium-sized model for accuracy
  • Voice and vision tasks use specialized models appropriate to the modality

    • This balances cost, performance, and capability. Simple tasks stay local; complex work goes to the cloud. The routing logic lives in the assistant's configuration, so I can adjust which model handles which task without changing the overall architecture. Ollama's transparent per-token pricing (introduced in August 2026) makes the cost analysis straightforward—you know exactly what each interaction costs.

    Automation is Where It Shines

    The real power of a self-hosted AI isn't chat—it's automation. Once you have a reliable AI infrastructure with memory and tool access, you can automate tasks that would otherwise require constant human attention. OpenClaw runs scheduled jobs that handle real-world tasks autonomously:

  • Media management: The AI monitors torrent health, identifies stalled downloads, removes dead torrents, and triggers re-searches automatically
  • Website analytics: Daily Cloudflare traffic reports are generated and posted to Discord every morning at 8 AM
  • Calendar and email monitoring: The AI checks for upcoming events and urgent emails, proactively notifying me when action is needed
  • System health checks: The AI monitors server status, disk usage, and service availability, alerting me to issues before they become critical
  • Memory maintenance: The AI periodically reviews its own memory graph, updates stale entries, and archives outdated information

    • These aren't scripted commands—they're AI-driven decisions. The media manager, for example, analyzes torrent health, decides what's dead, removes it, and triggers re-searches. All autonomously. The AI evaluates conditions, makes judgment calls, and takes action. This is the real promise of local AI: not a chatbot you talk to, but a digital employee that works alongside you.

    The Ecosystem in 2026

    The local AI ecosystem has evolved dramatically. Ollama, which serves 8.9 million developers as of 2026, has become the standard for running local models. The MLX engine for Apple Silicon delivers the highest performance yet, with improved memory efficiency and response quality. NVIDIA's Nemotron 3.5 Lightning (30B parameters, 3B active) is specifically designed for agents that stay running—gathering context, calling tools, and working through multi-step tasks on your own hardware.

    Meta's Muse Glimmer (30B multimodal, Apache 2.0) brought local coding agents to a new level. OpenJarvis v1.0, released in May 2026, provides an open-source framework for building personal AI agents with local Ollama support built in. The tools are mature, the models are capable, and the infrastructure is accessible. There's never been a better time to run your own AI.

    Lessons Learned

    If I had to distill everything I've learned into a few key takeaways:

  • Start with cloud models and add local inference later. Don't invest in expensive hardware before you know what you actually need.
  • Memory is more important than model size. A small model with good context beats a large model with no context.
  • Automate early. The sooner you start automating routine tasks, the sooner you'll see the real ROI of self-hosted AI.
  • Use a hybrid approach. Don't dogmatically insist on local-only or cloud-only. Use the right tool for each task.
  • Monitor your costs. Cloud API calls add up fast. Track usage and adjust routing to keep costs sustainable.
  • Invest in your memory architecture. The quality of your AI's memory determines the quality of its output. Spend time getting this right.
  • Security matters even locally. Just because your AI runs on your own hardware doesn't mean it's automatically secure. Implement proper access controls, keep credentials safe, and audit what your agents can access.

    • What's Next

    The roadmap includes expanding automation (email triage, calendar management, proactive research), improving memory (better entity extraction from conversations, automatic relationship inference, timeline awareness), and exploring multi-agent workflows (specialized AI assistants for different domains—one for coding, one for research, one for system administration).

    I'm also watching the open-source model space closely. Models like NVIDIA's Nemotron 3 Ultra, designed for high-throughput reasoning and long-running agent workflows, are making local agents increasingly viable for complex tasks. The gap between cloud and local capability is narrowing, and within a year or two, many tasks that currently require cloud inference will be handleable on consumer hardware.

    If you're building your own AI infrastructure, I'm happy to share what I've learned. Reach out via the contact page or connect on Discord. The local AI community is growing fast, and there's plenty of knowledge to share. The future of AI isn't just in the cloud—it's on your own hardware, under your own control, working for you.

    Related Posts

    When Hackers Defrost the Food Supply: The Hidden Risk in Smart Refrigeration

    A cyberattack on European supermarket refrigeration shows how connected operational technology can turn a routine IT breach into food loss and physical disruption.

    Oct 13, 2026• 9 min read

    The Zod 4.6.6 Scare: A Practical npm Supply-Chain Response

    A suspicious Zod commit triggered malware fears without proof of a poisoned npm release. Here is how teams should investigate, contain, and improve dependency security.

    Oct 12, 2026• 10 min read

    Talorys Turns Cloudflare’s Free Tier Into a Personal AI Agent

    Talorys runs a private personal AI assistant on Cloudflare Workers, Durable Objects, and Workers AI. Here is what its architecture gets right—and its limits.

    Oct 11, 2026• 9 min read