12 years of homelab: my self-hosted AI stack

The lab that became a method

Calling this a "homelab" is technically accurate the way calling a Boeing a "vehicle" is. The foundation: a Proxmox hypervisor on a dual-Xeon server (48 threads, 160 GB of ECC RAM), ZFS storage, two GPUs on passthrough (one for media, one dedicated to LLM inference) and 40+ services across 3 environments: the local host, an ARM VPS in the cloud, and LXC containers. Everything deployed from git, with the same GitOps pattern that runs my company's SaaS. This is where the pattern was proven first.

A number I find more telling than the 40+ active services: the directory of decommissioned stacks holds twice that. SSO providers, four generations of monitoring stack, orchestrators: tested seriously, compared, and retired with notes. A good homelab isn't the one that hoards services: it's the one that evaluates and kills them.

And it's real, daily use: syncs every 15 minutes, nightly synthesis jobs, an automated morning briefing on weekdays, daily backups. The lab isn't a showcase. It's infrastructure I depend on.

Networking: the answer to "which port did you open?" is none

No port-forwarding on the router. Public exposure goes out through Cloudflare Tunnel, with routing declared by labels on the containers themselves: bringing a service up already publishes it, with an access policy (SSO) in front of the admin panels. Internal traffic rides a Tailscale mesh (zero-trust, per-node identity). At the physical edge, an OPNsense firewall with dual WAN and automatic failover.

The practical effect: from the outside, the attack surface is that of someone with nothing running at home. On the inside, every access has an identity.

The AI stack

Inference: 24 GB of VRAM is the currency

A dedicated VM gets the RTX 3090 via GPU passthrough and runs llama.cpp with a model orchestrator in front, a "VRAM scheduler" that loads and unloads models on demand, with eviction cost rules based on measured real-world usage (in my logs, embedding requests outnumber chat by more than 20:1, so the embeddings model is never evicted; it stays hot 24/7).

The daily catalog: an embeddings model and a reranker resident, plus a quantized ~35B-parameter Qwen MoE with 128K context and a quantized KV cache, all of it coexisting in ~20 of the 24 GB. For heavy work, larger models with 256K context (one of them with vision) take the whole card, at 23.7 of 24 GB. This is LLMOps in practice: quantization as a conscious trade-off, a VRAM budget as capacity planning, and orchestration as scheduling: the same infrastructure disciplines, in a new currency.

Optimization with a number attached: moving the embeddings pipeline from CPU to GPU dropped the processing of a large document from ~38 s to ~4.6 s.

RAG and the personal knowledge graph

On top of the inference layer, a personal knowledge graph: 2,500+ pages, thousands of links and timeline entries, with hybrid search, vectors (pgvector) + Portuguese full-text, and 100% of the corpus embedded locally. Every night, a synthesis job walks the graph and writes reflections connecting what came in that day.

That nightly job is the best cost story in this case study: it first ran on cloud APIs, and hit rate limits, costing cents per run, every day. I moved the whole synthesis to the local GPU: from cost per call to the marginal cost of electricity, no rate limits, with data that never leaves the house. Cloud for the frontier, local for recurring volume. Knowing which is which is the job.

Agents: the homelab audits the company

The top floor of the stack is agent runtimes: an agent gateway with tools and memory, and, my favorite, an autonomous pentest agent (open-source, orchestrated by a workflow engine) that runs against the code of my own SaaS, using the local LLM as its brain. The lab at home doing continuous red-teaming of the company, at the cost of electricity. One of its findings became a postmortem and a patch sent upstream.

Run like production (otherwise it's just a pile of containers)

• Multi-layer observability: metrics with 2 years of retention, uptime monitors, multi-channel alerts with severity-based routing, and kernel and BMC watchdogs with deadman switches: silence also fires an alert.
• Backups: encrypted client-side, to storage on another continent, with 7 daily / 4 weekly / 12 monthly / 2 yearly retention, and a documented risk matrix that compares 7 destination providers and states in writing which SPOFs I accept and which I don't.
• Real postmortems: 15+ incidents documented with root cause analysis. Two favorites: the hardware watchdog that had never worked: the kernel module was blacklisted, and the "mitigation" I trusted was faith, not config (discovered the hard way, in a silent lockup); and the WAN failover with a latency threshold 10× more aggressive than the default, taking the network down every 3 hours: a health-check false positive, the same class of bug that takes down production at big companies.

Why self-host AI (and what it taught me)

• Structural privacy, not promised privacy. A personal knowledge graph is literally my brain, indexed. That data doesn't get shipped to a third-party API. When it runs on my metal, the privacy policy is physics.
• Cost has a shape. APIs are great for bursts and for the frontier; recurring, predictable load (embeddings, nightly syntheses, agents) amortizes hardware. Knowing the shape of your load is what decides, cloud vs bare-metal, all over again.
• LLMOps is infrastructure in a different currency. VRAM instead of RAM, quantization instead of compression, model eviction instead of pod scheduling. If you only call an API you never see the problem; if you serve the model, you solve it.
• Honest limits: 24 GB won't run a frontier model, and I don't pretend they will. The lab tells me exactly what local does better (volume, privacy, zero network latency) and what the cloud does better. That boundary, tested at home, is an engineering opinion, not a marketing one.
• Treat the lab like production (observability, backups, postmortems) and it stops being a hobby: it becomes the environment where production patterns are born and proven.