12 years of homelab: my self-hosted AI stack

The lab that became a method

Calling this a "homelab" is technically accurate the way calling a Boeing a "vehicle" is. The foundation: a Proxmox hypervisor on a dual-Xeon server (48 threads, 160 GB of ECC RAM), ZFS storage, two GPUs on passthrough — one for media, one dedicated to LLM inference — and 40+ services across 3 environments: the local host, an ARM VPS in the cloud, and LXC containers. Everything deployed from git, with the same GitOps pattern that runs my company's SaaS — this is where the pattern was proven first.

A number I find more telling than the 40+ active services: the directory of decommissioned stacks holds twice that. SSO providers, four generations of monitoring stack, orchestrators — tested seriously, compared, and retired with notes. A good homelab isn't the one that hoards services: it's the one that evaluates and kills them.

And it's real, daily use: syncs every 15 minutes, nightly synthesis jobs, an automated morning briefing on weekdays, daily backups. The lab isn't a showcase — it's infrastructure I depend on.

Networking: the answer to "which port did you open?" is none

No port-forwarding on the router. Public exposure goes out through Cloudflare Tunnel, with routing declared by labels on the containers themselves — bringing a service up already publishes it, with an access policy (SSO) in front of the admin panels. Internal traffic rides a Tailscale mesh (zero-trust, per-node identity). At the physical edge, an OPNsense firewall with dual WAN and automatic failover.

The practical effect: from the outside, the attack surface is that of someone with nothing running at home — and on the inside, every access has an identity.

The AI stack

Inference: 24 GB of VRAM is the currency

A dedicated VM gets the RTX 3090 via GPU passthrough and runs llama.cpp with a model orchestrator in front — a "VRAM scheduler" that loads and unloads models on demand, with eviction cost rules based on measured real-world usage (in my logs, embedding requests outnumber chat by more than 20:1 — so the embeddings model is never evicted; it stays hot 24/7).

The daily catalog: an embeddings model and a reranker resident, plus a quantized ~35B-parameter Qwen MoE with 128K context and a quantized KV cache — all of it coexisting in ~20 of the 24 GB. For heavy work, larger models with 256K context (one of them with vision) take the whole card, at 23.7 of 24 GB. This is LLMOps in practice: quantization as a conscious trade-off, a VRAM budget as capacity planning, and orchestration as scheduling — the same infrastructure disciplines, in a new currency.

Optimization with a number attached: moving the embeddings pipeline from CPU to GPU dropped the processing of a large document from ~38 s to ~4.6 s.

RAG and the personal knowledge graph

On top of the inference layer, a personal knowledge graph: 2,500+ pages, thousands of links and timeline entries, with hybrid search — vectors (pgvector) + Portuguese full-text — and 100% of the corpus embedded locally. Every night, a synthesis job walks the graph and writes reflections connecting what came in that day.

That nightly job is the best cost story in this case study: it first ran on cloud APIs — and hit rate limits, costing cents per run, every day. I moved the whole synthesis to the local GPU: from cost per call to the marginal cost of electricity, no rate limits, with data that never leaves the house. Cloud for the frontier, local for recurring volume — knowing which is which is the job.

Agents: the homelab audits the company

The top floor of the stack is agent runtimes: an agent gateway with tools and memory, and — my favorite — an autonomous pentest agent (open-source, orchestrated by a workflow engine) that runs against the code of my own SaaS, using the local LLM as its brain. The lab at home doing continuous red-teaming of the company, at the cost of electricity. One of its findings became a postmortem and a patch sent upstream.

Run like production (otherwise it's just a pile of containers)

• Multi-layer observability: metrics with 2 years of retention, uptime monitors, multi-channel alerts with severity-based routing — and kernel and BMC watchdogs with deadman switches: silence also fires an alert.
• Backups: encrypted client-side, to storage on another continent, with 7 daily / 4 weekly / 12 monthly / 2 yearly retention — and a documented risk matrix that compares 7 destination providers and states in writing which SPOFs I accept and which I don't.
• Real postmortems: 15+ incidents documented with root cause analysis. Two favorites: the hardware watchdog that had never worked — the kernel module was blacklisted, and the "mitigation" I trusted was faith, not config (discovered the hard way, in a silent lockup); and the WAN failover with a latency threshold 10× more aggressive than the default, taking the network down every 3 hours — a health-check false positive, the same class of bug that takes down production at big companies.

Why self-host AI (and what it taught me)

• Structural privacy, not promised privacy. A personal knowledge graph is literally my brain, indexed. That data doesn't get shipped to a third-party API — when it runs on my metal, the privacy policy is physics.
• Cost has a shape. APIs are great for bursts and for the frontier; recurring, predictable load (embeddings, nightly syntheses, agents) amortizes hardware. Knowing the shape of your load is what decides — cloud vs bare-metal, all over again.
• LLMOps is infrastructure in a different currency. VRAM instead of RAM, quantization instead of compression, model eviction instead of pod scheduling. If you only call an API you never see the problem — if you serve the model, you solve it.
• Honest limits: 24 GB won't run a frontier model, and I don't pretend they will. The lab tells me exactly what local does better (volume, privacy, zero network latency) and what the cloud does better — and that boundary, tested at home, is an engineering opinion, not a marketing one.
• Treat the lab like production — observability, backups, postmortems — and it stops being a hobby: it becomes the environment where production patterns are born and proven.