A whole AI stack on one used RTX 3090 (and what broke along the way)

In July I wrote that this wasn't possible

In my homelab case study, published in July, I wrote that "24 GB won't run a frontier model, and I don't pretend they will." For today's frontier model, that still holds. For the one from six months ago, it no longer does.

On August 14, Alibaba released Qwen3.8-27B, an open model with 27 billion parameters. On its official model card it beats Claude Opus 4.6 Max, which was at the frontier in March, on 16 of 24 benchmarks. It wins at agentic coding, instruction following and computer use. It loses at knowledge reasoning (GPQA Diamond, Humanity's Last Exam). The numbers are Alibaba's own and had no independent reproduction at launch, so read them as a ceiling.

The problem is size. In FP16 the model is 53.8 GB, more than twice the card. At 4 bits it fits, but it takes everything: the ceiling I measured was 192K context with all 24 GB in use and no room for the search models.

The model: weights that are −1, 0 or +1

PrismML's Bonsai 2 is the same Qwen3.8-27B compressed to ternary weights, with one FP16 scale per group of weights. The file is about 7 GB. On my card it runs 262K context in 13.0 GiB, against the 24 GiB the 4-bit version needed for 192K.

The next question is how much you lose. I ran a live A/B with 9 tasks checked by execution or against a known answer: code (interval merging, a buggy binary search, an LRU cache), trick problems, facts spread across a long context, and dense vision with arithmetic. Both got all 9 right, with the same answers. The only difference was thinking time: the 4-bit version thought about twice as long (22 vs 11 s on the LRU cache). PrismML says the loss shows up in long agent loops, and I haven't run that test yet. Until I do, the 4-bit version stays in the catalog for heavy vision work.

A detail that costs you if you miss it: no version of Bonsai 2 runs on upstream llama.cpp. It needs an activation transform (Hadamard) that hasn't landed in the main project yet and only exists in PrismML's fork. Two of the published bands fail with an error on upstream llama.cpp. The third one loads without a warning and outputs garbage. The machine now runs three llama.cpp builds side by side: one for embeddings and reranking, one for Qwen3.8, and the fork for Bonsai.

What runs on the card, and when

In front of the models sits llama-swap, a proxy that loads and unloads models on demand. I declare which sets can run together and an eviction cost per model. When a request arrives for a model that doesn't fit, it evicts the cheapest ones first.

• Daily: embeddings + reranker + Bonsai 2. That's 22.4 of 24.5 GB with all three loaded and answering. Bonsai alone takes 20.3 GB, because in production its cache runs at q8_0 and it carries the graft described further down.
• Image: embeddings + reranker + ComfyUI. That's 16.9 GB for a 1024×1024 image.
• Heavy: Qwen3.8-27B at 4 bits, alone on the card, for high-quality vision.

The embedder and reranker have the highest eviction cost: 40, against 10 for Bonsai and 20 for ComfyUI. In my logs, embedding requests outnumber chat 22 to 1, so those two only leave when the heavy model asks for the whole card.

When I ask for an image, llama-swap unloads Bonsai and starts ComfyUI. On the next question, it unloads ComfyUI and brings Bonsai back. The daily set leaves 2.1 GB free. Any new resident model, or more context for Bonsai, means measuring again.

Images inside the chat

The generator is Qwen-Image-2.1 in INT8 on ComfyUI: 7.26 GB of diffusion model, 9.35 GB of text encoder and 0.68 GB of VAE. Measured with the embedder and reranker loaded, a 1024×1024 image at 40 steps takes 49.96 s, and a 2048×1152 one at 50 steps takes 95.39 s. A cold start of ComfyUI takes 11 s.

The chat model doesn't pass the user's sentence along. It loads a prompt skill and writes a description of about twenty sentences, always in English, in the format Qwen-Image was trained on. Then it calls generate_image, and the image shows up in the conversation. The same tool has edit_image, to change an image sent in the chat.

What broke

HTTP 409 on every image

Open WebUI's built-in image generator opens a websocket to ComfyUI before it sends the job. llama-swap doesn't start a model on a websocket request. And ComfyUI was always off the card, because the chat model had just taken the card to write the prompt. The result was a 409 on every image. The fix was a tool of my own, qwen_image, which first sends a plain GET to ComfyUI's root. That request starts the model and waits until it's ready. The built-in image tools are turned off per model, so the chat model sees exactly one generate_image and one edit_image.

The database overrides the compose file

Web search in Open WebUI came back empty. The compose file said to use SearXNG, but after the first boot Open WebUI keeps its settings in its own database, and the database had a different search engine saved. The database wins over the environment variable. I fixed it in the database with the container stopped, because while it runs, Open WebUI writes its in-memory settings back on shutdown.

The wrong CUDA library

The fork's installer picked the build for CUDA 12.8, which looks for libcudart.so.12, on a machine with CUDA 13.2. It died with "cannot open shared object". The CUDA 13.3 build, with the library path set, ran without recompiling.

A graft: 28% faster with a part from another model

Qwen3.8 ships with a multi-token prediction (MTP) head, used for speculative decoding: a cheap part of the model proposes the next tokens and the full model checks them all at once. Bonsai ships without it. I took the head from the 4-bit Qwen3.8 that was already on disk, 16 tensors and 1,066,172,160 bytes, and grafted it onto Bonsai with the tools from the bonsai2-small-gpu repository. The rest of the model didn't change: removing the graft reproduces the original file's sha256, byte for byte.

• Speed: median from 60.1 to 77.2 tokens/s (1.28x). On code, 85.6.
• Cost: prompt processing gets 6% to 15% slower, and the daily set's free VRAM dropped from 5.1 to 2.1 GB.
• Not lossless: 5 of 6 long generations diverge at some point, because the fork's kernels don't give the same numbers in a batch as they do one token at a time. On a 70-question battery with known answers, the score was 56/70 with and without the graft, and no single item changed. All 14 misses are 3-digit multiplication, a limit of the ternary model, not of the graft.

What this opens up

• Privacy: my documents are indexed and searched on the card itself. Nothing leaves the house.
• Cost: the nightly job that works through my knowledge graph used to run on a cloud API, paid per call and hit rate limits every day. Now it costs electricity.
• Access: a model at the level of the frontier from six months ago, with vision, tools, search and images, fits on a used graphics card.
• Time: the base already existed (llama-swap, embeddings on the GPU since June, Open WebUI). Building this on top of it took 2 days.

References