Local AI is not Opus

Alex Ellis runs a bootstrapped software business (OpenFaaS, SlicerVM, Actuated, Inlets — all Golang, all self-hosted infra) and has been using AI coding tools since tab-completion. This post is his ledger, not a hot take: what a local model actually did for the business, what he still won't trust it to do, and the receipts behind both. His thesis is narrow and unfashionable — the popular claim that a local Qwen 27B is "near-Opus level" is wrong, but a well-tuned local model still delivers real, bounded value. It buys privacy, fixed cost, and protection against vendor risk. It does not replace a frontier subscription. See alex-ellis for the author.

The benchmark trap

The "near-Opus" claim leans on one number: Qwen 3.6 27B scores 77.2 on SWE-Bench Verified against Claude Opus 4.8's 88.6, so people announce that local is "only 12% behind SOTA" and a six-year-old GPU can retire a $200/mo subscription. Two problems. SWE-Bench is built from Python issues, and Python is mostly single-threaded and synchronous; Ellis's team writes distributed Go where channels, contexts, and structs span a large execution domain, which the benchmark never exercises. And because benchmarks are public, small models get tuned to score well on them — benchmaxxing inflates exactly the number the claim rests on. The frontier models are estimated at 0.5–2T parameters; a 27B dense model is on a different level regardless of one leaderboard.

"Local isn't about cost" is a position of privilege

The common line that local models "aren't about cost" assumes you can always reach for a subsidized $200/mo plan. Ellis points at where those subsidies bend: GitHub Copilot launched at 1500 requests for $39/mo, then quietly moved everyone to token-based pricing to loud backlash. Uber capped AI spend at $1500/mo per developer per tool — against a $330k median salary, two maxed-out tools is roughly 12% of annual comp. The subsidy is real today and not guaranteed tomorrow. Related economics: ai-subsidy-economics, no-silver-bullet-llms.

The real driver: sovereignty and vendor risk

What actually pulls a privacy-focused business toward local is data control and vendor risk. Ellis notes that Fable 5 was pulled overnight for users outside the US — local models are the answer to "what if the frontier labs do X?" And customer data can't legally go through cloud plans at all: running a customer's telemetry dump or diagnostic snapshot through ChatGPT Pro or Claude Max, even configured for 30-day retention, likely violates the contracts with those customers. So they wrote diag (a CLI that snapshots an OpenFaaS install), have operators email the dump, and run it through an airgapped local model in an ephemeral SlicerVM. One renewal review of a telemetry database found a customer had been under-reporting licenses and underpaying by 4–5x for over a year — revenue recovery that paid for the hardware, and that could never have gone through a cloud plan.

The hardware bet

Ellis dropped about $12,000 on an RTX 6000 Pro Blackwell with 96GB of VRAM (the price rose to ~$15,400 a couple of months later). It paid for itself — but through the revenue-recovery and support use cases, not by replacing Claude. Earlier he ran dual 3090s, which were a constant source of headaches: a 27B model doesn't fit at full fidelity on one 3090, forcing tradeoffs across weight quantization, context length, and KV-cache compression (rule of thumb: bad things start at Q4_0 on the keys; his most aggressive setting was Q8_0 keys, Q4_0 values). Even with NVLink and tensor parallelism, vLLM ran 3 tok/s slower than llama.cpp at generation — vLLM wins at production-scale concurrent serving, but for a single-user prosumer rig, startup time and latency matter more than aggregate throughput. See llm-quantization, local-llm-16gb-vram-tests.

Where it fails: long-horizon unsupervised work

The blade metaphor: heat-treating steel, if you overshoot the temper colour by one shade you start over. A local model runs so hot it shoots past the goal and loops, and nothing fixes it except killing the harness and hoping cleared context gives a different result. Concrete failures — asked what commands to add to faas-cli, it gave reasonable suggestions then repeated them for half an hour burning 600W; asked to write a Python reverse proxy, it got the indentation wrong, tried to fix it, corrupted the file, and went progressively off the rails; it read 27.3K as 273,000; it inferred a customer would churn from a low function count, ignoring that they ran those functions many times a day. The lesson: keep local models on analysis, not interpretation, and never on long-horizon unsupervised tasks. What's "creepy" about Claude and Codex is how long they work fully unattended while making real progress — that's the gap, not the leaderboard. This is the same shape as constraint-decay and the median-problem argument in no-silver-bullet-llms.

The working setup

Two independent llama.cpp instances on the RTX 6000, each keeping full 262144 context: the base Qwen 3.6 27B and Qwopus (a fine-tune layering chain-of-thought traces onto Qwen for better reasoning and coding). MTP speculative decoding gets ~93% acceptance, lifting sustained throughput from a stable 67 tok/s to 130–200 tok/s — it feels faster than a cloud model. Because two agents on one instance invalidate each other's cached prefix (the full prompt reprocesses every time), sharing the box turns into an operations problem: identity, access control, metering, quotas, routing, power monitoring. Ellis wrote "Toilgate," a vibe-coded opencode provider for access and metering, and put two Shelly Plus Plugs on the wall to watch power (the RTX 6000 pulls 600W under inference; the two 3090s draw ~750W combined and are loud). See speculative-decoding.

The trap once you can meter is comparing your per-million-token cost to GPT-5.5 API pricing — the wrong comparison, because the value was never the token price.

Where local actually earns its keep

• Reading and explaining codebases, even ones it can't write — Ellis calls this a superpower.
• Well-bounded maintenance driven by a detailed AGENTS.md: on alexellis/arkade, the local model added new CLIs faster than human contributors and tested its own work.
• Airgapped customer support and analysis where data can't leave the building.
• Respect the model card: temperature, context, and quantization all matter, and very low quantizations bite.
• Normalise running the same task through both a local and a cloud model.

Ellis skips 70B models (most are generations behind) and notes the 35-A3B Qwen variant is popular mainly because 3B active parameters look fast on a MacBook. Bigger models (GLM 5.2, Kimi 2.7, Minimax M3, Deepseek V4 Flash) need 4–6 of these cards even to load a quantized copy. His blunt summary: today's 27B dense models are not cut out to write Go all day; the unglamorous Grok Coder Fast 1 was cheaper and faster and served the team well for months before deprecation.

Cross-references

• titit-local-ai — Ted Neward's parallel argument for local AI in small businesses
• lucumr-local-models — Armin Ronacher on why the local UX gap is product polish, not model quality
• local-llm-16gb-vram-tests — the exact quantization / KV-cache operator knobs Ellis describes, worked out on consumer hardware
• speculative-decoding — the MTP draft mechanism behind his 130–200 tok/s
• llm-quantization — weight and KV-cache quantization tradeoffs
• no-silver-bullet-llms — models degrade away from the documented median; same conclusion from different evidence
• constraint-decay — the measured version of "fails at bounded, convention-heavy work"
• alex-ellis — the author