# Local AI is not Opus

Alex Ellis runs a bootstrapped software business (OpenFaaS, SlicerVM, Actuated, Inlets — all Golang, all self-hosted infra) and has been using AI coding tools since tab-completion. This post is his ledger, not a hot take: what a local model actually did for the business, what he still won't trust it to do, and the receipts behind both. His thesis is narrow and unfashionable — the popular claim that a local Qwen 27B is "near-Opus level" is wrong, but a well-tuned local model still delivers real, bounded value. It buys privacy, fixed cost, and protection against vendor risk. It does not replace a frontier subscription. See [[alex-ellis]] for the author.

## The benchmark trap

The "near-Opus" claim leans on one number: Qwen 3.6 27B scores 77.2 on SWE-Bench Verified against Claude Opus 4.8's 88.6, so people announce that local is "only 12% behind SOTA" and a six-year-old GPU can retire a $200/mo subscription. Two problems. SWE-Bench is built from Python issues, and Python is mostly single-threaded and synchronous; Ellis's team writes distributed Go where channels, contexts, and structs span a large execution domain, which the benchmark never exercises. And because benchmarks are public, small models get tuned to score well on them — benchmaxxing inflates exactly the number the claim rests on. The frontier models are estimated at 0.5–2T parameters; a 27B dense model is on a different level regardless of one leaderboard.

## "Local isn't about cost" is a position of privilege

The common line that local models "aren't about cost" assumes you can always reach for a subsidized $200/mo plan. Ellis points at where those subsidies bend: GitHub Copilot launched at 1500 requests for $39/mo, then quietly moved everyone to token-based pricing to loud backlash. Uber capped AI spend at $1500/mo per developer per tool — against a $330k median salary, two maxed-out tools is roughly 12% of annual comp. The subsidy is real today and not guaranteed tomorrow. Related economics: [[ai-subsidy-economics]], [[no-silver-bullet-llms]].

## The real driver: sovereignty and vendor risk

What actually pulls a privacy-focused business toward local is data control and vendor risk. Ellis notes that Fable 5 was pulled overnight for users outside the US — local models are the answer to "what if the frontier labs do X?" And customer data can't legally go through cloud plans at all: running a customer's telemetry dump or diagnostic snapshot through ChatGPT Pro or Claude Max, even configured for 30-day retention, likely violates the contracts with those customers. So they wrote `diag` (a CLI that snapshots an OpenFaaS install), have operators email the dump, and run it through an airgapped local model in an ephemeral SlicerVM. One renewal review of a telemetry database found a customer had been under-reporting licenses and underpaying by 4–5x for over a year — revenue recovery that paid for the hardware, and that could never have gone through a cloud plan.

## The hardware bet

Ellis dropped about $12,000 on an RTX 6000 Pro Blackwell with 96GB of VRAM (the price rose to ~$15,400 a couple of months later). It paid for itself — but through the revenue-recovery and support use cases, not by replacing Claude. Earlier he ran dual 3090s, which were a constant source of headaches: a 27B model doesn't fit at full fidelity on one 3090, forcing tradeoffs across weight quantization, context length, and KV-cache compression (rule of thumb: bad things start at Q4_0 on the keys; his most aggressive setting was Q8_0 keys, Q4_0 values). Even with NVLink and tensor parallelism, vLLM ran 3 tok/s slower than llama.cpp at generation — vLLM wins at production-scale concurrent serving, but for a single-user prosumer rig, startup time and latency matter more than aggregate throughput. See [[llm-quantization]], [[local-llm-16gb-vram-tests]].

## Where it fails: long-horizon unsupervised work

The blade metaphor: heat-treating steel, if you overshoot the temper colour by one shade you start over. A local model runs so hot it shoots past the goal and loops, and nothing fixes it except killing the harness and hoping cleared context gives a different result. Concrete failures — asked what commands to add to `faas-cli`, it gave reasonable suggestions then repeated them for half an hour burning 600W; asked to write a Python reverse proxy, it got the indentation wrong, tried to fix it, corrupted the file, and went progressively off the rails; it read 27.3K as 273,000; it inferred a customer would churn from a low function count, ignoring that they ran those functions many times a day. The lesson: keep local models on analysis, not interpretation, and never on long-horizon unsupervised tasks. What's "creepy" about Claude and Codex is how long they work fully unattended while making real progress — that's the gap, not the leaderboard. This is the same shape as [[constraint-decay]] and the median-problem argument in [[no-silver-bullet-llms]].

## The working setup

Two independent llama.cpp instances on the RTX 6000, each keeping full 262144 context: the base Qwen 3.6 27B and Qwopus (a fine-tune layering chain-of-thought traces onto Qwen for better reasoning and coding). MTP speculative decoding gets ~93% acceptance, lifting sustained throughput from a stable 67 tok/s to 130–200 tok/s — it feels faster than a cloud model. Because two agents on one instance invalidate each other's cached prefix (the full prompt reprocesses every time), sharing the box turns into an operations problem: identity, access control, metering, quotas, routing, power monitoring. Ellis wrote "Toilgate," a vibe-coded opencode provider for access and metering, and put two Shelly Plus Plugs on the wall to watch power (the RTX 6000 pulls 600W under inference; the two 3090s draw ~750W combined and are loud). See [[speculative-decoding]].

The trap once you can meter is comparing your per-million-token cost to GPT-5.5 API pricing — the wrong comparison, because the value was never the token price.

## Where local actually earns its keep

- Reading and explaining codebases, even ones it can't write — Ellis calls this a superpower.
- Well-bounded maintenance driven by a detailed AGENTS.md: on alexellis/arkade, the local model added new CLIs faster than human contributors and tested its own work.
- Airgapped customer support and analysis where data can't leave the building.
- Respect the model card: temperature, context, and quantization all matter, and very low quantizations bite.
- Normalise running the same task through both a local and a cloud model.

Ellis skips 70B models (most are generations behind) and notes the 35-A3B Qwen variant is popular mainly because 3B active parameters look fast on a MacBook. Bigger models (GLM 5.2, Kimi 2.7, Minimax M3, Deepseek V4 Flash) need 4–6 of these cards even to load a quantized copy. His blunt summary: today's 27B dense models are not cut out to write Go all day; the unglamorous Grok Coder Fast 1 was cheaper and faster and served the team well for months before deprecation.

## Cross-references

- [[titit-local-ai]] — Ted Neward's parallel argument for local AI in small businesses
- [[lucumr-local-models]] — Armin Ronacher on why the local UX gap is product polish, not model quality
- [[local-llm-16gb-vram-tests]] — the exact quantization / KV-cache operator knobs Ellis describes, worked out on consumer hardware
- [[speculative-decoding]] — the MTP draft mechanism behind his 130–200 tok/s
- [[llm-quantization]] — weight and KV-cache quantization tradeoffs
- [[no-silver-bullet-llms]] — models degrade away from the documented median; same conclusion from different evidence
- [[constraint-decay]] — the measured version of "fails at bounded, convention-heavy work"
- [[alex-ellis]] — the author