# Local AI is not Opus We've all heard people say that local Qwen 27B or 35-A3B is "near-Opus level", but I have receipts from a software business and open source projects, and am here to be transparent with you. > This post is long-form for a reason. It's not a cursory glance, an unsubstantiated claim on X about cancelling Claude Max, or a hobbyist report from a model running at single-digit tokens per second with a 32K context window. It isn't written by a famous CEO tweeting about coding from an airplane. > > It's my journey as a founder in a small software business, where local models have produced real, caveated value. I have skin in the game, but no incentive to push either cloud or local models, and a strong desire for local models to become capable and reliable. I'll cover how the card paid for itself in the first two or three months, how it keeps serving our specific business use case, why I still can't trust it unsupervised, and Qwen's worst trait: the infinite loops and hallucination risk. These show up most when you quantize it down to fit a consumer GPU. **On my use case for AI** My journey as a maintainer and founder started with OpenFaaS - built completely by hand, as was all software in 2016 up until recently. That meant laying down the core of the project on my own, then inviting others to participate through community. Around 2017 I tried to fund my time by joining VMware, and in 2019 after changes in the market, I needed a way to fund the work myself, so moved towards open-core and built a bootstrapped company. Today our small team maintains OpenFaaS, SlicerVM (AI sandboxes and "the missing API for Linux"), Actuated.com (self-hosted CI runners for GitHub/GitLab), and Inlets.com (self-hosted HTTP/TCP tunnels). These products are built around low-level infrastructure and Linux primitives: containers, Firecracker microVMs, network protocols, tunnels, CLIs, and Kubernetes. They're written in Go, and some have React-based UI components, landing pages, docs, agent skills, and CLIs. I've been using AI tools for as long as they've been available - from tab completion in VS Code, through ChatGPT generating chunks of code, to living in tmux 12 hours per day. I wrote a free tool Superterm.dev to keep track of my sessions. It's Claude or Codex that do the majority of my work, and whilst I insist on doing my own writing, I rarely write code by hand. **A turning point for frontier intelligence** I'd say it was roughly between November 2025 and January 2026 that we saw a turning point. Many developers on X started to espouse Claude Opus as having changed and how it was now capable of doing *all of* their work. The costs of the top-end coding plans settled at roughly 200 USD / mo for individuals. **What makes local models interesting** > There's an argument that says: "Why use anything less than the best you can afford?" The year of 2026 certainly is a new frontier: any idea can be cloned overnight by someone you've never heard of with a subscription in a developing nation. I've seen it happen to our SlicerVM product (originally written by hand in 2022) and Superterm (new in 2026, 100% written by coding agents). There are estimates that the leading models contain between 0.5-2T parameters. That's on a different level. Yet somehow, even a tiny dense model like Qwen 3.6 27B is able to score a reputable benchmark of 77.2 on SWE-Bench Verified vs 88.6% from Claude Opus 4.8. So you could be forgiven for taking to X and shouting loudly that "local is only 12% behind SOTA" and many have. You may go as far as claiming that a single 6-year old GPU can replace your 200 USD / mo ChatGPT Pro subscription. **Benchmaxxing** Benchmarks are a moving target, and since they're widely available, it's possible to educate and tune a model to obtain a higher score. The classic SWE-Bench Verified benchmark is based upon a set of Python issues. Python has threads and async, however most code you run into is single-threaded and synchronous. In contrast, we write distributed systems in Go, where channels, contexts, and structs span across a large execution domain. **Cost** There's a very popular take "local models aren't about cost" and that comes from a position of privilege. Individuals can use coding plans for 200 USD / mo. Coding plans are clearly subsidised, just look at what happened to GitHub Copilot plans. They started off by giving away 1500 requests for 39 USD / mo. Something changed at GitHub/Microsoft/Azure, and they moved everyone over to token-based pricing and the backlash was huge. Recently, Uber capped spend to 1500 USD / mo per developer per tool. The median salary at Uber is 330k USD annually, so if a developer used two tools to the maximum extent, it's roughly 12% of their annual compensation. **Sovereignty and privacy** We work with various enterprise customers that take data controls very seriously. If you squint at our product line, we're all about privacy and sovereignty. So naturally, we are drawn to local models. If you live outside of the US, the removal of Anthropic's Fable 5 model overnight might have come as a shock. In other words, there is serious vendor risk. Local models are the solution to "What if the frontier labs do X?" ## Tempering the blade I said that local models are not the same tool as SOTA. I build furniture using hand tools, and occasionally I'll make an edge tool like a chisel, a Sloyd knife for carving. There are two ways to work with steel: forging (heating a raw piece and hammering it) and "stock removal" (cutting sheet steel and grinding a bevel). But that's just the shaping. You then heat the steel and quench it in oil or water. This makes the steel become extremely hard, so hard that if you dropped it, it would shatter. So we scrub off the black scum, heat it up again, watching for a rainbow of colours. If we go one shade past where we need, we start the heat treating all over again. Our team's experience of local models is exactly like missing the temper colours. The model is running so hot, that it shoots past the goal and starts looping. Nothing can fix it, other than closing down the harness and hoping the cleared context gives a different result. I'd never leave Qwen 3.6 27B working on a long horizon task. **What I was looking for** I was looking for privacy, fixed costs and protection against vendor risk. Where I got let down is where I treat a local model inside opencode in the same way I treat Claude or Codex. It's almost creepy how long Claude/Codex can work fully unattended whilst making real progress towards a goal. **Sharp lessons from a 3090** I started off with a single 3090 card in 2023, and quickly realised I needed another to load models and have sufficient context. Qwen 3.5 was the first time I saw real work being done by agents. I could load a model in Q4 quantization with 200k context. I told the model "Explore this machine from every angle, complete a forensic report" - Qwen started reading every single file one by one, filled its context, then hallucinated the filenames and even tool calls (`~/faas-netes` became `~/faaned`). Scoping the task tightly gave a lucid report at 40-50 tokens per second. A 27B model doesn't fit at full fidelity into 1x 3090. The knobs are: quantization of weights, length of context, and compression of the KV cache. There's a well known rule of thumb that bad things start happening at Q4_0 on the keys part of the KV cache. The most aggressive I've been is Q8_0 for keys and Q4_0 for values. The 3090s were a constant source of headaches. My latest experiment was setting up vLLM (the gold standard for production and concurrent serving) and even with an NVLink (175GBP) and tensor parallelism, it was 3 tokens/second slower than llama.cpp during generation. vLLM is the right choice for production-scale serving with continuous batching and many concurrent users. But in a prosumer setup, startup time, simplicity, and single-user latency matter more than aggregate throughput. **Big spender** We dropped around 12000 USD on an RTX 6000 Pro Blackwell edition with 96GB of VRAM. A couple of months on, the price increased to around 15400 USD. It was a calculated bet, and it has paid off, but not because it replaces our Claude subscriptions - it can't do that. **Painless customer support, without leaking customer data** We wrote "diag", a CLI tool that captures a complete snapshot of an OpenFaaS installation on Kubernetes. Operators email this dump and we run it through an airgapped local model, in an ephemeral VM created by Slicer. **Revenue recovery** A renewal came up recently, and only because I fed the telemetry database into a local model, did we find out they'd been under-reporting licenses and under-paying by about 4-5x for over 12 months. That revenue recovery alone paid for the card. There's no way I would have in good conscience run the telemetry dump or a customer's diag output through any cloud plan. ChatGPT Pro and Claude Max can be configured for a 30 day retention period, but even that likely invalidates your contracts with customers. With earlier iterations of the model, I saw it fail at arithmetic - 27.3K counted as 273,000. Another time, the model inferred a customer was likely to churn because they had a small number of functions, ignoring that the customer ran that smaller number many times per day. So often it's better to have them focus on analysis, not interpretation. **Our current setup** I'm a big supporter of folks like Jack Rong and Kyle Hessling who have worked on fine-tunes of open weight models like Qwen. Qwopus attempts to layer Chain of Thought traces on top of Qwen to make it better at reasoning and coding. In our team we run both the latest generation of Qwopus, and the base 27B Qwen 3.6 model on the RTX 6000 rig. The models are served by two independent llama.cpp instances, which means they retain full context length. ```bash #!/bin/bash ~/llama.cpp/build/bin/llama-server \ -hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q8_K_XL \ --alias Qwen3.6-27B-Base \ --host 0.0.0.0 \ --port 8085 \ -ngl 99 \ -c 262144 \ --cache-type-k f16 \ --cache-type-v f16 \ --flash-attn on \ --parallel 1 \ --threads 16 \ -b 4096 \ -ub 2048 \ --jinja \ --reasoning-budget 2048 \ --temperature 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --presence-penalty 1.1 \ --reasoning on \ --spec-type draft-mtp \ --spec-draft-n-max 6 \ --chat-template-kwargs '{"preserve_thinking": true}' \ --chat-template-file chat_template.jinja \ --reasoning-budget-message "reasoning budget consumed, time to answer now" ``` We get about a 93% acceptance rate on our speculative decoding from MTP, and the speed increases from a stable 67 tok/s to 130-200 tok/s sustained over long periods. It feels faster than using a cloud model. With the Qwopus fine-tune, it works best with thinking turned off and the temperature really hot at 0.85-1.0. **About that looping** I asked Qwen what commands we should add to `faas-cli`, and it came back with reasonable suggestions, but got stuck and kept repeating them over and over, burning 600W of my electricity for a good half an hour. The same thing happened when I asked it to "add --json to all get and list commands". When it needed to write a reverse proxy in Python, the first version had bad indenting; when it realised the issue, it corrupted the file, and kept complaining that it didn't know how to fix it, going progressively off the rails. Han from my team has reported very similar looping - the model is stuck, at the edge of its ability, and won't ask for help. **Measuring and distributing access** Two agents hitting the same llama.cpp instance with unrelated contexts means each request invalidates the other's cached prefix — the full prompt gets re-processed from scratch every time. As soon as another person uses the model, it stops being a prototype: Who's on which llama.cpp instance? How much are they using? What has that cost in electricity? Rather than manually editing opencode.json, I wrote a provider for opencode ("Toilgate", 100% vibe-coded). Two Shelly Plus Plugs monitor power consumption at the wall. The RTX 6000 Pro pulls 600W during inference; the two 3090s are closer to 750W combined and extremely noisy. **The wrong comparison** The trap once you can measure is comparing input/output costs per million tokens to OpenAI's API pricing for GPT-5.5. That's the wrong comparison. This is where "local AI" turns into an operations problem. You need identity, access control, metering, quotas, model routing and power monitoring. ## Wrapping up Whilst local Qwen is not "near Opus levels", it is of value for certain tasks and workflows. Concrete things that help: - Match the local model and harness to specialised tasks - customer support, well bounded maintenance, and end-to-end testing - AGENTS.md - when I added detailed instructions to alexellis/arkade, the local model could add new CLIs more quickly than human contributors, and would test its work - Pay attention to the tuning notes on the model card - temperature, context settings, and quantization all matter. Beware of very low quantizations. - Local models can quickly read and explain codebases, even if they can't write them - this is a superpower - Fine-tunes like Qwopus exist - be willing to experiment - Normalise running the same task with a local and cloud model - Don't hand it long-horizon, unsupervised agentic work - that's where it loops, and even our almost 15k USD card couldn't fix that I've not mentioned 70B models - most are generations behind. The 35-A3B variant of Qwen is popular because it looks faster on MacBooks (only 3B active parameters at generation time). There are bigger models like GLM 5.2, Kimi 2.7, Minimax M3 and Deepseek V4 Flash, but you're often talking about 4-6 RTX 6000 Pro cards to even load a quantized version. Today, 27B dense models are not cut out to write Go all day long. The relatively unsexy Grok Coder Fast 1 was cheaper, and faster, and served us well for months before being deprecated.