Coding-Agent Tests of Local LLMs on 16 GB VRAM

A Russian-language Habr writeup by Vyacheslav, an enthusiast (not a professional developer) running local LLMs on consumer hardware. The setup: RTX 5070 Ti with 16 GB VRAM, 32 GB system RAM, CachyOS, llama.cpp built from source for the Blackwell architecture, OpenCode as the agent harness. The premise is that most local-LLM tests on YouTube use oversized models, default settings, and toy prompts, so the picture they give is misleading. Vyacheslav picks three MoE models that just barely fit at Q4 quantization, runs them through three agent-shaped tasks, and reports both the scores and the operational tricks that make the runs viable in the first place.

The three models

All three use Mixture-of-Experts architecture, all three are quantized to roughly 4 bits, none of them fit cleanly into 16 GB on their own:

See mixture-of-experts for why "26B vs 35B" is the wrong comparison once MoE is involved, and llm-quantization for the three quant families that show up here.

The fitting trick: --n-cpu-moe

The headline operational finding is that llama.cpp's --n-cpu-moe flag is a much better fit for MoE-on-low-VRAM than the classic --n-gpu-layers offload. Attention layers stay in VRAM for every token; only inactive expert weights spill into system RAM over PCIe. Because only a few experts fire per token, the PCIe traffic stays small.

Numbers from the article (generation tokens/sec):

For the full picture, see moe-cpu-offload.

The three tasks

1. Follow nonstandard AGENTS.md rules. The project's AGENTS.md requires an ai_gen_ filename prefix, strict typing, and a // Одобрено ИИ ("approved by AI") comment. The task itself — write a function that extracts unique emails — is trivial; the test is whether the model honors the contextual rules.
2. Tool calling. Read index.html, extract all <h1> tags, write them to headers.json. Multi-step tool use with a JSON-correctness trap (HTML entity decoding).
3. Refactor + unit tests. A file with five planted bugs (off-by-one, two divide-by-zeros, single-element return where a list is expected, unreadable names). Refactor and write tests.

Each task graded 4 binary criteria for 4 points; 12 total.

Results

Gemma 4 Fast is the only run that scores perfectly. Three findings stand out below.

Finding 1: Thinking mode hurts rule-following

Across all three models, the Thinking variant did worse than Fast on Test 1 (or tied). Qwen 3.6 Thinking — the lowest score in the matrix — ignored three of four AGENTS.md rules. Vyacheslav's hypothesis: when the model "thinks," it evaluates whether the nonstandard rules are actually load-bearing and quietly skips the ones it considers arbitrary. Fast mode follows literally without rationalizing. See thinking-mode-rule-erosion for the standalone concept and the prescriptive takeaway.

Finding 2: Qwen Coder Fast fails Tool Calling on an HTML entity

The single Tool Calling failure across six runs: Qwen Coder Fast wrote "Support & Contact" into the JSON instead of "Support & Contact". It read the raw HTML and never decoded the entity. Every other run, including Qwen Coder Thinking, did decode it. Two points lost over one character.

Finding 3: Qwen models break on long agentic sessions

Both Qwen models failed Test 3 in the same pattern: read file, start refactor, file-write tool errors on the large output, context compresses, model "forgets" what it was doing, restarts or asks the user. This is independent of Thinking. Vyacheslav also caught Qwen 3.6 Fast hallucinating in its final report that it had added the ai_gen_ prefix — when the code contained no such prefix. The lesson: the model's prose summary is not a trustworthy record of what it actually did.

A related artifact: both Qwen models created a new refactored_code.py instead of editing the existing buggy_code.py. Qwen 3.6 Thinking was the only Qwen run that edited in place — apparently the extra reasoning helped it understand the task shape.

Quantization sub-story

Vyacheslav also documents a smaller finding worth its own line: Unsloth's "Dynamic 2.0" UD-Q4_K_XL quants, marketed as the more advanced format, perform worse than classic Q4_K_M on Qwen MoE. Perplexity vs Q8_0 baseline:

The 1 GB disk saving costs almost 5× more perplexity error. Unsloth themselves later removed MXFP4 layers from their Qwen GGUF builds because of computational anomalies. Detail in llm-quantization.

256K context on 16 GB

A follow-up section after community comments: by switching --cache-type-k and --cache-type-v from q8_0 to q4_0 (halving KV cache cost per token) and pushing --n-cpu-moe higher, both Qwen 3.6 and Gemma 4 fit their declared 256K-token context windows in 16 GB VRAM. The Gemma 4 KV cache costs roughly 4× more per token than Qwen's because of its 30 layers × ~7 KV heads × 176 head-dim layout, so it needs --n-cpu-moe 12 instead of --n-cpu-moe 23 to hit 256K. See kv-cache-sizing.

Practical conclusions Vyacheslav draws

• For agentic IDE work on 16 GB VRAM: Gemma 4 Fast.
• For complex refactor + tests: Gemma 4 Thinking — the only model that wrote tests that actually ran.
• For strict style-guide adherence: any model in Fast mode. Never Thinking.
• For long multi-step sessions: avoid Qwen at current llama.cpp builds.
• Skip LM Studio and Ollama — up to 30% slower than llama.cpp built from source. The convenience costs measurable performance on hardware this constrained.
• Skip context-bloating skills (e.g., Context 7, Superpowers) on small local models. They work on large hosted models because the model has tokens to spare.

Connections

The article slots into existing lines of writing in the wiki:

• lucumr-local-models — Armin Ronacher's complaint about local-LLM fragmentation and tool-streaming gaps; Vyacheslav's setup is exactly the kind of expertise Ronacher says shouldn't be required.
• titit-local-ai — Ted Neward's case for local OSS LLMs on grounds of cloud fragility and economics; this article is a worked example of someone making the bet.
• hold-on-to-your-hardware — 16 GB consumer cards are the realistic ceiling; the operational tricks here are because of that ceiling.
• gemma-gem — separate Gemma 4 deployment via WebGPU in a Chrome extension; same model family in a very different surface.
• deepseek-v4-roleplay-instruct — another writeup about reasoning-mode behavior, with a different lesson (DeepSeek's <think> toggle is fine; here, Thinking mode itself is a hazard).

Tools introduced from this article: llama.cpp (entity-level, finally has its own toolbox page), OpenCode (the agent harness used here).

Related ingest (2026-05-13 batch)

• habr-local-llm-quantization-deep-dive — Vyacheslav's earlier primer article that lays out the conceptual ground this tests article assumes
• reddit-qwen3-6-mtp-12gb — practitioner Reddit thread on Qwen 3.6 + MTP at 80 t/s on 12 GB; same model family, adds speculative decoding
• little-coder-scaffold-model-fit — Inbarr shows scaffold redesign moves Qwen3.5-9B from 19.11% to 45.56% on Aider Polyglot; an orthogonal lever from the one studied here
• incompressible-knowledge-probes — Bojie Li's IKP framework places Qwen3.6, Gemma 4, and other locally-runnable models on the factual-capacity scale alongside frontier closed models
• interactive-turboquant — KV cache compression that improves on the q8_0 → q4_0 swap used in this article's 256K-context configuration