# Coding-Agent Tests of Local LLMs on 16 GB VRAM

A Russian-language Habr writeup by Vyacheslav, an enthusiast (not a professional developer) running local LLMs on consumer hardware. The setup: RTX 5070 Ti with 16 GB VRAM, 32 GB system RAM, CachyOS, llama.cpp built from source for the Blackwell architecture, OpenCode as the agent harness. The premise is that most local-LLM tests on YouTube use oversized models, default settings, and toy prompts, so the picture they give is misleading. Vyacheslav picks three MoE models that *just barely* fit at Q4 quantization, runs them through three agent-shaped tasks, and reports both the scores and the operational tricks that make the runs viable in the first place.

## The three models

All three use Mixture-of-Experts architecture, all three are quantized to roughly 4 bits, none of them fit cleanly into 16 GB on their own:

| Model | File | Total → active | Quant |
| --- | --- | --- | --- |
| Gemma 4 26B-A4B | `gemma-4-26B-A4B-it-MXFP4_MOE.gguf` | 26B → 4B | MXFP4 (Blackwell-native) |
| Qwen 3.6 35B-A3B | `Qwen_Qwen3.6-35B-A3B-Q4_K_M.gguf` | 35B → 3B | Q4_K_M (bartowski) |
| Qwen3-Coder 30B-A3B | `Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf` | 30B → 3B | Q4_K_M (bartowski) |

See [[mixture-of-experts]] for why "26B vs 35B" is the wrong comparison once MoE is involved, and [[llm-quantization]] for the three quant families that show up here.

## The fitting trick: `--n-cpu-moe`

The headline operational finding is that llama.cpp's `--n-cpu-moe` flag is a much better fit for MoE-on-low-VRAM than the classic `--n-gpu-layers` offload. Attention layers stay in VRAM for *every* token; only inactive expert weights spill into system RAM over PCIe. Because only a few experts fire per token, the PCIe traffic stays small.

Numbers from the article (generation tokens/sec):

| Model + mode | `--n-gpu-layers` | `--n-cpu-moe` | Gain |
| --- | --- | --- | --- |
| Qwen 3.6 Fast | 41.3 | **64.0** | +55% |
| Qwen 3.6 Thinking | 41.7 | **66.7** | +60% |
| Qwen Coder Fast | 51.3 | **57.6** | +12% |
| Qwen Coder Thinking | 53.0 | **60.2** | +14% |
| Gemma 4 (fits fully in VRAM) | n/a | **61.9 / 60.8** | — |

For the full picture, see [[moe-cpu-offload]].

## The three tasks

1. **Follow nonstandard AGENTS.md rules.** The project's `AGENTS.md` requires an `ai_gen_` filename prefix, strict typing, and a `// Одобрено ИИ` ("approved by AI") comment. The task itself — write a function that extracts unique emails — is trivial; the test is whether the model honors the contextual rules.
2. **Tool calling.** Read `index.html`, extract all `<h1>` tags, write them to `headers.json`. Multi-step tool use with a JSON-correctness trap (HTML entity decoding).
3. **Refactor + unit tests.** A file with five planted bugs (off-by-one, two divide-by-zeros, single-element return where a list is expected, unreadable names). Refactor and write tests.

Each task graded 4 binary criteria for 4 points; 12 total.

## Results

| Model | Mode | T1 | T2 | T3 | **Total** |
| --- | --- | --- | --- | --- | --- |
| Gemma 4 | Thinking | 3/4 | 4/4 | 4/4 | **11/12** |
| Gemma 4 | Fast | 4/4 | 4/4 | 4/4 | **12/12** |
| Qwen 3.6 | Thinking | 1/4 | 4/4 | 2/4 | **7/12** |
| Qwen 3.6 | Fast | 3/4 | 4/4 | 2/4 | **9/12** |
| Qwen Coder | Thinking | 2/4 | 4/4 | 2/4 | **8/12** |
| Qwen Coder | Fast | 2/4 | 2/4 | 2/4 | **6/12** |

Gemma 4 Fast is the only run that scores perfectly. Three findings stand out below.

### Finding 1: Thinking mode hurts rule-following

Across all three models, the Thinking variant did *worse* than Fast on Test 1 (or tied). Qwen 3.6 Thinking — the lowest score in the matrix — ignored three of four `AGENTS.md` rules. Vyacheslav's hypothesis: when the model "thinks," it evaluates whether the nonstandard rules are actually load-bearing and quietly skips the ones it considers arbitrary. Fast mode follows literally without rationalizing. See [[thinking-mode-rule-erosion]] for the standalone concept and the prescriptive takeaway.

### Finding 2: Qwen Coder Fast fails Tool Calling on an HTML entity

The single Tool Calling failure across six runs: Qwen Coder Fast wrote `"Support &amp; Contact"` into the JSON instead of `"Support & Contact"`. It read the raw HTML and never decoded the entity. Every other run, including Qwen Coder *Thinking*, did decode it. Two points lost over one character.

### Finding 3: Qwen models break on long agentic sessions

Both Qwen models failed Test 3 in the same pattern: read file, start refactor, file-write tool errors on the large output, context compresses, model "forgets" what it was doing, restarts or asks the user. This is independent of Thinking. Vyacheslav also caught Qwen 3.6 Fast *hallucinating* in its final report that it had added the `ai_gen_` prefix — when the code contained no such prefix. The lesson: the model's prose summary is not a trustworthy record of what it actually did.

A related artifact: both Qwen models created a new `refactored_code.py` instead of editing the existing `buggy_code.py`. Qwen 3.6 Thinking was the only Qwen run that edited in place — apparently the extra reasoning helped it understand the task shape.

## Quantization sub-story

Vyacheslav also documents a smaller finding worth its own line: Unsloth's "Dynamic 2.0" UD-Q4_K_XL quants, marketed as the more advanced format, perform *worse* than classic Q4_K_M on Qwen MoE. Perplexity vs Q8_0 baseline:

| Format | Size | Perplexity (WikiText-2) | Degradation |
| --- | --- | --- | --- |
| Q8_0 (baseline) | ~36.9 GB | 6.5342 | 0% |
| Q4_K_M | ~20.0 GB | 6.6688 | +2.1% |
| UD-Q4_K_XL | ~19.0 GB | 7.1702 | +9.7% |

The 1 GB disk saving costs almost 5× more perplexity error. Unsloth themselves later removed MXFP4 layers from their Qwen GGUF builds because of computational anomalies. Detail in [[llm-quantization]].

## 256K context on 16 GB

A follow-up section after community comments: by switching `--cache-type-k` and `--cache-type-v` from `q8_0` to `q4_0` (halving KV cache cost per token) and pushing `--n-cpu-moe` higher, both Qwen 3.6 and Gemma 4 fit their declared 256K-token context windows in 16 GB VRAM. The Gemma 4 KV cache costs roughly 4× more per token than Qwen's because of its 30 layers × ~7 KV heads × 176 head-dim layout, so it needs `--n-cpu-moe 12` instead of `--n-cpu-moe 23` to hit 256K. See [[kv-cache-sizing]].

## Practical conclusions Vyacheslav draws

- **For agentic IDE work on 16 GB VRAM:** Gemma 4 Fast.
- **For complex refactor + tests:** Gemma 4 Thinking — the only model that wrote tests that actually ran.
- **For strict style-guide adherence:** any model in Fast mode. Never Thinking.
- **For long multi-step sessions:** avoid Qwen at current llama.cpp builds.
- **Skip LM Studio and Ollama** — up to 30% slower than llama.cpp built from source. The convenience costs measurable performance on hardware this constrained.
- **Skip context-bloating skills** (e.g., Context 7, Superpowers) on small local models. They work on large hosted models because the model has tokens to spare.

## Connections

The article slots into existing lines of writing in the wiki:

- [[lucumr-local-models]] — Armin Ronacher's complaint about local-LLM fragmentation and tool-streaming gaps; Vyacheslav's setup is exactly the kind of expertise Ronacher says shouldn't be required.
- [[titit-local-ai]] — Ted Neward's case for local OSS LLMs on grounds of cloud fragility and economics; this article is a worked example of someone making the bet.
- [[hold-on-to-your-hardware]] — 16 GB consumer cards are the realistic ceiling; the operational tricks here are *because* of that ceiling.
- [[gemma-gem]] — separate Gemma 4 deployment via WebGPU in a Chrome extension; same model family in a very different surface.
- [[deepseek-v4-roleplay-instruct]] — another writeup about reasoning-mode behavior, with a different lesson (DeepSeek's `<think>` toggle is fine; here, Thinking mode itself is a hazard).

Tools introduced from this article: [[toolbox/llama-cpp|llama.cpp]] (entity-level, finally has its own toolbox page), [[toolbox/opencode|OpenCode]] (the agent harness used here).

## Related ingest (2026-05-13 batch)

- [[habr-local-llm-quantization-deep-dive]] — Vyacheslav's earlier primer article that lays out the conceptual ground this tests article assumes
- [[reddit-qwen3-6-mtp-12gb]] — practitioner Reddit thread on Qwen 3.6 + MTP at 80 t/s on 12 GB; same model family, adds speculative decoding
- [[little-coder-scaffold-model-fit]] — Inbarr shows scaffold redesign moves Qwen3.5-9B from 19.11% to 45.56% on Aider Polyglot; an orthogonal lever from the one studied here
- [[incompressible-knowledge-probes]] — Bojie Li's IKP framework places Qwen3.6, Gemma 4, and other locally-runnable models on the factual-capacity scale alongside frontier closed models
- [[interactive-turboquant]] — KV cache compression that improves on the q8_0 → q4_0 swap used in this article's 256K-context configuration