KV Cache Sizing

Every token in the context costs memory to hold attention state — the key and value vectors for each attention head, at each layer. The formula is straightforward:

KV bytes per token = 2 × num_kv_heads × head_dim × num_layers × bytes_per_element

The 2 is K plus V. The element size depends on the cache type: f16 is 2 bytes, q8_0 is 1 byte, q4_0 is half a byte. Halving cache precision halves cost per token at small quality impact.

Why the per-model cost varies so much

Architecture matters more than parameter count. Two models from Vyacheslav's tests:

*Gemma 4 has an irregular layout: most layers have 8 KV heads but every sixth layer has only 2, summing to 210 heads across 30 layers.

Gemma's larger head dimension and higher average head count make its KV cache roughly 4× more expensive per token than Qwen 3.6's, despite the smaller total parameter count. At 256K context this matters a lot: Gemma at 262K tokens needs more VRAM headroom for KV than the entire model takes up at Q4 quantization.

The q8_0 → q4_0 swap

The default cache type in many llama.cpp examples is q8_0 (1 byte per element). Switching to q4_0 (0.5 bytes per element) cuts cache cost in half with no measurable quality impact in practice — Vyacheslav reports neither model lost coherence at 200K+ token inputs. Combined with offloading more experts to RAM via `--n-cpu-moe`, this is how he fit each model's declared 256K-token maximum on 16 GB.

The 16 GB budget worked example

For Qwen 3.6 35B at 256K context on RTX 5070 Ti 16 GB:

For Gemma 4 26B at 262K context:

The Gemma peak at 15.3 GB leaves a thin margin — dropping --n-cpu-moe to 11 OOMs at 262K. Qwen's smaller KV cost means its peak stays well within 15 GB even at the full 256K.

Cost paid

Larger --n-cpu-moe means more experts processed via PCIe, so generation slows slightly. For Gemma 4 the difference between --n-cpu-moe=8 and --n-cpu-moe=12 is small (the model is compute-bound on this hardware). For Qwen 3.6 the slowdown is more noticeable, but in exchange for 8× the context.

Beyond q4_0 — TurboQuant on KV cache

A rotate-then-quantize KV cache compression family (random-rotation-quantization) matches full-precision recall at 4× compression. Compared on Llama-3.1-8B at the same memory budget:

llama.cpp's llama-cpp-turboquant fork exposes the TurboQuant codebook as --cache-type-k turbo4 --cache-type-v turbo3. The Reddit thread shows Qwen3.6-35B-A3B at 131K context fitting in 8 GB on a GTX 1070 partly because turbo4 brings KV cost to ~590 MB vs ~720 MB on q4_0. The byte budget is similar; the recall is meaningfully higher. See interactive-turboquant for the construction.

ik_llama also supports Q6 KV cache quantization, which is a different point on the size/quality curve and works only in that fork.

Cross-references

• moe-cpu-offload — the other lever in the 16 GB budget
• mixture-of-experts — why MoE makes this kind of fine-grained budgeting possible
• local-llm-16gb-vram-tests — the source article and the parent context
• random-rotation-quantization — TurboQuant family, the next step past q4_0
• interactive-turboquant — arkaung's tutorial on the rotation construction