# KV Cache Sizing

Every token in the context costs memory to hold attention state — the key and value vectors for each attention head, at each layer. The formula is straightforward:

```
KV bytes per token = 2 × num_kv_heads × head_dim × num_layers × bytes_per_element
```

The `2` is K plus V. The element size depends on the cache type: `f16` is 2 bytes, `q8_0` is 1 byte, `q4_0` is half a byte. Halving cache precision halves cost per token at small quality impact.

## Why the per-model cost varies so much

Architecture matters more than parameter count. Two models from Vyacheslav's tests:

| Model | KV heads | Head dim | Layers | KV per token (q4_0) |
| --- | --- | --- | --- | --- |
| Qwen 3.6 35B | 2 | 128 | 40 | ~10 KB |
| Gemma 4 26B | ~7 avg* | 176 | 30 | ~36 KB |

*Gemma 4 has an irregular layout: most layers have 8 KV heads but every sixth layer has only 2, summing to 210 heads across 30 layers.

Gemma's larger head dimension and higher average head count make its KV cache roughly 4× more expensive per token than Qwen 3.6's, despite the smaller total parameter count. At 256K context this matters a lot: Gemma at 262K tokens needs more VRAM headroom for KV than the *entire model* takes up at Q4 quantization.

## The q8_0 → q4_0 swap

The default cache type in many llama.cpp examples is `q8_0` (1 byte per element). Switching to `q4_0` (0.5 bytes per element) cuts cache cost in half with no measurable quality impact in practice — Vyacheslav reports neither model lost coherence at 200K+ token inputs. Combined with offloading more experts to RAM via [[moe-cpu-offload|`--n-cpu-moe`]], this is how he fit each model's declared 256K-token maximum on 16 GB.

## The 16 GB budget worked example

For Qwen 3.6 35B at 256K context on RTX 5070 Ti 16 GB:

| Setting | `--n-cpu-moe` | Idle VRAM | Peak at ~238K tokens |
| --- | --- | --- | --- |
| 32K context, q8_0 | 17 | 14.8 GB | — |
| 128K context, q4_0 | 20 | 14.0 GB | 14.1 GB |
| 256K context, q4_0 | 23 | 14.1 GB | 14.7 GB |

For Gemma 4 26B at 262K context:

| Setting | `--n-cpu-moe` | Idle VRAM | Peak at ~245K tokens |
| --- | --- | --- | --- |
| 32K context, q8_0 | 8 | 14.8 GB | — |
| 131K context, q4_0 | 10 | 14.1 GB | 14.5 GB |
| 262K context, q4_0 | 12 | 14.4 GB | 15.3 GB |

The Gemma peak at 15.3 GB leaves a thin margin — dropping `--n-cpu-moe` to 11 OOMs at 262K. Qwen's smaller KV cost means its peak stays well within 15 GB even at the full 256K.

## Cost paid

Larger `--n-cpu-moe` means more experts processed via PCIe, so generation slows slightly. For Gemma 4 the difference between `--n-cpu-moe=8` and `--n-cpu-moe=12` is small (the model is compute-bound on this hardware). For Qwen 3.6 the slowdown is more noticeable, but in exchange for 8× the context.

## Beyond q4_0 — TurboQuant on KV cache

A rotate-then-quantize KV cache compression family ([[random-rotation-quantization]]) matches full-precision recall at 4× compression. Compared on Llama-3.1-8B at the same memory budget:

| Method | NiaH recall |
| --- | --- |
| Full cache (FP16) | 0.997 |
| KIVI (integer w/ per-block scales) | 0.981 |
| **TurboQuant** | **0.997** |

llama.cpp's `llama-cpp-turboquant` fork exposes the TurboQuant codebook as `--cache-type-k turbo4 --cache-type-v turbo3`. The [[reddit-qwen3-6-mtp-12gb|Reddit thread]] shows Qwen3.6-35B-A3B at 131K context fitting in 8 GB on a GTX 1070 partly because turbo4 brings KV cost to ~590 MB vs ~720 MB on q4_0. The byte budget is similar; the recall is meaningfully higher. See [[interactive-turboquant]] for the construction.

ik_llama also supports Q6 KV cache quantization, which is a different point on the size/quality curve and works only in that fork.

## Cross-references

- [[moe-cpu-offload]] — the other lever in the 16 GB budget
- [[mixture-of-experts]] — why MoE makes this kind of fine-grained budgeting possible
- [[local-llm-16gb-vram-tests]] — the source article and the parent context
- [[random-rotation-quantization]] — TurboQuant family, the next step past q4_0
- [[interactive-turboquant]] — arkaung's tutorial on the rotation construction