# Local LLMs Deep Dive — Quants, MoE Offload, REAP, ik_llama, Speculative Decoding

Vyacheslav's second Habr article (1025132), published before the [[local-llm-16gb-vram-tests|three-model tests article]] (1033808) but covering the conceptual ground that article assumed. Where the tests article was a benchmark in narrative form, this one is the operator's primer: why BF16 won over FP16, how K-quants and I-quants differ, what `-fit` does that `--n-cpu-moe` did manually, why a Q3 quant can be slower than Q4 on the same model, what REAP buys you, and where ik_llama is and isn't a llama.cpp replacement.

## The pieces

### MoE naming and the Dense/MoE quality tradeoff

Standard MoE name is `Qwen3.6-35B-A3B`: major.minor model version, total parameter count, active parameter count. More active params means better quality but lower tokens/sec. Dense models activate everything; for the same total parameters they're "smarter" but much slower. See [[mixture-of-experts]].

### Why BF16, not FP16

The neural-network argument for keeping FP32's 8-bit exponent and shrinking the mantissa instead: training cares about dynamic range, not precision. Gradient descent visits magnitudes around $10^{-7}$ to $10^{-8}$; activation functions can't return infinity when values blow up; backpropagation has to make small corrections. FP16 narrows the exponent to 5 bits and the practical effect is that training stalls (weights stop moving) or diverges into NaN. The fix was Mixed Precision (FP32 + FP16), which costs more memory than FP32 alone — so the actual win came from BF16, which keeps FP32's 8-bit exponent and shrinks the mantissa from 23 bits to 7. Range preserved, size halved, training proceeds. This is the prehistory of every modern open-weight model. See [[bf16-vs-fp16]].

### What changed K-quants from naive Int4

llama.cpp's modern quantization story is ikawrakow's contribution. Three things on top of the basic algorithm:

- **Block quantization.** Model splits into blocks; each block has its own scale. Non-homogeneous models like LLMs benefit because outlier weights don't poison the whole tensor.
- **imatrix calibration.** A typical-usage dataset that activates different blocks. Blocks that respond strongly are quantized less aggressively; others, more.
- **Separate attn / ffn quantization levels.** Attention dominates "intelligence"; if attn is kept higher-precision while ffn is quantized hard, the overall quality stays high.

K-quants (`Q4_K_M`, etc.) are static + K-blocked. I-quants (`IQ4_XS`) use imatrix. Letter suffixes (S, M, L, XL) indicate how much higher attn is quantized than ffn body.

### UD-Q4_K_XL beats Q4_K_M; Q4_K_M ≈ UD-Q3_K_XL

KLD measurements (Kullback-Leibler Divergence vs the BF16 baseline) on Qwen3.6-35B-A3B:

- Classic Q4_K_M ≈ UD-Q3_K_XL (the dynamic Q3 quant is 4.4 GB smaller and roughly the same quality)
- UD-Q4_K_XL beats Q4_K_M, costing 1.2 GB more on disk
- UD-Q2_K_XL beats IQ2_M at the same byte budget

This updates the previously-recorded finding from [[llm-quantization]] — the earlier number (UD-Q4_K_XL *worse* than Q4_K_M on Qwen MoE by ~5×) came from a *perplexity-on-WikiText-2* measurement, not KLD vs BF16. The two metrics can point in different directions on the same comparison. Vyacheslav's KLD measurement is the more recent and architecture-specific one.

### Four dynamic quant creators

- **Unsloth** (UD-…-XL): keeps attn closer to original, dynamic ffn quantization. Released the [Unsloth Dynamic 2.0](https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs) scheme.
- **Ubergarm** (IQ4_KS and IQK family): builds for ik_llama specifically, dynamic quantization.
- **Bartowski**: standard Q4_K_M naming but custom imatrix; the imatrix is English-dominated, weaker on Russian.
- **mradermacher** (i1 prefix): imatrix tuned for multilingual.

All four use imatrix even for "static" K-quants. The standard recipe (ggml, ollama, LM Studio defaults) has been left behind.

See [[unsloth]] entity page.

### `-fit` is the new default

The big operational change: llama.cpp now ships `-fit` mode (default-on) that auto-detects MoE vs dense and picks `-ncmoe` vs `-ngl`. Memory accounting is automatic; specify `--fit-target 1024` to leave 1 GB free, or trust the default. The same flag the [[reddit-qwen3-6-mtp-12gb|Reddit thread]] uses as `-fitt 1536`. See [[moe-cpu-offload]] for the full mode table.

Three modes: `-ngl` (layer offload, best for Dense), `-cmoe` (full CPU offload of all MoE experts, smallest VRAM footprint), `-ncmoe N` (partial CPU offload of N expert layers).

### Why Ollama is 3× slower than llama.cpp

Ollama uses llama.cpp's GGML engine under the hood, so it *could* enable `-ncmoe` for MoE models. It doesn't. It uses `-ngl` for everything — Dense, where ngl is correct; and MoE, where ngl partial-loads to all 16 GB while leaving GPU at 30% effective utilization. Measured on RTX 4060 16GB with Qwen3.6-35B-A3B Q4_K_M:

| Engine | Mode | Speed |
| --- | --- | --- |
| ollama | ngl | 16 t/s |
| llama.cpp | ncmoe (auto via `-fit`) | **45 t/s** |

At 32K context the gap widens to 11 vs 36 t/s (3.2×). LM Studio loses similarly, both because of the engine choice and because it loads Vision weights even when not requested.

### Why UD-Q3_K_XL can be slower than UD-Q4_K_XL

Counterintuitive observation: a *smaller* quant can be *slower* on a hybrid CPU/GPU setup. The reason is the quant composition. UD-Q3_K_XL ships with i-quant blocks; UD-Q4_K_XL is all static K-quants. I-quants take ~2× more CPU compute to dequantize at inference time. If the smaller quant pushes more layers to GPU but the bottleneck is CPU expert layers, the bigger quant wins.

Reproduced on i7-14700 with `-t 2 -cmoe`: UD-Q4_K_XL at 25 t/s vs UD-Q2_K_XL at 18 t/s, both on the same Qwen3.6 model.

This generalizes: when you're CPU-bound on expert layers, quant *type* matters more than quant *size*. K-quants beat I-quants when the dequantization cost dominates.

### UD-Q2_K_XL is more capable than the conventional wisdom says

The folk knowledge that Q2 and Q3 quants are "not worth using" comes from static quants. Dynamic Q2 and SOTA quants like IQ2_KS are usable. Qwen3.6-35B-A3B at UD-Q2_K_XL (12.3 GB) fits fully on a 4060 16 GB at 68 t/s and successfully one-shots a procedural Minecraft clone with the help of qwen-code as agent harness — block destruction, multiple biomes, water flow, procedural graphics — over ~3 hours of agentic editing.

### REAP — expert pruning

REAP (Router-weighted Expert Activation Pruning) cuts the least-active experts from an MoE based on a target-domain calibration dataset. Common cuts: 20%, 50%, 75%. Variants are named `Qwen3.6-28B-REAP20-A3B-GGUF` (20% pruned, still 3B active). The 20% cut on Qwen 3.6 retains most agentic-coding capability — Vyacheslav successfully extended the Minecraft clone (added caves, diamonds, a generated house) using the REAP version. Russian-language quality degrades noticeably though: the REAP calibration dataset is English-coding-focused, so non-target capabilities fade first. Forgets who Chapaev is.

REAP is also a path to fitting *large* MoE models into RAM budgets — GLM-5.1 744B-A40B → 444B-A14B saves 40% of disk and active-compute. See [[reap-expert-pruning]].

### Speculative decoding overview

Four mechanisms for predicting multiple tokens per forward pass; the post acts as the index for [[speculative-decoding]]:

- **Draft model.** Use a small sibling model to predict next tokens; the big model verifies in batch. Worked example: Gemma-4 31B (62 t/s) + Gemma-4 E2B draft → 97 t/s on coding workloads (~55% draft acceptance). `--parallel 1` is necessary (the auto mode reserves 4× memory).
- **MTP (Multi-Token Prediction).** Extra blocks trained alongside the model that predict several tokens ahead. First shipped in DeepSeek V3/R1. llama.cpp PR #22673 brings it to Qwen 3.6 and Gemma 4.
- **EAGLE3.** No draft model and no MTP head; uses a lightweight autoregressive head fed by hidden states. PR #18039 in llama.cpp.
- **ngram-mod.** Pure-math draft, no model. Best for scenarios with heavy text reuse — e.g., editing one piece of code while the rest stays identical. `--spec-type ngram-mod`.

### Prompt-processing speedup

When the model is CPU-offloaded via cmoe, prompt processing (PP) is slow. The fix: bump batch sizes. `-ub 4096 -b 4096` jumps PP from 93 t/s to 475 t/s on Qwen3.6 with `-ncmoe 24`. Generation (TG) is unaffected. Worth knowing for agentic workloads where the model constantly re-ingests source files.

### ik_llama is not a drop-in replacement

ikawrakow split from ggerganov, forked llama.cpp into [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp), and ships SOTA IQK quants and aggressive PP optimizations. Trade-offs:

| | llama.cpp | ik_llama |
| --- | --- | --- |
| MXFP4 quants | full | partial (2–3× slower) |
| Vulkan / AMD GPUs | full | runs but very slow |
| AVX-512 / Intel CPUs | good | better |
| PP at long context | drops faster | holds up longer |
| KV cache Q6 quantization | no | yes |
| New IQK quants | no | yes |

IQ4_KS on Qwen3.5 measured ~3× better KLD than Q4_K_M of the same size, and matches UD-Q4_K_XL on quality at ~1 GB less. On large models the size win compounds — 20-30 GB matters when you're trying to fit GLM-5.1 or DeepSeek V3.2 into 192 GB RAM.

The maintenance trade-off: one person works on ik_llama. Optimizations don't flow in either direction with main llama.cpp. See [[toolbox/ik-llama-cpp]].

### Linux vs Windows, and iGPU offload

For RTX 50-series, CUDA 13.1 is the right toolchain — Windows defaults install CUDA 12, and the gap is measurable. CachyOS (Linux) frees ~2–3 GB compared to Windows, both because the desktop is leaner and because Windows shared memory model means crossing the VRAM ceiling drops you into RAM-backed VRAM silently, with order-of-magnitude slowdowns that look like "today nothing works."

iGPU offload: if the CPU has an integrated GPU, switch monitors to it and dedicate the discrete GPU to inference. Recovers the 2–3 GB Windows + browser + chat clients consume. On Windows 10 the setting is "Settings → Display → Graphics settings → Add app → Power saving GPU"; on Windows 11 there's a dropdown to set default GPU per app.

### 2026 model list

Vyacheslav's recommendation for early 2026:

**For home (single 16 GB consumer GPU):**

- Gemma 4 (E2B/E4B/26B-A4B/31B)
- Qwen 3.5 (eight variants from 0.8B dense to 122B-A10B MoE)
- Qwen 3.6 (35B-A3B and now 27B Dense)

**For home+ (workstations):**

- MiniMax-M2.7
- Step-3.5-Flash
- GLM-5.1
- Mistral-Small-4-119B-2603
- Kimi-K2.6

**Just released, no GGUF yet:**

- Tencent Hy3-preview
- DeepSeek-V4-Pro / V4-Flash

Qwen3.6 35B-A3B and 27B benchmark differently — the MoE version is significantly better at programming but worse at general knowledge per the IKP framework (see [[incompressible-knowledge-probes]]). Gemma 4 26B-A4B is "in coding much worse" than the Dense 31B, per Vyacheslav — confirms the tradeoff structure even for the same model family.

## Connections

- [[local-llm-16gb-vram-tests]] — the companion tests article from the same author. This article is the primer; that one is the experiment.
- [[mixture-of-experts]] — extended with IKP's total-params-vs-active result
- [[llm-quantization]] — updated with the BF16 history, K vs I quant computation cost, the four dynamic-quant creators
- [[moe-cpu-offload]] — updated with `-fit` / `-cmoe` / `-ncmoe` mode breakdown
- [[speculative-decoding]] — the family of techniques covered in §"Speculative decoding overview"
- [[reap-expert-pruning]] — REAP standalone concept
- [[toolbox/ik-llama-cpp]] — the fork
- [[toolbox/llama-cpp]] — main project, updated with `-fit` mode reference
- [[unsloth]] — the most prominent dynamic-quant creator
- [[bf16-vs-fp16]] — the exponent-vs-mantissa decision
- [[reddit-qwen3-6-mtp-12gb]] — the practitioner thread that uses the operational knowledge from this primer