# LLM Quantization Formats

Quantization compresses model weights from 16-bit floats to lower-precision numbers — usually 4-bit. Memory footprint drops 4×, and on supporting hardware, throughput can match. The catch is *how* you compress matters at least as much as the bit-width, and the format that ships with the most marketing copy is not always the format that wins on perplexity.

## The three formats from Vyacheslav's tests

### Q4_K_M — the established baseline

The standard GGUF format. Weights are grouped into 256-element blocks, each block carries its own scale and offset. Sensitivity-critical tensors (attention) stay at 6 bits; everything else drops to 4. No calibration data required. The `bartowski` GGUF builds use this format; it works predictably across architectures.

### UD-Q4_K_XL — Unsloth Dynamic 2.0

The newer-and-supposedly-smarter approach. Before quantizing, Unsloth pushes 300K to 1.5M tokens of real chat data through the model and measures per-layer sensitivity. Layers that are sensitive stay at 8 or 16 bits; the rest are pushed *more aggressively* to 4 bits. The pitch is "calibrated, data-driven, better quality at the same average bit-width."

### MXFP4 — Blackwell-native floating-point 4-bit

A floating-point 4-bit format (sign + small exponent + mantissa) instead of an integer one. Weights are grouped into blocks sharing an exponent. Nvidia's Blackwell tensor cores support MXFP4 natively — no emulation, direct hardware mapping. Gemma 4's GGUF uses this format and it works as intended.

## Where Dynamic 2.0 backfires

The "newer is better" intuition fails on Qwen MoE. Vyacheslav's perplexity measurements on WikiText-2:

| Format | Size | Perplexity | Degradation vs Q8_0 |
| --- | --- | --- | --- |
| Q8_0 (reference) | ~36.9 GB | 6.5342 | 0% |
| Q4_K_M | ~20.0 GB | 6.6688 | **+2.1%** |
| UD-Q4_K_XL | ~19.0 GB | 7.1702 | **+9.7%** |

For 1 GB less on disk, Dynamic 2.0 gives you nearly 5× the quality loss. The community consensus Vyacheslav cites is that Unsloth quants underperform specifically on MoE architectures with sensitive router dynamics — the aggressive 4-bit layers perturb routing decisions in ways that compound over long sequences.

Unsloth themselves later pulled MXFP4 layers from their Qwen GGUFs (Q2_K_XL, Q3_K_XL, Q4_K_XL) after reports of computational anomalies. So one of the marquee Dynamic 2.0 ingredients turned out to be incompatible with the dominant local MoE family. Q4_K_M kept working.

## Operational lesson

Don't pick a quant by feature list or by the publisher's claims. Check perplexity numbers for the *specific* model architecture you're running. The combination matters more than the format alone.

## A second measurement, different answer

[[habr-local-llm-quantization-deep-dive|Vyacheslav's deep-dive primer]] measures the same Qwen 3.6 35B-A3B quants by **KLD against the BF16 baseline** rather than perplexity on WikiText-2 and gets the opposite ranking:

- UD-Q4_K_XL **beats** Q4_K_M (UD-Q4_K_XL is 1.2 GB larger)
- Q4_K_M ≈ UD-Q3_K_XL (the dynamic Q3 is 4.4 GB smaller for the same KLD)
- UD-Q2_K_XL beats IQ2_M at the same byte budget

KLD vs BF16 is the more recent and architecture-specific measurement; the earlier perplexity number predates the BF16 baseline that newer quants are designed against. The takeaway sharpens: **the metric matters, the publisher matters, the architecture matters, the model version matters.** Don't pick a quant by its name. Find a measurement that compares the candidates against your reference precision on the specific model you intend to run.

## Beyond standard quants

Two additional quantization lines worth knowing about:

**IQK quants (ik_llama).** ikawrakow's fork ships SOTA quants not available in mainline llama.cpp. `IQ4_KS` on Qwen 3.5 shows ~3× better KLD than `Q4_K_M` at the same disk size and matches `UD-Q4_K_XL` quality at ~1 GB less. See [[toolbox/ik-llama-cpp]] and the [[unsloth]] entity for the alternative dynamic-quant publishers.

**TurboQuant (KV cache).** A rotate-then-quantize KV cache compression that hits 2–4 bits per coordinate at full-precision NiaH recall (4× compression). Standard llama.cpp ships `q4_0`/`q8_0`/`f16` KV cache; TurboQuant beats them on quality at the same byte budget. See [[random-rotation-quantization]] and [[interactive-turboquant]].

**Why I-quants can be slower than K-quants.** When a Q3 quant ships with I-quant blocks and the model is CPU-offloaded via `-cmoe`, the I-quant blocks dequantize at ~2× the CPU cost of K-quants. A larger Q4_K_M model can outrun a smaller UD-Q3_K_XL if the bottleneck is CPU expert layers. Quant *type* matters more than quant *size* when you're CPU-bound. See [[habr-local-llm-quantization-deep-dive]] for the worked example.

## Cross-references

- [[mixture-of-experts]] — explains why MoE is unusually sensitive to per-layer quantization
- [[local-llm-16gb-vram-tests]] — context where these quants were compared (perplexity metric)
- [[habr-local-llm-quantization-deep-dive]] — Vyacheslav's KLD-based measurement and the BF16 history
- [[lucumr-local-models]] — Ronacher's point that quantization choice is one of several invisible decisions that silently degrade local models
- [[bf16-vs-fp16]] — the precision baseline modern quants are measured against
- [[random-rotation-quantization]] — KV cache specifically; the rotate-then-quantize family
- [[toolbox/ik-llama-cpp]] — ikawrakow's fork with the SOTA IQK quants
- [[unsloth]] — the most prominent UD-Q*-XL publisher