LLM Quantization Formats

Quantization compresses model weights from 16-bit floats to lower-precision numbers — usually 4-bit. Memory footprint drops 4×, and on supporting hardware, throughput can match. The catch is how you compress matters at least as much as the bit-width, and the format that ships with the most marketing copy is not always the format that wins on perplexity.

The three formats from Vyacheslav's tests

Q4_K_M — the established baseline

The standard GGUF format. Weights are grouped into 256-element blocks, each block carries its own scale and offset. Sensitivity-critical tensors (attention) stay at 6 bits; everything else drops to 4. No calibration data required. The bartowski GGUF builds use this format; it works predictably across architectures.

UD-Q4_K_XL — Unsloth Dynamic 2.0

The newer-and-supposedly-smarter approach. Before quantizing, Unsloth pushes 300K to 1.5M tokens of real chat data through the model and measures per-layer sensitivity. Layers that are sensitive stay at 8 or 16 bits; the rest are pushed more aggressively to 4 bits. The pitch is "calibrated, data-driven, better quality at the same average bit-width."

MXFP4 — Blackwell-native floating-point 4-bit

A floating-point 4-bit format (sign + small exponent + mantissa) instead of an integer one. Weights are grouped into blocks sharing an exponent. Nvidia's Blackwell tensor cores support MXFP4 natively — no emulation, direct hardware mapping. Gemma 4's GGUF uses this format and it works as intended.

Where Dynamic 2.0 backfires

The "newer is better" intuition fails on Qwen MoE. Vyacheslav's perplexity measurements on WikiText-2:

For 1 GB less on disk, Dynamic 2.0 gives you nearly 5× the quality loss. The community consensus Vyacheslav cites is that Unsloth quants underperform specifically on MoE architectures with sensitive router dynamics — the aggressive 4-bit layers perturb routing decisions in ways that compound over long sequences.

Unsloth themselves later pulled MXFP4 layers from their Qwen GGUFs (Q2_K_XL, Q3_K_XL, Q4_K_XL) after reports of computational anomalies. So one of the marquee Dynamic 2.0 ingredients turned out to be incompatible with the dominant local MoE family. Q4_K_M kept working.

Operational lesson

Don't pick a quant by feature list or by the publisher's claims. Check perplexity numbers for the specific model architecture you're running. The combination matters more than the format alone.

A second measurement, different answer

Vyacheslav's deep-dive primer measures the same Qwen 3.6 35B-A3B quants by KLD against the BF16 baseline rather than perplexity on WikiText-2 and gets the opposite ranking:

• UD-Q4_K_XL beats Q4_K_M (UD-Q4_K_XL is 1.2 GB larger)
• Q4_K_M ≈ UD-Q3_K_XL (the dynamic Q3 is 4.4 GB smaller for the same KLD)
• UD-Q2_K_XL beats IQ2_M at the same byte budget

KLD vs BF16 is the more recent and architecture-specific measurement; the earlier perplexity number predates the BF16 baseline that newer quants are designed against. The takeaway sharpens: the metric matters, the publisher matters, the architecture matters, the model version matters. Don't pick a quant by its name. Find a measurement that compares the candidates against your reference precision on the specific model you intend to run.

Beyond standard quants

Two additional quantization lines worth knowing about:

IQK quants (ik_llama). ikawrakow's fork ships SOTA quants not available in mainline llama.cpp. IQ4_KS on Qwen 3.5 shows ~3× better KLD than Q4_K_M at the same disk size and matches UD-Q4_K_XL quality at ~1 GB less. See ik-llama-cpp and the unsloth entity for the alternative dynamic-quant publishers.

TurboQuant (KV cache). A rotate-then-quantize KV cache compression that hits 2–4 bits per coordinate at full-precision NiaH recall (4× compression). Standard llama.cpp ships q4_0/q8_0/f16 KV cache; TurboQuant beats them on quality at the same byte budget. See random-rotation-quantization and interactive-turboquant.

Why I-quants can be slower than K-quants. When a Q3 quant ships with I-quant blocks and the model is CPU-offloaded via -cmoe, the I-quant blocks dequantize at ~2× the CPU cost of K-quants. A larger Q4_K_M model can outrun a smaller UD-Q3_K_XL if the bottleneck is CPU expert layers. Quant type matters more than quant size when you're CPU-bound. See habr-local-llm-quantization-deep-dive for the worked example.

Cross-references

• mixture-of-experts — explains why MoE is unusually sensitive to per-layer quantization
• local-llm-16gb-vram-tests — context where these quants were compared (perplexity metric)
• habr-local-llm-quantization-deep-dive — Vyacheslav's KLD-based measurement and the BF16 history
• lucumr-local-models — Ronacher's point that quantization choice is one of several invisible decisions that silently degrade local models
• bf16-vs-fp16 — the precision baseline modern quants are measured against
• random-rotation-quantization — KV cache specifically; the rotate-then-quantize family
• ik-llama-cpp — ikawrakow's fork with the SOTA IQK quants
• unsloth — the most prominent UD-Q*-XL publisher