Evaluating Quantized Models for Deployment

ByteShape's index page for a three-part series on picking a quantized model. The complaint it opens with is that quants get compared on one convenient number — model size, bits per weight, perplexity, KL divergence, or tokens per second — and each of those answers a real question while none answers the deployment question, which is which quant to run on your hardware. This page covers the series overview; the three parts themselves were not ingested, and the evidence summarized here is ByteShape's own.

The framing splits measurement into three categories that get confused with each other. Fit metrics are model size, bits per weight, and context-memory requirements. Quality measurements are downstream task benchmarks. Deployment measurements are token-generation throughput on the hardware you will actually use. Fit decides which models are possible; quality and measured throughput decide which of the possible ones are practical.

Part 1 adds a distinction that does most of the work in the rest of the series: downstream quality is not the same thing as fidelity to a reference model. Perplexity and KL divergence measure how far a quant has moved from its BF16 baseline. Neither measures whether the quant answers correctly, follows instructions, or writes working code.

Fidelity metrics rank the wrong thing

Part 2 tests that distinction. Across the full model cohort, perplexity and KLD do look predictive of benchmark quality — and ByteShape's argument is that the correlation is carried almost entirely by the clearly degraded quants at one end. Among models that stay close to baseline, which is the set you are actually choosing from, the ranking signal collapses.

They report testing 14 fidelity-metric variants, varying the dataset, the context length, the aggregation method, and whether scoring covers the response only. The result held across all of them: fidelity metrics detect large departures from BF16 and do not reliably order near-baseline quants.

The mechanism they give for why is the clean part. At the prompt level, KLD says how often a quant's output distribution differs from BF16. It says nothing about whether those differences help, hurt, or change nothing. Their phrasing: KLD measures displacement, benchmark quality depends on the direction of that displacement.

Bits per weight ranks the wrong thing too

Part 3 makes the same shape of argument on the speed axis. Model size and BPW tell you whether a quant fits and give a broad performance trend, but among similarly sized quants they do not predict measured token-generation speed. Actual throughput depends on the quantization format, tensor shapes and group sizes, kernel support, memory-access behaviour, CPU and GPU architecture, workload and context length, and the inference engine's implementation. A smaller quant can be slower than a larger one, and a format that wins on one GPU can rank differently on another device. BPW measures storage cost.

The series ties both halves together with a scatter plot on an RTX 6000 in which each quant appears twice: a hollow circle at its proxy-ranked position, where lower KLD implies better quality and lower BPW implies more speed, and an arrowhead at its measured position, where a downstream benchmark score sets the quality rank and measured throughput sets the speed rank. If the proxies reproduced deployment ranking the two would overlap. Instead the arrows are long and point in varied directions — vertical movement is KLD disagreeing with benchmark quality, horizontal movement is BPW disagreeing with measured throughput, and some quants move substantially on both. The exact ordering is hardware-specific to that card; the general claim is that the quant winning the proxies is often not the quant winning in deployment.

The procedure that falls out

Filter by fit first, eliminating anything that exceeds available memory at the required context length. Benchmark what survives on tasks resembling the intended workload. Then measure throughput on the actual inference engine, hardware, context length, and generation settings. The choice is the quant that fits, performs well enough, and runs fast enough for the work.

How this sits against the rest of the quantization notes

llm-quantization catalogs the formats and already carries the finding that how you compress matters at least as much as the bit-width — Q4_K_M against UD-Q4_K_XL against MXFP4, where the newest and best-marketed format does not automatically win on perplexity. ByteShape's contribution is the next step out: even perplexity is the wrong tiebreaker once the candidates are all close to baseline. habr-local-llm-quantization-deep-dive is the practitioner version of the same reality, where offload flags, expert pruning, and the choice of inference build move real throughput in ways no bits-per-weight figure anticipates. random-rotation-quantization is one of the formats whose per-block metadata story would show up on the horizontal axis of that scatter.

The pattern generalizes past quantization. local-ai-is-not-opus is the same error made with a leaderboard instead of a proxy metric — a single SWE-Bench number standing in for a deployment question it does not cover, in a language and concurrency model the benchmark never exercises. scaffold-model-fit is the third instance: a coding-agent score measures model times scaffold, not the model, so the number does not transfer to a different harness. All three are the same failure, which is treating a screening metric as a ranking. For MoE models specifically, mixture-of-experts explains why the fit axis stops being a single number at all, since total parameters and active parameters pull memory and compute in different directions.

Series index: byteshape.com/blogs/Evaluating-Quantized-Models, published 2026-07-15.