Interactive TurboQuant — Rotate, Then Quantize

arkaung's interactive page walks through the math behind TurboQuant — a KV-cache and embedding compression family that hits 2–4 bits per coordinate without per-block metadata, without calibration, without training. The construction has a clean structure: rotate the input vector by a random orthogonal transform, then quantize each rotated coordinate against a single fixed codebook designed once for the post-rotation distribution. The page builds the construction from primitives (vectors, MSE, unbiased estimators, Lloyd–Max), justifies each step against the prior literature, and ends at the practical KV-cache result on Llama-3.1-8B.

For the standalone concept page, see random-rotation-quantization. This summary captures what arkaung's tutorial uniquely adds: the lineage, the worked steps, and the empirical floor against alternatives.

The eight primitives

Skippable for anyone who has done linear algebra: vector, length, inner product, MSE, unbiased estimator, rotation, Central Limit Theorem, high-dimensional concentration ($1/\sqrt{d}$ per-coordinate std of unit vectors), scalar quantization (one extra bit quarters the squared error).

The load-bearing one is high-dimensional concentration. A uniform point on the unit sphere in d dimensions has every coordinate distributed with mean 0 and standard deviation $1/\sqrt{d}$. At d = 1000 the standard deviation of any single coordinate is ~0.032. This is why one fixed grid works after a rotation: the rotation puts every input vector onto the sphere of radius $|x|_2$, and the marginal of any coordinate of a uniform point on that sphere is a known Beta density that converges to Gaussian as d grows.

The adversarial coordinate

Real trained embeddings are not flat. A few coordinates dominate ("outlier channels"). A fixed $[-L, L]$ grid either clips the outliers or wastes resolution on the bulk. Production quantizers (GPTQ, AWQ, KIVI, KVQuant) work around this by storing per-block scale + zero-point in full precision. At b=3 bits with block-size s=16, that overhead works out to b + 32/s = 5 effective bits per value, a 66% surcharge over the nominal b.

A random rotation smears the outlier across every coordinate. After rotation, no coordinate is an outlier — the magnitude is split roughly equally. The fixed grid now works without per-block metadata. Same storage cost as the naive approach, same reconstruction quality as the per-block approach.

The lineage

DRIVE introduced the construction for one-bit federated mean estimation. EDEN generalized to b bits with the Lloyd–Max codebook approach. RaBitQ arrived from a different research line (approximate nearest-neighbor search) with the same rotate-then-sign idea. TurboQuant is EDEN with the per-vector scale fixed to a constant, repackaged for KV-cache compression and inner-product retrieval. The relationship between the two lines is the subject of an ongoing public discussion in 2026 (arXiv 2604.18555 and 2604.19528).

The decoder split

The rotation is shared. The decoder is what varies. The same rotated coordinates feed three different reconstructions:

• Mean decoder (DRIVE): Store $\mathrm{sign}(\Pi x)$ and a scalar $S = |x|^2 / |\Pi x|_1$. Returns an unbiased estimate of $x$ itself.
• Inner-product decoder (RaBitQ / QJL): Store $\mathrm{sign}(\Pi x)$ and $|x|$. At query time, estimate $\langle q, x\rangle$ from $\langle \Pi q, \mathrm{sign}(\Pi x)\rangle$ with a normalizing scalar.
• MSE decoder (EDEN / TurboQuant): Snap each $(\Pi x)_i$ to the nearest of $2^b$ Lloyd–Max centroids from the universal codebook, then apply $\Pi^\top$ to recover $\tilde x$.

The page follows the MSE branch, which is what KV-cache compression uses.

The bias problem and TurboQuant-prod

MSE-optimal quantizers underestimate inner products in a measurable, fixable way: the Lloyd-Max codebook stores the conditional mean of each cell, which is smaller in magnitude than the cell's extreme values, so the reconstruction is a shrunken version of the input. Inner products against a shrunken reconstruction come out smaller. If the bias is a known constant, multiply it out. This is the construction that gives TurboQuant-prod its inner-product unbiasedness.

The residual chain in §7-8 is the same trick applied twice: quantize with a biased (b−1)-bit code, then quantize the residual with an unbiased 1-bit code (QJL). Two codebooks, one chain, b bits total.

Empirical floor

Needle-in-a-Haystack recall on Llama-3.1-8B-Instruct at 4× memory compression (TurboQuant paper Fig. 4):

TurboQuant matches full precision at 4× compression. SnapKV and PyramidKV (the two non-quantization methods, which prune attention keys instead) lose 10-14 pp. KIVI (integer quantization with per-block scales) loses 1.6 pp.

On LongBench-V1: TurboQuant at 3.5 bpv matches the full-precision average (50.06). At 2.5 bpv (6.4× compression), it stays within ~1% of full precision (49.44 vs 50.06).

Where this shows up in practice

The Reddit thread on Qwen3.6 + MTP on 12GB mentions --cache-type-k turbo4 --cache-type-v turbo3 — that's a llama.cpp fork (llama-cpp-turboquant) shipping the TurboQuant codebook as a KV cache type. The Still-Notice8155 comment runs Qwen3.6 at 128K on a GTX 1070 8GB partly because turbo4 brings the KV cache from ~720 MB to ~590 MB at 131K. Standard llama.cpp ships q4_0 / q8_0 / f16 KV cache types; TurboQuant is the next step out on quality at the same byte budget.

Connections

• random-rotation-quantization — the standalone concept page
• llm-quantization — KV cache and weight quantization in general; TurboQuant is one branch
• kv-cache-sizing — q4_0 vs q8_0 trade; TurboQuant sits at the same byte budget with strictly better recall
• reddit-qwen3-6-mtp-12gb — practical use of turbo4 on a GTX 1070
• arkaung-blog — the author's site