# Interactive TurboQuant — Rotate, Then Quantize

arkaung's interactive page walks through the math behind TurboQuant — a KV-cache and embedding compression family that hits 2–4 bits per coordinate without per-block metadata, without calibration, without training. The construction has a clean structure: rotate the input vector by a random orthogonal transform, then quantize each rotated coordinate against a single fixed codebook designed once for the post-rotation distribution. The page builds the construction from primitives (vectors, MSE, unbiased estimators, Lloyd–Max), justifies each step against the prior literature, and ends at the practical KV-cache result on Llama-3.1-8B.

For the standalone concept page, see [[random-rotation-quantization]]. This summary captures what arkaung's tutorial uniquely adds: the lineage, the worked steps, and the empirical floor against alternatives.

## The eight primitives

Skippable for anyone who has done linear algebra: vector, length, inner product, MSE, unbiased estimator, rotation, Central Limit Theorem, high-dimensional concentration ($1/\sqrt{d}$ per-coordinate std of unit vectors), scalar quantization (one extra bit quarters the squared error).

The load-bearing one is high-dimensional concentration. A uniform point on the unit sphere in *d* dimensions has every coordinate distributed with mean 0 and standard deviation $1/\sqrt{d}$. At d = 1000 the standard deviation of any single coordinate is ~0.032. This is why one fixed grid works after a rotation: the rotation puts every input vector onto the sphere of radius $\|x\|_2$, and the marginal of any coordinate of a uniform point on that sphere is a known Beta density that converges to Gaussian as *d* grows.

## The adversarial coordinate

Real trained embeddings are not flat. A few coordinates dominate ("outlier channels"). A fixed $[-L, L]$ grid either clips the outliers or wastes resolution on the bulk. Production quantizers (GPTQ, AWQ, KIVI, KVQuant) work around this by storing per-block scale + zero-point in full precision. At b=3 bits with block-size s=16, that overhead works out to b + 32/s = 5 effective bits per value, a 66% surcharge over the nominal b.

A random rotation smears the outlier across every coordinate. After rotation, no coordinate is an outlier — the magnitude is split roughly equally. The fixed grid now works without per-block metadata. Same storage cost as the naive approach, same reconstruction quality as the per-block approach.

## The lineage

| Idea | First introduced in |
| --- | --- |
| Random rotation + post-rotation Beta density analysis | DRIVE (Vargaftik et al., NeurIPS 2021) |
| Randomized Hadamard transform as the cheap practical rotation | DRIVE 2021; EDEN 2022 |
| Lloyd–Max codebook on $\mathcal{N}(0,1)$ for the rotated coordinates | EDEN (Vargaftik et al., ICML 2022) |
| Unbiased rotation-then-quantize via per-vector scale | DRIVE 2021, §4.2, Theorem 3 |
| The *b*-bit pipeline with codebook fixed | EDEN 2022 |
| Residual chain: biased (*b*−1)-bit + unbiased 1-bit | TurboQuant (Zandieh et al., 2025) |
| KV-cache and inner-product application framing | TurboQuant 2025; QJL (2024) for 1-bit |
| Sign + norm encoding for ANN inner-product retrieval | RaBitQ (Gao & Long, SIGMOD 2024) |
| Asymptotic optimality against Alon–Klartag bound | RaBitQ extended (Gao et al., 2024) |

DRIVE introduced the construction for one-bit federated mean estimation. EDEN generalized to *b* bits with the Lloyd–Max codebook approach. RaBitQ arrived from a different research line (approximate nearest-neighbor search) with the same rotate-then-sign idea. TurboQuant is EDEN with the per-vector scale fixed to a constant, repackaged for KV-cache compression and inner-product retrieval. The relationship between the two lines is the subject of an ongoing public discussion in 2026 (arXiv 2604.18555 and 2604.19528).

## The decoder split

The rotation is shared. The decoder is what varies. The same rotated coordinates feed three different reconstructions:

- **Mean decoder (DRIVE):** Store $\mathrm{sign}(\Pi x)$ and a scalar $S = \|x\|^2 / \|\Pi x\|_1$. Returns an unbiased estimate of $x$ itself.
- **Inner-product decoder (RaBitQ / QJL):** Store $\mathrm{sign}(\Pi x)$ and $\|x\|$. At query time, estimate $\langle q, x\rangle$ from $\langle \Pi q, \mathrm{sign}(\Pi x)\rangle$ with a normalizing scalar.
- **MSE decoder (EDEN / TurboQuant):** Snap each $(\Pi x)_i$ to the nearest of $2^b$ Lloyd–Max centroids from the universal codebook, then apply $\Pi^\top$ to recover $\tilde x$.

The page follows the MSE branch, which is what KV-cache compression uses.

## The bias problem and TurboQuant-prod

MSE-optimal quantizers underestimate inner products in a measurable, fixable way: the Lloyd-Max codebook stores the conditional mean of each cell, which is *smaller in magnitude* than the cell's extreme values, so the reconstruction is a shrunken version of the input. Inner products against a shrunken reconstruction come out smaller. If the bias is a known constant, multiply it out. This is the construction that gives TurboQuant-prod its inner-product unbiasedness.

The residual chain in §7-8 is the same trick applied twice: quantize with a biased (*b*−1)-bit code, then quantize the residual with an unbiased 1-bit code (QJL). Two codebooks, one chain, *b* bits total.

## Empirical floor

Needle-in-a-Haystack recall on Llama-3.1-8B-Instruct at 4× memory compression (TurboQuant paper Fig. 4):

| Method | NiaH |
| --- | --- |
| Full cache (FP16) | 0.997 |
| SnapKV | 0.858 |
| PyramidKV | 0.895 |
| KIVI | 0.981 |
| PolarQuant | 0.995 |
| **TurboQuant** | **0.997** |

TurboQuant *matches* full precision at 4× compression. SnapKV and PyramidKV (the two non-quantization methods, which prune attention keys instead) lose 10-14 pp. KIVI (integer quantization with per-block scales) loses 1.6 pp.

On LongBench-V1: TurboQuant at 3.5 bpv matches the full-precision average (50.06). At 2.5 bpv (6.4× compression), it stays within ~1% of full precision (49.44 vs 50.06).

## Where this shows up in practice

The Reddit thread on [[reddit-qwen3-6-mtp-12gb|Qwen3.6 + MTP on 12GB]] mentions `--cache-type-k turbo4 --cache-type-v turbo3` — that's a llama.cpp fork (`llama-cpp-turboquant`) shipping the TurboQuant codebook as a KV cache type. The Still-Notice8155 comment runs Qwen3.6 at 128K on a GTX 1070 8GB partly because turbo4 brings the KV cache from ~720 MB to ~590 MB at 131K. Standard llama.cpp ships q4_0 / q8_0 / f16 KV cache types; TurboQuant is the next step out on quality at the same byte budget.

## Connections

- [[random-rotation-quantization]] — the standalone concept page
- [[llm-quantization]] — KV cache and weight quantization in general; TurboQuant is one branch
- [[kv-cache-sizing]] — q4_0 vs q8_0 trade; TurboQuant sits at the same byte budget with strictly better recall
- [[reddit-qwen3-6-mtp-12gb]] — practical use of turbo4 on a GTX 1070
- [[arkaung-blog]] — the author's site