Google TurboQuant: 3-Bit KV Cache Compression Explained

TL;DR

Last updated October 7, 2026.

Google TurboQuant is a quantization algorithm that shrinks the KV cache of an LLM to about 3 bits per value with no training or calibration step. The paper (arXiv 2504.19874, an ICLR 2026 paper) reports quality neutrality at 3.5 bits per channel and marginal degradation at 2.5 bits. Google's announcement measured at least a 6x smaller KV cache and up to 8x faster attention-logit computation for 4-bit keys on H100 GPUs.

What Is Google TurboQuant?

TurboQuant comes from Amir Zandieh and Vahab Mirrokni's group at Google Research. The arXiv paper, TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate, was first posted April 28, 2025 by Zandieh, Majid Daliri, Majid Hadian, and Mirrokni. Google's blog post on March 24, 2026 brought it to a wide audience, alongside the two building blocks it uses: QJL (June 2024) and PolarQuant (February 2025, an AISTATS 2026 paper).

The problem it targets is memory overhead in ordinary quantization. Block-wise schemes store a scale and zero point in full precision for every small block of numbers, and Google estimates that bookkeeping adds 1 or 2 bits per number. At a 3-bit budget that overhead eats a large share of the savings. TurboQuant removes the per-block constants entirely.

The paper also proves a lower bound on the distortion any vector quantizer can reach, and shows TurboQuant lands within a constant factor of about 2.7 of that bound at every bit width. That guarantee holds for any input, because the method never looks at the data distribution before quantizing.

How TurboQuant Works: PolarQuant and QJL

Stage one is a random rotation. Multiplying a vector by a random orthogonal matrix spreads its energy evenly, and in high dimensions each coordinate of the rotated vector follows a concentrated Beta distribution whose shape is known ahead of time. Because the distribution is fixed, TurboQuant precomputes an optimal scalar quantizer (a codebook) for each bit width and applies it to every coordinate independently. Nothing data-dependent gets stored next to the codes.

Google's blog describes this stage through PolarQuant, which converts pairs of coordinates into a radius and an angle. After rotation the angles cluster in a predictable range, so the grid boundaries are known before any data arrives and the normalization step that block quantizers need disappears.

Stage two fixes a subtle problem. A quantizer tuned for mean squared error gives biased estimates of dot products, and attention scores are dot products between queries and cached keys. TurboQuant spends 1 more bit per coordinate on a Quantized Johnson-Lindenstrauss transform of the residual: it projects the leftover error and keeps only the sign of each projected value. QJL pairs that 1-bit sketch with a full-precision query, which makes the inner-product estimate unbiased.

TurboQuant Benchmarks

Google evaluated TurboQuant, PolarQuant, and QJL on LongBench, Needle In A Haystack, ZeroSCROLLS, RULER, and L-Eval using open models from the Gemma, Mistral, and Llama families. The needle test from the paper is the clearest single comparison. Every method ran at a memory compression ratio of 0.25, meaning it kept a quarter of the full KV cache.

TurboQuant matched the uncompressed cache exactly. Token-eviction methods (SnapKV, PyramidKV) lost more because they drop whole tokens, and a needle sitting in a dropped token cannot be recovered. On LongBench the paper ran TurboQuant at 2.5 and 3.5 bits during streaming generation, splitting channels into outlier and normal sets and giving outliers more bits, which is where the fractional bit widths come from.

For speed, Google measured attention-logit computation against an optimized JAX baseline: 4-bit TurboQuant keys ran up to 8x faster than 32-bit unquantized keys on H100 accelerators. Smaller keys mean less memory traffic, and decode-time attention is bound by memory bandwidth.

TurboQuant KV Cache Memory Savings

KV cache size per token is 2 × layers × KV heads × head dimension × bytes per value. Llama-3.1-8B has 32 layers, 8 KV heads, and a head dimension of 128, so it stores 65,536 numbers per token. The table below applies that to a full 128K-token context. It counts the quantized values only and leaves out small metadata such as vector norms. Our KV cache guide walks through the formula for other models.

Going from BF16 to 3.5 bits frees about 12.5 GiB per long sequence on an 8B model. On a server that memory turns into more concurrent requests or longer contexts on the same GPU, which is where the cost savings show up.

TurboQuant in vLLM

vLLM added TurboQuant as an attention backend in pull request 38479, merged April 15, 2026. It quantizes at store time with fused Triton kernels and needs no offline calibration or change to model weights. The implementation follows the paper's rotation idea with a Walsh-Hadamard transform and random sign flips, uses Lloyd-Max scalar quantization for keys, and stores values with uniform quantization. Pull request 39931 (merged May 5, 2026) added hybrid models and uniform quantization.

vllm serve Qwen/Qwen3-4B --kv-cache-dtype turboquant_k8v4

# keep boundary layers at full precision
vllm serve Qwen/Qwen3-4B --kv-cache-dtype turboquant_k8v4 \
  --kv-cache-dtype-skip-layers 0,1,34,35

The PR's throughput tests on four RTX PRO 6000 Blackwell GPUs put turboquant_k8v4 at 79% to 100% of baseline output tokens per second and turboquant_4bit_nc at 71% to 96%. Decode-heavy requests lose the most, landing at 65% to 79% of baseline across presets. Long prefill runs at parity: the 8,192-in, 64-out scenario hit 100% of baseline with k8v4. The trade is capacity for per-request speed, so TurboQuant suits workloads limited by KV memory, such as long agent contexts and high concurrency.

TurboQuant in llama.cpp and SGLang

llama.cpp mainline does not support TurboQuant as of October 7, 2026. The feature request opened March 25, 2026 was closed as not planned. Mainline did merge pull request 21038 on April 1, 2026, which adds a Hadamard rotation to the KV cache before quantization and cites TurboQuant. The TurboQuant+ project describes rotation plus the stock q4_0 cache as TurboQuant's rotation stage. The PolarQuant codebook and the rest of the method still live in forks.

SGLang received four TurboQuant KV cache pull requests between March and April 2026, including a fused Triton version reporting 3.88x compression. All four were closed without being merged.

TurboQuant GitHub Implementations

Google published the method through the paper and blog. Most code people use comes from independent repositories that appeared within days of the March announcement. Star counts below were checked October 7, 2026.

TurboQuant for Vector Search

The same quantizer compresses embeddings in a vector database. In the paper, TurboQuant beat the PQ and RabbiQ baselines named in Google's post on 1@k recall on GloVe (d=200), while indexing time dropped to nearly zero because there is no codebook to train on the dataset.

Qdrant 1.18 ships TurboQuant as a production quantization option. Qdrant's May 13, 2026 benchmarks found 4-bit TurboQuant within about 1 to 2 percentage points of its scalar quantization (which compresses 4x), and 2-bit and 1-bit TurboQuant giving higher recall than binary quantization at the same storage budget.

TurboQuant vs FP8 KV Cache and KIVI

FP8 is the safe default: half the memory and the widest kernel support. TurboQuant is the option when 2x is not enough and you need 4x or more. Against KIVI, a widely used 2-bit KV cache quantizer, the paper's needle test at a quarter of full memory scored 0.997 for TurboQuant and 0.981 for KIVI on Llama-3.1-8B-Instruct.

Pros and Cons

FAQ