TurboQuant Explained

A technical guide for ML engineers — based on arXiv:2504.19874 (ICLR 2026)

The KV Cache Problem

In autoregressive LLM inference, every generated token attends to all previous tokens via the attention mechanism. To avoid recomputing these attention keys and values at every step, they are cached — this is the KV cache.

For a model like Llama 3 70B running at 32K context with a batch of 8:

The bottleneck isn't compute — modern GPUs have plenty of TFLOPS. The bottleneck is memory bandwidth. Every new token requires reading the entire KV cache from HBM through the memory bus. For a 128K context, this means transferring hundreds of gigabytes per second just for attention.

Quantizing the KV cache to 3–4 bits reduces this bandwidth by 4–5× and directly accelerates token generation speed — not just memory usage.

KV Cache Growth vs Context

KV cache memory grows linearly with context length. TurboQuant at 3-bit reduces memory by ~6x compared to FP16 across all context sizes.

Quantizing the KV cache to 3–4 bits reduces this bandwidth by 4–5× and directly accelerates token generation speed — not just memory usage.

Why Existing Methods Fall Short

Existing KV cache quantization methods face two fundamental challenges:

The two key requirements for practical KV cache quantization are: (1) online operation — vectors must be quantized token-by-token as they arrive, with no offline calibration; and (2) unbiased inner products — quantization error must not systematically bias attention scores, or model quality degrades. No prior method satisfies both simultaneously at near-optimal distortion rates.

TurboQuant: Core Idea

TurboQuant's key insight: high-dimensional vector quantization is hard, but 1D scalar quantization is solved. If you can transform a correlated high-dimensional vector into independent scalar components, you can apply optimal 1D quantizers to each dimension independently.

Step 1: Random Rotation (PolarQuant)

Raw KV vectors have strongly correlated coordinates — some dimensions dominate, others carry little information. Quantizing such vectors directly wastes bits on redundant structure.

TurboQuant multiplies every KV vector by a pre-generated random orthogonal matrix Π:

After rotation, by the Johnson-Lindenstrauss lemma, each coordinate of y becomes approximately independent and follows a Beta distribution with known parameters. This transforms the problem from correlated d-dimensional quantization into d independent 1D quantization problems.

Step 2: Scalar Quantization

After rotation, each coordinate yᵢ is independently quantized using a Lloyd-Max optimal codebook — the theoretically optimal scalar quantizer for the known Beta distribution.

This achieves the Shannon-optimal distortion rate for scalar quantization. The codebook is pre-computed offline (one-time cost) and reused for all vectors.

Step 3: Inner Product Correction (QJL)

PolarQuant minimizes MSE, but quantization error introduces a systematic bias into inner product estimation. For attention: ⟨q, k̂⟩ ≠ ⟨q, k⟩. Accumulated over millions of tokens, this bias degrades model quality.

QJL (Quantized Johnson-Lindenstrauss) adds a 1-bit correction sketch:

The sketch s stores only m sign bits per vector (m ≪ d, typically 32–64 bits regardless of vector dimension). The total overhead is negligible — less than 0.5 bits per element in practice.

Performance Results

Bit Width vs Precision Tradeoff

TurboQuant achieves near-Shannon optimal distortion, maintaining high cosine similarity even at very low bit widths. At 3-bit, cosine similarity remains above 0.98.

Reference Implementation

def turboquant_encode(x, bits):
    # Step 1: Random rotation
    R = random_orthogonal_matrix(x.shape[-1])
    x_rotated = x @ R.T
    
    # Step 2: Scalar quantization (MSE-optimal)
    x_quantized = scalar_quantize(x_rotated, bits=bits)
    
    # Step 3: QJL correction for inner products
    sign_sketch = qjl_sketch(x_rotated)
    
    return x_quantized, sign_sketch, R

Use Cases

Quick Reference