The Future of LLM Compression: Beyond Weights and Activations

For years, LLM compression focused on weight quantization (GPTQ, AWQ) and activation pruning. TurboQuant opens a new frontier: vector-level quantization of the KV cache.

The Evolution of Compression

Weight quantization reduces model size but doesn't help inference memory after loading. Activation quantization helps during training but has limited impact on inference. KV cache quantization directly targets the memory bottleneck during long-context generation — the fastest-growing use case in production.

Why TurboQuant is Different

Unlike prior KV cache quantization methods that require per-model calibration or training, TurboQuant is:

• Online — quantizes vectors as they arrive, no offline phase
• Training-free — no model modification needed
• Unbiased — QJL correction preserves inner product accuracy
• Near-optimal — achieves Shannon distortion limit