SPECTRA Compresses LLM KV Cache Beyond 2-Bit Limit

As Large Language Models (LLMs) process increasingly long inputs, their inference memory becomes heavily reliant on the Key-Value (KV) cache, which stores attention keys and values. This cache grows with context length, consuming significant GPU memory. Traditional compression methods, like uniform quantization, struggle to go beyond two bits per value without severe quality degradation, hitting a "2-bit cliff" due to the cache's outlier-heavy data. The core challenge is that raw KV cache channels are highly correlated, making it difficult to prioritize which dimensions to compress more aggressively. This research reveals that by transforming the cache into a decorrelated coordinate system, a small subset of channels emerges as carrying most of the critical information. This transformation provides the necessary handle to allocate bit budgets more effectively. Based on this insight, the team developed SPECTRA, a novel, training-free, and drop-in codec. SPECTRA re-encodes the KV cache into this new coordinate system, focusing compression efforts on the most informative channels. Tested on models like Llama-3.1-8B and Qwen2.5-7B, SPECTRA achieved near-lossless 4x compression, remained competitive at 8x where uniform quantization failed, and even reached 12x compression. This breakthrough allows LLMs to handle much longer contexts and larger inference batches within existing GPU memory constraints.