New Quantization Method Boosts Low-Bit KV-Cache Accuracy

Long-context large language models (LLMs) rely on KV caches to manage memory and avoid redundant computations during inference. However, quantizing these caches to very low bitrates, such as one-bit, drastically reduces model quality, leading to significant accuracy drops. This research identifies that the primary cause of this degradation is not absolute logit error, but rather structured local misranking within the top-K logit distribution. To address this, the paper introduces Local Distribution Restoration, a new policy gradient method. This technique actively detects steps where the quantized logits pose a high risk of local distribution drift. When such a risk is identified, it selectively restores the distribution of the top-K candidate logits before token selection, effectively correcting the misranking. Implemented as DGAP, this method demonstrates remarkable improvements. On Llama-3.1-8B, 1-bit KV-cache quantization accuracy, which typically drops to 47.8%, is recovered to 83.2%. This recovery is achieved while maintaining the substantial memory footprint reduction of low-bit quantization and incurring only a modest decoding overhead, making efficient long-context LLM inference more viable.