NOVA-KV Optimizes LLM KV Cache with Attention-Preserving Quantization

Decoding large language models (LLMs) with long contexts is often bottlenecked by the bandwidth required to load the Key-Value (KV) cache. Reducing the cache size is crucial for improving decoding speed and serving capacity, but this must be done without compromising attention product accuracy or incurring high reconstruction costs. Existing quantization methods for KV caches often fall short, either being data-oblivious or using suboptimal transforms and scalar quantizers. This research formulates KV cache quantization as a transform coding problem, where the primary distortion criterion is the error in attention products. The authors derive closed-form optimal transforms for keys and values based on calibration statistics, showing that the optimal key transform is non-orthogonal and converts attention-aware distortion into mean-squared error (MSE) in the transform domain. This allows for the use of MSE-optimal vector quantizers directly on the transformed coefficients. To meet fixed-width layout requirements, the method groups coefficients into equal-volume partitions, enabling equal-size codebooks to achieve variable-rate optimality. The resulting method, NOVA-KV, recovers most of the long-context retrieval accuracy lost by scalar quantization methods at comparable throughput, demonstrating a significant advancement in efficient LLM serving.