PuzzleKV Compresses LLM KV Cache for Long Context Inference

Long-context inference in large language models (LLMs) is often bottlenecked by the substantial memory demands of the Key-Value (KV) cache. Existing KV cache compression techniques, such as global low-rank methods, may struggle to capture fine-grained information. This research introduces PuzzleKV, a novel approach that tackles this challenge by leveraging the observation of significant low-rank structure within fixed-length "logical pages" of the KV cache. PuzzleKV operates as a training- and calibration-free method, treating each completed page within a layer and KV head as an independent compression unit. It performs low-rank decomposition on these individual pages and computes attention directly over both dense and factorized pages. This incremental compression of newly eligible pages occurs dynamically during autoregressive decoding, optimizing memory usage on the fly. Experimental results across various models, context lengths, and benchmarks demonstrate PuzzleKV's effectiveness. At approximately 60% of the original KV cache storage, it achieved over 96% of the full KV performance. When combined with quantization, PuzzleKV could reduce storage to just 18.7% of the original while retaining over 93% performance, showcasing substantial gains over other methods like Global SVD.