FreqDepthKV Compresses LLM KV Caches, Boosts Long-Context Inference

A novel inference-time compression technique, FreqDepthKV, has been introduced to address the memory and bandwidth limitations of Key-Value (KV) caches in large language models (LLMs) during long-context inference. This method factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals. FreqDepthKV employs a lightweight online probe to dynamically assign attention heads to different cache modes (shared-depth, residual-depth, or exact) based on their contribution to attention logits, allowing the compression policy to adapt to varying prompt structures without requiring model retraining. Evaluations across multiple long-context benchmarks, including question answering, retrieval, summarization, and code generation, demonstrate that FreqDepthKV preserves task accuracy with significantly smaller cache budgets. It achieves substantial improvements in decoding throughput and reduces peak KV memory, offering an effective compression ratio of 3.9x.