CompressKV Reduces LLM KV-Cache Memory for Long Context Inference

Large language models (LLMs) processing long contexts face significant memory and computational challenges due to their key-value (KV) caches. Current methods for managing these caches often evict critical tokens, degrading performance. Researchers have introduced CompressKV, a novel framework designed to optimize KV-cache compression for LLMs, particularly those based on Grouped-Query Attention (GQA). CompressKV addresses the limitations of prior approaches by focusing on "Semantic Retrieval Heads" (SRHs) within the attention mechanism. These SRHs are crucial for identifying and preserving tokens that are semantically important, including initial, final, and key mid-context evidence. The framework also intelligently allocates cache budgets across different layers based on estimated eviction errors. Experimental results demonstrate CompressKV's superior performance, maintaining over 97% of full-cache performance with only 3% of the KV cache on question-answering tasks and achieving 90% accuracy with just 0.7% KV storage in "Needle-in-a-Haystack" tests. This innovation offers a much-improved trade-off between resource efficiency and performance for long-context LLM inference.