MosaicKV Compresses LLM KV Cache for Long Contexts

Serving large language models (LLMs) with extremely long contexts, often hundreds of thousands to millions of tokens, faces a significant bottleneck: the Key-Value (KV) cache. This cache grows linearly with context length, consuming vast GPU memory, limiting batch sizes, and reducing serving throughput. While previous compression techniques targeted only one dimension (sequence or channel), they offer limited gains for truly long contexts, and combining them directly leads to substantial accuracy loss. This paper introduces MosaicKV, a novel dynamic two-dimensional (2D) KV cache compression system specifically engineered for extremely long-context LLM serving. MosaicKV tackles the accuracy challenge by exploiting the non-uniform importance distribution within the KV cache. Instead of applying a uniform compression pattern, it dynamically identifies critical elements for each KV vector and selects appropriate compression strategies at a fine-grained segment level. To manage the overhead of fine-grained compression, MosaicKV incorporates a compressed KV cache management mechanism. This system intelligently utilizes underutilized GPU and CPU resources to maintain compressed caches and accelerate attention computation. Evaluations on an H800 GPU with multiple LLMs demonstrate impressive results: MosaicKV delivers up to 16x attention speedup, 4.8x lower decode latency, and 7.3x higher throughput, all while reducing memory usage by 3x and incurring only a minimal 1.76% average accuracy loss on standard benchmarks.