PolyKV Optimizes LLM KV Cache Compression for Long Contexts

The memory footprint of the KV cache is a major bottleneck for large language models (LLMs) when processing long contexts. Existing compression techniques typically apply a single, uniform policy and budget across all transformer layers. This uniform approach overlooks the fact that different layers within an LLM contribute differently during the prefill and decoding phases, meaning they may require distinct compression strategies and varying cache capacities for optimal performance. PolyKV addresses this by introducing a layer-wise KV cache optimization framework. It intelligently routes each transformer layer to the most suitable KV compression policy based on specific layer-level signals. Concurrently, PolyKV assigns non-uniform cache budgets to each layer while adhering to a fixed total memory budget. This heterogeneous composition allows for a more efficient and tailored application of existing KV cache methods. Evaluations on models like LLaMA-3.1-8B and Qwen3-8B demonstrate PolyKV's effectiveness. Under the same average KV budget, PolyKV significantly recovers the performance gap between the strongest single-policy baseline and a full KV cache, showing improvements of 54.5% and 25.7% respectively. Across various budget settings, PolyKV consistently outperforms baselines, recovering 40.0%-54.5% of the FullKV gap, proving its value in enhancing long-context inference efficiency.