KVpop Compresses AI Model Caches, Boosting Performance and Efficiency

Large language models face a significant bottleneck in memory and bandwidth due to the growing size of key-value (KV) caches during autoregressive decoding. Traditional methods for managing these caches often rely on static rules or proxy scores, which are not always effective at identifying which tokens will be most useful in the future. A new approach, KVpop, addresses this by learning an adaptive eviction policy. It directly supervises the decision to keep or drop KV cache entries, using a novel "future-attention" target for training. This allows the system to predict the utility of tokens more accurately without needing to compute dense attention maps. KVpop also incorporates a delayed memory-based scorer, which uniquely defers scoring to leverage near-future context. Testing on mathematical reasoning tasks, KVpop achieved 98% of full-attention performance with 75% KV cache compression on Qwen3-4B, and even stronger results on Qwen3-8B, demonstrating substantial memory savings without sacrificing quality.