SelKV Optimizes LLM KV Cache, Boosting Long Context Performance

The memory footprint of the Key-Value (KV) cache in Large Language Models (LLMs) is a significant bottleneck, growing linearly with context length during autoregressive text generation. Existing compression methods often indiscriminately merge tokens, which can degrade representations and lead to "attention sag"—where merged tokens receive disproportionate softmax mass. A new training-free framework, SelKV, addresses these issues with a dual-component approach. SelKV first employs a soft cosine gate that adaptively decides whether to merge or drop tokens based on their value-vector similarity, thereby preserving semantic fidelity. Second, it introduces an attention-ratio compensation mechanism that applies a decoding-time logit bias, correcting the softmax imbalance caused by merging. Evaluated on the LongBench dataset, SelKV maintains strong compressed performance even when retaining only 25% of the KV cache. It proves particularly robust for grouped-query attention (GQA) models, achieving near-lossless generation quality and outperforming full-cache baselines on complex multi-document QA tasks, all while delivering a 3.3x decoding speedup at 100k tokens.