Counter-Causal Surprise Improves LLM Key-Value Cache Management

This paper introduces an innovative approach to managing the key-value (KV) cache in large language models (LLMs), a critical component whose size grows linearly with context length and can quickly exhaust GPU memory. Efficient KV cache management is essential for reducing memory footprint and accelerating inference, especially for long prompts or generated outputs. The proposed method, termed "Counter-Causal Surprise," operates on the insight that past tokens which are highly predictable from more recent tokens are redundant and can be safely removed from the cache. To score entries for eviction, the model is run on tokens in their original order, reusing existing KV cache representations, but applying a counter-causal attention mask. This mask allows each position to attend only to its future context, identifying predictable past information. To further optimize cost, a fast single-layer approximation is also introduced, restricting the counter-causal pass to the last transformer layer for significant speedup with minimal accuracy loss. The strategy was evaluated on various open-source LLMs and benchmark datasets, demonstrating competitive or superior performance compared to other state-of-the-art methods in reducing KV cache size and improving inference speed.