QEvict Improves LLM Long-Context Decoding with Recoverable KV Cache.

Large Language Model (LLM) inference, especially with long contexts, is often bottlenecked by the memory footprint of the Key-Value (KV) cache. Current approaches to reduce this footprint typically evict tokens deemed unimportant based on attention scores. However, this "delete-or-retain" decision is irreversible and problematic because token importance can drift as new queries are generated during decoding. This "attention drift" causes standard policies to discard states that later become crucial, leading to information loss. To address this, researchers introduce QEvict, a novel three-tier KV-cache management scheme. Instead of binary eviction, QEvict employs a recoverable eviction strategy. It maintains high-confidence windows in full precision, stores intermediate windows in a quantized, recoverable tier, and only permanently deletes the lowest-confidence windows. During the decoding process, cumulative attention scores continuously update window importance. If a previously quantized window regains importance, it is dequantized and promoted back to the full-precision tier. This design allows QEvict to preserve a broader historical context under a fixed memory budget while ensuring that the most critical regions retain exact full precision. Experiments across various long-context benchmarks demonstrate that QEvict consistently outperforms existing eviction and quantization baselines, significantly reducing missed attention and improving overall information retention.