TaskPress Compresses LLM KV Cache for Efficient Long-Context Inference

Large Language Models (LLMs) face significant memory constraints during long-context inference due to the linear growth of the Key-Value (KV) cache with sequence length. Existing pruning techniques often optimize for specific queries, limiting their reusability. A new framework, TaskPress, addresses this by offering task-guided, query-agnostic KV cache eviction. TaskPress constructs a compact, reusable memory representation based on a high-level task guide. This guide acts as a meta-query during the prefill stage, filtering out irrelevant tokens before specific downstream queries are processed. The system also efficiently identifies important tokens using quantization scale factors as a zero-cost signal. Experiments across various long-context tasks demonstrate that TaskPress effectively creates a compact and reusable cache, improving efficiency for diverse queries.