GraceKV Optimizes LLM Cache Compression with Global Allocation

Large Language Models (LLMs) face significant bottlenecks due to the storage and repeated access of their Key-Value (KV) cache, especially with increasingly long contexts. Current compression methods often rely on fixed rules, focusing either on token eviction or merging, which limits their ability to dynamically allocate cache resources across different layers, attention heads, and context slots. This also prevents a joint optimization of local resolution and overall information coverage. To overcome these limitations, a novel approach called GraceKV has been introduced. GraceKV redefines the compression process as a global resource allocation problem within a fixed cache budget. It treats each combination of layer, KV head, and context slot as an atomic unit, constructing a prototype tree where leaf nodes represent token-level KV entries and internal nodes compress covered KV space. The system adaptively determines resource allocation by allowing candidate actions (expanding coverage or improving resolution) to compete globally for the shared budget. This method requires no additional training and performs all compression and inference directly on the GPU. Extensive experiments across various long-context tasks and compression ratios demonstrate GraceKV's superior performance, ranking first in most settings and maintaining robustness even with 128-fold compression, validating its global budget allocation strategy.