KV-Cache Eviction Methods Misrepresent Performance, Study Finds.

Research into KV-cache eviction strategies for large language models (LLMs) has uncovered a critical issue: many reported performance gains are not due to superior eviction logic but rather to how memory budgets are implicitly handled. The study found that methods often store per-head selections as masks, effectively retaining the entire cache rather than freeing memory, which inflates their perceived efficiency. When a strict memory budget is enforced, the performance of these methods drops significantly. The analysis shows that simply keeping the largest attention weights is already near-optimal, closing only a small gap to full attention. This suggests that the true differentiator among eviction methods lies in their memory management, not just their selection criteria. To address this, the paper introduces ContourKV, a novel training-free allocator based on the "dropped-mass" statistic. ContourKV demonstrates superior performance, winning against state-of-the-art methods in a majority of comparisons, especially when operating under realistic memory constraints. This highlights the importance of rigorous evaluation that accounts for actual memory usage.