Query Visibility Impacts KV-Cache Compression Performance Rankings

The effectiveness of KV-cache compression methods, vital for reducing memory in large language models, is heavily influenced by the evaluation protocol. Researchers conducted an audit comparing six published compression techniques under two scenarios: query-aware (where the query is known during compression) and query-agnostic (where compression happens before the query is seen). The study found that the rankings of these methods change considerably when evaluated in a query-agnostic manner, which is more representative of real-world deployment where a compressed cache is reused for multiple queries. For instance, methods like SnapKV, which perform well in query-aware settings, underperform trivial baselines when the query is hidden during compression. KeyDiff was identified as the only audited method that consistently outperformed trivial baselines in the query-agnostic protocol. The performance drop between the two protocols correlated with how much each method's scoring signal relied on query visibility, highlighting a critical distinction for practical applications.