KV-Cache Optimizations Benchmarked for Long-Context LLM Serving

Serving large language models (LLMs) with long input contexts is increasingly constrained by the rapid growth of the KV-cache, which stores key and value states. While various KV-cache compression techniques exist, comparing them has been difficult due to inconsistent evaluation methods across different models, tasks, and serving infrastructures. This paper introduces a comprehensive, workload-aware benchmark designed to evaluate representative KV-cache optimization mechanisms. These include techniques like quantization (e.g., KIVI, TurboQuant), pruning (e.g., SnapKV), and merging (e.g., CaM). The evaluation was conducted using Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3 across diverse long-context tasks such as multi-document QA, single-document QA, few-shot learning, and summarization. The benchmark measured critical metrics including task quality, average output throughput, time-to-first-token, and actual compression ratio across different context lengths. Key findings reveal that a high compression ratio does not always translate to superior end-to-end performance. For instance, KIVI4 demonstrated stable quality, SnapKV excelled in long-context throughput, and CaM showed significant gains on specific QA tasks but was sensitive to workload variations. These results emphasize the need for selecting KV-cache optimizations based on specific workload requirements rather than a universal approach.