SAKI Improves LLM KV Cache Efficiency for Long Contexts

Current low-rank Key-Value (KV) cache methods for large language models (LLMs) often focus on preserving model weights or key variance, which do not directly align with the attention scores used during inference. This misalignment can lead to suboptimal performance in long-context retrieval. A new approach, SAKI (Score-Aware Low-Rank Key Indexing), addresses this by deriving an objective that directly minimizes expected attention score distortion during rank-r key compression. SAKI offers a training-free KV cache index that directly preserves attention scores, leading to improved top-k recall. Experiments across various LLMs, including LLaMA 3.1 8B and Qwen 2.5 7B, show that SAKI consistently outperforms key PCA at all tested ranks. It significantly reduces recall error, particularly in deeper layers, demonstrating its effectiveness in optimizing for the attention score objective rather than just key reconstruction.