SemKV Quantizes KV Cache for Efficient Long-Context LLM Inference.

The Key-Value (KV) cache is a major memory bottleneck in large language models (LLMs), especially with long context lengths, as its size grows linearly. This research reveals a "quality cliff" phenomenon: uniform KV quantization works well down to a certain bit depth (e.g., 2.322 bits for Llama-3.1-8B) but then abruptly collapses below it. This cliff highlights the limitations of simple uniform quantization. To overcome this, SemKV proposes a semantic mixed-precision quantization approach. Instead of uniform precision, SemKV preserves all tokens and ranks them by a model-internal importance score, then assigns two adjacent above-cliff precisions. This method effectively interpolates the precision grid, achieving an average precision that uniform quantization cannot. SemKV demonstrates a 6.0x storage reduction without any statistically detectable quality difference compared to full FP16 KV cache, even outperforming FP16 token pruning with a larger memory budget. Further improvements are possible by using distortion-optimized quantizers, pushing the no-loss operating point to 7.9x reduction. The key is to first identify the quality cliff for a specific deployment and then interpolate above it with mixed precision.