RoPE-Aware Quantization Boosts KV-Cache Efficiency and LLM Performance

Large language models often face memory and speed bottlenecks due to the Key-Value (KV) cache, especially with long contexts. Existing quantization methods for KV-caches typically treat cached keys uniformly, which is suboptimal given the rotational positional embedding (RoPE) structure. RoPE's design means a key's contribution to attention logits varies across two-dimensional frequency blocks, making some blocks more sensitive to quantization errors. A new method, Block-GTQ, addresses this by implementing a RoPE-aware bit allocator for key-cache quantization. Built on TurboQuant-MSE, Block-GTQ assigns more bits to high-energy RoPE blocks that are more critical. This approach significantly preserves RoPE query-key logits, reducing per-layer Mean Absolute Error by 32-80% compared to uniform quantization. The practical benefits are substantial: Block-GTQ improves long-context retrieval, understanding, and reasoning. For instance, on Llama-3.1-8B-Instruct, it raised NIAH average from 70.6 to 97.4 and LongBench-EN from 36.87 to 53.31. It also enables 3.24x KV-cache compression with comparable quality to fp16, 1.34x faster inference at 128K context, and reduces peak memory from 56.31 GB to 19.85 GB, making 256K and 512K contexts feasible where fp16 would otherwise run out of memory.