OptR Improves INT2 KV-Cache Quantization for LLMs.

The Key-Value (KV) cache is a significant bottleneck for memory and bandwidth in large language models (LLMs) that handle long contexts, making ultra-low-bit quantization increasingly vital. Existing INT2 quantization methods, which use rotation, typically optimize based on cache statistics or proxy errors before the final attention readout. However, these methods often overlook the error propagation through the attention mechanism and the output projection ($W_O$), which ultimately affects the model's output. To address this, researchers propose OptR, an "output-aware rotation" method that directly minimizes the attention-output error *after* the $W_O$ projection. OptR achieves this by decomposing the post-$W_O$ error into key- and value-induced terms and learning per-head orthogonal corrections across the full INT2 quantization and attention path. Additionally, OptR employs an attention-equivalent key reparameterization to mitigate large channel-wise offsets without altering the softmax distribution. Across various models and benchmarks, OptR consistently outperforms existing methods like QuaRot and OSCAR, enhancing long-context retrieval capabilities while maintaining compatibility with the paged KV-cache format and incurring negligible inference overhead.