RaBitQCache Accelerates Long-Context LLM Inference with Quantization

Long-context Large Language Model (LLM) inference is frequently hampered by the substantial memory demands of its Key-Value (KV) cache. Current sparse attention techniques often fall short, either by employing static, fixed-budget retrieval (like Top-k) or by relying on proxy scores that are both computationally intensive and prone to bias. To overcome these critical limitations, researchers have developed RaBitQCache. RaBitQCache is a novel sparse attention framework that leverages randomized rotated binary quantization alongside high-throughput binary-INT4 arithmetic to efficiently estimate attention weights. This approach introduces an unbiased proxy score with a mathematically proven error bound, facilitating adaptive Top-p retrieval that dynamically adjusts the token budget based on the actual sparsity of attention. Furthermore, the system incorporates hardware-aware optimizations, including asynchronous pipelining and lazy updates, to effectively mask computational overhead. Evaluations confirm that RaBitQCache substantially boosts inference speed and reduces memory input/output operations, all while preserving the high quality of LLM generation, outperforming existing state-of-the-art methods.