BinaryPC Boosts LLM Decoding Throughput with Sparse Attention.

Large Language Models (LLMs) with long context windows are increasingly common, but the self-attention mechanism, especially during decoding, remains a major efficiency bottleneck. This is primarily due to the need to repeatedly process and store ever-growing key-value (KV) caches. While existing sparse attention methods aim to reduce computation by attending to fewer KV pairs, they often compromise accuracy, require additional training, or rely on computationally expensive hashing techniques. Researchers have developed BinaryPC, a novel training-free and data-aware sparse attention method designed for long-context LLMs. Unlike traditional Locality-Sensitive Hashing (LSH) which uses data-independent random projections, or learned non-linear hashing, BinaryPC generates compact binary hash codes and corresponding hash functions by directly computing binary principal components of the data. This approach explicitly preserves the structural information of the data without requiring any gradient-based training. Extensive experiments across various model families and long-context benchmarks demonstrate that BinaryPC maintains accuracy comparable to full attention. Crucially, it achieves superior performance compared to other sparse and hashing-based baselines. On modern GPUs, BinaryPC significantly improves end-to-end decoding throughput by 3.56 times over the highly optimized FlashAttention kernel, offering a substantial efficiency gain for deploying long-context LLMs.