LiteTopK Kernel Boosts Sparse Attention in LLMs and Vector Databases

The paper presents LITETOPK, an innovative kernel designed to optimize the Indexer-TopK operation, which is critical for sparse attention mechanisms in large language models and vector retrieval. Current GPU-based methods suffer from high memory traffic and synchronization costs. LITETOPK addresses these inefficiencies by exploiting the "curse of dimensionality," where high-dimensional vector distances tend to cluster. The core idea involves sampling data to estimate score ranges and then partitioning results into bins online. This allows LITETOPK to maintain a precise approximate threshold, only writing back promising candidates, thereby significantly reducing I/O and memory overhead while ensuring exact Top-k correctness. Experimental results demonstrate a 1.2x acceleration in the prefill stage of GLM 5.2 in real-world scenarios, alongside lower memory consumption.