Fast LapSum Enables Differentiable Top-k at Million Scale

The "top-k" operation is a fundamental component in modern sparse computation, crucial for tasks such as token routing, expert activation, memory selection, and attention pruning in large-scale models. However, traditional hard top-k operations block gradient flow, while existing continuous (soft) relaxations are often too computationally expensive for practical use in large models. This creates a bottleneck for training and optimizing sparse AI architectures. Researchers have introduced Fast LapSum, a groundbreaking exact-budget soft top-k primitive designed to overcome these limitations. Fast LapSum is unique in that it preserves an exact selection mass of 'k' while remaining fully differentiable from end-to-end. Its GPU solver operates in linear time after an initial sorting step, making it highly efficient. The solver combines a linear-time threshold computation with an analytical vector-Jacobian product. For extremely large scales, it employs probabilistic bracketing to sort only the uncertain middle band of kernel-noised scores, further optimizing performance. This results in almost negligible overhead, processing millions of scores in milliseconds. Fast LapSum makes exact soft top-k practical for various demanding applications, including generating megapixel sparse adversarial examples with significant speedups and training fully differentiable sparse image coders from scratch.