ATFlash Boosts LLM Inference Speed with Per-RoPE-Wavelength Attention

This research introduces ATFlash, a new technique designed to enhance the computational and memory efficiency of large language model (LLM) inference, particularly for long contexts. The core innovation is a "per-RoPE-wavelength distance window" that intelligently prunes query-key inner-product terms within the attention mechanism. This pruning is based on the observation that each 2D-rotation frequency pair in Rotary Position Embeddings (RoPE) has a wavelength that limits its positional discrimination. By pruning terms beyond a wavelength-proportional distance, ATFlash achieves significant reduction rates (37-48% on native context lengths) without sacrificing accuracy, maintaining 96-98% top-1 match rate and low KL divergence on benchmarks like LongBench-v2. The method is input-independent and compatible with existing sparse attention techniques. Implemented with minimal changes to FlashAttention-4 and FlashInfer, ATFlash demonstrates speedups up to 1.29x on Llama models at 128K context and 1.31x on Qwen2.5-7B-1M at 1M-token context, making long-context LLM inference more practical.