New Decoding Method Speeds Up Long-Context LLMs by 11.6x

Large Language Models (LLMs) face significant challenges when processing very long contexts, primarily due to memory bandwidth limitations and the quadratic computational cost of the attention mechanism during decoding. Existing solutions often involve trade-offs between memory overhead and computational efficiency. A new framework, Faster Flash Decoding (FFD), has been developed to tackle these issues through a novel hardware-algorithm co-design. FFD integrates the selector and computer components into a single, fused kernel, replacing external metadata with content-aware scanning using low-bit quantization. It also introduces a "top-delta" strategy for dynamic, distribution-adaptive sparsity without requiring global synchronization. This training-free, plug-and-play solution allows for the reuse of scanning results, leading to substantial performance gains. Empirical tests show FFD delivers up to an 11.6x kernel-level speedup and a 2.37x end-to-end throughput improvement, supporting context lengths up to 256K. Crucially, it maintains model accuracy while achieving high sparsity ratios, as validated on benchmarks like RULER and LongBench.