Gzip Compression Guides Parameter-Free Adaptive Sparse Attention

Adaptive sparse attention masks are known to outperform fixed patterns and even dense attention for long sequences by selectively focusing on relevant parts of the input. However, existing adaptive methods typically introduce additional learnable parameters, require custom gradient estimators, or rely on specialized CUDA kernels, adding complexity and computational overhead. This research proposes a remarkably simple yet effective solution: using classical data compression, specifically per-block gzip compression ratios, as a signal for content selection. The core idea is that content blocks that gzip cannot compress efficiently likely contain information not predictable from local repetition, making them ideal targets for long-range attention. This compression-based approach allows the sparse mask to adapt dynamically to the input content without any additional learned parameters, auxiliary losses, or custom kernels. In experiments on PG-19 byte-level language modeling with 92 million parameters and an 8K context, the method achieved 1.71 bits-per-byte (BPB), significantly outperforming dense attention (2.89 BPB) and other sparse attention baselines like BigBird (2.34 BPB) and Longformer (3.21 BPB). The performance advantage grows with sequence length, and convergence is 3.3 times faster, demonstrating a highly efficient and effective way to implement adaptive sparse attention.