BF1: Efficient Sparse Attention for Long-Context Transformers

Large Language Models (LLMs) often struggle with the computational cost of dense causal attention when processing very long contexts, even with highly optimized kernels. This research presents BF1, a novel deterministic block-aligned dyadic sparse-attention mechanism aimed at making long-context transformers more efficient. BF1 intelligently combines a small exact local neighborhood, a global first block, and logarithmically spaced historical blocks to manage attention over extended sequences. The BF1 approach is related to prior log-sparse and dilated attention patterns, but its key contribution lies in a correctness-gated retrofit for pretrained models, a matched topology-control study, and a detailed systems characterization. For a fixed block width, each converted layer uses O(n log n) token interactions and has O(log n) graph communication depth, significantly reducing computational load compared to the quadratic scaling of dense attention. Benchmarked on an NVIDIA RTX PRO 6000 Blackwell GPU, an optimized BF16 implementation shows substantial speedups, crossing dense attention performance between 2K and 4K tokens and achieving a 10.91x per-layer prefill speedup at 32K tokens. Retrofitting a Qwen3-0.6B model with BF1 in eight of its 28 attention layers reduced whole-model time to first token by up to 15.3% at 32K, while maintaining competitive perplexity. This establishes BF1 as a valuable sparse operator for real-world long-context applications.