Hierarchical Global Attention Boosts Long-Context Transformers

A novel attention mechanism called Hierarchical Global Attention (HGA) has been introduced as a direct replacement for dense causal attention in large language models (LLMs) designed for long contexts. HGA is particularly notable because it can be integrated into existing pretrained transformers without any retraining or modification of the original model parameters, making it a highly practical solution for extending context windows. HGA addresses the significant memory demands of long-context transformers by implementing a two-level hierarchical routing strategy. It first identifies relevant "chunks" of information using compact summaries and then refines this selection by routing only the most pertinent groups of tokens for exact, token-level attention. This approach drastically reduces the amount of data transferred to GPU memory, allowing models to process much longer contexts—up to 64K tokens on a single 32GB GPU—where traditional methods would fail due to memory constraints. The method stores the full historical key/value pairs in host RAM or NVMe storage, only bringing a small, routed working set to the GPU. Despite this aggressive sparsity (around 3%), HGA maintains attention quality very close to dense attention, with a minimal quality gap. This suggests that the hierarchical routing introduces negligible approximation errors, making it a powerful tool for scaling LLMs to unprecedented context lengths on consumer-grade hardware.