Gaussian Mixture Attention Offers Linear-Time Scaling for Long Contexts

Standard dot-product attention, a core component of Transformer models, presents a significant bottleneck when processing very long sequences due to its quadratic scaling in computation and memory. To address this, a new mechanism called Gaussian Mixture Attention (GMA) has been developed. GMA re-imagines sequence mixing by routing queries and keys through a fixed number of learned Gaussian mixture components, rather than performing explicit pairwise comparisons. In GMA, queries and keys are transformed into "responsibility" vectors within a shared latent space. The interaction between these vectors implicitly defines an affinity, and values are then processed via a K-slot latent memory. A key innovation is that GMA leverages matrix multiplication associativity to avoid constructing the full N x N affinity matrix, thereby achieving linear memory scaling (O(NK) instead of O(N^2)) for a fixed K. The paper details both bidirectional and causal versions of GMA, along with a differentiable parameterization for its Gaussian components. Empirical evaluations show that GMA successfully achieves its intended linear memory scaling and performs competitively against other attention-style baselines in long-context classification. While causal GMA improves upon some linear attention variants, it currently trails highly optimized causal SDPA and Mamba. The analysis of learned responsibilities also suggests that GMA offers a probabilistic and interpretable alternative for long-context processing.