Study Compares Linear Attention Architectures for Long Context Models

Researchers have conducted a comprehensive comparative study of various attention mechanisms, focusing on the trade-offs between traditional softmax attention and four recent recurrent linear-attention architectures: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2. The study aimed to clarify their differences in terms of expressivity, how they manage memory decay, training throughput, and overall implementation complexity, particularly for models handling long sequence lengths where quadratic costs become prohibitive. The architectures were expressed using a common recurrent-memory notation to highlight their distinct operational principles. Experiments involved 350M-parameter models trained on 15 billion tokens, including comparisons of optimizers and learning rates, hybrid versus pure stack configurations, and sequence-length runtime measurements. Larger DeltaNet runs at 1.3B and 3B parameters were also conducted, alongside a small set of downstream evaluations. Key findings indicate that Kimi Delta Attention, when paired with the Muon optimizer, achieved the lowest final validation loss among the 350M-parameter models. Conversely, a pure Gated DeltaNet stack trained with AdamW demonstrated the highest normalized training throughput. Hybrid stacks generally improved loss at the cost of throughput, and Muon consistently outperformed AdamW in reducing final validation loss. The paper also explored lightweight cross-layer routing mechanisms for DeltaNet-style memories, finding that routing the write value into the aligned hidden stream (CLVR) yielded modest improvements.