Maglev Introduces Sliding Recurrent Memory for Transformers

A new research paper introduces Maglev, an innovative recurrent Transformer architecture designed to enhance long-context processing while maintaining computational efficiency. Maglev features a fixed-size memory mechanism that extends the concept of sliding-window attention, crucially remaining parallelizable during the training phase. This architecture comprises two interconnected models: a "prefiller" (Q) that uses full or interleaved attention to generate memory targets, and a "decoder" (P) that relies on sliding-window attention and recurrent K/V injection to produce memories for predicting the next token. The training process for Maglev involves a memory consistency loss, which ensures that the decoder's memories align with those generated by the prefiller. This alignment allows the decoder (P) to operate independently during inference, leveraging its efficient sliding-window and recurrent mechanisms. Empirically, Maglev has demonstrated improvements in validation loss and performance on downstream pretraining benchmarks compared to existing sliding-window and latent recurrent transformer baselines. Furthermore, sharing parameters between the prefiller and decoder helps reduce memory footprint while largely preserving performance gains.