New Fast Weight Attention Improves Continual Learning in Neural Networks

The research explores a new framework for "fast-weight attention" within recurrent sequence models, specifically designed to enhance continual learning capabilities. This approach focuses on how models can efficiently compress an expanding context into a fixed-size recurrent state, effectively making the state transition an online learning rule. The paper details the derivation of normalized first-order updates for both squared-error regression and negative inner-product objectives, leading to variants like Falcon-1, Falcon-2, and Falcon-3, along with their inner-product counterparts. These variants are presented in recurrent, masked-parallel, and chunk-parallel forms, incorporating numerically stable positive-decay renormalization. The practical implications are significant: these methods remain competitive in language modeling tasks and demonstrate improved length extrapolation, particularly in tasks like variable-digit addition. This framework offers a structured way to manage temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent neural networks, addressing key challenges in continuous learning from sequential data.