New Attention Mechanism Improves AI Model Information Retention

This research identifies a limitation in standard attention mechanisms, specifically how weighted averaging can obscure the amount of accumulated evidence when patterns are repeated. This can lead to models achieving high task performance without necessarily retaining all prediction-relevant structural information in their internal representations. To address this, the authors introduce Mass-Aware Attention (MAA), a novel generalization of L1 normalization to an Lp family. MAA is designed to scale the numerator and denominator of attention at different rates when evidence patterns are repeated, thereby preserving the effective number of contributing inputs within the representation's magnitude. MAA requires no additional supervision, parameters, or hidden dimensions and reverts to standard attention at p=1. Across multiple dynamic graph models and datasets, MAA consistently improved future-link prediction accuracy and the linear recovery of basic graph statistics, demonstrating its effectiveness in enhancing the informativeness of AI model representations.