Faster Query-Key Learning Sharpens Transformer Attention

This paper investigates the dynamics of self-attention layers in Transformers, focusing on how different parameterizations of the query-key and output-value circuits influence attention patterns. The researchers found that certain configurations, particularly those inducing faster learning in the query-key circuit compared to the output-value circuit, result in sharper attention focused on task-relevant tokens. This sharpening occurs even while maintaining similar training loss. Through gradient-flow analysis, the study reveals that factorization implicitly rescales the learning rates of these two circuits. The derived closed-form dynamics show that query-key and output-value parameters move along a line, with their relative speeds dictated by their respective learning rates. Consequently, a faster query-key learning rate prompts the model to compensate for slower output-value learning by increasing attention mass on crucial tokens, thereby improving attention interpretability without sacrificing predictive performance.