Low-Rank Attention Residuals Improve LLM Performance and Efficiency.

This paper introduces Low-Rank Attention Residuals (LR-AttnRes), an architectural modification for large language models (LLMs) aimed at improving performance and efficiency. Traditional Attention Residuals replace the standard fixed residual sum with a depthwise attention mechanism over outputs from previous sub-layers. However, these often use full-dimensional keys and values, coupling routing with representation and making routing scores scale with the model's hidden width. LR-AttnRes addresses this by maintaining full-dimensional residual values but employing significantly lower-dimensional keys for routing. This design effectively decouples the routing mechanism from the content of the residual, allowing for more efficient and focused routing decisions. The research explores two variants: Projected LR-AttnRes, which learns low-rank keys from existing output projections, and Sliced LR-AttnRes, which uses a slice of the value dimensions as the key, further reducing FLOPs. Comprehensive experiments demonstrate that LR-AttnRes achieves better validation loss compared to other variants. The findings confirm that effective depthwise routing can be accomplished with far fewer dimensions than the model's full width, leading to performance improvements while potentially reducing computational overhead. The authors have released code and models to support further research and adoption.