Renormalization Group Analyzes Transformer Attention's Data Relevance

This paper employs Wilsonian renormalization group theory to examine the Transformer's attention mechanism, treating it as a perturbation to a trained MLP residual-stack fixed point. The core question is whether attention acts as a relevant, marginal, or irrelevant operator, depending on the characteristics of the input data. The findings indicate that attention's relevance is tied to the spectral structure of the data-generating process. For data with long correlation lengths, attention is strongly relevant, closing performance gaps that MLPs cannot, and causing significant representational shifts. Conversely, for data with short correlation lengths, attention is irrelevant, as the Transformer converges to similar performance and fixed-point geometry as an MLP. The study also highlights the disproportionate impact of the first-layer head in driving these representational changes.