Positional Encoding Shapes Attention Head's Spectral Algebra and Function.

The study delves into the fundamental mechanisms of attention heads in transformer models, specifically examining how positional encoding schemes influence their spectral algebra. Attention heads utilize a bilinear form, and their learned operator (M) is generally non-symmetric, leading to a complex eigenspectrum. The researchers investigated what this spectrum encodes across various pretrained models and positional schemes, including RoPE, learned-absolute, and ALiBi. They found that the positional scheme dictates the default spectral behavior of an attention head. For instance, RoPE-based models exhibit spectrally rotational previous-token heads, while learned-absolute and ALiBi models show non-rotational, content-like behavior. This distinct "fingerprint" emerges dynamically during training, rather than being a pre-imposed blueprint. The research also demonstrated that while specific spectral channels are not strictly necessary for capability (models can reroute around bans), the cost structure reveals each scheme's inherent bias, with RoPE maintaining directional routing even with symmetric operators due to its phase channel.