Deep Linear Transformers Exhibit Diverse Dynamical Behaviors

A new study delves into the inference-time behavior of deep linear encoder-only transformers, conceptualizing tokens as a system of interacting particles. The research demonstrates that within a two-dimensional embedding space, the dynamics of these transformers can be reinterpreted as a generalized Kuramoto-type model, characterized by pure second-harmonic coupling. This reformulation, leveraging Watanabe-Strogatz theory, indicates that the dynamics are inherently low-dimensional, irrespective of the specific key, query, and value matrices used. For certain token initializations, the parameter matrices induce a wide array of long-term behaviors, including token clustering, oscillations, and bifurcations, with oscillations and bifurcations linked to a newly discovered hidden Hamiltonian structure. The study also establishes that dynamics initialized near the Ott-Antonsen manifold exhibit similar long-term behaviors to those initialized precisely on it, suggesting robustness. Numerical experiments in higher dimensions, inspired by the two-dimensional theory, further imply that these characterized long-time behaviors persist, offering insights into the fundamental mechanisms governing transformer operations.