New Frustrated Synchronization Network Outperforms Transformers in Text.

This research introduces a new neural network architecture called the Frustrated Synchronization Network (FSN), which reinterprets attention mechanisms through the lens of frustrated synchronization. Unlike traditional synchronization, where perfect agreement yields no further computation, the FSN leverages structured departures from agreement to perform its calculations. In this model, token states are represented as phases on a torus, and the core computational pathway involves a learned complex coupling kernel incorporating harmonics and a one-step delay. The "frustration" in the FSN comes from several components: static Kuramoto-Sakaguchi angles, repulsive Daido components, and a delay term that links tokens to their successors, effectively implementing next-token prediction as synchronization frustrated by the input data. This novel design allows the FSN to achieve superior performance compared to tuned RoPE-SwiGLU transformers. Specifically, at a one-million-parameter scale and equivalent training budgets, the FSN consistently demonstrated lower validation loss on character-level text and code benchmarks. Its performance advantage persisted even when the transformer baseline was trained to convergence, with FSN runs converging to a lower loss. The FSN's advantage also holds at larger scales, up to four million parameters, suggesting a promising alternative to current transformer architectures.