SWave Retrospective Reveals Key Engineering Principles for Complex-Valued LMs.

SWave, a complex-valued recurrent language model, was developed based on the idea that representing language as complex waves could offer richer information encoding and better signal integrity over long contexts. Its architecture aimed to prevent state decay or explosion through a Cayley-parameterized unitary transition. During its development, the model underwent significant evolution. An initial "Resonance Head" design suffered from a failure mode called "cos-domination collapse," where the imaginary channel would collapse. This issue was resolved by replacing it with an untied head derived from the Phase-Associative Memory (PAM) architecture, enabling stable and extended training. The retrospective highlights several key engineering insights: components like ComplexNorm and the Wave Propagation Scan were consistently effective, while others, such as multi-scale retention concepts and certain auxiliary training objectives, proved non-essential. The study also offers a formal characterization of the collapse issue, a numerically stable parallel scan, six transferable engineering principles for complex-valued recurrent training, and a methodology for identifying structural divergences in code.