Complex-Valued Language Model SWave Undergoes Significant Architectural Evolution

This research paper provides a detailed account of the developmental journey of SWave, a recurrent language model that utilizes complex-valued representations for language processing. The model was initially conceived with the idea that complex waves could offer richer information encoding, that unitary transitions could prevent state instability, and that rotating hidden states could maintain signal integrity over long sequences. Over three distinct development phases, the SWave architecture underwent significant changes. Early components like the Resonance Head were found to suffer from a "cos-domination collapse" failure mode, leading to its replacement. The study highlights the successful retention of elements like ComplexNorm and Wave Propagation Scan, while other concepts, including multi-scale retention and certain auxiliary training objectives, were ultimately deemed ineffective. The investigation culminates in a formal description of the identified collapse mechanism, the introduction of a numerically stable parallel scan, and the distillation of six practical engineering principles specifically for training complex-valued recurrent models. It also proposes a plan-to-code traceability method to preempt structural divergences that traditional testing might miss.