New Side-Path Mitigates LLM Representation Collapse

Large Language Models (LLMs) often suffer from a fundamental issue known as representation collapse, which severely hampers their ability to process and understand long contexts. This problem manifests in two extremes: homogenization collapse, where tokens become too similar (e.g., attention sinks), and isolation collapse, where tokens lose connection (e.g., local attention). A spectral analysis of attention dynamics reveals an inherent trade-off between how efficiently information mixes and how much information capacity the model retains.To address this dilemma, a novel architectural intervention called the Topologically Regularized Side-Path (TRSP) has been introduced. TRSP is designed to be non-invasive, meaning it doesn't alter the core attention mechanism of the LLM. Instead, it uses a parameter-free Triangular Box mechanism, scaled by a lightweight gate, to regularize the interaction topology between tokens. This approach promotes a healthier geometric transition operator within the model.By integrating both proximal coupling to maintain effective rank and distal propagation to support non-degenerate mixing, TRSP effectively balances the spectral properties of the attention mechanism. Experiments demonstrate substantial improvements across general capabilities and, notably, on long-context benchmarks, with TRSP retaining high accuracy even at significantly increased training lengths compared to other advanced methods.