xHC Boosts Transformer Scaling Beyond Hyper-Connections.

Hyper-Connections (HC) have shown promise in scaling Transformers by expanding the residual stream into multiple parallel streams, offering a new dimension for memory scaling beyond traditional width and depth. However, existing HC methods, including Manifold-Constrained HC (mHC), typically hit a performance ceiling around four parallel streams (N=4) due to diminishing returns and rapidly increasing training costs. This research identifies two key bottlenecks: insufficient write-back information for a growing number of streams and a residual-mixing generation cost that scales cubically with N. To overcome these, the authors propose xHC (Expanded Hyper-Connections), which enables effective scaling beyond N=4, specifically demonstrating N=16. xHC integrates temporal feature augmentation to enrich write-back information and employs a sparse residual-stream architecture that updates only a subset (k=4) of the N=16 streams while maintaining dense access to the full residual state. Across 18B and 28B Mixture-of-Experts (MoE) models, xHC delivered consistent and strong downstream performance improvements, enhancing the average score by 4.0 points over mHC with only modest additional training FLOPs. Scaling-law experiments further showed that xHC is more compute-efficient, requiring less compute than vanilla and mHC to reach the same loss. Additionally, xHC-Flash is introduced to reduce memory traffic, making large-N residual-stream expansion practical for LLM pre-training.