CascadeFormer Optimizes Transformers with Depth-Tapered Architecture.

Deep Transformer models, despite their power, often suffer from inefficiencies where their deepest layers contribute minimally to overall performance. This research introduces two methods to address this: CascadeFormer and CascadeFlow Pruning, both stemming from a novel concept called Gradient Fan-in Asymmetry (GFA). CascadeFormer is an architecture that tapers its width as depth increases, aligning with the observed uneven information flow across layers. This design achieves comparable performance to uniform Transformer baselines while significantly reducing latency and increasing throughput. CascadeFlow Pruning, on the other hand, removes layers based on accumulated training gradients, outperforming standard pruning heuristics. The underlying principle, GFA, suggests that in Pre-LayerNorm residual stacks, the gradient at a given layer is a sum of an identity path and all downstream functional paths. This creates a gradient fan-in that decays linearly with depth, resulting in richer gradients for earlier layers and sparser ones for later layers. Empirical evidence supports GFA across various models, indicating that the structural asymmetry, rather than just gradient magnitude, is a key bottleneck for deeper layers.