Transformers Exhibit Stratified Computation Around Prediction Direction

New research delves into the internal workings of transformer models, uncovering a fundamental organizational principle within their residual streams. The study identifies a "prediction direction"—the unembedding direction of the token the model is currently predicting—as a critical, content-defined anchor. This anchor dictates a geometric and behavioral stratification of the residual stream. Across various transformer models, from 7B to 120B parameters, both dense and mixture-of-experts, a consistent pattern emerges: a narrow, scale-invariant "prediction interface" concentrates the structure most relevant for readout. In contrast, the much larger "prediction-distal complement" expands with model scale. Traditional variance-based analyses only partially capture this organization, especially with diverse prompts, because the prediction direction is nearly orthogonal to principal variance axes. The findings show a steep geometric gradient: regions close to the prediction are highly structured and group related prompts, while the complement is flatter and less discriminating. Disrupting directions near the prediction interface causes immediate and significant behavioral shifts, whereas disruptions further away have delayed effects. This suggests that while the interface is functionally decisive, the complement, despite being weakly readout-aligned per direction, carries significant causal and temporal load, with behavior driven by direction rather than magnitude.