Transformer FFN Linearity Varies, Not an Architectural Property

This research investigates the degree of linearity within the feed-forward network (FFN) blocks of Transformer models. By introducing a metric called linear recoverability (R^2_lin), which quantifies how much of an FFN block's output can be explained by a linear approximation, the study reveals that linearity is a highly variable and learned characteristic. The findings indicate that linearity is not a fixed architectural property or solely dependent on the activation function. Instead, individual FFN blocks within models like GPT-2 and Pythia-160m exhibit diverse linearity profiles, ranging from nearly linear to highly nonlinear, even between adjacent blocks. This suggests that the training process significantly shapes the computational nature of these components. The proposed measurement also offers practical implications for model compression, identifying blocks that are amenable to simpler linear replacements without substantial performance loss. It further highlights a methodological challenge in evaluating linear baselines for Transformer activations.