Induction Heads in Transformers Interpolate N-Grams for In-Context Learning

Induction heads are a fundamental component within transformer architectures, widely believed to be responsible for their impressive in-context learning capabilities. However, the precise statistical mechanisms they employ have remained somewhat opaque. This research delves into transformers trained on Markov chains to clarify how these heads function. The study identifies two complementary smoothing mechanisms at play. Firstly, at a finite attention-weight scale, induction heads act as a soft context-matching estimator. This means they aggregate information from both exact and partial context matches, weighting them based on their overlap. This process effectively creates a data-dependent interpolation across different context orders, akin to classical Jelinek-Mercer smoothing. Secondly, the presence of a beginning-of-sequence (BOS) token introduces additive pseudo-counts, which is analogous to Dirichlet-style smoothing. By constructing a disentangled transformer that explicitly implements these mechanisms, the researchers demonstrated that trained transformers naturally recover these predicted attention patterns. The findings bridge mechanistic interpretability with classical statistical smoothing, showing that transformers don't just count occurrences but actively learn to regularize their in-context estimations, often outperforming traditional count-based baselines.