Tracing LLM Behavior to Training Data with Next-Token Distributions.

This research delves into the fundamental relationship between a Large Language Model's (LLM) output behavior and the data it was trained on. Specifically, the study examines how closely an LLM's predicted next-token distribution aligns with the "empirical next-token distribution" (ENTD), which represents the actual next-token probabilities observed in the original training corpus for a given context. The ENTD is a crucial benchmark because it represents the ideal, unrestricted minimum for the cross-entropy loss function used during pretraining. The findings indicate that for a substantial portion of inputs, the LLM's distribution aligns almost perfectly with the ENTD, and this agreement generally improves with increased model scale and training compute. However, the study also identifies a significant "long tail" of input sequences where the LLM and ENTD diverge considerably. The authors explore potential reasons for these discrepancies, including aspects of the transformer architecture, the training procedure, and inherent noise in the ENTD estimation itself. This work encourages a new direction in "data-centric mechanistic interpretability," aiming to understand how model behaviors emerge directly from the training data rather than solely focusing on learned weights.