Token Influence in LLMs Decays by Power Law, Not Exponentially

New research investigates how the influence of input tokens diminishes over distance within autoregressive Transformer language models, impacting subsequent token predictions. Drawing inspiration from operator learning and differential equation solvers, the study empirically measured token dependencies using gradient profiles. Experiments conducted on models like Pythia and Qwen2.5-0.5B demonstrated that the sensitivity of a token's influence on later tokens follows a power-law decay, not an exponential one. This means that the impact of distant tokens persists more significantly than previously assumed. This power-law behavior was observed in coherent text and notably, even when syntax was disrupted, indicating it's a fundamental learned property of trained Transformers, unlike randomly initialized models. These findings suggest that large language models have developed internal mechanisms, possibly hierarchical or coarse-level, to effectively manage and exploit these long-range, slowly decaying dependencies. This understanding could inform future architectural designs for more efficient and powerful language models.