Entropic Bound Explains Transformer Capacity and Attention Rank

Researchers are exploring the fundamental question of minimum model capacity required for a given task, moving beyond neural scaling laws. They introduce the Entropic Bound, a spectral concept of task-intrinsic capacity specifically for Transformers. Initially, they prove that for a linear attention surrogate, the intrinsic rank (r*) of the token-mixing operator acts as a tight lower bound, meaning any model with lower rank incurs unavoidable risk, and this bound is achievable. They also show gradient descent can recover this rank and that r* is predictable from data before training. However, a direct transfer of this concept to real attention mechanisms proves problematic. The study identifies that the input-conditioned nature of attention's mixing operator, rather than softmax or rank constraints, is the reason a static weight kernel cannot adequately summarize capacity. To address this, an "attention-native intrinsic rank" is proposed, defined as the minimum query-key kernel rank that can realize the task within the attention class. With this new definition, the full Entropic Bound structure—including deficiency, achievability, and recovery—is restored for both linear and softmax attention, using the energy effective rank as a robust estimator. The work also maps the predictability frontier, showing that r* is exactly recoverable for linear QK attention even without the value map, while softmax attention allows only partial pre-training recovery due to its nonlinearities. These findings reframe the Entropic Bound from a descriptive metric to an attention-native capacity measure with a clearly defined predictability limit.