Kernel Attention Faces Exponential Feature Rank Challenge

Attention mechanisms are fundamental to modern neural networks, with full attention considering every token pair and kernel attention compressing sequences into a fixed-dimensional "sketch." This paper reveals a critical distinction between these two approaches, particularly concerning their feature requirements. The researchers show that this difference becomes exponentially significant even for very short sequences. Specifically, for tasks like Min-IP over Boolean inputs, rank-one normalized kernel attention can exactly solve sequences of length up to two. However, when faced with sequences of just three tokens containing two competing candidates, any single normalized nonnegative kernel-attention head requires an exponential number of features (2^Ω(m)) to achieve an error rate strictly below 1/2. This holds true even with flexible token-wise values and query-dependent readouts. In contrast, dense softmax attention can solve the same three-token task with only m-dimensional scores and constant temperature, demonstrating its superior efficiency. The study concludes that as context length increases, the lower bound for kernel attention's feature requirement approaches an exact 2^m realization, indicating a fundamental scaling limitation. This research provides crucial insights into the architectural trade-offs and inherent limitations of different attention mechanisms.