Attention Heads Quantified for Multi-Task Computation Efficiency.

Multi-head attention layers are fundamental components in modern neural networks, particularly for generating vector representations that can support various downstream tasks. This research investigates the efficiency of these layers by quantifying the minimum number of attention heads needed for specific multi-task scenarios. In one scenario, the study examines the task of creating a vector representation from which linear predictors can simultaneously identify both the smallest and largest numbers in a list. It confirms that two attention heads are sufficient with modest embedding dimensions and precision. Crucially, the research proves that attempting this with a single attention head would necessitate an exponentially larger embedding dimension or precision level, highlighting the efficiency gains of multiple heads. A second scenario explores computing the XOR function of an $n$-bit string. The findings indicate that the product of the number of attention heads and the polynomial degree of the predictor must be at least $n$. The paper also constructs multi-head attention layers that achieve this lower bound, providing practical insights into designing efficient attention mechanisms for complex Boolean functions.