Causal Evidence Improves Sparse Attention Routing in LLMs

Sparse attention mechanisms are crucial for reducing the computational cost of processing long contexts in large language models (LLMs) by allowing queries to focus on selected input parts. These selectors are typically trained by mimicking the attention patterns of a dense teacher model, assuming that attention directly indicates the context the teacher uses. However, this research challenges that assumption. Through experiments on retrieval tasks where the exact evidence for an answer is known, the study found a significant discrepancy between where a model attends and what information it causally depends on. Attention can be drawn to outdated or irrelevant facts that the model has learned to ignore, and attention patterns can even vary across training runs despite reliance on the same core evidence. When selectors were trained on these attention patterns, their performance suffered. The researchers demonstrated that by supervising sparse attention routing with "causal evidence sets" – identified by masking parts of the context and observing changes in the answer – accuracy dramatically improved. For instance, on a two-step reference task, a selector trained on attention achieved 41% accuracy, while the same selector trained on causal evidence reached 99%, matching the teacher model. This method requires no manual annotation, as evidence sets can be recovered from a frozen teacher through masking alone. This suggests that focusing on true causal dependence rather than mere attention patterns is a more effective strategy for developing efficient and accurate sparse attention mechanisms.