Linear Transformers Improve In-Context Learning Efficiency

Large language models based on transformers excel at in-context learning, adapting to new tasks without parameter updates by leveraging their attention mechanisms. However, the quadratic computational and memory demands of traditional softmax transformers, which scale with context length, pose a significant bottleneck for processing extensive data. Linear transformers were introduced to mitigate this by reducing complexity to a linear dependence on context length, but their theoretical underpinnings, particularly regarding feature mapping in linear attention, have been less clear. This research delves into the approximation and generalization capabilities of linear transformers within a domain generalization framework. It reveals that these models achieve in-context learning by effectively learning a mapping from context distributions to corresponding response functions. The study provides a dimension-independent convergence rate for generalization analysis, highlighting a trade-off between the regularity of data distributions and latent features. These theoretical insights offer a new perspective on designing activations and loss functions, which could facilitate the linearization of pre-trained softmax large language models, making them more efficient.