Attention Head Reweighting Enables Data-Efficient LLM Adaptation

Adapting large language models (LLMs) effectively with limited data is a critical challenge, especially in domains like security where labeled examples are scarce. While parameter-efficient adaptation methods have shown promise, LLMs still struggle with difficult tasks when only a few samples are available. This research introduces Attention Head Reweighting (AHR), a novel and highly data-efficient method. AHR adapts LLMs to new text classification tasks by learning only a single scalar value for each attention head. This approach drastically reduces the number of parameters that need to be learned, leveraging the functional specialization inherent in individual attention heads. Experiments conducted on diverse open-source text classification datasets demonstrate that AHR can outperform established baselines such as LoRA, even while utilizing 200 to 1000 times fewer trainable parameters (modifying only about 0.0001% of the model's parameters). Furthermore, the learned weights in AHR are easily interpretable, offering insights into which attention heads and mechanisms contribute most to the LLM's in-context learning abilities.