Super-Tuning Enables Efficient Sparse Fine-Tuning for LLMs

Fine-tuning large language models (LLMs) is computationally intensive and memory-demanding, requiring significant resources and storage for each task. This research explores whether saliency signals, typically used for model pruning, can be repurposed to identify which parameters in an LLM should be adapted during fine-tuning. The proposed method, Super, is a sparse parameter-efficient fine-tuning (PEFT) technique. It establishes a small, fixed set of trainable parameters by calculating a Wanda-style activation-weighted magnitude score from a calibration pass. This score helps determine the most critical parameters to update. Building on Super, the researchers also introduce Supra, a hybrid adapter. Supra combines the sparse updates from Super with the low-rank adaptation (LoRA) technique, maintaining a consistent trainable-parameter budget through a simple splitting rule. Experiments on arithmetic tasks with Llama-3.2-1B and Meta-Llama-3-8B showed that the best Super/Supra variants achieved the highest average accuracy among tested adapter configurations, suggesting that pruning-inspired orderings can effectively guide sparse PEFT, especially when integrated with low-rank adapters.