FrameFT Offers Efficient Fine-Tuning for Transformer Models

Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA are effective for adapting large pre-trained models, but their memory requirements scale with the model's hidden dimension and rank. This can still be substantial for very large models. A new proposal, FrameFT, addresses this by modeling the parameter update using a sparse coefficient matrix within a Fusion Frame basis. These Fusion Frames can be generated algorithmically and shared across different layers of the model, leading to highly efficient updates. By storing and optimizing only the sparse coefficients of the basis expansion, FrameFT drastically reduces the memory footprint. The inherent sparsity in both the coefficient matrix and the Fusion Frames also provides significant computational benefits. Formal convergence results support the method's efficacy. Experiments across various supervised fine-tuning benchmarks, including language and vision tasks, demonstrate that FrameFT matches or surpasses state-of-the-art PEFT techniques while requiring substantially fewer trainable parameters.