TopoTuner Offers Efficient, Topology-Guided LLM Fine-Tuning

Full fine-tuning remains a powerful method for adapting pretrained large language models (LLMs), but its cost and computational intensity stem from updating all model weights. While LoRA reduces trainable parameters, it doesn't explicitly identify which pretrained components are most critical to train versus freeze during adaptation. This research introduces TopoTuner, a novel topology-guided fine-tuning framework. TopoTuner selectively freezes attention projection matrices by treating each matrix as a "row cloud" and measuring its topological changes during fine-tuning using Wasserstein distances between persistence diagrams. This allows the framework to learn a reusable freezing profile from a source dataset. TopoTuner demonstrates its efficiency and effectiveness across LLaMA-3.1-8B, Mistral-7B-v0.3, and Qwen3-8B-Base models. It trains only 1-2% of model parameters, yet achieves performance competitive with full fine-tuning and outperforms LoRA in most settings. Furthermore, TopoTuner reduces training time by an average of 20.4% compared to full fine-tuning and 5.5% compared to LoRA, opening a new avenue for efficient and transferable fine-tuning strategies.