RW-LoRA Enables Efficient Decentralized LoRA Fine-Tuning.

Fine-tuning large foundation models efficiently, particularly with methods like LoRA (Low-Rank Adaptation), is critical. When scaling these processes to distributed environments, challenges arise from high communication overhead and synchronization issues, especially with centralized aggregation or repeated synchronization in gossip-based systems. This paper proposes RW-LoRA, a novel decentralized fine-tuning approach that sidesteps these problems. Instead of maintaining multiple model replicas and aggregating updates, RW-LoRA uses a single "model token" that moves across the network in a random walk. Each node updates this token sequentially using its local fine-tuning objectives. This design significantly reduces communication and computation costs by removing the need for global synchronization and avoiding aggregation errors. The authors provide theoretical convergence guarantees and demonstrate empirically that RW-LoRA achieves comparable task performance on NLP tasks while being substantially more efficient than gossip-based LoRA.