SeFoRA Improves Federated LoRA with Heterogeneous Client Ranks

Federated learning combined with parameter-efficient fine-tuning, specifically Low-Rank Adaptation (LoRA), presents significant challenges. When different clients in a federated network utilize varying LoRA ranks, their factor matrices become dimension-incompatible, making direct aggregation difficult. Furthermore, simple factor-wise averaging suffers from a "bilinear mismatch" problem, hindering effective model updates. To overcome these hurdles, researchers propose SeFoRA (Sketch-Aggregated Federated Low-Rank Adaptation). This algorithm allows each client to transmit a compact linear sketch of its local updates to the central federator. This sketching technique enables direct and efficient aggregation, mitigating the bilinear mismatch and facilitating aggregation within a smaller subspace of the full model. A specialized version, SeFoRA-Ho, is introduced for rank-homogeneous settings, demonstrating convergence guarantees. Experimental results, particularly on fine-tuning RoBERTa-Large on GLUE datasets, show that SeFoRA algorithms significantly outperform existing state-of-the-art federated LoRA methods, indicating a more robust and efficient approach to distributed model adaptation.