Dysco Boosts Federated LoRA Performance, Mitigating Interference

A new research paper presents Dysco (Dynamic Subspace Boosting), an innovative plug-in method designed to improve federated fine-tuning of large pre-trained models, particularly when using Low-Rank Adaptation (LoRA). Federated learning, where models are trained across decentralized devices, often faces instability when clients have diverse data, leading to "data-parameter interference" in LoRA adapter aggregation. Dysco addresses this by viewing aggregation not just as parameter averaging, but as a strategic allocation of subspaces. Dysco works by allowing clients to compute activation-insensitive subspaces from their local data and transmit only these bases to a central server. The server then constructs client-specific merged subspaces through a closed-form solution, optimizing for compatibility with other clients' update directions. To counter representation drift over time, Dysco employs multi-round subspace boosting, preserving past update directions while adapting to new data representations. The method includes a convergence analysis demonstrating that Dysco's server-fixed merged subspaces lead to a tighter upper bound on aggregation error. Empirical studies, including synthetic federated tasks and MIMIC-IV clinical-note classification with Llama-3.2-1B, show substantial reductions in interference and significant performance improvements (up to 4.3% on MIMIC) across various federated learning algorithms, with minimal overhead.