FedWeave Improves Federated Learning with Specialized MoE-LoRA

This research introduces FedWeave, a novel framework designed to enhance federated learning, particularly for Large Language Models (LLMs) adapted via Parameter-Efficient Fine-Tuning (PEFT) in decentralized, private data environments. Existing federated MoE-LoRA methods often struggle with task heterogeneity among clients, leading to interference and gradient conflicts during model aggregation. FedWeave addresses this by proposing an asymmetric aggregation strategy. The core insight is to separate the aggregation of specialized LoRA experts from the optimization of the routing mechanism. This allows experts to receive "pattern-coherent updates" that preserve their specialization, while routers benefit from "mixed-task observations" that improve expert comparison. FedWeave employs unsupervised prototype discovery to align local data buckets across clients, enabling prototype-level expert aggregation. During inference, it maintains high performance with sparse inference, activating only one expert. Theoretical analysis supports its advantages in controlling expert convergence and bounding sparse-inference risk, with empirical results showing consistent outperformance against strong baselines on heterogeneous multi-task benchmarks.