FedSLM Enables Federated Fine-Tuning for Compressed Models

Federated learning with large foundation models faces a significant challenge: many organizations with valuable domain-specific data lack the computational resources to host billion-parameter models. Existing federated approaches attempt to bridge this gap through methods like parameter-efficient tuning or knowledge distillation, but these often compromise on memory reduction, architectural independence, or representational accuracy.To resolve this, a new framework called FedSLM has been introduced, designed for federated fine-tuning with heterogeneous, compressed client models. FedSLM employs SVD-based decomposition to create self-contained client models whose low-rank subspaces are structurally compatible for aggregation. It then uses a two-stage protocol: synchronizing lightweight adapters within compression groups and fusing full-rank reconstructions across groups via structural alignment.Finally, a weak-to-strong elicitation step, incorporating an auxiliary confidence loss, transfers the aggregated knowledge to the full-scale server, while an explicit bias-variance trade-off helps mitigate compression artifacts. Experimental results on natural language and vision-language benchmarks demonstrate that FedSLM outperforms existing federated baselines, even with client models operating at roughly half the GPU memory of the full model.