Entropy Regularization Boosts Sparse Models in Federated Learning

Federated Learning (FL) faces challenges like data heterogeneity and partial client participation, particularly when aiming for sparse models. Learning sparse models is crucial for efficiency in FL, but it becomes difficult in scenarios with small sample sizes and high dimensionality, where optimization can lead to models that don't generalize well. Traditional magnitude-based pruning methods often fail to account for uncertainty in the parameter space. This work explores the use of entropy regularization on gate distributions within a probabilistic gate and L0 constraint framework. This mechanism helps maintain uncertainty during sparse federated optimization, preventing the model from committing too early to a specific sparse structure. The researchers investigate its impact under varying conditions of data heterogeneity, client participation, and desired sparsity levels. Experiments conducted on both synthetic and real-world benchmarks demonstrate that this entropy-regularized approach consistently outperforms existing methods like federated iterative hard thresholding (Fed-IHT) and pruning after dense federated averaging (FedAvg) training. The improvements are observed in both statistical performance on test data and the accuracy of sparsity recovery, indicating a more robust and effective way to achieve sparse models in challenging FL environments.