New Federated Learning Optimizer Improves Performance with Class Imbalance.

Federated learning often struggles with class imbalance, where underrepresented classes lead to poor model performance. Traditional solutions are difficult to apply due to privacy and data heterogeneity constraints inherent in federated settings. This research introduces FedCGNM, a novel client-side optimizer designed to mitigate this issue. It works by partitioning classes into groups, maintaining a normalized momentum for each group, and using the sum of these normalized momentums for updates, effectively balancing gradient magnitudes and reducing noise from rare classes. The paper also presents FedHOO, an X-armed-bandit algorithm for optimizing resampling rates in small-client federations. This method leverages federated parallelism to efficiently evaluate multiple rate combinations at a linear cost. Empirical tests on various long-tailed benchmarks and a proprietary dataset show that FedCGNM consistently outperforms existing baselines, with FedHOO providing additional benefits in smaller-scale federated environments.