TallyTrain Boosts Federated Learning Efficiency with Hard-Label Consensus.

A new federated learning protocol called TallyTrain has been developed to address the significant bandwidth limitations in distributed AI training. Federated learning typically faces constraints on two fronts: the size of the model, which dictates how often parameter-averaging methods can merge updates, and the number of classes, which makes soft-label distillation prohibitively expensive for large vocabularies. TallyTrain tackles the latter by drastically reducing communication for class information. Instead of transmitting full soft labels for each probe, TallyTrain only sends the argmax class index from each peer, collapsing the class-count axis to a mere $\lceil \log_2 C \rceil$ bits per probe. This compression is not just about efficiency; in non-IID training environments, TallyTrain can actually be superior to soft-label distillation. It achieves this by leveraging majority voting to filter out noise from under-trained peers, whereas soft-label averaging might amplify such noise. Across standard benchmarks, TallyTrain consistently matches or surpasses soft-label distillation while reducing communication by up to three orders of magnitude. The research also presents a bandwidth-bridge variant that combines this hard-label consensus with sparse parameter merges, demonstrating Pareto-dominance over established baselines like FedAvg, FedProx, and FedDF, making it a highly efficient solution for scalable federated learning.