Model Merging Boosts Distributed Learning Efficiency in DiLoCo.

Distributed learning frameworks, such as local SGD and DiLoCo, aim to reduce communication costs by periodically aggregating independently trained local models. However, their performance can degrade as the number of local models or training steps increases. This paper draws a parallel between the aggregation step in these methods and model merging techniques, which combine multiple fine-tuned models. The researchers evaluated several state-of-the-art model merging methods within this distributed optimization context. They identified Iso-C as particularly effective for improving DiLoCo, even surpassing momentum-based DiLoCo without its own momentum mechanism. Building on this, they developed IsoLoCo, which adapts Iso-C for distributed training by integrating Nesterov momentum. Empirical tests on language model pre-training across various worker counts demonstrated that IsoLoCo significantly outperforms DiLoCo. This performance gap widens with more workers and holds across different model sizes and inner step counts, confirming that merging-inspired aggregation is a potent strategy for low-communication distributed training.