Expert Training Duration Impacts LLM Model Merging Quality

This research investigates the impact of expert training duration on the quality of merged Large Language Models (LLMs), challenging the conventional wisdom of merging experts at their optimal validation loss. The study systematically fine-tuned experts across five domains and three model sizes, saving checkpoints at various training steps, from 25% to 500% of the optimal. The findings reveal a strong method-dependent pattern: simple averaging merging techniques degrade sharply if experts are overfit, while sparsification-based methods achieve their peak performance when experts are trained significantly beyond their validation optimum. This phenomenon is formalized through a bias-variance decomposition, drawing parallels to how random forests benefit from high-variance individual learners. The results emphasize that the choice of training duration and merging method should be interdependent, not independent, for optimal multi-task model merging.