UpgradeBench: Benchmarking LLM Specialist Upgrades Across Releases

Organizations frequently fine-tune large language models (LLMs) for specific tasks, creating "specialists." However, with the rapid release of new base models, a critical decision arises: how to effectively upgrade these specialists without losing performance or incurring excessive costs. This research presents UpgradeBench, a novel benchmark designed to simulate these real-world upgrade scenarios. UpgradeBench tracks four consecutive Qwen releases, a continuation checkpoint, and OLMo checkpoints, across six tasks and two model sizes. It systematically investigates three core questions: whether newer base models inherently improve retrained specialist performance, the portability of existing fine-tuning adapters across versions, and the efficacy of various recovery strategies, such as teacher relabeling. The findings reveal that upgrade gains are inconsistent, varying by task, scale, and release. Adapter portability is not guaranteed and decays with continued pretraining distance, even within the same model family. Crucially, the study shows that a fixed decision policy across 33 upgrade episodes can yield minimal quality regret while significantly reducing compute and labeling costs compared to full retraining. A lightweight CKA probe is also identified as a predictor for cross-version adapter portability.