No Universal Signal Predicts LLM Regression After Updates

Large Language Models are frequently updated, typically showing overall performance improvements. However, these aggregate gains can mask individual instances where an LLM's performance regresses, meaning a previously correct response becomes incorrect after an update. This research explores various inference-time signals to predict such sample-level regressions. The study compared single-model signals like confidence and logit margin with cross-version signals such as output KL divergence and representation drift across multiple benchmarks and model updates. Key findings indicate that the most effective signal for predicting regression is highly dependent on the specific task; for example, confidence works well for multiple-choice questions, while more complex tasks like math and code generation benefit from likelihood or KL divergence signals. Crucially, no single signal proved universally superior, highlighting the need for task-specific approaches.