Retraining Policies for Streaming ML Under Concept Drift

Production machine learning systems frequently encounter concept drift, where the underlying data distribution changes over time, leading to model degradation. Despite this common challenge, practitioners lack clear guidance on when to retrain models, especially given the costs, finite budgets, and inherent latency of training and deployment. This research presents a controlled empirical study comparing three practical model-refresh policies—periodic retraining, error-threshold triggering, and statistical drift-triggered retraining (ADWIN)—against a baseline of no retraining. The study evaluates these policies within a unified system model that explicitly accounts for retraining budgets and the combined latency of training and deployment. Across nearly 4,000 experiment runs, spanning various drift regimes, budget levels, latency levels, datasets, and learning modes, a critical finding emerged: the single most impactful design decision is whether the deployed model learns incrementally. When per-sample incremental updates are possible (as with the linear online learner and immediate labels studied), no retraining policy significantly outperforms the no-retrain baseline, even under extreme latency. However, in scenarios without incremental updates, the choice of policy becomes crucial, separating outcomes by 15-55 percentage points of post-drift accuracy. In these cases, simple periodic retraining consistently outperforms reactive policies (error-threshold and ADWIN) under abrupt and gradual drift, with reactive policies only showing an advantage under recurring drift. The study also identifies systematic failure modes of reactive policies and a latency-budget queueing interaction that can silently halve effective retraining budgets.