Time Series AI Models Fail in Critical Traffic Regimes, Benchmarks Hide Flaws

A new study highlights a critical flaw in how time series foundation models (TSFMs) are typically evaluated. Standard aggregate metrics used in benchmarks often obscure significant performance degradations that occur during specific, critical operating regimes. The researchers demonstrated this by applying a regime-stratified evaluation to traffic speed forecasting, where traffic patterns switch abruptly between free-flow and congested states. The findings indicate that TSFMs, while performing well overall, exhibit sharp declines in both accuracy and prediction-interval coverage during these transition periods. For instance, Mean Absolute Error (MAE) can jump from 3 mph overall to 11 mph during transitions, and 90% prediction interval coverage can drop to 55%. To address this, the paper proposes Bimodal Mixture Augmentation (BMA), a post-hoc method that combines TSFM forecasts with historical distributional knowledge, improving transition coverage while maintaining overall accuracy.