New Metric Proposed for Irregular Time Series Forecasting Evaluation

Current methods for evaluating irregular time series forecasts primarily rely on Mean Squared Error (MSE), which this paper argues is flawed. The issue stems from MSE being influenced by the specific timestamp sampling distributions of the data, leading to an inaccurate reflection of a model's true continuous-time predictive ability. This bias can obscure the actual performance of forecasting models. To address this, researchers propose the Continuous-time Squared Error (CSE). This new metric incorporates importance weighting to neutralize the impact of varying timestamp sampling distributions. Theoretically, CSE's asymptotic estimation error for continuous-time risk is proven to be no greater than MSE's, suggesting a more robust and unbiased evaluation. A comprehensive benchmark, including synthetic, semi-synthetic, and eight real-world datasets, was developed to validate CSE. Experiments confirm that CSE more accurately recovers continuous-time risk compared to MSE, indicating that relying solely on MSE may not fully capture model performance in practical irregular time series scenarios.