LiveHouse-TS: New Benchmark for Time Series Foundation Models

Time Series Foundation Models (TSFMs) are emerging as a promising solution for cross-domain zero-shot forecasting. However, existing evaluation methods typically rely on static benchmarks with fixed historical data, which only provide a snapshot of average performance. These benchmarks fail to capture how models perform in dynamic, real-world environments characterized by seasonal changes, distribution shifts, and unexpected events. To address this critical gap, researchers have developed LiveHouse-TS, an innovative open-world living benchmark infrastructure specifically for TSFMs. Unlike static leaderboards, LiveHouse-TS continuously evaluates models prequentially on real future data, emphasizing continuous temporal validity over snapshot accuracy. This infrastructure is designed to answer crucial long-term scientific questions, such as the stability of model rankings and robustness under distribution shifts. Extensive streaming evaluations across 11 domains and 17 datasets using LiveHouse-TS demonstrated a dramatic reshuffling of static model rankings under live protocols. This highlights that models performing well on historical data may not maintain their superiority in continuously evolving real-world scenarios, underscoring the need for dynamic evaluation.