SATS Improves Time Series Foundation Models with Scale-Aware Pretraining

Pretraining foundation models for time series data is complicated by the diverse sampling frequencies found across different datasets. Current approaches either use dataset-specific patch sizes, leading to fragmented representations, or enforce a fixed patch size, which ignores inherent temporal variations. To address these limitations, researchers propose SATS (Scale-Aware Token Alignment for Time Series). SATS introduces a novel scale-aware token alignment mechanism that explicitly treats patch size as a notion of scale. It incorporates a contrastive-inspired alignment regularizer to harmonize representation spaces across different scales while maintaining distinct modeling capabilities for each. Furthermore, SATS employs a hybrid masking strategy, combining both random and contiguous masking, to better capture multi-scale temporal structures within the data. Experimental results on LSTF benchmarks show SATS improving MSE by 9.2% and GIFT-Eval MASE by 8.3% over competitive baselines, while also boosting model efficiency by 65.6%.