Evolutionary Search Optimizes Transformers for Time Series

While Transformer architectures have shown promise in time-series forecasting, most approaches rely on fixed designs, despite the significant variability across different forecasting tasks and settings. This research addresses this limitation by proposing EVOTS (Evolutionary Transformer Search), a novel framework for evolutionary neural architecture search. EVOTS is designed to automatically discover Transformer-like models that are specifically adapted to the demands of multivariate time-series forecasting. The framework employs a modular genome representation, which allows for flexible composition of attention, feed-forward, and projection components within the architecture. A built-in repair mechanism ensures the structural validity of the models throughout the evolutionary process. This design enables EVOTS to effectively explore a wide and diverse space of potential architectures without needing predefined, hand-crafted design rules. EVOTS was rigorously evaluated on four benchmark datasets from the ETT family (ETTh1, ETTh2, ETTm1, and ETTm2) under multiple forecasting scenarios, including univariate-to-univariate, multivariate-to-univariate, and multivariate-to-multivariate predictions, with various forecasting horizons. In the critical multivariate-to-multivariate setting, the architectures evolved by EVOTS achieved competitive and, in several instances, superior mean squared error performance when compared to a robust Transformer-based baseline. The results highlight the effectiveness of evolutionary search in identifying flexible and high-performing Transformer-like architectures for time-series forecasting within practical computational limits.