Neural Architectures Show Varying Robustness to Temporal Data Drift

Machine learning systems deployed in real-world environments frequently encounter temporal distribution shifts, where data characteristics evolve over time. This phenomenon, known as "drift," can severely degrade model performance and reliability. A new empirical study systematically investigates how different neural network architectural choices influence their robustness to such temporal shifts. The research utilized a unified evaluation framework across three diverse, time-indexed domains: image classification, multi-label text classification, and text regression. Models, ranging from simple multilayer perceptrons to convolutional networks, recurrent networks, and pretrained Transformer-based encoders, were trained on cumulative historical data and then evaluated on both past and future time periods to quantify cross-temporal generalization. The findings indicate that architectural inductive biases play a significant role in temporal robustness. Models designed to leverage highly discriminative, localized features often achieve high in-distribution accuracy but degrade rapidly when data drifts. Conversely, pretrained encoders, which tend to capture broader, more stable representations, exhibit a more gradual degradation. These insights offer valuable guidance for selecting appropriate architectures for systems operating in dynamic, real-world conditions.