Self-Pretraining Boosts Transformer Performance in Medical Time Series

Recent advancements have shown that transformer architectures benefit greatly from Self-PreTraining (SPT) on long-context benchmarks. This research investigates whether similar performance gains can be achieved when applying SPT to various medical time-series data, including multimodal, multivariate, and even simple univariate datasets. The primary goal was to evaluate SPT's impact on transformer models' performance and scalability, especially in data-limited medical scenarios. The study evaluated transformer architectures across three distinct medical time-series tasks: rehabilitation robotics (Camargo dataset), stress detection (Non-EEG Stress), and Parkinson's disease detection (Gait Parkinson's Disease). Models were either trained from scratch or through SPT, utilizing four different masking-based objectives designed to foster temporal and cross-modal representation learning. The researchers also varied model depth to understand how model capacity interacts with the benefits of pre-training. Across all datasets and configurations, SPT consistently led to improvements in classification accuracy, ranging from 0 to 6 percentage points, depending on the masking strategy, dataset, and architecture. These gains were observed not only in complex multivariate settings but also with simple univariate inputs. Deeper models, capable of better exploiting the enriched temporal representations learned during pre-training, showed even greater improvements. These findings suggest that SPT is a straightforward and generalizable strategy to enhance transformer performance in medical time-series tasks, offering potential for improved robustness and accuracy in clinical settings where data is often scarce.