Homogeneous Train-Test Splitting Enhances AutoML Evaluation

This research highlights the critical role of train-test splitting methods in achieving accurate model evaluation within Automated Machine Learning (AutoML). Standard random splitting often assumes identical underlying distributions between partitions, an assumption frequently violated in real-world datasets characterized by class imbalance, clustering, or spatial autocorrelation. The study compares five established splitting strategies, including random, stratified, Kennard-Stone, Duplex, and SPXY, across fifteen UCI benchmark datasets. The analysis reveals that geometry-based methods, while aiming for coverage, can inadvertently introduce instability into downstream performance estimates by yielding near-zero Maximum Mean Discrepancy (MMD) scores, indicating a lack of diversity. To counter this, the paper proposes an "Optimised-Distribution" method that explicitly treats statistical similarity as an optimization objective. This new method consistently achieves the highest mean MMD similarity (89.0%) among all evaluated strategies, suggesting a more robust approach to creating homogeneous train-test splits for reliable AutoML model evaluation.