Dataset Selection Framework Preserves Model Rankings for Efficient Benchmarking.

Benchmarking machine learning models often involves evaluating performance across numerous datasets, which can be computationally expensive and time-consuming. To improve efficiency, it is desirable to select smaller, representative subsets of datasets that can still accurately reflect the global ranking of models. Current selection methods often rely on heuristics and lack robust analysis of how well they preserve these rankings. Researchers have developed a new framework specifically designed for selecting dataset subsets while rigorously evaluating the preservation of global model rankings. The framework incorporates bootstrap aggregation to provide valid confidence intervals, enabling a principled comparison of different selection strategies. It explores methods such as clustering, design criteria (A/D-optimality), random baselines, and greedy farthest-first (FAFI). For FAFI, the study derives upper bounds on selection quality in terms of ranking errors. Empirical results are compelling: in time series classification (TSC) and a natural language processing (NLP) benchmark, several strategies, including simple FAFI, significantly improve rank preservation compared to random subsets. For TSC, the best strategy achieved a Spearman correlation of 0.95 with full benchmark rankings using only five selected datasets. However, in recommender systems, the improvement over random selection was minimal. This indicates that the effectiveness of selection approaches depends on both the quality of dataset representations and the scale of the benchmarking regime.