Conformal Method Compares Models for Local Superiority

Standard methods for comparing machine learning models typically aggregate performance across an entire dataset, declaring a single "winner." This global approach can mask situations where different models excel in different parts of the data space. This paper introduces a novel framework called conformalized local model comparison, designed to identify regions of "local superiority." The method uses a split-sample approach, dividing data into three disjoint sets. These sets are used to fit competing models, estimate local performance centers and scales from out-of-sample scores, and then conformally calibrate residual uncertainty. At any given data point, the procedure declares a local winner only when a one-sided conformal bound definitively excludes a tie, providing statistical guarantees on erroneous declarations. The research demonstrates that aggregate global comparisons can significantly differ from the prevalence of local superiority, and provides a bias-variance decomposition to explain how model structure influences these local wins.