Frontier Learning Optimizes Models Under Distribution Shift

Modern machine learning often involves reusing pretrained or foundation models across various downstream tasks. However, deploying these models in new environments frequently encounters "distribution shift," where the target data differs from the training data, making it difficult to determine the best strategy among zero-shot application, fine-tuning, or training a new model from scratch. This research introduces "Frontier Learning," a framework that views a collection of candidate models—ranging from black-box predictors to white-box models with accessible internal representations—as complementary sources of information. Frontier Learning constructs a comprehensive target-domain feature by concatenating internal representations from white-box models with prediction outputs from black-box ones. A lightweight, regularized supervised learner is then trained on this combined representation using labeled target data. This approach is designed such that the resulting predictor is guaranteed to perform at least as well as any individual baseline strategy on the training sample, effectively learning the Pareto frontier of predictive models under distribution shift. Empirical evaluations across visual domain adaptation and clinical mortality prediction tasks demonstrate that Frontier Learning consistently matches or surpasses the strongest individual reuse strategy, particularly when no single baseline is reliably optimal.