Auditing Clinical Prediction: Separating Learner Gaps from Data Limits

In clinical prediction, models can reach a performance plateau for two distinct reasons: either the machine learning algorithm itself isn't fully utilizing the available information (a "learner gap"), or the inherent limitations of the recorded data prevent any model from achieving higher accuracy (a "measurement-channel ceiling"). This paper proposes a novel framework to precisely differentiate these two factors. The framework defines optimal balanced accuracy through total-variation separation, offering an architecture-agnostic approach to estimate the true performance frontier. It includes diagnostics like a label-permutation optimism floor and an underfit curve to help auditors understand where performance improvements are most likely to come from. Validation on real-world clinical datasets, such as UCI readmission and BRFSS diabetes, showed that well-tuned models often approach the estimated data frontier, while less capable models leave significant "learner gaps." The study also revealed that modest gains in metrics like AUROC can mask much larger improvements in Bayes decision-flip rates, and that combining different measurement channels can yield significant complementary gains. The findings suggest that saturation should prompt an auditable decision: improve the learner if headroom exists, or improve data collection if the ceiling has been reached.