New Framework Unifies Uncertainty, Improves AI Evaluation.

A new theoretical framework has been introduced to unify the understanding and evaluation of uncertainty in AI models, particularly crucial for safety-sensitive applications. The core idea is to define uncertainty as "pointwise posterior risk," which integrates Bayesian uncertainty over plausible functions with deviations caused by estimator choices, misspecification, and optimization errors. This comprehensive definition aims to clarify the often-inconsistent definitions of epistemic and aleatoric uncertainty found in literature. This unified view forms the basis for a novel, theory-backed benchmark that allows for the direct computation of "oracle" epistemic and aleatoric uncertainty. By using semi-synthetic datasets with real covariates and known generative processes, the benchmark avoids reliance on proxy tasks, which often provide incomplete insights. Empirical results from this benchmark demonstrate that models achieving high prediction accuracy do not necessarily provide reliable disentanglement of uncertainty types. The framework also highlights practical differences between methods and their sensitivity to datasets and modeling choices.