BACON Improves AI Judge Accuracy with Budgeted Human Calibration.

Evaluating AI models often relies on AI judges, which are scalable but can introduce biases compared to human preferences. These biases can distort decisions when used for model ranking or quality reporting. The BACON framework addresses this by integrating budgeted human calibration with outputs from multiple AI judges. BACON operates in four stages: it generates comprehensive auxiliary features for each item, including multi-judge scores and contextual embeddings. A small, sampled subset of items receives human labels, which are then used to train a cross-fitted outcome model. This model generates calibrated item-level predictions. These calibrated predictions support two main applications: estimating population-level summary metrics with valid confidence intervals, and providing individual-level surrogate scores for item ranking. By treating AI judges as auxiliary measurements and human labels as the calibration anchor, BACON significantly improves predictive accuracy, ranking consistency, and reduces bias and variance across various tasks and domains, even with limited human annotation budgets.