Metric Match Improves LLM Judge Reliability Estimation with Reduced Human Annotation.

Evaluating the reliability of Large Language Model (LLM) judges, which are increasingly used to assess open-ended text generation, typically requires extensive and expensive human annotations to ensure alignment with human raters. This research presents "Metric Match," a novel approach designed to streamline this process. The method intelligently selects a smaller, more representative subset of samples for human review, ensuring that the estimated reliability metrics for LLM judges accurately reflect the overall population. Empirical evaluations demonstrate that Metric Match significantly outperforms random subset selection. Across various correlation metrics and datasets, it achieved a win-rate of 0.838, reducing average estimation error by 18.7% and decreasing annotation needs by 32.5%. A practical cost model highlights substantial savings, such as over $1,000 in a medical case study involving expert annotations. Beyond reliability estimation, the method also proves effective for classifying whether an LLM judge meets a specific deployment reliability threshold. The project's code and an installable package are publicly available, making it accessible for broader adoption in AI evaluation workflows.