New Method Evolves AI Evaluators from Their Own Blind Spots

The development of effective AI agents is often hindered by the lack of reliable automatic metrics, especially for complex tasks like report generation where quality is hard to quantify. This research explores whether an evaluation metric can "write itself" by learning from its own deficiencies. The core idea is that while defining what makes an answer "good" is challenging, identifying what makes it "wrong" is often easier. The proposed method, EvalCEGAR, evolves a pool of small Python operators, each designed to flag a specific defect in a candidate output or abstain. Unlike direct prompting of models for operators, which yields limited diversity, EvalCEGAR borrows from counterexample-guided abstraction refinement (CEGAR) used in program verification. It treats the current pool of operators as an abstraction and actively searches for "collisions"—pairs of answers (one correct, one incorrect) that the current operators score identically. When such a collision is found, it serves as a specific "authoring request" to refine or create new operators that can differentiate the pair. If all attempts to resolve a collision fail, the system widens the scope of what an operator can read. Evaluated on MBPP+ and HumanEval+, EvalCEGAR successfully generated a 55-line operator that closed a significant gap towards perfect filtering on unseen tasks, outperforming a collection of hand-written operators. The method demonstrates that an LLM judge, given the same information, can achieve similar deltas on a nearly disjoint set of candidates, but at a continuous cost per call, whereas the evolved operator incurs no runtime cost.