New Diagnostic Assesses LLM Judge Competence for Skill Optimization

In text-space skill optimization, an agent's natural-language skill document is refined by accepting or rejecting candidate improvements via a validation gate. Traditionally, these gates rely on verifiable rewards, limiting their application to tasks with automatic verifiers. Replacing these with Large Language Model (LLM) judge gates could broaden applicability, but their effectiveness without ground truth is uncertain. This research addresses a crucial prior question: can we determine if an LLM judge can reliably distinguish correct from incorrect answers *before* it's integrated into the optimization loop? The study formalizes a reference-free judge as a "latent solver," meaning its evaluation capacity is bounded by its own ability to solve the task. This model provides a closed-form bound on discriminability (ROC-AUC) based on the judge's competence and the size of the answer space. A key finding is that a judge's benchmark accuracy can be misleading, often overstating the competence relevant for effective gating. A non-intervening probe was used to record judge scores during actual optimization runs without altering decisions. The results showed that discriminability was at chance when competence was low but usable when higher. Crucially, the diagnostic successfully predicted which types of gating errors would occur in a closed-loop study, offering a cheap pre-deployment method to assess LLM judge suitability.