New RubricForge Tool Improves AI Agent Evaluation by Reducing False Positives

Evaluating large language model agents often relies on other language models as judges, especially when real-world environment rewards are costly or unavailable. However, these AI judges frequently over-credit agents, marking fluent but ultimately unsuccessful actions as successes. This paper introduces RubricForge, a novel technique designed to create more reliable evaluation rubrics.RubricForge works by evolving a text-based judging rubric through reflective evolution against a small set of ground-truth labeled trajectories. This process grounds the rubric in actual outcomes, ensuring that the criteria used for evaluation are tied to real success or failure. Once optimized, this human-readable rubric is frozen and applied to new agent trajectories without needing further environment access.Empirical tests on benchmarks like tau-bench and WebShop showed that RubricForge significantly reduces the false-pass rate, cutting it by roughly half compared to generic G-Eval judges. While overall agreement with environment rewards wasn't statistically higher, the key benefit lies in improved faithfulness and a lower rate of incorrectly crediting failed agents, which is crucial for deploying reliable AI systems.