SkillTV-Bench Evaluates LLM Agent Performance on Complex Tasks

Evaluating Large Language Model (LLM) agents that perform complex, multi-step tasks, especially those using tools and interacting with environments, requires more than just checking the final output. It demands verification of the entire execution process, including the procedural knowledge embedded in "skills." Existing benchmarks often fall short by not combining task-time skills with directly inspectable artifacts or environments. To address this, SkillTV-Bench introduces a comprehensive benchmark of 681 real agent trajectories across 50 tasks and eleven domains. Its primary goal is to evaluate skill-aware trajectory verification for both LLM-as-a-Judge and Agent-as-a-Judge methods. This benchmark provides a robust platform for assessing how well judges can identify critical failures and inspect relevant evidence. Furthermore, the paper proposes SkillTV-Evolve, which externalizes verification knowledge into a reusable "JudgeSkill." This JudgeSkill guides an agent judge to plan specific inspections and issue evidence-grounded verdicts. An automated evolution loop refines this JudgeSkill using misjudged cases, leading to significant accuracy improvements. For instance, the refined skill increased an agent judge's accuracy by 14.8 percentage points on a development pool, demonstrating its effectiveness in improving agent evaluation and selection.