Biased LLM Judges Silently Disable Skill Retirement in Self-Evolving Agents.

The paper investigates a critical flaw in self-evolving AI agents: the silent disabling of their skill retirement mechanism when evaluated by biased Large Language Model (LLM) judges. Self-evolving agents rely on retiring ineffective skills to prevent their growing library from degrading below a baseline. This process assumes an unbiased reward system, which is often not the case with reference-free LLM judges. The researchers demonstrate that a biased judge doesn't just introduce noise; it fundamentally "switches off the curator" responsible for skill retirement. Through a corrupted-reward analysis and a behavioral study on a report-writing testbed, they show that "false-pass" bias – where failures are incorrectly marked as successes – disables contribution-based retirement beyond a certain threshold, irrespective of data volume. This mechanism failure is universal across domains and failure rates, affecting all but near-zero-false-pass verifier-like graders. Crucially, the downstream outcome is regime-dependent: evaluation quality only degrades if the same corruption also hinders skill synthesis; otherwise, it remains steady. This means the disabled curator is "silent," not surfacing in aggregate performance metrics, making it a behavioral safety result rather than a performance one. The paper proposes a cheap defect-injection audit to help operators identify this threshold before deployment.