Agent Leaderboards Misrepresent Capability, Study Finds

New research challenges the common assumption that AI agent leaderboards accurately reflect an agent's overall capability. The study, utilizing a Generalizability Theory framework across three open agent-trace benchmarks, found that the agent's inherent capability accounts for less than 3% of total variance. Instead, the interaction between the agent and specific tasks drives 7-23% of the variance, suggesting leaderboards primarily measure task specialization. The findings highlight several critical issues: reliability significantly drops on harder tasks, and evaluation designs appearing most reliable often replicate poorly in real-world scenarios. While population-level diagnostics transfer across benchmarks, individual agent rankings can invert. To address these challenges, the researchers propose a "Deployment Decision Reliability" (DDR) reporting discipline, offering a structured way for enterprises to make defensible deployment choices based on a deeper understanding of agent performance variance.