Benchmarking Reveals LLM Judge Reliability for Mobile Agent Evaluation

The increasing reliance on LLM-based judges for evaluating mobile agent task completion necessitates a thorough examination of their reliability. Researchers introduce MobileJudgeBench, a new benchmark designed to systematically assess LLM-as-judge methods using 931 human-annotated trajectories across various mobile agent benchmarks, models, and apps. The study yielded three key findings: firstly, a straightforward baseline judge utilizing sampled screenshots performed comparably to, and often surpassed, more elaborate purpose-built methods, indicating that complexity doesn't always equate to better judge quality. The underlying LLM backbone emerged as the primary determinant of performance among competitive methods. Secondly, benchmark quality metrics proved reliable in predicting real-world judge utility, correlating with both agent ranking fidelity and downstream performance when judges served as reward signals for reinforcement learning. Lastly, failure analysis revealed distinct, qualitatively opposite failure profiles (conservative vs. permissive) linked to the precision-recall characteristics of different LLM backbones.