CRATE Improves Mobile Agent Evaluation for Task Completion and Safety

This research presents CRATE, a novel two-stage VLM-as-judge framework designed for the automated evaluation of language-guided mobile agents. Current evaluation methods often struggle with context overload and primarily focus on task completion, neglecting operational safety. CRATE addresses these limitations by employing a step-level consequence reasoning mechanism. The framework independently extracts task-relevant visual clues and infers action-conditioned state changes at each step. This step-level textual evidence is then aggregated to provide an evidence-grounded evaluation of task completion. Furthermore, CRATE is extended to CRATE-S for assessing operational safety. Experiments demonstrate CRATE's effectiveness and robustness, with CRATE achieving an F1-score of 0.833 on AndroidWorld and CRATE-S reaching 0.697 on MobileRisk, showing strong alignment with ground truths and significant improvement over previous methods.