AndroidReality Benchmarks Mobile Agents for Real-World Robustness.

While mobile agents have shown promise on controlled online benchmarks, their performance often degrades significantly when deployed in real-world environments due to unpredictable interface conditions. To address this, researchers have introduced AndroidReality, a framework specifically designed to assess and improve the robustness of these agents. This framework categorizes real-world interface variability into a principled taxonomy of perturbations, spanning state, transition, and action dimensions. Building upon the existing AndroidWorld benchmark, AndroidReality integrates realistic and controllable perturbation injections, enabling a systematic evaluation of mobile agent resilience. Initial assessments using this framework have uncovered substantial robustness deficiencies and identified four common error types. These findings led to the development of a straightforward, training-free Test-Time Introspective Recovery (TTIR) mechanism, which effectively mitigates these failures in both perturbed and clean settings. The research emphasizes that robustness is a crucial, often overlooked, dimension in mobile agent evaluation, and benchmark perturbation is a valuable tool for stress testing and uncovering latent weaknesses.