New Reflection Method Boosts Desktop GUI Agents

This research addresses a key challenge in developing robust GUI agents for desktop environments: the difficulty of accurately reflecting on actions due to the large and often subtly changing visual interfaces. Traditional Planner-Operator-Reflector (POR) frameworks struggle because their reflector components combine change detection and outcome verification, leading to weakly grounded decisions. The authors propose Evidence-First Reflection (EFR), a novel two-stage reflector designed to explicitly separate these two processes. EFR first identifies the precise location of an action and candidate changed regions using Set-of-Marks annotations. It then describes and filters these action-relevant visual changes, providing clear evidence before the final judgment on the action's outcome is made. This decoupled design significantly reduces visual search complexity and reasoning burden for the agent. Experiments on OSWorld-Verified and WindowsAgentArena benchmarks demonstrate that EFR improves reflector accuracy by over 7% and boosts overall end-to-end task success rates by approximately 5-6%, making GUI agents more reliable in complex desktop environments.