RL for GUI Agents Enhanced by Autonomous Evaluation

Training Computer-Use Agents (CUAs) with reinforcement learning (RL) in open-ended desktop environments is challenging due to the scarcity of scalable, machine-readable reward signals. Task success in graphical user interfaces (GUIs) is often visually determined and difficult to specify with manual labels or handcrafted reward functions. The researchers propose an RL fine-tuning framework that utilizes autonomous vision-language evaluation to provide a scalable supervision signal for GUI agents. A Vision-Language Model (VLM) assesses task completion from a final screenshot and the original instruction, offering terminal feedback without requiring task-specific heuristics or manual labeling during policy optimization. Recognizing that autonomous evaluators are imperfect, their feedback is modeled as a noisy binary reward channel. A noise-corrected reward estimator is then derived for Proximal Policy Optimization. Experiments across macOSWorld, Windows Agent Arena, and OSWorld benchmarks demonstrated that these corrected evaluator rewards significantly outperformed both zero-shot baselines and raw evaluator rewards, leading to an average improvement of 12.6 percentage points over zero-shot performance and 5.1 points over raw evaluator fine-tuning. This suggests that autonomous evaluation, when noise is explicitly modeled, can serve as a practical and effective reward signal for RL in GUI environments.