EvoCUA-1.5 Boosts Online RL for Computer-Use Agents

Computer-use agents face the complex challenge of solving long, multi-step tasks by interacting with dynamic desktop environments. While existing methods like imitation learning provide a strong foundation, they struggle with the real-time causal feedback inherent in actual computer use, where each action alters the environment and future possibilities. EvoCUA-1.5 addresses this by transitioning these agents from offline experience learning to online reinforcement learning. This new framework allows policies to directly interact with sandbox environments and learn from the verifiable outcomes of tasks. Online RL in this context presents unique difficulties, including managing context in multi-turn interactions, dealing with sparse rewards, handling variable-length trajectories, and coping with slow environment feedback. EvoCUA-1.5 tackles these through several innovations. Key components include Step-Level Policy Optimization (STEPO) for maintaining advantage balance, policy-aware filtering for task calibration, and a Dynamic Tri-Adaptive Curriculum (DTAC) that intelligently combines learnable tasks, difficult positive replay, and controlled exposure to infeasible tasks. An asynchronous RL infrastructure further supports stability. These advancements significantly improve training stability and performance, with EvoCUA-1.5 achieving 63.2% success on OSWorld-Verified, outperforming comparable open-weight models and approaching larger parameter count models. This provides a practical pathway for scaling online RL in complex computer-use agent scenarios.