OSWorld 2.0 Benchmarks AI Agents on Complex Real-World Tasks

Existing benchmarks for evaluating AI agents in computer use often fall short in capturing the realism, complexity, and extended duration of real-world tasks. This limitation hinders the ability to fully understand the shortcomings of even the most advanced AI agents. To address this, OSWorld 2.0 has been introduced, a comprehensive benchmark comprising 108 long-horizon computer-use workflows. These tasks are designed to mirror everyday and professional scenarios, with human users typically taking over an hour and a half to complete them. The benchmark specifically targets challenging phenomena common in real workflows but underrepresented in previous evaluations, such as streaming interactions, dynamic environments, cross-source reasoning, inferring implicit states, and requiring visual-spatial precision. Evaluations using OSWorld 2.0 show that even frontier models like Claude Opus 4.8, despite using maximum thinking and batched tool calls, only complete about 20.6% of tasks. GPT-3.5, while more token-efficient, performs even lower. The results indicate that current agents are far from professional-level computer use, often failing due to losing track of constraints, missing mid-task information, guessing instead of asking, and skipping verification, particularly when tasks depend on hidden states they must recover.