DuMateBench Evaluates Autonomous Agents in Real-World Workflows

Autonomous agents are increasingly being adopted for complex, multi-tool workflows in real-world environments. However, existing benchmarks often separate tasks by application or capability and evaluate agents in cleaner, more stable settings than those encountered in practice. This creates a gap between benchmark performance and real-world utility. To address this, DuMateBench has been introduced as a real-session benchmark. It was reconstructed from anonymized and privacy-screened user sessions from a large-scale production agent platform. Each task in the benchmark retains the relevant pre-solution interaction history, persistent configurations, and workspace state, and is human-verified for accuracy. The benchmark comprises 200 tasks across 8 broad scenarios and 17 fine-grained capability categories, with most tasks requiring coordination of multiple capabilities. These tasks are executed in isolated Docker containers, injected with three types of real-world environmental complexity: Insufficient, Unstable, and Noisy conditions. Performance is assessed using a hybrid deterministic and LLM-as-Judge evaluation protocol. Experiments with five representative autonomous-agent frameworks and four state-of-the-art LLMs revealed significant shortcomings in strict task completion. Further analyses showed that performance under environmental perturbations is influenced by both the LLM's capabilities and the surrounding agent framework. The code and data are publicly available.