New Benchmark Evaluates AI Agents' User Interaction in Real Apps.

Current benchmarks for AI agents often overlook the complexity of real-world user interactions, focusing primarily on tool operation rather than dynamic communication. To address this, a new benchmark called AppWorld-UL has been developed. This benchmark simulates diverse user interactions, such as asking for clarification or confirming actions, within nine popular applications like Amazon and Spotify. AppWorld-UL features 516 tasks, many of which are intentionally ambiguous or constrained to necessitate agent-user dialogue. It uses an LLM to simulate user behavior with controlled knowledge boundaries, offering a more realistic interaction environment. Initial evaluations show that even advanced models like Claude Opus 4.7 struggle significantly, achieving less than 50% success, with performance dropping further on more complex, compositional tasks. The findings underscore the critical importance of robust user interaction capabilities for AI agents to effectively handle day-to-day digital tasks. The benchmark's difficulty and focus on diverse interactions position it as a valuable tool for advancing research in user-in-the-loop tool-use agents.