JarvisBench Evaluates Human-Agent Attention Coordination.

This paper introduces JarvisBench, a novel evaluation suite addressing the critical coordination challenge between humans and continuously operating, long-horizon AI agents. While agents can work autonomously for extended periods, human attention is intermittent. This creates a two-way problem: users need to query agents about their progress, and agents need to solicit human input for crucial decisions when the user isn't actively monitoring. JarvisBench models an "always-on" intermediary, named Jarvis, to mediate this interaction. The benchmark comprises 45 agentic task instances, including single-agent tasks and multi-agent projects across 19 diverse domains. A key feature is that the need for human attention arises naturally during agent execution, rather than being pre-scripted. JarvisBench evaluates two main aspects: the intermediary's accuracy and promptness in answering user questions about ongoing tasks, and its ability to detect when an agent requires human judgment, solicit it effectively, and route it back to improve task outcomes. Designed to integrate with various agent runtimes, JarvisBench provides a stable target for evaluating and improving human-agent collaboration as AI capabilities advance.