JarvisBench: A New Benchmark for Spoken AI Agent Mediation

This research introduces JarvisBench, a new benchmark designed to assess the effectiveness of "Jarvis-style" AI agent mediation in complex, long-running tasks. The core idea is to create an always-on, spoken interface that allows users to continuously interact with an AI agent, ask questions, receive progress updates, and inject guidance without interrupting the agent's primary workflow. JarvisBench features two tracks: one measures how mediation improves the agent's ability to complete tasks, and the other evaluates how it enhances user understanding and accessibility during ongoing execution. A modular prototype, built with various large language models, was tested on 34 tasks. Initial findings indicate that this mediation layer can provide context-aware responses to user queries and boost task performance when user input is strategically applied. The study also highlights that the performance of this mediator heavily depends on the underlying LLM, underscoring the potential of this "missing middle layer" in AI agent ecosystems and the need for further development.