New Benchmark Evaluates Human-Agent Systems with LLMs.

As large language models (LLMs) increasingly integrate into systems where humans are active collaborators, evaluating these "Human-Agent Systems" (HAS) becomes crucial. This research presents HAS-Framework, a graph-based model that treats humans and LLM agents as equal participants with defined roles, permissions, and communication paths. Building on this framework, HAS-Bench is introduced as a benchmark to assess HAS under various configurations of human involvement. It examines different levels of agency, interaction channels, and persona policies for human participation. The evaluation goes beyond just task completion, also measuring process-level collaboration metrics such as clarification quality, feedback utilization, control calibration, and interaction cost. Experiments across six diverse domains consistently demonstrate that human participation can substantially boost task completion rates and improve recovery from failures. However, the effectiveness of this human input is highly dependent on how, when, and by whom it is exercised, underscoring the need for careful design of human-agent collaboration.