OrchestraBench Diagnoses Multi-Agent System Failures and Recovery.

Researchers have introduced OrchestraBench, a novel benchmark designed to evaluate multi-agent orchestration frameworks beyond simple task accuracy. This benchmark focuses on diagnosing why a pipeline failed, identifying the origin of failure cascades, and assessing the quality of decomposition and recovery mechanisms. It employs a controlled, seed-reproducible failure-injection harness across templated enterprise workflows. OrchestraBench introduces new primary metrics such as cascade radius and per-failure-mode recovery rates. Initial findings using a real Claude agent revealed distinct failure-handling tiers: tool faults recovered fully, ambiguous delegation partially, while latent or semantic modes showed no recovery. This pattern persisted across different Claude models and workflow contexts. The research highlights that blind retries are ineffective for latent faults, emphasizing the need for robust detection and attribution. It also found that cascade radius increases with pipeline depth and that apparent containment gains often stem from trusted-state signals rather than autonomous detection. These insights are crucial for developing more robust and reliable multi-agent systems.