AGENTSCOPE Diagnoses LLM Agent Failures with Behavioral Abstractions

As large language model (LLM) agents become more prevalent, understanding and diagnosing their failures is crucial for ensuring their effectiveness and trustworthiness. Agent failures often manifest through lengthy and intricate operational trajectories, making manual diagnosis impractical. Traditional software bug diagnosis techniques are ill-suited for LLM agent failures, and relying solely on LLMs for judgment can lead to unreliable results. To address these challenges, this paper introduces AGENTSCOPE, a novel neuro-symbolic method for diagnosing agent failure modes. The core principle of AGENTSCOPE involves abstracting agent behavior, derived from its operational trajectories, into structured representations. Furthermore, AGENTSCOPE introduces "neural invariants" to specify expected agent behavior properties. It then employs LLM-guided reasoning, applied to these structured representations against the neural invariants, to precisely identify both the specific failure step and its type within the trajectory. The effectiveness of AGENTSCOPE has been demonstrated on publicly available agent failure datasets (Who&When) and a newly created, more comprehensive dataset (AgentErrata). AGENTSCOPE significantly surpasses the current state-of-the-art in both fault localization and attribution accuracy, highlighting how integrating structured abstractions with LLM-guided reasoning enables effective, reliable, and interpretable diagnosis of agent failures.