New Benchmark for LLM Agent Trajectory Attribution

Large Language Model (LLM) agents are increasingly performing complex, long-horizon tasks that involve user instructions, tool use, external observations, and memory. While existing benchmarks primarily evaluate the final outcomes of these agents, they offer limited support for understanding the detailed reasons behind an agent's actions—a process known as fine-grained attribution analysis. This lack of insight makes it difficult to diagnose failures, improve agent reliability, or ensure safety. To address this, researchers have developed a new benchmark and annotation framework specifically for long-horizon agent trajectory attribution. This framework standardizes heterogeneous agent trajectories using a unified component schema. It provides detailed annotations for the primary attribution component, along with attack and execution chains where relevant, covering aspects like task-aligned actions, unsafe actions, and safety refusals. The benchmark is instantiated with over 1,300 annotated trajectories from various agent settings, defining two evaluation tasks: primary attribution localization and attribution-chain recovery. Reference baselines demonstrate significant performance differences across these tasks, highlighting the challenges involved. Beyond this initial release, a reusable annotation skill is provided, allowing new agent models to be standardized, annotated, and evaluated within the same framework, fostering consistent and comprehensive analysis of agent behavior.