OAT Debugs LLM Agent Failures Using Success Trajectories.

Debugging and improving LLM-based agentic systems critically depends on identifying the specific steps within a failed task trajectory that led to the failure. Current methods for this "failure attribution" are often either computationally expensive, relying on extensive prompting, or require costly, difficult-to-scale step-level error annotations on failure data for post-training. This research proposes a more practical and efficient approach called OAT, which tackles the problem of unsupervised failure attribution. OAT is designed to be lightweight and trainable without the need for step-level supervision on failure data; instead, it learns exclusively from successful trajectories. OAT frames this problem as a one-class learning task, utilizing neural controlled differential equations to model the dynamic patterns observed in successful trajectories within a latent space. During inference, when a failure trajectory is encountered, each step is assigned an anomaly score based on how much it deviates from the learned dynamics of success. This score then helps identify the error steps. Experiments show that OAT, trained on just 100 successful trajectories, is 200-5000 times faster than prompting-based baselines and consistently outperforms them in F1 scores across both in-domain and out-of-distribution datasets, demonstrating its efficiency and effectiveness in diagnosing agentic system failures.