FSMs Predict LLM Agent Failures and Next Steps

LLM-based agents, while capable of executing complex multi-step tasks, often produce opaque behavioral traces that hinder safety auditing and runtime monitoring. Existing analysis methods typically focus on individual traces or successful outcomes, missing the overarching structural patterns that link next-step and failure prediction. This research addresses this by proposing a novel approach: collapsing an entire corpus of agent traces into a single, compact finite-state machine (FSM). These FSMs serve as a structural foundation for understanding the otherwise unpredictable behavior of LLM agents. Across twelve public datasets, the generated FSMs proved to be compact, accurately replaying held-out data with high fitness, and building rapidly. This structural substrate effectively supports both prediction goals. For next-step prediction, the FSM-state context consistently outperformed existing methods. For failure prediction, behavioral features derived from the FSM achieved high accuracy in identifying failing runs, enabling early stopping from partial traces. This suggests that the behavioral topology of LLM agents is significantly influenced by the deployment environment rather than solely the LLM itself, providing a model-agnostic primitive for enhancing safety and monitoring.