RL Framework Diagnoses, Recovers Autonomous LLM Agent Drift

Autonomous large language model (LLM) agents, increasingly deployed in complex real-world scenarios, are susceptible to "behavioral drift." This silent deviation from their intended task can lead to unintended and potentially irreversible consequences in external systems. Existing solutions primarily address drift at the prompt level, lacking granular mechanisms for step-by-step detection, risk evaluation, and recovery. This work proposes a plug-and-play recovery module, external to the main LLM agent, which is particularly useful since retraining large, expensive models for every deployment is impractical. The framework employs a graph-based structure where a single small language model (SLM) is trained via reinforcement learning to specialize at each node of a recovery graph. Each node has a specific function, such as drift classification, operation detection, risk evaluation, or final decision-making. The SLM learns to generate structured XML-formatted reasoning tailored to its node's role. Training combines rule-based structural rewards with an LLM-as-judge semantic-quality signal, ensuring the SLM is graded on both its output schema adherence and semantic appropriateness. Experiments on the AppWorld benchmark demonstrate that this method effectively utilizes information about the suspected drift onset to make accurate recovery decisions using a compact language model. The trained SLM consistently respects the prescribed output schema and produces semantically relevant content for its assigned role.