New Method Improves Multi-Turn AI Agent Guidance and Performance.

Training multi-turn AI agents, especially in interactive environments, often involves "privileged on-policy distillation." This technique uses a synchronized teacher model to provide dense supervision by re-scoring the student's responses at each turn, leveraging access to successful trajectories available only during training. However, a significant challenge arises when the student agent's actions diverge from the reference trajectory, leading to execution states not covered by the privileged guidance. This "state-reference mismatch" can make the teacher's feedback unreliable and counterproductive. To address this, a new framework called State-Matched Routing and Contextualized Self-Distillation (SMRC-SD) has been developed. SMRC-SD explicitly determines when and how privileged guidance should be applied. At each turn, it verifies if the student's current execution state matches a supported state within the reference trajectory. Distillation is only applied at these matched states, effectively filtering out turns where the reference lacks locally compatible guidance. Furthermore, for each matched state, SMRC-SD constructs state-conditioned teacher context from the successful trajectory, ensuring that the supervision is grounded in the actual state the student has reached. Experiments on ALFWorld and WebShop benchmarks show that SMRC-SD consistently outperforms traditional full-path distillation, significantly improving task success rates for agents. This highlights the importance of both selecting locally supported turns and providing state-compatible teacher context for effective agent training.