DASH-OPD Improves Multi-Turn Agent Training Efficiency

Training multi-turn AI agents using on-policy distillation (OPD) can be challenging because early errors by the student model can quickly lead the agent's trajectory away from the teacher's familiar domain, a problem known as exposure bias. Existing curriculum learning methods regulate teacher support based on training progress but lack the ability to dynamically determine when that support is most needed.To address this, the new DASH-OPD (Discrepancy-Aware Switching with Hysteresis for OPD) method has been developed. DASH-OPD allows for adaptive, bidirectional switching between student and teacher executors. On each turn, it calculates a mean log-probability ratio between the two executors to measure their discrepancy. Student-to-teacher ratios indicate drift, while teacher-to-student ratios signal recovery.These signals are normalized and accumulated over multiple turns, forming "drift" and "recovery" evidence. DASH-OPD switches executors only when this evidence surpasses predefined thresholds, incorporating hysteresis to prevent frequent, unstable switches caused by minor fluctuations. Experiments on the ALFWorld benchmark show that DASH-OPD outperforms all baselines, demonstrating superior training and deployment efficiency for multi-turn agents.