ATOD Boosts Multi-Turn Agent Training with Hybrid Distillation

Training small language models to act as autonomous agents in complex, multi-turn interactive tasks presents a dilemma: On-policy Distillation (OPD) offers fast initial learning but quickly plateaus, while Reinforcement Learning (RL) can achieve higher performance ceilings but is slow to learn due to sparse feedback. This research introduces ATOD (Annealed Turn-aware On-policy Distillation), a novel hybrid algorithm designed to leverage the strengths of both. ATOD employs an annealed schedule, where OPD dominates early training to quickly mimic a teacher's behavior, and then RL is gradually increased to drive exploration and optimize for environmental rewards. This allows the agent to benefit from dense supervision initially and then push beyond the teacher's performance. Furthermore, ATOD incorporates Turn-level Disagreement-Uncertainty Reweighting (T-DUR). This mechanism intelligently amplifies the importance of high-utility turns within long trajectories, providing more effective and targeted supervision. Experiments across various benchmarks demonstrate that ATOD consistently surpasses both pure OPD and RL methods, and even outperforms the teacher models, achieving significant improvements in success rates for multi-turn agents.