TurnOPD Boosts Long-Horizon AI Agent Training Efficiency

Training AI agents for complex, multi-step tasks often uses on-policy distillation (OPD), where a student model learns from a more capable teacher. However, traditional OPD struggles with long-horizon tasks due to two main issues: inefficient use of computational resources on less critical "tail turns" of a trajectory, and an imbalance in learning where early decisions are over-emphasized while later, deeper decisions are under-trained. To overcome these limitations, researchers developed TurnOPD. This new approach employs a turn-level budgeting system. It dynamically adjusts the length of agent rollouts based on real-time turn statistics and gradually shifts the focus of the learning loss from token-level to a more balanced, turn-level supervision. Experiments across various agentic environments like ALFWorld and WebShop demonstrated that TurnOPD significantly improves validation accuracy within the same training time budget compared to standard OPD. This advancement makes training more efficient and effective for agents tackling intricate, sequential decision-making problems.