On-Policy Distillation Outperforms Offline Learning with Noisy Experts

Imitation Learning (IL) is a fundamental approach for sequential decision-making, notably influencing language model training. A puzzling observation is that online methods, such as on-policy distillation (OPD), frequently outperform offline methods like supervised fine-tuning, despite theoretical claims of offline IL's optimality. This paper offers a theoretical explanation for this discrepancy by introducing a "noisy expert" model. In this model, the learner only has access to an imperfect version of the expert's policy, yet aims to match the performance of a clean, ideal expert. The research demonstrates a stark difference between offline and online IL in this scenario. Offline learning from noisy trajectories is shown to be inherently difficult, requiring sample complexity that grows exponentially with the task horizon to compete with a clean expert. Conversely, online interaction with the noisy expert via a novel variant of OPD achieves polynomial dependence on the horizon. The paper further shows that under specific, natural conditions on expert noise, a horizon-free sample complexity can be obtained, although with some statistical efficiency trade-offs. This analysis provides a theoretical foundation for why OPD is often more effective than supervised fine-tuning when training language models from imperfect teachers.