AsyncOPD Improves LLM Distillation Through Asynchronous Training.

Researchers have introduced AsyncOPD, a novel asynchronous on-policy distillation (OPD) training pipeline designed to enhance the efficiency of large language model (LLM) post-training. OPD, which trains a student model on its own rollouts guided by a teacher, often faces an on-policy systems bottleneck where rollout generation dominates training time. Asynchronous pipelines can alleviate this by decoupling rollouts from learner updates, but this introduces stale-policy data, an area previously underexplored in OPD. The study systematically investigates staleness in asynchronous OPD, particularly when teacher feedback uses local KL losses and full-vocabulary teacher logits are impractical. Key findings include that teacher-weighted forward KL is more robust to stale rollouts, while student-weighted reverse KL is vulnerable. For the vulnerable reverse-KL case, methods from asynchronous reinforcement learning did not outperform a simpler OPD-specific surrogate: recomputing the reverse-KL signal with the current student at learner time. Furthermore, the research analyzes how finite teacher-score caches create a bias-variance tradeoff for sparse and sampled reverse-KL OPD estimators, motivating the use of multi-sample Monte Carlo to reduce variance. AsyncOPD, built on these insights, demonstrates a 1.6x to 3.8x improvement in training throughput over strict synchronous training while maintaining comparable accuracy.