New Method Stabilizes Multi-Turn LLM Self-Distillation

Multi-turn on-policy self-distillation (OPSD) for Large Language Models (LLMs) can be fragile, especially when intermediate replies contain errors, leading to 'dirty-history' rollouts. Uniform distillation in such scenarios can waste computational effort on tokens that offer little corrective signal. Researchers introduce SMOPD (Selective Masking for On-Policy Distillation), a loss-only stabilization technique for multi-turn OPSD. SMOPD addresses this by ranking token positions by student entropy and removing the lowest-entropy 20% from the distillation loss for middle-turn replies. This approach ensures that the learning process focuses on tokens where the student is most uncertain, thereby providing a stronger corrective signal. Evaluations on the LiC benchmark with Qwen3 models show that SMOPD improves accuracy by 1.0-2.5 percentage points, outperforming baselines. The method adds no parameters and incurs zero inference-time overhead, suggesting that token-level uncertainty is a more reliable stabilization signal than scalar final-answer correctness in dirty-history OPSD settings.