On-Policy Self-Distillation Limits in Continual Learning Revealed

Continual post-training is a critical technique for enabling foundation models to acquire new knowledge while retaining existing capabilities. On-policy learning, particularly on-policy self-distillation, has been viewed optimistically as a method to mitigate forgetting. However, this study, using self-distillation policy optimization (SDPO), challenges that view. The findings indicate that while SDPO can accelerate in-domain specialization when teacher signals are stable, it performs poorly in out-of-distribution scenarios and can even cause models to collapse, leading to significant forgetting. In contrast, other on-policy reinforcement learning methods like GRPO adapt more conservatively and better preserve prior knowledge. Analysis suggests that denser self-distillation causes greater drift in both parameter and response spaces, potentially amplifying high-frequency artifacts through a self-reinforcing teacher-student loop. The research concludes that on-policy data alone is insufficient for robust continual learning, and dense self-distillation, while useful for specialization with stable targets, should not be considered a default stabilization strategy. The code is available.