Adaptive Anchoring Improves On-Policy Self-Distillation

On-policy self-distillation (OPSD) is a technique used to adapt language models by having a student model learn from a frozen teacher, based on trajectories generated by the student itself. A critical limitation of OPSD is its reliance on the quality of these student-generated trajectories. If the student's rollouts deviate significantly from the desired target trajectories, the teacher's guidance becomes less effective, a problem termed "rollout-conditioned signal degradation." To overcome this, a new unified training framework called Adaptive Supervised Anchoring has been proposed. This framework introduces two distinct but complementary supervision pathways. One pathway maintains the traditional rollout-conditioned distribution matching, providing guidance relevant to the states the student actually explores. The second pathway applies supervised cross-entropy directly on canonical ground-truth contexts, which prevents the issue of trying to impose target tokens on potentially erroneous student-generated prefixes. The strength of this "canonical-context anchor" is adaptively adjusted based on the token-level alignment between student rollouts and target trajectories. It is emphasized during the initial "cold start" phase when student quality is low and gradually relaxed as the student's rollout quality improves. Experiments across various model scales, task types, and reasoning benchmarks demonstrated that this adaptive approach significantly enhances task acquisition compared to standard OPSD, while also better preserving the model's general capabilities. This leads to a more balanced plasticity-stability tradeoff, highlighting context quality as a key bottleneck in OPSD.