Geometric Self-Distillation Improves LLM Out-of-Distribution Reasoning

On-policy distillation is a common technique for post-training large language models, where a teacher model provides supervision on the student's own generated text. A specific variant, privileged-context self-distillation, uses the same model as both teacher and student, but the teacher receives additional hints or full solution traces. While this provides abundant supervision, it can lead to issues: the teacher might be overly confident about continuations that are obvious with its privileged view but not yet justifiable by the student. This discrepancy can cause "drift" over many updates, degrading the student's ability to reason effectively on out-of-distribution data. To counter this, the researchers propose GeoSD, a geometric self-distillation objective. GeoSD addresses drift by treating it as movement in the student's predictive behavior. It incorporates two main components: first, a Hellinger loss scales each teacher preference based on the overlap the student already shares, reducing the pull on tokens the student cannot yet support. Second, a proximal term penalizes how far the student's predictions drift from a recent checkpoint, using a Fisher-Rao distance, which measures distance in the geometry of next-token distributions. Evaluations across mathematical reasoning benchmarks and three different model families (1.7B to 32B parameters) show that GeoSD not only maintains in-distribution performance gains but also significantly improves average out-of-distribution accuracy by 5.7-8.6 points compared to the base model. Analysis reveals that standard matching methods can lead to confident agreement on incorrect answers by draining mass from alternatives, whereas GeoSD preserves these alternatives, leading to more robust and generalizable reasoning.