PHF Improves LLM Reasoning by Distilling Teacher's Internal States

On-policy self-distillation (OPSD) is a technique used to train reasoning models by having them learn from their own generated outputs, guided by a "privileged teacher" that has access to correct solutions. Current OPSD methods primarily supervise only the output distribution, meaning the teacher's internal computational process isn't directly leveraged. A new approach, Privileged Hidden Flow (PHF), addresses this by additionally distilling how a privileged teacher's hidden states evolve along the same generated sequence. Instead of forcing exact hidden state matches, PHF aligns the token-to-token transition directions and the overall trajectory geometry of the hidden states. This method, which includes an all-layer recipe and adjacent-layer relations, consistently improves aggregate performance across various Qwen models, demonstrating a more effective way to transfer internal reasoning capabilities from a teacher to a student model.