VISTA Improves LLM Reasoning with Verifier-Informed Self-Distillation

On-policy self-distillation (OPSD) is a technique used to enhance the reasoning capabilities of large language models (LLMs). It involves training a "problem-only" student model using dense, token-level supervision from a "privileged" teacher model that has access to a reference solution. A limitation of standard OPSD is that it treats the teacher's distribution as a fixed target, updating only the student. This can misdirect the student if the teacher's output, even with privileged conditioning, isn't perfectly aligned with optimal problem-only reasoning. To address this, a new method called VISTA (Verifier-Informed Student-to-Teacher Adaptation) has been developed. VISTA maintains the standard OPSD student update process but introduces a crucial modification: it uses outcome-verified rollouts to adapt the teacher model towards the student's distribution. This adaptation is further refined within each verified rollout, focusing on the top-k positions where the teacher-student KL divergence is largest. Notably, VISTA reuses the existing rollout and loss function from standard OPSD, meaning it doesn't require additional sampling or a separate reward objective. Experiments conducted on AIME24, AIME25, and HMMT25 benchmarks using Qwen3 models of varying sizes (1.7B, 4B, and 8B) demonstrate VISTA's effectiveness. It consistently achieves the highest Avg@12 scores at every scale, improving over OPSD by 0.6, 0.7, and 2.1 points respectively. These results highlight the value of student supervision from verified rollouts and the potential of student-to-teacher adaptation in OPSD.