Weak-to-Strong Distillation Improves LLMs Beyond Stronger Teachers.

Researchers have developed a new method called Weak-to-Strong On-Policy Distillation (W2S-OPD) that allows a powerful large language model (LLM) to improve its capabilities by learning from multiple less capable models. Unlike traditional on-policy distillation, which typically requires a teacher model that is at least as strong as the student, W2S-OPD constructs a "proxy teacher" in the logit space. This proxy teacher is derived from the logit difference between a positive and a negative model, both of which are smaller and cheaper to obtain than the student. The core idea is that the logit difference between these weaker models can isolate specific capability directions, which are then added to the student's own base model. This creates a proxy teacher that is distributionally similar to the student but incorporates the desired skill. The student then distills this knowledge by minimizing the per-token reverse KL divergence on its own generated rollouts. The paper demonstrates three ways to instantiate these contrast pairs: using a post-RL expert against its pre-RL initialization, a larger base model against a smaller one, or a small base model with correct versus wrong hints. Across seven benchmarks spanning mathematical reasoning and code generation, W2S-OPD consistently outperforms standard on-policy distillation. Remarkably, it enables the student model to surpass the performance of even domain-specific teachers, continuing to improve the student even when all supervision sources are individually weaker. Analysis suggests that different contrast pairs emphasize distinct learning signals, such as reasoning frameworks or solving procedures.