D$^3$-MOPD Boosts Multi-Teacher Distillation Efficiency with Dynamic Scheduling.

Multi-teacher on-policy distillation (MOPD) is a technique where a single student model learns from multiple specialized "teacher" models, each expert in a different domain. A common issue with existing MOPD approaches is their reliance on a fixed data mixture across domains throughout training. This fixed approach is inefficient because different domains converge at varying speeds, leading to wasted computation on fast-converging domains and insufficient training for slower ones. To address this, researchers propose D$^3$-MOPD (Dynamic Domain ScheDuling for MOPD), a zero-overhead scheduler that intelligently adapts the domain mixture online. It repurposes the per-domain reverse-KL divergence signal, which is already generated during training, to track each domain's progress. An asynchronous off-process watcher monitors these signals, estimates remaining learning potential and current improvement rates, and then adjusts the domain sampling ratios without interfering with the core training loop. D$^3$-MOPD scales effectively to any number of domains, with benefits increasing as domain convergence patterns become more diverse. In experiments, a Qwen3.6-35B-A3B student model distilled from four domain-expert teachers using D$^3$-MOPD closed 97% of the average student-to-teacher performance gap, a significant improvement over vanilla MOPD's 63%. It also achieved the same peak performance with approximately a three-fold reduction in rollout steps and even surpassed specialist teachers on several benchmarks.