On-Device Distillation: Small Models Learn Differently from Diverse Teachers

Distilling large language models into smaller, on-device models offers significant advantages in terms of latency and cost for high-volume tasks like structured text extraction. This study investigates the effectiveness of such distillation, specifically mapping news articles to JSON objects with summaries and categorical labels. An 8B reasoning teacher model (deepseek-r1:8b) was distilled into a 0.6B student model (Qwen3-0.6B), with additional controls from a same-size non-reasoning teacher and a larger managed pipeline. The results show that the student model, running significantly faster than the teacher, recovered 58% of the summary quality gap, outperforming baselines. Crucially, the nature of the teacher model dictated the capabilities transferred: the reasoning teacher improved writing quality, while the managed pipeline enhanced label diversity. A same-size non-reasoning teacher yielded no improvement over the untuned base model for summary quality, indicating the importance of the teacher's reasoning capabilities. Interestingly, the study also observed that while the reasoning-lineage student excelled in quality, it sometimes fabricated information on short, thin-source articles, whereas a same-size instruction teacher's student remained more grounded. This suggests that no single teacher or distillation strategy is optimal for all tasks, necessitating a per-field routing map for on-device enrichment to leverage the strengths of different teacher lineages.