Multi-Task Distillation with Soft Prompts Boosts LLM Skills

On-policy self-distillation (OPSD) is a technique where a large language model (LLM) learns new skills by having a "teacher" model, often sharing the same backbone, supervise its own generated outputs. Existing OPSD methods either inject privileged context at the input, which can lead to post-hoc rationalization, or fine-tune model weights, which risks accumulating drift and forgetting across different tasks. These limitations make it challenging for a single student model to efficiently learn from a diverse set of skills. Researchers propose a new method called "Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context." In this approach, the teacher model differs from the student only by a learnable soft prompt. This prompt is trained on gold-standard data pairs with the student's backbone frozen, creating a task-specific teacher that preserves the student's exact representational geometry. This design avoids the issues of weight fine-tuning and input-level context injection. The method naturally extends to multi-task scenarios by routing each example in a combined dataset to its corresponding soft-prompt teacher. This allows a single student to simultaneously learn from multiple teachers, absorbing knowledge in parallel. During inference, all soft prompts are discarded, ensuring no added complexity. Experiments on models like Qwen3-1.7B-Base and Phi-4-mini-instruct across various tasks show that the single-task variant matches or exceeds full fine-tuning with significantly fewer parameters, while the multi-task variant achieves superior overall average performance without degrading general capabilities, unlike sequential fine-tuning.