New Scaling Laws for Task-Specific LLM Distillation Revealed

Researchers have investigated the empirical scaling laws governing the distillation of large language models (LLMs) for specific tasks. Their work focuses on understanding how performance in both domain-specific and general knowledge tasks changes as LLMs are compressed, considering factors like dataset size, compression ratio, and the type of supervision used. The study specifically compares logit-based and LoRA-based distillation techniques under iterative structural pruning, introducing a novel blended chain-of-thought supervision loss. This new loss function is shown to stabilize KL-divergence distillation over reasoning traces, proving crucial for maintaining general knowledge capabilities that might otherwise be lost during the pruning process. The findings indicate that while in-domain task quality predictably declines with compression, general knowledge benchmarks suffer much earlier. The supervision format, particularly chain-of-thought, is a key factor in mitigating this trade-off, actively helping to recover general knowledge. The team has released a dataset, FinHeadlineMix, and practical recommendations to aid in domain-specific compression decisions.