New Method Improves LLM Domain Adaptation Without General Data Loss

This research addresses the challenge of domain post-training for large language models (LLMs), where specializing a model for a target domain often degrades its general-purpose capabilities. The proposed solution is Self-Specialized Teacher Distillation (SSTD), a two-stage process designed to mitigate this "catastrophic forgetting" without requiring a representative replay corpus of general data. In the first stage, a copy of the base model is trained to become a domain-specific teacher, incorporating target supervision, base-aware key-token weighting, and distribution alignment to the frozen base model. The second stage involves distilling the teacher's token distribution to a student model using prefixes sampled from the student itself, ensuring domain feedback is applied to states the student is likely to encounter during inference. Evaluations across financial, medical, and legal reasoning tasks show SSTD significantly improves target domain performance while maintaining general suite scores, outperforming direct fine-tuning in preserving base model behaviors.