Distillation Improves Compact LLM Math Reasoning Accuracy

Researchers explored the effectiveness of knowledge distillation in enhancing the mathematical reasoning abilities of smaller language models. They successfully transferred knowledge from a powerful large reasoning model, DeepSeek-R1, to a more compact student model, Qwen2.5-7B. This was achieved by creating a specialized Chain-of-Thought (CoT) training dataset derived from historical mathematics competition problems, using a dual-agent framework. The student model was fine-tuned using Low-Rank Adaptation (LoRA) on this CoT corpus. While the base Qwen2.5-7B model had 64.67% accuracy and the teacher DeepSeek-R1 achieved 91.40%, the fine-tuned student model reached a mean accuracy of 69.43% on the competition dataset, representing a notable 4.76 percentage-point improvement. It also generalized well to the MATH-500 benchmark, achieving 73.1% accuracy. A crucial finding was the impact of response length on reasoning quality, with accuracy declining significantly as response length was reduced, particularly for complex reasoning levels.