Rubric-Conditioned Self-Distillation Enhances LLM Reasoning

Traditional methods for post-training reasoning language models, such as supervised distillation and reinforcement learning, often face limitations. Supervised distillation relies on expensive and potentially noisy chain-of-thought annotations, where even correct final solutions can be hindered by imperfect rationales. Reinforcement learning, on the other hand, typically compresses feedback into a single scalar reward, obscuring specific areas for improvement. To address these challenges, a new framework called Rubric-Conditioned Self-Distillation has been introduced. This approach integrates detailed rubrics as structured, fine-grained feedback for on-policy self-distillation. By conditioning the teacher model on criterion-level rubrics, it provides token-level guidance on the student model's sampled trajectories. This design moves beyond relying on a single reference rationale, instead using rubrics to specify the criteria for a strong response, enabling more precise credit assignment during the reasoning process. The framework, instantiated with a two-stage pipeline for rubric generation and rubric-guided reasoning, demonstrated superior performance on various science reasoning benchmarks, surpassing GRPO by 1.0 points and OPSD by 0.9 points on average.