CANON Improves LLM Reasoning with Label-Free Self-Distillation

Improving the reasoning accuracy of large language models (LLMs) without relying on expensive human-labeled data is a significant challenge. While sampling multiple solutions and taking a majority vote is a reliable way to boost accuracy, existing methods often use this consensus signal in limited ways, such as filtering or scalar rewards, discarding much of its rich information. Researchers have introduced CANON (Consensus-ANchored self-distillatiON), a novel label-free training technique that transforms this consensus into dense, token-level supervision. For any given unlabeled prompt, CANON generates multiple solutions, identifies the majority answer, and then conditions a frozen version of the model on a solution that aligns with this consensus. This "consensus-anchored teacher" then supervises the model's own rollouts at every token, providing fine-grained guidance. Experiments on mathematical and scientific reasoning benchmarks demonstrated that CANON substantially improves pass@1 scores, sometimes by as much as 12 points. It significantly outperforms label-free reinforcement learning methods, achieving similar performance to models trained with gold labels but at a fraction of the computational cost. Analysis indicates that CANON doesn't just sharpen the distribution of existing knowledge; it enables the model to solve problems it previously couldn't, and its internal majority vote becomes more accurate post-training.