Woodpecker Distillation Fixes LLM Reasoning Bugs with Weak Models

Large language models often struggle with reasoning tasks, not due to a lack of capability, but because of localized errors in intermediate steps. This new research introduces Woodpecker Distillation, a novel training framework designed to identify and correct these reasoning bugs in powerful models using insights from weaker "probe" models. The method involves inserting short, corrective patches generated by a weak model after a strong model's reasoning prefix. While direct fine-tuning on these patches doesn't reliably work, the key insight is that the useful signal lies in how these interventions reshape the strong model's future reasoning distribution. Woodpecker Distillation contrasts successful and unsuccessful weak-model patches at the same prefix to construct a corrective teacher distribution. This teacher distribution, derived from induced future token predictions, is then distilled into the strong model. Experiments on mathematical reasoning benchmarks demonstrate that Woodpecker Distillation consistently improves the performance of strong models, outperforming traditional direct imitation baselines by effectively learning from these contrastive local interventions.