Strategy-Guided Optimization Improves LLM Reasoning Beyond Imitation.

This research proposes Strategy-Guided Policy Optimization (SGPO), a novel approach to improve the reasoning capabilities of large language models (LLMs) by focusing on distilling reusable problem-solving strategies. Unlike traditional trajectory imitation, which often leads to memorization of specific steps, SGPO aims to transfer the "how to reason" rather than just "what to answer" from powerful models to weaker ones, thereby enhancing generalization to new problems. SGPO operates by extracting structured strategy descriptions and constructing both autonomous and strategy-guided trajectories for each problem, allowing for direct comparison. It employs a token-level forward-KL objective to selectively transfer the distributional shift induced by strategy conditioning into the unguided policy, ensuring stability with proximal constraints. Additionally, adaptive instance-level weighting adjusts guidance based on the model's evolving competence. Experiments across four mathematical benchmarks demonstrate that SGPO consistently surpasses existing methods like SFT and on-policy RL, significantly improving average scores and highlighting the effectiveness of strategy distillation over direct trajectory imitation.