SHAPE Framework Decodes LLM Math Reasoning, Improves Accuracy.

This paper presents SHAPE, a novel framework designed to dissect and understand the mathematical reasoning processes within large language models (LLMs) when they employ Chain-of-Thought (CoT). SHAPE analyzes CoT trajectories using two key concepts from mathematics education: semantic spaces, which represent the model's evolving mathematical interpretations, and heuristics, which are the specific mathematical actions taken. By applying SHAPE, the researchers found that the heuristics an LLM uses are a stronger predictor of correct answers than traditional CoT features. Furthermore, models tend to achieve correct solutions by concentrating their reasoning within a limited number of semantic spaces, mirroring human problem-solving strategies. The framework also served to evaluate the impact of post-training on mathematical proficiency, revealing that reinforcement learning can lead to mode-seeking in heuristic usage. The study successfully demonstrated that post-training LLMs by encouraging diverse heuristics can improve accuracy in mathematical reasoning tasks. SHAPE thus provides a theoretically grounded diagnostic tool for LLM reasoning and opens new avenues for enhancing LLM performance in mathematics.