New Framework Quantifies LLM Reasoning Effort in Chain-of-Thought Steps

A new research paper introduces the Step-Aware Reasoning Energy (SARE) framework, designed to provide a granular understanding of how large language models (LLMs) expend computational effort during their chain-of-thought (CoT) reasoning processes. Unlike previous methods that offer only a high-level view, SARE quantifies effort at the individual step level. The SARE framework leverages Centered Kernel Alignment (CKA) to analyze the hidden states of tokens across transformer layers, capturing the intricate relational structure between tokens. This allows for the identification of "phase-like transitions" in reasoning energy that are otherwise invisible to broader, trajectory-level metrics. Experiments across various benchmarks and LLMs revealed that reasoning energy is highly uneven across different step types. Notably, incorrect reasoning trajectories consistently exhibited lower energy at crucial decision points. SARE-based features also proved effective in predicting outcomes, often outperforming output-based confidence measures, suggesting that internal geometric dynamics hold significant predictive information.