ERR+ Improves LLM Reasoning Accuracy and Conciseness.

Large Language Models (LLMs) often achieve strong performance on complex tasks by generating detailed chain-of-thought (CoT) traces, typically optimized through reinforcement learning with verifiable rewards (RLVR). However, existing RLVR methods primarily focus on correctness, offering limited guidance on the quality or efficiency of the reasoning process itself. This research introduces ERR+, a novel two-phase RLVR framework designed to address these limitations. Empirical analysis revealed that correct reasoning traces exhibit more frequent and significant token-level entropy drops during the thinking phase compared to incorrect ones. Building on this insight, the first phase of ERR+ trains models using an Entropy Relief Reward (ERR). This reward is proportional to the cumulative token-level entropy drops, normalized by response length, encouraging the model to resolve uncertainty efficiently without stifling initial exploration. The second phase introduces the Robust Relative Efficiency Reward, which scores a response's length against co-generated peers, promoting conciseness. A formal analysis justifies this sequential design, showing that joint optimization initially leads to gradient conflicts. Experiments across five datasets and various model backbones demonstrate that ERR+ consistently improves both the accuracy and conciseness of LLM outputs, offering a more optimized approach to LLM reasoning.