ACPO Improves LLM Reasoning with Adaptive Credit Assignment

Reinforcement Learning (RL) has significantly boosted the reasoning capabilities of Large Language Models (LLMs). However, the challenge of assigning credit at the token level, especially with sparse outcome rewards, remains a major hurdle. Traditional scalable RL methods often apply trajectory-level rewards uniformly, while existing entropy-aware approaches either use detached heuristics or directly optimize true entropy, which can lead to misaligned gradients. This research proposes Adaptive Credit Policy Optimization (ACPO), a new framework designed for fine-grained, token-level credit assignment. ACPO leverages a mode-local surrogate entropy to adaptively adjust policy updates. It strategically emphasizes uncertain decisions in successful sequences and, conversely, targets overconfident tokens in failed sequences. The authors demonstrate that this surrogate entropy allows for deterministic bounds and, under specific conditions, preserves the policy-gradient direction. Experiments on demanding benchmarks like AIME 2025 for mathematical reasoning and HumanEvalPro for coding show that ACPO consistently outperforms strong RL baselines such as DAPO, GTPO, and SAPO, indicating a significant step forward in making LLMs more robust and intelligent.