RefGRPO Closes Agent Reflection Gap, Improves RL Performance.

Large Language Models increasingly function as agents interacting with environments and receiving feedback, such as execution results or error messages. A critical capability for these agents is to accurately self-assess their performance based on this feedback. However, a "reflection gap" often exists, where LLM agents misjudge their own outputs even after observing concrete outcomes, and standard Reinforcement Learning (RL) struggles to correct this due to credit-assignment issues. To bridge this gap, the researchers propose RefGRPO, a straightforward yet effective enhancement for standard RL algorithms. RefGRPO incorporates two main components: a "free calibration bonus" that is calculated by comparing the agent's self-reflection with the actual environmental outcome, requiring no additional reward models or external annotations; and a dynamic schedule for this bonus's coefficient. Compared to conventional RL baselines, RefGRPO simultaneously boosts reflection calibration, significantly reducing underconfidence rates, and improves task accuracy across various text-to-SQL benchmarks. This calibrated reflection transforms the agent into a self-verifier grounded in environmental feedback, enabling better self-improvement without outcome supervision and more effective selective prediction during testing.