New Policy Optimization Improves Multi-Turn LLM Agents

Training multi-turn evidence-reading agents using outcome-only reinforcement learning often leads to instability because intermediate steps receive insufficient direct feedback. A common issue, observed with GRPO on HotpotQA, is a "zero-advantage lock-in" where all sampled trajectories receive the same minimum penalty, causing relative advantages to vanish and optimization to stall. To overcome this, researchers propose Contextual Information-Gain Policy Optimization (CIGPO). This method injects variance into the reward signal by assigning per-turn rewards to intermediate evidence-reading steps. The per-turn signal is derived from the marginal increase in a frozen reference model's log-likelihood of the ground-truth answer, effectively preserving the reward variation needed for effective policy optimization. CIGPO, combined with separate normalization of information-gain and F1 rewards and an information-gain weight curriculum, achieved significant performance gains on HotpotQA. It reached a standard F1 score of 0.518, a substantial improvement over the best GRPO checkpoint and completely avoiding the optimization deadlock. This highlights reward-variance collapse as a critical failure mode and demonstrates the effectiveness of turn-level information-gain rewards.