New RL Optimization Breaks Exploration-Stability Dilemma for LLMs.

Reinforcement Learning (RL) is a key paradigm for enhancing the complex reasoning capabilities of Large Language Models (LLMs), but it faces an exploration-stability dilemma. Current RL frameworks rely on importance sampling (IS), which can lead to training instability, while standard clipping mechanisms used to mitigate this instability often restrict the policy update budget, stifling exploration. The paper formalizes the concept of Probability Capacity (Cap) and reveals that conservative clipping prematurely truncates update budgets for correct but low-confidence reasoning paths. To overcome this, researchers propose Unbounded Positive Asymmetric Optimization (UP), a universal and plug-and-play objective. UP fundamentally restructures the optimization process by anchoring the policy to its current state using a stop-gradient operator. This asymmetric design allows for unclipped, stable gradients for positive advantages, maximizing exploration, while retaining standard clipping safeguards for negative advantages to prevent instability. The formulation extends to different optimization granularities (token-level and sequence-level) and has been experimentally shown to enhance exploration capacity and achieve superior reasoning accuracy across diverse RL algorithms (DAPO, GSPO, GRPO), model architectures (Dense, MoE, vision-language), and training modalities (language and multimodal).