FADE Advantage Function Stabilizes LLM Reinforcement Learning

Reinforcement Learning (RL) is crucial for refining Large Language Models (LLMs), particularly for reasoning tasks, but it often suffers from training instability and a loss of diversity. Advantage functions are a common solution, reshaping the training objective and reweighting which experiences drive learning. However, the proliferation of different advantage methods makes it difficult to choose the right one. This research offers a unifying framework that decomposes any advantage function along two axes: sign (balancing entropy vs. weight geometry) and difficulty (focusing on hard problems vs. sample size). The authors observe that optimal trade-offs along these axes shift throughout training: early exploration benefits from balance and hard focus, while later exploitation favors suppression and medium focus. This insight motivates FADE (Focal Advantage with Dynamic Entropy), a novel, self-adapting advantage function. FADE automatically adjusts gradient weights by reading training dynamics. Empirical results show FADE significantly accelerates training, reaching peak performance up to 20,000 steps earlier than leading static baselines at the 7B scale and 2,000 steps earlier at 32B. It also achieves the best accuracy-diversity balance across various benchmarks, including LiveCodeBench and AIME, demonstrating a more stable and efficient RL training process for LLMs.