New Metric Improves LLM Reinforcement Learning with Verifiable Rewards.

Reinforcement Learning (RL) is increasingly vital for enhancing Large Language Models (LLMs) beyond simple imitation, particularly through paradigms like RL with Verifiable Rewards (RLVR) which aims to improve reasoning. However, there's been conflicting advice on which tokens to prioritize during RLVR training: some suggest high-entropy tokens, while others caution against low-probability tokens. Both approaches have shown empirical success, creating a tension in understanding optimal policy optimization dynamics. To resolve this, the researchers propose the Relative Surprisal Index (RSI), a principled, information-theoretic metric that effectively combines a token's entropy with its selected probability. RSI is shown to relate to the local ratio between logit-gradient norm and predictive entropy variations. Building on RSI, they introduce RSI Selection (RSI-S), an entropy-adaptive token filtering method. RSI-S intelligently filters out both redundant low-surprisal tokens and unstable high-surprisal tail tokens, reconciling previous contradictory paradigms. Empirical evaluations on AIME and AMC benchmarks demonstrate that RSI-S consistently improves average accuracy by 2-3 percentage points over existing methods like GRPO across various model scales, offering a promising new perspective for RLVR enhancement.