ExTra Improves Language Model Reasoning with Exploratory Trajectory Optimization

Reinforcement Learning with Verifiable Rewards (RLVR) for language models often struggles with tasks that are either too easy or too hard, leading to insufficient gradient signals. To address this, researchers developed ExTra, or Exploratory Trajectory Optimization, a new framework compatible with GRPO. ExTra aims to improve exploration during the learning process. The ExTra framework incorporates two main mechanisms. First, it uses a novelty reward that adds diversity bonuses based on embedding, encouraging the model to find varied correct solutions. Second, it employs entropy-guided prefix regeneration, which evaluates partial trajectories using entropy signals and allows for continued exploration from promising intermediate steps. This approach helps the model navigate complex problem spaces more effectively. Evaluations across six mathematical reasoning benchmarks showed significant improvements. ExTra boosted the Qwen3-1.7B model's performance over GRPO by approximately 5 points on pass@1 and 7 points on pass@16. These results highlight that incorporating trajectory-level exploration signals can substantially enhance both single-sample accuracy and the overall coverage during inference.