ADRS Enhances Agentic RL with Self-Distilled Reward Shaping

Agentic reinforcement learning (RL) allows Large Language Model (LLM) agents to learn through interaction, but often struggles with sparse, trajectory-level rewards that don't pinpoint which intermediate decisions were crucial. While privileged skills can offer denser supervision, existing methods lack joint calibration of teacher scores across steps, relating confidence to returns, and integrating this signal into RL's reward-to-advantage construction. ADRS (Agentic Reinforcement Learning with Self-Distilled Reward Shaping) addresses these limitations by providing return-associated token-level credit for multi-turn language agents. ADRS centers and normalizes privileged token scores, modulates them with a Teacher Value Advantage (TVA) gate based on confidence-return association, and incorporates this gated signal into the native RL credit path. This framework determines teacher preferences, their relevance to returns, and how they influence RL, all while keeping rollouts and inference skill-free. Experiments across three interactive benchmarks show ADRS consistently improves performance on long-horizon tasks, demonstrating gains across various RL backbones and data settings.