Influence Calibration Improves Agentic RL Self-Distillation.

In the realm of agentic Reinforcement Learning (RL), on-policy self-distillation (OPSD) is a technique where a language agent receives token-level supervision from a "self-teacher" based on its own trajectories. A critical limitation of current OPSD methods is their reliance on "teacher trust" to allocate supervision, which often fails to align with the actual policy objective. This "trust-utility mismatch" can lead to inefficient or even counterproductive learning. To address this, researchers developed Influence Calibration for Self-Distillation (ICSD). ICSD quantifies the first-order response of a token's contribution to the RL surrogate objective when perturbed by the teacher's guidance. This non-stationary signal is then converted into a bounded allocation weight, ensuring that supervision is directed towards tokens that genuinely support the policy's objective. Experiments across ALFWorld, WebShop, and Search-QA benchmarks, using various model families, showed that ICSD consistently improved aggregate metrics over trust-only allocation, achieving high success rates and significantly reducing the mass of teacher-supported tokens that opposed the objective.