New Policy Optimization Improves Deep Search Agents with Step-Level Guidance

Deep search agents often struggle with sparse outcome rewards in long, multi-step trajectories, making effective credit assignment difficult. While on-policy self-distillation (OPSD) attempts to provide dense token-level feedback, applying it to search agents can create issues because the teacher model often has privileged information, leading to a systematic difference in reasoning from the student. This paper addresses this challenge by introducing Step-Level Self-Distilled Policy Optimization (SSPO). SSPO's innovation lies in two key contributions. First, it uses "Evidence Anchors," which are concise, step-level evidence snippets extracted from the web, to provide crucial reasoning steps without revealing the full answer path. Second, SSPO converts the disagreement between the teacher and student into step-level advantage weights within the GRPO framework, applying these weights exclusively to incorrect trajectories. This design ensures that outcome rewards guide the direction of policy change, while the teacher modulates the magnitude of change at each step, preserving the diversity of correct trajectories. Experiments show SSPO consistently outperforms GRPO on various benchmarks, often with fewer gradient steps.