LAPO Improves Multi-Turn Search Reasoning with Self-Generated Rewards.

Reinforcement learning systems designed for multi-turn search reasoning often rely solely on final outcome rewards, which makes it difficult to differentiate between helpful, redundant, or even detrimental intermediate steps. To address this, a new method called LAPO (Leave-One-Turn Attribution for Self-Generated Process Rewards) has been introduced. LAPO employs a backward leave-one-turn attribution technique. For each step in a search sequence, it temporarily removes that step and its associated observation, then measures the impact on the policy's likelihood of reaching the correct answer. This approach allows for an estimation of each turn's contribution within the full reasoning context. The method also incorporates sign-consistency gating to refine these attributions, leading to more effective process supervision without the need for additional reward models or human judges. Experiments across several knowledge-intensive question-answering datasets show LAPO significantly improves exact-match scores compared to strong baselines.