Shapley Values Improve Reward Attribution for LLM Parallel Reasoning

Researchers have developed Parallel Shapley, a novel reinforcement learning (RL) framework designed to enhance multi-step reasoning in Large Language Models (LLMs). Current parallel reasoning approaches often struggle to accurately assess the individual contributions of different reasoning paths, leading to ambiguous learning signals when uniform rewards are assigned to the overall outcome. Parallel Shapley addresses this by treating each reasoning path as a player in a cooperative game. The framework leverages Shapley values to quantify the marginal contribution of each path, using a generative reward model to evaluate path utilities and Monte Carlo sampling for efficient approximation. Experiments on mathematical reasoning benchmarks demonstrate that Parallel Shapley not only outperforms existing baselines but also provides more stable and interpretable training. By effectively "fishing out free riders" and assigning rewards proportionally, this method significantly improves the quality of multi-path reasoning in LLMs.