New Method Improves LLM Process Reward Modeling with Learnable Credit Assignment.

Large Language Models (LLMs) benefit from process reward models (PRMs) that offer detailed feedback on their reasoning steps. However, training these PRMs typically demands costly, step-by-step human annotations. A more scalable approach, outcome-supervised PRMs, learns from only the final answer's correctness, but struggles with attributing credit to specific reasoning steps. This paper proposes Learnable Credit Assignment (LCA), a novel framework designed to overcome this "credit assignment" problem. LCA jointly learns how to assign credit and model rewards, operating on the principle that a reasoning chain is only as strong as its weakest link. It formalizes this as a Multiple Instance Learning problem and introduces Softmax-Weighted-Sum (SWS) pooling, a technique suited for situations where reasoning states are highly dependent and redundant. Extensive experiments demonstrate that LCA significantly outperforms existing outcome-supervised PRMs across various tasks and model architectures. This advancement allows for more effective and scalable training of LLMs by providing better feedback on their internal reasoning processes, ultimately leading to more robust and accurate models.