New Middleware Improves LLM Reasoning with Process-Supervised Reinforcement Learning

Training large language models (LLMs) to reason effectively often involves process-supervised reinforcement learning (RL), where feedback is provided at intermediate steps. A common technique, Group Relative Policy Optimization (GRPO), faces several challenges when integrating dense process signals, such as "channel contamination" from mixed signal streams, "resolution mismatch" between signal granularity and logical decisions, and a "cumulative trap" affecting exploration. To overcome these issues, a new middleware called PASS (Process Advantage Signal Shaping) has been developed. PASS introduces three key components: Advantage Fusion for independent signal standardization, Chunk-by-Value for credit assignment within value-homogeneous segments, and Divide-Length to convert cumulative objectives into average-value-density scores. Experiments across different domains and signal paradigms demonstrate that PASS consistently improves LLM reasoning performance over GRPO baselines.