PRISM Framework Optimizes Multi-Reward RL for LLMs

Large language models (LLMs) are increasingly expected to align with multiple human values and use cases, leading to the challenge of multi-reward reinforcement learning (RL). A common problem in this area is the "alignment tax," where different optimization objectives can conflict, leading to unstable and inefficient training. Traditional approaches often try to combine or mix these diverse rewards. This research proposes PRISM (Policy-space decomposition and composition for Multi-Reward RL), a novel multi-reward RL framework that tackles this issue by decomposing and composing policies rather than rewards. Instead of blending conflicting reward signals, PRISM optimizes a set of individual "positive" policies, each targeting a specific desired behavior, alongside a single "global negative" policy to avoid undesirable behaviors. This policy-space decomposition alleviates conflicts during optimization and offers enhanced controllability during inference through flexible policy composition. Experiments across scientific reasoning, tool-use reasoning, and helpfulness-safety alignment tasks demonstrate that PRISM consistently outperforms existing multi-reward RL baselines. It also provides additional control for fine-tuning preferences at inference time, making LLMs more adaptable to diverse user needs.