Thinking Checklist Reward Improves LLM Preference Alignment

Current methods for aligning Large Language Models (LLMs) with human preferences, often using reinforcement learning, typically rely on outcome-level rewards that evaluate only the final response. This approach can lead to coarse credit assignment, as it provides limited guidance on the reasoning process itself, especially when multiple responses receive similar scores. To address this, researchers propose the Thinking Checklist Reward (TCR), a novel process-oriented reward mechanism. TCR converts human preference pairs into specific "thinking checklists" for each sample. It then evaluates whether the LLM's generated reasoning trace adequately addresses the considerations implied by these preferences. To ensure TCR provides complementary guidance and avoids overlap with outcome-level supervision, it incorporates an exponential moving average (EMA) residual formulation. This isolates a "thinking surplus" that goes beyond what is predictable from the final outcome reward. Experiments across five models from three different model families demonstrate that TCR consistently enhances alignment performance on various benchmarks, with ablations confirming the importance of both the EMA-based residual formulation and the sample-specific checklist supervision.