SADQ Improves Q-Learning Stability by Regularizing TD Target Aggregation

Deep Q-Networks (DQNs) are a cornerstone of reinforcement learning, learning value functions through bootstrapped temporal-difference (TD) updates. A critical component of this is the greedy maximization over next-state action values, which approximates future returns. However, this aggregation rule is highly susceptible to estimation noise: when Q-values are uncertain, the maximization operator deterministically selects the largest estimate, regardless of its reliability. This can lead to amplified errors that propagate through bootstrapping, causing instability. To address this, researchers propose the Successor Rollout Aggregation Deep Q-Network (SADQ). SADQ is a straightforward modification to Q-learning that introduces regularization into the TD target formation process. It leverages one-step rollout predictions from a learned dynamics model to inform the comparison between candidate next-state actions. This additional structure helps guide the aggregation step without altering the fundamental learning framework. The resulting mixed Bellman update effectively attenuates unreliable maxima while preserving the standard fixed point under diminishing model error. Theoretical analysis shows that SADQ reduces bootstrap-induced overestimation pointwise. Empirically, SADQ consistently improves training stability across various benchmarks, including classical control tasks, real-world vector-based environments, and Atari games, outperforming strong DQN variants.