Dynamic Support Learning Enhances Reinforcement Learning Value Estimation

Traditional reinforcement learning (RL) often estimates value functions using regression, but distributional RL models a distribution of returns. A recent technique, Gaussian Histogram Loss (HL-Gauss), reframes value estimation as a classification problem by encoding scalar Bellman targets into Gaussian-smoothed categorical targets. However, a key challenge with histogram-based losses in RL is the need to pre-define a fixed support interval, which is difficult given the non-stationary and stochastic nature of RL target values. This research proposes a novel method that dynamically learns the lower and upper bounds of this support interval, rather than setting them beforehand. The objective function jointly optimizes these bounds with the categorical representation, forming a tighter upper bound on the mean-squared Bellman error. Empirical results show that this approach leads to more stable adaptation of the support interval and matches or improves performance on various continuous-control tasks compared to HL-Gauss, all without requiring prior specification of the support range.