New Offline RL Method Controls Local Correction Propagation

Offline Reinforcement Learning (RL) can benefit from exploring actions that are slightly outside the distribution of the training data (out-of-distribution or OOD actions). However, errors in estimating the value of these OOD actions can be significantly amplified through the bootstrapping process, leading to unstable or suboptimal policies. Existing methods either restrict the OOD region or manage the influence of generalized targets, often through separate and complex mechanisms. This research proposes Convex Hull Neighborhood Smooth Dual Generalization (CSDG), a new approach that explicitly separates and controls the contribution of OOD corrections. CSDG re-expresses the Bellman backup as a combination of an in-sample value target and a local correction derived from the Convex Hull Neighborhood. This formulation makes the generalized contribution clear and distinct from the reference path. The correction is smoothed using candidates sampled at different perturbation radii, and a mixture coefficient scales its impact on each backup. The paper provides theoretical analysis, including an exact one-step correction identity and bounds, and characterizes the implicit policies. The practical algorithm uses asymmetric bounded noise and expectile regression, avoiding the need for exact support classification or additional pessimistic OOD penalties. Experiments on Gym-MuJoCo and AntMaze demonstrate CSDG's strong aggregate performance and stable value estimation.