New RL Method Improves Subgoal Selection in Offline Learning

Hierarchical Implicit Q-Learning (HIQL), a method for offline goal-conditioned Reinforcement Learning (RL), often struggles with subgoal selection due to optimistic bias and mode collapse. Optimistic bias can misinterpret lucky outcomes as skillful choices, while mode collapse reduces diverse subgoal distributions to a single, potentially unreachable, Gaussian mean. To counter these issues, researchers propose NFTR (Normalizing Flows subgoal policies with Triangle-slack Reweighting). NFTR replaces the standard Gaussian policy with a conditional Normalizing Flow, which is identified as the minimal generative class for AWR-based subgoal selection, enabling provable mode-averaging. Additionally, NFTR incorporates a triangle slack score, derived from the architectural triangle inequality, to multiplicatively correct AWR weights. This mechanism downweights subgoals whose detour costs exceed average reachability, effectively vanishing on geodesics in deterministic environments and providing a conservative upper bound on composability violation in stochastic ones. These combined elements allow NFTR to avoid Gaussian collapse and maintain stability in stochastic dynamics.