DCRL Improves Long-Horizon Offline Reinforcement Learning.

Training reinforcement learning agents for complex, long-horizon tasks using pre-recorded offline data is challenging. Existing methods struggle with the propagation of errors in value estimates and the amplification of overestimation, especially when dependencies are long. DCRL addresses these issues by adopting a divide-and-conquer strategy. It breaks down each long trajectory into a balanced binary tree structure, then learns value functions by propagating information from the shortest segments (leaves) up to the longer segments (root). This recursive approach significantly shortens the dependency chain for value estimation, moving from a linear to a logarithmic depth, which empirically leads to much slower error accumulation. The method also effectively discovers shorter routes by propagating values across different trajectories, leading to substantial performance improvements over previous offline goal-conditioned RL techniques on challenging tasks.