CoDrift Improves Offline Reinforcement Learning with Compositional Drifting

Offline reinforcement learning (RL) inherently involves multiple objectives: policies must align with the fixed dataset's behavioral patterns while simultaneously prioritizing high-value actions. CoDrift, a novel compositional framework, addresses this by recasting these objectives as action-space motion fields. This perspective allows for the direct combination of heterogeneous learning objectives through field composition. Inspired by drifting models, CoDrift integrates three distinct objective-level fields into a unified policy field. These include a conditional field to preserve state-dependent behavioral structure, a marginal field for stable generative signals, and a value field to guide actions towards higher-value regions. The composed field then feeds into a stochastic generator, producing an action in a single forward pass during deployment. CoDrift demonstrated superior performance, achieving the best average rank across 73 tasks from OGBench and D4RL in both offline and offline-to-online settings, comparing favorably with state-of-the-art methods.