Reversal Q-Learning Improves Offline Reinforcement Learning Performance

Iterative generative modeling techniques, such as flow matching, are powerful tools for modeling complex behaviors, which is crucial for effective offline reinforcement learning (RL). This research proposes a novel off-policy RL algorithm called Reversal Q-learning (RQL), designed to train a flow policy using existing prior data. RQL operates within an "expanded" Markov decision process (MDP) framework, where individual flow refinement steps are treated as distinct actions. To enable off-policy RL in this context, the algorithm employs two key techniques: it generates virtual on-policy trajectories by "reversing" flows, making the framework compatible with prior data, and it applies a bias-and-variance reduction method to address the curse of horizon inherent in off-policy RL. The resulting RQL algorithm offers several advantages over previous flow-based RL methods, including avoiding backpropagation through time, making more effective use of the learned value function, and directly training a comprehensive, expressive flow policy. Experiments across 50 challenging simulated robotic tasks demonstrate that RQL achieves superior average offline RL performance compared to other state-of-the-art flow-based algorithms.