Decoupled Policy Extraction Boosts Offline Reinforcement Learning.

In Offline Reinforcement Learning (RL), the conventional approach involves jointly training an actor and a critic, where the critic guides the actor towards better actions. While effective in online RL where new data is continuously generated, this coupled paradigm presents challenges in offline settings due to fixed training data. Specifically, actor updates can lead to out-of-distribution (OOD) actions, amplifying critic overestimation, and creating a difficult trade-off between suppressing OOD actions and selecting high-value actions within the available data. To address these issues, a new paradigm called "decoupled policy extraction" is proposed. This method separates policy improvement from actor training. The actor is trained solely to model the behavior distribution, generating a set of behavior-supported action candidates. Policy improvement then occurs at inference time, where a separately learned critic reranks these proposals to select the highest-value actions. Extensive experiments demonstrate that this decoupled approach significantly outperforms both behavior cloning and traditional jointly learned offline RL methods. It remains effective even when using a simple Q-learning critic, showcasing its robustness and ability to overcome the limitations of coupled learning in offline settings.