Critic-Free Pretraining Boosts Online Reinforcement Learning.

Offline-to-online (O2O) reinforcement learning aims to take policies initially trained on static datasets and refine them through real-time interaction. A common challenge in this approach is that the "critic" — the component that estimates the value of states or actions — often carries biases from its offline training. As the policy and data distribution rapidly evolve in an online setting, these inherited value estimates can become misaligned, hindering effective policy improvement and exploration.To address this, researchers propose Critic-Free Pretraining (CFP). This novel paradigm completely foregoes the traditional step of training the critic during the offline phase. Instead, the critic is initialized fresh when the system transitions to online fine-tuning. This allows the critic to adapt directly to the dynamic online environment without being constrained by potentially inaccurate or outdated estimates from the offline data. CFP has been shown to be compatible with various O2O algorithms and consistently matches or improves performance, particularly on complex tasks, by enabling more accurate value estimation and efficient exploration.