New RL Method Improves Skill Learning from Offline Data

Reinforcement Learning (RL) often benefits from leveraging pre-existing datasets to enhance policy performance and reduce the need for extensive online interaction. A common strategy involves a two-stage approach: first, extracting diverse low-level skills from offline data, and then training a high-level policy to combine these skills to solve specific tasks. However, the quality of these extracted skills heavily depends on the quality of the initial dataset, which can be a significant limitation. To overcome this, researchers propose QDOS (Quality-Diversity Offline Skill learning), a novel framework designed for robust offline-to-online learning. QDOS incorporates an Advantage-Weighted Quality-Diversity pretraining objective. This objective intelligently weights skill extraction and diversity based on the estimated advantage of each trajectory segment, ensuring that the extracted skills are both diverse and highly valuable. Furthermore, QDOS employs a dual dataset reuse strategy, utilizing offline data not only for skill pretraining but also for populating the online replay buffer through pseudo-labeling. Experiments show that QDOS significantly outperforms existing methods in complex manipulation and locomotion tasks, accelerating exploration and improving final returns in environments with sparse rewards.