Neurosymbolic HRL Improves Sample Efficiency with Incremental Knowledge

Traditional Hierarchical Reinforcement Learning (HRL) often struggles with sparse reward environments and tasks requiring long-horizon reasoning because its knowledge representation is typically fixed. This new research proposes a neurosymbolic HRL framework that incorporates "Incremental Knowledge" (InK), enabling agents to continuously update their understanding of the environment as they explore. The system integrates high-level symbolic planning, which utilizes an updatable knowledge representation, with low-level goal-conditioned neural modules that learn basic motion primitives through experience and reward shaping. This combination allows for more dynamic and adaptive learning. Experiments on navigation tasks demonstrate that this ability to incrementally learn and reason about knowledge substantially boosts sample efficiency. The authors also developed Belief World Tree Search, a method for optimal symbolic planning that leverages prior knowledge about the world.