New RL Framework Optimizes Decision-Making with Environment Abstraction

Decision-making in large Markov Decision Processes (MDPs) often faces challenges due to vast state spaces. This research explores a novel approach to environment abstraction that prioritizes optimizing decision quality directly, rather than merely preserving the geometric or topological properties of the environment. The abstraction is conceptualized as a controlled approximation where the state space is aggregated, and a common action distribution is enforced within each of these aggregated states. For any given partition of the state space, the study provides a performance guarantee. This guarantee distinctly separates the error arising from value-function approximation from the performance loss introduced by sharing actions across aggregated states. This analytical insight forms the basis for a new multi-timescale reinforcement learning framework. This framework simultaneously adjusts both the decision-making policy and a tree-structured environment abstraction. The algorithm dynamically refines or coarsens regions of the state space based on discrepancies in Q-values, effectively balancing the desired performance level against the size and complexity of the abstraction. Empirical evaluations demonstrate that this method achieves substantial state compression, leading to improved sample efficiency and quicker replanning capabilities when compared to traditional actor-critic baseline algorithms.