New Analysis Improves Learning in Weakly-Coupled MDPs

A new study delves into the sample complexity of learning within average-reward weakly-coupled Markov decision processes (WCMDPs) and Restless Bandits (RBs), operating under a generative model. Traditional methods that reduce these problems to a tabular MDP often lead to prohibitively high complexity bounds, especially as the number of arms or components, denoted by N, grows exponentially. This research, however, capitalizes on the inherent weakly coupled structure of these systems. By exploiting this structure, the authors demonstrate that near-optimal policies can be learned with sample and computational complexities that are polynomial in N, a significant improvement. They specifically analyze a plug-in approach, where an efficient planning algorithm is applied to an empirical model derived from data. For fully heterogeneous WCMDPs, the work establishes the first finite-sample PAC (Probably Approximately Correct) guarantee, featuring polynomial complexity and an optimality gap of O(1/√N). Further, for homogeneous RBs, a smaller optimality gap is proven under specific structural assumptions. A key technical contribution is a novel Lyapunov-based analysis framework. Unlike classical approaches that struggle with bias functions, this framework employs an explicitly constructed Lyapunov function alongside a drift transfer technique between the true and empirical models. An important independent aspect of this framework is a fine-grained perturbation analysis for the underlying linear programming (LP) relaxation, offering a general tool for analyzing LP-based policies and weakly-coupled systems.