New Approach for Robust Decision-Making in Uncertain Environments

Traditional Robust Markov Decision Processes (RMDPs) often design a single optimal policy against a range of possible environmental dynamics. While this approach ensures robustness, it can be overly conservative, especially when the true environment dynamics are fixed but initially unknown, becoming clearer only after a system is deployed. This new research proposes an alternative: adaptive policy portfolios. An adaptive policy portfolio comprises a finite collection of pre-synthesized, memoryless randomized policies. These policies are developed offline and then paired with a lightweight online selector that chooses the most appropriate policy as more information about the environment becomes available. The quality of such a portfolio is measured by "robust regret," which quantifies the performance loss of the best portfolio member compared to a policy that would have been optimal if the environment were perfectly known from the start. The study delves into the computational complexity of both certifying an existing portfolio and synthesizing a new one. It finds that even for relatively simple scenarios, these tasks are computationally challenging. Despite the complexity, the research presents an offline construction method for these portfolios that can be specialized for runtime use, offering a more flexible and potentially less conservative approach to decision-making under uncertainty.