Risk-Aware GUMDPs Optimize Decisions with Entropic Risk Measures

Traditional Markov Decision Processes (MDPs) often focus on optimizing expected outcomes, which may not adequately capture risk preferences in decision-making. This research introduces Risk-Aware General-Utility Markov Decision Processes (GUMDPs), a framework where agents optimize a risk measure of the distribution of objective values, rather than just the mean. The objective function itself depends on the frequency of state visitation induced by the agent's policy. The paper specifically focuses on the entropic risk measure (ERM), which allows decision-makers to explicitly trade off expected performance with risk aversion. This enables a richer set of objectives to be considered within the GUMDP framework. To solve these complex risk-aware GUMDPs, the researchers propose an online planning approach based on Monte Carlo Tree Search (MCTS). This method is provably capable of solving risk-aware GUMDPs to any desired accuracy. Experimental results across diverse tasks, including standard MDPs, maximum state entropy exploration, imitation learning, and multi-objective MDPs, demonstrate the success of this approach in optimizing for a spectrum of risk-aware behaviors.