Noise-Robust Framework Optimizes Risk Policies via Inverse RL

A new research introduces a robust framework designed to infer and optimize risk-averse policies, particularly relevant for complex decision-making under uncertainty. This "elicit-to-optimize" approach seamlessly combines Inverse Reinforcement Learning (IRL) with standard Reinforcement Learning (RL) to handle a broad class of risk objectives defined by distortion riskmetrics. On the elicitation side, the framework employs an adaptive Bayesian IRL method. This method is capable of inferring an agent's hidden risk preferences even from noisy or suboptimal observed actions, explicitly accounting for the stochastic nature of human decisions. The researchers prove that a finite set of specific questions can identify the preferred distortion riskmetric, with a rapid convergence rate for the algorithm. For the optimization component, a model-free RL algorithm is developed to optimize policies under conditional distortion riskmetrics. This algorithm unifies various risk objectives by representing them as an integral of the conditional cost quantile function. It extends the Proximal Policy Optimization (PPO) algorithm by incorporating policy, value, and quantile neural networks. The quantile network is crucial for estimating the full conditional cost quantile function, enabling numerical evaluation of diverse risk objectives. Comprehensive empirical studies validate the framework's accuracy in elicitation and its effectiveness in challenging financial scenarios.