New RL Algorithm Optimizes Multi-Objective, Constrained Average-Reward Tasks

A new reinforcement learning (RL) algorithm, the Bias-Controlled Primal-Dual Natural Actor-Critic, has been introduced to tackle complex problems involving multiple conflicting objectives and safety constraints in infinite-horizon average-reward settings. Traditional methods struggle with bias introduced by the nonlinearity of utility and constraint functions, which propagates through policy gradient and actor-critic updates. This novel algorithm utilizes an MLMC-based approach to manage bias in scalarized objectives, constraint evaluation, and actor-critic estimation. Crucially, it achieves optimal global convergence and constraint-violation rates of `O(1/sqrt(T))` without needing prior knowledge of mixing times. This represents a significant advancement, being the first to establish optimal convergence for concave scalarized multi-objective RL, both with and without constraints, and without relying on mixing-time information.