AETDICE Unifies Nonlinear Multi-Objective Reinforcement Learning

Optimizing systems with multiple, often conflicting, objectives and nonlinear preferences is a significant challenge in multi-objective reinforcement learning (MORL). Historically, nonlinear MORL has been divided into two distinct paradigms: Scalarized Expected Return (SER) and Expected Scalarized Return (ESR), each with its own optimization complexities. This paper introduces the Aggregation-Expectation-Transformation (AET) framework, which successfully unifies these disparate criteria. The AET framework provides a principled foundation for general nonlinear MORL by decomposing scalarization into three parts. Building upon this, the researchers propose AETDICE, a tractable offline RL algorithm specifically designed for AET objectives. AETDICE leverages DICE-style density-ratio estimation within an augmented state space, enabling sample-based optimization from static datasets. This unified approach resolves long-standing barriers, allowing for more sophisticated and human-aligned AI decision-making that can capture complex trade-offs like risk aversion or fairness.