Explaining Multi-Objective Reinforcement Learning with Counterfactuals

Pareto-Conditioned Networks (PCNs) are a type of multi-objective reinforcement learning system that can learn diverse behaviors by adjusting a single policy based on a desired return command. However, the internal logic mapping these commands and states to specific actions often remains opaque, making it difficult for users to understand why an agent makes a particular decision. To address this, the researchers propose "command-space counterfactual explanations" for PCNs. Given a specific state, an original command, and an alternative "foil" action, the method searches for the minimal change in the desired return command that would cause the trained policy to choose that foil action. This provides an intuitive explanation: "If your preferences had shifted slightly towards X, the agent would have chosen Y." The contributions include formulating PCN explanations as return-command interventions, adapting adversarial machine learning techniques for reinforcement learning explanations, and introducing a boundary-seeded directional search method (CF-ZOO) that improves upon local optimization. These explanations are actionable and expressed in terms of user preferences, enhancing the interpretability of complex multi-objective AI agents.