LEMUR Aligns Multi-Objective RL with Human Preference Feedback

Traditional Reinforcement Learning (RL) typically relies on a single, well-defined scalar reward function, which is often difficult to specify for real-world tasks involving multiple, competing objectives like performance versus efficiency. While Multi-Objective RL (MORL) addresses these trade-offs by using vector rewards, it still assumes access to pre-specified reward functions for each objective. Preference-based RL (PbRL) has shown promise in learning without predefined reward functions by using human feedback, but it has largely been confined to single-objective settings. The LEMUR framework bridges this gap by introducing a novel approach that allows MORL agents to learn optimal multi-objective policies directly from the preferences of multiple humans. LEMUR jointly learns both the policies and multiple objective-specific reward models based on interactive human feedback. This enables agents to effectively balance competing objectives during the learning process, even when ground-truth reward functions are unavailable. Empirical results on various benchmark multi-objective tasks demonstrate LEMUR's superior performance compared to existing baseline methods, offering a promising direction for complex decision-making scenarios.