Multi-Agent Policy Composition Safety Rethought in RL.

A new study questions the safety of a common practice in multi-agent reinforcement learning (MARL): allowing individual agents to independently compose their policies from a learned library. While single-agent policy transfer using successor features offers guarantees, this paper demonstrates that in a multi-agent context, independent composition can lead to joint behaviors that are strictly worse than any policy in the original library. This failure occurs because changes in one agent's policy invalidate the value estimates relied upon by its teammates. To address this, the researchers propose MA-USFA (Multi-Agent Universal Successor Feature Approximators), a hierarchical method designed to achieve both safety and flexibility. MA-USFA features a lower layer where universal successor feature approximators predict each agent's successor features, conditioned on the objectives of its teammates. An upper composer then selects appropriate library entries for each agent and applies necessary cross-agent corrections, ensuring that the combined team behavior remains safe and effective even with dynamic, agent-specific objectives.