SAFE Improves Multi-Agent RL for Continuous Cooperative Tasks

A new multi-agent reinforcement learning (MARL) framework, named SAFE, has been developed to enhance cooperative tasks that involve continuous action spaces. Current methods for counterfactual credit assignment in MARL often introduce bias into policy gradients and struggle to guarantee convergence, particularly when dealing with continuous actions. SAFE tackles these issues by employing a counterfactual baseline that is conditioned on a "self-evolving default action." This default action is sampled directly from each agent's experience buffer, eliminating the need for additional simulations, reward models, or prior environmental knowledge. This design ensures that the baseline accurately quantifies each agent's contribution without biasing the deterministic policy gradient, thereby guaranteeing convergence to local optima. Extensive testing on cooperative vehicular tasks has demonstrated that SAFE consistently outperforms state-of-the-art models. This advancement offers a more robust and reliable approach for training multiple agents to collaborate effectively in complex environments with continuous control requirements.