Multi-Agent Actor-Critic Algorithms Explored for Parameterized Reinforcement Learning

A comparative study has investigated the performance of multi-agent extensions of popular actor-critic algorithms in the context of parameterized action reinforcement learning. This type of learning is crucial for environments that demand both discrete action selection and continuous parameterization, a common challenge in complex control systems. The research specifically examined Multi-Agent Greedy Actor-Critic (MAGAC), Multi-Agent Soft Actor-Critic (MASAC), and Multi-Agent Truncated Quantile Critics (MATQC). Unlike the traditional centralized training, decentralized execution paradigm, this framework utilizes independent actor-critic agents that share a common replay buffer while maintaining their own policy and value networks. Evaluated on benchmark tasks with varying numbers of agents (three, five, and ten), the study found that MAGAC consistently improved performance compared to its single-agent counterpart. MASAC and MATQC showed more modest gains. A key insight was the trade-off between performance and computational cost: increasing agents beyond five yielded limited additional performance, especially for MAGAC, highlighting scalability considerations for shared-experience multi-agent methods.