Decentralized Q-Learning Explores Multi-Player Information Asymmetry

This paper delves into the complexities of decentralized multi-player reinforcement learning within episodic tabular Markov Decision Processes, specifically focusing on scenarios where players operate with incomplete information. The researchers categorize information asymmetry into three types: unobserved actions with common rewards, observed actions with independent rewards, and unobserved actions with independent rewards. They aim to understand how agents can learn effectively without direct communication during the learning phase, relying only on a pre-agreed protocol. For the first two types of asymmetry, the study introduces algorithms called mQ-learning and mQ-learning-intervals, which achieve a regret bound comparable to single-agent Q-learning. For the most challenging scenario, unobserved actions with independent rewards, two-phase explore-then-commit algorithms, mEXC and mEXC-Bellman, are proposed. The findings indicate that decentralized learning under these conditions can achieve performance rates similar to centralized joint-action benchmarks, up to logarithmic factors. However, the practical implications are strongest for systems with a small number of players or limited per-player action sets, given the exponential growth of the joint action space.