Action Intersection Reduces Q-learning Overestimation Bias.

Q-learning, a fundamental reinforcement learning algorithm, often suffers from overestimation bias, especially when dealing with large discrete action spaces. This bias arises from the randomness in Q-value estimations, which can lead to suboptimal policy learning. Existing solutions, broadly categorized into coupling and decoupling paradigms, each have their own limitations. Coupling methods tend to overestimate, while decoupling methods often underestimate. This research proposes a novel "action intersection" strategy to mitigate these issues. This strategy introduces a semi-decoupling mechanism by allowing two Q-functions to share a fraction of trajectory data. When data is shared, Q-functions are updated using a coupling paradigm; otherwise, a decoupling paradigm is used. The power of action intersection lies in its ability to achieve a large bias range and fine-grained control over estimation bias by varying the data sharing fraction. Experiments in both tabular and deep reinforcement learning settings demonstrate that this method drastically outperforms several state-of-the-art baselines, providing a more stable and accurate Q-value estimation.