Gated Q-learning Improves Reinforcement Learning Sample Efficiency

This paper introduces Gated Q-learning, a new algorithmic framework designed to enhance sample efficiency in reinforcement learning. For decades, practitioners faced a binary choice in Q-learning: either eliminate off-policy bias at the cost of truncated eligibility traces (Watkins' Q($\lambda$)) or ignore the bias for faster learning, introducing detrimental errors (Peng's Q($\lambda$)). Gated Q-learning offers a solution by smoothly interpolating between these two extremes. Instead of relying on importance sampling, the new approach employs a continuous, state-action-dependent gating mechanism. This mechanism selectively attenuates eligibility traces in an exploration-aware manner, allowing for longer credit-assignment horizons without the pitfalls of traditional methods. The research provides a strong theoretical foundation, proving the operator remains a contraction mapping and deriving its fixed point, with empirical results showing faster initial learning than either historical extreme.