AlphaZero's Limits in Sparsely Rewarded Games Explored.

This research delves into the performance boundaries of AlphaZero, a neural-guided Monte Carlo Tree Search algorithm, particularly in games with sparse reward structures such as Connect Four and Chomp. While AlphaZero consistently achieves strong gameplay, the study highlights a gap between strong play and truly optimal, "perfect" play. It observes that vanilla AlphaZero struggles to maintain the exact trajectories required for optimal strategies, failing to preserve game-theoretic values in Connect Four and consistently restore Grundy-number invariants in Chomp. To address these limitations, the paper introduces AlphaZero Auxiliary Loss (AZAL), an enhancement that incorporates oracle-derived policy supervision into the training process. AZAL significantly improves the algorithm's consistency with optimal play across various game traces and state evaluations. For instance, in Chomp, AZAL achieves perfect full-game oracle consistency on certain board sizes and substantially improves performance in Connect Four, delaying mistakes and increasing oracle-match rates, though not always reaching perfect play.