AlphaZero Struggles with Perfect Play in Sparsely Rewarded Games.

AlphaZero has demonstrated remarkable capabilities in achieving superhuman performance in complex games through self-play and Monte Carlo Tree Search. However, this research delves into whether "strong play" equates to "perfect play," particularly in games characterized by sparse rewards, where optimal moves might not be immediately obvious or frequently reinforced. The study uses two oracle-evaluable games, Connect Four and Chomp, to explore this gap. The findings indicate that vanilla AlphaZero, while playing strongly, struggles to consistently adhere to the exact optimal trajectories required for perfect play. For instance, in Connect Four, it deviates from the optimal line, and in Chomp, it fails to reliably maintain the game's invariant (g=0). Even multi-frame inputs, tested on Chomp, did not fully resolve this issue. A significant improvement was observed with AlphaZero Auxiliary Loss (AZAL), which incorporates oracle-derived policy supervision. AZAL substantially enhanced the consistency with oracle play across various game traces and state evaluations. In Chomp, AZAL achieved perfect consistency on larger boards and high consistency on others, while in Connect Four, it improved the oracle-match rate and delayed mistakes, though it did not reach perfect play. This suggests that explicit guidance can help AlphaZero navigate the complexities of sparsely rewarded environments more effectively.