MCTS and Every-Visit Monte Carlo Control: A Unified View

Monte Carlo Tree Search (MCTS) and every-visit Monte Carlo (MC) control are typically presented as distinct algorithms, each with its own terminology and operational stages. MCTS is often described through its selection, expansion, simulation, and backup phases, while MC control is framed by trajectory sampling, return estimation, action-value updating, and policy improvement. However, this paper posits that, when examining the core mechanisms of trajectory generation and action-value updating, the differences between these two methods are largely semantic. The tree policy and rollout policy within MCTS can be seen as components of a single, evolving policy, representing learned and not-yet-learned parts, respectively. Under this interpretation, the expansion phase of MCTS aligns with the concept of first-visit and initialization in MC control. Furthermore, the backup operation in MCTS is essentially the standard every-visit Monte Carlo update. Thus, the four stages of MCTS can be reduced to two fundamental operations: sampling trajectories according to the current policy and performing every-visit Monte Carlo updates, suggesting a deep equivalence between the two approaches.