Real-Time RL Agents Learn Optimal Planning Budgets

In most Reinforcement Learning (RL) scenarios, the environment conveniently pauses, waiting indefinitely for an agent to make its decision. However, real-time environments present a significant challenge: deliberation takes time, and the environment continues to evolve during an agent's planning phase. This research delves into this critical aspect of real-time RL, where the environment's progression is not halted by the agent's internal processing. Building on existing real-time formalizations, the authors introduce a new concept: variable-delay real-time RL. In this setting, the agent itself must decide how long to deliberate at each decision point, acknowledging that this planning time has consequences as the environment advances. For planning agents, the optimal deliberation time is highly dependent on the current state, and attempting to plan *how long to plan* can lead to an infinite regress or "paralysis." To overcome this, the paper proposes an innovative solution: training a lightweight gating policy that operates in conjunction with a planner. This gating policy learns to dynamically select state-dependent planning budgets, effectively determining the optimal amount of "thinking time" for the agent in any given situation. Tested across a suite of real-time games including Pac-Man, Tetris, Snake, Speed Hex, and Speed Go, this gating policy consistently outperformed both fixed-budget and heuristic-based baselines. The approach also demonstrated successful transferability to a real-time setup involving two different GPUs for the environment and agent, highlighting its robustness and practical applicability.