Learning When to Plan: Meta-Reasoning for AI Agents

This research investigates how artificial agents can learn to dynamically choose between rapid, reactive decision-making and more deliberate, time-consuming planning, a capability known as meta-reasoning in humans. Reactive policies, often trained with reinforcement or imitation learning, are fast but may struggle with generalization. In contrast, model-based planning offers broader applicability but incurs computational delays. The authors propose a reinforcement learning method to train a meta-reasoning policy. This policy allocates computational resources by using a "reactive-policy uncertainty score." This score helps the agent predict when its reactive policy is likely to perform poorly, signaling the need to engage in more extensive, deliberative planning. Empirical studies conducted in motion planning and navigation environments demonstrate the effectiveness of this design. The meta-agent successfully learns to discern when a reactive action is sufficient versus when planning is essential. Furthermore, the research shows that as the reactive policy improves, the meta-agent can gradually shift towards a more fully reactive control strategy.