Experience Memory Boosts LLM Agent Sequential Decision-Making

While Large Language Models (LLMs) have shown impressive capabilities in single-shot reasoning, their performance in sequential decision-making tasks, particularly in games with clear rules and outcomes, remains a challenge. A study using fully-observable two-player zero-sum games, such as tic-tac-toe and Connect Four, revealed that LLMs consistently play suboptimally and lose to Monte Carlo Tree Search (MCTS) opponents. The research indicates that this performance gap isn't primarily due to a lack of memorized strategies, as obfuscating game tree surface forms didn't significantly alter outcomes. To address this, a new agentic framework was developed, enhanced with an "experience memory" specifically designed for sequential settings and credit assignment challenges. This framework incorporates post-game reflection and rule extraction mechanisms, allowing the LLM agent to learn from its past interactions. Demonstrations on tic-tac-toe showed measurable improvements in performance without requiring any modifications to the underlying model weights, highlighting the power of external memory and reflective processes for enhancing LLM agency.