New Benchmark Tests LLM Reasoning, Diplomacy Under Fog of War

A novel benchmark named "Age of LLM" has been introduced to evaluate large language models (LLMs) in a strategic, adversarial environment. This turn-based 1v1 game pits two LLMs against each other on a grid, with the objective of destroying the opponent's base. The benchmark incorporates several challenging elements: a "fog of war" that limits information, full diplomatic capabilities including messages and ceasefires, and a strict requirement for actions to conform to a JSON schema, with illegal actions being silently discarded. The engine is private, and each match uses a fresh random map and opponent to prevent data contamination. Initial findings from benchmarking 15 models across 54 matches indicate that a "nuclear rush" strategy dominates, often due to mechanical execution rather than cognitive deterrence failure. Diplomacy is frequent but rarely successful, and a significant portion of illegal actions stem from fog/state errors, providing a measure of belief-tracking. The research also suggests a weak link between reliability and winning, and the turn-by-turn traces offer a unique lens into LLM reasoning under uncertainty, including belief-tracking and spontaneous deception.