New Benchmark for AI Forecasting in Simulated Worlds

Developing and evaluating general-purpose AI forecasting systems often faces limitations inherent in real-world data, such as slow outcome resolution, infrequent tail events, and difficulty in assessing counterfactual scenarios. A new benchmark, ForecastBench-Sim, aims to address these challenges. This benchmark leverages game rollouts from Freeciv, a turn-based strategy game, to create a simulated environment. Forecasters are provided with a structured snapshot of the game state and are tasked with predicting future hidden states. The simulation then continues, and forecasts are scored. The simulated nature of ForecastBench-Sim enables the generation of diverse forecasting questions, including continuous, binary, conditional, and causal types, across various time horizons. It also facilitates the study of rare or disruptive outcomes and provides immediate resolution, making it a valuable complement to real-world forecasting benchmarks for research into probabilistic reasoning.