New Benchmark Tests AI World Modeling with Transformers in Mazes

This paper introduces the AGI Maze Prediction Datasets and Benchmark, a new lightweight testbed designed to assess the world-modeling capabilities of AI, particularly Transformers. The benchmark utilizes procedurally generated grid worlds to create tasks involving predicting transitions, future states, and sequential textual observations. Its design emphasizes learning transferable dynamics rather than memorizing specific layouts, achieved through source-maze-disjoint training and validation splits and exact-match evaluation. Initial baselines using byte-level Transformers were established, and these were compared against architectures augmented with working memory. While a generic latent-memory Transformer showed some fitting capacity, a pseudo-video spatial-memory Transformer, which initializes a 2D latent workspace from the input map and updates it from action history, achieved superior performance. This structured memory model reached perfect validation accuracy on certain fixed-horizon tasks and significantly improved textual trace prediction, suggesting that task-aligned working memory is more effective than simply increasing latent capacity for learning complex world dynamics.