InfraBench Evaluates AI Agents for Infrastructure Management

Managing modern computing infrastructure has become increasingly complex, prompting interest in AI agents for automation. However, a clear understanding of how well these agents handle real-world infrastructure challenges has been lacking. To address this, researchers introduced InfraBench, a comprehensive benchmark suite for evaluating AI agents across the entire system stack, operational lifecycle, and with detailed risk assessment. InfraBench includes a live leaderboard, a variety of tasks, and an evaluation harness, all publicly available. Initial experiments with 15 different agent-model configurations revealed that even the most capable agents could not achieve a perfect score. Mean effective scores ranged from approximately 40% to 88%, and repeating tasks multiple times showed that top configurations still failed a significant fraction of attempts. A key finding was a general failure pattern: agents often satisfied short-term objectives but left behind non-durable changes, broken distributed invariants, unsafe side effects, or uncleaned state. This highlights a critical gap in current AI agent capabilities for robust, production-grade infrastructure management, emphasizing the need for further development in areas like long-term state management and safety.