Predicting AI Task Difficulty Without Costly Simulations

Estimating the difficulty of a task for an AI agent is crucial for designing effective evaluation benchmarks and creating progressive training curricula. Traditionally, this often involves executing numerous costly simulations, or "rollouts," in complex, stateful environments. This becomes a significant computational bottleneck, especially for agents tackling long-horizon tasks. Prior methods for early difficulty prediction have been limited to simpler, static tasks or isolated coding environments, often relying on narrow features and potentially inaccurate evaluation metrics. This study investigates the prediction of task difficulty ex ante (before execution) across a broad range of 17 agentic benchmarks. These benchmarks span diverse domains, including coding, mathematics, machine learning, web navigation, and function calling. The researchers demonstrate that metrics like AUC can sometimes mask poor difficulty estimates, highlighting the need for more robust evaluation. A key finding is the identification of token-level entropy as a valuable predictive signal for task difficulty. Furthermore, the study shows that analyzing the residuals—the differences between expected and observed difficulty—can effectively expose hidden flaws within the environment, such as data contamination or task infeasibility. This capability allows for more reliable environment design and agent training.