CODA-BENCH Evaluates AI Agents on Data-Intensive Coding Tasks

As advanced AI agents increasingly demonstrate capabilities as autonomous engineers, there's a growing need for evaluation benchmarks that accurately reflect the complexities of real-world development. These scenarios typically involve intricate code alongside large-scale data, often within complex file systems. However, existing benchmarks tend to evaluate code-centric or data-centric abilities in isolation, failing to capture the integrated challenges of real development. To bridge this gap, researchers introduce CODA-BENCH, the first benchmark specifically designed to jointly evaluate both code and data intelligence within a data-intensive environment. This benchmark utilizes a data-intensive Linux sandbox based on the Kaggle ecosystem, featuring hundreds of datasets. Agents must actively navigate complex file hierarchies to identify relevant resources and then generate code for data-driven analytical tasks. CODA-BENCH comprises 1,009 tasks across 31 communities, with each task environment containing an average of 980 files, simulating realistic data scale and noise. Evaluations of top-performing agents on CODA-BENCH show a success rate of only 61.1%, indicating a substantial deficiency in their ability to effectively integrate data discovery with code execution. These findings point to critical areas for future research in agentic capabilities.