Evaluating Coding LLMs' Understanding of Software Execution

The research delves into evaluating the "software world models" implicitly held by coding Large Language Models (LLMs), moving beyond traditional code-execution benchmarks that primarily focus on control flow. The authors propose a broader evaluation by observing and predicting execution resources, including peak memory usage, wall-clock time, and ranked profiler outputs at method and line granularity. Using the SWE-bench Verified dataset, which contains real-world software engineering tasks, the study tested various frontier LLMs. The results indicated that all models exhibited only modest performance and brittle behavior when predicting these execution resource metrics. This suggests a significant gap in LLMs' understanding of how software actually executes, as opposed to merely generating syntactically correct code.