AI Code Agents Show Fragile Robustness to Code Transformations

AI code agents are increasingly used to fix software issues, but their reliability when faced with minor, semantically equivalent code changes is not well understood. This study investigates how robust these agents are when the surrounding codebase is rewritten using transformations that don't alter the code's meaning. Researchers developed a random variant sampler to apply common semantics-preserving transformations (SPTs), such as control-flow rewrites, dead-code injection, and identifier renaming, to create perturbed code variants. They then evaluated two agentic scaffolds (mini-SWE agent and OpenCode) powered by four leading models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B) on instances from SWE-bench Verified and SWE-bench Pro. The findings indicate a small but noticeable degradation in most configurations, with resolve rates dropping by up to 6.7 percentage points and statistically significant drops in 6 out of 16 configurations. Crucially, the study uncovered a "jagged robustness frontier," meaning no single model consistently ranked highest in robustness across different scaffolds. For example, Qwen performed robustly with mini-SWE agent but was brittle with OpenCode. Simpler scaffolds generally showed more robustness to these perturbations.