Coding Agents: Executable Models, Simplification, Verification Impact Performance

Previous research on ARC-AGI-3 agents combined several advanced techniques, making it unclear which components were most responsible for performance gains. This new study dissects these elements by testing four distinct Codex-based agents: a basic textual agent, an executable world model, the same model with scheduled simplification, and a full verification treatment incorporating simplification and exact reproduction. The primary evaluation used GPT-5.4 and GPT-5.5 models across varying reasoning efforts on public ARC-AGI-3 games. A consistent finding was that all agent variants benefited significantly from stronger underlying language models and increased reasoning effort. However, the impact of individual components was less straightforward. Interestingly, requiring a persistent executable deliverable was not universally beneficial, with the textual variant sometimes outperforming the flexible-interface executable variant. Simplification generally improved performance, while the complete verification treatment consistently ranked highest across all settings, albeit at a higher resource cost. A follow-up with GPT-5.6-sol showed the verification variant fully solving all public games, suggesting saturation of the public dataset.