Coding Agent Performance Varies Greatly with Harness Configuration

This research highlights the critical impact of a coding agent's "harness" on its overall performance, even when the underlying language model and task remain constant. A coding agent is defined as a combination of a model and a harness, where the harness manages what the model perceives, its tool access, and the continuation of its work. The study investigates whether altering the harness configuration alone can change the outcome. Researchers compared two configurations of the same harness on three coding benchmarks. The control setup provided the full conversation history in chronological order. In contrast, the treatment configuration, while maintaining the same record, mechanically shortened older tool results as the context window filled and adapted to repeated or stalled work. Under tight context constraints, the treatment harness significantly improved performance, increasing the mean per-task fail-to-pass fraction (F2PF) and the number of complete solutions on benchmarks like SWE-bench Verified and SWE-bench Pro. For instance, on a specific cohort with a tight context window, the treatment raised complete solutions from 43% to 72%. This effect was also observed across three other models with different designs, without specific retuning. The study concludes that coding agent evaluations must consider the model and harness as an integrated solver, as harness design profoundly influences what a model can achieve.