Analyzing AI Agent Trajectories Reveals Model-Specific Problem-Solving Behaviors

The performance of AI agents is not solely dependent on the underlying model but also significantly influenced by the "agent harness" – the structured execution layer that includes prompts, memory, tools, and interaction loops. A critical challenge is the "intent-execution gap," which describes the discrepancy between what the model aims to do and what the harness actually executes. Bridging this gap is as vital as other aspects of harness design. To investigate this, a customizable harness called "Simple Strands Agent" (SSA) was developed. SSA aims to identify common patterns across various large language model families while also accounting for model-specific preferences. Using SSA, researchers were able to reproduce or even enhance the pass@1 performance on several popular agentic benchmarks, including SWE-Pro and Terminal-Bench-2. Beyond simple pass rates, an extensive analysis of 138,000 trajectories generated by SSA revealed deeper insights into model behavior. By mapping these trajectories into code state-spaces, distinct problem-solving approaches emerged among different models. Finer-grained metrics, such as edit frequency, testing activity, and phase transitions, provided a detailed view of how individual models allocate their effort across the various stages of autonomous problem-solving.