Baseline LLMs Perform Well in Autonomous Penetration Testing.

This research emphasizes the importance of evaluating plain-agent baselines before attributing performance improvements in autonomous penetration testing to complex architectural designs. Many recent papers on this topic introduce multi-component security harnesses around frontier Large Language Models (LLMs), making it difficult to discern whether performance gains stem from the architecture or the underlying model. The study conducted a controlled experiment using the 104-task XBOW benchmark, comparing default coding CLI agents (Codex, OpenCode, Pi) with the same GPT-5 model, budget, and scoring rules. It found that these plain agents could solve a significant portion of the benchmark tasks. Furthermore, the research demonstrated that repeated plain-agent runs can achieve or even surpass the union coverage of some published architecture scores. Crucially, simply upgrading to newer models like GPT-5.2 and GPT-5.5 within the same basic scaffold led to substantial performance improvements. This suggests that while specialized harnesses can offer measurable lift and cost efficiency, the inherent capabilities of the LLM backbone are a major, often underestimated, factor in performance.