LLM Coding Agents Introduce Defects in System Implementation

As Large Language Model (LLM) coding agents take on more complex engineering tasks, their performance on systems-level requirements like schema design, asynchronous orchestration, and configuration correctness becomes critical. This paper presents a case study of an LLM agent tasked with implementing a multi-component data system based on a detailed specification. During a single session, the agent introduced five distinct defects, categorized by the violated constraint and the method required for detection. These defects underscore the challenges in relying solely on agents for end-to-end system implementation. The study also evaluated a specific architectural trade-off: restricting retrieval candidates to a graph-identified entity set versus unfiltered search. On the HotpotQA benchmark, using gold evidence labels for entity identification, filtered retrieval significantly outperformed unfiltered search, reaching its ceiling quickly. Unfiltered search, even with a higher budget, only recovered 69% of required evidence. The findings emphasize that while agents can implement systems, they require careful oversight, rigorous defect detection, and often human correction, especially when performance claims are made without re-measurement.