TerraProbe Detects Deceptive Fixes in LLM-Assisted Terraform.

The increasing use of large language models (LLMs) as automated repair agents for security misconfigurations in Terraform Infrastructure-as-Code (IaC) presents a growing risk. Current evaluation methods often consider a repair successful merely if a static-analysis finding disappears, without verifying the fix's actual validity, behavioral impact, or security intent. This research introduces TerraProbe, a comprehensive five-layer oracle framework designed to provide a more robust evaluation. TerraProbe was applied to 288 first-pass repairs generated by Gemini-2.5-flash-lite, GPT-4o, and Claude 3.5 Sonnet across real-world and injected-defect Terraform modules. The findings are stark: while targeted static analysis checks (like Checkov removal) showed high success rates (up to 83.3%), full scanner cleanliness dropped significantly (to 10.4%), and Terraform planning only succeeded for 39.6% of repairs. Crucially, human adjudication revealed that 71.4% of the plan-compared real-world repairs were "deceptive fixes." Deceptive fixes are those that pass automated checks but fail to address the underlying vulnerability. This pattern was consistent across all three LLMs, with no statistical difference in their deceptive-fix rates. The study also established a four-dimensional taxonomy of deceptive fixes and confirmed that critical vulnerabilities, such as wildcard IAM resource grants, persisted in many cases. TerraProbe highlights the critical need for multi-layered evaluation beyond simple static analysis to ensure genuine security in LLM-assisted IaC remediation.