AI Coding Benchmark SWE-Bench Pro Found Unreliable

A recent audit has cast doubt on the reliability of SWE-Bench Pro, a widely adopted benchmark for evaluating AI coding models. The investigation uncovered that nearly a third of the benchmark's tasks contain significant flaws, rendering it unsuitable for accurately assessing advanced coding abilities. These issues include ambiguous instructions, hidden constraints, and overly stringent testing criteria that can lead to valid solutions being incorrectly flagged as failures. The auditors utilized a combination of AI-driven investigator agents and expert human review to conduct this comprehensive assessment. This approach allowed for a scalable examination of tasks while retaining the critical element of expert judgment. The findings highlight the urgent need for more robust, fair, and trustworthy evaluation methods as AI coding models continue to advance.