Audit Finds SWE-Bench Pro Unreliable for AI Coding Benchmarking

A recent audit has cast doubt on the efficacy of SWE-Bench Pro, a prominent benchmark for evaluating AI coding models. The findings indicate that the benchmark is no longer a reliable indicator of advanced coding capabilities, primarily due to a significant noise ceiling of approximately 70%. This saturation point suggests that current models are hitting a limit imposed by the benchmark's design rather than their own performance. The audit identified numerous issues within SWE-Bench Pro's tasks, including ambiguous instructions, hidden requirements, and overly stringent testing criteria. These flaws can lead to correct solutions being incorrectly marked as failures, thereby distorting evaluation results. Consequently, the auditors have withdrawn their previous endorsement of SWE-Bench Pro as a leading evaluation tool for the research community. The report emphasizes the critical need for more robust, fair, and challenging benchmarks as AI coding models continue to advance. The audit itself employed a hybrid approach, combining model-based investigator agents with expert human review from five experienced software engineers, ensuring both scalability and qualitative judgment in their assessment.