New Benchmark Reveals LLMs Struggle with Legal Contract Review

A new benchmark called ContractScrub has been introduced to specifically test the capabilities of large language models (LLMs) in performing the final review of legal contracts, a process known as "scrubbing." This task involves meticulously checking transactional agreements for various errors, such as misuses of defined terms, incorrect references, and inconsistent language, which is typically a routine but painstaking job for lawyers. Despite the general perception that LLMs excel at text processing and tasks requiring long-context reasoning and consistency checks, the results on ContractScrub were unexpectedly poor. Even frontier models achieved a macro average recall of only 0.75, significantly underperforming compared to their strong results on more general benchmarks. This outcome underscores the critical importance of developing and utilizing narrowly targeted, domain-specific benchmarks to accurately gauge the practical limits and real-world impact of LLMs in specialized fields like law. It suggests that while LLMs have broad capabilities, their application in highly precise and critical domains still requires substantial improvement and tailored evaluation.