LLMs Fail to Identify Missing Information in Legal Queries

A new benchmark, InsufficiencyBench, has been developed to assess a critical failing of large language models (LLMs) in providing legal advice: their inability to handle underspecified user queries. Unlike existing evaluations that assume complete information, this benchmark tests whether LLMs can identify when a query lacks legally material facts, pinpoint the missing elements, and avoid drawing premature conclusions. The researchers formalized a taxonomy of eight missing-element categories across three structural failure modes, creating 202 benchmark items annotated by practicing attorneys. Evaluations of ten frontier models showed poor performance, with none exceeding an F2 score of 0.46 for identifying missing elements, and a median recall of only 0.44. This indicates that current LLMs either offer vague, non-committal responses or, more dangerously, provide answers based on assumptions they fabricate, rather than prompting the user for necessary details. This limitation poses a significant challenge for deploying LLMs reliably in legal AI systems where factual completeness is paramount.