New Framework Audits and Calibrates Legal AI Hallucinations

This new research introduces LegalHalluLens, a comprehensive auditing framework aimed at improving the trustworthiness of AI systems used in legal applications. The framework addresses the critical issue of AI hallucinations by moving beyond simple aggregate error rates, which often mask specific failure modes. It categorizes hallucinations into four legally relevant types: numeric, temporal, obligation/entitlement, and factual claims. A key component is the Risk Direction Index (RDI), which provides a single scalar to quantify the bias between omissions and inventions, offering a more actionable signal for compliance. The framework also includes a calibrated multi-agent debate pipeline designed to reduce fabricated detections. Evaluations across a large dataset of contracts demonstrated that hallucination rates vary significantly by claim category, with a substantial gap between obligation/numeric and temporal claims. The debate pipeline, informed by these diagnostics, achieved a 45% reduction in fabricated detections, outperforming generic debate approaches and matching commercial APIs with a smaller model.