New Benchmark Detects Regulatory Divergence in Life Sciences with LLMs.

This paper introduces RegDivergence-101, a pioneering benchmark designed to evaluate large language models (LLMs) in detecting regulatory contradictions across different jurisdictions, specifically focusing on the FDA and EMA in life sciences. Pharmaceutical sponsors currently perform this reconciliation manually, a labor-intensive process. The task involves classifying the relationship between an FDA and an EMA requirement on the same topic as AGREE, DIVERGE, or SILENT (where one agency is silent on a point the other regulates). The benchmark comprises 101 expert-grounded pairs, with labels derived from peer-reviewed studies and primary guidance texts. The study systematically characterized a baseline hierarchy of four methods. A lexical heuristic achieved a 0.511 macro-F1, while an NLI cross-encoder scored 0.233. An obligation-level Graph-RAG method improved to 0.663, but a flat LLM judge, specifically Claude Haiku, significantly outperformed others with a macro-F1 of 0.830. Key observations include the semantic detectability of "SILENT" relationships, the improvement of obligation graphs over lexical methods, and the indication that corpus-level graph construction is a promising architectural target for large-scale silent-detection. RegDivergence-101 establishes the task formulation and baseline for future research in this critical domain.