Validating AI Document Extraction: Addressing Risk Control Failures

This research addresses a critical challenge in AI document extraction: ensuring valid per-field selective risk control. This control mechanism dictates that if a field is accepted by the system, its error rate among all accepted fields must not exceed a predefined threshold, alpha. The study identifies three common failure modes in real-world applications, including issues related to document clustering, score-refit leakage, and tie-mass pathologies, which cause systems to silently violate this trust contract. To rectify these issues, the paper introduces a "validity ladder" of fixes, outlining a progression of methods to restore control. It demonstrates that a fit/validation split protocol can restore expected-selective-risk control for learned fusions, though it may not provide a strong certificate. More rigorously, Mondrian Learn-then-Test with exact binomial tails offers PAC certificates, but these are currently near-vacuous for document-level control. Crucially, the study finds that conditioning on a pre-specified provenance taxonomy (support-bin) significantly improves rigor across various tiers, particularly where simpler pooled thresholds fail to certify. This benefit is context-dependent, performing well on Claude Sonnet but not replicating on Haiku or Qwen, suggesting that conditioning helps precisely where pooled methods are insufficient. A human audit confirmed the practical tier's accepted-set risk was well within budget.