ExtractBench: New Benchmark for Enterprise Document Extraction

This paper presents ExtractBench, a novel benchmark specifically designed to evaluate schema-guided enterprise document extraction. This task is crucial for modern workflows where AI agents must accurately extract information from documents based on a user-defined schema, providing source evidence for grounding. The benchmark is comprehensive, featuring 4,869 pages from 370 enterprise documents across 8 business domains and 67 document types, with clear tags for different challenge scenarios. ExtractBench's evaluation system measures several key aspects: value accuracy (using order-insensitive value F1), record completeness at scale, grounding (with word- and page-level F1 for traceability), and the associated computational cost. Initial evaluations reveal that commercial Vision-Language Models (VLMs) perform well on shorter documents but often truncate record lists in longer ones. Coding agents achieve higher accuracy but at a significantly greater cost. Notably, LlamaExtract Agentic Plus emerged as the top performer across all three metrics, offering accuracy comparable to coding agents at a fraction of the expense.