Open Models Struggle with High-Risk Information Extraction.

This paper presents a comprehensive benchmark evaluating the performance of open-source Optical Character Recognition (OCR) engines, Large Language Models (LLMs), and Vision-Language Models (VLMs) in a complex, high-risk public sector application: extracting structured information from student applications for an international study program. This task is particularly critical given the EU AI Act's classification of such applications as high-risk. The study aimed to bridge the gap in systematic evaluations of end-to-end extraction pipelines using open models. The empirical evaluation revealed that while VLMs generally showed better performance than pipelines combining OCR with LLMs, the overall reliability of state-of-the-art open-source models in zero-shot settings remains a significant challenge. Only 4 out of 35 configurations achieved an F1 score above 0.5, with roughly 75% scoring below 0.25. The research also noted that model scale does not guarantee proportionally better results, and input quality, particularly the structural integrity of OCR output, is a crucial factor influencing downstream performance, often more so than the LLM or VLM capabilities themselves.