Paper deep dive
Evaluating Structured Information Extraction with Open Models in a High Risk Public Sector Application
Elias Schubert, Felix Bießmann
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/20/2026, 4:15:43 AM
Summary
This paper evaluates the end-to-end performance of open-source Optical Character Recognition (OCR) engines, Large Language Models (LLMs), and Vision-Language Models (VLMs) for structured information extraction from high-risk public sector documents (student applications). The study finds that while VLMs generally outperform OCR+LLM pipelines, overall performance is poor in zero-shot settings, with only 4 of 35 configurations achieving F1 scores above 0.5. Input quality, specifically the structural preservation of OCR output, is identified as a critical factor independent of downstream model capability. Model scale influences performance non-linearly, with larger models not guaranteeing proportionally better results.
Entities (14)
Relation Signals (8)
Input Quality → criticallyinfluences → Extraction Performance
confidence 96% · Input quality, particularly the structural preservation of OCR output, emerges as a critical factor independent of downstream model capability.
MinerU → achievesbestf1inpipeline → Qwen3:235B
confidence 95% · MinerU stands out as the strongest OCR engine, achieving an F1 score of 0.509, placing it on par with the best-performing VLM... The best- performing LLM was Qwen3:235B in all cases.
Student Applications → classifiedas → High Risk
confidence 95% · complex real-world document processing task classified as high risk: Student applications for an international study program.
VLMs → generallyoutperform → OCR+LLM Pipelines
confidence 94% · Our results reveal that while VLMs generally outperform OCR+LLM pipelines
Model scale → influencesperformancenonlinearly → Extraction Performance
confidence 93% · Model scale influences performance, yet the relationship is non-linear: substantially larger models do not guarantee proportionally better results.
Qwen2.5-VL → outperforms → Qwen3-VL
confidence 92% · Notably, Qwen2.5-VL marginally outperforms its successor Qwen3-VL (0.487)
LLaVA → performspoorly → Structured Extraction
confidence 90% · LLaVA (0.000) ... perform at or near zero, most likely due to malformed outputs
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The extraction of structured information from unstructured documents represents a critical component of digital transformations in all sectors. While proprietary solutions dominate commercial applications, a rapidly growing ecosystem of open-source Optical Character Recognition (OCR) engines, Large Language Models (LLMs), and Vision-Language Models (VLMs) offers accessible alternatives. However, systematic evaluations on realistic, multi-step extraction pipelines remain scarce. Responsible usage of such extraction tools require comprehensive evaluations on realistic tasks, especially as these solutions will be key components of applications in the public sector that the EU AI act categorizes as high risk. To address this gap we present a comprehensive benchmark assessing the end-to-end performance of open-source systems on a complex real-world document processing task classified as high risk: Student applications for an international study program. We conduct a comprehensive empirical evaluation with state-of-the-art OCR engines, LLMs and VLMs. Our results reveal that while VLMs generally outperform OCR+LLM pipelines, even state-of-the-art open-source models struggle to handle such tasks reliably in zero-shot settings. Only 4 of 35 configurations achieved F1 scores above 0.5, with the best OCR+LLM pipeline matching top VLM performance, though most OCR+LLM combinations performed substantially worse. Roughly 75\% of all configurations scored below 0.25. Model scale influences performance, yet the relationship is non-linear: substantially larger models do not guarantee proportionally better results. Input quality, particularly the structural preservation of OCR output, emerges as a critical factor independent of downstream model capability.
Tags
Links
- Source: https://arxiv.org/abs/2608.18289v1
- Canonical: https://arxiv.org/abs/2608.18289v1
Trouble viewing inline? Open PDF directly →
Full Text
34,979 characters extracted from source content.
Expand or collapse full text
Evaluating Structured Information Extraction with Open Models in a High Risk Public Sector Application Elias Schubert 1 and Felix Bießmann 1,2[0000−0002−3422−1026] 1 Berlin University of Applied Sciences, Luxemburger Str. 10, 13353 Berlin, Germany s92538,felix.biessmann@bht-berlin.de https://w.bht-berlin.de 2 Einstein Center Digital Future, Wilhelmstraße 67, 10117 Berlin, Germany https://w.digital-future.berlin Abstract. The extraction of structured information from unstructured documents represents a critical component of digital transformations in all sectors. While proprietary solutions dominate commercial appli- cations, a rapidly growing ecosystem of open-source Optical Charac- ter Recognition (OCR) engines, Large Language Models (LLMs), and Vision-Language Models (VLMs) offers accessible alternatives. However, systematic evaluations on realistic, multi-step extraction pipelines re- main scarce. Responsible usage of such extraction tools require compre- hensive evaluations on realistic tasks, especially as these solutions will be key components of applications in the public sector that the EU AI act categorizes as high risk. To address this gap we present a comprehensive benchmark assessing the end-to-end performance of open-source systems on a complex real-world document processing task classified as high risk: Student applications for an international study program. We conduct a comprehensive empirical evaluation with state-of-the-art OCR engines, LLMs and VLMs. Our results reveal that while VLMs generally out- perform OCR+LLM pipelines, even state-of-the-art open-source models struggle to handle such tasks reliably in zero-shot settings. Only 4 of 35 configurations achieved F1 scores above 0.5, with the best OCR+LLM pipeline matching top VLM performance, though most OCR+LLM com- binations performed substantially worse. Roughly 75% of all configu- rations scored below 0.25. Model scale influences performance, yet the relationship is non-linear: substantially larger models do not guarantee proportionally better results. Input quality, particularly the structural preservation of OCR output, emerges as a critical factor independent of downstream model capability. These findings provide practical guid- ance for practitioners selecting models for document processing work- flows while highlighting fundamental limitations in current open-source approaches to structured information extraction. Keywords: Optical Character Recognition· Large Language Model· Visual Large Language Model· Benchmark· Open-Source. arXiv:2608.18289v1 [cs.AI] 18 Aug 2026 2Schubert and Bießmann 1 Introduction The ability to extract structured information from unstructured or semi-structured sources has become an increasingly critical capability in modern data-driven workflows. Digital transformations across all sectors require these workflows to reliably integrate information from scanned documents or digitally edited forms. The rapid advancement of Optical Character Recognition (OCR), Large Lan- guage Models (LLMs), and Vision-Language Models (VLMs) has fundamen- tally transformed this landscape, making such extraction tasks more accurate, scalable, and accessible than ever before. Next to proprietary solutions, often marketed as all-encompassing general-purpose models, a rich ecosystem of free- to-use, open-source OCR engines, LLMs, and VLMs has emerged. These openly available models present substantial opportunities for processing large volumes of unstructured data, such as the ubiquitous PDF format, without incurring the costs or restrictions of commercial systems. Leveraging this potential requires more and systematic empirical evaluations of how freely available, off-the-shelf models perform on realistic extraction tasks. This is even more important given that most of this technology has been devel- oping faster than corresponding regulation. While many benchmarks and evalua- tions focus on tasks that are either simple or not classified as high-risk according to the EU AI Act [6], using the technology in high risk settings such as health care or workflows in the public sector requires extensive empirical evaluations. Even though these kinds of evaluations are more on the applied side of research we argue that such evaluations are contributions that can and should be made by researchers to build trust in open source technology in the public sector. To address this challenge, this work makes two main contributions. For one, we devise an evaluation protocol for extraction performance of a high risk use case in the public sector, student applications in an international study program. This task is representative for many other complex tasks: it requires processing of heterogeneous PDF documents containing a mixture of structured and unstruc- tured data, extracting targeted information, and returning it in a well-defined structured format. This workflow, illustrated in Figure 2 is representative of op- erational use cases encountered across numerous domains. Second we use this evaluation protocol for a benchmark assessing the capabilities of open OCR sys- tems combined with LLMs, as well as standalone VLMs. The results are discussed with respect to the extraction performance as well as other factors relevant for practical usage, such as computational efficiency. 2 Related Work While OCR engines, LLMs, and VLMs have each been evaluated extensively in single-step scenarios, benchmarks assessing their individual or combined per- formance on complex, multi-step extraction tasks in a zero-shot setting remain sparse. Such tasks, which involve converting documents into machine-readable text, extracting specified information, and returning it in a strictly defined struc- ture, closely reflect how these models are likely to be deployed in practice and Evaluating Open Structured Information Extraction3 constitute precisely the end-to-end pipeline evaluated in this work. Existing benchmarks tend to isolate individual components of this pipeline. With respect to OCR, Khan et al. [18] evaluated several engines on the CBC Reports Dataset, measuring accuracy, processing time and error rates. PaddleOCR and EasyOCR emerged as the strongest performers, a finding that directly informed the selection of OCR models used in this work. On the side of information extraction from unstructured and semi-structured documents, Ntinopoulos et al. [21] benchmarked mostly LLMs and table transformers on electronic health records in a zero-shot setting, finding that proprietary mod- els consistently achieved strong results with F1 scores above 0.9, outperforming several open-source models. They also found that table transformers achieved higher extraction performance than LLMs when only layout structure is consid- ered, but LLMs surpass table transformers once cell content becomes relevant. Similarly, Gao et al. [7] examined fine-grained extraction across multiple infor- mation types, observing that encoder-decoder architectures generalize better to unseen information types, while decoder-only architectures handle unseen task forms more effectively. Notably, model scale did not consistently predict perfor- mance, a theme that recurs across the literature. The interplay between OCR and multimodal LLMs for table extraction was investigated by Nunes et al. [22], who benchmarked OCR systems and MLLMs on extracting tables from images using a subset of non-complex tables from PubTables-1M. Although their setup does not include a full end-to-end extrac- tion pipeline, it closely aligns with the benchmark presented here and pro- vides valuable insights. Similar to this work, they enforce structured outputs by prompting models with a predefined Pydantic schema. Evaluating both pro- prietary and open models, they report strong performance across categories, with several models achieving F1 scores between 85 and 95, depending on the evaluation aspect. Regarding structured output generation, Tenckhoff et al. [28] benchmarked 22 open-source models across five prompting strategies on a synthetically gen- erated dataset, evaluating their ability to extract structured data and produce valid JSON from natural-language text, with GPT-4o included as a proprietary reference. The results reinforce that prompting strategy and model architecture are at least as influential as model scale, and that leading open-source models such as Gemma3-27B can rival proprietary systems. Finally, Roberts et al. [26] assessed VLMs on their ability to extract structure directly from images, using an automatic evaluation approach based on render- ing model outputs and comparing them to the original. Closed-weight models, in particular GPT-4o, significantly outperformed open-weight alternatives, and VLMs struggle to pick up visual nuances and were shown to be sensitive to prompt formulation, suggesting that zero-shot standardized prompting may be suboptimal for such tasks. Taken together, these works highlight meaningful progress across the individ- ual components relevant to document extraction pipelines, while also underscor- ing the absence of benchmarks that evaluate free and open models holistically on 4Schubert and Bießmann the complete three-step problem of document ingestion, information extraction and structured output generation that this paper directly addresses. 3 Methods In the following we first describe the study program application data and the ex- traction task, followed by a description of the extraction workflow and evaluation metrics. 3.1 Data The dataset used in this benchmark consists of 100 academic transcripts sub- mitted as part of applications to a German Data Science Master’s programme, provided as PDF documents. Of these, 94 are written entirely in English, three are bilingual in English and Turkish, two are in German, and one combines En- glish and Russian. The vast majority of applicants originate from Asian coun- tries, accounting for over 90% of the dataset. Each document typically contains structured data in the form of tables as well as unstructured elements such as free text, images and institutional sigils. As the documents contain personal information, the dataset cannot be published in its current form and must be anonymized prior to any release. An example document is shown in Figure 1. All documents were standardized to A4 format and additionally converted to JPG images to accommodate models requiring image input. Each document was then manually parsed to extract all courses deemed relevant to the extraction task, yielding a ground truth against which model outputs are evaluated. The extraction follows a ruleset designed to handle the heterogeneity and formatting inconsistencies typical of international academic documents: – Extract only Computer Science or Mathematics courses. – In multi-language documents, prefer the English version. – In case of multiple grade formats, prefer letter grades. – Missing values are filled with N/A. – Without a clear designation of total credits, select the first value in a row. – Formatting inconsistencies such as erroneous spacing or incorrect characters are preserved as-is in the ground truth. Table 1, which shows the ground truth belonging to the document shown in Figure 1, illustrates the application of these rules. With multiple credit columns given and none being explicitly designated, the first credit value for each course is selected. For example, "OPERATING SYSTEMS" appears with both the- ory (3.0) and practical (1.0) credits in the original document, but only 3.0 is extracted as it is the first credit value associated with that course. The extracted ground truth comprises four columns: academic field, course name, grade, and awarded credits, totaling 952 unique rows across all documents. The number of extracted rows per document ranges from 0 to 44, with no du- plicate entries within any single document. Evaluating Open Structured Information Extraction5 Fig. 1. Anonymized example document. Table 1. Manually extracted Ground Truth for the Document in Figure 1 Academic FieldCourse NameGradeAwarded Credits MathematicsMATHEMATICS FOR COMPUTER THEORY ENGINEERING & INFOR- MATION TECHNOLOGY B4.00 Computer ScienceOPERATING SYSTEMSB3.00 Computer ScienceDESIGN AND ANALYSIS OF ALGO- RITHMS C3.00 Computer ScienceAPPLICATIONDEVELOPMENT TOOLS B2.00 Computer SciencePROGRAMMING FOR SCIENTIFIC THEORY COMPUTING B+3.00 Computer ScienceNO SQL DATABASE SYSTEMSB+1.00 6Schubert and Bießmann 3.2 Extraction We compare the performance of OCR engines combined with LLMs against standalone VLMs in a three-step pipeline: digitalizing and preparing document content, extracting task-relevant information as specified in the prompt, and returning it as a structured output. All LLMs and VLMs are run using the Ollama platform, which provides consistent base settings across all models as well as an integrated format parameter that enforces structured output via a Pydantic schema. Fig. 2. Workflow of extraction and evaluation pipeline The OCR engines used in this benchmark were selected based on two crite- ria: popularity among practitioners, measured by GitHub star count, and perfor- mance reported in recent benchmarking literature [9] [20]. All models are used with pretrained weights without any further fine-tuning. The following engines were evaluated: Pytesseract [27], P-OCRv5 (PaddleOCR) [5], P-StructureV3 (PaddleOCR) [5], MinerU [30], EasyOCR [11], and docTR [15]. Table 2 provides an overview of each model’s input, output and language configuration. Table 3 details the architecture components for engines where explicit configuration was specified, with selections based on the highest-performing options as indicated in each engine’s official documentation and benchmarks. Evaluating Open Structured Information Extraction7 Table 2. OCR model configurations and output formats. ModelInputLanguage ConfigOutput Format docTRPDFauto-detectUnstructured; line breaks between text blocks EasyOCRImage (JPG)EnglishUnstructured; tabs between text blocks (manually added) MinerUPDFEnglishStructured; HTML tags for tables, line breaks between text rows PPOCRv5Image (JPG)EnglishUnstructured; tabs between text blocks (manually added) PPStructureV3PDFEnglishStructured; HTML tags for tables, line breaks between text blocks PytesseractImage (JPG)EnglishStructured; line breaks between text rows Table 3. OCR model architecture components (specified models only). ModelOrient. Class.Doc UnwarpingText DetectionTextline Orient.Text Recognition docTR—db_resnet50—master EasyOCR— MinerU— PPOCRv5P-LCNet_x1_0_doc_oriUVDocOCRv5_server_detPP-LCNet_x1_0_textline_oriPP-OCRv5_server_rec PPStructureV3P-LCNet_x1_0_doc_oriUVDocOCRv5_server_detPP-LCNet_x1_0_textline_oriPP-OCRv5_server_rec Pytesseract— For the extraction of relevant structured information from the OCR output, a single LLM was deliberately chosen to reduce variables and place the focus on the effect of model scale. Qwen3 [31], a modern model family released in 2025, was selected as it offers a wide range of parameter counts within a single architecture. Five parameter sizes were evaluated, as detailed in Table 4. Table 4. Qwen3 model architecture parameters [31]. Context lengths as specified in Ollama [23]. Q/KV heads refer to query and key-value attention heads [29]. ModelParametersContext LengthLayersHeads (Q / KV) Qwen30.6B40K2816 / 8 Qwen34B256K3632 / 8 Qwen314B40K4040 / 8 Qwen332B40K6464 / 8 Qwen3235B256K9464 / 4 VLM based Extraction VLMs were selected following the similar criteria as the OCR engines, drawing from both the most used models from Ollama model zoo and recent literature [14]. An effort was made to keep parameter 8Schubert and Bießmann counts comparable across models at approximately 30 billion parameters, though not all models offered variants at this scale. The following models were evalu- ated: LLaVA (7B) [17], Ministral-3 (14B) [16] , Gemma3 (27B) [8], Qwen2.5VL (32B) [2], and Qwen3VL (32B) [3]. As with the LLM based pipeline, structured outputs were generated using the same Pydantic schema via the Ollama format API. Table 5 provides a detailed overview of each model. Table 5. Vision-language model overview. ModelYearArchitectureParametersContextVision Encoder Gemma 32025Decoder-only27B128KSigLIP LLaVA2024Encoder-decoder7B32KCLIP Ministral 32025Decoder-only14B256KViT Qwen2.5-VL2025Encoder-decoder32B125KViT Qwen3-VL2025Encoder-decoder32B256KSigLIP-2 Prompting Strategy and Experimental Design The prompt design incor- porates several evidence-based strategies from recent literature: meaning-typed prompting via Pydantic schemas rather than rigid JSON definitions for reliable structured output [10], a verification phase where the model reviews its own output [25], delimiters to separate schema definitions and text sections [4], and concise, direct instructions [13]. Structured output is enforced via the following Pydantic schema: class Course(BaseModel ): academic_field: Literal["Computer␣Science", "Mathematics"] course_name: str grade: float | str | None = None awarded_credits: float | None = None class Courses(BaseModel ): courses: List[Course] The prompt follows a structured format: input data and goal definition, schema presentation with illustrative examples, output rules, context provision, and a verification section with output format specifications. Temperature is set to 0 for all models, following best practices for deterministic structured genera- tion as recommended in the Ollama documentation [24]. All experiments were executed on a compute cluster using a single NVIDIA H200 GPU with 141 GB VRAM, 32 GB system memory, and 8 CPU cores. System memory and CPU utilization remained well below capacity throughout all experiments. For the Qwen3:235b model, an NVIDIA B200 GPU with 192 GB VRAM was briefly utilized to accelerate inference. Evaluating Open Structured Information Extraction9 3.3 Evaluation Model outputs were first validated and normalized to ensure comparability with ground truth. Empty grade and credit cells columns were filled with N/A to match the ground truth format. When models failed to return data in the re- quired structure due to recursive reasoning patterns, additional text generation, inclusion of extra columns, or omission of required fields, an empty dataframe with the correct schema was substituted, effectively treating non-compliant out- puts as complete extraction failures. Both the extracted data and ground truth underwent identical normalization procedures: (1) index columns were dropped, (2) leading and trailing whitespace was stripped, (3) academic field and course name values were converted to lower- case, and (4) numerical values were cast to floats, while any alphabetic characters in grade or credit columns were also converted to lowercase. Evaluation Metrics For our evaluation we focus on exact string matches after lowercasing. Both the ground truth and extracted data were tokenized by con- verting each row, consisting of academic field, course name, grade, and awarded credits, into a single unique token as is illustrated in Figure 3. Given that ground truth contains no duplicate rows, both datasets were represented as sets of row tokens, enabling efficient set-based comparison. From these sets, a combined set of all unique tokens was constructed using set intersection. Fig. 3. Tokenization visualized Following standard practice in information or text extraction benchmarks [1, 12, 19] we computed Precision, Recall, and F1 scores. To determine True Posi- tives (TP), False Positives (FP), and False Negatives (FN) for these metrics, we adopt a set-theoretic approach based on Jaccard similarity [1]: – True Positives (TP): Tokens present in both extracted output and ground truth – False Positives (FP): Tokens in extracted output but absent from ground truth – False Negatives (FN): Tokens in ground truth but absent from extracted output In the subsequent analysis, F1 Score serves as the primary metric for model evaluation and comparison, as it provides a balanced measure of both extrac- tion accuracy and completeness, making it particularly suitable for assessing 10Schubert and Bießmann performance on imbalanced extraction tasks where both false positives and false negatives carry significant cost. 3.4 Results In Table 6 we show the overall results for each OCR+LLM and VLM model and Figure 4 shows the best F1 score per model, comparing the top-performing OCR+LLM combination for each OCR engine against all VLMs. VLMs outperform OCR+LLM approaches In the OCR+LLM pipeline, the best- performing LLM was Qwen3:235B in all cases. The results reveal a clear perfor- mance gap between OCR engines, largely attributable to the quality and struc- ture of their output. MinerU stands out as the strongest OCR engine, achieving an F1 score of 0.509, placing it on par with the best-performing VLM, Qwen2.5- VL (0.509). This can be attributed to MinerU’s output format, which preserves the spatial layout of the page by grouping text that appears on the same hori- zontal line into a single row, and representing tabular content using HTML tags to explicitly encode cell boundaries. Other OCR engines, by contrast, insert line breaks after each individual text block regardless of its position on the page, and largely fail to represent table structure semantically. This directly demonstrates the critical importance of input quality to downstream LLM performance: the structure of the OCR output is at least as consequential as the capability of the LLM processing it. On the VLM side, results are polarized. Qwen2.5-VL (0.509) and Ministral-3 (0.508) achieve the highest scores overall, while LLaVA (0.000) and Gemma3 (0.015) perform at or near zero, most likely due to malformed outputs that were treated as complete extraction failures under the evaluation protocol. Notably, Qwen2.5-VL marginally outperforms its successor Qwen3-VL (0.487), suggesting that a newer model version does not necessarily translate to improved task-specific performance. Overall, results across both pipelines are poor: the mean F1 score across all VLMs and the best OCR+LLM combination per engine is 0.282, with the highest observed score capped at 0.509. Parameter Size and Extraction Performance Figure 5 illustrates how F1 score varies across Qwen3 parameter sizes for each OCR engine. A consistent pattern emerges: the 0.6B model performs substantially worse than all larger variants across every OCR engine, frequently producing near-zero scores. This is likely not solely a function of model capacity, but also a known instability of the Qwen3:0.6B model, which has been documented to enter a repetitive generation loop, producing incoherent output that fails to conform to the required Pydantic schema. Beyond the 0.6B outlier, larger models generally yield modest but con- sistent improvements, with the 235B variant typically achieving the highest score per engine. However, this trend is not universal: for PPOCRv5 and docTR, the 14B model scores marginally higher than the 32B model, reinforcing the finding from related work that model scale and task performance do not maintain a strictly linear relationship. Evaluating Open Structured Information Extraction11 Table 6. Extraction performance metrics for OCR+LLM pipeline and VLM models. OCR EngineLLM ModelPrecisionRecallF1 OCR + LLM Pipeline docTRQwen3 0.6B0.0140.0220.016 docTRQwen3 4B0.1990.2170.201 docTRQwen3 14B0.2020.2380.215 docTRQwen3 32B0.2040.2220.209 docTRQwen3 235B0.3010.3290.311 EasyOCRQwen3 0.6B0.0070.0090.008 EasyOCRQwen3 4B0.0790.0880.080 EasyOCRQwen3 14B0.0790.1110.088 EasyOCRQwen3 32B0.1030.1250.108 EasyOCRQwen3 235B0.1040.1260.112 MinerUQwen3 0.6B0.0780.0810.075 MinerUQwen3 4B0.4150.4720.427 MinerUQwen3 14B0.4550.4990.465 MinerUQwen3 32B0.4710.5510.501 MinerUQwen3 235B0.4890.5420.509 PPOCRv5Qwen3 0.6B0.0020.0020.002 PPOCRv5Qwen3 4B0.1600.1990.170 PPOCRv5Qwen3 14B0.2070.2620.222 PPOCRv5Qwen3 32B0.1630.1840.169 PPOCRv5Qwen3 235B0.3120.3450.324 PPStructureV3Qwen3 0.6B0.0220.0200.020 PPStructureV3Qwen3 4B0.1320.1460.129 PPStructureV3Qwen3 14B0.1230.1330.124 PPStructureV3Qwen3 32B0.1570.1630.157 PPStructureV3Qwen3 235B0.1760.1910.179 PytesseractQwen3 0.6B0.0050.0050.005 PytesseractQwen3 4B0.0910.0950.092 PytesseractQwen3 14B0.0860.1000.091 PytesseractQwen3 32B0.0900.0960.092 PytesseractQwen3 235B0.1450.1620.151 VLM-Only Pipeline VLM ModelPrecisionRecallF1 Gemma3 27B0.0210.0180.015 LLaVA 7B0.0000.0000.000 Ministral-3 14B0.4800.5650.508 Qwen2.5-VL 32B0.4730.5740.509 Qwen3-VL 32B0.4410.5730.487 12Schubert and Bießmann Fig. 4. Best F1 score per model: top OCR+LLM combination (always Qwen3:235B) versus all VLMs. Fig. 5. F1 scores across Qwen3 parameter sizes for each OCR engine. Simplified Extraction with Fewer Attributes To investigate the sensitivity of the performance with reduced task complexity, we simplified the task and neglected the grade and awarded credits attributes. Figure 6 presents the improvement in F1 score (∆F1) when the grade column, the awarded credits column, or both are removed from both ground truth and model output prior to evaluation. Positive deltas indicate improved scores under the reduced column setting. Across all model configurations, the largest improvements are observed when both columns Evaluating Open Structured Information Extraction13 are removed, retaining only academic field and course name. Removing grades consistently yields a larger improvement than removing awarded credits, though the magnitude varies considerably across configurations, ranging from below 0.01 to above 0.1. This suggests that grade extraction poses a greater challenge, likely due to the diversity of grading formats across transcripts, including both letter and numerical grades, which introduces an additional layer of ambiguity even when explicit handling rules are provided in the prompt. These patterns hold consistently across both VLM and OCR+LLM configurations. Finally, as observed in Figure 5, the 0.6B Qwen3 configurations show minimal improvement even when columns are removed. This further supports the interpretation that the near-zero baseline scores for these configurations are driven primarily by output invalidity due to the model’s repetitive generation behavior, rather than extraction difficulty. Fig. 6. F1 improvement when removing grade, awarded credits, or both. 14Schubert and Bießmann 4 Conclusion This benchmark evaluated the end-to-end capability of open-source OCR en- gines, LLMs, and VLMs to extract structured information from heterogeneous real-world documents in a zero-shot setting, using all models in their pretrained state with best available configurations. The results reveal that none of the tested approaches can be considered a reliable out-of-the-box solution for complex ex- traction tasks of this kind, with roughly 75% of all configurations scoring below an F1 of 0.25. A central finding is that the quality of LLM output is strongly dependent on the quality of the context provided. OCR engines that preserve the spatial and structural layout of the source document, as MinerU does via row-wise text grouping and HTML-encoded tables, yield dramatically better downstream ex- traction results than engines that produce unstructured, block-fragmented out- put. This underlines that input preparation is at least as important as model capability in such pipelines. VLMs generally outperform OCR+LLM pipelines in this task, as they bypass the OCR step entirely and operate directly on the document image. However, their performance is far from reliable, with several models producing near-zero scores due to malformed outputs. It is also worth noting that a newer model is not always a better one: Qwen2.5-VL outperformed its successor Qwen3-VL on this task, and similarly, larger LLM parameter counts do not guarantee propor- tionally better results, though very small models such as Qwen3:0.6B are clearly insufficient for tasks of this complexity. These findings apply specifically to the zero-shot, off-the-shelf setting eval- uated here. The performance levels observed are not necessarily an inherent ceiling: improved preprocessing and data quality feeding into the LLM or VLM, more targeted prompting strategies, and refined schema definitions are all promis- ing directions that could meaningfully improve results. These do however require additional effort and experimentation, and suggest that deploying such pipelines reliably in practice demands more than simply selecting a capable model off the shelf. Improved data preparation, more targeted prompting strategies, and refined schema definitions are all promising directions that could meaningfully improve performance without requiring fine-tuning, and warrant further inves- tigation. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article. References 1. Adhikari, N.S., Agarwal, S.: A comparative study of PDF parsing tools across diverse document categories (2025), https://arxiv.org/abs/2410.09871 2. Bai,S.,etal.:Qwen2.5-VLtechnicalreport(2025), https://arxiv.org/abs/2502.13923 Evaluating Open Structured Information Extraction15 3. Bai,S.,etal.:Qwen3-VLtechnicalreport(2025), https://arxiv.org/abs/2511.21631 4. Chen, L.C., Weng, H.T., Pardeshi, M.S., Chen, C.M., Sheu, R.K., Pai, K.C.: Evaluation of prompt engineering on the performance of a large language model in document information extraction. Electronics 14(11), 2145 (2025). https://doi.org/10.3390/electronics14112145 5. Cui, C., et al: PaddleOCR 3.0 technical report (2025), https://arxiv.org/abs/2507.05595 6. European Parliament and Council: Regulation (eu) 2024/1689 of the european parliament (July 2024), https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng, oJ L 12.7.2024, p. 1–226 7. Gao, J., Zhao, H., Zhang, Y., Wang, W., Yu, C., Xu, R.: Benchmarking large lan- guage models with augmented instructions for fine-grained information extraction (2023), https://arxiv.org/abs/2310.05092 8. Gemma Team, Kamath, A., et al.: Gemma 3 technical report (2025), https://arxiv.org/abs/2503.19786 9. Heakl, A., Sohail, A., Ranjan, M., Hossam, R., Ahmad, G.S., El-Geish, M., Maher, O., Shen, Z., Khan, F., Khan, S.: KITAB-Bench: A comprehensive multi-domain benchmark for arabic OCR and document understanding (2025), https://arxiv.org/abs/2502.14949 10. Irugalbandara, C.: Meaning typed prompting: A technique for efficient, reliable structured output generation (2024), https://arxiv.org/abs/2410.18146 11. JaidedAI:EasyOCR:Ready-to-useOCRwith80+sup- portedlanguages.https://github.com/JaidedAI/EasyOCR(2020), https://github.com/JaidedAI/EasyOCR 12. Kim, D., Hong, Y.: Figure text extraction in biomedical literature. PloS one 6(1), e15338 (2011). https://doi.org/10.1371/journal.pone.0015338 13. Li, Y.: A practical survey on zero-shot prompt design for in-context learning. In: Proceedings of the Conference Recent Advances in Natural Language Processing - Large Language Models for Natural Language Processing. p. 641–647. INCOMA Ltd. (2023). https://doi.org/10.26615/978-954-452-092-2_069 14. Li, Z., Wu, X., Du, H., Liu, F., Nghiem, H., Shi, G.: A survey of state of the art large vision language models: Alignment, benchmark, evaluations and challenges (2025), https://arxiv.org/abs/2501.02189 15. Liao, H., RoyChowdhury, A., Li, W., Bansal, A., Zhang, Y., Tu, Z., Satzoda, R.K., Manmatha, R., Mahadevan, V.: DocTr: Document transformer for structured information extraction in documents (2023), https://arxiv.org/abs/2307.07929 16. Liu, A.H.e.a.: Ministral 3 (2026), https://arxiv.org/abs/2601.08584 17. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning (2023), https://arxiv.org/abs/2304.08485 18. Naqi Hadi, M.A., Gul, M., Khan, M., Alwakid, G.N., Zaman Jhanjhi, N.: Benchmarking performance analysis of optical character recognition techniques. In: 2024 26th International Multi-Topic Conference (INMIC). p. 1–6 (2024). https://doi.org/10.1109/INMIC64792.2024.11004392 19. Nasar, Z., Jaffry, S., Malik, M.: Information extraction from scientific articles: a survey. Scientometrics 117, 1931–1990 (2018). https://doi.org/10.1007/s11192- 018-2921-5 20. Nazeem, M., R, A., S, N., R. R, R.: Open-source OCR libraries: A comprehensive study for low resource language. In: Proceedings of the 21st International Confer- ence on Natural Language Processing (ICON). p. 416–421. NLP Association of India (NLPAI) (2024), https://aclanthology.org/2024.icon-1.48/ 16Schubert and Bießmann 21. Ntinopoulos, V., Rodriguez Cetina Biefer, H., Tudorache, I., Papadopoulos, N., Odavic, D., Risteski, P., Haeussler, A., Dzemali, O.: Large language models for data extraction from unstructured and semi-structured electronic health records: a multiple model performance evaluation. BMJ Health & Care Informatics 32(1), e101139 (2025). https://doi.org/10.1136/bmjhci-2024-101139 22. Nunes, G., Rolla, V., Pereira, D., Alves, V., Carreiro, A., Baptista, M.: Bench- marking table extraction: Multimodal LLMs vs traditional OCR. In: Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling (XLLM 2025). p. 8–15. Association for Computational Linguistics, Vienna, Aus- tria (2025). https://doi.org/10.18653/v1/2025.xllm-1.2 23. Ollama: Qwen3 — ollama library. https://ollama.com/library/qwen3 (2025), https://ollama.com/library/qwen3, accessed: 2025-03-03 24. Ollama:Structuredoutputs—ollamadocumentation. https://docs.ollama.com/capabilities/structured-outputs(2025), https://docs.ollama.com/capabilities/structured-outputs, accessed: 2025-03- 03 25. Polak, M.P., Morgan, D.: Extracting accurate materials data from research papers with conversational language models and prompt engineering. Nature Communi- cations 15(1), 1569 (2024). https://doi.org/10.1038/s41467-024-45914-8 26. Roberts, J.S., Lee, T., Wong, C.H., Yasunaga, M., Mai, Y., Liang, P.: Image2Struct: Benchmarking structure extraction for vision-language models. In: Advances in Neural Information Processing Systems. vol. 37, p. 115058–115097. Curran Asso- ciates, Inc. (2024). https://doi.org/10.52202/079017-3653 27. Smith, R.: An overview of the tesseract OCR engine. In: Ninth International Con- ference on Document Analysis and Recognition (ICDAR). vol. 2, p. 629–633. IEEE (2007). https://doi.org/10.1109/ICDAR.2007.4376991 28. Tenckhoff, S., Koddenbrock, M., Rodner, E.: LLMStructBench: Bench- marking large language model structured data extraction (2026), https://arxiv.org/abs/2602.14743 29. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need (2023), https://arxiv.org/abs/1706.03762 30. Wang, B., et al.: MinerU: An open-source solution for precise document content extraction (2024), https://arxiv.org/abs/2409.18839 31. Yang, A., et al.: Qwen3 technical report (2025), https://arxiv.org/abs/2505.09388