Paper deep dive

LAVA: Logic-Aware Validation and Augmentation Framework for Large-Scale Financial Document Auditing

Ruoqi Shu, Xuhui Wang, Isaac Wang, Yanming Mai, Bo Wan

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 77

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 8/23/2026, 2:41:13 AM

Summary

The paper introduces LAVA (Logic-Aware Validation and Augmentation), a modular, backbone-agnostic framework for auditing large-scale financial documents. LAVA addresses challenges in accuracy, consistency, and auditability by integrating four stages: document-rule retrieval, layout-preserving information extraction, auxiliary metadata enrichment, and auditable symbolic/arithmetic verification. It utilizes multimodal large language models (MLLMs) combined with external tools (like Python interpreters and checkers) to handle heterogeneous layouts, complex business rules, and numeric computations, significantly reducing hallucinations and improving traceability compared to baseline pipelines.

Entities (10)

LAVA · framework · 98%Arithmetic Processor · module · 95%Symbolic Reasoner · module · 95%Multimodal Large Language Models · technology · 92%Claude 3.7 Sonnet · model · 90%Financial Document Auditing · task · 90%AWS Textract · tool · 88%Tesseract OCR · tool · 88%Python Interpreter · tool · 85%LayoutLMv3 · model · 82%

Relation Signals (8)

LAVA → containsmodule → Arithmetic Processor

confidence 95% · bifurcated framework with two sub-modules: Arithmetic Processor... and Symbolic Reasoner

LAVA → containsmodule → Symbolic Reasoner

confidence 95% · bifurcated framework with two sub-modules: ... Symbolic Reasoner

LAVA → solvesproblem → Financial Document Auditing

confidence 95% · LAVA... framework for verifiable reasoning over semi-structured, layout-complex financial documents

LAVA → uses → Multimodal Large Language Models

confidence 95% · pipeline built on multimodal large language models

LAVA → usesmodel → Claude 3.7 Sonnet

confidence 90% · all baselines, LAVA... use Claude 3.7 Sonnet 1 as the validation component

Arithmetic Processor → usestool → Python Interpreter

confidence 90% · formula... is then executed by a deterministic external engine such as a Python interpreter

LAVA → usestool → Tesseract OCR

confidence 85% · experiment exclusively utilizes... Tesseract OCR

LAVA → usestool → AWS Textract

confidence 85% · processing of structural semantics... AWS Textract service was employed

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Financial document validation in production, such as payroll auditing, tax compliance, and loan underwriting, demands exceptional accuracy, consistency, and reproducibility under strict enterprise constraints. In practice, documents arrive with heterogeneous layouts and formats, semantically rich and context-dependent content, and embedded business rules that current pipelines struggle to process reliably. We introduce LAVA (Logic-Aware Validation and Augmentation), a modular, backbone-agnostic pipeline built on multimodal large language models, that integrates a four-stage design: document-rule retrieval, layout-preserving information extraction, auxiliary metadata enrichment, and auditable symbolic/arithmetic verification. LAVA supports robust rule grounding, fine-grained error attribution, and consistent, traceable end-to-end execution, capabilities essential for high-stakes deployment. Evaluated on a large real-world benchmark with diverse financial documents and dozens of expert-curated validation rules, LAVA outperforms baselines in hallucination control and edge-case handling while maintaining efficient token usage, demonstrating practicality for high-volume, time-critical validation.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

73,775 characters extracted from source content.

Expand or collapse full text

LAVA: Logic-Aware Validation and Augmentation Framework for Large-Scale Financial Document Auditing Ruoqi Shu Thanks: Equal contribution. Thanks: Corresponding author. Xuhui Wang11footnotemark: 1 22footnotemark: 2 Isaac Wang Thanks: Work done during an internship at BMO Financial Group. Yanming Mai Bo Wan Affiliation: BMO Financial Group Affiliation: ruoqi.shu, xuhuieric.wang@bmo.com, Affiliation: isaac.wang, yanming.mai, bo.wan@bmo.com Abstract Financial document validation in production—such as payroll auditing, tax compliance, and loan underwriting—demands exceptional accuracy, consistency, and reproducibility under strict enterprise constraints. In practice, documents arrive with heterogeneous layouts and formats, semantically rich, context-dependent content, and embedded business rules that current pipelines struggle to process reliably. We introduce LAVA (Logic-Aware Validation and Augmentation)—a modular, backbone-agnostic pipeline built on multimodal large language models—that integrates a four-stage design: document–rule retrieval, layout-preserving information extraction, auxiliary metadata enrichment, and auditable symbolic/arithmetic verification. LAVA supports robust rule grounding, fine-grained error attribution, and consistent, traceable end-to-end execution—capabilities essential for high-stakes deployment. Evaluated on a large real-world benchmark with diverse financial documents and dozens of expert-curated validation rules, LAVA outperforms baselines in hallucination control and edge-case handling while maintaining efficient token usage, demonstrating practicality for high-volume, time-critical validation. 1 Introduction Regulatory penalties, financial losses, and reputational damage can all result from a single error in financial document validation, making accuracy, consistency, and auditability non-negotiable. Financial institutions process millions of documents daily across workflows such as loan underwriting, payroll auditing, tax compliance, and fraud detection. The challenge is acute for semi-structured documents like statements, invoices, and tax slips, which span multiple pages, exhibit irregular layouts, encode domain-specific business logic, and often arrive as noisy scans or non-standard PDFs 3; 10; 41; 8. Recent years have witnessed rapid progress in visually rich document understanding through layout- and structure-aware pretraining of multimodal large language models (MLLMs). LayoutLM 41 pioneered spatial–textual joint encoding, followed by DocFormer 2, DocLLM 35, mPLUG-DocOwl2 15, and ROP 42, advancing multimodal architectures for structural representation. This shift moves beyond pure text modelling to multimodal document intelligence. However, evaluations remain dominated by perceptual and question answering (QA) tasks, probing reasoning in a narrow, task-specific manner while rarely accessing validation—reasoning that tests both interpretation and reliability under structured and cross-field constraints. Enterprise-grade document validation presents requirements beyond those of perception or reasoning alone. It calls for symbolic rule enforcement, cross-field consistency, and multi-step logical coherence—capabilities only partially reflected in existing benchmarks 37; 20; 4. Recent datasets have expanded evaluation to layout-aware perception and understanding 45; 39; 23; 31; 29, but compliance-critical validation logic remains underexplored. In practice, enterprises often patch the gap by pairing general-purpose models with rigid rule-based modules 28. Yet even state-of-the-art models that excel at extraction or understanding exhibit ongoing limitations when directly applied to validation: hallucinations remain common, and reasoning traceability is limited. Adaptation across regulatory schemas is fragile, and computational costs escalate with token usage at enterprise scale. These shortcomings make validation not merely an extension but a distinct and emerging frontier of document intelligence—one that demands frameworks where accuracy, efficiency, audit readiness, and robustness are central requirements. Our Work. To address these challenges, we present LAVA (Logic-Aware Validation and Augmentation), a modular and efficient framework for verifiable reasoning over semi-structured, layout-complex financial documents, designed for real-world applicability, fine-grained error attribution, and rapid adaptation and reproducibility across structurally similar collections. Agnostic to backbone models, LAVA extends beyond static benchmarks by integrating (i) layout-informed knowledge extraction preserving structural cues, (i) domain-aware augmentation with contextual metadata, and (i) arithmetic and symbolic verification ensuring factual alignment with business rules, orchestrated in a four-stage system of retrieval, extraction, augmentation, and hybrid reasoning. We evaluate LAVA on a real-world large-scale industrial benchmark with validation rules curated by senior industry experts reflecting real regulatory conditions. Results show competitive gains in factual accuracy and symbolic correctness, together with lower computational overhead than baseline MLLM pipelines, demonstrating robustness and cost-effectiveness in realistic financial validation scenarios. Our main contributions are: • Task and System. We formalize financial document validation as a multi-document reasoning task—largely absent in existing benchmarks—and instantiate it in LAVA, a novel modular framework designed for accurate and auditable validation of semi-structured financial documents. • Reasoning Strategy & Auditability. We design a controllable hybrid reasoning framework unifying factual/contextual templates with symbolic and arithmetic tasks via explicit formula generation, with an external checker fallback ensuring correctness, thereby enhancing accuracy, interpretability, and operational robustness. • Evaluation. We propose a comprehensive evaluation framework covering symbolic correctness, factual alignment, and hallucination control for fine-grained reasoning assessment in realistic validation workflows. 2 Related Work Visually Rich Document Understanding. Recent work has shifted from extraction pipelines toward LLM-centric modeling that integrates layout and visual cues. DocLayLLM 21 adds visual patches and 2D positional tokens, VisDoM 33 combines multimodal retrieval with consistency constraints, 3MVRD 11 aligns fine- and coarse-grained signals via multi-task distillation, and LayoutLLM 12 applies instruction tuning for unified document tasks. These advances improve representation and generalization; our work is complementary, focusing on auditable validation—explicit symbolic checks, cross-field consistency, and reproducible reasoning traces required in compliance-critical workflows. Our framework is model-agnostic, plugging into any MLLM backbone to introduce validation as a controllable layer. Layout-Guided Document Encoding. Backbones such as LayoutLMv3 16, FormNet 18; 19, and DocFormer 2 combine textual, spatial, and visual cues through large-scale pre-training. These excel at form-style entity extraction but remain embedding-level and not optimized for symbolic validation. In contrast, our framework leverages layout-derived structures to enable modular business rule checks and reproducible logic tracing, addressing auditability without massive training effort. LLM Verification and Rule-based Validation. Progress in LLM factuality—via self-checking 9, retrieval-augmented prompting 24, symbolic grounding 14, and rectification 17—has improved reliability in clean text and formal reasoning 22. Yet these methods falter on noisy, heterogeneous documents with long-range dependencies and embedded business rules. Traditional rule engines (e.g., Drools) provide transparency but fail under layout variation, while hybrid LLM–rule or knowledge graph systems 34; 26 trade flexibility for interpretability. Our framework instead couples symbolic verification with layout- and metadata-informed prompting, merging rule transparency with neural adaptability, making it fit for compliance-critical validation. 3 Method Figure 1: Overview of LAVA. The architecture comprises two parallel pipelines—document processing and rule grounding—interacting via bidirectional constraints (solid arrows: main data flow; dashed arrows: auxiliary exchange). The document-processing track performs retrieval, extraction, and augmentation to produce layout-preserving structured content with enriched metadata. The rule-grounding track retrieves, classifies, and dispatches applicable validation rules to either the Symbolic Reasoner or Arithmetic Processor, depending on reasoning type. Problem Formalization. We define financial document validation as a human-in-the-loop copilot task, where the system assists users (e.g., underwriters, compliance officers, fraud analysts) in verifying whether a set of financial documents (e.g., from a mortgage application) satisfies certain business rules under real-world conditions of layout complexity, domain-specific logic, and noisy input. Inputs. The system takes as input: (1) a document set =d1,…,dND=\d_1,…,d_N\ in scanned PDF or image format, where each did_i is a semi-structured financial document (e.g., bank statement, tax form), and (2) a validation intent q, a user-specified verification goal in natural language (e.g., Does gross income exceed loan threshold?). Objective. The system maps (q,)(q,D) to a set of validation outputs =vkV=\v_k\, where each vkv_k includes: a subset of supporting documents k⊆D_k , a set of retrieved business rules ℛk⊆ℛR_k from a predefined rule library, a binary verification label yk∈Pass,Faily_k∈\ Pass, Fail\, and an explanation trace eke_k for auditability. This formulation supports multi-document, logic-grounded reasoning while ensuring modular, interpretable validation aligned with enterprise workflows. Architecture. Figure 1 illustrate LAVA architecture, which employs a modular design paradigm to address the complexity of enterprise-grade production systems. Although this design involves the integration of multiple components, it yields critical advantages for deployment. Decoupling the workflow isolates potential points of failure within individual modules, enabling targeted and independent validation and thereby systematically reducing the long-term verification cost and overall operational overhead. Furthermore, the ability to debug, update, or replace modules without systemic disruption enhances maintainability—a stark contrast to the challenges of managing opaque, end-to-end models. This design philosophy is therefore foundational to building a robust, auditable, and scalable system fit for the rigors of real-world financial validation. 3.1 Document and Validation Retrieval We jointly describe the first two modules, as they operate in a tightly coupled fashion to determine relevant document-rule pairs for downstream extraction. Given a user-specified validation intent q and a document set =d1,…,dND=\d_1,…,d_N\, the modules select a subset v⊆D_v that is temporally valid and relevant to the task, along with a set of executable business rules ℛv=r1,…,rMR_v=\r_1,…,r_M\. Both subsets are tailored to the verification goal through a bidirectional constraint mechanism, ensuring only applicable document-rule pairs are forwarded for knowledge extraction and augmentation. Document Retrieval. Documents are first preprocessed to normalize layout and correct visual artifacts (e.g., OCR errors, rotation, skew) 5, preserving alignment and structural fidelity for downstream modules. Each document did_i is then classified into a predefined document type using a lightweight image-based classifier such as TinyViT 40, as financial documents of the same type generally share consistent page-level features. Recognized types are forwarded to Validation Retrieval to constrain rule applicability. Document-level metadata (e.g., date, coverage period) is extracted using a template-guided NER pipeline with regex patterns and rule-based heuristics. Temporal constraints parsed from the validation intent in the next module (e.g., “past 3 months”) are applied to filter out documents outside the relevant time window. In addition, applicable document types extracted from retrieved rules in Validation Retrieval are passed back to further prune documents irrelevant to all candidate rules. Validation Retrieval. Given q, this module retrieves a subset of rules ℛvR_v from the predefined library ℛR, based on semantic relevance and document compatibility. The rule library is enriched with metadata specifying applicable document types. Lightweight LLMs can parse q to extract temporal constraints, while a sentence encoder (e.g., Sentence-BERT 25) encodes q to retrieve top-K semantically relevant rules. Retrieved rules are then filtered using document-type constraints from Document Retrieval, and their own document-type metadata is fed back to further refine vD_v. In conclusion, temporal and semantic cues from q filter the document set, while recognized document types and rule metadata eliminate inapplicable rules. This closed-loop filtering minimizes irrelevant candidates on both sides, reduces reasoning load, and improves accuracy without sacrificing interpretability, ensuring downstream processing operates on the most relevant and valid document-rule pairs. 3.2 Knowledge Extraction This module transforms each filtered document di∈vd_i _v into a compact, layout-aware hybrid representation for downstream reasoning. Instead of flat key–value pairs, we produce a structured markup that encodes page layout, visual grouping, and field dependencies, serving as a bridge between scanned formats and language-model-friendly input. Structured Modality Conversion. To capture the rich visual and semi-structural semantics of financial documents, we adopt an HTML-like markup constructed from parsed layout and OCR signals. Prior work shows that retaining tabular alignments, hierarchical sections, and field groupings improves reasoning fidelity 32, but raw markup is insufficient for noisy scans. We therefore augment it with: (1) structural parsing via document analysis tools (e.g., LayoutParser 27, LayoutLMv3 16), AWS Textract; (2) OCR-based recovery (e.g., Tesseract OCR 30) for free-form or scattered content, including reconstruction of long paragraphs into coherent spans; (3) proximity-based grouping to merge fragmented tokens into coherent semantic units; and (4) visual region preservation for inherently non-textual content (e.g., charts, stamps, signatures), where candidate regions are identified from layout cues (e.g., low text coverage or OCR confidence) and retained as image patches for the MLLM input. Content Filtering. The extracted representation often contains much noise from headers, footers, boilerplate blocks, or placeholders. We prune such elements using DOM structure, field labels (e.g., Name, Address), and positional cues, reducing token usage while improving attention focus for model prompts. By combining structure-preserving markup, semantic recovery, selective visual preservation, and noise reduction, this module delivers a high-fidelity hybrid form that maintains interpretability while enabling reliable downstream reasoning. 3.3 Information Augmentation This module enriches downstream reasoning by injecting auxiliary signals from both documents and retrieved rules. It serves two purposes: routing rules to the appropriate verification pathway and augmenting prompts with rich metadata to improve model understanding. Each rule rk∈ℛvr_k _v is classified as symbolic (context-dependent logic) or arithmetic (numeric computation) via a lightweight LLM query, avoiding brittle heuristics. In parallel, metadata is additionally extracted from layout-preserving outputs: language (via token-based detection), document types (from retrieval module), and domain-specific terms (e.g., withholding or CPP 11 1 CPP refers to the Canada Pension Plan, a mandatory pension contribution in Canadian payroll systems.) detected through lexical and structural heuristics targeting low-frequency tokens, abbreviations, and left-hand-side predicates in field labels or rule expressions. This metadata is distilled into concise semantic clarifications for interpretability and disambiguation. The resulting signals are incorporated into prompt headers or reference blocks, sharpening alignment with rule semantics and improving precision in complex validation scenarios, while keeping the verification loop efficient. Figure 2: Arithmetic Processor. Given a validation rule, the system generates a formula and delegates evaluation to an external calculator. A checker LLM validates alignment between the rule and result. Failures trigger a fallback loop with negative feedback. 3.4 Validation To support diverse verification needs and enhance the reliability, we adopt a bifurcated framework with two sub-modules: Arithmetic Processor for numerical tasks (e.g., verifying tax deductions, calculating gross revenue), and Symbolic Reasoner for all other general rules requiring semantic and contextual reasoning. Arithmetic Processor. As illustrated in Figure 2, when rules involving arithmetic or numerical computation are routed to this sub-module, instead of relying on LLMs for direct computation—prone to hallucinations and numeric instability, we adopt a tool-use paradigm where the model is used solely for generating a task-specific formula, which is then executed by a deterministic external engine such as a Python interpreter 13; 7. To ensure alignment between the formula and the validation semantics, we introduce a fallback auditing loop: the rule and generated formula are reviewed by a secondary LLM "checker". If a mismatch is detected, the formula is regenerated, conditioned on the previous (incorrect) version as a negative example. This loop improves robustness by systematically detecting and correcting errors, mitigating hallucinated computations and flawed reasoning pathways. Symbolic Reasoner. This sub-module handles semantic or structural reasoning. In contrast to Arithmetic Processor, it directly delegates rules to a general-purpose model, as such reasoning falls within the model’s inherent strengths. This design choice reduces integration overhead and inference latency, while remaining sufficient for a wide range of non-arithmetic verification scenarios. Prompts. We adopt our prompt construction strategy in meta-prompting fashion 43 that presents task-relevant information in a step-wise and zero-shot format rather than relying on illustrative examples. Unlike chain-of-thought 38, few-shot prompting 6, or self-consistency 36, our approach avoids brittle curated examples, which struggle to generalize across heterogeneous financial documents 44, and removes high inference cost of example-heavy prompting. Prompt Excerpt for Symbolic Reasoner Excerpt: only key parts are shown below, simplified for illustration. You are an assistant verifying document_types documents (in languages), with extracted knowledge_names … Instructions: 1. Use structured_representation as references for identifying the layout… 2. Use semantic_fields as references for explicit context… … n. Output as follows: • Only output the fields requested in the validation rule… • Format: empty = ""; date = YYYYMMDD… Example: example_output Task Input: validation_rule • term_1 : definition_1, …, term_n : definition_n Output: Return JSON only. No explanation or preamble… Figure 3: Prompt excerpt for Symbolic Reasoner, dynamically built from the retrieved validation rule, extracted knowledge, and augmentation components. Full versions for both Arithmetic Processor and Symbolic Reasoner are provided in Appendix A.1.1 and A.1.2. To ensure verification prompts are precisely tailored to both the rules and associated documents, we implement a dynamic, template-based prompt construction system. It integrates supplemental information from upstream modules with the rule to populate flexible prompt templates. By leveraging templating libraries like Jinja, we employ programmatic, code-like logic to govern the inclusion and granularity of metadata based on both document layout and task semantics. As illustrated in Figure 3, this mechanism produces prompts that are maximally informative while remaining token-efficient and robust to both arithmetic and symbolic reasoning tasks. By structuring all inputs according to rule logic, metadata, and knowledge, our approach generalizes well across heterogeneous financial documents. The resulting system remains accurate, cost-efficient and context-aware, supporting scalable and reliable verification tasks. 4 Experiments We evaluate LAVA on real-world Canadian mortgage application documents sampled from a proprietary database to assess the system within a consistent validation scenario. The set includes multiple document types and around 1,000 scanned PDFs or images—such as tax forms, bank/investment statements, and legal agreements—selected for their semi-structured formats, diverse layouts, and rich logical dependencies (see A.3.3). This focused yet heterogeneous domain enables rigorous testing of LAVA’s ability to produce accurate outputs and avoid false positives, particularly where heuristic, zero-shot, and few-shot LLM-based methods struggle with layout variability or reasoning complexity. To ensure rigorous and interpretable evaluation, we adopt rule-level testing rather than intent-level, with validation rules drawn from a curated library provided by business stakeholders. Each rule corresponds to an atomic validation unit and is applied only to the document types for which it is defined. For fairness and comparability, document type metadata is included with the dataset, since accurate validation cannot be assessed without first matching each document to its correct rule set; this ensures that all pipelines are evaluated under the same conditions, without document and validation retrieval. Ground-truth outcomes are manually annotated by domain experts, and automatic evaluation is complemented by manual audits from business collaborators, ensuring both accuracy and institutional credibility in line with production-grade expectations for document validation. 4.1 Implementations To ensure fairness, all baselines, LAVA, and LAVA’s ablation variants use Claude 3.7 Sonnet 1 as the validation component. We configure the model with a maximum response length of 5120 tokens and enable reasoning (thinking) with a budget of 1024 tokens, allowing the model to better understand complex validation rules and generalize across documents with diverse layouts and edge scenarios. For LAVA’s Arithmetic Processor, the maximum retry number n in the fallback loop is set to 22, to ensure a practical trade-off between error correction and computational efficiency. And all other LLM parameters are kept identical across evaluations. For document analysis in experiments, we prioritized a methodology that ensures our findings are portable, reproducible, and immediately accessible. To this end, our experiment exclusively utilizes off-the-shelf tools that do not require custom training or dataset-specific fine-tuning. A foundational layer of raw text and spatial information was established for the experiment using the open-source Tesseract OCR 30. For the processing of structural semantics, such as tables and forms, the publicly available AWS Textract service was employed. This consistent and transparent tooling strategy not only ensures that the comparative evaluation in Sections 4.4 and 4.5 fairly assesses the core architectural and reasoning capabilities of the different pipelines, but also directly supports other researchers to easily replicate and build upon our results. 4.2 Validation Rules We group several dozen validation rules into five categories, reflecting distinct reasoning and computation demands. These categories assess the pipeline’s ability to extract information, integrate rules, and perform contextual reasoning. See A.2 for complete versions of some representative rules. 4.3 Evaluation Metrics Our evaluation metrics comprehensively assess reasoning quality across all pipeline stages. Given the distinct characteristics of our compliance-sensitive validation task compared to common document understanding settings, we design task-specific metrics with particular emphasis on suppressing false positives—a critical industrial concern causing costly investigations, delays, and loss of trust. We evaluate baselines from three perspectives to cover these aspects. Their formal definitions are given in Appendix A.3, along with visual examples showing how each metric manifests in an illustrative sample document. Hallucination Control. We report the percentage of responses with two failure types: • Factual Hallucination Rate: Content not grounded in the source document, such as fabricated values or formulas, and unsupported claims. • Numerical Infidelity Rate: Incorrect quantitative reasoning or numerical derivations, such as incorrect formula execution results and conclusions inconsistent with the given numerical evidence. Edge Case Handling. This metric measures failure rate on complex or exception-driven scenarios, about 10%–25% of document–rule pairs. Such cases require adaptive logic reasoning beyond fixed rules and broader logic coverage for diverse edge conditions. As these challenges often overlap, we report a single rate to capture both, where higher values indicate weaker generalization and reasoning flexibility. Token Cost. We measure input and output tokens per rule check to assess computational efficiency and deployment cost, highlighting trade-offs between reasoning quality and efficiency across pipelines. For each document-rule pair, token counts are recorded and aggregated over the full evaluation set. Metric Rule Group VLM + Field-Level OCR LLM + Field-Level OCR LLM + Enhanced OCR LAVA Factual C1C_1 0.03 0.03 0.01 0.01 Hallucination C2C_2 0.08 0.10 0.04 0.03 Rate C3C_3 0.31 0.33 0.30 0.18 C4C_4 0.05 0.08 0.05 0.03 C5C_5 0.28 0.30 0.15 0.05 Numerical C1C_1 N/A N/A N/A N/A Infidelity C2C_2 0.08 0.04 0.03 0.01 Rate C3C_3 0.33 0.31 0.20 0.10 C4C_4 0.11 0.08 0.05 0.00 C5C_5 0.25 0.10 0.10 0.00 Edge Case C1C_1 N/A N/A N/A N/A Error Rate C2C_2 0.27 0.25 0.23 0.02 C3C_3 0.86 0.90 0.82 0.17 C4C_4 0.46 0.34 0.43 0.11 C5C_5 0.89 0.92 0.86 0.08 Table 1: Performance of baselines across three metrics and five validation rule categories (C1C_1–C5C_5). C1C_1: content extraction; C2C_2: conditional logic reasoning; C3C_3: multi-step logic reasoning; C4C_4: unconstrained arithmetic consistency checking; C5C_5: constrained arithmetic consistency checking. 4.4 Comparative Baselines Here, Field-Level OCR denotes OCR output containing only individual field texts and their spatial information (e.g., from bounding boxes), without higher-level structure such as tables, sections, or relational links between fields. This representation preserves raw layout signals but omits the structured markup used in LAVA’s pipeline. We consider three representative settings: • VLM + Field-Level OCR: Document images with Field-Level OCR in a VQA setup, where OCR text aids visual grounding. • LLM + Field-Level OCR: Same OCR content without images, measuring performance from textual–spatial data alone in a QA setup. • LLM + Enhanced OCR (no structural markup): OCR refined with domain-specific cleanup and recovery steps identical to LAVA’s extraction, but without structured markup, isolating LAVA’s impact of structured representation and modular reasoning. Since OCR and markup information might introduce noise (e.g., when text is misrecognized), we also examine an OCR-free configuration to assess the impact of textual knowledge. This setting is evaluated in ablation study (Section 4.5), where all other modules are preserved but the model operates only on document images. It is excluded from the baseline set because the baselines are intended to preserve textual content in some form for fair comparison, whereas this variant probes the limits of visual-only reasoning and represents a more extreme ablation of the pipeline. As shown in Table 1, LAVA achieves the lowest failure rates across all metrics and categories, with especially large gains in multi-step logic reasoning (C3C_3) and constrained arithmetic consistency checking (C5C_5), reducing hallucination and numerical errors by over 10% compared to the best baseline. And it records approximately 0% Numerical Infidelity Rate in three of four categories. LAVA also substantially lowers edge case errors, indicating strong generalization under layout ambiguity and rare conditions. Common VQA and QA settings represented by the baselines underperform in this compliance-oriented validation task due to their limited handling of layout variability and domain-specific rules. These results highlight the benefit of structured prompts and modular reasoning in suppressing unsupported content generation and enabling reliable multi-step, context-aware reasoning over structural data. We also sampled a subset of DocVQA 23 documents with five manually defined rules each, where LAVA also achieved strong performance, confirming that our framework readily transfers to public data. More importantly, real-world validation tasks—where both documents and rules are richer in semantics, context, and reasoning complexity—pose substantially greater challenges, where our approach proves particularly effective under industry-level conditions. 4.5 Ablation Study To assess the contribution of each core module in LAVA, we performed an ablation study using a unified metric: the percentage of responses that fail to exactly match the ground truth. This stricter measure was chosen because LAVA’s earlier evaluations achieved low error rates in category-specific metrics, so a single comprehensive indicator better reveals performance drops when modules are altered or removed. We evaluated six configurations: the full LAVA system; three Knowledge Extraction (KE) variants; and versions without Info Augmentation (IA) or Arithmetic Processor (AP). The KE variants progressively reduce structural detail and change input format: 1. Markdown KE replaces LAVA’s structured markup with Markdown-like fashion. 2. Plain-text KE flattens layout into text aligned to mimic visual spacing but without structural tags. 3. No KE omits textual knowledge entirely, prompting the model with original document images. This configuration serves as the OCR-free, vision-only benchmark we referred to in the baseline comparison (Section 4.4). This stepwise removal strips explicit layout and relational cues, forcing reliance on surface text or raw images. For AP ablation, AP is replaced by Symbolic Reasoner, removing explicit calculation/result validation. The two Retrieval modules are not ablated since all experiments use a fixed rule set, making them independent of validation accuracy. Pipeline C1C_1 C2C_2 C3C_3 C4C_4 C5C_5 LAVA w/o KE 0.03 0.65 0.67 0.09 0.66 LAVA w/ Plain-text KE 0.02 0.63 0.58 0.08 0.48 LAVA w/ Markdown KE 0.01 0.25 0.54 0.05 0.55 LAVA w/o IA 0.01 0.15 0.45 0.05 0.12 LAVA w/o AP 0.01 0.03 0.29 0.10 0.56 LAVA 0.01 0.03 0.28 0.05 0.07 Table 2: Percentage of responses failing to exactly match ground-truth answers across five rule categories. The error patterns in table 2 shows that KE, IA, and AP play complementary roles, with KE being the most critical. Removing KE forces the model to rely almost entirely on visual reasoning, raising C2C_2, C3C_3, and C5C_5 failure rate to 0.65–0.67—over twice LAVA’s error in multi-step logic (C3C_3) and nearly tenfold in constrained arithmetic consistency checking (C5C_5) even with the validation module’s assistance. Two factors explain this: (1) Structural conversion: without KE’s structural format that preserves complex and nested layout, grouping, and field dependencies, multi-step logic and table reasoning lose explicit relational cues. Downgrading to Markdown or plain text progressively strips away these signals, with Markdown underperforming plain text in C5C_5 due to weaker preservation of columnar alignment in tables; (2) Content filtering: without KE’s filtering stage, noisy headers, footers, and irrelevant fields dilute attention and amplify hallucination risk. This gradient from fully structured to none shows that noise suppression, explicit spatial and relational cues are essential for stable reasoning, especially in multi-field and arithmetic-heavy rules for financial document validation tasks. IA mainly impacts logical reasoning: removing it raises C3C_3 error from 0.28 to 0.45, suggesting that metadata and domain cues help models resolve field semantics and linking conditions across steps. AP’s effect is computation-focused: removing it leaves C1C_1–C3C_3 unchanged but drives C4C_4/C5C_5 error to 0.10/0.56, reflecting the limits of unconstrained LLM reasoning in arithmetic tasks and the benefit of explicit calculation with fallback mechanism. Overall, the degradations are consistent with the design intent: KE supplies the structured substrate, IA enriches it with semantic context, and AP enforces numerical fidelity, justifying LAVA’s functionally complementary and modular composition. 4.6 Business Impact (a) LAVA and baselines. (b) LAVA and ablated variants. Figure 4: Average input and output token counts across five rule categories. While Section 4.4 and 4.5 highlight LAVA’s technical advantages in accuracy and robustness, real-world adoption also depends on its operational efficiency and cost-effectiveness. By consolidating token usage, we can further assess how LAVA scales under realistic operational constraints, using token consumption as a proxy for latency and API expenditure in high-volume production workflows. LAVA amortizes multi-step costs by extracting structured knowledge once and reusing it across rules. As shown in Figure 4(a), this design cuts input tokens by 25%–45% versus VLM and LLM baselines with Field-Level OCR, lowering inference costs and enabling faster turnaround for time-critical financial validation processes, while also achieving fewer errors (Table 1). Ablation results (Figure 4(b)) show that in non-computation tasks (C1C_1–C3C_3), excluding variant w/o KE, LAVA adds minimal tokens with stable output length due to the predefined schema-based response format. For computation tasks (C4C_4–C5C_5), the less faithful structural representation from plain-text and Markdown KE inflate inputs by causing more model output errors and retries in the validation loop. Across all ablations, full LAVA adds under 3k tokens per rule (∼$0.009 \$0.009 with Claude 3.7 or ∼$0.006 \$0.006 with GPT-4.1), an overhead outweighed by substantial gains in accuracy and reasoning stability. This demonstrates that LAVA delivers a dual advantage: (1) immediate computational savings from reduced token usage, and (2) long-term operational efficiency from its modular, auditable design. This combination suggests that LAVA’s architectural benefits outweigh the modular integration complexity, making it a cost-effective and scalable choice for high-volume, time-critical real-world financial workflows. 5 Conclusion We presented LAVA, a modular, interpretable and backbone- and domain-agnostic system for enterprise-grade financial document validation. Combining layout-preserving extraction, domain-aware augmentation, and symbolic and arithmetic verification, LAVA achieves high factual alignment and numerical reliability on a large real-world benchmark, with lower computational overhead than monolithic prompting. Evaluation with factual hallucination rate, numerical infidelity rate, and edge case handling shows that structured, multi-stage reasoning enables fine-grained error attribution, stronger robustness in compliance-sensitive workflows, and transfer to structurally similar data. Looking ahead, we aim to handle noisier, less-structured documents and to learn rule generalization across formats and domains. 6 Ethical Considerations This research was conducted using a proprietary dataset of financial documents, with all data fully anonymized and handled under strict institutional privacy protocols. We acknowledge the potential risk of algorithmic bias in financial decision-making. LAVA’s design as an accurate, scalable and auditable system is a deliberate step to mitigate this risk by enhancing transparency and ensuring human accountability. Nevertheless, any production deployment would necessitate rigorous, ongoing audits for demographic bias to ensure fair and equitable outcomes. References Anthropic (2024) Anthropic Claude 3 model family. Note: https://w.anthropic.com/news/claude-3-familyAccessed: 2025-08-07 Cited by: §4.1. Appalaraju et al. (2021) S. Appalaraju, B. Jasani, B. U. Kota, Y. Xie, and R. Manmatha DocFormer: end-to-end transformer for document understanding. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, p. 973–983. External Links: Document, Link Cited by: §1, §2. Bhattacharyya et al. (2025) A. Bhattacharyya, A. Tripathi, U. Das, A. Karmakar, A. Pathak, and M. Gupta Information extraction from visually rich documents using LLM-based organization of documents into independent textual segments. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 17241–17256. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §1. Borchmann et al. (2021) Ł. Borchmann, M. Pietruszka, T. Stanislawek, D. Jurkiewicz, M. Turski, K. Szyndler, and F. Graliński Due: end-to-end document understanding benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), Cited by: §1. Boudraa et al. (2020) O. Boudraa, W. K. Hidouci, and D. Michelucci Using skeleton and hough transform variant to correct skew in historical documents. Mathematics and computers in simulation 167, p. 389–403. Cited by: §3.1. Brown et al. (2020) T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), External Links: Link Cited by: §3.4. Chen et al. (2022) W. Chen, X. Ma, X. Wang, and W. W. Cohen Program of thoughts prompting: disentangling computation from reasoning for numerical reasoning tasks. Vol. abs/2211.12588. External Links: Link Cited by: §3.4. Chen et al. (2024) Y. Chen, J. Zhang, K. Peng, J. Zheng, R. Liu, P. Torr, and R. Stiefelhagen RoDLA: benchmarking the robustness of document layout analysis models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, p. 15556–15566. External Links: Document, Link Cited by: §1. Dhuliawala et al. (2024) S. Dhuliawala, M. Komeili, J. Xu, R. Raileanu, X. Li, A. Celikyilmaz, and J. Weston Chain-of-verification reduces hallucination in large language models. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 3563–3578. External Links: Link, Document Cited by: §2. Ding et al. (2024a) Y. Ding, J. Lee, and S. C. Han Deep learning based visually rich document content understanding: a survey. ArXiv preprint abs/2408.01287. External Links: Link Cited by: §1. Ding et al. (2024b) Y. Ding, L. Vaiani, C. Han, J. Lee, P. Garza, J. Poon, and L. Cagliero 3MVRD: multimodal multi-task multi-teacher visually-rich form document understanding. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 15233–15244. External Links: Document, Link Cited by: §2. Fujitake (2024) M. Fujitake LayoutLLM: large language model instruction tuning for visually rich document understanding. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), N. Calzolari, M. Kan, V. Hoste, A. Lenci, S. Sakti, and N. Xue (Eds.), Torino, Italia, p. 10219–10224. External Links: Link Cited by: §2. Gao et al. (2023) L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig PAL: program-aided language models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, p. 10764–10799. External Links: Link Cited by: §3.4. Hennigen et al. (2024) L. T. Hennigen, S. Shen, A. Nrusimha, B. Gapp, D. Sontag, and Y. Kim Towards verifiable text generation with symbolic references. External Links: 2311.09188, Link Cited by: §2. Hu et al. (2025) A. Hu, H. Xu, L. Zhang, J. Ye, M. Yan, J. Zhang, Q. Jin, F. Huang, and J. Zhou MPLUG-DocOwl2: high-resolution compressing for OCR-free multi-page document understanding. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 5817–5834. External Links: Document, ISBN 979-8-89176-251-0, Link Cited by: §1. Huang et al. (2022) Y. Huang, T. Lv, L. Cui, Y. Lu, and F. Wei LayoutLMv3: pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM International Conference on Multimedia, M ’22, New York, NY, USA, p. 4083–4091. External Links: ISBN 9781450392037, Link, Document Cited by: §2, §3.2. Kang et al. (2024) H. Kang, J. Ni, and H. Yao Ever: mitigating hallucination in large language models through real-time verification and rectification. External Links: 2311.09114, Link Cited by: §2. Lee et al. (2022) C. Lee, C. Li, T. Dozat, V. Perot, G. Su, N. Hua, J. Ainslie, R. Wang, Y. Fujii, and T. Pfister FormNet: structural encoding beyond sequential modeling in form document information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), S. Muresan, P. Nakov, and A. Villavicencio (Eds.), Dublin, Ireland, p. 3735–3754. External Links: Document, Link Cited by: §2. Lee et al. (2023) C. Lee, C. Li, H. Zhang, T. Dozat, V. Perot, G. Su, X. Zhang, K. Sohn, N. Glushnev, R. Wang, J. Ainslie, S. Long, S. Qin, Y. Fujii, N. Hua, and T. Pfister FormNetV2: multimodal graph contrastive learning for form document information extraction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, p. 9011–9026. External Links: Document, Link Cited by: §2. Li et al. (2025) S. Li, Y. Shen, X. Chen, J. Chen, H. Ju, H. Duan, S. Mao, H. Zhou, B. Zhang, B. Fu, P. Cai, L. Wen, B. Shi, Y. Liu, X. Cai, and Y. Qiao GDI-bench: a benchmark for general document intelligence with vision and reasoning decoupling. Vol. abs/2505.00063. External Links: Link Cited by: §1. Liao et al. (2025) W. Liao, J. Wang, H. Li, C. Wang, J. Huang, and L. Jin DocLayLLM: an efficient multi-modal extension of large language models for text-rich document understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 4038–4049. Cited by: §2. Liu et al. (2025) C. Liu, Y. Yuan, Y. Yin, Y. Xu, X. Xu, Z. Chen, Y. Wang, L. Shang, Q. Liu, and M. Zhang Safe: enhancing mathematical reasoning in large language models via retrospective step-aware formal verification. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 12171–12186. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §2. Mathew et al. (2021) M. Mathew, D. Karatzas, and C. Jawahar DocVQA: a dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, p. 2200–2209. Cited by: §1, §4.4. Qin et al. (2025) Y. Qin, S. Li, Y. Nian, X. V. Yu, Y. Zhao, and X. Ma Don’t let it hallucinate: premise verification via retrieval-augmented logical reasoning. Vol. abs/2504.06438. External Links: Link Cited by: §2. Reimers and Gurevych (2019) N. Reimers and I. Gurevych Sentence-BERT: sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan (Eds.), Hong Kong, China, p. 3982–3992. External Links: Document, Link Cited by: §3.1. Sadowski and Chudziak (2025) A. Sadowski and J. A. Chudziak Explainable rule application via structured prompting: a neural-symbolic approach. External Links: 2506.16335, Link Cited by: §2. Shen et al. (2021) Z. Shen, R. Zhang, M. Dell, B. C. G. Lee, J. Carlson, and W. Li LayoutParser: a unified toolkit for deep learning based document image analysis. In ICDAR, Cited by: §3.2. Shende et al. (2024) A. Shende, M. Mullapudi, and N. Challa Enhancing document verification systems: a review of techniques, challenges, and practical implementations. International Journal of Computer Engineering & Technology 15, p. 16–25. Cited by: §1. Šimsa et al. (2023) Š. Šimsa, M. Šulc, M. Uřičář, Y. Patel, A. Hamdi, M. Kocián, M. Skalickỳ, J. Matas, A. Doucet, M. Coustaty, et al. DocILE benchmark for document information localization and extraction. In International Conference on Document Analysis and Recognition, p. 147–166. Cited by: §1. Smith (2007) R. Smith An overview of the tesseract ocr engine. In ICDAR ’07: Proceedings of the Ninth International Conference on Document Analysis and Recognition, Washington, DC, USA, p. 629–633. External Links: ISBN 0-7695-2822-8, Link Cited by: §3.2, §4.1. Stanisławek et al. (2021) T. Stanisławek, F. Graliński, A. Wróblewska, D. Lipiński, A. Kaliska, P. Rosalska, B. Topolski, and P. Biecek Kleister: key information extraction datasets involving long documents with complex layouts. In Document Analysis and Recognition – ICDAR 2021, p. 564–579. External Links: Document, ISBN 9783030865498, ISSN 1611-3349, Link Cited by: §1. Sui et al. (2024) Y. Sui, M. Zhou, M. Zhou, S. Han, and D. Zhang Table meets llm: can large language models understand structured table data? a benchmark and empirical study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, p. 645–654. Cited by: §3.2. Suri et al. (2025) M. Suri, P. Mathur, F. Dernoncourt, K. Goswami, R. A. Rossi, and D. Manocha VisDoM: multi-document QA with visually rich elements using multimodal retrieval-augmented generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, p. 6088–6109. External Links: Document, ISBN 979-8-89176-189-6, Link Cited by: §2. Vertsel and Rumiantsau (2024) A. Vertsel and M. Rumiantsau Hybrid llm/rule-based approaches to business insights generation from structured data. External Links: 2404.15604, Link Cited by: §2. Wang et al. (2024) D. Wang, N. Raman, M. Sibue, Z. Ma, P. Babkin, S. Kaur, Y. Pei, A. Nourbakhsh, and X. Liu DocLLM: a layout-aware generative language model for multimodal document understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 8529–8548. External Links: Document, Link Cited by: §1. Wang et al. (2023a) X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, External Links: Link Cited by: §3.4. Wang et al. (2023b) Z. Wang, Y. Zhou, W. Wei, C. Lee, and S. Tata VRDU: A benchmark for visually-rich document understanding. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023, A. K. Singh, Y. Sun, L. Akoglu, D. Gunopulos, X. Yan, R. Kumar, F. Ozcan, and J. Ye (Eds.), p. 5184–5193. External Links: Document, Link Cited by: §1. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), External Links: Link Cited by: §3.4. Wu et al. (2023) A. Wu, L. Xiao, X. Wu, S. Yang, J. Xu, Z. Zhuang, N. Xie, C. Jin, and L. He DCQA: document-level chart question answering towards complex reasoning and common-sense understanding. Vol. abs/2310.18983. External Links: Link Cited by: §1. Wu et al. (2022) K. Wu, J. Zhang, H. Peng, M. Liu, B. Xiao, J. Fu, and L. Yuan Tinyvit: fast pretraining distillation for small vision transformers. In European conference on computer vision, p. 68–85. Cited by: §3.1. Xu et al. (2020) Y. Xu, M. Li, L. Cui, S. Huang, F. Wei, and M. Zhou LayoutLM: pre-training of text and layout for document image understanding. In KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, R. Gupta, Y. Liu, J. Tang, and B. A. Prakash (Eds.), p. 1192–1200. External Links: Link Cited by: §1, §1. Zhang et al. (2024) C. Zhang, Y. Tu, Y. Zhao, C. Yuan, H. Chen, Y. Zhang, M. Chai, Y. Guo, H. Zhu, Q. Zhang, and T. Gui Modeling layout reading order as ordering relations for visually-rich document understanding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 9658–9678. External Links: Link, Document Cited by: §1. Zhang et al. (2023) Y. Zhang, Y. Yuan, and A. C. Yao Meta prompting for ai systems. Vol. abs/2311.11482. External Links: Link Cited by: §3.4. Zhou et al. (2023) D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. V. Le, and E. H. Chi Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, External Links: Link Cited by: §3.4. Zhu et al. (2024) F. Zhu, Z. Liu, X. Y. Ng, H. Wu, W. Wang, F. Feng, C. Wang, H. Luan, and T. S. Chua MMDocBench: benchmarking large vision-language models for fine-grained visual document understanding. Vol. abs/2410.21311. External Links: Link Cited by: §1. Appendix A Appendix A.1 Prompt used in LAVA A.1.1 Prompt for Symbolic Reasoner Dynamically Generated Prompt You are an assistant who verifies financial documents. You are given document_types (in languages) and extracted knowledge_names. Your job is to verify the documents based on the validation rule. Please do not make up any values. <instructions> 1. Use structured representation as references for identifying the layout and structure of the documents. <html_tables>knowledge[tables]</html_tables> <forms>knowledge[forms]</forms> … 2. Use the semantic fields as references for providing explicit contextual information. <semantic_fields>knowledge[fields]</semantic_fields> … n. Output as follows: • Only output the fields requested in the validation rule, do not include any other fields stated in example_output. • Replace all spaces in keys with underscores "_". • If any field is not present, set value to "". • Output date in YYYYMMDD format. <example_output>example_output</example_output> </instructions> <validation_rule> validation_rule • domain_specific_term_1 : explanation_1 … • domain_specific_term_n : explanation_n </validation_rule> <output_format> Return solely the JSON object without any additional explanations, comments, preamble, or formatting like backticks. </output_format> A.1.2 Prompt for Arithmetic Processor Prompt for Formula Generation System Prompt: You are an assistant who generates a Python-style calculation formula to explain how a request field should be computed. You are given document_types (in languages), as well as knowledge extracted from the documents and a validation rule. <instructions> 1. Identify the requested field from the rule. Store its name in the JSON object with the key "field_name", delete any special symbols and replace all spaces in keys with underscores "_". 2. Extract the stated value for the requested field. • Store it in the JSON object with the key "stated" and make sure it is a valid float or integer. • If the requested field is not present or empty, set "stated" to "NaN". • Do not make up any values. 3. Identify the relevant numerical fields in the knowledge that are needed to compute the requested field. Consider how those fields logically combine to produce the value of the requested field. 4. Once you find an appropriate calculation expression, store it in Python-executable format in the JSON object with the key "formula": • Do not compute the result. • Do not include any functions like "round(…)" or similar. • Only use raw numerical operations. 5. Return solely the JSON object without any additional explanations, comments, preamble, or formatting like backticks. </instructions> <example_output> "field_name": "Current_Total_Income", "stated": 600, "formula": "100 + 200 + 300" </example_output> User Prompt: <knowledge>knowledge</knowledge> <validation_rule> validation_rule • domain_specific_term_1 : explanation_1 … • domain_specific_term_n : explanation_n </validation_rule> Prompt for Formula Correction System Prompt: You are the greatest financial auditor, logician and deducer. You are given document_types (in languages), as well as knowledge extracted from the documents, a validation rule, and a calculation expression. <instructions> 1. The calculation expression in response to the validation rule is wrong based on the knowledge extracted. 2. Check if there is any other correct calculation expression that can be derived from the data given. 3. Once you find an appropriate calculation expression, store it in Python-executable format: • Do not compute the result using equal sign. • Do not include any functions like "round(…)" or similar. • Only use raw numerical operations. 4. Return solely the calculation expression without any additional explanations, comments, preamble, or formatting like backticks. </instructions> <example_output_1>100 + 200 + 300</example_output_1> <example_output_2>80 * 100.5</example_output_2> User Prompt: <knowledge>knowledge</knowledge> <wrong_calculation>wrong_calculation<wrong_calculation> <validation_rule> validation_rule • domain_specific_term_1 : explanation_1 … • domain_specific_term_n : explanation_n </validation_rule> A.2 Representative Validation Rules and Output Examples Used in Experiment 1. Content Extraction Rule: (a) Are Current and YTD regular pay/salary amounts present in documents? (b) Is Current and YTD CPP/QPP (may appear as Government Pension) present in the paystub? (c) Is Social Insurance Number (SIN) present in the T4 document? Output Example "Employer_Name": "present": true, "value": "organization", "Current_Regular_Pay": "present": false, "value": "", 2. Conditional Logic Reasoning (a) Rule: <CRA_EI_data>EI_data</CRA_EI_data> <EI_verification_rules> • When Current EI is non-zero, skip the check and set "valid" to true. • When Current and YTD EI are both 0 or blank, skip the check and set "valid" to false. • When Current EI is 0 or blank, YTD EI is not 0 or blank. If YTD EI ≤ EI cap, set "valid" to true. Otherwise, set "valid" to false. </EI_verification_rules> Do the EI values comply with rules based on the Canada Revenue Agency (CRA) guidelines? Output Example "Pay_Date": "20241128", "EI": "Current": "100", "YTD": "200", "Cap": "1049.12", "valid": true (b) Rule: <numerical_verification_rules> • Cells with empty value, or only numbers, symbols, punctuation are valid. • Cells containing words (alphabetic characters) are invalid. • If all fields are valid, return an empty object . <numerical_verification_rules> Based on the rules provided, does every non-header cell in tables contain a valid numerical value? Output Example "This_Period_Regular": "amount", "YTD_CPP": "value" 3. Multi-step Logic Reasoning Rule: <pay_freq_cap> [[ "freq": "monthly", "cap": 12, "freq": "semimonthly", "cap": 24, "freq": "biweekly", "cap": 26, "freq": "weekly", "cap": 52 ]] </pay_freq_cap> <freq_verification_rules> • If start or end date is empty, set "stated" to "" and set "equal" to None. • If No. of pay period is empty, set "stated" to "" and set "equal" to None. • If No. of pay period is not empty, find its corresponding "frequency cap" from pay frequency rules: – Compute the expected "current period number": Use "Pay Frequency", "Start Date" and "frequency cap" to calculate how many periods have elapsed since the start of the year. – Extract current period number from No. of pay period stated (e.g., 22 on "22 of 26"). – Check if the "stated" current period number" equals the "expected current period number". If both equal, set "equal" to true. Otherwise, set "equal" to false. </freq_verification_rules> Based on the rules provided, does pay frequency and No. of pay period in the extracted date comply with the pay frequency rules? Output Example "Start_Date": "20240101", "End_Date": "20240131", "Pay_Frequency": "stated": "semimonthly", "expected": "monthly", "equal": false, "NO_Pay_Period": "stated": "11", "expected": "1", "equal": false 4. Unconstrained Arithmetic Consistency Checking Rule: (a) Is the YTD gross pay calculated correctly by summing up all earning items? (b) Is Line 23600 (Net Income) correctly calculated as Line 15000 (Total Income) minus Deductions from Total Income? (c) Is the ending balance correctly calculated as the starting balance plus total deposits minus total withdrawals for the period? Output Example "Current_Gross_Pay": "calculation": "100 + 200 + 300 = 600", "stated": "300", "equal": true, "YTD_Gross_Pay": "calculation": "1500 + 200 + 300 = 2000", "stated": "", "equal": null 5. Constrained Arithmetic Consistency Checking Rule: (a) Is the Current regular pay calculated correctly by rate * hours when rounding the result to 2 decimal places? Set formula to an empty string"" when rate or hours is not present or 0. (b) Is the current net pay correctly computed as the current gross pay minus all current deductions, with the result rounded to 2 decimal places? If the table includes a "Total Deduction" field, use it directly instead of summing individual deduction items. Output Example "Current_Gross_Pay": "calculation": "100 + 200 + 300 = 600", "stated": "300", "equal": true, "YTD_Gross_Pay": "calculation": "1500 + 200 + 300 = 2000", "stated": "", "equal": null A.3 Evaluation Metrics A.3.1 Factual Hallucination Rate (FHR). Given a fixed rule r with evaluation document set r=d1,…,dnrD_r=\d_1,…,d_n_r\, let ℰ^i,r E_i,r denote the predicted evidence subset in the model’s output explanation trace for (di,r)(d_i,r), and ℰ⁡(di,r)E(d_i,r) the gold evidence set from did_i. Let φ^i,r _i,r denote the predicted formula for (di,r)(d_i,r), and grg_r the corresponding gold formula. We mark a sample (di,r)(d_i,r) as factually hallucinated if either hold: (a) Evidence Hallucination: ℰ^i,r⊈ℰ⁡(di,r) E_i,r (d_i,r). (b) Formula Hallucination: φ^i,r≢gr _i,r ≡ g_r (tested via polynomial identity testing with Schwartz–Zippel). Formally, the sample-level indicator is FHi,r=[ _i,r=1\! [ (ℰ^i,r⊈ℰ⁡(di,r))∨ ( E_i,r (d_i,r))\;\; (1) (φ^i,r≢gr)]. ( _i,r ≡ g_r) ]. The rule-level rate is FHRr=1|r|∑di∈rFHi,r,FHR_r= 1|D_r| _d_i _rFH_i,r, (2) and the group-level average for C⊆ℛC is FHRCw=∑r∈C|r|⋅FHRr∑r∈C|r|.FHR^w_C= _r∈ C|D_r|·FHR_r _r∈ C|D_r|. (3) Containment checks normalize case, punctuation, numeric formatting, and unit conventions. Formula equivalence (≡) is evaluated by polynomial identity testing under randomized substitution. Figure 5: Sample document shown for clarity and error illustrations; metrics are document-agnostic and applied unchanged across all document types (sensitive content redacted). Example. Consider the rule “Is the YTD total deduction computed as the sum of all YTD items in the deduction table?” The pipeline is asked to extract relevant evidence, provide a calculation, and conclude the validation result for this task. Based on Figure 5, the ground truth is: Ground Truth Output "YTD_Total_Deduction": "calculation": "1427.28 + 4683.88 + 344.28 + 660.50 + 617.50 = 7733.44", "stated": "7633.44", "equal": false When factual hallucination occurs, the pipeline fabricates one or some of the variables in the calculation formula (e.g., changing 660.50660.50 to 560.50560.50) so that the sum exactly matches the stated total: Output with Factual Hallucination "YTD_Total_Deduction": "calculation": "1427.28 + 4683.88 + 344.28 + 560.50 + 617.50 = 7633.44", "stated": "7633.44", "equal": true , Consider the rule: “Are pay period start and end dates present in the document?” The pipeline is asked to check the presence of specific fields and extract the corresponding value. The ground truth for Figure 5 is: Ground Truth Output "Start_Date": "present": false, "value": "" "End_Date": "present": true, "value": "20111015" When factual hallucination occurs, the pipeline fabricates a start date by using the value of another field (in this case, the payment date) and incorrectly marking it as present as a consequence: Output with Factual Hallucination "Start_Date": "present": true, "value": "20111014" "End_Date": "present": true, "value": "20111015" These two examples illustrate distinct modes of factual hallucination. In the deduction case, the pipeline invents a new numerical value so that the arithmetic matches the stated total. In the date case, the pipeline fabricates a missing field by repurposing another field. In both scenarios, the error arises not from incorrect arithmetic or logical reasoning but from introducing evidence that is absent from the source document. A.3.2 Numerical Infidelity Rate (NIR). Given a fixed rule r with evaluation set r=d1,…,dnrD_r=\d_1,…,d_n_r\, let φ^i,r _i,r denote the predicted formula or reasoning process appearing in the model’s output explanation trace for (di,r)(d_i,r), and vi,r⋆v _i,r the gold numerical result. We mark a sample (di,r)(d_i,r) as numerically infidel if the evaluated result of the predicted formula or reasoning process deviates from the gold value: NIi,r=[eval(φ^i,r)≠vi,r⋆],NI_i,r=1\! [eval( _i,r)≠ v _i,r ], (4) with deviations judged within absolute/relative tolerances. The rule-level rate is NIRr=1|r|∑di∈rNIi,r,NIR_r= 1|D_r| _d_i _rNI_i,r, (5) and the group-level average for C⊆ℛC is NIRCw=∑r∈C|r|⋅NIRr∑r∈C|r|.NIR^w_C= _r∈ C|D_r|·NIR_r _r∈ C|D_r|. (6) Note that hallucinations in formula generation are already captured under FHR; NIR isolates purely numerical inconsistencies after formula or reasoning process generation. Example. Using the same rule for the YTD deduction value as in A.3.1: Output with Numerical Infidelity "YTD_Total_Deduction": "calculation": "1427.28 + 4683.88 + 344.28 + 660.50 + 617.50 = 7633.44", "stated": "7633.44", "equal": true , Here, all evidence terms are faithfully extracted, and the pipeline correctly generates the formula. But the result is miscomputed as 7633.447633.44 instead of the correct 7733.447733.44, possibly influenced by the stated value in the document. Unlike factual hallucination, this error does not arise from fabricating or misattributing values, but from applying the correct evidence while failing to maintain consistency in quantitative reasoning and arithmetic derivations. A.3.3 Edge Case Handling (ECH). For each rule r with evaluation set rD_r, we predefine a subset redge⊆rD^edge_r _r containing complex or exception-driven cases (typically 10%–25% of document–rule pairs). Such cases include missing or abnormal values, atypical field combinations, and boundary conditions requiring adaptive reasoning beyond commom patterns. For a document di∈redged_i ^edge_r, let y^i,r∈Pass,Fail y_i,r∈\ Pass, Fail\ be the model’s predicted label and yi,r⋆y _i,r the gold label. We define the indicator ECi,r=1,y^i,r≠yi,r⋆,0,y^i,r=yi,r⋆.EC_i,r= cases1,& y_i,r≠ y _i,r,\\[4.0pt] 0,& y_i,r=y _i,r. cases (7) The rule-level error rate is ECHr=1|redge|∑di∈redgeECi,r,ECH_r= 1|D^edge_r| _d_i ^edge_rEC_i,r, (8) and the group-level average is ECHCw=∑r∈C|redge|⋅ECHr∑r∈C|redge|.ECH^w_C= _r∈ C|D^edge_r|·ECH_r _r∈ C|D^edge_r|. (9) Higher values of ECHECH indicate weaker generalization and lower robustness on edge conditions. Example. Consider the rule: “Is the current gross pay calculated correctly by summing up the current amount of all earning items?” In the sample (Figure 5), the earnings table contains a subtotal of 1663.441663.44 followed by a negative adjustment of 38.4438.44. Correct handling requires the pipeline to exclude the subtotal from the formula and include the adjustment as a subtraction. The ground truth is: Ground Truth Output "Current_Gross_Pay": "calculation": "1625.00 + 32.50 + 5.94 - 38.44 = 1625.00", "stated": "1625.00", "equal": true However, the pipeline misinterprets the table: Output with Edge Case Error "Current_Gross_Pay": "calculation": "1625.00 + 32.50 + 5.94 + 1663.44 + 38.44 = 3365.32", "stated": "1625.00", "equal": false , In this output, the pipeline incorrectly reuses the subtotal 1663.441663.44 as if it were another earning item, and also flips the negative adjustment 38.4438.44 into a positive contribution. The error is not due to arithmetic mistakes but to misinterpreting atypical structures in the table. Unlike factual hallucination (fabricating values) or numerical infidelity (miscomputing a correct formula), this case reflects weaker robustness to edge conditions. Instead of applying rules mechanically, a reliable industrial pipeline should capture the underlying business logic and correctly adapt it to the diverse formats and exception cases present in financial documents. A.4 Evaluation Dataset Our evaluation uses a proprietary collection of production-level mortgage application documents from an active industrial workflow. The corpus spans a broad range of core and supporting document categories—covering proof of income, property appraisal, account statements, tax forms, and legal agreements—with file lengths varying from single-page forms to multi-dozen-page reports. Unlike public datasets that are often synthetic or visually uniform, our corpus retains the full heterogeneity, layout irregularities, OCR noise, and compliance constraints encountered in real underwriting. While individual files and categories vary across evaluations, the overall data composition and associated challenges are fully described, enabling reproducibility on comparable financial document collections. Table 3: Representative mortgage document types and key characteristics. Document Type Key Structural Features Key Semantic Features Valid Rule Categories Appraisal Multi-page PDF; photos + valuation tables; firm templates; checkboxes Market value; comparable properties; adjustments; effective date C1C_1, C2C_2 Bank Account Statement Multi-column ledgers; footnotes inside tables; embedded logos and stamps Balance progression; transaction category; cross-month reconciliation C1C_1 – C5C_5 Employment Letter Letterhead; signatures; uneven paragraphs; low-contrast scans Title; start date; compensation; pay frequency vs. paystubs C1C_1, C2C_2, C3C_3 Investment Account Statement Dense holdings tables; various formats; small-font disclosures and disclaimers Issue/maturity date; interest rate; holding list; balance C1C_1, C3C_3, C4C_4, C5C_5 Mortgage Statement Multi-section layout; various formats; tables with merged cells Outstanding balance; interest rate and type; payment due date; amount due C1C_1, C3C_3, C4C_4, C5C_5 Notice of Assessment CRA form; dense numeric blocks; tiny labels; presence of official headers Total/taxable income; refund amount; social insurance number; match T4 C1C_1, C2C_2, C3C_3, C4C_4 Paystub Multiple tables and forms; various layouts; faded scans Gross; net; deductions; start/end/pay date C1C_1 - C5C_5 Personal Income Tax Return Multipage; mixed sections; checkboxes; pre-filled and handwritten; cross-page linkage Address; citizenship; total/employment/net income C1C_1, C2C_2, C4C_4 Property Tax Bill Multiple tables; various layouts; inconsistent labelling for tax components; watermarks and logos Annual tax; installments/penalties; roll/assessment IDs; schedule math C1C_1 - C5C_5 Purchase Sales Agreement Long contract; initials/signatures; clause order varies; appendices with separate numbering Property address; price; closing date; buyer/seller name C1C_1, C2C_2, C3C_3 Realtor Listing Embedded images; dense tables and forms; inconsistent field ordering List price; property address; listing date and status; square footage C1C_1, C2C_2 Realtor Listing UW Listing + UW notes; handwritten overlays Adjusted price; remarks; reconcile with appraisal/APS C1C_1, C2C_2, C3C_3 Rental Lease Agreement Multi-page legal contract; various templates; small-font clauses; handwritten Monthly rent; term dates; payment plan; residence overlap C1C_1, C2C_2, C3C_3 Separation Agreement Multi-page legal document; dense narrative clauses; annex schedules Support obligations; asset division; enforceable clauses; child custody C1C_1, C2C_2, C3C_3 T4: Statement of Remuneration Paid Fixed CRA box layout; small-font numeric fields; bilingual field labels; scan noise Employment income; Province; EI/CPP exemptions; align paystub/NOA C1C_1, C2C_2, C4C_4 T4A: Statement of Pension, Retirement, Annuity, and Other Income Fixed CRA box layout; small-font numeric fields; bilingual field labels; scan noise Pension/annuity/self-employment income; income tax deducted; social insurance number C1C_1, C2C_2, C4C_4 T4A(P): Statement of Canada Pension Plan Benefits Fixed CRA box layout; small-font numeric fields; bilingual field labels; scan noise CPP benefits amount; Income tax deducted; C1C_1, C2C_2, C4C_4 T4RIF: Statement of Income From a Registered Retirement Income Fund Fixed CRA box layout; small-font numeric fields; bilingual field labels; scan noise Taxable amounts; Payer/issuer name; taxable amount; social insurance number C1C_1, C2C_2, C4C_4 T5: Statement of Investment Income Fixed CRA box layout; small-font numeric fields; bilingual field labels; scan noise Actual amount of dividends; Interest from Canadian sources; Reported income vs. bank statements C1C_1, C2C_2, C5C_5 T776: Statement of Real Estate Rentals Fixed CRA box layout; small-font numeric fields; bilingual field labels; scan noise Gross rents; Total expenses; Net income (loss) before adjustments; Capital cost allowance (CCA) claim C1C_1 - C5C_5

Instant research discovery