Paper deep dive
GUIDE: Governed Unified Intelligence for Document-to-Artifact Generation in Enterprise Settings
Shivali Dalmia, Sumukha Thoppanahalli, Mohammadreza Sediqin, Abhishek Mukherji
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Enterprise guideline documents are heterogeneous and multimodal, combining narrative text, complex tables, and embedded images. Existing LLM and VLM systems face hallucinated content, table structure degradation, and lack governed workflows extending beyond extraction to validation and artifact generation. This leaves enterprises to perform this manually, consuming 2-3 days per document. To address this, we introduce GUIDE, a governed multi-agent framework built on a shared versioned rule store with schema-validated inter-agent contracts and end-to-end provenance tracking. Six specialized agents handle parsing, VLM-driven extraction, consistency checking, evaluation, human-in-the-loop (HITL) escalation, and persona-tailored artifact synthesis. Evaluated on 120 real-world enterprise guideline documents, GUIDE achieves 96% document success, extracts 3,896 rules with 71.4% auto-approved, produces 812 deployment-ready artifacts, and reduces turnaround to 40-125 minutes per document.
Tags
Links
- Source: https://arxiv.org/abs/2608.12133v1
- Canonical: https://arxiv.org/abs/2608.12133v1
Trouble viewing inline? Open PDF directly ā
Full Text
22,732 characters extracted from source content.
Expand or collapse full text
GUIDE: Governed Unified Intelligence for Document-to-Artifact Generation in Enterprise SettingsCCS: Computing methodologies Natural language processingCCS: Computing methodologies Machine learningCCS: Information systems Information extraction Shivali Dalmia Note: Equal contribution Affiliation: Centific Research , Washington , USA email: shivali.dalmia@centific.com , Sumukha Thoppanahalli Affiliation: Centific Research , Washington , USA email: sumukhasharma.t@centific.com , Mohammadreza Sediqin Affiliation: Centific Research , New York , USA email: mohammadreza.s@centific.com and Abhishek Mukherji Affiliation: Centific Research , Washington , USA email: abhishek.mukherji@centific.com Abstract. Enterprise guideline documents are heterogeneous and multimodal, combining narrative text, complex tables, and embedded images. Existing LLM and VLM systems face hallucinated content, table structure degradation, and lack governed workflows extending beyond extraction to validation and artifact generation. This leaves enterprises to perform this manually, consuming 2ā3 days per document. To address this, we introduce GUIDE, a governed multi-agent framework built on a shared versioned rule store with schema-validated inter-agent contracts and end-to-end provenance tracking. Six specialized agents handle parsing, VLM-driven extraction, consistency checking, evaluation, human-in-the-loop (HITL) escalation, and persona-tailored artifact synthesis. Evaluated on 120 real-world enterprise guideline documents, GUIDE achieves 96% document success, extracts 3,896 rules with 71.4% auto-approved, produces 812 deployment-ready artifacts, and reduces turnaround to 40ā125 minutes per document. Keywords: multi-agent, document parsing, multimodal, vision language models, artifact generation, guideline extraction, human-in-the-loop ā authors: . VLDB Workshop Reference Format: VLDB 2026 Workshop: DASHSys: Systems for Data-centric Agents with Human-in-the-loop. ā This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing info@vldb.org. Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment. ISSN 2150-8097. 1. Introduction Enterprise annotation pipelines rely on unstructured guideline documents that must be converted into structured, executable work artifacts before labeling can begin. Quality managers (QMs) and project managers (PMs) currently perform this manuallyāreading guidelines, inferring rules, resolving ambiguities, and assembling annotator instructions and statements of work. Each document takes 2ā3 days, is error-prone, and must be redone whenever documents are updated (2; 12). At scale, this becomes a critical bottleneck limiting how quickly projects can be staffed, launched, and maintained. At its core, this is a data management challenge. Rules extracted from heterogeneous documents must be versioned, deduplicated, validated against a schema, and made traceable from source to artifact. Enterprise guideline documents compound this: text is encoded as positional tokens rather than semantic units; tables span pages, contain merged cells, or appear as images; and visual elements such as annotation examples and bounding box diagrams carry semantically critical information not captured in text (4). Any parsing error cascades into downstream rule quality. We introduce GUIDE, a governed multi-agent framework that treats guideline-to-artifact conversion as a data management problem: six specialized agents coordinate through a shared staging store that holds all extracted rules and intermediate results as structured, versioned data. This enforces contractual guarantees between stages, makes every artifact traceable to its source, and triggers human review only at explicit thresholds, the properties that make the pipeline governed. Our contributions are: ā¢ A versioned, schema-validated shared staging store with stable rule_id keying, inter-agent schema contracts, and end-to-end provenance tracking, forming the data management backbone of the pipeline. ā¢ GUIDE, a governed multi-agent framework integrating deterministic parsing, VLM-based extraction, structured rule modeling, and dependency-aware artifact generation across six specialized agents. ā¢ A two-stage evaluation framework combining structural validation (L1) and LLM-based semantic scoring (L2) with automated acceptance, targeted regeneration, and selective HITL escalation; zero-edit approvals feed back as calibration signals, reducing reviewer load. ā¢ A dependency-driven HITL workflow routing only flagged rules, gaps, and artifacts to the QM or PM workbench by object type, severity, and downstream dependency. ā¢ Empirical evaluation on 120 real-world enterprise guideline documents demonstrating strong performance across extraction fidelity, consistency control, and persona-specific artifact generation. 2. Related Work Document Parsing and Extraction. Document parsing has evolved from rule-based OCR pipelines (17) to hybrid vision-language approaches (13; 7), alongside discourse-level segmentation and structure-aware methods (16; 15). Table extraction remains difficult: rule-based methods rely on geometric heuristics while VLM-based approaches improve robustness but introduce instability on dense or borderless layouts (18; 5). Recent VLMs like Qwen-VL (19; 3) and LLaVA (8) advance multimodal extraction but vary in hallucination and visual grounding (21). LLM-Based Structured Extraction. GoLLIE (14) shows annotation guidelines in prompts improve zero-shot extraction; UIE (10) enhances robustness via instruction tuning. However, these methods treat extraction as a single-step process and lack validation, contradiction handling, or production-level structuring. Layout-aware parsers such as Docling (9) and Donut (5) focus on extraction as a terminal task with no governed output routing or artifact generation. Since their outputs are structurally incompatible with GUIDEās schema-constrained rule format, a direct end-to-end comparison is ill-posed; we instead compare against a monolithic single-model baseline with identical input and output structure. Multi-Agent and HITL Systems. Multi-agent frameworks (20) decompose complex tasks across specialized agents but coordinate via unstructured message passing, without schema enforcement or provenance guarantees. HITL systems improve annotation quality through selective routing (1), yet treat review as a flat queue without distinction by object type, severity, or dependency. GUIDE addresses these gaps via schema-enforced inter-agent contracts, multi-stage quality evaluation, and dependency-aware HITL escalation. 3. System Architecture GUIDE is architected around a central versioned rule store: schema-enforced tables keyed by stable rule_ids serving as the shared data layer. Agents read from and write to this store through Pydantic-validated contracts, ensuring no downstream agent ever consumes structurally invalid data. This yields three properties essential for enterprise deployment: provenance (every artifact traces to its source rules and originating document), versioning (rule updates across document revisions are reconciled rather than reprocessed), and auditability (every HITL decision is logged against a stable identifier). The system comprises six components: Parsing Agent, Rule Extraction Agent, Consistency Module, Evaluation Module, HITL Controller, and Artifact Generation Agent (Figure 1). Figure 1. Overview of the GUIDE pipeline. 3.1. Document Ingestion and Rule Extraction The Parsing Agent separates deterministic text extraction from VLM processing. Text is extracted from PDF (PyMuPDF), DOCX (internal XML), and PPTX (LibreOffice ā PDF) without language models. Visual content is processed using Qwen2.5-VL (19) after MD5-based image deduplication. Extraction quality is assessed via Q=1ā(garbage+mojibake+repetition+silent_skip)Q=1-(garbage+mojibake+repetition+silent\_skip) where each term is a normalized defect rate. Extracted content is segmented into semantic categories (task definition, evaluation criteria, edge cases, compliance requirements, workflow specifications) as a routing layer for downstream processing. The Rule Extraction Agent applies a two-stage pipeline: open-domain extraction identifying candidate rules with source spans and confidence scores, followed by normalization into a fixed 26-field schema. Rule type determines persona routing: evaluation-criteria, edge-case, and qa-process rules route to the QM workbench; worker-requirements and delivery-schema rules route to the PM workbench. Rules pass through the Consistency Module, which applies embedding-based similarity filtering followed by NLI classification (11) for deduplication and version alignment. Gap analysis identifies missing or ambiguous aspects as structured GapObjects; resolved gaps generate ClarificationRecords and new RuleUnits. For each approved rule, examples are extracted when available or inferred under strict adherence to rule semantics. 3.2. Evaluation Engine and HITL As ground truth annotations are unavailable for this corpus, all evaluation metrics are computed using standard signals: cosine similarity for grounding, schema validation for structural compliance, and LLM-as-judge scoring for semantic quality. All thresholds were selected empirically over the full corpus and fixed prior to all reported experiments; with only 120 diverse production documents, a held-out split would have reduced the diversity available for calibration. To validate the LLM judge, 300 rule-level annotations were independently labeled by expert annotators blind to judge outputs; the judge achieves precision 0.941, recall 0.974, F1 0.957, and Cohenās Īŗ=0.813Īŗ=0.813 against these labels, providing an independent check that the calibrated thresholds align with expert judgment rather than overfitting the calibration corpus. These metrics and thresholds are applied through a two-stage validation pipeline that every structured object must pass. L1 (Structural) applies deterministic Pydantic validation: 28 constraints for RuleUnit (required fields, valid enumerations, prohibition of instruction-source duplication), 4 for ExampleObject (valid rule linkage, separation of correct/incorrect outputs), and 8 for GapObject (valid gap types, resolved rule references). Failed objects are retried, corrected, or rejected by error severity. L2 (Semantic) scores passing objects across K quality dimensions using an LLM-as-judge: (1) Sā”(x)=1Kāāk=1Kskā(x),skā1,ā¦,5 S(x)= 1K _k=1^Ks_k(x), s_kā\1,ā¦,5\ with K=5K=5 for RuleUnit (clarity, persona fit, completeness, category fit, severity fit), K=3K=3 for ExampleObject (rule alignment, discriminability, input realism), and K=3K=3 for GapObject (question quality, severity calibration, gap type fit). Routing is by minimum dimension score: minkā”skā„4 _ks_kā„ 4 auto-approves; any skā2,3s_kā\2,3\ routes to HITL; any sk=1s_k=1 rejects. Objects with rule_source = inferred always route to HITL. Zero-edit HITL approvals are logged as calibration data for L2, progressively reducing reviewer load over deployment cycles. Every review decision is recorded against the corresponding rule_id, providing a complete audit trail from human action to source rule. HITL is staged and dependency-aware: Phase 1 reviews RuleUnits; Phase 2 reviews GapObjects conditioned on the approved rule set; Phase 3 reviews ExampleObjects conditioned on finalized rules and resolved gaps. This logically bounds the downstream review surface and ensures QM effort focuses on outputs grounded in high-quality, approved rules. 3.3. Artifact Generation The Artifact Generation Agent transforms approved rules, resolved gaps, and evaluated examples into eight deployment-ready artifacts via Pydantic-constrained templates: annotator guidelines, QA strategy, QA rubric, reviewer instructions, gaps document, QA agent specification (6), annotator SOW, and job description/requisition. Each artifact is evaluated via: (2) ART=w1ā RC+w2ā SC+w3ā PA+w4ā CSC =w_1Ā·RC+w_2Ā·SC+w_3Ā·PA+w_4Ā·CSC where RC (rule coverage), SC (structural conformance), PA (persona appropriateness via Flesch readability and LLM judgment), and CSC (cross-section contradiction, evaluated by Claude Sonnet 4.6) are combined with empirically tuned weights. ARTā„4.0ARTā„ 4.0 auto-approves; 3.5ā¤ART<4.03.5 <4.0 routes to human review; ART<3.5ART<3.5 triggers regeneration. 4. Evaluation 4.1. Dataset and VLM Selection Our corpus consists of 120 enterprise guideline documents from industrial clients (67 text, 23 speech, 16 multimodal, 8 image, 6 video; PDF/DOCX/PPTX formats), used as received without preprocessing; documents are governed by client confidentiality and non-disclosure agreements and cannot be released. Complexity tiers are defined by modality composition: Low (ā¼74 74 KB, 39ā46 min), Moderate (ā¼1.5 1.5 MB, 46ā70 min), and High (ā¼4.2 4.2 MB, 65ā125 min). Table 1 shows Qwen2.5-VL-32B achieves the best balance of evidence rate (77.8%), hallucination (20.2%), and throughput (232/355 images) and the lowest duplication (14.6%); we select it for all stages. Table 1. VLM benchmark results across document extraction tasks. Model Evidence Halluc. Quality Thruput Dupl. (%) (%) (%) (img) (%) Qwen2.5-VL-32B 77.8 20.2 64.9 232 14.6 Qwen3-32B 76.7 22.7 73.2 184 17.9 LLaVA-13B 64.1 35.9 63.6 159 42.0 4.2. Content Extraction, Rule Extraction, and Artifacts Content extraction across 120 documents (Table 3) achieves 96% document success with 99.2% page coverage, 97.1% figure recall, 88.3% table recall, and Q=1.00Q=1.00 on successfully processed documents. Failures are caused by VLM timeouts on image-heavy documents and poorly structured tables. Rule extraction on 115 documents (Table 4) yields 3,896 RuleUnits with 84.8% evidence rate, 82.6% coverage, and 3.2% hallucination. L1 passes 99.1%; the 0.9% flagged are structurally valid but insufficiently precise for direct execution. L2 auto-approves 71.4% with 28.6% routed to HITL and 0% rejected; most HITL cases arise from incomplete semantic coverage where rules capture the primary case but miss edge conditions. The consistency module identifies 26.7% gaps, 3.0% duplications, and 2.9% contradictions. The 2ā3 day manual baseline is an expert estimate by the QMs and PMs who perform this task, indicating an order-of-magnitude reduction rather than an exact head-to-head. Artifact generation produces 812 artifacts (Table 5). Cross-section contradiction (93.7%) and structural conformance (83.1%) reflect strong logical and structural consistency. Rule coverage (56.9%) and persona appropriateness (59.8%) remain the most challenging dimensions, the former reflects partial instantiation under constrained generation context, the latter reflects the difficulty of adapting technical rules for non-expert audiences. Only 29.8% of artifacts are auto-approved, with 52.0% routed to human review and 18.2% rejected, revealing two primary failure modes: incomplete rule propagation and persona adaptation gaps. 4.3. Comparison Against a Monolithic Baseline We compare GUIDE against a one-pass Qwen2.5-VL-32B baseline that produces rules and artifacts directly from parsed documents, without the rule store, L1/L2 validation, consistency module, or HITL routing, using the same parser, documents, and scoring. Removing governance degrades every dimension (Table 2): hallucination rises from 3.2% to 15.7%, duplication from 3.0% to 10.3%, contradictions from 2.9% to 7.8%, and L1 pass rate falls from 99.1% to 93.2%. A layer-wise ablation isolates each layer: L1/L2 routes 1,114 units (28.6%) to review; the consistency module removes 117 duplicates, routes 113 contradictions, and surfaces 1,040 otherwise-undetected gaps. Table 2. Ungoverned one-pass baseline vs. GUIDE (same model, documents, and scoring). Metric Baseline GUIDE Hallucination 15.7% 3.2% Duplication 10.3% 3.0% Contradiction 7.8% 2.9% L1 pass rate 93.2% 99.1% Table 3. Content extraction evaluation across 120 documents. Category Metric Value Coverage Page coverage 99.2% Figure recall 97.1% Table recall 88.3% Quality score Overall score 1.00 Garbage ratio 0.00% Mojibake ratio 0.00% Repetition ratio 0.01% Silent skip ratio 0.00% Success rate Document success 96% (115/120) Table 4. Rule extraction (3,896 RuleUnits). Category Metric Value Quality Evidence rate 84.8% Coverage 82.6% Hallucination 3.2% L1 Pass rate 99.1% Ambiguity 0.9% L2 Auto-approved 71.4% Human review 28.6% Rejected 0.0% Consistency Gaps 26.7% Duplications 3.0% Contradictions 2.9% Table 5. Artifact evaluation (115 documents, 812 artifacts). Category Metric Value HITL routing Auto-approved 29.8% Human review 52.0% Rejected 18.2% L1 Rule coverage 56.9% Structural conform. 83.1% Cross-sect. contra. 93.7% Gap completeness 66.7% L2 Persona approp. 59.8% Sequential correct. 65.2% 5. Limitations and Future Work Limitations. GUIDE demonstrates strong performance across content extraction, rule generation, consistency validation, and artifact generation on real-world enterprise documents. VLM extraction stability decreases on low-quality scans and borderless or merged-cell tables, a known challenge across current vision-language systems. Persona appropriateness and rule coverage remain the most challenging artifact dimensions, reflecting the inherent difficulty of adapting technical rules to non-expert audiences. The calibration mechanism relies on accumulating zero-edit HITL approvals over deployment cycles, so scoring stability in early cycles remains limited. The current evaluation also covers English enterprise guidelines only. While the 26-field rule schema and QM/PM persona routing are specific to annotation operations, the staging store, schema contracts, L1/L2 evaluation, and dependency-aware HITL routing are domain-agnostic. Future Work. We will improve table detection and VLM prompting for degraded layouts, develop persona adaptation modules that learn audience-specific language patterns from approved artifacts, and incorporate structured reviewer edits as additional fine-tuning signal to accelerate calibration convergence. GUIDE will further be extended to multilingual settings and broader domains including legal, clinical, and regulatory, with domain-adapted consistency models trained on enterprise rule pairs. 6. Conclusion We presented GUIDE, a governed multi-agent framework that transforms heterogeneous enterprise guideline documents into structured, deployment-ready artifacts via a schema-enforced shared staging store with end-to-end provenance. By coordinating six specialized agents through typed intermediate relations and applying threshold-driven HITL escalation, GUIDE reduces end-to-end turnaround from 2ā3 days to 40ā125 minutes while maintaining strong extraction fidelity and consistency. GUIDE shows how governed multi-agent pipelines serve as a principled foundation for data-aware, human-aligned agentic systems where reliability, traceability, and selective human oversight are first-class design goals. Acknowledgements. We thank Srinivasa Karthikeya Reddy Kovvuri for his contributions to this work. References Amershi et al. (2016) S. Amershi et al. Human-in-the-loop machine learning: a survey. arXiv preprint arXiv:1611.04477. Cited by: Ā§2. Anderson et al. (2024) E. Anderson J. Fritz et al. The design of an llm-powered unstructured analytics system. arXiv preprint arXiv:2409.00847. Cited by: Ā§1. Bai et al. (2025) S. Bai Y. Cai et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cited by: Ā§2. Ke et al. (2025) W. Ke et al. Large language models in document intelligence: a comprehensive survey, recent advances, challenges, and future trends. ACM. Cited by: Ā§1. Kim et al. (2022) G. Kim et al. OCR-free document understanding transformer (donut). In ECCV, Cited by: Ā§2, Ā§2. Kothari et al. (2026) S. Kothari et al. Position: early-stage quality assurance in annotation pipelines is more cost-effective than late-stage validation. arXiv preprint arXiv:2605.15714. Cited by: Ā§3.3. Li et al. (2025) Z. Li et al. Monkeyocr: document parsing with a structure-recognition-relation triplet paradigm. arXiv preprint arXiv:2506.05218. Cited by: Ā§2. Liu et al. (2023) H. Liu, C. Li, Q. Wu, and Y. J. Lee Visual instruction tuning. Advances in neural information processing systems 36. Cited by: Ā§2. Livathinos et al. (2025) N. Livathinos et al. Docling: an efficient open-source toolkit for ai-driven document conversion. arXiv preprint arXiv:2501.17887. Cited by: Ā§2. Lu et al. (2022) Y. Lu et al. Unified structure generation for universal information extraction. External Links: 2203.12277 Cited by: Ā§2. MacCartney (2009) B. MacCartney Natural language inference. Stanford University. Cited by: Ā§3.1. Perot et al. (2024) V. Perot et al. Lmdx: language model-based document information extraction and localization. In ACL 2024, Cited by: Ā§1. Poznanski et al. (2025) J. Poznanski et al. Olmocr: unlocking trillions of tokens in pdfs with vision language models. arXiv:2502.18443. Cited by: Ā§2. Sainz et al. (2024) O. Sainz et al. GoLLIE: annotation guidelines improve zero-shot information-extraction. External Links: 2310.03668 Cited by: Ā§2. Sediqin and Argamon (2025) M. Sediqin and S. E. Argamon RST-ndas: rst nuclei for discourse-aware abstractive summarization. In (DSAA), Cited by: Ā§2. Sediqin and Argamon (2026) M. Sediqin and S. E. Argamon LACES: lexical and contextual awareness for edu segmentation. In Advances in Natural Language Processing and Information Retrieval, External Links: ISBN 978-3-032-20897-2 Cited by: Ā§2. Smith (2007) R. Smith An overview of the tesseract ocr engine. In ICDAR 2007, Cited by: Ā§2. Verbovskiy (2025) A. Verbovskiy Comparing ocr and vlm techniques in processing tabular data. Masterās Thesis, A. Verbovskiy. Cited by: Ā§2. Wang et al. (2024) P. Wang et al. Qwen2-vl: enhancing vision-language modelās perception of the world at any resolution. arXiv preprint arXiv:2409.12191. Cited by: Ā§2, Ā§3.1. Wu et al. (2023) Q. Wu G. Bansal et al. AutoGen: enabling next-gen llm applications via multi-agent conversation. arXiv:2308.08155. Cited by: Ā§2. Zhu et al. (2024) F. Zhu et al. MMDocBench: benchmarking large vision-language models for fine-grained visual document understanding. External Links: 2410.21311 Cited by: Ā§2.