Paper deep dive

SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding

Abhigya Verma, Khyati Mahajan, Amit Kumar Saha, Shruthan Radhakrishna, Sagar Davasam, Vikas Yadav, Sai Rajeswar Mudumba

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 88

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%

Last extracted: 7/14/2026, 6:58:55 AM

Summary

This paper introduces SynthDocBench, a fully synthetic, controlled benchmark for evaluating long-context visual document understanding in Vision Language Models (VLMs). By systematically varying document length, layout structure, modality composition, and question difficulty, the benchmark isolates failure modes that existing real-world benchmarks obscure. Evaluation of eight frontier VLMs reveals sharp performance degradation with length, systematic positional sensitivity (middle sections are hardest), and breakdown of chart comprehension in long documents, suggesting current models overfit to benchmark artifacts rather than achieving robust reasoning.

Entities (8)

ServiceNow AI · organization · 95%SynthDocBench · benchmark · 95%Gemini 3.1 Pro · model · 90%Qwen3.5-VL-122B · model · 90%Vision-Language Models · model-type · 90%D3.js · software · 85%DocVQA · benchmark · 85%Positional Sensitivity · concept · 85%

Relation Signals (7)

ServiceNow AI → authored → SynthDocBench

confidence 95% · 1 ServiceNow AI ... We introduce SYNTHDOCBENCH, a fully synthetic benchmark for long-context visual document understanding

Gemini 3.1 Pro → achieveshighestaccuracyon → SynthDocBench

confidence 90% · Gemini-3.1-Pro (Google, 2026) leads all models by a substantial margin, achieving an overall ACC of 0.725

SynthDocBench → evaluates → Vision-Language Models

confidence 90% · Evaluating seven frontier VLMs, we uncover three failure modes that existing benchmarks cannot surface

Qwen3.5-VL-122B → rankssecondon → SynthDocBench

confidence 85% · Qwen3.5-VL-122B ranks second overall (ACC 0.655), demonstrating that a 122B MoE open-weight model can approach Gemini-level performance

SynthDocBench → reveals → Positional Sensitivity

confidence 85% · uncover three failure modes... a systematic positional sensitivity in which the middle third of a document is hardest for five of six models

D3.js → usedfor → Chart Generation

confidence 85% · Each visualization is generated in two aligned forms: (i) a visible D3.js rendering that appears in the document

SynthDocBench → comparesagainst → DocVQA

confidence 80% · Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that systematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constructed using a combinatorial design, each factor is varied independently across generated documents, enabling controlled analysis of model behavior. Documents are generated end to end using an LLM pipeline across six layout archetypes, with a 40 percent random override to prevent models from exploiting spurious correlations. Additionally, SynthDocBench spans long-context documents with substantially greater length and structural diversity than existing benchmarks. Evaluating seven frontier VLMs, we uncover three failure modes that existing benchmarks cannot surface: sharp degradation with document length, a systematic positional sensitivity in which the middle third of a document is hardest for five of six models and five of six models show a negative Early-to-Late trend (steepest decline: 8.3 percentage points), and breakdown of chart comprehension in long-document settings. These results suggest that current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

86,525 characters extracted from source content.

Expand or collapse full text

Published as a conference paper at COLM 2026 SynthDocBench: Controlled Benchmark for Long- Context Visual Document Understanding Abhigya Verma 1* , Khyati Mahajan 1* , Amit Kumar Saha 1* , Shruthan Radhakr- ishna 1 , Sagar Davasam 1 , Vikas Yadav 1 , Sai Rajeswar 1, 2, 3 1 ServiceNow AI 2 Mila 3 Université de Montréal Vision language models (VLMs) have achieved strong performance on visual docu- ment understanding benchmarks such as DocVQA, ChartQA, and MMLongBench- Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SYNTHDOCBENCH, a fully synthetic benchmark for long-context visual document understanding that sys- tematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constructed using a combi- natorial design, each factor is varied independently across generated documents, enabling controlled analysis of model behavior. Documents are generated end to end using an LLM pipeline across six layout archetypes, with a 40 percent random override to prevent models from exploiting spurious correlations. Additionally, SynthDocBench spans long-context documents with substantially greater length and structural diversity than existing benchmarks. Evaluating seven frontier VLMs, we uncover three failure modes that existing benchmarks cannot surface: sharp degradation with document length, a systematic positional sensitivity in which the middle third of a document is hardest for five of six models and five of six models show a negative Early→Late trend (steepest decline: 8.3 percentage points), and breakdown of chart comprehension in long-document settings. These results suggest that current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding. Correspondence: abhigya.verma@servicenow.com Code:https://github.com/ServiceNow/SynthDocBench Dataset:https://huggingface.co/datasets/ServiceNow-AI/ SynthDocBench * Equal contribution. 1 Introduction Understanding long, visually rich documents is a defining challenge for vision language models (VLMs). Real-world documents interleave text, tables, charts, and complex layouts across dozens or hundreds of pages, demanding both long-range retrieval and cross-modal reasoning. Benchmarks such as DocVQA (Mathew et al., 2021), ChartQA (Masry et al., 2022), and MMLongBench-Doc (Ma et al., 2024) have driven substantial progress, yet this progress conceals a fundamental diagnostic blind spot: when a model fails on a real document, it is difficult to know why. On single-page tasks, evaluation is approaching saturation. Frontier models exceed 95% on DocVQA (Bai et al., 2025a; Wang et al., 2025a) and 89% on ChartQA (Anthropic, 2025; Bai et al., 2025a) — though harder benchmarks such as ChartQAPro (Masry et al., 2025), Visu- Logic (Zhang et al., 2025), and ChartMuseum (Xu et al., 2025) show that chart understanding is far from solved, with degradations exceeding 30 percentage points. Yet all evaluate charts in isolation, abstracted from the multi-page contexts in which they naturally occur. Long- context benchmarks like MMLongBench-Doc (Ma et al., 2024) address the document-length dimension, targeting long PDF comprehension with rich visual content, and the strongest model achieves only 57%. LongDocURL (Deng et al., 2025) and M-LongDoc (Chia et al., 1 arXiv:2607.10400v1 [cs.CV] 11 Jul 2026 Published as a conference paper at COLM 2026 2025) extend to documents spanning 100s of pages, but prioritize breadth of coverage over controlled diagnosis: neither constructs questions requiring joint reasoning over charts and textually distributed evidence across distant pages. Furthermore, these benchmarks draw on real documents, confounding potential sources of difficulty (answer depth, presentation modality, layout density, cross-page evidence integration) which co-vary and cannot be disentangled. Synthetic benchmarks have a long history of enabling precisely this kind of decomposition. The bAbI tasks (Weston et al., 2016) and SCAN (Lake & Baroni, 2018) used program- matic generation to isolate reasoning primitives in NLP; CLEVR (Johnson et al., 2017) and RAVEN (Zhang et al., 2019) did the same for visual relational and analogical reasoning; and PuzzleVQA (Chia et al., 2024) recently applied synthetic abstract patterns to diagnose multimodal reasoning bottlenecks in VLMs. The common principle is that synthetic control trades ecological validity for interpretability, enabling attribution of failures to specific causes rather than an opaque bundle of confounds. No comparable instrument exists for long-context visual document understanding, an important setting where confounds are arguably most severe. We address this gap with SYNTHDOCBENCH, a fully synthetic, controlled benchmark that enables the first systematic decomposition of VLM failure modes in long-context document understanding by varying document length, page depth, modality composition, and question type as independent axes. We contribute: 1.SYNTHDOCBENCH, a controlled synthetic benchmark with independently variable length, depth, modality, and question-type axes, released as three task-specific subsets, which probes 24 distinct D3.js chart types (including visually ambiguous forms such as dumb- bell, lollipop, slope, and sparkline grids) requiring exact numerical reads that cannot be inferred from surrounding text;cross_modal, which places supporting charts and corroborating text in separate, non-adjacent sections to test cross-modal grounding over long-range dependencies; andcomplex, which requires combining 2 to 4 evidence units from text and charts across difficulty levels L1 to L5, ranging from direct value lookup to cross-section synthesis. 2.A fully automated LLM-based generation pipeline with a dual-layer document design: every chart is generated simultaneously as a rendered D3.js visualisation and as a hid- den structured metadata block used only for ground-truth derivation, making answers deterministic by construction across 24 chart types and 6 layout archetypes. 3.A systematic empirical analysis of seven frontier VLMs, validated with a cross-judge robustness check (GPT-5 and Gemini-as-judge agree within 3.5 ACC points,r ≥0.94, across all question types), revealing three concurrent, previously unobservable failure modes: sharp performance degradation with increasing evidence complexity and rea- soning depth (L1→L5), systematic positional sensitivity in which the middle section of a document is hardest for five of six models and five of six models exhibit a nega- tive Early→Late trend (steepest decline: 8.3 p), and collapse of precise chart-reading accuracy in long-document contexts, even for models that perform well on existing benchmarks. 2 Related Work Charts and Visual Reasoning. Chart understanding has been evaluated primarily in isolation from document context. ChartQA (Masry et al., 2022) established the standard evaluation setup for chart-oriented VQA, now approaching saturation. ChartQAPro (Masry et al., 2025) addresses this through harder, more diverse real-world charts, exposing over 30 percentage points of degradation for models that saturate ChartQA, though it retains the isolated-image setting. ChartGalaxy (Li et al., 2025) demonstrates that programmatic gener- ation can yield million-scale synthetic diversity spanning 75 chart types and 330 stylistic variations, providing a methodological precedent for SYNTHDOCBENCH’s synthetic ap- proach. VisuLogic (Zhang et al., 2025) and ChartMuseum (Xu et al., 2025) probe fine-grained logical and perceptual reasoning over isolated charts, while MultiChartQA (Zhu et al., 2025) extends this to simultaneous multi-chart reasoning. Collectively, these benchmarks establish 2 Published as a conference paper at COLM 2026 BenchmarkScopeAvg. Context#Docs/Charts#Questions Chart Benchmarks ChartQA (Masry et al., 2022)Isolated charts1 chart21,95332,701 ChartQAPro (Masry et al., 2025)Isolated charts1 chart1,3411,948 ChartGalaxy (Li et al., 2025)Isolated charts1 chart1,763,189— VisuLogic (Zhang et al., 2025)Isolated images1 image1,0001,000 ChartMuseum (Xu et al., 2025)Isolated charts1 chart9281,162 MultiChartQA (Zhu et al., 2025)Multi-chart2–3 charts655944 Document VQA Benchmarks DocVQA (Mathew et al., 2021)Single-page docs1 page12,76750,000 MP-DocVQA (Tito et al., 2023)Multi-page docs ≤20 pages—46,000 SlideVQA (Tanaka et al., 2023)Multi-page docs ∼20 pages2,61914,500 MMLongBench-Doc (Ma et al., 2024)Multi-page docs47.5 pages1351,082 LongDocURL (Deng et al., 2025)Multi-page docs ∼83 pages3962,325 M-LongDoc (Chia et al., 2025)Multi-page docs>200 pages—851 MMLongBench (Wang et al., 2025b)Multi-page docs8K–128K tokens13,331— SYNTHDOCBENCHMulti-page docsAvg. 51.1 pages200 / 3,3401,788 Figure 1: Landscape of benchmarks Top: benchmark comparison by average document length (pages) and average textual context (tokens). SynthDocBench occupies a unique region with both long multi-page context and high textual density. Bottom: comparison with existing chart and document VQA benchmarks. Prior benchmarks typically isolate either charts or long documents, whereas SYNTHDOCBENCH is designed to study their intersection under long contexts. that chart understanding is far from solved, yet none evaluate charts within the document contexts where they naturally occur and where interpretation requires engagement with surrounding text, tables, and figures. Long-Context Multimodal Benchmarks.Early long-context evaluation focused on needle- in-a-haystack retrieval (Wang et al., 2024), which provides insufficient difficulty ceilings for frontier models and correlates poorly with downstream reasoning performance (Wang et al., 2025b). Document-centric benchmarks have progressed from single-page evaluation via DocVQA (Mathew et al., 2021) through multi-page extensions including MP-DocVQA (Tito et al., 2023) and SlideVQA (Tanaka et al., 2023), though both are constrained to roughly twenty pages and principally assess span extraction. MMLongBench-Doc (Ma et al., 2024) 3 Published as a conference paper at COLM 2026 was the first benchmark targeting long PDF comprehension with interleaved visual content, and MMLongBench (Wang et al., 2025b) extended this framework to five task categories spanning up to 128K tokens. LongDocURL (Deng et al., 2025) and M-LongDoc (Chia et al., 2025) advance the state of the art through cross-element localisation and open-ended responses, but both prioritize breadth of coverage over diagnostic decomposition. SYN- THDOCBENCH is distinguished by its explicitly diagnostic design: document length, page depth, modality composition, and question type are varied as independent axes, with charts and figures foregrounded as primary reasoning targets whose interpretation requires engagement with arbitrarily distant contextual evidence. Vision-Language Models for Document Understanding.Recent vision-language models have made single-page document QA increasingly saturated: frontier systems such as Qwen3-VL (Bai et al., 2025a) and InternVL3.5 (Wang et al., 2025a) now approach ceiling performance on DocVQA, so these benchmarks provide limited diagnostic separation among leading models. In contrast, long-context visual document reasoning remains substantially harder: the best reported results on MMLongBench-Doc remain far below single-page performance. 1 This gap suggests that failures are driven by properties that emerge in multi- page settings e.g., long-range dependency tracking, retrieval over dispersed evidence, yet current evaluations do not isolate which document factors are most responsible. Figure 1 situates document VQA benchmarks by average document length (pages) and average token count. Single-page benchmarks (DocVQA, ChartQA) lie in the bottom-left, whereas multi- page benchmarks shift toward higher length and token regimes. SYNTHDOCBENCH lies near the frontier along both axes, comparable to LongDocURL in page count and substantially denser in tokens than MMLongBench, while using controlled synthetic documents and requiring retrieval and reasoning over multi-modal evidence across multiple pages. 3 The SYNTHDOCBENCH Benchmark The benchmark is generated via three coupled stages (Figures 2 and 3): (i) document gener- ation, mapping a topic seed to a styled visual report with an aligned structured manifest; (i) QA generation, converting the manifest into difficulty-controlled QA pairs; and (i) vision- only evaluation, measuring model performance on rendered page images with no access to the underlying metadata. Reference answers are derived deterministically from the same structured artifacts used to generate the documents, eliminating the annotation bottleneck of real-document benchmarks (Weston et al., 2016; Johnson et al., 2017). 3.1 Synthetic Visual Document Generation The document-generation pipeline maps a topic seedτto a styled visual reportDand a document-level QA manifestM, factorizing synthesis into content generation, visual grammar, visualization synthesis, and assembly (Figure 2). Topic-grounded content generation: Givenτ, the pipeline constructs a semantic backbone from retrieved topic-relevant evidence and reorganizes it into a structured intermediate rep- resentation that exposes section boundaries, data-bearing spans, and salient content units. Downstream stages operate over this representation rather than free-form text, enabling precise evidence tracing. Design and visual grammar: A layout archetypea ∈ Ais sampled with probability 0.6 from a topic-conditioned distribution and with probability 0.4 uniformly at random (see Appendix L.2), preventing trivial correlations between subject matter and layout. The archetype governs page grammar, chart placement strategy, and auxiliary component types (metric cards, timelines, pull quotes), yielding a realistic and diverse report distribution. Grounded visualization synthesis: Each visualization is generated in two aligned forms: (i) a visible D3.js rendering that appears in the document, and (i) a structured metadata objectV k recording chart semantics (axes, data points, derived insights). This dual-layer for- mulation ensures ground truth is available by construction without post-hoc chart parsing 1 https://huggingface.co/spaces/OpenIXCLab/mmlongbench-doc, accessed March 2026. 4 Published as a conference paper at COLM 2026 4 QA M et adat a & Qualit y Assuranc e Gat her HTM L Ar t if ac ts Validate Consistency Ex t rac t Table Programmat ic QA M anif est Generat ion 1 Content Generat ion Visualizat ion Pipeline 3 W iki Inf o Ex t rac t ion M et adat a Selec t ion Gen: Repor t Sec t ions Gen: Visual St y ling Topic s (Seed Input ) Design and St y ling 2 Summarize Content Build Ric h Components Plan Position Assign Type Visualizat ions St ruc t ure Char t Dat a & Feed Real Dat a Gen: Char ts & M et adat a (dual- out put ) QA M et adat a & Qualit y Assuranc e Assembly & Out put 5 Sy nt hDoc Benc h Chromium via Playwright Assembled HTM L Repor t HTM L M et adat a Quest ion Answ er M anif est Figure 2: Synthetic visual document generation pipeline. From a topic seed, the pipeline generates grounded report content, applies document-level visual styling, synthesizes visualizations, performs metadata and QA validation, and assembles the final HTML/PDF reports with a machine-readable QA manifest. Determinist ic Key Inf ormat ion Ex t rac t ion Quer y Generat ion Sy nt het ic Repor ts Prose Infographics Autocorrection St ruc t ured QA Out put HTM L + D3.js Char ts Ad versarial Verif ic at ion ba Diff iculty Verif ication Timelines Char t Queries Cross M odal Queries M ult i- Hop Queries QA Pairs GT Answer Sy nt hDoc Benc h Figure 3: QA generation pipeline. The pipeline parses the generated report into structured evidence channels, extracts and synthesizes key information, and generates chart-reading, cross-modal, and multi-hop questions. A verification stage filters weak or malformed items before serializing the final QA output. or human labeling (full schema in Appendix L.3). Validation and assembly: Numeric values in both chart and table metadata are recomputed from structured data and corrected before finalization; all validated metadata are aggre- gated intoM. The report is assembled as HTML and rendered to PDF using Playwright, 2 producing an aligned pair (D,M). 3.2 Question-Answer Generation The second stage converts a generated report into structured QA items via evidence recovery, synthesis, and generation with validation (Figure 3). The pipeline parsesDinto text, table, and visualization channels, recovering chart semantics directly from embedded metadata rather than pixels. Evidence units are then synthesized into higher-order compositions supporting aggregation, comparison, and cross-source reasoning. The benchmark produces three question families: chart-reading (direct chart or table seman- tics), cross-modal (joint reasoning over textual and visual evidence), and complex multi-hop (composition across 2–4 evidence units). Each question carries a difficulty labelL1–L5 (Table 13) and a structured evidence trace. A validation stage filters malformed, weakly supported, or ambiguous items; the serialized output schema is detailed in Appendix L.4. 2 https://github.com/microsoft/playwright 5 Published as a conference paper at COLM 2026 MetricMeanMed.MinMax Pages51.1492491 Words20,56820,45017,36725,138 Charts16.714532 PDF (MB)2.122.101.203.80 Table 1: Per-document statistics across all 200 reports. Three-layer quality control (numeric recomputation, automated consistency filtering, and 100-sample manual review with >96% acceptance rate) is described in Appendix F. 3.3 Benchmark Statistics The resulting benchmark comprises 200 synthetic reports and 1,788 questions distributed across three subsets: 597chartreading, 597complexmulti-hop, and 594cross_modal. Doc- uments average 51.1 pages, 16.7 charts, and≈20,568 words, placing SYNTHDOCBENCH firmly in the long-document regime. The benchmark spans 24 distinct chart types across 6 layout archetypes, ensuring models cannot exploit narrow visual or structural distributions. Figure 4 shows the distributions of page count, word count, and chart count per document: all three are unimodal and tightly concentrated, reflecting the controlled generation pipeline rather than the heavy-tailed distributions typical of real-world corpora. The tight ranges reflect the controlled generation pipeline: page count, word count, and chart count are all bounded by design, enabling ablations that hold document complexity constant while varying other axes (modality, question type, layout). 0255075 Pages 0 10 20 30 40 50 Documents Mean=51 01000020000 Word count 0 20 40 60 80 Documents Mean=19552 0102030 Charts 0 20 40 60 Documents Mean=17 (a) Document property distributions (pages, words, charts). Editorial 24% Academic 18% Infographic 16% Brutalist 15% Magazine 15% Dashboard 13% (b) Layout archetypes Figure 4: Document composition statistics across 200 reports. 4 Evaluation Setup Candidate models operate under a strict vision-only protocol: they receive only the rendered page-image sequenceIand never access HTML source, embedded metadata, orM. The end-to-end pipeline is illustrated in Figure 8 (Appendix A). Each PDF is rasterized at 144 DPI, capped at 120 pages, and concatenated into single-column 5-page vertical strips (max 7,900 px, 4 MB); we use 5-page strips as the default to satisfy the input constraints of all evaluated models simultaneously, noting that denser strips further improve performance where API limits permit. Full hyperparameters are in Table 19. Candidate models predict ˆ a = f θ (I ,q)at temperature 0 via a fixed system prompt requiring concise, vision-grounded 2 to 4 sentence answers (prompt in Appendix L.7). GPT-5 (Singh et al., 2025) scores each ( ˆ a,a ∗ )pair at temperature 0, returning a JSON score in[0, 10](rubric in Table 6). Parse failures receive score−1 and are excluded from all aggregates (Q valid ). To validate judge reliability, we re-scored a subset of responses using Gemini-3.1-Pro and Claude-Sonnet-4.5 as alternative judges (Appendix C): GPT-5 and Gemini-as-judge agree to within 3.5 ACC points (Pearsonr ≥0.94 across all question types), confirming that model rankings are robust to judge choice (see Table 9). 6 Published as a conference paper at COLM 2026 We report mean judge score and threshold accuracy: ACC( f θ ) = 1 |Q valid | ∑ q∈Q valid 1 h J ( ˆ a q , a ∗ q )≥ 6 i ,(1) whereτ=6 (“core answer correct”) aligns with MMLongBench-Doc (Ma et al., 2024). Both metrics are stratified by question family and difficulty level L1–L5. 4.1 Experiments and Results We evaluate the following eight vision-language models on SYNTHDOCBENCH: Gemini- 3.1-Pro (Google, 2026), GPT-5.4 (Singh et al., 2025), GPT-4o (OpenAI et al., 2024), Claude- Sonnet-4.5 (Anthropic, 2025), Qwen3.5-VL-122B (Qwen Team, 2025), Qwen3-VL-235B (Bai et al., 2025a), InternVL3-78B (Wang et al., 2025a), and Qwen2.5-VL-7B (Bai et al., 2025b). All models are evaluated with GPT-5 as judge. Prior BenchmarksSYNTHDOCBENCH (Ours) AccuracyOverallChartComplexCross-Modal Model DocVQA (1 pg) MMLong- Bench-Doc (47.5 pg) ACC Score ACC Score ACC Score ACC Score Proprietary Models Gemini-3.1-Pro (Google, 2026)93.445.10.7257.190.7597.620.7897.280.6286.65 GPT-5.4 (Singh et al., 2025)—0.4234.680.4254.240.4575.210.3874.59 GPT-4o (OpenAI et al., 2024)92.846.30.3864.380.4574.440.3604.660.3424.05 Claude-Sonnet-4.5 (Anthropic, 2025)92.040.10.3143.960.3534.110.3374.280.2503.49 Open-Weight Models Qwen3.5-VL-122B (Qwen Team, 2025)—0.6556.770.7137.210.6906.890.5616.22 Qwen3-VL-235B (Bai et al., 2025a)96.557.00.5866.180.6426.600.6116.230.5035.71 InternVL3-78B (Wang et al., 2025a)95.124.30.3834.390.4564.750.3974.730.2963.69 Qwen2.5-VL-7B (Bai et al., 2025b)93.725.10.0811.080.1621.760.0120.370.0671.12 Table 2: Performance on SYNTHDOCBENCH (200 reports, 1,788 questions) alongside two established benchmarks. DocVQA (Mathew et al., 2021) tests single-page understanding; MMLongBench-Doc (Ma et al., 2024) tests multi-page documents (avg. 47.5 pages). — = not publicly reported at time of writing. Bootstrap 95% CIs (2,000 resamples, seed 42) are ≤±0.023 overall and≤±0.041 per subset across all models; all pairwise ACC gaps between adjacent-ranked models exceed their combined CI half-widths. Per-model CI details are reported in Table 10. Table 2 presents ACC (τ=6) and mean judge score across all eight models, stratified by question subset. Gemini-3.1-Pro leads all models by a substantial margin, achieving an overall ACC of 0.725 and a mean judge score of 7.19. The ranking Gemini>Qwen3.5- VL-122B>Qwen3-VL-235B>GPT-5.4>GPT-4o≈InternVL3-78B>Claude-Sonnet-4.5 ≫Qwen2.5-VL-7B holds consistently across all three question subsets. Qwen3.5-VL-122B ranks second overall (ACC 0.655), demonstrating that a 122B MoE open-weight model can approach Gemini-level performance on long-context document understanding. GPT-5.4 ranks fourth overall (ACC 0.423), between Qwen3-VL-235B (0.586) and GPT-4o (0.386). GPT-4o (0.386) and InternVL3-78B (0.383) are statistically indistinguishable despite their different architectures, suggesting that neither parameter scale nor training approach alone determines long-context document performance on this benchmark. OCR + Text-Only Baseline. Chart-reading is the easiest subset for most models, while cross-modal questions are consistently the hardest, confirming that integrating evidence across text and charts within long documents remains an open challenge for all current VLMs. The gap between Gemini and the next-best model (Qwen3.5-VL-122B, 0.655) is 7.0 ACC points; we discuss a potential distribution-familiarity confound in Section 5. Model 7 Published as a conference paper at COLM 2026 OverallChartComplexCross- Modal 0.0 0.2 0.4 0.6 0.8 1.0 ACC ( = 6 ) 0.39 0.46 0.36 0.34 0.45 0.30 0.80 0.40 GPT-4o (vision) OCR + GPT-4o Figure 5: GPT-4o vision vs. OCR+GPT-4o (text-only) ACC by subset. Vision dominates on Chart; OCR dominates on Complex. Figure 6: Hard failures: 109 questions where all six models score≤3, broken down by error category and question subset (Ch. = chart-reading; Cx. = complex; XM. = cross- modal). rankings on SYNTHDOCBENCH correlate with MMLongBench-Doc (Spearmanρ=0.657, Pearson r=0.683), supporting external validity of the benchmark ordering (Appendix C). PyMuPDF extracts page text (up to 60,000 chars) and GPT-4o answers without images (n=1,525; 263 parse failures excluded). As Figure 5 shows, performance is highly asymmet- ric: complex multi-hop ACC is 0.798 for OCR vs. 0.360 for vision, confirming that complex evidence is largely text-recoverable; chart-reading reverses sharply (OCR 0.297 vs. vision 0.457), confirming that chart questions genuinely require pixel-level visual decoding. The 46 p gap between OCR chart ACC and Gemini’s (0.759) isolates visual perception as the primary bottleneck. 5 Analysis and Discussion L1L2L3L4L5 ModelACCSc.ACCSc.ACCSc.ACCSc.ACCSc. Proprietary Models Gemini-3.1-Pro0.7847.840.7317.530.6756.910.7667.190.6706.48 GPT-5.40.2312.370.4224.660.5015.290.4895.280.2594.02 GPT-4o0.2712.680.4554.780.4514.830.4004.640.1733.54 Claude-Sonnet-4.50.3823.810.3003.960.3123.990.3604.280.1543.18 Open-Weight Models Qwen3.5-VL-122B0.7077.160.6737.110.6396.660.6836.760.5286.02 Qwen3-VL-235B0.6486.480.5956.550.5686.010.6196.250.4575.36 InternVL3-78B0.4374.420.3824.350.3904.480.4274.630.1983.62 Qwen2.5-VL-7B0.0150.170.1231.480.1551.880.0330.710.0050.26 Table 3: Model performance stratified by difficulty levelL1–L5. ACC = fraction of questions with judge score≥6; Sc. = mean judge score (0–10). Best per column in bold. Bootstrap 95% CIs (2,000 resamples, seed 42) are≤±0.070 per level for all models; all adjacent-ranked model gaps exceed their combined CI half-widths. Per-level CI details are in Table 11. We analyse model performance across three dimensions: difficulty level, question category, and error type, surfacing failure modes hidden by aggregate scores. 8 Published as a conference paper at COLM 2026 Gemini-3.1-ProQwen3.5-VL-122BGPT-5.4GPT-4oClaude-S-4.5Qwen3-VL-235BInternVL3-78BQwen2.5-VL-7B Value reading Comparison Trend / pattern Verify w/ chart Integrate sources Compare repr. Historical/timeline Technical Impact / reception 0.780.710.230.270.380.650.440.01 0.760.690.310.380.270.620.330.07 0.730.740.740.720.410.650.600.40 0.620.560.400.400.280.480.320.04 0.690.590.370.300.260.550.290.03 0.580.540.390.330.210.480.280.13 0.910.870.680.550.510.780.600.01 0.640.480.180.110.140.430.160.00 0.810.720.510.420.360.620.440.02 Chart Reading Cross-Modal ComplexReasoning 0.0 0.2 0.4 0.6 0.8 1.0 ACC ( = 6 ) Figure 7: ACC (τ=6) by fine-grained question category. Rows are grouped by subset (Chart Reading, Cross-Modal, Complex Reasoning); columns are models ordered by overall ACC. The colour scale runs from red (0) to green (1). Full numerical values are in Table 12 (Appendix I). Difficulty-Stratified Results Table 3 shows ACC by levelL1–L5. All models except Gemini-3.1-Pro degrade monotonically towardL5; Claude-Sonnet-4.5 drops 23 p (L1→L5) while Gemini stays flat (0.670–0.784). Qwen3.5-VL-122B follows a similar flat-then-drop pattern (0.707 atL1, 0.528 atL5), tracking closely with Gemini across all levels. GPT-4o’s anomalousL1 dip (0.271) suggests precise value-extraction is harder for it than composi- tional reasoning. Question-Category Breakdown Figure 7 breaks down ACC by fine-grained category. Trend/pattern questions are easiest (perceptually salient direction cues), while value-reading and integrate-sources cross-modal questions are hardest — precise axis-label reading and multi-page evidence alignment each bottleneck distinct model families. The widest cross- model gap is in technical/quantitative complex questions (Gemini: 0.643 vs. GPT-4o: 0.111, InternVL3: 0.156), isolating quantitative multi-step reasoning as the primary frontier differ- entiator. Positional Bias: Evidence Location Within DocumentsQuestions are bucketed by relative chart positionp=k/Kinto equal thirds (Table 4; see Appendix G for the bar chart). The middle third is hardest for 5 of 8 models, dropping 5–18 p below Early. Qwen3.5-VL- 122B shows the steepest Early→Middle drop (−18.5 p) while partially recovering in Late; Claude-Sonnet-4.5 shows the steepest monotonic Early→Late decline (−11.7 p); Gemini- 3.1-Pro shows a U-shaped pattern echoing the lost-in-the-middle effect (Liu et al., 2023). Hard Failure and Domain Error Analysis Of 109 questions where all eight models score ≤3 (Figure 6; Appendix M), cross-modal failures dominate, confirming multi-modal evidence integration as the hardest challenge. Domain analysis (Figure 11, Appendix H) shows cross-modal ACC lags chart reading by 13-16 p in every topic domain. Visual hallucination dominates errors. Models return plausible-looking values absent from the ground-truth chart, concentrated on dense value-reading charts, dumbbell plots, and multi-series comparisons, localising the bottleneck to pixel-level decoding, not reason- ing. Figure-not-found and precision errors account for the remainder; both point to precise 9 Published as a conference paper at COLM 2026 ModelEarlyMiddleLate∆ Proprietary Gemini-3.1-Pro0.7880.7170.784 −0.004 GPT-5.40.4400.4190.468 +0.028 GPT-4o0.4890.4430.443 −0.046 Claude-Sonnet-4.50.4180.3420.301 −0.117 Open-Weight Qwen3.5-VL-122B0.8200.6350.660 −0.160 Qwen3-VL-235B0.6740.5780.693 +0.019 InternVL3-78B0.4510.4600.455 +0.003 Qwen2.5-VL-7B0.1740.1350.188 +0.014 n184237176— Table 4: Chart-reading ACC (τ=6) by position bucket.∆= Late−Early; negative = early- content advantage. See Figure 10 (Appendix G) for the bar chart. quantitative visual-textual alignment as a second bottleneck that prompting alone does not close (Appendix L.7). Effect of Image Presentation: Pages per Strip and Resolution All ablations use GPT- 5 as judge on the full benchmark (Table 5; full details in Appendix N). Gemini-3.1-Pro ACC increases monotonically from 0.369 (1 page/strip) to 0.792 (10 pages), with cross- modal benefiting most (0.339→0.707), confirming multi-page context is essential. 144 DPI is optimal; higher resolution degrades performance due to JPEG compression under the 4 MB API cap. GPT-4o and Claude-Sonnet-4.5 show model-specific optima at 2 and 5 pages respectively, indicating model-specific optimal context density. Gemini-3.1-Pro dominance and potential confounds. The 13.9 p gap between Gemini (0.725) and Qwen3-VL-235B (0.586) warrants caution: SynthDocBench uses web-rendered (HTML/D3.js) charts, which may be stylistically consistent with Gemini’s training distribu- tion. We cannot rule out rendering-familiarity as a partial confounder; future work should validate with alternative rendering backends to isolate long-context reasoning effects. 6 Conclusion We introduce SYNTHDOCBENCH, a long-context visual document understanding bench- mark, measuring VLM performance for the ability to locate multiple key information facts over long-contexts from a given document to reason and answer a multi-step question. It is generated synthetically with controlled difficulty axes by independently varying document length, layout complexity, modality composition, and question type across a combinato- rial design. Our results reveal three concurrent failure modes in frontier VLMs: sharp performance degradation with increasing evidence complexity and reasoning depth, sys- tematic positional sensitivity in which the middle third of a document is the hardest section for four of six models and three of six models show a negative Early→Late trend, with Claude-Sonnet-4.5 exhibiting the steepest monotonic decline (−11.7 p) and Gemini-3.1-Pro showing a distinctive U-shaped recovery (§5), and collapse of precise chart-reading accuracy in long-document contexts. SYNTHDOCBENCH reveals a clear disparity between benchmark performance and genuine long-context visual reasoning ability, serving as a diagnostic plat- form for future model development, providing clean, controlled signals needed to identify, understand, and ultimately improve model capabilities. Code and data will be publicly released upon acceptance. As future work, we plan to extend SYNTHDOCBENCH to broader multimodal reasoning settings, including richer document types (e.g., tables, forms, and mixed-layout reports), 10 Published as a conference paper at COLM 2026 Table 5: Rendering and prompting ablations on the full 200-report benchmark (judge: GPT- 5). OverallChartComplexCross-Modal SettingACCScoreACCScoreACCScoreACCScore (a) Pages per strip (concat-num) — Gemini-3.1-Pro, DPI fixed at 144 1 (single page)0.3694.040.4514.650.3183.480.3393.99 20.4745.120.5095.270.5315.600.3814.50 5 (default)0.7257.190.7597.620.7897.280.6286.65 100.7927.800.8388.320.8317.700.7077.37 (b) Rasterization resolution (DPI) — Gemini-3.1-Pro, concat-num fixed at 5 72 DPI0.6866.970.7537.500.7327.140.5726.25 144 DPI (default)0.7257.190.7597.620.7897.280.6286.65 216 DPI0.6836.920.7297.360.7477.180.5706.23 (c) Prompting strategy — Gemini-3.1-Pro Default0.7257.190.7597.620.7897.280.6286.65 Chain-of-thought0.7027.150.7317.320.7837.630.5916.50 No system prompt0.7237.250.7567.520.8207.710.5926.53 (d) Prompting strategy — GPT-4o Default0.3864.380.4574.440.3604.660.3424.05 Chain-of-thought0.3964.610.4584.490.3934.910.3364.41 No system prompt0.4204.720.5064.630.4195.220.3354.31 (e) Prompting strategy — Claude-Sonnet-4.5 Default0.3143.960.3534.110.3374.280.2503.49 Chain-of-thought0.3694.440.4034.360.3634.490.3404.47 No system prompt0.4224.900.3924.310.5155.730.3574.64 (f) Pages per strip — GPT-4o (DPI fixed at 144) 1 (single page)0.3744.170.4504.260.3374.140.3374.10 20.3964.390.4704.410.3814.630.3374.13 5 (default)0.3864.380.4574.440.3604.660.3424.05 100.3544.210.4384.350.3324.510.2923.75 (g) Pages per strip — Claude-Sonnet-4.5 (DPI fixed at 144) 1 (single page)0.2583.150.3033.510.2362.780.2353.15 20.2883.410.3613.920.2763.210.2273.11 5 (default)0.3143.960.3534.110.3374.280.2503.49 100.3123.700.3784.010.3144.060.2443.03 11 Published as a conference paper at COLM 2026 longer contexts, and more diverse reasoning tasks such as multi-hop aggregation and cross-document grounding. Reproducibility Statement The authors are committed to aiding researchers in reproducing our benchmark and re- sults. The evaluation source code is publicly available athttps://github.com/ServiceNow/ SynthDocBench , and the dataset is publicly available athttps://huggingface.co/datasets/ ServiceNow-AI/SynthDocBench. We also make every effort to disclose experiment hyperpa- rameters wherever necessary throughout the paper, and in the appendix. Ethics Statement Synthetic data and content generation.SYNTHDOCBENCH is constructed entirely from programmatically generated synthetic documents. No human subjects were involved, no personal data were collected, and no real-world documents were reproduced. Document content is grounded in broad, publicly available topic seeds (geopolitics, economics, en- vironmental science, technology, and related domains) and generated by large language models under structured constraints. We reviewed generated content to confirm that it does not contain personally identifying information, hate speech, or other harmful material. Because all textual content is machine-generated and factually non-binding, it should not be treated as authoritative. Model evaluation and API usage.Evaluation is performed via commercial APIs (Gemini, GPT-4o, Claude) and open-weight models under their respective terms of service. No model was fine-tuned on SYNTHDOCBENCH data during the study; all evaluations use frozen model weights. The GPT-5 judge is used solely to score candidate responses against deterministic reference answers; the rubric and prompts are fully disclosed in the appendix to allow independent replication. Environmental impact.Large-scale model evaluation carries a non-trivial computational and energy cost. We limited redundant evaluation runs through careful ablation design (Section 5) and report all configurations transparently so that future work can build on our results rather than repeating them. Estimated compute was approximately 150 GPU-hours for document rendering and 200 API-hours for model inference and judge scoring. Benchmark integrity and Goodhart’s Law.Releasing a controlled benchmark creates the risk that future models overfit to its specific design choices (chart types, layout archetypes, question templates). We mitigate this through (i) a 40% random layout override that prevents spurious topic–layout correlations and (i) programmatic generation that admits straightforward extension. We encourage the community to use SYNTHDOCBENCH as a diagnostic tool and to contribute extensions that preserve the benchmark’s diagnostic validity rather than optimizing against its current instantiation. Intended use.SYNTHDOCBENCH is intended for research on long-context visual docu- ment understanding. It is not intended for deployment in high-stakes decision-making systems. Results on SYNTHDOCBENCH characterize specific, controlled failure modes and should not be generalized uncritically to real-world document understanding performance. References Anthropic. Claude 4.5 Sonnet model card addendum.https://w-cdn.anthropic.com/ 963373e433e489a87a10c823c52a0a013e9172d.pdf, 2025. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025a. 12 Published as a conference paper at COLM 2026 Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Jialong Song, Peng Liu, et al. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025b. Yew Ken Chia, Vernon Toh, Deepanway Ghosal, Lidong Bing, and Soujanya Poria. Puz- zlevqa: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns. In Findings of the Association for Computational Linguistics: ACL 2024, p. 16259–16273, 2024. Yew Ken Chia, Liying Cheng, Hou Pong Chan, Maojia Song, Chaoqun Liu, Mahani Aljunied, Soujanya Poria, and Lidong Bing. M-longdoc: A benchmark for multimodal super-long document understanding and a retrieval-aware tuning framework. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 9244–9261, 2025. Chao Deng, Jiale Yuan, Pi Bu, Peijie Wang, Zhong-Zhi Li, Jian Xu, Xiao-Hui Li, Yuan Gao, Jun Song, Bo Zheng, et al. Longdocurl: a comprehensive multimodal long document benchmark integrating understanding, reasoning, and locating. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1135–1159, 2025. Google. Gemini 3.1 technical report. Technical Report, 2026. Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 2901–2910, 2017. Brenden M Lake and Marco Baroni. Generalization without systematicity: On the com- positional skills of sequence-to-sequence recurrent networks.In Proceedings of the 35th International Conference on Machine Learning, p. 2873–2882, 2018. URLhttps: //arxiv.org/abs/1711.00350. Zhen Li, Duan Li, Yukai Guo, Xinyuan Guo, Bowen Li, Lanxi Xiao, Shenyu Qiao, Jiashu Chen, Zijian Wu, Hui Zhang, Xinhuan Shu, and Shixia Liu. ChartGalaxy: A dataset for infographic chart understanding and generation. arXiv preprint arXiv:2505.18668, 2025. URL https://arxiv.org/abs/2505.18668. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023. Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. MMLongBench-Doc: Benchmarking long-context document understanding with visualizations. Advances in Neural Information Processing Systems, 37:95963–96010, 2024. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, p. 2263–2279, 2022. Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj, Firoz Kabir, Aarya- man Kartha, Md Tahmid Rahman Laskar, Mizanur Rahman, Shadikur Rahman, Mehrad Shahmohammadi, et al. ChartQAPro: A more diverse and challenging benchmark for chart question answering. In Findings of the Association for Computational Linguistics: ACL 2025, p. 19123–19151, 2025. Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. DocVQA: A dataset for VQA on document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 2200–2209, 2021. URL https://w.docvqa.org/. OpenAI, Aaron Hurst, Adam Lerer, Adam Goucher, et al. GPT-4o system card. arXiv preprint arXiv:2410.21276, 2024. Qwen Team.Qwen3.5-VL technical report.https://huggingface.co/Qwen/Qwen3. 5-VL-7B-Instruct, 2025. 13 Published as a conference paper at COLM 2026 Aaditya Singh, Lawrence Chan, Ted Li, Amelia Glaese, et al. OpenAI GPT-5 system card. arXiv preprint arXiv:2601.03267, 2025. Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: A dataset for document visual question answering on multiple images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, p. 13636–13645, 2023. Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. Hierarchical multimodal transform- ers for multipage docvqa. Pattern Recognition, 144:109834, 2023. Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, et al. Needle in a multimodal haystack. Advances in Neural Information Processing Systems, 37:20540–20565, 2024. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025a. Zhaowei Wang, Wenhao Yu, Xiyu Ren, Jipeng Zhang, Yu Zhao, Rohit Saxena, Liang Cheng, Ginny Wong, Simon See, Pasquale Minervini, et al. MMLONGBENCH: Benchmark- ing long-context vision-language models effectively and thoroughly. arXiv preprint arXiv:2505.10610, 2025b. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards AI-complete question answering: A set of prerequisite toy tasks. In Proceedings of the International Conference on Learning Representations, 2016. URL https://arxiv.org/abs/1502.05698. Liyan Xu et al. ChartMuseum: A benchmark for fine-grained chart understanding. arXiv preprint arXiv:2505.13444, 2025. URL https://arxiv.org/abs/2505.13444. Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 5317–5327, 2019. Weiye Zhang et al. VisuLogic: A benchmark for evaluating visual reasoning in multi-step logic problems. arXiv preprint arXiv:2504.15279, 2025. URLhttps://arxiv.org/abs/2504. 15279. Zifeng Zhu, Mengzhao Jia, Zhihan Zhang, Lang Li, and Meng Jiang. Multichartqa: Bench- marking vision-language models on multi-chart problems. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguis- tics: Human Language Technologies (Volume 1: Long Papers), p. 11341–11359, 2025. A Evaluation Pipeline Diagram Figure 8 details the end-to-end evaluation pipeline used to assess all candidate models on SYNTHDOCBENCH. Rendered PDFs are rasterized to page images at 144 DPI, concatenated into 5-page vertical strips, and passed directly to each candidate vision-language model at temperature 0 (vision-only; no OCR or metadata). Each candidate response is then scored against the reference answer by GPT-5 acting as judgeJ , using the Table 6 rubric. 14 Published as a conference paper at COLM 2026 Table 6: GPT-5 judge scoring rubric. Scores of−1 indicate parse failures and are excluded from aggregate statistics. ScoreCriterion 10All key facts present and fully correct 8–9Mostly correct, with only minor omissions or imprecision 6–7Core answer correct, but with some missing detail 4–5Partially correct, with significant gaps or errors 2–3Mostly incorrect or severely incomplete 0–1Incorrect, hallucinated, or contradictory to the reference −1Judge parse failure (excluded) 1 Sy nt hDoc Benc h Ev aluat ion 2 PDFs PDF Images (14 4 DPI) Candidate LLM J udge LLM Char t Based Queries Cross M odal Queries M ult i- Hop Queries QA Pairs c onver t eac h PDF p age to image Predic ted Answ er SCORES Figure 8: Evaluation pipeline. Rendered PDFs are converted to page images at 144 DPI, grouped into concatenated 5-page strips, and supplied directly to candidate models at temperature 0. Candidate answers are then scored against deterministic reference answers by GPT-5 acting as the judge modelJ . B Benchmark Comparison Table 7 situates SYNTHDOCBENCH relative to the most closely related benchmarks and reports frontier model scores on each. Despite strong numbers on DocVQA, ChartQA, and MMLongBench-Doc, coverage is sparse and inconsistent across models—a direct conse- quence of benchmarks built from heterogeneous real corpora with no controlled variation. Crucially, high scores on these benchmarks do not transfer: the same models that approach saturation on DocVQA and ChartQA exhibit substantial failure rates on our controlled sub- sets, demonstrating that SYNTHDOCBENCH exposes failure modes that existing benchmarks cannot surface. C Judge Validation To assess the reliability of GPT-5 as judge, we re-scored the full 1,788-question benchmark for four candidate–judge pairs using Gemini-3.1-Pro and Claude-Sonnet-4.5 as alternative judges. The same two candidates (GPT-4o and Qwen3-VL-235B) are evaluated under both alternative judges, enabling a direct comparison of judge behaviour on identical responses. Table 8 reports overall pairwise Pearson correlation, within-1-point agreement rate, and ACC delta between GPT-5 and each alternative judge; Table 9 further stratifies by question type. GPT-5 and Gemini-as-judge agree to within 3.5 ACC points (r ≥0.94,w 1 ≥0.84) across all tested candidates. Agreement is highest for chart-reading and cross-modal questions (r ≥0.95) and slightly lower for complex multi-hop questions (r ≥0.91), where scoring subjectivity is higher. These gaps are consistent across both candidate models and are well within the benchmark’s bootstrap CI half-widths, confirming that model rankings are robust to judge choice regardless of question type. Claude-Sonnet-4.5 is a systematically lenient judge (+11–16 ACC points overall, rising to+24–27 p on complex questions), consistent with the positivity bias documented for smaller LLM judges, and was excluded on this basis. The GPT-5/Gemini convergence addresses a potential vendor-conflict concern: 15 Published as a conference paper at COLM 2026 ModelDocVQAChartQAMathVistaMMMUMMLongBench-Doc Proprietary Models Gemini-3.1-Pro93.488.574.881.045.1 GPT-4o92.885.763.870.746.3 Claude-Sonnet-4.592.089.072.077.840.1 Open-weight Models Qwen3-VL-235B96.591.285.871.357.0 InternVL3-78B95.1—79.072.224.3 Qwen2.5-VL-7B93.783.868.158.025.1 Table 7: Performance of evaluated VLMs on established visual document understanding benchmarks. — = not publicly reported at time of writing. Table 8: Overall judge agreement statistics on the full SYNTHDOCBENCH (200 reports, 1,788 questions).r: Pearson correlation between score sequences.w 1 : fraction of responses where |score GPT-5 − score alt |≤ 1.∆ACC: ACC(alt judge)− ACC(GPT-5) atτ=6. CandidateAlt. Judgerw 1 ∆ACC GPT-4oGemini-3.1-Pro0.9420.838 −0.035 Qwen3-VL-235BGemini-3.1-Pro0.9600.893 −0.010 GPT-4oClaude-Sonnet-4.50.8810.654 +0.135 Qwen3-VL-235BClaude-Sonnet-4.50.9160.723 +0.156 Gemini-as-judge independently replicates GPT-5’s ranking with a maximum ACC deviation of 3.5 points. Ranking correlation with MMLongBench-Doc. To assess external validity, we com- pute the Spearman rank correlation between SynthDocBench overall ACC and published MMLongBench-Doc scores for the six models evaluated on both benchmarks. The cor- relation isρ =0.657 (r =0.683), indicating moderate positive agreement: models that perform well on our benchmark tend to perform well on MMLongBench-Doc, but the rankings are not identical. Notably, GPT-4o ranks higher on MMLongBench-Doc (#2) than on SynthDocBench (#3), suggesting that our benchmark’s emphasis on precise chart-value extraction and cross-modal alignment surfaces capabilities that MMLongBench-Doc’s more heterogeneous question set partially masks. The imperfect correlation (ρ <1) is itself evidence that SynthDocBench provides complementary diagnostic signal beyond existing benchmarks. Table 9: Judge agreement stratified by question type on the full SYNTHDOCBENCH (200 reports, 1,788 questions).r: Pearson correlation.w 1 : within-1-point agreement.∆ACC: ACC(alt)−ACC(GPT-5) atτ=6. Positive∆ACC means the alternative judge is more lenient than GPT-5. OverallChartComplexCross-Modal CandidateAlt. Judgerw 1 ∆ACCrw 1 ∆ACCrw 1 ∆ACCrw 1 ∆ACC Alt. Judge: Gemini-3.1-Pro GPT-4oGemini0.9420.838 −0.0350.9530.801 −0.0300.9150.869 −0.0540.9500.845 −0.020 Qwen3-VL-235BGemini0.9600.893 −0.0100.9680.910 −0.0220.9330.896 −0.0130.9670.872 +0.005 Alt. Judge: Claude-Sonnet-4.5 GPT-4oClaude0.8810.654 +0.1350.9130.647 +0.0220.8330.610 +0.2710.8950.705 +0.113 Qwen3-VL-235BClaude0.9160.723 +0.1560.9500.824 +0.0690.8490.618 +0.2410.9160.727 +0.159 16 Published as a conference paper at COLM 2026 Qualitative alignment with real-document failures. Consistent with our benchmark’s error taxonomy, the failure modes we identify in the controlled setting correspond to qualitatively similar patterns in MMLongBench-Doc: models that score lowest on our chart- reading subset also show the steepest degradation on MMLongBench-Doc’s figure-heavy questions, and the cross-modal integration failures we isolate (where GPT-4o drops to 0.342 ACC) align with the documented difficulty of questions requiring evidence from non-adjacent pages in real documents. The key distinction is attribution: in real documents these factors co-vary and failures cannot be cleanly assigned to a single cause, whereas SynthDocBench’s factorial design makes each failure mode individually observable. D Bootstrap Confidence Intervals Table 10 reports per-model bootstrap 95% confidence intervals (2,000 resamples, seed 42) for ACC on each question subset; Table 11 breaks these down by difficulty level. All pairwise ACC gaps between adjacent-ranked models exceed their combined CI half-widths, confirming the reliability of the rankings in Table 2. Table 10: Bootstrap 95% CI half-widths (±) for ACC atτ=6, computed over 2,000 resamples (seed 42) on the full SYNTHDOCBENCH (200 reports, 1,788 questions). ModelOverallChartComplexCross-Modal Gemini-3.1-Pro ±0.020 ±0.036 ±0.034 ±0.039 GPT-4o±0.022 ±0.040 ±0.039 ±0.038 Claude-Sonnet-4.5 ±0.022 ±0.041 ±0.037 ±0.034 Qwen3-VL-235B ±0.022 ±0.038 ±0.041 ±0.040 InternVL3-78B ±0.023 ±0.040 ±0.038 ±0.038 Qwen2.5-VL-7B ±0.013 ±0.029 ±0.008 ±0.021 Table 11: Bootstrap 95% CI half-widths (±) for ACC atτ=6 stratified by difficulty level, computed over 2,000 resamples (seed 42). ModelL1L2L3L4L5 Gemini-3.1-Pro ±0.055 ±0.044 ±0.040 ±0.036 ±0.069 GPT-4o±0.063 ±0.048 ±0.044 ±0.042 ±0.053 Claude-Sonnet-4.5 ±0.070 ±0.044 ±0.044 ±0.042 ±0.051 Qwen3-VL-235B ±0.070 ±0.049 ±0.044 ±0.044 ±0.069 InternVL3-78B ±0.070 ±0.046 ±0.042 ±0.043 ±0.056 Qwen2.5-VL-7B ±0.018 ±0.031 ±0.032 ±0.015 ±0.008 E Score Distribution per Model Figure 9 shows the distribution of raw judge scores (0–10) for each model. A bimodal distribution (mass near 0–2 and near 8–10) indicates a model that either fully recovers or fully misses an answer. A unimodal distribution centered at 6–7 indicates consistent but imprecise retrieval. Gemini-3.1-Pro is the only model with a strong right-skewed distribution, while Qwen2.5-VL-7B is concentrated near 0. 17 Published as a conference paper at COLM 2026 0246810 Judge score (010) Claude-Sonnet-4.5 Gemini-3.1-Pro GPT-4o GPT-5.4 InternVL3-78B Qwen2.5-VL-7B Qwen3.5-VL-122B Qwen3-VL-235B ACC threshold (6) Figure 9: Distribution of judge scores (0–10) per model. F Question Quality Control We apply three layers of quality assurance to the generated questions. Numeric recomputation. For every question whose ground-truth answer is a single nu- meric value (e.g. a chart data point, percentage, or count), we independently re-derive the answer from the structured metadataMused during generation, including rawchart_data arrays andkey_factstables. Any question whose recomputed answer differs from the LLM-generated answer by more than a 5% relative tolerance is flagged and either corrected or dropped. This filter removed fewer than 2% of chart-reading questions. Automated consistency filtering. All 1,788 questions pass a rule-based filter checking: (1) the question references an element present inM; (2) the ground-truth answer is non- empty and at least six tokens long; (3) no exact-match overlap between question text and answer text exceeding 40% of answer tokens (to screen out trivially answerable questions); and (4) difficulty levelLis consistent with the number of inferential steps encoded in the reasoning field. Manual review.A random sample of 100 questions (stratified by group type and difficulty level) was reviewed by two of the authors independently. Reviewers rated each question on three dimensions: answerability (is the answer determinable from the document?), an- swer correctness (is the ground-truth answer correct?), and question clarity (is the question unambiguous?). Inter-rater agreement wasκ =0.81. Overall acceptance rate across all three dimensions was 96%. The four rejected questions (two from complex and two from cross-modal) involved ambiguous referents that could not be resolved from the document alone; all four were removed from the final benchmark. G Positional Bias: Bar Chart Figure 10 visualises the ACC per position bucket from Table 4 in the main text. 18 Published as a conference paper at COLM 2026 Early (first third) Middle (second third) Late (final third) Evidence Position in Document 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy (score 6 ) Claude-Sonnet-4.5 Gemini-3.1-Pro GPT-4o GPT-5.4 InternVL3-78B Qwen2.5-VL-7B Qwen3.5-VL-122B Qwen3-VL-235B Figure 10: Chart-reading ACC (τ=6, judge: GPT-5) by evidence position bucket (n=597, 200 reports). Questions bucketed by rel- ative chart positionp = k/Kinto equal thirds. Middle third is hardest for 4 of 6 models; Claude-Sonnet-4.5 steepest decline (−11.7 p). AI & Technology Science & Environment Medicine & Health Economics & Policy Politics & Geopolitics Education 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Accuracy (score 6) 0.80 0.55 0.62 0.67 0.56 0.54 0.53 0.64 0.67 0.76 0.56 0.58 0.37 0.45 0.49 0.52 0.67 0.50 Chart Reading Complex Multi-hop Cross-Modal Overall acc () Figure 11: Gemini-3.1-Pro accuracy (ACC, τ=6) per topic domain, grouped by ques- tion type. Black diamonds = overall accu- racy per domain. Dashed line at 0.5. Based on the 57 domain-annotated reports (513 questions). H Domain Analysis Figure 11 shows Gemini-3.1-Pro’s accuracy stratified by six topic domains and three question types. Domain annotations cover 57 of the 200 reports (original curated set); the remaining 143 are not grouped by domain. Within the annotated subset, cross-modal accuracy falls 13–16 p below chart-reading and complex in every domain; the gap is widest in AI & Technology topics. I Question-Category Breakdown (Full Table) Table 12 provides the complete numerical values visualised in Figure 7. CategoryGemini-3.1 GPT-4o Claude-S-4.5 Qwen3-VL InternVL3 Qwen2.5-7B Chart Reading Value reading0.7840.2710.3820.6480.4370.015 Comparison0.7590.3770.2660.6230.3270.075 Trend / pattern0.7340.7240.4120.6530.6030.397 Cross-Modal Verify with chart0.6160.3990.2780.4800.3180.045 Integrate sources0.6920.2980.2600.5450.2930.030 Compare repr.0.5760.3280.2130.4850.2780.126 Complex Reasoning Historical/timeline0.9100.5530.5080.7840.5980.015 Technical0.6430.1110.1370.4270.1560.000 Impact / reception0.8140.4170.3640.6230.4370.020 Table 12: ACC (τ=6) by fine-grained question category (numerical companion to Figure 7). Best per row in bold. J Performance by Question Subset Figure 12 reports ACC (τ=6) broken down by question subset (chart-reading, complex, cross-modal) for all six models. Cross-modal integration is consistently the hardest subset 19 Published as a conference paper at COLM 2026 ChartComplexCross-Modal 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy (score 6 ) 0.35 0.34 0.25 0.76 0.79 0.63 0.46 0.36 0.34 0.42 0.46 0.39 0.46 0.40 0.30 0.16 0.01 0.07 0.71 0.69 0.56 0.64 0.61 0.50 Claude-Sonnet-4.5 Gemini-3.1-Pro GPT-4o GPT-5.4 InternVL3-78B Qwen2.5-VL-7B Qwen3.5-VL-122B Qwen3-VL-235B Figure 12: ACC (τ=6) by question subset. Cross-modal questions are consistently hardest across all models, confirming modality alignment as the primary bottleneck. across all models, confirming modality alignment as the primary bottleneck in long-context document understanding. 20 Published as a conference paper at COLM 2026 K Question Category Taxonomy Table 14 defines all nine fine-grained question categories used in the category-level break- down of Table 12. Each category belongs to one of the three question subsets and targets a distinct reasoning capability. The difficulty labelL1–L5 is orthogonal to category: any category can appear at any difficulty level depending on the complexity of the evidence chain. LevelModalityOperationExample L1: Direct lookupChart / TableRead-off What is the value for X in chart Y? L2: ComparisonChart / TableCompare Which is larger, A or B? L3: AggregationChart / TableCompute What is the total or average of . . .? L4: Domain reasoningChart + TextInfer Why does X outperform Y given the context? L5: Visual interpretationChart onlyInterpret What does colour C encode in chart Y? Table 13: Difficulty taxonomy for SYNTHDOCBENCH 21 Published as a conference paper at COLM 2026 Table 14: Fine-grained question category definitions for SYNTHDOCBENCH. Categories are grouped by subset; all nine are mutually exclusive within their subset. CategoryDefinition Chart Reading Value ReadingExtract one or more exact numerical values from a single chart or table element (e.g., read a bar height, a cell in a table, or a point on a line). No cross-source reasoning is required; the answer is localized to a single visual element. ComparisonCompare two or more values within the same chart or table to identify the larger, smaller, or closest item, or to compute a difference or ratio. The answer requires reading at least two elements from the same visual. Trend / Pattern Describe the directional behavior, distribution shape, or temporal pattern visible in a chart (e.g., “increasing trend,” “bimodal distribution,” “peaks in Q3”). Exact numerical precision is secondary to correctly characterizing the overall pattern. Cross-Modal Verify with ChartA specific quantitative claim appears in the document text. The model must locate the corresponding chart and confirm, refute, or quantify the claim using visual evidence. Integrate SourcesAnswer a question whose complete response requires combining a textual claim with a specific numerical value readable only from an associated chart. Neither source alone is sufficient; the model must execute a four-step pipeline: locate claim→ find chart→ parse value→ synthesize. Compare RepresentationsThe same phenomenon is described both in text (e.g., a percentage trend) and in a chart (e.g., a line plot). The model must reconcile or contrast these two representations, identifying any discrepancy or providing a richer joint characterization. Complex Reasoning Historical / TimelineSynthesize facts distributed across multiple sections or time periods to con- struct a causal or chronological narrative (e.g., sequence of policy changes, evolution of a metric over decades). Requires integrating at least two tempo- rally separated evidence units. Technical / QuantitativeApply domain-specific knowledge or multi-step arithmetic to evidence re- covered from the document (e.g., derive a compound growth rate, convert units, or evaluate a quantitative tradeoff between two technical approaches). Impact / ReceptionAssess the downstream effects, societal implications, or critical reception of a phenomenon discussed in the document. Requires synthesizing evaluative language from multiple sections rather than extracting a single localized fact. 22 Published as a conference paper at COLM 2026 L Additional Implementation Details This appendix summarizes implementation details that support reproducibility but are not strictly necessary for understanding the main methodological contributions. We include them here to document the design space of document layouts, the structured schema used to encode visualization semantics, the serialized format of generated QA items, and the rendering constraints imposed by the evaluation harness. L.1 Notation Glossary For readability, the main text introduces notation only when needed. Table 15 collects the full symbol glossary in one place. The notation spans the three benchmark stages: synthetic document generation, structured question generation, and image-based evaluation. Table 15: Notation used in the SYNTHDOCBENCH methodology. SymbolTypeDefinition τstringTopic seed used to initialize document generation AsetLayout archetype space a∼ P A (τ)sampleArchetype sampled from the topic-conditioned archetype distribution DdocumentFinal rendered document, represented as HTML and exported PDF V k objectStructured metadata object associated with the k-th visualization inD T k objectStructured metadata object associated with the k-th table inD MmanifestDocument-level QA manifest aggregating allV k andT k qstringNatural-language evaluation question a ∗ stringReference answer derived deterministically fromM ˆ astringModel-predicted answer for q from rendered page image(s) ofD f θ modelCandidate vision-language model parameterized byθ IsequenceRendered page-image sequence supplied to the evaluated model ℓintegerDifficulty level associated with a question JmodelJudge model used to score ( ˆ a, a ∗ ) L.2 Layout Archetypes The report-generation pipeline uses a small but diverse inventory of layout archetypes to vary document structure, rhetorical style, and chart placement strategy. These archetypes control the global page grammar of a report, including whether visualizations are embedded inline, arranged in grids, or surfaced as full-width elements. They also affect the types of auxiliary components that can appear, such as metric strips, pull quotes, or timeline bands. Table 16 lists the archetypes used in SYNTHDOCBENCH together with their distinctive design characteristics. L.3 Visualization Metadata Schema A key property of the benchmark is that every generated visualization is accompanied by a structured metadata object that records its semantic content independently of the rendered pixels. This metadata is used downstream for deterministic QA generation and for consistency validation between rendered charts and their underlying values. The schema is designed to capture both data semantics (e.g., chart values, axes, and derived insights) and presentation-level attributes (e.g., legend presence, color encoding, and highlighted items). Table 17 summarizes the fields stored for each chart. L.4 Question Output Schema Generated QA items are serialized in a structured format so that evaluation can be performed not only at the answer level but also at the level of question family, difficulty, and supporting 23 Published as a conference paper at COLM 2026 050100150200250 Count across all documents Bubble Chart Comparison Table Area Chart Sparkline Grid Treemap Radar Chart Heatmap Histogram Line Chart Dumbbell Chart Scatter Plot Gauge Chart Slope Chart Waterfall Chart Donut Chart Grouped Bar Chart Stacked Bar Chart Bar Chart Pie Chart Lollipop Chart 101 110 112 116 119 121 121 133 138 144 147 151 161 169 182 185 197 199 200 225 Figure 13: Distribution of chart types (top 20 shown). The corpus covers 24 distinct types spanning common (bar, line, scatter) and specialized (dumbbell, sankey, lollipop) forms. Table 16: Layout archetypes used in SYNTHDOCBENCH. ArchetypeViz layoutFont moodMax vizzesDistinctive features MagazineHero + smallerEditorial390vh hero, drop caps, gradient overlays, pull quotes Dashboard2× gridTechnical4KPI strip, metric cards, compact chart grids AcademicInlineScholarly2Abstract box, numbered sections, formal tables EditorialBreakout columnNarrative2Reading progress cues, chapter markers, block quotes InfographicFull-width stackBold4Icon strips, timeline bands, data callouts BrutalistFull bleedRaw2Thick borders, monospace, oversized numbers, stamp labels 24 Published as a conference paper at COLM 2026 Table 17: Visualization metadata schema used by the report-generation pipeline. FieldTypeContents vizIdstringUnique identifier of the form viz-section-index chartTypeenumChart type associated with the rendered visualization titlestringDescriptive chart title axes.x / axes.yobjectAxis label, type, optional unit, and optional range dataarray [ label, value, category, metadata ] — one entry per rendered mark insightsarray Typed facts such as maximum, minimum, comparison, gap, trend, outlier, or proportion visualPropertiesobjectcolorEncoding, colorMap, sortOrder, hasLegend, hasGridlines, high- lightedItems, annotationCount sourceContextstringAttribution to the source passage supplying the underlying values evidence. In addition to the surface question and reference answer, the output schema stores evidence traces such as required facts, chart indices, and required data points, which are useful for debugging, validation, and downstream error analysis. Table 18 describes the fields included in each serialized QA record. Table 18: Structured output schema for generated question-answer pairs. FieldQuestion typesContents questionAllNatural-language question string answerAllReference answer question_typeAllQuestion family/type label difficultyAllDifficulty level L1–L5 required_factsMulti-hopIndices into the extracted evidence list required_facts_textMulti-hopVerbatim supporting evidence fact_sourcesMulti-hopSource section for each supporting fact reasoningAllStepwise evidence-to-answer derivation chart_indexVisual, Cross-modalIndex of the referenced chart chart_titleVisual, Cross-modalTitle of the referenced chart required_data_pointsVisual, Cross-modal Specific label, value pairs required for the an- swer categoryVisual, Cross-modalFine-grained category label L.5 Rendering Hyperparameters The evaluation harness operates on rendered page images rather than HTML or PDF source, so rendering choices directly affect the visual evidence available to the model. The hyperparameters listed in Table 19 define the image construction process, including rasterization resolution, maximum page budget, batching of pages into concatenated strips, and compression constraints required to satisfy API limits across providers. These values therefore influence both evaluation efficiency and the effective difficulty of the visual inference problem. L.6 Inference and Judge Configuration Table 20 summarizes the fixed inference configuration applied to both candidate models and the judge. All models are queried at temperature 0 to ensure fully deterministic outputs across runs. 25 Published as a conference paper at COLM 2026 Table 19: Rendering and compression hyperparameters. ParameterDescriptionValue RESOLUTIONPDF rasterization DPI144 MAX_PAGESMaximum pages extracted per PDF120 CONCAT_NUMPages per concatenated image strip5 COLUMN_NUMColumns per concatenated image1 MAX_IMG_BYTESMaximum compressed image size4 MB MAX_DIMENSIONMaximum pixel dimension (longest side)7,900 px ACC_THRESHOLDMinimum judge score counted as correct6 Table 20: Inference configuration for candidate models and the judge. ParameterValueNotes Judge modelGPT-5Fixed across all experiments Temperature0Applied to both candidates and judge Candidate response2–4 sentencesEnforced via system prompt Judge formatJSON only "score": <0–10>, ... Parse failureScore =−1Excluded from Q valid L.7 Prompts All prompts are shown verbatim as supplied to the model APIs. Text in italics denotes fields populated at runtime. Candidate model — default (concise) prompt. System You are an expert analyst reviewing a PDF report. Answer the question using ONLY what is visible in the provided pages. Be concise and direct — 2–4 sentences maximum. Include specific numbers, dates, or names only if directly relevant. Do not restate the question. Do not explain your reasoning. Just answer. If the answer is not in the document, say: “Not found in the document.” User [Rendered page images supplied as vision input] Question: question Candidate model — chain-of-thought (CoT) prompt. System You are an expert analyst reviewing a PDF report. Answer the question using ONLY what is visible in the provided pages. Think step by step: first identify the relevant chart or section, then extract the key facts, then synthesize your final answer. Include specific numbers, dates, or names where relevant. If the answer is not in the document, say: “Not found in the document.” User [Rendered page images supplied as vision input] Question: question 26 Published as a conference paper at COLM 2026 Candidate model — no system prompt baseline.In the none condition no system prompt is supplied. The user turn contains only the rendered page images followed by the question, identical to the user turns above. Judge model prompt. System You are a factual accuracy evaluator. Score the CANDIDATE ANSWER against the REFERENCE ANSWER for a document-based Q&A task. Rubric (0–10): 10 = all key facts present and correct 8–9 = mostly correct, minor gap or imprecision 6–7 = core correct, some detail missing 4–5 = partial, significant gaps or errors 2–3 = mostly wrong or missing 0–1 = incorrect or hallucinated Respond with ONLY this JSON (no markdown, no extra text): "score": <0–10>, "judgement": "<2–3 sentences on accuracy>" User QUESTION: question REFERENCE ANSWER: gt_answer CANDIDATE ANSWER: candidate_answer Score 0–10 and explain. Return JSON only. M Hard Failure Error Category Definitions SubsetTotalCh.Cx.XM. Hard failures (all 6≤ 3)109371458 Table 21: Hard failure counts by question subset (all six models score≤3). Ch. = chart- reading; Cx. = complex; XM. = cross-modal. Table 22 defines the seven mutually exclusive error categories used to classify hard failures in Section 5. Each category was assigned by GPT-5 based on the question, ground-truth answer, and all model responses. The per-category count breakdown is in Table 21. 27 Published as a conference paper at COLM 2026 Table 22: Error category definitions for the hard failure analysis (Section 5). Categories are mutually exclusive; each failure is assigned exactly one. CategoryDefinition Visual HallucinationThe model confidently reads a wrong value from the chart. The figure is located and the relevant element identified, but the pixel- level value extraction is incorrect (e.g., reports 1,000 ppm instead of 30 ppm). Figure Not FoundThe model cannot locate the referenced figure in the document. It either returns “not found in the document” or retrieves a different, unrelated figure. Precision Error The model extracts the correct chart element but with the wrong exact value, scale, or unit (e.g., reads 29% instead of 27.5%, or reports billions instead of millions). Incomplete RetrievalThe model retrieves part of the required evidence but misses one or more key facts distributed across the document, producing an answer that is partially correct but substantively incomplete. Cross-Modal GroundingBoth the textual claim and the relevant chart are correctly iden- tified in isolation, but the model fails to perform the required quantitative alignment or comparison between the two modali- ties. Reasoning Error The necessary evidence is correctly retrieved from the document, but the model draws a wrong conclusion through a logical or arithmetic mistake in the final inference step. Question / Annotation AmbiguityThe question or ground-truth answer is ambiguous, under- specified, or requires information that is not recoverable from the visible document pages. N Ablation Study Details This section documents the exact experimental configuration used for each ablation study. All ablations use Gemini-3.1-Pro as the candidate model and GPT-5 as judge unless oth- erwise stated. Each condition is evaluated on a representative full 200-report benchmark (1,788 questions), and the same judge prompt is used throughout. N.1 Page Concatenation Ablation The evaluation harness renders each PDF page as a raster image and groups consecutive pages into a single vertical image strip before passing them to the model. Theconcat-num hyperparameter controls how many pages are stacked per strip. ParameterValueNote –concat-num1, 2, 5, 10Pages per image strip –resolution144DPI fixed –prompt-styledefaultConcise system prompt Modelgemini-3.1-proFixed candidate Total images/doc ⌈pages / concat-num⌉Varies Table 23: Concat-num ablation configura- tion. ParameterValueNote –resolution72, 144, 216PDF rasterization DPI –concat-num5Fixed (default) –prompt-styledefaultFixed Max bytes/image4 MBJPEG recompressed if exceeded Max dimension7,900 pxDownscaled if exceeded Table 24: DPI ablation configuration. At 72 DPI the resulting strip is typically around 1,200 px tall; at 144 DPI around 2,400 px; at 216 DPI around 3,600 px before any compression rescaling. Atconcat-num=1each page is sent individually, maximising resolution per image but fragmenting document context across many turns. Atconcat-num=10, up to ten pages are merged into one tall strip, providing wider context at reduced per-line resolution. The defaultconcat-num=5balances these trade-offs. 28 Published as a conference paper at COLM 2026 N.2 Rendering Resolution Ablation PDF rasterization converts each page to a pixel image at the specified DPI. Higher DPI produces finer text and chart detail but larger file sizes that may trigger per-image byte-limit compression. N.3 Prompting Strategy Ablation Across all three models, the no-prompt condition achieves the highest complex-question ACC (Gemini+5.3 pts, GPT-4o+5.9 pts, Claude+17.0 pts over default), with Claude’s complex ACC jumping from 0.380 to 0.550, suggesting the concise-answer framing actively constrains multi-step reasoning. CoT yields modest overall gains (Gemini:+1.6 pts; Claude: +3.0 pts) but reduces GPT-4o chart-reading ACC (0.520→0.468), where longer outputs introduce hallucinated intermediate steps. Chart-reading and cross-modal ACC vary by at most 5 points across all strategies, confirming these subsets are driven by visual perception rather than output formatting. ParameterValuesNote Judge modelsGPT-5, Gemini, ClaudeSame rubric/JSON Candidate modelsGPT-4o, Qwen3, InternVL3Pre-scored answers ACC thresholdτ4, 5, 6, 7, 8GPT-5 judge only Temperature0All judges Table 25: Judge sensitivity ablation configu- ration. StyleSystem promptDescription defaultConcise (App. L.7)2–4 sentence answer cotCoT (App. L.7)Step-by-step reasoning noneNoneNo system prompt Table 26: Prompting ablation configuration. N.4 Judge Sensitivity Ablation Judge sensitivity results are in Table L.7. Across all three models, the no-prompt condition achieves the highest complex-question ACC (Gemini+5.3 pts, GPT-4o+5.9 pts, Claude +17.0 pts over default), with Claude’s complex ACC jumping from 0.380 to 0.550, suggesting the concise-answer framing actively constrains multi-step reasoning. CoT yields modest overall gains (Gemini:+1.6 pts; Claude:+3.0 pts) but reduces GPT-4o chart-reading ACC (0.520→0.468), where longer outputs introduce hallucinated intermediate steps. Chart- reading and cross-modal ACC vary by at most 5 points across all strategies, confirming these subsets are driven by visual perception rather than output formatting. GPT-4oQwen3-VLInternVL3 Judge model GPT-5 (default)0.4340.5350.354 Gemini-3.1-Pro0.4170.5220.363 Claude-Sonnet-4.50.6070.7050.573 Thresholdτ τ = 40.6270.6970.568 τ = 50.4830.5750.398 τ = 6 (default)0.4340.5350.354 τ = 70.3600.4510.311 τ = 80.2990.4130.269 Table 27: Judge sensitivity results. Top: ACC for three candidates under three judge models. Bottom: ACC under GPT-5 at thresholdsτ ∈ 4, 5, 6, 7, 8. Rankings are fully preserved across all conditions. O Qualitative Model Response Examples We present four representative examples from SYNTHDOCBENCH spanning the three ques- tion subsets and dominant failure modes from Section 5. Each example shows (i) a zoomed- 29 Published as a conference paper at COLM 2026 in crop of the relevant evidence page, (i) the question and ground-truth answer, and (i) the annotated response from a representative subset of evaluated models, where green marks correct facts and red marks hallucinated or wrong values. 30 Published as a conference paper at COLM 2026 Example 1 — Chart Reading L1Mixed: Value Reading from Horizontal Bar Figure 14 from the document (p.19) — OpenAI AI Text Classifier Performance on AI Text. The bar shows 26% True Positive (Identified) and 74% False Negative (Missed). Question (Chart Reading, L1 — AI in the Classroom) According to Figure 14 (‘OpenAI AI Text Classifier Performance on AI Text (%)’), what percentage of AI- generated text resulted in a False Negative (Missed)? Ground Truth 74% of AI-generated text resulted in a False Negative (Missed). Gemini-3.1-Pro10/10 : According to Figure 14, 74% of AI-generated text resulted in a False Negative (Missed). GPT-4o0/10 : According to Figure 14, 43% of AI-generated text resulted in a False Negative (Missed). Claude-Sonnet-4.510/10 : Figure 14 shows that 74% of AI-generated text was classified as a False Negative (Missed), meaning the classifier failed to identify it as AI-written. Qwen3-VL-235B0/10 : Based on Figure 14, 26% of AI-generated text resulted in a False Negative (Missed). Qwen2.5-VL-7B0/10 : According to Figure 14, 26% of AI-generated text resulted in a False Negative (Missed). InternVL3-78B0/10 : According to Figure 14, the percentage of AI-generated text that resulted in a False Negative (Missed) is 16%. The chart clearly labels both segments: 26% (True Positive, blue) and 74% (False Negative, gray). Only Gemini and Claude read the correct gray segment. GPT-4o hallucinates 43%—a value absent from the chart. Both Qwen models confuse the two segments, reading the True Positive bar (26%) as the answer. InternVL3 hallucinates 16%. 31 Published as a conference paper at COLM 2026 Example 2 — Cross-Modal L2Failure: Visual Hallucination Figure 1 from the document (p.5) — Distribution of AI-Generated Code in Enterprise Files. The 40–50% bin has 120 files; the 50–60% bin has 40 files. Question (Cross-Modal, L2 — AI-Native Software Dev.) The text describes the distribution dropping ‘sharply after the 40–50% mark.’ How does Figure 1 quantify this as the exact file frequency drop between the 40–50% and 50–60% bins? Ground Truth 40–50% bin: 120 files→50–60% bin: 40 files. Drop of 80 files (66.7% reduction). Gemini-3.1-Pro0/10 : In Figure 1, the 40–50% bin shows a frequency of 200 files. The subsequent 50–60% bin drops to a frequency of 50 files. This represents an exact numerical drop of 150 files between the two bins. GPT-4o0/10 : Figure 1 indicates that the file frequency drops from approximately 17% in the 40–50% bin to approximately 8% in the 50–60% bin, representing a numerical decline of about 9 percentage points. Claude-Sonnet-4.51/10 : Not found in the document. While Figure 1 shows the distribution and the text describes a sharp drop after the 40–50% mark, the specific numerical values for file frequency in the two bins are not provided in the visible portions of the document. Qwen3-VL-235B2/10 : Figure 1 shows the frequency of files for the 40–50% bin is approximately 175. For the next bin, 50–60%, the frequency drops sharply to approximately 50. This represents a numerical drop of about 125 files. Qwen2.5-VL-7B0/10 : The figure shows that the number of files drops from approximately 250 in the 40–50% bin to around 150 in the 50–60% bin. InternVL3-78B0/10 : Not found in the document. Every model hallucinates a different plausible-looking but wrong pair of bar heights. GPT-4o conflates file counts with axis percentages. Claude and InternVL3 decline to answer despite the chart being visible on p.5. No model reads the correct 120→40 counts. 32 Published as a conference paper at COLM 2026 Example 3 — Cross-Modal L3Failure: Figure Not Found Figure 3 from the document (p.8) — GitHub Copilot Productivity Study. Dev 6 (control) reaches 180 min; Dev 6 (treatment) is at 85 min. Question (Cross-Modal, L3 — AI-Native Software Dev.) The text states the longest task time in the GitHub Copilot study was 180 min by a developer in the con- trol group. Using Figure 3, which specific developer recorded this time, and what was their time in the Treatment Group? Ground Truth Dev 6 (Control: 180 min→ Treatment: 85 min). Gemini-3.1-Pro0/10 : Not found in the document. GPT-4o0/10 : The document does not specifically identify which developer recorded the 180-minute maxi- mum time nor their completion time in the Treatment Group. Claude-Sonnet-4.50/10 : Not found in the document. While the document mentions the 180-minute maximum completion time in the control group, it does not identify which specific developer recorded this time or provide their individual completion time in the treatment group. Qwen3-VL-235B0/10 : Not found in the document. Qwen2.5-VL-7B0/10 : Not found in the document. InternVL3-78B0/10 : Not found in the document. Figure 3 is clearly visible on p.8 with per-developer labels on the x-axis (Dev 1 through Dev 10). The highest gray dot at 180 min sits at Dev 6. All six evaluated models fail to localise and resolve the individual data point label, returning a uniform “not found” despite the chart being visually present. 33 Published as a conference paper at COLM 2026 Example 4 — Complex Multi-Hop L5Failure: Incomplete Retrieval Two of three evidence figures from p.5 of the gender equality document: Figure 3 (Reproductive Rights Trajectories, top) and the Congress representation figure (bottom). A third chart on the same page shows child marriage rates 2013–2023. Question (Complex Multi-Hop, L5 — Gender Equality) How did women’s representation in US Congress change after the 1992 ‘Year of the Woman’? What was the exact decline in global child marriage rates 2013–2023? How does the US reproductive rights 50-point drop compare to changes in Ireland and Argentina over 2017–2023? Ground Truth House: 24→48 (doubled). Senate: 2→6 (tripled). Child marriage: 25→20 (drop of 5). US: 80→30. Ireland: 10→90. Argentina: 20→85. Gemini-3.1-Pro4/10 : Following the 1992 election, voters elected 4 new female senators (bringing the Senate total to 6) and 24 new women to the House of Representatives. Between 2013 and 2023, global child marriage rates declined from 25% to 19%. During the 2017–2023 timeframe, the 50-point drop in the US Reproductive Rights Index (from 75 to 25) contrasted with a 60-point increase in Ireland (from 20 to 80) and a 45-point increase in Argentina (from 30 to 75). GPT-4o2/10 : Following the 1992 ‘Year of the Woman’ election, the number of women in the U.S. Congress doubled. The global child marriage rate declined by 15% between 2013 and 2023. While the U.S. reproductive rights index dropped by 50 points from 2017 to 2023, during the same timeframe, Ireland’s index rose by 25 points and Argentina’s by 30 points. Claude-Sonnet-4.50/10 : Not found in the document. Qwen3-VL-235B3/10 : Following the 1992 ‘Year of the Woman’, the number of women in the US Congress increased from 4 to 6. Global child marriage rates declined from 25% in 2013 to 20% in 2023, a 5 percentage point drop. The US reproductive rights index dropped 50 points from 2017 to 2023, while Ireland’s index rose from 70 to 90 and Argentina’s rose from 60 to 85 during the same period. Qwen2.5-VL-7B2/10 : The number of women in the US Congress increased from 4 to 44 after the 1992 ‘Year of the Woman’ election. The global child marriage rate declined by approximately 25 percentage points between 2013 and 2023. The US reproductive rights index dropped by 50 points, while Ireland saw a 10-point increase and Argentina experienced a 10-point decrease during the same period. InternVL3-78B4/10 : 34 Published as a conference paper at COLM 2026 Following the 1992 ‘Year of the Woman’ election, the number of women elected to the U.S. House of Representatives increased by 24, marking a then-record high. The global child marriage rate declined by 12 percentage points between 2013 and 2023. The U.S. reproductive rights index experienced a 50-point drop, contrasting with Ireland and Argentina, which saw significant improvements in their index scores during the 2017–2023 period. The question requires six independent statistics from three charts on the same page. Gemini retrieves the Senate total and House count but uses wrong base scores for all three countries. Qwen3 correctly reads the child marriage drop and US delta but has wrong starting values for Ireland and Argentina. GPT-4o correctly identifies the Congress doubling and US 50-point drop but reports wrong deltas for the other countries. InternVL3 correctly retrieves the House count and US drop but vaguely describes Ireland/Argentina without numbers. Claude fails to retrieve any evidence. The Ireland (10→90) and Argentina (20→85) base scores are consistently missed. 35

Instant research discovery