Paper deep dive
Query-Driven Multimodal Information Extraction from Long Documents
Yikai Gao, Ding Xia, Xi Yang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/30/2026, 2:07:00 AM
Summary
This paper introduces a new task called query-driven image-text joint extraction from multimodal long documents, where models must extract specific textual attributes and their corresponding image bounding boxes based on user queries. The authors propose ITJoint, a manually annotated benchmark consisting of 2,455 pages of domain-specific documents, 316 queries, and 910 answer instances. They also design Q2IT, a multi-agent collaborative framework comprising Evidence, Page, and Figure agents to handle evidence collection, page selection, and target-image localization. Experiments show that Q2IT significantly outperforms standalone Vision-Language Models (VLMs), though a performance gap remains.
Entities (8)
Relation Signals (7)
Q2IT → containscomponent → Page Agent
confidence 95% · Q2IT consists of three progressively collaborating agents... the Page Agent... and the Figure Agent
Q2IT → containscomponent → Figure Agent
confidence 95% · Q2IT consists of three progressively collaborating agents... the Figure Agent
Q2IT → containscomponent → Evidence Agent
confidence 95% · Q2IT consists of three progressively collaborating agents: the Evidence Agent... the Page Agent... and the Figure Agent
DocVQA → contrastwith → Query-Driven Multimodal Information Extraction
confidence 93% · existing paradigms like DocVQA primarily focus on generating textual answers... rather than outputting query-specific textual attribute values and corresponding images
ITJoint → usedforevaluation → Q2IT
confidence 92% · evaluate it alongside standalone VLMs... Q2IT significantly improves performance on ITJoint
Q2IT → outperforms → Vision-Language Models
confidence 90% · experiments show that standalone VLMs struggle with this task, while Q2IT significantly improves performance on ITJoint
Figure Agent → usesmodel → Grounding DINO
confidence 88% · The Figure Extractor uses Grounding DINO (Liu et al., 2024) to obtain candidate boxes
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In domain-specific multimodal long documents, images and text jointly convey complex knowledge that cannot be fully captured by plain text alone. However, existing paradigms like DocVQA primarily focus on generating textual answers or localizing evidence regions, rather than outputting query-specific textual attribute values and corresponding images. To address this gap, we propose query-driven image-text joint extraction from long documents, requiring models to output query-requested textual attribute values and corresponding image bounding boxes. Based on challenges related to both user intent and document content, we designed a two-level taxonomy that operates at the query and instance levels. Further, we construct ITJoint, the first high-quality, manually annotated benchmark for this new task, comprising 2,455 pages of domain-specific documents with numerous non-decorative images, 316 queries, and 910 answer instances. Finally, we evaluate representative standalone Vision-Language Models from different providers and further design Q2IT, a multi-agent collaborative framework consisting of three progressively collaborating agents for evidence collection, page selection, and target-image localization. Using a joint evaluation approach that assesses both text extraction and image localization, our experiments show that standalone VLMs struggle with this task, while Q2IT significantly improves performance on ITJoint, although a substantial gap remains toward perfect results.
Tags
Links
- Source: https://arxiv.org/abs/2608.22214v1
- Canonical: https://arxiv.org/abs/2608.22214v1
Trouble viewing inline? Open PDF directly →
Full Text
61,905 characters extracted from source content.
Expand or collapse full text
Query-Driven Multimodal Information Extraction from Long Documents Yikai Gao 1 , Ding Xia 2 , Xi Yang 1,† 1 School of Artificial Intelligence, Jilin University 2 Graduate School of Information Science and Technology, The University of Tokyo ykgao25@mails.jlu.edu.cn dingxia1995,earthyangxi@gmail.com † Corresponding author Abstract In domain-specific multimodal long documents, images and text jointly convey complex knowl- edge that cannot be fully captured by plain text alone. However, existing paradigms like DocVQA primarily focus on generating textual answers or localizing evidence regions, rather than outputting query-specific textual attribute values and corresponding images. To address this gap, we propose query-driven image-text joint extraction from long documents, requir- ing models to output query-requested textual at- tribute values and corresponding image bound- ing boxes. Based on challenges related to both user intent and document content, we designed a two-level taxonomy that operates at the query and instance levels. Further, we construct ITJoint, the first high-quality, manually anno- tated benchmark for this new task, comprising 2,455 pages of domain-specific documents with numerous non-decorative images, 316 queries, and 910 answer instances. Finally, we evalu- ate representative standalone Vision-Language Models from different providers and further design Q2IT, a multi-agent collaborative frame- work consisting of three progressively collab- orating agents for evidence collection, page selection, and target-image localization. Us- ing a joint evaluation approach that assesses both text extraction and image localization, our experiments show that standalone VLMs strug- gle with this task, while Q2IT significantly im- proves performance on ITJoint, although a sub- stantial gap remains toward perfect results. 1 Introduction As foundation models have learned from massive amounts of web-scale data, their development is increasingly turning toward specialized documents that preserve knowledge accumulated through long- term human practice (Muennighoff et al., 2023; Vil- lalobos et al., 2024). Such domain-specific multi- modal long documents contain rich textual descrip- tions and images whose semantics are often tightly coupled and cannot be fully captured by plain text alone. However, transforming this unstructured multimodal information into structured image-text pairs still largely relies on time-consuming manual annotation. Therefore, query-driven image-text joint extraction provides a practical way to sat- isfy researchers’ targeted needs and to transform domain-specific long documents into high-quality multimodal datasets (Gadre et al., 2023). Document visual question answering (DocVQA) has made significant progress (Mathew et al., 2021; Tito et al., 2023; Van Landeghem et al., 2023; Ma et al., 2024; Deng et al., 2025), but it remains pri- marily designed to produce textual answers, even when visual grounding is introduced to localize sup- porting evidence regions (Yu et al., 2025). Besides, existing image-text extraction methods mostly rely on predefined rules for exhaustive extraction and lack query-driven, on-demand capabilities (Clark and Divvala, 2016; Li et al., 2019; Siegel et al., 2018; Song et al., 2025; Tang et al., 2024; Pang et al., 2024). Moreover, existing DocVQA datasets are largely built from academic papers with stan- dardized layouts or general documents where im- ages are mainly decorative or auxiliary, leaving real-world domain-specific long documents with complex layouts and tightly coupled image-text relations underexplored. To this end, we formally define a new task: query-driven image-text joint extraction from mul- timodal long documents. Unlike DocVQA, which asks the model to answer a question about a docu- ment, our task asks the model to construct query- specific multimodal instances by extracting textual descriptions together with their corresponding tar- get images (Figure 1). In other words, given a user query and a long document, the model is required to retrieve the requested textual attributes and return the corresponding image bounding boxes. How- ever, this task poses challenges from both user in- tent and the document itself. From the perspective 1 arXiv:2608.22214v1 [cs.AI] 23 Aug 2026 Please list the names and images of all porcelain wares from the Kangxi period of the Qing Dynasty. User Query: Document: · Answer DocVQA TaskBbox-DocVQA TaskOur Task Colorful Vase with Pattern of Flowers and Birds, Gourd-shaped Famille-rose Vase with ... Text Only? Where's the image? Colorful Vase with Pattern of Flowers and Birds, Gourd-shaped Famille-rose Vase with ... · · · Text Bbox? Need Image Bbox! 1 Colorful Vase with Pattern of Flowers and Birds Gourd-shaped Famille-rose Vase with Pattern of Pine Tree, Crane and Cock 2 · · · Exactly! Existing BenchmarksOur Benchmark Academic papers— Standardized layout General documents — Dispensable / Decorative images Insufficient for our task! Intra-page Non-linear LayoutAsymmetric Mapping Cross-page Dispersal 30 31353032114 Indispensable images Real-world complexity Sufficient! (a) Gap between our task and existing DocVQA tasks. (b) Document-level gap between ITJoint and existing DocVQA datasets. Figure 1: Comparison with existing tasks and datasets. (a) DocVQA tasks mainly return textual answers or evidence regions, while our task requires textual attributes and corresponding images. (b) Existing datasets mainly rely on academic papers or general documents, which are insufficient for evaluating our task. of user intent, real-world queries are not limited to clearly specified single-instance queries, but may also involve broadly referential set queries with an uncertain number of targets, requiring methods to balance precise extraction with comprehensive recall. From the document perspective, first, im- ages and text within a page often go beyond lin- ear arrangements and exhibit non-linear and multi- layer nested layouts, making rule-based associa- tion methods fragile in real-world domain-specific documents. Second, images and their descriptive text may be dispersed across pages rather than ap- pearing adjacently on the same page, requiring cross-page tracking and aggregation with multi- level context. Third, image-text mappings often go beyond one-to-one correspondences, including one-to-many and text-without-image cases, requir- ing robust mapping and hallucination suppression. To support systematic study and reliable evaluation, we design a two-level taxonomy at both the query and instance levels, and construct ITJoint, a high- quality manually annotated benchmark for query- driven image-text joint extraction from domain- specific multimodal long documents. Current approaches to address this task typically involve either standalone Vision-Language Models (VLMs) or carefully designed agent systems tai- lored for this specific problem. For the VLM-based methods, we selected four state-of-the-art models from different providers. Inspired by existing meth- ods, we also designed Q2IT, a multi-agent collabo- rative framework that incorporates various modules aimed at enhancing performance on our task. Q2IT consists of three progressively collaborating agents: the Evidence Agent, which retrieves relevant tex- tual information and image clues from the entire document; the Page Agent, which locates the target pages based on these clues; and the Figure Agent, which detects and crops the target images within those pages. To evaluate all methods comprehen- sively, we devised a joint evaluation approach that assesses both image localization and text extrac- tion. Results indicate that current standalone VLMs struggle to handle our task effectively. In contrast, the Q2IT workflow significantly improves perfor- mance, demonstrating greater capability in tackling our problem. However, there remains a substantial gap toward achieving perfect results, highlighting considerable room for improvement in both VLMs and multi-agent system designs. In summary, our contributions are as follows: •We define query-driven image-text joint ex- 2 traction from multimodal long documents, requiring models to return query-requested textual attributes and corresponding image bounding boxes. We further analyze its chal- lenges from both user intent and the document itself, and design a two-level taxonomy for systematic evaluation. •We construct ITJoint, the first high-quality manually annotated benchmark for this task. ITJoint targets real-world image-text coupling scenarios in domain-specific long documents, containing 2,455 document pages with an av- erage of 1.77 non-decorative images per page, 316 queries, and 910 answer instances. •We design Q2IT, a task-specific workflow for evidence collection, page selection, and target- image localization, and evaluate it alongside standalone VLMs. Joint assessment shows that Q2IT improves over direct inference, while substantial room remains for further im- provement. 2 Related Work Document Visual Question Answering and Benchmarks. DocVQA aims to answer user questions from document images (Mathew et al., 2021). Related benchmarks have evolved from single-page to multi-page documents (Tito et al., 2023), and further to multidomain, visually rich, and long-context multimodal document set- tings (Van Landeghem et al., 2023; Ma et al., 2024; Deng et al., 2025). Methods have also progressed from OCR- and layout-based document understand- ing models (Xu et al., 2021; Huang et al., 2022) to OCR-free models (Kim et al., 2021) and document- oriented multimodal large language models (Ye et al., 2023). BBox-DocVQA further introduces evidence bounding boxes for answer grounding (Yu et al., 2025). Nevertheless, existing DocVQA still centers on textual answers: even with bound- ing boxes, localization mainly serves as evidence grounding rather than extracting query-targeted textual attributes and corresponding image coordi- nates. Existing benchmarks are also dominated by structured academic papers or general documents where images are mostly auxiliary. Therefore, ex- isting DocVQA methods and benchmarks remain insufficient for query-driven image-text joint ex- traction. Image-Text Pair Extraction from Documents. Another line of work extracts images and associ- ated text from documents, including figure-caption extraction in scientific papers and multimodal data collection from domain-specific catalogs. Meth- ods such as PDFFigures 2.0 (Clark and Divvala, 2016), PDFigCapX (Li et al., 2019), and DeepFig- ures (Siegel et al., 2018) extract figures, captions, and coordinates from academic papers, while later work studies subfigure-subcaption alignment in compound figures (Song et al., 2025). For domain- specific catalogs, PDFChatAnnotator (Tang et al., 2024) supports multimodal data collection from PDF-format catalogs through human-LLM collab- oration, and VLM-based methods have been used to collect artifact images and attributes from ar- chaeological catalogs (Pang et al., 2024). How- ever, existing methods typically rely on predefined targets, fixed fields, or human-collaborative work- flows, and cannot select target instances on demand from open-ended user queries. Multi-Agent Frameworks for Complex Tasks. Multi-agent frameworks have been widely used to decompose complex tasks into collaborative sub- processes. General systems such as CAMEL (Li et al., 2023), AutoGen (Wu et al., 2024), and MetaGPT (Hong et al., 2024) support planning, execution, and refinement through role specializa- tion, conversation, or procedural intermediate arti- facts. In downstream applications, P2P (Sun et al., 2025) and Paper2Poster (Pang et al., 2026) decom- pose paper-to-poster generation into content pars- ing, visual organization, layout planning, and iter- ative refinement; AutoPrep (Fan et al., 2024) uses multiple agents for question-driven data prepara- tion; and oracle bone script interpretation adopts an agent-driven multimodal knowledge augmentation framework for domain-specific reasoning (Zhang et al., 2026). Inspired by these studies, we model image-text joint extraction from multimodal long documents as a staged collaborative process with multi-agents. 3 Task Definition and Benchmark 3.1 Task Formulation Given a user queryqand a long documentD = p i N i=1 , the model is required to return a set of answer instances: R = z j J j=1 , z j = (t j ,v j ).(1) Each answer instancez j consists of a textual at- tribute valuet j satisfying the query condition and 3 161 70 55 30 Single-Inst. Pos. Single-Inst. Neg. Set Pos. Set Neg. 166 567 177 1-to-N 1-to-0 1-to-1 Query TypesModality Cardinality 596 137 Visual Layout Compound Independent 438 295 Spatial Span Cross-page Intra-page 488 245 Alignment Clues Explicit Implicit Figure 2: Distributions of ITJoint under the query-level and instance-level taxonomies. an associated image set v j : v j = (π j,m ,b j,m ) M j m=1 ,(2) whereπ j,m andb j,m denote the page index and bounding box of them-th associated image, re- spectively. When textual evidence for an instance exists but no corresponding image is present, the system must still returnt j and explicitly output an empty image set, i.e.,v j = ∅, rather than omitting the instance or hallucinating an image match. 3.2 Challenge Taxonomy To systematically characterize the challenges of producingRand support fine-grained evaluation, we define a two-level taxonomy overqandz j = (t j ,v j ). Query-level taxonomy. To characterize chal- lenges from user intent, we define two query-level categories. Single-instance queries requireJ = 1 and request image-text information for one specific answer instance. Set queries require all answer instances satisfying a given condition, whereJ is not known in advance. Based on this distinc- tion, we further refine the conventional negative- query setting in existing benchmarks (Ma et al., 2024; Deng et al., 2025), where unanswerability is caused by the complete absence of textual evi- dence, into a task-specific negative setting where textual evidence exists but the corresponding image is missing. Specifically, positive queries require their an- swer instances to contain both the requested tex- tual attributes and corresponding images. Nega- tive queries differ across the two query types. For single-instance queries, the target instance has tex- tual evidence but no corresponding image, result- ing in a negative answer instance withv j = ∅. For set queries, image-bearing and text-only in- stances coexist inR, requiring the system to extract valid image-text pairs while explicitly identifying instances with v j = ∅. Instance-level taxonomy. To characterize chal- lenges from the document itself, we annotate each z j = (t j ,v j )with fine-grained labels. For modality cardinality, instances are labeled according toM j in Eq. 2: 1-to-0 ifM j = 0, 1-to-1 ifM j = 1, and 1-to-N ifM j > 1. Instances labeled as 1-to-0 con- tain textual evidence but no corresponding image and are therefore negative answer instances, while 1-to-1 and 1-to-N instances are positive answer in- stances. For positive answer instances, we further anno- tate three dimensions that describe howt j andv j are associated in the document. For visual layout, instances are labeled as independent or compound, distinguishing whether the target image inv j is a standalone image or is nested with other images in a compound image. For spatial span, instances are labeled as intra-page or cross-page according to whether the textual evidence and all image pages in v j together involve a single page or multiple pages. For alignment clues, instances are labeled as ex- plicit or implicit. Explicit alignment indicates that the document contains clear textual markers that directly establish the correspondence betweent j andv j , whereas implicit alignment indicates that such markers are missing or insufficient, requiring the system to infer the correspondence using addi- tional signals such as layout position, reading order, and visual content. Appendix A.2 provides visual examples of these instance-level labels. 3.3 ITJoint Benchmark Document Collection. We collect long docu- ments from multiple domain-specific fields, includ- ing medicine, agriculture, archaeology, biology, and art. These documents contain diverse image- text layouts, and include rich non-decorative core images. In addition, the selected documents cover both English and Chinese, enabling evaluation of models’ applicability across different language set- tings. Document examples are provided in Ap- pendix A.3. Annotation Workflow.We adopt a human-LLM collaborative annotation workflow. First, we ex- 4 Document: · · · According to the common histochemical stains table, please provide the specific color of the deposits in Granular corneal dystrophy when using the Masson trichrome stain, and provide all corresponding relevant images demonstrating this disease as referenced in the table. Query: Evidence Agent Evidence Planner Subquery 1:According to the common histochemical stains table, please provide the ... Strategy 1:single- Instance 1.Single-Instance retrieval 2.Set retrieval 3.Chapter Location Retrieved context chunks (source_page:image_29.png):M uscle: red dystrophy deposits: corneal dystrophy (see Fig 2-6I and Fig 5-22C)Oil ... Evidence Integrator q:Subquery 1, id:Granular corneal ..., val:red, clue:[Fig 2-6I,Fig 5-22C], src:[image_29.png] Evidence Checker Page Agent Clue Analyzer Keyword 1:[Fig 2-6I, Fig 2-6, 2-6I, 2-6] Tool 1:Keyword Match Keyword 2:[Fig 5-22C, Fig 5-22, 5-22C, 5-22] Tool 2:Keyword Match 1.Keyword Match 2.Page Mapping 3.Relative Offset 4.Source Fallback Candidate pages Keyword 1: image_27.png Keyword 2: image_110.png spatial desc: third row right column ... image_27.png spatial desc: bottom-right panel ... image_110.png Page Checker Figure Agent Figure ExtractorFigure Binder q:Subquery 1, id:Granular corneal ..., val:red, bbox-index:6 page:image_27.png bbox-index:1 page:image_110.png Figure Checker Text Answer: Red Figure Answer: T-I Joint Evaluation Figure 3: Overview of Q2IT: a three-agent framework for query-driven image-text joint extraction. The Evidence Agent performs evidence collection by collecting textual attribute values and image-reference clues, the Page Agent performs page localization by identifying target pages and coarse spatial descriptions, and the Figure Agent completes image-text binding by selecting target image regions and producing final answer instances. Table 1: Statistics of our collected ITJoint. StatisticNumber Documents Total pages2,455 Avg. pages59.88 Max / Min pages178 / 22 Avg. images / page1.77 Set Queries Total instances679 Avg. instances / query7.99 Max / Min instances / query62 / 1 Pos. / neg. instances in neg. sets148 / 107 1-to-N Instances Total images450 Avg. images / instance2.71 Max / Min images / instance13 / 2 tract the plain text from long documents and use a large language model to generate candidate queries in batches according to the query-level taxonomy. To improve annotation efficiency, we develop IT- Label, a visual annotation tool specifically de- signed for this task. During formal annotation, an- notators use ITLabel to manually clean candidate queries and annotate answers following predefined annotation guidelines. Specifically, annotators read through the docu- ment page by page to identify all answer instances satisfying the query conditions, with particular care taken to avoid missing instances for set queries whose target count is uncertain. They also assign the corresponding query-level and instance-level labels to each answer instance. During this pro- cess, candidate queries that do not satisfy the pre- defined rules are either filtered out or revised ac- cording to the actual document content. More de- tails about the ITLabel annotation tool, annotation guidelines, and annotation examples are provided in Appendix A.1. Dataset Statistics. Figure 2 and Table 1 sum- marize the statistics of ITJoint. ITJoint contains 2,455 domain-specific document pages, with an av- erage document length of 59.88 pages and 1.77 non-decorative images per page on average. It includes 316 queries and 910 answer instances. Among them, set queries correspond to 679 an- swer instances, with an average of 7.99 instances per set query and a maximum of 62 instances for a single query. For 1-to-N cases, the 166 instances involve 450 images in total, with an average of 2.71 images per instance and a maximum of 13 images. 4 Method As shown in Figure 3, given a queryqand a docu- mentD, Q2IT progressively constructs the answer setRthrough three stages: evidence collection, page selection, and target-image localization, han- dled by the Evidence Agent, Page Agent, and Fig- ure Agent, respectively. Before the agent workflow, Q2IT renders each page ofDas a 144-DPI image, extracts page- level text with an LLM, and segments the text into page-indexed chunks. These chunks are in- 5 dexed by sparse and dense retrieval and clustered with a Gaussian Mixture Model (GMM) to support set-query recall. More implementation details are provided in Appendix A.6. 4.1 Evidence Agent The Evidence Agent extracts candidate textual val- ues and image-reference clues. The Evidence Plan- ner first decomposesqinto subqueries ̃q a A a=1 and selects retrieval strategies according to query intent. For single-instance queries, it retrieves the top-krelevant chunks through hybrid retrieval. For set queries, it retrieves the top-kchunks, selects seed chunks with an LLM, and expands them to the corresponding GMM clusters. If a subquery specifies a section range, the system first locates section boundaries; if this fails, it falls back to re- trieval. The Evidence Integrator converts retrieved con- texts into a subquery-conditioned evidence collec- tion: E = ( ̃q a , E a ) A a=1 , E a = e a,r n a r=1 , e a,r = (id a,r , val a,r , clue a,r , src a,r ). (3) Here,id a,r denotes the candidate instance identi- fier,val a,r is the candidate textual value,clue a,r contains image-reference clues, andsrc a,r records source pages. The Evidence Checker removes irrelevant evidence and triggers supplementary re- trieval when textual values are missing or incom- plete. 4.2 Page Agent The Page Agent locates candidate target pages for each evidence item. It follows a clue-driven strategy: explicit labels are localized through key- word matching, printed page numbers through page-index mapping, relative references through offset computation, and intra-page spatial descrip- tions by directly using the evidence page. If clue- based localization fails, the system falls back to entity-based full-document matching and then to the source pages in src a,r . The Page Checker visually verifies candidate pages and their neighboring pages, producing veri- fied target pages and coarse spatial descriptions: L = L a A a=1 , L a = ℓ a,r n a r=1 , ℓ a,r = (ρ a,r,u , loc a,r,u ) U a,r u=1 . (4) Here,ρ a,r,u denotes a verified target page, and loc a,r,u denotes the coarse spatial description on that page for subsequent image selection. 4.3 Figure Agent The Figure Agent adopts a detect-then-select strat- egy to extract target images and complete image- text binding. For each verified pageρ a,r,u , the Fig- ure Extractor uses Grounding DINO (Liu et al., 2024) to obtain candidate boxes, followed by non- maximum suppression, area filtering, and validity filtering. The Figure Binder reformulates image binding as in-page candidate selection. Candidate boxes are numbered on the original page, and the model selects the target box usinge a,r , ̃q a , andloc a,r,u . This preserves page-level context such as captions, labels, and layout relations. Finally, the Figure Checker removes duplicate instances and merges complementary results from different subqueries. It combines the verified tex- tual value and selected image boxes into final an- swer instances: R = z j J j=1 , z j = (t j ,v j ).(5) 5 Experiments 5.1 Experimental Settings Metrics. Since this task requires both image- region localization and textual attribute extraction, we evaluate models from three perspectives: im- age localization, text extraction, and image-text joint correctness. For each query, we first greedily match predicted instances with gold instances that contain images based on instance-level bounding- box overlap. After all image-bearing instances are matched, the remaining 1-to-0 gold instances are matched with the remaining predictions based on textual equivalence. For a matched 1-to-0 instance, image localization is considered correct only if the prediction also outputs an empty image set; oth- erwise, it is counted as an image-text alignment hallucination. For image localization, we use an IoU threshold ofτ = 0.75and define two settings: Strict and Soft. Strict requires the number of predicted image boxes to exactly match the number of gold boxes, with every matched box satisfying the IoU threshold. Soft relaxes the cardinality constraint, allowing a predicted instance to contain fewer boxes than the gold instance, but requiring every predicted box to match a gold box of the same instance with IoU no lower thanτ. For text extraction, we adopt the LLM-as-a-Judge paradigm and assign each in- stance a binary correctness label. The joint metric 6 requires both image localization and text extraction to be correct. We use Strict Joint as the main evalu- ation setting, reporting macro-F1 at the query level and accuracy at the instance level. Soft results are provided in Appendix A.5. Baselines. We select four advanced multimodal large language models as baselines.For closed-source models, we use GPT-5.4 (Ope- nAI, 2026) and Gemini-3.1-Pro-Preview (Google, 2026). For open-source models, we use GLM- 4.5V (Hong et al., 2025) and Qwen3-VL-235B- A22B-Instruct (Bai et al., 2025). In the experi- ments, we refer to them as GPT, Gemini, GLM, and Qwen, respectively. Each baseline receives all page images of the document together with the query in a single call, and directly outputs struc- tured answer instances. Each document page is rendered as a 144-DPI PNG image. When the num- ber of pages exceeds the model’s single-call image limit, we follow the stitching strategy (Ma et al., 2024), arranging pages into composite images with a three-column grid and mapping the output coor- dinates back to the original page coordinates. Due to API limitations, GPT, GLM, and Qwen support up to 50 images per call, while Gemini supports up to 16. We set the temperature of all models to 0.0 for reproducibility. 5.2 Direct and Workflow-Based Results Table 2 compares two representative ways to ad- dress the proposed task: direct inference with stan- dalone VLMs and workflow-based extraction with Q2IT. Standalone VLMs. Under direct inference, dif- ferent VLMs show clear performance gaps. Gem- ini performs best across all query-level categories and most instance-level categories, indicating the strongest direct capability for long-document image-text joint extraction. Specifically, Gemini achieves 0.447 on single positive queries, 0.400 on set positive queries, 0.482 on 1-to-1 instances, and 0.291 on 1-to-N instances, while reaching 0.744 on 1-to-0 instances. Qwen is the strongest open-source standalone model, especially on positive image-bearing cate- gories. It achieves 0.230 on 1-to-1 instances and 0.100 on 1-to-N instances, outperforming GPT and GLM in these settings. GPT shows a more im- balanced pattern: it performs relatively better on negative or 1-to-0 cases, but struggles with positive instances requiring image localization, achieving only 0.113 on 1-to-1 and 0.024 on 1-to-N instances. GLM is the weakest overall, remaining below 0.10 on most positive image-bearing categories. Overall, Gemini leads by a clear margin among standalone VLMs, Qwen is more stable among open-source models, while GPT and GLM show substantial lim- itations in directly producing complete image-text answer instances. Q2IT workflow.Under the Q2IT setting, model differences remain clear, but their capability pro- files change. Gemini is the most stable backbone, achieving 0.692 on single positive queries, 0.943 on single negative queries, 0.876 on 1-to-0 instances, 0.563 on 1-to-1 instances, and 0.476 on 1-to-N in- stances. It also obtains the best or near-best results on difficult categories such as compound, cross- page, and implicit-alignment instances, showing strong overall reliability as the workflow backbone. GPT becomes highly competitive within Q2IT. It achieves the best score on set positive queries with 0.571, and reaches 0.534 on 1-to-1 instances and 0.452 on 1-to-N instances, close to Gemini. It also performs strongly on explicit-alignment instances, reaching 0.574. Open-source models still lag behind Gemini and GPT under Q2IT, but they show different strengths. Qwen performs slightly better than GLM on set positive queries, 1-to-N instances, and im- plicit alignment, suggesting an advantage in multi- instance and multi-image scenarios. GLM per- forms better on negative, 1-to-0, and some explicit- clue categories, indicating relatively stronger text- only or no-image judgment. Overall, closed-source models remain stronger in the Q2IT setting, with Gemini being more stable and GPT standing out on set queries and explicit alignment, while open- source models exhibit distinct but still limited ca- pabilities. Despite these gains, most categories remain be- low 0.60 even under Q2IT, indicating substantial room for improving both model capability and workflow design. 5.3 Decomposed Analysis Table 3 breaks down performance into query-level macro-F1 scores for Text, Page, Image, and Joint metrics, while Table 4 reports Text Acc. and Joint Acc. on 1-to-0 instances. From Page Localization to Image Localization. Page scores are much higher than Image scores, es- pecially for standalone VLMs. Gemini drops from 7 Table 2: Main results on ITJoint under the Strict Joint setting. We evaluate VLMs in two settings: standalone direct inference and as the backbone of Q2IT. Model Single Pos. Single Neg. Set Pos. Set Neg. 1-to-01-to-11-to-NIndep.Comp.IntraCrossExp.Imp. Standalone VLMs Gemini-3.1-Pro-Preview0.4470.7140.4000.5450.7440.4820.2910.4980.1820.5020.3440.4520.412 GPT-5.40.0990.3930.1040.3100.5410.1130.0240.1130.0070.0840.1050.0740.131 Qwen3-VL-235B-A22B-Instruct0.2490.3090.2250.2980.3870.2300.1000.2260.0930.2350.1490.2310.147 GLM-4.5V0.0740.3040.0460.1580.2730.0340.0780.0500.0150.0460.0410.0430.045 VLMs within Q2IT Gemini-3.1-Pro-Preview0.692 +0.244 0.943 +0.229 0.535 +0.136 0.714 +0.169 0.876 +0.132 0.563 +0.081 0.476 +0.185 0.577 +0.080 0.394 +0.212 0.539 +0.036 0.549 +0.206 0.572 +0.120 0.486 +0.074 GPT-5.40.608 +0.508 0.805 +0.412 0.571 +0.466 0.691 +0.381 0.802 +0.262 0.534 +0.422 0.452 +0.428 0.545 +0.433 0.387 +0.380 0.530 +0.445 0.495 +0.390 0.574 +0.500 0.400 +0.269 Qwen3-VL-235B-A22B-Instruct0.476 +0.226 0.623 +0.314 0.422 +0.197 0.407 +0.109 0.528 +0.141 0.382 +0.152 0.321 +0.221 0.395 +0.168 0.250 +0.157 0.388 +0.153 0.338 +0.189 0.410 +0.178 0.286 +0.139 GLM-4.5V0.485 +0.410 0.752 +0.448 0.403 +0.358 0.557 +0.398 0.644 +0.371 0.388 +0.355 0.277 +0.199 0.383 +0.332 0.277 +0.263 0.395 +0.349 0.315 +0.275 0.416 +0.373 0.257 +0.212 Table 3: Decomposed results on Text, Page, Image, and Joint metrics. ModelTextPageImageJoint Standalone VLMs Gemini-3.1-Pro-Preview0.6150.8810.5180.507 GPT-5.40.3060.8080.1870.185 Qwen3-VL-235B-A22B-Instruct0.4140.7760.2680.263 GLM-4.5V0.2430.6290.1310.128 VLMs within Q2IT Gemini-3.1-Pro-Preview0.787 +0.172 0.908 +0.027 0.793 +0.275 0.722 +0.215 GPT-5.40.763 +0.457 0.872 +0.063 0.725 +0.538 0.653 +0.468 Qwen3-VL-235B-A22B-Instruct0.669 +0.255 0.786 +0.011 0.553 +0.285 0.492 +0.229 GLM-4.5V0.668 +0.425 0.756 +0.126 0.595 +0.464 0.537 +0.409 Table 4: Results on 1-to-0 instances. ModelText Acc.Joint Acc.Gap↓ Standalone VLMs Gemini-3.1-Pro-Preview0.8490.7440.105 GPT-5.40.7670.5410.226 Qwen3-VL-235B-A22B-Instruct0.7140.3870.327 GLM-4.5V0.5970.2730.324 VLMs within Q2IT Gemini-3.1-Pro-Preview0.904 +0.055 0.876 +0.132 0.028 GPT-5.40.887 +0.120 0.802 +0.262 0.085 Qwen3-VL-235B-A22B-Instruct0.778 +0.064 0.528 +0.141 0.250 GLM-4.5V0.757 +0.160 0.644 +0.371 0.113 0.881 on Page to 0.518 on Image, while GPT drops from 0.808 to 0.187. This suggests that models can often reach the correct pages but fail to localize the target images. Q2IT substantially improves Image scores, from 0.518 to 0.793 for Gemini and from 0.187 to 0.725 for GPT, while the gains on Page are relatively smaller. This indicates that a key benefit of the workflow lies in converting page-level evi- dence into accurate bbox-level image localization. From Single-Side Correctness to Joint Correct- ness. Q2IT improves both Text and Image, but Joint remains lower than either metric. For exam- ple, Gemini with Q2IT achieves 0.787 on Text and 0.793 on Image, but 0.722 on Joint; GPT achieves 0.763, 0.725, and 0.653, respectively. This gap shows that correct text extraction and correct image localization do not necessarily form a correct an- swer instance. The remaining difficulty lies in bind- ing the textual attribute to the corresponding image, especially in multi-image, compound-layout, and implicit-alignment cases. Text-only Cases. The 1-to-0 setting further ex- poses hallucinated image binding. These instances contain textual evidence but no corresponding im- age, so the gap between Text Acc. and Joint Acc. reflects cases where the model identifies the textual instance but incorrectly assigns an image. Q2IT reduces this gap, such as from 0.226 to 0.085 for GPT and from 0.324 to 0.113 for GLM, but does not eliminate it. This shows that the task also re- quires models to recognize when no image-text association should be made. A qualitative example is provided in Appendix A.4. 5.4 Additional Ablation Analysis To further investigate the challenges discussed above, we provide additional ablation studies in Appendix A.7. 6 Conclusion We introduce query-driven image-text joint extrac- tion from multimodal long documents, a task that requires systems to retrieve textual attributes and corresponding images from domain-specific long documents according to user queries. To support this task, we analyze its core challenges from both user intent and the document itself, design a two- level taxonomy at the query and instance levels, and construct ITJoint, the first high-quality manu- ally annotated benchmark for this task. We further study two representative solution settings: stan- dalone VLMs and our task-specific multi-agent workflow, Q2IT. Experiments show that standalone VLMs struggle with this task, while Q2IT brings clear improvements but still remains far from per- fect. These findings demonstrate the difficulty of query-driven image-text joint extraction and high- light its value for driving future advances in multi- modal models and agent-based systems. 8 Limitations Although ITJoint covers diverse domain-specific long documents and complex image-text relations, its scale is still limited by the cost of manual anno- tation. Constructing the dataset requires annotators to read documents page by page, clean candidate queries, locate answer instances, and annotate im- age bounding boxes. This process is especially costly for set queries, where all instances satisfying the query condition must be exhaustively identified. Future work can expand the document sources, do- main coverage, and dataset scale to improve the representativeness of the benchmark. The benchmark also has several annotation- related limitations. Although ITJoint includes cross-page samples with varying spans, nearby- page cases account for a relatively large proportion, which may make the contribution of multi-strategy page search less pronounced in the overall results. In addition, some domain-specific images have am- biguous boundaries between the target object and its background. Although we use unified annota- tion guidelines to improve consistency, such bound- ary ambiguity may still affect fine-grained image localization evaluation. Methodologically, Q2IT adopts a structured three-stage workflow that models humans’ pro- gressive focusing behavior, rather than allowing multimodal large language models to perform long- document search and image-text matching fully autonomously. This design improves controllabil- ity and traceability, but the framework still depends on the quality of intermediate results across stages. Future work can explore more flexible planning and feedback mechanisms to further improve end- to-end coordination while maintaining reliability. Ethical Considerations Our dataset does not contain any personal informa- tion, and all source materials are obtained from pub- licly accessible channels. The dataset construction and refinement process is conducted with full re- spect for copyright and intellectual property rights. References Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, and 1 others. 2025. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Christopher Clark and Santosh Divvala. 2016. Pdffig- ures 2.0: Mining figures from research papers. In Proceedings of the 16th ACM/IEEE-CS on Joint Con- ference on Digital Libraries, pages 143–152. Chao Deng, Jiale Yuan, Pi Bu, Peijie Wang, Zhong- Zhi Li, Jian Xu, Xiao-Hui Li, Yuan Gao, Jun Song, Bo Zheng, and 1 others. 2025. Longdocurl: a com- prehensive multimodal long document benchmark integrating understanding, reasoning, and locating. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1135–1159. Meihao Fan, Ju Fan, Nan Tang, Lei Cao, Guoliang Li, and Xiaoyong Du. 2024. Autoprep: Natural language question-aware data preparation with a multi-agent framework. arXiv preprint arXiv:2412.10422. Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, and 1 others. 2023. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36:27092–27112. Google.2026.Gemini3.1ProPreview. https://ai.google.dev/gemini-api/docs/ models/gemini-3.1-pro-preview.Accessed: 2026-05-21. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Steven Yau, Zijuan Lin, Liyang Zhou, and 1 oth- ers. 2024. Metagpt: Meta programming for a multi- agent collaborative framework. In International Con- ference on Learning Representations, volume 2024, pages 23247–23275. Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, and 1 others. 2025. Glm-4.5 v and glm-4.1 v-thinking: Towards versatile multi- modal reasoning with scalable reinforcement learn- ing. arXiv preprint arXiv:2507.01006. Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutlmv3: Pre-training for doc- ument ai with unified text and image masking. In Proceedings of the 30th ACM international confer- ence on multimedia, pages 4083–4091. Geewook Kim, Teakgyu Hong, Moonbin Yim, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2021. Donut: Document understanding transformer without ocr. arXiv preprint arXiv:2111.15664, 7(15):2. Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" exploration of large language model society. Advances in neural information processing systems, 36:51991–52008. 9 Pengyuan Li, Xiangying Jiang, and Hagit Shatkay. 2019. Figure and caption extraction from biomedical docu- ments. Bioinformatics, 35(21):4381–4388. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jian- wei Yang, Hang Su, and 1 others. 2024. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pages 38–55. Springer. Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, and 1 others. 2024. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations. Advances in Neural Information Processing Systems, 37:95963–96010. Minesh Mathew, Dimosthenis Karatzas, and CV Jawa- har. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter con- ference on applications of computer vision, pages 2200–2209. Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. 2023. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358–50376. OpenAI. 2026.Introducing GPT-5.4.https:// openai.com/index/introducing-gpt-5-4/. Ac- cessed: 2026-05-21. Honglin Pang, Yi Chang, Tianjing Duan, and Xi Yang. 2024. An archaeological catalog collection method based on large vision-language models.arXiv preprint arXiv:2412.20088. Wei Pang, Kevin Qinghong Lin, Xiangru Jian, Xi He, and Philip Torr. 2026. Paper2poster: Towards multi- modal poster automation from scientific papers. Ad- vances in Neural Information Processing Systems, 38. Noah Siegel, Nicholas Lourie, Russell Power, and Waleed Ammar. 2018. Extracting scientific figures with distantly supervised neural networks. In Pro- ceedings of the 18th ACM/IEEE on joint conference on digital libraries, pages 223–232. Jifeng Song, Arun Das, Ge Cui, and Yufei Huang. 2025. Figex: Aligned extraction of scientific figures and captions. In Findings of the Association for Com- putational Linguistics: EMNLP 2025, pages 16558– 16571. Tao Sun, Enhao Pan, Zhengkai Yang, Kaixin Sui, Jiajun Shi, Xianfu Cheng, Tongliang Li, Wenhao Huang, Ge Zhang, Jian Yang, and 1 others. 2025. P2p: Au- tomated paper-to-poster generation and fine-grained benchmark. arXiv preprint arXiv:2505.17104. Yi Tang, Chia-Ming Chang, and Xi Yang. 2024. Pdfchatannotator: A human-llm collaborative multi- modal data annotation tool for pdf-format catalogs. In Proceedings of the 29th International Conference on Intelligent User Interfaces, pages 419–430. Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. 2023. Hierarchical multimodal transformers for mul- tipage docvqa. Pattern Recognition, 144:109834. Jordy Van Landeghem, Rubèn Tito, Łukasz Borchmann, Michał Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Jurkiewicz, Mickaël Coustaty, Bertrand Anck- aert, Ernest Valveny, and 1 others. 2023. Document understanding dataset and evaluation (dude). In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 19528–19540. Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Be- siroglu, Lennart Heim, and Marius Hobbhahn. 2024. Position: Will we run out of data? limits of llm scal- ing based on human-generated data. In Forty-first International Conference on Machine Learning. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, and 1 others. 2024. Au- togen: Enabling next-gen llm applications via multi- agent conversations. In First conference on language modeling. Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, and 1 others. 2021. Lay- outlmv2: Multi-modal pre-training for visually-rich document understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 2579–2591. Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chen- liang Li, Junfeng Tian, and 1 others. 2023. mplug- docowl: Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.02499. Wenhan Yu, Wang Chen, Guanqiang Qi, Weikang Li, Yang Li, Lei Sha, Deguo Xia, and Jizhou Huang. 2025.Bbox docvqa: A large scale bounding box grounded dataset for enhancing reasoning in document visual question answer. arXiv preprint arXiv:2511.15090. Jianing Zhang, Runan Li, Honglin Pang, Ding Xia, Zhou Zhu, Qian Zhang, Chuntao Li, and Xi Yang. 2026. Specializing large models for oracle bone script interpretation via component-grounded mul- timodal knowledge augmentation. arXiv preprint arXiv:2604.06711. 10 A Appendix A.1 Annotation Details ITLabel Annotation Tool. To improve annota- tion efficiency and consistency, we develop ITLa- bel, a visual annotation tool specifically designed for query-driven image-text joint extraction from long documents. As shown in Figure 4, ITLabel provides a unified document-level annotation inter- face, rather than treating each page as an isolated image. It supports page browsing, OCR-based text extraction, keyword search, candidate-query revi- sion, textual-attribute entry, image bounding-box annotation, and query-level and instance-level tax- onomy labeling. A key feature of ITLabel is its support for cross- page, multi-instance, and multi-image annotation. For a single query, annotators can create multiple answer instances; for a single answer instance, they can further link the textual attribute value to one or more image boxes across different pages. This design enables efficient annotation of set queries, cross-page cases, and 1-to-N image-text mappings. The OCR-based text extraction function also allows annotators to directly reuse document text when filling textual attributes, reducing manual transcrip- tion effort. Document Selection Rules.We collect long doc- uments from public resources to cover realistic doc- ument scenarios faced by researchers when acquir- ing image-text information. Document selection follows four principles: (1) image richness, where documents should contain abundant images that are indispensable for conveying core domain knowl- edge rather than merely decorative; (2) domain diversity, where documents should cover diverse domain-specific fields, such as medicine, agricul- ture, archaeology, biology, and art; (3) language coverage, where documents should cover different language settings, including English, Chinese, and bilingual documents; and (4) document length, where each document should contain a continuous page range from tens to over one hundred pages to form realistic long-range retrieval scenarios. Query Generation Rules. We adopt a human- LLM collaborative workflow to generate queries. Specifically, we first use an LLM to generate can- didate queries in batches based on the OCR text of each document. Human annotators then man- ually filter, revise, and supplement the candidate queries. Query generation follows seven princi- Figure 4: Interface of ITLabel. ples: (1) objectivity, where the target attribute of each query should have a deterministic answer in the document; (2) clarity, where the query should be clearly formulated so that annotators can understand the requested information without ambiguity; (3) diversity, where queries should cover diverse language styles and attribute dimen- sions while avoiding repetitive templates; (4) non- redundancy, where similar queries about the same entity should be avoided; (5) distribution balance, where the number of involved instances and the dis- tribution of categories should be kept as balanced as possible; (6) self-containment, where each query should be independently understandable without relying on additional context; and (7) multimodal dependency, where each query must require joint extraction of textual attributes and corresponding images rather than purely textual question answer- ing. Answer Annotation Rules. For each query, an- notators locate all entities satisfying the query con- dition in the document, record their textual at- tribute values, annotate image bounding boxes on the corresponding pages, and assign query-level categories and instance-level attributes. Answer an- notation follows four rules: (1) entity specificity, where each answer instance must refer to a specific individual entity rather than a generic concept; (2) textual faithfulness, where textual attribute values should be directly copied from the document and should not be manually summarized or inferred without evidence; (3) explicit negative annota- tion, where entities with textual evidence but no corresponding image must be explicitly recorded with an empty image set; and (4) independent im- age boxes, where each independent visual entity should be annotated with a separate bounding box, and multiple distinct images should not be merged into one box. 11 Query: The text describes a silver hairpin modeled to resemble a "wooden staff". According to the text, this dragon-head staff form is an attribute of which specific figures? Also, state the length of this hairpin (in cm), and provide its image. Ground Truth: text_answer: Attribute_Of = Taoist sages and immortals; Length = 25.1 cm image_answer: image_10.png, bounding box shown in red labels: single-instance / positive; 1-to-1 / independent / cross-page / explicit Figure 5: Ground-truth annotation example for a single- instance query in ITJoint. 1-to-N Compound Cross-page Explicit 3032114 1-to-1 Independent Cross-page Implicit 1-to-N Independent Intra-page Implicit 1-to-0 None None None a b c d 485051 Figure 6: Instance-level taxonomy examples in ITJoint. For set queries, annotators are required to ex- haustively identify all instances satisfying the query condition in the document. Particular attention is paid to valid instances involving cross-page rela- tions, implicit alignment, or weak visual clues, so as to avoid missing valid answer instances. Ground-Truth Annotation Example. Figure 5 shows the ground-truth annotation format for a single-instance query in ITJoint. The query re- quires a textual answer and its corresponding target image. The annotation records thetext_answer, theimage_answerwith the target bounding box, and the query-level and instance-level labels. This example further illustrates a cross-page case, where completing the answer instance requires informa- tion from multiple pages. For readability, we present a compact version of the annotation and omit implementation-specific fields such as local file paths. A.2 Instance-Level Taxonomy Examples Figure 6 provides visual examples of the instance- level taxonomy in ITJoint. Each panel corresponds to one answer instance and shows its labels along modality cardinality, visual layout, spatial span, and alignment clues. Red boxes indicate relevant textual evidence or target image regions. Panel (a) shows an answer instance associated with multiple target images, and is therefore la- beled as 1-to-N. The textual evidence and all target images together involve multiple pages, so the in- stance is also labeled as cross-page. Since the tar- get images are nested in compound visual layouts and the document provides explicit label references linking the text to the images, it is further labeled as compound and explicit. Panel (b) shows a 1-to-1 instance, where one tex- tual attribute value corresponds to a single target image. Although the target image itself is indepen- dent, the textual evidence and the image together involve multiple pages, requiring cross-page asso- ciation. The correspondence is implicit because the image-text relation cannot be established solely through an explicit marker and must be inferred from the surrounding context. Panel (c) illustrates an intra-page instance, where the textual evidence and all target images together involve only a single page. The target images are standalone visual regions rather than parts of a com- pound figure, so the instance is labeled as indepen- dent. Since multiple target image regions are as- sociated with the same textual attribute value, it is labeled as 1-to-N. Its alignment is implicit, as the association relies on page layout and local context rather than a direct textual reference. Panel (d) shows a 1-to-0 instance. The document contains textual evidence for the target instance, but no corresponding image is present. Therefore, the system is expected to return the textual attribute value with an empty image set, i.e.,v j = ∅, rather than hallucinating an image region. Since no image is associated with the instance, the visual layout, spatial span, and alignment-clue labels are not ap- plicable. A.3 Document Examples As described in the document collection process, ITJoint collects long documents from multiple domain-specific fields, including medicine, agri- culture, archaeology, biology, and art. These doc- uments cover both English and Chinese, contain diverse image-text layouts, and include rich non- decorative core images. We show examples of these documents below. 12 Figure 7: Medical document examples from ITJoint (English). A.4 Qualitative Example of Image-Text Alignment Hallucination Figure 14 shows a qualitative example of image- text alignment hallucination on a 1-to-0 instance using GPT as the backbone. The document con- tains textual evidence for the queried entity, but no corresponding target image exists. Under di- rect inference, GPT correctly identifies the textual answer but still assigns an image region, causing the instance to be correct under text accuracy but incorrect under joint accuracy. In contrast, GPT equipped with Q2IT returns the same textual an- swer with an empty image set, which is consistent with the ground truth. A.5 Additional Results under the Soft Joint Setting In the main experiments, we report results under the Strict Joint setting, which requires the predicted image set to exactly match the gold image set. This setting is appropriate for evaluating complete image-text joint extraction, but it may be overly conservative in scenarios where users do not neces- sarily need all corresponding images to complete the downstream task. In such cases, returning a subset of correct images can still be useful, as long as the system does not introduce incorrect or hallu- cinated image regions. Therefore, we additionally report results under the Soft Joint setting in Table 5. Soft Joint relaxes the image-box cardinality constraint by allowing the prediction to contain fewer boxes than the gold instance. For image-bearing instances, the predic- tion must contain at least one correctly matched box, and every predicted box must match a gold box of the same instance. For 1-to-0 instances, the prediction is still required to return an empty image set. Thus, Soft Joint rewards partially complete but correct image retrieval, while still penalizing irrel- evant or hallucinated images. Compared with the Strict Joint results in the main experiments, Soft Joint generally yields higher scores, especially for 1-to-N instances. Nevertheless, the overall trend remains consistent: Q2IT improves all backbone models across query-level and instance-level cat- egories, indicating that the effectiveness of our framework is robust across evaluation settings. A.6 Implementation Details We use Qwen3-Embedding-8B to encode page- indexed text chunks. For hybrid retrieval, FAISS and BM25 each return the top 30 candidates, which are fused and then reranked by BAAI/bge-reranker- v2-m3. The top 10 reranked chunks are used for evidence extraction. For section-restricted queries, after detecting a section title, Q2IT expands a 30- page window based on page metadata; if page in- dices cannot be reliably parsed, it falls back to the 13 Figure 8: Medical document examples from ITJoint (Chinese). next 50 chunks. A.7 Ablation Study Unless otherwise specified, all ablation experi- ments are conducted with GPT-5.4 as the backbone model under the Q2IT framework. The results are reported in Table 6. Page Agent ablation.We ablate two key designs in the Page Agent: the Page Checker and the multi- strategy page search. Removing the Page Checker causes the most significant drop in 1-to-0 accu- racy, from 0.802 to 0.531, a decrease of 27.1 per- centage points, indicating that this module is cru- cial for suppressing image-text alignment hallu- cinations. Accuracy on cross-page and implicit- alignment instances also drops by more than 10 percentage points, while the declines for intra-page and explicit-alignment instances are relatively mild. This comparison verifies our observation that when a target image is far from its descriptive text, long documents usually provide localizable clues, such as figure labels, printed page numbers, or relative page references. When such clues are absent, the target image is often located on the evidence page or nearby pages. Explicit references usually pro- vide captions or labels and can be localized through search, making them less dependent on the Page Checker; implicit cross-page images lack such an- chors and therefore rely more on visual verification. We further remove the multi-strategy page search after removing the Page Checker, using only the source pages output by the Evidence Agent as candidate pages. Page-level F1 drops from 0.799 to 0.740, and cross-page accuracy further decreases from 0.349 to 0.234. This confirms that target im- ages in long documents are not always located on the same pages as their descriptive text, and source pages cannot replace dedicated page search. Accuracy on 1-to-N instances also drops further, from 0.355 to 0.307, because multiple images cor- responding to the same entity are more likely to be distributed across different pages; without page search, some target-image pages cannot be located. External tool ablation. We examine the neces- sity of using an external open-set detector for pre- cise image-region extraction. Removing Ground- ing DINO leads to large drops across image-related instance categories: accuracy on 1-to-1 instances decreases from 0.534 to 0.170, accuracy on 1-to-N instances decreases from 0.452 to 0.078, and accu- racy on compound instances decreases from 0.387 to 0.073. This shows that current multimodal mod- els still lack sufficient direct coordinate-generation ability for precise region extraction, making a ded- icated open-set object detector necessary for this task. 14 Figure 9: Archaeological document examples from ITJoint (English). Table 5: Additional results of direct inference and Q2IT on ITJoint under the Soft Joint setting. ModelSingleSingleSetSet1-to-01-to-11-to-NIndep.Comp.IntraCrossExp.Imp. Pos.Neg.Pos.Neg. Closed-source Models Gemini-3.1-Pro-Preview0.4660.7140.4100.5450.7440.4820.3330.5090.1830.5050.3640.4580.429 Gemini-3.1-Pro-Preview (Q2IT)0.727 +0.261 0.943 +0.229 0.583 +0.173 0.722 +0.177 0.876 +0.132 0.563 +0.081 0.596 +0.263 0.592 +0.083 0.475 +0.292 0.550 +0.046 0.600 +0.236 0.607 +0.149 0.498 +0.069 GPT-5.40.0990.3930.1070.3100.5410.1130.0420.1180.0070.0910.1050.0800.131 GPT-5.4 (Q2IT)0.649 +0.550 0.805 +0.412 0.624 +0.517 0.701 +0.391 0.802 +0.262 0.536 +0.423 0.621 +0.578 0.579 +0.461 0.453 +0.445 0.555 +0.464 0.556 +0.451 0.623 +0.543 0.420 +0.290 Open-source Models Qwen3-VL-235B-A22B-Instruct0.2690.3090.2560.3050.3870.2300.2270.2580.1010.2650.1710.2580.176 Qwen3-VL-235B-A22B-Instruct (Q2IT)0.509 +0.240 0.623 +0.314 0.476 +0.220 0.425 +0.119 0.528 +0.142 0.382 +0.152 0.473 +0.246 0.424 +0.166 0.309 +0.208 0.406 +0.141 0.396 +0.225 0.444 +0.187 0.318 +0.143 GLM-4.5V0.0740.3040.0560.1580.2730.0340.1020.0570.0150.0480.0510.0450.057 GLM-4.5V (Q2IT)0.535 +0.460 0.752 +0.448 0.490 +0.434 0.580 +0.422 0.644 +0.371 0.388 +0.355 0.452 +0.349 0.416 +0.359 0.343 +0.329 0.418 +0.370 0.380 +0.329 0.459 +0.414 0.290 +0.233 Figure 10: Archaeological document examples from ITJoint (Chinese). 15 Figure 11: Agricultural document examples from ITJoint (English). Figure 12: Biology document examples from ITJoint (Chinese-English). Table 6: Ablation results of Q2IT on ITJoint with GPT-5.4 as the backbone model. Setting1-to-01-to-11-to-NIndep.Comp.IntraCrossExp.Imp.Page Page Agent Ablation Full Q2IT0.8020.5340.4520.5450.3870.5300.4950.5740.4000.872 w/o Page Checker0.531 −0.271 0.457 −0.078 0.355 −0.096 0.453 −0.092 0.350 −0.037 0.491 −0.039 0.349 −0.146 0.510 −0.064 0.282 −0.118 0.799 −0.072 w/o Page Search0.548 −0.254 0.436 −0.099 0.307 −0.145 0.431 −0.114 0.299 −0.088 0.523 −0.007 0.234 −0.261 0.469 −0.105 0.282 −0.118 0.740 −0.132 External Tool Ablation Full Q2IT0.8020.5340.4520.5450.3870.5300.4950.5740.4000.872 w/o Grounding DINO0.774 −0.028 0.170 −0.365 0.078 −0.374 0.166 −0.379 0.073 −0.314 0.151 −0.379 0.146 −0.349 0.121 −0.453 0.204 −0.196 0.888 +0.017 16 Figure 13: Art document examples from ITJoint (English). Query: The text details the principle of Self-glazing clay bodies. Please explain the specific role of fluxes during the firing of the body, and extract the image showing the surface effect after firing. Source Page Baseline output: "text_answer":"In self-glazing clay bodies, soluble fluxes migrate during drying from the interior of the clay body to the surface. After the piece is fired, these fluxes combine with the silica and alumina in the body to form a thin surface layer of glaze. This creates a glazed or semi-glazed fired surface as the characteristic surface effect.", "image_answer": ["page": "image_10.png","bbox": [0.09, 0.07, 1.0, 0.55]] Ground truth: "text_answer":"the fluxes combine with the silica and alumina in the body to create a thin layer of glaze on the surface of the ware.", "image_answer": [] Q2IT output: "text_answer":"When the piece is fired, the fluxes combine with the silica and alumina in the body to create a thin layer of glaze on the surface.", "image_answer": [] Figure 14: Qualitative example of image-text alignment hallucination. 17