Paper deep dive
Multi-Granularity Context-Enhanced RAG over Multimodal Knowledge Graphs
Zongyu Wu, Yilong Wang, Xiaochen Wang, Minhua Lin, Zhichao Xu, Fenglong Ma, Xiang Zhang, Suhang Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/29/2026, 3:46:05 AM
Summary
The paper introduces CEMMKG, a framework for constructing Context-Enhanced Multimodal Knowledge Graphs to improve Multimodal Retrieval-Augmented Generation (RAG). It addresses the semantic gap between images and text in existing methods by enriching visual elements with multi-granularity textual context (local and global) during graph construction, rather than processing modalities independently.
Entities (9)
Relation Signals (6)
CEMMKG → improves → MMKG
confidence 95% · The proposed CEMMKG enriches each image with complementary textual context... validate that CEMMKG is effective in leveraging contextual information to improve MMKG-based RAG performance.
RAG → mitigates → Hallucination
confidence 95% · Retrieval-augmented generation (RAG) is widely used to mitigate hallucination issues in large language models
CEMMKG → supports → GraphRAG
confidence 95% · we propose a novel framework for constructing a Context-Enhanced MMKG (CEMMKG) to better support multimodal GraphRAG.
local context → ispartof → CEMMKG
confidence 90% · We further introduce a multi-granularity design for the local context... The proposed CEMMKG enriches each image with complementary textual context at both local and global scopes.
global context → ispartof → CEMMKG
confidence 90% · The proposed CEMMKG enriches each image with complementary textual context at both local and global scopes.
MMKG → enables → GraphRAG
confidence 85% · recent studies have explored multimodal knowledge graphs (MMKGs) as knowledge bases for GraphRAG.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Retrieval-augmented generation (RAG) is widely used to mitigate hallucination issues in large language models (LLMs) and multimodal large language models (MLLMs). In particular, knowledge graph (KG)-based RAG leverages structured knowledge to provide (M)LLMs with high-quality external information. Building on these works, recent studies have explored multimodal knowledge graphs (MMKGs) as knowledge bases for GraphRAG. This enables Graph RAG to integrate knowledge across multiple modalities, thereby further enhancing its performance. However, existing MMKG-based RAG methods generally follow a common pipeline in which different modalities are largely processed independently before being fusion. As a result, textual context is only used to a limited extent during visual information extraction and subsequent multimodal knowledge fusion. This brings a semantic gap between images and text which limits the multimodal GraphRAG performance. To address this issue, we propose a novel framework for constructing a Context-Enhanced MMKG (CEMMKG) to better support multimodal GraphRAG. The proposed CEMMKG enriches each image with complementary textual context at both local and global scopes. Local context goes beyond the surrounding text by incorporating sentences that are semantically related to the image, while global context provides a summary of the entire passage. We further introduce a multi-granularity design for the local context, allowing it to capture semantically relevant information at different levels of detail. Extensive experiments on the selected vision-centric dataset validate that CEMMKG is effective in leveraging contextual information to improve MMKG-based RAG performance. Moreover, its effectiveness across different MMKG-based RAG methods demonstrates its broad applicability.
Tags
Links
- Source: https://arxiv.org/abs/2608.25986v1
- Canonical: https://arxiv.org/abs/2608.25986v1
Trouble viewing inline? Open PDF directly →
Full Text
70,723 characters extracted from source content.
Expand or collapse full text
Multi-Granularity Context-Enhanced RAG over Multimodal Knowledge Graphs CCS: Information systems Information retrievalCCS: Computing methodologies Natural language processing Zongyu Wu Note: Equal Contribution Affiliation: The Pennsylvania State University, University Park, Pennsylvania, USA email: zongyuwu@psu.edu , Yilong Wang Affiliation: The Pennsylvania State University, University Park, Pennsylvania, USA email: yvw5769@psu.edu , Xiaochen Wang Affiliation: The Pennsylvania State University, University Park, Pennsylvania, USA email: xcwang@psu.edu , Minhua Lin Affiliation: The Pennsylvania State University, University Park, Pennsylvania, USA email: mfl5681@psu.edu , Zhichao Xu Affiliation: University of Utah, Salt Lake City, Utah, USA email: zhichao.xu@utah.edu , Fenglong Ma Affiliation: The Pennsylvania State University, University Park, Pennsylvania, USA email: fenglong@psu.edu , Xiang Zhang Affiliation: The Pennsylvania State University, University Park, Pennsylvania, USA email: xzz89@psu.edu and Suhang Wang Affiliation: The Pennsylvania State University, University Park, Pennsylvania, USA email: szw494@psu.edu Received 5 June 2018 Abstract. Retrieval-augmented generation (RAG) is widely used to mitigate hallucination issues in large language models (LLMs) and multimodal large language models (MLLMs). In particular, knowledge graph (KG)-based RAG leverages structured knowledge to provide (M)LLMs with high-quality external information. Building on these works, recent studies have explored multimodal knowledge graphs (MMKGs) as knowledge bases for GraphRAG. This enables Graph RAG to integrate knowledge across multiple modalities, thereby further enhancing its performance. However, existing MMKG-based RAG methods generally follow a common pipeline in which different modalities are largely processed independently before being fusion. As a result, textual context is only used to a limited extent during visual information extraction and subsequent multimodal knowledge fusion. This brings a semantic gap between images and text which limits the multimodal GraphRAG performance. To address this issue, we propose a novel framework for constructing a Context-Enhanced MMKG (CEMMKG) to better support multimodal GraphRAG. The proposed CEMMKG enriches each image with complementary textual context at both local and global scopes. Local context goes beyond the surrounding text by incorporating sentences that are semantically related to the image, while global context provides a summary of the entire passage. We further introduce a multi-granularity design for the local context, allowing it to capture semantically relevant information at different levels of detail. Extensive experiments on the selected vision-centric dataset validate that CEMMKG is effective in leveraging contextual information to improve MMKG-based RAG performance. Moreover, its effectiveness across different MMKG-based RAG methods demonstrates its broad applicability. Keywords: Knowledge Graph, Retrieval-Augmented Generation, Multimodal Learning 1. Introduction Figure 1. An illustration of node-based Multimodal Knowledge Graph where image is also treated as a node.An illustration of node-based Multimodal Knowledge Graph. Although Multimodal Large Language Models (MLLMs) (Liu et al., 2023; Liu et al., 2024a; Yin et al., 2024) have demonstrated strong performance across a wide range of domains, they are still prone to generate hallucinated content (Bai et al., 2024) due to several factors such as outdated training knowledge. Retrieval-Augmented Generation (RAG), originally developed for LLMs (Gao et al., 2023; Lewis et al., 2020; Xu et al., 2026) and subsequently extended to multimodal settings (Abootorabi et al., 2025), has been widely adopted to enhance the performance of MLLMs and alleviate hallucinations. By retrieving relevant information from external data sources (Abootorabi et al., 2025), RAG can augment MLLMs with external knowledge which can support response generation . Among RAG approaches, GraphRAG (Peng et al., 2024) has gained increasing attention for leveraging structured data, such as knowledge graphs (KGs) (Ji et al., 2022), as external knowledge sources. By explicitly modeling entities and the relations among them, KGs provide structured knowledge that can effectively support information retrieval and model generation. Despite the success of KGs to represent knowledge, most existing GraphRAG methods (Edge et al., 2024) primarily operate on textual KGs, with graph construction and retrieval largely centered on textual information. This text-centric paradigm leaves rich information from other modalities, such as visual content and tabular data, largely underexplored. Information from different modalities could provide valuable knowledge beyond text, enabling a more comprehensive representation of KG. Consequently, the potential of GraphRAG remains constrained by its limited use of multimodal information. To incorporate multimodal information into KGs, recent works such as RAG-Anything (Guo et al., 2025) and MMGraphRAG (Wan and Yu, 2025) explore the construction of retrieval-oriented multimodal knowledge graphs (MMKGs) and modality fusion strategies for multimodal GraphRAG. As shown in Figure 1, an MMKG represents information from different modalities as nodes within a unified graph. For instance, the logo of the marathon event serves as a visual node in the MMKG and can be connected to other related entities. However, existing MMKG-based RAG methods (Wan and Yu, 2025; Guo et al., 2025) typically follow a pipeline in which information from each modality is largely processed separately before being fused into a unified MMKG. Such modality-specific processing may overlook the rich contextual relationships between information from different modalities. For example, when processing visual information, existing methods often utilize only limited textual context, such as surrounding text chunks. Visual information is often closely related to textual information distributed across different parts of a document. Such textual information may provide important context for interpreting the semantics of visual elements. An example is shown in Figure 2. Therefore, such limited textual context may result in suboptimal visual knowledge extraction and ineffective cross-modal knowledge fusion, thereby constraining the performance of MMKG-based RAG. Simply incorporating more textual context does not necessarily improve the quality of MMKG. Therefore, effectively leveraging textual context for selected images requires appropriate design throughout the MMKG construction process. Different context granularities capture information at varying levels of detail. Local context may offer details closely related to a visual element, whereas global context can provide a more holistic high-level understanding. Moreover, their effectiveness also depends on how they are utilized during MMKG construction. These considerations introduce several important questions that remain underexplored: (i) what textual information should be selected as context for visual elements, (i) what level of granularity is most appropriate for constructing such context, and (i) how the designed context can be effectively utilized during MMKG construction process. To answer these questions, we propose CEMMKG, a multi-granular context-enhanced MMKG construction framework that systematically explores the design and utilization of textual context for visual elements. Specifically, we first investigate different designs and granularities of contextual information for visual elements, considering both local and global context to facilitate cross-modal alignment. Second, we explore how the resulting textual context can be effectively leveraged at different stages of MMKG construction. Extensive experiments on a vision-centric subset selected from MMLongBench-Doc demonstrate the effectiveness of CEMMKG in improving the performance of MMKG-based RAG. The proposed CEMMKG is also applicable to different MMKG-based RAG methods. Figure 2. An illustration of textual context for a selected visual element, where useful contextual information may be distributed across different parts of a document rather than being limited to the immediate surroundings of the visual element. Such contextual information may provide richer semantics and strengthen the connections between image and text, thereby facilitating the construction of higher-quality MMKGs.An illustration of textual context for a selected visual element, where useful context information may be distributed across different parts of a document rather than being limited to the immediate surroundings of the visual element. Such contextual information may provide richer semantics for visual elements and strengthen the connections between image and text, thereby facilitating the construction of higher-quality MMKGs. In summary, this work has the following main contributions: • We present a comprehensive study of multi-granular context mapping between multi-modal elements in the MMKG construction process. • We propose CEMMKG, which systematically designs textual context for visual elements at different granularities and flexibly utilizes the resulting context across different stages of MMKG construction. • Extensive experiments across different context configurations on a vision-centric subset selected from MMLongBench-Doc demonstrate the effectiveness of CEMMKG in improving the performance of MMKG-based RAG. Furthermore, CEMMKG can be effectively integrated with different methods, demonstrating its broad applicability. 2. Related Work 2.1. Multimodal Large Language Models Large Language Models (LLMs) (OpenAI, 2023; Zhao et al., 2026) have demonstrated impressive capabilities, benefiting from techniques such as large-scale pretraining (Brown et al., 2020) and reinforcement learning (Schulman et al., 2017; Shao et al., 2024). Multimodal Large Language Models (MLLMs) (Yin et al., 2024; Liu et al., 2024a) extend the capabilities of LLMs to the visual domain by aligning a vision encoder (Radford et al., 2021) with an LLM backbone (Touvron et al., 2023), either through lightweight projection layers (Liu et al., 2023; Liu et al., 2024a; Bai et al., 2023; Bai et al., 2025) or through cross-attention and learned query modules (Li et al., 2023a; Chen et al., 2024b). Because they can reason jointly over interleaved images and texts, MLLMs have become a common tool for translating visual information into textual content, and are widely used to populates multimodal knowledge graphs with visual entities and relations. However, MLLMs inherit the hallucination problem of LLMs (Huang et al., 2025) and may describe objects, attributes, and relations that are absent from or inconsistent with the input image (Bai et al., 2024; Li et al., 2023b). Simply providing more textual information may not effectively address these errors, since these models may under-utilize evidence in the middle of long inputs (Liu et al., 2024b) and could be distracted by irrelevant passages (Cuconasu et al., 2024). 2.2. Multimodal RAG Multimodal RAG extends retrieval-augmented generation beyond text, so that the information in other modalities such as images and charts can also ground generation (Abootorabi et al., 2025). Early work retrieves image–text pairs with a jointly trained encoder and conditions a generator on the retrieved information (Chen et al., 2022; Yasunaga et al., 2023). Subsequent methods further improve retrieval for knowledge-intensive visual question answering through late interaction over fine-grained visual tokens (Lin et al., 2023), hierarchical retrieval from external knowledge sources (Caffagni et al., 2024), or learned filtering of the retrieved evidence (Ling et al., 2025). For documents containing rich visual information, two main strategies have emerged. One directly operates on visual content by encoding rendered document pages with vision–language retrievers, thereby preserving layout and visual information during retrieval (Yu et al., 2024; Faysse et al., 2025; Cho et al., 2024; Suri et al., 2024). The other converts documents into text using a parser (Wang et al., 2024a) and applies standard text retrieval, offering a more efficient and modality-agnostic solution at the cost of fine-grained visual information. Beyond these two strategies, recent work also explores routing queries to different retrieval sources and granularities based on their information needs (Yeo et al., 2025). Nevertheless, these approaches rely on flat retrieval of independently scored units, without explicitly modeling the relations among the retrieved information. Moreover, the concatenated evidence may not be fully utilized by the model (Liu et al., 2026). 2.3. Graph RAG Graph RAG addresses the fragmentation and redundancy in general RAG by retrieving over structure (Peng et al., 2024; Han et al., 2024), and existing methods divide into those that first induce a graph over an unstructured corpus and retrieve at varying granularity, from communities and summary trees to relational paths (Edge et al., 2024; Guo et al., 2024; Gutiérrez et al., 2024; Sarthi et al., 2024; Chen et al., 2026; Liang et al., 2025), and those that assume a curated graph and focus on reasoning over it via path planning, agentic exploration, subgraph selection, etc. (Wang et al., 2026b; Wang et al., 2026c; He et al., 2024; Wang et al., 2024b); both families remain text-centric. Combining structure with modality has therefore motivated multimodal GraphRAG, which builds MMKGs to support structure-aware multimodal retrieval (Wan and Yu, 2025; Guo et al., 2025; Yuan et al., 2026; Hsiao et al., 2026; Park et al., 2026; Dai et al., 2026; He et al., 2026; Wang et al., 2026a). Nevertheless, these methods majorly share one indexing pipeline in which visual and textual graphs are produced independently and merged only at a later fusion stage, so the textual context accompanying visual-to-graph conversion is either absent or restricted to a fixed window of positionally adjacent chunks. Context is thus selected by proximity rather than relevance, admitting unrelated neighboring text while ignoring a sentence that explicitly discusses the target figure from elsewhere in the document, which induces the modality gap we address. 2.4. Multimodal Knowledge Graphs Knowledge graphs (KGs) (Ji et al., 2022) are structured representations of knowledge that organize information as entities and their relations, providing a high-quality knowledge source for LLMs (Pan et al., 2024). Multimodal Knowledge Graphs (MMKGs) extend this representation so that visual evidence becomes part of the graph itself (Zhu et al., 2024; Chen et al., 2024c). Existing MMKGs can be roughly categorized along two categories. By representation, attribute-based MMKGs attach images to symbolic entities as an additional attribute (Liu et al., 2019; Wang et al., 2020; Alberts et al., 2021), without explicitly modeling the fine-grained content within each image, whereas node-based MMKGs promote visual content to first-class nodes so that objects and their relations are explicitly modeled and traversable (Dai et al., 2026; Wang et al., 2026a); we adopt the latter. By provenance, encyclopedic MMKGs are built by augmenting a pre-existing knowledge graph, which bounds their coverage and leaves them prone to becoming outdated (Park et al., 2025), whereas document-derived MMKGs are constructed from a target corpus by parsing a document into texts, tables, figures, and equations (Wang et al., 2024a) before extracting entities and relations from these units (Wan and Yu, 2025; Guo et al., 2025; Yang et al., 2025). Recent multimodal GraphRAG methods (Wan and Yu, 2025; Guo et al., 2025) mainly focus on node-based MMKGs constructed from multimodal documents, where textual context is important because visual elements are often difficult to fully understand in isolation and are closely related to the textual content that describes or discusses them. The quality of MMKG directly affect the performance of multimodal GraphRAG. However, existing works often utilize only limited textual context when processing visual information, which may not sufficiently capture the semantic connections between visual elements and the textual information. This motivates a more systematic design of textual context for visual information processing and modality fusion. Different from previous works, we provide a fine-grained definition of textual context and investigate how different forms of textual context can support MMKG construction. 3. Background and Preliminaries In this section, we start from the the general formulation of multimodal knowledge graph (MMKG)-based RAG and then progressively focus on the specific problem studied in this work. 3.1. MMKG-based RAG MMKG-based RAG extends conventional GraphRAG to support knowledge sources spanning multiple modalities, such as text and images. Given a multimodal document D, its textual content T and visual elements ℐ=I1,…,INI=I_1,…,I_N can be extracted using a document parser such as MinerU (Wang et al., 2024a). An MMKG-RAG system then constructs an MMKG M=(,ℰ)G_M=(V,E) based on D, where V and ℰE denote the sets of entities and relations, respectively. Given a user query q, the relevant multimodal knowledge in ℳG_M is then retrieved and provided to an LLM or MLLM ℱmodelF_model as external evidence to produce a more accurate response: (1) y^=ℱmodel(q,ℛ(q,ℳ)), y=F_model (q,R(q,G_M) ), where ℛR denotes the retriever and y y is the generated answer. 3.2. Multimodal Knowledge Graph Construction Eq. (1) highlights two key factors that influence the quality of the final answer: the retriever ℛR and the underlying knowledge graph MG_M on which it operates. Since retrieval can only operate on the knowledge represented in the graph, the quality of MG_M plays a fundamental role in the overall performance of MMKG-based RAG. Accordingly, we focus on the construction of MG_M from the document D, rather than on the retrieval stage. Existing MMKG-RAG methods construct ℳG_M by processing each modality separately and merging the results: a text-based graph tG^t is built from T and an image-based graph vG^v from ℐI, independently of each other. The two graphs are then integrated into a unified MMKG: (2) ℳ=ffusion(t,v),G_M=f_fusion (G^t,G^v ), where ffusionf_fusion is a modality fusion module which can identify entities of vG^v and tG^t that denote the same underlying object and then pair them. For ffusionf_fusion, we follow the fusion procedure proposed in previous work (Wan and Yu, 2025) and keep its overall workflow fixed throughout this study. For each visual entity, candidate textual entities are collected from neighboring text chunks and partitioned by clustering, after which an LLM identifies its textual counterpart from the most relevant cluster. Unmatched visual entities are enriched with textual context, and the resulting cross-modal alignments are used to fuse the image and text KGs into a unified MMKG. Constructing MG_M from D therefore involves constructing the textual and visual branches and subsequently fusing them. Among the two branches, textual graph construction inherits the well-established pipeline of text-based GraphRAG (Edge et al., 2024), whereas vision-based graph construction encounters a greater challenge, as visual information is more susceptible to information loss during its transformation into graph representations. Visual information also needs to be transformed into entities and relations, and information not captured during this process may be unavailable to subsequent stages, including modality fusion and information retrieval. We therefore further narrow our focus to the construction of the vision-based graph vG^v, while also considering its subsequent integration with the textual graph through modality fusion. Building vG^v, however, is not a purely visual problem. Although the pipeline above treats each modality largely independently, a figure or table can rarely be interpreted on its own, since much of its meaning is carried by the prose that introduces and discusses it. Existing methods consider such textual context only to a limited extent, primarily using text chunks located near the image in the image-to-graph module. However, richer contextual information that is semantically relevant to the image may be distributed across other parts of the document and remains largely underexplored. Textual context for visual elements therefore has the potential to improve both the quality of vG^v and its subsequent fusion with textual knowledge, yet its design and utilization remain underexplored by existing methods. Thus, starting from the end-to-end MMKG-based RAG formulation in Eq. (1), we progressively narrow our focus to the construction of ℳG_M, and more specifically, to the definition of textual context associated with visual elements and its utilization during vision-based graph construction and modality fusion. Accordingly, the problem studied in this work can be stated as follows: given a multimodal document D with textual content T and a set of visual elements ℐI, how can we establish meaningful and comprehensive textual context from T for each visual element Ii∈ℐI_i and effectively utilize such context to construct a higher-quality MMKG? Our framework for addressing this problem is presented in the following section. 4. Method Figure 3. An overview of the designed textual context in our work. Given a target image and texts in the document, we design different levels of contextual information, including local context with different granularities and global context.An overview of the designed textual context in our work. Given a target image and texts in the document, we design different levels of contextual information, including local context with different granularities and global context. Figure 4. An overview of context utilization during MMKG construction. The designed textual contexts can be incorporated into both the Image2Graph and modality fusion stages, individually or jointly, to support visual knowledge extraction and cross-modal knowledge fusion. In practice, an image-based graph is constructed for each visual element and subsequently fused with the text-based graph. For clarity, only one image-based graph is illustrated in the figure.An overview of context utilization during MMKG construction. The designed textual contexts can be incorporated into the Image2Graph and modality fusion stages, individually or jointly, to support visual knowledge extraction and cross-modal knowledge fusion. In this section, we present CEMMKG, our proposed framework for the problem defined in Section 3.2. Given a visual element of a document, CEMMKG constructs a textual context for it and then utilizes that context across the stages of MMKG construction. An overview of the framework is shown in Figure 4. Section 4.1 addresses what textual information should be selected as context for a visual element and at what granularity it should be organized, and Section 4.2 addresses how the resulting context is utilized during MMKG construction. Throughout this section, we reuse the textual content T and visual content set ℐI introduced in Section 3.1, and additionally leveraging their internal structure recovered by the parser. The textual content T is an ordered sequence of paragraphs, with each paragraph being an ordered sequence of sentences, and we use S to denote the set of all sentences in T. Each visual element Ii∈ℐI_i carries an identifier id(Ii)id(I_i) recovered from its caption, such as Figure 3 or Table 2. 4.1. Multi-Granularity Textual Context We construct the context from two complementary types of contextual information, distinguished by which part of T they draw upon: local context, which captures fine-grained details tied to a specific visual element, and global context, which captures the overall content of the document at a coarse-grained level. An overview of the designed context is shown in Figure 3. Next, we will introduce more details about each type. 4.1.1. Local Context For each visual element IiI_i, we define its local context as textual information drawn from a specific region of T, such as a set of sentences within a certain paragraph. Local context focuses on specific details of the document rather than its overall content, and is particularly important for understanding visual information, since figures and tables often cannot be fully interpreted in isolation. We draw it from two complementary sources: the text that immediately surrounds a visual element, and the text that explicitly refers to it. The latter source requires locating the sentences that mention the target element. A visual element is typically introduced at one point of a document and then discussed in detail elsewhere, and the discussing text is often far more informative about it than its positional neighbors. Using the identifier id(Ii)id(I_i) recovered during document parsing, we therefore define the reference set of IiI_i as (3) ℛi=s∈|id(Ii) is mentioned in s.R_i= \\,s \; |\;id(I_i) is mentioned in s\, \. That is, all sentences that contain a textual reference to IiI_i, such as “Figure 3 shows …”. Note that ℛiR_i is determined by mention rather than by position. Hence, its elements may lie far away from IiI_i in the document. On this basis, we define the components of local context. The first is inherited from previous work and always retained; the remaining three are reference-based and describe the same reference set at different scopes and information densities: • Surrounding Text ci,surrLc^L_i,surr: the text chunks lying immediately before and after IiI_i in the document. Previous work (Wan and Yu, 2025; Guo et al., 2025) has demonstrated the importance of involving the text surrounding an image as meaningful context of the image, since a visual element and its adjacent narrative are usually introduced together. To this point, we involve it as part of the local context to provide more comprehensive information concerning the image. This component captures the local context used by existing methods, as discussed in Section 3.2, while our overall context design further incorporates additional richer contextual information beyond this local scope. • Reference Sentence ci,sentLc^L_i,sent: for every sentence in ℛiR_i, that sentence together with the sentences immediately preceding and following it. This is the most concise form of reference-based context, providing fine-grained textual information that directly describes or complements the information presented in the corresponding image. • Reference Paragraph ci,paraLc^L_i,para: the full paragraphs in which the sentences of ℛiR_i appear. Paragraphs cover a broader scope, offering additional background and supporting information that may not be fully captured by the reference sentences alone, at the cost of a lower information density. • Reference Paragraph Summary ci,sumLc^L_i,sum: a summary of the reference paragraphs generated by a large language model (LLM), which is designed to retain the information relevant to the target image while filtering out redundant content from the original paragraphs. It thus sits between the two forms above, seeking the coverage of a paragraph at a density closer to that of a sentence. Since the three reference-based components provide contextual information for the same references at different granularities, they are considered alternative context configurations rather than being used jointly. Let ci,refLc^L_i,ref denote the selected reference-based component. The local context of IiI_i is then defined as: (4) ciL=ci,surrL,ci,refL,ci,refL∈ci,sentL,ci,paraL,ci,sumL.c^L_i= \c^L_i,surr,\;c^L_i,ref \, c^L_i,ref∈ \c^L_i,sent,\;c^L_i,para,\;c^L_i,sum \. The optimal granularity is not immediately clear, as a broader context could provide more supporting information but may also dilute the information most relevant to IiI_i. We therefore treat the granularity as a design dimension of CEMMKG and characterize its effect empirically in Section 5.2.1. Because a visual element may be referenced multiple times throughout a long document, ℛiR_i can become large, resulting in a correspondingly long context. As overly long context degrades how effectively (M)LLMs utilize the supplied evidence (Liu et al., 2024b; Cuconasu et al., 2024), we bound the local context in two ways. First, we retain only the leading reference sites in document order, and use fewer of them at the fusion stage than at the image-to-graph stage, since alignment needs only enough text to name the entities involved. Second, we cap the length of the reference-based component and that of the global context separately, so that neither can crowd out the other. 4.1.2. Global Context We define global context as textual information derived from the entire textual content T. In contrast to the previously defined local context, global context refers to a document-level semantic representation that captures the overarching knowledge of the entire document. Global context could provide broader semantic information that might help interpret visual elements beyond local-level information and facilitates cross-modal knowledge integration. To be specific, we define the following two alternative forms of global context: • Abstract cabsGc^G_abs: the document’s original abstract extracted from T and directly used as global context. • Document Summary csumGc^G_sum: a summary of the whole textual content T generated by a LLM, which captures and integrates information distributed throughout the document into a unified summary. Unlike the local components, which are used jointly, the two global forms are interchangeable and are determined by document availability: we use cabsGc^G_abs for documents that provide an abstract and fall back to csumGc^G_sum otherwise. Since many real-world documents fall outside the academic domain and may not contain an abstract, the latter case is common in practice. We use cGc^G to denote the selected form of global context. Note that cGc^G is defined at the document level and is therefore shared by all visual elements in D, whereas ciLc^L_i is specific to IiI_i. 4.1.3. Constructed Context Combining the two components above, the context constructed for visual element IiI_i is (5) C~i=fcontext(Ii,)=ciL,cG, C_i=f_context(I_i,T)= \c^L_i,\;c^G \, where fcontextf_context denotes the context construction procedure defined in this subsection. Every component of C~i C_i is grounded in the textual content of the document: ci,surrLc^L_i,surr, ci,sentLc^L_i,sent, ci,paraLc^L_i,para and cabsGc^G_abs are subsequences of T, whereas ci,sumLc^L_i,sum and csumGc^G_sum are LLM-generated compressions of such subsequences. The distinction between the two types therefore reduces to which part of T is drawn upon: local context is localized around IiI_i, either by position through ci,surrLc^L_i,surr or by reference through ℛiR_i, whereas global context spans T as a whole. The next subsection describes how the two components of C~i C_i are utilized across the stages of MMKG construction. 4.2. Multi-Stage Context Utilization Having constructed C~i C_i, we next introduce how it can be used for MMKG construction. As illustrated in Figure 4, an MMKG is assembled along two branches that meet at modality fusion, and the constructed context can be injected at the two points marked by ⊕ : when a visual element is turned into an image-based graph, and when that graph is fused with the text-based graph. Rather than incorporating the constructed context only once, we propose multi-stage context utilization, where different construction stages utilize different components of C~i C_i according to their specific objectives. For image-to-graph construction, both image-specific information and document-level context can facilitate a more comprehensive understanding of the visual element; therefore, we provide the full context C~i C_i at this stage. In contrast, the fusion stage aims to establish correspondences between image- and text-derived entities. We therefore provide only the local context ciLc^L_i, which provides more focused evidence for entity alignment. 4.2.1. Context-Augmented Image-to-Graph Construction Images in documents are often difficult to interpret in isolation, as their semantics are closely related to both specific textual descriptions and the overall topic of the document. We therefore augment each visual element with both local and global contextual information during image-to-graph construction. Existing methods build the image-based graph of a visual element IiI_i as iv=fv(Ii,Ci)G^v_i=f_v(I_i,C_i), where fvf_v is an image-to-graph module, such as the Image2Graph module of MMGraphRAG (Wan and Yu, 2025), and CiC_i is the limited surrounding text discussed in Section 3.2. We leave the backbone fvf_v unchanged and replace only its context argument with the constructed context: (6) iv=fv(Ii,C~i),G^v_i=f_v (I_i,\; C_i ), where ivG^v_i denotes the resulting image-based entity-relation graph. This corresponds to the left ⊕ of Figure 4. Inside fvf_v, the context conditions the MLLM that describes the visual element before entities and relations are extracted from that description, so a richer context propagates into every node and edge of ivG^v_i. Our modification is thus confined to the context supplied to the backbone. The two levels of context provide complementary semantic information for visual knowledge extraction. Local context provides fine-grained evidence directly related to the target image, supporting the interpretation of image-specific concepts and relations. In contrast, global context provides a document-level semantic reference that connects the image to the broader topic of the document. By jointly incorporating both levels of context, the image-to-graph process can extract visual knowledge that is consistent with both image-specific textual evidence and the overall semantics of the document. 4.2.2. Context-Guided Multimodal Knowledge Graph Fusion After constructing the image-based graphs, we explore how to better use the designed contexts to integrate them with the text-based knowledge graph extracted from the document. At this stage, the objective shifts from interpreting individual images to establishing correspondences between image- and text-derived knowledge. We utilize the local contexts as additional evidence to guide cross-modal knowledge integration. Let tG^t denote the text-based knowledge graph constructed from document D, and recall that |ℐ|=N|I|=N. The unified MMKG is then obtained as (7) ℳ=ffusion(t,ivi=1N,ciLi=1N),G_M=f_fusion (G^t,\;\G^v_i\_i=1^N,\;\c^L_i\_i=1^N ), where ffusion(⋅)f_fusion(·) denotes the cross-modal knowledge integration process. Relative to the generic formulation in Eq. (2), the local contexts ciLi=1N\c^L_i\_i=1^N are supplied as an additional argument. This corresponds to the right ⊕ of Figure 4, and we now make explicit how ciLc^L_i acts there. Recall from Section 3.2 that ffusionf_fusion aligns each visual entity with candidate textual entities derived from its associated textual context. In the backbone method, this context is limited to text chunks surrounding IiI_i, which may exclude relevant information distributed elsewhere in the document. We therefore augment the available context with the designed local context ciLc_i^L, which incorporates relevant information beyond these surrounding chunks. This allows a broader range of relevant textual entities to be considered as potential alignment candidates. The LLM then determines the final alignment following the original fusion procedure. In this way, ciLc_i^L facilitates the alignment of image-derived entities with their counterparts in the text-based knowledge graph, thereby supporting more effective cross-modal knowledge integration and reducing ambiguity during modality fusion. Across the two stages, the constructed context serves two complementary roles: it supports a more comprehensive understanding of each visual element during Image2Graph construction and provides relevant textual evidence for aligning the resulting visual entities with their textual counterparts during modality fusion. Table 1. Main results of different textual context designs on the VisionHeavy subset. Results are reported as strict full-credit accuracy / official MMLongBench-Doc soft accuracy (%). We additionally report results by visual-content type for a more fine-grained comparison, where each question may be associated with multiple types (Figure, Chart, and Table). Misc contains the remaining questions without any of these three labels. Method Additional Local Context Global Context Figure Table Chart Misc Overall Direct Inference – – 6.25 / 6.25 4.00 / 4.00 0.00 / 0.00 21.95 / 21.95 11.32 / 11.32 MMGraphRAG – – 6.25 / 6.25 20.00 / 22.46 31.25 / 31.25 34.15 / 34.15 23.58 / 24.17 Ours + MMGraphRAG Ref Paragraph Doc-level 12.50 / 12.50 16.00 / 16.00 31.25 / 31.25 26.83 / 26.83 22.64 / 22.64 Ref Paragraph Summary Doc-level 15.63 / 15.63 24.00 / 26.46 25.00 / 25.00 43.90 / 46.76 31.13 / 32.82 Ref Sentence Doc-level 21.88 / 24.70 24.00 / 24.00 31.25 / 31.25 51.22 / 54.01 34.91 / 36.84 5. Experiments In this section, we evaluate the effectiveness of our CEMMKG to answer the following research questions: (i) (RQ1): How do different granularities of local textual context defined in CEMMKG affect the performance of MMKG-based RAG? (i) (RQ2): Can CEMMKG be effectively applied to different MMKG-based RAG methods? 5.1. Experimental Setup 5.1.1. Datasets Following previous work (Wan and Yu, 2025; Guo et al., 2025), we evaluate our method on MMLongBench-Doc (Ma et al., 2024), a document question answering benchmark covering diverse document types. However, a portion of the questions in MMLongBench-Doc are either unanswerable or can be answered without visual information. Meanwhile, conducting a single experiment on the complete MMLongBench-Doc benchmark is highly time-consuming which makes it impractical to evaluate all model configurations. Therefore, we select a subset named VisionHeavy from MMLongBench-Doc. The selection follows two criteria: First, a document should be vision-intensive, meaning that a high proportion of its answerable questions require evidence from visual information such as figures. Second, a document should contain non-local evidence dependencies, meaning that answering its questions requires connecting visual elements with evidence beyond their neighboring textual context. The resulting VisionHeavy subset contains 106 questions. Among the answerable questions, 80.2% require visual information, and 43.2% rely on information distributed across multiple pages. The selected subset spans six of the seven document types in the original MMLongBench-Doc benchmark (Ma et al., 2024), including academic paper, administration&industry files, brochure, guideline, research report, and tutorial/workshop. 5.1.2. Baseline Methods We include the following two types of methods as baselines for comparison: • Direct Inference: A multimodal large language model directly answers the question based on the provided document. • MMKG-based RAG: Multimodal RAG methods that can construct unified multimodal knowledge graph and perform retrieval over the resulting graph. We consider RAG-Anything (Guo et al., 2025) and MMGraphRAG (Wan and Yu, 2025) in our work. 5.1.3. Evaluation Metrics Following the released MMLongBench-Doc evaluation protocol (Ma et al., 2024), we first use Llama-3.1-70B-Instruct to extract a canonical answer from each free-form response and then apply the released type-aware deterministic scorer. Let si∈[0,1]s_i∈[0,1] denote the resulting per-question score. We report the released benchmark accuracy, 1N∑i=1Nsi 1N _i=1^Ns_i, as soft accuracy, and additionally report a derived strict full-credit accuracy, 1N∑i=1N[si=1] 1N _i=1^N1[s_i=1]. Fractional scores arise from ANLS for eligible string and list answers, whereas integer, floating-point, and exact-match cases are scored binarily. Table 1 reports results as strict / soft accuracy. 5.1.4. Implementation Details All experiments are conducted on servers equipped with four NVIDIA A6000 48GB GPUs. Following previous works (Wan and Yu, 2025; Guo et al., 2025), we use MinerU (Wang et al., 2024a) to parse PDF documents and extract content from different modalities. In our experiments, we adopt both RAG-Anything (Guo et al., 2025) and MMGraphRAG (Wan and Yu, 2025) as backbone frameworks to demonstrate the effectiveness of the proposed CEMMKG. When using MMGraphRAG as the backbone, We employ Qwen2.5-72B-Instruct-AWQ (Team, 2024) for text-based knowledge graph construction, text summarization, and textual generation. InternVL2.5-38B-MPO-AWQ (Chen et al., 2024a) is used to generate image descriptions, and perform multimodal question answering. We use stella-en-1.5B-v5 (Zhang et al., 2024) to encode graph nodes and user queries. Llama-3.1-70B-Instruct (Grattafiori et al., 2024) is used to extract final answers from generated responses. We use deterministic decoding across all generation stages, setting the temperature to 0 and top-p to 1. We set the maximum output length of the answer extractor to 1,024 tokens. We retain up to five explicit textual references for image description and up to three for graph fusion. We modify the retrieval stage in MMGraphRAG by introducing a visual selection policy that prioritizes explicitly referenced Figure or Table identifiers. When no such reference is available, BM25 is used to identify visuals with strong lexical matches to the query; otherwise, the original dense graph retrieval results are retained. For RAG-Anything, we retain its native mix-mode retrieval and use GPT-4o-mini as the backbone model. To ensure a fair comparison, we use the same retrieval procedure and hyperparameters across all CEMMKG context configurations for each backbone framework. 5.2. Multimodal GraphRAG Performance 5.2.1. Overall Performance To asnwer RQ1, we conduct comprehensive experiments based on different textual context configurations. Table 1 presents the overall performance of using different textual context designs on the VisionHeavy subset. Among all evaluated configurations, incorporating the reference sentence(s) as additional local context together with document-level global context delivers the best overall performance, achieving a hard accuracy of 34.91% and a soft accuracy of 36.84%. This substantially outperforms MMGraphRAG, which achieves 23.58% hard accuracy and 24.17% soft accuracy. Similarly, when we use the reference paragraph summary, the performance is also strong with a hard accuracy of 31.13% and a soft accuracy of 32.82%. These results demonstrate that incorporating appropriately designed textual context during MMKG construction can effectively improve downstream multimodal GraphRAG performance. However, we can observe that directly using the full reference paragraph(s) as additional local context achieves an overall soft accuracy of only 22.64%, even lower than the 24.17% of MMGraphRAG. This result suggests that simply introducing more textual context does not necessarily lead to better performance. The possible reason is that the complete paragraph(s) could contain information that is only weakly related to the corresponding visual content, introducing irrelevant or redundant context during MMKG construction, which might affect both the visual information processing and modality fusion. This observation indicates that the effectiveness of local context depends not only on the amount of contextual information provided, but also on its granularity and information density. This suggests that effective local context should be appropriately scoped and closely aligned with the visual content, providing sufficient relevant information while minimizing contextual noise. Overall, these results underscore the importance of choosing an appropriate level of textual granularity for high-quality MMKG construction and improved downstream multimodal GraphRAG performance. 5.2.2. Performance across Different Types To better understand the effectiveness of CEMMKG, we further conduct experiments according to the types of visual content involved in each question, including Figure, Chart, and Table. Please note that these labels are not mutually exclusive, as a single question may involve multiple types of visual content. The per-type results are reported to provide a more fine-grained understanding of CEMMKG performance. As shown in Table 1, the reference-sentence context brings the most substantial improvement on Figure-related questions, increasing soft accuracy from 6.25% to 24.70%, while maintaining the same soft accuracy of 31.25% as MMGraphRAG on Chart-related questions. For Table-related questions, the reference paragraph summary achieves the best soft accuracy of 26.46%, compared with 22.46% for MMGraphRAG. These results suggest that different textual context designs could provide different benefits depending on the category of visual information involved. Interestingly, the proposed context design also yields substantial improvements on the Misc questions, increasing soft accuracy from 34.15% to 54.01% and hard accuracy from 34.15% to 51.22%. Please note that Misc refers to questions without Figure, Chart, or Table labels, but these questions may still involve multimodal information. One possible explanation for this improvement is that the designed contextual information enhances not only the representation of individual image-based graphs but also their integration with textual knowledge during MMKG construction. These improvements may therefore also benefit questions that rely on textual knowledge or cross-modal semantic connections, even when they do not explicitly require visual content. Table 2. Performance of CEMMKG with RAG-Anything as the backbone under different context configurations. Results are reported as strict / soft accuracy (%). Method Additional Local Context Strict / Soft Acc. (%) RAG-Anything – 28.89 / 36.18 Ours + RAG-Anything Ref Paragraph 28.89 / 34.49 Ours + RAG-Anything Ref Paragraph Summary 31.11 / 33.13 Ours + RAG-Anything Ref Sentence 35.56 / 41.12 5.3. Applicability to Different Methods The results of CEMMKG reported in Table 1 are primarily based on MMGraphRAG. To investigate whether CEMMKG is also applicable to other multimodal GraphRAG methods (RQ2), we further conduct experiments with RAG-Anything. We integrate CEMMKG into RAG-Anything’s visual representation stage by augmenting the description prompt for each image or table with the document-level summary and any available explicit reference sentence(s) before generating the structured visual description. The resulting description is then processed by RAG-Anything’s original multimodal graph construction pipeline. Unlike MMGraphRAG, RAG-Anything already supplies each visual with native contextual text that can span neighboring pages. Therefore, to better evaluate the effectiveness of the designed textual context, we select from VisionHeavy a subset of questions that require cross-page references. As shown in Table 2, the original RAG-Anything achieves a strict accuracy of 28.89% and a soft accuracy of 36.18%. Incorporating our designed reference sentence(s) as extra contextual information further improves these results to 35.56% and 41.12%, respectively, demonstrating its effectiveness even on a strong MMKG-based RAG baseline. We also observe a trend consistent with MMGraphRAG: performance improves progressively from reference paragraphs, to their LLM-generated summaries, and further to reference-sentence context. 6. Conclusion In this work, we systematically investigate the role of textual context in GraphRAG-oriented MMKG construction. We introduce CEMMKG, a framework that systematically defines and utilizes textual context in both visual information processing and modality fusion. Specifically, CEMMKG designs textual context for visual elements from complementary perspectives, including local context at different granularities and global context based on different document-level representations. We further explore how the designed context can be effectively incorporated across different stages of MMKG construction. Extensive experiments based on different context configurations demonstrate that appropriately designed textual context can effectively improve the performance of MMKG-based RAG. Moreover, CEMMKG can be effectively applied to different MMKG-based RAG methods, demonstrating its broader applicability. Overall, our findings highlight the importance of explicitly considering and leveraging contextual information when constructing MMKGs for multimodal Graph RAG. Future work could extend our framework to a broader range of modalities, such as video and audio, and investigate how contextual information should be defined and utilized for different modality combinations. Appendix A GenAI Usage Disclosure The authors used generative AI for writing assistance and all the generated content was reviewed by the authors. GenAI tools were also used to generate several icons for visualization purposes in the figures. AI is also an integral part of the proposed method, where it is used for textual graph generation, visual information processing, multimodal information fusion, and related tasks. These uses are described in detail in the Method section. GenAI tools were also used to assist with the implementation of parts of the code. In addition, AI was used to evaluate method performance, as described in the Experiments section. Ethical Considerations In this work, we study how to define and utilize textual context information to support multimodal knowledge graph construction. All experiments are conducted using publicly available models and datasets. We do not see any significant ethical concerns or negative societal impacts arising from this work. References Abootorabi et al. (2025) M. M. Abootorabi, A. Zobeiri, M. Dehghani, M. Mohammadkhani, B. Mohammadi, O. Ghahroodi, M. S. Baghshah, and E. Asgari Ask in any modality: a comprehensive survey on multimodal retrieval-augmented generation. In Findings of the Association for Computational Linguistics: ACL 2025, p. 16776–16809. Cited by: §1, §2.2. Alberts et al. (2021) H. Alberts, N. Huang, Y. Deshpande, Y. Liu, K. Cho, C. Vania, and I. Calixto VisualSem: a high-quality knowledge graph for vision and language. In Proceedings of the 1st Workshop on Multilingual Representation Learning (MRL), p. 138–152. Cited by: §2.4. Bai et al. (2023) J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou Qwen-VL: a versatile vision-language model for understanding, localization, text reading, and beyond. External Links: 2308.12966 Cited by: §2.1. Bai et al. (2025) S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin Qwen2.5-vl technical report. External Links: 2502.13923 Cited by: §2.1. Bai et al. (2024) Z. Bai, P. Wang, T. Xiao, T. He, Z. Han, Z. Zhang, and M. Z. Shou Hallucination of multimodal large language models: a survey. External Links: 2404.18930 Cited by: §1, §2.1. Brown et al. (2020) T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33, p. 1877–1901. Cited by: §2.1. Caffagni et al. (2024) D. Caffagni, F. Cocchi, N. Moratelli, S. Sarto, M. Cornia, L. Baraldi, and R. Cucchiara Wiki-LLaVA: hierarchical retrieval-augmented generation for multimodal LLMs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, p. 1818–1826. Cited by: §2.2. Chen et al. (2026) B. Chen, Z. Guo, Z. Yang, Y. Chen, J. Chen, Z. Liu, C. Shi, and C. Yang PathRAG: pruning graph-based retrieval augmented generation with relational paths. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 30183–30191. Cited by: §2.3. Chen et al. (2022) W. Chen, H. Hu, X. Chen, P. Verga, and W. W. Cohen MuRAG: multimodal retrieval-augmented generator for open question answering over images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 5558–5570. Cited by: §2.2. Chen et al. (2024a) Z. Chen, W. Wang, Y. Cao, Y. Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu, L. Gu, X. Wang, Q. Li, Y. Ren, Z. Chen, J. Luo, J. Wang, T. Jiang, B. Wang, C. He, B. Shi, X. Zhang, H. Lv, Y. Wang, W. Shao, P. Chu, Z. Tu, T. He, Z. Wu, H. Deng, J. Ge, K. Chen, K. Zhang, L. Wang, M. Dou, L. Lu, X. Zhu, T. Lu, D. Lin, Y. Qiao, J. Dai, and W. Wang Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. External Links: 2412.05271 Cited by: §5.1.4. Chen et al. (2024b) Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, B. Li, P. Luo, T. Lu, Y. Qiao, and J. Dai InternVL: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 24185–24198. Cited by: §2.1. Chen et al. (2024c) Z. Chen, Y. Zhang, Y. Fang, Y. Geng, L. Guo, X. Chen, Q. Li, W. Zhang, J. Chen, Y. Zhu, J. Li, X. Liu, J. Z. Pan, N. Zhang, and H. Chen Knowledge graphs meet multi-modal learning: a comprehensive survey. External Links: 2402.05391 Cited by: §2.4. Cho et al. (2024) J. Cho, D. Mahata, O. Irsoy, Y. He, and M. Bansal M3DocRAG: multi-modal retrieval is what you need for multi-page multi-document understanding. External Links: 2411.04952 Cited by: §2.2. Cuconasu et al. (2024) F. Cuconasu, G. Trappolini, F. Siciliano, S. Filice, C. Campagnano, Y. Maarek, N. Tonellotto, and F. Silvestri The power of noise: redefining retrieval for RAG systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 719–729. Cited by: §2.1, §4.1.1. Dai et al. (2026) S. Dai, Q. Huang, X. You, and J. Yu MG2^2-RAG: multi-granularity graph for multimodal retrieval-augmented generation. External Links: 2604.04969 Cited by: §2.3, §2.4. Edge et al. (2024) D. Edge, H. Trinh, N. Cheng, J. Bradley, A. Chao, A. Mody, S. Truitt, D. Metropolitansky, R. O. Ness, and J. Larson From local to global: a graph rag approach to query-focused summarization. External Links: 2404.16130 Cited by: §1, §2.3, §3.2. Faysse et al. (2025) M. Faysse, H. Sibille, T. Wu, B. Omrani, G. Viaud, C. Hudelot, and P. Colombo ColPali: efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations (ICLR), Cited by: §2.2. Gao et al. (2023) Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, M. Wang, and H. Wang Retrieval-augmented generation for large language models: a survey. External Links: 2312.10997 Cited by: §1. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, et al. The llama 3 herd of models. External Links: 2407.21783 Cited by: §5.1.4. Guo et al. (2025) Z. Guo, X. Ren, L. Xu, J. Zhang, and C. Huang RAG-anything: all-in-one rag framework. External Links: 2510.12323 Cited by: §1, §1, §2.3, §2.4, §2.4, 1st item, 2nd item, §5.1.1, §5.1.4. Guo et al. (2024) Z. Guo, L. Xia, Y. Yu, T. Ao, and C. Huang LightRAG: simple and fast retrieval-augmented generation. External Links: 2410.05779 Cited by: §2.3. Gutiérrez et al. (2024) B. J. Gutiérrez, Y. Shu, Y. Gu, M. Yasunaga, and Y. Su HippoRAG: neurobiologically inspired long-term memory for large language models. In Advances in Neural Information Processing Systems, Vol. 37, p. 59532–59569. Cited by: §2.3. Han et al. (2024) H. Han, Y. Wang, H. Shomer, K. Guo, J. Ding, Y. Lei, M. Halappanavar, R. A. Rossi, S. Mukherjee, X. Tang, Q. He, Z. Hua, B. Long, T. Zhao, N. Shah, A. Javari, Y. Xia, and J. Tang Retrieval-augmented generation with graphs (GraphRAG). External Links: 2501.00309 Cited by: §2.3. He et al. (2024) X. He, Y. Tian, Y. Sun, N. V. Chawla, T. Laurent, Y. LeCun, X. Bresson, and B. Hooi G-Retriever: retrieval-augmented generation for textual graph understanding and question answering. In Advances in Neural Information Processing Systems, Vol. 37, p. 132876–132907. Cited by: §2.3. He et al. (2026) X. He, Y. Wang, W. Fan, Q. Li, Q. Zhang, Y. Chang, and X. Wang HVM-GraphRAG: holistic-view multimodal graph retrieval-augmented generation on complex document. External Links: 2607.24861 Cited by: §2.3. Hsiao et al. (2026) C. Hsiao, Y. Wang, T. Lin, Y. Yeh, and C. Chen MegaRAG: multimodal knowledge graph-based retrieval augmented generation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §2.3. Huang et al. (2025) L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43 (2). External Links: ISSN 1046-8188 Cited by: §2.1. Ji et al. (2022) S. Ji, S. Pan, E. Cambria, P. Marttinen, and P. S. Yu A survey on knowledge graphs: representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems 33 (2), p. 494–514. Cited by: §1, §2.4. Lewis et al. (2020) P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. Kiela Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, Vol. 33, p. 9459–9474. Cited by: §1. Li et al. (2023a) J. Li, D. Li, S. Savarese, and S. Hoi BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning (ICML), p. 19730–19742. Cited by: §2.1. Li et al. (2023b) Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. Wen Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 292–305. Cited by: §2.1. Liang et al. (2025) L. Liang, Z. Bo, Z. Gui, Z. Zhu, L. Zhong, P. Zhao, M. Sun, Z. Zhang, J. Zhou, W. Chen, W. Zhang, and H. Chen KAG: boosting LLMs in professional domains via knowledge augmented generation. In Companion Proceedings of the ACM Web Conference 2025, p. 334–343. Cited by: §2.3. Lin et al. (2023) W. Lin, J. Chen, J. Mei, A. Coca, and B. Byrne Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering. In Advances in Neural Information Processing Systems, Vol. 36, p. 22820–22840. Cited by: §2.2. Ling et al. (2025) Z. Ling, Z. Guo, Y. Huang, Y. An, S. Xiao, J. Lan, X. Zhu, and B. Zheng MMKB-RAG: a multi-modal knowledge-based retrieval-augmented generation framework. External Links: 2504.10074 Cited by: §2.2. Liu et al. (2024a) H. Liu, C. Li, Y. Li, and Y. J. Lee Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 26296–26306. Cited by: §1, §2.1. Liu et al. (2023) H. Liu, C. Li, Q. Wu, and Y. J. Lee Visual instruction tuning. In Advances in Neural Information Processing Systems, Vol. 36, p. 34892–34916. Cited by: §1, §2.1. Liu et al. (2026) J. Liu, J. Gu, S. Chen, J. Chen, and Z. Wang Lost at the end: primacy bias in multimodal retrieval-augmented question answering. External Links: 2606.16494 Cited by: §2.2. Liu et al. (2024b) N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang Lost in the middle: how language models use long contexts. Transactions of the Association for Computational Linguistics 12, p. 157–173. External Links: Document Cited by: §2.1, §4.1.1. Liu et al. (2019) Y. Liu, H. Li, A. Garcia-Duran, M. Niepert, D. Onoro-Rubio, and D. S. Rosenblum MMKG: multi-modal knowledge graphs. In The Semantic Web – 16th International Conference (ESWC), p. 459–474. External Links: Document Cited by: §2.4. Ma et al. (2024) Y. Ma, Y. Zang, L. Chen, M. Chen, Y. Jiao, X. Li, X. Lu, Z. Liu, Y. Ma, X. Dong, P. Zhang, L. Pan, Y. Jiang, J. Wang, Y. Cao, and A. Sun MMLONGBENCH-doc: benchmarking long-context document understanding with visualizations. In Advances in Neural Information Processing Systems, Vol. 37, p. 95963–96010. External Links: Document Cited by: §5.1.1, §5.1.3. OpenAI (2023) OpenAI GPT-4 technical report. External Links: 2303.08774 Cited by: §2.1. Pan et al. (2024) S. Pan, L. Luo, Y. Wang, C. Chen, J. Wang, and X. Wu Unifying large language models and knowledge graphs: a roadmap. IEEE Transactions on Knowledge and Data Engineering 36 (7), p. 3580–3599. External Links: Document Cited by: §2.4. Park et al. (2025) H. Park, J. Seo, M. Jang, H. Park, H. D. Baek, G. Chang, H. Im, and S. Kim VAT-KG: knowledge-intensive multimodal knowledge graph dataset for retrieval-augmented generation. External Links: 2506.21556 Cited by: §2.4. Park et al. (2026) H. Park, J. Seo, J. Mun, H. Park, W. Byeon, S. J. Kim, H. Im, J. Lee, and S. Kim M3^3KG-RAG: multi-hop multimodal knowledge graph-enhanced retrieval-augmented generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.3. Peng et al. (2024) B. Peng, Y. Zhu, Y. Liu, X. Bo, H. Shi, C. Hong, Y. Zhang, and S. Tang Graph retrieval-augmented generation: a survey. External Links: 2408.08921 Cited by: §1, §2.3. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, p. 8748–8763. Cited by: §2.1. Sarthi et al. (2024) P. Sarthi, S. Abdullah, A. Tuli, S. Khanna, A. Goldie, and C. D. Manning RAPTOR: recursive abstractive processing for tree-organized retrieval. In The Twelfth International Conference on Learning Representations (ICLR), Cited by: §2.3. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. External Links: 1707.06347 Cited by: §2.1. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300 Cited by: §2.1. Suri et al. (2024) M. Suri, P. Mathur, F. Dernoncourt, K. Goswami, R. A. Rossi, and D. Manocha VisDoM: multi-document QA with visually rich elements using multimodal retrieval-augmented generation. External Links: 2412.10704 Cited by: §2.2. Team (2024) Q. Team Qwen2.5 technical report. External Links: 2412.15115 Cited by: §5.1.4. Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom Llama 2: open foundation and fine-tuned chat models. External Links: 2307.09288 Cited by: §2.1. Wan and Yu (2025) X. Wan and H. Yu MMGraphRAG: bridging vision and language with interpretable multimodal knowledge graphs. External Links: 2507.20804 Cited by: §1, §1, §2.3, §2.4, §2.4, §3.2, 1st item, §4.2.1, 2nd item, §5.1.1, §5.1.4. Wang et al. (2024a) B. Wang, C. Xu, X. Zhao, L. Ouyang, F. Wu, Z. Zhao, R. Xu, K. Liu, Y. Qu, F. Shang, B. Zhang, L. Wei, Z. Sui, W. Li, B. Shi, Y. Qiao, D. Lin, and C. He MinerU: an open-source solution for precise document content extraction. External Links: 2409.18839 Cited by: §2.2, §2.4, §3.1, §5.1.4. Wang et al. (2020) M. Wang, H. Wang, G. Qi, and Q. Zheng Richpedia: a large-scale, comprehensive multi-modal knowledge graph. Big Data Research 22, p. 100159. External Links: Document Cited by: §2.4. Wang et al. (2026a) X. Wang, B. Hoang, H. Liu, T. Wang, and F. Ma MKG-RAG-Bench: benchmarking retrieval in multimodal knowledge graph-augmented generation. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), Cited by: §2.3, §2.4. Wang et al. (2026b) X. Wang, Z. Wu, Y. Zhong, X. Zhang, S. Wang, and F. Ma GPR: empowering generation with graph-pretrained retriever. In Proceedings of the ACM Web Conference 2026, p. 8349–8352. Cited by: §2.3. Wang et al. (2026c) X. Wang, Y. Zhong, H. Wang, T. Wang, and F. Ma KAMR: grounding generation via knowledge-aligned multi-hop retrieval. External Links: 2607.27136 Cited by: §2.3. Wang et al. (2024b) Y. Wang, N. Lipka, R. A. Rossi, A. Siu, R. Zhang, and T. Derr Knowledge graph prompting for multi-document question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 19206–19214. External Links: Document Cited by: §2.3. Xu et al. (2026) Z. Xu, Z. Wu, Y. Zhou, A. Feng, K. Zhou, S. Woo, K. Ramnath, Y. Tian, X. Qi, W. Qiu, L. L. Cheong, and H. Ding Beyond correctness: rewarding faithful reasoning in retrieval-augmented generation. Transactions on Machine Learning Research. External Links: ISSN 2835-8856 Cited by: §1. Yang et al. (2025) M. Yang, Y. Ren, D. O. Opoku, R. Li, P. Ren, and C. Xing DSRAG: a domain-specific retrieval framework based on document-derived multimodal knowledge graph. In Web Information Systems and Applications – 22nd International Conference (WISA), Cited by: §2.4. Yasunaga et al. (2023) M. Yasunaga, A. Aghajanyan, W. Shi, R. James, J. Leskovec, P. Liang, M. Lewis, L. Zettlemoyer, and W. Yih Retrieval-augmented multimodal language modeling. In Proceedings of the 40th International Conference on Machine Learning (ICML), p. 39755–39769. Cited by: §2.2. Yeo et al. (2025) W. Yeo, K. Kim, S. Jeong, J. Baek, and S. J. Hwang UniversalRAG: retrieval-augmented generation over corpora of diverse modalities and granularities. External Links: 2504.20734 Cited by: §2.2. Yin et al. (2024) S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen A survey on multimodal large language models. National Science Review 11 (12). External Links: Document Cited by: §1, §2.1. Yu et al. (2024) S. Yu, C. Tang, B. Xu, J. Cui, J. Ran, Y. Yan, Z. Liu, S. Wang, X. Han, Z. Liu, and M. Sun VisRAG: vision-based retrieval-augmented generation on multi-modality documents. External Links: 2410.10594 Cited by: §2.2. Yuan et al. (2026) X. Yuan, L. Ning, Q. Ye, W. Fan, and Q. Li mKG-RAG: leveraging multimodal knowledge graphs in retrieval-augmented generation for knowledge-intensive VQA. In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval, External Links: Document Cited by: §2.3. Zhang et al. (2024) D. Zhang, J. Li, Z. Zeng, and F. Wang Jasper and stella: distillation of sota embedding models. External Links: 2412.19048 Cited by: §5.1.4. Zhao et al. (2026) W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, Y. Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J. Nie, and J. Wen A survey of large language models. Frontiers of Computer Science. External Links: Document Cited by: §2.1. Zhu et al. (2024) X. Zhu, Z. Li, X. Wang, X. Jiang, P. Sun, X. Wang, Y. Xiao, and N. J. Yuan Multi-modal knowledge graph construction and application: a survey. IEEE Transactions on Knowledge and Data Engineering 36 (2), p. 715–735. External Links: Document Cited by: §2.4.