Paper deep dive

MG$^2$-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation

Sijun Dai, Qiang Huang, Xiaoxing You, Jun Yu

Year: 2026Venue: arXiv preprintArea: cs.IRType: PreprintEmbeddings: 90

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%

Last extracted: 4/10/2026, 2:15:33 AM

Summary

MG^2-RAG is a lightweight, multi-granularity graph-based framework for Multimodal Retrieval-Augmented Generation (MM-RAG). It addresses the high computational costs and modality fragmentation of existing graph-based methods by using a hierarchical construction pipeline that fuses textual entities and visual objects into unified multimodal nodes. The framework employs a multi-granularity retrieval mechanism that propagates relevance across a heterogeneous graph, enabling structured multi-hop reasoning and achieving state-of-the-art performance with significant speed and cost improvements.

Entities (5)

MG^2-RAG · framework · 100%E-VQA · dataset · 95%Multimodal Large Language Models · model-architecture · 95%EVA-CLIP-8B · encoder · 90%SAM3 · segmentation-model · 90%

Relation Signals (3)

MG^2-RAG → improves → Multimodal Retrieval-Augmented Generation

confidence 95% · MG^2-RAG, a lightweight Multi-Granularity Graph RAG framework that jointly improves graph construction, modality fusion, and cross-modal retrieval.

MG^2-RAG → uses → SAM3

confidence 90% · we perform entity-driven segmentation... into an open-vocabulary segmentation model Φ(SAM3)

MG^2-RAG → uses → EVA-CLIP-8B

confidence 90% · all textual and visual elements are embedded into a shared space via a unified encoder Ψ(·) (i.e., EVA-CLIP-8B)

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Retrieval-Augmented Generation (RAG) mitigates hallucinations in Multimodal Large Language Models (MLLMs), yet existing systems struggle with complex cross-modal reasoning. Flat vector retrieval often ignores structural dependencies, while current graph-based methods rely on costly ``translation-to-text'' pipelines that discard fine-grained visual information. To address these limitations, we propose \textbf{MG$^2$-RAG}, a lightweight \textbf{M}ulti-\textbf{G}ranularity \textbf{G}raph \textbf{RAG} framework that jointly improves graph construction, modality fusion, and cross-modal retrieval. MG$^2$-RAG constructs a hierarchical multimodal knowledge graph by combining lightweight textual parsing with entity-driven visual grounding, enabling textual entities and visual regions to be fused into unified multimodal nodes that preserve atomic evidence. Building on this representation, we introduce a multi-granularity graph retrieval mechanism that aggregates dense similarities and propagates relevance across the graph to support structured multi-hop reasoning. Extensive experiments across four representative multimodal tasks (i.e., retrieval, knowledge-based VQA, reasoning, and classification) demonstrate that MG$^2$-RAG consistently achieves state-of-the-art performance while reducing graph construction overhead with an average 43.3$\times$ speedup and 23.9$\times$ cost reduction compared with advanced graph-based frameworks.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

90,192 characters extracted from source content.

Expand or collapse full text

MG 2 -RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation Sijun Dai 1,† , Qiang Huang 1,† , Xiaoxing You 2 , and Jun Yu 1,* 1 School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) daisijun@stu.hit.edu.cn, huangqiang, yujun@hit.edu.cn 2 School of Computer Science, Hangzhou Dianzi University youxiaoxing@hdu.edu.cn † Equal contribution * Corresponding author Abstract. Retrieval-Augmented Generation (RAG) mitigates halluci- nations in Multimodal Large Language Models (MLLMs), yet existing systems struggle with complex cross-modal reasoning. Flat vector retrieval often ignores structural dependencies, while current graph-based methods rely on costly “translation-to-text” pipelines that discard fine-grained visual information. To address these limitations, we propose MG 2 -RAG, a lightweight Multi-Granularity Graph RAG framework that jointly improves graph construction, modality fusion, and cross-modal retrieval. MG 2 -RAG constructs a hierarchical multimodal knowledge graph by combining lightweight textual parsing with entity-driven visual ground- ing, enabling textual entities and visual regions to be fused into unified multimodal nodes that preserve atomic evidence. Building on this repre- sentation, we introduce a multi-granularity graph retrieval mechanism that aggregates dense similarities and propagates relevance across the graph to support structured multi-hop reasoning. Extensive experiments across four representative multimodal tasks (i.e., retrieval, knowledge- based VQA, reasoning, and classification) demonstrate that MG 2 -RAG consistently achieves state-of-the-art performance while reducing graph construction overhead with an average 43.3×speedup and 23.9×cost reduction compared with advanced graph-based frameworks. Keywords: Multimodal Knowledge Graph· Retrieval-Augmented Gen- eration· Multimodal Large Language Models 1 Introduction Multimodal Large Language Models (MLLMs) have achieved remarkable success in tasks requiring complex cross-modal understanding and reasoning, leading to their rapid adoption across numerous applications [20,58,60]. Despite these ad- vances, deploying MLLMs in knowledge-intensive scenarios still raises important reliability concerns. In particular, MLLMs may produce multimodal hallucina- tions and often lack access to private or domain-specific knowledge, since their parameters are primarily trained on large-scale public corpora [33, 68, 79]. To arXiv:2604.04969v1 [cs.IR] 4 Apr 2026 2Sijun Dai et al. address these limitations, Multimodal Retrieval-Augmented Generation (M- RAG) augments MLLMs with external knowledge bases by retrieving relevant multimodal evidence to ground the generation process [5,32,74]. By incorporat- ing factual context at inference time, M-RAG significantly improves both the reliability and domain adaptability of MLLMs. ExistingWork:Text-CentricKnowledge GraphforMM-RAG MG 2 -RAG:Multi-Granularity Knowledge GraphforMM-RAG ObjectNode Expensive Graph ConstructionText-Centric Graph Topology Lightweight Graph ConstructionMultimodal Node Fusion EntityNode MLLM LLM Text Image Entityname:Cat Description:Acat lookingatlaptop ... Image Text Cat Fig. 1: Existing work employs text-centric graph extraction that discards visual details and incurs high costs. MG 2 -RAG efficiently fuses textual entities and visual objects into unified multimodal nodes while preserving atomic information. Existing M-RAG meth- ods generally fall into two paradigms: vector-based and graph-based approaches. Vector- based M-RAG encodes multi- modal inputs into a shared em- bedding space [57, 61, 64] and retrieves relevant evidence via similarity search [12,13]. While effective in many scenarios, this paradigm typically retrieves iso- lated multimodal elements and largely ignores the structural relationships among different pieces of evidence [9,31,70,78]. Consequently, two fundamen- tal limitations arise: First, the semantic gap between modal- ities can exacerbate modality fragmentation, weakening cross- modal alignment. Second, similarity-based retrieval cannot model logical depen- dencies among retrieved evidence, making complex multi-hop reasoning difficult. To overcome these limitations, recent work has explored graph-based M- RAG, which organizes multimodal knowledge into structured graphs with explicit relational dependencies. By representing entities and their relationships as graph nodes and edges, these approaches enable retrieval through structured traversal rather than flat similarity matching [48,65,73]. Such structured representations allow models to reason over interconnected multimodal facts and improve relia- bility in knowledge-intensive tasks [23,25,80]. As a result, graph-based retrieval has emerged as a promising direction for bridging multimodal alignment and structured reasoning. Despite its promise, existing graph-based M-RAG systems still face several practical challenges. –Expensive Graph Construction: Many existing methods rely heavily on MLLMs to extract relational triplets from multimodal data, resulting in significant overhead when processing large-scale knowledge bases [48,65,73]. –Text-Centric Graph Topology: Visual information is often converted into textual descriptions before constructing the graph [23, 65, 80]. This trans- formation discards fine-grained visual structures and overlooks independent visual entities that may contain important semantic information. MG 2 -RAG: Multi-Granularity Graph for M-RAG3 –Difficult Cross-modal Retrieval: Since these graphs lack unified multi- modal concepts, they struggle to effectively process queries containing both text and images, which limits semantic propagation across modalities and weakens multi-hop reasoning capabilities [48,70]. To address these challenges, we propose MG 2 -RAG, a lightweight Multi- Granularity Graph RAG framework that introduces unified multimodal node fusion together with efficient graph construction. Instead of relying on expensive MLLM-driven triplet extraction, MG 2 -RAG employs a lightweight pipeline that combines textual parsing with entity-driven visual grounding to efficiently con- struct multimodal knowledge graphs. To preserve modality-specific semantics, we fuse textual entities and visual objects into unified multimodal nodes while maintaining a fine-grained hierarchy of textual and visual atomic evidence. Building upon this representation, we further design a multi-granularity graph retrieval mechanism that aggregates dense similarities onto multimodal nodes and propagates relevance through the graph topology. This strategy enables unified processing of multimodal queries and facilitates structured multi-hop reasoning for MLLM generation. Extensive experiments on four representative multimodal tasks, including retrieval, knowledge-based VQA, reasoning, and classification, demonstrate that MG 2 -RAG achieves state-of-the-art performance while maintaining a highly efficient graph construction process. Our main contributions are summarized as follows: –Lightweight Multimodal Knowledge Graph Construction. We propose a lightweight pipeline that bypasses the expensive MLLM-driven triplet extraction process, significantly reducing the time and cost of multimodal knowledge graph construction. –Modality-Preserving Multimodal Node Fusion. We introduce a modality- preserving fusion strategy that integrates textual entities and visual objects into unified multimodal nodes while preserving fine-grained atomic evidence from both modalities. –Multi-Granularity Graph Retrieval. We design a retrieval mechanism that aggregates dense similarities onto multimodal nodes and propagates relevance through the graph topology, enabling unified multimodal retrieval and multi-hop reasoning for MLLM generation. 2 Related Work Retrieval-Augmented Generation (RAG) for LLMs. The paradigm of RAG has evolved from static retrieve-then-generate pipelines to dynamic, agentic reasoning systems. Early work augmented language models with external knowl- edge to mitigate hallucinations, progressing from sparse Wikipedia retrieval [7,39] to dense retrieval [26,42] and web-scale search integration [37]. Subsequent re- search addressed the rigidity of fixed retrieval patterns by introducing active and adaptive mechanisms, allowing models to determine when and what to retrieve, of- ten optimized via reinforcement learning and environmental feedback [2,11,19,30]. More recently, RAG has been tightly integrated with multi-hop reasoning and 4Sijun Dai et al. planning. Hierarchical query decomposition [40,76], iterative retrieval-reasoning loops [22,66], and agentic workflows [41,45,69] now enable models to emulate structured research processes. These advances substantially improve performance on complex, knowledge-intensive tasks. However, most existing approaches re- main grounded in flat vector retrieval, limiting their ability to model structured dependencies. Graph RAG. To overcome the structural limitations of flat retrieval, recent work has incorporated graph-based representations into RAG [16,28,53,62]. Graph RAG frameworks leverage entity-level knowledge graphs and community struc- tures to better capture global dependencies and multi-hop relationships [16,17]. Given the high computational cost of graph construction, subsequent research has focused on scalable indexing strategies, including dual-level retrieval [23], bipartite or relation-free graph designs [34,80], and hierarchical tree-based struc- tures [67,77]. Beyond efficiency, integrating graphs with LLMs has proven crucial for faithful multi-hop reasoning and long-term memory modeling. Iterative graph exploration [35, 63], planning over structured representations [8, 52], memory- inspired architectures [24, 25], and agentic frameworks [15] demonstrate that explicit relational modeling significantly enhances reasoning depth and inter- pretability. Nevertheless, these methods are predominantly text-centric and are not directly designed for multimodal settings. Multimodal RAG. Vision-Language Pre-training (VLP) models [44,61] have significantly advanced multimodal understanding, yet they remain constrained by parametric knowledge. Multimodal RAG (M-RAG) addresses this limitation by introducing external memory for grounding generation in retrieved visual–textual evidence [9, 32, 72]. Recent efforts improve retrieval granularity [18, 54] and incorporate adaptive or self-reflective strategies [5, 12, 75] to filter noise and control retrieval necessity. However, most systems rely on flat vector similarity, which struggles with structured multi-hop reasoning across modalities [36,49]. To address this, Multimodal Knowledge Graphs [38,48,73] have emerged as a promising direction. For example, MMGraphRAG [65] leverages scene graphs and clustering to link visual and textual elements. However, such approaches typically convert visual content into textual graph nodes via MLLMs, effectively reducing multimodal structure to text-centric representations. As a result, fine-grained visual evidence may be abstracted away, limiting faithful cross-modal reasoning. In contrast, constructing natively unified multimodal graph structures that preserve atomic visual regions alongside textual entities remains underexplored. 3 Methodology We propose MG 2 -RAG, a lightweight framework that transforms an unstruc- tured multimodal knowledge base into a multi-granularity multimodal knowledge graph (MMKG) for retrieval-augmented generation. As shown in Figure 2, the framework consists of two modules: (1) Hierarchical Multimodal Knowledge Graph Construction and (2) Multi-Granularity Graph Retrieval. MG 2 -RAG: Multi-Granularity Graph for M-RAG5 MultimodalKnowledge-Base TextualGraph Entity-Driven Segmentation MultimodalGraph Modality-Preserving Fusion SentenceNodeEntityNodeUnifiedMultimodalNodeTextualChunkNodeImageNode Semantic Edge Grounding Edge Contextual Edge ObjectNode 1 2 3 4 1 Textual Structure Extraction2 Entity-Driven Visual Grounding3 MultimodalNodeFusion4 Multi-Granularity Graph Integration ActivatedSeedNodes Multimodal Retrieval Query Top-kSeed Nodes 푴 !"# , 푴 $"# TextualQueryEmbeddingVisualQueryEmbeddingSentenceEmbeddingObjectEmbedding (a) Hierarchical Multimodal Knowledge Graph Construction kDocs (b) Multi - Granularity Graph Retrieval Images Text Meta SAM3 1 Multimodal Seed Node Retrieval 2 Multi-Granularity Graph Propagation Iteratively Updating Generation TextualChunkEmbeddingImageEmbedding ChunksImages Sentences Objects Dense Embedding Chunks Images Sentences Objects Fig. 2: Overview of MG 2 -RAG. This framework consists of two modules: (a) Multimodal Knowledge Graph Construction that transforms a multimodal knowledge base into a hierarchical fine-grained multimodal knowledge graph (MMKG) via textual parsing and entity-driven visual grounding; (b) Multi-Granularity Graph Retrieval that employs a multi-granularity retrieval mechanism, driven by graph propagation, to retrieve the top-K most relevant document chunks for MLLM generation. 3.1 Hierarchical Multimodal Knowledge Graph Construction Motivation. Existing MMKG construction strategies suffer from two key issues: (1) expensive LLM-based relation extraction and (2) modality fragmentation caused by converting images into textual descriptions. To address these issues, we construct a hierarchical MMKG that directly aligns textual entities with grounded visual objects, forming atomic unified multimodal nodes. We define the multimodal knowledge base asD=(c k ,I k ) N k=1 , where each unit consists of a textual document chunkc k and its associated image setI k =I k,1 ,· ,I k,m k . Rather than modeling text–image correspondence as loose cross-modal links, we explicitly extract and fuse semantic information to construct a compact hierarchical MMKG. Each unified multimodal node preserves fine-grained textual and visual components internally, enabling nuanced downstream reasoning. Textual Structure Extraction: NER and Dependency Parsing. We first extract structural signals from text using the transformer-based spaCy modelen_core_web_trf[29]. Each chunkc k is parsed into a sentence setS k = s k,1 ,· ,s k,n k , from which we extract a local entity setE k . Aggregating across all chunks yields the global sentence and entity sets: S = S N k=1 S k , E = S N k=1 E k . To capture structural relations between entities, we leverage token-level de- pendency parsing. Instead of relying on expensive LLM-based OpenIE extrac- 6Sijun Dai et al. tion [15, 24], we adopt a lightweight rule-based relation extraction strategy grounded in dominant grammatical patterns (e.g., predicate-centered and nominal- modifier structures). This significantly reduces construction cost while preserving high-precision relational structure. Details are provided in Appendix A.1. Entity-Driven Visual Grounding. To prevent the “translation-to-text” bot- tleneck, we directly ground textual entities in visual space. Given the local entity setE k and image setI k for each chunkc k , we perform entity-driven segmentation rather than global image parsing. Specifically, we batch entity names as semantic prompts into an open-vocabulary segmentation modelΦ(SAM3 [6]) to localize entity-relevant regions. For entity e∈E k , its grounded region set is: V e o =o| (o,σ)∈ Φ(e,I), I ∈I k , σ > τ,(1) whereois the segmented object,σdenotes grounding confidence for entitye on imageI, andτis a threshold. This targeted grounding ensures that visual objects are directly and semantically aligned with textual entities. Modality-Preserving Multimodal Node Fusion. For entities with valid detections (V e o ̸ =∅), we construct a modality-preserving multimodal nodev m = (e,V e o )∈V M , whereV M denotes the set of multimodal nodes. To explicitly model internal composition, we define: –Object-Multimodal Incidence Matrix:M OMI ∈0,1 |V O |×|V M | , where V O = S e∈E V e o . EntryM OMI [i,j] = 1 indicates that objecto i ∈V O belongs to multimodal node v m j ∈V M . –Sentence-Multimodal Incidence Matrix:M SMI ∈0,1 |S|×|V M | , where M SMI [i,j] = 1 indicates entitye j that is associated with the unified multi- modal node v m j appears in sentence s i ∈S. These matrices enable structured aggregation during retrieval. Multi-Granularity Graph Integration. Building upon the matrices of document chucksV C , imagesV I , and multimodal nodesV M , we construct multi- granularity multimodal graph: G = (V,E G ), V =V C ∪V I ∪V M . For each multimodal unit (c k ,I k ), nodes operate at three levels: chunk nodes v k c ∈V C , image nodesv k,j I ∈V I for each associated imageI k,j ∈I k , and unified multimodal nodesv m ∈V M . To represent rich structural and semantic interac- tions, the edge set E G is constructed across three complementary dimensions: (1)Contextual Edges: To preserve hierarchical provenance, each chunk nodev k c is linked to its associated image nodesv k,j I and multimodal nodesv m derived from e∈E k . These edges maintain document-level structural coherence. (2)Semantic Edges: To encode linguistic structure, we connect multimodal nodes (v m a ,v m b ) when their underlying textual entities exhibit direct syntac- tic dependencies. This enables structured reasoning over explicit relational cues extracted from text. (3) Grounding Edges: To model explicit cross-modal alignment, each visual objecto∈V e o encapsulated within a multimodal nodev m is linked to its source MG 2 -RAG: Multi-Granularity Graph for M-RAG7 image nodev k,j I . The edge weight is assigned as the grounding confidenceσ, reflecting the reliability of the localized visual evidence. These three edge types integrate contextual hierarchy, semantic structure, and visual grounding into a unified heterogeneous topology, laying the foundation for structured multi-granularity retrieval in Section 3.2 In parallel, all textual and visual elements are embedded into a shared space via a unified encoderΨ(·) (i.e.,EVA-CLIP-8B[64]), 3 which maps any elemental inputxto ad-dimensional vectorz x =Ψ(x)∈R d . We construct four embedding matrices corresponding to distinct levels of granularity: (1) Sentence matrix Z S ∈R |S|×d encoded from global sentencesS; (2) Chunk matrixZ C ∈R |V C |×d encoded from document chunksV C ; (3) Image matrixZ I ∈R |V I |×d encoded from source imagesV I ; and (4) Object matrixZ O ∈R |V O |×d encoded from segmented objectsV O . Together, the graph topology and dense embeddings enable multi-granularity retrieval. 3.2 Multi-Granularity Graph Retrieval Motivation. Flat vector retrieval treats modalities independently and ignores structural dependencies within multimodal knowledge bases. To enable structured cross-modal reasoning, we adopt a two-stage strategy: (1) activating fine-grained multimodal evidence as seed nodes, and (2) propagating their relevance across the graph topology to capture multi-hop interactions. Multimodal Seed Node Retrieval. Leveraging the multimodal index con- structed in Section 3.1, a textual queryq t or visual queryq v retrieves candidate evidence from all semantic levels. For each modalitym∈t,v, dense similarities with the embedding matricesZ S ,Z C ,Z I , andZ O produce four relevance vectors: s (m) S ∈R |S| , s (m) C ∈R |V C | , s (m) I ∈R |V I | , and s (m) O ∈R |V O | . Because sentencesSand visual objectsV O act as intermediate pivots rather than primary graph nodes, their scores are aggregated onto unified multimodal nodes V M using the incidence matrices M SMI and M OMI : s (m) M = D −1 S M ⊤ SMI s (m) S + D −1 O M ⊤ OMI s (m) O ,(2) whereD S , D O ∈R |V M |×|V M | are diagonal degree matrices performing mean pooling based on the number of mapped unified multimodal nodes. To balance evidence from different semantic levels, we introduce scaling factors ω C andω I for chunk and image nodes. Given the node partitionV=V C ∪V I ∪V M , the modality-specific activation vector isu (m) = [ω C s (m) C ;ω I s (m) I ;s (m) M ]. Finally, modality contributions are fused using weights λ t and λ v : r 0 = λ t u (t) + λ v u (v) .(3) To reduce noise before graph propagation, we retain only the top-kactivated nodes and normalize the resulting distribution, as detailed in Appendix A.3. 3 Without specialization, all embeddings are explicitlyℓ 2 -normalized, and their semantic proximity is computed directly via cosine similarity. 8Sijun Dai et al. Multi-Granularity Graph Propagation. Starting from the seed distribution r 0 , we propagate relevance across the heterogeneous graph using Personalized PageRank [27]. LetWdenote the column-normalized transition matrix derived from E G . The diffusion process iteratively updates node activations as: r ℓ+1 = αW r ℓ + (1− α)r 0 ,(4) until∆r=r ℓ+1 − r ℓ ≤ ε, whereα∈(0,1) controls the propagation strength and (1−α) ensures periodic restarts from the seed distributionr 0 . After convergence, the relevance scores of chunk nodes are extracted to select the top-kdocument chunks. These chunks are combined with the multimodal query (q t ,q v ) to form the final prompt for the MLLM, enabling grounded response generation. 4 Experiment 4.1 Experimental Setup Baselines. We compare MG 2 -RAG against three state-of-the-art graph-based M-RAG frameworks: VaLiK [48], mKG-RAG [73], and MMGraphRAG [65]. For fair comparison, we report results using the best-performing configurations from the original papers without dataset-specific fine-tuning. Additional compar- isons with other relevant approaches are summarized in Tables 1–4. Multimodal Tasks, Datasets, and Evaluation Metrics. We evaluate MG 2 - RAG on four representative multimodal tasks, covering retrieval, knowledge-based Visual Question Answering (VQA), reasoning, and classification. –Multimodal Retrieval: We evaluate retrieval performance on two Wikipedia- linked datasets: E-VQA [55] and InfoSeek [10]. E-VQA contains diverse fine- grained entities and includes single-hop and two-hop multi-page queries, making retrieval challenging. InfoSeek provides two evaluation splits: Unseen Entity (Unseen-E) and Unseen Question (Unseen-Q), where the former requires retrieving previously unseen entities. For both datasets, we construct MMKGs from the associated corpora and retrieve supporting evidence for each query. Performance is evaluated using Recall@K (R@K). –Knowledge-based VQA: Using the retrieved evidence, the model generates answers for both textual and visual queries. In E-VQA, questions may involve multiple associated images and require reasoning across one or two knowledge hops. For InfoSeek, evaluation on both Unseen-E and Unseen-Q assesses generalization to novel entities and question formulations. Performance is measured using BERT Matching Score (BEM) [4] for E-VQA, and standard accuracy [21] or relaxed accuracy [56] for InfoSeek. –Multimodal Reasoning: We further evaluate our method on ScienceQA [50], which contains around 21k multimodal science questions spanning physics, chemistry, and biology. Following VaLiK [48], the training set is used as the knowledge base to construct the MMKG. The task requires answering complex scientific questions through multimodal reasoning. Performance is measured using classification accuracy, reported across different question types, contextual modalities, and educational stages. MG 2 -RAG: Multi-Granularity Graph for M-RAG9 10 1 10 0 10 1 10 2 Time (h) 4.4 1.9 0.5 0.7 0.8 98.3 50.7 38.9 40.5 25.5 126.3 47.4 36.2 53.0 44.5 (22.2x) (24.9x) (77.1x) (61.4x) (31.1x) MG 2 -RAG (Ours) VaLiKMMGraphRAG E-VQA (5k)InfoSeek (5k) ScienceQACrisisMMD-BCCrisisMMD-MC 10 1 10 0 10 1 10 2 Cost ($) 3.6 1.6 0.4 0.6 0.7 43.0 20.2 17.6 15.7 16.1 33.8 20.8 59.6 21.4 22.2 (9.2x) (12.8x) (45.2x) (28.5x) (23.7x) Fig. 3: Graph construction efficiency. The top and bottom plots show construction time (hours) and cost ($), respectively. Red annotations above the MG 2 -RAG bars indicate the efficiency gain (speedup or cost reduction) relative to the strongest baseline. –Multimodal Classification: To assess real-world applicability, we evaluate MG 2 -RAG on CrisisMMD [1], a disaster-response dataset containing ap- proximately 35k noisy social-media image-text pairs. Following VaLiK [48], the training set is used to construct the MMKG, capturing modality correla- tions and noise patterns typical of user-generated content. We evaluate three classification tasks: (1) BC: binary classification for information relevance filtering, (2) MC: multi-class classification of fine-grained humanitarian cate- gories, and (3) MC-m: a simplified multi-class setting with merged categories. Performance is measured using standard classification accuracy. Further details on dataset preprocessing are provided in Appendix A.2. Implementation Details. For Hierarchical MMKG Construction, textual analysis is performed using spaCy (en_core_web_trf) [29], and entity-driven visual grounding is conducted with SAM3 [6]. All textual and visual elements are embedded into a shared semantic space usingEVA-CLIP-8B[64]. For Multi- Granularity Graph Retrieval, both dense similarity computation and Per- sonalized PageRank (PPR) propagation are implemented on GPUs to accelerate large-scale inference. Hyperparameters are empirically tuned for each task, with detailed configurations and prompts provided in Appendix A.3. All experiments are conducted on two NVIDIA RTX 6000 Ada GPUs (48GB). 4.2 Graph Construction Efficiency To evaluate the efficiency of our hierarchical MMKG construction strategy, we compare the graph construction time and cost of MG 2 -RAG with existing open-source graph-based M-RAG baselines. For large benchmarks (E-VQA and InfoSeek), we restrict the knowledge base to a 5k subset because baseline approaches rely on expensive MLLM-driven triplet extraction, making full-scale graph construction computationally and financially infeasible. In contrast, MG 2 - 10Sijun Dai et al. Table 1: Retrieval performance on E-VQA and InfoSeek. Retrieval modes include T→V (Text-to-Vision), V→T (Vision-to-Text), and M (Multimodal retrieval). Bold and underlineddenote the best and second-best results. Model Retrieval Modality E-VQAInfoSeek R@1↑ R@5↑ R@10↑ R@20↑ R@50↑ R@1↑ R@5↑ R@10↑ R@20↑ R@50↑ Single Modality Retrieval Nomic-Embed [57] T→T4.79.112.115.923.00.21.01.53.05.6 V→V16.4 26.933.638.444.5 31.4 48.854.560.066.1 CLIP ViT-L/14 [61] T→T1.02.02.53.24.40.10.20.20.40.7 V→V17.8 31.036.442.650.3 32.7 52.359.666.274.0 EVA-CLIP-8B [64] T→T2.24.15.27.09.50.10.71.42.34.5 V→V30.0 45.050.055.059.9 47.26570.274.479.6 Cross-Modality Retrieval CLIP ViT-L/14 [61] T→V1.63.34.87.010.90.20.61.12.35.3 V→T10.9 26.235.844.655.6 20.8 40.749.257.667.7 EVA-CLIP-8B [64] T→V1.84.66.38.512.90.20.40.82.04.9 V→T42.062.669.575.683.256.576.382.286.490.6 Multi-Modality Retrieval mKG-RAG [73]M– 49.7 71.678.082.589.1 MG 2 -RAG (Ours) M44.9 66.4 72.0 79.4 83.9 59.6 78.9 83.8 87.4 91.0 RAG remains efficient enough to construct graphs over the entire knowledge base, highlighting its scalability advantage. Figure 3 summarizes the results. By bypassing the bottleneck of MLLM-based relation extraction and instead combining lightweight textual parsing with entity- driven visual grounding, MG 2 -RAG achieves substantial efficiency gains across all evaluated datasets. The calculation method is detailed in Appendix A.2. –Construction Time: MG 2 -RAG dramatically accelerates graph construc- tion. Compared to the strongest baseline for each dataset, our framework achieves a maximum speedup of 77.1×on ScienceQA and an average speedup of 43.3× across all benchmarks (ranging from 22.2× to 77.1×). – Construction Cost: The lightweight strategy also significantly reduces financial overhead. MG 2 -RAG lowers construction costs by up to 45.2×on ScienceQA, with an average cost reduction of 23.9×across the five datasets. These results demonstrate that our MMKG construction strategy effectively tackles a major scalability bottleneck of existing graph-based M-RAG methods, enabling practical deployment without sacrificing downstream performance. 4.3 Multimodal Retrieval Performance Table 1 reports retrieval results on E-VQA and InfoSeek, comparing MG 2 - RAG with both cross-modal retrieval models and graph-based baselines. Our framework consistently achieves the highest recall across all metrics (R@1–R@50). For instance, MG 2 -RAG reaches R@1 scores of 44.9 and 59.6 on E-VQA and InfoSeek, respectively, as well as R@10 scores of 72.0 and 83.8. These results MG 2 -RAG: Multi-Granularity Graph for M-RAG11 Table 2: Knowledge-based VQA performance on InfoSeek and E-VQA. T and V denote Text and Vision modalities, respectively. ModelLLM/MLLM E-VQAInfoSeek Single-Hop↑ All↑ Unseen-Q↑ Unseen-E↑ All↑ Zero-Shot MLLMs InstructBLIP [14]Flan-T5XL11.90 12.008.907.40 8.10 LLaVA-v1.5 [46]LLaMA-3.1-8B16.00 16.908.308.90 7.80 Qwen2.5-VL-7B [13]Qwen2.5-VL-7B23.60 23.2022.8024.10 23.70 Gemini-3.1-Pro [20]Gemini-3.1-Pro29.36 27.9029.9222.52 25.70 GPT-5.2 [58]GPT-5.244.19 40.3032.6826.54 29.29 Qwen3.5-27B [60]Qwen3.5-27B29.79 25.7220.2516.75 18.33 Retrieval-Augmented Models RORA-VLM [59]Vicuna-7B–25.1027.30– EchoSight [70] (V→T)LLaMA-3.1-8B52.96 47.2330.0030.70 30.40 EchoSight [70] (V→V)LLaMA-3.1-8B46.32 41.2918.0019.80 18.80 Graph Retrieval-Augmented Models mKG-RAG [73]LLaMA-3.1-8B–32.9031.30 32.10 MG 2 -RAG(Ours)LLaMA-3.1-8B55.57 47.7732.3232.84 32.58 MG 2 -RAG(Ours)Qwen2.5-VL-7B57.33 48.5935.7835.18 35.48 MG 2 -RAG(Ours)Qwen3.5-27B61.65 52.1937.3638.40 37.87 MG 2 -RAG(Ours)GPT-5.268.96 60.3040.1638.4739.30 VaLiK (5k) [48]Qwen2.5-VL-7B16.16 15.223.192.07 2.51 MMGraphRAG (5k) [65] Qwen2.5-VL-7B19.12 16.520.690.39 0.50 MG 2 -RAG (5k) (Ours) Qwen2.5-VL-7B62.88 53.3639.1738.15 38.65 MG 2 -RAG (5k) (Ours) Qwen3.5-27B70.27 60.2442.4942.03 42.26 surpass the strongest cross-modal baseline EVA-CLIP-8B (V → T), which obtains 42.0 / 56.5 at R@1 and 69.5 / 82.2 at R@10. Compared with the multimodal graph baseline mKG-RAG, our framework achieves a further +9.9 improvement at R@1 on InfoSeek. These gains directly val- idate the effectiveness of our multi-granularity graph retrieval mechanism. By aggregating dense similarities onto unified multimodal nodes, MG 2 -RAG bridges textual and visual evidence within a shared graph structure, allowing re- trieval to capture cross-modal dependencies and structural relationships, thereby retrieving more accurate evidence than modality-isolated approaches. 4.4 Knowledge-based VQA Performance Table 2 presents results on knowledge-based VQA, where models must reason over retrieved multimodal evidence. MG 2 -RAG establishes a new state-of-the- art, consistently outperforming zero-shot MLLMs, standard RAG methods, and leading graph-based approaches. Notably, with the Qwen3.5-27B backbone, MG 2 - RAG attains 60.24 on E-VQA (5k) and 42.26 on InfoSeek (5k). Two key advantages explain these improvements. First, compared with stan- dard retrieval models like EchoSight [70], MG 2 -RAG consistently yields better results, achieving +0.54 accuracy on E-VQA and +2.18 on InfoSeek. This im- provement highlights the effectiveness of our modality-preserving multimodal 12Sijun Dai et al. Table 3: Multimodal reasoning performance on ScienceQA. Results are reported across different domains and contexts: NAT (natural science), SOC (social science), LAN (language science), TXT (text context), IMG (image context), NO (no context), G1-6 (grades 1-6), and G7-12 (grades 7-12). ModelLLM/MLLM SubjectContext ModalityGrade Avg.↑ NAT↑ SOC↑ LAN↑ TXT↑ IMG↑ NO↑ G1-6↑ G7-12↑ Human [50]-90.23 84.97 87.48 89.60 87.50 88.10 91.5982.42 88.40 Zero-shot/Few-shot Models CoT [51]GPT-485.48 72.44 90.27 82.65 71.49 92.89 86.6679.04 83.99 Chameleon [51]ChatGPT81.62 70.64 84.00 79.77 70.80 86.62 81.8676.53 79.93 Qwen2.5-VL-7B [3] Qwen2.5-VL-7B 88.77 88.86 83.82 87.39 88.05 86.06 89.9483.12 87.50 Gemini-3.1-Pro [20] Gemini-3.1-Pro 93.53 94.12 90.81 92.98 94.25 90.27 94.0390.93 92.90 GPT-5.2 [58]GPT-5.297.60 93.05 95.96 97.73 95.35 96.17 96.5495.88 96.30 Qwen3.5-27B [60]Qwen3.5-27B98.2795.1697.1897.90 96.43 97.4997.7696.5797.34 Graph Retrieval-Augmented Models MMKG [49]Qwen2.5-7B73.98 66.37 78.18 71.65 64.30 79.65 76.5168.03 73.47 Visual Genome [36] Qwen2.5-7B76.78 67.04 78.09 74.05 66.19 79.72 78.0869.68 75.08 VaLiK [48]Qwen2.5-7B84.15 75.14 87.64 82.99 73.18 89.69 84.4080.95 83.16 VaLiK [48]Qwen2.5-72B85.61 75.93 90.27 84.40 74.17 92.33 85.7982.98 84.77 MMGraphRAG [65] Qwen2.5-VL-7B 81.08 68.62 80.09 79.52 68.96 83.69 80.8773.43 78.21 MG 2 -RAG(Ours)Qwen2.5-7B85.17 81.33 85.82 83.43 76.10 89.41 86.1281.67 84.53 MG 2 -RAG(Ours)Qwen2.5-72B88.37 81.21 91.27 87.00 78.28 94.01 88.8485.43 87.62 MG 2 -RAG(Ours)Qwen2.5-VL-7B 90.81 87.96 87.09 89.93 86.61 88.92 90.6486.75 89.25 MG 2 -RAG(Ours)GPT-5.297.78 94.65 97.06 97.9396.2497.05 97.3396.43 97.00 MG 2 -RAG(Ours)Qwen3.5-27B98.45 95.28 98.73 98.44 96.43 98.68 98.42 96.84 97.85 node fusion, which retains both textual entities and visual objects as unified concepts. Second, MG 2 -RAG outperforms existing graph-based approaches even when using the same backbone models. For instance, with the LLaMA-3.1-8B backbone, our framework already surpasses mKG-RAG by 0.48 on InfoSeek, and the advantage becomes larger with stronger models. This demonstrates that our lightweight MMKG construction strategy produces a cleaner and more informative graph topology than traditional MLLM-generated triplet graphs. 4.5 Multimodal Reasoning Performance Table 3 reports results on ScienceQA, a widely used benchmark for evaluating multimodal reasoning across diverse scientific domains. MG 2 -RAG achieves state-of-the-art performance, consistently surpassing both zero-shot/few-shot MLLMs and existing graph-based RAG models. In particular, when paired with the Qwen3.5-27B backbone, MG 2 -RAG reaches an overall accuracy of 97.85, substantially exceeding human performance (88.40) [50]. Two observations highlight the effectiveness of our framework. First, MG 2 - RAG consistently outperforms prior graph-augmented models. Using the same Qwen2.5-VL-7B backbone, it improves over MMGraphRAG by 11.04 points in average accuracy and surpasses VaLiK by 1.37 points (7B) and 2.85 points (72B). These results demonstrate that our multi-granularity graph representation more effectively captures reasoning chains and structural relationships than MG 2 -RAG: Multi-Granularity Graph for M-RAG13 Table 4: Multimodal classification performance on CrisisMMD. Evaluation includes three tasks: BC (binary informativeness classification), MC (fine-grained humanitarian category classification), and MC-m (merged-category classification). MethodLLM/MLLM CrisisMMD BC↑ MC↑ MC-m↑ Avg.↑ Zero-Shot Models CLIP ViT-L/14 [61] CLIP ViT-L/14 43.36 17.8820.79 27.34 LLaVA-34B [47]LLaVA-34B56.44 25.1525.07 35.55 BLIP-2 [43]Flan-T561.29 40.8640.72 47.62 Qwen2.5-VL-7B [3] Qwen2.5-VL-7B 70.45 41.4442.33 51.41 Gemini-3.1-Pro [20] Gemini-3.1-Pro 71.30 49.3649.79 56.82 GPT-5.2 [58]GPT-5.271.00 53.00 53.6059.20 Qwen3.5-27B [60]Qwen3.5-27B69.83 48.8249.35 56.00 Graph Retrieval-Augmented Models LightRAG [23]Qwen2.5-7B67.49 45.1145.94 52.85 VaLiK [48]Qwen2.5-7B68.90 50.0250.69 56.54 VaLiK [48]Qwen2.5-72B68.89 49.7849.31 55.99 MMGraphRAG [65] Qwen2.5-VL-7B 68.40 44.0344.66 50.94 MG 2 -RAG(Ours)Qwen2.5-VL-7B 71.03 46.4547.25 54.91 MG 2 -RAG(Ours)Qwen2.5-7B69.6951.151.59 57.46 MG 2 -RAG(Ours)Qwen2.5-72B72.33 50.3450.83 57.83 MG 2 -RAG(Ours)Qwen3.5-27B72.28 52.3052.79 59.12 MG 2 -RAG(Ours)GPT-5.272.30 56.1056.70 61.70 conventional graph formulations. Second, MG 2 -RAG improves the reasoning capability of strong base MLLMs. Compared with zero-shot inference, integrating MG 2 -RAG yields average improvements of 1.75 points for Qwen2.5-VL-7B and 0.70 points for GPT-5.2. This indicates that our modality-preserving graph structure provides reliable multi-hop contextual evidence, reducing hallucination and enabling deeper reasoning over scientific knowledge. 4.6 Multimodal Classification Performance Table 4 presents results on CrisisMMD, evaluating multimodal classification in disaster scenarios. MG 2 -RAG achieves state-of-the-art performance. In particular, the combination of MG 2 -RAG and GPT-5.2 obtains the best overall accuracy of 61.70, demonstrating strong performance in both binary relevance filtering (BC) and multi-class humanitarian classification (MC and MC-m). Two insights emerge from the results. First, MG 2 -RAG consistently surpasses existing graph-based retrieval frameworks. Using the Qwen2.5-VL-7B backbone, our model improves over MMGraphRAG by 3.97 points in average accuracy. With Qwen2.5-7B, it outperforms LightRAG by 4.61 points and VaLiK by 0.92 points, while maintaining a 1.84-point advantage over VaLiK with the 72B backbone. These improvements highlight the effectiveness of our multi- granularity graph retrieval mechanism, which captures complex cross-modal relationships within noisy real-world data. Second, MG 2 -RAG substantially enhances the zero-shot capability of standard MLLMs. Compared with their 14Sijun Dai et al. Case (a) Q:When was this bridge built? Retrieved Evidence The bridge was designed by Calvert Vaux and Jacob Wrey Mould, andcompleted in 1862. It was built... Method Responses VaLiK:Needthe bridge name MMGraphRAG: Not the exact year Qwen3.5-27B: 1862 MG ퟐ -RAG:Built in 1862 Case (b) Q:What's this bird's wingspan? Retrieved Evidence Its total body length is 12–15 cm wingspan is 16–21 cm (6.3–8.3 in), and body mass is 9–14 g (0.32–0.49 oz). VaLiK:Pleasedescribe the bird MMGraphRAG: 27.9 to 32.0 cm Qwen3.5-27B: 16–19 cm MG ퟐ -RAG:16–21cm Method Responses Case (c) Q:Which country is this lake in? Retrieved Evidence The historic monuments, and the pristine lake environment... the North Macedonia and eastern Albania VaLiK:Northwest Territories MMGraphRAG: Nepal's SNP Qwen3.5-27B: Indonesia MG ퟐ -RAG:N. Macedonia, Albania Method Responses Q:Can I eat this plant or it fruits? Retrieved Evidence AMother tincture is made from the flowering parts of the plant. The green parts of the plant are eatenas vegetables. Method Responses VaLiK:Need toxicityanalysis MMGraphRAG: Itcan be toxic Qwen3.5-27B: You should not eat it MG ퟐ -RAG:Yes, parts of it are edible Case (d) Fig. 4: Case studies on Knowledge-based VQA and Multimodal Retrieval. MG 2 -RAG is compared with a baseline MLLM (Qwen3.5-27B) and graph-based ap- proaches (VaLiK and MMGraphRAG). The Retrieved Evidence illustrates the multimodal evidence retrieved by MG 2 -RAG for answer generation. respective zero-shot baselines, our framework improves performance by 3.50 points for Qwen2.5-VL-7B and 2.50 points for GPT-5.2. This demonstrates that our entity-driven visual grounding helps resolve ambiguity in social-media content and improves decision reliability. 4.7 Case Study Figure 4 presents qualitative comparisons on knowledge-based VQA and multi- modal retrieval tasks. Baseline MLLMs (e.g., Qwen3.5-27B) frequently produce vague or incorrect answers due to the absence of external knowledge. For example, in Case (c), the baseline model incorrectly predicts “Indonesia,” illustrating how hallucinations arise when multimodal evidence is missing. In contrast, MG 2 -RAG consistently produces accurate and grounded responses. By fusing textual en- tities and visual objects into unified multimodal nodes, the framework preserves fine-grained atomic evidence and ensures precise cross-modal alignment. This enables the model to correctly extract details such as the exact wingspan in Case (b) and plant edibility in Case (d). Moreover, our multi-granularity graph retrieval enables effective cross- modal reasoning. In Case (c), MG 2 -RAG correctly retrieves the key sentence “the pristine lake environment” and links it to the corresponding image through graph propagation. By jointly leveraging textual and visual evidence, the model accurately identifies the location as “North Macedonia, Albania,” successfully correcting the hallucinated prediction of the baseline. More qualitative examples for multimodal reasoning and classification are provided in Appendix B. MG 2 -RAG: Multi-Granularity Graph for M-RAG15 5 Conclusion In this paper, we introduce MG 2 -RAG, a lightweight Multi-Granularity Graph RAG framework that improves the reliability and efficiency of M-RAG. We propose a hierarchical MMKG construction strategy that bypasses expensive MLLM-driven triplet extraction by combining lightweight textual parsing with entity-driven visual grounding, enabling efficient fusion of textual entities and visual objects into unified multimodal nodes. Building on this representation, we further design a multi-granularity graph retrieval mechanism that aggregates dense similarities and propagates relevance through the graph topology to support structured multi-hop reasoning. Extensive experiments across four multimodal tasks (i.e., retrieval, knowledge-based VQA, reasoning, and classification) show that MG 2 -RAG consistently achieves state-of-the-art performance while maintain- ing high efficiency, providing an average 43.3×speedup and 23.9×cost reduction compared with existing graph-based baselines. Ultimately, MG 2 -RAG provides a scalable, efficient, and robust solution for M-RAG, enabling reliable deployment of MLLMs in complex, real-world knowledge-intensive applications. References 1. Alam, F., Ofli, F., Imran, M.: CrisisMMD: Multimodal Twitter Datasets from Natural Disasters. In: Proceedings of the International AAAI Conference on Web and Social Media (AAAI). vol. 12 (2018),https://ojs.aaai.org/index.php/ ICWSM/article/view/14983 2. Asai, A., Wu, Z., Wang, Y., Sil, A., Hajishirzi, H.: Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. In: The Twelfth International Conference on Learning Representations (ICLR) (2024),https://openreview. net/forum?id=hSyW5go0v8 3. Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923 (2025), https://arxiv.org/abs/2502.13923 4. Bulian, J., Buck, C., Gajewski, W., Börschinger, B., Schuster, T.: Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP). p. 291–305 (2022),https://aclanthology.org/2022.emnlp- main.20/ 5.Caffagni, D., Cocchi, F., Moratelli, N., Sarto, S., Cornia, M., Baraldi, L., Cucchiara, R.: Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 1818–1826 (2024),https://openaccess.thecvf. com/content/CVPR2024W/MMFM/html/Caffagni_Wiki- LLaVA_Hierarchical_ Retrieval-Augmented_Generation_for_Multimodal_LLMs_CVPRW_2024_paper. html 6. Carion, N., Gustafson, L., Hu, Y.T., Debnath, S., Hu, R., Suris, D., Ryali, C., Alwala, K.V., Khedr, H., Huang, A., Lei, J., Ma, T., Guo, B., Kalla, A., Marks, M., Greer, J., Wang, M., Sun, P., Rädle, R., Afouras, T., Mavroudi, E., Xu, K., Wu, 16Sijun Dai et al. T.H., Zhou, Y., Momeni, L., Hazra, R., Ding, S., Vaze, S., Porcher, F., Li, F., Li, S., Kamath, A., Cheng, H.K., Dollár, P., Ravi, N., Saenko, K., Zhang, P., Feichtenhofer, C.: SAM 3: Segment Anything with Concepts. arXiv preprint arXiv:2511.16719 (2025), https://arxiv.org/abs/2511.16719 7. Chen, D., Fisch, A., Weston, J., Bordes, A.: Reading Wikipedia to Answer Open- Domain Questions. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL). p. 1870–1879 (2017),https://aclanthology. org/P17-1171/ 8. Chen, L., Tong, P., Jin, Z., Sun, Y., Ye, J., Xiong, H.: Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphs. In: The Thirty- eighth Annual Conference on Neural Information Processing Systems (NeurIPS). vol. 37, p. 37665–37691 (2024),https://openreview.net/forum?id=CwCUEr6wO5 9. Chen, W., Hu, H., Chen, X., Verga, P., Cohen, W.: MuRAG: Multimodal Retrieval- Augmented Generator for Open Question Answering over Images and Text. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). p. 5558–5570 (2022),https://aclanthology.org/2022. emnlp-main.375/ 10. Chen, Y., Hu, H., Luan, Y., Sun, H., Changpinyo, S., Ritter, A., Chang, M.W.: Can Pre-trained Vision and Language Models Answer Visual Information-seeking Questions? In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). p. 14948–14968 (2023),https://aclanthology. org/2023.emnlp-main.925/ 11. Cheng, Q., Li, X., Li, S., Zhu, Q., Yin, Z., Shao, Y., Li, L., Sun, T., Yan, H., Qiu, X.: Unified Active Retrieval for Retrieval Augmented Generation. In: Findings of the Association for Computational Linguistics: EMNLP 2024. p. 17153–17166 (2024), https://aclanthology.org/2024.findings-emnlp.999/ 12.Cocchi, F., Moratelli, N., Cornia, M., Baraldi, L., Cucchiara, R.: Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 9199–9209 (2025),https://openaccess.thecvf. com/content/CVPR2025/html/Cocchi_Augmenting_Multimodal_LLMs_with_Self- Reflective_Tokens_for_Knowledge-based_Visual_Question_CVPR_2025_paper. html 13.Compagnoni, A., Morini, M., Sarto, S., Cocchi, F., Caffagni, D., Cornia, M., Baraldi, L., Cucchiara, R.: ReAG: Reasoning-Augmented Generation for Knowledge- based Visual Question Answering. arXiv preprint arXiv:2511.22715 (2025),https: //arxiv.org/abs/2511.22715 14. Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P.N., Hoi, S.: InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. In: Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS). vol. 36, p. 49250–49267 (2023),https://proceedings.neurips.c/paper_files/paper/2023/hash/ 9a6a435e75419a836fe47ab6793623e6-Abstract-Conference.html 15.Dong, J., An, S., Yu, Y., Zhang, Q.W., Luo, L., Huang, X., Wu, Y., Yin, D., Sun, X.: Youtu-GraphRAG: Vertically Unified Agents for Graph Retrieval-Augmented Complex Reasoning. arXiv preprint arXiv:2508.19855 (2025),https://arxiv.org/ abs/2508.19855 16. Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., Metropolitansky, D., Ness, R.O., Larson, J.: From Local to Global: A Graph MG 2 -RAG: Multi-Granularity Graph for M-RAG17 RAG Approach to Query-Focused Summarization. arXiv preprint arXiv:2404.16130 (2024), https://arxiv.org/abs/2404.16130 17.Edge, D., Trinh, H., Larson, J.: LazyGraphRAG: Setting a New Standard for Quality and Cost (2024),https://w.microsoft.com/en-us/research/blog/ lazygraphrag-setting-a-new-standard-for-quality-and-cost/ 18.Faysse, M., Sibille, H., Wu, T., Omrani, B., Viaud, G., HUDELOT, C., Colombo, P.: ColPali: Efficient Document Retrieval with Vision Language Models. In: The Thirteenth International Conference on Learning Representations (ICLR) (2025), https://openreview.net/forum?id=ogjBpZ8uSi 19.Gao, J., Li, L., Ji, K., Li, W., Lian, Y., Fu, Y., Dai, B.: SmartRAG: Jointly Learn RAG-Related Tasks From the Environment Feedback. In: The Thirteenth International Conference on Learning Representations (ICLR) (2025),https:// openreview.net/forum?id=OCd3cffulp 20.Google DeepMind Team: Gemini 3.1 Pro: Best for Complex Tasks and Bringing Creative Concepts to Life (2026),https://deepmind.google/models/gemini/pro/ 21.Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). p. 6904–6913 (2017),https://openaccess.thecvf.com/content_cvpr_ 2017/html/Goyal_Making_the_v_CVPR_2017_paper.html 22. Guan, X., Zeng, J., Meng, F., Xin, C., Lu, Y., Lin, H., Han, X., Sun, L., Zhou, J.: DeepRAG: Thinking to Retrieval Step by Step for Large Language Models. arXiv preprint arXiv:2502.01142 (2025), https://arxiv.org/abs/2502.01142 23.Guo, Z., Xia, L., Yu, Y., Ao, T., Huang, C.: LightRAG: Simple and Fast Retrieval- Augmented Generation. In: Findings of the Association for Computational Lin- guistics: EMNLP 2025. p. 10746–10761 (2025),https://aclanthology.org/2025. findings-emnlp.568/ 24.Gutiérrez, B.J., Shu, Y., Gu, Y., Yasunaga, M., Su, Y.: HippoRAG: neurobiologically inspired long-term memory for large language models. In: Proceedings of the 38th International Conference on Neural Information Processing Systems (NeurIPS). vol. 37, p. 59532–59569 (2024),https://openreview.net/forum?id=hkujvAPVsg 25. Gutiérrez, B.J., Shu, Y., Qi, W., Zhou, S., Su, Y.: From RAG to Memory: Non- Parametric Continual Learning for Large Language Models. In: Forty-second Inter- national Conference on Machine Learning (ICML). p. 21497–21515. PMLR (2025), https://openreview.net/forum?id=LWH8yn4HS2 26.Guu, K., Lee, K., Tung, Z., Pasupat, P., Chang, M.: Retrieval Augmented Language Model Pre-Training. In: Proceedings of the 37th International Conference on Machine Learning (ICML). p. 3929–3938. PMLR (2020),https://proceedings. mlr.press/v119/guu20a.html 27.Haveliwala, T.H.: Topic-sensitive pagerank. In: Proceedings of the 11th international conference on World Wide Web (W). p. 517–526 (2002),https://dl.acm. org/doi/abs/10.1145/511446.511513 28.He, X., Tian, Y., Sun, Y., Chawla, N., Laurent, T., LeCun, Y., Bresson, X., Hooi, B.: G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS). vol. 37, p. 132876–132907 (2024), https://openreview.net/forum?id=MPJ3oXtTZl 29.Honnibal, M., Montani, I., Van Landeghem, S., Boyd, A.: spaCy: Industrial-Strength Natural Language Processing in Python (2020), https://spacy.io/ 18Sijun Dai et al. 30.Hsu, S., Khattab, O., Finn, C., Sharma, A.: Grounding by Trying: LLMs with Reinforcement Learning-Enhanced Retrieval. In: The Thirteenth International Conference on Learning Representations (ICLR) (2025),https://openreview. net/forum?id=BPAZ6yW3K7 31.Hu, C.W., Wang, Y., Xing, S., Chen, C.J., Feng, S., Rossi, R., Tu, Z.: mrag: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation. arXiv preprint arXiv:2505.24073 (2025), https://arxiv.org/abs/2505.24073 32. Hu, Z., Iscen, A., Sun, C., Wang, Z., Chang, K.W., Sun, Y., Schmid, C., Ross, D.A., Fathi, A.: REVEAL: Retrieval-Augmented Visual-Language Pre-Training With Multi-Source Multimodal Knowledge Memory. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 23369– 23379 (2023),https://openaccess.thecvf.com/content/CVPR2023/html/Hu_ REVEAL_Retrieval-Augmented_Visual-Language_Pre-Training_With_Multi- Source_Multimodal_Knowledge_Memory_CVPR_2023_paper.html 33.Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., Liu, T.: A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions 43(2) (Jan 2025). https://doi.org/10.1145/3703155, https://doi.org/10.1145/3703155 34.Huang, Y., Zhang, S., Xiao, X.: KET-RAG: A Cost-Efficient Multi-Granular In- dexing Framework for Graph-RAG. In: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). p. 1003–1012 (2025), https://dl.acm.org/doi/abs/10.1145/3711896.3737012 35.Jin, B., Xie, C., Zhang, J., Roy, K.K., Zhang, Y., Li, Z., Li, R., Tang, X., Wang, S., Meng, Y., Han, J.: Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs. In: Findings of the Association for Computational Linguistics: ACL 2024. p. 163–184 (2024),https://aclanthology.org/2024. findings-acl.11/ 36. Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., Bernstein, M.S., Li, F.F.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision (IJCV) 123(1), 32–73 (2017),https: //link.springer.com/article/10.1007/S11263-016-0981-7 37.Lazaridou, A., Gribovskaya, E., Stokowiec, W., Grigorev, N.: Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115 (2022), https://arxiv.org/abs/2203.05115 38.Lee, J., Wang, Y., Li, J., Zhang, M.: Multimodal Reasoning with Multimodal Knowledge Graph. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL). p. 10767–10782 (2024),https://aclanthology. org/2024.acl-long.579/ 39.Lee, K., Chang, M.W., Toutanova, K.: Latent Retrieval for Weakly Supervised Open Domain Question Answering. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL). p. 6086–6096 (2019), https://aclanthology.org/P19-1612/ 40.Lee, M., An, S., Kim, M.S.: PlanRAG: A Plan-then-Retrieval Augmented Genera- tion for Generative Large Language Models as Decision Makers. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (NAACL). p. 6537–6555 (2024), https://aclanthology.org/2024.naacl-long.364/ 41. Lee, Z., Cao, S., Liu, J., Zhang, J., Liu, W., Che, X., Hou, L., Li, J.: ReaRAG: Knowledge-guided Reasoning Enhances Factuality of Large Reasoning Models with MG 2 -RAG: Multi-Granularity Graph for M-RAG19 Iterative Retrieval Augmented Generation. arXiv preprint arXiv:2503.21729 (2025), https://arxiv.org/abs/2503.21729 42. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küt- tler, H., Lewis, M., Yih, W.t., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval- Augmented Generation for Knowledge-Intensive NLP Tasks. In: Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS). p. 9459–9474 (2020),https://proceedings.neurips.c/paper/2020/ hash/6b493230205f780e1bc26945df7481e5-Abstract.html 43. Li, J., Li, D., Savarese, S., Hoi, S.: BLIP-2: Bootstrapping Language-Image Pre- training with Frozen Image Encoders and Large Language Models. In: Proceedings of the 40th International Conference on Machine Learning (ICML). p. 19730–19742. PMLR (2023), https://proceedings.mlr.press/v202/li23q 44. Li, J., Li, D., Xiong, C., Hoi, S.: BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In: Proceedings of the 39th International Conference on Machine Learning (ICML). p. 12888–12900. PMLR (2022), https://proceedings.mlr.press/v162/li22n.html 45. Li, X., Dong, G., Jin, J., Zhang, Y., Zhou, Y., Zhu, Y., Zhang, P., Dou, Z.: Search-o1: Agentic Search-Enhanced Large Reasoning Models. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP). p. 5420–5438 (2025), https://aclanthology.org/2025.emnlp-main.276/ 46.Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved Baselines with Visual Instruction Tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 26296–26306 (2024),https://openaccess.thecvf.com/ content/CVPR2024/html/Liu_Improved_Baselines_with_Visual_Instruction_ Tuning_CVPR_2024_paper.html 47. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual Instruction Tuning. In: Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS). vol. 36, p. 34892–34916 (2023),https://openreview.net/forum?id=w0H2xGHlkw 48.Liu, J., Meng, S., Gao, Y., Mao, S., Cai, P., Yan, G., Chen, Y., Bian, Z., Wang, D., Shi, B.: Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning. In: Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV). p. 981–992 (2025),https://openaccess.thecvf.com/content/ICCV2025/html/ Liu_Aligning_Vision_to_Language_Annotation-Free_Multimodal_Knowledge_ Graph_Construction_for_ICCV_2025_paper 49.Liu, Y., Li, H., Garcia-Duran, A., Niepert, M., Onoro-Rubio, D., Rosenblum, D.S.: MMKG: Multi-modal Knowledge Graphs. In: European Semantic Web Conference (ESWC). p. 459–474. Springer (2019),https://link.springer.com/chapter/10. 1007/978-3-030-21348-0_30 50. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.W., Zhu, S.C., Tafjord, O., Clark, P., Kalyan, A.: Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. In: Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS). vol. 35, p. 2507–2521 (2022), https://openreview.net/forum?id=HjwK-Tc_Bc 51.Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.W., Wu, Y.N., Zhu, S.C., Gao, J.: Chameleon: Plug-and-Play Compositional Reasoning with Large Lan- guage Models. In: Proceedings of the 37th International Conference on Neu- ral Information Processing Systems (NeurIPS). vol. 36, p. 43447–43478 (2023), https://openreview.net/forum?id=HtqnVSCj3q 20Sijun Dai et al. 52.Luo, L., Li, Y.F., Haffari, G., Pan, S.: Reasoning on Graphs: Faithful and Inter- pretable Large Language Model Reasoning. In: The Twelfth International Conference on Learning Representations (ICLR) (2024),https://openreview.net/forum?id= ZGNWW7xZ6Q 53.Luo, L., Zhao, Z., Haffari, G., Phung, D., Gong, C., Pan, S.: GFM-RAG: Graph Foundation Model for Retrieval Augmented Generation. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS) (2025), https://openreview.net/forum?id=0QNmAvQQqj 54.Luo, Y., Zheng, X., Li, G., Yin, S., Lin, H., Fu, C., Huang, J., Ji, J., Chao, F., Luo, J., Ji, R.: Video-RAG: Visually-aligned Retrieval-Augmented Long Video Compre- hension. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS) (2025), https://openreview.net/forum?id=QaZxGWlbgO 55. Mensink, T., Uijlings, J., Castrejon, L., Goel, A., Cadar, F., Zhou, H., Sha, F., Araujo, A., Ferrari, V.: Encyclopedic VQA: Visual Questions about Detailed Proper- ties of Fine-grained Categories. In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV). p. 3113–3124 (2023),https://openaccess. thecvf.com/content/ICCV2023/html/Mensink_Encyclopedic_VQA_Visual_ Questions_About_Detailed_Properties_of_Fine-Grained_Categories_ICCV_ 2023_paper.html 56. Methani, N., Ganguly, P., Khapra, M.M., Kumar, P.: PlotQA: Reasoning over Sci- entific Plots. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). p. 1527–1536 (2020),https://openaccess.thecvf. com/content_WACV_2020/html/Methani_PlotQA_Reasoning_over_Scientific_ Plots_WACV_2020_paper.html 57.Nussbaum, Z., Morris, J.X., Mulyar, A., Duderstadt, B.: Nomic Embed: Training a Reproducible Long Context Text Embedder. Transactions on Machine Learning Research (TMLR) (2025), https://openreview.net/forum?id=IPmzyQSiQE 58. OpenAI Team: Introducing GPT-5.2:The Most Advanced Frontier Model for Pro- fessional Work and Long-running Agents. (2026),https://openai.com/index/ introducing-gpt-5-2/ 59. Qi, J., Xu, Z., Shao, R., Chen, Y., Di, J., Cheng, Y., Wang, Q., Huang, L.: RoRA-VLM: Robust Retrieval-Augmented Vision Language Models. arXiv preprint arXiv:2410.08876 (2024), https://arxiv.org/abs/2410.08876 60.Qwen Team: Qwen3.5: Towards Native Multimodal Agents (2026),https://qwen. ai/blog?id=qwen3.5 61. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning Transferable Visual Models From Natural Language Supervision. In: Proceedings of the 38th International Conference on Machine Learning (ICML). p. 8748–8763. PMLR (2021), https://proceedings.mlr.press/v139/radford21a 62. Sarthi, P., Abdullah, S., Tuli, A., Khanna, S., Goldie, A., Manning, C.D.: RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval. In: The Twelfth International Conference on Learning Representations (ICLR) (2024),https:// openreview.net/forum?id=GN921JHCRw 63. Sun, J., Xu, C., Tang, L., Wang, S., Lin, C., Gong, Y., Ni, L., Shum, H.Y., Guo, J.: Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph. In: The Twelfth International Conference on Learning Representations (ICLR) (2024), https://openreview.net/forum?id=nnVO1PvbTv 64.Sun, Q., Wang, J., Yu, Q., Cui, Y., Zhang, F., Zhang, X., Wang, X.: EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters. arXiv preprint arXiv:2402.04252 (2024), https://arxiv.org/abs/2508.05318 MG 2 -RAG: Multi-Granularity Graph for M-RAG21 65.Wan, X., Yu, H.: MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs. arXiv preprint arXiv:2507.20804 (2025),https: //arxiv.org/abs/2507.20804 66. Wang, L., Chen, H., Yang, N., Huang, X., Dou, Z., Wei, F.: Chain-of-Retrieval Augmented Generation. arXiv preprint arXiv:2501.14342 (2025),https://arxiv. org/abs/2501.14342 67.Wang, S., Fang, Y., Zhou, Y., Liu, X., Ma, Y.: ArchRAG: Attributed Community-based Hierarchical Retrieval-Augmented Generation. arXiv preprint arXiv:2502.09891 (2025), https://arxiv.org/abs/2502.09891 68.Wasserman, N., Pony, R., Naparstek, O., Goldfarb, A.R., Schwartz, E., Barze- lay, U., Karlinsky, L.: Real-m-rag: A real-world multi-modal retrieval bench- mark. In: Proceedings of the 63rd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers). p. 31660–31683 (2025),https: //aclanthology.org/2025.acl-long.1528/ 69.Wu, J., Zhu, J., Liu, Y., Xu, M., Jin, Y.: Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Tools. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL). p. 28489–28503 (2025), https://aclanthology.org/2025.acl-long.1383/ 70. Yan, Y., Xie, W.: EchoSight: Advancing Visual-Language Models with Wiki Knowl- edge. In: Findings of the Association for Computational Linguistics: EMNLP 2024. p. 1538–1551 (2024), https://aclanthology.org/2024.findings-emnlp.83/ 71. Yang, W., Fu, J., Wang, R., Wang, J., Song, L., Bian, J.: OMGM: Orchestrate multiple granularities and modalities for efficient multimodal retrieval. In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T. (eds.) Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL). p. 24545–24563. Association for Computational Linguistics, Vienna, Austria (Jul 2025).https://doi. org/10.18653/v1/2025.acl-long.1198,https://aclanthology.org/2025.acl- long.1198/ 72. Yasunaga, M., Aghajanyan, A., Shi, W., James, R., Leskovec, J., Liang, P., Lewis, M., Zettlemoyer, L., Yih, W.t.: Retrieval-Augmented Multimodal Language Modeling p. 39755–39769 (2022), https://proceedings.mlr.press/v202/yasunaga23a.html 73.Yuan, X., Ning, L., Fan, W., Li, Q.: mKG-RAG: Multimodal Knowledge Graph- Enhanced RAG for Visual Question Answering. arXiv preprint arXiv:2508.05318 (2025), https://arxiv.org/abs/2508.05318 74.Zhang, R., Liu, C., Su, Y., Li, R., Huang, X., Li, X., Yu, P.S.: A comprehensive survey on multimodal RAG: all combinations of modalities as input and out- put. Authorea Preprints (2025),https://w.techrxiv.org/doi/full/10.36227/ techrxiv.176341513.38473003 75.Zhang, T., Zhang, Z., Ma, Z., Chen, Y., Qi, Z., Yuan, C., Li, B., Pu, J., Zhao, Y., Xie, Z., Ma, J., Shan, Y., Hu, W.: mR 2 AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA. arXiv preprint arXiv:2502.01113 (2024), https://arxiv.org/abs/2411.15041 76.Zhang, Z., Feng, Y., Zhang, M.: LevelRAG: Enhancing Retrieval-Augmented Gen- eration with Multi-hop Logic Planning over Rewriting Augmented Searchers. arXiv preprint arXiv:2502.18139 (2025), https://arxiv.org/abs/2502.18139 77. Zhao, Y., Zhu, J., Guo, Y., He, K., Li, X.: E 2 GraphRAG: Streamlining Graph-based RAG for High Efficiency and Effectiveness. arXiv preprint arXiv:2505.24226 (2025), https://arxiv.org/abs/2505.24226 78.Zhi Lim, Q., Poo Lee, C., Ming Lim, K., Kamsani Samingan, A.: UniRaG: Uni- fication, Retrieval, and Generation for Multimodal Question Answering With 22Sijun Dai et al. Pre-Trained Language Models. IEEE Access 12, 71505–71519 (2024).https: //doi.org/10.1109/ACCESS.2024.3403101,https://doi.org/10.1109/ACCESS. 2024.3403101 79. Zhou, Y., Zhang, T., Xu, S., Chen, S., Zhou, Q., Tong, Y., Ji, S., Zhang, J., Qi, L., Li, X.: Are They the Same? Exploring Visual Correspondence Shortcom- ings of Multimodal LLMs. In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV). p. 17663–17674 (October 2025),https: //openaccess.thecvf.com/content/ICCV2025/html/Zhou_Are_They_the_Same_ Exploring_Visual_Correspondence_Shortcomings_of_Multimodal_ICCV_2025_ paper.html 80.Zhuang, L., Chen, S., Xiao, Y., Zhou, H., Zhang, Y., Chen, H., Zhang, Q., Huang, X.: LinearRAG: Linear Graph Retrieval Augmented Generation on Large-scale Corpora. In: Proceedings of the 14th International Conference on Learning Representations (ICLR) (2026), https://arxiv.org/abs/2510.10114 MG 2 -RAG: Multi-Granularity Graph for M-RAG23 A Implementation Details A.1 Rule-based Relation Extraction To construct semantic relations between named entities during textual graph construction, we employ a lightweight rule-based extraction strategy built on the dependency parsing trees generated byspaCy[29]. Unlike prior graph-based RAG systems that rely on expensive LLM-driven triplet extraction, our approach directly leverages grammatical structures to identify high-precision relations while maintaining computational efficiency, consistent with the lightweight Multimodal Knowledge Graph (MMKG) construction pipeline described in the main paper. The extraction procedure operates sentence-by-sentence. For each sentence, we first identify all named entities and retain only relational entitiesEwhose labels excludeORDINAL,CARDINAL,PERCENT, orQUANTITY. When at least two relational entities are present (|E|≥2), we examine each unique entity pair (e 1 ,e 2 ) using dependency-root alignment and part-of-speech (POS) analysis. To capture the dominant grammatical patterns in natural language, the extraction rules are organized into two complementary categories: –Verb-Centric Structures, which capture relations expressed via predicates. –Noun-Centric Structures, which extract relations from nominal construc- tions when no governing verb is present. For consistency across syntactic variations, all relation predicates are normalized using the lemma form of the governing token. Verb-Centric Rules. Verb-centric rules handle sentences where the relation between two entities is mediated by a verbal predicate. When the dependency roots ofe 1 ande 2 share the same governing verb, or are connected through a verb-preposition construction, the relation is derived from the lemma of the verb. –Predicate-Centered Structures: If both entities attach to the same verbal root as a subject (nsubjorcsubj) and object (dobjorattr), the relation is extracted directly from the verb. If the verb contains a negation modifier (neg), the predicate is prefixed with not. The resulting triplet is denoted as: ⟨e 1 , [not] + verb lemma ,e 2 ⟩. This rule captures the most common active predicate constructions. –Adpositional Structures: When the dependency path between two entities includes an adposition (ADP), the relation predicate is expanded to incorpo- rate the preposition. For standard active patterns, we append the preposition to the verb lemma: ⟨e 1 , verb + prep,e 2 ⟩. For passive constructions, we apply a passive correction mechanism. If the dependency pattern indicates a passive subject (nsubjpass) and the preposi- tion by, the entity roles are inverted to reconstruct the corresponding active relation. This normalization ensures consistent relational representations regardless of syntactic voice. The rules also handle inverted prepositional structures detected through symmetric dependency patterns. 24Sijun Dai et al. Sentences (a)Verb-CentricRules(b)Noun-CentricRules Predicate-Centered Structures AdpositionalStructures Nominal-Modifier Structures Noun-Adposition-Noun Structures DependencyRootsandPOSAnalysis Founded (root) SteveJobs (nsubj) Apple (dobj) ⟨Steve Jobs, found, Apple⟩ Founded (root) Apple (nsubjpass) was (auxpass) by (prep) SteveJobs (pobj) ⟨Steve Jobs, found, Apple⟩ Operates (root) “Steve Jobsfounded Apple” “Apple operates in California”“Applewas founded by Steve Jobs” Apple (nsubj) in (prep) California (pobj) ⟨Apple, operate in, California⟩ “Paris, the capitalof France” Paris (root) capital (appos) of (prep) France (pobj) ⟨Paris, is_also, capital⟩ headquarters (root) “Googleheadquarters in Mountain View” Google (compound) in (prep) Mountain View (pobj) Extraction ExtractionExtractionNormalization Extraction Extraction ⟨Google headquarters, located_in, Mountain View⟩ e 횓 e 횔 e 횓 e 횔 e 횓 e 횔 e 횓 e 횔 e 횓 e 횔 Relational Entities Ve r b-CentricRules Noun-CentricRules 퓔 Fig. 5: Rule-based Relation Extraction Examples. The extraction framework is divided into two syntactic categories: (a) Verb-Centric Rules, which capture relations expressed through predicates and verb–preposition structures, and (b) Noun- Centric Rules, which derive relations from nominal modifiers and noun-preposition constructions when no governing verb is present. Noun-Centric Rules. When no governing verb connects two entities, we extract relations from nominal modifiers and adpositional attachments. –Nominal-Modifier Structures: If two entities are linked through an appo- sitional modifier (appos), the construction indicates semantic equivalence or renaming. In such cases, we extract the relation ⟨e 1 , is_also,e 2 ⟩, representing one entity functions as an alternative description of the other. –Noun-Adposition-Noun Structures: When two entities are connected via an adpositional phrase without an intervening verb, the adposition itself carries the relational meaning. In this case, the preposition is mapped to a predefined semantic predicate. For example, spatial prepositions like in or at are normalized to the relation located_in, producing triplets of the form ⟨e 1 , located_in,e 2 ⟩, Extraction Examples. Figure 5 illustrates several representative cases demon- strating the application of these rules. For verb-centric structures, the sentence “Steve Jobs founded Apple” contains a subject–verb–object pattern, yielding the triplet⟨Steve Jobs, found, Apple⟩. Similarly, the sentence “Apple operates in California” contains a verb–preposition construction, resulting in ⟨Apple, operate_in, California⟩. For passive constructions such as “Apple was founded by Steve Jobs,” the passive correction rule reverses the entity order, producing the same active relation ⟨Steve Jobs, found, Apple⟩. MG 2 -RAG: Multi-Granularity Graph for M-RAG25 For noun-centric structures, the appositive phrase “Paris, the capital of France” triggers the nominal modifier rule, yielding⟨Paris, is_also, capital⟩. Finally, in the phrase “Google headquarters in Mountain View,” the adposi- tional relation is mapped to a spatial predicate, producing⟨Google headquarters, located_in, Mountain View⟩. A.2 Dataset Pre-Processing Dataset Split Strategies. For the Multimodal Retrieval and knowledge-based Visual Question Answering (VQA) tasks, we evaluate our approach on two large- scale benchmarks: E-VQA [55] and InfoSeek [10]. To maintain a consistent retrieval scale across our experiments, we follow standard practices established for InfoSeek [12,70,73] and restrict the external knowledge base for both datasets to 100k documents. The specific splitting and pre-processing details for each dataset are as follows: –E-VQA [55]: This dataset comprises 221k question-answer pairs, each paired with up to 5 images and associated with 16.7k fine-grained entities. Questions are categorized as either single-hop or two-hop. Single-hop queries require a single Wikipedia page to answer, whereas two-hop queries necessitate sequential retrieval across multiple documents. The dataset is partitioned into training, validation, and test splits, containing 1M, 13.6k, and 5.8k samples, respectively. All our experiments are conducted on the test split, which includes 4.8k single-hop questions. The provided external knowledge base, derived from Wikipedia, contains approximately 2M pages. As mentioned above, we randomly sample a subset of 100k documents from this original knowledge base for our evaluation. –InfoSeek [10]: The InfoSeek dataset consists of approximately 1.3M image- question-answer triplets corresponding to around 11k distinct Wikipedia pages. It is divided into training, validation, and test splits, containing roughly 934k, 73k, and 348k samples, respectively. Notably, both the validation and test sets feature questions regarding unseen entities. InfoSeek provides an external knowledge base of around 6M Wikipedia entities. we follow previous works [12,70,73] and conduct our experiments using a subset of 100k pages. Baseline Comparison Settings. For the multimodal retrieval and knowledge- based VQA tasks, to facilitate comparisons with baseline models, we construct smaller dataset variants from the original benchmarks. Since baseline approaches [48,65] rely on expensive MLLM-driven triplet extraction, making full-scale graph construction computationally and financially infeasible, these smaller variants are necessary for evaluation. The detailed splitting and pre-processing strategies are as follows: For the E-VQA and InfoSeek benchmarks, we restrict the knowledge base to a 5k subset. For this restricted 5k subset, we ensure the inclusion of the necessary evidence required to answer the evaluation questions. To align the evaluation scales, the 26Sijun Dai et al. E-VQA test set remains unchanged at 5.8k samples, while we randomly extract 5.8k samples from the InfoSeek 73k validation set. Closed-Source Models Evaluation. For the evaluation of closed-source MLLMs [20,58], we evaluate on the E-VQA [55], InfoSeek [10], ScienceQA [50], and CrisisMMD [1] (including both binary informativeness classification and fine-grained humanitarian category classification) datasets. Rather than using any restricted subsets, we randomly sample 1k instances directly from the original evaluation splits of each dataset (e.g., the E-VQA test set and the InfoSeek validation set). Efficiency and Cost Evaluation. To ensure a fair comparison regarding time efficiency, both our proposed method and the baselines are evaluated on the same hardware infrastructure. For the cost evaluation, we specifically focus on the expenses incurred during the knowledge graph construction phase. The baseline approaches rely on LLMs to construct the graph; therefore, their cost is measured by the token consumption based on public API pricing (i.e., Qwen-2.5- VL-7B-Instruct via OpenRouter. 4 ) In contrast, our graph construction process is completely token-free. Consequently, our cost is solely determined by the GPU rental expenses accumulated during the processing time, calculated using standard rates from the AutoDL platform. 5 A.3 Hyperparameter Configurations and Prompts of MG 2 -RAG Multimodal Retrieval Task. As a comprehensive Multimodal Retrieval- Augmented Generation (M-RAG) framework, MG 2 -RAG fundamentally relies on its retrieval component to support all downstream reasoning tasks. In this stage, the framework performs multi-granularity retrieval over the constructed multimodal knowledge graph to identify fine-grained textual and visual evidence relevant to the query. To provide a detailed overview of the retrieval configura- tions across different datasets, we summarize the corresponding hyperparameter settings in Table 5. These parameters are adapted to the characteristics of each task to ensure effective retrieval. For instance, in VQA, textual queries such as “what is this” often contain limited semantic information and therefore contribute less to retrieval. To address this issue, we adjust modality weights and seed node activations to emphasize visual evidence. Despite these task-specific configurations, several parameters remain consistent across all settings. In particular, we fix the visual grounding threshold atτ= 0.5 and set the Personalized PageRank [27] convergence tolerance toε= 10 −6 . The multimodal evidence retrieved in this stage serves as the grounding context for the generation process, providing structured information that is subsequently integrated into the prompts for the VQA, reasoning, and classification tasks described below. 4 https://openrouter.ai/qwen/qwen-2.5-vl-7b-instruct. 5 https://w.autodl.com/market/list MG 2 -RAG: Multi-Granularity Graph for M-RAG27 Table 5: MG 2 -RAG hyperparameter configurations across different datasets. These parameters are specifically employed during the Multi-Granularity Graph Re- trieval process to accommodate the distinct characteristics of various tasks. ParameterE-VQA Infoseek ScienceQA Task-BC Task-MC Graph Propagation and Modality Fusion Damping factor (α) 0.20 0.15 0.85 0.85 0.70 Chunk Node Weight (ω C ) 0.8 1.2 0.05 0.2 1.0 Image Node Weight (ω I ) 1.6 0.5 1.0 1.0 1.0 Textual Fusion Weight (λ t ) 0.1 0.1 1.0 1.0 1.0 Visual Fusion weight (λ v ) 1.0 1.0 1.0 0.5 1.0 Activated Seed Nodes by Textual Query Chunk top-k 60 200 4 7 12 Sentence top-k 3 3 10 5 2 Image top-k 2 2 10 5 3 Object top-k 3 3 10 5 2 Activated Seed Nodes by Visual Query Chunk top-k 200 200 10 3 12 Sentence top-k 70 60 10 1 3 Image top-k 2 1 20 1 5 Object top-k 5 5 10 1 3 Knowledge-based VQA Task. For Knowledge-based VQA, the hyperparame- ter configurations emphasizes visual evidence during retrieval. Both E-VQA and InfoSeek assign a high visual fusion weight (λ v = 1.0) and a low textual fusion weight (λ t = 0.1), reflecting the dominant role of visual queries. In addition, both datasets use a low damping factor (α ≤0.20), favoring the retrieval of highly relevant local evidence rather than deep multi-hop propagation. Despite these similarities, the datasets require different node-weighting strate- gies. E-VQA assigns a higher weight to image nodes (ω I = 1.6) to support visual-centric queries, whereas InfoSeek prioritizes chunk nodes (ω C = 1.2) to retrieve external textual knowledge grounded in the visual input. The retrieved evidence is then combined with the original user query to construct prompts for downstream models. We adopt task-specific prompt templates, following EchoSight [70] for E-VQA and OMGM [71] for InfoSeek. Multimodal Reasoning Task. For multimodal reasoning on ScienceQA, the configuration reflects the need for multi-hop reasoning across heterogeneous evidence. This is indicated by a high damping factor (α= 0.85), which encourages deeper graph propagation to connect distributed multimodal cues. The task also requires balanced multimodal understanding, and therefore assigns equal fusion weights to textual and visual queries (λ t = 1.0,λ v = 1.0). After retrieval, the collected contextual evidence is integrated with the question and candidate 28Sijun Dai et al. Prompt 1: Multimodal Reasoning (ScienceQA) [System] You are an expert science educator. Answer the multiple-choice science question. [MLLM: Do not mention the visual content in your output; base your answer directly on the image and context.] [Instructions] 1. Evaluate Context: Retrieved knowledge is for reference. Use it if relevant to support reasoning; ignore it if noisy and rely on your own knowledge. 2. Reasoning: Briefly explain your logic step-by-step. 3. Format: The final answer MUST be the Option Letter only (e.g., A, B, C, D, E). [User Input] [MLLM: Query Image] Retrieved Context: $context_user Question: $question_options_user [Response] Reasoning: options. A unified prompt template is used to guide both LLMs and MLLMs in performing step-by-step reasoning and answer selection, as shown in Prompt 1. 6 Multimodal Classification Task. The multimodal classification setting in- cludes two sub-tasks: Task-BC (binary informativeness classification) and Task- MC (multi-class fine-grained humanitarian category classification). Both tasks benefit from broader contextual aggregation, which is reflected in relatively high damping factors (α = 0.85 for Task-BC and α = 0.70 for Task-MC). Unlike VQA tasks, social-media posts contain rich textual semantics. Conse- quently, both tasks adopt a maximum textual fusion weight (λ t = 1.0). The visual fusion weight is then adjusted according to the classification objective. Task-BC uses a moderate visual weight (λ v = 0.5) because determining informativeness often relies primarily on textual cues. In contrast, Task-MC setsλ v = 1.0 to capture fine-grained visual signals, such as distinguishing infrastructure damage from rescue operations, that are essential for accurate multi-class classification. After retrieval, the contextual evidence (e.g., related tweets and historical labels) is formatted as structured references to guide the classification process. The prompt templates used for this stage are provided in Prompts 2 and 3. 6 Italicized instructions in the prompt indicate modality-specific guidance. MG 2 -RAG: Multi-Granularity Graph for M-RAG29 Prompt 2: Multimodal Classification (Task-BC) [System] You are an expert in crisis response and social media analysis. LLM: Answer the multiple-choice classification question [MLLM: Classify the crisis-related tweet based on its text and the provided image]. [Instructions] 1. Context Usage: Retrieved tweets and labels are reference criteria. Use them if relevant; otherwise, rely on your own judgment. 2. Reasoning: Briefly explain your logic step-by-step [MLLM: explicitly considering both visual and textual evidence]. 3. Format: The final answer MUST be the Option Letter only. [User Input] [MLLM: Query Image] Retrieved Context: $context_user Target Tweet: $question_user Options: [A] not_informative | [B] informative [Response] Reasoning: B Additional Case Study Examples Multimodal Reasoning Task. Figure 6 presents additional qualitative exam- ples for multimodal reasoning, where models must interpret complex interactions between visual and textual information. Baseline approaches often struggle in these scientific settings, either predicting incorrect entities or producing halluci- nated answers due to weak cross-modal alignment and limited logical reasoning. By employing modality-preserving multimodal node fusion, MG 2 -RAG tightly aligns fine-grained visual features with domain-specific textual knowledge. For instance, in Case (a), baseline models hallucinate geographical entities such as “Maryland” or “New Jersey.” In contrast, MG 2 -RAG explicitly links the visually highlighted region with the retrieved textual evidence “Pennsylvania is farthest north,” enabling the model to correctly identify the colony. Similarly, in Case (b), the framework connects the visual appearance of the animal with the retrieved biological description “hard outer shell,” allowing it to distinguish the Nautilus from visually similar alternatives such as a snake. Beyond improved alignment, multi-granularity graph retrieval enables structured multi-hop reasoning rather than superficial fact matching. In Case (c), baseline models are misled by the visual similarity of the flasks and guess between 30Sijun Dai et al. Prompt 3: Multimodal Classification (Task-MC) [System] You are an expert in crisis response and humanitarian aid. [LLM: Answer the multiple-choice classification question.] [MLLM: Classify the humanitarian category of the tweet based on both its text and the provided image.] [Instructions] 1. Reasoning: Briefly explain your logic step-by-step [MLLM: explicitly considering both visual and textual evidence]. 2. Format: The final answer MUST be the Option Letter only. [User Input] [MLLM: Query Image] Retrieved Context: $context_user Target Tweet: $question_user Options: [A] infrastructure_and_utility_damage | [B] not_humanitarian [C] other_relevant_information [D] rescue_volunteering_or_donation_effort [E] vehicle_damage | [F] affected_individuals [G] injured_or_dead_people | [H] missing_or_found_people [Response] Reasoning: “Solution A” or “Solution B.” MG 2 -RAG instead retrieves the governing scientific rule (concentration = particles/volume) and combines it with the visual evidence to correctly infer “Neither.” Likewise, in Case (d), the retrieval process connects multiple pieces of textual and visual evidence related to trading constraints, allowing the model to accurately determine the exchanged items (“Tomatoes and broccoli”). These examples illustrate how the combination of precise cross-modal alignment and multi-hop graph reasoning enables MG 2 -RAG to provide reliable and interpretable solutions for complex scientific reasoning tasks. Multimodal Classification Task. Figure 7 further demonstrates the robustness of MG 2 -RAG in multimodal classification, particularly under noisy real-world conditions such as social media posts. In such settings, visual and textual signals are often ambiguous, inconsistent, or emotionally charged, which frequently leads baseline models to produce generic or incorrect predictions. Through modality-preserving multimodal node fusion, MG 2 -RAG extracts precise semantic cues from noisy multimodal inputs. For instance, in Case (a), integrates visual evidence of street debris with the textual reference to “clean up,” retrieving infrastructure-related context and correctly predicting MG 2 -RAG: Multi-Granularity Graph for M-RAG31 Case (d) Method Responses VaLiK:Tomatoesand almonds MMGraphRAG: Different items Qwen3.5-27B: Orangesandbroccoli MG ퟐ -RAG:Tomatoesandbroccoli Retrieved Evidence Q:What can Sandeep and Tracy trade to each get what they want? Look at the labeled part of the images.Sandeep has broccoli. ...for broccoli to both be happier. Case (a) Method Responses VaLiK:Maryland MMGraphRAG: Maryland Qwen3.5-27B: New Jersey MG ퟐ -RAG:Pennsylvania Retrieved Evidence Q:What is the name of the colony shown? To find the answer, on most maps, north is at the top of the map. Pennsylvania is farthest north. Case (b) VaLiK:rat snake MMGraphRAG: rat snake Qwen3.5-27B: Nautilus MG ퟐ -RAG:Nautilus Retrieved Evidence Q:Which is better for protection against a sharp teeth predator? The nautilus has a hard outer shell. Its body is adapted for protection against a predator with sharp teeth. Method Responses Case (c) Q:Which solution has a higher concentration of blue particles? Retrieved Evidence Concentrationcan be described in terms of ...concentration = particles of solute / volume of solvent Method Responses VaLiK:SolutionB MMGraphRAG: SolutionA Qwen3.5-27B: SolutionB MG ퟐ -RAG:Neither Fig. 6: Case studies on Multimodal Reasoning. MG 2 -RAG is compared with a base- line MLLM (Qwen3.5-27B) and graph-based approaches (VaLiK and MMGraphRAG). The Retrieved Evidence illustrates the multimodal evidence retrieved by MG 2 -RAG for answer generation. Hurricane Irma: Residential trash, recycling pickup to be suspended (Infrastructure and utility damage) Retrieved Evidence VaLiK:Rescuevolunteeringor donationeffort MMGraphRAG: Otherrelevant information Qwen3.5-27B: Rescuevolunteering ordonationeffort MG ퟐ -RAG:Infrastructureand utilitydamage Method Responses Case (a) Before Hurricane Irma: Quinn Street flood clean up... TweetContent: Has Trump donated his $1M for Harvey relief yet?(Rescue volunteering or donation effort) Retrieved Evidence VaLiK:Nothumanitarian MMGraphRAG: Otherrelevant information Qwen3.5-27B: Otherrelevant information MG ퟐ -RAG:Rescuevolunteeringor donationeffort Method Responses Case (b) Y'allspresident has a networthof 3.5 billion dollars, but could only donate 1 mill to Harvey victims? TweetContent: There‘s another hurricanecoming...for Caribbeans, this hurricane (Other relevant information) Retrieved Evidence VaLiK:Affectedindividuals MMGraphRAG: Rescue volunteeringordonationeffort Qwen3.5-27B: Affectedindividuals MG ퟐ -RAG:Otherrelevant information Method Responses Case (c) TweetContent: So much to process when people are still in recovery mode Harvey!! Heartbreaking!! Praying... Pics of NOAA Southeast Science Center in Miami, post Irma....(Not humanitarian) Retrieved Evidence VaLiK:Otherrelevantinformation MMGraphRAG: Otherrelevant information Qwen3.5-27B: Other relevant information MG ퟐ -RAG:Nothumanitarian Method Responses Case (d) Bacardi visitor centrereopens after Hurricane Maria... TweetContent: Fig. 7: Case studies on Multimodal Classification. MG 2 -RAG is compared with a baseline MLLM (Qwen3.5-27B) and graph-based approaches (VaLiK and M- GraphRAG). The Retrieved Evidence illustrates the multimodal evidence retrieved by MG 2 -RAG for answer generation. “Infrastructure and utility damage.” Similarly, in Case (b), the system aligns textual commentary about donations with the visual news content, successfully identifying the underlying topic as “Rescue volunteering or donation effort.” In addition, multi-granularity graph retrieval provides a structural rea- soning mechanism that helps filter misleading contextual signals. In Case (c), emotionally charged text such as “Heartbreaking!! Praying...” misleads baseline models into predicting categories like “Affected individuals” or “Rescue volun- teering.” MG 2 -RAG instead retrieves factual meteorological updates, correctly grounding the post in “Other relevant information.” Likewise, in Case (d), the 32Sijun Dai et al. presence of the word “Hurricane” leads baseline models toward generic disaster labels. In contrast, MG 2 -RAG retrieves contextual evidence about a building reopening, enabling the model to correctly classify the post as “Not humanitarian.” Together, multimodal node fusion and multi-granularity graph retrieval form a complementary mechanism that enables MG 2 -RAG to perform reliable and explainable reasoning in challenging multimodal scenarios.

Instant research discovery