Paper deep dive

GraphLoom: Reliability-Calibrated Graph Evidence Routing for Multimodal KG-RAG

Zafar Ali, Asad Khan, Aalia Malik, Pavlos Kefalas

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 72

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 8/18/2026, 5:19:57 AM

Summary

GraphLoom is a reliability-calibrated multimodal knowledge-graph retrieval-augmented generation (KG-RAG) framework designed to improve answer quality and evidence faithfulness in multimodal question answering. It constructs instance-level multimodal knowledge graphs from grounded scene descriptions, extracted relational triples, and external commonsense knowledge. GraphLoom employs bounded subgraph retrieval, interleaved multi-hop retrieval, and a reliability-calibrated router that selects high-utility evidence slots for injection into a frozen language model via joint graph-sequence attention, thereby reducing noise and hallucination compared to standard long-context RAG approaches.

Entities (12)

GraphLoom · framework · 100%Llama-3.1-8B-Instruct · model · 95%MultiModalQA · dataset · 95%OK-VQA · dataset · 95%Qwen3-VL-Instruct · model · 95%REBEL · model · 95%ScienceQA · dataset · 95%ConceptNet · knowledge-base · 90%HieraSlot · component · 90%KG-JSA++ · mechanism · 90%Visual Genome · dataset · 90%MiniCheck · tool · 85%

Relation Signals (11)

GraphLoom → evaluatedon → MultiModalQA

confidence 95% · We evaluate GraphLoom on ... MultiModalQA

GraphLoom → evaluatedon → ScienceQA

confidence 95% · We evaluate GraphLoom on ScienceQA

GraphLoom → evaluatedon → OK-VQA

confidence 95% · We evaluate GraphLoom on ... OK-VQA

GraphLoom → uses → Qwen3-VL-Instruct

confidence 95% · grounded scene descriptions generated by Qwen3-VL-Instruct

GraphLoom → uses → REBEL

confidence 95% · relational triples extracted by REBEL

GraphLoom → usesdecoder → Llama-3.1-8B-Instruct

confidence 95% · frozen Llama-3.1-8B-Instruct decoder

GraphLoom → enricheswith → ConceptNet

confidence 90% · commonsense knowledge from ConceptNet

GraphLoom → enricheswith → Visual Genome

confidence 90% · enriched with visual relation priors derived from Visual Genome (VG150)

GraphLoom → implements → HieraSlot

confidence 90% · Retrieved triples are converted into compact key–value slot memories, called HieraSlot.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Multimodal retrieval-augmented generation (RAG) systems often rely on long unstructured contexts or aggressively expanded evidence graphs, which can introduce noisy evidence, weaken multi-hop reasoning, and increase unsupported generation. We present GraphLoom, a reliability-calibrated multimodal knowledge-graph RAG framework for compact and faithful evidence routing. Given a question and its associated multimodal input, GraphLoom constructs an instance-level multimodal knowledge graph from grounded scene descriptions, extracted relational triples, and external commonsense knowledge. Instead of injecting all retrieved evidence into the generator, GraphLoom performs reliability-aware subgraph retrieval with bounded expansion and selectively routes high-utility evidence through hierarchical graph memory slots and joint graph-sequence attention in a frozen language model. To improve robustness in complex reasoning settings, GraphLoom further combines interleaved retrieval with budgeted corrective retrieval, enabling adaptive multi-hop evidence refinement under noisy retrieval conditions. We evaluate GraphLoom on ScienceQA, MultiModalQA, and OK-VQA, including large distractor evidence pools that approximate noisy external knowledge retrieval. Experimental results show consistent gains in answer quality and evidence faithfulness over strong multimodal RAG, graph-retrieval, and open-source vision-language baselines, with improved retrieval quality on MultiModalQA and stable performance under noisy evidence pools. Additional analyses using MiniCheck-based verification, human evaluation, and latency profiling show that reliability-calibrated graph evidence routing provides an effective alternative to long-context multimodal evidence injection.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

68,884 characters extracted from source content.

Expand or collapse full text

GraphLoom: Reliability-Calibrated Graph Evidence Routing for Multimodal KG-RAGConference: Proceedings of the 35th ACM International Conference on Information and Knowledge Management; November 7–11, 2026; Rome, ItalyCCS: Information systems Information retrievalCCS: Information systems Question answeringCCS: Information systems Multimedia and multimodal retrievalCCS: Computing methodologies Knowledge representation and reasoningCCS: Computing methodologies Natural language processing Zafar Ali Affiliation: School of Computer Science and Engineering, Southeast University , Nanjing , China email: zafar_ali@seu.edu.cn , Asad Khan Affiliation: School of Computer Science and Engineering, Southeast University , Nanjing , China email: asadkhan@seu.edu.cn , Aalia Malik Affiliation: School of Computer Science and Engineering, Southeast University , Nanjing , China email: aaliamalik@seu.edu.cn and Pavlos Kefalas Affiliation: Dashub & Department of Informatics, Aristotle University , Thessaloniki , Greece email: pavloskefalas@gmail.com 2026; © , 2026 Abstract. Multimodal retrieval-augmented generation (RAG) systems often rely on long unstructured contexts or aggressively expanded evidence graphs, which can introduce noisy evidence, weaken multi-hop reasoning, and increase unsupported generation. We present GraphLoom, a reliability-calibrated multimodal knowledge-graph RAG framework for compact and faithful evidence routing. Given a question and its associated multimodal input, GraphLoom constructs an instance-level multimodal knowledge graph from grounded scene descriptions, extracted relational triples, and external commonsense knowledge. Instead of injecting all retrieved evidence into the generator, GraphLoom performs reliability-aware subgraph retrieval with bounded expansion and selectively routes high-utility evidence through hierarchical graph memory slots and joint graph–sequence attention in a frozen language model. To improve robustness in complex reasoning settings, GraphLoom further combines interleaved retrieval with budgeted corrective retrieval, enabling adaptive multi-hop evidence refinement under noisy retrieval conditions. We evaluate GraphLoom on ScienceQA, MultiModalQA, and OK-VQA, including large distractor evidence pools that approximate noisy external knowledge retrieval. Experimental results show consistent gains in answer quality and evidence faithfulness over strong multimodal RAG, graph-retrieval, and open-source vision–language baselines, with improved retrieval quality on MultiModalQA and stable performance under noisy evidence pools. Additional analyses using MiniCheck-based verification, human evaluation, and latency profiling show that reliability-calibrated graph evidence routing provides an effective alternative to long-context multimodal evidence injection. Keywords: Multimodal question answering, knowledge-graph retrieval, retrieval-augmented generation, multimodal KG-RAG, reliability-calibrated evidence routing 1. Introduction Multimodal question answering requires grounded reasoning over heterogeneous evidence, including textual descriptions, tables, images or diagrams, and structured relational knowledge. Knowledge-graph question answering (KG-QA) (16; 13) and retrieval-augmented generation (RAG) (6; 11) offer complementary strengths: graphs expose explicit relational structure for multi-hop reasoning, while neural retrieval supports evidence access over large and noisy information spaces. However, reliable multimodal KG-assisted QA remains challenging because relevant evidence is often distributed across visual content, textual or tabular context, external knowledge, and inferred relations. This makes the problem not only a reasoning task, but also an evidence retrieval and knowledge-management problem. Despite rapid progress in vision–language models (22; 19), current multimodal QA pipelines still face several limitations. First, multimodal knowledge-graph reasoning remains sensitive to incomplete knowledge and noisy or imperfectly fused multimodal evidence, which can impair multi-hop inference (40). Second, large or broadly expanded graph contexts can exceed practical context budgets and make evidence selection difficult (11), while multimodal generators remain susceptible to visually ungrounded hallucinations (18). Third, scalable multi-hop retrieval and long-context decoding remain expensive. Large KG search spaces motivate retrieval of compact supporting subgraphs (13; 11), while retaining historical key–value states during decoding incurs substantial memory cost (37). Sparse-attention mechanisms can reduce this inference burden (36). Fourth, many retrieval and correction pipelines depend on multiple thresholds and confidence scores, yet their robustness under parameter variation is often unclear. Finally, using the same verifier for both answer revision and final faithfulness assessment can risk circular evaluation. These issues are especially important for information and knowledge management settings, where systems must retrieve from noisy evidence pools, remain efficient, and provide credible grounding diagnostics. To address these limitations, we propose GraphLoom, a reliability-calibrated multimodal KG-RAG framework for controlled evidence retrieval, organization, and injection. Given a question and its associated multimodal input, GraphLoom constructs an instance-level MMKG from available evidence, including grounded scene descriptions generated by Qwen3-VL-Instruct (4) and relational triples extracted by REBEL (12). The graph is enriched with visual relation priors derived from Visual Genome (VG150) (15) and commonsense knowledge from ConceptNet (31). A unified multimodal embedder, Qwen3-VL-Embedding (20), performs task-conditioned subgraph retrieval with bounded expansion, producing compact supporting evidence without unbounded graph traversal. To evaluate scalability beyond a small local graph, we further test retrieval under large distractor evidence pools that simulate noisy external knowledge sources. For multi-hop questions, GraphLoom performs interleaved retrieval, where intermediate hypotheses trigger an additional bounded retrieval round before decoding (35). When the retrieved evidence appears insufficient, a lightweight adequacy evaluator triggers at most one budgeted corrective action, following the corrective-retrieval principle of CRAG (38). GraphLoom considers targeted expansion, query reformulation, or evidence re-scoring as bounded corrective actions. After generation, a verification–revision stage edits unsupported spans using retrieved evidence (8). To avoid circular faithfulness claims, we evaluate final grounding with an independent verifier and human validation rather than relying only on the verifier used during revision. A key design goal of GraphLoom is decoder-time evidence control under transparent computation budgets. Retrieved triples are converted into compact key–value slot memories, called HieraSlot. A reliability-calibrated router estimates the utility and reliability of candidate slots and activates only a small subset at each decoding step. These selected slots, together with multimodal prefix memories, are injected into a frozen Llama-3.1-8B-Instruct decoder through KG-JSA++ joint graph–sequence attention. This design supports stepwise grounding without relying on long unstructured prompts or broad evidence expansion. We also distinguish full-pipeline latency from online latency under cached graph construction, making the efficiency profile explicit. We evaluate GraphLoom on ScienceQA, OK-VQA, and MultiModalQA across multiple-choice and open-ended QA settings. Beyond answer correctness, we analyze retrieval quality, evidence faithfulness, scalability under distractor evidence, threshold robustness, and latency. We compare GraphLoom with multimodal RAG, graph-based retrieval, interleaved/corrective retrieval, and recent vision–language baselines. Our main contributions are: • We introduce GraphLoom, a reliability-calibrated multimodal KG-RAG framework that treats evidence retrieval, graph organization, and decoder-time evidence use as a controlled knowledge-management problem. • We propose a controlled evidence injection interface that converts retrieved graph evidence into multimodal prefix memories and hierarchical key–value graph slots, enabling compact grounding in a frozen language model. • We develop a reliability-calibrated slot routing mechanism with KG-JSA++ joint graph–sequence attention, allowing useful evidence slots to compete with sparse self-context during decoding while suppressing noisy or low-confidence evidence. • We provide a comprehensive evaluation covering answer accuracy, retrieval quality, faithfulness, scalability, latency, and robustness. Results on ScienceQA, MultiModalQA, and OK-VQA show improved answer quality and evidence grounding over strong multimodal RAG, graph-retrieval, and vision–language baselines, with retrieval gains further demonstrated on MultiModalQA. 2. Related Work Multimodal QA spans vision–language modeling, structured knowledge integration, retrieval-augmented generation, and faithful answer generation. Instruction-following VLMs and multimodal LLMs, including BLIP-2 (19), LLaVA (22), InstructBLIP (7), MiniGPT-4 (41), Flamingo (1), Qwen3-VL (4), and proprietary models such as GPT-5 and Gemini 2.5 (28; 9), have substantially improved image–text reasoning. However, their predictions often depend on dense visual tokens, long prompts, or implicit parametric knowledge, making evidence attribution difficult for multi-hop or knowledge-intensive questions. GraphLoom therefore focuses on explicit evidence retrieval and controlled use during decoding. Multimodal knowledge graphs (MMKGs) provide a complementary route to structured reasoning. MMKGR (40) performs multi-hop reasoning over MMKGs, while Lee et al. (17) integrate MMKG representations with LLM-based multimodal reasoning. Other work studies MMKG construction from cross-modal evidence (23). Pythia-RAG (2) constructs a unified MMKG from textual, visual, and commonsense evidence and retrieves query-conditioned subgraphs, whereas EvoGraph-R1 (21) supports iterative retrieval, expansion, and graph refinement over evolving multimodal knowledge hypergraphs. GraphLoom differs by focusing on which retrieved graph memories should reach the decoder at each generation step. Recent RAG methods increasingly emphasize structured, iterative, and utility-aware retrieval. G-Retriever, SKURG, and IRCoT combine graph retrieval, structured multimodal evidence, or intermediate reasoning with evidence acquisition (11; 39; 35), while CRAG adds retrieval-quality assessment and corrective retrieval (38). VisDoM (32) combines visual and textual retrieval for multimodal document QA, and Talk2Doc (14) uses weighted knowledge graphs within RAG for patient QA. More recent systems include HiKEY (30), which performs hierarchical coarse-to-fine multimodal retrieval with token-efficient evidence subgraphs, and Luo et al. (25), who rank visual evidence by estimated downstream utility rather than semantic similarity alone. RARR (8) complements these approaches by retrieving supporting evidence and revising unsupported outputs. GraphLoom builds on these ideas through reliability-calibrated, decoder-time routing of graph memories. Efficiency is also important in retrieval-augmented reasoning because long contexts, repeated retrieval, and growing KV caches increase latency and memory use. Attention-sink retention and sparse-attention methods reduce these costs (37; 36), but are not designed for graph-evidence routing. GraphLoom combines bounded subgraph retrieval, sparse self-context, and decoder-time slot selection, while reporting both full-pipeline and cached online latency. Faithfulness remains central to evidence-grounded QA. FActScore (27) evaluates generations through atomic factual claims, while MiniCheck (34) provides claim-level verification against grounding documents. Revision-based methods such as RARR (8) can further improve attribution and correct unsupported content. GraphLoom separates revision-time verification from final assessment through an independent automatic verifier and human evaluation. Overall, GraphLoom differs from prior multimodal QA, MMKG, and RAG systems in three respects: it treats multimodal KG-RAG as controlled evidence management rather than simple retrieve-and-read; it performs reliability-calibrated graph-slot routing inside a frozen decoder; and it evaluates answer quality together with retrieval effectiveness, noisy-evidence scalability, threshold robustness, latency, and independently assessed faithfulness. Figure 1. GraphLoom architecture. Given a question and its associated multimodal input, GraphLoom constructs an instance-level multimodal knowledge graph, retrieves a compact task-conditioned evidence subgraph, and organizes the retrieved evidence into multimodal prefix memories and hierarchical key–value graph slots. A reliability-calibrated router selects high-utility slots, which are injected into a frozen decoder through KG-JSA++ joint graph–sequence attention. Interleaved retrieval and budgeted corrective retrieval improve multi-hop evidence coverage, while verification–revision supports faithful answer generation. 3. Methodology: GraphLoom Figure 1 illustrates GraphLoom, a reliability-calibrated multimodal KG-RAG framework for controlled evidence retrieval, organization, and decoder-time injection. Given a question Q and its associated multimodal input, including image I when available, GraphLoom constructs an instance-level multimodal knowledge graph (MMKG) from grounded scene descriptions, extracted relational triples, and external visual/common-sense knowledge. It then retrieves a compact task-conditioned evidence subgraph using multimodal dense retrieval with bounded expansion. Retrieved triples are encoded as HieraSlot memories with a multimodal prefix. During decoding, a reliability-calibrated router selects Top-k graph slots using combined utility–reliability scores, and KG-JSA++ injects the selected slots and prefix memories into a frozen Llama-3.1-8B-Instruct decoder through joint graph–sequence attention. GraphLoom separates evidence construction from online evidence use. For static multimodal collections, scene descriptions, triples, entity links, and enriched instance graphs can be precomputed and indexed. At query time, the online path includes query-conditioned retrieval, slot routing, decoder-time evidence injection, and conditional correction/revision. This enables the experiments to report both full end-to-end latency and cached online latency, clarifying where computational cost occurs. For multi-hop questions, GraphLoom performs at most one additional hypothesis-guided retrieval round. If evidence is insufficient, a lightweight adequacy evaluator triggers at most one budgeted corrective action. After generation, a verification–revision stage edits unsupported spans using retrieved evidence. The revision-time verifier is treated as an internal component, while final faithfulness is evaluated with independent evaluators and human validation. 3.1. Augmented Multimodal KG Construction For each input instance, we construct an instance-level multimodal knowledge graph MMKG=(V,E)G_MMKG=(V,E) that represents extracted multimodal evidence under a shared (s,r,o)(s,r,o) schema. The graph construction is independent of the question Q, which is introduced later during task-conditioned retrieval. The graph is local to the input instance, but its triples are embedded in the same retrieval space as external candidate evidence, allowing the retrieval module to be evaluated under both compact local and larger noisy evidence-pool settings. Scene description and triple extraction. When an image I is available, Qwen3-VL-Instruct (4) generates a concise scene description D grounded in the image. We then extract structured triples t=(s,r,o)t=(s,r,o) from D using REBEL (12). Each extracted triple is associated with a normalized extraction-confidence score confext(t)∈[0,1]conf_ext(t)∈[0,1], derived from the model’s generation likelihood. Entity mentions are canonicalized using a lightweight named-entity and alias-linking step. Available textualized input evidence that is not represented as graph triples is retained separately for the evidence-bound revision context in Section 3.7. For static image repositories, this construction step is cacheable and does not need to be repeated for every question over the same image. Confidence-aware filtering. To reduce extraction noise, we retain only triples satisfying (1) E=t∣confext(t)≥τedge.E= \t _ext(t)≥ _edge \. The default value is τedge=0.5 _edge=0.5. All filtering thresholds are selected using validation data only and then kept fixed during testing. We do not tune thresholds separately for individual test sets. The experimental section reports robustness under ±20%± 20\% perturbations of the main evidence-selection and verification thresholds. External KG enrichment. We enrich the instance graph with relevant neighbors from VG150 visual relations and ConceptNet commonsense knowledge (31). For each entity e∈Ve∈ V, we consider neighbors within at most two hops and retain relations whose normalized source score exceeds τKB=0.7 _KB=0.7, yielding (2) MMKG+=(V∪ΔV,E∪ΔE).G^+_MMKG= (V∪ V,\;E∪ E ). The enrichment is depth-bounded to two hops, while the subsequent retrieval stage applies explicit degree and total-edge budgets to prevent uncontrolled evidence expansion. In the standard setting, retrieval is performed over MMKG+G^+_MMKG. In the scalability setting, MMKG+G^+_MMKG is mixed with an external distractor evidence pool to test retrieval robustness under noisy knowledge-management conditions. 3.2. Compact Evidence Subgraph Retrieval GraphLoom retrieves evidence from a candidate pool ⁡(Q,I)C(Q,I): (3) ⁡(Q,I)=ℰMMKG+∪ℰext,C(Q,I)=E^+_MMKG _ext, where ℰMMKG+E^+_MMKG denotes triples from the enriched instance graph and ℰextE_ext denotes optional external or distractor triples used in scalability experiments. In the standard instance-level setting, ℰext=∅E_ext= . This formulation keeps the core retrieval mechanism unchanged while allowing evaluation under larger and noisier candidate evidence pools. Each candidate triple t is verbalized as text⁡(t)text(t) and scored by multimodal similarity: (4) score⁡(t,Q,I)=cos⁡(fembed(I,Q),fembed(text⁡(t))),score(t,Q,I)= \! (f_embed(I,Q),f_embed(text(t)) ), where fembedf_embed denotes Qwen3-VL-Embedding (20); when no image is available, the query is embedded in text-only mode. For large candidate pools, triple embeddings are precomputed and indexed before query-time retrieval. This affects retrieval efficiency but does not change the scoring function. (5) sub(Q)=(Vsub,ℰsub(Q)),|ℰsub(Q)|≤Nmax.G_sub(Q)= (V_sub,E_sub(Q) ), |E_sub(Q)|≤ N_ . This bounded retrieval design prevents the decoder from receiving large flattened neighborhoods. The budgets K, dmaxd_ , and NmaxN_ are selected on validation data and kept fixed during testing. 3.3. Interleaved Multi-hop Retrieval To recover missing connectors for multi-hop reasoning, GraphLoom performs at most two bounded retrieval rounds. From the initial subgraph sub(1)(Q)G_sub^(1)(Q), we select bridge nodes using degree and centrality signals. Candidate bridge nodes satisfy deg⁡(v)≥2 (v)≥ 2 and are ranked by betweenness centrality within the retrieved subgraph. A frozen Llama-3.1-8B-Instruct decoder then generates a concise intermediate query Q~ Q conditioned on Q and the selected bridge evidence. To limit autoregressive overhead, Q~ Q is capped at 15 tokens and only one additional retrieval round is allowed. We rerun retrieval with (I,Q~)(I, Q) to obtain sub(2)(Q~)G_sub^(2)( Q). The merged edge set is represented using following equation: (6) ℰmrg=ℰsub(1)(Q)∪ℰsub(2)(Q~).E_mrg=E_sub^(1)(Q) _sub^(2)( Q). From this merged set, we retain the top-NmaxN_ edges according to their original question-conditioned scores: (7) ℰsub(Q)=TopNt∈ℰmrg⁡(score⁡(t,Q,I),min⁡(Nmax,|ℰmrg|)).E_sub(Q)=TopN_\,t _mrg (score(t,Q,I), \! (N_ ,|E_mrg| ) ). This interleaved step improves evidence coverage for multi-hop questions while keeping the number of retrieval and generation calls bounded. Its cost is reported separately from initial retrieval and final decoding. 3.4. Evidence Memories: Fusion and HieraSlot Retrieved evidence conditions the frozen Llama-3.1-8B-Instruct decoder through two complementary memory channels: a global multimodal prefix and structured graph-slot memories. Global multimodal prefix. We concatenate the question Q with a concise facts block ℱF derived from sub(Q)G_sub(Q), yielding T=[Q;ℱ]T=[Q;F]. Text tokens are encoded by the frozen Qwen3-VL-Embedding text encoder to obtain Htext∈ℝLT×dembH_text ^L_T× d_emb, while visual tokens Hvis∈ℝLV×dvisH_vis ^L_V× d_vis are extracted by the frozen Qwen3-VL-Instruct vision encoder. We fuse the two modalities with a lightweight cross-attention module: (8) Hfus=CrossAttn⁡(Htext,Hvis,θfus)∈ℝLT×d,H_fus=CrossAttn(H_text,H_vis; _fus) ^L_T× d, where θfus _fus are trainable. The fused representation is pooled to a fixed prefix length Lp=32L_p=32 and converted into prefix key/value pairs (Kfus,Vfus)(K_fus,V_fus) for KG-JSA++. Structured slot memories (HieraSlot). Each retrieved triple t=(s,r,o)∈ℰsub(Q)t=(s,r,o) _sub(Q) is represented by two complementary slots: an entity slot with key ke=Emb⁡(s)k_e=Emb(s) and value ve=Emb⁡(r⊕o)v_e=Emb(r o), and a relation slot with key kr=Emb⁡(r)k_r=Emb(r) and value vr=Emb⁡(s⊕o)v_r=Emb(s o). The embedding projection is defined as: (9) Emb⁡(x)=Wslotfembedtext(x)+bslot,Emb(x)=W_slotf_embed^text(x)+b_slot, which maps the text embedding into the decoder hidden space. For M retrieved triples, the slot bank is given as follows: (10) SHieraSlot=(ke(i),ve(i))i=1M∪(kr(i),vr(i))i=1M.S_HieraSlot= \(k_e^(i),v_e^(i)) \_i=1^M∪ \(k_r^(i),v_r^(i)) \_i=1^M. Entity slots capture entity-specific attributes, whereas relation slots encode relational patterns. During routing, these slot types compete independently, enabling fine-grained control over whether entity-level or relation-level evidence is used at a decoding step. The selected slot representations are subsequently mapped to the layer-specific key/value space used by KG-JSA++. 3.5. Reliability-Calibrated Slot Routing At decoding step t, GraphLoom selects a small active set of graph slots from SHieraSlotS_HieraSlot. For each candidate slot j, the router uses slot content, retrieval score, extraction confidence, source type, graph centrality, and the current decoder state to estimate a utility score uju_j and reliability score c^j c_j: (11) uj=MLPu(ϕj),c^j=σ⁡(MLPc(ϕj)),u_j=MLP_u( _j), c_j=σ\! (MLP_c( _j) ), where ϕj _j denotes the feature vector for slot j and σ⁡(⋅)σ(·) is the sigmoid function. Because the decoder-state component changes with decoding step t, uju_j, c^j c_j, and ρj _j are step-dependent; the index t is suppressed for readability. The final routing score combines semantic utility and reliability: (12) ρj=uj+βlogit(c^j), _j=u_j+β\,logit( c_j), where β controls the strength of reliability calibration. The active slot set is defined as follows: (13) t=TopKj∈SHieraSlot⁡(ρj,k).A_t=TopK_j∈ S_HieraSlot ( _j,k ). Only slots in tA_t are passed to KG-JSA++. This keeps decoder-time evidence use compact and reduces exposure to noisy triples. The active-slot budget k is fixed after validation, and its sensitivity is reported experimentally. 3.6. KG-JSA++: Joint Attention with Sparse Self-KV KG-JSA++ integrates external graph evidence with decoder self-attention through a single joint attention operation inside the frozen Llama-3.1-8B-Instruct decoder. Unlike separate cross-attention, it lets routed graph memories and internal context compete directly for attention, enabling fine-grained and reliability-aware evidence use. At decoding layer ℓ and step t, GraphLoom forms prefixed memory by concatenating the fused multimodal prefix with the routed slot memories: (14) Kpref(ℓ)(t)=[Kfus(ℓ);K~t(ℓ)],Vpref(ℓ)(t)=[Vfus(ℓ);V~t(ℓ)],K_pref^( )(t)= [K_fus^( ); K_t^( ) ], V_pref^( )(t)= [V_fus^( ); V_t^( ) ], where (K~t(ℓ),V~t(ℓ))( K_t^( ), V_t^( )) are the key/value pairs corresponding to active slots in tA_t. To improve decoding efficiency, we sparsify the self-attention key–value cache instead of attending to the full token history. Following the attention-sink principle of StreamingLLM (37), the retained cache includes: (i) recent tokens, (i) sink tokens from the beginning of the sequence, and (i) high-similarity prior tokens selected by cosine similarity to the current query qt(ℓ)q_t^( ). Their union is defined as: (15) It(ℓ)=Irecent∪Isink∪Isim.I_t^( )=I_recent∪ I_sink∪ I_sim. KG-JSA++ computes attention logits over both external memories and sparse self-context: (16) Spref S_pref =qt(ℓ)(Kpref(ℓ)(t))⊤dh+Bpref(t), = q_t^( ) (K_pref^( )(t) ) d_h+B_pref(t), Sself S_self =qt(ℓ)(Kself(ℓ)(t)[It(ℓ)])⊤dh+Mcausal[It(ℓ)], = q_t^( ) (K_self^( )(t)[I_t^( )] ) d_h+M_causal[I_t^( )], where dhd_h is the attention-head dimension and McausalM_causal enforces causal masking. To encourage reliable evidence without overriding semantic relevance, slot positions receive a reliability-aware logit bias: (17) Bpref(t)[j]=γlogit(c^j),j∈slot positions,0,j∈fused-prefix positions,B_pref(t)[j]= casesγ\,logit( c_j),&j positions,\\ 0,&j -prefix positions, cases where γ is fixed after validation. A single softmax is applied over the concatenated prefix and self-attention scores: (18) αt(ℓ)=softmax⁡([Spref;Sself]). _t^( )=softmax ( [S_pref;S_self ] ). The output is computed as follows: (19) ot(ℓ)=αt(ℓ)[Vpref(ℓ)(t)Vself(ℓ)(t)[It(ℓ)]].o_t^( )= _t^( ) bmatrixV_pref^( )(t)\\ V_self^( )(t)[I_t^( )] bmatrix. For fixed head dimension and a fixed retained sparse context, the per-step attention-score computation scales as: (20) ⁡(Lp+|t|+|It(ℓ)|),O (L_p+|A_t|+|I_t^( )| ), compared with ⁡(t)O(t) for full causal self-attention. Thus, the attention cost depends on the fixed prefix length, active slot budget, and retained sparse self-context size rather than directly on the full generated history. 3.7. Corrective Retrieval and Evidence-Bound Revision To mitigate failures from imperfect retrieval, GraphLoom uses a lightweight evidence-adequacy evaluator that can trigger budgeted corrective retrieval, followed by a verification–revision pass over the generated answer. The evaluator computes: (21) Eeval=σ⁡(MLP⁡(ggraph⊕gquery⊕ghistory)),E_eval=σ\! (MLP (g_graph g_query g_history ) ), where Eeval∈(0,1)E_eval∈(0,1), ggraphg_graph summarizes subgraph statistics, gqueryg_query captures question difficulty, and ghistoryg_history encodes prior retrieval actions. If Eeval<τevalE_eval< _eval, GraphLoom executes at most one corrective action. We consider three bounded actions: (i) Expansion Boost, which temporarily increases NmaxN_ by a fixed validation-selected ratio and reruns bounded expansion; (i) Query Reformulation, which generates a concise clarified query conditioned on Q and sub(Q)G_sub(Q); and (i) Verbalization Repair, which generates alternative verbalizations for low-confidence triples and re-scores them with the multimodal embedder. The action selector trades off expected utility against the remaining evidence budget. No iterative correction loop is used. After answer generation, a verification–revision stage enforces evidence faithfulness. The draft answer AdraftA_draft is decomposed into atomic claims ci\c_i\. Each claim is checked using a DeBERTa-v3 NLI verifier (10) against a textualized evidence context composed of the retrieved subgraph, grounded scene description, and available textual evidence: (22) ℰrev=Textualize⁡(sub(Q))⊕D⊕Ctext,ei=NLIrev(ci,ℰrev),E_rev=Textualize\! (G_sub(Q) ) D C_text, e_i=NLI_rev (c_i,E_rev ), where CtextC_text denotes available textualized input evidence, including serialized table context when applicable. Claims with ei<τente_i< _ent trigger retrieval of the top relevant triples from MMKG+G^+_MMKG, followed by a constrained revision prompt to the frozen decoder. The prompt allows edits only to unsupported spans and preserves supported spans whenever possible. Importantly, the revision-time verifier is not used as the sole final faithfulness evaluator. The experimental section reports final faithfulness using independent verification and human validation, preventing the H-Rate results from being tied only to the same NLI model used inside revision. 3.8. Answer Prediction Multiple-choice QA. Given answer options o(j)j=1n\o^(j)\_j=1^n, each option is scored by its length-normalized log-likelihood under the frozen decoder conditioned on the memory context ℳM: (23) S⁡(o(j))=1|o(j)|∑i=1|o(j)|log⁡P⁡(yi(j)∣y<i(j),ℳ).S(o^(j))= 1|o^(j)| _i=1^|o^(j)| P\! (y_i^(j) y_<i^(j),M ). The predicted answer is computed as: (24) y^=arg⁡maxj⁡S⁡(o(j)). y= _jS(o^(j)). Open-ended QA. For free-form generation, we use autoregressive decoding conditioned on ℳM. To keep evaluation reproducible, decoding settings are fixed across all controlled comparisons. If sampling is used in any setting, the random seed and number of runs are fixed consistently. Evidence insufficiency. GraphLoom includes an evidence-safety fallback that can emit a special [NO_EVIDENCE] token when the retrieved evidence does not satisfy the adequacy requirement. In this case, decoding stops and the system returns “Insufficient evidence.” To avoid treating abstention as a source of artificial performance gain, such outputs are counted as incorrect for answer correctness, while their occurrence is tracked separately from claim-level faithfulness evaluation. 3.9. Computation and Latency Accounting GraphLoom contains both cacheable graph-construction and query-time inference stages. We therefore report three complementary costs. Full-pipeline latency includes scene description generation, triple extraction, entity linking, KG enrichment, retrieval, optional interleaved/corrective retrieval, decoding, verification, and revision. Online latency assumes cached instance-level graph construction and includes only query-conditioned retrieval, slot routing, decoding, and conditional correction/revision. Decoder-time cost reports the active slot count |t||A_t|, prefix length LpL_p, sparse self-context size |It(ℓ)||I_t^( )|, and the resulting per-step attention cost. This separation distinguishes cacheable preprocessing from the cost introduced during online evidence routing and generation. 3.10. Training Objective All backbone models, including the Qwen3-VL encoders and the frozen Llama-3.1-8B-Instruct decoder, remain fixed during training. The main end-to-end training objective optimizes the lightweight multimodal fusion projection, HieraSlot projections, and reliability router. It combines answer generation, router calibration, contrastive slot utility, and router–attention alignment: (25) ℒ=λgenℒgen+λcalℒcal+λcfℒcf+λalignℒalign.L= _genL_gen+ _calL_cal+ _cfL_cf+ _alignL_align. The validation-selected weights are λgen,λcal,λcf,λalign=1.0,0.5,0.3,0.2\ _gen, _cal, _cf, _align\=\1.0,0.5,0.3,0.2\ and are fixed for test evaluation. ℒgenL_gen is a label-smoothed negative log-likelihood loss over the gold answer sequence conditioned on the memory context ℳM produced by the multimodal prefix and routed graph slots. For multiple-choice QA, it is applied to the gold option; for open-ended QA, it is applied to the reference answer. ℒcalL_cal calibrates router confidence c^j c_j using binary cross-entropy against a binary utility target zjz_j, indicating whether a slot provides supporting evidence for the target answer. To further separate useful and misleading slots, ℒcfL_cf applies a margin loss to positive–negative slot pairs: (26) ℒcf=1||∑(j+,j−)∈max⁡(0,m−ρj++ρj−),L_cf= 1|P| _(j^+,j^-) \! (0,\,m- _j^++ _j^- ), where m is a validation-selected margin and P contains useful–misleading slot pairs. Finally, ℒalignL_align aligns the router distribution with the decoder attention assigned to the selected evidence slots: (27) ℒalign=DKL(proute(⋅)∥pattn(⋅)).L_align=D_KL (p_route(·)\,\|\,p_attn(·) ). Here, pattnp_attn is obtained by averaging attention to selected slots across decoder heads, layers, and answer-token steps and renormalizing over the same slot set as proutep_route. This encourages selected evidence to align with evidence actually used during generation without using the final faithfulness evaluator as a training signal. The trainable modules are optimized with AdamW, linear warmup, cosine decay, and gradient clipping at 1.0. Early stopping uses Sval=0.7Sans+0.3SfaithS_val=0.7S_ans+0.3S_faith, where SansS_ans is the validation answer score and SfaithS_faith is a validation-only grounding score. All loss weights, thresholds, and routing/retrieval budgets are selected on validation data and fixed before test evaluation. 4. Experimental Evaluation We evaluate GraphLoom along six axes: (i) answer correctness, (i) retrieval quality under strict budgets, (i) independently assessed faithfulness, (iv) scalability under noisy candidate pools, (v) robustness to threshold variation, and (vi) efficiency with separate full-pipeline and online latency accounting. 4.1. Datasets and Evaluation Protocol Datasets. We evaluate on three complementary multimodal QA benchmarks. ScienceQA (24) is a multiple-choice benchmark with image and context inputs (4,241 test instances) and no gold supporting evidence. OK-VQA (26) is an open-ended benchmark requiring external knowledge, with 5,046 test instances. MultiModalQA (MMQA) (33) requires reasoning across text, tables, and images. Because answers and supporting-context annotations for the public MMQA test split are unavailable, we report MMQA answer and retrieval results on the official labeled development split (2,441 instances). Standard benchmark splits are otherwise retained. Answer correctness. We report Accuracy on ScienceQA, standard VQA accuracy on OK-VQA, and Exact Match (EM) and token-level F1 on MMQA using the corresponding evaluation scripts. Retrieval quality. On the labeled MMQA split, we evaluate the retrieved subgraph subG_sub using Precision@5, Recall@5, and MRR against gold supporting evidence. Semantic alignment between retrieved triples and gold evidence is measured with an evaluator not used during answer revision. We also evaluate retrieval under larger candidate pools containing distractor triples, as described in Section 4.5. Faithfulness. We assess grounding using Hallucination Rate (H-Rate) following atomic-fact verification (27). Generated answers are decomposed into atomic claims and checked against retrieved and input evidence. Because GraphLoom uses an NLI verifier during revision, final H-Rate is evaluated with an independent automatic verifier and complementary human validation. Lower H-Rate indicates fewer unsupported claims. Efficiency. We report decoder-time cost through the retrieved-edge count |ℰsub||E_sub|, active HieraSlot count |t||A_t|, sparse self-KV size |It(ℓ)||I_t^( )|, and per-step attention-FLOP reduction. Online latency assumes cached graph construction and includes query-conditioned retrieval, slot routing, decoding, and conditional correction/revision. Full-pipeline latency additionally includes scene description generation, triple extraction, entity linking, and KG enrichment. 4.2. Baselines We compare GraphLoom against retrieval-free VLMs, multimodal RAG/KG-QA systems, retrieval-control and revision frameworks, and targeted ablations. Vision–Language Models (VLMs). We include LLaVA-1.5 (22) as a widely used open-source baseline, together with recent models Qwen2.5-VL (5), InternVL3 (42), and LLaVA-OneVision-1.5 (3). Qwen3-VL-Instruct (4), which is also used in GraphLoom’s graph-construction pipeline, is evaluated as an additional open-source baseline. We further include GPT-5 (28) and Gemini 2.5 Pro (9) as closed-source reference models; because their APIs may change over time, they are not treated as controlled baselines. Multimodal RAG/KG-QA systems. We compare with MuRAG (6) and SKURG (39) to assess the benefit of instance-level MMKG construction and compact slot-based evidence injection over document-level or prebuilt-graph retrieval. Retrieval-control / revision frameworks. We re-implement IRCoT-style interleaved retrieval and the corrective trigger from CRAG (35; 38) using the same retriever and decoder backbone. These variants use Qwen3-VL-Embedding for retrieval and Llama-3.1-8B-Instruct for decoding, but exclude the reliability-calibrated router and KG-JSA++. For the faithfulness comparison, we additionally include RARR (8) as a revision-based baseline. This isolates the contribution of GraphLoom’s controlled evidence-routing and revision mechanisms. Ablations. Flat-RAG injects retrieved evidence as plain text using the same retriever and evidence sources, measuring the effect of structured slot memories. CLIP-RAG replaces Qwen3-VL-Embedding with CLIP-ViT-L/14 (29) while keeping the remaining pipeline unchanged, isolating gains beyond retriever choice. Additional ablations remove interleaved retrieval, corrective retrieval, post-hoc revision, reliability-calibrated routing, and KG-JSA++. 4.3. Overall Results Tables 1 and 2 summarize QA performance on ScienceQA, MMQA, and OK-VQA. All controlled open-source methods and GraphLoom variants use the same evaluation splits, prompts, and protocol, with mean ± std reported over five runs. Closed-source API models are reported only as reference comparisons. Comparative Analysis. GraphLoom achieves 92.5% accuracy on ScienceQA, 55.2 EM / 66.5 F1 on MMQA, and 70.5% VQA accuracy on OK-VQA. The gain is modest on ScienceQA, where recent VLMs already perform strongly, but larger on MMQA and OK-VQA, which require external knowledge, multimodal evidence alignment, or multi-hop reasoning. These results support the benefit of explicit graph retrieval and controlled evidence injection for knowledge-intensive multimodal QA. Table 1. QA results on ScienceQA and MMQA. Bold indicates the best controlled open-source result; closed-source models are reference-only comparisons. Model ScienceQA Acc ↑ MMQA EM ↑ MMQA F1 ↑ LLaVA-1.5 (7B) (22) 89.3 ± 0.2 44.8 ± 0.3 54.9 ± 0.4 Qwen2.5-VL (5) 91.2 ± 0.2 50.6 ± 0.4 61.9 ± 0.4 InternVL3 (42) 91.8 ± 0.2 51.4 ± 0.4 62.8 ± 0.4 LLaVA-OneVision-1.5 (3) 91.5 ± 0.3 50.9 ± 0.5 62.3 ± 0.4 Qwen3-VL-Instruct (2B) (4) 89.5 ± 0.2 46.4 ± 0.4 55.9 ± 0.3 MuRAG (6) 87.9 ± 0.3 47.8 ± 0.5 57.8 ± 0.4 SKURG (39) 88.4 ± 0.3 48.3 ± 0.4 58.3 ± 0.5 IRCoT (35) 89.7 ± 0.2 49.3 ± 0.4 59.3 ± 0.3 CRAG (38) 89.4 ± 0.3 48.9 ± 0.5 58.9 ± 0.4 Closed-Source Reference Models GPT-5 (28) 92.8 54.2 65.7 Gemini 2.5 Pro (9) 93.1 54.8 66.3 GraphLoom Ablations Flat-RAG 90.1 ± 0.3 49.8 ± 0.4 61.2 ± 0.4 CLIP-RAG 89.8 ± 0.2 49.0 ± 0.5 60.5 ± 0.5 GraphLoom 92.5 ± 0.2 55.2 ± 0.4 66.5 ± 0.3 Table 2. Results on OK-VQA. GraphLoom achieves the strongest controlled open-source score and remains competitive with closed-source reference models. Model VQA Accuracy ↑ LLaVA-1.5 (7B) (22) 61.2 ± 0.4 Qwen2.5-VL (5) 66.2 ± 0.4 InternVL3 (42) 67.4 ± 0.3 LLaVA-OneVision-1.5 (3) 66.8 ± 0.4 Qwen3-VL-Instruct (2B) (4) 63.8 ± 0.3 MuRAG (6) 59.5 ± 0.5 SKURG (39) 62.1 ± 0.4 IRCoT (35) 64.3 ± 0.4 CRAG (38) 63.9 ± 0.5 GPT-5 (28) 68.7 Gemini 2.5 Pro (9) 69.5 GraphLoom 70.5 ± 0.3 Compared with IRCoT and CRAG, GraphLoom achieves higher MMQA scores, suggesting that reliability-calibrated routing and KG-JSA++ provide benefits beyond interleaved or corrective retrieval alone. Flat-RAG and CLIP-RAG also perform below GraphLoom, indicating that the gains are associated not only with retrieving evidence but also with organizing it into structured slots and selectively routing graph memories during decoding. GraphLoom also remains competitive with GPT-5 and Gemini 2.5 Pro under our evaluation protocol, although these closed-source systems are treated only as reference comparisons. Faithfulness. Detailed faithfulness results are reported in Section 4.7 using independent automatic H-Rate and complementary human validation. GraphLoom achieves the lowest unsupported-claim rate among the evaluated systems, consistent with the intended roles of reliability-calibrated routing and evidence-bound revision. 4.4. Ablation Studies We ablate the main GraphLoom components on MMQA to determine whether the gains arise from retrieval alone or from the proposed evidence-routing design. The ablations remove interleaved retrieval, corrective retrieval, post-hoc revision, reliability-calibrated routing, and KG-JSA++; we also analyze routing features, external knowledge sources, and router calibration. Table 3 shows that all components contribute to performance, with the largest drops occurring when the reliability-calibrated router or KG-JSA++ is removed. Without the router, H-Rate increases from 10.5 to 17.3, consistent with reduced suppression of low-confidence or weakly relevant evidence. Removing KG-JSA++ causes the largest degradation in both answer quality and faithfulness, indicating the importance of jointly attending to routed graph memories and decoder context. Overall, the results suggest that GraphLoom benefits not only from retrieving evidence, but also from controlling how that evidence is selected and injected during decoding. Table 3. Ablation on MMQA. Removing key components lowers EM/F1 and increases H-Rate, highlighting the importance of controlled evidence routing and joint graph–sequence attention. Variant EM ↑ F1 ↑ H-Rate ↓ Full GraphLoom 55.2 ± 0.4 66.5 ± 0.3 10.5 ± 0.4 w/o Interleaved Retrieval 53.6 ± 0.5 64.6 ± 0.4 11.1 ± 0.5 w/o Corrective Retrieval 54.1 ± 0.4 65.0 ± 0.5 11.8 ± 0.6 w/o Post-hoc Revision 54.7 ± 0.4 65.7 ± 0.4 13.9 ± 0.5 w/o Reliability-Calibrated Router 51.8 ± 0.6 61.6 ± 0.5 17.3 ± 0.7 w/o KG-JSA++ 50.5 ± 0.7 60.3 ± 0.6 18.9 ± 0.8 Interleaved and corrective retrieval provide smaller but consistent gains. Removing interleaved retrieval mainly reduces EM/F1, consistent with the additional hypothesis-guided retrieval round recovering useful multi-hop connectors. Corrective retrieval has a larger effect on H-Rate, while post-hoc revision produces only modest EM/F1 changes but substantially reduces unsupported claims, consistent with its role as an evidence-bound correction step. Fine-grained ablations. Removing the retrieval score (-ret-score) produces the largest single-feature degradation (EM −1.6-1.6, H-Rate +2.8+2.8), indicating that query–evidence alignment is a particularly important routing signal. Removing contextual features (-context) also reduces performance (EM −1.2-1.2, H-Rate +2.1+2.1), while extractor confidence (-confext) and centrality (-centrality) have smaller but consistent effects. These results support combining semantic relevance, evidence reliability, and graph-structural cues rather than relying on a single signal. Knowledge sources. VG150 and ConceptNet provide complementary external evidence. Removing VG150 increases H-Rate by 0.8 points, consistent with its contribution to visual-relation grounding. Removing ConceptNet reduces EM by 1.3 points, with larger effects on questions requiring commonsense links beyond directly visible content. Removing both sources (No-KG) yields the largest degradation, reducing EM by 2.4 points and increasing H-Rate by 4.8 points. This indicates that external knowledge contributes to performance when combined with bounded retrieval and reliability-aware routing. Router calibration. Figure 2 shows that the calibrated router is better aligned with empirical slot utility than the uncalibrated variant, reducing ECE from 0.087 to 0.032. This supports the reliability-calibration objective and is consistent with the improved faithfulness observed in Table 3. 000.20.20.40.40.60.60.80.811000.20.20.40.40.60.60.80.811Lower ECE indicatesmore reliable slot selectionPredicted confidenceObserved slot utilityPerfect calibrationCalibrated router, ECE = 0.032Uncalibrated router, ECE = 0.087 Figure 2. Reliability diagram for slot routing. The calibrated router lies closer to the perfect-calibration diagonal, reducing ECE from 0.087 to 0.032. 4.5. Scalability under Noisy Evidence Pools To evaluate GraphLoom beyond compact instance-level graphs, we test retrieval under progressively larger candidate pools. For each MMQA instance, the enriched instance graph is mixed with distractor triples sampled from other MMQA instances and external KG triples unrelated to the current question. Triple embeddings are precomputed and indexed, so only the candidate-pool size changes; the scoring function and bounded-expansion mechanism remain fixed. Table 4. Scalability under noisy candidate evidence pools on MMQA. GraphLoom degrades gradually as the retrieval space grows to one million distractor triples. Candidate pool Recall@5 ↑ MRR ↑ MMQA F1 ↑ Online latency (ms) ↓ Instance graph only 78.4 0.742 66.5 119 +10K distractor triples 77.6 0.734 66.2 124 +100K distractor triples 75.9 0.713 65.8 132 +1M distractor triples 73.2 0.684 65.1 148 Table 4 shows gradual degradation as the candidate pool grows. Moving from the instance graph to one million distractor triples reduces Recall@5 by 5.2 points and MRR by 0.058, while MMQA F1 drops by only 1.4 points. This suggests that bounded expansion and reliability-aware slot selection help limit the effect of noisy candidates on answer generation. Online latency increases from 119 ms to 148 ms as retrieval operates over a larger indexed pool. However, the decoder-time budget remains fixed through NmaxN_ and the active-slot limit k, so generation cost does not scale with candidate-pool size. These results indicate that GraphLoom remains effective under substantially larger noisy retrieval spaces. 4.6. Threshold Robustness To assess sensitivity to threshold selection, we perturb the main evidence-selection and verification thresholds by ±20%± 20\% around their validation-selected values while keeping all other settings fixed. This evaluates the stability of confidence filtering, KG enrichment, corrective retrieval, and evidence verification. Table 5. Robustness under ±20%± 20\% threshold perturbations on MMQA. Performance remains stable under moderate changes to the main evidence-selection and verification thresholds. Setting MMQA EM ↑ MMQA F1 ↑ H-Rate ↓ Default 55.2 66.5 10.5 τedge−20% _edge-20\% 54.8 66.1 11.2 τedge+20% _edge+20\% 54.5 65.8 10.9 τKB−20% _KB-20\% 54.7 66.0 11.3 τKB+20% _KB+20\% 54.4 65.7 11.0 τeval−20% _eval-20\% 54.6 65.9 11.8 τeval+20% _eval+20\% 54.9 66.2 10.8 τent−20% _ent-20\% 54.9 66.0 12.2 τent+20% _ent+20\% 54.6 65.7 10.3 Table 5 shows that MMQA EM and F1 remain within 0.8 points of the default setting across all perturbations, indicating limited sensitivity to moderate threshold changes. Lowering τedge _edge or τKB _KB admits more evidence but also increases H-Rate slightly, whereas higher values reduce noise at the cost of some useful evidence. Reducing τeval _eval triggers fewer corrective actions and increases H-Rate, while increasing it produces a modest faithfulness improvement. Similarly, a higher τent _ent makes revision stricter, lowering H-Rate but slightly reducing answer quality. Overall, the default setting provides the best accuracy–faithfulness balance. 4.7. Quantitative Hallucination Assessment We assess unsupported assertions using complementary automatic and human evaluation, with the same prompts, decoding settings, and claim-splitting procedure across models. Independent automatic H-Rate. H-Rate is the fraction of atomic factual claims unsupported by the retrieved subgraph subG_sub or textualized input evidence. Because GraphLoom uses DeBERTa-v3 only during verification–revision, final H-Rate is computed with MiniCheck (34), which is not used during training or revision. Each claim is paired with an evidence context containing verbalized triples, retrieved facts, and the grounded scene description. We evaluate 1,500 examples, with 500 sampled from each dataset, and report mean ± std across runs. Human validation. We additionally evaluate 300 randomly sampled outputs, with 100 examples from each dataset. Three NLP researchers independently assess each output using an evidence-grounding rubric and majority voting. Visual claims are considered supported only when the corresponding visual or retrieved evidence is explicit. Thus, the human evaluation complements MiniCheck by directly assessing visual as well as textualized grounding. Error categories. Residual hallucinations mainly involve fabricated relations, where no supporting edge exists in subG_sub, and mislocalized entities, where a plausible predicate is attached to the wrong entity. Table 6. Hallucination assessment. Automatic H-Rate is computed with MiniCheck, independently of the revision-time verifier; human evaluation uses 300 balanced examples. Model Independent H-Rate (%) ↓ Human Hallucination-Free (%) ↑ LLaVA-1.5 (22) 25.2 ± 0.8 39.5 ± 1.2 Qwen2.5-VL (5) 17.6 ± 0.6 56.2 ± 1.1 InternVL3 (42) 16.8 ± 0.5 57.9 ± 1.0 SKURG (39) 20.3 ± 0.7 49.8 ± 1.1 CRAG (38) 18.1 ± 0.6 54.7 ± 1.3 RARR (8) 15.2 ± 0.5 59.8 ± 1.0 Flat-RAG 14.8 ± 0.6 61.0 ± 1.1 GraphLoom 10.5 ± 0.4 68.5 ± 0.9 Overall, GraphLoom achieves a lower unsupported-claim rate than the evaluated VLM, graph-RAG, corrective-retrieval, and revision baselines. Its improvement over Flat-RAG is consistent with contributions from reliability-calibrated slot routing and evidence-bound revision. Using an automatic evaluator independent of the revision-time verifier, together with human validation, reduces the risk of circular faithfulness evaluation. 4.8. Implementation Details and Hyperparameter Analysis GraphLoom is implemented in PyTorch with Hugging Face Transformers. All backbone models—Qwen3-VL-Instruct, Qwen3-VL-Embedding, and Llama-3.1-8B-Instruct—remain frozen during training. The end-to-end decoder-side trainable modules are the multimodal fusion projection (∼ 0.8M parameters), HieraSlot projections (∼ 0.3M), and reliability router (∼ 0.5M), totaling approximately 1.6M parameters. The code is available at11 1 https://github.com/Zafar-southeast/GraphLoom. Training configuration. We train with AdamW (β1=0.9 _1=0.9, β2=0.95 _2=0.95, weight decay =0.01=0.01). The learning rate warms up for 1,000 steps to 3×10−43× 10^-4 and then follows cosine decay to 1×10−61× 10^-6. We use batch size 64, gradient accumulation 2, gradient clipping at 1.0, and early stopping on the validation score defined in Section 3.10. Training uses one NVIDIA A100 80GB GPU. Hyperparameters. Validation tuning selects K=15K=15, two retrieval rounds, dmax=5d_ =5, Nmax=50N_ =50, k=4k=4, Lp=32L_p=32, Lrecent=64L_recent=64, τedge=0.5 _edge=0.5, τKB=0.7 _KB=0.7, τeval=0.6 _eval=0.6, and τent=0.7 _ent=0.7. All values are fixed during testing. Performance is most sensitive to k and NmaxN_ , which control the trade-off between evidence coverage and computation. 11223344556648485050525254545656Active slots kkEM (%)101015152020H-Rate (%) (a) Effect of active slots k on MMQA. 20203535505070706060626264646666Edge budget NmaxN_ F1 (%)8080100100120120140140160160Latency (ms) (b) Effect of edge budget NmaxN_ on MMQA. Figure 3. Sensitivity analysis of active slots and edge budget. The selected defaults (k=4k=4, Nmax=50N_ =50) balance answer quality and computation. 4.9. Latency and Efficiency Breakdown We report full-pipeline and online latency to separate cacheable graph construction from query-time evidence use. Full-pipeline latency includes scene description generation, triple extraction, entity linking, KG enrichment, initial and optional interleaved/corrective retrieval, decoding, verification, and revision. Online latency assumes cached instance-level graph construction and includes query-conditioned retrieval, slot routing, decoding, and conditional correction/revision. Conditional modules are reported as amortized per-query costs over the evaluation set, with non-triggered cases contributing zero additional cost. At inference, GraphLoom retrieves 38.2 triples per query on average, activates at most |t|≤4|A_t|≤ 4 graph slots per decoding step, and retains |It(ℓ)|≤88|I_t^( )|≤ 88 sparse self-KV tokens. At t=512t=512, this reduces per-step attention FLOPs by approximately 76% relative to full causal attention. Table 7 reports the module-level latency breakdown. Table 7. Latency breakdown of GraphLoom on one NVIDIA A100 80GB GPU. Conditional modules are reported as amortized per-query costs. Module Latency / query (ms) ↓ Scene description generation 142 Triple extraction 44 Entity linking + KG enrichment 18 Initial retrieval 9 Interleaved retrieval 13 Corrective retrieval 7 Slot routing + KG-JSA++ decoding 66 Verification–revision 24 Full-pipeline latency 323 Online latency with cached graph 119 Most full-pipeline cost comes from cacheable graph construction, particularly scene description generation and triple extraction. With cached instance-level graphs, online latency is 119 ms per query, dominated by slot routing, KG-JSA++ decoding, and verification–revision. This separation clarifies the cost of online evidence use while showing that decoder-time attention remains controlled by the active-slot and sparse self-KV budgets. 5. Conclusion and Future Work We presented GraphLoom, a reliability-calibrated multimodal KG-RAG framework that retrieves compact evidence subgraphs and injects selected graph memories into a frozen decoder through HieraSlot and KG-JSA++. Experiments on ScienceQA, MultiModalQA, and OK-VQA show consistent gains in answer quality and faithfulness, with additional improvements in MMQA retrieval. Scalability, robustness, and latency analyses further demonstrate stable performance under noisy evidence pools and moderate threshold variation. These results suggest that reliability-aware evidence routing can provide an effective alternative to larger models or longer contexts. Future work will explore adaptive retrieval, stronger multimodal indexing, and extensions to document, chart, table, and video QA. GenAI Usage Disclosure Generative AI components used within GraphLoom are described in the methodology and experimental setup. OpenAI ChatGPT was additionally used to assist with code generation, debugging, experimental output preparation, LaTeX, language refinement, and consistency checking. All generated code and reported outputs were reviewed and validated by the authors, who made the methodological and technical decisions and take full responsibility for the final work. Acknowledgements. This work was supported by the NSFC under Grant No. 6509009704. References Alayrac et al. (2022) J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, Cited by: §2. Ali et al. (2026) Z. Ali, Y. Huang, A. Khan, G. Qi, Y. Zhang, J. Feng, C. Deng, and P. Kefalas Pythia-rag: retrieval-augmented generation over a unified multimodal knowledge graph for enhanced qa. Knowledge-Based Systems 335, p. 115200. External Links: Document Cited by: §2. An et al. (2025) X. An, Y. Xie, K. Yang, W. Zhang, X. Zhao, Z. Cheng, Y. Wang, S. Xu, C. Chen, D. Zhu, C. Wu, H. Tan, C. Li, J. Yang, J. Yu, X. Wang, B. Qin, Y. Wang, Z. Yan, Z. Feng, Z. Liu, B. Li, and J. Deng LLaVA-OneVision-1.5: fully open framework for democratized multimodal training. External Links: 2509.23661 Cited by: §4.2, Table 1, Table 2. Bai et al. (2025) S. Bai et al. Qwen3-VL technical report. External Links: 2511.21631 Cited by: §1, §2, §3.1, §4.2, Table 1, Table 2. Bai et al. (2025) S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin Qwen2.5-VL technical report. External Links: 2502.13923 Cited by: §4.2, Table 1, Table 2, Table 6. Chen et al. (2022) W. Chen, H. Hu, X. Chen, P. Verga, and W. Cohen MuRAG: multimodal retrieval-augmented generator for open question answering over images and text. In EMNLP, p. 5558–5570. Cited by: §1, §4.2, Table 1, Table 2. Dai et al. (2023) W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi InstructBLIP: towards general-purpose vision-language models with instruction tuning. In NeurIPS, Cited by: §2. Gao et al. (2023) L. Gao, Z. Dai, P. Pasupat, A. Chen, et al. RARR: researching and revising what language models say, using language models. External Links: 2210.08726 Cited by: §1, §2, §2, §4.2, Table 6. Google (2025) Google Gemini 2.5: our most intelligent AI model. Note: Google Blog Cited by: §2, §4.2, Table 1, Table 2. He et al. (2021) P. He, J. Gao, and W. Chen DeBERTaV3: improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. External Links: 2111.09543 Cited by: §3.7. He et al. (2024) X. He, Y. Tian, Y. Sun, N. V. Chawla, T. Laurent, Y. LeCun, X. Bresson, and B. Hooi G-Retriever: retrieval-augmented generation for textual graph understanding and question answering. In NeurIPS 37, Cited by: §1, §1, §2. Huguet Cabot and Navigli (2021) P. Huguet Cabot and R. Navigli REBEL: relation extraction by end-to-end language generation. In Findings of ACL: EMNLP, p. 2370–2381. Cited by: §1, §3.1. Jiang et al. (2023) J. Jiang, K. Zhou, X. Zhao, and J.-R. Wen UniKGQA: unified retrieval and reasoning for solving multi-hop question answering over knowledge graphs. In ICLR, Cited by: §1, §1. Khan et al. (2025) A. Khan, Z. Ali, Irfanullah, A. Aziz, and P. Kefalas Talk2Doc: a patient q&a system using retrieval-augmented generation with weighted knowledge graphs and llms. In Proceedings of the 21st International Conference on Intelligent Computing (ICIC 2025), Ningbo, China, p. 595–610. External Links: Document Cited by: §2. Krishna et al. (2017) R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L. Li, D. A. Shamma, M. S. Bernstein, and L. Fei-Fei Visual genome: connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123, p. 32–73. External Links: Document Cited by: §1. Lan et al. (2023) Y. Lan, S. Wang, Y. Yin, W. Wu, H. Wang, and W. Lu Complex knowledge base question answering: a survey. IEEE Trans. Knowl. Data Eng. 35, p. 11196–11215. Cited by: §1. Lee et al. (2024) J. Lee, Y. Wang, J. Li, and M. Zhang Multimodal reasoning with multimodal knowledge graph. In ACL, p. 10767–10782. Cited by: §2. Leng et al. (2024) S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In CVPR, p. 13872–13882. Cited by: §1. Li et al. (2023) J. Li, D. Li, S. Savarese, and S. Hoi BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, Cited by: §1, §2. Li et al. (2026) M. Li, Y. Zhang, D. Long, K. Chen, S. Song, S. Bai, Z. Yang, P. Xie, A. Yang, D. Liu, J. Zhou, and J. Lin Qwen3-vl-embedding and qwen3-vl-reranker: a unified framework for state-of-the-art multimodal retrieval and ranking. arXiv preprint arXiv:2601.04720. Cited by: §1, §3.2. Lin et al. (2026) J. Lin, C. Jiang, X. Lin, R. Zhang, X. Zhu, J. Liu, C. Tang, Y. Du, S. Gao, J. Ning, L. Liu, Z. Huang, T. Li, J. Ye, and J. He EvoGraph-r1: self-evolving multimodal knowledge hypergraphs for agentic retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 756–765. Cited by: §2. Liu et al. (2023) H. Liu, C. Li, Q. Wu, and Y. J. Lee Improved baselines with visual instruction tuning. External Links: 2310.03744 Cited by: §1, §2, §4.2, Table 1, Table 2, Table 6. Liu et al. (2025) J. Liu, S. Meng, Y. Gao, S. Mao, P. Cai, G. Yan, Y. Chen, Z. Bian, B. Shi, and D. Wang Aligning vision to language: text-free multimodal knowledge graph construction for enhanced LLM reasoning. External Links: 2503.12972 Cited by: §2. Lu et al. (2022) P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S. Zhu, O. Tafjord, P. Clark, and A. Kalyan Learn to explain: multimodal reasoning via thought chains for science question answering. In NeurIPS, Cited by: §4.1. Luo et al. (2026) W. Luo, Z. Hu, X. Wang, Z. Yu, H. Zhang, and Z. Huang Utility-oriented visual evidence selection for multimodal retrieval-augmented generation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), San Diego, California, United States, p. 35091–35124. External Links: Document Cited by: §2. Marino et al. (2019) K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi OK-VQA: a visual question answering benchmark requiring external knowledge. In CVPR, Cited by: §4.1. Min et al. (2023) S. Min, K. Krishna, X. Lyu, M. Lewis, W.-T. Yih, P. W. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi FActScore: fine-grained atomic evaluation of factual precision in long-form text generation. In EMNLP, p. 12076–12100. Cited by: §2, §4.1. OpenAI (2025) OpenAI Introducing GPT-5. Note: OpenAI Blog Cited by: §2, §4.2, Table 1, Table 2. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML, p. 8748–8763. Cited by: §4.2. Shin et al. (2026) J. Shin, G. Shim, J. Park, J. Seo, and H. Lim HiKEY: hierarchical multimodal retrieval for open-domain document question answering. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), San Diego, California, United States, p. 17967–17987. External Links: Document Cited by: §2. Speer et al. (2017) R. Speer, J. Chin, and C. Havasi ConceptNet 5.5: an open multilingual graph of general knowledge. In AAAI 31, Cited by: §1, §3.1. Suri et al. (2025) M. Suri, P. Mathur, F. Dernoncourt, K. Goswami, R. Rossi, and D. Manocha VisDoM: multi-document QA with visually rich elements using multimodal retrieval-augmented generation. In NAACL, p. 6088–6109. Cited by: §2. Talmor et al. (2021) A. Talmor, O. Yoran, A. Catav, D. Lahav, Y. Wang, A. Asai, G. Ilharco, H. Hajishirzi, and J. Berant MultiModalQA: complex question answering over text, tables and images. In ICLR, Cited by: §4.1. Tang et al. (2024) L. Tang, P. Laban, and G. Durrett MiniCheck: efficient fact-checking of LLMs on grounding documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 8818–8847. Cited by: §2, §4.7. Trivedi et al. (2023) H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. External Links: 2212.10509 Cited by: §1, §2, §4.2, Table 1, Table 2. Xiao et al. (2025) E. Xiao, C. Li, Y. Zhang, G. Neubig, and A. Bertsch Efficient many-shot in-context learning with dynamic block-sparse attention. In ACL, Cited by: §1, §2. Xiao et al. (2023) G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis Efficient streaming language models with attention sinks. External Links: 2309.17453 Cited by: §1, §2, §3.6. Yan et al. (2024) S. Yan, Z. Zhu, X. Cheng, B. Wang, P. Zhao, et al. Corrective retrieval augmented generation. External Links: 2401.15884 Cited by: §1, §2, §4.2, Table 1, Table 2, Table 6. Yang et al. (2023) Q. Yang, Q. Chen, W. Wang, B. Hu, and M. Zhang Enhancing multi-modal multi-hop question answering via structured knowledge and unified retrieval-generation. In ACM M, p. 5223–5234. Cited by: §2, §4.2, Table 1, Table 2, Table 6. Zheng et al. (2023) S. Zheng, W. Wang, J. Qu, H. Yin, W. Chen, and L. Zhao MMKGR: multi-hop multi-modal knowledge graph reasoning. In ICDE, p. 96–109. Cited by: §1, §2. Zhu et al. (2023) D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny MiniGPT-4: enhancing vision-language understanding with advanced large language models. External Links: 2304.10592 Cited by: §2. Zhu et al. (2025) J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y. Duan, W. Su, J. Shao, Z. Gao, E. Cui, X. Wang, Y. Cao, Y. Liu, X. Wei, H. Zhang, H. Wang, W. Xu, H. Li, J. Wang, N. Deng, S. Li, Y. He, T. Jiang, J. Luo, Y. Wang, C. He, B. Shi, X. Zhang, W. Shao, J. He, Y. Xiong, W. Qu, P. Sun, P. Jiao, H. Lv, L. Wu, K. Zhang, H. Deng, J. Ge, K. Chen, L. Wang, M. Dou, L. Lu, X. Zhu, T. Lu, D. Lin, Y. Qiao, J. Dai, and W. Wang InternVL3: exploring advanced training and test-time recipes for open-source multimodal models. External Links: 2504.10479 Cited by: §4.2, Table 1, Table 2, Table 6.

Instant research discovery