Paper deep dive
GraMRAG: Orchestrating Multi-Agent Multi-Step Reasoning via Graph Memory with Reinforcement Learning
Zhongyu Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/20/2026, 4:16:44 AM
Summary
The paper introduces GraMRAG, a graph memory-guided multi-agent Retrieval-Augmented Generation (RAG) framework designed for complex multimodal reasoning. It addresses limitations in existing systems such as state blindness and inadequate retrieval depth by employing a vision-text bridged reasoning paradigm, a dynamic multimodal memory graph (DAG) to model action-observation dependencies, and Topology-Aware Policy Optimization (TAPO) for fine-grained credit assignment in reinforcement learning.
Entities (10)
Relation Signals (9)
GraMRAG → uses → Multimodal Memory Graph
confidence 95% · we propose GraMRAG... that integrates a dynamic multimodal memory graph
GraMRAG → uses → TAPO
confidence 95% · we propose Topology-Aware Policy Optimization (TAPO)... enabling fine-grained credit assignment
GraMRAG → evaluatedon → SQuAD
confidence 90% · Experiments... SQuAD
GraMRAG → evaluatedon → HotpotQA
confidence 90% · Extensive experiments on challenging multimodal benchmarks... HotpotQA
Multimodal Memory Graph → mitigates → state blindness
confidence 90% · explicitly modeling action-observation dependencies to mitigate state blindness
Evidence Validator → partof → GraMRAG
confidence 90% · the Evidence Validator aggregates and verifies... within the GraMRAG framework
Visual Explorer → partof → GraMRAG
confidence 90% · the Visual Explorer performs hierarchical region proposal generation... within the GraMRAG framework
Text Researcher → partof → GraMRAG
confidence 90% · forwarded to the Text Researcher... within the GraMRAG framework
TAPO → solves →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Although existing multi-agent Retrieval-Augmented Generation (RAG) systems have demonstrated promise on complex multimodal reasoning tasks, they remain fundamentally limited in reasoning depth and memory structure, suffering from inadequate retrieval and state blindness when answering knowledge-intensive questions. To address these limitations, we propose GraMRAG, a graph memory-guided multi-agent RAG framework that integrates a dynamic multimodal memory graph to enable stable, multi-step multimodal reasoning. We introduce a vision-text bridged reasoning paradigm that unifies multi-scale entity cropping with a ReAct-style visual toolchain, enhancing the long-horizon cross-modal reasoning capability. We further construct a multimodal memory graph that formalizes agent reasoning as a dynamic directed acyclic graph (DAG), explicitly modeling action-observation dependencies to mitigate state blindness and suppress redundant retrieval. Moreover, we propose Topology-Aware Policy Optimization (TAPO) that leverages graph topology for critical path identification and targeted node pruning, enabling fine-grained credit assignment across multi-step reasoning trajectories. Extensive experiments on challenging multimodal benchmarks demonstrate that our approach consistently outperforms existing baselines and achieves state-of-the-art performance on complex long-horizon reasoning tasks.
Tags
Links
- Source: https://arxiv.org/abs/2609.14066v1
- Canonical: https://arxiv.org/abs/2609.14066v1
Trouble viewing inline? Open PDF directly →
Full Text
46,338 characters extracted from source content.
Expand or collapse full text
GraMRAG: Orchestrating Multi-Agent Multi-Step Reasoning via Graph Memory with Reinforcement Learning Zhongyu WangBeihang University Email: wangzhongyu@buaa.edu.cn Abstract Although existing multi-agent Retrieval-Augmented Generation (RAG) systems have demonstrated promise on complex multimodal reasoning tasks, they remain fundamentally limited in reasoning depth and memory structure, suffering from inadequate retrieval and state blindness when answering knowledge-intensive questions. To address these limitations, we propose GraMRAG, a graph memory-guided multi-agent RAG framework that integrates a dynamic multimodal memory graph to enable stable, multi-step multimodal reasoning. We introduce a vision-text bridged reasoning paradigm that unifies multi-scale entity cropping with a ReAct-style visual toolchain, enhancing the long-horizon cross-modal reasoning capability. We further construct a multimodal memory graph that formalizes agent reasoning as a dynamic directed acyclic graph (DAG), explicitly modeling action-observation dependencies to mitigate state blindness and suppress redundant retrieval. Moreover, we propose Topology-Aware Policy Optimization (TAPO) that leverages graph topology for critical path identification and targeted node pruning, enabling fine-grained credit assignment across multi-step reasoning trajectories. Extensive experiments on challenging multimodal benchmarks demonstrate that our approach consistently outperforms existing baselines and achieves state-of-the-art performance on complex long-horizon reasoning tasks. 22footnotetext: Corresponding author. 1 Introduction Figure 1: Comparison between our GraMRAG and existing multi-agent RAG systems. “Never memorize something that you can look up.” — Albert Einstein Retrieval-Augmented Generation (RAG) extends the knowledge boundaries of Multimodal Large Language Models (MLLMs) by retrieving relevant information from external knowledge bases Wang (2026); Fan et al. (2026); Hu et al. (2026). Recent advances in MLLMs have convincingly demonstrated that multimodal agentic RAG systems are capable of efficient retrieval and reasoning over large-scale corpora comprising interleaved textual and visual content Zhang et al. (2026); Guan et al. (2025). MLLM-based multi-agent architectures, leveraging specialized collaboration among components, exhibit superior performance over single-agent frameworks on complex multimodal RAG tasks Zhou et al. (2026); Chang et al. (2025); Xu et al. (2025b). However, existing multi-agent RAG systems face three key limitations. First, the average number of retrieval rounds in current systems remains limited, predisposing agents to prematurely terminate exploration and settle for partially available evidence on complex tasks Jiang et al. (2025); Wu et al. (2025a); Team et al. (2025). This fundamentally impedes the transfer of the long-horizon planning capability from text-based deep research to multimodal scenarios, rendering these systems inadequate for tens of iterative reasoning steps demanded by real-world deep research. Second, prevailing linear or iterative summarization-based memory structures Yao et al. (2022); Zhou et al. (2025); Xu et al. (2025a); Wu et al. (2025b), lacking explicit modeling of agent reasoning states, are prone to context expands and state blindness as interaction rounds accumulate, resulting in repetitive queries and redundant retrieval in multi-hop tasks. Third, conventional outcome-based sparse rewards coarsely propagate terminal-state signals across entire trajectories Huang et al. (2026); Zhou et al. (2025), causing effective retrieval steps to be erroneously penalized in negative samples while redundant operations are undeservedly rewarded in positive ones, preventing fine-grained credit assignment across multi-step reasoning. To address the mentioned limitations, we propose GraMRAG, a novel graph memory-guided multi-agent RAG framework. First, to bridge the long-horizon reasoning gap, we introduce a vision-text bridged reasoning paradigm that orchestrates multi-scale entity cropping with a ReAct-style visual toolchain, seamlessly transferring the iterative planning capability of text-only deep research models into the multimodal domain to enable long-horizon cross-modal reasoning trajectories. Second, to overcome state blindness and context expansion in linear memory paradigms, we construct a multimodal memory graph that formalizes the reasoning process as a dynamic directed acyclic graph (DAG), explicitly encoding agent actions and multimodal observations with their temporal and logical dependencies to mitigate redundant retrieval and state blindness. Third, to tackle the credit assignment problem from coarse terminal rewards, we introduce a Topology-Aware Policy Optimization (TAPO) algorithm that leverages the graph topology to identify critical reasoning paths and perform targeted node pruning, precisely masking false positive and false negative nodes during policy updates to enable fine-grained credit assignment across multi-step reasoning. Our contributions are summarized as follows: • We introduce a vision-text bridged reasoning paradigm that unifies multi-scale entity cropping with a ReAct-style visual toolchain, enabling long-horizon cross-modal reasoning. • We construct a multimodal memory graph that formalizes agent reasoning as a dynamic DAG to systematically mitigate state blindness and redundant retrieval. • We develop a TAPO algorithm leveraging graph topology for critical path identification and node pruning, enabling fine-grained credit assignment across multi-step reasoning. • We conduct extensive experiments on challenging multimodal benchmarks to evaluate our approach, demonstrating state-of-the-art performance over existing baselines. 2 Related Work 2.1 Multi-Agent Systems in Multimodal RAG Multi-agent systems have demonstrated promise on complex multimodal reasoning tasks Wang and Liu (2026); Dong et al. (2026); Li et al. (2026); Xinjie et al. (2025); Zhang et al. (2025a). By deploying multiple specialized agents that each focus on distinct facets of a problem, these architectures enable collaborative problem-solving that surpasses the capabilities of single-agent frameworks Ni and Ye (2026); Sinha et al. (2026); Siingh et al. (2025). However, existing multi-agent RAG systems conduct insufficient retrieval rounds, causing agents to prematurely terminate exploration on complex tasks Geng et al. (2025); Narayan et al. (2025). Such shallow reasoning trajectories are fundamentally insufficient for demanding deep research scenarios that require tens of iterative reasoning steps. To address this challenge, we introduce a vision-text bridged reasoning paradigm integrating multi-scale entity cropping with a visual toolchain, using image descriptions as a cross-modal bridge to enable long-horizon planning in the multimodal domain. 2.2 Memory Structures of Agent Systems The memory architectures of existing agent systems can broadly be categorized into two paradigms Zhang et al. (2025b); Ye et al. (2025): the history accumulation paradigm, exemplified by ReAct Yao et al. (2022), which linearly concatenates all interaction records, and the iterative summarization paradigm, exemplified by Mem1 Zhou et al. (2025), which controls context length by compressing observations. However, as the number of interaction rounds grows, both paradigms cause agents to progressively lose track of historical retrieval paths, a phenomenon we term state blindness, which in turn induces redundant queries and repetitive retrieval in multi-hop tasks. To address this issue, we construct a multimodal memory graph that formalizes the reasoning process as a dynamic DAG, explicitly encoding the temporal and logical dependencies between actions and observations, thereby effectively mitigating state blindness and redundant retrieval. Figure 2: Overview of the GraMRAG framework. 3 Method 3.1 Vision-Text Bridged Reasoning Paradigm Multi-Scale Visual Entity Grounding. Given an input image I, a user query q, and a visual induction prompt pvp_v, the Visual Explorer performs hierarchical region proposal generation at each reasoning step t. Rather than submitting the full image as a single retrieval unit, which is prone to failure under background noise interference, the Visual Explorer simultaneously generates coarse-level contextual regions and fine-grained entity-level patches, forming a candidate bounding-box set ℬt=bt1,bt2,…,btNtB_t=\b_t^1,\ b_t^2,\ …,\ b_t^N_t\, where each btib_t^i represents a region crop at a distinct spatial scale or entity granularity, and NtN_t denotes the number of proposals at step t. This candidate set induces a visual retrieval action Atv=VSearch(ℬt)A_t^v=VSearch(B_t), which is processed sequentially by a two-stage visual toolchain: an image search module that matches each cropped region against a web-scale index and returns candidate URLs and a webpage retrieval module that fetches page content in structured markdown format. The observation obtained at step t is denoted: Otv=VisChain(Atv)O_t^v=VisChain(A_t^v) (1) Upon completing the current step, the Visual Explorer applies the induction prompt pvp_v again to generate the next search context, producing the visual reasoning trajectory: τvis=I,q,pv,R1,A1v,O1v,…,pv,RTv,ATvv,OTvv split _vis=\&I,\ q,\ p_v,\ R_1,\ A_1^v,\ O_1^v,\\ &…,\ p_v,\ R_T_v,\ A_T_v^v,\ O_T_v^v\ split (2) where RtR_t denotes the chain-of-thought reasoning generated prior to action AtvA_t^v, and TvT_v is the total number of visual retrieval steps. Following each visual retrieval action, an independent Hit/No Hit Judge evaluates the observation OtvO_t^v obtained at the current step, conditioned on the original image I and query q, outputs a binary acceptance signal: st=Judge(I,q,Otv)∈0,1s_t=Judge(I,\ q,\ O_t^v)∈\0,1\ (3) When st=1s_t=1 (Hit), the retrieval is considered successful, meaning the retrieved content is semantically consistent with the query entity, and OtvO_t^v is added into the accumulated valid evidence pool: ℰv=Otv∣st=1,t=1,…,TvE^v=\O_t^v s_t=1,\ t=1,\ …,\ T_v\ (4) When st=0s_t=0 (No Hit), the retrieval is considered unsuccessful, triggering the Visual Explorer to re-crop at a finer spatial granularity or to reformulate the search query before initiating a new retrieval attempt. The visual exploration phase terminates when |ℰv||E^v| reaches a predefined sufficiency threshold or when the total step count reaches TvT_v. Decoupling the Hit/No Hit judgment from the main agents and assigning it to an independent judge achieves a clean separation between exploration strategy and evidence evaluation, enhancing the robustness of the multi-scale iterative retrieval process. Modality Bridging and Textual Reasoning. To transfer the evidence accumulated during the visual exploration phase into the textual reasoning domain, we introduce a modality bridging step. A captioning module first generates a comprehensive textual description D of the input image I, covering key entities, spatial relationships, and scene-level semantics. The visual trajectory τvis _vis is then transformed into a bridged context: I is replaced by D, all visual induction prompts pvp_v are removed, while all chain-of-thought reasoning RtR_t, actions AtvA_t^v, and judge-verified observations (i.e., entries in ℰvE^v) are preserved. The bridged context is forwarded to the Text Researcher, a deep research LLM equipped with web search tools, which generates the subsequent textual reasoning trajectory: τtext= _text=\ D,q,R1,A1v,O1v,…,RTv,ATvv, D,\ q,\ R_1,\ A_1^v,\ O_1^v,\ …,\ R_T_v,\ A_T_v^v, OTvv,RTv+1,ATv+1t,OTv+1t, O_T_v^v,\ R_T_v+1,\ A_T_v+1^t,\ O_T_v+1^t, …,RTv+Tt,ATv+Ttt …,\ R_T_v+T_t,\ A_T_v+T_t^t\ (5) where TtT_t denotes the number of textual reasoning steps executed by the Text Researcher. Evidence Aggregation and Complete Trajectory Construction. The Evidence Validator aggregates and verifies the complete collection of evidence produced across both reasoning phases. Taking as input all judge-verified visual observations in ℰvE^v and the textual reasoning trajectory τtext _text produced by the Text Researcher, the Evidence Validator performs a cross-modal reasoning pass to resolve evidence conflicts, filter hallucinated content, and generate the final answer aouta_out. By grounding its reasoning in the graph-structured memory rather than a flattened context window, the Evidence Validator avoids the state-blindness inherent in linear memory paradigms, ensuring that the complete body of evidence across both modalities is faithfully incorporated into the final answer. The complete multimodal deep research trajectory is obtained by merging τvis _vis and τtext _text: τ= τ=\ I,q,R1,A1v,O1v,…,RTv,ATvv, I,\ q,\ R_1,\ A_1^v,\ O_1^v,\ …,\ R_T_v,\ A_T_v^v, OTvv,RTv+1,ATv+1t,OTv+1t, O_T_v^v,\ R_T_v+1,\ A_T_v+1^t,\ O_T_v+1^t, …,RTv+Tt,ATv+Ttt,aout …,\ R_T_v+T_t,\ A_T_v+T_t^t,\ a_out\ (6) 3.2 Multimodal Memory Graph Figure 3: Inference pipeline of the GraMRAG framework. We construct a multimodal memory graph t=(t,ℰt)G_t=(V_t,\,E_t) that formalizes the multi-agent reasoning process as a dynamic DAG. This graph explicitly encodes action-observation dependencies across both the visual retrieval and textual research phases via a topological structure, fundamentally mitigating the state blindness inherent in linear memory paradigms and providing the structured foundation required by the TAPO algorithm. As illustrated in Figure 2(b), the memory graph evolves incrementally throughout reasoning, faithfully recording the exploration trajectory of each agent through node insertions and edge connections at each step. Node Schema and Modality Classification. Each node ni∈tn_i _t encodes a discrete reasoning unit as a typed quadruple: ni≜(μi,κi,σi,νi)n_i ( _i,\ _i,\ _i,\ _i ) (7) where μi∈nroot,nvis,nbrd,ntxt,nans _i∈\n_root,\,n_vis,\,n_brd,\,n_txt,\,n_ans\ is the modality type tag; κi _i is the search query or action descriptor driving the creation of this node; σi _i is a concise textual summary of retrieved or generated content; and νi _i is the visual feature bank, populated only for nvisn_vis-type nodes and set to empty for all other types. Five node types constitute the full reasoning pipeline. The root node (nrootn_root) is initialized with user query q and input image I, serving as the global anchor of the reasoning graph and permanently residing in tV_t. Visual nodes (nvisn_vis) are spawned by the Visual Explorer at each retrieval step t∈[1,Tv]t∈[1,T_v], storing the Hit/No Hit judgment and the corresponding evidence summary, with νi _i preserving multimodal content from matched web pages. The bridge node (nbrdn_brd) is a singleton node representing the modality bridging step, storing image description D as the semantic interface between the two reasoning phases. Text nodes (ntxtn_txt) are spawned by the Text Researcher at each retrieval step t∈[Tv+1,Tv+Tt]t∈[T_v+1,T_v+T_t], storing web-retrieved textual evidence summaries. The answer node (nansn_ans) is spawned by the Evidence Validator, storing the final generated answer aouta_out, whose insertion marks the termination of the reasoning process. The edge set ℰtE_t connects temporally adjacent nodes , encoding the sequential dependency of the reasoning flow. Iterative Graph Expansion Protocol. The memory graph tG_t evolves through sequential node insertions jointly driven by the three main agents and the Hit/No Hit Judge. The expansion protocol consists of four phases as follows. During visual expansion (t∈[1,Tv]t∈[1,T_v]), the Visual Explorer executes retrieval action Atv=VSearch(ℬt)A_t^v=VSearch(B_t) and the Hit/No Hit Judge outputs binary signal sts_t; a visual node nt=(nvis,κt,σt,νt)n_t=(n_vis,\, _t,\, _t,\, _t) is then inserted, where σt _t encodes the evidence summary alongside the sts_t judgment, νt _t preserves visual content from matched web pages, and a sequential edge (nt−1,nt)(n_t-1,n_t) connects it to its predecessor. During modality bridging, upon termination of the visual phase, the bridge node nbrd=(nbrd,∅,D,∅)n_brd=(n_brd,\, ,\,D,\, ) is inserted and connected to the final visual node via the cross-modal bridge edge (nTv,nbrd)(n_T_v,n_brd), completing the semantic context transition from the visual to the textual domain. During textual expansion (t∈[Tv+1,Tv+Tt]t∈[T_v+1,T_v+T_t]), the Text Researcher executes web retrieval actions; text nodes nt=(ntxt,κt,σt,∅)n_t=(n_txt,\, _t,\, _t,\, ) are inserted with sequential edges, and the bridge node is connected to the first text node via edge (nbrd,nTv+1)(n_brd,n_T_v+1), fully closing the cross-modal transition path. During answer generation, once the Evidence Validator determines that the accumulated evidence is sufficient, it inserts the answer node nans=(nans,∅,aout,∅)n_ans=(n_ans,\, ,\,a_out,\, ) and the reasoning process terminates. Upon completion of all four phases, the memory graph captures the complete multimodal reasoning trajectory as a structured DAG rather than compressing it into a one-dimensional sequence, thereby enabling each agent to efficiently query the full historical reasoning state and simultaneously providing the topological backbone required for fine-grained credit assignment in the reinforcement learning phase. 3.3 Topology-Aware Policy Optimization Figure 4: Overview of Topology-Aware Policy Optimization. As illustrated in Figure 4, we propose Topology-Aware Policy Optimization (TAPO) to address the systematic credit assignment problem in sparse reward reinforcement learning training, leveraging the topological structure of the memory graph to perform selective gradient masking that simultaneously rectifies both forms of bias across positive and negative trajectories with step-level precision. Graph-Structured Trajectory Decomposition. We ground the reinforcement learning framework in the memory graph structure defined in Section 3.2. At each reasoning step t, the policy model πθ _θ receives the graph-serialized context: Ct=ssys,q,HtC_t=\s_sys,\;q,\;H_t\ (8) where ssyss_sys is the system instruction and Ht=Serialize(t−1)H_t=Serialize(G_t-1) is the graph-linearized memory context. The policy model produces an reasoning segment that corresponds precisely to the construction of a single memory graph node ntn_t: st=(Ct,ζt,At,Ot)⟶nts_t=(C_t,\; _t,\;A_t,\;O_t)\; \;n_t (9) where ζt _t denotes the chain-of-thought reasoning preceding action AtA_t, and OtO_t is the environmental observation returned by the external tools. The terminal segment takes the form sans=(Cans,ζans,aout)s_ans=(C_ans,\; _ans,\;a_out). A complete rollout trajectory decomposes into an ordered sequence of segments Ψ=s1,s2,…,sTv+Tt,sans =\s_1,\;s_2,\;…,\;s_T_v+T_t,\;s_ans\. This node-based decomposition establishes a direct structural correspondence between policy gradient signals and graph topology: each gradient signal maps precisely to a specific graph node, providing the basis for the topology-driven credit assignment. Topology-Driven Credit Assignment. For each rollout, we assign a binary outcome reward r∈0,1r∈\0,1\ based on whether aouta_out matches the ground-truth answer atruea_true. We then exploit the DAG topology of TG_T to perform targeted gradient masking under two distinct scenarios as illustrated in Figure 4(b). Given a correct rollout (r=1r=1), we identify the critical reasoning path ρ∗ρ^* by traversing TG_T backwards from nansn_ans to nrootn_root along the directed edges. Nodes absent from ρ∗ρ^* constitute topological dead ends, namely exploratory steps that contribute no causal evidence to the correct answer. Propagating positive reward signals to these nodes would erroneously reinforce aimless exploration, so we suppress their gradient contributions. Given an incorrect rollout (r=0r=0), we identify the set of epistemically valuable nodes val⊆TN_val _T. For nvisn_vis-type nodes, a node ntn_t is designated as valuable when the Hit judgment sts_t stored within σt _t records st=1s_t=1, confirming that the visual retrieval successfully located semantically relevant content. For ntxtn_txt-type nodes, a dedicated relevance evaluator further determines whether the textual evidence retrieved at step t is directly relevant to the query. Excluding valN_val from the negative policy gradient update prevents the model from being penalized from performing demonstrably productive retrievals on trajectories that ultimately fail due to shortcomings in other reasoning steps. Integrating both scenarios, we define the gradient mask ψt∈0,1 _t∈\0,1\ for segment sts_t as: ψt=(r=1)⋅(nt∉ρ∗)+(r=0)⋅(nt∈val) _t=I(r=1)·I(n_t∉ρ^*)+I(r=0)·I(n_t _val) (10) where (⋅)I(·) is the indicator function; ψt=1 _t=1 suppresses the gradient contribution of segment sts_t during the policy update, while ψt=0 _t=0 permits normal gradient update. The TAPO objective is defined as: TAPO(θ)=[1∑g=1Gng∑g=1G∑i=1ng(1−ψg,i) _TAPO(θ)=E\! [ 1 _g=1^Gn_g _g=1^G _i=1^n_g(1- _g,i) . ⋅ℒclip(ωg,i(θ),A^g,i)−β⋅KL[πθ∥πref]] .·L_clip\! ( _g,i(θ),\, A_g,i )-β·KL\! [ _θ\,\|\, _ref ] ] (11) where ωg,i(θ) _g,i(θ) is the probability ratio of segment sg(i)s_g^(i) under the current and old policies; ngn_g denotes the number of segments in the g-th rollout. The (1−ψg,i)(1- _g,i) factor selectively gates gradient contributions based on the topological analysis of the memory graph: in positive trajectories, topological dead-end steps are masked to prevent their gradients from reinforcing redundant exploration; in negative trajectories, valuable retrieval steps are masked to prevent their gradients from penalizing effective retrieval behavior. Through this design, TAPO achieves precise step-level credit assignment across the entire multi-step reasoning trajectory, significantly outperforming conventional sparse-reward methods that rely solely on terminal outcome signals. 4 Experiments 4.1 Datasets and Evaluation Metrics Method General Text Image & Visual Document Large-Scale Long-Context Video Corpus Overall HotpotQA SQuAD ViDoSeek SlideVQA MMLongBench LVBench WikiHowQA SyntheticQA XVBench Qwen3-VL-4B-Instruct ReAct 62.8 63.9 40.7 45.8 13.4 13.0 13.2 29.1 22.5 33.8 Vanilla RAG 62.4 63.2 44.9 43.4 14.9 12.8 12.1 33.7 27.1 34.9 UniversalRAG 51.7 65.3 40.3 14.6 5.2 14.9 3.8 22.9 6.9 25.1 VideoRAG 56.4 63.1 41.5 34.9 16.0 18.1 20.5 44.3 28.3 35.9 MemAgent 66.2 70.3 46.4 42.7 10.9 20.1 20.2 33.4 23.7 37.1 Mem1 70.9 66.2 45.4 50.7 26.5 17.8 13.0 41.8 30.8 40.3 VimRAG 75.7 74.0 48.3 52.9 28.3 23.5 21.5 52.4 33.5 45.6 GraMRAG 78.4 77.1 51.2 55.6 30.8 26.0 24.1 57.3 36.0 48.5 Qwen3-VL-8B-Instruct ReAct 70.7 64.7 41.3 50.4 14.7 15.8 21.7 34.2 25.1 37.6 Vanilla RAG 63.4 65.3 47.0 49.0 17.4 15.6 14.8 37.7 29.7 37.8 UniversalRAG 56.3 66.7 45.2 16.9 6.9 20.4 10.7 24.0 8.7 28.4 VideoRAG 62.5 62.3 43.1 36.0 19.3 23.8 27.1 48.6 32.1 39.4 MemAgent 72.3 74.1 48.1 44.8 13.5 23.0 23.4 36.8 28.3 40.5 Mem1 71.5 68.6 44.2 57.0 34.0 22.5 20.8 44.0 31.4 43.8 VimRAG 78.6 76.6 53.3 63.1 34.7 23.2 29.0 54.5 37.7 50.1 GraMRAG 81.3 79.8 56.1 66.4 38.5 26.2 33.4 59.0 41.2 53.5 Table 1: Performance comparison across different benchmarks. We evaluate GraMRAG on a comprehensive set of benchmarks covering diverse task categories: general text benchmarks including HotpotQA Yang et al. (2018) and SQuAD Rajpurkar et al. (2016); image and visual document benchmarks including ViDoSeek Wang et al. (2025a), SlideVQA Tanaka et al. (2023) and MMLongBench Ma et al. (2024); large-scale long-context video corpus benchmarks including LVBench Wang et al. (2025b), WikiHowQA and SyntheticQA Jeong et al. (2025), and XVBench Wang et al. (2026). We employ a unified binary model-based metric (0 or 1) across all tasks, where a reward model performs semantic matching between the generated and reference answers to ensure consistent and fair evaluation. 4.2 Implementation Details We conduct experiments using Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct Bai et al. (2025) as the agent backbones. We use Qwen3-Max Yang et al. (2025) as the reward model to evaluate the quality and relevance of generated responses. Training is divided into two stages. In the SFT stage, we employ LlamaFactory Zheng et al. (2024) with LoRA Hu et al. (2022) of rank 32 to fine-tune the language model while keeping the vision tower and multimodal projector frozen. Training spans 3 epochs with a batch size of 8 and gradient accumulation steps of 8, a learning rate of 1.0e-4, and a cosine learning rate scheduler. In the RL stage, we use rLLM Tan et al. (2025) with the TAPO objective, an actor learning rate of 1.0e-6, a total of 2 training epochs, 8 agent groups, and a batch size of 32. All experiments are conducted on 8 NVIDIA H100 GPUs. 4.3 Comparison with State-of-the-Art Methods We compare GraMRAG against representative baselines: ReAct Yao et al. (2022) iteratively invokes external retrieval tools in a think-then-act paradigm; Vanilla RAG uses the original question as a query and performs direct MLLM inference over retrieved results; UniversalRAG Yeo et al. (2025) formulates cross-modal retrieval as a unified routing problem; VideoRAG Jeong et al. (2025) extracts significant visual information via key-frame selection; MemAgent Yu et al. (2025) manages context by sequentially feeding search results to the model; Mem1 Zhou et al. (2025) dynamically compresses agent memory through a cyclical retrieval-then-memorization process; and VimRAG Wang et al. (2026) models reasoning process to manage large-scale multimodal contexts. As presented in Table 1, linear history accumulation paradigms such as ReAct rapidly exhaust the context window under the heavy token overhead of visual data; task-specific methods including VideoRAG and UniversalRAG exhibit limited generalizability due to fixed inference pipelines; and summarization-based memory paradigms such as MemAgent and Mem1, lacking explicit modeling of agent reasoning states, progressively lose track of historical retrieval paths as interaction rounds accumulate; and VimRAG models the reasoning process but lacks long-horizon reasoning capability. Our GraMRAG achieves consistently state-of-the-art performance across all benchmarks, achieving overall average scores of 48.5% and 53.5% under the Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct settings respectively, validating the effectiveness of formalizing the multi-step reasoning process as a dynamic DAG with topology-driven fine-grained credit assignment for multimodal knowledge-intensive reasoning tasks. 4.4 Ablation Study Reasoning Horizon Memory Paradigm Policy Optimization Acc Standard Long Iterative Graph GRPO TAPO ✓ ✓ 43.8 ✓ ✓ 44.2 ✓ ✓ 47.5 ✓ ✓ ✓ 49.3 ✓ ✓ ✓ 53.5 Table 2: Ablation study across all benchmarks. To validate the contribution of each component in GraMRAG, we conduct ablation studies across all benchmarks. As shown in Table 2, the three components exhibit clear progressive complementarity: long-horizon reasoning enables sustained multi-round retrieval across extended cross-modal reasoning trajectories; the multimodal memory graph demonstrates that formalizing reasoning as a dynamic DAG effectively mitigates state blindness and suppresses redundant retrieval; and replacing the standard sparse-reward method Group Relative Policy Optimization (GRPO) Shao et al. (2024) with TAPO delivers a further improvement, directly confirming the critical role of topology-guided credit assignment in fine-grained policy optimization. The complete model achieves an average score of 53.5%, an improvement of over 9 points above the baseline, comprehensively validating the necessity and complementarity of each component. 5 Analysis and Discussion Figure 5: Analysis of retrieval performance. Retrieval quality depends on reasoning depth. High-quality answer generation in complex multimodal tasks depends on sustaining precise multi-round retrieval across a long reasoning horizon. Traditional memory paradigms cause agents to progressively lose track of historical retrieval paths and accumulate redundant searches that severely constrain effective reasoning depth. GraMRAG addresses this by transferring the iterative planning capability into the multimodal domain via the vision-text bridged reasoning paradigm, supporting trajectories spanning tens of steps; the multimodal memory graph further encodes action-observation dependencies explicitly, directing each retrieval attempt precisely toward unexplored evidence space. As shown in Figure 5, this design yields substantially higher hit rate across all three task categories, directly validating the decisive role of long-horizon reasoning depth in determining retrieval quality. Graph memory mitigates state blindness. The dynamic DAG representation of the multimodal memory graph explicitly encodes action-observation dependencies as directed edges, enabling each agent to efficiently access the complete historical reasoning trajectory and effectively mitigating state blindness. As shown in Table 2, replacing iterative summarization-based memory with the multimodal memory graph yields a 3.3-point performance improvement, validating that the multimodal memory graph effectively mitigates state blindness and suppresses redundant retrieval, confirming the necessity of structured memory in long-horizon multimodal reasoning. Figure 6: Analysis of training efficiency. Graph pruning accelerates training convergence. As demonstrated in Figure 6, TAPO achieves faster convergence throughout training compared to the baseline Group Sequence Policy Optimization (GSPO) Zheng et al. (2025), indicating that topology-guided graph pruning drives the model toward faster convergence by supplying cleaner credit assignment signals. This yields two critical insights into agentic reinforcement learning: first, optimization stability fundamentally depends on ensuring correct positive gradient signals while eliminating ambiguous updates from negative trajectories; second, the quality of rollout samples, specifically their preference alignment, is more decisive for performance than simply scaling up the training set. By simultaneously masking dead-end nodes in positive trajectories and preserving valuable nodes in negative ones, TAPO achieves fine-grained step-level credit assignment rather than coarse terminal-reward-level optimization. 6 Conclusion In this paper, we introduce GraMRAG, a novel graph memory-guided multi-agent RAG framework for complex multimodal long-horizon reasoning tasks. Our approach integrates three key innovations: a vision-text bridged reasoning paradigm that unifies multi-scale entity cropping with a ReAct-style visual toolchain for long-horizon cross-modal reasoning; multimodal memory graph construction that formalizes agent reasoning as a dynamic directed acyclic graph (DAG) to fundamentally mitigate state blindness and suppress redundant retrieval; and Topology-Aware Policy Optimization (TAPO) algorithm that leverages graph topology for fine-grained credit assignment across multi-step reasoning trajectories. Extensive experiments on diverse multimodal benchmarks demonstrate that our method consistently outperforms existing baselines and achieves state-of-the-art performance. Future work will investigate the scalability of the proposed framework to larger architectures and broader multimodal scenarios. 7 Limitations Although GraMRAG achieves significant performance improvements on complex multimodal multi-step reasoning tasks, the multi-agent collaboration framework introduces some inference overhead compared to traditional RAG approaches. Notably, such a trade-off is generally acceptable in deep research scenarios where users prioritize reasoning accuracy over response speed. Nevertheless, it may hinder direct deployment in resource-constrained environments. Therefore, optimizing the interaction strategy among agents and designing more efficient lightweight graph memory mechanisms constitute key directions for future work. References Bai et al. (2025) Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, and 1 others. 2025. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Chang et al. (2025) Chia-Yuan Chang, Zhimeng Jiang, Vineeth Rakesh, Menghai Pan, Chin-Chia Michael Yeh, Guanchu Wang, Mingzhi Hu, Zhichao Xu, Yan Zheng, Mahashweta Das, and 1 others. 2025. Main-rag: Multi-agent filtering retrieval-augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2607–2622. Dong et al. (2026) Jiangwen Dong, Zehui Lin, Wanyu Lin, and Mingjin Zhang. 2026. S-dag: A subject-based directed acyclic graph for multi-agent heterogeneous reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 29394–29402. Fan et al. (2026) Xuanbo Fan, Tianqi Zhao, Yi Cheng, Chi Xiu, Jiaxin Guo, Boci Peng, Bingjing Xu, Jessica Zhang, Feng Sun, and Yan Zhang. 2026. Segmem-rag: Adaptive memory for retrieval-augmented generation in open-ended knowledge environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 30647–30655. Geng et al. (2025) Xinyu Geng, Peng Xia, Zhen Zhang, Xinyu Wang, Qiuchen Wang, Ruixue Ding, Chenxi Wang, Jialong Wu, Yida Zhao, Kuan Li, and 1 others. 2025. Webwatcher: Breaking new frontier of vision-language deep research agent. arXiv preprint arXiv:2508.05748. Guan et al. (2025) Ziyi Guan, Jason Chun Lok Li, Zhijian Hou, Pingping Zhang, Donglai Xu, Yuzhi Zhao, Mengyang Wu, Jinpeng Chen, Thanh-Toan Nguyen, Pengfei Xian, and 1 others. 2025. Kg-rag: Enhancing gui agent decision-making via knowledge graph-driven retrieval-augmented generation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 5396–5405. Hu et al. (2022) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3. Hu et al. (2026) Hao Hu, Yifan Feng, Ruoxue Li, Rundong Xue, Xingliang Hou, Zhiqiang Tian, Yue Gao, and Shaoyi Du. 2026. Cog-rag: Cognitive-inspired dual-hypergraph with theme alignment retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 31032–31040. Huang et al. (2026) Wenxuan Huang, Yu Zeng, Qiuchen Wang, Zhen Fang, Shaosheng Cao, Zheng Chu, Qingyu Yin, Shuang Chen, Zhenfei Yin, Lin Chen, and 1 others. 2026. Vision-deepresearch: Incentivizing deepresearch capability in multimodal large language models. arXiv preprint arXiv:2601.22060. Jeong et al. (2025) Soyeong Jeong, Kangsan Kim, Jinheon Baek, and Sung Ju Hwang. 2025. Videorag: Retrieval-augmented generation over video corpus. In Findings of the Association for Computational Linguistics: ACL 2025, pages 21278–21298. Jiang et al. (2025) Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Pengshuo Qiu, Pan Lu, Zehui Chen, Guanglu Song, Peng Gao, Yu Liu, and 1 others. 2025. Mmsearch: Unveiling the potential of large models as multi-modal search engines. In The Thirteenth International Conference on Learning Representations. Li et al. (2026) Yuanhao Li, Mingshan Liu, Hongbo Wang, Yiding Zhang, Yifei Ma, and Wei Tan. 2026. Draft-rl: Multi-agent chain-of-draft reasoning for reinforcement learning-enhanced llms. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 29530–29537. Ma et al. (2024) Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, and 1 others. 2024. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations. Advances in Neural Information Processing Systems, 37:95963–96010. Narayan et al. (2025) Kartik Narayan, Yang Xu, Tian Cao, Kavya Nerella, Vishal M Patel, Navid Shiee, Peter Grasch, Chao Jia, Yinfei Yang, and Zhe Gan. 2025. Deepmmsearch-r1: Empowering multimodal llms in multimodal web search. arXiv preprint arXiv:2510.12801. Ni and Ye (2026) Yiran Ni and Deshi Ye. 2026. Multi-agent corridor reasoning for multi-agent path finding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 29617–29624. Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 2383–2392. Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Siingh et al. (2025) Shikhhar Siingh, Abhinav Rawat, Chitta Baral, and Vivek Gupta. 2025. Getreason: Enhancing image context extraction through hierarchical multi-agent reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29779–29800. Sinha et al. (2026) Sanchit Sinha, Guangzhi Xiong, Zhenghao He, and Aidong Zhang. 2026. Concept-rulenet: Grounded multi-agent neurosymbolic reasoning in vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 29678–29686. Tan et al. (2025) Sijun Tan, Michael Luo, Colin Cai, Tarun Venkat, Kyle Montgomery, Aaron Hao, Tianhao Wu, Arnav Balyan, Manan Roongta, Chenguang Wang, and 1 others. 2025. rllm: A framework for post-training language agents. Tanaka et al. (2023) Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. 2023. Slidevqa: A dataset for document visual question answering on multiple images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13636–13645. Team et al. (2025) Tongyi DeepResearch Team, Baixuan Li, Bo Zhang, Dingchu Zhang, Fei Huang, Guangyu Li, Guoxin Chen, Huifeng Yin, Jialong Wu, Jingren Zhou, and 1 others. 2025. Tongyi deepresearch technical report. arXiv preprint arXiv:2510.24701. Wang et al. (2025a) Qiuchen Wang, Ruixue Ding, Zehui Chen, Weiqi Wu, Shihang Wang, Pengjun Xie, and Feng Zhao. 2025a. Vidorag: Visual document retrieval-augmented generation via dynamic iterative reasoning agents. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9124–9145. Wang et al. (2026) Qiuchen Wang, Shihang Wang, Yu Zeng, Qiang Zhang, Fanrui Zhang, Zhuoning Guo, Bosi Zhang, Wenxuan Huang, Lin Chen, Zehui Chen, and 1 others. 2026. Vimrag: Navigating massive visual context in retrieval-augmented generation via multimodal memory graph. arXiv preprint arXiv:2602.12735. Wang et al. (2025b) Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Ming Ding, Xiaotao Gu, Shiyu Huang, Bin Xu, and 1 others. 2025b. Lvbench: An extreme long video understanding benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22958–22967. Wang (2026) Zhongyu Wang. 2026. Mdocrag-rl: Empowering multi-modal document rag via complex visual reasoning with reinforcement learning. In Findings of the Association for Computational Linguistics: ACL 2026, pages 8641–8651. Wang and Liu (2026) Zhongyu Wang and Pengbo Liu. 2026. Mars-rl: Enhancing multi-agent rag systems for multi-modal documents via strategic reasoning with reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9674–9683. Wu et al. (2025a) Jinming Wu, Zihao Deng, Wei Li, Yiding Liu, Bo You, Bo Li, Zejun Ma, and Ziwei Liu. 2025a. Mmsearch-r1: Incentivizing lmms to search. arXiv preprint arXiv:2506.20670. Wu et al. (2025b) Xixi Wu, Kuan Li, Yida Zhao, Liwen Zhang, Litu Ou, Huifeng Yin, Zhongwang Zhang, Xinmiao Yu, Dingchu Zhang, Yong Jiang, and 1 others. 2025b. Resum: Unlocking long-horizon search intelligence via context summarization. arXiv preprint arXiv:2509.13313. Xinjie et al. (2025) Zhao Xinjie, Fan Gao, Xingyu Song, Yingjian Chen, Rui Yang, Yanran Fu, Yuyang Wang, Yusuke Iwasawa, Yutaka Matsuo, and Irene Li. 2025. Reagent: Reversible multi-agent reasoning for knowledge-enhanced multi-hop qa. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 4067–4089. Xu et al. (2025a) Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. 2025a. A-mem: Agentic memory for llm agents. arXiv preprint arXiv:2502.12110. Xu et al. (2025b) Yangyang Xu, Jinpeng Hu, Zhuoer Zhao, Zhangling Duan, Xiao Sun, and Xun Yang. 2025b. Multiagentesc: A llm-based multi-agent collaboration framework for emotional support conversation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 4665–4681. Yang et al. (2025) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2369–2380. Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. Ye et al. (2025) Rui Ye, Zhongwang Zhang, Kuan Li, Huifeng Yin, Zhengwei Tao, Yida Zhao, Liangcai Su, Liwen Zhang, Zile Qiao, Xinyu Wang, and 1 others. 2025. Agentfold: Long-horizon web agents with proactive context management. arXiv preprint arXiv:2510.24699. Yeo et al. (2025) Woongyeong Yeo, Kangsan Kim, Soyeong Jeong, Jinheon Baek, and Sung Ju Hwang. 2025. Universalrag: Retrieval-augmented generation over corpora of diverse modalities and granularities. arXiv preprint arXiv:2504.20734. Yu et al. (2025) Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, and 1 others. 2025. Memagent: Reshaping long-context llm with multi-conv rl-based memory agent. arXiv preprint arXiv:2507.02259. Zhang et al. (2025a) Taolin Zhang, Dongyang Li, Qizhou Chen, Chengyu Wang, and Xiaofeng He. 2025a. Belle: A bi-level multi-agent reasoning framework for multi-hop question answering. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4184–4202. Zhang et al. (2026) Wenchuan Zhang, Jingru Guo, Hengzhe Zhang, Penghao Zhang, Jie Chen, Shuwan Zhang, Zhang Zhang, Yuhao Yi, and Hong Bu. 2026. Patho-agenticrag: towards multimodal agentic retrieval-augmented generation for pathology vlms via reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 29921–29929. Zhang et al. (2025b) Zeyu Zhang, Quanyu Dai, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. 2025b. A survey on the memory mechanism of large language model-based agents. ACM Transactions on Information Systems, 43(6):1–47. Zheng et al. (2025) Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, and 1 others. 2025. Group sequence policy optimization. arXiv preprint arXiv:2507.18071. Zheng et al. (2024) Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 3: system demonstrations), pages 400–410. Zhou et al. (2026) Yuxiang Zhou, Jichang Li, Yanhao Zhang, Haonan Lu, and Guanbin Li. 2026. Mobile-agent-rag: Driving smart multi-agent coordination with contextual knowledge empowerment for long-horizon mobile automation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 29939–29947. Zhou et al. (2025) Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, and Paul Pu Liang. 2025. Mem1: Learning to synergize memory and reasoning for efficient long-horizon agents. arXiv preprint arXiv:2506.15841.