Paper deep dive
Dual-Latent Memory Routing for Vision-Language Reasoning
Hao-Xuan Ma, Jin-Fei Qi, Yicheng Xiao, Han-Jia Ye
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 88%
Last extracted: 9/9/2026, 4:12:48 AM
Summary
The paper introduces Dual-Latent Memory Routing (DLMR), a parameter-efficient mechanism for Multimodal Large Language Models (MLLMs) to address long-horizon reasoning degradation. DLMR employs two separate latent memories: a visual memory for compressing image evidence and a reasoning memory for tracking intermediate constraints. A lightweight router dynamically selects which memory to activate and the token budget for injection during inference, preserving visual grounding and coherence without modifying the frozen base MLLM backbone.
Entities (10)
Relation Signals (9)
DLMR → uses → Visual Memory
confidence 95% · DLMR instantiates the two ingredients above with Dual Latent Memories... a visual memory that compresses image evidence
DLMR → uses → Reasoning Memory
confidence 95% · DLMR instantiates the two ingredients above with Dual Latent Memories... a reasoning memory that tracks intermediate conclusions and constraints
DLMR → uses → Router
confidence 95% · A Router then dynamically decides which memory and how much to reuse during inference
DLMR → improves → long-horizon reasoning
confidence 90% · yielding substantial gains on both general and reasoning benchmarks... maintaining coherent long-horizon reasoning
DLMR → evaluatedon → MMVet
confidence 85% · We evaluate DLMR on a diverse set of vision-language benchmarks... MMVet
DLMR → evaluatedon → MathVision
confidence 85% · For reasoning-centric evaluation, we use ... MathVision
DLMR → trainedwith → GRPO
confidence 85% · DLMR (RL) uses GRPO-based training
DLMR → builton → Qwen2.5-VL-7B
confidence 80% · using two MLLMs (Qwen2.5-VL-7B and InternVL-3-8B)
DLMR → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal large language models (MLLMs) have recently made strong progress in vision-language reasoning, yet their performance often degrades as generations grow longer. A key factor is that they frequently lose track of earlier visual evidence and intermediate constraints under a monolithic growing context. Inspired by how humans separately recall what they see and what they infer when solving complex tasks, we propose DLMR, a parameter-efficient mechanism that equips MLLMs with Dual Latent Memories: a visual memory that compresses image evidence and a reasoning memory that tracks intermediate conclusions and constraints. A Router then dynamically decides which memory and how much to reuse during inference, preserving visual grounding while maintaining coherent long-horizon reasoning. DLMR is trained in three stages, from latent memory construction to selective router learning, while keeping the base MLLM frozen, yielding substantial gains on both general and reasoning benchmarks with only a small number of additional trainable parameters. Analyses further show interpretable, state-dependent routing with specialized memory roles and reduced decoding tokens over long generations. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.05539v1
- Canonical: https://arxiv.org/abs/2609.05539v1
Trouble viewing inline? Open PDF directly →
Full Text
76,404 characters extracted from source content.
Expand or collapse full text
Dual-Latent Memory Routing for Vision-Language Reasoning Hao-Xuan Ma Affiliation: School of Artificial Intelligence, Nanjing University, China Affiliation: National Key Laboratory for Novel Software Technology, Nanjing University, China Jin-Fei Qi Affiliation: School of Artificial Intelligence, Nanjing University, China Affiliation: National Key Laboratory for Novel Software Technology, Nanjing University, China Yicheng Xiao Affiliation: Institute of Automation, Chinese Academy of Sciences, Beijing, China Han-Jia Ye Affiliation: School of Artificial Intelligence, Nanjing University, China Affiliation: National Key Laboratory for Novel Software Technology, Nanjing University, China Correspondence to: yehj@lamda.nju.edu.cn Abstract Multimodal large language models (MLLMs) have recently made strong progress in vision-language reasoning, yet their performance often degrades as generations grow longer. A key factor is that they frequently lose track of earlier visual evidence and intermediate constraints under a monolithic growing context. Inspired by how humans separately recall what they see and what they infer when solving complex tasks, we propose DLMR, a parameter-efficient mechanism that equips MLLMs with Dual Latent Memories: a visual memory that compresses image evidence and a reasoning memory that tracks intermediate conclusions and constraints. A Router then dynamically decides which memory and how much to reuse during inference, preserving visual grounding while maintaining coherent long-horizon reasoning. DLMR is trained in three stages, from latent memory construction to selective router learning, while keeping the base MLLM frozen, yielding substantial gains on both general and reasoning benchmarks with only a small number of additional trainable parameters. Analyses further show interpretable, state-dependent routing with specialized memory roles and reduced decoding tokens over long generations. Code is available at https://github.com/Hunter-Wrynn/DLMR. Keywords: Machine Learning, ICML †affiliationnotice: Equal contribution 1 Introduction Multimodal large language models (MLLMs) have recently achieved impressive progress in integrating visual and textual information for complex reasoning tasks (Shao et al., 2024a; Wen et al., 2025). However, as reasoning chains lengthen, MLLMs increasingly fail to preserve early visual evidence and intermediate reasoning states over extended decoding (Sun et al., 2025; Zhou et al., 2025; Tu et al., 2026). A key factor is that most MLLMs rely on a single growing context: the image is encoded once as a prefix, while subsequent states are carried only through generated text (Ren et al., 2025; Xu et al., 2025a). Consistent with this limitation, attention to image tokens steadily decays as generation proceeds (Fig. 1). Prior work encourages revisiting relevant information via prompting-based guidance or training-based alignment (Shao et al., 2024a; Gao et al., 2024; Liu et al., 2025a; Yu et al., 2024b), yet these approaches remain bound to the same growing-context mechanism and thus do not fundamentally prevent long-horizon forgetting—models can still lose effective access to early multimodal cues as decoding length increases. Figure 1: Normalized attention to image tokens through generation step. Compared to the vanilla MLLM (Qwen2.5-vl-7B on MathVision Benchmark), DLMR preserves substantially more visual attention in later decoding, enabling more reliable revisiting of early visual evidence and often requiring fewer reasoning tokens to reach the final answer. A natural remedy is to move beyond the single-stream, text-carried context. Cognitive science suggests that robust multi-step reasoning relies on a disentangled working memory (Wolpert et al., 2011), where humans keep separate buffers for observations and constraints and revisit them on demand. Current MLLMs lack a structured alternative to the growing-context paradigm, hindering selective storage and reuse as reasoning progresses. This calls for two architectural ingredients: (i) a disentangled representation that can store reusable visual evidence and evolving reasoning constraints, and (i) a state-dependent controller that dynamically determines what and how much to retrieve during generation. Motivated by this, we propose DLMR, a parameter-efficient mechanism for long-horizon multimodal reasoning. DLMR instantiates the two ingredients above with Dual Latent Memories and a learned Router: it separates reusable observations from evolving task state, and selectively revisits either source only when needed. This enables structured, state-dependent reuse—the model can verify cues from the image when grounding is required, or reuse previously derived state when progressing through multi-step inference, while allocating more computation to harder steps. For instance, in a geometry problem, one step may require checking a diagram attribute, whereas the next step primarily updates previously established relations. Concretely, DLMR maintains two learnable latent vectors (visual and reasoning) as disentangled memories. A lightweight memory injector converts selected latents into memory tokens that are appended during inference, leaving the base MLLM unchanged. A discrete memory router then selects which memory to activate and sets a token budget for the current step. As a diagnostic, while the vanilla MLLM rapidly attenuates attention to visual tokens during decoding, DLMR sustains stronger late-stage visual attention with fewer generated tokens on average (Fig. 1). We also introduce a three-stage training recipe from latent memory construction to selective router learning while keeping the backbone frozen, making DLMR easy to integrate into existing MLLMs with a small parameter overhead. Experiments on challenging long-horizon multimodal reasoning benchmarks show that DLMR improves task performance under controlled inference budgets. Finally, we provide analyses of routing behavior and memory specialization, illustrating when the model prefers visual memory or reasoning memory and how it allocates its thinking budget over the course of generation. In summary, our contributions are as follows: • We propose Dual-Latent Memory Routing (DLMR), a parameter-efficient multimodal memory mechanism that separates reusable perceptual evidence from evolving logical constraints for long-horizon reasoning. • We design a lightweight memory injector and a memory router that, conditioned on the current context, selects the memory source and token budget for controlled reuse with minimal backbone changes. • We introduce a three-stage training recipe and provide empirical results and analyses that demonstrate improved long-horizon performance and better ability to revisit visual evidence. 2 Related Work 2.1 Vision Language Reasoning Recent multimodal large language models (MLLMs) have substantially improved vision-language reasoning by aligning visual representations with instruction-following language models and scaling up multimodal supervision (Bai et al., 2025; Lu et al., 2025; Agarwal et al., 2025). Building on these foundations, multimodal reasoning is commonly enhanced from three perspectives: (1) CoT-based methods elicit multi-step reasoning at inference time via chain-of-thought prompting and test-time scaling, including multimodal variants that ground rationales in visual evidence (Shao et al., 2024a; Xu et al., 2025a; Gao et al., 2024; Jiang et al., 2025; Gao et al., 2025; Wu et al., 2025b); (2) Training-based methods improve reasoning through instruction tuning and targeted data curation, further refined by reinforcement learning with task-level feedback or verifiable rewards (Shen et al., 2025; Wen et al., 2025; Wu et al., 2025c; Yu et al., 2024b; Peng et al., 2025); and (3) RAG-based methods augment prompts with external evidence (documents, images, or knowledge) to support grounded reasoning and long-context tasks (Kim et al., 2025; Yu et al., 2024a; Ming and Li, 2024; Hao et al., 2025; Tanaka et al., 2025). Despite these advances, long-horizon multimodal reasoning remains challenging as models often struggle to reliably reuse early visual evidence and intermediate constraints over long generations (Sun et al., 2025; Zhou et al., 2025; Tu et al., 2026; Wu et al., 2025a; Li et al., 2025). 2.2 Memory Systems Memory-augmented modeling has been widely studied in LLMs and is increasingly transferred to multimodal settings, where the core goal is to preserve reusable information beyond a single growing context (Hu et al., 2025; Zhang et al., 2025a; Long et al., 2025; Yan et al., 2025). Early designs rely on explicit retrieval, with multimodal variants extending retrieval to images or cross-modal evidence (Kim et al., 2025; Yu et al., 2024a). A complementary direction introduces latent memory by learning persistent memory tokens or latent contexts that can be attended to during generation, reducing dependence on long textual traces and enabling parameter-efficient reuse (Yu et al., 2025; Zeng et al., 2025; Zhang et al., 2024b; Liu et al., 2025b). These lines of work motivate memory mechanisms for MLLMs, but existing transfers typically treat memory as a single undifferentiated source—either retrieved context or a monolithic latent space—and provide limited state-dependent control over what to reuse and how much to think when reasoning unfolds. In contrast, DLMR explicitly separates memory into two specialized latent buffers for visual evidence and reasoning constraints, and employs a discrete router to condition reuse decisions. Figure 2: Overview of DLMR. Given an image I and text prompt x, a frozen MLLM backbone ℳθM_θ performs autoregressive decoding. DLMR maintains two compact latent-space memories (v)Z^(v) (visual evidence) and (r)Z^(r) (reasoning constraints). At delimiter-eligible steps, a discrete router selects the memory type st∈v,rs_t∈\v,r\ and an injection budget kt∈+k_t ^+ (or a null action). A lightweight memory injector contextualizes the selected latents into ktk_t memory tokens tM_t, which are appended to the current context before predicting the next token. This enables state-dependent, budget-aware reuse without modifying the backbone. 3 Preliminary Problem Setup. We study memory-augmented multimodal reasoning, where each instance consists of an image I and a tokenized textual instruction (or dialogue history) =(x1,…,xL)x=(x_1,…,x_L). The goal is to generate an answer =(y1,…,yT)y=(y_1,…,y_T) consistent with both modalities. We consider an autoregressive multimodal large language model (MLLM) parameterizing P(∣,)=∏t=1TP(yt∣,,<t).P(y ,x)= _t=1^TP(y_t ,x,y_<t). (1) We build on a Transformer-based MLLM backbone ℳθM_θ with hidden size d and keep θ frozen. Following standard MLLMs, I is encoded into a sequence of visual tokens and concatenated with text tokens to form a unified multimodal input sequence to ℳθM_θ. Why long-horizon forgetting arises. In standard MLLMs, an image I is encoded into LvL_v visual tokens =(v1,…,vLv)v=(v_1,…,v_L_v) and concatenated with text prompt x as a single prefix 0=[;]s_0=[v;x]; the model then generates =(y1,…,yT)y=(y_1,…,y_T) autoregressively. At decoding step t, the visible context is t=[0;<t]s_t=[s_0;y_<t] with length nt=Lv+L+t−1n_t=L_v+L+t-1, which grows with t. In a decoder-only Transformer, the next-token state is updated by causal self-attention over all previous tokens: αt,i=exp(zt,i)∑j=1ntexp(zt,j),zt,i=⟨qt,ki⟩/d. _t,i= (z_t,i) _j=1^n_t (z_t,j), z_t,i= q_t,k_i / d. (2) Since ∑i=1ntαt,i=1 _i=1^n_t _t,i=1, the average attention weight is 1/nt1/n_t; unless the model becomes increasingly peaked on the early visual prefix, the total attention mass on the fixed image-token set =1,…,LvV=\1,…,L_v\ tends to shrink as the context grows: Atimg=∑i∈αt,i≈O(Lvnt)→t→∞0.A^img_t= _i _t,i≈ O\! ( L_vn_t ) [t→∞]0. (3) Therefore, under a single growing context, early visual evidence becomes progressively harder to reuse, which aligns with long-context retrieval failures (Liu et al., 2024; Hsieh et al., 2024) and the empirically observed visual grounding decay in MLLMs, where attention to image tokens weakens as generation proceeds and correlates with hallucinations (Xu et al., 2025b). Motivated by this, DLMR introduces an explicit memory-and-control interface: it stores reusable visual evidence and evolving reasoning constraints in dual latent memories and injects compact memory tokens on demand, mitigating long-horizon forgetting without repeatedly restating state in text. 4 Method As illustrated in Fig. 2, DLMR adds three lightweight components on top of the frozen backbone: (i) dual latent memories for visual evidence and reasoning constraints, (i) a memory injector that converts selected latents into step-specific memory tokens, and (i) a discrete router that decides when/what/how much to inject. We describe memory generation in Sec. 4.1, routing in Sec. 4.2, and the training recipe in Sec. 4.3. 4.1 Memory Generation Dual latent memories. To enhance long-horizon reasoning without modifying the backbone, we introduce two compact, input-agnostic latent-space memories (s)∈ℝMs×d,s∈v,r,Z^(s) ^M_s× d, s∈\v,r\, (4) where v denotes a visual latent-space memory and r denotes a reasoning latent-space memory. These memories are shared across all inputs and will be contextualized into step-specific memory tokens during decoding. Given a token budget k∈+k ^+ (e.g., 4,8,16\4,8,16\), we select the first k latent vectors from the chosen memory set, denoted (s)1:kZ^(s)_1:k. Memory injector. Let t∈ℝLt×dE_t ^L_t× d denote the current multimodal token embeddings (image tokens plus the text prefix up to <ty_<t). Conditioned on (t,1:k(s),k)(E_t,Z^(s)_1:k,k), a lightweight memory injector gϕg_φ contextualizes the selected latents into k memory tokens: t=gϕ(t,1:k(s),k)∈ℝk×d.M_t=g_φ\! (E_t,Z^(s)_1:k,k ) ^k× d. (5) For the implementation of gϕg_φ, we adopt a parameter-efficient approach by integrating LoRA into a replica of the frozen base MLLM. We first map the current context embeddings to the injector space: t(w)=Pθ→w(t)E_t^(w)=P_θ→ w(E_t), and let the latent vectors attend to the context through self-attention. Specifically, we construct the augmented sequence ~t(w)=[t(w);1:k(s)]∈ℝ(Lt+k)×dw, E_t^(w)=[E_t^(w);Z^(s)_1:k] ^(L_t+k)× d_w, (6) and pass it through the injector to obtain hidden states ~t(w) H_t^(w). We then take the last k states as contextualized memory tokens: t=Pw→θ(~t(w)[Lt+1:Lt+k])∈ℝk×d.M_t=P_w→θ\! ( H_t^(w)[L_t+1:L_t+k] ) ^k× d. (7) These tokens will be appended to the decoding context when the router invokes an injection (Sec. 4.2). 4.2 Memory Routing Memory-augmented decoding. We allow memory injection only at a subset of eligible generation steps to improve stability and control overhead. Let (<t)⊆1,…,TT(y_<t) \1,…,T\ denote the eligible set, which may depend on the current decoded prefix. Concretely, we define a small set of delimiter patterns D, and mark step t as eligible if the current decoded prefix ends with any delimiter in D. We further cap the number of routed injections by enforcing a per-sample limit of at most NmaxN_ injections. At each t∈(<t)t (y_<t), the router chooses a discrete action at=(st,kt),st∈v,r,kt∈+,a_t=(s_t,k_t), s_t∈\v,r\,\;k_t ^+, (8) where +K^+ is a small set of positive token budgets (e.g., 4,8,16\4,8,16\). If t is not eligible (or the injection cap is reached), we deterministically perform no injection and fall back to standard decoding. Routing policy. We model routing as a lightweight, context-dependent policy πψ(at∣t) _ψ(a_t _t). In practice, we implement πψ _ψ as a parameter-efficient LoRA-augmented head on top of the backbone that reuses the hidden states from the ongoing decoding pass. Given the current multimodal prefix, we summarize the decoding state using a compact representation (the final-layer hidden state of the latest prefix token) and map it to a distribution over actions. At inference we use greedy selection for reproducibility, while during router training we sample actions for policy optimization. When the router selects at=(st,kt)a_t=(s_t,k_t), we generate memory tokens t=gϕ(t,1:kt(st),kt).M_t=g_φ\! (E_t,Z^(s_t)_1:k_t,k_t ). (9) We append tM_t to the current context, and then predict the next token with the frozen backbone. This yields a controllable inference-time overhead governed by the injection frequency and budgets kt\k_t\. Table 1: Comparison of DLMR with strong baselines on 3 general and 4 reasoning benchmarks using two MLLMs (Qwen2.5-VL-7B and InternVL-3-8B). DLMR (SFT) trains the injector with supervised learning, while DLMR (RL) uses GRPO-based training. Method General Reasoning MMVet MMStar RealWorldQA Avg MMMU MathVerse MathVision MathVista Avg Qwen2.5-VL-7B Vanilla 63.30 62.60 68.10 64.67 53.11 42.64 22.37 68.30 46.61 CoT 64.55 63.73 68.88 65.72 53.78 43.24 23.02 68.80 47.21 CCoT 64.67 64.11 69.12 65.97 53.33 43.31 24.67 67.90 47.30 SFT 64.61 62.98 69.27 65.62 54.16 43.38 23.68 68.90 47.53 GRPO 68.06 65.33 70.41 67.93 57.74 45.39 28.33 69.70 50.29 Visual-RFT 68.77 65.27 71.09 68.38 58.54 45.27 29.12 70.10 50.76 RCTS-RAG 66.02 65.56 69.74 67.11 55.27 44.76 26.94 69.80 49.19 DLMR SFT 74.77 68.26 71.33 71.45 61.00 46.95 35.32 72.10 53.84 DLMR RL 73.85 68.67 70.70 71.07 62.77 48.98 39.34 74.70 56.45 InternVL-3-8B Vanilla 75.08 68.46 70.98 71.51 57.22 37.44 27.30 71.10 48.27 CoT 76.98 70.13 72.04 73.05 58.44 42.13 33.55 71.90 51.51 CCoT 78.02 71.87 73.16 74.35 58.28 41.06 32.14 71.80 50.82 SFT 77.63 69.74 72.75 73.37 58.28 41.06 32.14 71.80 50.82 GRPO 81.55 72.85 73.21 75.87 62.11 45.71 35.81 73.70 54.33 Visual-RFT 82.17 72.51 73.43 76.04 63.78 46.25 35.43 74.60 55.02 RCTS-RAG 78.29 71.62 72.37 74.09 58.94 41.57 33.97 72.10 51.65 DLMR SFT 86.32 76.54 74.88 79.25 67.45 49.31 37.18 77.20 59.04 DLMR RL 85.91 77.10 74.02 79.01 66.33 52.14 38.56 81.30 63.08 4.3 Training Recipe We adopt a three-stage training scheme that progressively learns (i) informative latent-space memories, (i) an effective memory injector, and (i) an adaptive router. Across all stages, the backbone parameters θ are kept frozen; we only optimize lightweight memory-related components. Stage 1: Latent-space memory pre-warm. We initialize the latent-space memories (v)Z^(v) and (r)Z^(r) with a self-supervised alignment objective, while keeping the injector and router fixed (or disabled). Given an input (,)(I,x), we first run the frozen backbone without any routed injection to obtain teacher hidden states, and compute pooled representations that summarize (i) the visual evidence from image tokens and (i) the textual reasoning state from the prefix, denoted (v,r)(t_v,t_r). We then form a student pass by injecting latent-space memories at the prompt end (always visual) and at delimiter-eligible points (Sec. 4.2) using randomized memory types s∈v,rs∈\v,r\ and budgets k∈+k ^+. From the injected memory tokens we obtain pooled student representations (v,r)(z_v,z_r), and optimize only (v)Z^(v) and (r)Z^(r) via a two-teacher alignment loss: ℒstage1= _stage1= ∑s∈v,r(1−cos(s,s)) _s∈\v,r\ (1- (z_s,t_s) ) (10) +λneg∑s≠s′[cos(s,s′)−m]+ + _neg _s≠ s [ (z_s,t_s )-m ]_+ +λsep|cos(v,r)|, + _sep | (z_v,z_r) |, where λneg _neg weights a cross-teacher hinge term (margin m) that penalizes mismatched alignment, and λsep _sep weights a separation term that discourages similarity between vz_v and rz_r. Stage 2: Injector training. We train the injector parameters ϕφ together with the latent-space memories =(v),(r)Z=\Z^(v),Z^(r)\ on downstream multimodal reasoning data, while keeping the router disabled. To make both memories robust under different overhead constraints, we expose the model to mixed injections (e.g., alternating or random) by varying the memory type s∈v,rs∈\v,r\ and budget k∈+k ^+ at eligible points during training. We optimize either (i) supervised next-token likelihood when gold outputs are available, or (i) GRPO when only task-level feedback is available. The objective can be written compactly as maxϕ,(,,)∼[logPθ,ϕ,(∣,)]−ϵℒpreserve, _φ,Z\;E_(I,x,y) [ P_θ,φ,Z(y ,x) ]-ε\,L_preserve, (11) where θ is frozen and ℒpreserveL_preserve is a weak specialization regularizer to prevent the two memories from drifting toward a single shared representation (we use a small cross-branch hinge term plus a separation term, consistent with Stage 1). Stage 3: Router learning. Finally, we freeze the injector and latent-space memories, and train the router policy πψ _ψ to adaptively choose (st,kt)(s_t,k_t) at eligible steps (Sec. 4.2) using GRPO. To balance accuracy and efficiency, we optimize a cost-aware objective: maxψτ∼πψ[Rtask(τ)+λeffReff(τ)]−βKL(πψ∥πref), _ψ\;E_τ _ψ [\,R^task(τ)+ _eff\,R^eff(τ) ]-β\,KL\! ( _ψ\,\|\, _ref ), (12) where τ denotes a sampled decoding trajectory (including routing actions), and πref _ref is a fixed reference router for stabilization. In our implementation, Reff(τ)R^eff(τ) encourages smaller average injection budgets and is only counted when the answer is correct (see Appendix B). 5 Experiments 5.1 Experimental Setup Benchmarks. We evaluate DLMR on a diverse set of vision-language benchmarks that cover both general multimodal understanding and long-horizon reasoning. For general capability, we report results on MMVet (Yu et al., 2023), MMStar (Chen et al., 2024), and RealWorldQA. For reasoning-centric evaluation, we use MMMU (Yue et al., 2024), MathVerse (Zhang et al., 2024a), MathVision (Wang et al., 2024a), and MathVista (Lu et al., 2023). Details are in Appendix C.3. Baselines. We compare DLMR against a set of strong baselines spanning three common paradigms: (i) CoT-based inference-time prompting, including standard CoT and CCoT (Cheng and Van Durme, 2024); (i) training-based post-training methods, including SFT, GRPO (Shao et al., 2024b), and Visual-RFT (Liu et al., 2025a); and (i) RAG-based augmentation represented by RCTS-RAG (Yang et al., 2025). Details are in Appendix C.2. Implementation Details. All experiments are implemented on Qwen2.5-VL-7B (Bai et al., 2025) and InternVL3-8B (Zhu et al., 2025). During the three-stage training procedure, we use the same training dataset. Initially, we include the training split dataset of the selected benchmarks and retain their original data division. For benchmarks without a training phase, we use them solely for evaluation. Additionally, we incorporate the OpenMMReasoner dataset (Zhang et al., 2025b), improving the reasoning. Notably, since Stage 2 admits two training variants (SFT and GRPO), we also report corresponding baseline variants trained with SFT or RL. Figure 3: Disentanglement ablation on Qwen2.5-VL-7B evaluated on six benchmarks (two general: MMVet, MMStar; four reasoning: MMMU, MathVerse, MathVision, MathVista). We compare using the dual latent memory ((v),(r)Z^(v),Z^(r)) and a single shared memory (Z). 5.2 Main Results We first present the main results showing that DLMR improves both general VQA and reasoning benchmarks (Tab. 1). We then conduct three ablations to identify the source of gains: [Abl.1] removes memory disentanglement, which collapses (v),(r)\Z^(v),Z^(r)\ into a single latent set Z (Fig. 3); [Abl.2] disables injector training by keeping the injector gϕg_φ non-trainable while retaining the same injection interface (Tab. 2); and [Abl.3] disables adaptive routing via fixed budgets k∈4,8,16k\!∈\!\4,8,16\ to examine the accuracy–cost trade-off (Tab. 4). DLMR enables reliable long-horizon multimodal reasoning. As shown in Tab. 1, DLMR consistently outperforms prompt-based reasoning, post-training, and retrieval-based baselines, and the gains cannot be attributed solely to stronger prompting or additional parameter updates. Instead, DLMR introduces an explicit memory interface that separates reusable visual evidence from evolving intermediate constraints and selectively reuses them during generation. The improvements are most pronounced on long-horizon reasoning benchmarks (e.g., MathVision/MathVista), where conventional methods often lose early visual cues or drift from intermediate constraints as decoding proceeds; by revisiting the appropriate memory stream on demand, DLMR mitigates such drift and yields larger gains in precisely these challenging regimes. Meanwhile, DLMR also improves general VQA performance, suggesting no trade-off between general perception and reasoning. Empirically, DLMR (SFT) performs best on general benchmarks, while DLMR (RL) achieves the strongest reasoning results (Tab. 1), indicating that SFT stabilizes broad visual grounding whereas RL further strengthens multi-step decision-making and constraint maintenance. Table 2: Injector ablation on Qwen2.5-VL-7B evaluated on four reasoning benchmarks. We compare the full model with a frozen injector variant (no injector training) while keeping the same dual memories and routing/injection interface. Variant MMMU MVerse MathV MVista Avg Frozen injector 58.14 44.32 27.74 69.96 50.44 Trainable injector 61.00 46.95 35.32 72.10 53.84 Table 3: Cross-backbone generalization of DLMR on three additional MLLMs (LLaVA-OV-1.5-4B, Qwen2.5-VL-3B, InternVL-3-2B), evaluated on General and Reasoning benchmarks. ↑ indicates the performance improvement over the corresponding base model. DLMR yields consistent gains across all three backbones, indicating that the proposed memory augmentation is largely base-model agnostic. Base Model General Reasoning MMVet MMStar RealWorldQA Avg MMMU MathVerse MathVision MathVista Avg LLaVA-OV-1.5-4B 66.0 62.6 68.1 64.6 56.0 42.6 22.3 68.3 46.6 + DLMR 75.1 ↑ 9.1 68.9 ↑ 6.3 71.3 ↑ 3.2 71.4 ↑ 6.8 63.9 ↑ 7.9 46.9 ↑ 4.3 35.3 ↑ 13.0 72.1 ↑ 3.8 53.8 ↑ 7.2 Qwen2.5-VL-3B 61.2 55.1 65.3 60.6 51.2 31.2 21.9 61.2 41.4 + DLMR 69.8 ↑ 8.6 60.6 ↑ 5.5 70.1 ↑ 4.8 66.8 ↑ 6.2 58.9 ↑ 7.7 33.2 ↑ 2.0 36.5 ↑ 4.6 68.6 ↑ 7.4 49.3 ↑ 7.9 InternVL-3-2B 65.1 61.3 64.5 63.6 48.6 24.1 21.7 54.7 37.3 + DLMR 70.4 ↑ 5.3 66.1 ↑ 4.8 69.5 ↑ 4.0 68.7 ↑ 5.1 55.3 ↑ 6.7 30.3 ↑ 6.2 29.1 ↑ 8.4 62.1 ↑ 7.4 44.2 ↑ 6.9 [Abl.1] Disentanglement is crucial. We ablate memory disentanglement on Qwen2.5-VL-7B by collapsing the dual buffers into a single shared latent memory Z, and comparing it with the full dual-latent design ((v),(r))(Z^(v),Z^(r)) on six benchmarks (two general and four reasoning). As shown in Fig. 3, the dual-latent design consistently outperforms the shared-memory variant, improving the overall six-benchmark average from 52.05 to 59.73 (+7.68); restricted to the four reasoning benchmarks, the gain widens from 46.61 to 53.84 (+7.23). Notably, the largest single-benchmark gain occurs on the most long-horizon benchmark MathVision (22.37→ 35.32), where models are more prone to losing early visual evidence or drifting from intermediate constraints, suggesting that separating visual evidence and intermediate constraints directly addresses the key failure mode we target. In contrast, collapsing both information types into a single undifferentiated buffer limits specialization and can introduce interference, leading to weaker and less stable performance across tasks. Figure 4: Accuracy–length frontier under different router budget caps kmaxk_ on Qwen2.5-VL-7B. General tasks peak at kmax=16k_ =16, while Reasoning tasks favor a larger headroom and peak at kmax=32k_ =32, indicating that adaptive routing uses high budgets only on hard states. [Abl.2] A learnable injector is necessary. We freeze the memory injector while keeping the same dual memories and injection interface (Tab. 2). This variant still appends latent memories and retrieves the corresponding hidden states as memory tokens, but it can no longer adapt how these latents are contextualized into task-useful signals. Freezing the injector causes a clear performance drop on reasoning benchmarks relative to the full model, showing that where memory is injected is insufficient—a trainable interface is needed to translate latent buffers into actionable, context-aligned tokens. We attribute the degradation to a contextualization mismatch: without training, injected tokens become less informative (or partially redundant), weakening the model’s ability to reliably revisit visual evidence and maintain intermediate constraints in long generations. Overall, this ablation confirms that DLMR’s gains rely on both disentangled memories and a learnable injector that grounds and shapes memory tokens for downstream reasoning. [Abl.3] Adaptive routing yields a better accuracy–cost frontier. We replace state-dependent budget allocation with a fixed injection budget k∈4,8,16k∈\4,8,16\ at every eligible insertion point, while keeping the same dual memories and injector. As shown in Tab. 4, larger fixed budgets substantially increase generation length (avg tokens: 664→732→765664→ 732→ 765 for k=4,8,16k=4,8,16), yet the accuracy is not monotonic. In particular, k=8k=8 achieves the best fixed-budget accuracy (52.71), whereas k=16k=16 produces longer outputs but slightly worse accuracy (52.04), indicating diminishing returns and possible redundancy from over-injection. We further vary the router’s maximum allowable budget kmaxk_ and report the resulting accuracy and cost in Fig. 4. Interestingly, the optimal frontier differs by task family: General benchmarks achieve their best trade-off at kmax=16k_ =16, whereas Reasoning benchmarks benefit from a larger budget headroom and peak at kmax=32k_ =32. This suggests that general VQA queries rarely require aggressive memory injection, while multi-step reasoning problems more often profit from occasionally allocating a larger budget on hard states, with the router still selecting small budgets on easy ones. Table 4: Budget ablation (Qwen2.5-VL-7B). We compare fixed injection budgets k (no adaptive routing) with the full DLMR router. Avg Acc is averaged over four reasoning benchmarks; Avg token length is the average number of generated tokens per sample. Variant Avg Acc Avg token length No router, fixed k=4k=4 51.55 664 No router, fixed k=8k=8 52.71 732 No router, fixed k=16k=16 52.04 765 Full DLMR (adaptive ktk_t) 53.84 677 5.3 Detailed Analysis. We provide additional analyses to better understand when and why DLMR helps. Specifically, we analyze (i) whether DLMR brings backbone-agnostic gains on general vision–language benchmarks, (i) whether the learned memory routing mechanism transfers to text-only reasoning, and (i) whether DLMR improves token efficiency by reducing redundant long-form generations. More analyses on router interpretability, wall-clock inference overhead, and transfer to unseen benchmarks are provided in the appendix A. [Ana.1] Model generalization. Tab. 3 shows that DLMR yields consistent gains across three distinct base MLLMs (LLaVA-OV-1.5-4B, Qwen2.5-VL-3B, and InternVL-3-2B), indicating that the proposed memory augmentation is largely base-model agnostic. On general VQA benchmarks, DLMR improves the average score by +6.8 on LLaVA-OV-1.5-4B, +6.2 on Qwen2.5-VL-3B, and +5.1 on InternVL-3-2B, with consistent gains on MMVet, MMStar, and RealWorldQA. More importantly, DLMR delivers even larger improvements on reasoning-centric benchmarks, boosting the average by +7.2, +7.9, and +6.9, respectively; the gains are particularly pronounced on MMMU and MathVision, suggesting that explicitly modeling reusable visual evidence and evolving reasoning constraints benefits long-horizon multimodal reasoning across diverse base MLLMs. Figure 5: Cross-modality transfer of memory routing to text-only benchmarks (MMLU, GSM8K, ARC-Challenge) on Qwen2.5-VL-7B, compared against the vanilla backbone, GRPO, and Visual-RFT baselines. [Ana.2] Cross-modality transfer of memory routing. Although DLMR is motivated by vision-language reasoning, its core idea—separating reusable context from evolving constraints and enabling state-dependent, budget-aware reuse—is modality-agnostic. To test transfer beyond multimodal inputs, we apply the same routing and injection mechanism to text-only benchmarks (MMLU (Hendrycks et al., 2020), GSM8K (Cobbe et al., 2021) and ARC-Challenge (Clark et al., 2018)) without any vision features. Benchmark details are in Appendix C.3. As shown in Fig. 5, DLMR consistently improves over the vanilla backbone and a GRPO baseline, with the largest gains on multi-step reasoning tasks. In particular, DLMR achieves +15.39+15.39 on GSM8K (68.08→ 83.47) and +11.08+11.08 on ARC-Challenge (44.11→ 55.19), indicating that the learned latent memories and router can preserve and reuse intermediate reasoning state even in purely textual settings. Overall, these results suggest that DLMR provides a transferable mechanism for more reliable long-horizon decoding. Figure 6: Average generated tokens per sample on four reasoning-centric benchmarks using Qwen2.5-VL-7B. By selectively reusing latent memories, DLMR shortens generations compared to strong baselines. [Ana.3] Improved token efficiency via selective memory reuse. We measure token efficiency as the average number of generated tokens per sample on reasoning benchmarks. For completeness, we include the tokens used to generate the latent memories (both visual and reasoning) when computing DLMR’s total generated length. As shown in Fig. 6, DLMR produces shorter or comparable generations to strong baselines and achieves the best overall average token efficiency. In particular, DLMR reduces average generated tokens by 9.7% compared to Visual-RFT (the closest competitor in capability), and by 4.2% compared to SFT (the closest baseline in token usage), indicating a more compact decoding trajectory by replacing redundant textual re-derivation with selective latent revisitation. 6 Conclusion We presented DLMR, a parameter-efficient memory augmentation mechanism for long-horizon vision–language reasoning. DLMR equips a frozen MLLM with dual latent memories that disentangle reusable visual evidence from evolving reasoning constraints, and couples them with a lightweight injector plus a discrete, state-dependent router that selects what to reuse and how much to inject at each step. With a simple three-stage training recipe, DLMR delivers consistent gains on both general VQA and reasoning benchmarks across multiple backbones, with the largest improvements on tasks that demand reliable visual revisiting and constraint maintenance over long generations. Beyond multimodal settings, DLMR transfers to text-only reasoning and improves token efficiency, while the learned router exhibits interpretable, budget-aware behavior that concentrates computation on harder states. Acknowledgements This work was supported in part by the National Key Research and Development Program of China (No. 2024YFE0202800), the Basic Research Program of Jiangsu under Grant No. BK20253021, the National Natural Science Foundation of China (NSFC) under Grants No. 62522605 and No. 62376118, the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (No. JYB2025XDXM118), the “111 Center” (No. B26023), and the Collaborative Innovation Center of Novel Software Technology and Industrialization. Impact Statement This paper presents Dual-Latent Memory Routing (DLMR), a parameter-efficient memory augmentation for multimodal large language models (MLLMs) that aims to improve long-horizon vision–language reasoning by storing reusable visual evidence and intermediate reasoning constraints in disentangled latent memories and selectively re-injecting them during generation. If deployed responsibly, DLMR can benefit applications such as education and tutoring on visual problems, accessibility tools that require grounded image understanding, and scientific/engineering assistants that must preserve intermediate constraints across long solutions; selective reuse may also reduce unnecessary generation, lowering latency and energy under fixed inference budgets. However, improved grounding and long-horizon reasoning can also increase capability in harmful settings (e.g., surveillance enablement or more convincing misinformation), and stronger-looking outputs may be over-trusted despite remaining hallucinations, biases, or brittleness under distribution shift inherited from the backbone and data. As with many vision–language methods, data provenance may raise privacy or copyright concerns depending on training and deployment practices. We position DLMR as a research mechanism rather than an autonomous decision system for high-stakes use, and recommend standard safeguards in any deployment, including human oversight, dataset documentation and bias auditing, privacy-preserving data handling, and safety filtering/policy enforcement to mitigate misuse. References Agarwal et al. (2025) S. Agarwal, L. Ahmad, J. Ai, S. Altman, A. Applebaum, E. Arbus, R. K. Arora, Y. Bai, B. Baker, H. Bao, et al. Gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925. Cited by: §2.1. Bai et al. (2025) S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: §C.2, §2.1, §5.1. Chen et al. (2024) L. Chen, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin, et al. Are we on the right way for evaluating large vision-language models?. Advances in Neural Information Processing Systems 37, p. 27056–27087. Cited by: Table A4, §5.1. Cheng and Van Durme (2024) J. Cheng and B. Van Durme Compressed chain of thought: efficient reasoning through dense representations. arXiv preprint arXiv:2412.13171. Cited by: §C.2, §5.1. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: Table A4, §5.3. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: Table A4, §5.3. Gao et al. (2025) J. Gao, Y. Li, Z. Cao, and W. Li Interleaved-modal chain-of-thought. In CVPR, p. 19520–19529. Cited by: §2.1. Gao et al. (2024) T. Gao, P. Chen, M. Zhang, C. Fu, Y. Shen, Y. Zhang, S. Zhang, X. Zheng, X. Sun, L. Cao, et al. Cantor: inspiring multimodal chain-of-thought of mllm. In ACM M, p. 9096–9105. Cited by: §1, §2.1. Hao et al. (2025) H. Hao, J. Han, C. Li, Y. Li, and X. Yue RAP: retrieval-augmented personalization for multimodal large language models. In CVPR, p. 14538–14548. Cited by: §2.1. Hendrycks et al. (2020) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. Cited by: Table A4, §5.3. Hsieh et al. (2024) C. Hsieh, Y. Chuang, C. Li, Z. Wang, L. Le, A. Kumar, J. Glass, A. Ratner, C. Lee, R. Krishna, et al. Found in the middle: calibrating positional attention bias improves long context utilization. In Findings of the Association for Computational Linguistics: ACL 2024, p. 14982–14995. Cited by: §3. Hu et al. (2025) M. Hu, T. Chen, Q. Chen, Y. Mu, W. Shao, and P. Luo Hiagent: hierarchical working memory management for solving long-horizon agent tasks with large language model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 32779–32798. Cited by: §2.2. Hurst et al. (2024) A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. Cited by: §C.2. Jiang et al. (2025) J. Jiang, C. Ma, X. Song, H. Zhang, and J. Luo Corvid: improving multimodal large language models towards chain-of-thought reasoning. In ICCV, p. 3034–3046. Cited by: §2.1. Kim et al. (2025) G. I. Kim, J. W. Kim, and B. Jang UniRAG: a unified rag framework for knowledge-intensive queries with decomposition, break-down reasoning, and iterative rewriting. In EMNLP, p. 18795–18810. Cited by: §2.1, §2.2. Li et al. (2025) Y. Li, Z. Lan, and J. Zhou Text or pixels? it takes half: on the token efficiency of visual text inputs in multimodal llms. arXiv preprint arXiv:2510.18279. Cited by: §2.1. Liu et al. (2024) N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang Lost in the middle: how language models use long contexts. Transactions of the association for computational linguistics 12, p. 157–173. Cited by: §3. Liu et al. (2025a) Z. Liu, Z. Sun, Y. Zang, X. Dong, Y. Cao, H. Duan, D. Lin, and J. Wang Visual-rft: visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785. Cited by: §C.2, §1, §5.1. Liu et al. (2025b) Z. Liu, Y. Du, T. Fu, S. Su, C. Ho, and C. Wang Vision-language memory for spatial reasoning. arXiv preprint arXiv:2511.20644. Cited by: §2.2. Long et al. (2025) L. Long, Y. He, W. Ye, Y. Pan, Y. Lin, H. Li, J. Zhao, and W. Li Seeing, listening, remembering, and reasoning: a multimodal agent with long-term memory. arXiv preprint arXiv:2508.09736. Cited by: §2.2. Lu et al. (2023) P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K. Chang, M. Galley, and J. Gao Mathvista: evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255. Cited by: Table A4, §5.1. Lu et al. (2025) S. Lu, Y. Li, Y. Xia, Y. Hu, S. Zhao, Y. Ma, Z. Wei, Y. Li, L. Duan, J. Zhao, et al. Ovis2. 5 technical report. arXiv preprint arXiv:2508.11737. Cited by: §2.1. Ming and Li (2024) Y. Ming and Y. Li Understanding retrieval-augmented task adaptation for vision-language models. arXiv preprint arXiv:2405.01468. Cited by: §2.1. Peng et al. (2025) Y. Peng, G. Zhang, M. Zhang, Z. You, J. Liu, Q. Zhu, K. Yang, X. Xu, X. Geng, and X. Yang Lmm-r1: empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536. Cited by: §2.1. Ren et al. (2025) S. Ren, Q. Yu, J. He, X. Shen, A. Yuille, and L. Chen Beyond next-token: next-x prediction for autoregressive visual generation. arXiv preprint arXiv:2502.20388. Cited by: §1. Shao et al. (2024a) H. Shao, S. Qian, H. Xiao, G. Song, Z. Zong, L. Wang, Y. Liu, and H. Li Visual cot: advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. NeurIPS 37, p. 8612–8642. Cited by: §1, §2.1. Shao et al. (2024b) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §C.2, §5.1. Shen et al. (2025) H. Shen, P. Liu, J. Li, C. Fang, Y. Ma, J. Liao, Q. Shen, Z. Zhang, K. Zhao, Q. Zhang, et al. Vlm-r1: a stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615. Cited by: §2.1. Sun et al. (2025) H. Sun, Z. Sun, H. Peng, and H. Ye Mitigating visual forgetting via take-along visual conditioning for multi-modal long cot reasoning. arXiv preprint arXiv:2503.13360. Cited by: §1, §2.1. Tanaka et al. (2025) R. Tanaka, T. Iki, T. Hasegawa, K. Nishida, K. Saito, and J. Suzuki Vdocrag: retrieval-augmented generation over visually-rich documents. In CVPR, p. 24827–24837. Cited by: §2.1. Tu et al. (2026) C. Tu, P. Ye, D. Zhou, L. Bai, G. Yu, T. Chen, and W. Ouyang Attention reallocation: towards zero-cost and controllable hallucination mitigation of mllms. IJCV 134 (1), p. 22. Cited by: §1, §2.1. Wang et al. (2024a) K. Wang, J. Pan, W. Shi, Z. Lu, H. Ren, A. Zhou, M. Zhan, and H. Li Measuring multimodal mathematical reasoning with math-vision dataset. NeurIPS 37, p. 95095–95169. Cited by: Table A4, §5.1. Wang et al. (2024b) P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. Cited by: §C.2. Wen et al. (2025) X. Wen, Z. Liu, S. Zheng, S. Ye, Z. Wu, Y. Wang, Z. Xu, X. Liang, J. Li, Z. Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms. arXiv preprint arXiv:2506.14245. Cited by: §1, §2.1. Wolpert et al. (2011) D. M. Wolpert, J. Diedrichsen, and J. R. Flanagan Principles of sensorimotor learning. Nature reviews neuroscience 12 (12), p. 739–751. Cited by: §1. Wu et al. (2025a) J. Wu, Y. Xiong, X. Li, Y. Xia, R. Wang, Y. Wang, T. Yu, S. Kim, R. A. Rossi, L. Yao, et al. Mitigating visual knowledge forgetting in mllm instruction-tuning via modality-decoupled gradient descent. arXiv preprint arXiv:2502.11740. Cited by: §2.1. Wu et al. (2025b) Q. Wu, X. Yang, Y. Zhou, C. Fang, B. Song, X. Sun, and R. Ji Grounded chain-of-thought for multimodal large language models. arXiv preprint arXiv:2503.12799. Cited by: §2.1. Wu et al. (2025c) Y. Wu, Y. Zhou, Z. Ziheng, Y. Peng, X. Ye, X. Hu, W. Zhu, L. Qi, M. Yang, and X. Yang On the generalization of sft: a reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508.05629. Cited by: §2.1. Xu et al. (2025a) G. Xu, P. Jin, Z. Wu, H. Li, Y. Song, L. Sun, and L. Yuan Llava-cot: let vision language models reason step-by-step. In ICCV, p. 2087–2098. Cited by: §1, §2.1. Xu et al. (2025b) X. Xu, H. Chen, M. Lyu, S. Zhao, Y. Xiong, Z. Lin, J. Han, and G. Ding Mitigating hallucinations in multi-modal large language models via image token attention-guided decoding. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 1571–1590. Cited by: §3. Yan et al. (2025) S. Yan, X. Yang, Z. Huang, E. Nie, Z. Ding, Z. Li, X. Ma, K. Kersting, J. Z. Pan, H. Schütze, et al. Memory-r1: enhancing large language model agents to manage and utilize memories via reinforcement learning. arXiv preprint arXiv:2508.19828. Cited by: §2.2. Yang et al. (2025) Q. Yang, C. Zhang, L. Fan, K. Ding, J. Ye, and S. Xiang Re-ranking reasoning context with tree search makes large vision-language models stronger. arXiv preprint arXiv:2506.07785. Cited by: §C.2, §5.1. Yu et al. (2024a) S. Yu, C. Tang, B. Xu, J. Cui, J. Ran, Y. Yan, Z. Liu, S. Wang, X. Han, Z. Liu, et al. Visrag: vision-based retrieval-augmented generation on multi-modality documents. arXiv preprint arXiv:2410.10594. Cited by: §2.1, §2.2. Yu et al. (2024b) T. Yu, Y. Yao, H. Zhang, T. He, Y. Han, G. Cui, J. Hu, Z. Liu, H. Zheng, M. Sun, et al. Rlhf-v: towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In CVPR, p. 13807–13816. Cited by: §1, §2.1. Yu et al. (2023) W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang Mm-vet: evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490. Cited by: Table A4, §5.1. Yu et al. (2025) X. Yu, C. Xu, G. Zhang, Z. Chen, Y. Zhang, Y. He, P. Jiang, J. Zhang, X. Hu, and S. Yan Vismem: latent vision memory unlocks potential of vision-language models. arXiv preprint arXiv:2511.11007. Cited by: §2.2. Yue et al. (2024) X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. Mmmu: a massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, p. 9556–9567. Cited by: Table A4, §5.1. Zeng et al. (2025) S. Zeng, D. Qi, X. Chang, F. Xiong, S. Xie, X. Wu, S. Liang, M. Xu, and X. Wei Janusvln: decoupling semantics and spatiality with dual implicit memory for vision-language navigation. arXiv preprint arXiv:2509.22548. Cited by: §2.2. Zhang et al. (2025a) G. Zhang, M. Fu, and S. Yan Memgen: weaving generative latent memory for self-evolving agents. arXiv preprint arXiv:2509.24704. Cited by: §2.2. Zhang et al. (2025b) K. Zhang, K. Wu, Z. Yang, B. Li, K. Hu, B. Wang, Z. Liu, X. Li, and L. Bing Openmmreasoner: pushing the frontiers for multimodal reasoning with an open and general recipe. arXiv preprint arXiv:2511.16334. Cited by: §5.1. Zhang et al. (2024a) R. Zhang, D. Jiang, Y. Zhang, H. Lin, Z. Guo, P. Qiu, A. Zhou, P. Lu, K. Chang, Y. Qiao, et al. Mathverse: does your multi-modal llm truly see the diagrams in visual math problems?. In ECCV, p. 169–186. Cited by: Table A4, §5.1. Zhang et al. (2024b) Y. Zhang, W. Zhu, H. Tang, Z. Ma, K. Zhou, and L. Zhang Dual memory networks: a versatile adaptation approach for vision-language models. In CVPR, p. 28718–28728. Cited by: §2.2. Zhou et al. (2025) D. Zhou, Y. Zhang, Y. Wang, J. Ning, H. Ye, D. Zhan, and Z. Liu Learning without forgetting for vision-language models. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §1, §2.1. Zhu et al. (2025) J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y. Duan, W. Su, J. Shao, et al. Internvl3: exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479. Cited by: §C.2, §5.1. This supplementary material provides additional details supporting the main paper, organized as follows: • Section A: Additional Experiments and Analyses. Extended experimental results and ablations beyond those reported in the main paper. • Section B: Method details. Additional formulations and algorithmic descriptions of the proposed routing framework, including objective definitions and design rationales that complement the main text. • Section C: Training details. Full training configurations and hyperparameters, data preprocessing, optimization settings, and implementation choices to ensure reproducibility. Appendix A More experiments Interpretable routing behavior. Fig. A1 visualizes the router’s injection frequency over the relative generation position, separately aggregated on reasoning benchmarks (top) and general benchmarks (bottom). A clear and intuitive pattern emerges: on reasoning tasks, the router invokes the reasoning memory more frequently than the visual memory, whereas on general tasks it preferentially activates the visual memory. This aligns with the differing information demands across task families: multi-step reasoning requires repeatedly revisiting intermediate constraints and partial conclusions, while general perception-heavy queries benefit more from re-accessing detailed visual evidence. Beyond the overall preference, the curves also exhibit position-dependent dynamics: both memories are injected more often in the earlier and middle stages of decoding and gradually taper off near the end, suggesting that the router concentrates memory reuse when establishing key evidence/constraints and relies less on injection once the answer becomes determinate. Inference efficiency on the benchmark suite. Tab. A1 reports end-to-end inference time and accuracy aggregated over the General and Reasoning benchmark suites, evaluated on two backbones. Accuracy is the average over the three General benchmarks (MMVet, MMStar, RealWorldQA) and the four Reasoning benchmarks (MMMU, MathVerse, MathVision, MathVista); time is the average end-to-end inference latency per sample. Across both models, DLMR consistently improves accuracy while maintaining (or reducing) inference time relative to strong SFT baselines. On Qwen2.5-VL-7B, DLMR improves general accuracy from 65.6%65.6\% to 71.4%71.4\% while slightly reducing time (→5.45.6\!→\!5.4s), and yields a larger gain on reasoning (47.5%→53.8%47.5\%\!→\!53.8\%) together with a notable speedup (→11.514.0\!→\!11.5s, ≈ 18%). On InternVL3-8B, DLMR similarly boosts accuracy on both general (73.4%→79.3%73.4\%\!→\!79.3\%) and reasoning (50.8%→59.0%50.8\%\!→\!59.0\%) with comparable latency to SFT (3.53.5 vs. 3.73.7s; 13.413.4 vs. 13.113.1s). These results suggest that DLMR’s selective latent reuse can reduce redundant token-level re-derivation in long generations: it preserves or improves wall-clock efficiency while strengthening correctness, especially on multi-step reasoning where repeated constraint maintenance would otherwise inflate decoding time. Figure A1: Router frequency over generation position. We plot the routing ratio (injection frequency) of the visual memory and the reasoning memory as a function of relative decoding position, aggregated over reasoning benchmarks (top) and general benchmarks (bottom). The router invokes reasoning memory more often on reasoning tasks, while it prefers visual memory on general tasks, indicating task-adaptive and interpretable routing. Table A1: Inference time and accuracy on the General/Reasoning benchmark suites. Accuracy is averaged over the three General benchmarks (MMVet, MMStar, RealWorldQA) and the four Reasoning benchmarks (MMMU, MathVerse, MathVision, MathVista) from Tab. 1; time is the average end-to-end inference time per sample. Model & Method General Reasoning Time (s) Acc (%) Time (s) Acc (%) Qwen2.5-VL-7B Vanilla 10.1 64.7 28.5 46.6 SFT 5.6 65.6 14.0 47.5 DLMR 5.4 71.4 11.5 53.8 InternVL3-8B Vanilla 8.4 71.5 21.1 48.3 SFT 3.5 73.4 13.4 50.8 DLMR 3.7 79.3 13.1 59.0 Table A2: Transfer to unseen benchmarks. We evaluate models trained under the default setting on unseen datasets spanning reasoning (LogicVista) and general perception/OCR (OCR-Bench, MMT-Bench). All numbers are task scores in % (higher is better). Method LogicVista OCR-Bench MMT-Bench Vanilla 43.1 86.1 51.5 CoT 47.5 89.1 55.0 SFT 49.0 90.1 56.5 GRPO 50.2 88.4 58.0 DLMR 55.0 92.0 62.5 Transfer to unseen benchmarks. Tab. A2 evaluates whether the learned memory routing generalizes beyond the training distribution on unseen benchmarks, covering a reasoning dataset (LogicVista) and two perception/OCR datasets (OCR-Bench, MMT-Bench). We compare against prompting baselines (Vanilla, CoT) and training-based baselines (SFT, GRPO). Overall, DLMR achieves the best performance on all three unseen datasets, improving LogicVista to 55.055.0 (vs. 50.250.2 for GRPO), OCR-Bench to 92.092.0 (vs. 90.190.1 for SFT), and MMT-Bench to 62.562.5 (vs. 58.058.0 for GRPO), indicating that the disentangled memories transfer to both reasoning and perception/OCR tasks outside the training distribution. Appendix B Methodology Details This appendix provides additional implementation and modeling details for DLMR to complement Secs. 4.1–4.3. B.1 Injection Eligibility and Budget Cap Delimiter-based eligibility. We restrict routed injections to a subset of steps to improve stability and control overhead. Let D be a small delimiter set (e.g., =,,., D=\ ,,\, .,\, n\). At decoding step t, we mark the step as eligible if the decoded prefix <ty_<t ends with any pattern in D, i.e., t∈(<t)⇔∃d∈:<t ends with d.t (y_<t) ∃ d :\;y_<t ends with d. (13) If t∉(<t)t (y_<t), we deterministically perform no injection. Per-sample injection cap. To avoid excessive overhead, we cap the number of routed injections to at most NmaxN_ per sample. Once the cap is reached, the router is no longer queried and decoding proceeds without further injections. B.2 Injection Placement Fixed visual injection at prompt end. In addition to routed injections during decoding, we apply a fixed prompt-end injection using visual latents (s=vs=v) to provide stable perceptual cues. Importantly, for chat-style MLLMs, we insert the prompt-end memory tokens before the assistant marker (rather than after it). This keeps the final token immediately preceding generation a real text token from the prompt, while still exposing the injected memory tokens to the subsequent autoregressive pass. B.3 Router Action Space and Parameterization Discrete action encoding. Let +=k(1),…,k(m)K^+=\k^(1),…,k^(m)\ be the candidate budget set. We define a routed action set =v,r×+A=\v,r\×K^+ with ||=2m|A|=2m actions. At each eligible step, the router outputs either a null decision (no injection) or an action at=(st,kt)∈a_t=(s_t,k_t) . For implementation convenience, we index actions by a single integer a∈0,…,2m−1a∈\0,…,2m-1\: s(a)=v,a<m,r,a≥m,k(a)=k(1+(amodm)).s(a)= casesv,&a<m,\\ r,&a≥ m, cases k(a)=k^(1+(a m)). (14) Policy network. We parameterize the routing policy as πψ(at∣t) _ψ(a_t _t) using a lightweight classifier head on top of the backbone hidden states. Concretely, at step t we take the final-layer hidden state of the most recent prefix token (or an equivalent pooled representation) and map it to 2m2m logits via a linear layer. We optionally enable LoRA adapters on the router branch for parameter efficiency while keeping the backbone weights θ frozen. Sampling vs. greedy. During router training, we sample actions from πψ _ψ to enable policy optimization. At inference time, we use greedy action selection for reproducibility. B.4 Memory Injector Architecture Injector as a LoRA-augmented copy. We implement the injector gϕg_φ as a copy of the backbone with LoRA adapters enabled in the injector branch. Given current context embeddings t∈ℝLt×dE_t ^L_t× d, selected latents (s)1:k∈ℝk×dZ^(s)_1:k ^k× d, and budget k, we form ~t(w)=[Pθ→w(t);1:k(s)]∈ℝ(Lt+k)×dw, E^(w)_t=[P_θ→ w(E_t);Z^(s)_1:k] ^(L_t+k)× d_w, (15) run the injector Transformer to obtain ~t(w) H^(w)_t, and extract the last k states as contextualized memory tokens. If dw≠d_w≠ d, we use learned linear projections Pθ→wP_θ→ w and Pw→θP_w→θ to map between embedding spaces. Cache invalidation. When memory tokens are injected mid-generation, the effective prefix changes. In practice we reset the key–value cache after an injection to ensure correctness, and continue decoding from the augmented prefix. B.5 Training Details Stage 1: teacher–student pooling. We compute teacher representations (v,r)(t_v,t_r) by running the frozen backbone without injection and pooling hidden states over (i) image-token positions and (i) non-image (text) positions, respectively. We compute student representations (v,r)(z_v,z_r) by pooling the injected memory-token hidden states corresponding to the visual and reasoning latent banks. We use a masked mean pooling: pool(,)≜∑imii∑imi,pool(H,m) _im_iH_i _im_i, (16) where m is a binary mask over sequence positions. Stage 1: injection mixing and budget coverage. To make both banks usable under different compute budgets, in the student pass we always inject visual latents at the prompt end, and at delimiter-eligible points we alternate (or randomly sample) the latent type s∈v,rs∈\v,r\ and the budget k∈+k ^+. This matches the inference-time interface while avoiding dependence on any supervised output labels. Stage 2: preserve regularizer. During injector training, we optionally add a weak specialization regularizer ℒpreserve≜ _preserve [cos(v,r)−m]+ [ (z_v,t_r)-m ]_+ (17) +[cos(r,v)−m]+ + [ (z_r,t_v)-m ]_+ +|cos(v,r)|, + | (z_v,z_r) |, which mirrors Stage 1 by discouraging cross-branch confusion and collapse while allowing task learning to dominate. Stage 3: GRPO for routing. We freeze the injector and both latent banks, and optimize the router with GRPO. For each input, we sample a group of trajectories under the current policy, compute a task reward and an efficiency reward, and form group-relative advantages for policy updates. We include a KL penalty to a reference policy πref _ref for stabilization, and only query the router at delimiter-eligible steps until the injection cap is reached. Efficiency reward. We define ReffR^eff to prefer smaller budgets and apply it only when the model is correct: for example, if kmax=max(+)k_ = (K^+) and k¯ k is the average injected budget over the trajectory, we use Reff(τ)=Rtask(τ)=1⋅(1−k¯(τ)/kmax).R^eff(τ)=I\R^task(τ)=1\· (1- k(τ)/k_ ). (18) Appendix C Training Details This section describes full training configurations, data preprocessing, optimization, and implementation choices used in our experiments. We omit the specific backbone identity (model name / parameter count) since our method is backbone-agnostic. C.1 Configuration and Reproducibility Configuration system. All experiments are specified by a YAML configuration and optional command-line overrides. Key model-level knobs include the prompt-end injection length MvM_v, the per-step routed injection length MrM_r, the budget set +K^+, and the maximum number of routed injections NmaxN_ . Seeding. We fix a global random seed (default: 4242) for Python, NumPy, and PyTorch, and enable deterministic CuDNN where applicable. For stochastic training procedures (e.g., GRPO rollouts), we keep sampling enabled and report average results across the evaluation split. Precision and distributed training. We train with mixed-precision activations (BF16 where supported) and use the accelerate launcher with DeepSpeed ZeRO stage-2 for data parallel training. We keep the backbone frozen throughout; only memory-related parameters are trainable. Backbone architectures. Tab. A3 summarizes the key architectural hyperparameters of the two backbone MLLMs used in our main experiments. Table A3: Backbone model configurations. Key architectural hyperparameters of the two backbone MLLMs used in our experiments/analysis: Qwen2.5-VL-7B-Instruct and InternVL3-8B. Configurations Parameters Qwen2.5-VL-7B InternVL3-8B Text backbone hidden_size d 3584 3584 num_layers L 28 28 num_heads H 28 28 num_kv_heads HkvH_kv 4 4 mlp_intermediate dffd_f 18944 18944 max_position_emb 128000 32768 rope_theta 10610^6 10610^6 Vision backbone vision_hidden_size 1280 1024 vision_num_layers 32 24 vision_num_heads 16 16 vision_mlp_intermediate 3420 4096 image_size 448 448 patch_size 14 14 Tokenization vocab_size 152064 151674 dtype / attn_impl bfloat16 / SDPA bfloat16 / SDPA C.2 Baselines We select a total of 7 baselines, including the vanilla model; 2 CoT-based inference-time prompting methods: CoT, CCoT (Cheng and Van Durme, 2024); 3 direct training-based post-training methods: SFT, GRPO (Shao et al., 2024b), Visual-RFT (Liu et al., 2025a); 1 RAG-based augmentation method: RCTS-RAG (Yang et al., 2025). All the baselines are implemented on two base models: Qwen2.5-VL-7B (Bai et al., 2025) and InternVL3-8B (Zhu et al., 2025). For strategies initially implemented on other base models, e.g. GPT-4o (Hurst et al., 2024) and Qwen2- VL(Wang et al., 2024b), we transfer them to Qwen2.5-VL-7B (Bai et al., 2025) and InternVL3-8B (Zhu et al., 2025) for fair comparison. C.3 Data Format and Preprocessing Benchmarks. We evaluate on a diverse suite of vision-language benchmarks spanning general multimodal understanding and long-horizon reasoning, plus three text-only benchmarks for cross-modality transfer. Tab. A4 summarizes the benchmarks; we follow each benchmark’s official evaluation protocol. Table A4: Evaluation benchmarks. # Eval refers to the size of the split we evaluate on. Benchmark Domain # Eval Answer fmt. Metric General (vision–language) MMVet (Yu et al., 2023) Composite VL skills 218 Open-ended LLM-judge acc. MMStar (Chen et al., 2024) Vision-indispensable VL 1,500 MCQ Accuracy RealWorldQA Real-world VQA 765 MCQ / short ans. Accuracy Reasoning (vision–language) MMMU (Yue et al., 2024) College-level multidisciplinary 900 (val) MCQ / open Accuracy MathVerse (Zhang et al., 2024a) Visual math (diagram variants) 3,940 (testmini) Open-ended Accuracy MathVision (Wang et al., 2024a) Competition math w/ figures 3,040 Open-ended Accuracy MathVista (Lu et al., 2023) Visual math (testmini) 1,000 MCQ / open Accuracy Text-only (cross-modality transfer) MMLU (Hendrycks et al., 2020) Knowledge & reasoning (57 subjects) 14,042 MCQ Accuracy GSM8K (Cobbe et al., 2021) Grade-school math word problems 1,319 Open-ended Exact match ARC-Challenge (Clark et al., 2018) Hard science QA 1,172 MCQ Accuracy Dataset splits. For datasets that provide official train/validation/test splits, we keep the original splits unchanged. For benchmarks that do not provide a training split (or are intended as evaluation-only), we use them solely for evaluation. For datasets used for training but lacking a predefined validation set, we reserve a small portion of the training split for validation. We optionally subsample a fixed number of examples for quick ablations. Training corpus. Across the three-stage training procedure, we use a unified training corpus that (i) aggregates the available training splits of the selected benchmarks and (i) incorporates the OpenMMReasoner dataset to strengthen reasoning skills. Stage 2 admits two variants (SFT and GRPO), and we report corresponding baseline variants trained with SFT or RL on the same training corpus and with matched decoding configurations when applicable. Multimodal input representation. Each example provides either (i) a plain-text instruction prompt or (i) a chat-style message list with interleaved text and image references. Images are loaded from a dataset-specific root directory and passed to the MLLM processor to obtain visual tensors. If an example references a missing image and a blank placeholder image is available in the dataset directory, we fall back to the placeholder for robust training; otherwise we raise an error. Tokenization and supervision masks (SFT). We build a full training sequence using the backbone chat template, and construct labels by masking out: (i) all tokens that belong to the prompt (only the completion is supervised), (i) padding tokens, and (i) special vision placeholder tokens used by the MLLM tokenizer. This yields standard next-token cross-entropy on the assistant completion tokens. Length filtering. To avoid out-of-memory issues and unstable delimiter injection, we filter out samples whose tokenized prompt length exceeds a stage-specific maximum length. For stage-1 cold-start we set a larger maximum length (e.g., 20482048 tokens); for GRPO we constrain the prompt length by max_prompt_length. C.4 Model and Injection Hyperparameters Table A5: Hyperparameters across three training stages. Stage 1: latent cold-start (optimize latent banks only). Stage 2: injector SFT (PEFT, backbone frozen). Stage 3: router RL (GRPO, backbone frozen). Values shared across both backbones (Qwen2.5-VL-7B and InternVL3-8B) unless stated otherwise. Batch sizes are per-device; gradient accumulation is used when needed. Configurations Parameters Stage 1 Stage 2 Stage 3 Core backbone Qwen2.5-VL-7B / InternVL3-8B Qwen2.5-VL-7B / InternVL3-8B Qwen2.5-VL-7B / InternVL3-8B visual_latents_len MvM_v 8 8 – reasoning_latents_len MrM_r 16 16 – max_visual_aug_num 1 1 1 max_reasoning_aug_num NmaxN_ 5 5 5 random_reasoning_aug True – – reasoning_latents_source – alternate – reasoning_latents_budgets – [4, 8, 16] – reasoning_latents_visual_prob – 0.5 – action_budget_set – – [4, 8, 16] num_actions |||A| – – 6 (Visual/Reasoning × 4,8,16) router_trainset – – Random 50% subset of train split router_validset – – Random 50% subset of valid split Loss neg_weight 1.0 – – sep_weight 0.1 – – margin 0.2 – – Augmentation delimiter_set [,, ., ] – – budget_set [4, 8, 16] – – branch_choice alternate (visual/reasoning) – – Preserve sft_preserve_epsilon – 0.001 – sft_preserve_margin – 0.2 – LoRA rank r 16 16 16 α 32 32 32 drop_out_rate 0.1 0.1 0.1 target_module [q_proj, v_proj] [q_proj, v_proj] [q_proj, v_proj] Training method Self-supervised alignment SFT GRPO max_length 2048 2048 – max_prompt_length – – 1024 max_completion_length – – 1024 batch_size 1 – – per_device_batch_size – 1 4 grad_accum 1 1 – epoch 3 8 2 num_generations (group size) – – 8 num_iterations – – 1 loss_type – – BNPO kl_coefficient β – – 0.0 efficiency_reward_weight λeff _eff – – 1.0 temperature – – 1.0 learning_rate 1×10−31× 10^-3 1×10−51× 10^-5 1×10−51× 10^-5 warmup_ratio 0.1 0.1 0.1 optimizer AdamW AdamW AdamW scheduler Cosine Cosine Cosine precision bf16 bf16 bf16 deepspeed ZeRO-2 ZeRO-2 ZeRO-2 attention_impl SDPA SDPA SDPA Latent banks and budgets. We maintain two learnable latent banks with sizes MvM_v (visual) and MrM_r (reasoning). At each injection, we select the first k latents from the chosen bank with k∈+k ^+. We use a small discrete budget set, typically +=4,8,16K^+=\4,8,16\. Injection cap and eligibility. We cap the number of routed injections per sample to NmaxN_ (e.g., Nmax=5N_ =5). Router queries are restricted to delimiter-eligible steps defined by a small delimiter set D (e.g., punctuation and newline). LoRA configuration (injector and router). We use parameter-efficient LoRA adapters for both the injector branch and the router branch. Unless stated otherwise, we use rank r=16r=16, scaling α=32α=32, dropout 0.10.1, and apply LoRA to attention projection modules (e.g., query/key/value projections) while keeping all backbone weights frozen. C.5 Stage 1: Latent Cold-Start Trainable parameters. We optimize only the two latent banks (v)Z^(v) and (r)Z^(r); LoRA adapters and router parameters are disabled. Objective and pooling. We compute teacher representations from a forward pass without injection and pool hidden states over image-token positions and non-image positions to obtain (v,r)(t_v,t_r). We compute student representations (v,r)(z_v,z_r) by pooling over injected memory-token positions corresponding to each bank. We optimize the two-teacher alignment loss in Eq. (6) with hyperparameters (λneg,λsep,m)( _neg, _sep,m). Injection schedule during cold-start. We always inject visual latents at the prompt end. At delimiter-eligible points, we alternate (or randomly sample) the memory type s∈v,rs∈\v,r\ and budget k∈+k ^+ to ensure both banks learn to function under different compute budgets. Optimization. We use AdamW with a cosine learning-rate schedule and linear warmup. A typical setting is: learning rate 10−310^-3, warmup ratio 0.10.1, batch size 11 (per-device), and 3 epochs. We disable gradient checkpointing in this stage for stability with multimodal processors, but it can be enabled if memory becomes a bottleneck. C.6 Stage 2: Injector Training (SFT/GRPO) Trainable parameters. We enable the injector LoRA adapters and jointly optimize injector parameters ϕφ and latent banks Z. The router is disabled. SFT settings. We optimize next-token cross-entropy on completion tokens with AdamW and cosine schedule. Typical settings include: learning rate 10−510^-5, warmup ratio 0.10.1, BF16, maximum sequence length 10241024–20482048, and stage-specific batch sizes with gradient accumulation to match the available hardware. Mixed injections during training. To match inference-time behavior, we inject at the prompt end and at delimiter-eligible points. At eligible points, we mix visual and reasoning injections (alternating or random) and sample budgets k∈+k ^+. Specialization preserve regularizer. Optionally, we apply a small-weight regularizer to discourage the two banks from collapsing into the same representation. In our implementation, this corresponds to a hinge-based cross-branch penalty with margin m plus an inter-bank separation term, scaled by ϵε (e.g., ϵ=10−3ε=10^-3). GRPO settings (if used for injector learning). When training the injector with GRPO, we generate multiple rollouts per prompt (num_generations), compute group-relative advantages, and include an optional KL penalty to a reference policy for stabilization. We set generation sampling enabled during training (nonzero temperature) to ensure reward variance within each prompt group. C.7 Stage 3: Router Training (GRPO) Trainable parameters. We freeze the injector and latent banks and optimize only the router parameters ψ (classification head and optional router LoRA adapters). Action space. The router predicts a discrete action over =v,r×+A=\v,r\×K^+ at eligible steps; non-eligible steps use a null action. GRPO hyperparameters. We use group sampling with num_generations rollouts per prompt and optimize a cost-aware objective that adds an efficiency reward favoring smaller budgets when the answer is correct. We cap prompt length (max_prompt_length) and completion length (max_completion_length) for stable training and bounded compute. We use AdamW with cosine schedule, BF16, and gradient accumulation as needed. Implementation notes. Since routed injections modify the effective prefix, we reset the decoding key–value cache after each injection and continue generation from the augmented sequence. We also recommend inserting prompt-end memory tokens before the assistant marker in chat-formatted prompts to avoid degenerate early termination.