Paper deep dive
ReMAP: Restoring the Perceptual Cycle with Reasoning-Time Latent Visual Memory
Hao Jiang, Zhanyu Guo, Chenwei Wu, Yichen Guo, Qizhe Zhang, Junchi Yao, Jixian Wu, Jinhao You, Kai Tang, Jiajun Cao, Tinghao Wang, Mengyu Wang, Leo Anthony Celi, Shanghang Zhang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As multimodal large language models (MLLMs) reason for longer, attention to the initial visual input diminishes, weakening visual grounding. Visual memory reintroduces visual evidence during reasoning. We conduct a controlled analysis of visual memory along three axes: curation, organization, and access. We find that local evidence benefits from global context, compact latent representations balance accuracy and visual-context cost, and the utility of memory access depends on the reasoning state. Guided by these findings, we propose ReMAP (Reasoning-Time Memory-Augmented Perception), which couples two complementary latent memories: a static, question-conditioned Global memory that preserves scene and cross-image context, and a dynamic Local memory that uses this context as an anchor while selecting and re-encoding region-level evidence according to the current reasoning state. Both memories return compact latent tokens inserted into the reasoning sequence, and a reinforcement-learning access policy trained with branched rollouts decides when to continue reasoning or invoke Global or Local memory. On ten benchmark families, ReMAP outperforms prior visual-memory methods on all four multi-image benchmarks, exceeding the strongest prior results on MuirBench and MIMIC by 8.38 and 14.84 percentage points. Across four backbone families, enabling memory access improves over the same trained model with memory disabled, and on shared V*Bench, CV-Bench-2D, and MuirBench questions ReMAP reduces the visual tokens entering the reasoning sequence by 51.0-76.8% relative to the native-resolution backbone. Further analyses show that Global and Local memory form distinct yet complementary latent representations. Together, these components restore the perceptual cycle by letting the reasoning state trigger targeted visual retrieval, with the retrieved evidence guiding subsequent reasoning.
Tags
Links
- Source: https://arxiv.org/abs/2610.05097v1
- Canonical: https://arxiv.org/abs/2610.05097v1
Trouble viewing inline? Open PDF directly โ
Full Text
112,710 characters extracted from source content.
Expand or collapse full text
ReMAP: Restoring the Perceptual Cycle with Reasoning-Time Latent Visual Memory Hao Jiang 1* , Zhanyu Guo 1* , Chenwei Wu 2โ , Yichen Guo 3,4โ , Qizhe Zhang 4 , Junchi Yao 5 , Jixian Wu 6 , Jinhao You 1 , Kai Tang 4 , Jiajun Cao 4 , Tinghao Wang 1 , Mengyu Wang 7 , Leo Anthony Celi 8 , Shanghang Zhang 4B 1 University of Electronic Science and Technology of China, 2 University of Michigan, Ann Arbor, 3 Nanyang Technological University, 4 Peking University, 5 Mohamed bin Zayed University of Artificial Intelligence, 6 New York University, 7 Harvard University, 8 Massachusetts Institute of Technology Abstract: As multimodal large language models (MLLMs) reason for longer, attention to the initial visual input diminishes, weakening visual grounding. Visual memory reintroduces vi- sual evidence during reasoning. We conduct a controlled analysis of visual memory along three axes: curation, organization, and access. We find that local evidence benefits from global con- text, compact latent representations balance accuracy and visual-context cost, and the utility of memory access depends on the reasoning state. Guided by these findings, we propose ReMAP (Re asoning-Time Memory-Augmented Perception), which couples two complementary la- tent memories: a static, question-conditioned Global memory that preserves scene and cross- image context, and a dynamic Local memory that uses this context as an anchor while selecting and re-encoding region-level evidence according to the current reasoning state and the question. Both memories return compact latent tokens that are inserted into the reasoning sequence, and a reinforcement-learning access policy trained with branched rollouts decides when to continue reasoning or invoke Global or Local memory. On ten benchmark families, ReMAP outperforms prior visual-memory methods on all four multi-image benchmarks, exceeding the strongest prior results on MuirBench and MIMIC by 8.38 and 14.84 percentage points, respectively. Across four backbone families, enabling memory access improves over the same trained model with memory disabled, and on shared V*Bench, CV-Bench-2D, and MuirBench questions ReMAP reduces the visual tokens entering the reasoning sequence by 51.0โ76.8% relative to the native- resolution backbone. Further analyses show that Global and Local memory form distinct yet complementary latent representations and play coordinated roles in interleaved vision-language reasoning. Together, these components restore the perceptual cycle by letting the reasoning state trigger targeted visual retrieval, with the retrieved evidence guiding subsequent reasoning. Code coming soon. 1INTRODUCTION Visionโlanguage models are increasingly applied to visual tasks covering larger visual contexts and longer reasoning trajectories. These tasks require models to draw on visual evidence at different reasoning stages such as inspecting fine-grained details, comparing observations across images, or verifying intermediate conclusions. As visual inputs and generated text accumulate, maintaining ac- cess to relevant visual evidence is increasingly challenging. Recent studies report visual forgetting, where attention to visual inputs diminishes during extended generation (Sun et al., 2025; Liu et al., 2025), highlighting the difficulty of sustaining visual grounding throughout reasoning. Visual memory methods have emerged as a natural solution to retain visual evidence during reason- ing. Existing approaches explore several routes with distinct trade-offs: reinspecting source pixels offers fine-grained details but requires extra image processing (Zheng et al., 2026; Su et al., 2025; Jiang et al., 2025), replaying encoded features reuses existing representations but limits available ev- idence to their initial coverage (Chu et al., 2025; Gao et al., 2025; Zhang et al., 2026a), constructing * Equal contribution. โ Project leader. B Corresponding author. Correspondence to: Hao Jiang (2024010906013@std.uestc.edu.cn), Shanghang Zhang (shanghang@pku.edu.cn). arXiv:2610.05097v1 [cs.CV] 4 Oct 2026 Preprint. Under review. (c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity (c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity (c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity(c) Global context resolves local ambiguity Human Qwen Human Qwen Human Qwen What is shown in this image? Briefly explain your answer. 1,244ยญtoken analysis Manual crop replay Look closely. What are these pieces made of? Densely packed cylinders of different heights. Most likely plastic, possibly wood. + Global information The background of this photo looks like a kitchen. With that context, they are likely spaghetti. Full image Local crop Local crop Full image Early (20%) Middle (50%) Late (80%) Replay position 60 65 70 75 Accuracy (%) 65.9 64.8 67.0 72.572.5 71.4 +6.6 p +7.7 p +4.4 p Frozen latent Crop pixels Fresh latent 0 20 40 60 80 100 Accuracy (%) 0 0.25 0.50 0.75 1.00 1.25 Rel. memory (ร) 0.22ร 1.00ร 0.49ร 41.7% 66.7% 50.0% Frozen featuresFresh features AccuracyReplay overhead What color is the helmet? Negative gainPositive gain CoT ImagesQuery 0 What changed? Which animal is this? Memory Memory 1Memory 2 (a) Global information helps interpret local details.(b) Latent memory balances accuracy and replay overhead. (c) Memory invocation is not always beneficial, and its optimal timing varies across samples. Figure 1: Investigating memory curation, organization, and access. (a) An interaction with Qwen2.5-VL-7B-Instruct on a MuirBench example illustrates how global context helps resolve am- biguous local evidence. (b) A comparison of frozen latents, crop pixels, and fresh latents on a random 25% subset of V*Bench examines the trade-off between accuracy and incremental GPU- memory overhead. (c) An easy recognition example and cases from V*Bench and MuirBench show how the need for memory and its invocation timing vary across queries and reasoning states. latent memories provides compact representations at the cost of interpretability (Zhang et al., 2026b; Chen et al., 2026c; Yu et al., 2026). These approaches differ in evidence selection, representation, and delivery timing. We therefore ask: what design choices of a visual memory system truly improve visual reasoning? We characterize visual memory along three design axes: curation (what evidence enters memory), organization (how evidence is represented), and access (when to access memory and which memory to use). Under this framework, we conduct preliminary experiments combining qualitative analysis with controlled empirical evaluations, as illustrated in Figure 1. These analyses yield three key findings: โถ Memory curated from local visual evidence alone can be insufficient for contextual reasoning, as illustrated in Figure 1(a). โท Memory organization affects both reasoning accuracy and computational overhead, as illustrated by the trade-off between image-crop replay and compact latent representations in Figure 1(b). โธ Reintroducing visual evidence is not always beneficial, and its utility depends on when it is accessed during reasoning (Figure 1(c)). Guided by these findings, we propose ReMAP (Reasoning-Time Memory-Augmented Perception). For memory curation, ReMAP combines Global memory, which preserves scene and cross-image context from the input images and question, with Local memory, which selects and re-encodes image regions according to the current reasoning state. For memory organization, Global and Local memory return compact latent tokens that are inserted into the reasoning sequence to balance reasoning accuracy and computational overhead. For memory access, we treat memory invocation as an agentic tool call so that the model can obtain relevant visual evidence when needed. We leverage branched rollouts before each invocation to learn whether, when, and which memory to invoke. These rollouts compare the outcomes of requesting memory versus continuing from the same reasoning state and teach when memory helps. Inference follows a single trajectory. We evaluate ReMAP against existing visual memory methods on ten benchmark families span- ning single-image perception, multi-image reasoning, and broader visual understanding. ReMAP achieves state-of-the-art overall performance across the benchmark suite. Notably, ReMAP sig- 2 Preprint. Under review. nificantly improves multi-image reasoning, outperforming existing visual-memory methods on all four benchmarks and exceeding the strongest prior results on MuirBench and MIMIC by 8.38 and 14.84 percentage points, respectively. Beyond these aggregate gains, our analyses characterize reasoning-time visual memory: ReMAPโs memory access improves performance across four back- bone families, Global and Local evidence provide complementary benefits, and compact latent re- turns reduce visual-context cost while maintaining leading accuracy. Further analyses make Global and Local memory interpretable through their distinct yet complementary latent representations and coordinated roles in interleaved vision-language reasoning. Our contributions are threefold: โถ We identify three principles for reasoning-time visual memory: contextualize local evidence with global context, compress representations, and adapt access to the evolving reasoning state. โท We introduce ReMAP, a framework for reasoning-time visual memory. ReMAP com- bines query-conditioned Global context with state-guided Local evidence in compact latent returns and learns when and which memory to invoke through branched rollouts. โธ We demonstrate strong multi-image reasoning with interpretable memory roles. ReMAP outperforms prior visual-memory methods on all four multi-image benchmarks, while representation and trajectory analyses reveal the complementary functions of Global and Local memory. 2RELATED WORK Memory curation spans pixel reacquisition, feature replay, and learned memory formation, with dis- tinct trade-offs between evidence coverage and processing overhead (Zheng et al., 2026; Su et al., 2025; Zhang et al., 2026b). Memory organization represents evidence through images, text, en- coded features, or latent tokens, with different trade-offs in interpretability and context cost (Hu et al., 2024; Zhang et al., 2026d; Huang et al., 2026; Yu et al., 2026). Memory access uses heuristic triggers, uncertainty criteria, or learned policies (Sun et al., 2025; Liu et al., 2026b; Peng et al., 2026), and related policy optimization methods improve tool use and visual reasoning through train- ing feedback (Dong et al., 2026a; Li et al., 2026c). ReMAP bridges these dimensions by combining pixel reacquisition with compact latent memory and preserving scene and cross-image context to interpret newly acquired local evidence. Its access training branches before memory invocation to compare requesting memory with continuing from the same reasoning state, providing feedback on the utility of memory access. A detailed review and comparison are provided in Appendix A.1. 3REASONING-TIME MEMORY-AUGMENTED PERCEPTION In this section, we introduce ReMAP, which interleaves autoregressive reasoning with access to la- tent visual memory to restore the perceptual cycle. Figure 2 provides an overview of the framework. Given imagesI =I i N i=1 and a question Q, the model begins reasoning from a coarse visual prefix V c = E c (I). During reasoning, the model can continue generating text or invoke Global or Local latent memory. After each memory return, the model resumes reasoning with the retrieved evidence. Specifically, Section 3.1 describes how Global and Local memories are formed and integrated into the reasoning sequence. Section 3.2 describes memory access and its training policy. 3.1MEMORY CURATION AND ORGANIZATION Global memory. Global memory provides a compact representation of image content and, for multi-image inputs, relationships across images. We construct it in two stages: the Image Weaver summarizes each image, and the Relation Weaver integrates these summaries into relational context. Both stages are conditioned on the input question. Let V i denote the encoded visual features of image I i . We use two sets of learnable query tokens, Q G and Q R , to aggregate per-image content and relational context, respectively. The corresponding modules, W img and W rel , apply cross-attention followed by feed-forward layers. The Image Weaver attends to each imageโs features separately, while the Relation Weaver attends to the concatenated image summaries. Both include an encoded representation of the question in their attention context, 3 Preprint. Under review. The first two images show Porsche cars, and the third reports sales by model. What is the total sales volume of the two cars? The first two images contain the target vehicles, while the third organizes sales by model; each vehicle must be matched to its corresponding chart row. Global Memory Token Sequence ...... The total sales is 2280. Global Memory Formation ํฝ ํ ํฝ ํ ํฝ ํ ํธ ํฎ Image Weaver ํธ ํน ํธ ํฎ Image Weaver ํธ ํฎ Image Weaver ํต ํํํ =ํ ํํํ ํ ํํํ ,ํ 1 ํํํ ;โฏ;ํ ํ ํํํ Relation Weaver ROI Re-Encoder ... ... ํ ํ ํ ํ ํ ํ Local Weaver แ ํ ํก,ํ ํฟ =ํ ํฟ เท โ ํก ,ํ ํก,ํ ;ํ ํบ Twig Localization ํน ํ,ํ ํน ํ,ํ ํน ํ,ํ Local Memory Formation Vision Encoder Hidden States ํธ ํ ํ + ํ ํ + ํ ํ + ํ ํ ํํํ ํ ํ ํํํ ํ ํ ํํํ ํฐ ํ ํฐ ํ ํฐ ํ Local Memory ํ ํณ ํ ํฎ ํ ํบ =ํ 1 ํํํ ;โฏ;ํ ํ ํํํ ;ํต ํํํ Figure 2: Overview of ReMAP. Global memory forms a question-conditioned summary of individ- ual images and their relations before decoding. During reasoning, a Local request uses the current state to select sources and regions, re-encodes selected pixels, and compresses them with cached Global context. Global and Local returns are inserted as continuous latent blocks after their request markers, followed by end markers before autoregressive decoding resumes. denoted by the conditioning on Q below: Z img i = W img (Q G ,V i ;Q), Z rel = W rel Q R , [Z img 1 ;ยท ;Z img N ];Q .(1) The Global memory block Z G = [Z img 1 ;ยท ;Z img N ;Z rel ] concatenates the image summaries and relational context. Constructed and cached before decoding, Z G depends on the input images and question and remains fixed throughout reasoning. When the decoder generates <mem g>, the run- time inserts Z G followed by <memgend> and resumes reasoning. Local memory. Figure 3(a) shows how Local memory dynamically retrieves fine-grained evidence as reasoning evolves. At state h t , the Source Selector uses the layer-12 request state h 12 t to select imagesS t , and Twig uses the layer-18 prefix states H 18 <t to localize a crop in each selected image. Twig extends the ROI predictor of SD-RPN (Shi et al., 2025) and Q-Zoom (Shi et al., 2026) from the input question to the current reasoning prefix, so successive requests can localize different regions. A frozen ROI encoder maps the crops to region features R t,i . The Local Weaver takes the cached Global memory Z G as an additional input while compressing each region: e Z L t,i = W L (h t ,R t,i ;Z G ), Z L = e Z L t,i iโS t .(2) Z G anchors each crop in the image-level and cross-image context that local evidence alone can miss, consistent with our first finding. A <mem l> request similarly inserts Z L followed by <memlend> before generation resumes. The concatenated block contains K Local latents. 3.2MEMORY ACCESS Latent memory keeps retrieved visual evidence within a fixed token budget. Replaying whole im- ages at every request would repeatedly expand the context and compete with the ongoing reasoning sequence. Moreover, memory does not help at every step, since its value depends on when it is ac- cessed, consistent with our third finding. The content of a return can be supervised directly, whereas the value of its timing appears only in later reasoning. We therefore separate memory formation from memory access. Stage I uses supervised fine-tuning to form useful memory and warm up request handling, while Stage I uses reinforcement learning to learn whether and when to invoke Global or Local memory. 4 Preprint. Under review. Source selector Layer 18 Layer 12 Layer 1 ... ... Frozen Backbone LLM ... Image tokens Text tokens Decoder blocksร2 Q/k projection+mRoPE Twig Localization LoRA ํ ํก ํพ ํํํ ํ ํก ํพ ํํํ ํ Query condition Cross-attention FFN Formation Local Weaver K (a) Local memory formation q roi ํ ํณ (c) Memory invocation policy learning 1. Branch-point selection2. Shared-prefix rollouts and credit ํํ ํ ,ํ ํ =ํโํ ํ ํ ํ ํ What is the total sales volume of the two cars? Answer ํด 1 ํด 2 ํด 3 G L N Shared prefix Branch Step Action type <mem_g> Decision 1 <mem_l> Decision2 Decision3 ํ ํ ํ ํ ํ ํ Text ... ํ ํ ํฎ ํ ํ ํณ ํ ํ ํต (no request) ํข ํ ํ ํ,ํ ํน ํ,ํ ํฌ ํซํจํข ํฏ <ํ ํํ ํ ํ ํํ ํ ํ ํํ ํ ํฎ ํธ ํณ 1. Memory content training h t Local Weaver LLM โ suffix = CE( เทก ํ, Y) ํ ํณ 2. ROI localization โ ROI = โ wBCE (ฤ, q roi )+(1-softIoU) Twig q roi ํฏ <ํ ํํ 3. Student-state adaptation Teacher review What is the total sales volume of the two cars? ํฐ ํ ํ ํ,ํ Student prefix + 4. Interface warm-up <mem_x_end> โ call โ end Latent memory Student prefix โ later ... <mem_g> <mem_l> ํน ํํํํํํํ ํ ํฎ (b) Stage1: memory and interface training Sampled continuations Y:Teacher continuation ฤ:Annotated ROI Same loss โ ROI Same loss โ suffix in Student-state Figure 3: Local memory and adaptive invocation. (a) The request state h 12 t selects source images S t , Twig uses the prefix states H 18 <t to localize a crop in each, and the Local Weaver compresses the crops with the Global anchor Z G into K Local latents. (b) Stage I trains the Local Weaver withL suffix and Twig withL ROI , adapts both to teacher-reviewed student states, and warms up the memory tokens. (c) Each root branches where siblings are most likely to take a different route, and the invocation gain compares requesting memory with the default continuation N at the same state. Stage I: Supervised learning and warm-up. Stage I isolates memory formation from access- policy learning, so that each return localizes the right region and carries evidence that supports sub- sequent reasoning. The vision encoder, pretrained LLM, and decoder LoRA remain frozen through- out this stage. Existing datasets lack ROI annotations that change along the reasoning chain, so we annotate reasoning steps with target regions, together with reference CoTs from a teacher VLM, per-image facts, cross-image relations, and answers. We first optimize memory quality by training the Image and Relation Weavers and then the Local Weaver. Their returns are inserted into teacher CoTs, and the continuation cross-entropyL suffix through the frozen LLM, together with fact, rela- tion, and answer supervision, trains them to encode evidence for the teacherโs subsequent reasoning. We then train Twig to localize the annotated regions withL ROI . Because student states differ from teacher states, we next train continuation on student rollouts, where the teacher reviews the studentโs prefixes and crops to provide continuation and ROI targets for the same two losses. Finally, with the formation modules fixed, a warm-up lets the model emit the memory special tokens, updating only the input embeddings of the four markers and the output-layer rows that score <mem g> and <meml>. Figure 3(b) illustrates these signals, and Appendix A.3.3 gives the objectives. Stage I: Reinforcement learning for memory access. Guided by our third finding, Stage I makes memory access helpful by learning when and which memory to request, with formation modules frozen. At a request boundary, the decoder emits a Global or Local request token or con- tinues generating text. A trajectoryโs final reward cannot show whether a given request helped. We therefore branch from a shared prefix and compare requesting with continuing at the same state. Following ARPO (Dong et al., 2026a), each root branches at most once, at the state maximizing u(s t ,a t ) = 1โ p t (a t ) m ,(3) where p t (a t ) is the probability of the realized route a t , with p t (N) = 1โ p t (G)โ p t (L) for no request, and m is the number of sibling rollouts. This score is the probability that at least one sibling takes a different route, so the branch goes where siblings likely diverge from the root, as illustrated in Figure 3(c). A sibling i that requests memory at branch b receives the invocation gain โ call i = R i โ B b , where R i is its final-answer reward and B b is the mean reward of siblings that continue without a request. These siblings may still request memory later, so the gain measures the value of requesting now. Normalized invocation gains update the request-token embeddings and 5 Preprint. Under review. Table 1: Comparison with existing visual memory methods across ten benchmark families. Results cover single-image perception, multi-image reasoning, and broader visual understanding. MIMIC Avg. is the mean across four subsets. The top-2 results in each column are highlighted as first andsecond , with the best results also shown in bold. Single-image evidence acquisitionMulti-image relational reasoningHeld-out generalization MethodV โ CV-2D CV-3D HR-4K HR-8K BLINK Muir MMSI MIMIC Avg.RWQA MathVista MMVet Native76.44 74.4176.3372.8867.7556.62 57.85 28.1045.1767.9759.9046.06 Coarse67.54 73.1075.4261.1354.6257.79 58.15 28.4043.5667.1959.5047.06 Pixel reinspection Q-Zoom79.06 76.3574.0873.0066.6353.97 50.38 25.4042.2970.8554.6047.43 DeepEyes84.2978.6179.4274.6368.7549.24 48.50 24.1039.2769.6762.0051.01 Pixel Reasoner 81.15 75.0081.5873.5065.3858.50 53.15 27.9044.83 71.9071.4048.07 ReGround73.30 73.7377.0068.1360.6354.65 56.12 25.4028.0166.9369.3043.21 Encoded-feature reuse ICoT56.02 68.2233.6766.1365.2524.02 20.96 26.3029.9135.9527.4036.79 MINT-CoT41.36 72.8474.8354.0043.0049.79 54.04 26.8034.0562.4862.1047.98 Latent representations DMLR44.50 68.5266.7546.0042.3842.32 39.81 22.0023.7454.9051.4035.87 VaLR70.68 74.7971.7551.0047.8844.88 45.08 19.5017.9345.7549.3033.21 V-Reflection79.06 75.2177.7573.2566.5054.21 46.65 25.1053.1670.8568.7037.94 VisMem60.21 76.6482.5056.1347.8856.95 57.00 21.7041.1968.1061.1045.55 SFT58.64 75.0475.5856.6248.2554.13 55.7329.0052.7070.5972.1055.05 Stage I sft 76.44 77.9683.3375.0068.3858.92 57.81 27.3068.3771.2470.1048.35 Ours 82.2079.2883.9275.5069.1360.6565.3829.3068.0071.7673.4064.27 output rows, and group-relative final-answer advantages (Shao et al., 2024b) update the decoder LoRA. Appendix A.3.4 gives the objectives. 4EXPERIMENTS We evaluate ReMAP on single-image perception, multi-image reasoning, and broader visual under- standing. We then examine the contributions of memory curation, organization, and access through ablations, latent visualizations, reasoning trajectories, and visual-context cost. 4.1EXPERIMENTAL SETUP Benchmarks. We use ten benchmark families in three categories. Single-image perception includes V*Bench (Wu & Xie, 2024), CV-Bench-2D/3D (Tong et al., 2024), and HR-Bench- 4K/8K (Wang et al., 2025b). Multi-image reasoning includes BLINK (Fu et al., 2024), Muir- Bench (Wang et al., 2025a), MMSI-Bench (Yang et al., 2026a), and MIMIC (Das et al., 2026). Broader visual understanding (Yu et al., 2024; Lu et al., 2024; xAI, 2024) uses M-Vet, Math- Vista, and RealWorldQA for generalization tests. Baselines and implementation. Following the approaches discussed in the introduction, we com- pare pixel reinspection methods Q-Zoom (Shi et al., 2026), DeepEyes (Zheng et al., 2026), Pixel Reasoner (Su et al., 2025), and ReGround (Peng et al., 2026), encoded-feature reuse methods ICoT (Gao et al., 2025) and MINT-CoT (Chen et al., 2025), and latent-representation methods DMLR (Liu et al., 2026a), VaLR (Jeon et al., 2026), V-Reflection (Zhou et al., 2026), and Vis- Mem (Yu et al., 2026). Native and Coarse provide backbone controls at native resolution and at most 1,024 initial visual tokens per image. ReMAP uses the Coarse input budget and returns 32 latents per Local call. All ReMAP rows and the Native and Coarse controls in Table 1 use Qwen2.5- VL-7B-Instruct (Bai et al., 2025); prior methods keep their released backbones. ReMAP is trained 6 Preprint. Under review. Table 2: Memory access across backbone families. Base denotes the original backbone, and OFF denotes the trained ReMAP model with memory invocation disabled. โ reports ReMAPโBase differences in score points. Evaluation details appear in Appendix A.6. BackboneMethodV โ CV-3DHR-8KBLINKMuirBenchMMSIMIMIC Avg. Qwen2.5-VL-3B Base51.8350.8342.1243.6150.0829.4028.21 OFF56.0270.5849.7546.6650.7728.7036.25 ReMAP58.6475.5851.2548.4555.2330.5050.11 โโ +6.81โ +24.75โ +9.12โ +4.84โ +5.15โ +1.10โ +21.90 LLaVA-OneVision-7B Base70.6845.0856.6244.6641.8528.9044.05 OFF71.7370.1756.2551.6041.6527.5052.79 ReMAP76.4476.0059.2553.4548.6529.7059.52 โโ +5.76โ +30.92โ +2.62โ +8.78โ +6.81โ +0.80โ +15.47 Qwen3-VL-8B Base62.3092.2559.0064.3973.7727.9056.99 OFF63.8791.5057.1265.3372.6931.6058.70 ReMAP66.4993.8360.0067.0774.3132.3074.20 โโ +4.19โ +1.58โ +1.00โ +2.68โ +0.54โ +4.40โ +17.21 InternVL3-8B Base59.1668.6762.8853.3451.8127.3036.94 OFF61.2668.3363.1253.8151.4229.2043.00 ReMAP65.4569.2564.1255.4452.8830.3057.00 โโ +6.28โ +0.58โ +1.25โ +2.10โ +1.08โ +3.00โ +20.06 primarily on 16 NVIDIA A800 GPUs. Evaluation protocols, baseline configurations, and training details appear in Appendices A.6, A.4, and A.5. 4.2OVERALL PERFORMANCE โถ ReMAP outperforms prior methods on all four multi-image benchmarks. In Table 1, ReMAP surpasses the best prior results on MuirBench and MIMIC by 8.38 and 14.84 percentage points, re- spectively. These gains span cross-image comparison, spatial reasoning, and evidence aggregation. ReMAP also improves over the native-resolution backbone on all five single-image metrics, extend- ing its benefits to fine-grained visual perception. โท ReMAPโs learned memory access transfers across backbone families. We isolate the contri- bution of reasoning-time memory invocation by comparing each backboneโs original Base model, its trained Memory-OFF counterpart, and ReMAP with memory enabled (Table 2). ReMAP improves over Memory-OFF on both single-image and multi-image metrics for all four backbones, showing that the gains come from dynamic evidence access during reasoning rather than from the trained model weights alone. 4.3INTERPRETING LATENT MEMORY ORGANIZATION Global and Local form two distinct types of latent memory. Figure 4 visualizes their benchmark- conditioned t-SNE projections. Local is benchmark-specific: its distribution changes across bench- marks. Global mixes across benchmarks because it captures abstract scene-level and cross-image relations. In the multi-image view, Global is more dispersed, suggesting that it represents a wider range of inter-image relations. 7 Preprint. Under review. (a) High-resolution benchmarks โ50โ25025 t-SNE Component 1 โ40 โ20 0 20 40 t-SNE Component 2 High-resolution benchmarks Global Local CV-Bench-3D CV-Bench-2D V*Bench HR-Bench-4K HR-Bench-8K (b) Multi-image benchmarks โ50โ25025 t-SNE Component 1 โ40 โ20 0 20 40 t-SNE Component 2 Multi-image benchmarks Global Local MIMIC BLINK MuirBench MMSI-Bench Figure 4: Global and Local memory representations. (a) High-resolution and (b) multi-image benchmark groups. Joint t-SNE projections show mean-pooled Global and Local memory vectors; colors denote benchmark groups, circles and triangles denote Global and Local, respectively, and shaded contours show density. 4.4MEMORY ACCESS AND THE PERCEPTUAL CYCLE HR-Bench-4K #0347 What is the speed limit indicated on the sign in the image? A. 20 km/h B. 40 km/h C. 60 km/h D. 30 km/h No memory Answer B With memory Answer D Original input (a) Local gaze completion perceptual cycle # Local ROI1 MMSI #0377 Where is the potted plant relative to the longest sofa? The statue is north of the sofa. # Global relation & anchors Raw heatmaps Sharpened peaks (b) GlobalโLocal interleaved perceptual cycle 23 A1 A2 B1 B2 # Local ROI2 # Local ROI3 A. Due south B. Northeast C. Due east D. Southwest No memory Answer C With memory Answer B Original input # Local ROI1 # Local ROI2 To determine the speed limit indicated on the sign in the image, letโs follow these steps:โ <mem_l><mem_l_end>The sign is a standard German speed limit sign, circular with a white background and black border, โก<mem_l><mem_l_end>indicating a speed limit.โข <mem_l><mem_l_end>The sign shows a number in the center, which is the speed limit in kilometers per hour.The number on the sign is "30".Given these observations, the speed limit indicated on the sign is 30 km/h. Answer: D To determine the position of the potted plant relative to the longest sofa, let's analyze the image step by step: Identify the longest sofa: It is the couch on the right side of the image. โ <mem_l><mem_l_end>Find the statue: It is on the table located on the left side of the image. โก <mem_l><mem_l_end>Draw a line from the statue towards the potted plant and deduce the direction it points: it points towards the northeast.โข<mem_g><mem_g_end> Identify the position of the potted plant: It is located to the northeast of the longest sofa. Answer: B Figure 5: Local refinement and GlobalโLocal interaction. (a) HR-Bench-4K #0347: repeated Local calls narrow the inspection from the street scene to a speed-limit sign, yielding 30 km/h rather than 40 km/h without memory. (b) MMSI #0377: two Local calls followed by a Global call accompany the northeast answer, whereas the paired no-call trajectory answers due east. Local provides fine-grained visual evidence, Global provides broader scene and cross-image context, and their interleaved calls restore the perceptual cycle. We analyze how the learned policy routes between these two forms of visual evidence as reasoning evolves. Figure 5 shows that memory returns are inserted into ongoing reasoning and change the subsequent reasoning state, yielding a state-dependent evidenceโreasoning loop rather than a fixed access schedule. Figure 6(b) 8 Preprint. Under review. NativeQ-Zoom DeepEyes VisMem ReMAP 0 5,000 10,000 15,000 20,000 Visual tokens entering the sequence 4,418 1,340 4,454 4,418 1,024 906 626 1,114 906 444 7,099 4,016 9,197 7,099 3,389 0 25 50 75 100 Performance (%) 76.44 79.06 84.29 60.21 82.20 74.41 76.35 78.61 76.64 79.28 57.85 50.38 48.50 57.00 65.38 V*BenchCV-Bench-2DMuirBench (a) 020406080100 Normalized CoT position (%) 0% 5% 10% 15% 20% 25% 30% Invocation ratio CV-Bench HR-Bench MMSI MathVista M-Vet Global Local (b) Figure 6: Memory invocation and visual-context cost. (a) Mean total visual tokens entering the sequence (bars) and performance (lines) on shared V*Bench, CV-Bench-2D, and MuirBench ques- tions; the shaded column marks ReMAP. (b) Global (solid) and Local (dashed) invocation ratios over the normalized reasoning trajectory. Table 3: Training, curation, and access ablations (%). Base is Coarse; BLINK reports micro accuracy and M-Vet mean graded scores. MethodV โ CV-3DMuirBLINKMM-Vet Base67.5475.4258.1557.8147.06 SFT58.6475.5855.7354.1355.05 Stage I sft 76.4483.3357.8158.9248.35 Memory curation w/o Memory79.0680.0061.0059.0257.80 w/ Global memory79.5883.0064.0059.5060.55 w/ Local memory81.6881.0062.0060.0259.63 Memory access Fixed76.9683.0061.4659.3958.72 Random79.5879.7561.5858.8157.34 Entropy79.0681.1762.0059.3959.63 ReMAP82.2083.9265.3860.6564.27 shows that Local calls concentrate early and in the middle of the trajectory, whereas Global calls persist later. 4.5COMPONENT, CAPACITY, AND ACCESS ABLATIONS Next, we study the gains from different components and design choices in ReMAP, including train- ing, memory curation, memory access, and latent capacity. Global and Local specialize by scope, and their combination is best. We analyze how curation scope affects benchmark performance. Global is stronger on CV-Bench-3D, MuirBench, and M- Vet, whereas Local is stronger on V โ Bench and BLINK. Combining both memories is best on every benchmark, exceeding the better of the Global-only and Local-only variants by 3.72 points on M- Vet, as shown in Table 3. Learned invocation is more reliable than fixed or heuristic schedules. We analyze whether the learned access policy selects more useful request points than Fixed, Random, and Entropy sched- ules. ReMAP is highest on every benchmark, exceeding the strongest of these alternatives by 3.38 points on MuirBench and 4.64 on M-Vet. Fixed invocation falls below w/o Memory on V โ Bench, showing that a fixed schedule can make added evidence harmful; Table 3 reports the access variants. The control definitions are given in Appendix A.9. Appendix A.10 examines branch-point selection and routing diversity. 9 Preprint. Under review. 48163264 20 30 40 50 60 Local latent tokens (K) Performance (%) 2N+24N+48N+8 Global latent tokens HR-Bench 4KMuirBenchMMSI-BenchMacro average Figure 7: Latent-capacity sensitivity. Performance on 20% holdout splits from HR-Bench-4K, MuirBench, and MMSI-Bench. Curves show benchmark scores and their equal-weight macro av- erage; shaded bands are item-level 95% intervals from one training seed. Dashed guides mark the selected K = 32 and 4N + 4 settings. ReMAP remains effective across a range of latent return sizes, making a compact capacity sufficient. We sweep the Local and Global return sizes to determine the default configuration (Fig- ure 7). Performance is broadly stable across the tested capacities, while increasing the number of returned tokens does not bring consistent gains. We therefore use K = 32 Local tokens and 4N + 4 Global tokens in the remaining experiments. 4.6VISUAL-CONTEXT COST ReMAP maintains strong performance with substantially lower visual-context cost. We ana- lyze the accuracyโcontext-cost tradeoff using the matched profiles in Figure 6(a). ReMAP remains competitive while using fewer visual tokens than Native, and its compact memory returns refresh visual evidence without repeatedly expanding the decoder context. Detailed accounting is provided in Appendix A.12. 5LIMITATIONS We note three limitations of ReMAP that we hope to address in future work. First, ReMAP operates within a fixed memory and access budget: each trajectory permits at most five memory requests, each Local request re-encodes at most three selected images, and each Global block has a fixed 4N + 4 layout. These limits keep computation low but may omit evidence when an input contains many images or when a task requires extended visual search. Second, Twig is trained only in Stage I on teacher-reviewed ROI targets and receives no further training dedicated to localization. When its box misses the relevant region, the subsequent zoom-in contributes less useful detail, so we expect a stronger localizer, for example one trained with dedicated grounding data, to further improve Local memory. Third, the memory modules are attached at fixed positions of the backbone: the Source Selector reads layer 12, and Twig reads layer 18. Internal layer sweeps selected these taps from the quality of the tapped representations, but we have not yet studied how placing the components at other layers affects end-to-end performance, and we leave this analysis to future work. 6CONCLUSION ReMAP shows that visual memory becomes effective when the scope and timing of visual evidence follow the reasoning state. Global memory preserves question-conditioned scene and cross-image context, while Local memory reacquires fine-grained pixels as the trajectory demands. A learned policy coordinates these two forms of evidence within compact latent returns, improving over prior methods on all four multi-image benchmarks and over the Native backbone on all five single-image 10 Preprint. Under review. metrics while reducing the profiled visual context. Together, these components restore the perceptual cycle by letting each reasoning step use its current evidence to guide the next visual inspection. We hope this view helps future multimodal models treat perception as part of reasoning. AI USE STATEMENT We used generative AI tools to aid and polish the writing, retrieve and discover related literature, and draft sections of the manuscript. We reviewed all AI-assisted material and take responsibility for the final content of this paper. ETHICS STATEMENT This work uses public research datasets and no human subjects or newly collected personal data. ReMAP does not remove inherited model biases or failure modes; additional visual operations can also make source-allocation or localization errors appear confident. REPRODUCIBILITY STATEMENT Details of the memory interface, learning objectives, training configuration, data, evaluation proto- cols, and ablations are provided in the appendices. We will publicly release our source code and model weights. REFERENCES Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL technical report, 2025. URL https://arxiv.org/abs/2502.13923. Chao Chen, Zhixin Ma, Yongqi Li, Yupeng Hu, Yinwei Wei, Wenjie Li, and Liqiang Nie. Reasoning in the dark: Interleaved vision-text reasoning in latent space. In Findings of the Association for Computational Linguistics: ACL 2026, p. 39117โ39129, San Diego, California, United States, July 2026a. Association for Computational Linguistics. doi: 10.18653/v1/2026.findings-acl.1949. URL https://aclanthology.org/2026.findings-acl.1949/. Harold Haodong Chen, Xinxiang Yin, Wen-Jie Shu, Hongfei Zhang, Zixin Zhang, Chenfei Liao, Litao Guo, Qifeng Chen, and Ying-Cong Chen. Show, donโt tell: Morphing latent reasoning into image generation. arXiv preprint arXiv:2602.02227, 2026b. URL https://arxiv.org/ abs/2602.02227. Tao Chen, Kun Zhang, Qiong Wu, Xiao Chen, Chao Chang, Xiaoshuai Sun, Yiyi Zhou, and Rongrong Ji.Scaling the long video understanding of multimodal large language models via visual memory mechanism.In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), p. 31877โ31888, June 2026c.URL https://openaccess.thecvf.com/content/CVPR2026/html/ Chen_Scaling_the_Long_Video_Understanding_of_Multimodal_Large_ Language_Models_CVPR_2026_paper.html. Xinyan Chen, Renrui Zhang, Dongzhi Jiang, Aojun Zhou, Shilin Yan, Weifeng Lin, and Hongsheng Li.MINT-CoT: Enabling Interleaved Visual Tokens in Mathemati- cal Chain-of-Thought Reasoning.In Advances in Neural Information Processing Sys- tems, volume 38, p. 69110โ69139. Curran Associates, Inc., 2025.doi:10.52202/ 085713-2324.URL https://proceedings.neurips.c/paper_files/paper/ 2025/file/6403d7e0824e7be2e699f47d9703ed80-Paper-Conference.pdf. Xu Chu, Xinrong Chen, Guanyu Wang, Zhijie Tan, Kui Huang, Wenyu Lv, Tong Mo, and Weiping Li. Qwen Look Again: Guiding vision-language reasoning models to re-attention visual informa- tion, 2025. URL https://arxiv.org/abs/2505.23558. 11 Preprint. Under review. Anurag Das, Adrian Bulat, Alberto Baldrati, Ioannis Maniadis Metaxas, Bernt Schiele, Geor- gios Tzimiropoulos, and Brais Martinez. More images, more problems? a controlled analysis of VLM failure modes. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jur- gens (eds.), Findings of the Association for Computational Linguistics: ACL 2026, p. 7423โ 7442, San Diego, California, United States, July 2026. Association for Computational Lin- guistics. ISBN 979-8-89176-395-1. doi: 10.18653/v1/2026.findings-acl.366. URL https: //aclanthology.org/2026.findings-acl.366/. Yizhuo Ding, Mingkang Chen, Zhibang Feng, Tong Xiao, Wanying Qu, Wenqi Shao, and Yanwei Fu. VTPerception-R1: Enhancing multimodal reasoning via explicit visual and textual percep- tual grounding. arXiv preprint arXiv:2509.24776, 2025. URL https://arxiv.org/abs/ 2509.24776. Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, Guorui Zhou, Yutao Zhu, Ji-Rong Wen, and Zhicheng Dou. Agentic reinforced policy optimization. In The Fourteenth International Con- ference on Learning Representations, 2026a. URL https://iclr.c/virtual/2026/ poster/10009306. Shuai Dong, Siyuan Wang, Xingyu Liu, Chenglin Li, Haowen Hou, and Zhongyu Wei. Interleaved latent visual reasoning with selective perceptual modeling. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 29316โ 29335, San Diego, California, United States, July 2026b. Association for Computational Linguis- tics. doi: 10.18653/v1/2026.acl-long.1351. URL https://aclanthology.org/2026. acl-long.1351/. Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: Multimodal large language models can see but not perceive. In Computer Vision โ ECCV 2024, p. 148โ166, 2024. doi: 10.1007/ 978-3-031-73337-6 9. URL https://w.ecva.net/papers/eccv_2024/papers_ ECCV/html/3356_ECCV_2024_paper.php. Jun Gao, Yongqi Li, Ziqiang Cao, and Wenjie Li.Interleaved-Modal Chain-of-Thought.In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 19520โ19529, June 2025.URL https://openaccess.thecvf.com/content/ CVPR2025/html/Gao_Interleaved-Modal_Chain-of-Thought_CVPR_2025_ paper.html. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models, 2021. URL https: //arxiv.org/abs/2106.09685. Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah Smith, and Ranjay Krishna.Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models.In Advances in Neural Information Processing Sys- tems, volume 37, p. 139348โ139379. Curran Associates, Inc., 2024.doi: 10.52202/ 079017-4423.URL https://proceedings.neurips.c/paper_files/paper/ 2024/file/fb82011040977c7712409fbdb5456647-Paper-Conference.pdf. Siyuan Huang, Xiaoye Qu, Yafu Li, Tong Zhu, Zefeng He, Muxin Fu, Daizong Liu, Wei-Long Zheng, and Yu Cheng. Persistent visual memory: Sustaining perception for deep generation in LVLMs, 2026. URL https://arxiv.org/abs/2605.00814. Byungwoo Jeon, Yoonwoo Jeong, Hyunseok Lee, Minsu Cho, and Jinwoo Shin. Vision-aligned latent reasoning for multi-modal large language model. arXiv preprint arXiv:2602.04476, 2026. URL https://arxiv.org/abs/2602.04476. Chaoya Jiang, Yongrui Heng, Wei Ye, Han Yang, Haiyang Xu, Ming Yan, Ji Zhang, Fei Huang, and Shikun Zhang. VLM-R 3 : Region recognition, reasoning, and refinement for enhanced multimodal chain-of-thought. arXiv preprint arXiv:2505.16192, 2025. doi: 10.48550/arXiv.2505.16192. URL https://arxiv.org/abs/2505.16192. 12 Preprint. Under review. Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max W. F. Ku, Qian Liu, and Wenhu Chen. MAN- TIS: Interleaved multi-image instruction tuning. Transactions on Machine Learning Research, 2024. URL https://openreview.net/forum?id=skLtdUVaJa. Houcheng Jiang, Jiajun Fu, Junfeng Fang, Chen Gao, Xiang Wang, Xiangnan He, and Yong Li. UniVLR: Unifying text and vision in visual latent reasoning for multimodal LLMs. arXiv preprint arXiv:2605.11856, 2026. doi: 10.48550/arXiv.2605.11856. URL https://arxiv.org/ abs/2605.11856. Boyi Li, Yifan Shen, Yuanzhe Liu, Yifan Xu, Jiateng Liu, Xinzhuo Li, Zhengyuan Li, Jingyuan Zhu, Yunhan Zhong, Fangzhou Lan, Jianguo Cao, James M. Rehg, Heng Ji, Ismini Lourentzou, and Xu Cao. Toward cognitive supersensing in multimodal large language model. arXiv preprint arXiv:2602.01541, 2026a. URL https://arxiv.org/abs/2602.01541. Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. LLaVA-NeXT-Interleave: Tackling multi-image, video, and 3D in large multimodal models. arXiv preprint arXiv:2407.07895, 2024. URL https://arxiv.org/abs/2407.07895. Yifan Li, Zhenghao Chen, Ziheng Wu, Kun Zhou, Ruipu Luo, Can Zhang, Zhentao He, Yufei Zhan, Wayne Xin Zhao, and Minghui Qiu.Unleashing perception-time scaling to multimodal reasoning models.In International Conference on Learning Representa- tions, 2026b. URL https://proceedings.iclr.c/paper_files/paper/2026/ hash/499b12df1531fe8e0febcf08381f3a4-Abstract-Conference.html. Yunheng Li, Hangyi Kuang, Hengrui Zhang, Jiangxia Cao, Zhaojie Liu, Qibin Hou, and Ming- Ming Cheng. Rethinking token-level policy optimization for multimodal chain-of-thought. arXiv preprint arXiv:2603.22847, 2026c. URL https://arxiv.org/abs/2603.22847. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Pi- otr Doll ฬ ar, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In Com- puter Vision โ ECCV 2014, volume 8693 of Lecture Notes in Computer Science, p. 740โ755. Springer, 2014. doi: 10.1007/978-3-319-10602-1 48. URL https://doi.org/10.1007/ 978-3-319-10602-1_48. Chengzhi Liu, Zhongxing Xu, Qingyue Wei, Juncheng Wu, James Zou, Xin Eric Wang, Yuyin Zhou, and Sheng Liu. More thinking, less seeing? assessing amplified hallucination in multimodal reasoning models, 2025. URL https://arxiv.org/abs/2505.21523. Chengzhi Liu, Yuzhe Yang, Yue Fan, Qingyue Wei, Sheng Liu, and Xin Eric Wang. Reason- ing within the mind: Dynamic multimodal interleaving in latent space.In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings, p. 9225โ9236, June 2026a.URL https://openaccess.thecvf.com/content/ CVPR2026F/html/Liu_Reasoning_Within_the_Mind_Dynamic_Multimodal_ Interleaving_in_Latent_Space_CVPRF_2026_paper.html. Xu Liu, Yongheng Zhang, Qiguang Chen, Yao Li, Sheng Wang, and Libo Qin. Letโs think with images efficiently! an interleaved-modal chain-of-thought reasoning framework with dynamic and precise visual thoughts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, p. 32213โ32221, 2026b. doi: 10.1609/aaai.v40i38.40494. URL https://ojs. aaai.org/index.php/AAAI/article/view/40494. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Rep- resentations, 2024. URL https://proceedings.iclr.c/paper_files/paper/ 2024/hash/663bce02a0050c4a11f1eb8a7f1429d3-Abstract-Conference. html. Lei Peng, Shuai Lv, and Wei Hu. ReGround: Restoring visual grounding in multi-step reasoning through self-diagnosis and visual re-examination. In Proceedings of the 34th ACM International Conference on Multimedia, 2026. URL https://arxiv.org/abs/2608.04385. 13 Preprint. Under review. Tan-Hanh Pham and Chris Ngo. Multimodal chain of continuous thought for latent-space reasoning in vision-language models. arXiv preprint arXiv:2508.12587, 2025. URL https://arxiv. org/abs/2508.12587. Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual CoT: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. arXiv preprint arXiv:2403.16999, 2024a. URL https://arxiv.org/abs/2403.16999. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models, 2024b. URL https://arxiv.org/abs/ 2402.03300. Yuheng Shi, Xiaohuan Pei, Minjing Dong, and Chang Xu. Catching the details: Self-distilled RoI predictors for fine-grained MLLM perception. arXiv preprint arXiv:2509.16944, 2025. URL https://arxiv.org/abs/2509.16944. Yuheng Shi, Xiaohuan Pei, Linfeng Wen, Minjing Dong, and Chang Xu.Q-Zoom: Query- aware adaptive perception for efficient multimodal large language models.arXiv preprint arXiv:2604.06912, 2026. URL https://arxiv.org/abs/2604.06912. Alex Su, Haozhe Wang, Weiming Ren, Fangzhen Lin, and Wenhu Chen.Pixel Rea- soner:IncentivizingPixelSpaceReasoningviaCuriosity-DrivenReinforcement Learning.In Advances in Neural Information Processing Systems, volume 38, p. 8222โ8251. Curran Associates, Inc., 2025.doi:10.52202/085713-0280.URL https://proceedings.neurips.c/paper_files/paper/2025/file/ 0c38f54740062529a4117a04b583f3c-Paper-Conference.pdf. Hai-Long Sun, Zhun Sun, Houwen Peng, and Han-Jia Ye. Mitigating visual forgetting via take- along visual conditioning for multi-modal long CoT reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 5158โ 5171. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.acl-long.257. URL https://aclanthology.org/2025.acl-long.257/. Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fer- gus, Yann LeCun, and Saining Xie.Cambrian-1: A fully open, vision-centric exploration of multimodal llms.In Advances in Neural Information Processing Systems, volume 37, 2024.URL https://proceedings.neurips.c/paper_files/paper/2024/ hash/9e3a664ccfeabc0da16ac6f1f1cfe59-Abstract-Conference.html. Fei Wang, Xingyu Fu, James Y. Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, Tianyi Lorena Yan, Wenjie Jacky Mo, Hsiang-Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai-Wei Chang, Dan Roth, Sheng Zhang, Hoi- fung Poon, and Muhao Chen.MuirBench: A comprehensive benchmark for robust multi- image understanding. In The Thirteenth International Conference on Learning Representations, 2025a. URL https://proceedings.iclr.c/paper_files/paper/2025/hash/ 9cf6139382f98623d08c595622f3fb1-Abstract-Conference.html. Jiacong Wang, Zijian Kang, Haochen Wang, Haiyong Jiang, Jiawen Li, Bohong Wu, Ya Wang, Jiao Ran, Xiao Liang, Chao Feng, and Jun Xiao.VGR: Visual Grounded Reasoning. In International Conference on Learning Representations, volume 2026, p. 89506โ89528, 2026a. URL https://proceedings.iclr.c/paper_files/paper/2026/file/ 90aeee6dfe75ab6b5d6958bce40d3e16-Paper-Conference.pdf. Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image per- ception in multimodal large language models. In Proceedings of the AAAI Conference on Ar- tificial Intelligence, volume 39, p. 7907โ7915, 2025b. doi: 10.1609/aaai.v39i8.32852. URL https://ojs.aaai.org/index.php/AAAI/article/view/32852. 14 Preprint. Under review. Yubo Wang, Juntian Zhang, Yichen Wu, Yankai Lin, Nils Lukas, and Yuhan Liu. Forest before trees: Latent superposition for efficient visual reasoning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 11272โ 11288, San Diego, California, United States, July 2026b. Association for Computational Linguis- tics. doi: 10.18653/v1/2026.acl-long.517. URL https://aclanthology.org/2026. acl-long.517/. Mingyuan Wu, Jingcheng Yang, Jize Jiang, Meitang Li, Kaizhuo Yan, Hanchao Yu, Minjia Zhang,Chengxiang Zhai,and Klara Nahrstedt.VTool-R1:Vlms learn to think with images via reinforcement learning on multimodal tool use.In The FourteenthInternationalConferenceonLearningRepresentations,2026.URL https://proceedings.iclr.c/paper_files/paper/2026/hash/ 7e6f445a74cdb71931aac64f1e3f49c9-Abstract-Conference.html. Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, p. 13084โ13094, 2024. URL https://openaccess.thecvf.com/content/ CVPR2024/html/Wu_V_Guided_Visual_Search_as_a_Core_Mechanism_in_ Multimodal_CVPR_2024_paper.html. xAI. Grok-1.5 Vision Preview. Official xAI announcement introducing RealWorldQA, 2024. URL https://x.ai/news/grok-1.5v. Sihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang, Mo Li, Jingli Lin, Chenming Zhu, Xiaochen Chen, Haodong Duan, Xiangyu Yue, Dahua Lin, Tai Wang, and Jiang- miao Pang.MMSI-Bench:A benchmark for multi-image spatial intelligence. InTheFourteenthInternationalConferenceonLearningRepresentations,2026a. URL https://proceedings.iclr.c/paper_files/paper/2026/hash/ fe5d4bd3e2af823701b5d8f70a3c7602-Abstract-Conference.html. Zeyuan Yang, Xueyang Yu, Delin Chen, Maohao Shen, and Chuang Gan.Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 33510โ33520, June 2026b. URL https://openaccess.thecvf.com/content/CVPR2026/html/ Yang_Machine_Mental_Imagery_Empower_Multimodal_Reasoning_with_ Latent_Visual_Tokens_CVPR_2026_paper.html. Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, Qiaobo Chen, Yinyuting Yin, Hao Zhang, Tengfei Shi, Liang Wang, Qiang Fu, Wei Yang, and Lanxiao Huang. Mastering complex control in MOBA games with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020. URL https://arxiv.org/abs/1912.09729. Runpeng Yu, Xinyin Ma, and Xinchao Wang. Auto-Controlled Image Perception in MLLMs via Vi- sual Perception Tokens. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 21822โ21831, October 2025. URL https://openaccess.thecvf. com/content/ICCV2025/html/Yu_Auto-Controlled_Image_Perception_ in_MLLMs_via_Visual_Perception_Tokens_ICCV_2025_paper.html. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. M-Vet: Evaluating large multimodal models for integrated capabilities. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 57730โ57754. PMLR, 2024. URL https: //proceedings.mlr.press/v235/yu24o.html. Xinlei Yu, Chengming Xu, Guibin Zhang, Zhangquan Chen, Yudong Zhang, Yongbo He, Peng-Tao Jiang, Jiangning Zhang, Xiaobin Hu, and Shuicheng Yan. VisMem: Latent vision memory unlocks potential of vision-language models.In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 31544โ31555, June 2026. URL https://openaccess.thecvf.com/content/CVPR2026/html/Yu_VisMem_ Latent_Vision_Memory_Unlocks_Potential_of_Vision-Language_ Models_CVPR_2026_paper.html. 15 Preprint. Under review. Guanghao Zhang, Tao Zhong, Yan Xia, Mushui Liu, Zhelun Yu, Haoyuan Li, Wanggui He, Dong She, Yi Wang, and Hao Jiang. CMMCoT: Enhancing Complex Multi-Image Comprehension via Multi-Modal Chain-of-Thought and Memory Augmentation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(15):12430โ12438, 2026a. doi: 10.1609/aaai.v40i15.38236. URL https://ojs.aaai.org/index.php/AAAI/article/view/38236. Guibin Zhang,Muxin Fu,and Shuicheng Yan.MemGen:Weaving Genera- tiveLatentMemoryforSelf-EvolvingAgents.InInternationalConference onLearningRepresentations,volume2026,p.22555โ22588,2026b.URL https://proceedings.iclr.c/paper_files/paper/2026/file/ 26407588dfb08e48c459c074ab6adb7d-Paper-Conference.pdf. Xin Zhang, Qiqi Tao, Jiawei Du, Moyun Liu, and Joey Tianyi Zhou. Visual latents know more than they say: Unsilencing latent reasoning in MLLMs. arXiv preprint arXiv:2605.02735, 2026c. URL https://arxiv.org/abs/2605.02735. Yang Zhang, Xiaoshuai Sun, Rui Zhao, Wujin Sun, Yidong Chen, Jiayi Ji, Qian Chen, and Rongrong Ji. Look on demand: A cognitive scheduling framework for visual evidence acquisition in mul- timodal reasoning. In Proceedings of the 43rd International Conference on Machine Learning, 2026d. URL https://arxiv.org/abs/2605.28160. Ziwei Zheng, Minghao Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu.DeepEyes:Incentivizing โthinking with imagesโ via reinforce- ment learning.In The Fourteenth International Conference on Learning Representations, 2026. URL https://proceedings.iclr.c/paper_files/paper/2026/hash/ cdb347d7516d52a9280ea9d6708911f6-Abstract-Conference.html. Jiazhou Zhou, Yucheng Chen, Hongyang Li, Qing Jiang, Hu Zhou, Ying-Cong Chen, and Lei Zhang. V-Reflection: Transforming mllms from passive observers to active interrogators, 2026. URL https://arxiv.org/abs/2604.03307. AAPPENDIX A.1RELATED WORK AND POSITIONING Memory curation: acquiring and selecting visual evidence. Memory curation determines which evidence reasoning acquires. DeepEyes learns crop requests (Zheng et al., 2026), while Pixel Reasoner combines image cropping and video-frame selection (Su et al., 2025). VLM-R 3 interleaves region selection, cropping, and zooming with textual reasoning (Jiang et al., 2025). VPT supports token-guided cropping and state-conditioned full-image re-encoding (Yu et al., 2025). CMMCoT generates image indices and boxes to re-encode entity regions during multi-image reasoning (Zhang et al., 2026a). Qwen Look Again, ICoT, and MINT-CoT instead select or reuse encoded visual to- kens (Chu et al., 2025; Gao et al., 2025; Chen et al., 2025). VTPerception-R1 grounds reasoning in explicit perceptual descriptions (Ding et al., 2025), while perception-time scaling decomposes per- ceptual estimates into symbolic intermediate steps (Li et al., 2026b). VisMem uses state-conditioned formers to produce perceptual and semantic memory (Yu et al., 2026). V-Reflection trains latent rea- soning states with spatial supervision (Zhou et al., 2026). Memory organization: representing visual evidence. Visual Sketchpad represents intermediate evidence as generated sketches (Hu et al., 2024), VTool-R1 as annotated images (Wu et al., 2026), and CSMR as text (Zhang et al., 2026d). Take-along Visual Conditioning pools visual tokens (Sun et al., 2025), while Persistent Visual Memory projects image embeddings into a lower-dimensional space (Huang et al., 2026). CMMCoT retrieves cross-image cues from cached per-layer visual keyโ value features (Zhang et al., 2026a). FlexMem compresses and retrieves visual KV memories for long-video understanding (Chen et al., 2026c). MemGen and VisMem represent memory as latent token sequences (Zhang et al., 2026b; Yu et al., 2026). Laser and Mirage represent intermediate reasoning through continuous hidden states or self-generated visual tokens (Wang et al., 2026b; Yang et al., 2026b). MCOUT-Multi refines continuous thoughts by attending to multimodal input embeddings (Pham & Ngo, 2025), while Cognitive Supersensing predicts answer-image latents and 16 Preprint. Under review. Table 4: Qualitative positioning of representative visual-memory methods. Green checks indi- cate explicit support, red crosses indicate non-core mechanisms, and gray dashes indicate dimen- sions not emphasized in the cited work. Method Pixel / Region Reacquisition Compact Visual Memory State- Conditioned Access Cross-Image Context Global / Local Split Invocation vs. Continuation DeepEyesโโโโ Pixel Reasonerโโโโ VLM-R 3 โโโโ CMMCoTโโโโ Qwen Look Againโโโโ ICoT / MINT-CoTโโโโ MemGenโโโโ VisMemโโโโ FlexMemโโโโ ReGroundโโโโโ ReMAP (ours)โ uses them to guide rationale training (Li et al., 2026a). IVT-LR combines recurrent hidden states with selected input-image embeddings for latent reasoning (Chen et al., 2026a). ILVR interleaves text with generated visual latents supervised by selected features from intermediate helper images (Dong et al., 2026b). UniVLR supervises latent reasoning with visual encodings of rendered rea- soning text and auxiliary images (Jiang et al., 2026). Zhang et al. (2026c) identify underused visual latents and optimize them at inference time with a frozen backbone. Memory access: timing and selecting memory use. Take-along Visual Conditioning and Qwen Look Again reactivate visual evidence at reflection events (Sun et al., 2025; Chu et al., 2025). Re- Ground uses self-diagnosis to guide re-examination (Peng et al., 2026), and VGR triggers region- specific replay from precomputed visual features (Wang et al., 2026a). DaP-ICoT inserts selected object images when the reasoning stepโs mean logit margin falls below a threshold (Liu et al., 2026b). MemGen learns a memory trigger (Zhang et al., 2026b). LatentMorph learns when to invoke latent reasoning over compact visual memory during image generation (Chen et al., 2026b). VisMem inserts perceptual or semantic latent memory through invocation tokens (Yu et al., 2026). Its two memories differ in abstraction level rather than in spatial scope, and a request is not compared with continuing from the same state. CSMR invokes a separate perception module (Zhang et al., 2026d). For policy training, ARPO allocates entropy-guided rollout branches after tool feedback (Dong et al., 2026a). PEPO combines visual hidden-state similarity and token entropy to reweight token advantages without auxiliary branches (Li et al., 2026c). Positioning ReMAP. ReMAP bridges these dimensions by connecting active visual evidence ac- quisition with latent memory and learned access. Within memory curation, it emphasizes the com- plementarity between newly acquired local details and the scene and cross-image context needed to interpret them. Within memory organization, it brings pixel-level reacquisition into compact latent memory, connecting renewed perception with an efficient representation of visual evidence. Within memory access, it focuses on the utility of requesting evidence at a given reasoning state, using comparisons between invocation and continuation to guide selective access. Together, these choices support sustained visual grounding throughout reasoning. A.2RUNTIME AND MEMORY INTERFACE Memory transactions. ReMAP begins autoregressive reasoning from a coarse visual prefix of the input images. Ordinary autoregressive generation continues by default. A memory transaction is performed only when the decoder emits a Global or Local request token. The corresponding transactions are โจmem gโฉ; Z G ; โจmemgendโฉ , โจmemlโฉ; Z L ; โจmemlendโฉ .(4) 17 Preprint. Under review. After the decoder emits a request token, the runtime inserts the corresponding latent block and end marker into the current decoder sequence. Generation resumes from the position following the end marker. End markers are not sampled as vocabulary outputs, and returned latents enter the decoder as continuous inputs. Each reasoning trajectory permits at most five memory requests. After the request budget is exhausted or the model commits an answer, Global and Local request tokens are masked. Inserted latents and end markers occupy successive positions in the decoder sequence and KV cache. Request tokens and these inserted positions do not count toward the ordinary-text budget, but they still consume decoder context capacity. Global memory. Global curation forms a question-conditioned latent block from the ordered in- put images. For an input containing N images, four tokens are allocated to each image and four additional tokens encode cross-image relations: K G (N ) = 4N + 4.(5) The resulting block is formed before decoding, cached throughout the reasoning trajectory, and inserted at a new sequence position whenever the decoder emits a Global request. Repeated Global requests neither re-encode the input images nor regenerate the cached block from a later reasoning state. Local memory. The Local request uses the current reasoning state to select image sources and regions. The selector retains at most three sources, stopping when cumulative selection probability reaches 0.85. A frozen vision encoder maps the selected crops to region features, and the Local Weaver combines these features with the reasoning state and cached Global memory: Z L = [W L (h t ,R t,i ;Z G )] iโS t .(6) Each Local request returns 32 latent tokens. Global memory supplies context for Local compression but is not concatenated into the 32-token Local output. The 4N + 4 Global layout, the 32-token Local return, the three-source cap, and the 0.85 threshold define the memory interface used in all main experiments. The Source Selector reads the layer-12 state exposed at a request boundary, and Twig uses layer-18 prefix states selected by a layer sweep. Stage I rollout branching follows the partial-rollout strategy of ARPO (Dong et al., 2026a) and is specified with the access objective below. A.3MEMORY FORMATION AND ACCESS OBJECTIVES A.3.1GLOBAL LAYOUT AND CACHING Each per-image Image-Weaver span contains four latent queries and is restricted by its attention mask to the visual activations of its own source image. Thus Z img i โ R 4รd . The Relation Weaver uses four queries over the concatenated image summaries [Z img 1 ;ยท ;Z img N ]; its relation span can attend to all sources, givingZ rel โ R 4รd . Both weavers are conditioned on the encoded question, as in Equation 1. The Global block is therefore Z G = [Z img 1 ;ยท ;Z img N ;Z rel ]โ R (4N +4)รd ,(7) which is the K G (N ) = 4N + 4 layout used in the capacity sweep of Figure 7. We compute this block once from the prompt-time images and question, retain it as Z G , and reuse the same latent values at every later Global request. Only the decoder insertion position and its resulting KV entries are new; later reasoning states determine whether a request is made, but do not recompute or alter the cached block. A.3.2LOCAL SELECTION AND COMPRESSION The Source Selector uses the layer-12 request state h t = h 12 t to rank the source images. If p t,i denotes the normalized score of source i, the selector orders sources by decreasing score and keeps 18 Preprint. Under review. the shortest prefix whose cumulative score reaches 0.85, with a maximum of three sources. When the threshold is not reached within three sources, it keeps the three highest- scoring sources. The layer-12 request tap and layer-18 Twig tap are ReMAP settings selected by internal layer sweeps: layer 12 supplies the state at a request boundary, while layer 18 provides the spatially arranged activations used for localization. Twig Localization uses the layer-18 prefix states H 18 <t to locate a region b t,i in each selected image I i . Let V c i denote the coarse visual features of image I i . Source selection, region localization, and frozen crop encoding are written as S t = f sel h t ,V c i N i=1 , b t,i = f loc H 18 <t ,V c i , R t,i = E roi I i [b t,i ] . (8) Here, I i [b t,i ] is the selected crop, and E roi is the frozen vision encoder that produces its region features R t,i . After processing H 18 <t with its decoder blocks, Twig applies query/key projections and multimodal rotary position embeddings (mRoPE). In Figure 3(a), q h t denotes the query at the current reasoning position for head h, and K h img contains the keys at the selected imageโs visual-token positions. The ROI heatmap is the head-averaged scaled dot product reshaped to the imageโs coarse spatial grid: q roi = reshape G h รG w 1 H H X h=1 q h t (K h img ) โค โ d h ! .(9) For readout, q roi is treated as a logit map consistent with the weighted-BCE localization objective. We retain grid cells with q roi > 0 (equivalently, sigmoid probability at least 0.5), take the minโmax box of their row and column coordinates, map the box from the coarse grid to image pixels, and clip it to the image boundary. The resulting box is b t,i for crop encoding. For local compression, the Local Weaver conditions its learnable compression queries Q L on the reasoning state h t and cached Global memory Z G . Twigโs q t is used only for region localization. The crop featuresR t,i provide fine-grained evidence, whileZ G supplies image summaries and cross- image relational context. Equation 2 summarizes source-specific compression and concatenation into a return with K = 32 latent tokens in the default configuration. A.3.3FORMATION AND WARM-UP OBJECTIVES Continuation distillation. The Image, Relation, and Local Weavers are trained through the frozen LLM. For a teacher trajectory with prefixP t and continuation Y , we insert a Global or Local trans- action at position t and minimize L suffix =โ 1 |Y| |Y| X k=1 logp LLM Y k |P t , [โจmemโฉ;Z;โจmemendโฉ], Y <k ,(10) where Z is Z G or Z L and the markers match its memory type. The LLM is frozen, so gradients reach only the weaver that produced Z. The loss therefore trains each weaver to encode the evidence that the teacherโs subsequent reasoning uses. Together withL suffix , fact, relation, and answer supervision from the Stage I annotations (Appendix A.5) trains the weavers. ROI localization. Twig produces ROI logits q roi on the coarse grid of each selected image. The teacher-reviewed box is rasterized to a binary target ฬ A on the same grid. With P = sigmoid(q roi ), the localization objective is L ROI =L wBCE ( ฬ A,q roi ) + 1โ softIoU( ฬ A,P ), softIoU( ฬ A,P ) = P ฬ Aโ P P ฬ A + P P โ P ฬ Aโ P . (11) Q-Zoom (Shi et al., 2026) trains its ROI predictor with binary cross-entropy. We weight positive cells by the ratio of negative to positive cells, so that a small target region is not dominated by background cells. We cap this weight at 50 based on our experience. The soft-IoU term matches the overall extent of the heatmap to the target box. 19 Preprint. Under review. Student-state adaptation. Teacher trajectories do not contain the states that the student reaches at inference. We therefore run the student autoregressively and apply the same two objectives to its states. For a student state s t with complete text-and-latent prefixP t , the teacher reviewsP t together with the selected crop I i [b t,i ] and returns an ROI target ฬ A and a continuation target Y . Twig is updated withL ROI , and the weavers withL suffix conditioned onP t . Interface warm-up. Each warm-up sequence either contains no memory transaction or contains one Global or Local transaction at a request boundary. The frozen formation modules compute Z, which enters the sequence as a constant input. Warm-up minimizes next-token cross-entropy over the supervised positionsT : L warm =โ 1 |T| X kโT logฯ(y k | y <k ), T =T call โชT end โชT later .(12) Here T call contains the request token, T end the first text token after the end marker, and T later the remaining reasoning and answer tokens. The three sets giveL call ,L end , andL later in Figure 3(b). Prompt tokens, inserted latents, and end markers are excluded fromT . Sequences without a trans- action supervise ordinary text at states where no memory is requested, so the decoder does not learn to request memory at every state. Only the four marker input rows and the two request output rows are updated. The end-marker input rows receive gradients through the text predicted after the return. A.3.4ACCESS LEARNING OBJECTIVE Policy parameters. Let ฯ ฯ,ฯ denote the decoding policy. The symbol ฯ denotes the decoder LoRA parameters, and ฯ = (ฮท G ,ฮท L ,w G ,w L ) denotes the request interface, where ฮท G and ฮท L are the request-token input embeddings and w G and w L are their output-classifier rows. All other pa- rameters, including the end-marker embeddings and the memory-formation modules, stay fixed. Let V t be the legal vocabulary and h ฯ,ฮท (P t ) the decoder state over the complete text-and-latent prefix. The request rows share the output softmax with the frozen ordinary rows w v : โ(v | s t ) = w โค v h ฯ,ฮท (P t ), ฯ ฯ,ฯ (v | s t ) = expโ(v | s t ) P uโV t expโ(u| s t ) .(13) This softmax gives p t (G) and p t (L), and the remaining mass is p t (N) = 1โ p t (G)โ p t (L). An ordinary output z has probability p ฯ,ฯ (N | s t )q ฯ,ฮท (z | s t , N). Updating w G and w L changes the request logits and the shared normalizer, and updating ฮท G and ฮท L changes the hidden states after a request. Invocation credit therefore acts on the whole suffix after a request, including the text generated after the returned latents. Rollouts and branch selection. Each prompt samples four root trajectories from the behavior pol- icy ฯ old . Candidate branch states precede answer commitment and admit at least one legal request, including the initial state. Each root branches once, at the candidate with the largest u(s t ,a t ) in Equation 3, with ties resolved by the earliest position. From the shared prefix, two siblings (m = 2) are sampled independently, each with its own subsequent requests and reasoning. Requests in the shared prefix count toward the five-request budget. The roots and their siblings form the natural groupG, so 4โค|G|โค 12. Rewards. For a completed trajectory ฯ i , the terminal reward scores its extracted answer หy(ฯ i ) against the reference y โ : R i = Score(หy(ฯ i ),y โ )โ [0, 1].(14) Typed answer checks are used where applicable, and semantic evaluation gives graded scores; for example, listing questions receive per-example F1. A prompt enters the policy losses only when all natural trajectories and the required continuation controls have valid scores. Unavailable scores are not treated as zero. Same-state continuation baseline. For branch b, the siblings at b are the root and its resampled continuations, and G b contains them. Let D b = i โ G b : a i,b = N contain the siblings that continue without a request at b, which can include the root. The baseline B b in Section 3.2 is their mean reward: B b = |D b | โ1 P iโD b R i , D b ฬธ= โ , R(ฯ aux b ),D b = โ , (15) 20 Preprint. Under review. where ฯ aux b starts from the same branch state and is forced to emit an ordinary token at its first decision. Its later decisions follow ฯ old and may request memory. The auxiliary continuation only supplies B b and does not enterG. Advantages. The decoder uses the group-relative advantage of GRPO (Shao et al., 2024b), and the request interface uses the normalized gain over continuation at the same branch state: ห A dec i = R i โ ฬ R G ฯ G + ฮด , ห A call i = R i โ B b(i) ฬฯ b(i) , ฬฯ b = ฯ b + ฮด, ฯ b > ฮด, 1,ฯ b โค ฮด. (16) Here ฬ R G and ฯ G are the mean and sample standard deviation of rewards inG, and b(i) is the branch at which trajectory i requests memory. The branch scale ฯ b is the sample standard deviation of the rewards in G b together with one additional entry B b . We set ฮด = 10 โ6 . Equal rewards give zero advantage in both terms. Policy objectives. Let z ij be the vocabulary output at decision j of trajectory i, including ordinary text, request tokens, and EOS. Its likelihood ratio ฯ ij = ฯ ฯ,ฯ (z ij | s ij )/ฯ old (z ij | s ij ) uses the same legal-output mask as sampling. We use the dual-clipped policy loss (Ye et al., 2020): โ clip (ฯ,A) =โ minฯA, clip(ฯ, 1โ ฮต, 1 + ฮต)A, โ dc (ฯ,A) = minโ clip (ฯ,A),โcA, A < 0, โ clip (ฯ,A),Aโฅ 0, (17) where ฮต is the ratio-clipping width and c > 1 bounds the loss for negative advantages. The decoder term uses all T i sampled decisions of each natural trajectory, including request tokens, EOS, and the shared prefix of each sibling, with N dec = P iโG T i . The invocation term uses the set C of decisions from natural trajectories that request memory at their branch, from the request through answer commitment; it includes ordinary text as well as later requests. Returned latents and end markers are inserted by the runtime and contribute no policy term. With N call = max(1,|C|), the two objectives are L dec = 1 N dec X iโG T i X j=1 โ dc (ฯ ij , ห A dec i ) + ฮป CE L CE , L access = 1 N call X (i,j)โC โ dc (ฯ ij , ห A call i ) + ฮป pref L pref . (18) The policy part ofL access is zero whenC is empty. Only ฯ receives gradients fromL dec , and only ฯ receives gradients from L access . Each objective is computed per prompt and averaged over the prompts eligible for it. The values of ฮต, c, ฮป CE , and ฮป pref are listed in Table 5. Post-call answer supervision. LetP + t be the complete prefix just after a memory return, including the request, its latents, and its end marker. The target Y contains the answer-format prefix, the reference answer, and EOS. With memory requests masked, the auxiliary loss is L CE =โ 1 |Y| |Y| X k=1 logฯ text ฯ,ฯ (Y k |P + t ,Y <k ).(19) The return is taken from the last pre-answer request of the first root that makes one. If no root requests memory, a forced initial Global or Local probe supplies the return. The prefixP + t contains no later returns, reasoning suffix, or reference answer. This term trains the decoder to answer from returned evidence and needs only a valid reference, independent of the rollout reward. Initial-state request preferences. Three probes start from the initial state s 0 . One is forced to emit an ordinary token, and the other two emit the Global and Local request tokens; later decisions follow ฯ old . Let โ a = R(ฯ a 0 )โ R(ฯ N 0 ) for aโG, L. The preference loss is L pref = 1 2 X aโG,L |โ a | softplus โ sgn(โ a ) log p ฯ (a| s 0 ) p ฯ (N| s 0 ) .(20) 21 Preprint. Under review. The loss raises the probability of a request relative to continuation when that request improves the reward, and lowers it otherwise. The initial hidden state is detached, so only w G and w L receive gradients. The probes do not enter either policy term. A prompt uses this term only when all three probes have valid scores. A.4TRAINING CONFIGURATION Figure 8 shows the five training steps. The vision encoder and the pretrained LLM weights stay frozen in every step, and each step updates only the modules marked as trainable. The Stage I objectives are defined in Appendix A.3.3, and the Stage I objectives in Appendix A.3.4. updated / trainablefrozen 1 Weaver Distillation 2 Twig Training 3 Student-State Adaptation 4 Interface Warm-up 5 Access Learning (Stage I) ํฟ ํํํํ =ํถํธํกํํํโํํํถํํ +ํํํํกํ +ํํํํํกํํํํ +ํํํ ํคํํํ Global / Local formation memory Teacher-reviewed ROI targets Twig Localization Vision encoder + trained weavers ํฟ ํ ํํผ =ํฟ ํคํตํถํธ +1โํ ํํํกํผํํ ROI heatmap / crop box Student-generated prefixes +selected crop Twig + memory weavers Vision encoder + pretrained LLM + decoder LoRA ํฟ ํ ํกํขํํํํก =ํฟ ํ ํํผ +ํฟ ํ ํขํํํฅ adapted returns request / return markers <request> ... <return> 4 marker input rows + 2 request output rows ordinary vocabulary rows + formation modules ํฟ ํํํํํ =ํถํธํํํํขํํ ํก+ ํถํธํกํํฅํกํํํกํํํํกํขํํ protocol-ready model โ root trajectories + CRD siblings ... Decoder LoRA ฯ Request interface ฯ backbone + weavers + end-marker rows ํฟ ํํํ =ํํขํํโํํํํํด ํ ํํ+ํ ํถํธ ํฟ ํถํธ group rewards / invocation gains separate gradient scopes: ฯ โ L_dec, ฯ โ L_access Teacher CoT + facts / relations/answers Q: ... ? A: ... CoT: ... Facts: ... Image Weaver + Relation Weaver + Local Weaver Vision encoder + pretrained LLM LLM <prefix> ... โ LLM LoRA ํฟ ํ ํขํํํฅ =ํถํธลถ,ํ LLM ํฟ ํํํํ =ํํขํํโํํํํํด ํ ํํ+ ํ ํํํํ ํฟ ํํํํ colored blocks denote schematic formation modules or latent groups; labels give their role Color key: Figure 8: Five-step training pipeline. Each row lists the inputs of one step, the updated modules (flame), the frozen components (snowflake), the objective, and the output. Steps 1โ4 form Stage I, and step 5 is Stage I access learning, in which the decoder LoRA ฯ and the request interface ฯ receive gradients only fromL dec andL access , respectively. Optimization. Table 5 lists the optimizer settings. The ratio-clipping width ฮต = 0.2 and the dual- clip constant c = 3 follow Ye et al. (2020), and ฮด = 10 โ6 only stabilizes the normalization in Equation 16. The decoder LoRA (Hu et al., 2021) is attached to the query, key, value, output, and MLP projections of the top four decoder layers, with rank r = 16 and ฮฑ = 32. We set the learning rates, the LoRA configuration, the auxiliary loss weights, and the gradient-norm limit based on our experience. The Global Weavers use a separate learning rate for the output projection that maps their latents to the decoder embedding space. Each Stage I step makes one pass over its supervision. The random seed is 20260911. Rollouts and training length. Stage I runs on 16 A800 GPUs with one prompt per GPU and no gradient accumulation, so each step schedules 16 prompts. Each prompt samples four root trajec- tories at temperature 1, and each eligible root adds two siblings, so a step samples 64โ192 natural trajectories. Continuation controls and preference probes are sampled in addition. Rollouts use a 768-token ordinary-text budget, a 64-token answer-completion grace period, and at most five mem- ory requests. These rollout sizes and budgets are set empirically. The 40,000-example schedule contains 2,500 steps. Checkpoint selection. We evaluate the model every 100 steps and at the end of training on a fixed development set whose task list is set before training. The selected checkpoint maximizes the equally weighted mean of per-task rewards, and ties select the earlier checkpoint. The same weights are used for all final benchmarks, and final benchmark scores do not enter the selection. 22 Preprint. Under review. Table 5: Optimization settings for memory formation and access learning. StageParameter group or settingValue I: Global formationImage and Relation Weavers10 โ6 Output projection10 โ5 I: Local formationSource Selector, Twig, and Local Weaver 10 โ6 I: interface warm-upMarker and request rows10 โ4 I: access learningDecoder LoRA10 โ6 Decoder LoRA rank r / ฮฑ16 / 32 Decoder LoRA layerstop 4 of 28 Request rows3ร 10 โ4 Ratio-clipping width ฮต0.2 Dual-clip constant c3 Answer-loss weight ฮป CE 0.1 Preference-loss weight ฮป pref 0.1 All stagesOptimizer / weight decayAdamW / 0 Maximum gradient norm1 A.5TRAINING DATA Sources and allocation. We construct 50,000 imageโquestion examples for Local memory forma- tion, 30,000 for Global memory formation, and 40,000 for Stage I access learning. Each example contains one or more images, a question, and a reference answer. Table 6 lists the source datasets. All examples come from the training splits of these sources and are deduplicated against every eval- uation unit in Table 7, so there is no leakage between training and evaluation data. Table 6: Training data sources. Source datasets provide questionโanswer pairs, region annotations, or instance annotations from which additional questions are constructed. SourceContent and use Visual-CoT (GQA) (Shao et al., 2024a) Object, attribute, and spatial-relation questions with region annotations. COCO train2017 (Lin et al., 2014)Multi-image questions built from instance annotations. Mantis-Instruct (Jiang et al., 2024): ChartQA, Spot-the-Diff, ImageCoDe, NLVR2, and Birds-to-Words Chart reading and arithmetic, image comparison, description-to-image matching, and cross-image relational judgments. Q-Zoomโs VStar-COCO data (Shi et al., 2026) Spatial-relation questions on COCO train2017 images. ScanQA via M4-Instruct (Li et al., 2024) Multi-view scene question answering, restricted to examples from the original ScanQA training split. Stage I annotations. Each Stage I example carries a reference CoT, per-image facts, cross-image relations, the answer, and target regions. Qwen3-VL-32B-Instruct generates the reference CoTs. Answers come from the source datasets. Target regions come from source region annotations and from model-generated step annotations, which are checked against the source images and the pro- posed crops. Per-image facts and cross-image relations are derived from source annotations and from model-generated pseudo-labels. Facts and relations supervise what Global memory encodes, region annotations supervise Local source selection and localization, and answers supervise task completion. Student-state targets. For student-state adaptation, we roll out Qwen2.5-VL-7B-Instruct and record each text-and-latent prefix with its source selections, crops, and memory returns. The teacher reviews each prefix together with its crop and returns only the ROI target ฬ A and the continuation target Y . We do not reuse the full teacher CoT as the continuation of a student state, because that CoT need not follow from the studentโs prefix. 23 Preprint. Under review. Access-learning data. Stage I examples cover single-image perception, multi-image reasoning, and general visual question answering. Rewards and the post-call answer loss use the source an- swers, and the Stage I memory annotations are not used in Stage I. A.6EVALUATION SPLITS AND SCORING The evaluation suite contains ten benchmark families, divided into fifteen evaluation units with 19,913 examples in total (Table 7). Paired comparisons evaluate both conditions on the same exam- ples in every unit. BLINK uses its labelled validation split, and MathVista uses test-mini. Mecha- nistic subsets and development probes are reported separately from these evaluation units. Table 7: Evaluation units and aggregation rules. Scores are reported as percentages. The example count is the full evaluation denominator. Evaluation unitExamplesAggregated metric Single-image perception V*Bench, released test191Micro accuracy CV-Bench, test-2D1,438Mean source accuracy CV-Bench, test-3D1,200Micro accuracy HR-Bench, 4K800Mean category accuracy HR-Bench, 8K800Mean category accuracy Multi-image reasoning BLINK, labelled validation1,901Mean subtask accuracy MuirBench, released test2,600Micro accuracy MMSI-Bench, non-circular set1,000Micro accuracy MIMIC, Common test1,000Accuracy MIMIC, Counting test5,000Accuracy MIMIC, Odd-One-Out test1,000Accuracy MIMIC, Listing test1,000Corpus micro set F1 Broader visual understanding M-Vet v1, released set218Mean fractional judge credit MathVista, test-mini1,000Accuracy RealWorldQA, released test765Micro accuracy Total19,913 Aggregation. CV-Bench-2D averages the accuracies of its two sources with equal weight. Each HR-Bench split averages four category accuracies, and BLINK averages its fourteen subtask accu- racies. These macro averages differ from micro accuracy when components differ in size. MIMIC keeps its released metrics: accuracy for Common, Counting, and Odd-One-Out, and corpus-level micro set F1 for Listing, which is not an average of per-example F1 scores. The MIMIC Avg. col- umn in the main tables is the arithmetic mean of these four scores and is not an official MIMIC metric. Unresolved predictions stay in the denominator and receive no credit. Judged benchmarks. M-Vet is scored by the mean fractional credit from a fixed DeepSeek-V4- Flash judge with a fixed JSON prompt. MathVista uses deterministic scoring for multiple-choice an- swers and the same judge for mathematical equivalence of free-form answers. These judges replace the online GPT-based judging of the original benchmarks. Comparison protocol. All methods in Table 1 are evaluated on the same examples with the same released questions, answer options, image order, answer-format instructions, scoring rules, and judge. Each baseline uses its authorsโ released checkpoint, tools, stopping rules, and generation caps and is not retrained on our data; single-image methods receive an ordered contact sheet for multi-image inputs. Because baselines differ in backbone and training data, the main table also re- ports the backbone controls Native and Coarse and the same-data controls SFT and Stage I sft , which share ReMAPโs backbone and training data. Prompts and generation settings. Native-protocol comparisons keep the released question, an- swer options, image order, and answer-format instruction. Baselines that accept only one image 24 Preprint. Under review. receive an ordered contact sheet of all input images, marked with a dagger in Table 8. Generation budgets, stopping rules, and the memory-request cap are reported with each evaluation condition. Additional baseline configurations. MINT-CoT and ICoT use their Qwen2-VL-7B implementa- tions, and Pixel Reasoner uses its released Qwen2.5-VL-7B policy. ICoT keeps all input images but replays visual tokens from the last image through its released selection rule. Pixel Reasoner uses a 65,536-token runtime context for multi-image inputs, caps each crop at 1,024 visual tokens and each generation round at 1,024 output tokens, keeps a 5,120-token cap on initial images, and preserves its released crop tools and stopping rules. Q-Zoom caps its initial image or ordered contact sheet and each RoI at 576 visual tokens, with at most 576 output tokens. These caps apply to each input or crop, not to the whole trajectory. Cross-backbone evaluation. In Table 2, Base denotes the original backbone, OFF the trained model with Global and Local requests disabled, and ReMAP the full model.The six benchmark columns are computed as 100n/N from the correct counts n, with N = 191, 1200, 800, 1901, 2600, 1000 in column order. For HR-Bench-8K and BLINK, these are mi- cro accuracies rather than the official four-category and fourteen-subtask macro averages used in the main table. For every backbone, MIMIC Avg. is the mean over the four MIMIC subsets. Differences are computed from unrounded ratios and rounded to two decimals. Paired scoring. The mechanistic comparisons share source examples, decoding settings, and scor- ing rules; each study specifies its memory interface, request positions, and content controls. Every expected example stays in the denominator, and an unresolved prediction is scored as wrong. We re- port aggregate accuracy and paired HELP/HARM counts. Confidence intervals use a sample-clustered bootstrap that resamples all states of an example together. The full-cohort visual-influence and static pixel-budget analyses use 5,000 percentile resamples with seed 20260828, and the state-conditioned utility and content-control analyses use 4,000 resamples with seed 20260830. Budget-sweep inter- vals are pointwise across the five tested values of K. MIMIC components. Table 8 lists the four MIMIC component scores behind the MIMIC Avg. column of the main table. Table 8: MIMIC component scores. A dagger denotes an ordered contact-sheet adaptation. Vis- Mem is the public-code reproduction used in the main table. MethodCommonCountingOdd-One-OutListingMean Native62.6044.9860.4012.6945.17 Coarse61.1038.4254.7020.0243.56 Q-Zoom โ 51.9037.4454.8025.0042.29 DeepEyes โ 52.6040.3853.2010.9039.27 Pixel Reasoner64.5043.3430.0041.4644.83 ReGround53.6036.7613.008.6628.01 ICoT48.8015.3854.900.5529.91 MINT-CoT54.3030.0618.5033.3334.05 DMLR โ 38.7016.1417.7022.4123.74 VaLR38.1012.3820.600.6217.93 V-Reflection85.1031.0458.2038.2853.16 VisMem60.3037.8255.1011.5541.19 ReMAP92.0071.7680.0028.2468.00 A.7VISUAL INFLUENCE AND STATE-CONDITIONED UTILITY The two probes in this section measure how much the initial images still affect predictions as rea- soning proceeds, and whether the value of renewed visual access depends on the reasoning state. Both use a fixed Q-Zoom-adapted Qwen2.5-VL-7B checkpoint. 25 Preprint. Under review. A.7.1VISUAL INFLUENCE DECAY For each example, we generate one deterministic reasoning trajectory. We then teacher-force the same text while replacing all input images with a same-layout patch-shuffled or blank version, and compare the next-token distributions p t and ep t under the original and perturbed images: f t = 1[arg maxp t ฬธ= arg maxep t ], d t = JS(p t ,ep t ).(21) The flip indicator f t records a change of the most likely next token, and the JensenโShannon diver- gence d t also records changes that keep that token. Positions are normalized by trajectory length and pooled into eight bins. Both statistics measure the size of the visual influence, not its direction. Figure 9 plots the flip rate across the eight bins, and Table 9 compares the first and last parts of each trajectory. 00.250.50.751 0% 5% 10% 15% 20% Normalized reasoning position Next-token flip rate (a) Pooled, 697 trajectories Patch-shuffled Blank 00.250.50.751 0% 5% 10% 15% 20% Normalized reasoning position (b) Patch-shuffled, by cohort V*Bench BLINK MuirBench Figure 9: Visual influence decays along the reasoning trajectory. Rate at which the next-token top-1 decision changes when the input images are replaced, in eight bins of normalized reasoning position. (a) Patch-shuffled and blank images, pooled over V*Bench, BLINK, and MuirBench, with 95% bootstrap intervals. (b) Patch-shuffled images for each cohort. Table 9: Patch-shuffle visual influence across reasoning. Eligible/selected counts the trajectories analyzed out of the sampled examples. Firstโ last bin gives the flip rate in the first and eighth bins. Paired change is the per-example mean flip rate over the last quarter of the trajectory minus that over the first quarter, with a 95% bootstrap interval. CohortEligible / selectedFirstโ last binPaired change V*Bench189/19115.43%โ 2.36% โ8.79 [โ10.48,โ7.00] p BLINK254/25613.27%โ 2.45% โ10.88 [โ12.15,โ9.63] p MuirBench254/2567.43%โ 1.96% โ4.50 [โ5.48,โ3.55] p Pooled697/70311.73%โ 2.25% โ7.98 [โ8.78,โ7.21] p Blank images give the same pattern, with the flip rate falling from 14.82% to 2.50%. Under patch shuffling, the mean JS divergence falls from 0.0320 to 0.0039. The signed change in the log prob- ability of the generated token is not consistent across cohorts: pooled, it is โ0.0211 with interval [โ0.0450, 0.0026], and V*Bench has the opposite sign from BLINK and MuirBench. The initial images therefore influence later predictions less, but removing them does not uniformly lower the probability of the generated text. A.7.2STATE-CONDITIONED UTILITY We insert visual content at selected states of the same deterministic trajectory and compare the gold- answer log probability. Candidate states are the prompt-only state and the positions at 20%, 40%, 60%, and 80% of the chain of thought. All arms share the textual prefix, the native visual user-turn interface, and a 16-token return budget. The pair of states in each contrast is chosen on development examples, and all estimates and intervals in Table 10 are computed on an independent confirmation split. 26 Preprint. Under review. Table 10: State and content contrasts in gold-answer log probability. Lateโearly contrasts insert the same content at the later and the earlier selected state. Content contrasts compare the correct region or scene with a random wrong region, a matched wrong scene, or a shuffled scene. NS marks an interval that contains zero. CohortContrastEstimate95% CI V*Bench (n = 125)lateโearly correct-content utility+0.1771[+0.1327, +0.2227] V*Bench (n = 125)correctโrandom-wrong RoI โ0.0112[โ0.0754, +0.0486] NS CV-Bench-3D (n = 152)lateโearly current-scene utility+0.0726[+0.0405, +0.1050] CV-Bench-3D (n = 152)currentโmatched-wrong scene+0.2265[+0.1361, +0.3171] CV-Bench-3D (n = 152)currentโshuffled scene+0.0264[โ0.0358, +0.0882] NS The same correct content is more useful at the later state on both V*Bench (+0.1771) and CV- Bench-3D (+0.0726), so the value of identical evidence depends on when it is accessed. On CV- Bench-3D, the current scene is also more useful than a matched wrong scene (+0.2265). The con- trasts against a random wrong region on V*Bench and against a shuffled scene on CV-Bench-3D are not significant. A.8MEMORY-CONTENT REQUIREMENTS AND CONTROLS The two probes in this section test which memory content helps: freshly encoded pixels of a region versus features from the initial encoding, and correct relational context across images versus content- matched controls. Both use the fixed checkpoint of Appendix A.7. A.8.1SAME-BOX PIXEL RE-ENCODING On the 191 V*Bench examples, both arms use the gold RoI, the same insertion point in the prefill, and the same number K of inserted tokens. The feature arm pools the RoI features from the initial encoding. The pixel arm crops the region from the source image, encodes it at high resolution, and compresses the result to K tokens. Figure 10 reports five values of K. 48163264 55 60 65 70 75 80 Inserted tokensK Accuracy (%) (a) Accuracy Feature pooling Pixel re-encoding 48163264 โ0.1 0 0.1 0.2 0.3 0.4 Inserted tokensK (b) Pixelโfeature gold log prob. Figure 10: Pixel re-encoding versus feature pooling on V*Bench. Both arms return the gold RoI with K inserted tokens at the same insertion point on 191 examples. (a) Accuracy. (b) Pixel minus feature gold-answer log probability, with 95% CIs. Pixel re-encoding matches the feature arm at K = 4 (โ0.0007 in gold-answer log probability) and exceeds it from K = 8 onward. At K = 32, it raises the log probability by 0.2243 (95% CI [0.1241, 0.3325]) and accuracy from 62.30% to 71.73%. The high-resolution RoI pass uses 278.4 visual tokens on average, compared with 1,244.4 for the high-resolution full image. At K = 16, pixel memory also exceeds wrong-RoI, shuffled, blank, and zero controls. Splitting the examples into target-size tertiles gives the same direction in every tertile, with small differences between tertiles. 27 Preprint. Under review. A.8.2RELATIONAL-ROLE PROBE We build 192 non-overlapping partโwhole image pairs from COCO train2017 instance annotations, covering 800 unique images. In the role-changing condition, the focal image stays pixel-identical while its relation to the other image changes. In the appearance control, the appearance changes while the focal role stays fixed. Sensitivity is the total-variation distance between the modelโs answer distributions before and after a change, and fixation is the rate at which the model keeps predicting the focal option after the roles are swapped. Table 11: Relational-role probe. The upper rows use total-variation sensitivity or fixation rate on 192 pairs. The lower rows compare the oracle relational context with each control in gold-answer log probability on the 71 selected late failure states. NS marks an interval that contains zero. ContrastEstimate95% CI Role changeโ appearance control+0.0482[+0.0254, +0.0725] Lateโ early role sensitivity+0.0039[โ0.0056, +0.0138] NS Lateโ early fixation rate+0.0182[โ0.0026, +0.0391] NS Oracleโ no context+2.5797[+2.1222, +2.9881] Oracleโ blank block+2.4765[+2.1251, +2.8061] Oracleโ zero block+2.8028[+2.5116, +3.0891] Oracleโ object list only+1.3852[+1.1268, +1.6433] Oracleโ another sample+1.2927[+1.0276, +1.5654] Oracleโ wrong role+3.1493[+2.7849, +3.5274] The model is more sensitive to a role change than to an appearance change (+0.0482), so its predic- tions depend on cross-image relations and not only on the focal image (Figure 11(a)). Neither role sensitivity nor fixation changes significantly between early and late states. 20%40%60%80% 0 0.05 0.1 0.15 0.2 Reasoning position Total-variation distance (a) Sensitivity along reasoning Role changeAppearance change 0204060 Oracle Blank block Zero block Another sample Object list only Wrong role No context 57.7 53.5 52.1 49.3 47.9 35.2 0 Accuracy (%) (b) Repair of 71 failure states Figure 11: Relational-role probe. (a) Total-variation sensitivity to a role change and to an appear- ance change at four reasoning positions on 192 pairs, with 95% intervals. (b) Accuracy on the 71 selected late failure states with the oracle relational context and six controls; without context, every state is wrong by construction. To test whether correct relational context can repair these errors, we select the 71 late states at which the model predicts the focal option for both the true and the role-swapped prefix, so that its role- swapped prediction is wrong. At each state, we insert either the oracle relational context or one of six controls: no context, a blank block, a zero block, an object list only, another sampleโs context, or the wrong role. Without context, all 71 states are wrong by construction. The oracle context repairs 41 of them, an accuracy of 57.7%, compared with 53.5%, 52.1%, 47.9%, 49.3%, and 35.2% for the blank, zero, object-list, other-sample, and wrong-role controls (Figure 11(b)). In gold-answer log probability, it exceeds every control (Table 11). Because the states are selected as failures and the relation comes from an oracle, this probe measures what correct relational content can contribute, not the effect of the learned Global memory. 28 Preprint. Under review. A.9ADDITIONAL ABLATIONS Shared evaluation setting. The memory-curation and memory-access variants in Table 3 use the complete ReMAP weights and share the input examples, preprocessing, scoring, and decoding lim- its. Base is the Coarse backbone with at most 1,024 visual tokens per image. The table uses all 191 V โ Bench, 1,200 CV-Bench-3D, 2,600 MuirBench, 1,901 BLINK, and 218 M-Vet examples. V โ Bench, CV-Bench-3D, MuirBench, and BLINK report micro accuracy, and M-Vet reports the mean graded score. BLINK in this table therefore uses micro accuracy rather than the mean-subtask aggregation of Appendix A.6. Unresolved answers count as incorrect. Memory-curation variants. w/o Memory masks both request tokens. w/ Global memory and w/ Local memory enable only one request type and keep the decoder adapter, request parameters, and answer path of ReMAP. When Global requests are disabled, Local still uses the cached Global memory Z G , and the unused return capacity is not reassigned. Each Local request returns 32 latent tokens, including on multi-image examples. All variants use a 768-token ordinary-text budget, a 64-token answer grace period, and at most five memory requests. Memory requests do not count toward the text budget. Training variants. SFT distills the teacher CoT without memory, and Stage I sft distills both the CoT and the memory. ReMAP adds access learning after Stage I. These variants differ in training. w/o Memory instead shares all of ReMAPโs weights, including the Stage I decoder LoRA, and masks Global and Local requests only at inference. The gap between ReMAP and w/o Memory therefore measures the contribution of reasoning-time memory access on top of the full SFT and RL training with parameters held fixed; it is an inference-time ablation, not a model trained without memory. Invocation controls. Fixed, Random, and Entropy use the complete ReMAP weights and change only when requests are issued at inference. Fixed requests Global at positions 0 and โ0.7Bโ and Local at โ0.3Bโ and โ0.5Bโ of the text budget B = 768, that is, at positions 0, 230, 384, and 537. Random and Entropy consider a request after each generated delimiter, excluding position 0. Random issues a request with a fixed probability, following the random insertion of MemGen and VisMem (Zhang et al., 2026b; Yu et al., 2026); as in VisMem, it samples requests at delimiters. Entropy issues a request when the entropy of the next-token distribution over ordinary text tokens exceeds a fixed threshold. Memory tokens are excluded from this distribution, and EOS is retained. A triggered request chooses Global or Local with equal probability, as in VisMem. In all three con- trols, committing an answer or emitting EOS cancels later requests, and the five-request cap applies. Because each policy decides its own requests, the number of requests differs across policies. A.10BRANCH-POINT SELECTION DIAGNOSTIC This diagnostic compares four rules for choosing the state at which a root trajectory branches, with the policy frozen. The rules are the score u(s t ,a t ) in Equation 3, labeled CRD in Figure 12; the route entropy H p t (N),p t (G),p t (L) ; uniform random selection among eligible states; and the entropy-change signal of ARPO (Dong et al., 2026a). The ARPO-style signal is the mean entropy of the last four ordinary tokens minus the mean entropy of the first four generated tokens. We branch at its maximum under the shared budget instead of using ARPOโs stochastic branching gate. Unlike the route entropy, u(s t ,a t ) conditions on the realized route and measures how likely a sibling is to depart from it. All rules receive the same root trajectories and eligible candidate states. The cohort contains 192 training prompts, 32 from each of six ability categories: attribute, counting, existence, local detail, relation, and spatial reasoning. The prompts are selected without using rewards or correctness. Each prompt has four root trajectories, and each root yields a family of the original continuation and two siblings resampled from the selected state, giving 12 trajectories per rule and prompt. Routing Yield@H16 is the fraction of families whose trajectories produce different sets of Global and Local requests within the first 16 generated positions after the branch. Joint EAY multiplies this diversity indicator by the mean absolute deviation of the familyโs final-answer rewards, so it is positive only when routing diversity comes with different rewards. Both metrics are averaged 29 Preprint. Under review. 0255075100 (a) Routing Yield@H16 (%) โ CRD Route Entropy Random ARPO-style ฮH 0.00.10.20.30.4 (b) Joint EAY โ Figure 12:Branch-point selection with a fixed trajectory budget. (a) Routing Yield@H16, the fraction of branch families whose trajectories make different Global and Local requests within 16 positions after the branch. (b) Joint EAY, the reward variation that co-occurs with this routing diversity. Each point averages four root families of one prompt; diamonds and bars mark the mean and 95% bootstrap interval. All rules use the same roots, candidate states, and 12 trajectories per prompt on 192 prompts, with the policy frozen. ARPO-style โH is the fixed-budget adaptation of ARPOโs entropy-change signal. over the four families of a prompt, and we report prompt means. Confidence intervals use 10,000 prompt-level bootstrap resamples stratified by ability category, with seed 26090601 and the same resamples for all rules. The score u(s t ,a t ) gives the highest routing yield, 75.91% (95% CI [72.92, 78.78]), compared with 68.36% [65.23, 71.48] for the route entropy, 38.54% [35.03, 41.93] for random selection, and 32.16% [28.65, 35.68] for the ARPO-style signal. Its Joint EAY of 0.144 [0.131, 0.159] is close to that of the route entropy, 0.143 [0.129, 0.157], and above those of random selection, 0.084, and the ARPO-style signal, 0.067. Route-based selection therefore exposes more alternative memory deci- sions at the branch, and these alternatives come with larger reward differences than those found by random or entropy-change selection. A.11COMPLETE QUALITATIVE CASE TRACES Figure 13 shows the complete reasoning of the two main-text cases, with and without memory. A.12VISUAL-CONTEXT ACCOUNTING Figure 6(a) compares the visual tokens that enter the decoder sequence with the performance in Table 1. For each question, we count the initial image tokens and every crop token or memory latent returned during reasoning. A repeated Global request inserts the cached block again, and its latents are counted again. Ordinary text tokens, request and end markers, and representations used only inside the vision encoder or the memory weavers are excluded. The count therefore measures the visual context of the decoder, not total inference compute. The comparison uses the questions measured for all five methods: 78 V*Bench, 60 CV-Bench-2D, and 107 MuirBench questions. Means are unweighted over these questions and include questions without memory requests. The performance lines use the full-benchmark scores of the main table. ReMAP starts from at most 1,024 visual tokens per image and returns 32 latents per Local request and 4N + 4 latents per Global request. Its counts come from the recorded memory insertions of its evaluation run. For Q-Zoom and DeepEyes on MuirBench, the counts sum the passes over each input image. 30 Preprint. Under review. Question & options Image No memory With memory Question & options Image Case 1 :HR-Bench-4K #0347 A. 20 km/h B. 40 km/h C. 60 km/h D. 30 km/h No request Answer B Memory calls: local โ local โ local Case 2 :MMSI #0377 What is the position of the potted plant relative to the longest sofa (given that the statue on thetable is to the north of the longest sofa)? A. Due south B. Northeast C. Due east D. Southwest No request To determine the position of the potted plant relative to the longest sofa, we need to analyze the positioning of the sofa and the potted plant in the image. Identify the longest sofa: The longest sofa is the one on the far right of the image, which extends the entire length Identify the statue on the table: The statue is located on a small table to the left of the sofa. Determine the direction of the potted plant: The potted plant is located to the right and slightly ahead of the sofa, Given these observations: -The potted plant is to the east of the sofa. Therefore, the correct answer is C. Answer: C Memory calls: local โ local โ global What is the speed limit indicat- ed on the sign in the image? To determine the speed limit indicated on the sign in the image, let's follow these steps: <mem_l> ... <mem_l_end> The sign is a standard German speed limit sign, circular with a white background and black border, <mem_l> ... <mem_l_end> indicating a speed limit. <mem_l> ... <mem_l_end> The sign shows a number in the center, which is the speed limit in kilometers per hour. The number on the sign is "30". Given these observations, the speed limit indicated on Answer: D the sign is 30 km/h. No memoryAnswer C per hour. To determine the speed limit indicated on the sign in the image, let's follow these steps: Identify the sign: The sign in question appears to be a circular sign with a red border, which is typically used for speed limit signs in many countries. Read the numbers: The numbers on the sign are "40 km/h". This indicates the speed limit in kilometers Match with the options: The options provided are: -A. 20 km/h -B. 40 km/h -C. 60 km/h -D. 30 km/h Compare the sign with the options: The number "40" matches exactly with option B. Therefore, the speed limit indicated on the sign is Answer: B 40 km/h. of the wall. relative to the statue. Answer D With memory Answer B of the image. To determine the position of the potted plant relative to the longest sofa, let's analyze the image step by step: Identify the longest sofa: It is the couch on the right side <mem_l> ... <mem_l_end> Find the statue: It is on the table located on the left side <mem_l> ... <mem_l_end> Draw a line from the statue towards the potted plant and deduce the direction it points: it points towards the north- <mem_g> ... <mem_g_end> Identify the position of the potted plant: It is located to the northeast of the longest sofa. Answer: B of the image. east Figure 13: Complete paired reasoning traces. Cases 1 and 2 expand panels (a) and (b) of Figure 5 with the full recorded reasoning of the paired no-call and memory-enabled trajectories. Each mem- ory return is marked by its request and end tokens. Case 1 (HR-Bench-4K #0347): without memory, the model reads 40 km/h and answers B; after three Local calls, it reads 30 km/h and answers D. Case 2 (MMSI #0377): without memory, the model answers due east (C); after two Local calls and one Global call, it answers northeast (B). 31 Preprint. Under review. Table 12 lists the per-question means. ReMAP adds 16.0, 18.7, and 29.3 tokens per question through memory returns, and it uses 76.8%, 51.0%, and 52.3% fewer visual tokens than Native on V*Bench, CV-Bench-2D, and MuirBench, respectively. Table 12: Visual tokens entering the decoder sequence per question. Means over the questions shared by all five methods, with the question count given for each benchmark. Added counts crop tokens and memory latents returned during reasoning. Totals are computed before rounding. BenchmarkMethodInitialAddedTotal V*Bench (n = 78)Native4,417.90.04,417.9 Q-Zoom1,008.0331.81,339.8 DeepEyes4,417.935.94,453.8 VisMem4,417.90.04,417.9 ReMAP1,008.016.01,024.0 CV-Bench-2D (n = 60)Native906.00.0906.0 Q-Zoom450.6175.3625.9 DeepEyes906.0208.11,114.1 VisMem906.00.0906.0 ReMAP425.318.7444.0 MuirBench (n = 107)Native7,099.50.07,099.5 Q-Zoom3,515.4500.84,016.2 DeepEyes7,099.52,097.69,197.0 VisMem7,099.50.07,099.5 ReMAP3,359.929.33,389.2 32