Paper deep dive
Remember by Asking: Retrieval-Induced Memory Evolution for LLM Agents
Wanqi Zhou, Jiawei Lu, Yang Wang, Zhaolong Xing, Zhen Chen, Ai Han, Haoyue Shi
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Long-term memory is essential for language agents to maintain coherent and effective behavior over extended, multi-session interactions. Existing memory systems mainly use retrieval at read time, while write-time memory formation still relies on direct extraction or compression. However, when future information needs are unknown, compressing an entire interaction in one pass can overlook locally important details that may matter later. To this end, we introduce RIME, a retrieval-induced memory framework that shifts memory construction from monolithic compression toward evidence-centered integration. RIME uses generic self-questions to retrieve focused dialogue evidence and grounds memory formation in both the retrieved evidence and relevant historical memories, which are jointly reconciled into an evolving memory bank with temporal and provenance information. At inference time, compressed memory serves as the primary rather than the sole source of evidence: when it cannot support an answer, RIME retrieves relevant source dialogue together with its local context to recover information omitted during memory formation, without resorting to full-history processing. Extensive experiments on LoCoMo with Qwen3-235B-A22B and GPT-5.6 Sol show that RIME consistently achieves the best performance across all three quality metrics among the compared methods, while requiring substantially fewer query-time LLM tokens.
Tags
Links
- Source: https://arxiv.org/abs/2609.34438v1
- Canonical: https://arxiv.org/abs/2609.34438v1
Trouble viewing inline? Open PDF directly →
Full Text
21,197 characters extracted from source content.
Expand or collapse full text
Remember by Asking: Retrieval-Induced Memory Evolution for LLM Agents Wanqi Zhou Jiawei Lu Yang Wang Zhaolong Xing Zhen Chen Ai Han Haoyue Shi †thanks: 🖂 Corresponding author: haoyueshi@chd.edu.cn†thanks: * These authors contributed equally to this research. Abstract Long-term memory is essential for language agents to maintain coherent and effective behavior over extended, multi-session interactions. Existing memory systems mainly use retrieval at read time, while write-time memory formation still relies on direct extraction or compression. However, when future information needs are unknown, compressing an entire interaction in one pass can overlook locally important details that may matter later. To this end, we introduce RIME, a retrieval-induced memory framework that shifts memory construction from monolithic compression toward evidence-centered integration. RIME uses generic self-questions to retrieve focused dialogue evidence and grounds memory formation in both the retrieved evidence and relevant historical memories, which are jointly reconciled into an evolving memory bank with temporal and provenance information. At inference time, compressed memory serves as the primary rather than the sole source of evidence: when it cannot support an answer, RIME retrieves relevant source dialogue together with its local context to recover information omitted during memory formation, without resorting to full-history processing. Extensive experiments on LoCoMo with Qwen3-235B-A22B and GPT-5.6 Sol show that RIME consistently achieves the best performance across all three quality metrics among the compared methods, while requiring substantially fewer query-time LLM tokens. Index Terms: Long-term memory, Language agents, Dialogue memory †address: 1JD.com 2Chang’an University 1 Introduction Memory is fundamental for language agents operating over extended interactions. Without persistent memory, agents cannot reliably retain user-specific information, connect events across sessions, or reuse past experience in future decisions. Prior work has therefore developed mechanisms for storing, retrieving, and managing information beyond the immediate context window, ranging from memory streams and memory banks to more structured and adaptive memory systems [10, 18, 9, 1, 15, 6]. Despite this progress, most systems still form memories by directly extracting or compressing conversational context. This can overlook useful evidence, while details discarded at formation time may later become important for unforeseen queries. Relying only on compressed memory prevents such information from being recovered, whereas full-history fallback incurs substantial inference cost [17]. These limitations motivate selective access to source dialogue during both memory formation and memory use. To this end, we introduce RIME, a retrieval-induced framework for long-term dialogue memory. As illustrated in Fig. 1, rather than compressing an entire interaction at once, RIME uses generic self-questions to retrieve focused dialogue evidence, which is combined with local context and relevant historical memories to incrementally form an evolving memory bank. By grounding memory formation in retrieved evidence, RIME surfaces potentially important information before consolidation, reducing the risk that it is overlooked in the full conversational context. At inference time, RIME treats compressed memory as the primary rather than the sole information source, selectively retrieving relevant source dialogue and surrounding context when needed instead of processing the full interaction history. Across LoCoMo experiments with Qwen3-235B-A22B [16] and GPT-5.6 Sol [8], RIME consistently improves both memory effectiveness and inference efficiency over existing memory systems. Our contributions are threefold. (1) We introduce a question-guided retrieve-then-consolidate mechanism for memory formation, where generic self-questions first retrieve focused dialogue evidence before memory construction, rather than relying on a single global compression of the full interaction. (2) We introduce selective source-context recovery for memory use, allowing compressed memory to serve as the primary but not exclusive information source and recovering relevant dialogue context only when additional evidence is needed. (3) We conduct extensive experiments on LoCoMo with Qwen3-235B-A22B and GPT-5.6 Sol, where RIME achieves the strongest performance across LLM-Judge, F1, and BLEU-1 among the compared methods while requiring substantially fewer query-time LLM tokens than deliberative memory baselines. 2 Related Work Long-term memory for language agents. Long-term memory systems have evolved from persistent external memory stores [10, 18, 9] toward more sophisticated mechanisms for constructing and maintaining compact memory. Mem0 [1] consolidates extracted facts through explicit memory operations, while A-MEM [15] organizes memories as dynamically linked and evolving notes. Nemori [6] and FadeMem [14] focus on selective retention, through semantic distillation and adaptive forgetting, respectively. SimpleMem [5] and LightMem [2] improve efficiency through structured compression, consolidation, and hierarchical memory management. Despite their different designs, these methods mainly optimize how observed interaction content is transformed, organized, or maintained as memory. RIME intervenes one step earlier: through generic self-questions, it brings retrieval into the memory-writing process, recalling dialogue evidence relevant to each question before consolidation to better preserve important information that may otherwise be overlooked. Retrieval and memory-based reasoning. Beyond memory construction, recent work has explored richer mechanisms for accessing and reasoning over long-term memory. Hindsight [3] combines multiple retrieval strategies with explicit reflection over structured memories, while SAGE [13] and REALM [12] use retrieval feedback to further evolve or reconsolidate the memory structure. Most closely related to RIME, D-Mem [17] adopts a dual-process design that falls back to Full Deliberation over the interaction history when compact memory is insufficient. RIME also revisits the original dialogue, but retrieves only query-relevant source turns together with local context rather than deliberating over the broader history. Figure 1: Overview of RIME. (a) During memory formation, generic formation questions are used as retrieval cues to identify focused dialogue evidence, which is expanded with local context and combined with relevant historical memories for joint memory formation and reconciliation. Unlike conventional direct compression, RIME retrieves relevant evidence before consolidation. (b) During memory use, RIME first answers from retrieved semantic memories and selectively returns to the source dialogue only when the memory-based answer is insufficient. 3 Methodology 3.1 Retrieval-based Memory Formation RIME introduces complementary innovations in both memory formation and memory use. Question-guided evidence recall. RIME maintains a fixed set of generic formation questions =qrr=1R,Q=\q_r\_r=1^R, (1) which are shared across all sessions and independent of downstream queries. In our implementation, R=6R=6, covering people and relationships, events, activities, plans, states and preferences, and specific factual details. Let ϕ(⋅)φ(·) denote the text embedding function and s(a,b)=ϕ(a)⊤ϕ(b)‖ϕ(a)‖2‖ϕ(b)‖2s(a,b)= φ(a) φ(b)\|φ(a)\|_2\|φ(b)\|_2 (2) denote cosine similarity. For each formation question qrq_r, RIME retrieves the kfk_f most relevant dialogue turns from the current session tS_t: ℛt(r)=argtopkx∈tkfs(qr,x).R_t^(r)= *arg\,topk_x _t^k_fs(q_r,x). (3) The formation questions therefore act as retrieval cues rather than summarization instructions: each question independently surfaces dialogue evidence associated with a particular type of information before memory construction. We first merge the turns recalled by all formation questions: ℛt=⋃r=1Rℛt(r).R_t= _r=1^RR_t^(r). (4) Because some recalled turns depend on nearby context, RIME performs two rounds of local context expansion: ℛ~t(ℓ+1)=ℛ~t(ℓ)∪t(ℛ~t(ℓ)),ℓ=0,1, R_t^( +1)= R_t^( ) _t\! ( R_t^( ) ), =0,1, (5) with ℛ~t(0)=ℛt R_t^(0)=R_t, where t(⋅)N_t(·) adds neighboring turns needed to resolve local dialogue dependencies. The final evidence set is ℰt=ℛ~t(2).E_t= R_t^(2). (6) Historical memory grounding. For each formation question qrq_r, RIME combines the question with its recalled evidence ℛt(r)R_t^(r) to construct a query for retrieving relevant historical memories: zt(r)=Concat(qr,ℛt(r)),z_t^(r)=Concat (q_r,R_t^(r) ), (7) where the dialogue turns in ℛt(r)R_t^(r) are concatenated in decreasing order of retrieval similarity. The corresponding historical memories are retrieved as ℋt(r)=argtopkm∈ℳt−1khs(zt(r),m),ℋt=⋃r=1Rℋt(r).H_t^(r)= *arg\,topk_m _t-1^k_hs\! (z_t^(r),m ), _t= _r=1^RH_t^(r). (8) Joint memory formation and reconciliation. Given the recalled current-session evidence ℰtE_t and the retrieved historical memories ℋtH_t, RIME performs fact extraction, within-session deduplication, reference resolution, and historical reconciliation jointly in a single LLM call: t=Fmem(ℰt,ℋt).O_t=F_mem (E_t,H_t ). (9) Each predicted operation o∈to _t takes one of four forms: op(o)∈Add,Update,Delete,NoOp.op(o)∈\ Add, Update, Delete, NoOp\. (10) Add creates a new memory, Update revises or augments an existing memory describing the same underlying fact, Delete removes a memory explicitly contradicted as erroneous, and NoOp is used when the historical memory already captures the current evidence. Each memory also retains its temporal information and dialogue provenance. Finally, the predicted operations are applied to obtain the updated memory bank: ℳt=Apply(ℳt−1,t).M_t=Apply (M_t-1,O_t ). (11) 3.2 Selective Memory Use Given a user query qiusrq_i^usr, let y~i=Fans(qiusr,ℳi) y_i=F_ans(q_i^usr,M_i) denote the initial answer from the retrieved semantic memories ℳiM_i. If y~i=NONE y_i= NONE, RIME retrieves relevant source turns using TF–IDF [11]: ℛisrc=argtopkx∈⋃t=1Ttksψ(qiusr)⊤ψ(x),ℰisrc=ℛisrc∪Ctxw(ℛisrc),R_i^src= *arg\,topk_x∈ _t=1^TS_t^k_sψ(q_i^usr) ψ(x), _i^src=R_i^src _w(R_i^src), (12) where Ctxw(⋅)Ctx_w(·) adds up to w preceding and following turns within the same session. The final prediction is y^i=y~i,y~i≠NONE,Fans(qiusr,ℰisrc),y~i=NONE. y_i= cases y_i,& y_i≠ NONE,\\[2.0pt] F_ans(q_i^usr,E_i^src),& y_i= NONE. cases (13) 4 Experiments 4.1 Experimental Setup Dataset. We evaluate on LoCoMo [7], a benchmark for very long-term conversational memory. Following the paper-comparable setting, we use 1,540 questions from four categories: single-hop, multi-hop, temporal, and open-domain, excluding the adversarial category. Baselines. We compare RIME with RAG [4], Mem0 [1], A-MEM [15], Nemori [6], and D-Mem [17], covering retrieval-based, structured, adaptive, and deliberative long-term memory systems. Models. We evaluate all methods with two LLM backbones, Qwen3-235B-A22B and GPT-5.6 Sol. For each setting, the same backbone is used for memory construction and answer generation, and the corresponding model is also used as the LLM judge. For answer generation and evaluation, we follow the prompting protocol used by D-Mem [17] and Nemori [6], including the answer and LLM-judge prompts. Memory and Retrieval Configuration. RIME uses six generic formation queries covering person, event, activity, plan, state, and specific detail. For each query, we retrieve the top 10 dialogue turns and top 5 relevant historical memories. At inference, we retrieve the top 30 semantic memories using OpenAI’s text-embedding-3-small. If the initial answer is NONE, TF–IDF retrieval selects the top 20 source-dialogue turns, each expanded with a context window of 15 turns. No raw dialogue is provided during the initial memory-based answer. Unless otherwise specified, all baselines use the same retrieval settings; A-MEM retains its original all-MiniLM-L6-v2 embedding model following its official configuration. The source code, including the exact prompts and evaluation configuration, is available at https://anonymous.4open.science/r/RIME_official-4BC7. 4.2 Main Results Table 1 compares RIME with representative long-term memory systems under two LLM backbones. RIME consistently achieves the best performance across all three quality metrics with both Qwen3-235B-A22B and GPT-5.6 Sol. Notably, these gains are achieved with substantially lower query-time LLM cost than deliberative memory baselines such as A-MEM and Nemori, demonstrating that selective access to source dialogue provides an effective alternative to expensive deliberation over long histories. Table 1: Main results on the 1,540 non-adversarial LoCoMo questions. Higher is better for Judge, F1, and BLEU-1; lower is better for inference tokens. †D-Mem results are directly reported from the original paper using Qwen3-235B-Instruct, as its implementation is not publicly available. Method Judge↑ F1↑ BLEU-1↑ Tokens↓ Qwen3-235B-A22B RAG 31.36 20.87 17.46 3.64K Mem0 53.12 35.76 30.49 2.20K A-MEM 68.77 44.12 37.86 63.19K Nemori 77.34 46.01 39.88 20.92K D-Mem† 78.60 51.00 42.60 15.57K RIME 81.82 51.46 44.22 5.11K GPT-5.6 Sol RAG 40.00 23.94 20.29 3.31K Mem0 70.52 46.45 39.59 2.24K A-MEM 80.78 53.77 46.54 52.14K Nemori 83.05 52.64 45.88 17.74K RIME 84.29 56.41 48.85 5.21K 4.3 Ablation and Analysis Effect of question-guided memory formation. To isolate the effect of question-guided retrieval, we compare three formation strategies with the same memory-use stage. Direct Compression uses the Mem0 extraction prompt over the full session, while Question-Prompted Compression additionally uses the six formation dimensions as extraction instructions without question-specific retrieval. RIME instead uses these questions as retrieval cues before memory consolidation. As shown in Table 2, adding the six dimensions to the compression prompt yields no consistent gain over direct compression, whereas RIME improves all three metrics. This suggests that question-guided memory formation is effective. Effect of selective source retrieval. We further examine whether access to the source dialogue is necessary when semantic memory is insufficient. Disabling selective source retrieval causes a substantial performance drop on both backbones, with the Judge score decreasing by 10.59 points on Qwen3-235B-A22B and 3.32 points on GPT-5.6 Sol. Source retrieval is triggered more frequently with Qwen than with GPT-5.6 (16.8% vs. 6.0%), consistent with the larger degradation observed when this mechanism is removed. Table 2: Effect of memory formation strategies on LoCoMo using Qwen3-235B-A22B (n=1,540n=1,540). All variants use the same selective memory-use procedure. Formation Strategy Judge↑ F1↑ BLEU-1↑ Direct Compression 71.62 49.26 42.32 Question-Prompted Compression 72.66 48.87 42.11 RIME 81.82 51.46 44.22 Table 3: Ablation of selective source retrieval on LoCoMo. Method Judge↑ F1↑ BLEU-1↑ Qwen3-235B-A22B RIME w/o selective retrieval 71.23 44.73 38.41 RIME 81.82 51.46 44.22 GPT-5.6 Sol RIME w/o selective retrieval 80.97 54.27 46.96 RIME 84.29 56.41 48.85 5 Conclusion We presented RIME, a retrieval-induced framework for long-term dialogue memory. RIME uses generic questions as retrieval cues to identify relevant evidence before memory consolidation and selectively returns to the source dialogue when compressed memory is insufficient. Experiments on LoCoMo with Qwen3-235B-A22B and GPT-5.6 Sol show that RIME achieves the best performance across LLM-Judge, F1, and BLEU-1 among the compared methods, while using substantially fewer query-time LLM tokens than deliberative memory baselines. Further analysis confirms the effectiveness of both question-guided memory formation and selective source retrieval for robust and efficient long-term memory. References [1] P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav (2025) Mem0: building production-ready AI agents with scalable long-term memory. In ECAI, Frontiers in Artificial Intelligence and Applications, p. 2993–3000. Cited by: §1, §2, §4.1. [2] J. Fang, X. Deng, H. Xu, Z. Jiang, Y. Tang, Z. Xu, S. Deng, Y. Yao, M. Wang, S. Qiao, et al. (2026) Lightmem: lightweight and efficient memory-augmented generation. In International Conference on Learning Representations, Vol. 2026, p. 98706–98729. Cited by: §2. [3] C. Latimer, N. Boschi, A. Neeser, C. Bartholomew, G. Srivastava, X. Wang, and N. Ramakrishnan (2026) HINDSIGHT: structured agent memory that retains, recalls, and reflects. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), p. 275–285. Cited by: §2. [4] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, et al. (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33, p. 9459–9474. Cited by: §4.1. [5] J. Liu, Y. Su, P. Xia, S. Han, Z. Zheng, C. Xie, M. Ding, and H. Yao (2026) Simplemem: efficient lifelong memory for llm agents. arXiv preprint arXiv:2601.02553. Cited by: §2. [6] W. Ma, J. Nan, and W. Wu (2026) What deserves memory: adaptive memory distillation for llm agents. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 34789–34812. Cited by: §1, §2, §4.1. [7] A. Maharana, D. Lee, S. Tulyakov, M. Bansal, F. Barbieri, and Y. Fang (2024) Evaluating very long-term conversational memory of llm agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 13851–13870. Cited by: §4.1. [8] OpenAI (2026) GPT-5.6 system card. Note: https://deploymentsafety.openai.com/gpt-5-6 Cited by: §1. [9] C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez (2023) Memgpt: towards llms as operating systems. arXiv preprint arXiv:2310.08560. Cited by: §1, §2. [10] J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein (2023) Generative agents: interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, p. 1–22. Cited by: §1, §2. [11] G. Salton and C. Buckley (1988) Term-weighting approaches in automatic text retrieval. Information processing & management 24 (5), p. 513–523. Cited by: §3.2. [12] Y. Song, Y. Wang, X. Ma, Z. Fu, J. Lin, W. Liu, J. Wang, H. Deng, Y. Yu, and W. Zhang (2026) Retrieval-driven memory reconsolidation for long-term llm agents. arXiv preprint arXiv:2609.16053. Cited by: §2. [13] S. Wang, D. Brahma, and R. Henao (2026) SAGE: a novelty gate for efficient memory evolution in agentic llms. arXiv preprint arXiv:2605.30711. Cited by: §2. [14] L. Wei, X. Dong, X. Peng, N. Xie, and B. Wang (2026) Fademem: biologically-inspired forgetting for efficient agent memory. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 4011–4015. Cited by: §2. [15] W. Xu, Z. Liang, K. Mei, H. Gao, J. Tan, and Y. Zhang (2026) A-mem: agentic memory for llm agents. Advances in Neural Information Processing Systems 38, p. 17577–17604. Cited by: §1, §2, §4.1. [16] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §1. [17] Z. You, J. Yuan, and J. Cai (2026) D-mem: a dual-process memory system for llm agents. arXiv preprint arXiv:2603.18631. Cited by: §1, §2, §4.1. [18] W. Zhong, L. Guo, Q. Gao, H. Ye, and Y. Wang (2024) Memorybank: enhancing large language models with long-term memory. In Proceedings of the AAAI conference on artificial intelligence, Vol. 38, p. 19724–19731. Cited by: §1, §2.