Paper deep dive
TransMem: Transforming Hidden States into Memory for Large Language Models
Haodong Lei, Junming Liu, Yirong Chen, Pinlong Cai, Botian Shi, Ding Wang, Hongsong Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/3/2026, 1:50:51 AM
Summary
The paper introduces TransMem, a lightweight inference-time parametric memory module that transforms sparse historical hidden states from a frozen Large Language Model (LLM) backbone into reusable memory representations. It employs a gating network to dynamically apply latent interventions and uses evidence-conditioned self-distillation to learn transferable memory utilization. Experiments on LoCoMo, HotpotQA, and MemoryAgentBench demonstrate significant improvements in long-context reasoning and memory retention compared to external and parametric memory baselines.
Entities (9)
Relation Signals (8)
TransMem → evaluatedon → LoCoMo
confidence 95% · Experiments on LoCoMo, HotpotQA, and MemoryAgentBench demonstrate consistent improvements
TransMem → evaluatedon → HotpotQA
confidence 95% · Experiments on LoCoMo, HotpotQA, and MemoryAgentBench demonstrate consistent improvements
TransMem → evaluatedon → MemoryAgentBench
confidence 95% · Experiments on LoCoMo, HotpotQA, and MemoryAgentBench demonstrate consistent improvements
TransMem → uses → Evidence-Conditioned Self-Distillation
confidence 95% · To learn transferable memory utilization rather than task-specific knowledge, we introduce evidence-conditioned self-distillation.
TransMem → appliedto → LLaMA3.1-8B-Instruct
confidence 90% · Table 1 lists results for Llama3.1-8B-Instruct with TransMem.
TransMem → appliedto → Qwen3-4B-Instruct
confidence 90% · Table 1 lists results for Qwen3-4B-Instruct with TransMem.
TransMem → outperforms → BM25 RAG
confidence 90% · Table 1 shows TransMem achieving higher F1 scores on LoCoMo and HotpotQA compared to BM25 RAG.
TransMem → outperforms → δ-Mem
confidence 90% · Table 1 shows TransMem achieving higher F1 scores on LoCoMo and HotpotQA compared to δ-Mem.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language model (LLM) agents increasingly operate over long interaction histories, where effective reasoning requires identifying and exploiting task-relevant evidence distributed across past observations and actions. However, useful information encoded in previously computed representations is often underutilized during subsequent generation. We propose \textbf{TransMem}, a lightweight inference-time parametric memory module that transforms sparse historical hidden states from a frozen LLM backbone into reusable memory representations. TransMem uses a lightweight gating network to dynamically apply the latent intervention to the current hidden states, without repeatedly encoding the preceding context. To learn transferable memory utilization rather than task-specific knowledge, we introduce evidence-conditioned self-distillation. A memory-augmented student processes the full context and matches the predictive distribution of an evidence-only teacher that shares the same frozen backbone. Experiments on LoCoMo, HotpotQA, and MemoryAgentBench demonstrate consistent improvements across different model architectures and scales. TransMem yields gains of 11.58--29.25 $F_1$ on LoCoMo and 10.20--13.03 $F_1$ on HotpotQA, while improving the average MemoryAgentBench accuracy from 29.54\% to 40.00\%. These results establish sparse historical hidden states as an effective and efficient memory substrate for long-context LLM agents. Our code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2607.29032v1
- Canonical: https://arxiv.org/abs/2607.29032v1
Trouble viewing inline? Open PDF directly →
Full Text
57,856 characters extracted from source content.
Expand or collapse full text
TransMem: Transforming Hidden States into Memory for Large Language Models Haodong Lei1,2, Junming Liu2, Yirong Chen2, Pinlong Cai2, Botian Shi2, Ding Wang2 , Hongsong Wang1 Abstract Large language model (LLM) agents increasingly operate over long interaction histories, where effective reasoning requires identifying and exploiting task-relevant evidence distributed across past observations and actions. However, useful information encoded in previously computed representations is often underutilized during subsequent generation. We propose TransMem, a lightweight inference-time parametric memory module that transforms sparse historical hidden states from a frozen LLM backbone into reusable memory representations. TransMem uses a lightweight gating network to dynamically apply the latent intervention to the current hidden states, without repeatedly encoding the preceding context. To learn transferable memory utilization rather than task-specific knowledge, we introduce evidence-conditioned self-distillation. A memory-augmented student processes the full context and matches the predictive distribution of an evidence-only teacher that shares the same frozen backbone. Experiments on LoCoMo, HotpotQA, and MemoryAgentBench demonstrate consistent improvements across different model architectures and scales. TransMem yields gains of 11.58–29.25 F1F_1 on LoCoMo and 10.20–13.03 F1F_1 on HotpotQA, while improving the average MemoryAgentBench accuracy from 29.54% to 40.00%. These results establish sparse historical hidden states as an effective and efficient memory substrate for long-context LLM agents. Our code is available at https://github.com/Haodong-Lei-Ray/TransMem. Introduction Large language models (LLMs) are increasingly expected to support sustained interactions over long contexts, particularly in applications such as coding agents (Yang et al. 2024), long-horizon agent systems (Shi et al. 2026; Yu et al. 2026a), and complex reasoning (Agashe et al. 2025). The crucial information needed to answer a query may be distributed across a long interaction history or document (Wu et al. 2025). The challenge is therefore not merely to accept longer inputs, but to identify and exploit the evidence that matters for the current prediction (Du et al. 2025). A straightforward response is to enlarge the context window or scale the backbone model (Ding et al. 2024; Zhao et al. 2025). However, model predictions can be sensitive to the position of relevant evidence, and long-context reasoning often degrades when useful information is distant from the generation point (Liu et al. 2024; Hsieh et al. 2024). External memory addresses this limitation by summarizing, retrieving, or storing information outside the current computation (Lei et al. 2026a; Yang et al. 2026). Parametric memory encodes information into auxiliary latent spaces, which helps mitigate limitations imposed by the native computational structure of Transformers (Wei et al. 2026; Lei et al. 2026b). These mechanisms are useful when information must persist across tasks or sessions. However, they do not directly address a distinct test-time problem, which is how an LLM can better exploit the representations already computed for the current long context. Based on this question, we observe two properties of causal transformers. (1) Hidden states naturally aggregate information from their preceding tokens (Li et al. 2024; DeepSeek-AI et al. 2025). As shown in Figure 1(a), a hidden state near the end of a context segment can therefore serve as a compact representation of that segment and its prefix. (2) These representations are position-sensitive. Because attention tends to favor nearby tokens (Liu et al. 2024), representations formed at different positions preserve different parts of the context with different fidelity. This effect becomes more pronounced as the context length increases. As illustrated in Figure 1(b), even when the information most relevant to the query appears earlier in the context, the Transformer may still assign higher attention scores to more recent tokens. Consequently, information from a long context is distributed across hidden states at different positions, rather than being fully retained by representations near the current generation position. These properties suggest that a small set of strategically selected hidden states can provide complementary views of a long input, without recomputing the entire long context as another memory. Consequently, substantial task-relevant information may remain latent in previously computed hidden states but underutilized during subsequent generation. Figure 1: Two properties of contextual representations in LLMs. (a) Hidden states aggregate preceding context. Owing to the autoregressive computation of causal Transformers, each hidden state is computed from its preceding tokens. Consequently, later hidden states can serve as contextual representations of the preceding sequence. (b) Transformer representations are region-sensitive. Excluding the initial sink position, evidence appearing later in the context generally receives higher attention, reflecting a recency bias. Based on these observations, we propose TransMem, an inference-time parametric memory module that transforms sparse historical hidden states into task-specific memory representations to enhance long-context reasoning. Rather than learning to store particular knowledge representations, TransMem learns the memory capability itself through an evidence-conditioned self-distillation training method. Specifically, it learns how to extract useful information from existing hidden states to answer the current query. As a result, TransMem does not memorize knowledge tied to a specific long context, but instead acquires a general capability for processing and reasoning over long contexts. It offers a scalable route toward improving memory capacity independently of the backbone model size. Our contributions are summarized as follows: • We propose TransMem, a lightweight inference-time parametric memory module that transforms previously computed sparse hidden states from a frozen LLM backbone into reusable memory representations. • We introduce an evidence-conditioned self-distillation training method. By exposing the teacher and student to different input contexts, the parameterized memory module learns to recover evidence-conditioned predictions from full long contexts. • Extensive experiments on LoCoMo (Maharana et al. 2024), HotpotQA (Yang et al. 2018), and MemoryAgentBench (Hu et al. 2026) have shown consistent improvement on long-context reasoning across backbone architectures and scales. Related Work External Memory for LLMs The integration of long-term memory mechanisms (Zhang et al. 2025c; Srivastava et al. 2026) marks a fundamental paradigm shift in the evolution of LLM agents (Zhang et al. 2025c). Early frameworks like MemGPT (Packer et al. 2024) pioneered operating-system-inspired paging and segmentation to manage extended contexts. Building upon this, scalable architectures like Mem0 (Chhikara et al. 2025) dynamically consolidate memory states to mitigate the severe limitations of fixed context windows. Retrieval-augmented memory systems frequently suffer from critical information omissions and semantic noise (Latimer et al. 2025). To address this issue, recent studies have explored iterative retrieval paradigms, including MemCoT (Lei et al. 2026a), MemEvolve (Zhang et al. 2025b), SimpleMem (Liu et al. 2026), and ReasoningBank (Ouyang et al. 2026). These approaches substantially improve retrieval accuracy by enabling progressive evidence refinement and adaptive memory exploration. However, they often suffer from limited robustness and increased computational overhead due to the repeated retrieval process (Srivastava et al. 2025). Moreover, these methods fundamentally do not eliminate the information loss introduced during memory extraction. Instead, they remain vulnerable to premature termination within the retrieval loop, which can prevent the model from locating the necessary evidence and ultimately lead to retrieval failure. Parametric Memory for LLMs Compared with memory mechanisms that store information in external knowledge repositories, parameterized memory provides a more expressive representation space by directly encoding knowledge into model parameters. Therefore, it represents a more advanced paradigm for enhancing the memory capacity of LLMs. Early attempts along this direction typically employ auxiliary models to explicitly or implicitly encode additional information, including MeMo (Quek et al. 2026), MemVerse (Liu et al. 2025), Memory Decoder (Cao et al. 2025), MemSifter (Tan et al. 2026), MEMTS (Yu et al. 2026b), and Mem-π (Wang et al. 2026). More sophisticated approaches instead design specialized architectures that introduce dedicated representations to enhance long-context reasoning capabilities (Zhang et al. 2025a; Ji et al. 2026). For example, DRIFT (Xie and Wang 2026) reveals that the last-layer feature may already contain highly compressed information from long contexts, suggesting that intermediate representations can serve as effective memory carriers. Meanwhile, the decoupling of memory capability from reasoning capability has emerged as an increasingly important research direction. MLPMemory (Wei et al. 2026) maps last-layer features into a latent distribution to inject long-context reasoning ability, while δ-mem (Lei et al. 2026b) adopts a heterogeneous delta-network architecture to incorporate long-context reasoning capabilities into layer-wise representations. Figure 2: The overall architecture of the TransMem framework. (a) Inference with TransMem. TransMem is applied after the frozen LLM backbone produces the hidden states at the K-th layer. The selected historical hidden states are transformed into memory representations by the TransMem module. (b) Evidence-conditioned self-distillation. The student model consists of the frozen LLM backbone augmented with TransMem and receives the full long context C. The teacher model shares the same frozen backbone parameters but receives only the evidence E and question Q, providing an evidence-conditioned supervision signal. The student is trained to recover the teacher prediction distribution from the full long context. Method In this section, we present the TransMem framework, including its inference and training procedures. Figure 2 provides an overview of the framework. During inference, the frozen backbone performs its standard forward computation, while TransMem abstracts a small set of historical hidden states into a memory shift. This shift influences current output hidden states to better preserve key information from long contexts. We train TransMem through self-distillation to recover the predictive distribution of a teacher model conditioned on gold evidence, enabling the memory module to preserve crucial information while suppressing contextual noise. Preliminaries During the prefill stage, the input long context is denoted as C. Let Q⊆CQ C denote the question and E⊆CE C denote the evidence that supports the answer. At position t, the tokenized input is x1:t=[C;A≤t]x_1:t=[C;A_≤ t], where C=x1:LC=x_1:L is the long context and A≤t=(yL+1,…,yt)A_≤ t=(y_L+1,…,y_t) is the generated answer prefix. The next token is yt+1y_t+1. The hidden states produced by the K-th layer for x1:tx_1:t are: h1:tK=[HC;HA]=[h1:LK;hL+1:tK],h^K_1:t=[H_C;H_A]=[h^K_1:L;h^K_L+1:t], (1) where htK∈ℝ1×dh_t^K ^1× d, with d denoting the hidden dimension of the frozen backbone model. The context features HCH_C are computed during prefill, during which TransMem does not intervene in generation. With the KV cache, each layer only needs to receive the final hidden state at each decoding step. In a standard LLM forward pass, transformer layer K+1K+1 produces the hidden state of the next layer as htK+1=fθK+1(htK),h^K+1_t=f^K+1_θ(h^K_t), (2) where fθK+1f^K+1_θ represents the (K+1)(K+1)-th Transformer block. Evidence-Conditioned Self-distillation Training We train TransMem with self-distillation. Our goal is to improve the ability of LLMs to identify and exploit query-relevant evidence from long contexts. Therefore, instead of training the model to memorize the entire context, we aim to encourage the model to suppress irrelevant information and focus on evidence that supports the answer. To achieve this goal, we construct the teacher by providing only the evidence E and query Q. Since the teacher has access to the evidence without irrelevant context noise, its output distribution represents a great behavior on evidence. In contrast, the student is equipped with learnable TransMem. It receives the full context C and learns to recover the teacher’s evidence-conditioned predictions through distillation. The teacher runs the frozen backbone on (E,Q)(E,Q), while the student runs the TransMem-augmented backbone on C. The teacher generates the answer trajectory A, and the student follows its answer prefix during teacher forcing. Both paths share the same frozen LLM backbone, so no separate teacher model is introduced. Let pT(⋅∣A<i,E,Q)p_T(· A_<i,E,Q) denote the teacher distribution along its answer trajectory. A student model augments the frozen backbone with TransMem and produces pS(⋅∣A<i,C)p_S(· A_<i,C). As shown in Figure 2(b), the training objective is the forward KL divergence at the language model output. We optimize the TransMem module by minimizing the forward KL divergence between the teacher and student distributions: ℒSD=1|A|∑i=1|A|KL(pT(⋅∣A<i,E,Q)∥pS(⋅∣A<i,C)).L_SD= 1|A| _i=1^|A|KL\! (p_T(· A_<i,E,Q)\, \|\,p_S(· A_<i,C) ). (3) Since the teacher provides soft supervision, we use the teacher distribution as the target distribution. The student distribution is read through the frozen language model head. Gradients from Eq. (3) pass through the frozen upper layers into every lower TransMem module. The backbone receives no parameter updates, while each correction is optimized for its final token-level effect. A correction at layer K changes the input distribution of every higher layer. Training modules against independently cached layer targets therefore creates a mismatch once lower-layer corrections become active. In-loop training keeps the frozen backbone in the computation graph and optimizes all memory modules jointly. The resulting upper-layer states include the corrections from every active lower module (Zhang et al. 2026). Inference with TransMem The TransMem module at the K-th layer consists of a Transformer block fmKf_m^K, a learnable projection WΔKW^K_ , and a learnable gating network fgKf_g^K. TransMem applies its correction only while generating A≤tA_≤ t. Once htKh^K_t is obtained, we divide HCH_C into segments of size n and use the final hidden state of each segment as a memory feature. This gives HM=hnK,h2nK,…,hLKH_M=h^K_n,h^K_2n,...,h^K_L. The size of HMH_M remains small, with the optimal value being only 5, which ensures the sparsity of the memory representation. As illustrated in Figure 2(a), we feed HMH_M and HAH_A into TransMem. They first undergo standard autoregressive attention to produce the memory feature hmh_m from the different segments. A learnable projection WΔK∈ℝd×dW^K_ ^d× d then transforms hmh_m into the final memory shift Δhm h_m. The complete process is hm=fmK(HM,h1:tA),h_m=f^K_m(H_M,h^A_1:t), (4) Δh=WΔKhm. h=W^K_ h_m. (5) Not every memory shift Δh h is useful, and noisy shifts may adversely affect the output. Therefore, we use the learnable gating network fgKf^K_g to scale the memory shift. Specifically, gm=fgK(hm)=α⋅σ(Wghmτ),g_m=f^K_g(h_m)=α·σ ( W_gh_mτ ), (6) where σ(x)=11+e−xσ(x)= 11+e^-x, α=2α=2, τ is gate temperature, and Wg∈ℝd×dW_g ^d× d is learnable gate weight. The core correction formula is h^tK=htK+gm⊙Δh, h^K_t=h^K_t+g_m h, (7) where t≥L+1t≥ L+1, indicating that the memory correction is applied only during answer generation. Inference with the TransMem memory correction then becomes: htK+1=fθK+1(h^tK).h^K+1_t=f^K+1_θ( h^K_t). (8) Therefore, our method enhances long-context reasoning through a lightweight network without extensively recomputing long text segments. Our method does not store additional information from the preceding context. Instead, its contextual awareness relies on the already computed vectors in HMH_M. The features in HMH_M can serve as memory because of the computational properties of a transformer. In particular, hLKh^K_L encodes transformed information from x1:Lx_1:L and therefore acts as a feature representation of that prefix. A similar operation can also be found in EAGLE (Li et al. 2024) and DeepSeek MTP (DeepSeek-AI et al. 2025). In this sense, the method parameterizes part of the transformer’s memory capability in an independent network. It therefore takes a further step toward separating the reasoning and memory capabilities of an LLM. TransMem is not applied to all layers. Empirically, we find that inserting memory modules into only the last four layers of the LLM is sufficient to achieve strong performance. It further makes our method more lightweight. Furthermore, unlike existing latent memory methods, TransMem is invoked only when the model begins generating the answer sequence A. This sparse inference strategy further reduces the additional computation. To construct sparse memory representations, we adopt a dynamic segmentation strategy. Specifically, we partition the long context into B segments and set the segment length to n=LBn= LB. We note that this choice is not necessarily optimal, and the segment length can be adapted to different tasks or application scenarios. When the long context C exceeds the maximum context window of the frozen backbone model, we adopt a chunk-wise memory extraction strategy. Specifically, C is divided into S chunks according to the backbone context window, denoted as C1,…,CS\C_1,…,C_S\. Each chunk independently produces a sparse memory representation HMiH_M^i. During answer generation, the final memory is formed by aggregating all chunk-level memories: HM=Concat(HM1,…,HMS).H_M=Concat(H_M^1,…,H_M^S). This strategy enables TransMem to exploit information beyond the backbone context window while introducing only a small amount of additional computation. Experiments Table 1: Main benchmark results comparing different memory mechanisms. Model LoCoMo HotpotQA Avg. Multi-Hop Temporal OpenDomain Single-Hop EM F1 Qwen3-4B-Instruct 40.79 38.39 32.89 10.77 48.05 42.35 56.00 External Memory + BM25 RAG 36.68-4.11 38.12-0.27 20.34-12.55 9.99-0.78 45.47-2.58 40.35-2.00 52.83-3.17 + LLMLingua-2 40.98+0.19 39.07+0.68 30.13-2.76 10.98+0.21 49.19+1.14 36.93-5.42 50.03-5.97 + MemoryBank 38.14-2.65 37.88-0.51 21.76-11.13 13.35+2.58 47.31-0.74 39.16-3.19 51.25-4.75 Parametric Memory + Context2LoRA 48.11+7.32 37.95-0.44 34.99+2.10 16.75+5.98 60.11+12.06 37.85-4.50 50.88-5.12 + MemGen 40.05-0.74 32.93-5.46 33.30+0.41 12.67+1.90 48.13+0.08 5.36-36.99 16.27-39.73 + MLP Memory 40.21-0.58 32.87-5.52 21.72-11.17 13.81+3.04 52.75+4.70 44.92+2.57 59.23+3.23 + δ-Mem 46.53+5.74 42.14+3.75 37.20+4.31 13.35+2.58 55.36+7.31 49.41+7.06 63.66+7.66 + TransMem (Ours) 54.11+13.32 52.04+13.65 45.62+12.73 20.64+9.87 61.87+13.82 48.55+6.20 66.20+10.20 Qwen2.5-14B-Instruct 42.74 32.59 27.36 13.50 55.35 50.75 60.20 External Memory + BM25 RAG 42.25-0.49 32.17-0.42 30.77+3.41 23.21+9.71 52.19-3.16 50.46-0.29 61.2+1.00 + LLMLingua-2 43.43+0.69 34.12+1.53 31.15+3.79 19.48+5.98 53.98-1.37 45.45-5.30 59.10-1.10 + MemoryBank 44.64+1.90 34.50+1.91 44.70+17.34 23.86+10.36 50.39-4.96 46.66-4.09 58.51-1.69 Parametric Memory + Context2LoRA 48.28+5.54 35.16+2.57 28.34+0.98 17.23+3.73 63.85+8.50 48.15-2.60 59.98-0.22 + MemGen 43.64+0.90 29.36-3.23 29.47+2.11 17.54+4.04 56.83+1.48 10.56-40.19 21.45-38.75 + MLP Memory 45.33+2.59 35.65+3.06 25.46-1.90 21.39+7.89 58.90+3.55 47.35-3.40 58.84-1.36 + δ-Mem 47.83+5.09 34.99+2.40 27.47+0.11 27.37+13.87 62.25+6.90 51.38+0.63 65.36+5.16 + TransMem (Ours) 54.32+11.58 41.20+8.61 41.44+14.08 30.38+16.88 66.38+11.03 58.01+7.26 72.34+12.14 Llama3.1-8B-Instruct 22.39 20.30 19.42 25.86 23.84 49.77 58.60 External Memory + BM25 RAG 19.25-3.14 12.28-8.02 11.58-7.84 31.02+5.16 23.18-0.66 44.95-4.82 55.12-3.48 + MemoryBank 23.53+1.14 23.31+3.01 13.01-6.41 31.05+5.19 26.77+2.93 50.15+0.38 65.97+7.37 + SimpleMem 27.47+5.08 30.44+10.14 18.77-0.65 31.23+5.37 29.37+5.53 52.06+2.29 66.15+7.55 + Memory-R1-GRPO 26.84+4.45 32.87+12.57 16.72-2.70 8.81-17.05 30.75+6.91 48.54-1.23 62.15+3.55 + TriMem 38.70+16.31 34.56+14.26 32.36+12.94 32.71+6.85 43.20+19.36 55.36+5.59 68.11+9.51 Parametric Memory + δ-Mem 46.21+23.82 36.64+16.34 31.28+11.86 24.87-0.99 57.56+33.72 56.27+6.50 69.47+10.87 + TransMem (Ours) 51.64+29.25 47.27+26.97 37.29+17.87 28.69+2.83 61.21+37.37 56.57+6.80 71.63+13.03 Table 2: Results on Memory Agent Bench comparing different memory mechanisms. Model Avg. AR TTL LRU SF Qwen3-4B-Instruct 29.54 35.30 26.14 47.08 14.37 External Memory + BM25 RAG 24.43 32.20 9.74 37.86 15.00 + LLMLingua-2 15.63 21.45 1.43 38.45 8.62 + MemoryBank 17.65 22.65 7.67 36.36 9.88 Parametric Memory + Context2LoRA 32.53 40.00 29.86 25.15 17.75 + MemGen 29.61 34.85 28.45 44.30 14.38 + MLP Memory 28.80 35.35 26.00 31.19 14.38 + δ-Mem 38.85 44.40 47.29 41.55 17.00 + TransMem (Ours) 40.00 41.60 54.64 36.19 24.00 Experimental Setup Evaluation and Benchmarks. We evaluate both general reasoning ability and memory effectiveness. We evaluate multi-hop question answering performance on the HotpotQA test set (Yang et al. 2018), which assesses multi-hop reasoning. Memory effectiveness is evaluated on LoCoMo (Maharana et al. 2024) and MemoryAgentBench (Hu et al. 2026), which measure the retention, retrieval, and use of information over extended interaction histories. LoCoMo contains contexts with an average length of approximately 16K tokens, making it suitable for evaluating long-context reasoning capabilities. In contrast, MemoryAgentBench includes many scenarios with context lengths exceeding 256K tokens, providing a more challenging evaluation of latent-space memory utilization. Following mem0 (Chhikara et al. 2025), we exclude the adversarial question category from LoCoMo. LoCoMo comprises 841 single-hop (SH), 282 multi-hop (MH), 96 open-domain (OD), and 321 temporal (TP) reasoning challenges. Baselines. We compare TransMem with representative memory approaches based on different memory mechanisms, including external memory and parametric memory methods. For external memory methods, we include BM25 RAG (Lewis et al. 2020), which retrieves relevant passages from historical contexts and appends them to the input, LLMLingua-2 (Pan et al. 2024), which compresses long contexts into shorter textual representations, MemoryBank (Zhong et al. 2024), Memory-R1-GRPO (Yan et al. 2026) and TriMem (Sun et al. 2026), which maintain historical information as retrievable textual memories. For parametric memory methods, we compare with Context2LoRA (Back et al. 2026), MemGen (Zhang et al. 2025a), MLP Memory (Wei et al. 2026) and δ-Mem (Lei et al. 2026b), which introduce additional trainable components to encode or transform contextual information into memory representations. All experiments are conducted with a fixed random seed of 42. Implementation Details. To evaluate the generality of TransMem, we conduct experiments on LLM backbones with different architectures and parameter scales, including Qwen3-4B-Instruct (Qwen Team 2025b), Qwen2.5-14B-Instruct (Qwen Team 2025a), and Llama3.1-8B-Instruct (Grattafiori et al. 2024). TransMem is a learnable memory component, which is trained only on the HotpotQA training set. We follow the same data processing strategy as AgeMem (Yu et al. 2026c) to ensure the effective utilization of the HotpotQA training set. Notably, we ensure that the HotpotQA training set contains no content overlapping with the test set, thereby avoiding unfair performance gains due to test-set leakage. TransMem is inserted into the final four transformer layers of the student LLM model. Its transformer block consists of a single layer that follows the Qwen3 transformer architecture. Training uses one NVIDIA A100 Tensor Core GPU and 15 gradient accumulation steps per rank, resulting in a global batch size of 30. Main Results We evaluate TransMem on models with different scales and architectures. As shown in Table Experiments, TransMem consistently improves long-context reasoning performance across all evaluated models. On LoCoMo, it achieves relative improvements of at least 27% over the corresponding backbone models. On HotpotQA, TransMem yields relative EM improvements of approximately 14%. We further observe that the effectiveness of TransMem may depend on the underlying capability of the backbone model. For example, although Qwen3-4B-Instruct has fewer parameters than Qwen2.5-14B-Instruct, its more advanced architecture enables comparable performance on LoCoMo. Moreover, TransMem demonstrates robustness to substantial changes in the backbone architecture. Although Llama3.1-8B-Instruct does not achieve performance comparable to the other two backbones, the transformer block within TransMem adopts the more advanced Qwen3 architecture, allowing it to achieve an F1F_1 score of 51.64 on LoCoMo and 71.63 on HotpotQA. These results are comparable to those of Qwen2.5-14B-Instruct with a larger parameter scale. As shown in Table Experiments, we further evaluate TransMem on Memory Agent Bench, which contains substantially longer contexts than LoCoMo, with many examples exceeding 256K tokens. Since TransMem reuses historical hidden representations that have been truncated from the current context window, it can preserve information beyond the effective context length of the backbone model. As a result, it consistently improves performance across different memory settings, demonstrating that the proposed memory mechanism remains effective under ultra-long-context reasoning. Notably, although TransMem is trained only on the HotpotQA training set, it generalizes effectively to both LoCoMo and Memory Agent Bench. This phenomenon demonstrates that the learned memory capability transfers well across diverse long-context reasoning tasks, rather than memorizing knowledge specific to the HotpotQA dataset. Ablation Studies Table 3: Module Ablation Results on LoCoMo with Qwen3-4B-Instruct as the backbone. Model Avg. MH TP OD SH TransMem 54.11 52.04 45.62 20.64 61.87 w/o Gating Network 53.09 48.04 45.09 21.57 61.45 w/o Transformer Block 40.79 38.39 32.89 10.77 48.05 To investigate the contribution of each component in TransMem, we conduct an ablation study by removing individual modules. As shown in Table 3, the results demonstrate that the proposed transformer block is the primary contributor to the overall performance of TransMem. Although the Gating Network provides a relatively modest improvement in the overall score, it yields a noticeably larger gain on multi-hop reasoning. This observation suggests that the gating mechanism facilitates the retrieval and utilization of relevant memory representations, thereby improving the model’s ability to exploit stored information during complex reasoning. Further Study Efficiency. To evaluate the efficiency of TransMem, we measure the additional computational cost and inference latency under different context lengths and latent memory configurations. We report the additional latency introduced by the memory module, excluding the frozen backbone prefill computation. As shown in Figure 3(a), we compare the additional inference costs of TransMem, δ-Mem, and MLPMemory on Qwen3-4B-Instruct as the context length increases from 10K to 100K tokens. δ-mem introduces an approximately linear increase in computational overhead, where every additional 10k tokens in the latent memory leads to nearly 100 GFLOPs of extra computation. Although this overhead remains relatively small compared with the backbone model, the computational cost of TransMem is independent of the context length and only depends on the number of selected memory slots B, equivalently |HM||H_M|. Therefore, TransMem maintains a constant computational overhead regardless of the input context length. Moreover, since TransMem is inserted into only a subset of backbone layers rather than all layers, its computational cost remains lower than δ-mem even with a 10k-token context. TransMem also achieves lower inference latency, as shown in Figure 3(b). Unlike methods that repeatedly encode historical memory representations, TransMem performs sparse memory computation only during answer generation without additional processing of previous contexts. Therefore, increasing the context length does not introduce additional inference latency, demonstrating the scalability of TransMem for long-context reasoning. Figure 3: Efficiency comparison in terms of computation cost and inference latency. (a) Additional computational cost introduced by the memory module, shown on a logarithmic scale. (b) Average additional inference latency introduced by the memory module. Table 4: LoCoMo F1F_1 under different TransMem injection depths and layer positions. Experiments are conducted on Qwen3-4B-Instruct. Model Avg. MH TP OD SH Last-2 53.14 49.91 44.70 18.35 61.41 Last-4 54.11 52.04 45.62 20.64 61.87 Last-8 53.27 50.29 44.31 20.44 61.43 Middle 28-31 52.85 50.18 43.74 20.07 60.96 Middle 22-25 52.32 47.24 43.26 20.67 61.09 Middle 18-21 51.54 45.18 44.48 20.17 59.96 Middle 12-16 51.11 44.00 42.92 20.57 60.11 Insertion Depth. We further investigate the impact of inserting TransMem at different layers. As illustrated in Table 4, we find that increasing the number of inserted layers does not always lead to better performance, and earlier inserted layers are not necessarily beneficial. The Last-2 to Last-8 settings denote inserting TransMem into the last 2 to 8 layers of the backbone, respectively. Since Qwen3-4B-Instruct contains 36 layers, the Middle settings insert TransMem into selected intermediate layers. We observe that inserting TransMem into the last four layers achieves the best performance, particularly on the Multi-Hop task. Compared with inserting TransMem into only the last two layers, Last-4 provides more effective memory enhancement. However, further increasing the number of inserted layers to Last-8 does not bring additional improvements. Similarly, inserting TransMem into layers 28 to 31 does not provide noticeable gains. Overall, inserting TransMem into the last layers generally yields better performance than inserting it into the middle layers. Earlier insertion locations may tend to provide weaker improvements, suggesting that effective memory formation may occur primarily in deeper layers. Table 5: Effect of different training objectives on LoCoMo F1F_1. Experiments are conducted on Qwen3-4B-Instruct. All training sets are from HotpotQA. Method Avg. MH TP OD SH SFT 48.82 47.06 40.66 19.45 55.89 GRPO 53.18 51.36 43.92 17.42 61.40 OPD 52.80 51.17 45.01 18.96 60.19 ECSD 54.11 52.04 45.62 20.64 61.87 Training Method. In addition to evidence-conditioned self-distillation (ECSD), we explored supervised fine-tuning (SFT) (Ouyang et al. 2022) and two subsequent policy-optimization objectives. As illustrated in Table 5, direct SFT on golden answers performs substantially worse than ECSD. Starting from the SFT checkpoint, both on-policy distillation (OPD) (Li et al. 2026) and group relative policy optimization (GRPO) (Shao et al. 2024) improve cross-domain LoCoMo performance, reaching 52.80 and 53.18 F1, respectively. However, neither matches ECSD, which achieves 54.11 F1. The best GRPO variant reaches 53.18 F1F_1, compared with 54.11 for ECSD. This result suggests that answer-level reinforcement learning does not compensate for a weaker memory-representation objective: ECSD’s token-level supervision from a privileged-evidence teacher remains important for learning the TransMem module. Conclusion We propose TransMem, a lightweight parametric memory module that improves test-time long-context reasoning without explicit memory construction or backbone scaling. TransMem transforms sparse, position-sensitive historical hidden states into reusable memory representations and injects them into current reasoning. Evidence-conditioned self-distillation trains the module to recover an evidence-only teacher’s predictions from full contexts. Experiments on LoCoMo, MemoryAgentBench, and HotpotQA show consistent gains across backbones with sparse memory representations and context-independent overhead, supporting efficient decoupling of memory capability from backbone reasoning capacity. Future work will explore adaptive MoE memory extensions for TransMem and applications to long-horizon agent scenarios. Appendix A Appendix Appendix B More Experiments Effect of the Train Set We next compare different training corpora while fixing the student to Qwen3-4B and inserting dynamically gated TransMem modules into its last four layers. We evaluate all checkpoints on the same 1,540 questions from LoCoMo and report token-level F1F_1 in Table 6. The HotpotQA–LongMemEval mixture reaches 53.95 F1F_1, only 0.160.16 points below the main HotpotQA recipe. In contrast, aggressively downsampling HotpotQA and adding the small LoCoMo-train split reduces overall F1F_1 to 51.80, with the largest degradation on temporal questions. The last train set contains 84 LoCoMo questions that also occur in the full evaluation set; removing them at evaluation gives 51.87 F1F_1, essentially the same conclusion. Hence, the decrease is not explained by evaluation overlap. Since the last recipe changes both corpus composition and the number of HotpotQA examples, the results do not show that LoCoMo supervision is intrinsically harmful; rather, this particular mixture provides no improvement. Table 6: LoCoMo F1F_1 with different training corpora for Qwen3-4B TransMem. HQA, LME, and LoCoMo-T denote HotpotQA, LongMemEval, and LoCoMo-train. MH, Temp., Open, and SH denote multi-hop, temporal, open-domain, and single-hop questions. Train set Overall MH Temp. Open SH HQA 54.11 52.04 45.62 20.64 61.87 HQA + LME 53.95 50.40 45.54 21.85 62.02 15% HQA + LME + LoCoMo-T 51.80 50.53 37.20 19.13 61.53 Gate Parameterization We compare three gate parameterizations on Qwen3-4B with the Last-4 configuration, keeping HotpotQA as the train set. Let z=Wghm/τz=W_gh^m/τ. Besides a fixed residual scale, we evaluate the non-negative centered sigmoid used by our main model, g=2σ(z)g=2σ(z), and a signed alternative, g=3σ(z)−1g=3σ(z)-1. The latter can suppress the residual with a negative correction. Table 7: LoCoMo F1F_1 under different gate parameterizations. All dynamic gates are initialized at g=1g=1. Gate Overall MH Temp. Open SH g=1g=1 53.09 48.04 45.09 21.57 61.45 g=2σ(z)g=2σ(z) 54.11 52.04 45.62 20.64 61.87 g=3σ(z)−1g=3σ(z)-1 54.01 51.47 45.87 20.87 61.76 Both learned gates improve over the fixed residual scale. The main 2σ(z)2σ(z) gate performs best overall, improving the fixed-gate model by 0.460.46 F1F_1 points. Allowing negative corrections does not provide an additional benefit: 3σ(z)−13σ(z)-1 is 0.100.10 points below the main parameterization, although it attains the best temporal score. We therefore use 2σ(z)2σ(z) for its slightly stronger overall result and its simpler non-negative modulation. Early Final-Hidden TransMem Before introducing layer-wise injection, we studied a final-hidden variant that applies a single TransMem module after the backbone’s final normalization and before the language-model head. Unlike the current method, the module itself contains either two or four Transformer blocks, denoted by LTML_TM. It uses a fixed residual scale and has no learned gate. Table 8 summarizes this early architecture on held-out HotpotQA examples. Table 8: HotpotQA results (%) with Qwen3-4B-Instruct (the 2507 checkpoint). The first five rows use Final-hidden and report Exact Match/Contains (E/C) on the archived 128-example split; LTML_TM is the number of Transformer blocks inside its single memory module. The final row gives the main TransMem result under the paper’s official EM/F1F_1 protocol and is included as a reference rather than a directly controlled comparison. Method Memory layout Train set HQA score Final-hidden LTM=4L_TM=4 HQA+LME 39.8/50.8 Final-hidden LTM=2L_TM=2 HQA 39.1/50.8 Final-hidden LTM=2L_TM=2 HQA+LME 39.1/48.4 Final-hidden LTM=4L_TM=4 LME 36.7/50.0 Final-hidden LTM=2L_TM=2 LME 38.3/52.3 TransMem (Ours) Last-4 HQA 48.55/66.20 Changing the internal depth from two to four does not yield a consistent improvement on HotpotQA. The two mixed-corpus configurations with LTM=4L_TM=4 and LTM=2L_TM=2 also differ in training duration, so they should not be interpreted as a strictly controlled depth comparison. More broadly, these results suggest that merely scaling up the memory parameters at a single final-hidden location has limited effect. This observation motivated the current design: instead of concentrating capacity at one location, we inject small memory modules at multiple LLM layers so that memory corrections can interact with different stages of the backbone computation. Additional Hidden-State Supervision We also tested whether directly matching the teacher’s final-layer feature improves knowledge transfer. In addition to the self-distillation objective, the Last-4 model was trained with ℒ=ℒSD+β‖Hlast′−Hlasttea‖22.L=L_SD+β H _last-H_last^tea _2^2. Here, Hlast′H _last is the answer-position feature from the student LLM’s final layer after the TransMem update, and HlastteaH_last^tea is the corresponding final-layer feature produced by the privileged-evidence teacher. The prime therefore denotes the TransMem-updated student representation. All variants use Qwen3-4B and the same QASPER–HotpotQA multi-domain train set. Table 9: LoCoMo F1F_1 when adding hidden-state regression to the self-distillation objective. All rows use the Qwen3-4B Last-4 architecture. β Overall MH Temp. Open SH 0 54.11 52.04 45.62 20.64 61.87 0.3 51.91 46.94 44.12 19.16 60.28 1.0 52.18 46.38 44.26 19.29 60.90 3.0 51.79 45.29 43.62 20.89 60.62 Hidden-state supervision provides no benefit in this ablation: every β>0β>0 setting performs below the no-regression variant (β=0β=0). One likely explanation is that the output-distribution objective already supplies the task-relevant behavioral signal, whereas imposing additional Euclidean alignment over-constrains the student’s hidden space. In other words, supervising hidden states too strongly may restrict functionally equivalent representations without improving cross-domain memory use. TransMem Feature Source Our default Last-4 model computes the memory update for LLM layer K from the output HKH^K of that layer. We compare it with transmem-before, which computes the same update from the layer input HK−1H^K-1 while still adding the resulting residual to HKH^K. All other architectural and training choices remain unchanged. Table 10: Feature-source ablation for Qwen3-4B Last-4 TransMem. Variant Source state Added to LoCoMo F1F_1 Default HKH^K HKH^K 54.11 Before HK−1H^K-1 HKH^K 53.92 Reading the post-block state is 0.19 F1F_1 points better, although the small gap indicates that TransMem is not highly sensitive to this choice. We retain HKH^K because it exposes each memory module to the representation already transformed by its corresponding LLM layer. Gate Statistics on Qwen3-4B Table 11 compares the learned gate distributions under architectural, parameterization, and train-set changes. Every experiment in this table uses Qwen3-4B. The reported mean and standard deviation aggregate validation answer positions across all injected layers; because the gate is token-dependent, they are descriptive statistics rather than fixed model parameters. Table 11: Gate statistics and LoCoMo F1F_1 for Qwen3-4B. HQA, LME, and LoCoMo-F denote HotpotQA, LongMemEval, and the full LoCoMo train set. Setting Train set Gate μ/σμ/σ F1F_1 Last-4 HQA 2σ(z)2σ(z) 1.137/0.384 54.11 Last-8 HQA 2σ(z)2σ(z) 0.605/0.319 53.27 Last-4 HQA 3σ(z)−13σ(z)-1 1.200/0.398 54.01 Last-4 (before) HQA 2σ(z)2σ(z) 1.237/0.313 53.92 Last-4 LME+ LoCoMo-F 2σ(z)2σ(z) 1.561/0.143 50.68 The Last-4 and Last-8 models learn qualitatively different global scales: Last-4 mildly amplifies memory updates on average, whereas Last-8 suppresses them. The signed gate and transmem-before variants remain close to the main model. Training on the much smaller LME–LoCoMo mixture produces a high-mean, low-variance gate and a lower F1F_1 despite containing target-domain examples, consistent with overfitting rather than improved memory use. Layer-Wise Gate Behavior Aggregate statistics hide a consistent division of labor among the four injected layers. Figure 4 reports the token-level gate distribution for the main Qwen3-4B Last-4 checkpoint on 128 in-domain HotpotQA development questions and a 386-question cross-domain LoCoMo diagnostic subset. Figure 4: Layer-wise gate values for Qwen3-4B Last-4 TransMem. Curves show means and shaded regions show one standard deviation across answer tokens. The dashed line is the fixed residual scale g=1g=1. The gate forms a monotonic amplification-to-suppression profile: Layer 32 strongly amplifies the TransMem update, Layers 33–34 progressively reduce it, and Layer 35 suppresses the direct intervention. More importantly, the same profile is preserved on LoCoMo. The corresponding means from Layers 32 to 35 are 1.618,1.332,0.975,0.6101.618,1.332,0.975,0.610 on HotpotQA and 1.472,1.298,0.996,0.5991.472,1.298,0.996,0.599 on LoCoMo. This cross-domain stability suggests that the gate learns a layer-specific correction schedule rather than a dataset-specific binary switch. More Models We additionally train Last-4 TransMem with HotpotQA on two other student backbones. Table 12 reports their absolute LoCoMo scores together with Qwen3-4B as a reference. Table 12: LoCoMo F1F_1 with additional student backbones. Student backbone Injection Train set F1F_1 Qwen2.5-7B Last-4 HQA 49.98 Qwen3-8B Last-4 HQA 50.76 The method can be trained without changing the Last-4 design across these model sizes and families. Absolute performance does not scale monotonically with parameter count under this protocol, so these results support architectural portability rather than a claim that a larger frozen student necessarily yields a larger gain. Sensitivity to the Number of Memory Segments We first study the memory-segmentation granularity, denoted by B in the paper. We reuse the Qwen3-4B TransMem checkpoint trained with B=4B=4 and vary only B at inference time. Thus, this experiment measures inference-time sensitivity rather than the effect of retraining a separate model for every value of B. Table 13 reports the official task metrics (percentage points) on the deterministic subsets of MemoryAgentBench. The results are nearly invariant over B∈4,8,16,32B∈\4,8,16,32\: increasing B from 4 to 32 changes EventQA by only 0.20.2 points and improves RULER-1 by 1.01.0 point, while the other three scores remain unchanged. This suggests that TransMem is not brittle to the precise segmentation granularity. We therefore retain B=4B=4 as the default because it uses fewer memory states. Table 13: Inference-time sensitivity to the number of memory segments B. FC-SH and FC-MH denote the single-hop and multi-hop FactConsolidation subsets, respectively. B RULER-1 RULER-2 EventQA FC-SH FC-MH 4 40.00 30.00 47.40 43.00 5.00 8 40.00 30.00 47.20 43.00 5.00 16 40.00 30.00 47.20 43.00 5.00 32 41.00 30.00 47.60 43.00 5.00 References S. Agashe, K. Wong, V. Tu, J. Yang, A. Li, and X. E. Wang (2025) Agent s2: a compositional generalist-specialist framework for computer use agents. In Second Conference on Language Modeling, External Links: Link Cited by: Introduction. S. Back, D. Lee, N. Kang, T. Lee, S. K. Hong, Y. Gwon, and S. Ahn (2026) Understanding lora as knowledge memory: an empirical analysis. External Links: 2603.01097, Link Cited by: Baselines.. J. Cao, J. Wang, R. Wei, Q. Guo, K. Chen, B. Zhou, and Z. Lin (2025) Memory decoder: a pretrained, plug-and-play memory for large language models. In Advances in Neural Information Processing Systems, Cited by: Parametric Memory for LLMs. P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav (2025) Mem0: building production-ready ai agents with scalable long-term memory. External Links: 2504.19413 Cited by: External Memory for LLMs, Evaluation and Benchmarks.. DeepSeek-AI, A. Liu, B. Feng, et al. (2025) DeepSeek-v3 technical report. External Links: 2412.19437, Link Cited by: Introduction, Inference with TransMem. Y. Ding, L. L. Zhang, C. Zhang, Y. Xu, N. Shang, J. Xu, F. Yang, and M. Yang (2024) LongRoPE: extending LLM context window beyond 2 million tokens. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 11091–11104. Cited by: Introduction. Y. Du, W. Huang, D. Zheng, Z. Wang, S. Montella, M. Lapata, K. Wong, and J. Z. Pan (2025) Rethinking memory in llm based agents: representations, operations, and emerging topics. External Links: 2505.00675, Link Cited by: Introduction. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, et al. (2024) The llama 3 herd of models. External Links: 2407.21783, Link Cited by: Implementation Details.. C. Hsieh, Y. Chuang, C. Li, Z. Wang, L. Le, A. Kumar, J. Glass, A. Ratner, C. Lee, R. Krishna, and T. Pfister (2024) Found in the middle: calibrating positional attention bias improves long context utilization. In Findings of the Association for Computational Linguistics: ACL 2024, p. 14982–14995. External Links: Link, Document Cited by: Introduction. Y. Hu, Y. Wang, and J. McAuley (2026) Evaluating memory in LLM agents via incremental multi-turn interactions. In The Fourteenth International Conference on Learning Representations, Cited by: 3rd item, Evaluation and Benchmarks.. F. Ji, Z. Xie, J. Yang, F. Zhang, Z. Song, and X. Chen (2026) Parametric memory decoding for zero-shot routing in lora-based external parametric memory. External Links: 2607.04118, Link Cited by: Parametric Memory for LLMs. C. Latimer, N. Boschi, A. Neeser, C. Bartholomew, G. Srivastava, X. Wang, and N. Ramakrishnan (2025) Hindsight is 20/20: building agent memory that retains, recalls, and reflects. External Links: 2512.12818 Cited by: External Memory for LLMs. H. Lei, J. Liu, Y. Chen, D. Wang, and H. Wang (2026a) MemCoT: test-time scaling through memory-driven chain-of-thought. External Links: 2604.08216, Link Cited by: Introduction, External Memory for LLMs. J. Lei, D. Zhang, J. Li, W. Wang, K. Fan, X. Liu, Q. Liu, X. Ma, B. Chen, and S. Poria (2026b) δ-Mem: efficient online memory for large language models. External Links: 2605.12357 Cited by: Introduction, Parametric Memory for LLMs, Baselines.. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, et al. (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33, p. 9459–9474. Cited by: Baselines.. Y. Li, Y. Zuo, B. He, J. Zhang, C. Xiao, C. Qian, T. Yu, H. Gao, W. Yang, Z. Liu, and N. Ding (2026) Rethinking on-policy distillation of large language models: phenomenology, mechanism, and recipe. External Links: 2604.13016, Link Cited by: Training Method.. Y. Li, F. Wei, C. Zhang, and H. Zhang (2024) EAGLE: speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning, Cited by: Introduction, Inference with TransMem. J. Liu, Y. Su, P. Xia, S. Han, Z. Zheng, C. Xie, M. Ding, and H. Yao (2026) SimpleMem: efficient lifelong memory for llm agents. External Links: 2601.02553, Link Cited by: External Memory for LLMs. J. Liu, Y. Sun, W. Cheng, H. Lei, Y. Chen, L. Wen, X. Yang, D. Fu, P. Cai, N. Deng, Y. Yu, S. Hu, B. Shi, and D. Wang (2025) MemVerse: multimodal memory for lifelong learning agents. External Links: 2512.03627 Cited by: Parametric Memory for LLMs. N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang (2024) Lost in the middle: how language models use long contexts. Transactions of the Association for Computational Linguistics 12, p. 157–173. Cited by: Introduction, Introduction. A. Maharana, D. Lee, S. Tulyakov, M. Bansal, F. Barbieri, and Y. Fang (2024) Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), p. 13851–13870. External Links: Document Cited by: 3rd item, Evaluation and Benchmarks.. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe (2022) Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22. External Links: ISBN 9781713871088 Cited by: Training Method.. S. Ouyang, J. Yan, I. Hsu, Y. Chen, K. Jiang, Z. Wang, R. Han, L. Le, S. Daruki, X. Tang, V. Tirumalashetty, G. Lee, M. Rofouei, H. Lin, J. Han, C. Lee, and T. Pfister (2026) ReasoningBank: scaling agent self-evolving with reasoning memory. In International Conference on Learning Representations, External Links: Link Cited by: External Memory for LLMs. C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez (2024) MemGPT: towards llms as operating systems. External Links: 2310.08560 Cited by: External Memory for LLMs. Z. Pan, Q. Wu, H. Jiang, M. Xia, X. Luo, J. Zhang, Q. Lin, V. Ruhle, Y. Yang, C. Lin, H. V. Zhao, L. Qiu, and D. Zhang (2024) LLMLingua-2: data distillation for efficient and faithful task-agnostic prompt compression. In Findings of the Association for Computational Linguistics ACL, L. Ku, A. Martins, and V. Srikumar (Eds.), p. 963–981. Cited by: Baselines.. R. W. H. Quek, S. Lee, A. W. L. Leong, A. Verma, A. Prakash, N. F. Chen, B. K. H. Low, D. Rus, and A. Solar-Lezama (2026) MeMo: memory as a model. External Links: 2605.15156, Link Cited by: Parametric Memory for LLMs. Qwen Team (2025a) Qwen2.5 technical report. External Links: 2412.15115, Link Cited by: Implementation Details.. Qwen Team (2025b) Qwen3 technical report. External Links: 2505.09388, Link Cited by: Implementation Details.. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: Training Method.. Y. Shi, Y. Chen, S. Wang, S. Li, H. Cai, Q. GU, X. Wang, and A. Zhang (2026) Look back to reason forward: revisitable memory for long-context LLM agents. In International Conference on Learning Representations, Cited by: Introduction. G. Srivastava, S. Cao, and X. Wang (2025) ThinkSLM: towards reasoning in small language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), p. 32612–32662. External Links: Document Cited by: External Memory for LLMs. G. Srivastava, A. Hussain, C. Wang, Y. C. Lin, and X. Wang (2026) EffGen: enabling small language models as capable autonomous agents External Links: 2602.00887, Link Cited by: External Memory for LLMs. J. Sun, J. Zhu, J. Yao, T. Liu, and B. Han (2026) Rethinking how to remember: beyond atomic facts in lifelong llm agent memory. arXiv preprint arXiv:2605.19952. Cited by: Baselines.. J. Tan, Z. Dou, L. Zhang, Y. Hu, Y. Cheng, and J. Wen (2026) MemSifter: offloading llm memory retrieval via outcome-driven proxy reasoning. External Links: 2603.03379, Link Cited by: Parametric Memory for LLMs. X. Wang, C. Wang, H. Nekoei, C. Pal, A. Lacoste, S. Gella, B. Liu, and P. Taslakian (2026) Mem-π: adaptive memory through learning when and what to generate. External Links: 2605.21463, Link Cited by: Parametric Memory for LLMs. R. Wei, J. Cao, J. Wang, J. Kai, Q. Guo, B. Zhou, and Z. Lin (2026) MLP memory: a retriever-pretrained memory for large language models. In International Conference on Learning Representations, Cited by: Introduction, Parametric Memory for LLMs, Baselines.. D. Wu, H. Wang, W. Yu, Y. Zhang, K. Chang, and D. Yu (2025) LongMemEval: benchmarking chat assistants on long-term interactive memory. In International Conference on Learning Representations, Cited by: Introduction. W. Xie and X. Wang (2026) Decoupled reasoning with implicit fact tokens (DRIFT): a dual-model framework for efficient long-context inference. In Findings of the Association for Computational Linguistics: ACL, M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), p. 38047–38067. Cited by: Parametric Memory for LLMs. S. Yan, X. Yang, Z. Huang, E. Nie, Z. Ding, Z. Li, X. Ma, J. Bi, K. Kersting, J. Z. Pan, H. Schuetze, V. Tresp, and Y. Ma (2026) Memory-r1: enhancing large language model agents to manage and utilize memories via reinforcement learning. In Annual Meeting of the Association for Computational Linguistics (Long Papers), p. 12805–12825. External Links: Document Cited by: Baselines.. J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press (2024) SWE-agent: agent-computer interfaces enable automated software engineering. NIPS ’24. Cited by: Introduction. W. Yang, J. Xiao, H. Zhang, Q. Zhang, Y. Wang, and B. Xu (2026) Coarse-to-fine grounded memory for llm agent planning. External Links: 2508.15305 Cited by: Introduction. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning (2018) HotpotQA: a dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing, Cited by: 3rd item, Evaluation and Benchmarks.. H. Yu, T. Chen, J. Feng, J. Chen, W. Dai, Q. Yu, Y. Zhang, W. Ma, J. Liu, M. Wang, and H. Zhou (2026a) MemAgent: reshaping long-context LLM with multi-conv RL-based memory agent. In International Conference on Learning Representations, Cited by: Introduction. X. Yu, L. fan, X. Qiu, N. Dong, Y. Huang, H. Qi, G. Pu, W. Ouyang, X. Chen, and J. Hu (2026b) MEMTS: internalizing domain knowledge via parameterized memory for retrieval-free domain adaptation of time series foundation models. External Links: 2602.13783, Link Cited by: Parametric Memory for LLMs. Y. Yu, L. Yao, Y. Xie, Q. Tan, J. Feng, Y. Li, and L. Wu (2026c) Agentic memory: learning unified long-term and short-term memory management for large language model agents. In Annual Meeting of the Association for Computational Linguistics (Long Papers), p. 21457–21483. External Links: Document Cited by: Implementation Details.. G. Zhang, M. Fu, and S. Yan (2025a) MemGen: weaving generative latent memory for self-evolving agents. External Links: 2509.24704 Cited by: Parametric Memory for LLMs, Baselines.. G. Zhang, H. Ren, C. Zhan, Z. Zhou, J. Wang, H. Zhu, W. Zhou, and S. Yan (2025b) MemEvolve: meta-evolution of agent memory systems. External Links: 2512.18746, Link Cited by: External Memory for LLMs. X. Zhang, Z. Ding, T. Pan, R. Yang, C. Kang, X. Xiong, and J. Gu (2026) OPSDL: on-policy self-distillation for long-context language models. External Links: 2604.17535, Link Cited by: Evidence-Conditioned Self-distillation Training. Z. Zhang, Q. Dai, X. Bo, C. Ma, R. Li, X. Chen, J. Zhu, Z. Dong, and J. Wen (2025c) A survey on the memory mechanism of large language model-based agents. ACM Trans. Inf. Syst. 43 (6). External Links: ISSN 1046-8188, Link, Document Cited by: External Memory for LLMs. H. Zhao, J. Li, Y. Pan, S. Liang, X. Yang, F. Dou, T. Liu, and J. Lu (2025) HELENE: hessian layer-wise clipping and gradient annealing for accelerating fine-tuning LLM with zeroth-order optimization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), p. 26044–26067. External Links: Document Cited by: Introduction. W. Zhong, L. Guo, Q. Gao, H. Ye, and Y. Wang (2024) Memorybank: enhancing large language models with long-term memory. In Proceedings of the AAAI conference on artificial intelligence, Vol. 38, p. 19724–19731. Cited by: Baselines..