Paper deep dive
MemOPD: On-Policy Distillation through Memory State Alignment for Long-Horizon Agents
Zhiyuan Liu, Tinghong Ye, Chenghao Liu, Yizhuo Li, Songfang Huang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/10/2026, 4:08:41 AM
Summary
The paper introduces MemOPD (Memory-Aligned On-Policy Distillation), a training framework for long-horizon agents using compact memory. It addresses the state mismatch problem where context rewriting during memory compression breaks the alignment required for valid on-policy distillation. MemOPD reconstructs the exact invocation state for sampled actions, ensuring teacher supervision matches the rollout state, and improves F1 scores significantly over standard PPO.
Entities (7)
Relation Signals (5)
MemOPD → improves → F1 Score
confidence 95% · MemOPD-3B improves F1 over PPO by up to 416.2%
MemOPD → uses → On-Policy Distillation
confidence 95% · MemOPD records the inputs and sampled outputs... The teacher provides full-vocabulary supervision
MemOPD → solves → State Mismatch
confidence 90% · We identify and formulate the state mismatch problem introduced by context rewriting in compact memory agents.
PPO → usedin → MemOPD
confidence 90% · The teacher provides full-vocabulary supervision at the sampled action positions, while PPO preserves the final task objective.
RCE → verifies → State Alignment
confidence 90% · RCE therefore tests whether packed training reproduces the conditional computation used during rollout
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Long-horizon agents accumulate growing contexts during interaction, impairing performance and stability. Compact memory mitigates this problem by compressing and rewriting the history retained between model invocations. Learning what to retain typically relies on proximal policy optimization (PPO) with final task rewards, but sparse rewards provide little guidance for individual memory updates. This limitation motivates on-policy distillation (OPD), which supplies dense teacher supervision on student rollouts. For such supervision to be valid, the teacher must evaluate each sampled action under the same state in which it was generated. However, the context rewriting performed during memory compression can break this alignment. When sampled responses are retained and re-encoded for later invocations, flattening the interaction into a persistent history may cause the teacher to score the action under a state that the student never visited during rollout. The action therefore remains on-policy by provenance, but not necessarily by state. We therefore propose Memory-Aligned On-Policy Distillation (MemOPD). MemOPD records the inputs and sampled outputs of each model invocation, restores its original token positions and causal visibility, and packs the reconstructed invocations for efficient teacher scoring. The teacher provides full-vocabulary supervision at the sampled action positions, while PPO preserves the final task objective. Experiments verify state alignment across several context updates and show that it improves F1 by 7.0% over persistent-history teacher scoring in a matched control. Overall, MemOPD-3B improves F1 over PPO by up to 416.2%, while packing yields up to a 1.63x speedup in actor computation during training. The code for this work is publicly available at: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.07068v1
- Canonical: https://arxiv.org/abs/2608.07068v1
Trouble viewing inline? Open PDF directly →
Full Text
52,596 characters extracted from source content.
Expand or collapse full text
MemOPD: On-Policy Distillation through Memory State Alignment for Long-Horizon Agents Zhiyuan Liu1 , Tinghong Ye2 , Chenghao Liu1 , Yizhuo Li3, Songfang Huang1 Abstract Long-horizon agents accumulate growing contexts during interaction, impairing performance and stability. Compact memory mitigates this problem by compressing and rewriting the history retained between model invocations. Learning what to retain typically relies on proximal policy optimization (PPO) with final task rewards, but sparse rewards provide little guidance for individual memory updates. This limitation motivates on-policy distillation (OPD), which supplies dense teacher supervision on student rollouts. For such supervision to be valid, the teacher must evaluate each sampled action under the same state in which it was generated. However, the context rewriting performed during memory compression can break this alignment. When sampled responses are retained and re-encoded for later invocations, flattening the interaction into a persistent history may cause the teacher to score the action under a state that the student never visited during rollout. The action therefore remains on-policy by provenance, but not necessarily by state. We therefore propose Memory-Aligned On-Policy Distillation (MemOPD). MemOPD records the inputs and sampled outputs of each model invocation, restores its original token positions and causal visibility, and packs the reconstructed invocations for efficient teacher scoring. The teacher provides full-vocabulary supervision at the sampled action positions, while PPO preserves the final task objective. Experiments verify state alignment across several context updates and show that it improves F1 by 7.0% over persistent-history teacher scoring in a matched control. Overall, MemOPD-3B improves F1 over PPO by up to 416.2%, while packing yields up to a 1.63×1.63× speedup in actor computation during training. The code for this work is publicly available at: https://github.com/TPssp/MemOPD. 1 Introduction Large language models (LLMs) have driven rapid advances in language agents, enabling them to reason, plan, and interact with external tools (Wang et al. 2024; Yao et al. 2023; Nakano et al. 2022). These agents have demonstrated strong performance in a wide range of long-horizon tasks such as information-seeking workflows, where multiple rounds of reasoning, action execution, and environment feedback are required (Yao et al. 2022; Deng et al. 2023; Gur et al. 2024; Jin et al. 2025; Zheng et al. 2025). During such interactions, each model invocation produces new responses and observations that may influence future decisions. As a result, maintaining the complete interaction history leads to continuously growing contexts, increasing Transformer computation costs and making it more difficult for the model to identify and utilize relevant information (Vaswani et al. 2017; Liu et al. 2024; Wu et al. 2025; An et al. 2025). To address this challenge, existing approaches introduce retrieval mechanisms, external memory systems, or context compression strategies to control the amount of information provided to the model (Lewis et al. 2020; Karpukhin et al. 2020; Borgeaud et al. 2022; Park et al. 2023; Zhong et al. 2023; Packer et al. 2024; Chhikara et al. 2025; Xu et al. 2025; Yoon et al. 2024; Li et al. 2023; Cao et al. 2024; Lee et al. 2024; Jiang et al. 2023, 2024). Among them, compact memory provides a more direct solution by learning to rewrite the context retained between model invocations, selectively keeping useful information and removing outdated content. For example, MEM1 (Zhou et al. 2025) learns a compact internal state that jointly supports memory consolidation and reasoning, allowing the agent to operate with a bounded context while solving long-horizon tasks. Figure 1: Motivation for MemOPD. (A) During compact memory interaction, the behavior policy retains selected content from each response and combines it with the next observation, thereby controlling context growth. PPO learns from the final task reward, which provides limited guidance for individual memory updates. (B) Directly flattening these calls changes the state used to score a sampled action. Content retained as later context can have different roles, positions, and visibility, causing the reconstructed training state σ~t+1,k σ_t+1,k to differ from the rollout state σt+1,k _t+1,k. This mismatch motivates memory state alignment. A common strategy for training such agents is to optimize the entire interaction trajectory with proximal policy optimization (PPO) using the final task reward (Schulman et al. 2017; Jin et al. 2025; Zhou et al. 2025). However, the reward is often sparse and delayed, providing limited supervision for individual memory updates. Teacher distillation provides an alternative source of dense learning signals by transferring the teacher’s output distribution to the student model (Hinton et al. 2015; Kim and Rush 2016). Recent language model studies further extend distillation to student-generated sequences (Gu et al. 2024; Ko et al. 2024; Agarwal et al. 2024). On-policy distillation (OPD) can therefore guide sampled memory decisions while PPO preserves the final task objective. However, directly applying OPD in this setting introduces an additional requirement: the state used for teacher evaluation must exactly match the state in which the student generated the action. Standard OPD assumes that the autoregressive prefix that produces an action remains unchanged when the action is scored, but context rewriting can break this assumption. During interaction, the agent may generate a response, retain only part of it as memory, and later insert the retained content into a new context. If training simply flattens these interactions into a persistent history, the same response tokens may have different positions, causal visibility, or prediction locations. Consequently, an action can remain on-policy in terms of its origin, since it was sampled from the student policy, while the reconstructed training state no longer corresponds to any state visited during rollout. The teacher may therefore evaluate an action under an incorrect context, causing the distillation objective to optimize a different conditional distribution from the one that produced the behavior. In our 3B model audit, persistent-history reconstruction produced a p99 log probability error of 1.774, changed the top prediction at 651 sampled action positions, and falsely triggered PPO clipping for 13.29% of the actions. This observation motivates Memory-Aligned On-Policy Distillation (MemOPD), which reconstructs the call state that produced each sampled action before applying teacher supervision. Instead of treating the interaction history as a single flattened sequence, MemOPD records the exact input and output of every model invocation during rollout, restores the original token positions, causal visibility, and prediction locations, and packs the reconstructed invocations into an efficient training representation. The teacher can then provide dense full vocabulary supervision under the state that produced each action, while PPO continues to optimize the final task objective. To verify that the reconstruction preserves the original computation, we introduce rollout context equivalence (RCE), which requires the packed representation to produce the same action logits as independently executing each model invocation. RCE therefore tests whether packed training reproduces the conditional computation used during rollout rather than merely forming a valid tensor. Moreover, the reconstructed representation enables efficient batched execution by sharing common computation across invocations without sacrificing state fidelity. We evaluate MemOPD on long-horizon retrieval tasks in which the agent uses compact memory. Across five independently trained seeds, MemOPD-3B raises F1 over PPO by up to 416.2%. In a matched Q2 control, it further improves F1 over persistent-history teacher scoring by 7.0%. Its packing strategy yields a maximum 1.63×1.63× speedup in actor computation during training. Our contributions are summarized as follows: 1. We identify and formulate the state mismatch problem introduced by context rewriting in compact memory agents. We show that student-generated actions alone are insufficient for valid on-policy distillation, and introduce memory state alignment as a necessary condition for applying OPD to long-horizon agents. 2. We propose MemOPD, a training framework that reconstructs each model invocation, separates sampled actions from their later context occurrences, and packs the reconstructed states for efficient teacher-guided optimization. 3. We introduce RCE as a verification criterion for memory-aware training and show that MemOPD exactly recovers computation across diverse context-update strategies. Experiments on long-horizon retrieval tasks demonstrate consistent gains in performance and efficiency. 2 Related Work Memory management for long-horizon agents. Long-horizon agents combine reasoning with environment actions across web navigation, information seeking, and other multi-turn tasks (Yao et al. 2023, 2022; Nakano et al. 2022; Deng et al. 2023; Gur et al. 2024). To keep interaction histories bounded, agent systems maintain episodic or hierarchical memories outside the active context (Park et al. 2023; Zhong et al. 2023; Packer et al. 2024; Chhikara et al. 2025; Xu et al. 2025), while prompt compression and learned reading shorten the context presented to the model (Yoon et al. 2024; Lee et al. 2024; Jiang et al. 2023, 2024). Other approaches learn context management inside the agent through trajectory folding, explicit memory operations, bounded memory control, or memory specific training signals (Sun et al. 2025; Yu et al. 2026; Wang et al. 2026; Li et al. 2026b; Zhou et al. 2025; Li et al. 2026a). MEM1, for example, learns a consolidated internal state and observes that position changes prevent its packed representation from exactly recovering rollout computation (Zhou et al. 2025). These methods mainly study what information an agent should retain, retrieve, or compress. They do not provide the state aligned OPD interface needed to score a sampled decision after its context has been rewritten. MemOPD addresses this training condition by recovering the exact state that produced each decision. Learner states and on-policy distillation. Imitation learning has long recognized that learner decisions visit states that differ from a fixed expert trajectory. DAgger therefore queries an expert on states visited by the learner (Ross et al. 2011). This strategy determines where supervision is collected, but it does not ensure that a stored trajectory later reproduces the same conditioning state. Knowledge distillation matches a student to a teacher distribution, while sequence level methods transfer supervision to generated sequences (Hinton et al. 2015; Gou et al. 2021; Kim and Rush 2016). Language model methods further use reverse Kullback–Leibler (KL) objectives or student generated data (Gu et al. 2024; Ko et al. 2024). Generalized knowledge distillation formulates OPD by querying a teacher on student generated sequences (Agarwal et al. 2024). Standard OPD assumes that the autoregressive prefix used to generate an action remains available when the action is scored. Context rewriting breaks this assumption because a response can later reappear as context with different positions or visibility. Consequently, applying standard OPD directly does not guarantee valid teacher supervision. MemOPD instead reconstructs the conditioning state and preserves the sampled action domain; its reverse KL term is unchanged. 3 Method 3.1 Overview An agent with compact memory repeatedly invokes a trainable student policy, executes an external action, and rewrites the context for the next invocation. If training flattens these invocations into one persistent sequence, action likelihoods and teacher targets can be evaluated under inputs that the student never encountered. The resulting objective no longer represents the policy trajectory that produced the data, even though the batch remains syntactically valid. MemOPD prevents this mismatch by recording each student invocation, reconstructing its memory state, and packing the independent computations. At each update, a frozen snapshot πold _old of the trainable student πθ _θ generates the rollout, whereas the frozen teacher πT _T only scores recorded actions after reconstruction. OPD then supplies local guidance on these actions, while PPO retains the task objective. The reference policy πref _ref and critic VϕV_φ use the same reconstruction, so every objective uses consistent state and action semantics. Figure 2 illustrates the pipeline. Figure 2: Overview of MemOPD. Here, a call is a model invocation. (1) MemOPD reconstructs each call, separates sampled actions from later context occurrences, and packs the aligned computations under a shared task prefix. (2) RCE compares packed and independent full vocabulary logits within numerical tolerance. (3) The aligned batch supports PPO over the sampled action domain and full vocabulary teacher guidance over the teacher supervision mask. 3.2 Compact Memory Rollouts To identify the state that training must reproduce, we first formalize how compact memory generates each sampled action. We write the fixed task and system prefix as q; it specifies the task and the interaction protocol. Before invocation t, the mutable context xtx_t contains the memory and latest observation available to the student. Conditioned on [q,xt][q,x_t], the behavior policy samples a response ata_t that updates memory and selects a search or final-answer action. The environment executes this action and returns an observation oto_t, a task reward rtr_t, and a terminal indicator dtd_t. The context update rule U then constructs the input for the next invocation: at a_t ∼πold(⋅∣q,xt), _old(\,· q,x_t), (ot,rt,dt) (o_t,r_t,d_t) =ℰ(at), =E(a_t), (1) xt+1 x_t+1 =U(xt,at,ot). =U(x_t,a_t,o_t). The student policy, the environment, and U jointly determine the states visited during rollout. The teacher does not generate a separate trajectory; it is queried only after these student states have been reconstructed. Our end-to-end study instantiates U with the compact-memory protocol of MEM1 (Zhou et al. 2025). The stable prefix contains the task and interaction instructions. Each response contains reasoning, retained memory, and either a search command or a final answer. After a search, the returned evidence becomes the new observation. The runtime keeps the response content required for the next invocation, appends the new evidence, and removes the earlier mutable context. A three-invocation interaction therefore has the form [q]→a0,[q,m0,o0]→a1,[q,m1,o1]→a2,[q]→ a_0, [q,m_0,o_0]→ a_1, [q,m_1,o_1]→ a_2, (2) where mtm_t is the content retained from response ata_t. The third invocation receives [q,m1,o1][q,m_1,o_1], not an accumulated history such as [q,a0,o0,a1,o1][q,a_0,o_0,a_1,o_1]. This distinction is central to training because a sampled response and its later context copy can have different token positions and visibility. Training first uses supervised fine tuning (SFT) to teach the required reasoning, memory, search, and answer format. The frozen behavior policy πold _old then samples complete student interactions, and PPO optimizes the student using the final task reward. The reference policy limits drift, while the critic and generalized advantage estimation (GAE) provide token-level learning signals. Before any objective scores an action, MemOPD reconstructs the invocation that produced it. The frozen teacher then adds dense supervision without generating the trajectory or replacing the task reward. 3.3 Memory State Alignment Rollout and training states. Valid training requires every sampled action to be scored under the invocation that produced it. For invocation t, let zt=[q,xt]z_t=[q,x_t] be the exact tokenized input. We define its realized memory state as σt=(zt,ρt,Vt,γt), _t=(z_t, _t,V_t, _t), (3) where ρt _t gives token positions, VtV_t specifies causal visibility, and γt(k) _t(k) identifies the position whose output distribution predicts action token at,ka_t,k. Thus, a memory state is the complete model input presented at an invocation, including its token IDs and computation structure. It is neither the retained text alone nor a Transformer hidden state, and identical decoded text can represent different memory states. The state used to predict action token at,ka_t,k also includes the earlier tokens at,<ka_t,<k from the same response. Let σt,k=(zt,at,<k,ρt,Vt,γt(k)) _t,k=(z_t,a_t,<k, _t,V_t, _t(k)) denote this rollout state and σ~t,k σ_t,k the state reconstructed during training. Memory state alignment requires σ~t,k=σt,k σ_t,k= _t,k for every sampled action token. When this equality fails, behavior likelihoods, PPO ratios, and teacher targets no longer describe the sampled decision, so action provenance alone is insufficient. Reconstruction and packing. To enforce this equality, MemOPD records ztz_t and the exact sampled token IDs of ata_t during rollout rather than decoding and tokenizing the response again. Keeping these IDs preserves the sampled action and its token boundaries throughout reconstruction. The reconstruction compiler places the recorded invocations in one physical sequence, y=[q,x0,a0,x1,a1,…,xT−1,aT−1],y=[q,\,x_0,a_0,\,x_1,a_1,…,x_T-1,a_T-1], (4) while treating physical adjacency and causal visibility separately. The stable prefix q has one physical occurrence, keeps positions 0,…,|q|−10,…,|q|-1, and is visible to every invocation block. This sharing is valid because its token IDs and positions are identical in all independent executions. A token in xtx_t sees only q and preceding tokens in the context of invocation t. A token in ata_t additionally sees preceding tokens of ata_t, while tokens assigned to other invocations are blocked. Finally, positions restart from the indices used by the corresponding independent invocation instead of increasing across the packed sequence. When a response is retained for a later invocation, the compiler represents it twice because its two occurrences serve different purposes. The first is the exact sampled action in the policy trajectory. The second is context for a later invocation and therefore receives the positions and visibility of that invocation. If a sequence limit would remove a conditioning token or sampled action, the compiler rejects the example rather than silently changing the training state. A correct reconstruction should reproduce, for every sampled action token, the same full vocabulary logits as independent execution. We call this property rollout context equivalence (RCE). Let g(t,k)g(t,k) denote the position in the packed sequence whose output distribution predicts action token at,ka_t,k. Let ℓg(t,k)pack ^pack_g(t,k) and ℓt,kind ^ind_t,k be the corresponding full vocabulary logits from packed and independent execution. RCE requires ‖ℓg(t,k)pack−ℓt,kind‖∞≤ϵnumfor every (t,k), \| ^pack_g(t,k)- ^ind_t,k \|_∞≤ _num every (t,k), (5) where ϵnum _num is the numerical tolerance for the execution dtype and backend. We test this condition directly against independent invocation logits for the same recorded actions. Sampled action domain. Correct conditioning does not determine which physical tokens are policy decisions. Retained response copies and environment observations condition future actions, but they were not sampled at those physical positions. We mark these decisions as follows: mjact=[yj lies in the sampled occurrence of a recorded at].m^act_j=1[\,y_j lies in the sampled occurrence of a recorded a_t\,]. (6) The selected positions form the sampled action domain. Although a later context occurrence may repeat the same token IDs, it occupies different physical positions and receives mact=0m^act=0. Policy loss, entropy, reference regularization, value learning, and GAE (Schulman et al. 2016) use this domain. GAE advances across ordered sampled actions and skips positions used only for context. Teacher supervision has an indicator mTm^T satisfying mjT≤mjactm^T_j≤ m^act_j. The teacher mask may select a subset of the action domain; in the main experiment, it covers every sampled response token, so mT=mactm^T=m^act. Model Q2 Q8 Q16 EM ↑ F1 ↑ Peak ↓ Time ↓ EM ↑ F1 ↑ Peak ↓ Time ↓ EM ↑ F1 ↑ Peak ↓ Time ↓ Qwen2.5-14B-Inst 0.734 0.904 15.45±0.1715.45\!±\!0.17 5.55±0.175.55\!±\!0.17 1.552 1.872 44.52±0.3344.52\!±\!0.33 16.11±0.2516.11\!±\!0.25 0.567 0.704 38.18±0.7238.18\!±\!0.72 29.63±0.6229.63\!±\!0.62 Qwen2.5-7B-Inst 0.267 0.365 19.64±0.3319.64\!±\!0.33 4.65±0.094.65\!±\!0.09 0.872 1.102 49.57±0.4049.57\!±\!0.40 14.04±0.1814.04\!±\!0.18 0.166 0.214 43.10±0.5343.10\!±\!0.53 15.58±0.2215.58\!±\!0.22 Qwen2.5-7B-Inst (A-MEM) 0.286 0.372 14.17±0.1114.17\!±\!0.11 24.66±0.4924.66\!±\!0.49 1.131 1.431 18.53±0.1118.53\!±\!0.11 54.16±1.1054.16\!±\!1.10 0.733 0.966 18.87±0.1518.87\!±\!0.15 92.21±2.4592.21\!±\!2.45 Qwen2.5-7B-Inst (truncate) 0.261 0.335 8.32±0.058.32\!±\!0.05 6.02±0.176.02\!±\!0.17 0.968 1.228 11.78±0.1011.78\!±\!0.10 11.63±0.1911.63\!±\!0.19 0.396 0.498 13.28±0.1413.28\!±\!0.14 22.12±0.5922.12\!±\!0.59 Search-R1 0.454 0.533 13.05±0.0813.05\!±\!0.08 4.29±0.224.29\!±\!0.22 0.064 0.080 24.74±0.1724.74\!±\!0.17 4.33±0.164.33\!±\!0.16 0.009 0.011 20.91±0.0320.91\!±\!0.03 4.66±0.174.66\!±\!0.17 DeepResearcher 0.532 0.646 21.79±0.4121.79\!±\!0.41 4.01±0.084.01\!±\!0.08 0.731 0.902 51.87±0.3851.87\!±\!0.38 11.16±0.1611.16\!±\!0.16 0.071 0.106 48.31±0.6048.31\!±\!0.60 15.79±0.1815.79\!±\!0.18 MEM1-QA (7B) 0.708 0.836 6.39±0.026.39\!±\!0.02 6.49±0.066.49\!±\!0.06 1.860 2.301 8.05±0.068.05\!±\!0.06 8.61±0.118.61\!±\!0.11 1.883 2.285 10.31±0.0810.31\!±\!0.08 8.71±0.108.71\!±\!0.10 Mixed-horizon SFT (3B) 0.469 0.606 4.85±0.084.85\!±\!0.08 4.44±0.184.44\!±\!0.18 0.913 1.146 6.56±0.336.56\!±\!0.33 14.43±0.2314.43\!±\!0.23 1.048 1.352 12.26±0.0512.26\!±\!0.05 18.23±0.1318.23\!±\!0.13 PPO (3B) 0.658±0.0150.658\!±\!0.015 0.806±0.0200.806\!±\!0.020 6.49±0.036.49\!±\!0.03 5.53±0.225.53\!±\!0.22 0.538±0.0150.538\!±\!0.015 0.655±0.0150.655\!±\!0.015 7.93±0.657.93\!±\!0.65 22.32±0.1822.32\!±\!0.18 0.442±0.0110.442\!±\!0.011 0.549±0.0240.549\!±\!0.024 14.42±0.1614.42\!±\!0.16 24.60±0.1424.60\!±\!0.14 MemOPD (3B) 0.756±0.0200.756\!±\!0.020 0.921±0.0240.921\!±\!0.024 7.29±0.037.29\!±\!0.03 5.91±0.165.91\!±\!0.16 1.913±0.0241.913\!±\!0.024 2.510±0.0192.510\!±\!0.019 8.20±0.118.20\!±\!0.11 20.77±0.1420.77\!±\!0.14 2.430±0.0292.430\!±\!0.029 2.834±0.0192.834\!±\!0.019 9.66±0.099.66\!±\!0.09 21.14±0.1121.14\!±\!0.11 Table 1: Multi-objective multi-hop retrieval on Q2, Q8, and Q16. EM and F1 are summed across questions. Peak context is in 10210^2 tokens and time is in seconds per trajectory. PPO and MemOPD report mean ± standard deviation over five seeds. This sampled-action-domain contract is separate from state reconstruction. Correct visibility can still mark a retained response copy as another action, causing the actor, critic, and GAE to count one decision twice. Conversely, a correct action domain cannot repair logits computed with deleted history or wrong positions. State reconstruction preserves conditioning, while the action domain preserves which tokens count as decisions. 3.4 Teacher Guidance and Policy Optimization Once the states and action domain are aligned, MemOPD combines local teacher guidance with task-level policy optimization. The student and frozen teacher receive the same tokens, positions, and visibility and share a tokenizer and vocabulary V. For a supervised action position j containing token at,ka_t,k, let pθ,jp_θ,j and pT,jp_T,j be the distributions read at g(t,k)g(t,k) from the student and teacher. We use full vocabulary reverse KL divergence, ℒOPD=1∑jmjT∑jmjT∑v∈pθ,j(v)logpθ,j(v)pT,j(v).L_OPD= 1 _jm^T_j _jm^T_j _v p_θ,j(v) p_θ,j(v)p_T,j(v). (7) The divergence itself is standard in language model distillation (Gu et al. 2024; Ko et al. 2024; Agarwal et al. 2024). Its role here is to expose the complete teacher distribution only after the student memory state has been recovered. The reference policy is separate from both the teacher and the behavior snapshot. It remains frozen at the SFT initialization, whereas πold _old is the policy that generated the current rollout batch. Following the KL regularization commonly used in reinforcement learning from human feedback (Ouyang et al. 2022), the reference policy constrains drift by adding a token-level KL penalty to the task reward. Let ℓold,j _old,j and ℓref,j _ref,j be the log probabilities assigned to the sampled action token by the behavior and reference policies, respectively. We define the reference KL penalty and the resulting token-level reward as KLjref=[ℓold,j−ℓref,j]+,r~j=rj−βKLjref.KL^ref_j= [ _old,j- _ref,j ]_+, r_j=r_j-β\,KL^ref_j. (8) Here, β controls reference regularization. The scalar task reward is placed on the final sampled action token, while the reference penalty is defined at every sampled action position. The critic predicts values at these positions, and GAE propagates the resulting signal backward through the ordered student actions. Let τ1,…,τN _1,…, _N be the positions for which mact=1m^act=1. The recursion follows these decisions rather than adjacent storage positions: δn _n =r~τn+γVτn+1−Vτn, = r_ _n+γ V_ _n+1-V_ _n, (9) Aτn A_ _n =δn+γλGAEAτn+1. = _n+γ _GAEA_ _n+1. Here, γ is the discount factor and λGAE _GAE is the GAE decay parameter. The value after a terminal action is zero. Context copies and observations do not introduce additional decisions or value targets. For each sampled action position j containing token at,ka_t,k, let σj=σt,k _j= _t,k and σ~j=σ~t,k σ_j= σ_t,k. Because another state would mix a policy change with a change in conditioning, both policies evaluate aj=at,ka_j=a_t,k at g(t,k)g(t,k) under the aligned state σ~j=σj σ_j= _j. We therefore define ϱj(θ)=πθ(aj∣σ~j)/πold(aj∣σ~j) _j(θ)= _θ(a_j σ_j)/ _old(a_j σ_j) and ϱ¯j=clip(ϱj,1−ϵ,1+ϵ) _j=clip( _j,1-ε,1+ε), where ϵε is the PPO clipping threshold. The PPO loss is ℒPPO=−1∑jmjact∑jmjactmin(ϱjAj,ϱ¯jAj).L_PPO=- 1 _jm^act_j _jm^act_j \! ( _jA_j, _jA_j ). (10) The combined actor objective is ℒactor=ℒPPO−αℋ+λℒOPD,L_actor=L_PPO- + _OPD, (11) where ℋH is policy entropy, and α and λ control entropy regularization and teacher guidance, respectively. The teacher supplies dense local preferences, but it does not determine whether a sequence of memory updates and retrieval actions succeeds. PPO retains the task reward as the supervision signal for complete interaction success and can counteract local teacher preferences that harm the final answer. 4 Experiments 4.1 Experimental Setup Benchmarks. Following MEM1, we use a multi-objective question answering benchmark constructed from HotpotQA and Natural Questions (Zhou et al. 2025; Yang et al. 2018; Kwiatkowski et al. 2019). Each example combines several questions into one query, and the agent must retrieve evidence over multiple turns from a local Wikipedia corpus before answering them all. Q2, Q8, and Q16 contain 2, 8, and 16 questions, creating progressively longer retrieval and memory update sequences. PPO and MemOPD are optimized on Q2, while Q8 and Q16 test transfer to longer horizons. System EM ↑ F1 ↑ Peak ↓ Dependency ↓ Time ↓ Qwen2.5-7B-Inst (truncate) 0.296 0.386 6.31±0.056.31\!±\!0.05 1.647±0.041.647\!±\!0.04 2.241±0.042.241\!±\!0.04 Qwen2.5-7B-Inst (A-MEM) 0.246 0.372 8.51±0.118.51\!±\!0.11 0.932±0.030.932\!±\!0.03 11.319±0.3611.319\!±\!0.36 Qwen2.5-7B-Inst 0.271 0.390 9.26±0.219.26\!±\!0.21 1.151±0.051.151\!±\!0.05 2.302±0.042.302\!±\!0.04 Qwen2.5-14B-Inst 0.431 0.533 8.99±0.208.99\!±\!0.20 2.260±0.092.260\!±\!0.09 6.870±0.206.870\!±\!0.20 Search-R1 0.452 0.518 11.12±0.2711.12\!±\!0.27 1.473±0.051.473\!±\!0.05 2.218±0.142.218\!±\!0.14 DeepResearcher 0.428 0.512 13.43±0.3613.43\!±\!0.36 6.844±0.326.844\!±\!0.32 3.780±0.083.780\!±\!0.08 MEM1-QA (SFT) 0.302 0.356 6.57±0.056.57\!±\!0.05 3.296±0.133.296\!±\!0.13 4.943±0.204.943\!±\!0.20 MEM1-QA 0.409 0.470 5.62±0.035.62\!±\!0.03 0.742±0.020.742\!±\!0.02 3.751±0.073.751\!±\!0.07 PPO (3B) 0.409±0.0200.409\!±\!0.020 0.486±0.0210.486\!±\!0.021 6.00±0.056.00\!±\!0.05 1.049±0.1901.049\!±\!0.190 2.781±0.0532.781\!±\!0.053 MemOPD (3B) 0.434±0.0220.434\!±\!0.022 0.522±0.0300.522\!±\!0.030 6.13±0.056.13\!±\!0.05 0.717±0.0280.717\!±\!0.028 1.988±0.0481.988\!±\!0.048 Table 2: Answer quality and efficiency on Wiki-RAG. Peak context and dependency are in 10210^2 and 10510^5, respectively, and time is in seconds per trajectory. PPO and MemOPD report mean ± standard deviation over five seeds. We also evaluate single-objective Wiki-RAG (Jin et al. 2025), where the agent repeatedly retrieves passages from a Wikipedia datastore before answering one open-domain question. This benchmark tests transfer from multi-objective retrieval to a standard single-objective setting. Metrics. Exact match (EM) and token F1 measure answer quality. For Q2, Q8, and Q16, both metrics are summed over the questions in each example. We also report peak context, dependency, and inference time per trajectory. PPO and MemOPD results are the mean and standard deviation over five independently trained seeds; efficiency uncertainty for fixed models is the standard error across test trajectories. Dependency is ∑s|s|(|ps|+⌊|s|/2⌋) _s|s|(|p_s|+ |s|/2 ) over generated segments s, where psp_s is the visible prefix preceding that segment. Models and training. Our student is Qwen2.5 3B (Yang et al. 2025). We use gpt-oss-120b (OpenAI 2025) to generate agent trajectories, retaining only those with correct final answers and valid interaction formats. We convert each response turn into the corresponding model call format, yielding 20,036 turn-level examples. The student receives one epoch of SFT on this data before PPO or MemOPD training. Across five seeds, PPO and MemOPD use the same initialization, data order, reconstructed packed calls, sampled-action mask, and evaluation protocol. MemOPD additionally uses the frozen 7B teacher with λ=0.02λ=0.02, selected on the Q2 development evaluation. For the matched Q2 control, PPO omits teacher guidance, while the two OPD variants use the same frozen teacher and coefficient and differ only in whether teacher scoring uses persistent history or reconstructed call states. All training runs use four NVIDIA H200 GPUs. 4.2 Multi-Objective Retrieval Table 1 compares our 3B training variants with established systems on Q2, Q8, and Q16. MemOPD-3B achieves the highest EM and F1 at all horizons. Averaged over five seeds, MemOPD improves F1 over PPO by 14.3%, 283.2%, and 416.2% on Q2, Q8, and Q16, respectively. On Q16, it reduces peak context by 33.0% and inference time by 14.1%. The growing advantage shows that aligned teacher guidance is more useful at longer horizons. 4.3 Single-Objective Wiki-RAG Transfer Table 2 compares the resulting PPO and MemOPD models with established systems on Wiki-RAG. On Wiki-RAG, MemOPD improves EM and F1 over PPO by 6.1% and 7.4%, respectively, when averaged over five seeds. It also has the lowest dependency and inference time in the table. Relative to PPO, MemOPD reduces dependency by 31.6% and inference time by 28.5%, while peak context increases by 2.2%. 4.4 State Alignment Verification We next test the mechanism behind these gains by scoring the same recorded actions under persistent history, independent invocations, and reconstructed packing. Independent invocations reproduce the original call states and serve as the reference, allowing us to measure how each training representation changes the action probabilities. The audit contains 64 real trajectories, 199 model invocations, and 12,083 sampled action tokens. Representation p99 |Δlogp|↓| p| Top-1 ↑ False clip ↓ Persistent history 1.7738 94.61 13.29 Batching numerical floor 5.31×10−55.31\!×\!10^-5 100.00 0.00 Reconstructed packing 3.43×10−53.43\!×\!10^-5 100.00 0.00 Table 3: State reconstruction on 64 student trajectories. Errors compare the same sampled actions with independent invocations in float32. False clipping is the fraction of unchanged-policy ratios outside the PPO clipping interval. Context update Tokens p99 |Δlogp|↓| p| Top-1 ↑ Full response 3,329 3.43×10−53.43\!×\!10^-5 100.00 Suffix retention 3,329 3.83×10−53.83\!×\!10^-5 100.00 Summary replacement 3,329 3.59×10−53.59\!×\!10^-5 100.00 Sliding window 3,329 3.72×10−53.72\!×\!10^-5 100.00 Retrieval refresh 3,329 3.81×10−53.81\!×\!10^-5 100.00 MEM1 12,083 3.43×10−53.43\!×\!10^-5 100.00 Table 4: RCE across five controlled context updates and native MEM1 trajectories. Figure 3: Experiment diagnostics. (a) Q2 results for PPO, persistent-history OPD, and MemOPD. (b) Log probability error, top-1 changes, and false clipping under state corruption. (c) Actor time saved relative to independent invocations. Persistent history changes the top prediction at 651 action positions and falsely clips 13.29% of the actions even before any policy update. Reconstructed packing instead matches independent invocations at the numerical floor. The mismatch also changes the supervision from the actual 7B teacher. Before the first memory update, exact and persistent states give identical teacher distributions. Across later invocations, however, top-1 agreement falls to 81.20%, the p99 sampled log probability difference reaches 19.854, and the mean KL divergence reaches 0.793, changing the teacher prediction at 444 action positions. Having established this effect on MEM1 trajectories, we next test whether the same reconstruction interface supports other forms of context update. Table 4 tests whether the interface depends on one memory topology. In every case, the compiler receives the realized token context of each invocation rather than symbolic memory roles. It therefore preserves RCE across several context updates, while MEM1 remains the complete downstream training instance. 4.5 Ablation Studies Matched teacher-state control. We compare PPO, OPD with persistent-history teacher scoring, and MemOPD under the same Q2 protocol. PPO uses reconstructed student-side calls and the sampled-action mask without teacher guidance. The persistent-history variant adds the frozen teacher but scores sampled actions under a flattened interaction history, whereas MemOPD scores them under reconstructed call states. All other training and evaluation settings are fixed. As shown in Figure 3a, persistent-history teacher scoring improves F1 and EM over PPO by 5.6% and 4.4%, respectively, showing that dense teacher guidance remains useful despite the state mismatch. Reconstructing the teacher state further improves F1 and EM by 7.0% and 10.0%, respectively. Overall, MemOPD improves F1 and EM over PPO by 13.0% and 14.8% in this matched control. Reconstruction requirements. Figure 3b independently corrupts visibility and positions while keeping the recorded actions fixed. Visibility is the larger source of error, but incorrect positions alone still change 260 top predictions and falsely clip 4.97% of the actions. State reconstruction is also distinct from identifying which tokens are decisions. The exact action domain selects 12,083 sampled action tokens, whereas trajectory and response masks add 66,830 and 74,357 spurious positions. Packing Efficiency. Figure 3c compares reconstructed packing with correct independent invocations across nine configurations. Packing achieves up to a 1.63×1.63× actor speedup while preserving RCE. This speedup comes from shared computation without changing the optimized states. 5 Discussion MemOPD separates action provenance from state validity. A student rollout guarantees that the learner generated each action, but compact memory may delete, replace, or reencode its conditioning context before training. The resulting batch can therefore contain genuine student actions evaluated under states that never occurred during interaction. Memory state alignment enforces this requirement, and RCE audits the reconstructed logits against independent invocations. Teacher guidance and PPO serve different roles in the objective. The teacher provides dense guidance at sampled action tokens, whereas PPO evaluates whether the complete interaction solves the task. The matched control makes this distinction measurable: persistent-history teacher scoring improves F1 over PPO by 5.6%, showing that teacher guidance remains useful before state alignment, while reconstructing the teacher state adds a further 7.0% improvement. Beyond the optimization objective, state correctness also determines how packing can be used. MemOPD restores each invocation before sharing stable prefixes across otherwise independent computations. RCE verifies that this sharing preserves the action logits, so the measured speedup does not depend on an approximate rollout state. The same reconstruction interface is not tied to semantic roles defined by MEM1. Because the compiler consumes realized token contexts rather than a fixed memory topology, it preserves RCE under full response retention, suffix retention, summary replacement, sliding windows, and retrieval refresh. These results show that memory state alignment depends on the actual computation presented to the model rather than the particular form of memory rewriting, allowing MemOPD to support diverse context update mechanisms through a unified training interface. 6 Conclusion Compact memory can separate sampled actions from the states later used to train them. MemOPD resolves this mismatch by reconstructing every invocation, distinguishing sampled actions from later context copies, and combining state-aligned teacher guidance with task-level PPO. RCE verifies the packed computation, while the matched control shows that teacher guidance already improves F1 over PPO by 5.6% and state alignment adds a further 7.0%. Across five seeds, MemOPD improves F1 over PPO by up to 416.2%, while packing yields up to a 1.63×1.63× speedup in actor computation during training. References R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem (2024) On-policy distillation of language models: learning from self-generated mistakes. In Proceedings of ICLR, External Links: 2306.13649, Link Cited by: §1, §2, §3.4. C. An, J. Zhang, M. Zhong, L. Li, S. Gong, Y. Luo, J. Xu, and L. Kong (2025) Why does the effective context length of llms fall short?. In Proceedings of the International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1. S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. V. D. Driessche, J. Lespiau, B. Damoc, A. Clark, D. D. L. Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simonyan, J. Rae, E. Elsen, and L. Sifre (2022) Improving language models by retrieving from trillions of tokens. In Proceedings of the 39th International Conference on Machine Learning, External Links: Link Cited by: §1. Z. Cao, Q. Cao, Y. Lu, N. Peng, L. Huang, S. Cheng, and J. Su (2024) Retaining key information under high compression ratios: query-guided compressor for llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), External Links: Document, Link Cited by: §1. P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav (2025) Mem0: building production-ready ai agents with scalable long-term memory. External Links: Document, 2504.19413, Link Cited by: §1, §2. X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su (2023) Mind2web: towards a generalist agent for the web. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §1, §2. J. Gou, B. Yu, S. J. Maybank, and D. Tao (2021) Knowledge distillation: a survey. International Journal of Computer Vision 129, p. 1789–1819. External Links: Document, 2006.05525, Link Cited by: §2. Y. Gu, L. Dong, F. Wei, and M. Huang (2024) MiniLLM: knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, External Links: 2306.08543, Link Cited by: §1, §2, §3.4. I. Gur, H. Furuta, A. Huang, M. Safdari, Y. Matsuo, D. Eck, and A. Faust (2024) A real-world webagent with planning, long context understanding, and program synthesis. In Proceedings of the International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1, §2. G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. External Links: 1503.02531, Link Cited by: §1, §2. H. Jiang, Q. Wu, C. Lin, Y. Yang, and L. Qiu (2023) LLMLingua: compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 13358–13376. External Links: Document, 2310.05736, Link Cited by: §1, §2. H. Jiang, Q. Wu, X. Luo, D. Li, C. Lin, Y. Yang, and L. Qiu (2024) LongLLMLingua: accelerating and enhancing LLMs in long context scenarios via prompt compression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1658–1677. External Links: Document, 2310.06839, Link Cited by: §1, §2. B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han (2025) Search-r1: training llms to reason and leverage search engines with reinforcement learning. External Links: 2503.09516, Link Cited by: §1, §1, §4.1. V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. Yih (2020) Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Link Cited by: §1. Y. Kim and A. M. Rush (2016) Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, p. 1317–1327. External Links: Document, Link Cited by: §1, §2. J. Ko, S. Kim, T. Chen, and S. Yun (2024) DistiLLM: towards streamlined distillation for large language models. In Proceedings of the 41st International Conference on Machine Learning, Vol. 235, p. 24872–24895. External Links: 2402.03898, Link Cited by: §1, §2, §3.4. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov (2019) Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics 7, p. 452–466. External Links: Document, Link Cited by: §4.1. K. Lee, X. Chen, H. Furuta, J. Canny, and I. Fischer (2024) A human-inspired reading agent with gist memory of very long contexts. In The Twelfth International Conference on Learning Representations, External Links: 2402.09727, Link Cited by: §1, §2. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. Kiela (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §1. R. Li, X. Zhang, H. Yu, S. Duan, X. Li, W. Xiang, C. Liao, X. Guo, Y. Li, and J. Suo (2026a) MemPO: self-memory policy optimization for long-horizon agents. External Links: 2603.00680, Link Cited by: §2. X. Li, T. Lyu, Y. Yang, L. Shan, S. Yang, L. Zhang, Z. Huang, Q. Liu, and Y. Li (2026b) Escaping the context bottleneck: active context curation for llm agents via reinforcement learning. External Links: 2604.11462, Link Cited by: §2. Y. Li, B. Dong, F. Guerin, and C. Lin (2023) Compressing context to enhance inference efficiency of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Link Cited by: §1. N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang (2024) Lost in the middle: how language models use long contexts. Transactions of the Association for Computational Linguistics 12, p. 157–173. External Links: Document, Link Cited by: §1. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman (2022) WebGPT: browser-assisted question-answering with human feedback. External Links: 2112.09332, Link Cited by: §1, §2. OpenAI (2025) Gpt-oss-120b & gpt-oss-20b model card. External Links: 2508.10925, Link Cited by: §4.1. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe (2022) Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, External Links: 2203.02155, Link Cited by: §3.4. C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez (2024) MemGPT: towards llms as operating systems. External Links: Document, 2310.08560, Link Cited by: §1, §2. J. S. Park, J. C. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein (2023) Generative agents: interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, External Links: Document, 2304.03442, Link Cited by: §1, §2. S. Ross, G. J. Gordon, and J. A. Bagnell (2011) A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 15, p. 627–635. External Links: Link Cited by: §2. J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel (2016) High-dimensional continuous control using generalized advantage estimation. In Proceedings of the 4th International Conference on Learning Representations (ICLR), External Links: 1506.02438, Link Cited by: §3.3. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. External Links: 1707.06347, Link Cited by: §1. W. Sun, M. Lu, Z. Ling, K. Liu, X. Yao, Y. Yang, and J. Chen (2025) Scaling long-horizon llm agent via context-folding. External Links: 2510.11967, Link Cited by: §2. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §1. L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, W. X. Zhao, Z. Wei, and J. Wen (2024) A survey on large language model based autonomous agents. Frontiers of Computer Science 18 (6), p. 186345. External Links: Document, Link Cited by: §1. X. Wang, M. Li, P. Lu, X. Chang, L. Shang, J. Li, F. Mi, P. Parthasarathi, and Y. Cui (2026) InfMem: learning system-2 memory control for long-context agent. External Links: 2602.02704, Link Cited by: §2. D. Wu, H. Wang, W. Yu, Y. Zhang, K. Chang, and D. Yu (2025) LongMemEval: benchmarking chat assistants on long-term interactive memory. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1. W. Xu, Z. Liang, K. Mei, H. Gao, J. Tan, and Y. Zhang (2025) A-mem: agentic memory for llm agents. In Advances in Neural Information Processing Systems, External Links: 2502.12110, Link Cited by: §1, §2. A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu (2025) Qwen2.5 technical report. External Links: 2412.15115, Link Cited by: §4.1. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning (2018) HotpotQA: a dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Link Cited by: §4.1. S. Yao, H. Chen, J. Yang, and K. Narasimhan (2022) WebShop: towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §1, §2. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2023) ReAct: synergizing reasoning and acting in language models. In Proceedings of the International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1, §2. C. Yoon, T. Lee, H. Hwang, M. Jeong, and J. Kang (2024) CompAct: compressing retrieved documents actively for question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), External Links: Link Cited by: §1, §2. Y. Yu, L. Yao, Y. Xie, Q. Tan, J. Feng, Y. Li, and L. Wu (2026) Agentic memory: learning unified long-term and short-term memory management for large language model agents. External Links: 2601.01885, Link Cited by: §2. Y. Zheng, D. Fu, X. Hu, X. Cai, L. Ye, P. Lu, and P. Liu (2025) DeepResearcher: scaling deep research via reinforcement learning in real-world environments. External Links: 2504.03160, Link Cited by: §1. W. Zhong, L. Guo, Q. Gao, H. Ye, and Y. Wang (2023) MemoryBank: enhancing large language models with long-term memory. External Links: Document, 2305.10250, Link Cited by: §1, §2. Z. Zhou, A. Qu, Z. Wu, S. Kim, A. Prakash, D. Rus, J. Zhao, B. K. H. Low, and P. P. Liang (2025) MEM1: learning to synergize memory and reasoning for efficient long-horizon agents. External Links: Document, 2506.15841, Link Cited by: §1, §1, §2, §3.2, §4.1.