Paper deep dive
Stop When Memory Suffices: Evidence-Conditioned Progressive Execution for LLM Agents
Yidan Lin, Kaixiang Wang, Jiong Lou, Jie Li
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The continued development of LLMs toward persistent and adaptive intelligence increasingly requires long-term memory mechanisms that preserve and reuse information across interactions. Existing memory systems either compress and structure histories for efficient access or perform deep research over broader trajectories. The former lowers online cost but may omit temporal, causal, or cross-step dependencies, while the latter improves evidence coverage at substantial latency and inference cost. This raises a key question: can a memory system achieve strong answer quality while maintaining low online latency? We introduce Router-Mem, an evidence-conditioned progressive execution framework for long-horizon agent memory. Router-Mem first applies a shared low-cost retrieval prefix to obtain evidence. A lightweight sufficiency router then predicts whether the context supports early termination, which enable a single-token decision at inference time. It is trained with evidence-level supervision and rationale-conditioned representation distillation. When evidence is insufficient, Router-Mem reuses retrieval hits to expand memory blocks and perform deeper analysis and aggregation. Experiments on AMA-Bench and BEAM show that Router-Mem achieves 55.17\% and 38.77\% score while reducing average inference time by 27.3\% and 25.5\% compared with full memory execution.
Tags
Links
- Source: https://arxiv.org/abs/2608.01285v1
- Canonical: https://arxiv.org/abs/2608.01285v1
Trouble viewing inline? Open PDF directly →
Full Text
44,119 characters extracted from source content.
Expand or collapse full text
Stop When Memory Suffices: Evidence-Conditioned Progressive Execution for LLM Agents Yidan Lin, Kaixiang Wang, Jiong Lou, Jie Li Abstract The continued development of LLMs toward persistent and adaptive intelligence increasingly requires long-term memory mechanisms that preserve and reuse information across interactions. Existing memory systems either compress and structure histories for efficient access or perform deep research over broader trajectories. The former lowers online cost but may omit temporal, causal, or cross-step dependencies, while the latter improves evidence coverage at substantial latency and inference cost. This raises a key question: can a memory system achieve strong answer quality while maintaining low online latency? We introduce Router-Mem, an evidence-conditioned progressive execution framework for long-horizon agent memory. Router-Mem first applies a shared low-cost retrieval prefix to obtain evidence. A lightweight sufficiency router then predicts whether the context supports early termination, which enable a single-token decision at inference time. It is trained with evidence-level supervision and rationale-conditioned representation distillation. When evidence is insufficient, Router-Mem reuses retrieval hits to expand memory blocks and perform deeper analysis and aggregation. Experiments on AMA-Bench and BEAM show that Router-Mem achieves 55.17% and 38.77% score while reducing average inference time by 27.3% and 25.5% compared with full memory execution. Introduction Large language models are evolving from static generators into interactive systems that reason, use tools, learn from feedback, and operate over extended tasks (Yao et al. 2023; Schick et al. 2023; Shinn et al. 2023). This creates a growing need to retain and reuse information beyond a single context window (Wang et al. 2024; Zhou et al. 2024). Long-running applications accumulate conversations, tool calls, observations, errors, and intermediate decisions (Liu et al. 2024b; Yang et al. 2024). Later queries must recover relevant evidence while preserving temporal and cross-step dependencies (Li et al. 2024; Guo et al. 2024a). Although recurrent memory, sparse attention, and efficient exact attention have extended context capacity (Dai et al. 2019; Beltagy et al. 2020), full-history processing remains costly (Zaheer et al. 2020; Dao et al. 2022), and performance is still sensitive to evidence position, sequence length, and task complexity (Liu et al. 2024a; Bai et al. 2024). These limitations motivate dedicated memory mechanisms for selectively retrieving and organizing query-relevant experience (Hsieh et al. 2024; Zhang et al. 2024). Figure 1: The Existing Memory Paradigms VS Router-mem. Existing agent memory systems broadly follow two processing paradigms. Preprocessing-oriented methods extract, summarize, or restructure interaction histories into compact representations for efficient access (Zhang et al. 2025b; Hu et al. 2025)(as shown in Fig. 1a). Mem0 (Chhikara et al. 2025) consolidates salient conversational facts, A-MEM (Xu et al. 2025) organizes memories as dynamically linked notes, and G-Memory (Zhang et al. 2025a) builds hierarchical graphs for multi-agent histories. These designs reduce online retrieval cost, but early compression may omit temporal, causal, or cross-step details needed by future queries. Query-time reconstruction methods instead retain broader histories and perform iterative search and reasoning after a query arrives. GAM (Yan et al. 2025) uses a researcher agent to construct query-specific context, while E-mem (Wang et al. 2026) activates episodic segments and aggregates analyses from multiple memory agents. They improve evidence coverage, but can impose unnecessary latency when local memory already suffices, exposing a fundamental quality–efficiency trade-off (Du 2026). However, memory-query processing need not follow a fixed paradigm. Queries whose supporting evidence is already localized should not incur the cost of deep memory research, whereas queries with incomplete or dispersed evidence may not be answered reliably through compact retrieval alone. This raises a central question: can a memory system infer the required processing depth from the query and the evidence recovered by lightweight retrieval, thereby allocating computation more effectively? To address this gap, we propose Router-Mem, an evidence-conditioned progressive execution framework for long-horizon agent memory (as shown in Fig. 1b). For every query, Router-Mem first runs a shared low-cost retrieval prefix to obtain an initial evidence context. This prefix is shared by both execution paths and provides the key evidence for deciding whether the system can return early. A lightweight sufficiency router then evaluates the query together with the retrieved context and predicts whether the current evidence supports early termination. The router is trained with evidence-level supervision and rationale distillation, which transfers complex sufficiency judgments into a single-token decision at inference time. When the retrieved evidence is sufficient, the system answers directly and skips the remaining computation. Otherwise, it reuses the retrieval hits as localization signals, expands the relevant memory blocks, and invokes deeper analysis and evidence aggregation. The fast and slow paths are coupled through the shared prefix, which avoids restarting memory search from scratch. This design preserves inexpensive access for locally answerable queries while allocating costly memory reasoning to cases that still lack decisive evidence, yielding a better balance between answer quality, latency, and inference cost. Experiments on AMA (Zhao et al. 2026) and BEAM (Tavakoli et al. 2026) show that Router-Mem achieves 55.17% and 38.77% score, while reducing average inference time by 27.3% and 25.5% relative to full memory processing. Across different routing thresholds, it consistently provides favorable quality–latency trade-offs and forms a strong empirical Pareto frontier among compared methods. Our main contributions are as follows: • We formulate long agentic memory based query as an evidence-conditioned progressive execution problem. The required processing depth is determined by the query and the evidence recovered at runtime. • We propose Router-Mem, an evidence-conditioned progressive framework that addresses fixed computation in long-horizon memory access. It combines a reusable retrieval prefix, evidence-sufficiency routing, and retrieval-guided continuation to stop when evidence is sufficient and invoke broader memory analysis only when needed, balancing efficiency with deep memory reasoning quality. • We evaluate Router-Mem on AMA-Bench and BEAM across different routing thresholds and model settings. The results demonstrate strong answer quality with substantially lower inference time and establish a favorable quality–latency Pareto frontier. Related Work Retrieval-Augmented Generation. Retrieval-augmented generation (RAG) grounds LLMs in external knowledge and has become a standard paradigm for knowledge-intensive tasks (Lewis et al. 2020). Some works like Adaptive-RAG, Stop-RAG, further selects among no retrieval, single-step retrieval, and iterative retrieval according to predicted query complexity (Jeong et al. 2024; Park et al. 2025; Li et al. 2026). However, this query-level routing does not account for the evidence actually recovered at runtime: a difficult query may be resolved by successful initial retrieval, whereas a seemingly simple query may still lack decisive evidence. Recent work also improves retrieval through structured representations. HippoRAG combines knowledge graphs with Personalized PageRank for associative and multi-hop retrieval (Gutiérrez et al. 2024), while GraphRAG and LightRAG use graph-based indexing for global and hierarchical knowledge discovery (Edge et al. 2024; Guo et al. 2024b). Despite improving retrieval planning and evidence organization, these methods do not directly determine whether the currently retrieved evidence is sufficient to terminate further computation. Memory Systems for LLM Agents. Long-term agent memory systems aim to efficiently preserve and access past experiences. Preprocessing-based approaches construct compact memory representations before queries arrive. Mem0 extracts and consolidates salient conversational information (Chhikara et al. 2025), A-MEM organizes experiences as evolving structured notes (Xu et al. 2025), and G-Memory builds hierarchical graphs for multi-agent interactions (Zhang et al. 2025a). ReasoningBank further distills reusable reasoning strategies from agent trajectories (Ouyang et al. 2026). These methods reduce retrieval cost but may lose fine-grained temporal and causal dependencies. In contrast, query-time reconstruction methods retain broader histories and perform deeper reasoning when queries arrive. GAM employs researcher agents to construct query-specific contexts (Yan et al. 2025), while E-mem uses episodic retrieval and multi-agent analysis for context reconstruction (Wang et al. 2026). Although these methods improve evidence coverage, they may introduce unnecessary computation when lightweight retrieval already provides sufficient evidence. Our work connects these two regimes through a shared retrieval prefix and an evidence-sufficiency router, which terminates early when the current context supports an answer and preserves deeper processing for queries that require it. Figure 2: Training Process of Router-mem. Method In this section we systematically presents the design of Router-Mem including training process and system pipeline. More details can be found in technical supplement. Progressive Memory Execution Long-horizon agents continuously accumulate heterogeneous memories, including conversations, intermediate decisions, tool calls, environment observations, and execution feedback. Queries over such memories require different levels of processing. Some queries can be answered from a small set of locally relevant memories recovered by lightweight retrieval. Others depend on evidence that is dispersed across time, execution steps, or interaction trajectories, and therefore require broader memory expansion and aggregation. Applying only lightweight retrieval to every query may miss such dependencies, whereas executing the full memory-processing pipeline for every query introduces unnecessary latency and inference cost. We therefore organize memory access as a progressive execution process. Every query first performs a shared low-cost retrieval prefix. If the retrieved context already contains sufficient evidence, the system directly generates an answer and skips the remaining computation. Otherwise, it continues with broader memory processing. The fast and slow paths are coupled through the shared retrieval prefix, as the expensive stage continues from the retrieved evidence and reuses its localization signals for broader memory processing. Determining whether the retrieved context is sufficient is itself non-trivial. A direct solution is to prompt a general-purpose LLM to inspect the query and retrieved memories and explicitly reason about whether further processing is needed. However, a reliable judgment may require checking for missing evidence, temporal updates, action–feedback relations, and conflicting records. Although the final output is only a binary decision, the underlying reasoning can be complex. Performing such deliberation online often requires a capable model, a long context prefill, and additional reasoning tokens. This cost is undesirable because the router is introduced precisely to avoid unnecessary computation. An expensive routing decision may substantially reduce the efficiency gain obtained from early termination. To address this issue, we train a lightweight evidence-sufficiency router that directly predicts whether the current retrieval evidence state supports early termination. Given a query q and an ordered memory M=(z1,…,zT)M=(z_1,…,z_T), the shared retriever first returns Cf=Rf(q,M),C_f=R_f(q,M), (1) where CfC_f is the initially retrieved memory context. We define the context-dependent termination label as y(q,Cf)=[Cf provides sufficient evidence to answer q],y(q,C_f)=1 [C_f provides sufficient evidence to answer q ], (2) and train the router to estimate sθ(q,Cf)=Pθ(y=1∣q,Cf).s_θ(q,C_f)=P_θ (y=1 q,C_f ). (3) Importantly, the decision is conditioned on the execution state (q,Cf)(q,C_f) rather than on the query alone. The same query may terminate after one retrieval result but require further processing after another result that omits a decisive memory step. During training, we use evidence-level supervision and rationale distillation to transfer the required evidence judgment into the lightweight router. At inference time, the router reads only (q,Cf)(q,C_f) and produces a single-token termination decision. Evidence-Sufficiency Data Construction Training the router requires supervision that distinguishes evidence sufficiency from semantic relevance (as shown in fig. 2). We collect long-horizon agent trajectories from diverse interaction settings, including tool use, software engineering, web navigation, games, and long conversations. These training sources are disjoint from the downstream evaluation benchmarks, while preserving similar interaction patterns such as multi-step actions, environment feedback, and temporally dispersed evidence. For each trajectory, we construct and verify a query–answer–evidence tuple (qi,ai,Ei)(q_i,a_i,E_i), where EiE_i contains the memory steps that support the reference answer aia_i. We then apply evidence intervention to produce contexts with different sufficiency labels. A positive context retains all required evidence and mixes it with distractor steps: Ci+=Ei∪Di,C_i^+=E_i∪ D_i, (4) where DiD_i contains randomly sampled memory steps from the same trajectory. Since the complete supporting evidence is preserved, Ci+C_i^+ is labeled as sufficient. A negative context removes part or all of the required evidence and fills the remaining context with semantically related hard distractors: Ci−=(Ei∖Eidrop)∪Dihard.C_i^-= (E_i E_i^drop )∪ D_i^hard. (5) The hard distractors may share entities, tools, or nearby events with the query, but do not preserve the complete evidence chain. These examples teach the router that relevance alone does not imply answerability. Evidence removal does not always make a context insufficient, because redundant or alternative support may remain. We therefore evaluate each candidate negative with multiple answer models and discard it if any model still produces a correct answer. For every retained sample, a teacher model generates a rationale that explains which evidence is present and which decisive information is missing. The resulting dataset consists of tuples (qi,Ci,yi,ri)(q_i,C_i,y_i,r_i), where yi∈yes,noy_i∈\yes,no\ is the sufficiency label and rir_i is the teacher rationale. We split the data by trajectory to prevent related questions, neighboring steps, and context variants from leaking across training and validation sets. Training Process Given a constructed sample (qi,Ci,yi,ri)(q_i,C_i,y_i,r_i), we form the student input as xi=Prompt(qi,Ci).x_i=Prompt(q_i,C_i). (6) The router retains the causal language-model interface and predicts the sufficiency label through its next-token distribution. Let VyesV_yes and VnoV_no denote token groups corresponding to the normalized outputs “yes” and “no.” We optimize the grouped-token classification loss ℒclsi=−log∑v∈VyiPθ(v∣xi).L_cls^i=- _v∈ V_y_iP_θ(v x_i). (7) The binary label specifies the desired decision but provides limited supervision about the underlying evidence judgment. We therefore use rationale-conditioned representation distillation. A frozen teacher additionally reads the rationale rir_i, while the student observes only (qi,Ci)(q_i,C_i). Let hsih_s^i denote the student representation before the decision token, and let htih_t^i denote the teacher representation after processing the rationale. We align them at a selected intermediate layer using ℒrepi=1−(hsi)⊤hti∥hsi∥2∥hti∥2.L_rep^i=1- (h_s^i) h_t^i h_s^i _2 h_t^i _2. (8) The final training objective is ℒrouter=ℒcls+λreps0ℒrep,L_router=L_cls+ _reps_0L_rep, (9) where λrep _rep controls the contribution of representation distillation and s0s_0 calibrates the scales of the two losses. The teacher and rationale are used only during training. At inference time, the student reads (q,Cf)(q,C_f) and produces a single decision token without generating an explanation. Figure 3: System Pipeline of Router-mem. Table 1: Results on AMA-Bench with two backbone LLMs: DeepSeek-V4-Flash and Qwen3.5-35B-A3B. DeepSeek-V4-Flash Qwen3.5-35B-A3B Method Score (%) Early (%) TmemT_ mem (s) Te2eT_ e2e (s) TbuildT_ build (s) Score (%) Early (%) TmemT_ mem (s) Te2eT_ e2e (s) TbuildT_ build (s) External Memory Baselines RAG 40.46 – 0.057 2.49 10.15 45.21 – 0.059 4.72 9.88 LightMem 35.34 – 0.058 2.30 1157.78 36.58 – 0.055 2.49 6531.13 Mem0 35.90 – 0.159 3.50 46.12 43.96 – 0.156 2.61 51.37 Long Context 47.08 – 3.595 5.18 0.001 53.95 – 20.520 22.80 0.002 GAM 40.21 – 22.450 25.289 123.57 43.96 – 24.497 28.40 141.52 Fixed Execution Endpoints Fast Prefix Only 44.91 100.0 0.054 3.70 5.07 48.09 100.0 0.054 3.63 4.56 Full Completion (Slow) 56.49 0.0 11.341 14.47 5.07 56.41 0.0 15.816 20.28 4.66 Router-Mem: Evidence-Conditioned Early Termination Router-Mem (τ=0.1τ=0.1) 51.48 53.7 5.608 9.21 4.93 52.50 54.6 8.414 12.80 4.57 Router-Mem (τ=0.3τ=0.3) 52.68 38.5 7.215 10.53 4.90 53.65 39.2 9.351 13.10 4.65 Router-Mem (τ=0.5τ=0.5) 55.17 30.0 8.248 11.57 5.08 55.06 30.2 10.581 14.49 4.60 Router-Mem (τ=0.7τ=0.7) 54.73 22.1 9.045 12.26 4.98 55.52 22.3 13.888 18.67 4.71 Router-Mem (τ=0.9τ=0.9) 55.89 12.3 10.273 13.51 4.94 55.52 12.5 14.423 18.96 4.61 Shared retriever training. We train the shared retriever with the same QA–evidence supervision. Evidence-complete memory views are treated as positives, while semantically related but incomplete views are used as negatives. Using an L2-normalized encoder e(⋅)e(·), we optimize a multi-positive contrastive objective: ℒreti=−1|Pi|∑p∈Pilogexp(e(qi)⊤e(p)/T)∑c∈iexp(e(qi)⊤e(c)/T).L_ret^i=- 1|P_i| _p∈ P_i (e(q_i) e(p)/T ) _c _i (e(q_i) e(c)/T ). (10) This objective encourages the shared prefix to retrieve answer-supporting evidence. Algorithm 1 Progressive Memory Execution 1:Query q, memory M, threshold τ 2:Answer a a 3:(Cf,Hf)←Rf(q,M)(C_f,H_f)← R_f(q,M) ⊳ Shared retrieval prefix 4:if CfC_f fits the router input budget then 5: s←sθ(q,Cf)s← s_θ(q,C_f) ⊳ Evidence-sufficiency score 6: if s≥τs≥τ then 7: a^←A(q,Order(Cf)) a← A(q,Order(C_f)) 8: return a a ⊳ Optimistic early return 9: end if 10:end if 11:ℬf←MapToBlocks(Hf)B_f (H_f) 12:ℬs←SelectBlocks(ℬf,M)B_s (B_f,M) 13:for all b∈ℬsb _s in parallel do 14: ub←MemAgent(q,b)u_b (q,b) 15:end for 16:Cs←Aggregate(ub:b∈ℬs)C_s (\u_b:b _s\ ) 17:a^←A(q,Cs) a← A(q,C_s) 18:return a a Progressive Memory System After training the retriever and termination router, we integrate them into a progressive memory execution system. The system first constructs aligned retrieval and processing views of the same memory (as shown in fig. 3). It then performs a shared retrieval step for every query and invokes broader memory processing only when the retrieved evidence is predicted to be insufficient. Aligned memory views. Given an ordered memory M=(z1,…,zT)M=(z_1,…,z_T), we construct two aligned views. The retrieval view indexes individual memory steps and stores their corresponding step and block identifiers. The processing view groups temporally adjacent steps into overlapping memory blocks. This alignment allows each retrieval hit to directly locate the memory blocks used by subsequent processing. The two views therefore support different stages of one execution process, rather than two independent memory systems. Shared retrieval and termination. For a query q, the shared retriever first returns an initial context CfC_f and its retrieval hits HfH_f: Cf,Hf=Rf(q,M).C_f,H_f=R_f(q,M). (11) The router evaluates the sufficiency of CfC_f through its next-token distribution. We aggregate the probability assigned to the “yes” and “no” token groups: pyes=∑v∈VyesPθ(v∣q,Cf),pno=∑v∈VnoPθ(v∣q,Cf),p_yes= _v∈ V_yesP_θ(v q,C_f), p_no= _v∈ V_noP_θ(v q,C_f), (12) and compute the normalized termination score s(q,Cf)=pyespyes+pno.s(q,C_f)= p_yesp_yes+p_no. (13) Given a threshold τ, the system follows πτ(q,Cf)=terminate,s(q,Cf)≥τ,continue,s(q,Cf)<τ. _τ(q,C_f)= casesterminate,&s(q,C_f)≥τ,\\ continue,&s(q,C_f)<τ. cases (14) The threshold is selected on development data and fixed during evaluation. Varying τ yields different empirical trade-offs between answer quality and execution cost. If the retrieved context exceeds the router input budget, the system conservatively selects continuation. Early return and execution continuation. If the termination condition is satisfied, the retrieved steps are restored to their original temporal order and passed to the answer model: a^f=A(q,Cf). a_f=A(q,C_f). (15) This branch is an early return from the full memory execution, rather than the output of an independently selected fast system. Otherwise, the system reuses the retrieval hits to continue the same execution process. Let ℬfB_f denote the blocks associated with HfH_f. We expand these anchors into a bounded set of processing blocks: ℬs=SelectBlocks(ℬf,M).B_s=SelectBlocks(B_f,M). (16) The selected blocks include the retrieved anchor regions and their temporal neighborhoods, which helps recover local action–feedback and cross-step dependencies. Each block is analyzed independently by a memory agent: ub=MemAgent(q,b),b∈ℬs.u_b=MemAgent(q,b), b _s. (17) An aggregator combines the block-level results into a consolidated context: Cs=Aggregate(ub:b∈ℬs),a^s=A(q,Cs).C_s=Aggregate (\u_b:b _s\ ), a_s=A(q,C_s). (18) The continuation stage therefore preserves and extends the computation performed by the shared retrieval prefix. It is invoked only when the current evidence does not support early termination. Table 2: End-to-end results on BEAM with Model DeepSeek-V4-Flash. Method Score (%) Early Term. (%) TmemT_mem (s) Te2eT_e2e (s) TbuildT_build (s) External Memory Baselines RAG 30.29 – 0.094 1.66 217.01 LightMem 27.12 – 0.058 1.55 1174.31 Mem0 22.09 – 0.151 1.75 982.20 Long Context 38.01 – 14.304 17.08 0.05 GAM 35.23 – 25.817 28.435 1334.78 Fixed Execution Endpoints Fast Prefix Only 30.02 100.0 0.054 2.51 31.81 Full Completion (Slow) 43.26 0.0 14.611 18.43 31.90 Router-Mem: Evidence-Conditioned Early Termination Router-Mem (τ=0.1τ=0.1) 33.94 56.9 5.675 8.51 32.21 Router-Mem (τ=0.3τ=0.3) 36.53 39.4 8.781 11.72 31.84 Router-Mem (τ=0.5τ=0.5) 38.77 26.6 10.883 14.11 31.88 Router-Mem (τ=0.7τ=0.7) 39.81 16.7 12.512 16.18 32.01 Router-Mem (τ=0.9τ=0.9) 42.09 8.0 13.422 16.93 31.86 Experiment We evaluate Router-Mem against state-of-the-art memory systems for LLM agents, focusing on effectiveness, routing efficiency, and robustness. Our experiments investigate three key research questions: RQ1: Comparative Effectiveness. Does Router-Mem provide advantages over existing memory mechanisms in terms of answer quality, inference latency, and overall efficiency? RQ2: Quality–Latency Trade-off. Can Router-Mem achieve a favorable balance between memory answer performance and inference latency, and how does the routing threshold control this trade-off? RQ3: Training Effectiveness. Do the proposed router training and embedding training improve retrieval quality, routing decisions, and end-to-end system performance? Experimental Setup. Dataset. We evaluate Router-Mem on two representative long-term memory benchmarks for LLM agents: AMA-Bench (Zhao et al. 2026) and BEAM (Tavakoli et al. 2026). AMA-Bench is designed to evaluate memory capabilities in long-horizon agentic scenarios, covering diverse interactions where agents must retrieve and reason over previously observed experiences. It emphasizes accurate memory access under complex multi-step dependencies. BEAM focuses on extremely long-context memory understanding and evaluates whether agents can effectively recover relevant information from large-scale conversational histories. Together, these benchmarks cover complementary challenges: AMA-Bench tests structured agent memory reasoning, while BEAM stresses scalability under massive context lengths. They provide a comprehensive evaluation of both memory effectiveness and efficient execution. Settings. We compare Router-Mem with representative memory baselines, including standard RAG (Lewis et al. 2020), compact memory systems such as LightMem (Fang et al. 2026) and Mem0 (Chhikara et al. 2025), Long Context, and deep query-time memory reconstruction methods GAM (Yan et al. 2025). For evaluation, we use DeepSeek-V4-Flash as the backbone model and report answer quality measured by the official LLM judge provided by each benchmark. We evaluate both efficiency and effectiveness by reporting memory construction time, retrieval latency, and end-to-end inference latency. Further experiments analyze the impact of routing thresholds τ, router training components, and embedding model choices to evaluate the robustness and scalability of Router-Mem. We also evaluate the E2E token cost of Router-mem against existing agentic memory baselines in technical supplement. Figure 4: Quality–latency under different routing thresholds. Main Results. We compare Router-Mem with representative memory mechanisms from three categories. RAG serves as a lightweight retrieval baseline that directly retrieves relevant contexts without persistent memory modeling. LightMem and Mem0 represent compact memory systems that extract and organize historical information into efficient memory representations. Long Context provides a full-context baseline that directly exposes the entire history to the language model. We further compare against two execution endpoints: Fast Prefix Only, which always terminates after the shared retrieval stage, and Full Completion (Slow), which always performs complete memory reconstruction. These endpoints characterize the lower-cost and upper-performance boundaries of progressive memory execution. Results on AMA-Bench. Table 1 shows that Router-Mem consistently achieves a favorable quality–latency trade-off between the fast and slow endpoints. The Fast Prefix Only baseline achieves the lowest memory latency (0.0540.054s) but suffers from incomplete evidence coverage, obtaining 44.91% score. In contrast, Full Completion reaches the highest score of 56.49% with substantially larger latency. By adaptively routing queries, Router-Mem approaches the performance of full completion while avoiding unnecessary deep memory processing. For example, Router-Mem (τ=0.5τ=0.5) achieves 55.17% score with only 8.25s memory latency, reducing the cost by 27.3% compared with full completion. Different thresholds provide controllable operating points, demonstrating that the learned router effectively balances answer quality and execution efficiency. Results on BEAM. Results on BEAM further demonstrate the scalability of Router-Mem under extremely long conversational histories. The Fast Prefix Only baseline obtains only 30.02% score, while Full Completion improves performance to 43.26% at the cost of 14.61s memory latency. Router-Mem bridges this gap by allocating deeper reasoning only when necessary. With τ=0.7τ=0.7, it achieves 39.81% score while reducing memory latency to 12.51s, and with τ=0.9τ=0.9, it approaches the full completion performance (42.09%) with lower inference cost. Across thresholds, Router-Mem forms a smooth quality–cost frontier, validating that evidence-conditioned routing enables adaptive computation allocation for long-term agent memory. Table 3: Router training comparison on AMA-Bench and BEAM. ET denotes the early-termination rate. AMA-Bench BEAM τ Router Score ET Te2eT_e2e Score ET Te2eT_e2e (%) (%) (s) (%) (%) (s) 0.3 Base 55.63 8.0 13.68 29.71 90.5 4.15 Trained 52.68 38.5 10.53 36.53 39.4 11.72 0.5 Base 55.52 4.9 14.21 29.62 86.5 5.15 Trained 55.17 30.0 11.57 38.77 26.6 14.11 0.7 Base 55.79 2.2 14.39 31.78 80.5 5.67 Trained 54.73 22.1 12.26 39.81 16.7 16.18 Table 4: Effect of embedding training on AMA-Bench. Setting Embedding Score (%) Fast hit (%) TmemT_mem (s) Te2eT_e2e (s) Fast-only Base 42.00 100.0 0.055 3.41 Fast-only Trained 44.91 100.0 0.054 3.70 Router (τ=0.5τ=0.5) Base 54.79 21.8 9.17 12.16 Router (τ=0.5τ=0.5) Trained 55.17 30.0 8.25 11.57 Slow-only Base 55.73 0.0 11.29 14.32 Slow-only Trained 56.49 0.0 11.34 14.47 Ablation Study Memory Performance and Latency Trade-off. Figure 4 illustrates how the routing threshold controls the quality–latency trade-off on AMA-Bench. Increasing τ makes the router more conservative and sends more queries to the deeper memory-processing path. This generally improves answer quality at the cost of additional online memory time. Among the resulting operating points, τ=0.5τ=0.5 provides a particularly favorable Pareto-efficient trade-off. At a comparable online memory time, this operating point also achieves a substantially higher AMA-Bench score than the large model API Router baseline. It achieves a score of 55.17%, only 1.32 points below the Full Completion endpoint (56.49%), while reducing TmemT_mem from 11.341s to 8.248s and Te2eT_e2e from 14.47s to 11.57s. These correspond to 27.3% and 20.0% latency reductions, respectively. Thus, τ=0.5τ=0.5 retains most of the accuracy benefit of full memory reconstruction while avoiding a substantial fraction of its computational overhead. Effect of Router Training. As shown in Table 3, router training primarily calibrates the fast–slow decision boundary rather than uniformly favoring one execution path. On AMA-Bench, the base router is strongly biased toward the slow path, with only 4.9% early termination at τ=0.5τ=0.5. Training raises this rate to 30.0% and reduces Te2eT_e2e from 14.21s to 11.57s, while preserving a comparable score (55.52% vs. 55.17%). In contrast, the base router on BEAM is over-confident in the fast path, terminating 86.5% of queries early and achieving only 29.62%. Training corrects this bias, reducing early termination to 26.6% and improving the score to 38.77%. Moreover, the trained router responds consistently to changes in τ, enabling flexible control over computation. Without training, the decision boundary collapses toward opposite fixed endpoints across the two benchmarks, making threshold adjustment largely ineffective. Effect of Embedding Training. We study the effect of training the embedding model on AMA-Bench. As shown in Table 4, the trained embedding improves the Fast-only score from 42.00 to 44.91 without changing the memory latency. Under the practical Router-Mem setting at τ=0.5τ=0.5, it increases the score from 54.79 to 55.17 and the fast-hit rate from 21.8% to 30.0%, while reducing TmemT_mem from 9.17s to 8.25s and Te2eT_e2e from 12.16s to 11.57s. The Slow-only endpoint also gains 0.76 points with nearly unchanged runtime. These results indicate that embedding training primarily improves retrieved evidence quality and routing efficiency rather than increasing computation. Table 5: Effect of jointly training the router and retriever on AMA-Bench. τ Training Score (%) Early Term. (%) TmemT_mem (s) Te2eT_e2e (s) 0.3 None 54.48 9.4 10.352 13.453 Joint 52.68 38.5 7.215 10.534 0.5 None 53.96 6.1 10.842 13.987 Joint 55.17 30.0 8.248 11.570 0.7 None 56.25 3.5 11.073 14.219 Joint 54.73 22.1 9.045 12.260 Effect of Joint Training. We also evaluate the joint effect of training both the router and retriever. As shown in Table 5, without training, the system behaves similarly to the Slow-only endpoint: early termination remains between 3.5% and 9.4%, and varying τ provides limited control over computation. Joint training increases this range to 22.1%–38.5% and consistently reduces both memory and end-to-end latency while preserving comparable answer quality. At τ=0.5τ=0.5, it improves the score from 53.96% to 55.17%, raises early termination from 6.1% to 30.0%, and reduces TmemT_mem from 10.842s to 8.248s. This broader operating range indicates that the learned components jointly calibrate retrieval quality and termination confidence across thresholds. These results show that joint training more reliably identifies safely answerable queries and makes τ an effective control knob for the quality–latency trade-off. Conclusion In this paper, we propose Router-Mem to address the challenge of efficiently accessing long-horizon memories in LLM agents, where existing methods often trade off between low-cost retrieval and expensive deep memory reasoning. Router-Mem is an evidence-conditioned progressive memory execution framework that learns when the current retrieved context is sufficient to terminate further computation. It first performs a shared retrieval prefix and employs a lightweight sufficiency router to make an early-termination decision. In particular, the shared prefix couples the fast and slow paths, allowing the system to preserve retrieved evidence and avoid restarting memory search when deeper processing is required. When evidence is insufficient, it reuses retrieval anchors to guide memory expansion and deeper evidence aggregation. Experiments on AMA-Bench and BEAM demonstrate that Router-Mem achieves strong answer quality while substantially reducing inference cost. We believe our approach provides a practical direction toward adaptive memory systems that allocate reasoning resources according to the actual evidence requirements of each query. References Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, and J. Li (2024) LongBench: a bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, p. 3119–3137. Cited by: Introduction. I. Beltagy, M. E. Peters, and A. Cohan (2020) Longformer: the long-document transformer. arXiv preprint arXiv:2004.05150. Cited by: Introduction. P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav (2025) Mem0: building production-ready AI agents with scalable long-term memory. arXiv preprint arXiv:2504.19413. Cited by: Introduction, Memory Systems for LLM Agents., Settings.. Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. Le, and R. Salakhutdinov (2019) Transformer-XL: attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 2978–2988. Cited by: Introduction. T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré (2022) FlashAttention: fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, Cited by: Introduction. P. Du (2026) Memory for autonomous llm agents: mechanisms, evaluation, and emerging frontiers. arXiv preprint arXiv:2603.07670. Cited by: Introduction. D. Edge, H. Trinh, N. Cheng, J. Bradley, A. Chao, A. Mody, S. Truitt, D. Metropolitansky, R. O. Ness, and J. Larson (2024) From local to global: a graph RAG approach to query-focused summarization. arXiv preprint arXiv:2404.16130. Cited by: Retrieval-Augmented Generation.. J. Fang, X. Deng, H. Xu, Z. Jiang, Y. Tang, Z. Xu, S. Deng, Y. Yao, M. Wang, S. Qiao, H. Chen, and N. Zhang (2026) LightMem: lightweight and efficient memory-augmented generation. External Links: 2510.18866, Link Cited by: Settings.. T. Guo, X. Chen, Y. Wang, R. Chang, S. Pei, N. V. Chawla, O. Wiest, and X. Zhang (2024a) Large language model based multi-agents: a survey of progress and challenges. arXiv preprint arXiv:2402.01680. Cited by: Introduction. Z. Guo, L. Xia, Y. Yu, T. Ao, and C. Huang (2024b) LightRAG: simple and fast retrieval-augmented generation. arXiv preprint arXiv:2410.05779. Cited by: Retrieval-Augmented Generation.. B. J. Gutiérrez, Y. Shu, Y. Gu, M. Yasunaga, and Y. Su (2024) HippoRAG: neurobiologically inspired long-term memory for large language models. In Advances in Neural Information Processing Systems, Cited by: Retrieval-Augmented Generation.. C. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg (2024) RULER: what’s the real context size of your long-context language models?. In Conference on Language Modeling, Cited by: Introduction. Y. Hu, S. Liu, Y. Yue, G. Zhang, B. Liu, F. Zhu, J. Lin, H. Guo, S. Dou, Z. Xi, et al. (2025) Memory in the age of ai agents. arXiv preprint arXiv:2512.13564. Cited by: Introduction. S. Jeong, J. Baek, S. Cho, S. J. Hwang, and J. Park (2024) Adaptive-RAG: learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 7036–7050. Cited by: Retrieval-Augmented Generation.. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. Kiela (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, Cited by: Retrieval-Augmented Generation., Settings.. X. Li, S. Wang, S. Zeng, Y. Wu, and Y. Yang (2024) A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth 1 (1), p. 9. Cited by: Introduction. Y. Li, Z. Yan, and T. Käfer (2026) RASER: recoverability-aware selective escalation router for multi-hop question answering. arXiv preprint arXiv:2606.02488. Cited by: Retrieval-Augmented Generation.. N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang (2024a) Lost in the middle: how language models use long contexts. Transactions of the Association for Computational Linguistics 12, p. 157–173. Cited by: Introduction. X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, and J. Tang (2024b) AgentBench: evaluating LLMs as agents. In International Conference on Learning Representations, Cited by: Introduction. S. Ouyang, J. Yan, I. Hsu, Y. Chen, K. Jiang, Z. Wang, R. Han, L. T. Le, S. Daruki, X. Tang, V. Tirumalashetty, G. Lee, M. Rofouei, H. Lin, J. Han, C. Lee, and T. Pfister (2026) ReasoningBank: scaling agent self-evolving with reasoning memory. In International Conference on Learning Representations, Cited by: Memory Systems for LLM Agents.. J. Park, S. Cho, and J. Lee (2025) Stop-RAG: value-based retrieval control for iterative RAG. arXiv preprint arXiv:2510.14337. Cited by: Retrieval-Augmented Generation.. T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom (2023) Toolformer: language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, Cited by: Introduction. N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, Cited by: Introduction. M. Tavakoli, A. Salemi, C. Ye, M. Abdalla, H. Zamani, and J. R. Mitchell (2026) Beyond a million tokens: benchmarking and enhancing long-term memory in LLMs. In International Conference on Learning Representations, Cited by: Introduction, Dataset.. G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar (2024) Voyager: an open-ended embodied agent with large language models. Transactions on Machine Learning Research. Cited by: Introduction. K. Wang, Y. Lin, J. Lou, Z. Zhou, B. Suvonov, and J. Li (2026) E-mem: multi-agent based episodic context reconstruction for llm agent memory. External Links: 2601.21714, Link Cited by: Introduction, Memory Systems for LLM Agents.. W. Xu, Z. Liang, K. Mei, H. Gao, J. Tan, and Y. Zhang (2025) A-MEM: agentic memory for LLM agents. In Advances in Neural Information Processing Systems, Cited by: Introduction, Memory Systems for LLM Agents.. B. Y. Yan, C. Li, H. Qian, S. Lu, and Z. Liu (2025) General agentic memory via deep research. arXiv preprint arXiv:2511.18423. Cited by: Introduction, Memory Systems for LLM Agents., Settings.. J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press (2024) SWE-agent: agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, Cited by: Introduction. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2023) ReAct: synergizing reasoning and acting in language models. In International Conference on Learning Representations, Cited by: Introduction. M. Zaheer, G. Guruganesh, A. Dubey, J. Ainslie, C. Alberti, S. Ontañón, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed (2020) Big bird: transformers for longer sequences. In Advances in Neural Information Processing Systems, Cited by: Introduction. G. Zhang, M. Fu, G. Wan, M. Yu, K. Wang, and S. Yan (2025a) G-Memory: tracing hierarchical memory for multi-agent systems. arXiv preprint arXiv:2506.07398. Cited by: Introduction, Memory Systems for LLM Agents.. X. Zhang, Y. Chen, S. Hu, Z. Xu, J. Chen, M. K. Hao, X. Han, Z. L. Thai, S. Wang, Z. Liu, and M. Sun (2024) ∞bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, Cited by: Introduction. Z. Zhang, Q. Dai, X. Bo, C. Ma, R. Li, X. Chen, J. Zhu, Z. Dong, and J. Wen (2025b) A survey on the memory mechanism of large language model-based agents. ACM Transactions on Information Systems 43 (6), p. 1–47. Cited by: Introduction. Y. Zhao, B. Yuan, J. Huang, H. Yuan, Z. Yu, H. Xu, L. Hu, A. Shankarampeta, Z. Huang, W. Ni, Y. Tian, and J. Zhao (2026) AMA-Bench: evaluating long-horizon memory for agentic applications. In Proceedings of the 43rd International Conference on Machine Learning, Cited by: Introduction, Dataset.. S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig (2024) WebArena: a realistic web environment for building autonomous agents. In International Conference on Learning Representations, Cited by: Introduction.