Paper deep dive
D$^2$ACCI: A Dual-Loop Diagnostic Protocol for Evidence-Preserving Agent Memory
Xule Liu, Yijun Liu, Chao Li, Shao Kun
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 8/19/2026, 5:26:47 AM
Summary
The paper introduces D2ACCI, a dual-loop diagnostic protocol for evaluating and iterating on LLM agent memory systems. It addresses the difficulty of localizing failures in multi-stage memory pipelines by using paired statistical comparisons, protected-slice monitoring, and trace-level localizability. The protocol is instantiated in MemStack and evaluated on benchmarks like LoCoMo, LongMemEval, and PersonaMem-V2, demonstrating that trace-enriched diagnostics significantly improve root-cause agreement and enable robust, regression-aware memory system evolution.
Entities (14)
Relation Signals (11)
D2ACCI → evaluatedon → LongMemEval
confidence 95% · ...evaluate on three public benchmarks, achieving 90.93% on LongMemEval...
D2ACCI → evaluatedon → PersonaMem-v2
confidence 95% · ...evaluate on three public benchmarks, achieving 57.20% on PersonaMem-V2.
D2ACCI → evaluatedon → LoCoMo
confidence 95% · ...evaluate on three public benchmarks, achieving 93.59% on LoCoMo...
D2ACCI → institutedin → MemStack
confidence 95% · We instantiate the protocol in MemStack and evaluate on three public benchmarks...
DCR → measureslocalizabilityin → D2ACCI
confidence 92% · We further introduce DCR, a graded observability metric that measures whether failures remain localizable...
BM25/RRF → ismonitoredfeatureflagin → D2ACCI
confidence 90% · In contrast, BM25/RRF is retained as a monitored feature flag...
Session-Memory Retrieval → yieldssignificantgainin → D2ACCI
confidence 90% · Five paired ablations show that... session-memory retrieval... yield statistically significant gains...
Supplement Extraction → yieldssignificantgainin → D2ACCI
confidence 90% · Five paired ablations show that supplement extraction... yield statistically significant gains...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Memory is a key capability of LLM agents. Persistent memory extends this across sessions---enabling recall, revision, and personalization. Yet its multi-stage pipeline (ingestion, retrieval, filtering, generation) makes failures difficult to localize: end-to-end evaluation reveals that an error occurred, but not which stage caused it. Existing evaluations often report aggregate performance without paired statistical comparisons, slice-level non-regression checks, or stage-level diagnostic traces. We propose D$^2$ACCI (Diagnostic-Driven Artifact-based Closed-loop Controlled Iteration), a dual-loop protocol whose outer diagnostic gate promotes, feature-flags, or rejects memory interventions based on paired evidence, protected-slice monitoring, and trace-level localizability. We further introduce DCR, a graded observability metric that measures whether failures remain localizable, and D$^2$ACCI-Eval, a reusable artifact for gate replay. We instantiate the protocol in MemStack and evaluate on three public benchmarks, achieving 93.59% on LoCoMo, 90.93% on LongMemEval, and 57.20% on PersonaMem-V2. Five paired ablations show that supplement extraction, session-memory retrieval, and Forget Guard yield statistically significant gains (+1.9 to +3.7pp, all p $\le$ .003). In contrast, BM25/RRF is retained as a monitored feature flag---a distinction invisible to aggregate-only evaluation. A diagnostic audit shows enriched traces substantially improve root-cause agreement over result-only relabeling. Diagnostic artifacts reach 98--100% DCR@3 versus 0% for results-only logs. These results establish that robust memory-system iteration demands traceable, statistically grounded, and regression-aware evidence---exactly the gap D$^2$ACCI fills.
Tags
Links
- Source: https://arxiv.org/abs/2608.17756v1
- Canonical: https://arxiv.org/abs/2608.17756v1
Trouble viewing inline? Open PDF directly →
Full Text
47,383 characters extracted from source content.
Expand or collapse full text
D2ACCI: A Dual-Loop Diagnostic Protocol for Evidence-Preserving Agent Memory Xule Liu Yijun Liu Chao Li Shao Kun Thanks: Corresponding author. Abstract Memory is a key capability of LLM agents. Persistent memory extends this across sessions—enabling recall, revision, and personalization. Yet its multi-stage pipeline (ingestion, retrieval, filtering, generation) makes failures difficult to localize: end-to-end evaluation reveals that an error occurred, but not which stage caused it. Existing evaluations often report aggregate performance without paired statistical comparisons, slice-level non-regression checks, or stage-level diagnostic traces. We propose D2ACCI (Diagnostic-Driven Artifact-based Closed-loop Controlled Iteration), a dual-loop protocol whose outer diagnostic gate promotes, feature-flags, or rejects memory interventions based on paired evidence, protected-slice monitoring, and trace-level localizability. We further introduce DCR, a graded observability metric that measures whether failures remain localizable, and D2ACCI-Eval, a reusable artifact for gate replay. We instantiate the protocol in MemStack and evaluate on three public benchmarks, achieving 93.59% on LoCoMo, 90.93% on LongMemEval, and 57.20% on PersonaMem-V2. Five paired ablations show that supplement extraction, session-memory retrieval, and Forget Guard yield statistically significant gains (+1.9 to +3.7p, all p≤.003p≤.003). In contrast, BM25/RRF is retained as a monitored feature flag—a distinction invisible to aggregate-only evaluation. A diagnostic audit shows enriched traces substantially improve root-cause agreement over result-only relabeling. Diagnostic artifacts reach 98–100% DCR@3 versus 0% for results-only logs. These results establish that robust memory-system iteration demands traceable, statistically grounded, and regression-aware evidence—exactly the gap D2ACCI fills. Introduction LLM agents are increasingly expected to operate over long time horizons, where the information needed for a later interaction may no longer fit within a limited context window. Persistent memory addresses this limitation by storing, organizing, and reusing information across interactions, and is becoming a first-class component of LLM agents (14; 3; 16). A deployed assistant must recall facts across months of interaction, distinguish stable preferences from transient context, respect revised information, and avoid using memories that users ask to forget (12; 24; 8). These requirements are difficult to satisfy reliably because persistent-memory systems involve multiple processing stages. When a memory-augmented answer is incorrect, the reason may lie in failed extraction, incorrect consolidation with another memory, irrelevant retrieval, over-aggressive filtering, or omission of a critical update from the final prompt. End-to-end evaluation reveals that an error occurred, but not which stage caused it or what improvements should be applied. Figure 1: (a) Current end-to-end evaluation reports only aggregate accuracy; internal pipeline states are hidden and failures cannot be attributed to a specific stage. (b) D2ACCI emits per-stage traces, enabling paired gates to localize failures and drive targeted repairs. We therefore argue that agent memory should be evaluated as an evolvable runtime system. A long-lived agent is repeatedly updated as developers revise retrieval policies, consolidation rules, deduplication thresholds, and memory management logics (6). Without explicit regression controls, these updates can silently degrade the user experience or performance of task slices, such as temporal inference or preference fidelity. This risk is especially pronounced for personalized memory because improvements are often non-monotonic: broader retrieval may recover temporally relevant evidence while introducing conflicting preferences, whereas stronger deduplication may reduce redundancy while deleting rare but crucial facts. For agent builders, this makes memory iteration a deployment problem rather than only a benchmark problem. Ideal iteration requires more than aggregate component ablations. Prior evaluations of agent memory rarely combine paired comparisons, protected slice non-regression checks, null result preservation, and stage-level diagnostics (3; 16; 17). Thus, an aggregate improvement may obscure trade offs or regressions under specific benchmarks. Reliable memory system evolution requires a reproducible procedure that links each intervention to a testable failure hypothesis, statistically supported paired evidence, protected slice monitoring, and traces capable of preserving stage-level failure localization. To address this gap, we introduce D2ACCI (Diagnostic-Driven Artifact-based Closed-loop Controlled Iteration), a dual-loop protocol that separates an inner runtime loop from an outer diagnostic-evolution loop. The inner loop executes the memory-augmented agent, whereas the outer loop evaluates candidate changes and determines whether they should be promoted, feature flagged, or rejected. D2ACCI defines a decision contract that ties these decisions to paired statistical evidence, protected slice non-regression checks, and trace-level localizability. Additionally, we introduce DCR, a graded trace-coverage metric for measuring whether failures remain localizable at the relevant stage, and D2ACCI-Eval, a reusable evaluation artifact supporting paired statistical analysis, protected-slice gates, discordance export, and deterministic gate replay. The artifact preserves both positive and null findings, allowing later iterations to reuse prior evidence. We instantiate D2ACCI in MemStack, a diagnosable memory kernel with multi-granularity storage and feature-flagged retrieval. We evaluate on three public benchmarks, achieving 93.59% on LoCoMo, 90.93% on LongMemEval, and 57.20% on PersonaMem-V2. Across five paired feature-gated configurations, supplement extraction, session-memory retrieval, and Forget Guard yield statistically significant gains of +2.71, +3.67, and +1.92 percentage points, respectively, whereas BM25/RRF yields no statistically significant improvement. Diagnostic artifacts achieve 98–100% DCR@3, compared with 0% for results-only logs. On an external Mem0-compatible slice, trace-adapted and results-only outputs achieve the same task accuracy (43.33%) but differ substantially in diagnostic coverage, demonstrating artifact portability beyond task performance. Our contributions are as follows: 1. D2ACCI protocol and decision contract. We introduce a dual-loop diagnostic protocol that promotes, feature-flags, or rejects memory interventions based on paired statistical evidence, protected-slice non-regression checks, and stage-level trace sufficiency. The individual primitives (paired runs, traces, feature flags) are familiar; the contribution is the decision contract that binds them into acceptance criteria—changing outcomes that aggregate-only reasoning cannot (e.g., issuing contradictory BM25/RRF verdicts across benchmarks, or promoting components without slice monitors). 2. D2ACCI-Eval artifact and DCR metric. We provide a reusable evaluation artifact for paired comparisons, protected slice gates, discordance export, and deterministic gate replay, together with a graded metric for measuring failure localizability. 3. MemStack case study. We instantiate D2ACCI in a diagnosable memory kernel with multi-granularity storage and feature-gated retrieval, and evaluate it on three public benchmarks with distinct failure profiles. A separate Mem0-compatible slice demonstrates artifact portability beyond the primary system. Algorithm 1 D2ACCI dual-loop memory-system iteration 1: Inner loop: run θP_θ to ingest, update, retrieve, constrain, assemble, and answer while emitting did_i. 2: Specify outer-loop hypothesis HtH_t, target slice, and expected failure mechanism. 3: Implement candidate θ′θ in the shared core or behind a feature flag. 4: Run baseline/candidate and collect details JSONL plus diagnostics. 5: Compute paired outcomes: I, R, BwB_w, BcB_c. 6: Label root causes in R∪BwR∪ B_w using the evidence-loss ladder. 7: Run paired gate: McNemar/bootstrap statistics, slice checks, and DCR. 8: Outer loop: accept, rollback, or gate the feature; archive artifacts and rejected priors. Figure 2: MemStack under the D2ACCI dual-loop protocol. Top: the inner memory loop processes queries through feature-flagged stages (F; independently togglable by the outer gate): ingestion, multi-layer storage, retrieval, constraint filtering, context assembly, and generation; each emits a typed diagnostic trace (IDs, layers, ranks, filters, budget, judge). Ablation targets (✗) mark paired ablations in Table 3. Bottom: the outer diagnostic loop feeds traces into D2ACCI-Eval (paired stats, slice deltas, DCR, discordance, audit), producing a gate decision that updates inner-loop feature flags for iteration t+1t+1. The iteration timeline shows automatic gate replay including the t3t_3 rejection. D2ACCI Framework Problem Formulation Let =sii=1NS=\s_i\_i=1^N denote an evaluation set, where each sample si=(qi,hi,yi∗)s_i=(q_i,h_i,y_i^*) contains a query, interaction history, and target answer. A memory configuration θ defines a pipeline θP_θ that produces y^i=θ(qi,hi) y_i=P_θ(q_i,h_i). An evaluator J yields a score mi=J(y^i,yi∗)m_i=J( y_i,y_i^*) and aggregate metric ℳ(θ)=N−1∑imiM(θ)=N^-1 _im_i. Aggregate performance alone does not reveal why a memory-augmented answer fails. D2ACCI therefore records a diagnostic state for each sample: di=(eiraw,eistore,ritop−k,fifilter,cictx,gigen,ji),d_i=(e_i^raw,e_i^store,r_i^top-k,f_i^filter,c_i^ctx,g_i^gen,j_i), (1) where eirawe_i^raw denotes evidence in the history, eistore_i^store denotes stored memory, ritop−kr_i^top-k denotes retrieved items, fifilterf_i^filter denotes filtering or reranking decisions, cictxc_i^ctx denotes assembled context, gigeng_i^gen denotes the generated answer, and jij_i denotes the final judgment. A candidate θ′θ is compared against a baseline θ on paired samples, partitioning them into I (improved), R (regressed), BwB_w (both wrong), and BcB_c (both correct). For each material regression or persistent error, the earliest actionable failure stage is labeled as ingestion miss, retrieval miss, context-assembly error, constraint error, generation error, component hurt, or unresolved ambiguity. Algorithm 2 first flags incomplete reports, then applies five ordered checks: reject significant harm, feature-flag low DCR(Diagnostic Coverage Rate), feature-flag inconclusive statistics, accept-with-monitor protected-slice regressions, or accept otherwise. The DCR check precedes the inconclusive fallback, so positive but poorly observable candidates are still flagged. Protected slices Π are pre-specified from benchmark category structure (e.g., multi-hop, temporal, sensitive-preference) before any gate is evaluated. Dual-loop decomposition The key design choice is to separate two coupled but different loops. The inner memory loop runs at inference or ingestion time: it extracts candidate memories from user interactions, writes or updates multi-granularity stores, retrieves evidence for a query, applies update/forget constraints, assembles context, and generates an answer. The outer diagnostic loop runs over paired artifacts: it compares a feature-bearing run against an ablation, checks statistical and slice gates, inspects traces for the earliest evidence-loss stage, and then accepts, rolls back, or keeps a feature behind a flag. From an agent-architecture perspective, the outer loop is a metacognitive self-regulation layer: the agent monitors its own competence boundaries, detects protected-slice regressions, and gates modifications to its own harness before they propagate. This differs from a single metric-driven development loop because the outer loop can reject a component that raises a raw dashboard number but lacks a trace-supported mechanism or regresses a protected slice. It also differs from pure prompt/program optimization: the outer loop does not search only over prompts, but controls architectural features such as supplement extraction, deduplication thresholds, retrieval channels, and forget-guard injection. Once instrumented, the gate can be triggered automatically after each candidate run, enabling automatic gate replay for candidate memory-policy changes with statistical regression controls. Algorithm 1 summarizes the complete dual-loop iteration from candidate construction to gated promotion, rollback, or feature-flagging. Gate replay is deterministic under fixed thresholds and pre-specified protected slices. Algorithm 2 D2ACCI-Eval gate-replay decision contract 0: ℛ=(Δ,p,[L,U],Δπ∈Π,DCRτ)R=( ,p,[L,U],\ _π\_π∈ ,DCR_τ); IDs, traces, Π , α,ρmin,ϵα, _ ,ε 0: Decision D, rationale r, archive A 1: if ℛR incomplete then 2: (D,r)←(Feature-Flag,incomplete report)(D,r)←( Feature-Flag,incomplete report) 3: else if Δ<0∧p≤α∧U<0 <0 p≤α U<0 then 4: (D,r)←(Reject,significant harm)(D,r)←( Reject,significant harm) 5: else if DCRτ<ρminDCR_τ< _ then 6: (D,r)←(Feature-Flag,insufficient trace)(D,r)←( Feature-Flag,insufficient trace) 7: else if p>α∨L≤0≤Up>α L≤ 0≤ U then 8: (D,r)←(Feature-Flag,inconclusive evidence)(D,r)←( Feature-Flag,inconclusive evidence) 9: else if ∃π∈Π:Δπ<−ϵ∃π∈ : _π<-ε then 10: (D,r)←(Accept-with-Monitor,slice guard)(D,r)←( Accept-with-Monitor,slice guard) 11: else 12: (D,r)←(Accept,positive paired evidence)(D,r)←( Accept,positive paired evidence) 13: end if 14: A←Archive(ℛ,Π,D,r,checks, monitors, priors)A (R, ,D,r,checks, monitors, priors) 15: return D,r,AD,r,A MemStack System Instantiation MemStack instantiates D2ACCI as a shared memory kernel with thin benchmark adapters (Figure 2). We retain only the design choices needed to interpret the ablations: L0 atomic facts, L1 topic rollups, L2 user profiles, and session memory are separate because each layer isolates a different failure mode. No Supplement disables the long-session supplement stage and conservative deduplication; No Session disables answer-time session-memory retrieval; No Forget Guard disables constraint-cache filtering. Design lessons. Three choices enable stage-level failure localization: (1) layer separation—each ablation maps onto a distinct failure family; (2) feature flags—retrieval channels, supplement extraction, and constraint injection are toggled per-run for cheap paired ablation; (3) trace emission at every stage boundary—without all three, a pipeline may improve accuracy while hiding whether the gain came from better evidence or lucky prompt formatting. Diagnosable ingestion, retrieval, and constraints. L0 facts are extracted from raw turns with source-turn IDs; L1 rollups aggregate session-level patterns; L2 profiles distill stable preferences from L1. Session memory is flushed at session boundaries as a structured record (raw turns, summary, keywords, timestamps, multi-granularity embeddings); long-session repair is handled by supplement extraction, which writes additional source-grounded L0 facts. At answer time, vector/BM25/RRF/session retrieval produces candidates; constraint-cache filtering removes forgotten or superseded memories; context assembly packs constraints before ordinary evidence under a token budget. Traces record source IDs, memory IDs, top-k scores, filtered items, packed/dropped context, and the final prompt at each stage boundary; these are exactly the fields consumed by DCR and gate replay. D2ACCI-Eval and Diagnostic Coverage Ordinary A/B testing answers whether a candidate is better on average; it does not say whether the pipeline preserved enough evidence to debug the errors it still makes. D2ACCI-Eval is the artifact layer for this distinction. Given paired details files, it joins examples by stable IDs, computes aggregate and slice-level deltas, exports Full-only and Ablated-only discordances, runs McNemar tests and bootstrap confidence intervals, and creates a root-cause audit queue. Its additional observability metric is Diagnostic Coverage Rate (DCR). For a failure or discordance set F, stage set S, trace did_i, and a per-stage actionability predicate A(di,s)A(d_i,s), let C(di)=∑s∈S[A(di,s)]C(d_i)= _s∈ S1[A(d_i,s)] be the number of actionable stages. We define DCRτ(F;S)=1|F|∑i∈F[C(di)≥τ].DCR_τ(F;S)= 1|F| _i∈ F1 [C(d_i)≥τ ]. (2) Here A is true when the trace contains the minimum fields needed to inspect a stage: source-turn or memory IDs for ingestion, top-k items and scores for retrieval, filter decisions for reranking, final packed context for assembly, constraint IDs for forget/update guards, and output plus judge metadata for generation. In the artifact, A(di,s)A(d_i,s) is schema-validated rather than free-form: a stage is actionable only when the required IDs, scores, decisions, and metadata are present. DCR therefore measures observability rather than accuracy. A high-accuracy system with low DCR may be hard to repair; a lower-accuracy but high-DCR run can still be valuable if it localizes missing evidence. Unless otherwise stated, gate replay uses DCR@3, requiring at least three actionable stages, and a fixed minimum coverage threshold of 0.90 set before inspecting paired results; sensitivity is reported at DCR@1, @5, and @6. To make the number interpretable, D2ACCI-Eval includes a results-only contrast that retains the answer and score but removes stage traces; this contrast receives zero coverage under the default three-stage predicate. Design properties of DCR. Three properties distinguish DCR from an ad-hoc coverage count and justify its use as a gate criterion: (1) Stage monotonicity—adding instrumentation stages can only increase C(di)C(d_i), so DCR never penalizes a more observable system (DCRτ(F,S′)≥DCRτ(F,S)DCR_τ(F;S ) _τ(F;S) for S′⊃S ⊃ S). (2) Accuracy-orthogonal separation—DCR captures information that accuracy cannot: the Mem0-compatible slice achieves 43.33% accuracy under both trace-adapted and result-only configurations, yet DCR@3 is 100% vs. 0%. (3) Gate soundness—under Algorithm 2, a null component (true Δ=0 =0) is promoted with probability at most α, because promotion requires McNemar p≤αp≤α with CI excluding zero; the false-promotion rate is bounded by the test’s Type-I control. Together, these ensure DCR is well-behaved and non-redundant, and that the outer-loop gate provides a statistical guarantee against adopting inert components—distinguishing it from metric-only dashboard decisions. Benchmark Eval set Ours MemBrain Δ Representative slices (category / (count, accuracy)) LoCoMo 1540 non-adv QA 93.59% 93.25% +0.34p Single-hop Multi-hop Temporal Open (841, 96.2) (282, 89.5) (321, 92.4) (96, 86.8) LongMemEval 500/500 questions 90.93% 85.60% +5.33p Single-user Knowledge-update Multi-session Temporal (70, 99.5) (78, 93.6) (133, 85.0) (133, 89.0) PersonaMem-V2 200 personas, 5000 MCQ 57.20% 55.72% +1.48p Sensitive Forget Neutral Therapy (511, 86.3) (1048, 66.9) (858, 51.2) (627, 44.5) Table 1: Main results and representative slice breakdowns. External rows are reference points, not fully normalized same-stack baselines. Slice-level reporting is required because memory interventions are frequently non-monotonic. Experiments Experimental Setup We evaluate the same shared core across three public regimes. LoCoMo tests long-term conversational QA over multi-session dialogues (12). We report non-adversarial judge accuracy on 1540 questions from 10 conversations. LongMemEval evaluates chat assistants on long-term interactive memory and contains 500 curated questions in the LongMemEval-S setting (24). PersonaMem-V2 evaluates implicit persona memory with 5000 multiple-choice questions over 200 personas (8). Unless otherwise stated, the answer model is GPT-4.1-mini, the open-ended judge is GPT-4o-mini with three deterministic runs per question (majority vote), and embeddings use BGE-M3 (1). PersonaMem-V2 uses exact-match multiple-choice accuracy. Paired confidence intervals use BCa bootstrap with 10,000 resamples; McNemar tests use the exact two-sided binomial form. External rows are reference points from published papers or public summaries; our strongest claims rely on internal paired comparisons using the same artifacts, prompts, and model settings. MemBrain is our primary public reference point on all three benchmarks; its numbers are taken from its maintained evaluation summary under the same splits (5). We treat published external numbers as reference context, not controlled baselines; the controlled evidence comes exclusively from internal paired ablations where the only variable is one feature flag. Concurrent systems (Mem0 Platform 92.5%/94.4% on LoCoMo/LongMemEval (3); EverMemOS 92.3%/83.0% (16)) are not directly comparable due to differences in answer models, judge prompts, and retrieval budgets. We retain MemBrain as the primary reference point because it documents conditions closest to ours across all three benchmarks. Each run produces per-question JSONL and summary metadata; D2ACCI-Eval consumes to produce paired statistics, slice deltas, discordance lists, DCR reports, and audit templates. We report results in terms of these artifacts below. Condition vs. Human A Fine κ Coarse κ Enriched GPT-4o trace relabel 0.571 0.619 Result-only GPT-4o relabel 0.258 0.272 Human B 0.674 0.711 Table 2: Audit-label contrast on the 60-case packet. Human B is an independent annotator; all rows are pre-adjudication. Main Results and Paired Ablations Table 1 summarizes the evaluated configuration and representative slice breakdowns. The LoCoMo margin over the reference point is within noise (+0.34p) and is not claimed as a contribution; the paper’s value comes from the ablation, diagnostic-coverage, and audit analyses below. The system is strongest on single-hop LoCoMo and single-session-user LongMemEval questions, but weaker on open-domain LoCoMo, multi-session LongMemEval, and therapy/background PersonaMem-V2 cases. These weaknesses define protected-slice monitors and motivate the root-cause audit. Table 3 reports completed paired ablations only. The sign convention is Full minus Ablated, so a positive delta means the evaluated component helps. Negative/null results are intentionally retained—they are central to D2ACCI: a component should not be promoted unless paired evidence supports its mechanism. The No-Session replay disables session-memory retrieval and session-context injection at answer time while retaining all stored memories, BM25/RRF, agentic retrieval, model settings, and judge settings; it isolates whether cross-session state contributes to multi-session and temporal questions without a re-ingestion confound. Benchmark Comparison n Ablated Δ Paired evidence Gate decision p 95% CI Full wins / Ablated wins Metric-only D2ACCI gate LoCoMo No BM25/RRF 1,540 94.00% −0.41-0.41 p .4426 [−1.36-1.36, 0.56] 27/34 Demote Feature-flag LoCoMo No Supplement 1,540 90.89% +2.71+2.71 p .0009 [1.26, 4.20] 90/50 Promote Accept; MH monitor LME No BM25/RRF 500 89.93% +1.00+1.00 p .4583 [−1.13-1.13, 3.13] 17/12 Promote Feature-flag LME No Session 500 87.27% +3.67+3.67 p .0026 [1.33, 6.00] 28/9 Promote Accept; KU monitor PMem No Forget Guard 5,000 55.28% +1.92+1.92 p .0030 [0.64, 3.18] 560/464 Promote Accept; slice monitor Table 3: Completed paired diagnostic ablations with metric-only vs. D2ACCI gate decisions. CI is bootstrap 95% for Full−-Ablated; Full wins / Ablated wins count correct examples. Metric-only: accept if Δ>0 >0, demote if Δ<0 <0; D2ACCI adds statistical and slice-level checks. The completed ablations change the interpretation of the system. Each ablation targets the benchmark whose category structure most directly stresses the removed component: supplement extraction addresses long-session evidence gaps central to LoCoMo’s multi-hop questions; session retrieval addresses cross-session temporal reasoning in LongMemEval; Forget Guard addresses preference revision central to PersonaMem-V2. BM25/RRF is tested on both open-ended benchmarks and is null on both, retained only as a monitored feature flag. The three accepted components are validated with slice monitoring for the small non-monotonic regressions in Table 3; cross-benchmark replication (e.g., No Session on LoCoMo) is supported identically by the protocol but not yet complete. The audit-label contrast in Table 2 provides the key validation; supplementary material includes per-queue details and a separate 110-example LoCoMo run. This supports our central claim: useful memory changes require paired, slice-level diagnostics, not aggregate wins alone. BH correction over the five tests preserves the accepted rows at q=0.01q=0.01 (adjusted p≤.0050p≤.0050) and leaves BM25/RRF null; we therefore read BM25/RRF as not promotable, not as zero-effect proof. An alternate GPT-4.1-mini replay over all 267 original open-ended discordances preserves the four open-ended gate outcomes (BM25/RRF feature-flag p=.175/.115p=.175/.115; Supplement/Session accept p=.0014/.0009p=.0014/.0009), with concordant examples left unrescored. Diagnostic Coverage and Gate Validation Trace availability and root cause agreement. Table 2 compares root-cause labeling under two conditions on the 60-case packet: full enriched traces (retrieval results, assembly context, constraint records) versus result-only information (answer and score). Human–human pre-adjudication agreement across the 60 labeled cases is 44/60 exact matches (κ=0.674κ=0.674 fine, 0.7110.711 coarse; bootstrap 95% CIs [0.534,0.797][0.534,0.797] and [0.564,0.841][0.564,0.841]). The trace effect remains large: enriched-trace GPT-4o relabeling reaches κ=0.571κ=0.571 fine (0.6190.619 coarse; CIs [0.418,0.715][0.418,0.715] and [0.464,0.764][0.464,0.764]) versus κ=0.258κ=0.258 (0.2720.272; CIs [0.159,0.355][0.159,0.355] and [0.175,0.376][0.175,0.376]) for result-only. Under the result-only GPT-4o relabeling condition, 36/60 cases are assigned to “unresolved”, because answer-level evidence alone does not identify failing stage. This validates the core premise: trace availability transforms root-cause labeling from guesswork into an auditable task. The following LoCoMo discordance illustrates that result-only outputs show only a date flip, whereas traces reveal that supplement extraction preserved the decisive temporal evidence. At the aggregate level, the gate accepts Supplement with a multi-hop monitor over 1540 pairs (+2.71p, p=.0009p=.0009, DCR@3=99.47%). Case Study (LoCoMo conv1:q038) Question: When did Gina go to a dance class with friends? Gold Response: 21 Jul 2023. Ablated Response(No Supplement): Jul 14. Final context carries a wrong 2023-07-14 temporal memory. × Full Response(Supplement enabled): Jul 21. Trace keeps source-backed 2023-07-21 evidence. ✓ D2ACCI Diagnosis: Result-only outputs show a date flip but not its cause; traces show the supplement stage preserves the decisive temporal evidence. Diagnostic coverage and trace masking. The audit contrast above establishes that trace availability materially improves diagnostic consistency in the 60-case packet (κ jumps from 0.258 to 0.571). DCR formalizes this as a measurable proxy: it tracks whether each failure retains enough stage evidence for inspection, not whether the trace itself is a correct root-cause label. Crucially, DCR is graded, not binary. Counterfactual trace masking produces monotonic degradation across the five paired ablations without requiring additional model runs. Result-only records achieve 100.0% at DCR@1 but 0.0% at @3, @5, and @6. Retrieval-only traces achieve 98.4–100.0% at @1 and @3 but 0.0% at @5 and @6. Adding stored-memory, retrieval, and assembled-context traces extends the 98.4–100.0% coverage through @5, whereas only full traces maintain 98.4–100.0% coverage at @6. The 98.4% vs. 100% variation across ablations is the most informative signal—it identifies samples that bypass a given stage (e.g., short conversations that never trigger supplement extraction) and are therefore only partially diagnosable. These partial-observability blind spots would be invisible under a binary “logged vs. not-logged” check; DCR surfaces them as concrete instrumentation targets for the next iteration. Pair-level DCR@3 is 99.21% over 127 LoCoMo No-BM25/RRF discordances, 99.47% over 190 LoCoMo No-Supplement cases, 98.63% over 73 LongMemEval No-Session cases, and 100.00% over 2700 PersonaMem-V2 No-Forget-Guard cases; the separate 110-example LoCoMo result-only check is 0/110, confirming that scalar outcomes alone provide no localizable evidence. Beyond diagnostic coverage, we validate that the gate itself produces meaningful decisions. The last two columns of Table 3 contrast the dual-loop gate against a metric-only policy. The metric-only policy issues contradictory BM25/RRF verdicts across benchmarks (demote on LoCoMo, promote on LongMemEval); the dual-loop gate identifies both as null and retains a monitored feature-flag. For the three accepted components, metric-only promotes unconditionally, while the dual-loop gate attaches explicit slice monitors (MH −2.6-2.6p, KU −1.3-1.3p, multi-slice non-monotonicity in PersonaMem-V2) that prevent silent regressions from propagating. A threshold sweep confirms that the accept/feature-flag split is stable for α≥0.005α≥ 0.005; only monitoring annotations change as ϵε varies. Trace-Guided Failure Analysis and Iteration The preceding sections validate the gate statistically; we now show how traces guide specific failure localization and repair. Failure boundary analysis. A rule-based trace pass over 2,140 wrong PersonaMem-V2 cases localizes the performance boundary: generation/option-scoring accounts for 1,223 cases (57.1%), constraint handling for 637 (29.8%), retrieval misses for 278 (13.0%), and ingestion misses for 2 (0.1%). PersonaMem-V2 is included precisely because it stress-tests the protocol under conditions where memory-only improvements hit diminishing returns. This diagnosis is itself a protocol output: without per-question retrieval and assembly traces, one cannot distinguish “never stored” from “stored but model picked the wrong option.” The practical payoff is preventing wasted optimization: D2ACCI localizes the boundary between memory-addressable and generation-addressable errors (87% lie beyond retrieval), a conclusion invisible under aggregate-only evaluation and one that redirects engineering effort toward the actual bottleneck. Ingestion stage over-merge. Traces for affected questions (e.g., “What breed is James’s second dog?”) revealed that L0 contained a single merged memory “James has dogs” instead of three separate records. Retrieval returned this merged record (rank 1, score 0.71) but lacked breed information. Failure localized to ingestion-stage over-merge; the repair (threshold 0.92) recovered 4.48p. Constraint stage filtering. In the PersonaMem follow-up audit (e.g., persona136:q7), the full trace’s forget scan preserves the Diwali option and the model selects the gold answer, while the no-guard variant drifts to a different cultural-event option. This localizes the gain to the constraint stage rather than retrieval: the evidence was already present, but the guard prevented a forgotten-topic spillover. Figure 3: LoCoMo iteration trajectory with automatic gate-replay decisions. Each transition is produced by the replay script (<<1 s per candidate); all five decisions match the human-in-the-loop outcome. The dashed red segment marks t3t_3: traces localized over-merging, triggering automatic rejection. AP∗ Trigger Observed Behavior Resolution OM Threshold (0.75)(0.75) Facts collapse into a single memory representation Reverted DA Option-aware filter Retrieval surfaces distractor evidence Rejected NME Gated expansion Recovers some targets, introduces off-target noise (57.34%→57.24%57.34\%→ 57.24\%) Slice-gated ∗ AP: Anti-pattern Table 4: Representative archived anti-patterns (OM: Over-merge, DA: Distractor amplification, NME: Non-monotonic expansion) exposed by D2ACCI traces. Figure 3 shows the LoCoMo trajectory; the t3t_3 rejection (detailed in the case study above) is the trace-guided outer-loop example. The No-Supplement row in Table 3 is its paired statistical counterpart (+2.71+2.71p, p=.0009p=.0009). To verify the outer loop requires no human intervention, we replay the gate-decision script over LoCoMo iterations (t0t_0–t5t_5) using only paired reports (deltas, p-values, slice vectors, DCR) as input. The script produces decisions matching human judgment in <<1 s. Figure 3 annotates each transition with its automatic gate-replay outcome; all five decisions match the human-in-the-loop process, including the critical t3t_3 rejection. Once instrumented, the gate script reproduces human decisions without per-candidate manual review. Table 4 catalogs rejected interventions as reusable anti-patterns. Related Work Agent memory benchmarks and systems. Long-term memory benchmarks evaluate agents’ ability to retain, retrieve, and update information across extended interactions (12; 24; 8). Corresponding systems explore virtual context hierarchies, add/search/update APIs, dynamically linked notes, temporal or graph stores, and multi-granular representations (14; 38; 18; 3; 27; 9; 7; 22; 28; 25). RL-based work further optimizes memory management, utility estimation, retrieval, answer generation, and proactive querying (36; 29; 35; 31; 21). These systems make memory decisions adaptive, but task-level rewards and aggregate scores can conflate ingestion, retrieval, constraint, and generation failures; D2ACCI is complementary because it supplies paired, slice-level, trace-preserving acceptance criteria. Self-evolving memory architectures and LM programs. Recent agents evolve memories, reusable skills, or the memory mechanism itself from interaction feedback, reflective search, accumulated experience, and failure signals (19; 20; 37; 23; 34; 2; 26; 33; 15; 11). Similarly, DSPy, MIPRO, and TextGrad optimize language-model programs or instructions under task objectives (10; 13; 32). D2ACCI differs by making paired statistical testing, protected-slice monitoring, null-result preservation, and trace coverage explicit promotion criteria rather than prescribing a search space. Observability and regression-controlled evolution. Observability tools and self-evolving-agent research highlight the need for reliable evaluation and capability preservation (6; 30). MemTrace performs post-hoc attribution of failures to memory operations (4). D2ACCI complements such tools by combining stage-level traces with paired gates, protected-slice checks, archived null results, and DCR to test whether a modification is reproducible, non-regressive, and diagnostically auditable. Conclusion We introduce D2ACCI in this paper, a dual-loop protocol that turns memory-system changes into auditable decisions grounded in paired statistics, slice monitors, and stage traces. Across three public agent memory benchmarks, the protocol distinguishes statistically validated improvements from null changes. When instantiated in MemStack, D2ACCI achieves 93.59% accuracy on LoCoMo, 90.93% on LongMemEval, and 57.20% on PersonaMem-V2. The three accepted components yield statistically significant gains of 1.92–3.67 percentage points, while the corresponding diagnostic artifacts achieve 98–100% DCR@3, preserving sufficient stage-level evidence to localize failures and support trace-guided repair rather than blind tuning. The protocol is portable to any memory stack that emits stable sample IDs and comparable stage traces. In the future, the protocol will be extended to online settings and cross-stack comparisons. In addition, we will develop automated trace verification to complement manual audits. References Chen et al. (2024) J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu BGE M3-embedding: multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL, Cited by: Experimental Setup. Cheng et al. (2026) Z. Cheng, Z. Liu, Y. Shan, X. Wang, X. Zhu, Y. Ma, H. Wang, Y. Guo, W. Lin, and Y. Wang Mem2^2evolve: towards self-evolving agents via co-evolutionary capability expansion and experience distillation. External Links: 2604.10923, Link Cited by: Self-evolving memory architectures and LM programs.. Chhikara et al. (2025) P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav Mem0: building production-ready AI agents with scalable long-term memory. Note: arXiv preprint arXiv:2504.19413 Cited by: Introduction, Introduction, Experimental Setup, Agent memory benchmarks and systems.. Deng et al. (2026) X. Deng, R. Zhong, H. Peng, X. Lu, Y. Wu, G. Li, B. Xu, Y. Yao, J. Fang, H. Cao, J. Guo, Y. Yuan, Z. Ma, Y. Yu, R. Hu, B. Dong, H. Zhu, and N. Zhang MemTrace: tracing and attributing errors in large language model memory systems. Note: arXiv preprint arXiv:2605.28732 Cited by: Observability and regression-controlled evolution.. FeelingAI Team (2026) FeelingAI Team MemBrain: agent-native memory for ai agents. Note: https://github.com/feelingai-team/MemBrainSoftware and maintained evaluation summary Cited by: Experimental Setup. Gao et al. (2026) H. Gao, J. Geng, W. Hua, M. Hu, X. Juan, H. Liu, S. Liu, J. Qiu, X. Qi, Y. Wu, H. Wang, H. Xiao, Y. Zhou, S. Zhang, J. Zhang, J. Xiang, Y. Fang, Q. Zhao, D. Liu, Q. Ren, C. Qian, Z. Wang, M. Hu, H. Wang, Q. Wu, H. Ji, and M. Wang A survey of self-evolving agents: what, when, how, and where to evolve on the path to artificial super intelligence. External Links: 2507.21046, Link Cited by: Introduction, Observability and regression-controlled evolution.. Hu et al. (2026) C. Hu, X. Gao, Z. Zhou, D. Xu, Y. Bai, X. Li, H. Zhang, T. Li, C. Zhang, L. Bing, and Y. Deng EverMemOS: a self-organizing memory operating system for structured long-horizon reasoning. External Links: 2601.02163, Link Cited by: Agent memory benchmarks and systems.. Jiang et al. (2025) B. Jiang, Y. Yuan, M. Shen, Z. Hao, Z. Xu, Z. Chen, Z. Liu, A. R. Vijjini, J. He, H. Yu, R. Poovendran, G. Wornell, L. Ungar, D. Roth, S. Chen, and C. J. Taylor PersonaMem-v2: towards personalized intelligence via learning implicit user personas and agentic memory. Note: arXiv preprint arXiv:2512.06688 Cited by: Introduction, Experimental Setup, Agent memory benchmarks and systems.. Kang et al. (2025) J. Kang, M. Ji, Z. Zhao, and T. Bai Memory os of ai agent. External Links: 2506.06326, Link Cited by: Agent memory benchmarks and systems.. Khattab et al. (2024) O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts DSPy: compiling declarative language model calls into state-of-the-art pipelines. In International Conference on Learning Representations, Cited by: Self-evolving memory architectures and LM programs.. Liu et al. (2026) J. Liu, X. Ye, P. Xia, Z. Zheng, C. Xie, M. Ding, and H. Yao EvolveMem:self-evolving memory architecture via autoresearch for llm agents. External Links: 2605.13941, Link Cited by: Self-evolving memory architectures and LM programs.. Maharana et al. (2024) A. Maharana, D. Lee, S. Tulyakov, M. Bansal, F. Barbieri, and Y. Fang Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, Note: LoCoMo benchmark Cited by: Introduction, Experimental Setup, Agent memory benchmarks and systems.. Opsahl-Ong et al. (2024) K. Opsahl-Ong, M. J. Ryan, J. Purtell, D. Broman, C. Potts, M. Zaharia, and O. Khattab Optimizing instructions and demonstrations for multi-stage language model programs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 9340–9366. External Links: Document Cited by: Self-evolving memory architectures and LM programs.. Packer et al. (2023) C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez MemGPT: towards LLMs as operating systems. Note: arXiv preprint arXiv:2310.08560 Cited by: Introduction, Agent memory benchmarks and systems.. Pan et al. (2026) W. Pan, S. Liu, X. Zhou, S. Zhang, W. Shi, M. Xu, and X. Jia M⋆ : every task deserves its own memory harness. External Links: 2604.11811, Link Cited by: Self-evolving memory architectures and LM programs.. Patel et al. (2026) A. Patel, M. Chen, K. Zhang, Y. Wang, and J. Liu EverMemOS: a self-organizing memory operating system for LLM agents. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, Cited by: Introduction, Introduction, Experimental Setup. Rasmussen et al. (2025a) D. Rasmussen G. Frisoni et al. Zep: a temporal knowledge graph architecture for agent memory. Note: arXiv preprint arXiv:2501.13956 Cited by: Introduction. Rasmussen et al. (2025b) P. Rasmussen, P. Paliychuk, T. Beauvais, J. Ryan, and D. Chalef Zep: a temporal knowledge graph architecture for agent memory. External Links: 2501.13956, Link Cited by: Agent memory benchmarks and systems.. Shinn et al. (2023) N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao Reflexion: language agents with verbal reinforcement learning. External Links: 2303.11366, Link Cited by: Self-evolving memory architectures and LM programs.. Wang et al. (2023) G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar Voyager: an open-ended embodied agent with large language models. External Links: 2305.16291, Link Cited by: Self-evolving memory architectures and LM programs.. Wang et al. (2026a) S. Wang, B. Liu, Z. Gao, L. Ma, X. Wang, Y. Xie, and X. Tan Explore with long-term memory: a benchmark and multimodal llm-based reinforcement learning framework for embodied exploration. External Links: 2601.10744, Link Cited by: Agent memory benchmarks and systems.. Wang et al. (2026b) S. Wang, E. Yu, O. Love, T. Zhang, T. Wong, S. Scargall, and C. Fan MemMachine: a ground-truth-preserving memory system for personalized AI agents. Note: arXiv preprint arXiv:2604.04853 Cited by: Agent memory benchmarks and systems.. Wei et al. (2026) T. Wei, N. Sachdeva, B. Coleman, Z. He, Y. Bei, X. Ning, M. Ai, Y. Li, J. He, E. H. Chi, C. Wang, S. Chen, F. Pereira, W. Kang, and D. Z. Cheng Evo-memory: benchmarking llm agent test-time learning with self-evolving memory. External Links: 2511.20857, Link Cited by: Self-evolving memory architectures and LM programs.. Wu et al. (2025) D. Wu, H. Wang, W. Yu, Y. Zhang, K. Chang, and D. Yu LongMemEval: benchmarking chat assistants on long-term interactive memory. In International Conference on Learning Representations, Note: arXiv:2410.10813 Cited by: Introduction, Experimental Setup, Agent memory benchmarks and systems.. Xia et al. (2026) M. Xia, X. Zhang, S. Dixit, P. Harimurugan, R. Wang, V. Ruhle, R. Sim, C. Bansal, and S. Rajmohan Memora: a harmonic memory representation balancing abstraction and specificity. Note: arXiv preprint arXiv:2602.03315 Cited by: Agent memory benchmarks and systems.. Xiong et al. (2026) Y. Xiong, S. Hu, and J. Clune Learning to continually learn via meta-learning agentic memory designs. External Links: 2602.07755, Link Cited by: Self-evolving memory architectures and LM programs.. Xu et al. (2025) W. Xu, Z. Liang, K. Mei, H. Gao, J. Tan, and Y. Zhang A-mem: agentic memory for llm agents. External Links: 2502.12110, Link Cited by: Agent memory benchmarks and systems.. Xu et al. (2026) Y. Xu, Y. Sun, Y. Liu, M. Zhou, J. Qiao, L. Ma, K. Tang, W. Wang, X. Jiang, and G. Jiang From passive retrieval to active memory navigation: learning to use memory as a structured action space. Note: arXiv preprint arXiv:2607.05794 Cited by: Agent memory benchmarks and systems.. Yan et al. (2026) S. Yan, X. Yang, Z. Huang, E. Nie, Z. Ding, Z. Li, X. Ma, J. Bi, K. Kersting, J. Z. Pan, H. Schütze, V. Tresp, and Y. Ma Memory-r1: enhancing large language model agents to manage and utilize memories via reinforcement learning. External Links: 2508.19828, Link Cited by: Agent memory benchmarks and systems.. Yu et al. (2026a) Y. Yu, X. Yuan, H. Jin, H. Liu, Y. Yu, and H. Wang Do self-evolving agents forget? capability degradation and preservation in lifelong llm agent adaptation. External Links: 2605.09315, Link Cited by: Observability and regression-controlled evolution.. Yu et al. (2026b) Y. Yu, L. Yao, Y. Xie, Q. Tan, J. Feng, Y. Li, and L. Wu Agentic memory: learning unified long-term and short-term memory management for large language model agents. External Links: 2601.01885, Link Cited by: Agent memory benchmarks and systems.. Yuksekgonul et al. (2024) M. Yuksekgonul, F. Bianchi, J. Boen, S. Liu, Z. Huang, C. Guestrin, and J. Zou TextGrad: automatic “differentiation” via text. Note: arXiv preprint arXiv:2406.07496 Cited by: Self-evolving memory architectures and LM programs.. Zhang et al. (2025) G. Zhang, H. Ren, C. Zhan, Z. Zhou, J. Wang, H. Zhu, W. Zhou, and S. Yan MemEvolve: meta-evolution of agent memory systems. External Links: 2512.18746, Link Cited by: Self-evolving memory architectures and LM programs.. Zhang et al. (2026a) H. Zhang, Q. Long, J. Bao, T. Feng, W. Zhang, H. Yue, and W. Wang MemSkill: learning and evolving memory skills for self-evolving agents. External Links: 2602.02474, Link Cited by: Self-evolving memory architectures and LM programs.. Zhang et al. (2026b) K. Zhang, S. Gui, S. Yang, W. Chen, and Y. Feng Learning to remember: end-to-end training of memory agents for long-context reasoning. External Links: 2602.18493, Link Cited by: Agent memory benchmarks and systems.. Zhang et al. (2026c) S. Zhang, J. Wang, R. Zhou, J. Liao, Y. Feng, Z. Li, Y. Zheng, W. Zhang, Y. Wen, Z. Li, F. Xiong, Y. Qi, B. Tang, and M. Wen MemRL: self-evolving agents via runtime reinforcement learning on episodic memory. External Links: 2601.03192, Link Cited by: Agent memory benchmarks and systems.. Zhao et al. (2024) A. Zhao, D. Huang, Q. Xu, M. Lin, Y. Liu, and G. Huang ExpeL: llm agents are experiential learners. External Links: 2308.10144, Link Cited by: Self-evolving memory architectures and LM programs.. Zhong et al. (2023) W. Zhong, L. Guo, Q. Gao, H. Ye, and Y. Wang MemoryBank: enhancing large language models with long-term memory. External Links: 2305.10250, Link Cited by: Agent memory benchmarks and systems..