Paper deep dive

Persistent Memory in Multi-Agent LLM Inference: What It Costs, What It Buys, and When You Can Tell

Hochan Son, Kyungdoe Han, Jaehan Koh, Xiaowu Dai, Wenlu Xu, Guang Cheng

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 50

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 10/13/2026, 4:27:24 AM

Summary

This paper evaluates a three-tier multi-agent LLM architecture (CoA-PKV) that decomposes long-context inference to bound the active KV cache. While decomposition significantly reduces peak KV memory usage (14.3 MiB vs ~35.5 MiB for baselines), the addition of a persistent memory tier for storing and recalling reasoning traces incurs a small memory cost (+0.368 MiB) but yields no statistically significant accuracy improvement. The authors argue this null result is structural: standard single-question benchmarks do not allow for cross-query memory transfer, and previous reports of memory benefits likely suffered from measurement artifacts such as trace contamination, order confounding, and component execution failures.

Entities (11)

KV Cache · memory-component · 99%CoA-PKV · system · 98%Flat · baseline · 95%RAG · baseline · 95%HotpotQA · dataset · 92%NVIDIA DGX Spark · hardware · 90%NVIDIA Jetson Orin Nano Super · hardware · 90%Qwen3.5-2B · model · 90%Qwen3.5-35B-A3B-FP8 · model · 90%Qdrant · technology · 85%Redis · technology · 85%

Relation Signals (10)

CoA-PKV → reduces → peak KV working set

confidence 98% · Decomposition delivers: peak KV working set of 14.3 MiB per query against 35.5 and 35.3 MiB for single-pass and retrieval-augmented baselines.

Flat → haspeakkv → 35.5 MiB

confidence 95% · Flat... peak KV per query (MiB) 35.5

RAG → haspeakkv → 35.3 MiB

confidence 95% · RAG... peak KV per query (MiB) 35.3

CoA-PKV → uses → KV Cache

confidence 95% · Decomposing long-context inference across cooperating agents bounds the active KV cache per call

Persistent Tier → hasnoeffecton → Accuracy

confidence 92% · produces no detectable accuracy change (+0.015, 95% CI [-0.011, +0.046])

Persistent Tier → increases → peak cache

confidence 90% · costs +0.368 MiB [+0.167, +0.590] of peak cache

CoA-PKV → runson → NVIDIA Jetson Orin Nano Super

confidence 90% · The Reasoner and Synthesiser run on the vLLM node; the Decomposer... run on the Jetson’s smaller model

CoA-PKV → runson → NVIDIA DGX Spark

confidence 90% · an NVIDIA DGX Spark running vLLM

→ →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Decomposing long-context inference across cooperating agents bounds the active KV cache per call rather than total evidence, which matters when KV-cache memory binds. Many such systems add a persistent tier storing and recalling reasoning traces, usually validated by an ablation reporting an accuracy gain. We measure both on one three-tier agent architecture. Decomposition delivers: peak KV working set of 14.3 MiB per query against 35.5 and 35.3 MiB for single-pass and retrieval-augmented baselines. The persistent tier does not: across eight controlled dataset pairs at n=100 per arm it costs +0.368 MiB [+0.167, +0.590] of peak cache and produces no detectable accuracy change (+0.015, 95% CI [-0.011, +0.046]). We argue the null is structural: single-question benchmarks supply each item with its own evidence and score it independently, and correctness requires resetting stored traces between conditions, so recall has nothing informative to retrieve. Reaching it took four measurement corrections -- three inflating the apparent benefit, the fourth making an effect that size look resolvable -- none visible in the results table. We give the conditions an agent-memory ablation must satisfy and detection procedures that need no knowledge of the specific defect.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

46,201 characters extracted from source content.

Expand or collapse full text

Persistent Memory in Multi-Agent LLM Inference: What It Costs, What It Buys, and When You Can Tell Hochan Son †thanks: Lead contributor and corresponding author: hochanson@g.ucla.edu. Kyungdoe Han Jaehan Koh Affiliation: University of California, Los Angeles University of Wisconsin HCLTech America Xiaowu Dai Wenlu Xu Guang Cheng †thanks: Corresponding author: guangcheng@stat.ucla.edu. Abstract Decomposing long-context inference across cooperating agents bounds the active KV cache per call rather than total evidence, which matters when KV-cache memory binds. Many such systems add a persistent tier storing and recalling reasoning traces, usually validated by an ablation reporting an accuracy gain. We measure both on one three-tier agent architecture. Decomposition delivers: peak KV working set of 14.314.3 MiB per query against 35.535.5 and 35.335.3 MiB for single-pass and retrieval-augmented baselines. The persistent tier does not: across eight controlled dataset pairs at n=100n=100 per arm it costs +0.368+0.368 MiB [+0.167,+0.590][+0.167,+0.590] of peak cache and produces no detectable accuracy change (+0.015+0.015, 95% CI [−0.011,+0.046][-0.011,+0.046]). We argue the null is structural: single-question benchmarks supply each item with its own evidence and score it independently, and correctness requires resetting stored traces between conditions, so recall has nothing informative to retrieve. Reaching it took four measurement corrections—three inflating the apparent benefit, the fourth making an effect that size look resolvable—none visible in the results table. We give the conditions an agent-memory ablation must satisfy and detection procedures that need no knowledge of the specific defect. 1 Bounding the working set, and the tier that is hard to test The KV cache is the memory bottleneck of long-context inference: it grows with context, is provisioned against its peak, not its average, and routinely exceeds an edge accelerator’s VRAM, which is why cache management is a systems problem in its own right (Kwon et al., 2023). One response is architectural: if a task is decomposed across agents, each handling part of the evidence and passing distilled state onward (Zhang et al., 2024), no single call holds the whole context and the active working set stays bounded as total evidence grows. Systems in this family commonly add a further tier: a persistent store keeping each query’s reasoning trace and retrieving relevant ones later. The argument is intuitive—an agent that remembers should need less context—and is usually supported by an ablation that disables the tier and reports an accuracy drop. We measure both on one such system. The first holds; the second does not—and we argue the way these systems are evaluated makes it very difficult either for it to hold or for a reported gain to mean what it appears to. Figure 1: The measured system and the single axis the ablation varies; Appendix A.1 gives the block-level call sequence. What is new here. Retrieval augmentation and agent decomposition are routinely treated as interchangeable answers to long context. For memory they are not: RAG changes which evidence enters a call; decomposition changes how many calls it is spread across, and only the second bounds the peak KV. Our retrieval-matched RAG baseline still concatenates its passages into one generation call and measures a peak indistinguishable from closed-book single-pass (Table 1): filtering evidence better does not shrink the largest call. To our knowledge no prior work reports an agent architecture’s peak KV as a measured, ablated systems quantity (§6); that decomposition alone bounds it, independent of any persistent-recall tier, is our first contribution and §3–4’s null the second. We do not claim persistent recall improves accuracy: that is untested, not disproven. The architecture. Figure 1 shows the system. L1 is the ephemeral per-call KV cache. L2 is a key–value store of typed Communication Units, so an agent receives distilled predecessor output rather than raw context. L3 is a vector store whose two roles—semantic evidence filtering before anything enters a cache, and (q,k,v)(q,k,v) trace persistence for cross-query recall—are separable in code; only the second is the persistent-memory claim. We call the full system CoA-PKV—chain of agents over a persistent key–value tier—and the arm with trace recall and writes disabled CoA-NoKV; they differ in exactly those two operations, L1, L2 and evidence filtering untouched in both. 2 The decomposition bounds the working set We measure peak KV as the largest single call’s (prompt+generated)(prompt+generated) footprint, converted at the served model’s 10,24010,240 B/token KV geometry (not total device memory; Appendix A.1), against a closed-book single-pass baseline (Flat) and a retrieval-matched RAG pipeline (Lewis et al., 2020) sharing the same reasoner, critic and evidence pool. All arms pool the same two cells (HotpotQA, LB-HotpotQA) at n=100n=100, averaging each arm’s replicate runs (RAG 3 per cell at seeds 42/43/44, CoA-PKV 2, Flat 1); intervals are the Appendix A.3 cluster bootstrap. Table 1: Peak KV and latency: Flat, RAG, and CoA-PKV (ours). Brackets: 95% CI. Flat‡ RAG (Lewis et al., 2020) CoA-PKV (ours) vs Flat vs RAG peak KV per query (MiB) 35.5 [34.0, 37.0] 35.3 [30.9, 39.8] 14.3 [13.0, 15.4] −59.9%-59.9\% −59.7%-59.7\% latency per query (s)† 67 52 29 −56.7%-56.7\% −44.2%-44.2\% Peak KV is 2.5×2.5× lower and latency 1.81.8–2.3×2.3× lower: our CoA-PKV interval clears both baselines with no overlap, while Flat and RAG overlap each other, as §1 anticipates. Flat has one run per cell, so its interval covers question sampling only, which by §5, C4 makes it optimistic; ‡closed-book single-pass prompting, no retrieval, hence no citation. †Latency is from one contiguous service window, before the ≈3.5×≈3.5× serving-stack drift of the later campaign (§5, C3); peak KV is token-count-derived and unaffected by serving speed, which is why we lead with it (per-dataset: Appendix A.7). Bounded per-call working sets are bought with more calls—cumulative KV traffic is ≈1.6×≈1.6× higher—so the claim is lower peak at higher total traffic, the trade that matters when KV-cache memory binds. Accuracy against retrieval-matched RAG is parity, every difference smaller than the reproducibility band; that the decomposition works is what makes the memory tier worth testing (Appendix A.6). 3 The persistent tier costs memory and buys nothing measurable We ablate trace recall across eight dataset pairs spanning multi-hop QA (Yang et al., 2018; Ho et al., 2020; Trivedi et al., 2022), long-document QA (Bai et al., 2024) and graduate-level multiple choice (Rein et al., 2023; Wang et al., 2024). Both arms are matched on dataset, n, seed, evidence scoping and endpoint, traces reset between conditions, and isolation verified live by a probe asserting the recall path is present when enabled and absent when disabled—absence alone is also what an unconfigured store looks like. Pairing is per question; intervals are percentile bootstrap (B=104B=10^4), the across-cell one a cluster bootstrap resampling cells then questions, since questions within a cell share a difficulty profile. Table 2: Cost and accuracy effect of enabling persistent recall. effect 95% CI peak KV cost of enabling recall +0.368+0.368 MiB [+0.167,+0.590][+0.167,+0.590] accuracy change from enabling recall +0.015+0.015 [−0.011,+0.046][-0.011,+0.046] The cost interval excludes zero: recall prepends retrieved traces to the reasoner’s prompt, so prompts grow—≈2%≈2\% against per-query peaks of 13.613.6–21.921.9 MiB (Appendix A.7), the multiple-choice cells sitting above §2’s 14.314.3. The accuracy interval includes zero. We do not claim the effect is zero; we claim an effect larger than ≈0.046≈0.046 is excluded, and that the point estimate is not separable from noise. That bound is not in tension with the ±0.09±0.09 replicate band of §5, C4: the band measures run-to-run drift in absolute scores, whereas this delta is paired per question, so drift common to both arms differences out; drift between them, which C3’s order confound leaves open, does not. It is also convention-dependent: scoring failed questions as 00 gives +0.015+0.015, dropping those either arm failed gives +0.003+0.003, and the gap is the memory-off arms’ extra failures, which C3’s order confound leaves unattributed (§5). It is also an intention-to-treat estimate—recall could fire for only one question in four—which §4 quantifies. 4 Why the null is structural The result is not that our persistent memory is poor, but that the regime we—and many systems papers—evaluate in cannot exercise it. That multi-session interaction is required to test memory is established: LoCoMo (Maharana et al., 2024) and LongMemEval (Wu et al., 2025) exist because single-session QA does not probe it. We quantify that observation’s consequence: what a correctly isolated ablation returns when a system is evaluated on independent-item benchmarks anyway, and what makes it return something else. Two properties compose to force the null. Items are independent and self-supplied. Benchmarks of this kind present each question with its own gold or retrieved evidence and score it in isolation. Nothing a system learns on question i is intended to be useful for question j. Correctness forces the reset. Recall retrieves by question embedding, so at a fixed seed a re-run retrieves its own prior conclusions (§5, C2). Preventing that requires clearing stored traces between conditions. But then within-condition recall can only reach traces from other, unrelated items in the same run. Measured, not assumed. Reproducing the retriever’s predicate exactly—same embedding model, same 0.60.6 threshold, only traces written earlier in the same run—recall was reachable for 24% of questions as sent (20%20\% once each cell’s shared instruction boilerplate is stripped; Appendix A.5); for the rest it provably returned nothing, so §3’s figure is intention-to-treat. Restricted to questions where recall could fire the effect is −0.012-0.012 [−0.073,+0.046][-0.073,+0.046] (n=138n=138)—a wider interval than the aggregate—while questions where it provably could not act as a built-in control and give +0.009+0.009 [−0.007,+0.026][-0.007,+0.026], consistent with zero as required. The aggregate is thus correctly measured but uninformative about efficacy, and the subgroup that tests efficacy is underpowered—a distinction applying to any such ablation. The transferable point is the contrapositive, uncomfortable because the practice is common: a non-zero memory effect reported on an independent-item benchmark is more likely an artifact than a finding. Systems papers routinely support a memory tier with exactly such an ablation; we did, with four artifacts (§5) present at once—three inflating the effect, the fourth making it look resolvable. Testing the claim requires a regime where items genuinely share state—multi-session dialogue, a corpus queried repeatedly, an agent revisiting entities—and where the reset that prevents contamination does not destroy the signal. 5 Conditions an agent-memory ablation must satisfy We reached §3’s measurement only after four corrections, stated as conditions rather than bugs: each was satisfied in appearance and violated in fact. C1–C3 each independently produced an apparent memory benefit; C4 made effects that size look resolvable. C1 — The component executed. Our persistent tier wrote traces nothing ever read: both arms of every prior memory ablation ran with no memory. Three further components were similarly inert—a client-side ablation flag for a server-side agent, a reranker whose 404s became “keep existing order”, a cost accumulator off the benchmark path that zeroed 38 published cells. Each is correct code implementing a degradation that was total, permanent and indistinguishable from success. Detection: emit a counter for every component whose presence is a claim and assert it non-zero in the run summary—not a log line, a counter the summary refuses to omit. C2 — Runs were independent. With traces keyed by question embedding, a fixed-seed re-run retrieves its own previous answers; we measured 100/100100/100 of a run’s questions already carrying a stored trace, from 48,91548,915 traces over 2,8562,856 questions. The signature is detectable retrospectively without instrumentation: within groups sharing code version, n and seed, score rises monotonically with run order in 6 of 6 groups on our most entity-dense dataset (p≈0.0006p≈ 0.0006), with no such pattern where cross-item entity reuse is lower. Correcting it moved one dataset 0.2949→0.22820.2949→ 0.2282—downward, the direction sanity-checking never catches. Detection: query the store with the evaluation questions before running; any hit above threshold is contamination. C3 — Only the named axis varied. Our original ablation switch disabled memory and moved execution from a service to in-process and changed the evidence source and the dependency set. The cost is concrete: an uncorrected MMLU-Pro pair read +0.370+0.370, a 3737-point apparent memory win produced by an option-letter extraction step fixed in one arm and not the other; with both arms on the same parser the cell reads +0.010+0.010. A subtler instance is time: execution was systematically order-confounded—the memory-enabled arm ran first in all eight pairs—while our serving stack slowed ≈3.5×≈3.5× across a multi-day campaign. Within a pair the arms ran at the same speed, but memory-off failed more: 20 orchestrator give-ups to 1, client timeouts level at 3 (Appendix A.2); with order and arm confounded, the record cannot say why. Detection: diff the arms’ serialised requests—they must differ in exactly one field—then counterbalance order, since a diff cannot see time. C4 — The effect exceeds the measurement floor. Pooled within-configuration standard deviation—across replicate groups sharing code version, n and seed—is 0.0460.046 and 0.0520.052 on our two multi-hop datasets, giving 95% bands of ±0.09± 0.09–0.100.10; every architecture-vs-baseline delta we had been prepared to quote was smaller. Detection: estimate the band from replicates, not a within-run bootstrap, which measures sampling error only and understated our uncertainty twofold (Bouthillier et al., 2021). Under C2’s reset protocol the band is far tighter (Appendix A.4). Why trust a paper reporting four confounds not to have a fifth? The corrections moved cells both ways—two up, one down, two reverted—the signature of measurement, not tuning (Appendix A). 6 Related work Agent decomposition (Zhang et al., 2024) and retrieval augmentation (Lewis et al., 2020) are established responses to context limits; we combine both with a persistent tier, and both motivate decomposition by accuracy, not memory. KV cache management (Kwon et al., 2023) optimises the cache for a given workload rather than asking whether an architectural choice changes its peak demand—complementary to our question. The agent-memory literature (§4) establishes that multi-session regimes are required, not what happens outside one—our subject. Our bands follow Bouthillier et al. (2021); C2’s contamination channel has no counterpart in single-model benchmarking. 7 Limitations and Future Work Single system, single team; the conditions are general in aim, not frequency, and reconstructed rather than derived. Bands rest on 2–4 replicates and §2’s Flat arm is single-run. Future work: replicate every §2 arm to RAG’s three runs per cell, a context-length scaling study, and §4’s regime—where agent memory’s accuracy becomes testable, not merely untested. References Bai et al. (2024) Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, and J. Li LongBench: a bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, p. 3119–3137. External Links: Link Cited by: §3. Bouthillier et al. (2021) X. Bouthillier, P. Delaunay, M. Bronzi, A. Trofimov, B. Nichyporuk, J. Szeto, N. Mohammadi Sepahvand, E. Raff, K. Madan, V. Voleti, S. Ebrahimi Kahou, V. Michalski, T. Arbel, C. Pal, G. Varoquaux, and P. Vincent Accounting for variance in machine learning benchmarks. In Proceedings of Machine Learning and Systems (MLSys), Vol. 3. External Links: Link Cited by: §5, §6. Ho et al. (2020) X. Ho, A. Duong Nguyen, S. Sugawara, and A. Aizawa Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics (COLING), Barcelona, Spain (Online), p. 6609–6625. Note: Introduces the 2WikiMultihopQA dataset External Links: Document, Link Cited by: §3. Kwon et al. (2023) W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP), p. 611–626. External Links: Document Cited by: §1, §6. Lewis et al. (2020) P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. Kiela Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: Table 1, §2, §6. Maharana et al. (2024) A. Maharana, D. Lee, S. Tulyakov, M. Bansal, F. Barbieri, and Y. Fang Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, Bangkok, Thailand, p. 13851–13870. External Links: Document, Link Cited by: §4. Rein et al. (2023) D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman GPQA: a graduate-level google-proof Q&A benchmark. arXiv preprint arXiv:2311.12022. External Links: Document, Link Cited by: §A.7, §3. Trivedi et al. (2022) H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal MuSiQue: multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics (TACL) 10, p. 539–554. External Links: Link Cited by: §3. Wang et al. (2024) Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen MMLU-Pro: a more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, External Links: 2406.01574, Link Cited by: §A.7, §3. Wu et al. (2025) D. Wu, H. Wang, W. Yu, Y. Zhang, K. Chang, and D. Yu LongMemEval: benchmarking chat assistants on long-term interactive memory. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §4. Yang et al. (2018) Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning HotpotQA: a dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), Brussels, Belgium, p. 2369–2380. External Links: Document, Link Cited by: §3. Zhang et al. (2024) Y. Zhang, R. Sun, Y. Chen, T. Pfister, R. Zhang, and S. Ö. Arik Chain of agents: large language models collaborating on long-context tasks. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2406.02818, Link Cited by: §1, §6. Appendix A Appendix This appendix records what is needed to judge the measurements in the main text. It is deliberately confined to protocol, statistics, per-cell numbers and the schematic query flow needed to read them. Agent prompts, the controller implementation and the typed state schema carried between agents are the subject of a companion paper and are not reproduced here; Appendix A.1 gives only the block-level structure the ablation in §2–4 already refers to in prose. Per-sample outputs, the isolation audit, the ablation probe and all interval code will be released with the paper. A.1 Reproducibility and implementation The controller runs six nodes—Decomposer, Retriever, Reasoner, Critic, Synthesiser and a separate answer critic—over the three memory tiers named throughout this paper: L1, a flat, ephemeral per-call context window (not to be confused with the Flat baseline arm in §2, which is an entire inference pipeline rather than a memory tier); L2, a Redis-backed key-value store carrying typed Communication Units between agents; and L3, a Qdrant vector database performing both evidence filtering and, when enabled, persistent trace recall and write. Inference is served from two heterogeneous edge accelerators rather than a single data-center GPU: an NVIDIA Jetson Orin Nano Super (8 GB) running llama.cpp with a Qwen3.5-2B parameter model, and an NVIDIA DGX Spark running vLLM with Qwen3.5-35B-A3B-FP8 (35B total parameters, ≈3≈3B active). Its hybrid attention keeps a growing KV cache on only 10 of 40 layers, giving the 2×10×2×256×12× 10× 2× 256× 1 B (fp8) = 10,240=\,10,240 B/token used to convert every KV figure in this paper. Both are on-premises edge devices rather than a cloud API—the deployment regime the peak-KV argument in §2 is meant to address. The Reasoner and Synthesiser run on the vLLM node; the Decomposer, Retriever, Critic and answer critic run on the Jetson’s smaller model, whose per-token KV footprint is lower. Our accumulator records the peak call’s size but not which agent issued it, so every call is converted at the vLLM model’s geometry. That can only overstate our own peak—a Jetson call scored at the larger model’s geometry is counted heavier than it was—and the Flat and RAG baselines’ single generation call runs on the vLLM node in any case. The codebase—agents, controller, memory tiers, and the benchmarking and audit tooling used throughout this paper—will be released as an open-source repository. Peak KV is not device occupancy. Every peak-KV figure counts the attention cache of one call, on the 10 full-attention layers above. It leaves out the per-sequence state the other 30 (linear-attention) layers carry, which the 10,24010,240 B/token constant does not include; the rounding of cache allocation to fixed-size pages; the model weights; and the serving engine’s limit on concurrent sequences, which can bind before the KV cache does. A lower peak KV is therefore headroom in the KV cache, not a proportional gain in sequences per device. QueryDecomposerproduces sub-questionsRetrieversearch_reranked (L3)Reasonersearch_reranked; search_outputs recall if kv_enabled;then store_qkv reasoning_loop_n if kv_enabledCriticscore vs. thresholdSynthesisersearch_reranked; then store_qkv final_answer if kv_enabledAnswer criticFinal answeracceptreject:re-reasonreject:re-retrieve Query path through the controller. One query’s path from arrival to final answer, drawn from the controller rather than from a narrative summary. Three details matter for reading the ablation and are easy to get wrong. First, search_reranked is not a single step at the Retriever: it runs again inside the Reasoner and again inside the Synthesiser, and it is ungated at all three. Second, store_qkv is not a terminal step—it fires once per Reasoner loop (key reasoning_loop_n) and once more at synthesis (key final_answer), so a query with two Reasoner loops writes three traces, not one. Third, the Critic is a router, not a gate: on rejection it can return to the Reasoner or back to the Retriever, and a separate answer critic runs after synthesis. Only the two kv_enabled operations—search_outputs and store_qkv—differ between CoA-PKV and CoA-NoKV; every other box above executes identically in both arms. Ablation isolation protocol A cell is admitted only if both arms satisfy all of the following. The checks are mechanical and are applied by a script over the stored run records, not by inspection. 1. The memory-disabled arm was produced by the server-side request field, not by any client-side switch. Provenance is recorded per run; an unreadable record counts as failing, since guessing “clean” is how a confounded run re-enters. 2. Both arms record the same dataset, subset, n, seed, evidence-scoping flag and service endpoint. 3. The reference arm has memory enabled, so the pair has contrast. 4. Stored traces were cleared before each arm, and the evidence pool was not. Clearing evidence would drive retrieval precision toward 1.01.0 and collapse the axis under study; clearing traces is what prevents the contamination in condition C2. 5. Isolation was verified live in the same window: a probe issues one query with memory enabled and one disabled, and requires the recall path to appear in the first and be absent from the second. A one-directional check is insufficient, because an unconfigured store also produces no recall. Runs failing any check are reported as gaps rather than numbers. Eight cells pass and are pooled in Table 3; LB-All passes but is reported separately because it overlaps three of them. The remainder are excluded rather than mixed in. A.2 Per-cell results Paired per question, memory-enabled minus memory-disabled. Each delta carries its own n: accuracy pairs every question both arms scored (failed questions score 00, so they are included), while peak KV pairs only questions for which both arms recorded a cache figure—a failed question has no peak to compare, so the KV n is lower. Table 3: Per-cell ablation results. cell n Δ peak KV (MiB) 95% CI n Δ accuracy 95% CI HotpotQA 96 +0.321+0.321 [+0.145,+0.504][+0.145,+0.504] 100 +0.002+0.002 [−0.026,+0.029][-0.026,+0.029] GPQA 89 +0.640+0.640 [+0.083,+1.217][+0.083,+1.217] 100 +0.090+0.090 [+0.010,+0.170][+0.010,+0.170] MMLU-Pro 98 +0.748+0.748 [+0.314,+1.189][+0.314,+1.189] 100 +0.010+0.010 [−0.060,+0.080][-0.060,+0.080] LB-HotpotQA 95 +0.028+0.028 [−0.161,+0.226][-0.161,+0.226] 100 +0.029+0.029 [−0.015,+0.077][-0.015,+0.077] LB-2WikiMultihopQA 98 +0.221+0.221 [−0.018,+0.492][-0.018,+0.492] 100 +0.007+0.007 [−0.031,+0.047][-0.031,+0.047] LB-MuSiQue 99 +0.283+0.283 [−0.049,+0.635][-0.049,+0.635] 100 +0.004+0.004 [−0.037,+0.045][-0.037,+0.045] NarrativeQA 95 +0.071+0.071 [−0.198,+0.344][-0.198,+0.344] 96 +0.004+0.004 [−0.027,+0.033][-0.027,+0.033] Qasper 95 +0.628+0.628 [+0.408,+0.859][+0.408,+0.859] 98 −0.029-0.029 [−0.066,+0.002][-0.066,+0.002] across cells +0.368+0.368 [+0.167,+0.590][+0.167,+0.590] +0.015+0.015 [−0.011,+0.046][-0.011,+0.046] LB-All (not pooled) 82 −0.007-0.007 [−0.236,+0.230][-0.236,+0.230] 100 +0.071+0.071 [+0.017,+0.132][+0.017,+0.132] Why LB-All is reported but not pooled. It is the harness’s combined LongBench setting, drawing items from the same three subsets that already appear as LB-HotpotQA, LB-2WikiMultihopQA and LB-MuSiQue. The cluster bootstrap treats cells as independent units, so pooling it would resample the same questions under two labels and understate the interval. The exclusion follows from that overlap alone. It is nonetheless the sharpest illustration of condition C3 in our data. Its arms differ by 1 (memory-on) versus 17 (memory-off) failed questions—the largest asymmetry we recorded. The 17 are connections refused on the run’s last questions, during an orchestrator outage in the second arm; they never reached the pipeline. Since failed questions score zero, its accuracy delta reads +0.071+0.071 under our primary convention and −0.006-0.006 with every question either arm failed dropped. Had we pooled it, the across-cell accuracy interval would have shifted to [+0.001,+0.065][+0.001,+0.065] and excluded zero: a marginally significant memory benefit produced entirely by a serving outage during one arm. We record this because it is precisely the failure the conditions are meant to catch, and it would have been easy to accept. Among pooled cells the GPQA delta is the largest and also carries the most extreme failure asymmetry (1 memory-on versus 11 memory-off, all orchestrator give-ups); dropping every question either arm failed reduces it to +0.023+0.023, and the across-cell figure from +0.015+0.015 to +0.003+0.003. This is the convention dependence noted in the main text; we report the failures-score-zero figure as primary because it is the convention under which the arms are scored identically, and disclose the other because the difference is not negligible. What the record calls a timeout. The per-question flag both conventions read marks every failed question, whatever the cause. Over the eight pooled pairs’ 794794 paired questions, memory-on and memory-off fail 77 and 2323 times: 33 and 33 client timeouts (900900 s), 11 and 2020 orchestrator give-ups (HTTP 500, “Pipeline returned no answer after 3 attempts”, with no pipeline step or cause recorded), and 33 and 00 server disconnects. The memory-off give-ups all sit in HotpotQA, GPQA and LB-HotpotQA, run on 20 August: memory-on runs finished 09:08–12:42 UTC, memory-off runs 17:36–21:24. Within every pair the arms ran at the same speed (median per-question latency ratio, memory-off over memory-on, on questions both answered: 1.001.00–1.031.03), so the campaign’s ≈3.5×≈3.5× slowdown does not explain the gap. With memory-on always first, and on that day also hours earlier, the record cannot say whether the arm, the order or the time of day caused it; only a counterbalanced repeat can. A.3 Intervals Deltas are paired per question rather than computed from run-level means. Question difficulty varies far more than the ablated component does, and pairing removes that variance; matching is on question text rather than row position, since a dropped sample would otherwise silently pair question i against question i+1i+1. Per-cell intervals are percentile bootstrap over the paired deltas, B=104B=10^4. The across-cell interval is a two-stage cluster bootstrap: resample cells with replacement, then questions with replacement within each drawn cell. Pooling all questions into one sample would treat several hundred correlated observations as independent and understate the interval, because questions within a cell share a dataset, a prompt format and a difficulty profile. The cell is the independent unit. What the paired columns cover. The two columns of Table 3 are not computed over the same set, so each carries its own n. Accuracy pairs every question both arms scored—a failed question scores 00 and is included, which is the failures-score-00 convention §3 quotes—so n is 100100 except where a question is absent from a run entirely. Peak KV pairs only questions for which both arms recorded a cache figure; a failed question produces no peak, so its KV n is 100100 minus roughly each cell’s failure count. The released kv_layer_cost_ci.py prints the resolved run ids and both per-cell n values, so the set behind every column is re-derivable rather than taken on trust. Peak KV is the largest single call’s (prompt+generated)(prompt+generated) footprint within a query, not the sum over calls. The sum answers a different question—total tokens processed—and is roughly four times larger; using it would overstate the resource being claimed. Both are recorded per query. A.4 Reproducibility bands Bands are estimated from replicate groups: runs sharing code version, n and seed, so that neither code nor sample composition can contribute to the spread. Pooled within-group standard deviation is 0.0460.046 across six groups on MuSiQue and 0.0520.052 across four groups on 2WikiMultihopQA, giving 95% bands of ±0.09± 0.09 and ±0.10± 0.10. Group sizes are two to four runs, so the bands are themselves imprecise; they are sufficient to establish that deltas below 0.060.06 are not resolvable, which is the use we make of them, and not to quote to three digits. These MuSiQue groups serve twice: they estimate the band above and supply condition C2’s run-order test below. If run order moves scores within a group, the within-group spread absorbs that movement, so the band is inflated rather than tight—the conservative direction, given that we use it only to rule deltas unresolvable. The run-order test in condition C2 compares each group’s scores in chronological order. On MuSiQue all six groups are strictly increasing. With group sizes (2,3,3,2,3,2)(2,3,3,2,3,2) and orderings assumed uniform, the probability of that pattern arising by chance is (1/2!)3(1/3!)3≈5.8×10−4(1/2!)^3(1/3!)^3≈ 5.8× 10^-4. The comparison dataset shows one of four, consistent with chance. Post-submission measurement. The bands above are the historical spread, from replicate groups that predate the trace-reset protocol of condition C2. Re-measuring the same two datasets under the protocol—four replicates each, fixed seed, traces reset between runs—gives a within-configuration standard deviation of 0.0030.003 (MuSiQue) and 0.0070.007 (2WikiMQA), an order of magnitude tighter, and on a subsequent repair of the serving stack the four replicates of each were identical (σ=0σ=0). As 95% bands that is ±0.006±0.006–0.0140.014 rather than ±0.09±0.09, comfortably inside the ≈0.046≈0.046 bound of §3. The floor quoted in C4 is therefore an upper bound inherited from the contaminated regime, not a property of the measurement: under the protocol, run-to-run drift on these datasets is not the binding uncertainty. We report the historical figure because it is what the deltas in this paper were judged against, and the tighter one because it is what a reader reproducing the work should expect. What the corrections cost The obvious objection to a paper reporting four confounds is that a fifth may remain. Our evidence against a systematic residue is that the corrections moved cells in both directions—the signature of measurement rather than tuning. Two corrections raised scores, one lowered them, and two intended improvements were regressions we reverted. Table 4: What the corrections cost. correction effect output schema + aligned prompt +0.145+0.145 scoped retrieval +0.082/+0.032/+0.045+0.082/+0.032/+0.045 trace-reset protocol −0.067-0.067 disabling reasoner CoT (reverted) −0.209-0.209 lexical reranker (reverted) −4-4 of 40 gold-in-top-5 A separate reversal came from sample size alone, with no defect involved: a comparison reading 0.704>0.6140.704>0.614 at n=50n=50 read 0.622<0.6460.622<0.646 at n=100n=100. We include it because it is the same category of error—evidence that looked settled and was not. A.5 Recall reachability Reproduces the retriever’s own predicate rather than proxying it: search_outputs returns stored traces within cosine 0.60.6 of the current question under sentence-transformers/all-MiniLM-L6-v2. Because resets clear the store between conditions and items are processed in a fixed order, a question can only reach traces written earlier in its own run; we count those. A count of zero means recall provably returned nothing, which is what makes the unreachable group a control rather than a comparison. This analysis covers the six multi-hop and multiple-choice cells it was originally computed on, plus LB-All; NarrativeQA and Qasper, added to the ablation later, are not re-embedded here. The reachability rate in §4 is over the six cells (136136 of 575575 answered questions reachable as-sent); its effect split also counts LB-All’s two reachable questions, giving n=138n=138. Neither covers all eight cells in Table 3. Table 5: Recall reachability by cell. cell as-sent content-only Δ reachable Δ unreachable HotpotQA 0% 0% — −0.006-0.006 GPQA 27% 24% +0.087+0.087 +0.000+0.000 MMLU-Pro 86% 86% −0.012-0.012 +0.000+0.000 LB-HotpotQA 2% 2% +0.000+0.000 +0.017+0.017 LB-2WikiMultihopQA 10% 1% −0.045-0.045 +0.019+0.019 LB-MuSiQue 16% 7% −0.088-0.088 +0.022+0.022 LB-All 2% 2% −0.400-0.400 +0.004+0.004 As-sent versus content-only. Multiple-choice datasets wrap every item in identical instruction text—the stored query for MMLU-Pro is 1,499 characters opening with the same preamble for all 100 items—and that boilerplate carries enough embedding mass to clear the threshold on its own. Stripping the prefix and suffix shared by every question in a cell halves the apparent reachability of LB-2WikiMultihopQA (10%→1%10\%→ 1\%) and LB-MuSiQue (16%→7%16\%→ 7\%): roughly half their recall was firing on template text. “Reachable” should therefore be read as the retriever would have returned something, not as something useful was available. MMLU-Pro is unchanged at 86%86\% because its items genuinely share subject vocabulary, and it is also the cell where recall fires most and still does not help (−0.012-0.012). Two caveats. The reachability figure is threshold- and embedding-dependent; at a looser threshold more questions become reachable and the subgroup grows. And LB-All’s −0.400-0.400 rests on two questions—noise, reported only for completeness. Failed questions are excluded throughout, since they score zero regardless of memory and their cross-arm asymmetry is confounded with order (Appendix A.2). A.6 What is not reported here Three things are measured but deliberately outside this paper’s scope: the full accuracy comparison against the retrieval-matched baseline, the cumulative-traffic and throughput characterisation, and the architecture’s design rationale. The first two are summarised in §2 only to the extent needed to establish that the decomposition delivers a bounded working set, which is the premise of the memory question. A context-length scaling study, which would connect the working-set result to an asymptotic cost model, has not been run and is stated as a limitation rather than claimed. A.7 Per-dataset KV measurement, apples to apples §2 quotes one aggregate triple (Flat/RAG/CoA-PKV). This section breaks the same comparison out per dataset, using the seed-4242, n=100n=100 (or n=99n=99 where one item failed) slice of the runs behind §2, so every entry below comes from the same question set run under the same model. These per-dataset figures need not reproduce §2’s aggregate exactly—§2 pools two of these cells and averages each arm’s replicates (RAG over seeds 42/43/44, CoA-PKV over two runs) while this table is seed-4242 only—but both are computed identically: peak KV is the largest single call’s (prompt+generated)(prompt+generated) footprint, converted at the served model’s 10,24010,240 B/token KV geometry (Appendix A.1). Table 6: Per-dataset KV measurement, apples to apples. dataset Flat RAG CoA-PKV vs Flat vs RAG est/peak‡ peak KV (MiB) (CoA-PKV) HotpotQA 35.00 30.18 13.59 −61.2%-61.2\% −55.0%-55.0\% 12.2×12.2× GPQA 42.13 n/a∗ 19.89 −52.8%-52.8\% — 8.9×8.9× MMLU-Pro 36.76 n/a∗ 21.92 −40.4%-40.4\% — 6.3×6.3× LB-HotpotQA 36.09 40.06 15.49 −57.1%-57.1\% −61.3%-61.3\% 11.1×11.1× LB-2WikiMultihopQA n/a† 38.84 14.34 — −63.1%-63.1\% 9.1×9.1× LB-MuSiQue n/a† 44.12 14.55 — −67.0%-67.0\% 9.0×9.0× ∗RAG does not apply: GPQA and MMLU-Pro are closed-book multiple choice with no external passage pool to retrieve against (Rein et al., 2023; Wang et al., 2024). †Not a null result: KV-cost instrumentation was not yet wired into the Flat backend for these two LongBench subsets at the time these particular runs were made, and no alternate seed-4242 Flat run with the instrumentation exists for either. We report the gap rather than substituting a number from a different seed or a different dataset. Every reduction above is real regardless: it holds against whichever baseline (Flat, RAG, or both) is actually instrumented for that dataset. Measured, not estimated. The ‡ column is the ratio between each query’s pre-run token estimate—summing (input+max-output)(input+max-output) tokens declared for every call in the query, before any call executes—and its measured peak, at the same 10,24010,240 B/token geometry. For CoA-PKV this ratio is 6.36.3–12.2×12.2×: most of the ten-or-so calls a decomposed query makes generate far below the budget declared for them, and the estimate has no way to know this in advance, while the peak is whichever single call actually turned out largest. The same ratio computed identically for Flat and RAG, which make one to two calls per query, is 1.11.1–1.7×1.7× across these same six datasets. Both ratios come from the same completions that produced the peak and cumulative figures elsewhere in this paper—not from a separate formula—so the gap between them is direct evidence that the 2.5×2.5× reduction in §2 reflects what the served model actually generated for these questions, not a bound computed before any query ran. A.8 Detailed conversation log: good and weak scenarios Table 3 answers whether memory helps on average; it cannot show what one question’s trace looks like, which is where §4 is easiest to misread. This section works through the only HotpotQA triple sharing seed 4242 across CoA-PKV, CoA-NoKV and RAG (r720/r724/r692), so all three answer the identical draw—one question every arm answers correctly, and one every arm gets wrong. Good scenario (accept): parity, not a memory effect. Question: “My Neighbor Totoro was produced by a Japanese animation film studio founded in what year?” Gold: 1985. All three arms agree with gold. CoA-PKV and CoA-NoKV cost nearly identical KV—the recall path found nothing to recall on this question (a matter of question order and topic, not architecture; see Appendix A.5). Query → Decomposer: “My Neighbor Totoro was produced by a Japanese animation film…”Decomposer → L3 (Qdrant): retrieve evidence for “production studio”L3 → Reasoner: passages (+ recalled trace if kv_enabled)Decomposer → L3: retrieve evidence for “founding year”L3 → Reasoner: passages (+ recalled trace if kv_enabled)Reasoner: reason, loop with CriticReasoner → Synthesiser: distilled conclusionSynthesiser → Query: ‘1985’ arm answer F1F_1 elapsed (s) calls peak/cum. KV (MiB) prompt/gen (tok) CoA-PKV 1985 1.00 25.8 10 16.3 / 73.3 6,624 / 880 CoA-NoKV 1985 1.00 25.5 10 16.8 / 72.7 6,532 / 908 RAG 1985 1.00 24.5 2 21.4 / 24.0 1,154 / 1,302 Reasoning trace (CoA-PKV and CoA-NoKV are identical on this question). sub-question: production studio—“[loop 1] The original question asks for the founding year of the Japanese animation film studio that produced ‘My Neighbor Totoro’. From Evidence [1], I can confirm that ‘My Neighbor Totoro’ was produced by Studio Ghibli. From Evidence [5], I can find the specific founding details of Studio Ghibli, which states it was founded on 15 June 1985. Combining these facts, the studio is Studio Ghibli and its founding year is 1985.” sub-question: founding year—“[answer_critic loop 1] score=0.00 threshold=0.70 — needs improvement: Answer is factually incorrect.” This second entry is not a fresh answer attempt: it is the answer-critic’s verdict on the first, correct reasoning pass, logged into the same list. The Reasoner answered both sub-questions correctly in its first pass; the Critic rejected it once before the pipeline nonetheless emitted the correct final answer (F1=1F_1=1). A critic that scores a correct answer as “factually incorrect” and is overridden downstream is genuine system behaviour, not a rendering artifact, and is worth its own investigation. Weak scenario (reject): three different failure mechanisms behind one metric. Question: “Who used a Barrack buster to shoot down a British Army Lynx helicopter”. Gold: IRA. This is not a memory-ablation story: all three reasoning traces independently converge on the correct entity, and each arm then fails for a different reason—exactly what a single F1F_1 number cannot show. Query → Decomposer: “Who used a Barrack buster to shoot down a British Army Lynx…”Decomposer → L3 (Qdrant): retrieve evidence for “Who”L3 → Reasoner: passages (+ recalled trace if kv_enabled)Reasoner: reason, loop with CriticReasoner → Synthesiser: distilled conclusionSynthesiser → Query: ‘Seán Kelly’ arm answer F1F_1 elapsed (s) calls peak/cum. KV (MiB) prompt/gen (tok) CoA-PKV Seán Kelly 0.00 31.2 10 15.6 / 70.5 6,090 / 1,132 CoA-NoKV Provisional IRA 0.00 33.4 10 16.8 / 71.2 6,024 / 1,264 RAG Provisional IRA 0.00 76.3 2 46.9 / 49.5 931 / 4,140 Reasoning trace (CoA-PKV and CoA-NoKV are identical on this question). sub-question: Who—“[loop 1] The evidence provides specific details about the incident. Evidence [1] states that on 20 March 1994, a British Army Lynx helicopter was shot down by the Provisional Irish Republican Army (IRA) in Northern Ireland. It specifies that a unit of the IRA’s South Armagh Brigade fired an improvised mortar at the British Army base in Crossmaglen. Evidence [5] defines ‘Barrack buster’ as the colloquial name for several improvised mortars developed in the 1990s by the engineering group of the Provisional Irish Republican Army (IRA). Combining these two pieces of evidence, the ‘improvised mortar’ mentioned in the 1994 shootdown incident is the ‘Barrack buster’ described in the definition. Therefore, the entity that used the Barrack buster to shoot down the helicopter was the Provisional Irish Republican Army (IRA), specifically the South Armagh Brigade unit.” All three reasoning traces converge on the same correct entity (the IRA) before answering; the failures happen downstream and are not the same failure. CoA-PKV’s Synthesiser diverges from its own correct reasoning and outputs an unrelated name (Seán Kelly). CoA-NoKV and RAG both produce the semantically right answer, Provisional Irish Republican Army, in a surface form that token-F1F_1 against the gold string IRA scores at zero. One F1=0F_1=0 collapses three distinct failure modes into the same number, which is the case for reading per-sample traces before trusting an aggregate.

Instant research discovery