Paper deep dive
Bookkeeping, Composition, or Unreachable Gold? Reading MemoryAgentBench's Conflict-Resolution Scores Against a Frozen Last-Write Resolver
Egor Pakhomov, Erik Nijkamp
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 10/8/2026, 7:10:38 AM
Summary
This paper critiques the MemoryAgentBench Conflict Resolution split, arguing that its scores for 'selective forgetting' are largely driven by simple last-write resolution rather than complex memory capabilities. The authors implement a frozen, zero-learning resolver that answers 80.25% of questions correctly by simply keeping the newest fact. They identify 67 items where the benchmark's gold answer contradicts the last-write rule (e.g., an overwritten fact is the correct answer), which the resolver fails. Experiments with Gemini 2.5 Pro, Flash, and a BM25 agent show that model performance drops significantly on these 'ambiguous' items compared to items solvable by the last-write rule, suggesting the benchmark measures reachability and parser scope more than true memory composition.
Entities (10)
Relation Signals (8)
Erik Nijkamp → affiliatedwith → Salesforce AI Research
confidence 99% · Erik Nijkamp Affiliation: Salesforce AI Research
Egor Pakhomov → affiliatedwith → Salesforce AI Research
confidence 99% · Egor Pakhomov Affiliation: Salesforce AI Research
Gemini 2.5 Pro → testedon → MemoryAgentBench
confidence 99% · We ran Gemini 2.5 Pro on all 800 questions under the benchmark’s long-context prompt
Gemini 2.5 Flash → testedon → MemoryAgentBench
confidence 99% · Gemini 2.5 Flash reproduces the class ordering
MemoryAgentBench → hassplit → Conflict Resolution
confidence 98% · MemoryAgentBench’s Conflict Resolution split is read as measuring “selective forgetting”.
Last-Write Resolver → achievesaccuracyon → MemoryAgentBench
confidence 95% · Under the official metric the rule answers 80.25% of the questions
BM25 Agent → testedon → MemoryAgentBench
confidence 95% · our pre-registered approximate re-implementation of the benchmark’s BM25 agent... score 41.6%
StateMemBench → relatedto → MemoryAgentBench
confidence 85% · The closest work, StateMemBench [16]... we ask the converse of an existing split
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:MemoryAgentBench's Conflict Resolution split is read as measuring "selective forgetting". We execute the benchmark's own rule - the newest statement about a fact wins - as a zero-learning resolver frozen on one of the four fact lists. Under the official metric the rule answers 80.25% of the questions (74.5% on the three held-out lists). Of the rest, 67 items have a released gold that the last-write graph cannot reach but overwritten statements would ("The capital of India is New Delhi." superseded by "The capital of India is Grosseto."; gold New Delhi); such items are a third of the multi-hop questions at 262K. Two long-context models and our pre-registered approximate re-implementation of the benchmark's BM25 agent, one retained run per item and outcomes only, score 84.7%, 82.6% and 41.6% on the items the rule solves against 10.4%, 11.9% and 6.0% on those 67. The failures are a reachability split plus a small parser-scope residual; the per-item split, not the aggregate, is the unit at which a score here can be read.
Tags
Links
- Source: https://arxiv.org/abs/2610.09193v1
- Canonical: https://arxiv.org/abs/2610.09193v1
Trouble viewing inline? Open PDF directly →
Full Text
78,591 characters extracted from source content.
Expand or collapse full text
Interpreting Agent Behavior (IAB) Bookkeeping, Composition, or Unreachable Gold? Reading MemoryAgentBench’s Conflict-Resolution Scores Against a Frozen Last-Write Resolver Egor Pakhomov Affiliation: Salesforce AI Research Email: epakhomov@salesforce.com Erik Nijkamp Affiliation: Salesforce AI Research Email: erik.nijkamp@salesforce.com Abstract MemoryAgentBench’s Conflict Resolution split is read as measuring “selective forgetting”. We execute the benchmark’s own rule—the newest statement about a fact wins—as a zero-learning resolver frozen on one of the four fact lists. Under the official metric the rule answers 80.25% of the questions (74.5% on the three held-out lists). Of the rest, 67 items have a released gold that the last-write graph cannot reach but overwritten statements would (“The capital of India is New Delhi.” superseded by “The capital of India is Grosseto.”; gold New Delhi); such items are a third of the multi-hop questions at 262K. Two long-context models and our pre-registered approximate re-implementation of the benchmark’s BM25 agent, one retained run per item and outcomes only, score 84.7%, 82.6% and 41.6% on the items the rule solves against 10.4%, 11.9% and 6.0% on those 67. The failures are a reachability split plus a small parser-scope residual; the per-item split, not the aggregate, is the unit at which a score here can be read. 1 Introduction Once a label such as “selective forgetting” is attached to a benchmark number, a low score reads as a missing capability and a gap between agents as a behavioral difference; whether that reading is licensed is a property of the item set and scoring rule, not of the agent [2, 3]—a construct-validity question [4] that audits answer with trivial baselines and item-level log reading [5, 6]. Partial-input and shallow-cue baselines [7, 8, 9, 10, 11] and Agentless [12] measure how far a system lacking the purported skill gets; multi-hop QA is often answerable without the composition it names [13, 14, 15]; we apply the move to an agent-memory competency. The closest work, StateMemBench [16], builds state-tracking probes with replay-defined gold whose traps defeat “lazy reader” policies—trust-most-recent first, the rule executed here; we ask the converse of an existing split: how much of it that policy already solves. Behavioral taxonomies read trajectories with an LLM coder [17]; we read items and outcomes. This paper is a benchmark-score validity instrument; its behavioral evidence is outcome-only item-level runs of two long-context models and our pre-registered approximate re-implementation of the benchmark’s BM25 configuration; memory systems with learned or structured stores were not run (Limitations), so for them the reinterpretation is by analogy. MemoryAgentBench [1] evaluates memory-equipped agents on four competencies; its Conflict Resolution split (dataset FactConsolidation) presents a numbered list of fact statements, later ones contradicting earlier ones, and asks single- and multi-hop questions about the final state. The prompt states the rule: “Each fact in the knowledge pool is provided with a serial number at the beginning, and the newer fact has larger serial number. You need to solve the conflicts of facts in the knowledge pool by finding the newest fact” (v1, App. B, Fig. 5). The rule is executable—parse, keep the greatest position per (subject, relation) key, answer by lookup or composition—so wherever it matches gold a handcrafted zero-learning implementation suffices (bookkeeping plus our grounding and path heuristics): the item operationalizes forgetting as explicit last-write state tracking and bounds what more it demands; where it fails, an audit of why tells what else the item demands. Contributions. (i) A frozen zero-learning last-write resolver under the official metric with a pre-declared held-out split, beside the cited published scores (§2–3.2). (i) A labeling of every resolver failure under a documented six-category rule (categories fixed in the study’s pre-execution specification; ordering and operationalization documented at labeling time, App. A), separating bookkeeping, misses on items whose gold the last-write graph reaches, and items whose gold it does not reach (§3.3). (i) Two pre-registered outcome-only experiments (two long-context models; one retrieval agent) over all 800 items, their per-item outcomes aligned to those labels (§3.4). Neither (i) nor (i) appears in the source paper or its repository. Model-facing result: Table 1’s stratification of three systems’ per-item accuracy by these labels; interpretive: what a “selective forgetting” score can be read as measuring; we claim nothing about mechanism (one retained run per item, no trace read) or memory systems in general. 2 Setup Corpus. We use the released Conflict Resolution parquet (App. G): 8 rows, 800 gold-answerable questions, but only 4 distinct context strings—one fact list per tier (455 / 2,310 / 4,580 / 18,332 statements; 6K–262K after the benchmark), each appearing once with a single-hop and once with a multi-hop question set. Parser and resolver. The parser is a fixed list of 35 anchored full-line patterns over the numbered statements, mapping to 34 relations (App. C); a statement yields a tuple only when exactly one pattern matches; per (subject, relation) key the greatest-position tuple is kept (last write). A question is answered by literal grounding of the question entity and a graph path of at most six edges scored by fixed lexical cues and penalties; otherwise the output is ABSTAIN. Resolver execution and metric scoring make no LLM API calls. Freeze protocol. Parquet rows 0 and 4 were declared the development set in the pipeline’s freeze record—an internal freeze (a hashed, timed specification file; no third-party registry)—and were the only rows whose content was exported before it; they are the same 6K fact list, so the parser was developed on one of the four fact lists, frozen (App. G), and only then run on the other three and on all 8 question sets unchanged. On the three held-out lists the frozen parser parses 24,871 of 25,222 distinct statements (98.6%); the 351 unparsed statements are three forms outside the pattern list (App. C); no independent semantic audit of parses exists (App. G). Scoring. Headline scoring is the official repository’s substring_exact_match (commit fe1735de; App. G), with ABSTAIN counted wrong and strict equality also reported. Intervals are exact Clopper–Pearson [18] at the answer level, conditional on the four lists; the four-list bootstrap (App. D) is a width indicator only; clustered uncertainty is the honest scale here [19]. 3 Results 3.1 What the frozen symbolic pipeline recovers The frozen resolver answers 642/800 = 80.25% of questions correctly (answer-level 95% CI 77.3–83.0; four-list interval 67.3–92.8, App. D), with 32 ABSTAINs; held-out (six question sets), 447/600 = 74.50% (70.8–77.9). Per list (Table 1, Figure 1): single-hop 88–100% at every length; multi-hop 95% on the 6K list, lower on the three larger lists and lowest at 262K, whose multi-hop set contributes 64 of the 158 failures. The overwrite relation is exercised: resolving with the first write per key instead of the last loses 526 of the 642 resolver-solved items (answered differently and scored wrong; all 266 multi-hop); 116 single-hop items are solvable under either policy, and 7 of the 67 ambiguous items have the first write as gold (App. D). Table 1: Resolver accuracy (%) by hop type (SH = single-hop, MH = multi-hop) and fact list (n=100n=100 per list cell); pooled columns carry exact 95% CIs (answer-level, conditional on the four lists; per-list CIs: Fig. 1, App. E); normalized (official) = strict in every cell; 6K = development list. 6K (dev) 32K 64K 262K All four Held-out (32K–262K) Single-hop 100 95 93 88 94.0 (91.2–96.1) 92.0 (88.3–94.8) Multi-hop 95 62 73 36 66.5 (61.6–71.1) 57.0 (51.2–62.7) ABSTAIN (SH / MH) 0 / 0 2 / 7 5 / 4 4 / 10 11 / 21 11 / 21 3.2 Position relative to published scores The tier-matched comparison (App. F, Table 8; Figure 1) is a reading aid, not a leaderboard (whole-list reading vs. chunked ingestion; prompting, retrieval and formatting differ—a harness-versus-model confound [20]): at 262K the resolver’s 88 / 36 exceeds every main-table system of the benchmark’s arXiv v1 (at most 60 / 7; v4, the ICLR camera-ready, changes some rows: App. F), the camera-ready’s GPT-5-mini (78 / 28), and GPT-4.1-mini under its always-prefer-later prompt (Policy A: 36 → 40 SH, 5 → 4 MH, n=100n=100 per cell; the camera-ready’s App. K.2, Table 19). 3.3 The operational split: what the failures demand Figure 1: (Left.) Resolver accuracy per question set (exact 95% CIs) and published points at their tiers (App. F; whole-list reading vs. chunked ingestion, not directly comparable, §3.2). Table 2: Operational split (§3.3; App. A/B/E); cells: accuracy / share of own errors (%), pooled over hop type (last row: accuracy on multi-hop items only, resolver-correct (266) / ambiguous (60)); Pro, Flash = Gemini 2.5; BM25 = our pre-registered approximate re-implementation of the benchmark’s BM25 configuration (512-token chunks, top 10, GPT-4o-mini; §3.4, App. I). Item class (n; share of 158 failures) Pro Flash BM25 Resolver-correct (642) 84.7 / 49.0 82.6 / 54.1 41.6 / 72.8 Last-write-reachable miss (78; 49.4%) 52.6 / 18.5 59.0 / 15.5 12.8 / 13.2 Ambiguous gold, oper. (67; 42.4%) 10.4 / 30.0 11.9 / 28.5 6.0 / 12.2 Parser scope (13; 8.2%) 61.5 / 2.5 69.2 / 1.9 30.8 / 1.7 All (800) 75.0 / 100 74.1 / 100 35.6 / 100 Multi-hop only (266 / 60) 74.4 / 8.3 66.2 / 11.7 11.7 / 0.0 One row (App. B, B5): multi-hop 6K #67 asks for the capital of the country that gave rise to Rogério Ceni’s sport; statement 158 reads “The capital of India is New Delhi.”, statement 341 reads “The capital of India is Grosseto.”, and the released gold is New Delhi—the earlier, overwritten value. Label production. The 158 failures were labeled by a classification script over a row-level evidence workbench: 13 rows were designated as parser-scope failures in an LLM-assisted batch review under the authors’ direction (a required query edge occurs only in an unparsed statement; App. A); the script labeled every other row from the workbench’s last-write reachability flag—reachable → last-write-reachable miss; not reachable → ambiguous gold. Its override lists for the other three categories were left empty; category names were pre-specified; ordering, the reachability operationalization and the 13 designations were fixed during labeling (App. A); 158/158 labeled (Table 1). Every failure’s gold occurs in the context, so the empty category follows from the rule: “genuine beyond last write” can hold only an item whose gold is absent from the context (App. A). Half the residue: last-write-reachable misses. 78 failures (49.4%; 67 multi-hop, 11 single-hop, 22 ABSTAINs) have a gold reachable from a question mention by some last-write path of at most six edges—the flag does not test that the path answers the question—and the resolver did not return it; we call them last-write-reachable misses (mostly, we expect, grounding or path-selection errors of our resolver, which a stronger grounder could reduce; the flag alone does not establish that). 8 of the 78 fail a cue-consistency screen (App. A), one of them (multi-hop 262K #77) clearly. Two-fifths of the residue is operationally ambiguous gold. 67 failures (42.4%; none ABSTAIN) have a released gold that occurs in the context but is not reachable from a question mention in the frozen last-write graph; for all 67 it becomes reachable within six edges once overwritten tuples are re-admitted (base rate 25% for other questions’ golds, App. A)—in 50 the gold-valued tuple itself was overwritten, in 17 only an intermediate one; a row-by-row reading confirms all 67 (App. A). The label is operational (relative to the frozen parser’s graph), not a claim that the released gold is wrong; 3 of the 67 also have the gold inside an unparsed statement (App. B; a fuller parser could relabel two). App. B collects the provenance of these items: MQuAKE and its known conflict issues [21, 22, 23], ripple effects of edits [24], and in-context knowledge conflicts [25, 26]. Any procedure answering from the exact last-write graph is scored incorrect on them (66 of 67 under the substring surface, App. A); 60 of 67 are multi-hop, concentrated at 262K (App. E). The remaining 13 failures (8.2%) need an edge found only in an unparsed statement (App. B). 3.4 Three systems, item by item We ran Gemini 2.5 Pro on all 800 questions under the benchmark’s long-context prompt and assembly (deviations: App. I), one retained scored output per item (pre-registered; outcome stratification, not mechanism evidence); it scores 75.0% (single-hop 90.0%, multi-hop 60.0%; held-out 68.5%; CIs in App. I). Per class (Table 1): 84.7% vs. 10.4% on resolver-solved vs. ambiguous items (8.4% of questions, 30% of its 200 errors). Its largest error block is elsewhere: 98 misses on items the rule solves, 61 at 262K (Flash: 112 and 40)—consistent with, but not a test of, length-dependent degradation [27, 28]. Gemini 2.5 Flash reproduces the class ordering, not the per-tier values (Table 1; κ=0.53κ=0.53 with Pro). Our pre-registered approximate BM25 re-implementation (App. I) scores 35.6% (62.8% single-hop, 8.5% multi-hop); its top-10 chunk set matches the closest official-code variant’s on 134 of 800 items, so 56 / 4 at 262K vs. the published 56 / 3 is only a coarse fidelity check. Per class it follows the same ordering (Table 1): 6.0% (4/67) on the operationally ambiguous items (12% of its errors), 73% of its errors on items the rule solves. For the retrieval agent, 496 of 800 items had every resolver-used statement in the top-10 chunks yet only 262 were answered correctly, and a gold-valued statement was retrieved for 336 of its 515 errors: the resolver’s statements were often retrieved, but with outcomes only the post-retrieval cause cannot be localized. Class membership is confounded with hop type and with length (60 of the 67 ambiguous items are multi-hop, 34 of them at 262K); within multi-hop items, pooled over lengths, resolver-solved vs. ambiguous accuracy is 74.4% vs. 8.3% (Pro, 198/266 vs. 5/60), 66.2% vs. 11.7% (Flash, 176/266 vs. 7/60) and 11.7% vs. 0% (agent, 31/266 vs. 0/60): the pooled gap persists for the long-context models but shrinks to 12 points for the agent; at multi-hop 262K alone it is 4/36 vs. 1/34 for Pro (App. I). Three runs of the retrieval agent under the identical protocol put its accuracy on the operationally ambiguous items at 6.0–9.0% and on resolver-solved items at 40.3–41.7%, with per-item agreement between runs κ = 0.90 (35–37 of 800 items change outcome); the Gemini scores are single runs (App. I). 4 Discussion On this split, single-hop: 94.0% of items (92.0% held-out) are solved by one-edge greatest-position lookup, and the two long-context models solve 90% and 92%. Multi-hop: composition over a growing graph plus about a sixth of the multi-hop half (a third at 262K) whose gold the rule cannot reach, on which accuracy falls to 8.3%, 11.7% and 0% (§3.4); under the documented rule no multi-hop miss was attributed to semantics beyond last write (§3.3)—true of this split, not of conflict resolution in general, for which last-write-defeating probes exist [16]; read multi-hop scores against the ambiguous fraction per tier (App. E). Limitations. (1) Four fact lists, no memory store: intervals are conditional on the four lists (App. D); no memory system with a learned or structured store was analyzed at the item level. (2) Label production: 145 of the 158 labels follow from the reachability flag; the 13 parser-scope rows were designated in an LLM-assisted batch review (GPT-5.6) and are the sole discretionary labels: a re-labeling blind to the frozen labels (flag shown) agreed on 147/158 (κ 0.88 [29]) but on 6 of the 13, so agreement on the 145 measures rule-following; one without the explicit flag (evidence capped and path-seeded; last-write tags visible) agreed on 141/158 (κ 0.82) and recovered 12 of 13; no per-row record of the review is available; the mechanical criterion is post hoc; the frozen labels are unchanged. (3) One split, one rule, one parser: the 35-pattern parser is a lower bound on a symbolic procedure; its heuristics are ours (App. G). (4) Within-corpus held-out: developed on the 6K list; the held-out lists share its templates. Agenda. Align agents’ traces, not just outcomes, with the labels; re-adjudicate the 67 items with the maintainers; test other splits. LLM use: App. H. Acknowledgments and Disclosure of Funding This work was funded by Salesforce. References [1] Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory in LLM agents via incremental multi-turn interactions. In The Fourteenth International Conference on Learning Representations (ICLR), 2026. URL https://openreview.net/forum?id=DT7JyQC3MR. arXiv:2507.05257; v1 (July 2025) used for all quoted numbers unless marked v4. [2] Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. AI and the everything in the whole wide world benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/forum?id=j6NxpQbREA1. [3] Samuel R. Bowman and George E. Dahl. What will it take to fix benchmarking in natural language understanding? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 4843–4855, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.385. URL https://aclanthology.org/2021.naacl-main.385/. [4] Andrew M. Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan, Chris Schmitz, Karolina Korgul, Hunar Batra, Oishi Deb, Emma Beharry, Cornelius Emde, Thomas Foster, Anna Gausen, María Grandury, Simeng Han, Valentin Hofmann, Lujain Ibrahim, Hazel Kim, Hannah Rose Kirk, Fangru Lin, Gabrielle Kaili-May Liu, Lennart Luettgau, Jabez Magomere, Jonathan Rystrøm, Anna Sotnikova, Yushi Yang, Yilun Zhao, Adel Bibi, Antoine Bosselut, Ronald Clark, Arman Cohan, Jakob Foerster, Yarin Gal, Scott A. Hale, Inioluwa Deborah Raji, Christopher Summerfield, Philip H. S. Torr, Cozmin Ududec, Luc Rocher, and Adam Mahdi. Measuring what Matters: Construct Validity in Large Language Model Benchmarks. In Advances in Neural Information Processing Systems 38 (NeurIPS 2025), Datasets and Benchmarks Track, 2025. URL https://arxiv.org/abs/2511.04703. arXiv:2511.04703. [5] Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Jasjeet Sekhon, Jacob Steinhardt, Antony Kellermann, Sarah Schwettmann, Matei Zaharia, Ion Stoica, Percy Liang, and Daniel Kang. Establishing Best Practices for Building Rigorous Agentic Benchmarks. arXiv:2507.02825, 2025. URL https://arxiv.org/abs/2507.02825. [6] Peter Kirgis, Sayash Kapoor, Stephan Rabanser, Nitya Nadgir, Cozmin Ududec, Magda Dubois, J Allaire, Conrad Stosz, Marius Hobbhahn, Jacob Steinhardt, and Arvind Narayanan. Log analysis is necessary for credible evaluation of AI agents. arXiv:2605.08545, 2026. URL https://arxiv.org/abs/2605.08545. [7] Danqi Chen, Jason Bolton, and Christopher D. Manning. A thorough examination of the CNN/Daily Mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL), pages 2358–2367, Berlin, Germany, 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1223. URL https://aclanthology.org/P16-1223/. [8] Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, Volume 2 (Short Papers) (NAACL-HLT), pages 107–112, New Orleans, Louisiana, 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-2017. URL https://aclanthology.org/N18-2017/. [9] Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics (*SEM), pages 180–191, New Orleans, Louisiana, 2018. Association for Computational Linguistics. doi: 10.18653/v1/S18-2023. URL https://aclanthology.org/S18-2023/. [10] Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. What makes reading comprehension questions easier? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4208–4219, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1453. URL https://aclanthology.org/D18-1453/. [11] Divyansh Kaushik and Zachary C. Lipton. How much reading does reading comprehension require? A critical investigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5010–5015, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1546. URL https://aclanthology.org/D18-1546/. [12] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Demystifying LLM-based software engineering agents. Proceedings of the ACM on Software Engineering, 2(FSE):801–824, 2025. doi: 10.1145/3715754. Also arXiv:2407.01489, “Agentless: Demystifying LLM-based Software Engineering Agents”. [13] Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. Compositional questions do not necessitate multi-hop reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4249–4257, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1416. URL https://aclanthology.org/P19-1416/. [14] Jifan Chen and Greg Durrett. Understanding dataset design choices for multi-hop reasoning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, Volume 1 (Long and Short Papers) (NAACL-HLT), pages 4026–4032, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1405. URL https://aclanthology.org/N19-1405/. [15] Yichen Jiang and Mohit Bansal. Avoiding reasoning shortcuts: Adversarial evaluation, training, and model development for multi-hop QA. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 2726–2736, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1262. URL https://aclanthology.org/P19-1262/. [16] Xinyi Fan, Miri Liu, Ruozhen Yang, Siru Ouyang, and Jiawei Han. Can Agent Memory Systems Track Evolving State? arXiv:2608.19652, 2026. URL https://arxiv.org/abs/2608.19652. [17] Jie Gao, Kaiser Sun, Jen tse Huang, Katherine Van Koevering, Sijie Ji, Heyuan Huang, Weiyan Shi, Zhuoran Lu, Ziang Xiao, Daniel Khashabi, and Mark Dredze. How to Interpret Agent Behavior. arXiv:2605.13625, 2026. URL https://arxiv.org/abs/2605.13625. [18] C. J. Clopper and E. S. Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404–413, 1934. doi: 10.1093/biomet/26.4.404. [19] Evan Miller. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640, 2024. URL https://arxiv.org/abs/2411.00640. [20] Yunbei Zhang, Janet Wang, Yingqiang Ge, Weijie Xu, Jihun Hamm, and Chandan K. Reddy. Stop Comparing LLM Agents Without Disclosing the Harness. arXiv:2605.23950, 2026. URL https://arxiv.org/abs/2605.23950. [21] Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 15686–15702, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.971. URL https://aclanthology.org/2023.emnlp-main.971/. [22] Shaochen Zhong, Yifan Lu, Lize Shao, Bhargav Bhushanam, Xiaocong Du, Yixin Wan, Yucheng Shi, Daochen Zha, Yiwei Wang, Ninghao Liu, Kaixiong Zhou, Shuai Xu, Kai-Wei Chang, Louis Feng, Vipin Chaudhary, and Xia Hu. MQuAKE-remastered: Multi-hop knowledge editing can only be advanced with reliable evaluations. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=m9wG6ai2Xk. [23] MQuAKE repository. MQuAKE GitHub repository: README. https://github.com/princeton-nlp/MQuAKE, 2024. Accessed 2026-09-02; main branch at commit fb43dad. [24] Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics, 12:283–298, 2024. doi: 10.1162/tacl_a_00644. URL https://aclanthology.org/2024.tacl-1.16/. [25] Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7052–7063, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.565. URL https://aclanthology.org/2021.emnlp-main.565/. [26] Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. Knowledge conflicts for LLMs: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8541–8565, Miami, Florida, USA, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.486. URL https://aclanthology.org/2024.emnlp-main.486/. [27] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024. doi: 10.1162/tacl_a_00638. URL https://aclanthology.org/2024.tacl-1.9/. [28] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling (COLM), 2024. [29] Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46, 1960. doi: 10.1177/001316446002000104. [30] Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. In The Thirteenth International Conference on Learning Representations (ICLR 2025), 2025. URL https://arxiv.org/abs/2410.10813. arXiv:2410.10813. [31] Bjarni Haukur Bjarnason, André Silva, and Martin Monperrus. On Randomness in Agentic Evals. arXiv:2602.07150, 2026. URL https://arxiv.org/abs/2602.07150. [32] Berk Atil, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, Zhe Wu, Lixinyu Xu, and Breck Baldwin. Non-Determinism of "Deterministic" LLM Settings. arXiv:2408.04667, 2024. URL https://arxiv.org/abs/2408.04667. [33] Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference. In Advances in Neural Information Processing Systems 38 (NeurIPS 2025), 2025. URL https://arxiv.org/abs/2506.09501. arXiv:2506.09501. [34] Dongming Jiang, Yi Li, Songtao Wei, Jinxin Yang, Ayushi Kishore, Alysa Zhao, Dingyi Kang, Xu Hu, Feng Chen, Qiannan Li, and Bingzhe Li. Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations. arXiv:2602.19320, 2026. URL https://arxiv.org/abs/2602.19320. [35] Wei Zhou, Xuanhe Zhou, Shaokun Han, Hongming Xu, Guoliang Li, Zhiyu Li, Feiyu Xiong, and Fan Wu. Are We Ready For An Agent-Native Memory System? arXiv:2606.24775, 2026. URL https://arxiv.org/abs/2606.24775. Appendix A Taxonomy rule, label production, and reachability check Decision rule (six categories fixed before execution in the study’s pre-execution specification—internally the “gate specification”, an excerpt of which is available from the authors on request; the ordering and the reachability operationalization were documented in the analysis log (the “gate report”) by the LLM-based pipeline that operated the analysis under the authors’ direction (the “operating agent”; App. H) when the labels were produced). required query edge present only in a frozen-parser unparsed statement → parse failure; released gold inconsistent with or unreachable under greatest-position overwrite despite context occurrence → ambiguous gold; gold reachable in the frozen last-write graph but query grounding/path selection chooses another output → multi-hop composition failure; demonstrable semantics not expressible by greatest-position overwrite → genuine beyond-last-write; scoring-surface-only difference → normalization artifact; no isolated cause → unclassifiable. The rule’s names are quoted as frozen; this paper writes parse failure as parser scope and multi-hop composition failure as last-write-reachable miss (the label strings in the audit file stay parse_failure and multi_hop_composition_failure; the second name predates the finding that the class holds 11 single-hop rows), and uses no other aliases. Workbench. For each of the 158 headline failures a row-level evidence workbench was built by script before any label existed; the workbench file (available on request) is a deterministic regeneration of the one shown to the labeler in batches. Each row carries the question, the released gold strings, the resolver’s prediction and trace reason, the predicted start entity, the predicted relation path and the statements it used, the entity mentions and relation/terminal cues found in the question, whether the gold is reachable from some question mention in the frozen last-write graph, every gold-reaching path with its statement indices, the parsed context statements containing the gold string, the unparsed context statements containing the gold string, and every context line containing the gold string. Label production. The labels were produced by the classification script over this workbench. In an LLM-assisted review (GPT-5.6) directed by the authors, the workbench was read in batches and 13 rows (residual ids—row numbers in the 158-row audit file—13, 17, 30, 48, 56, 67, 108, 138, 141, 142, 153, 157, 158) were designated as parse failures, on the ground that a required query edge occurs only in an unparsed statement; these ids are an explicit list in the script. A post hoc mechanical proxy—whether an unparsed statement’s whole subject phrase occurs in the question text (whole-word match after the official normalization, tolerating a possessive s, e.g. “Attorney General’s”)—recovers exactly the 13 designated rows and no other failure. The proxy was formulated after the designations, to test them; it is not part of the documented rule, and it does not test whether the matched statement supplies a required query relation or whether the subject lacks other parsed outgoing edges (two designated subjects have parsed edges of other relations): the required-edge reading of these rows comes from the designation review. Every other row was labeled by the script from the workbench’s last-write reachability flag: gold reachable in the frozen last-write graph → last-write-reachable miss (78 rows; the script’s label string is multi_hop_composition_failure, kept in the audit file; earlier drafts called the class grounding/path-selection failure; the category holds 67 multi-hop and 11 single-hop rows, 22 of them ABSTAINs); not reachable → ambiguous gold (67 rows). The flag tests reachability from some question mention by any relation sequence of at most six edges, without regard to the chain the question asks for: all 78 last-write-reachable-miss rows are reachable from a question mention, 56 of them from the resolver’s chosen start entity (the other 22 are ABSTAINs with no start entity). A screen requiring some gold-reaching last-write path to contain every relation the resolver cued from the question flags 8 of the 78; in one of them (multi-hop 262K #77) the resolver’s own path covers every cued relation and reaches the last-write answer while the gold sits on a relation sequence the question does not cue (author → died_city), so under a chain-consistent test that row is operationally ambiguous; the other seven are ABSTAINs or paths that themselves miss a cue. The two models’ accuracy on the 8 screen-flagged rows is 5/8 (Pro) and 4/8 (Flash) versus 36/70 and 42/70 on the other 70 last-write-reachable-miss rows. The frozen labels are reported unchanged. The 11 single-hop rows in this category are resolver defects: five ABSTAIN with an empty cue set (no entry of the frozen cue table matches the question form “Which language was X written in?”), and six prefer a 3–4-edge path over the 1-edge gold path because a cued relation is credited each time it appears on a path. The script also carries explicit override lists for the categories genuine conflict resolution beyond last write, normalization artifact and unclassifiable; all three were left empty. The script writes one uniform audit note per label: last-write-reachable miss—“gold is reachable in the frozen last-write graph, but query grounding/path selection chose another output”; ambiguous gold—“released gold is not reachable from the question entity in the frozen last-write graph despite appearing in the context”; parse failure—“required query edge appears only in an unparsed statement”. 158/158 rows received a label; 0 were unclassifiable. The pre-execution specification called for every failure to be classified by the operating agent; what was executed was a script over the reachability flag plus 13 agent designations—a deviation we disclose. Blind re-adjudication. After the labels were frozen, a second language model, of a different family from the one used in the batch review (Claude Opus 4.8), received the same eight workbench batches (which contain no label) and the same six-category rule in the same order, and returned a label and a one-sentence reason for every row. Agreement with the frozen labels: 147/158 (93.0%), Cohen’s κ=0.875κ=0.875. By frozen label: last-write-reachable miss 76/78, ambiguous gold 65/67, parse failure 6/13. Confusion: 2 last-write-reachable-miss rows → ambiguous gold (the second labeler required the gold-reaching path to match the question’s relation chain); 1 ambiguous-gold row → last-write-reachable miss; 1 ambiguous-gold row → parse failure (the row whose gold occurs in 31 unparsed “original language” statements); 7 parse-failure rows → ambiguous gold: all 7 satisfy the mechanical criterion above (their first hop—the head coach of a club, the President of a country, the pope, a state attorney general—exists only in unparsed statements); the second labeler applied the reachability test before the parse-scope test, and in several of its reasons it notes the missing head-coach or office edge and still chooses ambiguous gold (e.g., “gold_reachable_lww False and no head coach edge grounds to Australia’s sport, so gold unreachable under overwrite”), so these are disagreements about the order in which the rule’s tests are applied. The per-row labels and reasons are recorded in the audit file, available on request. No frozen label was changed. Because 145 of the 158 reference labels are a deterministic function of the reachability flag, which the second labeler saw, agreement on those rows measures rule-following; the informative part is the 13 parser-scope rows. Re-adjudication B. Because re-adjudication A showed the labeler the reachability flag that determines 145 of the 158 reference labels, a second blind re-labeling withheld the explicit flag. A third model family (Gemini 3.1 Pro (preview)) saw, per row, the question, gold, resolver output and path, and a set of candidate context statements, and applied the same six-category rule. The candidate builder selected statements mentioning a question, path or gold entity and, where present, entities on the computed gold-reaching paths, so evidence selection was itself reachability-conditioned; it targeted 60 candidates per row while retaining every unparsed candidate (one row therefore retained 121); each candidate was tagged parsed (with subject, relation, object and whether it is the last write for its key) or unparsed. Agreement with the frozen labels: 141/158 (89.2%), Cohen’s κ=0.82κ=0.82. It labeled 12 of the 13 designated parser-scope rows parse failure and one further row (a last-write-reachable miss under the frozen labels) parse failure; 8 rows it called unclassifiable: 7 on rows whose evidence hit the cap (2 of them say so), 1 (#77) on an untruncated row whose block showed the gold-reaching died_city path but not the spouse → citizenship edges the labeler reported absent, which the workbench’s triggers omit; 6 ambiguous-gold rows it called last-write-reachable misses, reasoning about paths whose last-write status it misjudged; one last-write-reachable-miss row it called ambiguous gold and the remaining designated parser-scope row a last-write-reachable miss (17 disagreements in all); the two re-labelings agree with each other on 134/158 (κ=0.74κ=0.74). Its disagreements concentrate on rows whose evidence hit the 60-statement cap: 52 of 158 rows hit the cap; 7 of the 8 unclassifiable rows and 14 of the 17 disagreements are truncated rows (disagreement 26.9% on truncated vs 2.8% on untruncated rows). The per-statement last-write tags remained visible, so a diligent labeler could reconstruct aggregate reachability; only the precomputed flag was withheld. Frozen labels unchanged. Construction note. Every one of the 158 rows has the gold string in at least one context line. Given the rule’s ordering, an item whose gold occurs in the context is assigned to parse failure, ambiguous gold or multi-hop composition failure before “genuine beyond-last-write” is considered; that category can therefore be reached only by an item whose gold is absent from the context, and under the documented rule it is empty (§3.3). Additional per-item reachability check (not part of the labeling). For each ambiguous-gold row, a breadth-first search over subject → object edges of all parsed tuples of the context (overwritten tuples re-admitted), depth ≤6≤ 6, from the resolver’s start entity, reaches a gold string under the official normalization in 67/67 rows, and in 0/67 rows when only last-write tuples are used (reproducing the workbench flag). In 50 of the 67 some re-admitted path of at most six edges ends in a gold-valued tuple that a later statement overwrote (existence over paths); in the other 17 only intermediate tuples on such paths were overwritten and the gold-valued tuple at the end of the path is itself a last write; on the shortest re-admitted path the split depends on tie-breaking and the matching surface, so we report the existence version only. Reachability here tests exact equality of the normalized node with a normalized gold, as the workbench does; under the scoring rule’s substring test one row (multi-hop 262K, gold “Vienna”) is reachable at depth one through the unrelated node “Academy of Fine Arts Vienna”, so the count is 66/67 under that looser test. Base rate: from the same start entities, the golds of the other 99 questions in the same set are reachable within six edges in 5.6% of pairs under last-write and 25.4% once overwritten tuples are re-admitted (6,633 pairs), and sit at the end of a superseded edge on some such path in 12.7%; the corresponding rates for the 67 rows’ own golds are 1.5% (1/67), 100% (67/67) and 74.6% (50/67) (substring surface throughout). Row-by-row check of the 67 rows. A workbench (available on request) lists, for each of the 67 rows, the question, the released gold, the resolver’s answer and the statements it used, every parsed statement whose object is the gold together with the last-write statement on the same key, and the shortest gold-reaching path once overwritten statements are re-admitted. Two models read it independently under the authors’ direction (Claude Opus 4.8 with file access; GPT-5.6 with the workbench inlined) and judged, row by row, whether the released gold is a value that a later statement on the same key superseded, or is reachable only through such a statement: 67 of 67 confirmed by both, no disagreement (verdict files available on request). This checks the raw statements, not the parser’s tuples, so it does not replace the parse audit that App. G says is missing. The re-admitted graph reaches a median of 47 nodes from a start entity (87 at 262K) against 8 under last-write. Per question set: multi-hop 262K 34, 32K 13, 64K 9, 6K 4; single-hop 262K 3, 64K 2, 32K 2. All 158 failures have at least one context line containing the gold string. Three ambiguous-gold rows also have the gold string inside an unparsed statement (residual ids 73, 86, 132; App. B). Appendix B Examples per category Each example gives question → gold → resolver output, with the question set and question index. Last-write-reachable miss. (B1) “From which country does the musical genre Michael Mantler is associated with come from?” → Canada → post-punk (multi-hop 6K, #38; the path stopped one edge short—gold is reachable via music_type → created_country, statements 298 and 267). (B2) “What is the occupation of the chairperson of the Palestine Liberation Organization?” → basketball player → Regina Ip (multi-hop 32K, #4). (B3) “Who holds the title of chief public representative in the country where Jason Motte is a citizen?” → Roch Marc Christian Kaboré → Ireland (multi-hop 32K, #7). Ambiguous gold (operational). (B4) “Which city serves as the capital of the country where the sport that Mike D’Antoni specialized in hails from?” → London → Oderzo (multi-hop 6K, #44). (B5) “Which city serves as the capital of the country that gave rise to the sport played by Rogério Ceni?” → New Delhi → Grosseto (multi-hop 6K, #67; context statements 158 “The capital of India is New Delhi.” and 341 “The capital of India is Grosseto.”). (B6) “What is the name of the capital city of the country where the founder of the manufacturer of Ford F-Series hailed from?” → New Delhi → Grosseto (multi-hop 6K, #86). The three rows whose gold string also occurs in an unparsed statement: multi-hop 262K #4 (gold “English”; 31 unparsed occurrences, “The original language of … is English.” forms), multi-hop 262K #20 (gold “Sergio Mattarella”; unparsed “The President of Italy is Sergio Mattarella.”), multi-hop 262K #96 (gold “Australia”; inside the subject phrase of two unparsed head-coach statements, “The head coach of Football Federation Australia is …”, where it is not a value). Provenance of the fact lists. The benchmark’s construction statement (v1 App. A.4; identical in the v4 camera-ready’s App. B) says: “For each edit pair, the sentence representing outdated information (the distractor) is given a smaller number, while the sentence representing more recent information (the one containing the answer) is given a larger number.” The 50 rows in which a later statement on the same key supersedes the gold-valued statement are the concrete content of “operationally ambiguous” relative to this construction; neither v1 nor v4 names the MQuAKE subset used. The MQuAKE repository’s README documents a knowledge-conflict issue in MQuAKE-CF-3k under multi-edit use and points to the corrected subset [23]; a later audit found a large share of MQuAKE questions and labels corrupted and its multi-hop editing evaluations unreliable without repairs [22]; and conflicting evidence—between a context and parametric memory [25], or between statements inside one context [26]—is a recognized failure surface for language models in its own right. Whatever their origin, the 67 operationally ambiguous items are ones on which a procedure that reproduces the frozen last-write graph is scored incorrect under exact node match (66 of 67 under the substring surface, App. A). Parser scope. (B7) “What is the country of origin of the sport to which the head coach of Western Sydney Wanderers FC belongs?” → Australia → ABSTAIN (multi-hop 32K, #23; “head coach” form unparsed). (B8) “What is the name of the partner of the President of Azerbaijan?” → Mahidol Adulyadej → ABSTAIN (multi-hop 32K, #29; titled-office form unparsed). (B9) “What is the name of the capital of the country where the sport coached by the head coach of the Cleveland Indians was invented?” → Amposta → Grosseto (multi-hop 32K, #70). Appendix C Parser and resolver Statement segmentation. Each line of the context is tested against the full-line regular expression *( +)\. +(.+?) *; a matching line contributes one statement with the captured integer as its numbered position. The unnumbered heading is excluded. Patterns. Table 3 lists the 35 frozen patterns in their order of application, with the relation key each yields. Every pattern is applied as a full-line match against the statement text; a statement yields a tuple (subject == group s, relation, object == group o, position) only when exactly one pattern matches, and is otherwise recorded as unparsed. The order is deliberate: more syntactically specific patterns precede generic ones. Two surface forms (head_government and the “Prime Minister” form) share the relation key head_government, so the 35 patterns map to 34 relation keys. Table 3: The 35 frozen full-line patterns (Python re syntax), listed in order of application, with the relation key each yields; spellings such as “univeristy” and “origianl” reproduce the benchmark’s statement templates. Relation key Pattern chairperson The chairperson of (?P<s>.+) is (?P<o>.+)\. director The director of (?P<s>.+) is (?P<o>.+)\. headquarters_city The headquarters of (?P<s>.+) is located in the city of (?P<o>.+)\. author The author of (?P<s>.+) is (?P<o>.+)\. ceo The chief executive officer of (?P<s>.+) is (?P<o>.+)\. educated_at The univeristy where (?P<s>.+) was educated is (?P<o>.+)\. capital The capital of (?P<s>.+) is (?P<o>.+)\. head_state The name of the current head of state in (?P<s>.+) is (?P<o>.+)\. head_government The name of the current head of the (?P<s>.+) government is (?P<o>.+)\. head_government The Prime Minister of (?P<s>.+) is (?P<o>.+)\. music_type The type of music that (?P<s>.+) plays is (?P<o>.+)\. official_language The official language of (?P<s>.+) is (?P<o>.+)\. produced_by The company that produced (?P<s>.+) is (?P<o>.+)\. original_broadcaster The origianl broadcaster of (?P<s>.+) is (?P<o>.+)\. born_city (?P<s>.+) was born in the city of (?P<o>.+)\. died_city (?P<s>.+) died in the city of (?P<o>.+)\. plays_position (?P<s>.+) plays the position of (?P<o>.+)\. located_continent (?P<s>.+) is located in the continent of (?P<o>.+)\. worked_city (?P<s>.+) worked in the city of (?P<o>.+)\. spouse (?P<s>.+) is married to (?P<o>.+)\. founded_by (?P<s>.+) was founded by (?P<o>.+)\. associated_sport (?P<s>.+) is associated with the sport of (?P<o>.+)\. citizen_country (?P<s>.+) is a citizen of (?P<o>.+)\. performed_by (?P<s>.+) was performed by (?P<o>.+)\. founded_city (?P<s>.+) was founded in the city of (?P<o>.+)\. created_country (?P<s>.+) was created in the country of (?P<o>.+)\. child (?P<s>.+)'s child is (?P<o>.+)\. created_by (?P<s>.+) was created by (?P<o>.+)\. religion (?P<s>.+) is affiliated with the religion of (?P<o>.+)\. famous_for (?P<s>.+) is famous for (?P<o>.+)\. employed_by (?P<s>.+) is employed by (?P<o>.+)\. speaks_language (?P<s>.+) speaks the language of (?P<o>.+)\. work_field (?P<s>.+) works in the field of (?P<o>.+)\. developed_by (?P<s>.+) was developed by (?P<o>.+)\. written_language (?P<s>.+) was written in the language of (?P<o>.+)\. Last-write graph. For each (subject, relation) key, the tuple with the greatest numbered position is retained; all earlier tuples for that key are discarded. Subjects and objects are used as written (after stripping surrounding whitespace); no entity linking is performed. Entity grounding. Every subject and object of the last-write graph is a node. A node is a mention of the question if its normalized surface (Unicode NFKC, case-folded, curly quotes unified, whitespace collapsed) occurs as a substring of the normalized question; when mentions overlap, a longer literal dominates any literal whose span lies within it. Path search and scoring. From each mention, all cycle-free paths of 1 to 6 edges in the last-write graph are enumerated (outgoing edges ordered by relation, then object). Each path receives a deterministic score: min(ℓ,60)/100 ( ,60)/100, where ℓ is the mention length in characters; plus, for each relation on the path, the weight of the strongest fixed lexical cue for that relation found in the question (a fixed table of regular expressions per relation, weights 1.0–5.0); plus a fixed terminal preference of 12.0 when the question matches one of a fixed list of question constructions associated with the path’s last relation; minus 0.550.55 per edge; minus 0.450.45 per relation on the path with no cue in the question; plus 0.350.35 per distinct cued relation on the path; minus 6.06.0 if the terminal value’s normalized surface already occurs in the question. The highest-scoring path’s terminal value is the answer; ties are broken in favour of the later mention position in the question (larger start index), then by terminal value, then by relation sequence. The cue and preference tables are fixed in the frozen source and were not changed after the freeze. ABSTAIN conditions. The resolver outputs the literal ABSTAIN when no node is mentioned in the question or no path leaves any mentioned node, or when the best path’s score is ≤0≤ 0, or when the question triggers no relation cue and no terminal preference. ABSTAIN is scored as wrong in every headline number. Unparsed-statement census. Over the three distinct held-out fact lists (32K, 64K, 262K: 2,310 + 4,580 + 18,332 = 25,222 statements), the frozen parser leaves 351 statements unparsed (29 + 60 + 262): 166 of the form “The head coach of X is Y”, 69 of the form “The original language of X is Y”, and 116 titled-office statements of the form “The ⟨ ⟩ is ⟨ ⟩ ” (e.g., “The President of Azerbaijan is Ilham Aliyev.”). Each held-out list occurs in two parquet rows, so the per-row totals are 702 unparsed of 50,444 (49,742 parsed, 98.6%). The development list (455 statements) is parsed in full. Appendix D Sensitivities Table 4: Sensitivity of the headline accuracy. All entries are normalized (official) scoring; strict scoring gives identical counts in every row. Variant Accuracy (%) All 800 answers, ABSTAIN wrong (headline) 642/800 = 80.25 ABSTAIN excluded (768 answers) 642/768 = 83.59 Held-out six question sets only (three fact lists) 447/600 = 74.50 Leave one question set out (8 sets = 4 lists × 2 hop types), omitted set 0–7 78.14 / 82.86 / 81.29 / 86.57 / 77.43 / 78.14 / 78.43 / 79.14 range 77.43–86.57 (width 9.14 p) Leave one fact list out (both of its question sets), dropped 6K / 32K / 64K / 262K 74.50 / 80.83 / 79.33 / 86.33 Fact-list-clustered percentile bootstrap, 4 clusters of 200 answers, 10,000 replicates, seed 20260901 80.25 [67.25, 92.75]; half-width 12.75 p Overwrite policy. Re-resolving every question with the frozen parser and heuristics but keeping the first write per (subject, relation) key instead of the last (analysis script available on request; the last-write rerun reproduces 642/800 exactly): 526 of the 642 resolver-solved items are lost under first-write—answered differently and scored wrong (260 single-hop, all 266 multi-hop; one further single-hop item is answered differently but both answers contain the gold), 116 single-hop items are solved under both policies (115 because every key on their path has a single write; one because both policies’ answers contain the gold), 7 of the 67 operationally ambiguous items are solved only under first-write (their gold is the earlier value), and the remaining 151 failures are solved by neither. The split therefore does exercise the overwrite relation on 82% of the items the rule solves; what it does not require is anything beyond taking the newest write. Parquet rows 0–7 are, in order, multi-hop 6K, 32K, 64K, 262K and single-hop 6K, 32K, 64K, 262K. With only 4 clusters the bootstrap resampling distribution has 256 equiprobable outcomes; the interval is reported as a width indicator only and supports no precision claim. The frozen gate record’s clustered bootstrap used the 8 question sets as clusters and gave [65.1, 93.0]; because the 8 sets are 4 fact lists × 2 question sets, the paper reports the 4-list version, [67.25, 92.75]. Appendix E Accuracy grid with intervals; residual splits Table 6 repeats the accuracy grid of Table 1 with exact 95% intervals; Table 7 splits the 158 failures by question set and class, and Table 5 cross-tabulates them by hop and output type. Table 5: Residual taxonomy of the 158 failures by hop type and by output type (counts from the row-level audit file); Model correct = Gemini 2.5 Pro (§3.4). Label n (share) Multi-hop / single-hop ABSTAIN / committed Model correct Last-write-reachable miss 78 (49.4%) 67 / 11 22 / 56 41 Ambiguous gold (operational) 67 (42.4%) 60 / 7 0 / 67 7 Parser scope 13 (8.2%) 7 / 6 10 / 3 8 Genuine beyond last write 0 – – – Normalization artifact 0 – – – Unclassifiable 0 – – – Table 6: Per-list accuracy (%) with exact Clopper–Pearson 95% CIs (n=100n=100 per hop cell, 200 per both-hops cell); the pooled and held-out columns with their CIs are in Table 1. 6K (dev) 32K 64K 262K Single-hop 100 (96.4–100) 95 (88.7–98.4) 93 (86.1–97.1) 88 (80.0–93.6) Multi-hop 95 (88.7–98.4) 62 (51.7–71.5) 73 (63.2–81.4) 36 (26.6–46.2) Both hops 97.5 (94.3–99.2) 78.5 (72.2–84.0) 83.0 (77.1–87.9) 62.0 (54.9–68.8) ABSTAIN (SH / MH) 0 / 0 2 / 7 5 / 4 4 / 10 Table 7: Residual label by question set (8 sets = 4 fact lists × multi-hop, single-hop); n=100n=100 questions per set. Counts from the row-level audit file. Question set Failures Last-write-reachable miss Ambiguous gold Parser scope Multi-hop 6K 5 1 4 0 Multi-hop 32K 38 22 13 3 Multi-hop 64K 27 15 9 3 Multi-hop 262K 64 29 34 1 Single-hop 6K 0 0 0 0 Single-hop 32K 5 2 2 1 Single-hop 64K 7 3 2 2 Single-hop 262K 12 6 3 3 All 158 78 67 13 Appendix F Published comparison: all cited rows All values are transcribed from Hu et al. [1], version v1 (arXiv, 7 Jul 2025), and are not recomputed; they are percentages under the benchmark’s accuracy / substring exact match (SubEM) mapping. Table 8: Tier-matched comparison used in §3.2: cited published Conflict Resolution scores beside the resolver, by tier; transcribed from Hu et al. [1] v1 (last row: v4 camera-ready), not recomputed. Main table = the benchmark’s v1 Table 2 / App. Table 7 (262K tier); v1 App. Table 11 = per-tier long-context rows; v1 Table 3 = its reasoning-model run on short lists. System Tier Single-hop (%) Multi-hop (%) Frozen last-write resolver (this work) 6K (dev) / 32K / 64K / 262K 100 / 95 / 93 / 88 95 / 62 / 73 / 36 GPT-4o, long-context (the benchmark’s v1 Tables 3, 11, 2) 6K / 32K / 64K / 262K 92 / 88 / 85 / 60 28 / 10 / 13 / 5 o4-mini, reasoning model (v1 Table 3) 6K / 32K 100 / 61 80 / 14 Best RAG or memory system, v1 main table 262K 56 (BM25) 7 (Contriever) GPT-5-mini, long-context (v4) 262K 78 28 The benchmark’s own runs cap FactConsolidation output at 10 tokens (v1 Table 14) and use a chunk size of 512 (v1 Table 15). The ICLR 2026 camera-ready (arXiv v4, 28 Jun 2026) renames the competency Selective Forgetting, relabels the metric Accuracy (still substring exact match), revises BM25 single-hop from 56 to 48, drops NV-Embed-v2, and adds GPT-5-mini 78 / 28, Qwen3-Embedding-4B 29 / 3, MemoRAG 21 / 7, Zep 7 / 3, MIRIX 14 / 2 and MIRIX (4.1-mini) 20 / 3 at 262K; every other number cited here is unchanged except the per-tier correction noted next. In v1 App. Table 11 the Claude-3.7-Sonnet multi-hop entry at 262K is 0 where v1 Table 2 / App. Table 7 give 2 for the same cell; the v4 camera-ready corrects the per-tier table to 2. Table 9: All 17 systems of the benchmark’s v1 main results table (v1 Table 2 / App. Table 7; 262K tier). Categories as in the source; all non-long-context rows use a GPT-4o-mini backbone. System Category Single-hop Multi-hop GPT-4o long-context 60 5 GPT-4o-mini long-context 45 5 GPT-4.1-mini long-context 36 5 Gemini-2.0-Flash long-context 30 3 Claude-3.7-Sonnet long-context 43 2 BM25 simple RAG 56 3 Contriever embedding RAG 18 7 Text-Embed-3-Small embedding RAG 28 3 Text-Embed-3-Large embedding RAG 28 4 NV-Embed-v2 embedding RAG 55 6 RAPTOR structure-augmented RAG 14 1 GraphRAG structure-augmented RAG 14 2 HippoRAG-v2 structure-augmented RAG 54 5 Mem0 structure-augmented RAG 18 2 Cognee structure-augmented RAG 28 3 Self-RAG agentic memory 19 3 MemGPT agentic memory 28 3 Frozen last-write resolver (this work, 262K list only) deterministic, no learned component 88 36 Pooled over all four lists the resolver scores 94.0 single-hop / 66.5 multi-hop (92.0 / 57.0 on the three held-out lists); only its 262K row is tier-matched to the main table. At 6K, the source’s own reasoning-model run (o4-mini, v1 Table 3) reaches 100 / 80, within 15 p of the resolver’s 95 multi-hop on the development list. The camera-ready’s App. K.1 states the reading that the item-level analysis in this paper qualifies: “We clarify that while prompt deltas can shift outcomes, our benchmark applies a unified and standardized prompt template across all evaluated agents (Long-context, RAG, and Agentic). This ensures that the performance gaps observed (e.g., the failure of RAG agents in multi-hop Selective Forgetting) are attributed to the limitations of their memory mechanisms rather than prompt inconsistency.” The camera-ready’s App. G, item 4, states the same reading from the 6K runs: “As demonstrated in our ablation study, long-context agents with strong reasoning capabilities achieve near-perfect performance on short-context (6K tokens) versions of the dataset. This result directly confirms that performance degradation on long-context, full-length task inputs stems not from flaws in the task definition itself, but from the fundamental limitations of current memory agents in long-range reasoning—specifically, their inability to accurately identify and discard outdated information across extended interaction histories.” On this split a gap can also reflect items whose gold the stated rule does not reach (§3.3) and the failure at 262K of both long-context models on items the rule solves (§3.4); the item-level analysis qualifies both readings. Table 10: Per-tier published rows. Top: v1 Table 3 (“Performances of reasoning models on the dataset FactConsolidation”). Bottom: v1 App. Table 11 (“Performance comparison on different context length”), FactConsolidation columns only. Single-hop Multi-hop v1 Table 3 6K 32K 6K 32K GPT-4o 92 88 28 10 o4-mini 100 61 80 14 Single-hop Multi-hop v1 App. Table 11 32K 64K 262K 32K 64K 262K GPT-4o 88 85 60 10 13 5 GPT-4o-mini 63 58 45 10 5 5 GPT-4.1-mini 82 72 36 7 9 5 Gemini-2.0-Flash 49 62 30 7 9 3 Claude-3.7-Sonnet 46 45 43 2 2 0 Mem0 22 8 18 3 2 2 Cognee 39 31 28 4 5 3 Frozen last-write resolver (this work) 95 93 88 62 73 36 Appendix G Freeze and reproducibility record • Dataset: the released Conflict Resolution parquet, SHA-256 24d5c3f0… (1,491,588 bytes); 8 rows, 4 distinct context strings, 800 questions. • Development rows 0 and 4 (the 6K fact list with its multi-hop and single-hop question sets) declared before any content was inspected; no question, answer or statement text from the other six rows was inspected before the freeze. • Frozen parser/resolver source: SHA-256 c4ea4e1…; freeze time 2026-09-02T00:25:35Z; the all-row evaluation refuses to run unless the source hash matches the freeze record; no edits after the freeze. Development-set diagnostic before the freeze: 195/200 correct, 0 ABSTAIN, 910/910 statements parsed. • Parse check: the pipeline’s script recorded a 30-statement sample (seed 20260901) as faithful without a per-row check (all 30 rows marked correct by the script); a later re-application of the same patterns to the same sample, drawn from the parser’s own output, reproduces the tuples by construction and is a consistency check that cannot fail; no independent semantic audit of parses exists. The same seed is used for the clustered bootstrap. • Resolver execution and metric scoring (segmentation, parsing, graph construction, resolution, scoring, workbench construction, classification script) are local deterministic Python and make no LLM API calls; the 13 parser-scope designations consumed by the classification script were made in the LLM-assisted batch review (App. A), and the long-context experiment of §3.4 retains one scored output per question (retries: App. I). • Scoring: the official repository’s substring_exact_match at commit fe1735de, re-implemented verbatim—lowercase, strip ASCII punctuation, drop the articles a/an/the, collapse whitespace, then test whether any normalized gold string is a substring of the normalized prediction; strict case-sensitive equality reported alongside (identical counts in every group). • Erratum: the frozen record parser-freeze.md says 32 patterns; the frozen source has 35 patterns mapping to 34 relation keys (App. C). • Novelty check against the source paper (v1, 23 pages, and the v4 camera-ready) and the official repository at commit fe1735de: neither version contains a deterministic zero-learning baseline on this split or a row-level residual taxonomy; the nearest construction elsewhere, StateMemBench [16], generates its own scenarios rather than auditing an existing benchmark’s items; LongMemEval’s knowledge-update questions [30] are the natural next target for the instrument. • Available from the authors on request: the frozen source, the per-question results, the parsed and unparsed statement files, the 158-row workbench and labeled audit file, and the classification script. Appendix H LLM-use disclosure Language models enter this work in two roles. As instruments, in three labeling steps disclosed in App. A—the batch review that designated the 13 parser-scope rows (GPT-5.6) and the two blind re-labelings (Claude Opus 4.8, re-adjudication A; Gemini 3.1 Pro (preview), re-adjudication B)—and in the row-by-row check of the 67 ambiguous-gold rows (Claude Opus 4.8 and GPT-5.6, independently). As subjects: the long-context models of §3.4, Gemini 2.5 Pro and Gemini 2.5 Flash, and the retrieval agent’s GPT-4o-mini backbone. LLM assistance was also used in the engineering of the resolver and analysis code and in manuscript preparation, under the authors’ direction and review; the authors take full responsibility for all content, claims, and measurements. Parsing, resolution and scoring were produced by scripted pipelines with no LLM call. The pipeline’s audit script recorded a 30-statement parse sample as faithful without a per-row check, and a later re-application of the same patterns to the same sample is a consistency check that cannot fail; no independent semantic audit of parses exists (App. G). The 158 residual-failure labels were produced by the classification script over a row-level evidence workbench: the 13 designated rows are an explicit list in the script; every other row was labeled by the script from the workbench’s last-write reachability flag (gold reachable in the frozen last-write graph → last-write-reachable miss; not reachable → ambiguous gold); the script’s explicit override lists for the categories genuine beyond last write, normalization artifact and unclassifiable were left empty. The ambiguous-gold labels are corroborated by the programmatic reachability check and the row-by-row check in App. A. The re-derivation script that reproduces every reported number from the frozen artifacts, its output, and the reference-verification record are available from the authors on request. Appendix I Long-context and retrieval-agent experiments: protocol and tables Design (pre-registered before any call; amendments and deviations listed below). All 800 questions, one retained output per question (retry protocol below), in the benchmark’s long-context setting reproduced from the official repository at commit fe1735de (utils/templates.py, BASE_TEMPLATES[’factconsolidation’]; the long-context agent in agent.py): the system message “You are a helpful assistant that can read the context and memorize it for future retrieval.”; the fact list split at line boundaries into chunks of about 16,000 characters (≈ 4,096 tokens, the repository’s configured chunk size at that commit; the paper’s v1 Table 15 lists 512), each wrapped in the repository’s memorize template (“Dialogue between User and Assistant … <User> The following context is the facts I have learned: … <Assistant> I have learned the facts and I will answer the question you ask.”) with a fixed time stamp; then the repository’s query template, whose rule sentence reads “Each fact in the knowledge pool is provided with a serial number at the beginning, and the newer fact has larger serial number. You need to solve the conflicts of facts in the knowledge pool by finding the newest fact with larger serial number.”, followed by the question and “Answer:”. Fact list first, question last; no tools. The paper’s §1 quotes the benchmark paper’s Figure 5 wording, which ends at “finding the newest fact”; the repository template used here adds “with larger serial number”. Model selection. Fixed before any result by a cost rule: candidates in the order gpt-5.5, claude-sonnet-5, gemini-2.5-pro, gemini-2.5-flash; a two-call smoke test per candidate (one 6K item, one 262K item); the primary model is the first in that order whose projected 800-item cost is at or below a pre-set budget; a cheaper secondary model (gemini-2.5-flash) is run as a robustness check if it fits a smaller pre-set budget. The rule selected Gemini 2.5 Pro as primary and Gemini 2.5 Flash as secondary; both were run under the identical protocol. Sampling and output budget. Temperature is the provider default (a fixed value of 0.0 was rejected by two of the four candidate models), so outputs are not deterministic; one run per item per model, no best-of-n. Each item was requested with up to three attempts at a 4,000-token output budget; empty outputs (hidden reasoning exhausting the budget) and transport/rate-limit errors were retried, non-empty outputs were never discarded; items still unresolved—11 for Pro (6 empty, 5 transport) and 17 for Flash (all transport)—were re-requested under the same loop with a 16,000-token budget, under an amendment declared before the retry; two smoke-test items per candidate model preceded the run and are not scored. Single runs vary run to run even at temperature 0 [31, 32], and across inference configurations for numerical reasons [33]; the repeat-run analysis below measures the former for the retrieval agent. The long-context runs are the full-context baseline that memory-system evaluations are advised to report [34]; this paper audits one update workload and harness and compares no memory architectures [35]. Repeat runs. Under pre-registration amendment R-A1 the BM25 agent was run three times under the identical protocol (the retained first run plus two repeats; temperature 0.7). Long-context amendments A3/A4 pre-registered a repeat of Gemini 2.5 Flash; that run was stopped at 368 of 800 items after provider errors and rate limiting cut its throughput (amendment A5, decided before any of its results were analyzed), so the two Gemini models are reported as single runs. Table 11 gives per-class accuracy per run for the agent; the range across runs is the run-to-run variation a single run hides. Table 11: Repeat runs: accuracy (%) per item class and run; last two columns are the per-item agreement between runs (κ; the three pairwise values round to the same number) and the number of items whose outcome differs between a pair of runs (range). System class run 1 run 2 run 3 κ (pairs) differing items BM25 + GPT-4o-mini resolver-correct 41.6 40.3 41.7 0.90 35–37 last-write-reachable miss 12.8 10.3 11.5 ambiguous gold 6.0 9.0 9.0 parser scope 30.8 23.1 7.7 all items 35.6 34.5 35.5 The run files contain 811 (Pro) and 817 (Flash) records for 800 items; the long-context pre-registration’s amendment A2 described the 11 Pro retries as empty outputs, the run file shows 6 empty and 5 transport failures, and a dated note in the pre-registration records the correction. Both models returned a visible answer for 800 of 800 items (every finish reason “stop”); provider-reported completion tokens, including hidden reasoning, average 554 (Pro) and 439 (Flash), while the visible outputs average 1.8 and 2.3 words. The retrieved template and agent source files are kept alongside the prompt file in the analysis record. Scoring. The official substring_exact_match applied (a) to the full visible output and (b) to the first line truncated to 10 words (the benchmark caps generation at 10 tokens). For Pro, (a) and (b) give identical counts on every group; for Flash they differ on one item (593 vs. 592 correct overall). The pre-registration named scoring (b) as the headline; the paper reports (a) because the two differ on one Flash item and (a) is the alignment quantity; both are in the scored files. Every deviation from the pre-registration: the repository’s system message was used (the pre-registration said none); the output budget was raised from 200 to 4,000 tokens with a single 16,000-token retry for still-empty or errored items; and scoring (a) rather than (b) is the headline; temperature is the provider default as pre-registered (the smoke test had sent a fixed 0.0, which two candidates rejected). Also recorded, under the same normalization: whether the output contains the resolver’s last-write prediction for the item. Table 12: Accuracy (%) per question set and pooled under the official metric, with exact Clopper–Pearson 95% CIs, for both models. Group n Pro correct Pro accuracy (95% CI) Flash correct Flash accuracy (95% CI) Single-hop 6K 100 99 99.0 (94.6–100.0) 100 100.0 (96.4–100.0) Single-hop 32K 100 98 98.0 (93.0–99.8) 96 96.0 (90.1–98.9) Single-hop 64K 100 98 98.0 (93.0–99.8) 95 95.0 (88.7–98.4) Single-hop 262K 100 65 65.0 (54.8–74.3) 77 77.0 (67.5–84.8) Multi-hop 6K 100 90 90.0 (82.4–95.1) 83 83.0 (74.2–89.8) Multi-hop 32K 100 70 70.0 (60.0–78.8) 61 61.0 (50.7–70.6) Multi-hop 64K 100 71 71.0 (61.1–79.6) 56 56.0 (45.7–65.9) Multi-hop 262K 100 9 9.0 (4.2–16.4) 25 25.0 (16.9–34.7) Single-hop, all four lists 400 360 90.0 (86.6–92.8) 368 92.0 (88.9–94.5) Multi-hop, all four lists 400 240 60.0 (55.0–64.8) 225 56.3 (51.2–61.2) Held-out (32K–262K), both hops 600 411 68.5 (64.6–72.2) 410 68.3 (64.4–72.0) All 800 600 75.0 (71.8–78.0) 593 74.1 (70.9–77.1) Table 13: Model outcome (correct / wrong) by resolver item class, overall and per fact list, for both models. No item is missing. Model Item class All 6K 32K 64K 262K Pro Resolver-correct (642) 544 / 98 188 / 7 141 / 16 152 / 14 63 / 61 Last-write-reachable miss (78) 41 / 37 1 / 0 20 / 4 13 / 5 7 / 28 Ambiguous gold (67) 7 / 60 0 / 4 3 / 12 1 / 10 3 / 34 Parser scope (13) 8 / 5 0 / 0 4 / 0 3 / 2 1 / 3 All (800) 600 / 200 189 / 11 168 / 32 169 / 31 74 / 126 Flash Resolver-correct (642) 530 / 112 182 / 13 133 / 24 131 / 35 84 / 40 Last-write-reachable miss (78) 46 / 32 1 / 0 19 / 5 14 / 4 12 / 23 Ambiguous gold (67) 8 / 59 0 / 4 2 / 13 2 / 9 4 / 33 Parser scope (13) 9 / 4 0 / 0 3 / 1 4 / 1 2 / 2 All (800) 593 / 207 183 / 17 157 / 43 151 / 49 102 / 98 Accuracy by class with intervals. Pro: resolver-correct items 84.7% (81.7–87.4; per list 96.4 / 89.8 / 91.6 / 50.8 at 6K / 32K / 64K / 262K); last-write-reachable-miss items 52.6% (40.9–64.0; 100 / 83.3 / 72.2 / 20.0 on 1 / 24 / 18 / 35 items); ambiguous-gold items 10.4%; parser-scope items 61.5% (31.6–86.1); shares of its 200 errors 49.0 / 18.5 / 30.0 / 2.5% in that class order. Flash: resolver-correct items 82.6% (79.4–85.4; per list 93.3 / 84.7 / 78.9 / 67.7); last-write-reachable-miss items 59.0% (47.3–70.0; 100 / 79.2 / 77.8 / 34.3); ambiguous-gold items 11.9%; parser-scope items 69.2% (38.6–90.9); shares of its 207 errors 54.1 / 15.5 / 28.5 / 1.9% in that class order. The 67 operationally ambiguous items. Pro’s output contains the resolver’s last-write prediction in 17 of 67 (95% CI 15.5–37.5%), the released gold in 7 (4.3–20.3%), both in 0, and neither in 43 (51.5–75.5%); per list, outputs containing the last-write prediction: 6K 4 of 4, 32K 7 of 15, 64K 4 of 11, 262K 2 of 37. Flash: last-write prediction in 16 (14.3–35.9%), gold in 8 (5.3–22.2%), both in 0, neither in 43 (51.5–75.5%); per list 6K 2 of 4, 32K 6 of 15, 64K 3 of 11, 262K 5 of 37. On the 78 last-write-reachable-miss items neither model’s output ever contains the resolver’s (wrong) prediction; on the 13 parser-scope items each does once. Two-model agreement. The two models’ per-item outcomes agree on 657 of 800 items (82.1%), Cohen’s κ=0.53κ=0.53. Table 15 gives the cross-tab by item class: on the 67 ambiguous-gold items both are wrong on 53, on the 642 resolver-correct items both are right on 484. Retrieval agent (Experiment R). Our pre-registered approximate re-implementation of the benchmark’s BM25 configuration (BM25 over 512-token chunks, top 10, GPT-4o-mini; the main table’s Simple RAG + BM25 row, 56 / 3 at 262K in v1, 48 / 3 in v4) was run on all 800 questions under a protocol mirroring the repository at commit fe1735de as far as documented. Ingestion: the fact list is split into consecutive 512-token windows (tiktoken o200k_base; the paper’s v1 Table 15 chunk size, the yaml says 4,096); the official chunker is not in the snapshot, so windows may split inside a fact line; each chunk is wrapped in the memorize template (fixed time stamp), and the wrapped text is what BM25 indexes (12 / 64 / 128 / 524 chunks per list at 6K / 32K / 64K / 262K). Retrieval: BM25Okapi over lowercase whitespace tokens with the question text as query; the top 10 chunks in rank order, each prefixed “Memory i:” as in the repository’s agent code (the top 10 cover about 83% / 16% / 8% / 2% of the four lists). Generation: the repository’s RAG query template and system message; temperature 0.7 (the benchmark’s setting); maximum output 200 tokens; one retained output per question with the same retry loop as above (800 items, 800 returned, one retry used, mean completion length 2.8 tokens); two smoke-test items preceded the run and are not scored. Four departures from the official code: lowercased BM25 tokens and the bare question as query (both pre-registered; the official retriever uses text.split() without lowercasing and a longer query string), a guessed chunker that may split statements (the official chunker is not in the snapshot), and a 200-token output budget; offline, the top-10 chunk set equals the closest official-code variant’s on 134 of 800 items (ranked order on 3), with hit rates within about 2 points. Scoring (a) as above; the agent scores 285/800 = 35.6% (95% CI 32.3–39.1); held-out 196/600 = 32.7% (28.9–36.6); single-hop 251/400 = 62.8% (57.8–67.5); multi-hop 34/400 = 8.5% (6.0–11.7); per set (single-hop / multi-hop, correct of 100): 6K 70 / 19, 32K 65 / 7, 64K 60 / 4, 262K 56 / 4, with 95% CIs 60.0–78.8 / 11.8–28.1, 54.8–74.3 / 2.9–13.9, 49.7–69.7 / 1.1–9.9, 45.7–65.9 / 1.1–9.9; its 262K row (56 / 4) sits beside the benchmark’s published 56 / 3—a coarse fidelity check, not a replication. Table 14: Retrieval agent (BM25 + GPT-4o-mini): outcome (correct / wrong) by resolver item class, overall and per fact list. No item is missing. Item class All 6K 32K 64K 262K Resolver-correct (642) 267 / 375 89 / 106 66 / 91 59 / 107 53 / 71 Last-write-reachable miss (78) 10 / 68 0 / 1 3 / 21 3 / 15 4 / 31 Ambiguous gold (67) 4 / 63 0 / 4 1 / 14 1 / 10 2 / 35 Parser scope (13) 4 / 9 0 / 0 2 / 2 1 / 4 1 / 3 All (800) 285 / 515 89 / 111 72 / 128 64 / 136 60 / 140 Per class: resolver-correct items 41.6% (37.7–45.5), last-write-reachable misses 12.8%, parser-scope items 30.8%, ambiguous-gold items 6.0% (4/67); shares of its 515 errors 72.8 / 13.2 / 12.2 / 1.7%. On the 67 ambiguous-gold items its output contains the resolver’s last-write prediction in 4, the released gold in 4, and neither in 59. Per-item outcome agreement with the long-context models is low: κ=0.20κ=0.20 with Pro and 0.230.23 with Flash. Retrieval diagnostic (no LLM call): for 496 of the 800 items every statement the resolver used lies in the agent’s top-10 chunks; on those items the agent scores 262/496 = 52.8% (48.3–57.3). Hop stratification: of the 67 operationally ambiguous items 60 are multi-hop, of the 642 resolver-correct items 376 are single-hop; within multi-hop items, accuracy on resolver-correct (266) vs. operationally ambiguous (60) items is 198/266 (74.4%) vs. 5/60 (8.3%) for Pro, 176/266 (66.2%) vs. 7/60 (11.7%) for Flash and 31/266 (11.7%) vs. 0/60 for the agent; at multi-hop 262K (36 resolver-correct, 34 ambiguous items), 4/36 vs. 1/34, 13/36 vs. 4/34 and 4/36 vs. 0/34. Table 15: Per-item outcome cross-tab of the two models (Pro outcome × Flash outcome) by resolver item class. Item class Both correct Pro only Flash only Both wrong Resolver-correct (642) 484 60 46 52 Last-write-reachable miss (78) 33 8 13 24 Ambiguous gold (67) 1 6 7 53 Parser scope (13) 7 1 2 3 All (800) 525 75 68 132 1