Paper deep dive

MemToC: Benchmarking Memory-Tool Conflict Resolution in Large Language Models

Arseniy Varlamov, Rishat Zinnatullin, Elisei Rykov, Alexander Panchenko, Ilseyar Alimova

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 143

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%

Last extracted: 8/28/2026, 3:33:07 AM

Summary

The paper introduces MemToC, a benchmark for evaluating how large language models arbitrate between their parametric memory and tool returns when they conflict. Using 6,504 episodes across five 7-9B models, the study finds that models strongly prefer tool returns, often ignoring correct parametric memory. The authors propose mitigation methods using SFT and DPO with cross-fitting over the ToolHop dataset, showing that while fine-tuning can improve correct-answer retention, it often reduces abstention on errors, highlighting the need for joint evaluation of correctness, tool use, and robustness.

Entities (10)

Gemma 2 9B IT · model · 99%Llama-3.1-8B-Instruct · model · 99%Mistral-7B-Instruct-v0.3 · model · 99%Qwen2.5-7B-Instruct · model · 99%MemToC · benchmark · 98%DPO · method · 95%SFT · method · 95%ToolHop · dataset · 95%Parametric Memory · concept · 90%tool return · concept · 90%

Relation Signals (8)

tool return → conflictswith → Parametric Memory

confidence 95% · Tool-augmented LLMs must arbitrate between two fallible sources when a tool return conflicts with their parametric memory

MemToC → evaluates → Llama-3.1-8B-Instruct

confidence 95% · Across five open-weight 7–9B models... Llama-3.1-8B-Instruct

MemToC → evaluates → Qwen2.5-7B-Instruct

confidence 95% · Qwen2.5-7B-Instruct

MemToC → evaluates → Gemma 2 9B IT

confidence 95% · gemma-2-9b-it

MemToC → evaluates → Mistral-7B-Instruct-v0.3

confidence 95% · Mistral-7B-Instruct-v0.3

MemToC → usesdatafrom → ToolHop

confidence 95% · MemToC is built from ToolHop (Ye et al. 2025), whose 995 chains decompose into 3,912 sub-questions

SFT → isusedfor → MemToC

confidence 90% · We compare prompting with SFT and DPO... SFT and DPO meet this criterion

DPO → isusedfor → MemToC

confidence 90% · We compare prompting with SFT and DPO... SFT and DPO meet this criterion

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Tool-augmented LLMs must arbitrate between two fallible sources when a tool return conflicts with their parametric memory, yet existing evaluations measure source preference without establishing source correctness. We introduce MemToC, a controlled benchmark for post-tool-return arbitration with executable tools. MemToC comprises 6,504 evaluation episodes constructed from 542 quality-controlled factual questions, independently elicited model-specific closed-book answers, and controlled tool returns of known correctness. These components instantiate four source-correctness cases; tool-error and no-tool conditions are separate controls. Across five open-weight 7-9B models, tool returns strongly dominate elicited closed-book answers. The four instruction-tuned models retain a verified-correct answer against an incorrect tool in only 6.5-17.1% of eligible cases, follow a correct tool in 86.0-93.1%, and repeat the tool return in 78.4-86.0% of cases where both sources are wrong. No cross-model ordering remains stable across three instruction-wording variants with the question and episode content held fixed. We compare prompting with SFT and DPO using chain-level cross-fitting over ToolHop, so questions sharing an underlying fact never straddle training and evaluation. We apply an asymmetric success criterion: correct-answer retention must improve without a detected reduction in correct-tool following. SFT and DPO meet this criterion on the same two of four instruction-tuned backbones. Improvements rarely come cleanly: 19 of 20 tested method-model combinations reduce abstention after tool errors or on unanswerable inputs. Transfer beyond MemToC is positive but partial and depends on the model and presentation frame. Correctness-conditioned arbitration can be improved through fine-tuning, but gains must be evaluated jointly with correct tool use, abstention, and robustness to formulation.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

139,990 characters extracted from source content.

Expand or collapse full text

MemToC: Benchmarking Memory–Tool Conflict Resolution in Large Language Models Arseniy Varlamov1 Rishat Zinnatullin2 Elisei Rykov3 Alexander Panchenko3,4 Ilseyar Alimova3 1Central University, Moscow 2Ural Federal University, Yekaterinburg 3Skolkovo Institute of Science and Technology, Moscow 4AIRI, Moscow Abstract Tool-augmented LLMs must arbitrate between two fallible sources when a tool return conflicts with their parametric memory, yet existing evaluations often measure source preference without establishing source correctness. We introduce MemToC, a controlled benchmark for post-tool-return arbitration with executable tools. MemToC comprises 6,504 evaluation episodes constructed from 542 quality-controlled factual questions, independently elicited model-specific closed-book answers, and controlled tool returns of known correctness. These components instantiate four source-correctness cases, while tool-error and no-tool conditions serve as separate controls. The resulting metrics distinguish appropriate source use from indiscriminate source preference. Across five open-weight 7–9B models, tool returns strongly dominate elicited closed-book answers. The four instruction-tuned models retain a verified-correct answer against an incorrect tool in only 6.5–17.1% of eligible cases, follow a correct tool in 86.0–93.1%, and repeat the tool return in 78.4–86.0% of cases where both sources are wrong. No cross-model ordering remains stable across three instruction-wording variants with the question and episode content held fixed. Motivated by these failures, we compare prompting with SFT and DPO using chain-level cross-fitting over ToolHop, ensuring that questions sharing an underlying fact never straddle training and evaluation. We apply an asymmetric success criterion: correct-answer retention must improve without a detected reduction in correct-tool following. SFT and DPO meet this criterion on the same two of four instruction-tuned backbones, suggesting that outcomes depend more strongly on the backbone than on the optimization objective within our model set. Improvements rarely come cleanly: 19 of 20 tested method–model combinations reduce abstention after tool errors or on unanswerable inputs. Transfer beyond MemToC is positive but partial and depends on the model and presentation frame. Thus, correctness-conditioned arbitration can be improved through fine-tuning, but gains must be evaluated jointly with correct tool use, abstention, and robustness to formulation. Figure 1: Schematic illustration of the four source-correctness cases. The displayed combination of answer values is hypothetical and does not represent a single observed MemToC episode; a fixed preference for either source cannot succeed across all four cases. 1 Introduction LLM agents increasingly rely on tools whose outputs are presented as authoritative evidence, yet tools can be stale, misconfigured, or resolve a query to the wrong entity. When a tool return disagrees with a model’s parametric memory, the system must arbitrate between two fallible sources, and evaluating that requires knowing which source is correct: a metric recording only which source is followed cannot separate useful from harmful deference (Figure 1). Existing work lacks explicit control over source correctness. TMC (Cheng et al. 2026) identifies conflicts as disagreements between memory-only and tool-conditioned responses. Because these conflicts emerge naturally rather than being constructed, neither source correctness nor the appropriate arbitration decision is known in advance. Other work constructs conflicts through entity substitution (Longpre et al. 2021) or counterfactual augmentation (Neeman et al. 2023), and recent benchmarks stratify textual evidence by source correctness (Sun et al. 2026; Xie et al. 2024). However, these settings do not study executable tool returns and omit tool-error and no-tool controls. Agentic evaluations likewise show models overwhelmingly adopting tool outputs despite access to reliability diagnostics (Wang and Vemuri 2026). We instead study the controlled post-tool-call setting, where tool selection and invocation are held fixed while only the observed tool return is intervened upon. Our approach We construct MemToC from the ToolHop (Ye et al. 2025) dataset, which provides decomposed multi-hop questions, locally executable tool calls, and verifiable intermediate answers. From these data, we derive a quality-controlled benchmark set of self-contained factual questions with short, verifiable answers and plausible same-type distractors. For each model and question, we first elicit a closed-book answer and then present the same question with a correct tool return, an incorrect return, an error payload, or no return. Tool-return correctness is controlled by injecting either a correct or an incorrect return constructed from externally verified answers and human-validated distractors, whereas LLM memory correctness is measured independently for each model by comparing its closed-book answer against the verified answer. Because we supply the return rather than observe it, we know whether the tool is right before the model answers, instead of having to judge it afterwards. Together with the separately measured closed-book answer, this fixes the correct response for every episode, which is what makes arbitration — rather than source preference — measurable. Tool-error and no-tool instances serve as separate controls. This design evaluates resistance to incorrect returns separately from appropriate use of correct returns, rather than rewarding a fixed preference for either source. We make three contributions. 1. Benchmark. We introduce MemToC, a controlled benchmark for tool–memory arbitration comprising 542 quality-controlled factual questions with executable calls, verified answers, model-specific closed-book answers, and controlled correct and incorrect returns. 2. Evaluation. We evaluate five open-weight 7–9B models across four source-correctness cases, complemented by tool-error and no-tool controls, extraction-first answer matching, and three instruction-wording variants with fixed question text. 3. Mitigation methods. We compare prompting with SFT and DPO using cross-fitting over ToolHop chains, so that every reported fine-tuned prediction is obtained on questions held out from training. 2 Related Work Tool–memory conflicts TMC (Cheng et al. 2026) introduced tool–memory conflicts by identifying disagreements between memory-only and tool-conditioned responses, and showed that existing mitigation strategies only partially resolve them. Subsequent work finds that models may defer strongly to tool outputs and that failures can arise from the tool-use protocol itself (Wang and Vemuri 2026; Zhang et al. 2026). Concurrent studies examine tool reliability and failure modes directly (Tian et al. 2026; Zhang et al. 2024; Yan et al. 2026; Zhu et al. 2026a; Soni 2026). However, these studies do not independently establish the correctness of both the model’s closed-book answer and the tool return, and therefore cannot evaluate arbitration across source-correctness cases. Controlling source preference Direct Preference Optimization (DPO) (Rafailov et al. 2023) learns from preferred and rejected responses without training an explicit reward model. Context-DPO (Bi et al. 2025) applies this objective to following contextual evidence, while KnowPO (Zhang et al. 2025) learns adaptive source selection over constructed conflicts. Both methods primarily optimize reliance on contextual information rather than condition source selection on the correctness of competing sources. Correctness-aware methods in RAG use gating, regime-specific models, or conflict-aware decoding (Zhu et al. 2026b; Wang et al. 2026a; Jiang et al. 2026). Other approaches steer models between parametric memory and contextual evidence (Zhao et al. 2024; Xin et al. 2025; Li et al. 2026), consolidate evidence at inference time (Wang et al. 2025), or control whether a tool should be invoked (Wang et al. 2026b; Chen et al. 2026; Zeng et al. 2026). None of these methods evaluates correctness-conditioned arbitration after a tool return. Figure 2: Overview of MemToC. (a) ToolHop sub-questions are filtered, augmented with type-consistent near and far incorrect returns, and quality-controlled to yield 542 factual items. (b) For each model and prompt formulation, a closed-book pass obtains m, followed by a tool-conditioned pass that produces y under controlled return r. Comparing m and r with verified answer g stratifies responses into four source-correctness cases, separating answer retention from correct- and incorrect-tool following; control conditions are evaluated separately. 3 The MemToC Benchmark MemToC separates source correctness from source preference (Figure 2): questions, verified answers, executable calls, and controlled incorrect returns are constructed and quality-controlled before any model is evaluated. The closed-book answer is then elicited separately for each model and prompt formulation, making case membership model- and formulation-specific while the benchmark artifacts remain fixed. Each benchmark question includes a verified answer, an executable tool call, and at least one curated incorrect return; combining 542 quality-controlled questions with four tool conditions and three instruction-wording variants yields 6,504 fixed evaluation episodes. 3.1 Source data MemToC is built from ToolHop (Ye et al. 2025), whose 995 chains decompose into 3,912 sub-questions, each with an executable tool call and a verified intermediate answer. We use the sub-questions and discard each chain’s terminal answer, which in 602 of the 995 chains is computed over earlier steps rather than being an independently queryable fact. The retained questions have short, verifiable answers spanning named entities such as people, places, organizations, and creative works, temporal values such as dates, years, and time zones, and other short textual values. 3.2 Construction pipeline The left side of Figure 2 summarizes the construction pipeline in three stages. All manual reviews and authoring decisions were performed independently by two annotators. Candidate extraction and filtering We flatten ToolHop’s 3,912 sub-questions and deduplicate identical formulations, yielding 1,948 unique candidates. We retain 605 factual questions whose answers can be retrieved independently rather than computed from earlier chain steps. Related and near-duplicate questions are grouped for fold assignment, after which structural and answer verification removes 30 questions that are not self-contained, reveal their answer, or have an invalid verified answer. This produces a 575-question construction pool. Distractor construction Each question receives a plausible near distractor and, where possible, a far distractor. Near distractors match the verified answer in type, domain, and plausibility, whereas far distractors preserve the answer type but differ more strongly in period or country. Distractors are generated automatically for 426 records using Wikidata constraints or deterministic temporal transformations. For the remaining 149 records, where structured metadata is insufficient to identify a plausible substitute, two annotators author the distractors. The size of this share is a measured outcome rather than a stylistic choice: blinded audits of sampled records leave the automatic branch with a residual rejection rate of about 12% against a target of 3%, concentrated on pre-modern and legendary figures whose Wikidata records carry no citizenship. Both automatically generated and human-authored distractors subsequently undergo the same blinded quality-control procedure. Quality control Programmatic checks require every distractor to be non-empty, distinct from the verified answer and from the paired distractor, and absent from Wikidata’s alternative valid values for the queried property. Two annotators then review all 575 records for self-containedness, answer leakage, verified-answer validity, distractor type, and plausibility. The review yields 503 passes, 39 distractor repairs, and 33 exclusions, producing the final 542-question benchmark. After quality control, far distractors are available for 443 of the 542 benchmark questions. Their coverage is lower among records with human-authored distractors and absent for creative works and other short textual values. Consequently, near–far comparisons are restricted to a non-random subset weighted toward automatically constructed distractors. 3.3 Evaluation procedure Figure 2(b) summarizes the evaluation procedure in three stages. Closed-book baseline For each model and prompt formulation, we first elicit an answer m without tool access. The prompt requires the model to provide its best guess rather than abstain, ensuring that m can be evaluated against the verified answer g. The correctness of m determines the row of the source-correctness matrix. Tool-conditioned run We then present the same question together with a controlled tool return r and collect the model’s final response y. The question text, verified answer, executable call, and return are fixed benchmark artifacts, while m and y are model-specific. Each item is evaluated under a reference prompt and two wording variants of the closed-book and with-tool instructions. Return conditions are a correct return, a near incorrect return, a structured error payload, and no return. A far incorrect return is evaluated separately, on the 443 questions that have one. NoopTool serves as a separate interface control that preserves the tool-call format without providing a competing factual value. Correctness-case assignment The correctness of m and r relative to g assigns each model–formulation pair to one of four cases. When both sources are correct, the target is to return g. When only the closed-book answer is correct, the target is to retain m. When only the tool return is correct, the target is to follow r. When neither source is correct, the target is to reject both answers, explicitly surface the conflict, and abstain. Structured tool errors, no-return instances, and NoopTool are evaluated as controls outside this four-case matrix. Case assignment is completed before the final response y is scored. 3.4 Validation of response targets To validate the response targets independently of the benchmark labels, two annotators labeled the preferred assistant action in 100 conflict scenarios without seeing the assigned source-correctness case, reaching four-class inter-annotator agreement (Cohen’s κ=0.785κ=0.785). The annotators generally preferred responses that explicitly surfaced a disagreement rather than silently selecting one source. When neither source was correct, they preferred an explicit statement of uncertainty to an unexplained refusal. These annotations support treating conflict acknowledgment as a separate outcome and flagged abstention as the target when neither source is correct. 4 Experimental Setup We evaluate arbitration behavior on five open-weight LLMs using the proposed benchmark and a preregistered evaluation protocol. Beyond measuring baseline performance, we investigate two classes of mitigation methods: prompting and parameter-efficient fine-tuning, and assess the robustness of the resulting behavior to changes in prompt formulation, presentation format, and evaluation setting. Unless stated otherwise, all experiments follow the same inference and scoring protocol described below. 4.1 Models and inference We evaluate five open-weight models: Llama-3.1-8B11 1 https://hf.co/meta-llama/Llama-3.1-8B, Llama-3.1-8B-Instruct22 2 https://hf.co/meta-llama/Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct33 3 https://hf.co/Qwen/Qwen2.5-7B-Instruct, gemma-2-9b-it44 4 https://hf.co/google/gemma-2-9b-it, and Mistral-7B-Instruct-v0.355 5 https://hf.co/mistralai/Mistral-7B-Instruct-v0.3. These models cover four families and include a direct comparison between the base and instruction-tuned Llama checkpoints. All models are served with vLLM on a single NVIDIA A100 80GB GPU and decoded greedily at temperature 0. 4.2 Evaluation implementation and metrics Closed-book prompt sensitivity The primary evaluation uses the required-best-guess prompt defined in the benchmark protocol. As a sensitivity analysis, we also evaluate a prompt that allows the model to respond that it does not know. Allowing abstention leaves some source-correctness cases nearly empty for refusal-prone models, so the required-best-guess condition is used for the primary analysis. Retention under this condition should therefore be interpreted as an upper bound on the model’s ability to retain a correct closed-book answer under ordinary interaction. Response scoring We request responses in the format FINAL: <answer> and extract the marked span, falling back to the final line when the marker is absent. Normalization- and alias-tolerant matching compares the extracted answer with the verified answer, the closed-book answer, and the observed tool return. Responses are classified as following the tool, retaining the closed-book answer, mentioning both, abstaining, or selecting another answer. The same extraction and matching procedure is applied to closed-book and tool-conditioned responses. For predefined ambiguous abstentions that cannot be resolved by the rule-based parser, a fixed LLM-based extractor returns a normalized span to the same matcher. Against blinded human labels on 167 responses, the deterministic scorer reaches κ=0.806κ=0.806, with recall 1.00 for tool following, 0.939 for retention, and 0.706 for abstention. The LLM-based normalization is restricted to these ambiguous abstentions and changes retention by at most 3.1 percentage points and incorrect-tool following by at most 0.6 percentage points. Metrics Correct-answer retention measures how often the model returns the verified answer when its closed-book answer is correct and the tool return is incorrect. Correct-tool following measures how often it follows a verified-correct return when its closed-book answer is incorrect. Conflict-resolved accuracy measures verified-answer accuracy across these two cases. Incorrect-tool following measures adoption of the tool return when neither source is correct, and tool-error abstention measures abstention after a structured tool error. Higher values are desirable for all metrics except incorrect-tool following. Agreement-case accuracy and conflict acknowledgment are reported as secondary metrics. 4.3 Mitigation methods Success criterion We consider a mitigation method successful if correct-answer retention improves significantly and no statistically significant reduction in correct-tool following is detected. This criterion rules out methods that resist incorrect returns only by inducing general distrust of tool outputs. All comparisons use common question support. Prompting We evaluate three prompting strategies on the four instruction-tuned models: a warning that the tool return may be unreliable, a source-assessment instruction asking the model to determine which source is more likely correct, and an abstain-and-flag instruction allowing the model to withhold an answer while reporting the disagreement. Prompted and standard-prompt runs receive identical questions, returns, formulations, and source-correctness labels. Fine-tuning We compare SFT and DPO (Rafailov et al. 2023) using LoRA adapters over frozen backbone models. Each question supplies a verified answer g, a near distractor dneard_near, and two balanced return conditions. When the tool return is incorrect, g is preferred and the return is rejected; when the tool return is correct, the return is preferred and dneard_near is rejected. SFT uses the same inputs and preferred completions without rejected responses. Training uses only near distractors, while far distractors are reserved for evaluation. Cross-fitting We partition questions into two folds while keeping complete ToolHop chains and related-question groups together. Fold A contains 287 questions from 154 chains, and fold B contains 288 questions from 155 chains, with no chain shared between folds. For each model and training objective, we train on one fold and evaluate on the other, then reverse the direction. Matched SFT and DPO runs use the same checkpoint, LoRA configuration, hyperparameters, and initial seed. Because fine-tuning can change the closed-book answer and hence source-correctness case membership, each fine-tuned checkpoint is compared with its corresponding original checkpoint only on questions assigned to the relevant case for both checkpoints. Folds are cut on the 575-question construction pool, before the semantic exclusions; the 33 excluded questions are therefore present during training but never scored. 4.4 Robustness analyses Each item is evaluated under a reference prompt and two instruction-wording variants while holding the question text, verified answer, tool call, and return fixed. Primary estimates pool the three formulations, while formulation-specific estimates measure sensitivity to wording. A cross-model ordering is considered stable only if it holds in the pooled analysis and under every formulation. We also compare three presentations of the same external value: a structured tool return with its schema, the same return without the schema, and a retrieval-style passage. Finally, we test transfer in both directions using an independently constructed evaluation set based on the FaithEval counterfactual split and verified ARC-Challenge questions (Ming et al. 2025). 4.5 Statistical analysis Prompted runs are compared with standard-prompt runs of the same checkpoint, whereas fine-tuned checkpoints are compared with their corresponding original checkpoints. Confidence intervals are obtained by bootstrap resampling with questions as clusters. Paired comparisons use sign-flip permutation tests with Holm correction within each preregistered family, while unpaired cross-model comparisons use two-sample proportion tests. Bootstrap intervals quantify estimation uncertainty, and the corrected permutation tests determine statistical significance. Model Adaptation Ret.↑ Tool↑ Wrong↓ Err.↑ Llama-3.1-8B-Instruct – 17.1 93.1 78.6 84.4 SFT 31.631.6 92.892.8 78.678.6 70.1†70.1 DPO 22.322.3 92.192.1 78.078.0 85.985.9 gemma-2-9b-it – 9.2 86.3 79.4 99.5 SFT 26.926.9 91.891.8 83.6†83.6 97.9†97.9 DPO 18.618.6 86.586.5 77.2†77.2 98.4†98.4 Qwen2.5-7B-Instruct – 6.5 91.8 86.0 80.6 SFT 3.33.3 97.797.7 92.0†92.0 77.7†77.7 DPO 6.36.3 91.291.2 85.785.7 79.3†79.3 Mistral-7B-Instruct-v0.3 – 10.9 86.0 78.4 73.9 SFT 53.453.4 57.957.9 37.6†37.6 16.2†16.2 DPO 16.416.4 81.581.5 69.9†69.9 70.2†70.2 Table 1: Performance before and after fine-tuning on MemToC, pooled across three prompt formulations. Paired changes reported in the text use the corresponding original checkpoint restricted to the same support and need not equal raw differences between displayed rows. Ret., Tool, Wrong, and Err. denote correct-answer retention, correct-tool following, incorrect-tool following, and tool-error abstention, respectively. A dash denotes the original instruction-tuned checkpoint with no additional adaptation, evaluated using the standard benchmark prompt. These rows report full pooled rates, whereas SFT and DPO rows report held-out cross-fitted rates on adaptation-specific case-stable common support. Bold marks a Holm-significant change on a decision metric, † marks an uncorrected p<0.05p<0.05 on a diagnostic metric, and green or red indicates a desirable or undesirable direction. 5 Results We first characterize baseline arbitration behavior under controlled memory–tool conflicts, then evaluate whether supervised fine-tuning and preference optimization improve conflict resolution without degrading appropriate tool use. We next assess the robustness of these findings to changes in prompt formulation and present additional analyses, controls, dataset transfer experiments and model size. The main quantitative results are summarized in Table 1. 5.1 Default arbitration behavior Source selection across correctness cases Across the four instruction-tuned models, correct-answer retention ranges from 6.5% to 17.1%, whereas correct-tool following ranges from 86.0% to 93.1%. When neither source is correct, the models still repeat the incorrect tool return in 78.4% to 86.0% of eligible cases. Together, these results show that the models generally favor the tool return regardless of whether it is correct. Error handling Tool-error abstention varies from 73.9% for Mistral-7B-Instruct-v0.3 to 99.5% for gemma-2-9b-it. This variation does not correspond to behavior under factual conflicts: a model may abstain reliably after an explicit tool error while still following a plausible but incorrect return. The base Llama-3.1-8B checkpoint is excluded from pooled cross-formulation comparisons because the two alternative instruction wordings leave only 6 to 13 eligible questions in the relevant cases. 5.2 Fine-tuning results SFT The preregistered success criterion is met for Llama-3.1-8B-Instruct and gemma-2-9b-it. Retention increases by 14.3 points for Llama while no statistically significant change in correct-tool following is detected. For gemma, retention increases by 17.0 points and correct-tool following by 5.6 points, indicating improved discrimination rather than reduced tool use. SFT on Qwen does not improve retention and increases both correct-tool following by 5.9 points and incorrect-tool following by 5.5 points, indicating greater responsiveness to tool returns regardless of correctness. SFT on Mistral increases retention by 41.8 points but reduces correct-tool following by 27.6 points, incorrect-tool following by 40.7 points, and tool-error abstention by 57.7 points. It therefore improves resistance to incorrect returns by inducing a broad reduction in tool use rather than better arbitration. DPO The criterion is likewise met for Llama and Gemma, increasing retention by 5.8 and 9.3 points, respectively, with no detected reduction in correct-tool following. DPO on gemma additionally reduces incorrect-tool following by 2.4 points with uncorrected p<0.05p<0.05. DPO on Qwen produces no significant change in either decision metric. DPO on Mistral increases retention by 6.0 points but reduces correct-tool following by 4.5 points, reproducing the trade-off observed under SFT at a smaller magnitude. Cross-model pattern SFT and DPO satisfy the criterion on the same two models and fail on the same two models. Within this model set, fine-tuning outcomes therefore vary more consistently with the model than with the optimization objective. Tool-error abstention decreases at the uncorrected level in seven of the eight model–objective combinations; DPO on Llama is the only exception, with a non-significant increase of 1.4 points. 5.3 Prompt formulation changes model rankings Across the three formulations, original-checkpoint retention varies by 7 to 15 percentage points within an instruction-tuned model — comparable with or larger than most cross-model differences. The ranges overlap and rankings reverse: Qwen2.5-7B-Instruct has the highest retention on the Reference formulation and the lowest on Paraphrase A. Incorrect-tool following varies by 11 to 18 points, and no ordering holds both pooled and under every formulation. Sensitivity to wording is not confined to the question. The with-tool prompt carries one protocol sentence stating that the tool cannot be called again, with no epistemic content about sources. Removing it leaves arbitration untouched — retention on Llama-3.1-8B-Instruct moves by −1.9-1.9 points ([−6.8,+3.1][-6.8,+3.1], p=0.65p=0.65, n=162n=162) — but it does move behavior where the tool is right: correct-tool following falls by 9.3 points ([−13.8,−4.8][-13.8,-4.8], p=0.0002p=0.0002, n=378n=378) and no-conflict accuracy by 5.6 ([−9.5,−1.1][-9.5,-1.1], p=0.011p=0.011). A sentence carrying no information about source reliability therefore buys compliance with a correct tool rather than resistance to an incorrect one. This also qualifies the fine-tuning conclusions: only SFT on gemma satisfies the criterion under every formulation separately, while DPO on Mistral fails the pooled criterion but satisfies it under Paraphrase A. The stable conclusions are therefore directional rather than rank-based. 5.4 Additional analyses and controls Prompting is not a clean alternative On Llama-3.1-8B-Instruct the warning and source-assessment prompts increase retention by 13 and 32 points but reduce correct-tool following by 7 and 9 points and tool-error abstention by 61 and 54, while the abstain-and-flag prompt preserves abstention but cuts correct-tool following from 88% to 24%. Across four models and three strategies, pooled retention rises in 11 of 12 combinations, but only three increases persist across all formulations, and every strategy reduces tool-error abstention for every model: prompting shows the same trade-off as fine-tuning. Distractor distance changes measured deference How far the incorrect return sits from the verified answer is a construction choice, and it moves the measurement: a deliberately distant distractor of the same type raises retention on Llama-3.1-8B by 19.4 points (Holm p=0.0015p=0.0015, n=103n=103) while none of the four instruction-tuned models shows a significant change (Holm p=1.0p=1.0), and every model repeats a distant incorrect value less often when neither source is correct. Distance is thus registered by all five models but translated into higher retention only by the pretrained Llama checkpoint (Appendix E). A separate probe finds the model’s own correct answer near the top of the representation on two of four families even as it follows the wrong return, so the failure there is not one of availability (Appendix H). Presentation format changes measured deference Replacing the executed-tool frame with a retrieval-style passage carrying the same payload reduces incorrect-tool following by 11 points for Llama-3.1-8B-Instruct and 35 for Qwen2.5-7B-Instruct, with null contrasts for the other three. It is not entirely selective: for Llama-Instruct, following decreases by 12 points when the external value is correct and 11 when it is incorrect, consistent with general distrust of the frame, while Qwen retains a larger selective component (−25-25 correct, −35-35 incorrect). Text-based evaluations may therefore understate deference in executed-tool settings. Tool-choice protocol changes measured reliability A dev-stage probe replays the memory-right slice of this benchmark set (324 episodes) through three tool-choice protocols on the native function-calling channel. Reliability is a model × protocol interaction: Llama-3.1-8B-Instruct follows a named schema-guided call on 0.992 of episodes but 0.328 under free, unconstrained auto choice (Δ −0.664-0.664 [−0.744,−0.576][-0.744,-0.576], replicated under both instruction-wording variants, in an independent re-run, and under sampling at T=0.7T=0.7), while Qwen2.5-7B is unmoved. The traces locate the failure in schema-invalid arguments; constraining the decoder to the schema on the same weights repairs it (0.328→0.9920.328→ 0.992) where preference fine-tuning on our pairs does not (0.3600.360) — a repair on the interface, not on arbitration (Appendix G). Conflict resolution is silent Two annotators independently and blind labeled 120 incorrect-tool responses for explicit acknowledgment of the disagreement. None acknowledges it: 0 of 120, an exact 95% binomial interval of [0.000,0.030][0.000,0.030], zero positives on all five models and in both conflict cases, raw agreement 118/120; a chance-corrected coefficient is uninformative when the positive class is empty. The same measurement on the previous, pre-curation version found 11 in 118, 0.0930.093 [0.048,0.161][0.048,0.161] — disjoint intervals, so the decrease is not sampling noise, but the benchmark set and the rubric moved together and it cannot be attributed to either alone. Detectors remain screening tools rather than prevalence estimators: on those labels a regex detector recovered 2 of the 11 and an LLM-based detector reached recall 0.73 at precision 0.38, and the two-stage estimate they support, 1% to 16% per arm, lies above the rate measured directly on the current benchmark set. Transfer beyond MemToC is partial On an independently constructed FaithEval and ARC set, DPO on gemma raises retention from 66.9% to 78.1% under a direct-value frame and from 31.4% to 36.1% under a passage frame, and SFT on Llama raises it from 24.1% to 28.5% under the passage frame. Both adapters also reduce abstention on unanswerable questions, from 28.2% to 20.6% and from 17.4% to 10.6% respectively. In reverse, adapters trained only on FaithEval and ARC — sharing no MemToC components — raise retention on the untouched MemToC set for gemma-2-9b-it by 4.6 points (Holm p<0.001p<0.001) while preserving correct-tool following, and produce no significant change for Llama-3.1-8B-Instruct (1.2 points, Holm p=0.35p=0.35): the learned behavior is not exclusively a signature of MemToC, yet its generalization remains model-dependent. The multiple-choice matcher was repaired after the fine-tuning results had been inspected, per the preregistered offline-rescoring remedy; this flipped one preregistered retention gate for Llama with SFT from fail to pass, and left the earlier independent gate and the gemma estimate unchanged. Confidence intervals for all estimates are in Appendix F.1. Scale does not resolve the trade-off A single 70B probe (Llama-3.3-70B-Instruct, served as a public AWQ-int4 requantization, one seed) defends correct memory better than any 8–9B instruct model — retention 21% pooled against 6.5–17.1% — and follows a correct tool most often (96%). It is, however, more deferent when neither source is correct (85%) and markedly worse at refusing a broken tool (44% against 74–99%). At 21% retention it still abandons its own verified-correct answer in roughly four conflict cases out of five. Scale is confounded with model generation and quantization here, and this is a single point, so we read it descriptively (Appendix F.2). 6 Limitations The final results come from a single benchmark core, ToolHop, and from open-weight models, primarily in the 7–9B range; no proprietary model is evaluated. The authored distractor layer is one construction realization, most machine-authored distractors (426 of 575) came from a single generator, and all are typed synthetic substitutions in executed returns rather than naturally occurring database ambiguities; a second quality-controlled realization remains future work. The question-level rebuild changes cell composition, preventing comparison across revisions. Additional training seeds were evaluated only for the two models that satisfied the criterion, so seed-specific failures cannot be ruled out for the other two, and we did not evaluate training-free conflict-aware decoding (Jiang et al. 2026), named as a baseline in the preregistration. The matched-presentation analysis uses a single seed and all frames retain agentive wording. Transfer is evaluated on one external multiple-choice dataset, on small verified pools (279 value-frame and 185 passage-frame episodes), so null results within a frame may reflect limited power; all transfer estimates are restricted to human-verified examples, since a further 687-row extension has not been verified. Forced elicitation makes the measured retention rates upper bounds, and neither protocol gives direct access to a model’s beliefs. Tool-error abstention is measured with a single error payload, a structured service-unavailable return, and the rate turns on which error is shown. On the two models we could re-run, using one prompt formulation and 542 paired questions per model, that payload gives the highest abstention of the three kinds tested, and the two group the kinds differently: Qwen2.5-7B-Instruct abstains on 0.89 of service-unavailable returns against 0.43 under both timeout and permission-denied, whereas Llama-3.1-8B-Instruct returns the same pooled rate under timeout as under service-unavailable, 0.88 for both, and falls to 0.51 under permission-denied alone. That equality is a property of the rate rather than of the behavior: 84 of that model’s 542 episodes change response category between those two payloads, in offsetting directions. Nor is a high rate desirable in itself: on that Llama, among the 162 memory-correct questions, a permission error raises fall-back to the correct closed-book answer from 0.09 to 0.35 (paired +0.25+0.25, 95% interval [+0.19,+0.33][+0.19,+0.33]); among the 378 memory-wrong questions, it lowers abstention from 0.90 to 0.54 (paired −0.36-0.36, 95% interval [−0.41,−0.31][-0.41,-0.31]). We therefore read the reported rate as specific to the payload it was measured on rather than as a property of tool failure, and note that models differ both in how readily they abstain and in how sensitive that is to the kind of error. The core human labels were produced by two annotators who are also authors, so external validation remains pending; the acknowledgment measurement has an empty positive class, bounding the rate from above rather than estimating it. An LLM-based assistant was used under author direction; the authors verified every reported number against the released artifacts. 7 Conclusion MemToC reframes tool–memory conflict as correctness-conditioned arbitration: controlling the correctness of an executable tool return while independently measuring each model’s closed-book answer separates appropriate tool use from harmful deference, and correct-answer retention from indiscriminate distrust. Across five open-weight 7–9B models tool returns strongly dominate elicited closed-book answers, conflict resolution is silent — 0 of 120 annotated conflict responses acknowledge the disagreement, 95% upper bound 3% — and rankings change across prompt formulations and presentation frames, so tool-deference rates are not stable model properties. Arbitration can be improved, but model-dependently and rarely selectively: cross-fitted SFT and DPO satisfy our asymmetric criterion on the same two of four instruction-tuned backbones, and transfer beyond MemToC is partial. More importantly, 19 of 20 tested method–model combinations reduce abstention after tool errors or on unanswerable inputs, so evaluating conflict mitigation only by resistance to incorrect returns can reward broad distrust, not better arbitration. Future work should target selective improvements that preserve correct tool use and failure handling, validated across independent constructions and interaction formats. 8 Data and Code Availability The benchmark, the construction and evaluation code, the scoring layer the reported numbers are read from, and a per-claim map from each number to the file that carries it are released under Apache-2.0 at https://github.com/ArceniyVarlamov/memtoc. The release carries a verification script that recomputes 180 of the values reported here from that scoring layer and diffs each against the printed number; the values it cannot reach are named there rather than omitted. The three-payload tool-error comparison of §6 is a separate run and is not part of that release. References Bi et al. (2025) B. Bi, S. Huang, Y. Wang, T. Yang, Z. Zhang, H. Huang, L. Mei, J. Fang, Z. Li, F. Wei, W. Deng, F. Sun, Q. Zhang, and S. Liu Context-DPO: aligning language models for context-faithfulness. In Findings of the Association for Computational Linguistics: ACL 2025, p. 10280–10300. External Links: 2412.15280, Link, Document Cited by: §2. Chen et al. (2026) Y. Chen, V. Siu, Y. Liu, D. Song, and C. Wang Controlling tool use with heading-specific activation steering. arXiv preprint arXiv:2607.05790. External Links: 2607.05790 Cited by: §2. Cheng et al. (2026) J. Cheng, R. Pan, and H. Amiri Investigating tool-memory conflicts in tool-augmented large language models. Note: Workshop on Reliable and Responsible Foundation Models @ ICML 2025 External Links: 2601.09760 Cited by: §1, §2. Goodfire (2025) Goodfire Sparse autoencoders for llama-3.1-8b-instruct. Note: https://w.goodfire.ai/blog/sae-open-source-announcement Cited by: Appendix C. He et al. (2024) Z. He, W. Shu, X. Ge, L. Chen, J. Wang, Y. Zhou, F. Liu, Q. Guo, X. Huang, Z. Wu, Y. Jiang, and X. Qiu Llama Scope: extracting millions of features from llama-3.1-8b with sparse autoencoders. External Links: 2410.20526 Cited by: Appendix C. Jiang et al. (2026) R. Jiang, T. Wu, Y. Wang, B. Zhu, and L. Huang From context-aware to conflict-aware: generalizing contrastive decoding for knowledge conflict in LLMs. arXiv preprint arXiv:2606.10298. External Links: 2606.10298 Cited by: §2, §6. Li et al. (2026) R. Li, P. Huang, Z. Liu, Y. Yan, H. Xie, Y. Gu, G. Yu, and M. Sun SHIFT: gate-modulated activation steering for knowledge conflict mitigation in retrieval-augmented generation. External Links: 2606.27786 Cited by: §2. Lieberum et al. (2024) T. Lieberum, S. Rajamanoharan, A. Conmy, L. Smith, N. Sonnerat, V. Varma, J. Kramár, A. Dragan, R. Shah, and N. Nanda Gemma Scope: open sparse autoencoders everywhere all at once on gemma 2. External Links: 2408.05147 Cited by: Appendix C. Lin (2026) J. Lin Neuronpedia: open source interpretability platform. Note: https://w.neuronpedia.orgAccessed 2026-07-12 Cited by: Appendix C. Longpre et al. (2021) S. Longpre, K. Perisetla, A. Chen, N. Ramesh, C. DuBois, and S. Singh Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, External Links: Link Cited by: §1. Ming et al. (2025) Y. Ming, S. Purushwalkam, S. Pandit, Z. Ke, X. Nguyen, C. Xiong, and S. Joty FaithEval: can your language model stay faithful to context, even if “the moon is made of marshmallows”. In The Thirteenth International Conference on Learning Representations (ICLR), External Links: 2410.03727 Cited by: Appendix B, §F.1, §4.4. Neeman et al. (2023) E. Neeman, R. Aharoni, O. Honovich, L. Choshen, I. Szpektor, and O. Abend DisentQA: disentangling parametric and contextual knowledge with counterfactual question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), External Links: Link Cited by: §1. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn Direct preference optimization: your language model is secretly a reward model. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Cited by: §2, §4.3. Soni (2026) H. Soni ToolFailBench: diagnosing tool-use failures in LLM agents. External Links: 2607.04686 Cited by: §2. Sun et al. (2026) Y. Sun, W. Shang, W. Gao, X. Huang, and J. Ma Diagnosing LLM arbitration behavior over pre-evidence epistemic states in RAG-based fact-checking. arXiv preprint arXiv:2606.01120. External Links: 2606.01120 Cited by: §1. Tian et al. (2026) Y. Tian, Z. Shi, Y. Zhou, and B. Zhao Beyond function calling: benchmarking tool-using agents under tool-environment unreliability. arXiv preprint arXiv:2606.25819. External Links: 2606.25819 Cited by: §2. Wang et al. (2026a) B. Wang, H. Huang, Y. Li, Y. Zhou, J. Teng, Z. Yang, G. Shi, and C. Feng Regime-aware peer specialization for robust RAG under heterogeneous knowledge conflicts. arXiv preprint arXiv:2606.30518. External Links: 2606.30518 Cited by: §2. Wang et al. (2025) F. Wang, X. Wan, R. Sun, J. Chen, and S. Ö. Arık Astute RAG: overcoming imperfect retrieval augmentation and knowledge conflicts for large language models. In Annual Meeting of the Association for Computational Linguistics (ACL), External Links: 2410.07176 Cited by: §2. Wang et al. (2026b) Y. Wang, R. Zhou, Y. Ma, R. Fu, J. Liang, S. Cao, M. Huang, T. Fang, and L. Pan ASA: backbone-training-free representation engineering for tool-calling agents. External Links: 2602.04935 Cited by: §2. Wang and Vemuri (2026) Z. Wang and P. Vemuri When the tool decides: LLM agents defer blindly to graph neural network tools, and stronger backbones defer more. External Links: 2606.14476 Cited by: §1, §2. Wu et al. (2025) Z. Wu, A. Arora, A. Geiger, Z. Wang, J. Huang, D. Jurafsky, C. D. Manning, and C. Potts AxBench: steering LLMs? even simple baselines outperform sparse autoencoders. External Links: 2501.17148, Link Cited by: Appendix C. Xie et al. (2024) J. Xie, K. Zhang, J. Chen, R. Lou, and Y. Su Adaptive chameleon or stubborn sloth: revealing the behavior of large language models in knowledge conflicts. In The Twelfth International Conference on Learning Representations (ICLR), External Links: 2305.13300 Cited by: §1. Xin et al. (2025) C. Xin, S. Zhou, H. Zhu, W. Wang, X. Chen, X. Guan, Y. Lu, H. Lin, X. Han, and L. Sun Sparse latents steer retrieval-augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 4547–4562. External Links: Link Cited by: Appendix C, Appendix C, §2. Yan et al. (2026) L. Yan, R. Li, X. Han, W. Li, B. Wang, L. Wang, C. Lyu, and G. Chen Trust no tool: evaluating and defending LLM agents under untrusted tool feedback. External Links: 2605.17453 Cited by: §2. Ye et al. (2025) J. Ye, Z. Du, X. Yao, W. Lin, Y. Xu, Z. Chen, Z. Wang, S. Zhu, Z. Xi, S. Yuan, T. Gui, Q. Zhang, X. Huang, and J. Chen ToolHop: a query-driven benchmark for evaluating large language models in multi-hop tool use. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, p. 2995–3021. External Links: Link, Document Cited by: §1, §3.1. Zeng et al. (2026) Y. Zeng, S. You, Y. Liu, Q. Du, X. Ding, Y. Hou, Y. Wang, W. Ning, H. Song, D. Tu, B. Cai, and T. Liu The tool-overuse illusion: why does LLM prefer external tools over internal knowledge?. External Links: 2604.19749 Cited by: §2. Zhang et al. (2026) K. Zhang, Z. Xiong, M. Zhong, Z. Jiang, Z. Yuan, Z. Li, and Y. Lin Are tools all we need? unveiling the tool-use tax in LLM agents. External Links: 2605.00136 Cited by: §2. Zhang et al. (2025) R. Zhang, Y. Xu, Y. Xiao, R. Zhu, X. Jiang, X. Chu, J. Zhao, and Y. Wang KnowPO: knowledge-aware preference optimization for controllable knowledge selection in retrieval-augmented language models. In Proceedings of the AAAI Conference on Artificial Intelligence, External Links: 2408.03297 Cited by: §2. Zhang et al. (2024) Y. Zhang, J. Chen, J. Wang, Y. Liu, C. Yang, C. Shi, X. Zhu, Z. Lin, H. Wan, Y. Yang, T. Sakai, T. Feng, and H. Yamana ToolBeHonest: a multi-level hallucination diagnostic benchmark for tool-augmented large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, External Links: Link Cited by: §2. Zhao et al. (2024) Y. Zhao, A. Devoto, G. Hong, X. Du, A. P. Gema, H. Wang, X. He, K. Wong, and P. Minervini Steering knowledge selection behaviours in llms via sae-based representation engineering. External Links: 2410.15999, Link Cited by: Appendix C, §2. Zhu et al. (2026a) D. Zhu, X. Ma, Y. Shen, X. Li, Y. Zhao, S. Wang, L. Yan, and D. Yin When tools fail: benchmarking dynamic replanning and anomaly recovery in LLM agents. External Links: 2606.05806 Cited by: §2. Zhu et al. (2026b) X. Zhu, Z. Wang, K. Mei, W. Xu, M. Guo, B. Yang, J. Fan, and D. N. Metaxas Trust or abstain? a self-aware RAG approach. arXiv preprint arXiv:2605.18792. External Links: 2605.18792 Cited by: §2. Appendix A Prompt Templates, by Experiment This appendix gives every prompt the evaluation uses, each labelled with the experiment it belongs to and the point in the protocol at which it is applied. All templates are shown in the Reference formulation; Paraphrase A and Paraphrase B differ from it in the wording of both the closed-book and with-tool instructions, while the question text and episode content remain fixed, and all three ship with the released material. Placeholders in braces are filled per episode. Display note: the blocks below are wrapped for column width; in the actual templates each instruction paragraph is a single line, and newlines occur only before the Tool schema:, Tool output:, End your reply, and Question: lines. A1. One benchmark episode, verbatim. What a single question looks like after construction (§3.2), with the model’s two answers to it: the tool schema is elided and the text is display-wrapped. Question: Which actor played Colonel Sherman T. Potter in "M*A*S*H"? [gold: Harry Morgan] Tool schema: "name": "actor_lookup", ... Tool output: "result": "McLean Stevenson" [curated distractor: the actor of a *different* M*A*S*H commanding officer] Closed-book (same model, forced elicit.): FINAL: Harry Morgan played Colonel Sherman T. Potter in "M*A*S*H". With the wrong tool return: FINAL: McLean Stevenson played Colonel Sherman T. Potter in "M*A*S*H". A2. Main evaluation prompt. Used for every condition that carries a tool return — correct return, incorrect return, tool error and no conflict — in the default-behaviour results of §5.1, in the fine-tuning evaluation of §5.2, and as the untreated baseline of both intervention experiments. You called the tool below; its output is shown. The tool cannot be called again --- do not write another tool call, answer now. Use the output together with your judgment to answer the question. Tool schema: schema Tool output: tool_output End your reply with one line that starts with ’FINAL: ’ followed by your answer. Question: question A3. Closed-book elicitation prompt. Run once per model and formulation before any evaluation prompt, with no tool present, to record what the model answers from its own knowledge. That answer defines which correctness case each question falls into for that model (§3.3), and is the reference against which correct-answer retention is scored. Answer the question using ONLY your own internal knowledge. Do not assume any external tool or document. Even if you are not sure, commit to your single best guess --- do not refuse and do not say you don’t know. End your reply with one line that starts with ’FINAL: ’ followed by your answer. Question: question A4. Closed-book elicitation, permissive variant. The comparison arm of the elicitation-protocol choice reported in Appendix D: identical to A3 except that the third, best-guess sentence is replaced by "If you do not know, write UNKNOWN." A5. Prompting-strategy sentences. The three prompting interventions of §4.3, whose results are in §5.4 and Appendix E. Each is a single sentence inserted into prompt A2 — one strategy per arm — immediately after the Tool output: line and before the FINAL-line instruction. Nothing else in the prompt changes. • warn: "Note: tool outputs may be wrong or stale; weigh the tool output against your own knowledge before answering." • source-priority: "If you are confident in your own knowledge and it contradicts the tool output, trust your own knowledge; follow the tool output only when you have no confident knowledge of your own." • abstain-and-flag: "If the tool output conflicts with your own knowledge, state the discrepancy explicitly in one sentence before your final line. If you cannot tell which side is right, answer ’FINAL: UNKNOWN'." A6. Retrieval-passage prompt. The third arm of the presentation-format comparison of §4.4, results in §5.4. It replaces the header of prompt A2 with the one below; the payload, the question and the FINAL-line instruction are identical, so the only difference from A2 is how the same external value is presented. The second arm of that comparison is prompt A2 with the tool schema omitted. You retrieved the snippet below from a document search; it is shown. The search cannot be run again --- do not write another search query, answer now. Use the snippet together with your judgment to answer the question. Retrieved snippet: tool_output Question Which actor played Colonel Sherman T. Potter in “M*A*S*H”? verified: Harry Morgan Closed-book answer same model, no tool FINAL: Harry Morgan correct Observed tool condition the same question under four observations: Correct return carries the verified answer both sources correct Incorrect return typed distractor closed-book only correct "result": "McLean Stevenson" — another M*A*S*H commander, same entity type FINAL: McLean Stevenson wrong tool adopted Tool error "error": ... desired: use verified memory No tool no return observed desired: answer from memory Figure 3: One MemToC episode under four tool-observation conditions. The question and the independently elicited closed-book answer are held fixed while the observed tool return changes. Correct and incorrect factual returns instantiate cells of the correctness matrix; tool error and no tool are evaluated as separate controls. The displayed model outputs are verbatim. Appendix B Experimental Setup and Reproducibility This appendix states how the runs behind the reported numbers were configured, gives the intervention and robustness protocols of §4.3 and §4.4 in full, defines how each metric is computed, describes the earlier benchmark versions that some measurements were taken on, and says which released file carries which number. It adds no result. Serving, training and software stack. All inference runs on a single NVIDIA A100-80GB (shared JupyterHub node) under vLLM 0.8.5.post1 with torch 2.6.0+cu124 and transformers 4.51.3, bf16 — the three-payload tool-error comparison of §6 is the one run in fp16 — temperature 0, one process per arm; the 70B point is the only run using a public AWQ-int4 requantization. LoRA training uses the transformers Trainer with peft adapters (deliberately not trl, to avoid version-fragile collators), the deployment chat template and prompt-token masking to −-100. Hyperparameters: LoRA r=16 on q/k/v/o, α=32, lr 1e-5, 1 epoch, DPO β=0.1, 574/576 correctness-balanced pairs per fold; each cross-fitted fold is a separate training run. The judge layer is Qwen2.5-7B-Instruct served identically (temperature 0, max 32 new tokens). Every run records its seed, code version, and the sha256 of both the weights and the episode file. Scoring and all aggregation are deterministic and re-runnable offline on CPU from the released per-episode records and summary layer, which is how every table in this paper is regenerated. Metric conventions. Correct-answer retention, correct-tool following, incorrect-tool following and case accuracy are read from the judge-normalized layer. Tool-error abstention and no-conflict accuracy are deterministic-scorer values; where the offline recomputation differs from the aggregator on a tool-error cell it does so by at most 0.010.01, and the tables report the aggregator’s value. Correct-tool following is pooled over the whole correct-return condition. Pooling across the three formulations is the unweighted mean of the three, not an n-weighted mean. Statistical procedure. The reading rule for each stress test was fixed before computation. Confidence intervals are percentile bootstrap with questions as clusters; paired comparisons use sign-flip permutation tests with Holm correction inside each preregistered family, and unpaired cross-model comparisons use two-proportion tests. Intervals quantify uncertainty and do not by themselves decide significance. The bootstrap interval and the Holm-corrected permutation p disagree in ten of 908 contrasts, none of them a family metric on the headline slice; those readings are reported as directional in the text. Intervals are rounded once, from the four-digit values in the released files, not from printed three-digit output. Baseline cross-model rates are computed on each model’s own eligible source-correctness cases. On the final canon, pairwise common-support tests are used for the preregistered base-versus-instruction-tuned contrasts; the headline ranges do not use an all-model intersection. Intervention and robustness protocols (§4.3 and §4.4). Success criterion. Prompting and fine-tuning interventions use the same preregistered asymmetric criterion: correct-answer retention must improve without a detected reduction in correct-tool following. This rule excludes interventions that resist incorrect tool returns only by inducing general distrust of all tool outputs. Each intervention is evaluated against the corresponding untreated checkpoint on common question support. Incorrect-tool following, tool-error abstention, agreement-case accuracy, and conflict acknowledgment are diagnostic outcomes rather than components of the success criterion. Prompting. We evaluate three prompting strategies on the four instruction-tuned models. The warning prompt states that the tool return may be unreliable, the source-assessment prompt asks the model to assess which source is more likely to be correct, and the abstain-and-flag prompt permits the model to withhold an answer and report the disagreement. Prompted and untreated variants receive the same questions, tool returns, formulations, and fixed source-correctness labels. Complete prompt templates are provided in Appendix A. Fine-tuning. We compare supervised fine-tuning (SFT) and direct preference optimization (DPO) using LoRA adapters while keeping the backbone weights frozen. We do not collect a separate preference-annotation set; instead, preference data are generated automatically from the benchmark. Each question provides a verified answer g, a plausible near distractor dneard_near, and two balanced tool-return conditions: tool_right returns g, whereas tool_wrong returns dneard_near. These conditions generate two DPO pairs per question. Under tool_wrong, g is preferred and the tool return is rejected; under tool_right, the tool return is preferred and dneard_near is rejected. Thus, DPO does not teach unconditional trust in either source: it teaches the model to agree with the tool when it is correct and to override it when it is incorrect. Only near, plausible distractors are used for training; far distractors are reserved for evaluation. SFT uses the same condition-specific inputs and preferred completions, but without rejected responses. The SFT loss is computed only over completion tokens, not over the prompt or tool-return context. Matched SFT and DPO runs use the same base checkpoint, hyperparameters, LoRA configuration, and initial random seed, so the training objective is the intended difference. Cross-fitting and seeds. To prevent identical or related questions from appearing in both training and evaluation, we split the data at the level of complete ToolHop chains rather than individual example IDs. The first fold contains 287 questions from 154 chains, and the second contains 288 questions from 155 chains; no chain is shared between the folds. These folds yield 574 and 576 DPO pairs, respectively. For each of the four instruction-tuned models and each training objective, we train one version on the first fold and evaluate it on the second, then reverse the direction, resulting in 16 training runs in total. Each version starts from the same corresponding base model, and only questions held out from that version’s training are included in its evaluation. Combining the two held-out prediction sets covers all 575 questions. This number refers to the pool entering data construction and cross-fitting, whereas the final metrics are computed on the approved 542-question analysis set; the remaining 33 questions are excluded only when results are calculated, not during training. The distinction is necessary because the earlier split procedure compared only example identifiers, producing overlaps of 184 questions from 120 chains with the new evaluation set and 130 questions from 94 chains with one cleaned version. The current procedure keeps related question groups within the same fold. Prompt formulation. Each question is evaluated under three prompt formulations — Reference, Paraphrase A and Paraphrase B — with the question text, verified answer, tool call and tool return held byte-identical across them; what varies is the wording of the closed-book and with-tool instructions (Appendix A). Primary intervention results are pooled across the three formulations, and formulation-specific estimates are reported as a sensitivity analysis. A cross-model ordering is considered stable only if it holds in the pooled analysis and under every formulation. Cross-formulation comparisons focus on the four instruction-tuned models because Meta-Llama-3.1-8B produces too few usable closed-book cases under the two wording variants. Presentation format. We compare three matched presentations of the same external value: the standard executed-tool return with its schema, the same return without the schema, and a retrieval-style passage. The question, payload, and required answer format are held fixed. Correct-value and incorrect-value following are evaluated separately to distinguish selective rejection of incorrect evidence from a general shift in source trust. Measurement validation. Before inspecting aggregate results, we compare the scoring pipeline with blinded human labels on a stratified sample of 167 responses, evaluating final-answer correctness and conflict acknowledgment separately. The acknowledgment labelling, the detector measurements retained from the previous benchmark version, and the prevalence estimator they support are described with their result in Section 5.4 and in Appendix D. Transfer. We test transfer using an independently constructed counterfactual evaluation based on the FaithEval counterfactual split and verified ARC-Challenge questions (Ming et al. 2025). For forward transfer, selected MemToC-trained adapters are evaluated on the external set with conflicting information presented either as a direct value or in a passage. For reverse transfer, adapters trained exclusively on FaithEval and ARC examples are evaluated on the untouched MemToC set. These adapters contain no ToolHop questions, MemToC distractors, or other MemToC construction components. Earlier versions of the benchmark. The benchmark passed through several cleaning revisions before the released one. Every revision was a full re-scoring or re-run rather than a partial patch, and each was recorded with the shift it produced. Five were small: 76 malformed values were replaced in the substitution pool (cells shift ≤ 0.018); a normalization defect that truncated parenthesized suffixes was repaired (≤ 0.015); 24 questions whose verified answer was wrong were excluded, 13 found by human review, 3 by a full manual audit and 8 by a structural gold-judge with recall 1.00 on the known-bad set, all human-confirmed; 18 cross-kind substitutions were re-drawn, after which an automated type audit reports zero cross-kind and incorrect-tool following shifts by at most 0.03; and the judge’s span normalization was validated, raising outcome agreement from κ 0.707 to 0.803 while moving only the abstention channel elsewhere (|Δ|≤0.011| |≤ 0.011). Three revisions moved results enough to matter, and the main text reports the affected findings as construction-scoped. First, 271 of 276 substituted incorrect values were replaced by blind double annotation: incorrect-tool following rose by 0.15 to 0.22 on all five models, and prompting gains that had been present disappeared. Second, the benchmark was rebuilt at the level of the individual question rather than the reasoning-chain step — the construction described in §3.2 and Appendix D — after which incorrect-tool following rose to 0.71–0.83, the pretrained checkpoint moved from the bottom to the top of the retention spectrum, and retention on the arbitration case fell to 0.05–0.21. Third, every fine-tuned arm was retrained under group cross-fitting, because the preference pairs are drawn from the same question pool as the evaluation; the DPO retention gain on Llama-3.1-8B-Instruct fell from +0.24+0.24 to +0.058+0.058, SFT reached the success criterion on two models,the 70B retention figure fell from 0.837 to 0.213,and the tool-error abstention leak under DPO reversed sign to no detected leak. The main paper and Appendices D–H are measured on the released benchmark, with three exceptions that are labelled wherever they appear: the representation-steering probe (Appendix C), the acknowledgment-detector validation and the prevalence estimator it supports (Appendix D), and the switch-composition analysis (Appendix E). Those three were measured on the previous version, and no number from them is presented as evidence about the released one. Three further quantities changed between versions, and the paper reports the current value in each case: the effect of substitution plausibility, now measured by construction as the near-versus-far contrast and present on the pretrained checkpoint alone; the one-sentence protocol ablation, previously about 25 points on the arbitration case, which does not reproduce and relocates to correct-tool obedience; and the conflict-acknowledgment rate, previously 11 of 118, which is 0 of 120 under direct annotation on the released benchmark. Where each reported number comes from. Every number in the paper and in this document is recomputed from one of the released files, listed in Table 2. The released CLAIM-TO-FILE.md carries the same mapping at the level of individual claims, and tools/verify_paper_numbers.py re-derives 180 of the published values and diffs each against the printed one. Released file What is read from it benchmark/ the evaluated episodes, one file per formulation, plus the repair overlay results/construction/ the frozen substitution mapping: §3.2 branch counts, answer types, far coverage results/quality_control/ Step-7 review verdicts, the 542-question analysis set, cross-fitting folds results/scored_episodes/ per-episode scoring records for the 15 evaluated arms; the untreated rows of Table 1 recompute from these results/summaries/ every cell of Table 1, the prompting results, pooling across formulations, and all paired comparisons with their intervals results/finetuning/ the paired fine-tuning changes, the same contrasts as levels, and the 70B scale point results/distractor_distance/ the near-versus-far contrast and the protocol-sentence ablation results/acknowledgment/ the two-annotator acknowledgment round Table 2: Which released file carries which reported number. Raw model output text is not released; what the scored-episode files hold is the extracted answer span and the scoring decisions taken on it. Two families cannot be recomputed from the release and are named there rather than omitted: the absolute levels of the fine-tuned rows of Table 1, which need the 48 cross-fitted arms, and the three measurement layers taken on the previous benchmark version. Appendix C Steering the Representation with Sparse Autoencoders A dev-stage probe of SAE steering finds the representation-level knob direction-asymmetric: knocking the model off the tool is easy, re-inducing its parametric answer is not. We port the SpARE recipe (Zhao et al. 2024) to the tool setting on the two SAE-covered anchors (Llama Scope (He et al. 2024) with base→ transfer, Goodfire’s native-IT layer-19 dictionary (Goodfire 2025), GemmaScope IT and pre-trained dictionaries (Lieberum et al. 2024)). Features separating "kept memory" from "followed tool" are mined on the 506 held-out tool-wrong hops underlying the DPO pair set (zero overlap with evaluation, asserted at build), from a behavioural split under forced elicitation — without forcing, the kept-memory side is nearly empty — selected by mutual information with a frequency-separation cross-check (the two rankings agree on 16–19 of the top-20 features per layer; (Xin et al. 2025)), and applied as SpARE’s clamped remove/add edit at the last input position, edit strengths 1–12, on dev sets of 40 episodes per direction (deterministic scorer, standard evaluation prompt). Every dictionary–layer combination used for editing is measured against a reconstruction gate on our episodes’ activations (full-sequence FVU < 0.5): Llama Scope 0.42–0.51 across layers 12–15 — layer 12 marginally exceeds the bar and the multi-layer arm’s median (0.48) passes — GemmaScope IT 0.23/0.34 at layers 9/20, the pre-trained GemmaScope transfer 0.31–0.34 per arm at 23–26, and Goodfire 0.41 at layer 19; the base→ transfer costs about 0.1 to 0.13 FVU more than the other anchor’s native dictionary, consistent with (Xin et al. 2025). Stopping rules were pre-registered in stages: the first sweep closed the branch under an honest-negative rule (a knob no stronger than the prompt sentence ends the branch, no re-sweeps); the qualification run registered a direction-agnostic revival bar of half the prompting effect (+0.15 on the dev flip); and the final widening run, sanctioned separately, was registered with that bar restricted to the memory-ward direction a correctness-targeted policy needs. The outcome is asymmetric. Steering toward the tool works on gemma once the edit reaches the layer band where SpARE operates in ODQA (23–26, reachable only by transferred pre-trained dictionaries; gate passed at FVU 0.31–0.34): tool-following on kept-memory episodes rises 0.03→ 0.28 (+0.25 at strength 5) with format fully intact — past the +0.15 bar, but in the direction the policy does not need: the reverse direction of H3, realized at representation level. That widening run (feature budget K 0.5→ 0.7, strength to 12, same layers) pushes the tool-ward flip further still (+0.35) and the memory-ward flip not at all (the +0.05 ceiling is intact): the asymmetry widens with knob strength rather than closing. Steering toward memory — the direction arbitration-by-correctness needs — never exceeds +0.05 at the last input position on either anchor at any strength, layer set, feature budget, or dictionary, including Llama’s native-IT dictionary, which rules out transfer noise as the explanation. Pushed hard (strengths 8–12) the model demonstrably lets go of the tool (followed-tool −-0.30…−-0.35 on the multi-layer and pre-trained-transfer arms; the single-layer native dictionary manages only −-0.10; zero degeneration throughout) but lands on neither answer; even in the easy direction kept-memory drops by −-0.53 while tool-following rises only +0.25. A last probe, sanctioned separately under the same memory-ward bar, moves the edit from the input’s last position onto the tool-output span itself — the serialized payload the model would otherwise copy (6–19 tokens, all layers of the working band). Wherever generation stays intact the picture does not change: the memory-ward flip reaches +0.10 on gemma (strength 5) and +0.125 on Llama (strength 12, generation intact throughout), both under the +0.15 bar. On gemma at strengths 8–12 the nominal flip does cross the bar (+0.20–0.225 raw; +0.15–0.175 after hand-verification of every scored flip removes two scorer artifacts) — but the crossing is not usable steering: tool-following is annihilated outright (1.00→ 0.00, against −-0.30…−-0.35 at the same strengths at the last position), a third of the arm’s outputs collapse into schema echo, and the parametric answers that do surface frequently arrive with confabulated provenance — "the tool output states Grace Kelly" where it states another name. The edit corrupts the evidence representation until the model reads its own answer into it; it does not restore arbitration. Our reading, offered as interpretation rather than mechanism: in a tool chain the conflicting evidence stays in context and is re-read at every generation step, so an activation edit can suppress the copy-from-tool behaviour but cannot inject an answer the model must produce from parameters — a structural difference from single-shot ODQA, where SpARE steers both ways; editing the evidence span itself probes the same reading from the other side, and what appears past the stability limit is hallucinated reading of the corrupted span, not source arbitration. This matches benchmark-scale evidence that SAE steering is not competitive with simple baselines (Wu et al. 2025), and sharpens it: in this setting the failure is not uniform weakness but a missing direction — precisely the one a correctness-targeted policy would need to actuate. A feature-level reading of the mined dictionaries supports this interpretation. We retrieved automatically generated feature explanations and top activating tokens for all edited feature sets (Neuronpedia (Lin 2026); 426 features, 100% label coverage) and manually adjudicated the eighty top-MI features of the working dial against their activating tokens (all 426 features carry labels; the adjudication log is not part of this appendix). Entity- or fact-content features are rare there (5/80) and split across both sides of the separator: the dictionary offers essentially no content feature for the memory-ward edit to clamp. What the kept-memory side does contain — the only fully token-consistent cluster in the adjudication (11 features, 11/11 label–token agreement) — is epistemic-evaluation markers: contrast (differs, distinct), incorrectness (incorrect), validity (valid, lawfully), evidence sufficiency (evidence, sufficient), reliance on sources (reliance, rely), certainty (definitely), interpretation, hedged generality (usually, always), and inference (therefore, rather). The followed-tool side is dominated by reading-and-quoting functions: online search, direct quotation, question–answer format, document structure. The separator the SAE exposes is thus a dissent posture versus a copy posture, not stored content versus tool content — consistent with the behavioural asymmetry: posture features can be clamped, knocking the model off the tool or onto it, but no feature encodes the parametric answer that the memory-ward direction needs to induce. The label profile of the inert dictionaries is qualitatively similar, so the working dial is distinguished by its layer band rather than by an obviously richer feature inventory; and half of the labelled features (49%) activate on template- or token-specific patterns, consistent with the late-layer token-specificity caveat (Xin et al. 2025) — individual autointerp labels are weak evidence, and we rely only on the aggregate pattern. Appendix D Benchmark Construction, Quality Control, and Scorer Validation Table 3 inventories every human annotation pass behind the benchmark, with the reliability figure appropriate to each; the rubrics and the per-pass detail follow in this section. Where Pass Reliability i Step 2 Filter review, 54 borderline 19 restored i Step 4 Wording review, 605 candidates 30 removed i Step 5 Distractor authoring, 152 106 differ iv Step 7 Semantic review, 575 0.948 agreement v Step 5 Generator audits, 2 rounds 12% rejected vi §3.4 Target behaviour, 100 κ=0.785κ=0.785 vii §3.4 Full-instance audit, 150 κ=0.88κ=0.88–0.920.92 viii §5.4 Acknowledgment, 120 118/120 raw Table 3: The eight human annotation passes. Reliability in pass iv is measured on a blinded replication of 58 of the 575 records. In pass i divergence is expected rather than erroneous — several incorrect values can be equally plausible — so it is resolved by a rule fixed before adjudication, not reported as agreement. Pass vii also found 24 instances with incorrect verified answers, excluded from the corresponding analyses. Pass viii has an empty positive class, so no chance-corrected coefficient is informative. Typing fidelity (two numbers, two constructs). The automated audit — consistency with the type map — initially read 9.9% off-type in same-type arms (pilot v0: 58%); review traced genuine cross-kind errors to 3.9% of the near arms (12 episodes) and 2.0% of far (6): a person/place/organization gold paired with a work-valued distractor, from truncated Wikipedia disambiguation titles the NER pass had mis-typed. These were corrected at source (a documented type-map correction rule plus enumerated NER fixes), the affected distractors re-drawn from the clean same-type pool, and every affected arm re-run; after correction the automated audit is 0 cross-kind, with a residual 6% that is within-kind labelling noise (film vs. album vs. band). A stricter human QC on the pre-correction pool put cross-type contamination at 13% (near) and 18% (far) — an upper noise bound, since a full human re-audit of the corrected pool is future work. v2 curation mechanics. Candidate replacement values were drawn from ToolHop’s own value pool and screened with the binary plausibility check ("could a stale or buggy tool realistically return this value?") in three ranked rounds plus a shortlist curation round; final-round agreement 96.7%, κ 0.89, every decision recorded. The orthogonal structural-validity sweep covered all 149 unique question–gold pairs (two annotators, agreement 148/149, κ 0.93) and surfaced 8 malformed targets — questions whose gold cannot satisfy the question’s type — and a web-grounded factcheck of the remaining 141 golds (every verdict with a source URL) surfaced 2 world-incorrect ones; the resulting 28 episodes are excluded from all conditions by sibling exclusion at the hop level. Human QC sample composition (previous benchmark version). The executed pass covers 150 episodes and 120 conflict answers drawn from the validation-slice runs; of the 120 answer-level labels, 114 carry an agreed outcome label and 118 an acknowledgment label. The 6 remaining outcome disagreements are left unresolved. These are the denominators of the legacy acknowledgment anchor and of the detector-validation numbers below; the acknowledgment rate reported in §5.4 comes instead from the separate round described next, run on the released benchmark. Direct acknowledgment round on the released benchmark (§5.4). Two annotators independently and blind labelled 120 incorrect-tool responses sampled from the released benchmark’s own runs under an equal per-model quota (24 each for base Llama, Llama-Instruct, Qwen, gemma and Mistral) and spanning both conflict categories. The rubric is the narrowed v2 rubric described under Rubric history below: the response must reference the model’s own knowledge, so a bare denial of the tool value, or a complaint that the tool output is irrelevant to the question, does not count. Raw agreement is 118/120; the two disagreements were resolved to no by the rubric fixed before annotation. The acknowledged count is 0, giving an exact Clopper–Pearson 95% interval of [0.000,0.030][0.000,0.030], with zero on every model and in both categories. We deliberately do not report a chance-corrected coefficient: with an empty positive class the chance-correction term is degenerate, and the informative quantities are the raw agreement and the prevalence. Two caveats travel with this number. It bounds the rate from above rather than estimating it. And it differs from the legacy 11/118 anchor along two axes at once — the benchmark version and the rubric — so the drop cannot be attributed to curation alone; the rubric revision by itself flipped 44 of 77 previously agreed positives when it was introduced. The deployment-slice spot-check sampled 50 answers from the previous benchmark version’s standard-prompt arms (5 models × 10, conflict cells, oversampling the scorer’s "neither" blind spot); a sampler defect let 15 tool_error answers into the sample, where agreement is higher still (κ 0.84). Gold exclusions: 13 bad-gold hops surfaced in the human QC sample, 3 more by a full manual audit of all hops, 8 by the structural gold-judge, human-confirmed (Appendix B). Answers from neither source. Where neither the tool nor the model’s own memory is correct, the desired behaviour is abstain-and-flag, but a small share of answers are the world-correct third entity, supplied by neither the tool nor the elicited memory: 0.053 on base Llama, 0.047 Mistral, 0.032 Llama-Instruct, 0.027 Qwen and 0.007 gemma (Reference formulation, judge-normalized). These land in the neither taxonomy’s third-entity branch (§3.3) and are a second, independent sign that forced elicitation undercounts what a model holds (§4.2): the parametric answer was unavailable when we asked for it and surfaced when a wrong tool value was present. They also bound how much of that case could in principle be answered correctly rather than abstained on. Protocol-choice evidence. Three measurements motivate design choices described in the benchmark section and are reported here rather than there, so that the construction section carries no model-level results. Elicitation protocol. Under a prompt that permits “UNKNOWN”, Qwen2.5-7B-Instruct declines 104 of 107 pilot questions, yet produces the verified answer on some of those same questions once an incorrect tool return is shown; forced elicitation is therefore used for the primary evaluation, and agreement between the two protocols on shared questions ranges from 0.60 to 0.90 across models. Scoring. Re-scoring a pilot Llama run under naive token-level F1 instead of extraction-first matching moves the measured incorrect-tool follow rate from 0.47 to 0.06; the inversion is a property of answer verbosity, not of arbitration. Format compliance. The requested FINAL: line is produced in 98–100% of instruction-tuned responses and approximately 69–90% of base-model responses, which is why base arms rely on the fallback extractor more often. Construction-step details (Section 3.2). The main text keeps the eight-step ladder and its counts; the rule sets, distance constants and per-branch audits behind the steps are reproduced here verbatim from the pre-compression draft. The pipeline turns ToolHop sub-questions into MemToC records in eight steps. Each step below states what it does and why, and reports how many questions survive it. Steps 2, 4 and 7 are performed by human annotators and are labelled manual; Steps 1, 5 and 6 are executed by code and are labelled automatic; Step 3 combines an automatic pass with manual adjudication. Every manual step is carried out by two annotators working independently. Step 1 (automatic). Flatten chains and deduplicate. This step establishes the unit of the benchmark. We take the sub-question, not the chain step, as the unit, because the same sub-question is reused across chains — one formulation occurs in 17 different chains — and step-level sampling therefore produces a pool with far less variety than its size suggests. We flatten all 3,912 sub-questions into a single list and collapse those whose normalized text is identical. 3,912 → 1,948 unique questions. Step 2 (manual + automatic). Separate factual from computational questions. This step keeps only questions for which answering without a tool is meaningful, since a question whose answer is computed rather than recalled cannot create a conflict with parametric knowledge. A rule-based filter rejects a question when its answer is produced by an operation over an earlier answer rather than looked up: counting letters, reading ASCII codes, forming initials, extracting a name component, taking a difference between dates, comparing two values, or measuring a length. Because such a filter can over-reject, two annotators then manually reviewed all 54 questions whose rejection was not clear-cut and asked whether the rejection was justified; 19 were restored — a city’s time zone, for instance, is a fact about a place rather than an arithmetic result. 1,948 → 605 factual questions (1,343 computational questions set aside). Step 3 (automatic + manual adjudication). Control redundancy and leakage between questions. This step prevents two questions that share the same underlying fact from being split across a training and an evaluation fold, which would leak the answer. Exact duplicates are already removed in Step 1. A second automatic pass identifies near-identical formulations that differ only in a named argument (the time zone of Orleans, Charles and Rockland counties). A third, also automatic, pass identifies knowledge clusters: linked questions about the same family or narrative in which the answer to one question becomes the main entity of another. These clusters cross ToolHop chains. 412 of the 605 factual candidates belong to a cluster of size greater than one, and two annotators manually adjudicated the 58 questions whose cluster membership was ambiguous. No question is removed at this step; its output is the grouping used for cluster bootstrap and for fold assignment. 605 → 605 questions, now grouped. Step 4 (manual). Structural review of question wording. This step ensures that each retained question can be answered on its own and does not give away its answer. We did not rewrite or regenerate any question: we preserved the original ToolHop wording, and self-containedness was a selection criterion rather than an editing operation. ToolHop sub-questions are already phrased as standalone questions, so the review decides which of them to keep. Two annotators independently reviewed all 605 candidates and removed 4 questions that refer to an earlier step, 1 that contains its answer in the question text, 7 dynasty questions whose answer follows from an ordinal in a title (the father of the sixth Earl of Exeter is the fifth Earl), and 18 whose verified answer was disputed. The verified answers themselves were checked in three ways, two automatic and one manual: 397 were compared automatically against Wikidata through the property identifier used by the question (329 exact matches, 11 partial, 3 fuzzy, 12 mismatches, 28 with no such claim, 14 with an unresolvable subject), 208 were checked automatically by a web-browsing language model, and 74 anchor answers were checked manually by the authors against primary sources. 605 → 575 questions with fixed wording, executable calls, and verified answers. Step 5 (automatic + manual). Answer-aware distractor construction. This step produces, for each question, the incorrect value that a tool return will carry. A credible incorrect value has to match the question’s domain and period as well as the answer type, which a draw from ToolHop’s own pool of verified answers — that is, some other question’s gold answer of the same nominal type — does not deliver. Each substitution is therefore constructed from the verified entity itself, through one of three branches; the manual review that established this is reported below. (a) Wikidata branch (246 questions). For person-valued answers we resolve the verified entity through the relation-specific Wikidata property used by the question rather than by name search — the answer to a question about a person’s father is resolved through property P22 — which prevents an unrelated namesake from being selected. Candidate distractors are drawn from other real entities matched on occupation (P106), gender (P21), year of birth (P569) and citizenship (P27), excluding the verified answer and every other true value of that property for the subject. A near distractor lies within 45 years of the verified person’s birth year; a far distractor is separated by 150–600 years or by country. Candidates with more than 40 Wikipedia language editions are excluded, so that a model does not reject a substitution because the entity is exceptionally prominent rather than because the fact is wrong. (b) Temporal branch (173 questions). Dates, years and time zones are handled by a deterministic procedure. The substituted value differs from the verified answer but is rendered in the verified answer’s own surface format, so that the evaluation does not conflate factual conflict with format recognition. Distance is set on the calendar: a near date is offset by 15–1,500 days and a near year by 1–15 years, while a far date is offset by 18,000–110,000 days and a far year by 80–400 years. (c) Human-authored branch (149 questions). The remaining questions are routed to human authors when the verified entity cannot be resolved unambiguously in Wikidata or the metadata needed for automatic matching are absent: works, historical figures without reliable country information, and people whose profession or period cannot be established automatically. Two authors independently proposed distractors for 152 questions. They proposed the same value for 46 and different values for 106. Because several incorrect values can be equally plausible, we do not treat disagreement as annotation error and do not report it as consensus: both proposals are retained, and the benchmark value is selected by a rule fixed before adjudication. This branch supplies context-sensitive substitutions that answer type alone cannot yield, such as replacing a character from Father Ted with another character from the same series, or a pharaoh with another pharaoh. After structural filtering, 149 human-authored records remain. Near and far. All three branches share one rule: the answer type is always preserved, a near distractor must be indistinguishable from the verified answer in type and plausibility, and a far distractor must be obviously different in period or country. What realizes “obviously different” is branch-specific and is stated with each branch above — birth-year distance or citizenship for entities, calendar distance for dates and years. The distinction is a property of the distractor pair, not of the question, and it is frozen in the released mapping rather than recomputed at run time. Seven further records received targeted automatic repairs, giving 575 records: 246 Wikidata, 173 temporal, 149 human-authored and 7 repaired, of which 426 are constructed automatically and 149 carry human-authored substitutions. Step 6 (automatic). Programmatic validity checks. This step catches malformed records before any human reads them. The checks are assertions in the build script and involve no human judgement: every distractor is non-empty; no distractor equals the verified answer; the near and far distractors of a question differ from each other; and no distractor coincides with a value recorded in Wikidata as an alternative true value of the property the question asks about. All 575 records pass. 575 → 575 records. Step 7 (manual). Blinded semantic review. This step checks the properties that code cannot check — whether a question makes sense on its own and whether its incorrect value is believable — before any model is run. Two annotators reviewed all 575 question–answer–distractor records under five checks, blind to each other’s labels: whether the question is self-contained, whether it leaks the verified answer, whether the verified answer is valid, whether the distractor has the required answer type, and whether the substitution is plausible. A blinded replication of 58 records was annotated a second time to measure reliability (Section 3.4 of the main paper). The review returns 503 pass, 39 repair and 33 exclude verdicts. A repair changes only the substituted incorrect value; question wording and verified answers stay frozen byte-for-byte, including 169 inherited capitalization typos, which we document rather than correct. Removing the exclude records leaves the shared analysis set. 575 → 542 questions. Step 8 (automatic). Instantiation into tool conditions. This step turns each quality-controlled record into the episodes a model actually sees, so that what varies between conditions is the observed tool return and nothing else. Each retained question is instantiated with (i) a correct return containing the verified answer g; (i) an incorrect return containing the near distractor dneard_near; (i) an incorrect return containing dfard_far, where one is available; (iv) a structured tool-error payload; and (v) no tool return. The question, verified answer, executable call and available distractors are fixed across models. Conditions (i), (i), (iv) and (v) exist for every question and are the four that make up the 6,504 episodes quoted in the main text (542×4×3542× 4× 3); condition (i) is available for only 463 of the 575 questions and is analysed separately. Each factual return condition is evaluated under three instruction-wording variants with the question text held fixed. NoopTool is evaluated separately as an interface control. Figure 3 instantiates these conditions for one benchmark question. The correct and incorrect factual returns enter the correctness matrix in Figure 2 of the main paper; tool error and no tool are separate controls outside that matrix. Dataset composition (Section 3.3). Reproduced from the pre-compression draft. Chain provenance and position. The 605 factual questions are attributed to 310 ToolHop chains: when the same question text occurs in several chains, it is assigned to the chain in which it first occurs. The remaining 685 chains are therefore not free of factual questions — we verified that all 995 chains contain at least one question from the pool — their texts simply collapsed during deduplication. Multiple questions per chain are the norm rather than the exception: of the 310 contributing chains, 279 yield two MemToC candidates, 8 yield three and 23 yield one. We retain every eligible sub-question instead of selecting one representative per chain. Factual questions sit early in the reasoning chain: 298 are first-step questions, 288 second-step and 19 third-step. After the structural review of Step 4, the 575 retained questions come from 309 of those chains. Training folds are cut by whole chain (fold A: 287 questions from 154 chains; fold B: 288 from 155; zero intersection). Topics. ToolHop labels each chain with a domain, and a question inherits the label of the chain it is attributed to. Over the 575-question pool these labels reduce to 47 distinct topics, out of 62 in ToolHop as a whole. The distribution is long-tailed and biographical: history 118 questions (20.5%), film 94 (16.3%), genealogy 80 (13.9%), time zones 44 (7.7%), mathematics 26 (4.5%), computing 19 (3.3%), time conversion 17 (3.0%), literature 15 (2.6%), music 14 (2.4%) and entertainment 12 (2.1%). These ten topics carry 439 questions, 76.3% of the pool; the other 37 topics contribute 136 questions, none with more than twelve. Two properties of this label matter when reading the results. It describes the chain rather than the individual question, which is why topics such as mathematics survive the factual filter of Step 2 at all — a chain about a calculation can still contain a lookup step. And the concentration in biographical domains, with history, film and genealogy alone accounting for 292 questions (50.8%), is what makes people the dominant answer type: the verified answers of the 575-question pool are 304 people, 111 dates, 50 places, 44 years, 23 organizations, 18 time zones, 17 other strings and 8 works. That mix is also why the Wikidata branch of Step 5 covers the largest share of the pool, and it bounds the domains over which our conclusions were measured. Answer types. ToolHop does not record an answer type per step: its previous_answer_type field is chain-level, covers five values (person 498, date 274, year 151, place 44, organization 28) and cannot be attached to a specific step when a chain branches. The category of each verified answer is therefore derived from its Wikidata P31 (instance of) property, which is what the distractor branches of Step 5 dispatch on. The automatically assembled type map this replaced, and the audit that retired it, are recorded below. Coverage of the far condition. A far distractor exists for 463 of the 575 questions (443 after the semantic exclusions of Step 7). It is missing for 105 human-authored records, for all 17 other_string answers and for all 8 work answers. Near–far comparisons are therefore restricted to this subset, which is skewed towards automatically constructed items — only 44 of the 149 authored records are covered — and is not a random sample of the benchmark set. Superseded construction steps. Two elements of the pipeline were replaced during development. The main text describes only the final construction; they are recorded here as provenance. Same-type sampling. The first procedure drew each incorrect value uniformly from ToolHop’s own pool of verified answers of the same nominal type — that is, another question’s gold answer. A manual review of all 605 candidates rejected 167 of them: approximately 106 name fragments, 40 substitutions of the wrong gender for the relation asked about, and 9 strings containing part of the verified answer, with the remainder type-compatible but implausible for the question’s domain or period — a chemist replaced by a canal engineer. Surface defects can be filtered automatically, contextual implausibility cannot, which is what motivated resolving the verified entity itself and constraining candidates on occupation, period and country. Answer-type map. Those defects traced to an automatically assembled type map: GLiNER (gliner_small-v2.1, threshold 0.5) was run over the 807 unique normalized verified-answer strings, its person, location, organization and creative-work labels mapped onto our four categories by highest-scoring span, and ToolHop’s own field preferred where it applied, overriding 24 predictions. A later manual audit measured category mismatches in 13% of near distractors and 18% of far distractors, mostly genealogical relations routed into the place-and-organization branch. The current pipeline derives the category from Wikidata P31 instead. Generator audits. Two blinded sampled audits bracket the automatic branch: annotator agreement κ=0.53κ=0.53 with 12% of inspected distractors rejected before the type-map fix, and κ=0.67κ=0.67 with the rejection rate unchanged after it, 9 of the 11 remaining failures being pre-modern or legendary figures without citizenship metadata. Scorer-v3 extension. The regex extension for passive abstain phrasings was validated against the same human labels with zero over-fires on 120 QC answers and moves only the abstain channel: retention, following, accuracy and incorrect-tool following stay within |Δ |≤ 0.011 across all 15 arms. Scorer gate on the released benchmark (v1.3). Before any current metric was read, the deterministic scorer was validated against blind human labels drawn from this benchmark’s own responses, under a pre-registered bar (outcome κ ≥ 0.8; recall ≥ 0.9 on the TOOL-followed and GOLD-kept classes). Three probes failed and drove scorer iterations (quoted FINAL markers as mentions; prefixed refusals; comma-headed aliases; question-echo, "is not known", "did not provide", tool-re-run echoes; date-format equivalence) — each iteration regression-checked on all previously labelled rows. The final enlarged probe (167 stratified rows, zero overlap with the failed probes, population re-scored by v1.3 before sampling) passed: κ 0.806, TOOL recall 23/23, GOLD recall 31/33 = 0.939; the two annotators agree at 161/167 (κ 0.952), six disagreements adjudicated, and the adjudicated conventions (agree-value answers are BOTH, not GOLD; degenerate schema echoes, "does not have X in the provided data", and refusal-FINALs anchor to ABSTAIN; year-only dates match full dates bidirectionally) are frozen in the released packet README. The disclosed blind zone: ABSTAIN recall 0.706 (long polite refusals read as OTHER), which the judge layer targets (§5.4). CAR detector details (previous benchmark version). Everything in this paragraph — the detector’s precision, the two-stage estimator and its per-model brackets — was measured on the previous benchmark version, where positive acknowledgments existed at all. It is reported as evidence about the detectors, not as an estimate of the acknowledgment rate: the brackets sit above the direct measurement on the released benchmark, which is 0/120. The deployed detector is restricted to tool_wrong answers, where the LLM judge is more precise and its false positives do not cluster on tool_error; precision against human labels is 0.38 on tool_wrong (0.27 across all conditions). Two honesty notes: the 11-positive validation set was reused across both judge iterations, so judge validation numbers are in-sample, and prompt iteration was stopped there to avoid fitting the detector to its own test. Since judge over-firing plausibly correlates with answer verbosity, no cross-model CAR claims are made from the judge screen alone. In the two-stage estimator, the per-arm rate is recovered as flag-rate × human-precision + (1−-flag-rate) × human-miss-rate over 2,403 screened tool-wrong answers; the per-model brackets are gemma [0.01, 0.05], base Llama [0.02, 0.07], Llama-Instruct [0.02, 0.08], Mistral [0.05, 0.09], Qwen [0.05, 0.16]. Rubric history: the first rubric left open whether "the tool result does not match the question" counts as acknowledgment (κ 0.196); the second — fixed before any re-annotation — requires the answer to reference the model’s own knowledge, and flipped 44 of 77 previously agreed acknowledgments (κ 0.351 at raw agreement 0.87, depressed by the now-rare positive class). Desired-behaviour validation details. Actions ∈ follow tool, use own knowledge, present both, abstain plus a binary must-flag axis, under an enterprise-policy framing (the tool is an authoritative-but-fallible internal database), blind to correctness and cell. On the arbitration case neither annotator ever chose to follow the tool (0/30 each). The must-flag axis agrees at 0.94 raw but κ 0.237 — a rare-class κ paradox (both annotators flag near-universally). Cell sizes (released benchmark). The closed-book cell map is fixed per (model, prompt formulation) by one forced elicitation and shared by every arm of that model and formulation — standard, presentation and prompting alike — which is what makes the treatment statistics exactly question-paired. Cross-formulation conflict cases: arbitration 67–162, neither-correct 293–414 across the five models (the correct-return case mirrors it at 293–414 — both hold the closed-book-wrong questions; tool_error and no-conflict span all 542; mem_absent 2–125, excluded from conflict metrics); the judge’s memory-correctness relabel moves only the pretrained checkpoint (162→ 136 — echo-style closed-book answers re-labelled). Its cases under Paraphrase A and B collapse to n=18/10 (closed-book degeneration, §5.1) and are not reported. The finetuning arms are cross-fitted, so each carries its own out-of-fold support: the paired fine-tuning contrasts reported in the main paper run on 76–201 shared questions (184–445 formulation-pooled units) on the arbitration case and 382–481 shared questions on the correct-return case. Because a cell is defined by the arm’s own parametric answer, treatment moves cell membership; contrasts are computed on the intersection of supports and the drift (questions entering and leaving) is reported next to every delta in the released summary. The largest drift is on Mistral’s and gemma’s SFT arms in the correct-return case, which is consistent with their large movements in incorrect-tool following (§5.2). Appendix E Additional Stability Analyses Distractor distance (Section 5.4), full estimates. The contrast is question-paired and read on the Reference wording, the only formulation for which both conditions exist; a far distractor exists for 463 of the 575 questions (443 after the semantic exclusions of Step 7), and the covered subset is skewed towards automatically constructed items, so it is not a random sample of the benchmark set. Replacing the near distractor with one separated by 150 to 600 years or by country for entities, and by a comparable margin for dates, raises correct-answer retention on Meta-Llama-3.1-8B from 20.6% to 40.8% (+19.4+19.4 points, 95% CI [9.7,29.1][9.7,29.1], Holm-adjusted p=0.0015p=0.0015, n=103n=103 questions), while all four instruction-tuned models are null (|Δ|≤4.2| |≤ 4.2 points, Holm p=1.0p=1.0). When neither source is correct, every model repeats a distant incorrect value less often, from −8.2-8.2 points on the base model to −19.7-19.7 on gemma-2-9b-it (all p≤0.003p≤ 0.003). The deterministic and judge-normalized layers agree on the verdict for every model; on Meta-Llama-3.1-8B-Instruct they differ in sign while both are null, so we claim agreement of verdicts rather than of signs. On a separate axis, Figure 4 shows correct-answer retention under the three prompt formulations on untreated models: the ranges overlap and the model ranking reverses between them (§5.3). Correct-answer retention Llama-3.1-8B-Instruct gemma-2-9b-it Qwen2.5-7B-Instruct Mistral-7B-Instruct-v0.3 00.10.2Reference Paraphrase A Paraphrase B Figure 4: Correct-answer retention across three instruction-wording variants with fixed question text on untreated models. Markers show the Reference, Paraphrase A, and Paraphrase B estimates; gray segments show their minimum–maximum range and are not confidence intervals. The ranges overlap and model rankings change across formulations. Qwen2.5-7B-Instruct has the highest retention on the Reference formulation and the lowest on Paraphrase A, so we do not include cross-model retention rankings among the core claims. Unpaired base-versus-instruct retention family (Reference formulation). The per-cell two-proportion family (planned, Holm-corrected; cell sizes as as reported below) agrees with the question-paired base-vs-instruct design on three of four pairs: base vs gemma Holm-p 0.0007, vs Llama-Instruct 0.017 and vs Mistral 0.041, but not vs Qwen (0.129, on the smallest arbitration case, n=67). The paired family, which is the pre-registered one, separates all four. Prompting, all four models and all three strategies (§5.4). Table 4 gives the levels behind the prompting result. One sentence changes per arm and nothing else; because the arms of a block share that model’s own closed-book answers, differences within a block are exactly question-paired. Model Prompt Ret.↑ Tool↑ Wrong↓ Err.↑ Llama-3.1-8B-Instruct – 9.3 87.8 71.4 89.3 Warning 21.0 75.1 65.3 16.4 Source-priority 50.6 79.4 60.9 29.7 Abstain-and-flag 12.3 23.5 18.8 89.7 gemma-2-9b-it – 4.8 80.2 74.1 100.0 Warning 17.7 80.2 65.5 89.7 Source-priority 25.0 82.6 75.1 81.9 Abstain-and-flag 21.0 58.4 39.2 97.0 Qwen2.5-7B-Instruct – 11.9 82.6 75.4 88.8 Warning 40.3 70.0 56.5 11.4 Source-priority 37.3 75.1 62.6 22.5 Abstain-and-flag 1.5 25.1 18.8 100.0 Mistral-7B-Instruct-v0.3 – 10.2 88.2 82.9 75.6 Warning 8.7 88.2 78.2 23.1 Source-priority 18.9 85.8 74.7 58.5 Abstain-and-flag 43.3 35.3 26.8 52.8 Table 4: Prompting interventions on the four instruction-tuned models, Reference formulation. A dash denotes the same checkpoint under the standard evaluation prompt. Ret., Tool, Wrong and Err. are as in Table 1 of the main paper: correct-answer retention, correct-tool following, incorrect-tool following and tool-error abstention. These are single-formulation levels and therefore differ from the formulation-pooled rows of Table 1; the paired changes with intervals and Holm tests, pooled over the three formulations, are in the released summary layer. Retention, correct-tool following and incorrect-tool following are judge-normalized, tool-error abstention is a deterministic-scorer value. Reading. Every model gains retention from at least two strategies (eleven of twelve pooled comparisons are Holm-significant, §5.4) while paying a model-specific cost. Llama-Instruct and Qwen give up correct-tool following under the warning and source-priority prompts; gemma and Mistral preserve it there but still lose tool-error abstention. Abstain-and-flag collapses correct-tool following on every model and is Qwen’s one null, with retention at 1.5. gemma is the only model whose tool-error abstention survives the warning prompt nearly intact, 100.0 to 89.7. The one-sentence protocol ablation (§5.3). The standard with-tool prompt carries a protocol sentence — "the tool cannot be called again — answer now" — that has no epistemic content about sources. We remove it and rerun the anchor model on the full released benchmark (identical episodes, temperature 0, question-paired, judge-normalized). On the arbitration case the sentence does nothing: retention changes by −0.019-0.019, 95% CI [−0.068,+0.031][-0.068,+0.031], p=0.65p=0.65, n=162n=162, and the deterministic layer agrees (−0.025-0.025, p=0.43p=0.43). What it does move is obedience where the tool is right: following a correct tool return falls by 0.093 [0.048,0.138][0.048,0.138] (p=0.0002p=0.0002, n=378n=378) and no-conflict accuracy by 0.056 (p=0.011p=0.011); tool-error abstention is borderline and we read it as null (−0.033-0.033, p=0.069p=0.069). The sentence buys compliance, not deference. On the previous benchmark version’s validation slice the same ablation was measured at roughly 25 points on the arbitration case (retention 0.75→ 0.47, incorrect-tool following 0.16→ 0.41, tool-error abstention 0.23→ 0.52), and was described in earlier drafts as the largest single-intervention movement we had measured there. It does not survive the released benchmark. We record the discrepancy rather than dropping it: it is one of the construction-scope effects this paper reports, and the earlier estimate was taken on a slice whose arbitration case was both smaller and differently composed. Seed robustness of the LoRA effect (§5.2). The primary adapters use seed 20260705; two further seeds (42 and 20260719) were trained and evaluated under the identical cross-fitted protocol on the two anchors that meet the criterion. Judge-normalized, formulation-pooled change in retention on the arbitration case: Llama-Instruct +0.058 / +0.077 / +0.086, gemma +0.093 / +0.111 / +0.140; all four additional arms are Holm-significant at p 0.0004 within family T3, all three seeds agree in sign per model, and the per-model CIs share a common interval ([+0.056, +0.088] on the anchor, [+0.096, +0.131] on gemma). The gain is therefore not a single-seed artifact. The two models that fail the criterion were not seed-replicated (§6). Per-formulation verdicts of the asymmetric criterion (§5.2). The criterion (retention rises at Holm p<0.05p<0.05 with no significant fall in correct-tool following) was pre-registered on the pooled slice and, separately, on each prompt formulation. Pooled it is met by Llama-Instruct and gemma under both losses. Per formulation, in Table 5: Arm Reference Par. A Par. B Pooled Llama-3.1-8B-I ⋅· SFT met failt met met Llama-3.1-8B-I ⋅· DPO failk failk met met gemma-2-9b-it ⋅· SFT met met met met gemma-2-9b-it ⋅· DPO failk met met met Qwen2.5-7B-I ⋅· SFT fail fail fail fail Qwen2.5-7B-I ⋅· DPO fail fail fail fail Mistral-7B-I ⋅· SFT fail fail fail fail Mistral-7B-I ⋅· DPO fail met fail fail Table 5: The asymmetric criterion evaluated per prompt formulation and pooled (judge-normalized layer; Holm correction inside each slice’s own family). k failed because the retention gain does not reach Holm p<0.05p<0.05 (0.28, 0.051 and 0.44 respectively); t failed on a significant loss in correct-tool following (−-0.045, p 0.028). The deterministic layer agrees except on Llama-Instruct’s DPO arm under Paraphrase A, which it reads as met. Only gemma’s SFT arm meets the criterion under every formulation; Mistral’s DPO arm, a failure when pooled, meets it under Paraphrase A. This is the same instability the stability rule of §5.3 detects in the untreated orderings, which is why §5.2 reports the 2/4 result as a pooled finding whose passing set is formulation-scoped. The prompting family behaves the same way: of the eleven pooled-significant retention gains in the four-model × three-strategy matrix, only Llama-Instruct’s warn arm, gemma’s source-priority arm and Mistral’s abstain-and-flag arm hold on all three formulations separately, and Mistral’s warning arm is negative under the Reference formulation (−-0.016) against ++0.065 pooled. Switch compositions (previous benchmark version). An oracle prompt-switch inherits the knob it turns, and a realizable, observables-only switch captures most of that ceiling, bounded by the conflict cells in which no elicitable memory exists to disagree with the tool. On gemma the oracle switch buys +0.380 accuracy on the arbitration case over the standard-prompt arm (decided-only Y +0.295 [+0.198, +0.402]) and the realizable switch +0.350 (Y +0.254 [+0.155, +0.352]), leaving a gap of only +0.041 [−-0.003, +0.092] between what an oracle could do and what observable signals actually achieve; on the anchor the whole family is indistinguishable from zero (oracle +0.091 accuracy, Y +0.050 [−-0.053, +0.158]; realizable +0.070, Y +0.041 [−-0.047, +0.130]), because the knob it composes is itself null on that model and formulation. The switch’s safety property is exact on both anchors — zero false fires on the agree cell — and its structural ceiling is the 41–42% of conflict cells the observable gate cannot see at all, where no elicitable memory exists to disagree with the tool. Any observables-based policy inherits that ceiling. Formulation-spread composition block (previous benchmark version). Re-running the prompting arms and recomputing every composition under the other two protocol formulations separates what is structural from what the knob contributes. This block is the one composition not re-run on the curated version (its control arms exist only under the Reference formulation there); the Reference-formulation pattern it brackets reproduced on the curated version. The switch’s safety properties replicate exactly: zero false fires on the agree cell under all three prompt formulations of both anchors, and the observable gate’s blind spot sits at 40–42% of conflict cells everywhere. The gains inherit the prompt knob’s formulation sensitivity: on gemma the oracle buys +0.11–+0.31 decided-only accuracy across formulations (always CI-positive) and the realizable switch +0.10–+0.28 (gap to the ceiling 0.01–0.04); on the anchor the knob is indistinguishable from zero under one formulation (oracle Δ −-0.02 [−-0.09, +0.06]) and strong under another (+0.26 [+0.17, +0.36]), where the realizable switch captures only about half the ceiling (gap +0.12 [+0.05, +0.18]). The 40–42% of conflict cells invisible to the observable gate (no elicitable memory to disagree with) are a structural ceiling for any observables-based policy. OOD label provenance (Appendix F.1). The full external-set label layer (981 counterfactual + 500 unanswerable rows) was produced by two LLM labeling passes — Claude Opus 4.8 Extra High (A) and ChatGPT/GPT-5.6-sol (B) — with prompts, decoding settings and request counts released; it is not a human double annotation, and we compute no human–human agreement for it. Human evidence for this branch is confined to the separate 300-row human pilot audit (which defines the verified 279/185 deployment pools) and to the scorer gate (which validates the matcher, not item semantics). The two passes are not independent evidence. Both were run on 2026-07-19, after the joint human adjudication of 2026-07-18, from a task file whose two "previous annotator" columns carried the human labels in plain view. The consequence is measurable. The human pilot drew 300 of 1,000 rows at random (subsample share 0.26–0.34 across deciles of the join order, expected 0.294), so the 294 rows humans inspected and the 687 they did not are exchangeable and must carry the same defect rate. Adjudication confirmed 15 gold-non-uniqueness defects inside the inspected rows — at least 5.1%, hence about 35 expected among the 687. Each pass flags exactly those 15 rows and 0 (A) / 1 (B) outside them; binomial p = 2.4e-16 and 9.0e-15. Neither pass flags any of the 11 rows adjudication cleared, nor any of the 83 rows one annotator flagged that adjudication never reviewed. The passes replay the adjudicated verdict where it exists and answer "clean" elsewhere. Their 966/981 mutual agreement is therefore trivially high and is not a quality signal; the 687-row extension is unverified by anything. Nothing here disturbs the Appendix F.1 estimates, which are computed on the human-verified 279 and 185 pools. A blind re-run was attempted and did not qualify. We rebuilt the task blind — items only, hint columns and adjudication access removed, IDs tokenized, rows shuffled — over 494 rows (all 294 humans had inspected plus 200 drawn from the 687 they had not), under a threshold fixed before any label: recall ≥ 0.60 against the 15 jointly adjudicated defects with ≤ 0.30 false flags. The blinding sanity check passed — flag rates 0.218 inside the human-inspected part against 0.190 outside (Fisher p = 0.50), against 15 versus 0–1 for the label-visible passes, which independently confirms the diagnosis above — but the utility threshold failed: recall 0.400 [0.198, 0.643], false flags 1/11. The extended pool therefore cannot be rescued by an LLM screen, and every reading stays on the human-verified pools. Two by-products are worth recording: the blind pass flags 19.0% [14.2, 25.0] of the 200 uninspected rows against a 5.1% adjudicated defect rate in the inspected part — a flag is not a confirmed defect, but this is the only quantitative signal we have about the extended pool — and on the contested context_supports criterion the blind pass lands between our two annotators (19.6% against 0% and 28%), so the stricter reading is a systematic property of FaithEval’s soft support rather than one annotator’s idiosyncrasy, which is why the passage frame is reported as a sensitivity reading throughout. OOD pool construction (Appendix F.1). The serialization filter removed 19/1000 label-desynced rows (answerKey not among the choice labels); the human audit of a 300-row subset surfaced 21 defects (adjudicated jointly), in two families: a programmatically filterable serialization artifact and gold-non-uniqueness, which motivates per-item uniqueness checks for any wholesale use of the source set. Closed-book elicitation over the 981 filtered questions fixes each anchor’s memory-right set (819/981 anchor, 890/981 gemma, on the revised matcher of Appendix F.1); both frames are read on the common 185-pool support (n=158/169 memory-right episodes per model), the value frame additionally on the 279-pool as secondary (n=240/256). The abstention axis uses FaithEval’s unanswerable split (500 questions); closed-book retention is measured on the full elicitation set. Appendix F Out-of-Distribution Transfer and the 70B Scale Point This appendix carries the two result families moved out of the main text for length. Both are summarized in §5.4. F.1 Out-of-distribution transfer An out-of-distribution transfer test finds the training gains are not a construction signature — they transfer, but partially, attenuated, and frame-dependently, and which frame carries the transfer depends on the loss. The strongest objection to any in-benchmark control result is that the model learned to distrust the construction’s own signature rather than arbitration-by-correctness. We test this on an external conflict set built from FaithEval’s counterfactual split (Ming et al. 2025) joined back to ARC-Challenge for world-true gold: multiple-choice science questions whose counterfactual answer comes from authored evidence — a different task format, domain and conflict provenance from everything in the training pairs. The deployment pools are 279 value-frame and 185 passage-frame episodes: the same counterfactual answer presented either as a bare tool return value or as the full passage inside the tool return (pool construction and supports in Appendix E). Because the MC matcher is a new scorer domain, arm metrics were embargoed behind blind two-annotator scorer gates, and all four pre-registered thresholds passed (κ 0.975 between human consensus and scorer, κ 0.987 between annotators, stratified abstention recall 1.00 at precision 1.00). The enriched stratum of the second gate also forced a matcher revision, disclosed below. Label provenance. The 981 counterfactual plus 500 unanswerable rows were labeled in two LLM passes, not by independent human double annotation, and a provenance audit shows the passes were run with the human pilot’s labels visible: they reproduce that pilot where humans looked and answer "clean" almost everywhere else, flagging 0 and 1 of the 687 rows no human inspected against about 35 expected at the audited 5.1% defect rate. The 687-row extension is therefore unverified, every reading below stands on the human-verified 279/185 pools, and the human gates validate the matcher and scorer, not the semantic validity of every item. The pre-registered readings (paired episode bootstrap, B=2,000), on adapters trained leakage-free over the full 575-question pool, are given in Table 6: Arm Value frame Passage frame Abstention Retention Llama-3.1-8B-Instruct ⋅· SFT +0.051 +0.044 −-0.068 +0.025 Llama-3.1-8B-Instruct ⋅· DPO +0.070 0.000 −-0.020 +0.017 gemma-2-9b-it ⋅· DPO +0.112 +0.047 −-0.076 +0.004 Table 6: Out-of-distribution transfer (FaithEval-counterfactual × ARC), adapters trained leakage-free over the full 575-question pool, paired episode bootstrap B=2,000B=2,000. Every cell is a change against the corresponding original checkpoint, with its 95% interval; bold marks an interval excluding zero. The two conflict frames present the same counterfactual value either as a bare tool return or inside a passage, and both are read on the common 185-item support so that they are comparable. The 95% intervals are given in the text below. Abstention is measured on the unanswerable split. Retention is the change in closed-book accuracy, and all three arms fall inside the preregistered ±0.03± 0.03 band. The 95% intervals, in the order of the table: value frame [−0.019,+0.127][-0.019,+0.127], [+0.006,+0.133][+0.006,+0.133] and [+0.065,+0.166][+0.065,+0.166]; passage frame [+0.006,+0.082][+0.006,+0.082], [−0.038,+0.038][-0.038,+0.038] and [+0.012,+0.083][+0.012,+0.083]; abstention [−0.096,−0.038][-0.096,-0.038], [−0.038,−0.002][-0.038,-0.002] and [−0.102,−0.052][-0.102,-0.052]. Both conflict frames are read on the common 185-item support (n = 158 memory-right items on the anchor, 169 on gemma) so that the two frames are directly comparable; on the wider 279-item value pool the same arms give +0.038, +0.046 and +0.086. Retention is the closed-book accuracy change on the 981-item elicitation set against a pre-registered ± 0.03 band. Transfer is real but frame-scoped, and the frame depends on the loss: the anchor’s DPO arm transfers in the value frame only, its SFT arm in the passage frame only — opposite frames — while gemma’s DPO transfers in both, so the presentation frame is a first-order axis even for a trained effect, echoing §5.4 from the treated side. The abstention leak is confirmed on all three arms, not on SFT alone as the previous benchmark version suggested, making it the most consistent side effect we measure anywhere. It is also where the in-distribution reading must be reconciled: the anchor’s DPO arm shows no detectable tool-error leak inside MemToC (+0.014, p = 0.062, §5.2) and still loses abstention here (−-0.020). These are different measurements — abstaining on a broken tool call versus on an unanswerable question in a new domain — and both stand; the abstention axis is the fragile one on seven of our eight in-domain arms and on all three transferred arms. Per-frame nulls remain power-bounded (minimal detectable effect 0.11 to 0.16), not demonstrated absences, and every positive reading on the anchor’s DPO arm is a narrow interval we report as directional. Transfer also runs in the reverse direction. Adapters trained by correctness-labelled LoRA-DPO on FaithEval/ARC rows only — with zero ToolHop content of any kind — and evaluated on the untouched MemToC benchmark set raise kept memory on gemma by Δ +0.046 [+0.025, +0.072] (Holm p 0.0008, n = 158 shared questions) with the right-tool leg intact (+0.010), and leave the anchor null (+0.012 [−-0.009, +0.030], n = 197). Because the training data shares no construction, no question pool and no distractor mapping with MemToC, this is the cleanest available evidence that the trained behaviour is not a signature of our construction — but it holds on one of two models, so it supports a model-dependent replication rather than a dataset-independent property of the recipe. Disclosure: the MC matcher was revised after the arms were read. The enriched gate stratum exposed a failure mode the pre-registered thresholds did not cover: where the matcher matched an answer to no option, both annotators nonetheless read an explicit option in 67–73% of cases, and the unmatched fraction differed across arms (7.65% on one untreated arm against 0.71% on its SFT arm), so the miss biased deltas and not only levels. Following the pre-registration’s remedy — corrections by offline rescoring only, never by regenerating responses — we repaired the matcher and rebuilt both the new and the legacy summaries on it; the tables above are the rebuilt values. What mitigates this: the direction of the correction was predicted before rescoring from the differential unmatched rate; an independent earlier gate on different arms and episodes, annotated before this rule existed, is unchanged at 40/40; gemma, whose unmatched rate was already near zero, moved by exactly 0.0000; and the rule’s thresholds are flat across the coverage × margin grid we swept. What it does not undo: the repair flipped one retention gate from fail to pass (the anchor’s SFT arm, +0.045 → +0.025 against a ± 0.03 band), which is why that arm is readable at all above, and the residual bias points the same way. Residual matcher error on the gate is 4/55. Full tables and the replication script are in the released provenance for this run. OOD transfer scope. The out-of-distribution test is one external set, multiple-choice and science-domain, so format transfer is tested but free-form transfer is not; its verified conflict pools are small, per-frame nulls are power-bounded (minimal detectable effect 0.11 to 0.16) rather than demonstrated absences, and several positive readings are narrow intervals we report as directional. All OOD claims are restricted to the human-verified 279/185 pools because the 687-row extension is unverified (Appendix E). Transferred magnitudes are direction-, not size-preserving. F.2 The 70B scale point A 70B scale probe (one model, one quantization, no seeds). Llama-3.3-70B-Instruct, run as a public AWQ-int4 requantization with its own fresh elicitation on the same benchmark set, buys a real but partial improvement, unevenly across axes. It defends correct memory better than any 8–9B instruction-tuned model (retention on the arbitration case 0.21 formulation-pooled against their 0.07–0.17; paired against Llama-Instruct on common support, Δ +0.133 [+0.076, +0.196]) and follows the right tool most (correct-tool following 0.96 pooled, Δ +0.029). That contrast carries a large support drift — 401 of the 70B’s arbitration questions have no counterpart in the anchor’s cell and 48 the reverse, so the pairing runs on the 185 questions where both models hold the correct answer. The other two axes move the wrong way: with no correct memory to defend it is more deferent (incorrect-tool following 0.85 pooled, Δ +0.068), and it is markedly worse at refusing a broken tool (tool-error abstention 0.49 under the Reference formulation against 0.76–1.00 for the 8–9B instruction-tuned models, Δ −-0.410). The reading is a trade, not a lift: at 0.21 pooled retention this model still abandons its own correct answer in roughly four of five conflict cells, beating the anchor’s 0.17 pooled but nowhere near solving the problem. Scale is confounded here with model generation and quantization, and it is a single point, so we read it descriptively. Appendix G Tool-Choice Protocols in an Executed Agent Loop This appendix carries the dev-stage probe referenced in §5.4. It supports exactly one claim there — that reliability is a model × protocol interaction, and that constraining the decoder to the schema repairs the Llama failure — and nothing beyond it. The remaining rows are reported for the same reason as the SAE-steering probe, as a bounded result on an axis the benchmark itself does not vary. Setup. We replay the memory-right slice of this benchmark set — the 162 question pairs (324 episodes) on which the evaluated model’s forced closed-book answer is correct — through a live agent runtime in which the model emits a tool call over its own native function-calling template, the call is executed, and the (overwritten) return is fed back. The runtime is vLLM 0.8.5.post1 with native tool calling enabled and the per-model tool-call parser, so the protocol effects below are measured as a standard serving stack realizes them. Only the tool-choice protocol varies: a1, a named schema-guided call; a2, free auto choice; a4, a two-stage decide-then-call prompt. Decoding is greedy. Intervals are percentile bootstrap, B=2,000B=2,000, over the per-model memory-right slice. Table 7 reports the arms. Model Guided Auto Two-stage n Qwen2.5-7B-Instruct 1.000 1.000 0.984 61 Qwen2.5-3B-Instruct 1.000 0.979 0.979 48 Qwen2.5-1.5B-Instruct 1.000 0.970 0.576 33 Llama-3.1-8B-Instruct 0.992 0.328 0.976 125 Llama-3.2-3B-Instruct 1.000 0.291 0.367 79 Mistral-7B-Instruct-v0.3 0.104† 0.885 0.250† 96 Hermes-3-Llama-3.1-8B 1.000 0.964 1.000 110 Table 7: Answer agreement with the executed tool return on the memory-right slice, by model and tool-choice protocol. The three protocols are a named schema-guided call, free auto choice, and a two-stage decide-then-call prompt. Bold marks a failing arm. †Mistral’s two schema-guided arms are confounded with a serving defect described below and must not be read as a model property. Hermes-3 is a third-party tool-finetuned checkpoint of the same Llama-3.1-8B base, included as an uncontrolled comparison. Reading. Which protocol fails depends on the model and its scale — Llama under auto, Mistral under the guided call, Qwen2.5-1.5B under the two-stage prompt — and the failures are mechanically distinct rather than one bug. Four mechanisms appear in the traces: (T1) argument type coercion, in the Llama family under auto, 458 type errors dominated by booleans serialized as strings; (T2) unrenderable arguments, in Mistral’s guided arms, where 302 of 302 rejected calls fail to parse as JSON at all; (T3) stage-one decision-parse failure, in the two-stage protocol; and (T4) silent abstention from calling the tool. Two repairs bracket T1: preference fine-tuning on our own pairs does not remove it (444 type errors remaining, 0.328→0.3600.328→ 0.360), whereas constraining the decoder to the schema does (0.328→0.9920.328→ 0.992) and a third-party tool-finetuned checkpoint shows 14 type errors against 458, with no stringified values at all. The controlled repair we own is the schema-constrained decoder; the tool-finetuned checkpoint is an observation, not an ablation. Metric caveat. The score above is answer agreement: it matches the final answer against the tool’s value. On the memory-right slice the tool’s value and the model’s own memory coincide, so an episode that never calls the tool can still score. Against the tool-execution rate recovered from the traces, 16 of 21 arms agree to within 1.5 percentage points — the collapses above are unaffected, and on every Llama arm the two quantities are equal to three decimals — but two do not: Mistral under auto scores 0.885 while executing the tool in 0.385 of episodes, and Qwen2.5-1.5B under the two-stage prompt scores 0.576 while executing the tool in none. We therefore report both quantities, and read the small-model two-stage failure as silent abstention rather than as malformed calls. In particular, Mistral does not "recover" under auto: it mostly does not call the tool. Reproducibility and sampling. Decoding is greedy, so a second run under a different seed emits byte-identical tool calls in 324 of 324 episodes. This establishes that the pipeline reproduces end to end; on its own it is not evidence about sampling. We therefore re-ran the memory-right slice with the closed-book elicitation held greedy—which fixes the evaluation subset—and every tool-choice arm sampled at T=0.7T=0.7, on three seeds. The collapse persists: Llama-3.1-8B follows the named schema-guided call on 1.0001.000 of episodes and drops to 0.2680.268, 0.2680.268 and 0.3230.323 under auto (Δ −0.732-0.732, −0.732-0.732, −0.677-0.677; mean −0.714-0.714, sd 0.0320.032; every 95% interval excludes zero), against −0.664-0.664 [−0.744,−0.576][-0.744,-0.576] under greedy decoding, while Qwen2.5-7B stays flat (+0.000+0.000, +0.016+0.016, +0.000+0.000). Sampling did vary the runs—per-episode outcomes differ on 30 to 40 of 324 episodes across seed pairs—so the aggregate is stable while individual episodes are not. Two caveats: the sampled runs score 127 memory-right episodes rather than 125, because greedy elicitation is not bit-stable across runs on this server (two earlier greedy runs already differ from each other by three episodes, while the three sampled seeds share an identical subset); and this is a single temperature, not a curve. Serving confound on the guided arms. Under the vLLM version used here, Mistral’s native tokenizer path hard-errors on the guided render and the fallback path soft-fails, so its guided-arm figure is a property of the serving stack and not of the model. The same class of defect produced one unusable Qwen2.5-7B cell under a prompt formulation, where the constrained decoder looped on JSON whitespace and 238 of 260 calls never parsed, while the same model on the same episodes under auto executed 260 of 260 cleanly. Guided-arm numbers are reported with this caveat attached, and the free-choice arm is used as the clean comparison point. We tried to isolate the defect by re-running the affected arms under a different constrained-decoding backend, and report the attempt because it failed informatively. On this vLLM version the backend selector is inert: switching it reproduced the unusable cell to the episode (238 of 260 rejected either way, and 42 of 260 on the two-stage arm), and the only other selectable value is the one the default already resolves to, so the implementation cannot be varied here at all. The failure is therefore established as deterministic rather than flaky, but its attribution to the decoder rests on the cross-arm and cross-formulation evidence above, not on a backend swap. This does not reach the §5.4 claim: the arm that collapses there is free auto choice, which sets no schema and invokes no constrained decoder, so that collapse cannot be an artifact of one. The same re-run reproduced it independently, at 0.984→0.3120.984→ 0.312, Δ −0.672-0.672 [−0.752,−0.584][-0.752,-0.584]. Appendix H Whether the Model’s Own Answer Is Still Available A natural question about the arbitration case is whether the model’s own correct answer is still available to it at the moment it adopts the wrong tool value, or whether the tool return displaces it. This appendix reports a dev-stage probe of that question. It supports no claim in the main paper: it covers two model families of four, one construction, and one reading position, and the quantity it measures is a property of the representation, not of behaviour. Measure. We read the residual stream through a Jacobian lens at the final input position, and record the rank of the row corresponding to the model’s own elicited closed-book answer in the resulting vocabulary ordering. The rank is taken as the minimum across a band of source layers (8–28 for Llama, 10–38 for gemma, 7–24 for Qwen; the band tracks dmodeld_model). Episodes are the arbitration case of the conflict construction — the model’s closed-book answer is correct, the tool return carries the typed distractor — split by what the model actually did, from the behavioural labels of the same runs. As a control we take, for the same prompts, the elicited-answer rows of other questions, for which the model holds no such answer here. Table 8 reports the ranks. Followed the tool Kept memory Control Model Median Top-20 Median Top-20 Median Llama-3.1-8B-Instruct, seed 1 1 38/41 1 53/55 421 Llama-3.1-8B-Instruct, seed 2 1 33/37 1 59/63 421 gemma-2-9b-it 2 32/36 1 36/39 538 Qwen2.5-7B-Instruct 26,145 0/25 34,800 0/66 36,598 Table 8: Rank of the model’s own elicited closed-book answer in the Jacobian-lens reading at the final input position, on the arbitration case, by what the model then did. Lower is closer to the top of the distribution. The control column applies the same measure to answers the model does not hold for the question at hand, and is a per-model quantity, so the two Llama seeds share it. Reading. On both Llama seeds the model’s own correct answer sits at rank 1 whether or not the model goes on to produce it: adopting the wrong tool value does not push the parametric answer down the distribution, against a control median of 421. On gemma the pattern holds with one asymmetry — rank 1 when it keeps its answer, rank 2 when it follows the tool. On Qwen there is nothing to see: the median rank is around 2.6×1042.6× 10^4, no episode places the answer in the top 20, and the control sits at the same order of magnitude. This is not an artifact of how the answer is tokenized: scoring the best rank over all subtokens of the answer span rather than its first row moves the Qwen medians only to 25,548 and 27,378, still with nothing in the top 20, while leaving the Llama numbers unchanged. Read narrowly, the probe says that on two of four families the failure in the arbitration case is not a failure of availability. The answer is present and near the top of the read-out at the moment the model writes the tool’s value instead. That is consistent with the behavioural picture in §5.2 — interventions can move the choice without teaching the model anything new — but it is a suggestion from a probe, not evidence for it. Four caveats, all load-bearing. First, the rank is that of the memory answer — the string the model produced under forced closed-book elicitation — and not of the gold answer as such; on this cell the two coincide by construction, but the measure follows the model’s answer, and the distinction matters wherever they could differ. Second, in a majority of the kept-memory episodes the answer string also occurs somewhere in the prompt, where a top rank is much less surprising; restricted to episodes where it does not, the counts are smaller but the picture is unchanged (Llama seed 1: 25/26 in the top 20 after following the tool, 11/11 after keeping memory; gemma 21/24 and 8/8; Qwen 0/7 and 0/7). Third, the effect is absent on Qwen and only partial on gemma, so it is not a property of transformers here but of some of them. Fourth, a lens read at one position is a measurement of the representation under a particular projection, and we make no claim that it identifies a mechanism.

Instant research discovery