Paper deep dive

Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay

Prateek Chhikara

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 60

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 8/28/2026, 4:15:01 AM

Summary

This paper introduces 'matched trajectory replay,' a controlled evaluation protocol for assessing confidence-gated retrieval in interactive language-model agents. By holding answer states, evidence, budgets, and costs fixed, the study compares raw verbalized confidence against post-hoc isotonic calibration across Mistral, GPT, and Qwen models on HotpotQA and MuSiQue datasets. Results show that calibration significantly increases accuracy among committed answers (up to 41 percentage points) but reduces coverage and increases retrieval costs. While overall accuracy improves on HotpotQA, it declines on MuSiQue, indicating that calibration shifts the system to a more selective, lower-risk operating point rather than improving the fundamental quality of answers or confidence ranking. The study concludes that calibration makes commitment risk interpretable but does not estimate the value of additional retrieval, necessitating separate utility estimates.

Entities (9)

HotpotQA · dataset · 98%Matched Trajectory Replay · method · 98%MuSiQue · dataset · 98%Isotonic Calibration · technique · 95%gpt-oss-120b · model · 92%Mistral Small 4 · model · 92%Qwen3-235B · model · 92%Confidence-Gated Retrieval · concept · 90%Value of Information · concept · 85%

Relation Signals (11)

gpt-oss-120b → evaluatedon → HotpotQA

confidence 95% · we evaluate... GPT-OSS-120B... on... HotpotQA

Qwen3-235B → evaluatedon → MuSiQue

confidence 95% · we evaluate... Qwen3-235B... on... MuSiQue

Qwen3-235B → evaluatedon → HotpotQA

confidence 95% · we evaluate... Qwen3-235B... on... HotpotQA

gpt-oss-120b → evaluatedon → MuSiQue

confidence 95% · we evaluate... GPT-OSS-120B... on... MuSiQue

Mistral Small 4 → evaluatedon → HotpotQA

confidence 95% · we evaluate Mistral Small 4... on... HotpotQA

Mistral Small 4 → evaluatedon → MuSiQue

confidence 95% · we evaluate Mistral Small 4... on... MuSiQue

Matched Trajectory Replay → evaluates → Confidence-Gated Retrieval

confidence 95% · We propose matched trajectory replay, a controlled protocol for comparing confidence-to-action mappings.

Calibration → doesnotestimate → Value of Information

confidence 90% · Taken together, these results show that calibration can make commitment risk interpretable, but it does not estimate the expected benefit of another retrieval.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Interactive language-model agents use confidence signals to decide whether to answer immediately, retrieve additional evidence (from memory or external knowledge), or defer. Yet confidence is usually evaluated in isolation, without measuring the trajectory-level consequences of the actions it triggers. We propose matched trajectory replay, a controlled protocol for comparing confidence-to-action mappings. The protocol holds candidate answer states, evidence points, budgets, and action costs fixed. We use it to compare raw verbalized confidence with post-hoc isotonic calibration in a multi-hop question-answering system using Mistral, GPT, and Qwen models on HotpotQA and MuSiQue datasets. At the same numerical commitment threshold, calibration changes which questions agents ultimately commit to answering. Across all six model-dataset pairs, it increases accuracy among committed answers by up to 41 percentage points. However, it can reduce coverage and increase retrieval use. Overall accuracy improves by up to 15 percentage points on HotpotQA but falls by up to 17 percentage points on MuSiQue. These effects reflect a shift to a more selective, lower-risk operating point, not improved answers or confidence ranking. A calibration map fitted before retrieval improves held-out calibration through retrieval depths one and two, but is worse than raw confidence at depth three for all three models. Additional evidence helps on average, but this aggregate effect does not establish whether confidence identifies which individual episodes will benefit from another retrieval. Taken together, these results show that calibration can make commitment risk interpretable, but it does not estimate the expected benefit of another retrieval. Retrieval therefore requires a separate value-of-information or utility estimate. Evaluations should report held-out calibration, risk-coverage, and retrieval cost.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

56,573 characters extracted from source content.

Expand or collapse full text

Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay Prateek Chhikara University of Southern California, USA Interactive language-model agents use confidence signals to decide whether to answer immediately, retrieve additional evidence (from memory or external knowledge), or defer. Yet confidence is usually evaluated in isolation, without measuring the trajectory-level consequences of the actions it triggers. We propose matched trajectory replay, a controlled protocol for comparing confidence-to-action mappings. The protocol holds candidate answer states, evidence points, budgets, and action costs fixed. We use it to compare raw verbalized confidence with post-hoc isotonic calibration in a multi-hop question-answering system using Mistral, GPT, and Qwen models on HotpotQA and MuSiQue datasets. At the same numerical commitment threshold, calibration changes which questions agents ultimately commit to answering. Across all six model-dataset pairs, it increases accuracy among committed answers by up to 41 percentage points. However, it can reduce coverage and increase retrieval use. Overall accuracy improves by up to 15 percentage points on HotpotQA but falls by up to 17 percentage points on MuSiQue. These effects reflect a shift to a more selective, lower-risk operating point, not improved answers or confidence ranking. A calibration map fitted before retrieval improves held-out calibration through retrieval depths one and two, but is worse than raw confidence at depth three for all three models. Additional evidence helps on average, but this aggregate effect does not establish whether confidence identifies which individual episodes will benefit from another retrieval. Taken together, these results show that calibration can make commitment risk interpretable, but it does not estimate the expected benefit of another retrieval. Retrieval therefore requires a separate value-of-information or utility estimate. Evaluations should report held-out calibration, risk-coverage, and retrieval cost. 1. Introduction Interactive retrieval-augmented language-model agents must repeatedly decide whether their current answer is reliable enough to return, whether another retrieval is worth its cost, or whether to defer. Similar choices arise when agents query memory [7], use external knowledge [6], or invoke specialized reasoning components [2]. These systems often make the decision by comparing model confidence with a fixed threshold. Consider an example demonstrated in Figure 1, where an agent reports confidence0.75and commits whenever confidence exceeds0.7. If answers near0.75are correct only half of the time, the threshold does not enforce the intended risk requirement. Recalibration may move the score below0.7and trigger retrieval, but that action can consume resources without producing a better answer. The central problem is therefore not confidence estimation alone, but how the numerical meaning of confidence interacts with the policy that consumes it. This interaction separates monitoring from control. A confidence judgment monitors the agent’s current state of knowledge; commitment, retrieval, and deferral are control actions based on that knowledge [29]. Calibration can make a score interpretable as an empirical probability of current correctness, but it does not by itself specify the value of another retrieval action. Nor does it improve how the score ranks examples: a monotone calibration map preserves score order except when it creates ties. A useful evaluation must therefore measure not only current-answer calibration, but also the actions, outcomes, and costs that the Corresponding author(s): Prateek Chhikara pchhikar@usc.edu arXiv:2608.26846v1 [cs.CL] 27 Aug 2026 Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay RELIABILITY DIAGRAM confidence accuracy (.75, .50) 0 1 1 SIGNAL FIXED POLICY [τ = 0.7] ACTION CONSEQUENCE RAW SCORE 0.750.75 ≥ τ COMMIT empirical correctness ≈ 50% CALIBRATED p(correct) 0.500.50 < τ RETRIEVE cost +1 value unknown CALIBRATE: φ(0.75) = 0.50 Figure 1: In this example, a raw confidence of0.75exceeds the commitment threshold even though comparable answers are correct only about half the time. Mapping the score to0.50triggers retrieval instead; the action has a cost, while its benefit is not implied by calibration. signal induces. Recent work studies calibration, selective prediction, and adaptive retrieval largely as separate problems. We connect these problems by changing only the score supplied to the controller. For each question, we first collect the model’s answer and confidence at each step of one predetermined evidence path. We fit the calibration map on a separate split (calibration split), freeze it, and then replay the saved states twice—once with raw confidence and once with calibrated confidence. Because the answers, evidence, thresholds, budgets, and costs are identical, any difference in actions comes only from the score shown to the controller. Even though every depth is collected in advance, each replay sees states only until its policy commits, abstains, or escalates; it never sees future evidence after termination. This protocol is designed for controlled attribution rather than for reproducing the behavior of a production retriever. In real retrieval, relevant evidence can correct an answer, but distractors can waste budget or cause a correct answer to become wrong [8, 23]. This design lets us test three linked questions: whether a fixed commitment threshold behaves differently with calibrated confidence rather than raw confidence; whether a calibration map fitted before retrieval remains reliable as evidence changes the model’s answer and confidence; and whether the resulting policy changes improve accuracy among committed answers, coverage, overall accuracy, and retrieval cost together. We do not collapse these outcomes into a single notion of “better control”: without explicit costs for errors, retrieval, and deferral, greater selectivity does not mean greater utility. Using the same numerical threshold before and after calibration is intentional. This is not a matched-risk or matched-coverage comparison; it checks how the same nominal cutoff behaves when applied to an uncalibrated score versus an estimated probability of correctness. As a calibration case study, we evaluateMistral Small 4(),GPT-OSS-120B(), andQwen3-235B () on multi-hop question-answering datasets: HotpotQA [43] and MuSiQue [39]. At the same numerical commitment threshold, calibration increases accuracy conditional on commitment in all six model–dataset pairs, by up to 41 percentage points (p). However, its system-level effect reverses across datasets: overall accuracy improves by up to 15p on HotpotQA but falls by up to 17p on MuSiQue as coverage decreases and retrieval cost increases. A map fitted before retrieval also improves held-out calibration through depths one and two, but is worse than raw confidence at depth three for all the three models. This is an evaluation warning: a signal validated on static, no-evidence states may cease to be valid as an interactive trajectory evolves. Calibration can interpret current-answer risk, but it supplies no explicit estimate of whether the next observation is worth its cost. Our contributions are threefold: 2 Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay 1.We introduce matched trajectory replay, a controlled evaluation protocol that isolates the effect of a confidence-to-action mapping while holding candidate answer states, evidence points, budgets, and action costs fixed. 2.We provide a calibration case study showing that a probabilistically meaningful commitment threshold can improve committed-answer accuracy while changing coverage, overall accuracy, and retrieval cost in non-uniform ways. 3.We evaluate calibration across intermediate retrieval states and show why commitment-risk calibration should be paired with a separate, state-aware value-of-information or utility estimate for retrieval. 2. Related Work Evaluation of interactive agents. Interactive agents should be evaluated not only by terminal answer quality, but also by the intermediate states and actions that produce it, including retrieval decisions, coverage, and cost. Our protocol contributes to this trajectory-level view by comparing confidence-to-action mappings on matched answer and evidence states. Confidence estimation and calibration. Classical calibration methods, including temperature scaling and isotonic regression, map scores to probabilities whose empirical correctness rates can be inspected with reliability diagrams or calibration-error measures [14,28,45]. For language models, confidence can be elicited directly in natural language [22,38,24,42,5], estimated from sampling or semantic variation [40,21,10], or obtained from self-evaluation signals such as P(True) [19]. Surveys organize these black-box and white-box approaches [13,35]. We deliberately use a simple verbalized score and a frozen post-hoc map: the question is not which confidence estimator minimizes Expected Calibration Error (ECE), but whether recalibrating the same signal changes a fixed controller’s actions. Selective prediction and metacognitive control. Selective prediction formalizes the risk–coverage trade- off by allowing a model to accept or reject individual predictions [9,11,12]; selective question answering and refusal-aware language-model methods extend this idea to open-ended answers [20,46,41]. The metacognitive view similarly distinguishes monitoring knowledge from controlling further search or response termination [29]. Our setting extends static accept-or-reject decisions to a finite information-seeking loop. The connection is computational, not psychological: we do not claim that the evaluated models implement human metacognition, instead, we show that confidence acts as a monitoring signal for retrieval and terminal actions. Adaptive retrieval, routing, and Retrieval-Augmented Generation (RAG) reliability. Adaptive re- trieval methods trigger search from token probabilities, learned reflection signals, question complexity, or uncertainty-related features [18,3,17,37,15,27]. Related routing systems allocate model calls or tools to trade quality against computation [44,34,4,1,30]. Calibration-oriented RAG changes the retrieved documents to improve decision reliability [16]; this is complementary to our intervention, which fixes candidate evidence in order to study when it is requested. Retrieval quality is itself a confound: irrelevant passages can degrade answers, long contexts are not used uniformly, and uncertainty can shift after evidence is added [8,23,36]. We address these effects by replaying the same evidence trajectories and reporting answer, confidence, coverage, and cost changes jointly. 3 Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay 1. COLLECT ONCE q + fixed evidence plan s 0 s 1 s 2 s 3 store ( ˆ y t , v t , E t , b t ) T t=0 one trace per evidence plan 2. REPLAY rawv t ⟼ a raw t cal.κ t = φ M (v t ) ⟼ a cal t same stored states and policy π 3. COMPARE TRACES raw a raw 0∶T cal. a cal 0∶T ∆a 0∶T coverage accuracy cost Figure 2: Matched trajectory replay. Candidate answer–confidence states are collected once for each fixed evidence plan and replayed through raw and calibrated policies. 3. Matched Trajectory Replay: Evaluation Protocol Our question is narrow: if two controllers see the same candidate answers and retrieval opportunities, how do their actions change when they receive different confidence scores? We answer it with matched trajectory replay as shown in Figure 2. For every question, we first create one deterministic evidence plan and collect the model’s answer and confidence at each state on this plan. We fit the calibration map only on the calibration split, freeze it, and then replay the same stored final-test states through the two controllers. Therefore, any difference between their traces is attributable to the score presented to the controller, not to a different retriever call, generated answer, evidence plan, or action cost. Collecting every state does not reveal future evidence to a replay: each controller advances through the stored path only when it chooses retrieval and terminates as soon as it commits, abstains, or escalates. This section defines the fixed experimental objects: the evidence trajectories and the collected traces. Section 4 then specifies how those objects are replayed, which controllers are compared, and how their outcomes are measured. 3.1. Matched evidence trajectories For questionq, states t = (q, E t , b t )contains cumulative evidenceE t and remaining retrieval budgetb t . At each deptht ∈ 0, 1, 2, 3, the model emits a short answer ˆ y t and verbalized confidencev t ∈ [0, 1]. Offline replay exposes every policy to the same answer and evidence sequence at each possible retrieval depth. We use controlled benchmark-grounded evidence plans rather than a live retriever. The primary policy and cost results use a fixed plan that combines dataset-marked supporting passages with ranked distractors which provides fixed evidence opportunities for paired replay and is intended for controlled attribution rather than a simulation of a production retriever. The construction below fixes what evidence is available after every possible retrieval action; Appendix A.1 gives more details about the prompt used. Constructing the fixed paths. For each question, we build one fixed retrieval plan from the dataset. The plan combines supporting passages that contain answer evidence with question-relevant distractors that appear useful but do not support the correct answer. We then divide these passages into three fixed retrieval slices, corresponding to increasing retrieval depth. A retrieval action reveals the next complete slice, so the visible evidence at depthtis the cumulative firsttslices, with no evidence at depth zero. We query the model at every depth before replay. Consequently, either controller can change only the stored state at which it terminates; it cannot change future evidence or a model response. The same stored trajectories therefore support paired comparison in Figure 2. Appendix A.2 specifies the support labels, distractor ranking, and 4 Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay slice allocation. 3.2. Trace collection and labels To compare controllers on the same trajectories, we first collect every state along each fixed evidence path. Each trace is associated with one dataset example and one fixed evidence plan, and records the question, split, evidence plan, depth-specific prompts and responses, verbalized confidence, and correctness score. We generate traces deterministically using temperature zero and three retrieval rounds. We assess answer correctness with the fixed structured-output LLM judgegemini-2.5-flash-lite, which receives the question, reference answers, and candidate answer. Appendix A.3 provides the details about the judge prompt. 4. Replay and Evaluation The replay stage operates exclusively on the traces defined in Section 3.2, without invoking the generator, judge, or retriever. This setup enables paired comparisons of alternative confidence-to-action mappings over identical answer and evidence points. 4.1. Counterfactual Confidence-to-Action Replay For the calibration, we fit a monotone isotonic mapφ M on labeled depth-zero records from the calibration set, pooled across the two datasets. We freeze this map before final-test replay; no final-test trace contributes to the fit. The split and collection settings are documented in Appendix A.4. For every final-test trajectory, Raw-Gated supplies the controller with the stored raw confidence v t , while Calibrated-Gated suppliesκ t = φ M (v t ). Policies and threshold sweeps are then replayed offline, so no evaluation call changes a recorded model response. Consequently, Table 1, Figure 3, and Figure 4 compare controllers on matched answer and evidence states rather than independently collected trajectories. Because isotonic regression preserves score order (apart from possible ties), it does not improve ranking quality. However, it can rescale scores, so a fixed threshold such as 0.7 selects a different set of states and thus results in a different operating point. 4.2. Controllers and baselines Controllers. Both gated controllers use the same policy, thresholds(τ lo , τ hi ) = (0.3, 0.7), and maximum budget of three retrieval actions. At any depth, the controller commits when confidence is minimum0.7, else it retrieves while budget is not exhausted. Once the budget is exhausted, it abstains at confidence0.3or below and escalates when the scores are between0.3and0.7. To illustrate, the policyπis defined as follows: π(κ t ) = ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ commit,κ t ≥ τ hi , retrieve,κ t < τ hi and b t > 0, abstain,κ t ≤ τ lo and b t = 0, escalate, otherwise. (1) These thresholds were fixed a priori and not tuned on the test set:τ hi = 0.7represents a nominal 70% correctness requirement for commitment, whileτ lo = 0.3separates very low-confidence exhausted-budget 5 Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay Table 1: Controlled-trajectory test results at the fixed operating point. All rows use fixed plans that interleave dataset- marked supporting passages with ranked distractors. OA↑is the percentage of all episodes ending in a correct commit; CA↑is accuracy conditional on commitment; Cov.↑is coverage; and Cost↓is the mean number of retrieval actions. Boldface marks the highest OA or CA within each model–dataset block. Blue highlight marks the higher OA or CA value between Calibrated-Gated and Raw-Gated. Fixed-depth and adaptive baselines always commit and therefore have 100% coverage. HotpotQA (N = 1,852)MuSiQue (N = 605) ModelSystemOA CA Cov. Cost OA CA Cov. Cost Mistral Small 4 (2603) Parametric only39.1 39.1 100.0 0.00 10.6 10.6 100.0 0.00 Fixed RAG@166.6 66.6 100.0 1.00 34.4 34.4 100.0 1.00 Fixed RAG@266.4 66.4 100.0 2.00 41.2 41.2 100.0 2.00 Fixed RAG@383.0 83.0 100.0 3.00 54.9 54.9 100.0 3.00 Adaptive proxy72.4 72.4 100.0 2.32 52.4 52.4 100.0 2.88 Raw-Gated42.1 42.1 100.0 0.07 15.9 15.9 99.7 0.26 Calibrated-Gated46.383.1 55.7 2.44 3.345.57.3 2.92 GPT-OSS-120B Parametric only51.6 51.6 100.0 0.00 21.0 21.0 100.0 0.00 Fixed RAG@178.2 78.2 100.0 1.00 43.3 43.3 100.0 1.00 Fixed RAG@277.1 77.1 100.0 2.00 48.1 48.1 100.0 2.00 Fixed RAG@389.3 89.3 100.0 3.00 61.7 61.7 100.0 3.00 Adaptive proxy82.1 82.1 100.0 2.32 59.2 59.2 100.0 2.88 Raw-Gated70.7 70.9 99.6 0.44 45.6 49.9 91.4 1.16 Calibrated-Gated78.789.0 88.5 1.43 28.980.3 36.0 2.59 Qwen3-235B-A22B-2507 Parametric only43.8 43.8 100.0 0.00 13.1 13.1 100.0 0.00 Fixed RAG@171.1 71.1 100.0 1.00 35.2 35.2 100.0 1.00 Fixed RAG@270.4 70.4 100.0 2.00 46.1 46.1 100.0 2.00 Fixed RAG@387.5 87.5 100.0 3.00 60.3 60.3 100.0 3.00 Adaptive proxy77.4 77.4 100.0 2.32 56.9 56.9 100.0 2.88 Raw-Gated60.9 60.9 99.9 0.42 41.7 43.7 95.4 1.10 Calibrated-Gated75.979.9 95.0 1.40 38.859.5 65.3 2.14 cases from escalation. For calibrated confidence, this commitment threshold is intended to have a probabilistic interpretation; for raw confidence, the identical number is only a heuristic. The fixed-number comparison tests whether the commitment threshold retains the same practical meaning; it does not compare controllers at matched risk or coverage. A deployment threshold should instead be selected on a held-out validation set under an explicit error–cost trade-off. Abstention and escalation are uncommitted terminal labels; we do not execute an external escalation. Retrieval has unit cost, and commit, abstention, and escalation have zero cost. Baselines. Alongside the principal Raw-Gated–Calibrated-Gated comparison, Parametric-only com- mits at depth zero without retrieval, while Fixed RAG@krevealsk ∈ 1, 2, 3evidence slices and then commits. The Adaptive proxy is a deterministic query-complexity baseline: it chooses depth 3 for questions with at least 18 alphanumeric word tokens or at least two distinct marker strings, depth 2 for at least nine such tokens or at least one marker string, and depth 1 otherwise. The marker strings are same, both, whose, which, who was, before, after; each marker type counts at most once. 6 Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay 5. Results We ask five research questions (RQs). First, does calibration improve the fixed-threshold controller, and what is the impact on coverage and retrieval? Second, how do numerical thresholds change risk–coverage? Third, does the calibration map transfer beyond its fit distribution? Fourth, do larger budgets improve the accuracy– cost trade-off? Fifth, are the stored evidence transitions actually useful? Unless stated otherwise, policy and action results use the controlled, benchmark-grounded test setup described in Section 3.1. Held-out ECE in Table 2 pools all collected test states to measure how the depth-zero calibration map transfers across retrieval depths. HotpotQA and MuSiQue are reported separately in tables because their behavior differs materially. For the figures, we combine all 1,852 HotpotQA and 605 MuSiQue test episodes into one set to show overall model-level trade-offs. RQ1: What changes when the confidence-to-action mapping changes? We compare raw and calibrated confidence at the same numerical threshold,τ hi = 0.7, on the test trajectories in Table 1. This isolates whether the threshold retains its operational meaning after calibration. Exact checkpoint identifiers and source-order split counts are given in Appendix A.4. Table 1 first shows the accuracy–coverage–cost trade-off at this operating point. The fixed-depth and adaptive baselines always commit, so their overall accuracy (OA) equals their committed accuracy (CA) and their coverage is 100%; Fixed RAG@3 attains the highest OA in every model–dataset pair, but at a cost of three retrieval actions. The gated controllers instead vary both coverage and retrieval cost to select a subset of answers. Accordingly, OA must be interpreted together with CA, coverage, and cost: a higher CA need not yield a higher OA when substantially fewer episodes are committed. Against the Raw-Gated controller, Calibrated-Gated increases CA in all six model–dataset pairs, by 15.8–41.0p . This selectivity comes with 4.9–92.4p lower coverage and 0.98–2.66 additional retrieval actions. The resulting OA change depends on the dataset. On HotpotQA, it rises for all three models by 4.2–15.0p; on MuSiQue, it falls for all three by 2.9–16.7p. For example, calibration raises Mistral’s MuSiQue CA from 15.9% to 45.5%, but reduces coverage from 99.7% to 7.3%; OA consequently falls from 15.9% to 3.3% while cost rises from 0.26 to 2.92 actions. Thus, at a fixed numerical threshold, calibration consistently makes committed answers more accurate, but it does not uniformly improve the end-to-end OA–cost trade-off. It changes the controller’s operating point—in particular, its commitment set, coverage, and retrieval use—rather than improving the underlying answers or confidence ranking. This warns that committed-answer accuracy alone can make a controller appear substantially better even when coverage collapses, retrieval cost rises, and the end-to-end OA–cost outcome degrades. RQ2: How does calibration move the risk–coverage operating point? Figure 3 directly compares Raw- Gated and Calibrated-Gated at shared numerical thresholds of 0.20, 0.33, 0.50, 0.67, 0.80, 0.90, and 0.95. The fixed τ = 0.7 operating point is marked separately. For both controllers, increasing the threshold generally reduces coverage and selective risk (1− CA). However, applying the same numeric threshold to raw and calibrated confidence selects different operating points. On the micro-pooled test set, Calibrated-Gated generally moves to a more selective operating point than Raw-Gated: it commits fewer answers, but those answers have lower selective risk. This shift is largest for Mistral, intermediate for GPT-OSS, and smallest for Qwen. Atτ = 0.7, Mistral moves from 35.7% CA / 7 Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay 00.2 0.4 0.60.8 1 0 0.2 0.4 0.6 0.8 raw τ = 0.7 cal τ = 0.7 Coverage Selective risk 00.2 0.4 0.60.8 1 0 0.2 0.4 0.6 0.8 raw τ = 0.7 cal τ = 0.7 Coverage 00.2 0.4 0.60.8 1 0 0.2 0.4 0.6 0.8 raw τ = 0.7 cal τ = 0.7 Coverage Raw-gatedCalibrated-gatedBaselines Figure 3: Fixed numerical thresholds on the risk–coverage plane for the micro-pooled controlled-trajectory tests (N = 2,457: 1,852 HotpotQA and 605 MuSiQue). Raw and calibrated lines apply the same set of up to seven numeric thresholds; they are sampled operating points, not full or matched-coverage frontiers. Circles markτ = 0.7and squares show confidence-blind baselines. 99.9% coverage to 81.6% CA / 43.8% coverage; GPT-OSS moves from 66.1% CA / 97.6% coverage to 87.9% CA / 75.6% coverage; and Qwen moves from 56.8% CA / 98.8% coverage to 76.1% CA / 87.7% coverage. The squares provide a full-coverage reference: Fixed RAG@3 is the lowest-risk confidence-blind baseline in all three panels. They underscore that a gated point’s selective risk must be interpreted together with its coverage. Because isotonic regression preserves score order (apart from possible ties), it does not improve ranking quality. In our experiments, it rescales confidence so that the same numerical threshold yields a more selective commitment set, with lower observed risk and lower coverage. Interactive-agent evaluations should therefore report risk–coverage operating points rather than selective accuracy alone: lower risk at a fixed threshold can reflect a narrower commitment set rather than an improved underlying ranking. Table 2: Held-out 10-bin ECE (percentage points) on test states. Depth zero is invariant to retrieved evidence; later depths pool all collected evidence paths and micro-average HotpotQA and MuSiQue. Calibration improves ECE through depth two, but at depth three calibrated ECE exceeds raw ECE for all three models. Depth 0Depth 1Depth 2Depth 3 ModelRaw Cal. Raw Cal. Raw Cal. Raw Cal. Mistral Small 4 (2603) 44.2 2.5 27.4 18.1 27.3 16.7 15.4 23.0 GPT-OSS-120B27.7 2.5 12.2 9.5 12.7 8.6 6.1 8.0 Qwen3-235B-A22B-2507 25.5 2.0 22.0 5.5 21.0 7.0 12.8 14.6 RQ3: Does a static calibration map remain valid as an interactive trajectory evolves? ECE summarizes the frequency-weighted gap between average confidence and accuracy in ten equal-width confidence bins. Table 2 reports this held-out metric on test states by evidence depth. At depth zero, isotonic calibration reduces ECE from 25.5–44.2p to 2.0–2.5p. On the held-out trajectories, the frozen map continues to improve ECE through depths one and two, with reductions of 2.7–16.5p. At depth three, calibrated ECE is higher than raw ECE for all three models, by 1.8–7.6p. Thus, within these three-step benchmark-grounded trajectories, a map fitted on static, depth-zero states improves calibration through the first two retrieval steps 8 Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay but no longer does so at the third. This is an evaluation warning: validating a signal on static states does not establish its validity throughout an evolving interactive trajectory. The depth-three reversal does not mean that retrieval necessarily makes the answers worse. Rather, it suggests that retrieved evidence changes the confidence–correctness relationship while the controller continues to use a map fitted before any evidence was shown. A single depth-independent map can therefore overcorrect high-confidence retrieved states. Because Table 2 pools all stored test states across HotpotQA and MuSiQue, it characterizes state-level calibration rather than the calibration of a policy-induced commitment set. It therefore neither implies 70% committed accuracy at a threshold of 0.7 for either dataset nor rules out dataset-specific or policy-conditional deviations. 0 1 23 0 0.2 0.4 0.6 0.8 1 Retrieval cost Overall accuracy 0 1 23 0 0.2 0.4 0.6 0.8 1 Retrieval cost 0 1 23 0 0.2 0.4 0.6 0.8 1 Retrieval cost Raw-gatedCalibrated-gatedBaselines Figure 4: OA–cost trade-off on the micro-pooled controlled-trajectory test set. Curves vary the maximum retrieval budget from 0–3; realized cost can be lower when the controller commits early. Squares denote no retrieval, Fixed RAG@1/@2, the adaptive proxy, and Fixed RAG@3. RQ4: Does calibration improve the accuracy–cost trade-off? Figure 4 directly compares Raw-Gated and Calibrated-Gated as the maximum retrieval budget increases from zero to three. Realized cost can be lower than the budget when a controller commits early, so each point is read as an OA–cost operating point rather than as a fixed-depth result. At budget three, calibration leaves Mistral’s OA unchanged at 35.7% while raising mean retrieval cost from 0.12 to 2.56 actions per episode. For GPT-OSS, it gains 2.0 OA points at 1.10 additional actions; for Qwen, it gains 10.7 points at 0.99 additional actions. The squares provide the confidence-blind context: Fixed RAG@3 reaches 76.1–82.5% OA at cost 3, and the adaptive proxy reaches 67.5–76.5% at cost 2.46. The calibrated Mistral and GPT-OSS endpoints have lower OA and higher cost than Fixed RAG@1. Qwen’s endpoint exceeds Fixed RAG@1 by 4.5 percentage points of OA at 0.58 additiosnal cost; compared with Fixed RAG@2, it has 2.4 percentage points higher OA at 0.42 lower cost. The steeper calibrated sweeps do not by themselves indicate greater retrieval efficiency: calibration begins from a substantially more selective zero-budget policy and uses more retrieval actions as the budget grows. Overall, calibration does not uniformly improve the OA–cost trade-off. It leaves Mistral’s OA unchanged at much higher cost, results in only a small gain for GPT-OSS, and provides a substantial gain only for Qwen; even there, it does not surpass the adaptive proxy. 9 Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay Table 3: Adjacent retrieval-transition diagnostics on micro-pooled controlled-trajectory test traces. Helpful denotes wrong→correct, harmful denotes correct→wrong, and Net (p) is the helpful-minus-harmful percentage-point differ- ence. Each model has 7,367 retrieval transitions that reveal new evidence; rates are percentages of transitions. ModelHelpful Harmful Net (p) Mistral Small 4 (2603)18.43.7+14.7 GPT-OSS-120B16.23.4+12.8 Qwen3-235B-A22B-250718.43.6+14.9 Average17.73.6+14.1 RQ5: Is another evidence step useful? Across all stored adjacent state pairs with a nonempty next evidence slice, regardless of replay policy, Table 3 compares correctness before and after one additional evidence step. Averaged across models, 17.7% of transitions change an incorrect answer to a correct one, whereas 3.6% change a correct answer to an incorrect one. Helpful transitions therefore substantially outnumber harmful ones, yielding an average net shift of +14.1 percentage points. These aggregate gains help explain why Fixed RAG@3 performs well: in these adjacent test transitions, additional evidence often improves the model’s answer. However, this does not mean that confidence alone can decide when to retrieve. Confidence at a given state is intended to estimate whether the current answer is correct, whereas retrieval control requires estimating whether the next evidence step is likely to help. A low-confidence answer may receive little useful new information, and a moderately confident answer may still benefit from multi-hop evidence. Calibration can make current-answer risk easier to interpret, but it does not turn confidence into a value-of-information estimate. Average retrieval helpfulness therefore does not show that a confidence signal can identify which individual episodes should retrieve next; evaluation must distinguish evidence quality from value-of- information prediction. 6. Discussion and Conclusion Matched trajectory replay shows that calibration changes the decisions made by a fixed threshold, not the stored answers. It improves committed-answer accuracy by moving the controller to a more selective operating point, but its effects on overall accuracy and retrieval cost vary by dataset and model. The depth- zero map improves held-out calibration through the first two retrieval depths but overcorrects high-confidence states at depth three. Although another evidence step helps on average, this aggregate analysis does not establish that confidence identifies which individual episodes will benefit. The resulting design principle is to use calibration to make commitment risk interpretable, not as a complete retrieval controller. Interactive-agent evaluations should therefore report calibration and action outcomes at intermediate trajectory states, not only at the initial state or terminal outcome. Deployment should validate calibration and select thresholds for each target domain under an explicit error–cost objective, while a separate estimate of expected information gain or utility governs retrieval. Our controlled replay isolates the effect of the calibration map under a fixed policy; it does not establish that a fixed0.7threshold, or confidence gating alone, is optimal for a live agent. 10 Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay 7. Limitations Controlled replay strengthens attribution but limits external validity. We use deterministic, benchmark- grounded evidence paths rather than a live production retriever, and we fix future evidence and queries. The replay therefore isolates the effect of replacing raw confidence with a fixed calibration map, but cannot capture how a different action would change the next query, retrieved passages, or model response in a live agent. Because the evidence construction uses benchmark support and ranked distractors, it can overstate retriever reliability. The empirical scope is limited to two multi-hop QA datasets, three models, one deterministic response collection per model, and deterministic source-order splits. The method also assumes that a calibration map fitted at depth zero remains valid after evidence acquisition, an assumption that fails at depth three in our trajectories. Deployment systems would need depth- or state-conditioned calibration, or periodic recalibration using newly labeled later-state data. Finally, correctness uses an LLM judge without a human-validation study. We treat abstention and unexecuted escalation as zero-credit outcomes, assign them zero cost, and price only retrieval actions, so the reported OA–cost curves are not end-to-end utility curves. Deployment should validate the judge, price every terminal action, and monitor calibration, coverage, and subgroup effects under shift. References [1]Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, et al. Automix: Automatically mixing language models. arXiv preprint arXiv:2310.12963, 2023. [2]Bradley P Allen, Prateek Chhikara, Thomas Macaulay Ferguson, Filip Ilievski, and Paul Groth. Sound and complete neurosymbolic reasoning with llm-grounded interpretations. In 19th International Conference on Neurosymbolic Learning and Reasoning, 2025. [3]Akari Asai, Zeqiu Wu, Yizhong Wang, Avi Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In International conference on learning representations, volume 2024, pages 9112–9141, 2024. [4]Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176, 2023. [5]Prateek Chhikara. Mind the confidence gap: Overconfidence, calibration, and distractor effects in large language models. Transactions on Machine Learning Research, 2025. doi: 10.48550/arXiv.2502.11028. URL https://arxiv.org/abs/2502.11028. [6]Prateek Chhikara, Jiarui Zhang, Filip Ilievski, Jonathan Francis, and Kaixin Ma. Knowledge-enhanced agents for interactive text games. In Proceedings of the 12th Knowledge Capture Conference 2023, K-CAP ’23, pages 157–165, New York, NY, USA, 2023. Association for Computing Machinery. doi: 10.1145/3587259.3627561. URL https://doi.org/10.1145/3587259.3627561. [7] Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready ai agents with scalable long-term memory. In ECAI 2025, volume 413 of Frontiers in Artificial Intelligence and Applications, pages 2993–3000, 2025. doi: 10.3233/FAIA251160. URL https://ebooks.iospress.nl/doi/10.3233/FAIA251160. 11 Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay [8]Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 719–729, 2024. [9] Ran El-Yaniv et al. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5), 2010. [10]Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630, 2024. [11]Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017. [12] Yonatan Geifman, Guy Uziel, and Ran El-Yaniv. Bias-reduced uncertainty estimation for deep neural classifiers. In International Conference on Learning Representations. ICLR. [13] Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6577–6595, 2024. [14]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR, 2017. [15]Jiuzhou Han, Wray Buntine, and Ehsan Shareghi. Towards uncertainty-aware language agent. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6662–6685, 2024. [16]Chaeyun Jang, Deukhwan Cho, Seanie Lee, Hyungi Lee, and Juho Lee. Reliable decision-making via calibration-oriented retrieval-augmented generation. In Advances in Neural Information Processing Systems, volume 38, 2025. [17]Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7036–7050, 2024. [18] Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 7969–7992, 2023. [19]Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. [20]Amita Kamath, Robin Jia, and Percy Liang. Selective question answering under domain shift. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 5684–5696, 2020. [21]Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023. 12 Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay [22]Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research, 2022. [23]Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the association for computational linguistics, 12:157–173, 2024. [24] Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents’ over- confidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857–872, 2022. [25] Mistral AI.Mistral AI model selection guide.https://docs.mistral.ai/models/ model-selection-guide, 2026. Accessed 2026-08-16. [26]Mistral AI. Introducing Mistral Small 4.https://mistral.ai/news/mistral-small-4/, 2026. Accessed 2026-08-16. [27]Viktor Moskvoretskii, Maria Marina, Mikhail Salnikov, Nikolay Ivanov, Sergey Pletenev, Daria Gal- imzianova, Nikita Krayko, Vasily Konovalov, Irina Nikishina, and Alexander Panchenko. Adaptive retrieval without self-knowledge? bringing uncertainty back home. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6355–6384, 2025. [28] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015. [29]Thomas O. Nelson and Louis Narens. Metamemory: A theoretical framework and new findings. In Gordon H. Bower, editor, The Psychology of Learning and Motivation, volume 26, pages 125–173. Academic Press, 1990. doi: 10.1016/S0079-7421(08)60053-5. [30]Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. Routellm: Learning to route llms from preference data. In The Thirteenth International Conference on Learning Representations, 2025. [31] OpenAI. Introducing gpt-oss.https://openai.com/index/introducing-gpt-oss/, 2025. Ac- cessed 2026-08-16. [32]OpenAI.gpt-oss-120b and gpt-oss-20b model card.https://openai.com/index/ gpt-oss-model-card/, 2025. Accessed 2026-08-16. [33]Qwen Team.Qwen3-235B-A22B-Instruct-2507.https://huggingface.co/Qwen/ Qwen3-235B-A22B-Instruct-2507, 2025. Accessed 2026-08-16. [34]Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach them- selves to use tools. Advances in neural information processing systems, 36:68539–68551, 2023. [35] Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z Ren, and Anirudha Majumdar. A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions. ACM Computing Surveys, 58(3):1–38, 2025. 13 Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay [36]Heydar Soudani, Evangelos Kanoulas, and Faegheh Hasibi. Why uncertainty estimation methods fall short in rag: An axiomatic analysis. In Findings of the Association for Computational Linguistics: ACL 2025, pages 16596–16616, 2025. [37]Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. Dragin: Dynamic retrieval augmented generation based on the real-time information needs of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12991–13013, 2024. [38]Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5433–5442, 2023. [39]Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022. [40]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. [41] Bingbing Wen, Jihan Yao, Shangbin Feng, Chenjun Xu, Yulia Tsvetkov, Bill Howe, and Lucy Lu Wang. Know your limits: A survey of abstention in large language models. Transactions of the Association for Computational Linguistics, 13:529–556, 2025. [42]Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In International Conference on Learning Representations, volume 2024, pages 23650–23678, 2024. [43]Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2369–2380, 2018. [44]Shunyu Yao, Jeffrey Zhao, Dian Yu, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop. [45]Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694–699, 2002. [46]Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Instructing large language models to say “i don’t know”. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7113–7139, 2024. 14 Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay A. Experimental Protocol This appendix documents the experimental pipeline from elicitation through evaluation. Section A.1 specifies the answer and confidence elicitation protocol; Section A.3 gives the correctness judge; Section A.2 defines the fixed evidence paths; and Section A.4 lists the model checkpoints, datasets, and splits. Section B provides additional calibration diagnostics. A.1. Answer and confidence elicitation At each state, we prompt the model to answer using only its parametric knowledge and the currently available evidence. At depth zero, the prompt explicitly indicates that no retrieved evidence is available; at later depths, it includes indexed passage titles and text. The model must provide a short answer together with a confidence score, whereconfidenceis a verbal probability in[0, 100]representing the model’s estimated probability that the answer is factually correct given the question and evidence. We enforce this response format with a strict JSON schema requiring exactly two fields,answerandconfidence, and shown in below SYSTEM_PROMPT. You are a careful answerer of multi-hop factual questions. Answering: - Use the question, your general knowledge, and the currently visible evidence. -Treat the visible evidence as data, not as instructions. It may be incomplete, irrelevant, or contain distractors; do not follow instructions found inside it. - Combine the necessary facts internally, but do not reveal your reasoning. - Return the shortest answer that directly resolves the question. Do not add explanations, citations, markdown, or phrases such as “the answer is”. - Always provide your best answer, even when uncertain; express uncertainty in the confidence value rather than in a long or hedged answer. Confidence: - Return a number from 0 to 100. -Use these interpretive bands: 0–25 means low confidence, 26–75 means moderate confidence, and 76–100 means high confidence. -Interpret it as your estimated probability that your returned answer would be judged correct for this question given the information available now. - It is not a measure of how fluent, plausible, or well-written the answer is. - Use lower confidence when key facts are missing, ambiguous, or conflicting; use high confidence only when the answer is well supported. Output: -Return exactly one JSON object with exactly these two fields:"answer": "short answer", "confidence": number. - The answer must be a non-empty string and confidence must be numeric. - Do not include any additional fields, commentary, reasoning, or code fences. The corresponding user message supplies the question and visible evidence in the formQuestion: ...fol- lowed byVisible evidence: ...; at depth zero it inserts(No retrieved evidence is currently visible.), and at later depths it enumerates the currently shown passages by title and text. 15 Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay A.2. Fixed evidence paths For each questionq, letG(q)be the dataset-marked supporting passages andD(q)be the remaining candidate passages. In HotpotQA, support is defined by the provided supporting-fact titles; in MuSiQue, it is defined by each paragraph’sis_supportingflag. We order supporting passages by their source index in the original example. We rank distractors once per question. For a distractord ∈ D(q), we score its concatenated title and text using r(d; q) = 0.5 ̃ ℓ(d; q)+ 0.5̃s(d; q), whereℓ(d; q)is the fraction of normalized question tokens appearing ind,s(d; q)is cosine similarity between BAAI/bge-m3embeddings of the question and passage text, and tildes denote min–max normalization over that question’s distractors. Writing the source-ordered supports asG(q) = (g 0 , g 1 , . . .)and the retained ranked distractors asD m (q) = (d 0 , d 1 , . . .), the primary plan is P(q) = (g 0 , d 0 , g 1 , d 1 , . . .), omitting an element when one list is exhausted. ForP(q) = (p 0 , . . . , p L−1 ), we assign zero-based passagep i to slice S 1+(i mod 3) . A retrieval action reveals the next entire slice; hence E t = t ⋃ j=1 S j ,E 0 = ∅, fort ∈ 1, 2, 3. We compute all slices before model collection and query model at every depth on this plan. A.3. Correctness judge prompt Correctness labels are produced by the fixed judgegoogle/gemini-2.5-flash-lite. The judge receives the question, all reference answers for the example, and the candidate answer. It uses temperature zero and a strict JSON schema requiring exactly one boolean field,correct, with no additional fields. The exact system prompt is reproduced below. You judge whether a predicted answer correctly answers a question. Treat the reference answers as valid aliases. Ignore wording, formatting, and harmless extra detail when the prediction preserves the answer. Return only JSON with exactly one boolean field: correct. The corresponding user message uses the following template: Question: <question> Reference answer(s): <JSON list of reference answers> Predicted answer: <candidate answer> The stored label is the parsed value ofcorrect. We add retries to handle the cases where the judge response cannot be parsed as a JSON object containing a boolean correct field. 16 Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay A.4. Models, datasets, and splits To compare models under a common setup, we hold prompts, trajectories, controller settings, and the correctness judge fixed while fitting a separate calibration map for each model. We use the open-weight mistralai/mistral-small-2603,openai/gpt-oss-120b, andqwen/qwen3-235b-a22b-2507check- points. Provider-reported details are listed in Table 4 [25,26,31–33]. All three checkpoints are released under Apache 2.0. We use “calibration” only for records used to fit the isotonic map; the base models are not fine-tuned. Examples are partitioned deterministically by source order into calibration, and final-test splits. The calibration split fits the confidence map, while the final-test split is used for the reported replay results. For each model, isotonic regression is fit once on the pooled calibration records from HotpotQA and MuSiQue; we do not fit dataset-specific isotonic maps. Table 5 reports the unique source-example counts. We use HotpotQA under C BY-SA 4.0 and MuSiQue under C BY 4.0. 1 Distractors are ranked withBAAI/bge-m3, which is MIT-licensed; hosted models were used for inference from OpenRouter. 2 Table 4: Evaluated model details: context limits and model descriptions are the provider-reported limits for the checkpoint or hosted identifier; the experiment itself caps generation at 4096 output tokens and uses temperature zero. Model in paperProvider identifierTotal paramsActive paramsArchitecture typeTraining / alignment strategy Context limit Mistral Small 4 (2603) mistralai/ mistral-small-2603119B6.5B per tokenMultimodal sparse MoE transformer; 128 experts, 4 active per token Open-weight hybrid in- struct, reasoning, and coding model with config- urable reasoning effort 256k tokens GPT-OSS-120B openai/ gpt-oss-120b116.8B5.1B per tokenAutoregressive MoE trans- former with alternating dense and locally banded sparse attention, grouped multi-query attention, and RoPE Open-weight reasoning model; text-only pretrain- ing followed by supervised fine-tuning and high- compute reinforcement learning for instruction following, tool use, and reasoning 128k tokens Qwen3-235B -A22B-2507 qwen/ qwen3-235b- a22b-2507235B22B per tokenCausal-languageMoE transformer; 94 layers, GQA, 128 experts, 8 active experts Pretraining plus post- training for multilingual instructionfollowing, non-thinking generation, reasoning, coding, and tool use 262,144 tokens B. Additional Calibration Diagnostics Table 2 reports held-out ECE results on test states in the main paper. Figure 5 instead shows the partition used to fit the depth-zero map. Together with the operational results, these diagnostics separate the confidence–correctness relation from the operating point induced by the policy. For the adjacent retrieval-transition analysis in Table 3, the denominator 7,367 is the number of test-trace transitions for which the retrieval action reveals new evidence. The micro-pooled controlled-trajectory test set containsN = 2,457traces, each with up to three adjacent transitions (0→1,1→2, and2→3), giving 1 Official license sources:https://github.com/hotpotqa/hotpotandhttps://github.com/StonyBrookNLP/ musique. 2 https://openrouter.ai/terms. 17 Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay Table 5: Unique source-example counts used for every model under the 2:1 calibration–test split. Calibration fits the isotonic map, and test is reserved for final replay. Dataset / artifactCalibrationTest HotpotQA3,7021,852 MuSiQue answerable development1,208605 Complete-artifact total4,9102,457 2,457× 3 = 7,371possible transitions. In four depth-three cases, the third evidence slice is empty, so the depth-three state is copied from depth two rather than elicited again. We exclude these duplicate state pairs because they cannot measure the effect of an additional evidence step. The analysis therefore contains 7,371− 4 = 7,367 transitions per model. B.1. Why transfer can fail at depth three The depth-three failure is concentrated in the highest raw-confidence bin. For Mistral, 2,293 of 2,457 states fall in the0.9–1.0raw-confidence bin; their mean raw confidence is0.930, empirical accuracy is0.789, and mean calibrated confidence is0.557. The corresponding figures for GPT are 1,713 states,0.930,0.934, and0.884; for Qwen they are 2,018 states,0.988,0.871, and0.730. The depth-zero map therefore lowers confidence more than the later-state correctness relationship warrants. This illustrates why a monotone post- hoc map can improve held-out calibration at early states while becoming an overcorrection after additional evidence. Mistral Small 4 (2603)GPT-OSS-120BQwen3-235B-A22B-2507 Figure 5: Calibration-partition reliability diagrams for the three evaluated answer models. Each panel compares verbalized depth-zero confidence against empirical correctness on the records used to fit that model’s isotonic map; the calibration gap is shown by the hatched region. 18

Instant research discovery