Paper deep dive
Beyond Confidence: Test-Time Scaling for Multi-Turn Search Agents via Retrieval Grounding
Hyunho Kook, Junhyuk So, Tianyu Fu, Haizhong Zheng, Beidi Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 8/26/2026, 5:23:16 AM
Summary
The paper identifies 'copy inflation' as a failure mode in confidence-based voting for multi-turn search agents, where retrieved documents inflate token log probabilities, flattening confidence scores. It proposes Retrieval-Grounded Voting (RGV), which weights rollouts by lexical overlap between the answer and retrieved documents, significantly outperforming DeepConf across four benchmarks and five LLMs.
Entities (13)
Relation Signals (13)
Retrieval-Grounded Voting ā evaluatedon ā BrowseComp-Plus
confidence 99% Ā· Across four search-agent benchmarks... RGV consistently outperforms
Retrieval-Grounded Voting ā evaluatedon ā FRAMES
confidence 99% Ā· Across four search-agent benchmarks... RGV consistently outperforms
Retrieval-Grounded Voting ā evaluatedon ā BrowseComp
confidence 99% Ā· Across four search-agent benchmarks... RGV consistently outperforms
Retrieval-Grounded Voting ā evaluatedon ā GAIA
confidence 99% Ā· Across four search-agent benchmarks... RGV consistently outperforms
Retrieval-Grounded Voting ā outperforms ā DeepConf
confidence 98% Ā· RGV consistently outperforms confidence-based voting, with gains of up to +5.4% accuracy
Copy Inflation ā causesfailurein ā DeepConf
confidence 97% Ā· identify the underlying failure reason as copy inflation... This flattens confidence scores within each question and weakens the resulting weighted vote.
Retrieval-Grounded Voting ā evaluatedon ā Tongyi DeepResearch
confidence 96% Ā· Across four search-agent benchmarks and five LLMs, RGV consistently outperforms
Retrieval-Grounded Voting ā usesmetric ā lexical_overlap
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Confidence-based voting aggregates parallel LLM rollouts by weighting each with internal signals such as token log probabilities, and has been actively studied for single-turn reasoning. However, modern LLMs increasingly act as multi-turn search agents that retrieve and condition on external documents. In this paper, we show that confidence-based voting transfers poorly to this multi-turn setting, and identify the underlying failure reason as copy inflation: when retrieved documents are appended to an agent's context, tokens copied from those documents receive systematically inflated log probabilities. This flattens confidence scores within each question and weakens the resulting weighted vote. To address this issue, we propose Retrieval-Grounded Voting (RGV), which scores each rollout by the lexical overlap between its final answer and the documents it retrieved. By computing the signal outside the contaminated context, RGV sidesteps both token log probabilities and additional LLM calls. Across four search-agent benchmarks and five LLMs, RGV consistently outperforms confidence-based voting, with gains of up to +5.4% accuracy and +35% on minority-correct questions, where the correct answer appears in only 1-2 of 8 rollouts.
Tags
Links
- Source: https://arxiv.org/abs/2608.24024v1
- Canonical: https://arxiv.org/abs/2608.24024v1
Trouble viewing inline? Open PDF directly ā
Full Text
96,738 characters extracted from source content.
Expand or collapse full text
Beyond Confidence: Test-Time Scaling for Multi-Turn Search Agents via Retrieval Grounding Hyunho Kook Affiliation: University of Southern California Email: haizhonz@andrew.cmu.edu Junhyuk So Affiliation: Pohang University of Science and Technology (POSTECH) Email: beidic@andrew.cmu.edu Tianyu Fu Affiliation: Tsinghua University Haizhong Zheng Affiliation: Carnegie Mellon Universityhyunho.kook@usc.edu junhyukso@postech.ac.kr fuvty@outlook.com Beidi Chen Affiliation: Carnegie Mellon Universityhyunho.kook@usc.edu junhyukso@postech.ac.kr fuvty@outlook.com Abstract Confidence-based voting aggregates parallel LLM rollouts by weighting each with internal signals such as token log probabilities, and has been actively studied for single-turn reasoning. However, modern LLMs increasingly act as multi-turn search agents that retrieve and condition on external documents. In this paper, we show that confidence-based voting transfers poorly to this multi-turn setting, and identify the underlying failure reason as copy inflation: when retrieved documents are appended to an agentās context, tokens copied from those documents receive systematically inflated log probabilities. This flattens confidence scores within each question and weakens the resulting weighted vote. To address this issue, we propose Retrieval-Grounded Voting (RGV), which scores each rollout by the lexical overlap between its final answer and the documents it retrieved. By computing the signal outside the contaminated context, RGV sidesteps both token log probabilities and additional LLM calls. Across four search-agent benchmarks and five LLMs, RGV consistently outperforms confidence-based voting, with gains of up to +5.4% accuracy and +35% on minority-correct questions, where the correct answer appears in only 1-2 of 8 rollouts. Figure 1: Overview. (a) Each rollout is one trajectory of the ReAct-style search agent, looping over tool calls (search, page visit) and emitting a predicted answer; N parallel rollouts may differ. (b) The weighted vote sums per-rollout weights over predicted answers; argmax wins. (c) Confidence-based voting weights each rollout by its token logprobs, inflated when the model parrots context. RGV (ours) weights each rollout by the lexical overlap between its answer prose and the retrieved documentsāa signal read outside the contaminated context. 1 Introduction Confidence-based voting, weighting parallel rollouts by an aggregate of their token-level logprobs, has been widely adopted for test-time scaling in single-turn LLM reasoning (Fu et al., 2025; Taubenfeld et al., 2025), and can outperform majority voting on non-retrieval interactive agent tasks (Wang et al., 2024). But as LLMs are deployed as multi-turn search agents that condition on retrieved documents (Yao et al., 2023; Nakano et al., 2021; Chen et al., 2025), this logprob-based signal transfers poorly. Why confidence voting fails in this setting, and how to address it at the voting layer, remains largely understudied. Existing work addresses the problem from three perspectives, but each leaves it unresolved. Confidence-based voters (Fu et al., 2025; Taubenfeld et al., 2025; Wang et al., 2024) are designed for single-turn LLM settings and offer no diagnosis for search agents. Trajectory-aware aggregators (Lee et al., 2026; Li et al., 2025) sidestep the confidence signal by spending an extra LLM call per question (an aggregator on top of the parallel rollouts), adding inference cost. RL-based recalibration (Xuan et al., 2026) adjusts confidence via RL fine-tuning but leaves the source of miscalibration in search agents largely unexplained. An ideal treatment should first diagnose why confidence signals degrade in search agents, then leverage that diagnosis to design an efficient, broadly applicable voting-layer solution. To address this gap, we make two contributions. First, we identify and quantify a mechanism we call copy-inflation: once retrieved documents are appended to the agentās context, copied tokens receive inflated logprobs, compressing confidence scores within a question so that weighted majority voting collapses to simple majority (Ā§3). Second, the diagnosis points to a solution: read the voting signal from outside the contaminated context. We propose Retrieval-Grounded Voting (RGV): weight each rollout by the lexical overlap between its answer prose and the documents it retrieved (Ā§4). Because the retrieval log is environment-derived, the signal is independent of the modelās hidden state and requires no logprobs, no fine-tuning, and no extra LLM calls. We publicly release both our code11 1 https://github.com/kookhh0827/copy-inflation-search-agents and our data22 2 https://huggingface.co/datasets/kookhh0827/copy-inflation-search-agents: the full rollout trajectories, the per-token log-probabilities, and the judge outputs. Across four search agent benchmarks and five LLMs, RGV beats logprob-based confidence voting (DeepConf; Fu et al., 2025), by up to +5.4%+5.4\% accuracy and +35%+35\% on minority-correct questions. RGV at four rollouts already matches DeepConf at eight, halving the rollout budget at equal accuracy. The gains are robust to the choice of overlap metric and add negligible CPU cost. 2 Background Multi-turn search agent. We study ReAct-style agents (Yao et al., 2023; Schick et al., 2023) whose tools retrieve text documents and whose final answer is derived from the retrieved content. We call these multi-turn search agents, or search agents for short. Such an agent interleaves reasoning and tool calls under a budget of T turns. With context c0=qc_0=q, at turn k the policy ĻĪø _Īø samples (uk,xk)ā¼ĻĪø(ā ā£ckā1)(u_k,x_k) _Īø(Ā· c_k-1), where uku_k is the modelās thought span at turn k and xkx_k is either a tool invocation (e.g. search to query a search engine, or visit to fetch a web page) or the final answer. On a tool call the environment returns an observation oko_k and the context grows by ck=ckā1|ukāāxkāāokc_k=c_k-1\|u_k\|x_k\|o_k. Retrieved snippets are thus appended to the same context the model conditions on at later turns. Rollout. A rollout tit_i is one trajectory of this loop, terminating on a final answer or budget exhaustion. We sample N rollouts t1,ā¦,tN\t_1,ā¦,t_N\ in parallel from q. The rolloutās record contains every tool call and its response, every thought span the model emitted, and a final answer turn whose output is one last thought span, followed by the predicted answer string. Three names from this record recur below: the predicted answer a^i a_i, the answer prose PiP_i (the final answer the model produces on the answer turn, excluding its internal chain-of-thought), and the retrieval log i=d1,ā¦,dmiD_i=\d_1,ā¦,d_m_i\ (the documents the agent fetched). Weighted majority vote. The N rollouts may give different answers, so a voting rule returns a single final answer. Each rollout is scored by a non-negative weight wiw_i, and the chosen answer is the one whose rollouts carry the largest total weight: Vā”(a)=āi=1Nwiā(a^i=a),a^=argā”maxaā”Vā”(a).V(a)= _i=1^Nw_i\,I( a_i=a), a= _aV(a). Different choices of wiw_i give different methods. Simple majority (wiā”1w_i\!ā”\!1) just counts rollouts per answer. DeepConf (Fu et al., 2025), our confidence baseline, reduces per-token confidences Ct=ā1kāj=1klogpjC_t=- 1k _j=1^k p_j (top-k averaged log-prob, negated; higher means more peaked) over the tokens the model produces during the rollout to a single wiw_i via a sliding-window aggregation. We use the Lowest-Group reduction (sliding-window minimum) at window size W=1024W=1024 as the headline; the full sweep over reductions and window sizes is in Appendix F. Our method (Ā§4) replaces wiw_i with a direct grounding check between the answer prose PiP_i and the retrieved docs iD_i. 3 Motivation When the only input is the question, DeepConf (Fu et al., 2025), a logprob-based confidence voter (Ā§2) is well-motivated: peaked next-token posteriors correlate with correctness (Kadavath et al., 2022; Tian et al., 2023). In a search agent, however, that correlation breaks down because the tokens DeepConf reads are systematically inflated by the retrieved docs appended to the context. We document this on BrowseComp-Plus rollouts from Tongyi-DeepResearch in three observations (Fig. ) and trace the consequence at the rollout level (Fig. ). Tool outputs in context inflate logprobs. A search agent appends every retrieved snippet and visited page to its context window; when it later writes prose, any token it copies from those appended documents has very high pā”(tkā£context)p(t_k ) because the document is right there. The tokenās logprob says the copy step is well-calibrated; it does not say the copied token is the right answer. Confidence-based voting cannot tell the difference between a rollout that copied the right entity and one that copied an irrelevant one. (i) Copy tokens carry inflated confidence (Fig. ). Recall (Ā§2) that DeepConf reads the tokens the model produces during the rollout. For each such token we check whether its surface form is a substring of any retrieved doc the agent fetched on this rollout; copy tokens have a mean logprob +0.50+0.50 nats higher than non-copy tokens (730k tokens from a subset of rollouts where per-token logprobs were re-collected). The signal DeepConf reads is thus contaminated by the appended tool outputs regardless of what those tools returned. The gap is robust to the choice of copy-token definition (Appendix G). (i) At the rollout level, copy-inflation flattens DeepConf scores within a question (Fig. ). We measure DeepConfās within-question share of total score variance, i.e. how much of the scoreās spread comes from differences among rollouts of the same question, rather than between questions. Weighted majority voting needs this within-question spread to prefer one rollout over another; if it vanishes, the vote degenerates into plain simple-majority. As copy fraction grows, DeepConfās within-question share shrinks 0.39ā0.200.39ā 0.20: same-question rollouts get nearly the same DeepConf score and the weight tiebreaker disappears, precisely on the copy-heavy questions where rollouts disagree most. (i) And this is not an edge case; copy ratios are very high. Across our rollouts, the median copies 93%93\% of its content tokens from the docs it retrieved, 81%81\% copy at least 90%90\%, and 85%85\% of questions have a mean copy fraction ā„0.90ā„0.90 (Fig. , x-axis). The two failure modes above therefore apply to nearly the entire benchmark, not a corner case. While copy-inflation can in principle arise whenever tool outputs enter the agentās context; multi-turn search agents are the dominant regime. Two interventions confirm the direction of causation rather than mere association: masking the copied tokens restores DeepConfās within-question spread without restoring its discriminative power, and removing the documents from context collapses copied-token log-probabilities about twice as much as non-copy ones (Appendix I). Consequence: the weighted vote no longer tracks correctness. Figure makes the rollout-level consequence concrete. (a) Stratifying questions by oracle difficulty (the fraction of their N rollouts that turn out correct), DeepConfās mean per-question score remains at 87%87\% of its peak even on questions where every rollout is wrong: DeepConf cannot tell an impossible question from a solvable one, so the cluster-weighted vote inherits this miscalibration at the voting layer; The pattern replicates across every benchmark and model we test (Appendix D). (b) The same pattern holds at the rollout level: DeepConfās per-rollout score distributions for correct vs. wrong rollouts overlap heavily (overlap =54.2%=54.2\%), as copy-inflation lifts wrong rollouts into the same high-confidence zone as correct ones, leaving the confidence weight nothing to discriminate. The principle. The shared cause of failure modes is that the voting signal is computed inside the contaminated context, not outside it. Robust voting therefore requires a signal whose measurement target is independent of the modelās hidden state; something derived from outside the model. The next section turns this principle into a method. 4 From principle to method: Retrieval-Grounded Voting To address the copy-inflation problem identified in Ā§3, we propose reading the voting signal from outside the contaminated context, specifically, from the retrieval log. The key intuition, shared with faithfulness evaluation in summarisation and RAG (Maynez et al., 2020; Min et al., 2023; Es et al., 2024), is that a rollout is more trustworthy when its final answer is lexically anchored in the documents it retrieved. The RGV score. For each rollout i on a given question, we have its answer prose PiP_i (defined in Ā§2: the answer turnās output text) and the set i=d1,ā¦,dmiD_i=\d_1,ā¦,d_m_i\ of documents the agent retrieved during that rollout (one djd_j per search or visit call). We map a text span X to a token set ā”(X)T(X) using a fixed tokenisation/normalisation rule (Appendix A). The rolloutās RGV weight is the maximum prose-recall, the fraction of the answer-prose tokens that are anchored in some retrieved document, wiRGV=maxdāiā”|ā”(Pi)ā©ā”(d)||ā”(Pi)|.w_i^RGV\;=\; _d _i |T(P_i) (d)||T(P_i)|. This is the standard grounding primitive of the faithfulness literature (Ā§7), instantiated with two design choices, max-over-docs and answer-side normalisation, which we motivate next. Appendix E shows that these choices matter only at the margin, while the signal source carries the bulk of the gain. Why max-over-docs. We score against the single best-matching document rather than the union of all retrieved documents to avoid dilution: as the retrieval bag grows, an irrelevant-doc union washes out a strong match to the truly supporting passage. This ātake an extreme, not an averageā design mirrors local-extreme confidence reductions in DeepConf (Fu et al., 2025), which prefer the most-uncertain window-group over an average across all windows. Why the prose-side denominator. With the numerator fixed at |ā”(Pi)ā©ā”(d)||T(P_i) (d)|, the design choice is the denominator. Symmetric Jaccard (Jaccard, 1912) uses |ā”(Pi)āŖā”(d)||T(P_i) (d)|, which over-penalises when the retrieved doc is long, exactly the regime we target. Normalising by |ā”(Pi)||T(P_i)| keeps the score length-invariant on the doc side: a well-anchored answer is not punished for retrieving a long supporting passage. Other overlap functions (Lin, 2004; Robertson and Zaragoza, 2009) yield similar headline numbers (Appendix E); the gain comes from the signal source, not the choice of denominator. Voting rule. We plug wi=wiRGVw_i\!=\!w_i^RGV into the weighted majority vote of Ā§2; answer strings are clustered after a light normalisation (lowercase, whitespace collapse, leading-article strip) and the cluster with the largest summed weight wins. What is and is not computed inside the modelās context. The token set ā”(Pi)T(P_i) is, of course, generated by the model. The token set ā”(d)T(d) is read directly off the retrieval log: it is what the environment returned, regardless of how the model used it. Even though the docs were appended to the modelās context, their token set as a measurement target does not depend on the modelās hidden state, and a rollout cannot inflate its RGV weight through verbosity or self-assuredness; RGV asks only whether the answer prose is anchored in the documents the rollout actually retrieved. 5 Experimental setup Benchmarks. We evaluate on four multi-turn search agent benchmarks where the agent gathers evidence from retrieved documents and derives a grounded answer: BrowseComp-Plus (Chen et al., 2025) (fixed-corpus retrieval), BrowseComp (Wei et al., 2025) (open-web browsing), GAIA (Mialon et al., 2023) (general-assistant tasks dominated by web search), and FRAMES (Krishna et al., 2025) (multi-hop retrieval requiring synthesis across multiple Wikipedia articles). For each benchmark, we randomly sample up to 150150 questions, held constant across models. Licences and intended-use terms for every benchmark, corpus and model are listed in Appendix O. Models. We evaluate five LLMs across the four benchmarks above: gpt-oss-120b (OpenAI et al., 2025), MiniMax-M2.7 (MiniMax et al., 2025), GLM-5.1 (GLM-5-Team et al., 2026), Kimi-K2.5 (Team et al., 2025a), and Tongyi-DeepResearch (Team et al., 2025b). The resulting benchmark Ć model coverage is reported in Table 1. Rollout protocol. The agent is given two tools per benchmark: search (issue a text query, receive ranked snippets) and get_doc (fetch a full document by ID) on BrowseComp-Plus (fixed-corpus retrieval), and search and visit (fetch and render a URL) on the other three benchmarks. We sample N=8N=8 rollouts per question. Full hyperparameters, tool truncation rules, and the force-final / recovery procedure are in Appendix A. Judge. A single judge is used across all datasets: Qwen3-32B at temperature 00, applied with the official BrowseComp-Plus grading template (Chen et al., 2025). The judge sees only the question, the gold answer, and the rolloutās final answer; it does not see voting outcomes or other rollouts. Voting protocol. All voting methods plug a per-rollout weight wiw_i into the weighted majority vote of Ā§2; answer strings are clustered under the strict normalisation of Ā§4. Methods compared: Simple-majority (SM, wiā”1w_i\!ā”\!1); DeepConf (DC; Lowest-Group reduction with window W=1024W=1024, chosen on the full reduction Ć window-size grid in Appendix F); RGV (Ā§4). Two reference columns in Table 1 are non-voting: single = mean per-rollout accuracy, and oracle = any-rollout-correct upper bound. 6 Results Dataset Model Single Avg. Simple Maj. DeepConf RGV (Ours) Oracle BrowseComp-Plus Tongyi-DeepResearch 51.6 62.0 65.7 71.1 74.2 OSS-120B 47.3 53.3 54.7 56.0 66.7 MiniMax-M2.7 67.8 75.3 75.3 78.7 83.3 Kimi-K2.5 80.0 80.7 80.7 81.3 88.7 GLM-5.1 74.9 80.7 80.7 82.0 90.0 GAIA Tongyi-DeepResearch 70.0 79.6 79.6 80.6 92.2 OSS-120B 60.4 69.9 68.9 71.8 83.5 MiniMax-M2.7 80.3 87.4 88.3 93.2 97.1 Kimi-K2.5 73.1 78.6 79.6 81.6 92.2 GLM-5.1 83.3 87.4 87.4 89.3 92.2 BrowseComp Tongyi-DeepResearch 41.2 52.0 52.7 54.0 68.7 OSS-120B 22.7 30.0 29.3 34.0 41.3 MiniMax-M2.7 37.1 40.7 46.0 50.7 59.3 Kimi-K2.5 35.0 52.8 56.6 58.5 73.6 GLM-5.1 57.8 72.6 73.7 78.9 92.6 FRAMES Tongyi-DeepResearch 84.0 87.3 88.0 88.0 94.7 OSS-120B 78.7 82.0 82.7 85.3 91.3 MiniMax-M2.7 86.5 88.7 89.3 89.3 94.0 Kimi-K2.5 85.7 88.0 88.7 89.3 97.3 GLM-5.1 88.3 90.0 90.0 90.7 98.0 Table 1: Headline voting accuracies (%) across 4 multi-turn search agent benchmarks Ć 5 LLMs, N=8N=8 rollouts per question. Table 1 shows that RGV is best in all 20 benchmarkĆmodel cells. The pattern is consistent with the copy-inflation story of Ā§3: whenever confidence weights lose within-question spread, grounding is the remaining discriminator. Gains are largest where voting is hardest: low single-rollout accuracy (e.g. +5.4%+5.4\% on BrowseComp-Plus Tongyi at 51.6%51.6\% single) and noisy open-web retrieval (BrowseComp, up to +6.2%+6.2\%). They shrink where majority already saturates (FRAMES; ā„84%ā„ 84\% single), but remain nonzero even when DeepConf matches simple majority (e.g. GLM-5.1 on FRAMES: 90.0%90.0\% vs. 90.7%90.7\%), suggesting that lexical grounding adds information beyond both count (simple-majority votes) and logprob confidence. All margins are stable under within-cell 3-fold splits (Appendix B). We probe the boundaries of this result in three appendices: degraded retrieval, prompt and answer-format sensitivity, and selection versus voting, in Appendices H, J and K. For the mechanism and ablation analyses that follow (Ā§6.1 ā Ā§6.3), we anchor on Tongyi-DeepResearch on BrowseComp-Plus (N=8N=8, 830 questions), where retrieval comes from a fixed corpus (no search-engine noise) and the agent is representative of the class we target. 6.1 Is the signal genuine grounding? Ā§4 interprets overlap as grounding, but a spurious alternative is that it rewards verbosity or common tokens. If so, RGV would succeed by accident, and overlap could not support the copy-inflation account of Ā§3. We therefore test whether the score reflects genuine retrieval grounding. At the rollout level, RGV separates judge-correct from judge-wrong trajectories better than DeepConf (ROC AUC (Fawcett, 2006) 0.9080.908 vs. 0.8370.837; +0.071+0.071). To explain why, we analyse (i) which token classes carry the gap, (i) controlled shuffles and nulls, and (i) correlation with an independent retrieval-quality measure. (a) rare-entity matching (b) null controls Figure 2: Why RGV works (BrowseComp-Plus Tongyi-DeepResearch). y-axis: CāW mean gap (mean RGV score of correct rollouts minus that of wrong rollouts). (2(a)) Rare-entity tokens carry the signal. (2(b)) The signal survives a within-Q doc shuffle, collapses 3Ć3Ć across questions, and 53Ć53Ć under a random-token null. Rare-entity matching drives the signal. Figure 2(2(a)) decomposes the CāW mean gap, the difference in mean RGV score between judge-correct (C) and judge-wrong (W) rollouts, by token class. Digit tokens (years, codes, identifiers) give the largest gap (0.1450.145), capitalised entity names follow (0.0920.092), and common content words contribute little (0.0100.010). The pattern is intuitive: common words appear in every rolloutās prose regardless of correctness, so they carry no discriminative power; rare entities appear only when the rollout actually retrieved the relevant document. This aligns with the rare-token weighting in classical IR (Robertson and Zaragoza, 2009) and atomic-faithfulness metrics (Min et al., 2023). Null controls confirm question-specific grounding. To distinguish genuine grounding from incidental overlap, we construct three counterfactuals (Figure 2(2(b))). The CāW mean gap is 0.0800.080. (i) A within-question doc shuffle (re-pair each rolloutās prose with another rolloutās docs from the same question) preserves the gap at 0.0800.080, since same-question rollouts share relevant retrievals. (i) An across-question shuffle collapses the gap to 0.0250.025 (3Ć3Ć drop). (i) A length-matched random-token null collapses it to 0.0020.002 (53Ć53Ć). Together, they rule out the two most plausible confounds: (i) shows the signal is not generic text similarity but depends on the question-document match; (i) shows it is not an artifact of answer length. Gold-document validation. BrowseComp-Plus provides gold supporting documents per question (mean ||=2.9|G|=2.9). For each rollout we compute gold recall, |iā©|/|||D_i |/|G|. Judge-correct rollouts have mean gold recall 0.940.94, vs. 0.310.31 for judge-wrong (Pearson r=0.72r=0.72). RGV scores correlate with gold recall at r=0.57r=0.57, vs. 0.360.36 for DeepConf. In other words, a high RGV score is a reliable indicator that the rollout found the right documents, not merely that it produced verbose prose. This closes the validation loop: RGV works because it measures retrieval quality, the very property that copy-inflation prevents DeepConf from reading. 6.2 Where the advantage concentrates Figure 3: Voting accuracy by question difficulty (# correct of N=8N=8) on BrowseComp-Plus -DeepResearch. Most of the gap concentrates on the 1ā2/81-2/8 bucket. The copy-inflation analysis of Ā§3 makes a natural prediction: because copy-inflation compresses DeepConfās within-question spread on exactly the questions where rollouts disagree, RGVās advantage should concentrate on those same hard-to-vote questions. We test this prediction directly. Figure 3 stratifies the 830 questions by oracle difficulty (the number of the eight rollouts that the judge marks correct). On easy questions (ā„3/8ā„ 3/8 correct) all methods converge. Most of the accuracy gap between RGV and DeepConf concentrates on the 122122 minority-correct (1ā2/81-2/8) questions, where one or two of eight rollouts found the right answer but the majority did not: RGV scores 84.4%84.4\% vs. DeepConfās 49.2%49.2\% (+35.2%+35.2\%). DeepConf fails here because copy-inflation (Ā§3) gives the wrong majority nearly the same confidence as the correct minority; RGV breaks the tie through grounding. This is also the bucket where the gold-doc recall gap of Ā§6.1 is largest (RGV-argmax 84%84\% vs. DeepConf 71%71\%, +13%+13\%), confirming that the voting advantage traces back to retrieval quality. RGVās value is concentrated precisely where it is needed: the ambiguous questions where rollouts disagree and the majority answer may be wrong. 6.3 Scaling, cost, and design choices Figure 4: Accuracy vs. rollout budget N on BrowseComp-Plus with Tongyi-DeepResearch. RGV is uniformly above DeepConf. Two practical questions remain: does RGVās advantage persist under tighter rollout budgets, and is the max-over-docs design of Ā§4 the right choice? Scaling with N. Sub-sampling the eight rollouts to Nā1,ā¦,8Nā\1,ā¦,8\ at 6060 random partitions per N (Figure 4), RGV at N=4N=4 (65.9%65.9\%) already matches DeepConf at N=8N=8 (65.7%65.7\%): the same accuracy at half the rollout cost. The gap grows monotonically from +3.4%+3.4\% at N=2N=2 to +5.4%+5.4\% at N=8N=8, meaning RGV benefits more, not less, from additional rollouts. This is the regime where test-time scaling (Snell et al., 2024; Brown et al., 2024) is most relevant, and where a better voting rule yields compounding returns. Cost. RGV uses only the emitted prose and the retrieved snippets already in the trajectory: no extra call, no logprobs, ā¼ 0.3 ms per rollout on one CPU thread. By contrast, DeepConf requires per-token logprobs, and aggregator methods (Lee et al., 2026; Chen et al., 2023) add at least one rollout-equivalent of inference. When combined with the iso-accuracy scaling above, RGV offers a favourable costāaccuracy trade-off. Max-over-docs ablation. Reduction Accuracy (%) Simple Majority (no RGV) 62.0 mind _d 69.3 meandmean_d 70.5 maxdāmind _d- _d 70.5 maxd _d (headline RGV) 71.1 Table 2: Doc-set reduction ablation (N=8N=8, BrowseComp-Plus, Tongyi-DeepResearch). Ā§4 argues for max over the retrieved-document set to avoid dilution from irrelevant documents. Table 2 validates this: max consistently leads, though all four reductions beat simple majority by +7+7 to +9%+9\%. The fact that even min (the worst-matching document) still yields a +7.3%+7.3\% gain over simple majority is revealing: the bulk of RGVās advantage comes from the signal source itself, i.e. reading prose-vs-retrieval overlap outside the contaminated context. The choice of reduction is secondary; max adds a further +0.6+0.6 to +1.8%+1.8\% by focusing on the single most relevant document and avoiding dilution from irrelevant retrievals. 6.4 Error analysis RGV selects the correct answer on 590590 of 616616 questions where a correct rollout exists (95.8%95.8\%). The 2626 failures (4.2%4.2\%) follow a well-characterised pattern: in 2424 cases a wrong rollout retrieves documents about the right topic but draws the wrong conclusion, receiving a higher grounding score because it echoes topically relevantābut factually incorrectāevidence. We call this well-grounded but wrong. This residual is small relative to the +5.4%+5.4\% overall gain; it concentrates on high-diversity questions (mean 4.94.9 unique clusters vs. 4.04.0) where many plausible answers compete. Appendix N contrasts three representative successes with three failure cases. 7 Related work Confidence-based aggregation for single-turn reasoning. Sample-and-aggregate test-time compute (Wang et al., 2022b; Snell et al., 2024; Brown et al., 2024; Wang et al., 2025) typically weights the vote by a confidence proxy: self-reported confidence (Wang et al., 2024; Taubenfeld et al., 2025), sliding-window token logprobs (Fu et al., 2025), an extra LLM judge (Chen et al., 2023; Thirukovalluru et al., 2024), or saturation-based budget (Aggarwal et al., 2023). These assume the question is the only conditioning, with no tool outputs in context. The breakdown in multi-turn tool-using agents. Within the broader class of tool-using agents, we focus on search agents, the subclass where retrieved documents dominate the context. Recent work reports that confidence transfers poorly to this setting, but examines a different signal class. Xuan et al. (2026) document a confidence dichotomyāevidence tools induce overconfidence in verbalised confidence and recalibrate it via RL fine-tuning; BrowseConf (Ou et al., 2025) uses verbalised confidence as a retry trigger. Both target the verbalised channel rather than the token-logprob signal used by logit-based voters, and neither operates at the voting layer. Wang et al. (2024) soften majority voting with token logprobs on interactive tasks without retrieval. Broader miscalibration accounts (Kadavath et al., 2022; Tian et al., 2023; Xiong et al., 2024; Kuhn et al., 2023) share our framing; our contribution is to address the mechanism at the voting layer without retraining or a second model. Trajectory-aware aggregators. A complementary thread adds an extra model: AggAgent (Lee et al., 2026) aggregates parallel trajectories; ParallelMuse (Li et al., 2025) compresses partial ones for an aggregator; PRM-style verifiers (Cobbe et al., 2021; Lightman et al., 2023) and self-correction agents (Shinn et al., 2023; Zhou et al., 2024) score intermediate steps. These help but add at least one rollout-equivalent of inference; RGV adds none. Faithfulness and grounding metrics. RGVās lexical-overlap signal is a voting-layer use of a familiar intuition from summarisation and RAG evaluation (Maynez et al., 2020; Lin, 2004; Min et al., 2023; Es et al., 2024; Gao et al., 2024). The primitive itself is long-standing: Grusky et al. (2018) measure extractive coverage as the fraction of summary tokens drawn from the source article, the same answer-side normalisation we adopt in Ā§4, and Shuster et al. (2021) use unigram overlap between a response and its grounding knowledge as a hallucination proxy, with a Rare-F1 variant that discounts common words for exactly the reason Figure 2(2(a)) finds them uninformative. Those metrics are evaluated post-hoc against a single output. Our contribution is the layer at which the primitive is applied, weighted voting over parallel rollouts, and the reason it has to be read outside the model at all: copy-inflation (Ā§3) is what breaks the internal alternative. 8 Conclusion We identified copy-inflation, the mechanism by which retrieved documents inflate logprob-based confidence in search agents, and proposed RGV, which reads the voting signal from outside the contaminated context. Across four benchmarks and five LLMs, RGV outperforms both simple majority and confidence-based voting while adding minimal cost. Our findings suggest that when retrieved evidence contaminates the model context, aggregation signals drawn from environment-derived records offer a more reliable foundation than internal model signals. The diagnosis is not specific to voting: any mechanism that consumes an agentās token-level confidence, whether for early stopping, routing, abstention, confidence-shaped rewards, or logprob-based hallucination detection, reads the same contaminated signal and inherits the same failure. We believe this direction may extend beyond voting to other settings where context and evidence are no longer separable. Limitations RGV inherits retrieval quality. The score is read from the retrieval log, so it can be no better than what retrieval returned. Appendix H maps this boundary with four cells. Where a weaker retriever genuinely degrades the agent (BM25 in place of the dense retriever), the grounded vote keeps its margin; where the weaker stack still suffices for the benchmark, all aggregation rules converge and RGV neither helps nor hurts; under outright corpus mismatch it is the only rule that stays above the single-rollout average, though its own margin is thin. Degradation is graceful, and the copy fraction of a cell predicts which regime applies. But on a task where retrieval contributes little, so does RGV. Grounded is not correct, and short answers weaken the signal. RGV estimates retrieval success, not truth. Section 6.4 measures the residual: on 2626 of 616616 questions (4.2%4.2\%) a wrong rollout retrieves documents on the right topic and outscores a correct one. The signal is surface-level overlap with no entailment check, so this is intrinsic to the design rather than a tuning artefact. Relatedly, because the score normalises by the answer prose, it weakens as that prose becomes very short: forcing single-entity answers reduces it to answer containment and the vote merely matches DeepConf (Appendix J). Scope. We study multi-turn search agents, where retrieved documents are the primary evidence source; agents relying on non-retrieval tools such as code interpreters would need a different grounding signal, and on reasoning-intensive tasks the gains shrink (Appendix L). All experiments are in English. Finally, RGV trusts the retrieval log, so an adversarially poisoned corpus would raise the score of rollouts that copy from it. This vulnerability is shared by any method that reads that log. Accidental noise is not a problem (injecting up to 1616 irrelevant documents per rollout leaves the vote unchanged), but adversarial robustness remains open. Acknowledgments This work was partially supported by Google Research Award, Google ML & System Junior Faculty Award, Amazon Research Award, Fireworks AI, Intel, Li Auto, Moffett AI, and CMU CyLab Seed funding. This material is also based upon work supported by the National Science Foundation under Grant No. 2504353. Any opinions, findings, and conclusions or recommendations expressed are those of the authors and do not necessarily reflect the views of the National Science Foundation. This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via 560000C260017. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (RS-2022-00143911, AI Excellence Global Innovative Leader Education Program) References Aggarwal et al. (2023) Pranjal Aggarwal, Aman Madaan, Yiming Yang, and Mausam. 2023. Letās sample step by step: Adaptive-consistency for efficient reasoning and coding with llms. Preprint, arXiv:2305.11860. Brown et al. (2024) Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher RĆ©, and Azalia Mirhoseini. 2024. Large language monkeys: Scaling inference compute with repeated sampling. Preprint, arXiv:2407.21787. Chen et al. (2023) Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2023. Universal self-consistency for large language model generation. Preprint, arXiv:2311.17311. Chen et al. (2025) Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Sahel Sharifymoghaddam, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, and Jimmy Lin. 2025. Browsecomp-plus: A more fair and transparent evaluation benchmark of deep-research agent. Preprint, arXiv:2508.06600. Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168. Es et al. (2024) Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150ā158, St. Julians, Malta. Association for Computational Linguistics. Fawcett (2006) Tom Fawcett. 2006. An introduction to roc analysis. Pattern Recognition Letters, 27(8):861ā874. ROC Analysis in Pattern Recognition. Fu et al. (2025) Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao. 2025. Deep think with confidence. Preprint, arXiv:2508.15260. Gao et al. (2024) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-augmented generation for large language models: A survey. Preprint, arXiv:2312.10997. GLM-5-Team et al. (2026) GLM-5-Team, :, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzheng Pan, Hao Zeng, Haoke Zhang, Haoran Wang, and 168 others. 2026. Glm-5: from vibe coding to agentic engineering. Preprint, arXiv:2602.15763. Grusky et al. (2018) Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 708ā719, New Orleans, Louisiana. Association for Computational Linguistics. Jaccard (1912) Paul Jaccard. 1912. The distribution of the flora in the alpine zone. The New Phytologist, 11(2):37ā50. Jin et al. (2025) Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516. Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022. Language models (mostly) know what they know. Preprint, arXiv:2207.05221. Krishna et al. (2025) Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. 2025. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4745ā4759. Kuhn et al. (2023) Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664. Lee et al. (2026) Yoonsang Lee, Howard Yen, Xi Ye, and Danqi Chen. 2026. Agentic aggregation for parallel scaling of long-horizon agentic tasks. Preprint, arXiv:2604.11753. Li et al. (2025) Baixuan Li, Dingchu Zhang, Jialong Wu, Wenbiao Yin, Zhengwei Tao, Yida Zhao, Liwen Zhang, Haiyang Shen, Runnan Fang, Pengjun Xie, Jingren Zhou, and Yong Jiang. 2025. Parallelmuse: Agentic parallel thinking for deep information seeking. Preprint, arXiv:2510.24698. Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Letās verify step by step. Preprint, arXiv:2305.20050. Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74ā81, Barcelona, Spain. Association for Computational Linguistics. Luan et al. (2021) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021. Sparse, dense, and attentional representations for text retrieval. Transactions of the Association for Computational Linguistics, 9:329ā345. Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004ā9017, Singapore. Association for Computational Linguistics. Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 1906ā1919. Mialon et al. (2023) GrĆ©goire Mialon, ClĆ©mentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. Preprint, arXiv:2311.12983. Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. Preprint, arXiv:2305.14251. MiniMax et al. (2025) MiniMax, :, Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, Chengjun Xiao, Chengyu Du, Chi Zhang, Chu Qiao, Chunhao Zhang, Chunhui Du, Congchao Guo, and 109 others. 2025. Minimax-m1: Scaling test-time compute efficiently with lightning attention. Preprint, arXiv:2506.13585. Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, and 1 others. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. OpenAI et al. (2025) OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others. 2025. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925. Ou et al. (2025) Litu Ou, Kuan Li, Huifeng Yin, Liwen Zhang, Zhongwang Zhang, Xixi Wu, Rui Ye, Zile Qiao, Pengjun Xie, Jingren Zhou, and Yong Jiang. 2025. Browseconf: Confidence-guided test-time scaling for web agents. Preprint, arXiv:2510.23458. Phan et al. (2025) Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, and 1 others. 2025. Humanityās last exam. arXiv preprint arXiv:2501.14249. Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333ā389. Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto DessĆ, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: language models can teach themselves to use tools. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ā23, Red Hook, NY, USA. Curran Associates Inc. Sciavolino et al. (2021) Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021. Simple entity-centric questions challenge dense retrievers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6138ā6148, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Shinn et al. (2023) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36:8634ā8652. Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784ā3803, Punta Cana, Dominican Republic. Association for Computational Linguistics. Snell et al. (2024) Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. Preprint, arXiv:2408.03314. Taubenfeld et al. (2025) Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. 2025. Confidence improves self-consistency in llms. In Findings of the Association for Computational Linguistics: ACL 2025, pages 20090ā20111. Team et al. (2025a) Kimi Team, Yifan Bai, Yiping Bao, Y. Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, and 181 others. 2025a. Kimi k2: Open agentic intelligence. Preprint, arXiv:2507.20534. Team et al. (2025b) Tongyi DeepResearch Team, Baixuan Li, Bo Zhang, Dingchu Zhang, Fei Huang, Guangyu Li, Guoxin Chen, Huifeng Yin, Jialong Wu, Jingren Zhou, Kuan Li, Liangcai Su, Litu Ou, Liwen Zhang, Pengjun Xie, Rui Ye, Wenbiao Yin, Xinmiao Yu, Xinyu Wang, and 38 others. 2025b. Tongyi deepresearch technical report. Preprint, arXiv:2510.24701. Thirukovalluru et al. (2024) Raghuveer Thirukovalluru, Yukun Huang, and Bhuwan Dhingra. 2024. Atomic self-consistency for better long form generations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12681ā12694. Tian et al. (2023) Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5433ā5442. Wang et al. (2024) Han Wang, Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. 2024. Soft self-consistency improves language models agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 287ā301. Wang et al. (2025) Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Y Zou. 2025. Mixture-of-agents enhances large language model capabilities. In International Conference on Learning Representations, volume 2025, pages 33944ā33963. Wang et al. (2022a) Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022a. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533. Wang et al. (2022b) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022b. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171. Wei et al. (2025) Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. 2025. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516. Xiong et al. (2024) Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In International Conference on Learning Representations, volume 2024, pages 23650ā23678. Xuan et al. (2026) Weihao Xuan, Qingcheng Zeng, Heli Qi, Yunze Xiao, Junjue Wang, and Naoto Yokoya. 2026. The confidence dichotomy: Analyzing and mitigating miscalibration in tool-use agents. Preprint, arXiv:2601.07264. Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369ā2380, Brussels, Belgium. Association for Computational Linguistics. Yao et al. (2023) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. Preprint, arXiv:2210.03629. Zhou et al. (2024) Pei Zhou, Jay Pujara, Xiang Ren, Xinyun Chen, Heng-Tze Cheng, Quoc V Le, Denny Zhou, Swaroop Mishra, Huaixiu S Zheng, and 1 others. 2024. Self-discover: Large language models self-compose reasoning structures. Advances in Neural Information Processing Systems, 37:126032ā126058. Appendix A Implementation details Infrastructure. Tongyi-DeepResearch rollouts are generated locally on 8Ć8Ć NVIDIA RTX PRO 6000 GPUs; all other models are served via Fireworks AI.33 3 https://fireworks.ai Web search is powered by the Serper API44 4 https://serper.dev and page visits use Crawl4AI.55 5 https://github.com/unclecode/crawl4ai Token sets for RGV. Given a text span X, we form the token set ā”(X)T(X) by (i) lowercasing and Unicode-NFKC-normalising the string; (i) stripping markdown markup, URLs, code fences, and JSON delimiters; (i) whitespace-splitting; (iv) removing a small English stopword list of ā¼130 130 function words; and (v) dropping any token whose normalised length is below two characters or whose normalised form is purely punctuation. No lemmatisation or stemming is used. Each document djd_j is taken to be the raw text returned by the tool, not the modelās quoted snippets, so ā”(d)T(d) reflects what the corpus served and is independent of how the model rendered it. Computing the max-over-docs. For each rollout we cache ā”(Pi)T(P_i) once and iterate over d1,ā¦,dmi\d_1,ā¦,d_m_i\, computing |ā”(Pi)ā©ā”(d)|/|ā”(Pi)||T(P_i) (d)|/|T(P_i)| via a single set intersection per document. End-to-end runtime is Oā”(|Pi|+āj|dj|)O(|P_i|+ _j|d_j|) and finishes in ā¼0.3 0.3 ms per rollout on one CPU thread for a typical BrowseComp-Plus rollout. Strict answer clustering. Predicted answer strings are normalised by lowercasing, collapsing whitespace, stripping a leading article (the/a/an) and surrounding punctuation, and removing matching outer quotes. Two rollouts join the same cluster iff their normalised strings are byte-equal. DeepConf computation. Per-token confidence is Ct=ā1kāj=1klogpjC_t=- 1k _j=1^k p_j, the negated mean log-prob over the top-k candidates at step t. We compute CtC_t over all of the rolloutās generated tokens (reasoning and answer). The headline DeepConf score is the Lowest-Group reduction with window size W=1024W=1024: slide a window of 10241024 tokens with stride 11 across the generated sequence, compute the mean CtC_t inside each window, and take the minimum. Other reductions (Bottom-10%10\%, Tail) and window sizes Wā1024,2048,4096W\!ā\!\1024,2048,4096\ are reported in Appendix F. Force-final and recovery. Inside the rollout loop, before each LLM call, we check whether the prompt token count exceeds 85%85\% of the modelās context window. If so, the last assistant content is replaced with a truncation marker and a single follow-up call is issued with tools disabled and a markdown final-answer instruction (verbatim in Appendix M). The same force-final is triggered when the agent exhausts max_iterations (200200), the cumulative output budget (max_total_output_tokens, 3030K). No post-hoc compressed recovery is performed: if the single force-final call itself fails, the rollout is recorded with status api_error and an empty final_text, and is treated as parse_error by the judge. Per-model sampling parameters. The same protocol is used across all models. Parameters not listed Table 3 take standard defaults. The judge is Qwen3-32B at temperature 0.00.0. Parameter gpt-oss-120b MiniMax-M2.7 GLM-5.1 Kimi-K2.5 Tongyi-DeepResearch rollout temperature 0.85 0.85 0.85 0.85 0.85 top_p 1.0 1.0 1.0 1.0 1.0 presence penalty 0.0 0.0 0.0 0.0 1.1 max output tokens / turn 8,192 8,192 8,192 8,192 8,192 max total output budget 30,000 30,000 30,000 30,000 30,000 max tool calls 200 200 200 200 200 context window 131,072 196,607 202,752 256,000 131,072 Table 3: Per-model sampling parameters. All models use the same system prompt (Appendix M), confidence solicitation protocol, and Qwen3-32B judge at temperature 0.00.0. Appendix B Variance estimation via 3-fold split The headline Table 1 reports a single mean accuracy per cell over the 150150-question sample. To attach uncertainty without changing the protocol, we split each datasetās 150150 questions into three disjoint folds of 5050 (strata by the random seed already used for sampling: folds use the first 5050, next 5050, and last 5050 qids in their sampling order), recompute each voting methodās accuracy on each fold, and report the fold-mean Ā± fold-standard-deviation in Table 4 below. Folds are fixed across methods so that paired-fold differences are well-defined. Dataset Model Simple Maj. DeepConf RGV Oracle BrowseComp-Plus Tongyi-DeepResearch 62.0Ā±1.162.0Ā±1.1 65.7Ā±1.065.7Ā±1.0 71.1Ā±1.971.1Ā±1.9 74.2Ā±0.574.2Ā±0.5 OSS-120B 53.3Ā±4.753.3Ā±4.7 54.7Ā±4.154.7Ā±4.1 56.0Ā±2.856.0Ā±2.8 66.7Ā±4.166.7Ā±4.1 MiniMax-M2.7 75.3Ā±6.875.3Ā±6.8 75.3Ā±7.475.3Ā±7.4 78.7Ā±8.278.7Ā±8.2 83.3Ā±1.983.3Ā±1.9 Kimi-K2.5 80.7Ā±5.080.7Ā±5.0 80.7Ā±4.180.7Ā±4.1 81.3Ā±5.781.3Ā±5.7 88.7Ā±0.988.7Ā±0.9 GLM-5.1 80.7Ā±4.580.7Ā±4.5 80.7Ā±5.280.7Ā±5.2 82.0Ā±4.882.0Ā±4.8 90.0Ā±1.690.0Ā±1.6 GAIA Tongyi-DeepResearch 79.6Ā±5.879.6Ā±5.8 79.6Ā±5.879.6Ā±5.8 80.6Ā±3.980.6Ā±3.9 92.2Ā±4.292.2Ā±4.2 OSS-120B 69.9Ā±3.169.9Ā±3.1 68.9Ā±4.568.9Ā±4.5 71.8Ā±3.871.8Ā±3.8 83.5Ā±2.983.5Ā±2.9 MiniMax-M2.7 87.4Ā±3.687.4Ā±3.6 88.3Ā±5.088.3Ā±5.0 93.2Ā±2.793.2Ā±2.7 97.1Ā±2.497.1Ā±2.4 Kimi-K2.5 78.6Ā±3.778.6Ā±3.7 79.6Ā±4.879.6Ā±4.8 81.6Ā±2.181.6Ā±2.1 92.2Ā±2.892.2Ā±2.8 GLM-5.1 87.4Ā±0.287.4Ā±0.2 87.4Ā±2.487.4Ā±2.4 89.3Ā±2.289.3Ā±2.2 92.2Ā±0.092.2Ā±0.0 BrowseComp Tongyi-DeepResearch 52.0Ā±2.852.0Ā±2.8 52.7Ā±3.852.7Ā±3.8 54.0Ā±2.554.0Ā±2.5 68.7Ā±3.468.7Ā±3.4 OSS-120B 30.0Ā±0.930.0Ā±0.9 29.3Ā±0.929.3Ā±0.9 34.0Ā±1.634.0Ā±1.6 41.3Ā±3.441.3Ā±3.4 MiniMax-M2.7 40.7Ā±5.240.7Ā±5.2 46.0Ā±4.346.0Ā±4.3 50.7Ā±6.250.7Ā±6.2 59.3Ā±3.859.3Ā±3.8 Kimi-K2.5 52.8Ā±10.452.8Ā±10.4 56.6Ā±8.156.6Ā±8.1 58.5Ā±7.858.5Ā±7.8 73.6Ā±13.373.6Ā±13.3 GLM-5.1 72.6Ā±6.072.6Ā±6.0 73.7Ā±5.373.7Ā±5.3 78.9Ā±5.378.9Ā±5.3 92.6Ā±0.992.6Ā±0.9 FRAMES Tongyi-DeepResearch 87.3Ā±3.187.3Ā±3.1 88.0Ā±2.888.0Ā±2.8 88.0Ā±3.588.0Ā±3.5 94.7Ā±2.194.7Ā±2.1 OSS-120B 82.0Ā±4.782.0Ā±4.7 82.7Ā±5.082.7Ā±5.0 85.3Ā±5.785.3Ā±5.7 91.3Ā±3.491.3Ā±3.4 MiniMax-M2.7 88.7Ā±1.988.7Ā±1.9 89.3Ā±2.889.3Ā±2.8 89.3Ā±2.589.3Ā±2.5 94.0Ā±1.694.0Ā±1.6 Kimi-K2.5 88.0Ā±2.888.0Ā±2.8 88.7Ā±3.488.7Ā±3.4 89.3Ā±4.789.3Ā±4.7 97.3Ā±1.397.3Ā±1.3 GLM-5.1 90.0Ā±2.890.0Ā±2.8 90.0Ā±2.890.0Ā±2.8 90.7Ā±1.990.7Ā±1.9 98.0Ā±0.998.0Ā±0.9 Table 4: Headline accuracies (%) with fold-meanĀ± under a 3-fold split of each benchmarkās question pool. This is a coarse estimator: with three folds the standard deviation is itself noisy. We use it as a stability indicator (does the margin survive a different 50-question slice?) rather than for formal hypothesis tests. The point estimates in Table 1 are computed on all 150 questions. Appendix C Verbalised confidence as a voting weight The main text compares RGV against logprob-based confidence (DeepConf), which is the dominant paradigm in the voting literature (Fu et al., 2025; Wang et al., 2022b; Snell et al., 2024). A natural question is how verbalised confidenceāasking the model to self-report a numerical confidence scoreāperforms as an alternative voting weight (Taubenfeld et al., 2025; Xuan et al., 2026; Ou et al., 2025). Verbalised confidence is arguably closer to an āoutside-contextā signal than raw logprobs, since it does not directly read per-token probabilities from the contaminated generation; however, the self-assessment is still produced by the same model reasoning under the same context, so it remains susceptible to the overconfidence patterns documented by Xuan et al. (2026). We include it here for completeness as a third signal class, distinct from both the logit-based (DeepConf) and the retrieval-based (RGV) families. Verbalised-confidence elicitation. After each rolloutās final answer is produced, the agent is issued one deterministic follow-up call at temperature 00, with tools disabled (tool_choice="none") and max_tokens=1024. The prompt asks for a single integer between 00 and 100100 representing the verbalised confidence that the just-given answer is correct (verbatim text in Appendix M). The reply is parsed for the last labelled or bare integer in [0,100][0,100] and stored as a per-rollout score; this score is used only in Appendix C. The protocol elicits a verbalised confidence viā[0,100]v_i\!ā\![0,100] on every rollout (Ā§5, Appendix M). We report it separately from the main-text comparison to keep the signal-class contrast clean (logit-based vs. retrieval-based); this section evaluates it as a voting weight under the same weighted majority rule as the other methods. Dataset Model SM DeepConf Verb RGV DeepConf+RGV Verb+RGV BrowseComp-Plus Tongyi-DeepResearch 62.0 65.7 68.0 71.1 70.7 71.3 OSS-120B 53.3 54.7 56.7 56.0 56.0 58.0 MiniMax-M2.7 75.3 75.3 74.0 78.7 78.0 78.0 Kimi-K2.5 80.7 80.7 80.0 81.3 81.3 82.0 GLM-5.1 80.7 80.7 82.0 82.0 82.0 82.7 GAIA Tongyi-DeepResearch 79.6 79.6 73.8 80.6 80.6 79.6 OSS-120B 69.9 68.9 71.8 71.8 71.8 72.8 MiniMax-M2.7 87.4 88.3 91.3 93.2 92.2 93.2 Kimi-K2.5 78.6 79.6 82.5 81.6 82.5 85.4 GLM-5.1 87.4 87.4 86.4 89.3 89.3 88.3 BrowseComp Tongyi-DeepResearch 52.0 52.7 53.3 54.0 53.3 54.0 OSS-120B 30.0 29.3 35.3 34.0 34.0 34.7 MiniMax-M2.7 40.7 46.0 48.0 50.7 52.7 52.0 Kimi-K2.5 52.8 56.6 60.4 58.5 60.4 62.3 GLM-5.1 72.6 73.7 78.9 78.9 74.7 78.9 FRAMES Tongyi-DeepResearch 87.3 88.0 87.3 88.0 88.0 87.3 OSS-120B 82.0 82.7 82.7 85.3 84.7 84.0 MiniMax-M2.7 88.7 89.3 88.0 89.3 89.3 89.3 Kimi-K2.5 88.0 88.7 84.0 89.3 88.7 88.0 GLM-5.1 90.0 90.0 90.0 90.7 90.0 90.7 Table 5: Voting accuracies (%) including the verbalised-confidence weight (Verb). Verbalised confidence occupies an intermediate position: it does not directly read per-token logprobs, but the self-assessment is still produced under the same context that contains the retrieved documents. As Xuan et al. (2026) and Ou et al. (2025) also observe, this makes it partially susceptible to the overconfidence patterns of Ā§3, though to a lesser degree than logprob-based signals. Combining signals. The DC+RGV and Verb+RGV columns in Table 5 report a simple combination: within each question, we z-normalise each signal (zero mean, unit variance) and form the voting weight wi=zā”(RGVi)+Ī±āzā(otheri)w_i=z(RGV_i)+Ī±\,z(other_i), where Ī± is selected from 0.05,0.1,0.15,0.2,0.3,0.5,1.0\0.05,0.1,0.15,0.2,0.3,0.5,1.0\ per cell to maximise voting accuracy. Verb+RGV improves over RGV alone in 99 of 2020 cells (up to +3.8%+3.8\% on GAIA Kimi-K2.5 and BrowseComp Kimi-K2.5), suggesting that verbalised confidence captures signal orthogonal to lexical grounding. DC+RGV improves in 33 of 2020 cells, indicating that logprob-based confidence is largely subsumed once grounding is accounted for. The Verb+RGV combination is a promising direction for settings where a follow-up confidence call is affordable. Appendix D Cross-dataset evidence for copy-inflation The mechanism analysis in Ā§3 uses BrowseComp-Plus with Tongyi-DeepResearch as the deep-dive configuration. To verify that the copy-inflation pattern is not specific to that setting, we replicate the key diagnostic figures across all twenty benchmarkĆmodel cells. Figure 5: DeepConf overconfidence across all 20 benchmarkĆmodel cells (cf. Figure a). Each panel stratifies questions by oracle difficulty and reports the mean DeepConf score as a percentage of its peak. On every configuration, impossible questions (0%0\% correct, red) receive 8282ā97%97\% of the peak score. Rows: datasets; columns: models. Figure (a) showed that DeepConf gives impossible questions 87%87\% of its peak score on BrowseComp-Plus with Tongyi. Figure 5 extends this across all twenty configurations. On every combination, impossible questions receive 8282ā97%97\% of the peak, confirming that copy-inflation is a general property of logprob-based confidence in multi-turn search agents, not an artefact of one benchmark or model. Severity across model families. Figure 5 shows the behavioural signature; the mechanism itself can be measured directly. Table 6 reports, per model family, the median fraction of generated tokens that are copies and the copy-vs-non-copy logprob gap, adding a DeepSeek cell (deepseek-v4-flash on GAIA, 103103 questions Ć 88) to the five families of Table 1. Family Cell copy % logprob gap GPT-OSS FRAMES 82.1 +0.15+0.15 GPT-OSS BrowseComp-Plus 91.9 +0.07+0.07 Tongyi FRAMES 85.2 +0.69+0.69 Tongyi BrowseComp-Plus 91.9 +0.11+0.11 GLM FRAMES 87.8 +0.16+0.16 Kimi FRAMES 87.7 +0.20+0.20 MiniMax FRAMES 84.8 +0.40+0.40 DeepSeek GAIA 89.7 +0.04+0.04 Table 6: Copy-inflation severity by model family. The gap is positive in every cell; copying accounts for 7878ā92%92\% of generated tokens across all six families (the 78%78\% lower end is the wiki-18 cell of Appendix H). The mechanism replicates in every family we tested, including a reasoning-tuned DeepSeek model. Severity tracks retrieval-bag size and tool style more closely than training objective: the DeepSeek cell has the smallest gap (+0.04+0.04 nats) yet still copies 89.7%89.7\% of its tokens, and the within-question flattening of Appendix I holds there too. Disentangling training objective from tool style would need controlled pairs, which we leave to future work. Appendix E Robustness to alternative lexical-overlap variants The headline method (Ā§4) uses prose-recall under the strict clustering rule. To check that the gain comes from the signal source rather than the specific overlap function, we swap the per-rollout score wiRGVw_i^RGV for other members of the set-overlap family (and a few neighbours), keeping everything else fixed (Table 7). Dataset Model prose-recall Jaccard ROUGE-2 ROUGE-L BM25 TF-IDF BrowseComp-Plus Tongyi-DeepResearch 71.1 71.1 70.2 71.3 71.7 69.0 OSS-120B 56.0 56.0 56.7 55.3 55.3 52.7 MiniMax-M2.7 78.7 77.3 74.0 76.0 75.3 75.3 Kimi-K2.5 81.3 80.0 79.3 79.3 77.3 78.7 GLM-5.1 82.0 79.3 80.0 80.7 79.3 78.7 GAIA Tongyi-DeepResearch 80.6 70.9 72.8 71.8 71.8 73.8 OSS-120B 71.8 69.9 68.9 68.0 68.0 67.0 MiniMax-M2.7 93.2 91.3 91.3 91.3 91.3 89.3 Kimi-K2.5 81.6 78.6 77.7 76.7 79.6 78.6 GLM-5.1 89.3 88.3 88.3 87.4 88.3 87.4 BrowseComp Tongyi-DeepResearch 54.0 53.3 54.0 52.7 50.7 52.0 OSS-120B 34.0 33.3 32.7 32.7 31.3 30.7 MiniMax-M2.7 50.7 48.0 45.3 45.3 46.0 46.0 Kimi-K2.5 58.5 52.8 58.5 52.8 50.9 52.8 GLM-5.1 78.9 82.1 77.9 76.8 69.5 81.1 FRAMES Tongyi-DeepResearch 88.0 88.0 86.7 87.3 86.0 86.7 OSS-120B 85.3 79.3 82.0 82.0 80.7 80.0 MiniMax-M2.7 89.3 89.3 88.0 88.7 88.0 87.3 Kimi-K2.5 88.7 89.3 87.3 88.0 86.7 87.3 GLM-5.1 90.7 90.0 89.3 90.0 89.3 88.7 Table 7: Voting accuracy (%) across lexical-overlap variants on the main cells. All variants use the same max-over-docs reduction; only wiRGVw_i^RGV changes. The highest value in each row is bolded. Definitions. Jaccard: |ā”(P)ā©ā”(d)|/|ā”(P)āŖā”(d)||T(P) (d)|/|T(P) (d)| (symmetric variant of prose-recall). ROUGE-2: bigram F1F_1 between prose and doc. ROUGE-L: unigram F1F_1 (approximating LCS-based ROUGE-L). BM25: scored as the maximum BM25 score of P against each d under the per-rollout retrieval set as the corpus. TF-IDF: cosine of TF-IDF vectors (same corpus). A semantic scorer in place of a lexical one. Every variant above is lexical, which leaves open whether the gain depends on surface overlap. We therefore replace the score with a frozen dense encoder, intfloat/e5-base-v2 (Wang et al., 2022a) (109109M parameters, no fine-tuning), and set wi=maxdāiā”cosā”(Eā”(Pi),Eā”(d))w_i= _d _i \! (E(P_i),E(d) ), keeping the max-over-docs reduction and everything else fixed. Method FRAMES OSS BM25 OSS BM25 Tongyi Simple majority 82.0 38.7 58.7 DeepConf 82.7 40.7 59.3 Embedding cosine 83.3 42.0 60.0 RGV 85.3 44.0 61.3 The two BM25 columns are the degraded-retrieval cells of Appendix H. On all three, the embedding voter beats both simple majority and DeepConf but does not reach the lexical score, at roughly five orders of magnitude more compute (encoder minutes versus ā¼0.3 0.3 ms per rollout). Two known properties of dense encoders are consistent with the ordering: they degrade on exactly the rare entities that carry our signal (Figure 2(2(a))) (Sciavolino et al., 2021), and fixed-dimension embeddings lose precision on long documents (Luan et al., 2021); a similar pattern is reported for hallucination detection, where an n-gram check outperforms grey-box log-probability baselines in most setups (Manakul et al., 2023). The point is not that lexical overlap is optimal. It is that two scorers with nothing in common except where they read from land on the same side of the baselines, which is what the copy-inflation account predicts: the signal source carries the gain, not the metric. Appendix F DeepConf variant grid Fu et al. (2025) propose three reductions of sliding-window group confidences: Lowest-Group (sliding-window minimum, the extreme case of Bottom-q%q\% as qā0q\!ā\!0), Bottom-10%10\%, and Tail. We compute the per-token confidence as they do, Ct=ā1kāj=1klogpjC_t=- 1k _j=1^k p_j, and sweep all three reductions across three window sizes Wā1024,2048,4096W\!ā\!\1024,2048,4096\ on the rolloutās generated tokens; per-rollout weights are then plugged into the weighted majority vote of Ā§2 (Table 8). Lowest-Group (min ) Bottom-10% Tail Dataset Model 1024 2048 4096 1024 2048 4096 1024 2048 4096 BrowseComp-Plus Tongyi-DeepResearch 65.7 65.4 64.8 65.7 65.5 64.9 61.8 61.9 61.4 OSS-120B 54.7 54.0 53.3 54.7 54.0 53.3 53.3 52.7 52.0 MiniMax-M2.7 75.3 74.7 74.0 75.3 74.7 73.3 73.3 72.7 72.0 Kimi-K2.5 80.7 80.0 79.3 80.7 80.0 79.3 78.0 78.0 77.3 GLM-5.1 80.7 80.0 79.3 80.7 80.0 79.3 78.7 78.0 77.3 GAIA Tongyi-DeepResearch 79.6 78.6 77.7 79.6 78.6 77.7 76.7 76.7 75.7 OSS-120B 68.9 67.0 65.0 68.9 67.0 65.0 64.1 64.1 63.1 MiniMax-M2.7 88.3 87.4 86.4 88.3 87.4 86.4 85.4 84.5 83.5 Kimi-K2.5 79.6 78.6 77.7 79.6 78.6 77.7 76.7 75.7 74.8 GLM-5.1 87.4 86.4 85.4 87.4 86.4 85.4 85.4 84.5 83.5 BrowseComp Tongyi-DeepResearch 52.7 51.3 50.0 52.7 51.3 50.0 49.3 48.7 47.3 OSS-120B 29.3 28.7 28.0 29.3 28.7 28.0 27.3 27.3 26.7 MiniMax-M2.7 46.0 44.7 43.3 46.0 44.7 43.3 42.0 41.3 40.7 Kimi-K2.5 56.6 54.7 52.8 56.6 54.7 52.8 52.8 50.9 49.1 GLM-5.1 73.7 72.6 71.6 73.7 72.6 71.6 69.5 68.4 67.4 FRAMES Tongyi-DeepResearch 88.0 87.3 86.7 88.0 87.3 86.7 86.0 85.3 84.7 OSS-120B 82.7 82.0 81.3 82.7 82.0 81.3 80.0 80.0 79.3 MiniMax-M2.7 89.3 88.7 88.0 89.3 88.7 88.0 86.7 86.0 85.3 Kimi-K2.5 88.7 88.0 87.3 88.7 88.0 87.3 86.0 85.3 84.7 GLM-5.1 90.0 89.3 88.7 90.0 89.3 88.7 88.0 87.3 86.7 Table 8: DeepConf voting accuracy (%) over the 3 reductions Ć 3 window sizes grid (generated tokens). The Lowest-Group Ć W=1024W=1024 cell matches the DC headline in Table 1. The highest value in each row is bolded. Appendix G Robustness of the copy-inflation gap The headline number in Ā§3 (Fig. ) is the per-token mean logprob gap between copy and non-copy content tokens: +0.50+0.50 nats overall, +0.61+0.61 on judge-correct rollouts, +0.38+0.38 on judge-wrong ones. To check the gap is not an artefact of any single token-subset choice we re-measure under four alternative rules. All measurements use the same pool of n=307n=307 rollouts drawn from the BrowseComp-Plus Ć Tongyi-DeepResearch runs (830830 questions, 88 rollouts each). The pool is constructed by uniformly sampling 5050 questions per rollout seed and keeping those whose token-level statistics and retrieved-doc texts are both available (307/400307/400); the random seed is fixed for reproducibility. Common procedure. For each rollout we form two artefacts. (i) The retrieved-doc text D: a lowercased, whitespace-collapsed concatenation of everything the search and visit tools returned to the agent. (i) The generated-token stream: every per-token logprob the agent emitted inside its chain-of-thought reasoning span. Each generated token is normalised by stripping subword-piece prefixes, lower-casing, and keeping only alphanumeric characters; tokens that normalise to fewer than two characters are discarded. A token is a copy token if its normalised surface is a substring of D, else non-copy. Substring matching makes the test robust to BPE splits of rare entities. For a token-subset rule with predicate Ļā”(ā )Ļ(Ā·) and weight Ļā”(ā )Ļ(Ā·) we pool every kept token across rollouts and report Ī= \;= ātācopy,Ļā”(t)Ļā”(t)ālogā”pā”(t)ātācopy,Ļā”(t)Ļā”(t) _t ,\,Ļ(t)Ļ(t) p(t) _t ,\,Ļ(t)Ļ(t) ā \;- ātācopyĀÆ,Ļā”(t)Ļā”(t)ālogā”pā”(t)ātācopyĀÆ,Ļā”(t)Ļā”(t), _tā copy\,,Ļ(t)Ļ(t) p(t) _tā copy\,,Ļ(t)Ļ(t), the same token-level aggregation as the main-text headline of +0.50+0.50 nats. Variants. ā¢ All alnum tokens: Ļā”TrueĻ , Ļā”1Ļā” 1. Baseline; reproduces the main-text headline. ā¢ Stopword-removed: Ļ(t)=[wtā]Ļ(t)=[w_t ] with S a ā¼120 120-word English stopword list. ā¢ IDF-weighted: ĻĻ as in stopword-removed, Ļā”(t)=logā”(N/dfā”(wt))Ļ(t)= (N/df(w_t)) with sample IDF over N=307N=307 documents-as-rollouts. ā¢ Digits or capitalised-leading: digit tokens of length ā„2ā„ 2 or BPE-stripped tokens starting uppercase, Ļā”1Ļā” 1. ā¢ Length ā„8ā„ 8: Ļ(t)=[|wt|ā„8]Ļ(t)=[|w_t|ā„ 8], Ļā”1Ļā” 1. Token subset |copy||copy| |copyĀÆ|| copy| gap all alnum (baseline) 936 332 84 584 +0.51+0.51 stopword-removed 558 491 74 714 +0.54+0.54 IDF-weighted 558 491 74 714 +0.69+0.69 digits / cap-leading 232 855 21 690 +0.44+0.44 length ā„8ā„ 8 92 233 25 935 +0.62+0.62 Table 9: Copy-vs-non-copy logprob gap under five token-subset rules. All gaps positive at pāŖ0.001p 0.001 (paired bootstrap over rollouts). All five rules produce a significantly positive gap, confirming that the copy-inflation gap is not a side-effect of any particular token-subset choice. The gap is largest under IDF-weighting, consistent with the visual story that rare, entity-like tokens (the kind retrieval brings into context) are exactly the ones the agent copies. Appendix H Degraded retrieval: weaker retriever and corpus mismatch RGV reads its signal from the retrieval log, so its behaviour under a weaker retrieval stack is a boundary worth measuring rather than assuming. We report four additional cells, each 150150 questions Ć 88 rollouts under the protocol of Ā§5 with the same Qwen3-32B judge, varying only the retrieval component. Two degradation modes. In the first, we replace the dense retriever (Qwen3-Embedding-8B) on BrowseComp-Plus with BM25 (Robertson and Zaragoza, 2009), holding the corpus and every other part of the agent fixed. In the second, we adopt the Search-R1 retrieval stack (Jin et al., 2025): the wiki-18 corpus (2121M passages) with the E5 retriever (Wang et al., 2022a), top-55 passages per call. We run that stack both on HotpotQA (Yang et al., 2018), whose questions are Wikipedia-native, and on the same FRAMES questions used in Table 1, where the corpus no longer covers the questions. Retrieval stack Model Single SM DeepConf RGVprose_prose RGVJac_Jac Oracle BM25 ā dense (BCP+) gpt-oss-120b 31.4 (47.3) 38.7 40.7 37.3 44.0 52.7 BM25 ā dense (BCP+) Tongyi-DeepResearch 48.8 (58.9) 58.7 59.3 60.7 61.3 79.3 wiki-18 + E5, HotpotQA gpt-oss-120b 72.7 74.7 75.3 73.3 72.7 80.7 wiki-18 + E5, FRAMES questions gpt-oss-120b 59.6 (78.7) 58.7 58.0 60.0 59.3 72.0 Table 10: Voting accuracy (%) under degraded retrieval. Parenthesised values are the single-rollout accuracy of the corresponding stock cell, i.e. how far the swap moved the agent. We report both members of the overlap family (Appendix E). Length-weighted voting scores 38.038.0, 56.056.0, 74.774.7 and 60.760.7 on the four rows respectively. Where degraded retrieval bites, grounding wins. On the two BM25 rows the swap genuinely degrades the agent (single accuracy ā31.447.3\!ā\!31.4 and ā48.858.9\!ā\!48.8), and the grounded vote keeps a clear margin: +5.3/+3.3+5.3/+3.3 over simple majority and DeepConf on gpt-oss-120b, and +2.6/+2.0+2.6/+2.0 on Tongyi-DeepResearch. Where it does not, methods converge. On HotpotQA the Search-R1 stack turns out to be sufficient for Wikipedia-native questions: single accuracy is 72.772.7 with only 88 points of headroom to the oracle, and all four aggregation rules sit inside a 2.62.6-point band. This is also the lightest-copying cell we measured (78.4%78.4\% of generated tokens, versus 91.9%91.9\% on BrowseComp-Plus) and the one with the healthiest DeepConf (rollout-level AUC 0.820.82), exactly as the copy-inflation account of Ā§3 predicts: less copying, less contamination, more usable internal confidence. Under corpus mismatch, only grounding stays above single. Running the same stack on FRAMES questions is the harshest setting: retrieval falls out of domain and single-rollout accuracy drops ā59.678.7\!ā\!59.6. Here simple majority (58.758.7) and DeepConf (58.058.0) fall below the single-rollout average, because errors become correlated when every rollout retrieves from the wrong corpus, and a vote over correlated errors amplifies rather than cancels them. RGV (60.060.0) is the only aggregation rule that stays above it. Taken together, degradation is graceful and, more usefully, it is predictable in advance: the copy fraction of a cell and its remaining headroom to the oracle tell which regime applies before any voting rule is chosen. Appendix I Causal interventions on copy-inflation Ā§3 establishes copy-inflation by correlation: copy-heavy rollouts have flatter DeepConf scores. This appendix reports two interventions that test the causal direction. I.1 Intervention 1: masking copied tokens If copying causes the flattening, then recomputing DeepConf on non-copy tokens only should restore within-question spread. And if the internal signal were merely diluted rather than corrupted, discrimination should return along with the spread. We recompute DeepConf over the complement of the copy set (the substring rule of Appendix G) on ten cells spanning six model families and four corpora. rollout AUC within-Q variance share Corpus / cell Model family copy % DC DCmasked_masked DC DCmasked_masked RGV FRAMES GPT-OSS 82.1 0.68 0.49 0.35 0.52 0.46 FRAMES Tongyi 85.2 0.59 0.37 0.49 0.57 0.57 FRAMES GLM 87.8 0.78 0.75 0.16 0.30 0.31 FRAMES Kimi 87.7 0.67 0.47 0.46 0.59 0.53 FRAMES MiniMax 84.8 0.69 0.57 0.51 0.50 0.58 BrowseComp-Plus Tongyi 91.9 0.67 0.43 0.52 0.48 0.67 BrowseComp-Plus GPT-OSS 91.9 0.73 0.43 0.18 0.42 0.59 GAIA DeepSeek 89.7 0.70 0.72 0.16 0.24 0.48 wiki-18 / HotpotQA GPT-OSS 78.4 0.82 0.49 0.13 0.29 0.49 wiki-18 / FRAMES GPT-OSS 81.9 0.79 0.44 0.12 0.28 0.41 Table 11: Masking copied tokens restores variance but not validity. copy % is the median fraction of generated tokens that are copies. rollout AUC separates judge-correct from judge-wrong rollouts; within-Q variance share is the quantity weighted voting consumes (Ā§3). Masking raises the variance share in 8/108/10 cells (up to 2.4Ć2.4Ć on BrowseComp-Plus / GPT-OSS, ā0.420.18\!ā\!0.42) while AUC falls in 9/109/10. Three readings of Table 11. First, masking does restore the spread, which confirms that copied tokens are what flattens the score. Second, the restored spread carries no correctness signal: AUC falls in nine of ten cells, and voting accuracy does not systematically improve (up in 22 cells, flat in 33, down in 55). The discriminative information lives in precisely the tokens the repair removes. Third, the retrieval-grounded score holds 1.11.1ā3.8Ć3.8Ć (median 1.6Ć1.6Ć) more within-question variance share than DeepConf in 10/1010/10 cells. This is a negative result we consider more consequential than the method itself: the contamination resists internal repair, so any consumer of agent token-confidence, whether for early stopping, routing, abstention or reward shaping, inherits it, and no simple adjustment recovers a usable weight. I.2 Intervention 2: removing the documents The second intervention removes the cause instead of the symptom. We take 100100 BrowseComp-Plus / Tongyi-DeepResearch rollouts and teacher-force the model over the same answer tokens twice: once with the retrieved documents present in context, and once with each document replaced by the placeholder [document removed] (trailing-context budget 2424K tokens). If the documents are what prop up copied-token confidence, removing them should cost copied tokens more than non-copy tokens. Token class Ī logprob (mean) Ī logprob (median) copied ā0.221-0.221 ā0.187-0.187 non-copy ā0.106-0.106 ā0.068-0.068 Copied tokens lose roughly twice as much log-probability, and the copy drop exceeds the non-copy drop in 8282 of 100100 rollouts. Together with the null controls of Figure 2(2(b)) and the token-subset robustness of Appendix G, this closes the causal chain: documents in context inflate the confidence of the tokens copied from them. Appendix J Prompt and answer-format sensitivity A grounding-based weight invites two worries: that a prompt could inflate it, and that it could break on short answers. We test both directly with gpt-oss-120b on BrowseComp-Plus, 150150 questions Ć 88 rollouts per arm, changing only one line of the prompt. Arm Single SM DC RGV med. prose tok. stock 47.3 53.3 54.7 56.0 254 ++ grounding instr. 48.6 55.3 57.3 59.3 254 ++ entity-only 44.2 52.7 53.3 53.3 2 A prompt cannot buy weight. The grounding arm appends āyour final answer must be grounded in the retrieved documentsā. The mean voted weight is unchanged (0.091ā0.0910.091ā 0.091) and rollout-level AUC barely moves (0.840ā0.8440.840ā 0.844). The reason is structural rather than empirical: an instruction attached to the question reaches all N rollouts of that question equally, and weighted voting consumes only the within-question ordering of the weights, which a common shift leaves intact. This is the same property that makes the weight robust to a rollout being merely verbose (Ā§4). Short answers degrade gracefully. The entity-only arm forces a bare entity as the final answer, taking the median answer prose from 254254 tokens to 22. At that length prose-recall reduces to answer containment: whether the predicted entity occurs in the rolloutās own documents. Rank AUC dips from 0.710.71 to 0.670.67 on the truly short subset (77%77\% of rollouts), and 99%99\% of correct entities are anchored in their own retrieval versus 80%80\% of wrong ones. Every method loses accuracy in this arm because the model is weaker without its report format (single 47.3ā44.247.3ā 44.2); within it, the grounded vote still matches DeepConf and leads simple majority. This arm also isolates the design choice of Ā§4: a symmetric, document-side-normalised overlap such as Jaccard collapses to 42.042.0 here, because a two-token answer can never cover a long document. The answer-side denominator is what keeps the score meaningful at this extreme. Appendix K Selection versus voting RGV is presented as a voting weight, but the same score can select a single rollout. Separating the two tells us whether the gain comes from the signal or from the aggregation frame. On the BrowseComp-Plus / Tongyi-DeepResearch replay cell we compare weighted voting against Best-of-N selection (take the single highest-scoring rollout), sub-sampling to Nā1,2,4,8Nā\1,2,4,8\ with 6060 random partitions per N. N Single SM vote RGV vote DC argmax RGV argmax DC Oracle 1 58.9 58.9 58.9 58.9 58.9 58.9 58.9 2 58.9 58.9 63.3 60.3 63.3 60.2 70.1 4 59.4 64.3 69.6 65.2 67.4 61.5 79.7 8 59.2 67.3 74.0 70.0 69.3 60.7 86.7 Two conclusions. RGV works as a standalone rollout-quality scorer: argmax-RGV beats argmax-DeepConf at every N, and at N=8N=8 it also beats simple majority (69.369.3 vs 67.367.3). But voting dominates selection by +4.7+4.7 points at N=8N=8, because argmax stakes the whole answer on one rollout and is therefore fully exposed to the well-grounded-but-wrong failure mode quantified in Ā§6.4; summing weights over an answer cluster averages that risk away. Notably argmax-DeepConf decreases from N=4N=4 to N=8N=8: with more rollouts to choose from, a flattened confidence score is more likely to pick a confidently wrong one. Appendix L Preliminary results on HLE Our main evaluation focuses on search agent benchmarks where retrieved documents are the primary evidence source (Ā§5). Here, we report one exploratory cell on Humanityās Last Exam (Phan et al., 2025), an expert-reasoning benchmark where retrieval plays a minor role: Single SM DC RGV Oracle OSS-120B 25.6 31.2 31.8 32.5 49.0 RGV still outperforms DeepConf (+0.7%+0.7\%), though the margin is smaller than on search-centric benchmarks. This is expected: HLE questions often require domain-expert reasoning beyond what retrieval supplies, so the grounding signal captures only part of the answer quality. Appendix M Worked examples: prompts, thought spans, and clustering This appendix shows the exact text the agent and the judge see, plus illustrative excerpts of what a rollout produces. The examples are lightly redacted (line breaks, bracketed abbreviations) for readability; nothing functional is changed. M.1 Agent system prompt Below is the system prompt for BrowseComp-Plus (fixed-corpus retrieval with search and get_doc). For the open-web benchmarks (BrowseComp, GAIA, FRAMES) the tool names are search and visit and references to [docid] become [url]; the rest of the prompt is identical. You are a deep research assistant. Your core function is to conduct thorough, multi-source investigations into any topic. You must handle both broad, open-domain inquiries and queries within specialized academic fields. For every request, synthesize information from credible, diverse sources to deliver a comprehensive, accurate, and objective response. # Tools You have access to search and get_doc tools. Use search to look up snippets; use get_doc(docid) to read full documents you find via search. You may call tools many times until you have gathered sufficient evidence. # Final answer format When you have enough evidence, produce a comprehensive markdown report containing: -- A bolded direct answer to the question -- ## Step-by-Step Reasoning and Evidence section with sub-sections walking through each clue -- A markdown table or bullet list summarizing how each criterion is satisfied -- Inline citations with [docid] for every nontrivial claim Do not give a one-line answer; produce a structured research report. M.2 Force-final prompt Issued once when the token-count guard fires, max-iterations is hit, the output budget is exhausted, or the API returns a non-retryable error. Tools are disabled for this call. The conversation has been truncated. Based on the evidence you have gathered so far, write the final markdown report now. Use the format from the system prompt (bolded direct answer, ## Step-by-Step Reasoning and Evidence, a summary table, inline [docid] citations). End the report with a single line of the form Confidence: N%. M.3 Verbalised-confidence solicitation Issued once after the rollout has produced a final answer (either naturally or via force-final), at temperature 00 with tools disabled. Reply with ONLY a single integer between 0 and 100 representing your confidence (in percent) that the answer you just gave is correct. No words, no %, no labels --- just the integer. Example output: 73 M.4 Judge prompt The judge (Qwen3-32B at temperature 00) receives the question, the gold answer, and the rolloutās final markdown report, and decides correctness. We use the official BrowseComp-Plus GRADER_TEMPLATE for every dataset. Question: question Gold answer: gold_answer Model response: model_response Read the modelās response. First, extract the modelās final short answer in the extracted_final_answer field (or output None if the model did not produce one). Then decide whether the extracted answer matches the gold answer up to obvious paraphrase / equivalent name. Output a JSON object with fields extracted_final_answer, reasoning, and judge_correct ā\ yes, no \. Do not output anything else. M.5 Example thought span vs. answer prose A single rollout on the BrowseComp-Plus question āThere is an influencer whose career has lasted over a decadeā¦What is the full name of the influencer?ā (gold answer: TĆ³ke Makinwa). The thought span is what DeepConf readsāchain-of-thought across all turns; the answer prose is what RGV readsāthe assistantās output on the final answer turn (no internal thought). Thought span (excerpt, what DeepConf reads). Need an influencer with a >>10-year career who anchored a 2022 talk segmentā¦The host is born in the 1980s and has served as a host at major events. Possible candidates: Tope Mark-Odigie, TĆ³ke Makinwa, Lilly Singhā¦TĆ³ke Makinwa is listed as 2nd of 4 siblings on Wikipedia [docid 118], became brand ambassador for Pulse Nigeria in 2016ā¦Lilly Singh is also 2nd of 4 but the 2022 talk-segment host criterion fits TĆ³ke betterā¦ Answer prose (what RGV reads). TĆ³ke Makinwa. ## Step-by-Step Reasoning and Evidence -- Long-career influencer with a YouTube vlog series: [docid 118] records that TĆ³ke Makinwa launched the Toke Moments vlog in 2014, more than a decade after her 2004 radio debut. -- 2022 talk-segment anchor: [docid 118] notes she anchored the talk segment of The Buzz, hosted by Ebuka Obi-Uchendu (born 1982). -- Second of four siblings: [docid 118] lists her as the second of four. -- 2016 ambassador for an MIT-founded brand: [docid 118] records the Pulse Nigeria ambassadorship; Pulseās founder Tunde Kehinde is an MIT Sloan alumnus. ā”(Pi)T(P_i) is built from the Answer prose block only; DeepConfās per-token confidences are computed over the Thought span only. The two operate on disjoint parts of the rolloutās output. M.6 Example clustering on one question For the same question, the eight rollouts produced the following predicted-answer strings, normalised under the strict rule: # raw predicted answer normalised cluster 1 TĆ³ke Makinwa. toke makinwa A 2 The answer is Toke Makinwa. toke makinwa A 3 Lilly Singh lilly singh B 4 Lilly Singh. lilly singh B 5 Lilly Singh lilly singh B 6 Toke Makinwa toke makinwa A 7 Shannon LaNier shannon lanier C 8 Bhuvan Bam bhuvan bam D Strict clustering recovers four clusters with |A|=3|A|=3, |B|=3|B|=3, |C|=|D|=1|C|=|D|=1. Simple-majority ties A and B at weight 33; under DeepConf the eight rollouts all receive nearly identical weights and the tie persists; under RGV the three A-cluster rollouts carry the largest prose-recall (their answer prose lexically aligns with the TĆ³ke Makinwa Wikipedia entry the agent actually fetched), so the weighted vote selects cluster A. Appendix N Success and failure cases We first show three representative successesāminority-correct questions where RGV rescues the answer that DeepConf missesāthen five failures exhibiting the āwell-grounded but wrongā pattern of Ā§6.4. N.1 Success cases: RGV rescues the correct minority DeepConf gives all rollouts near-identical confidence and follows the wrong majority; RGV elevates the correctly grounded rollout. Success 1 (qid 301): royal family identification. Q: āName the royal family a certain individualās spouse was born intoā¦ā (multiple biographical constraints). Gold: Nwoko. RGV DC Predicted answer r2 ā 0.184 6.54 Nwoko Royal Family r6 ā 0.118 6.66 Nwoko Royal Family r7 ā 0.071 6.46 Greek royal family r1 ā 0.026 6.70 Greek royal family r5 ā 0.024 6.68 British royal family r0 ā 0.009 6.75 House of Orange-Nassau r4 ā 0.008 6.59 House of Windsor Seven distinct answers. The two correct rollouts retrieve the Nwoko family page and echo it in their prose (RGVā„0.118RGVā„ 0.118); all wrong rollouts name famous European royal families without retrieving relevant evidence (RGV ā¤0.071ā¤ 0.071). DeepConf range: 6.466.46ā6.756.75, unable to discriminate. Success 2 (qid 424): architect identification. Q: āName the architect who was a WWII veteran and TV broadcaster, designed a building completed 1977ā1987ā¦ā Gold: Raffaele Contigiani. RGV DC Predicted answer r4 ā 0.152 6.98 Raffaele Contigiani r6 ā 0.133 6.72 Raffaele Contigiani r1 ā 0.095 6.75 Victor Alfred Lundy r3 ā 0.092 6.99 Denys Lasdun r2 ā 0.073 6.76 Denys Lasdun r0 ā 0.018 6.48 Denys Lasdun Three wrong rollouts converge on Denys Lasdun (a famous British architect who fits some but not all constraints). The correct rollouts retrieve Contigianiās Italian biography and echo it in prose, producing a clear RGV separation (0.1520.152 vs. 0.0950.095). Success 3 (qid 56): movie identification. Q: āMovie with a ReFrame Stamp, 2018ā2023, about a festival, with a cast member who appeared in a heist filmā¦ā Gold: Last Christmas. RGV DC Predicted answer r2 ā 0.115 7.53 Last Christmas r5 ā 0.028 7.96 Crazy Rich Asians r4 ā 0.026 7.54 Crazy Rich Asians r3 ā 0.026 7.69 Crazy Rich Asians r7 ā 0.018 7.78 Rifkinās Festival r1 ā 0.018 7.37 Rifkinās Festival r6 ā 0.016 7.70 A Simple Favor Three rollouts vote for Crazy Rich Asians with high DeepConf (7.547.54ā7.967.96); DeepConf selects this wrong majority. RGV: the single correct rollout retrieves the filmās page (0.1150.115 vs. ā¤0.028ā¤0.028) and wins. N.2 Failure cases: well-grounded but wrong Failure 1 (qid 1185): cricket match. Q: āMatch number, tournament, year satisfying nine batting constraintsā¦ā Gold: 31st match, IPL 2013. RGV DC Predicted answer r7 ā 0.235 6.93 2019 ICC World Cup r5 ā 0.174 6.98 Bangladesh vs South Africa r2 ā 0.141 7.30 31st match, IPL 2013 r7 retrieves extensive World Cup documentation and echoes cricket statistics (RGV=0.235RGV=0.235), outscoring the correct rollouts (0.1650.165, 0.1410.141) despite naming the wrong tournament. Failure 2 (qid 790): writerās work identification. Q: āA writer born in July, article posted April 2023, smoked on Christmasā¦name the work.ā Gold: āManosā. RGV DC Predicted answer r0 ā 0.229 6.96 āKilling Jokeā r3 ā 0.206 6.68 āManosā r1 ā 0.094 6.71 (different work) r7 ā 0.008 6.70 āmouseā r4 ā 0.000 6.80 āgirlsā r5 ā 0.000 6.83 āThugsā r0 retrieves an article about the same writer but identifies a different work (0.2290.229 vs. 0.2060.206). Both rollouts ground their answer in the writerās bibliography; the wrong one happens to echo a more frequently cited title. Failure 3 (qid 773): clothing colour. Q: āA child reported missing multiple times 2014ā2018. What colour shirt in the 2018 police description?ā Gold: Red. RGV DC Predicted answer r7 ā 0.209 7.41 White t-shirt r3 ā 0.190 7.27 (different colour) r4 ā 0.180 7.68 (different colour) r2 ā 0.179 7.46 (different colour) r5 ā 0.166 7.88 (different colour) r6 ā 0.142 8.05 (different colour) r1 ā 0.109 7.11 Red r0 ā 0.000 6.45 (different colour) All seven wrong rollouts retrieve articles about the same missing-person case and echo detailed police descriptions, receiving higher RGV scores (0.1420.142ā0.2090.209) than the single correct rollout (0.1090.109). The correct answer (āRedā) is a single word with minimal lexical overlap; wrong rollouts produce longer prose quoting extensively from the retrieved reports. Pattern and future directions. Each failure shares a structure: the wrong rollout retrieves documents about the right domain (same writer, same sport, same missing-person case) but identifies the wrong specific entity within that domain. RGV measures whether the answer is anchored in retrieved evidence; it cannot verify that the evidence answers the specific question asked. Closing this gap likely requires moving beyond surface-level lexical overlap toward semantic grounding checks, for example, entailment verification between the question constraints and the retrieved passages, or cross-referencing the answer against multiple independent sources. Such extensions could complement RGVās efficiency with deeper reasoning at a modest additional cost. Appendix O Artifacts, licences, and intended use Table 12 lists every external artifact this work uses, with the licence under which it is released. We used each within the research use its original release specifies. Artifact Type Licence Note on use BrowseComp-Plus (Chen et al., 2025) benchmark MIT questions, corpus, gold-evidence qrels BrowseComp (Wei et al., 2025) benchmark MIT via openai/simple-evals FRAMES (Krishna et al., 2025) benchmark Apache-2.0 questions and gold answers GAIA (Mialon et al., 2023) benchmark gated, none stated validation split; redistribution not permitted HLE (Phan et al., 2025) benchmark MIT authors ask that it not be re-uploaded HotpotQA (Yang et al., 2018) benchmark C BY-SA 4.0 Appendix H only wiki-18 (Jin et al., 2025) corpus C BY-SA Wikipedia derivative; Appendix H only gpt-oss-120b (OpenAI et al., 2025) model Apache-2.0 open weights, served via API Tongyi-DeepResearch-30B-A3B (Team et al., 2025b) model Apache-2.0 open weights, served locally (vLLM, FP8) MiniMax-M2.7 (MiniMax et al., 2025) model per model card served via API GLM-5.1 (GLM-5-Team et al., 2026) model per model card served via API Kimi-K2.5 (Team et al., 2025a) model per model card served via API Qwen3-32B judge Apache-2.0 served locally (vLLM) e5-base-v2 (Wang et al., 2022a) encoder MIT Appendix E baseline Table 12: External artifacts and their licences. What we release. We release our code, the full rollout trajectories, the per-token log-probabilities, and the judge outputs for every cell reported in this paper. Our own contributions in that release, namely the trajectories, scores, judge labels and derived annotations, are made available under C BY 4.0. Text that the environment returned (retrieved corpus passages and fetched web pages) remains under its original terms and is redistributed with its source identifiers retained; we provide a contact for removal requests. Benchmarks we do not redistribute. Two of the benchmarks we evaluate on restrict redistribution, and we respect those terms. GAIA is access-gated and states that its validation split may not be reshared; HLEās authors ask that the benchmark not be publicly re-uploaded, to protect it from contamination. For these two, our release carries question identifiers rather than question and gold-answer text, so that researchers who have obtained the benchmarks through their official channels can join our records against them, while the release itself does not republish the benchmark. Data characteristics. All questions and answers are in English. The benchmarks consist of factual information-seeking questions and contain no personal or offensive content by construction. The agentsā retrieved documents come from a fixed corpus (BrowseComp-Plus, wiki-18) or the live web via a commercial search API (BrowseComp, GAIA, FRAMES); we scan the released tool outputs for credentials and personally identifying information before publication.