Paper deep dive
Judged Useless, Queried Anyway: Tool-Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisions
Chubin Zhang, Zhenglin Wan, Xingrui Yu, Jingxuan Wu, Yaxin Zhou, Ivor Tsang, Bo An
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:An agent whose tool keeps returning nothing useful should stop relying on it. In a retrieval environment with controlled source failures, we separate how agents judge results from what they do. We compare stopping at the same step after longer and shorter runs of results the agent judged useless; this contrast is zero for clock- or deadline-driven stopping. Where we record their judgments, the seven agents we test call a failing source's results useless 97-100% of the time, yet most of them rarely stop on that judgment. Prompt cues change when they stop but not what they stop on. Permission to answer from memory and a reasoning mode can bring early stops regardless of evidence, a stated budget moves the 7-8B models' stops to the deadline, and a stopping rule or call cost in the prompt is followed at most partly. Stopping follows the evidence only when the harness enforces an integration step that makes the agent answer after five consecutive results it judged useless. This step raises failing-source success for every model, keeps the stopping point fixed when the budget doubles, and needs no extra judgment call when the agent states its judgments. A pre-registered replication on 300 fresh questions confirms the dissociation and the rule's effect.
Tags
Links
- Source: https://arxiv.org/abs/2610.06191v1
- Canonical: https://arxiv.org/abs/2610.06191v1
Trouble viewing inline? Open PDF directly β
Full Text
132,799 characters extracted from source content.
Expand or collapse full text
Judged Useless, Queried Anyway yu_xingrui@a-star.edu.sg β‘ Corresponding author. ://github.com/bennidict23/judged-useless-queried-anyway Judged Useless, Queried Anyway: Tool-Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisions Chubin Zhang Affiliation: Nanyang Technological University, Singapore Zhenglin Wan Affiliation: National University of Singapore, Singapore Xingrui Yu Affiliation: CFAR, Agency for Science, Technology and Research, Singapore Jingxuan Wu Affiliation: Department of Statistics and Operations Research, UNC-Chapel Hill, United States Yaxin Zhou Affiliation: Carnegie Mellon University, United States Ivor Tsang Affiliation: Nanyang Technological University, Singapore Affiliation: CFAR, Agency for Science, Technology and Research, Singapore Affiliation: IHPC, Agency for Science, Technology and Research, Singapore Bo An Affiliation: Nanyang Technological University, Singapore Abstract An agent whose tool keeps returning nothing useful should stop relying on it. In a retrieval environment with controlled source failures, we separate how agents judge results from what they do. We compare stopping at the same step after longer and shorter runs of results the agent judged useless; this contrast is zero for clock- or deadline-driven stopping. Where we record their judgments, the seven agents we test call a failing sourceβs results useless 97β100% of the time, yet most of them rarely stop on that judgment. Prompt cues change when they stop but not what they stop on. Permission to answer from memory and a reasoning mode can bring early stops regardless of evidence, a stated budget moves the 7β8B modelsβ stops to the deadline, and a stopping rule or call cost in the prompt is followed at most partly. Stopping follows the evidence only when the harness enforces an integration step that makes the agent answer after five consecutive results it judged useless. This step raises failing-source success for every model, keeps the stopping point fixed when the budget doubles, and needs no extra judgment call when the agent states its judgments. A pre-registered replication on 300 fresh questions confirms the dissociation and the ruleβs effect. 1 Introduction Figure 1: Left: a real held-out trajectory of Claude Haiku 4.5 with a persistently failing source (selected by a fixed rule, Appendix B.3). It calls every result irrelevant and recalls at step 6 that Barlow is in a band, yet searches until its budget runs out. With the enforced rule, it answers correctly. Right: share of side-channel judgments that call a failing sourceβs results useless (red), and share of questions with five such judgments in a row on which the unaided agent answers (gray). Judgments are replayed on the unaided agentβs own trajectories, except for Claude Haiku 4.5 (*), whose judgments come from its enforced-rule run on the same questions. Tools often fail silently. When a search index goes stale or a retriever drifts off topic, the agent still receives text, but the text no longer helps. Figure 1 shows what a capable agent does then. In the example, Claude Haiku 4.5 tries to answer a two-hop question with a search tool that has stopped working. It calls every result irrelevant and recalls at step 6 that Barlow is in a band, yet it keeps searching until its budget runs out. Most agents we test behave this way. After their own judgments have called five results in a row useless, five of the seven answer on at most 7% of questions. Yet the answer is often within reach. When forced to answer at that point, Claude Haiku 4.5 is right on 114 of the 284 questions where this happens, whereas on its own it answers only one of them. Leaving a source is a sequential decision. One useless result is weak evidence that the source has failed, because a reformulated query may succeed and the source may recover. An agent that leaves at the first useless result forgoes the evidence of every source that would have recovered; one that never leaves spends its whole budget on every source that would not. A good policy must therefore integrate its judgments: the longer the current run of results it has judged useless, the more likely it should be to stop. It should track the run rather than the total, because the run restarts when the source recovers. Prior work documents that search-augmented models over-search (Xie et al., 2026) and that deep-research agents keep browsing results that have turned out irrelevant, which Zhang et al. (2026b) attribute to distorted judgment. Two questions remain open. Does the failure lie in judging the evidence, in believing that continuing pays, or in turning either into an action? And can it be fixed simply by telling the agent more, for instance that it may answer without evidence or how many steps remain? To answer them, we build a retrieval environment in which we control when the source fails and whether it recovers (Β§2), and we measure the agentβs judgments, beliefs and actions separately. Success alone cannot reveal what stopping responds to. An agent that stops at a fixed step, one that stops at the deadline and one that stops after enough useless evidence can score alike and differ only when conditions change. The difference still matters, because an agent that waits for the deadline spends every call up to it and moves its stopping point whenever the budget changes. We therefore start from what integration predicts. If an agent integrates its own judgments, then at the same step it should stop more often after an unbroken run of results it judged useless than after a history containing a result it judged useful (Figure 2). We test this prediction with the time-matched contrast Ξ , which is positive for an agent that integrates, zero for one that stops by the clock or the deadline, and negative for one whose stops follow useful evidence instead. Our main findings are: β’ Agents judge correctly but do not act on their judgments (Β§3). Every model whose judgments we recorded calls a failing sourceβs results useless 97β100% of the time, and the judgments the open models state in their own reasoning largely agree. Yet their stopping ignores these judgments. Every unaided open model has a negative Ξ , and the 7β8B models rarely answer even where their own beliefs favor answering. β’ Telling agents more changes when they stop but not what they stop on (Β§4). Permission to answer from memory and a reasoning mode (Β§7) can bring early stops regardless of the evidence. A stated budget moves the 7β8B modelsβ stops to the deadline, which shifts whenever the budget does. A stopping rule or call cost written into the prompt is followed at most partly, and showing agents their running count of useless judgments does not make them stop on it. β’ An enforced integration step makes stopping follow the evidence (Β§5). We let the harness apply the rule, so that only the finish action remains after five consecutive results the agent judged useless. Success on a failing source then rises for every model, and the stopping point stays put when the budget doubles. For agents that state their judgments in their reasoning, the harness can read the judgments there instead of asking for them. β’ The pattern replicates and is not specific to 7β8B models (Β§6, Β§7). It holds with Ξ as the pre-registered primary endpoint on 300 fresh questions and transfers to fact verification. A larger open model, a reasoning mode and an RL-trained search agent still largely fail to stop on the evidence, and only Claude Sonnet 5 partly does so without being prompted. 2 Setting To study this decision, we need to know at every step whether the latest result helped and whether the source will recover. Real tools reveal neither, so we inject failures into question answering over small knowledge bases in which both are known. Tasks and tools. Questions come from the HotpotQA distractor development set (Yang et al., 2018). Each question has its own knowledge base of the ten paragraphs HotpotQA provides. Two of them hold the supporting facts, so we know whether each result is useful. The agent follows the ReAct format (Yao et al., 2023) with search (title match, then word overlap), lookup (next sentence with a keyword) and finish. A budget of eight actions is enough to answer a two-hop question after several failed searches. An answer is correct at token F1 β₯0.6β₯ 0.6. The base prompt states neither the budget nor that an unanswered question counts as wrong. Because this alone could make continued search reasonable, Β§4 tests prompts that state both. For transfer, we use 292 balanced FEVER claims (Thorne et al., 2018), where a random guess is right half the time. Each claimβs knowledge base holds its evidence pages and eight from other claims. Source failures. A failed observation is replaced by a paragraph from another questionβs knowledge base. Such a paragraph is almost always useless, so we can score the agentβs judgments. Section 6 tests harder, on-topic failures. Six regimes vary when failure happens, so that neither always leaving nor never leaving is right: clean (no failure), persistent (every observation fails), recover_after_c (the first cβ1,2,3cβ\1,2,3\ fail, which rewards persistence) and late_onset_from_3 (failure from the third observation on, which separates the length of a failure run from elapsed time). Success is averaged over the six with equal weight (mean6). Judgments, beliefs and actions. At the decision point after the t-th result, we record three things. The first is the agentβs judgment of the latest result. Agents often state it in their next thought, though not always explicitly. To obtain a judgment at every step without changing the trajectory, we also ask a one-word question (USEFUL or USELESS) on a scratch copy of the conversation that the agent never sees again (the side channel). When a condition acts on these judgments, they are recorded during the run; otherwise they are collected afterwards by replaying the recorded trajectories (Claude Haiku 4.5βs come from its enforced-rule run). The second is its beliefs. On another scratch copy, the agent reports its probability of success if it continues and if it answers now. They let us test whether the agent searches on because it thinks continuing pays, and we compare them with the realized success of the recorded trajectory and of an answer-now branch replayed from the same history. The third is its action. Figure 2: What Ξ compares at a fixed step. The enforced rule answers in (a) and continues in (b). Measuring what stopping responds to. If an agent integrates its judgments, its probability of answering should rise with the length of the current run of results it has judged useless. The obvious test is our pre-registered integration index I=hβ‘(4,5)βhβ‘(1,2)I=h(\4,5\)-h(\1,2\), which compares the probability h of answering after four or five consecutive failed results with that after one or two. But when the source fails from the start, the run grows with time. An agent that merely stops later also scores I>0I>0, and a simulated clock-driven agent reaches +0.5+0.5. We therefore compare histories at the same step (Figure 2). The time-matched contrast Ξ is the probability of answering when every result so far was judged useless (a), minus that when the latest result was judged useless but an earlier one useful (b). We pool it over decisions after three to six results with the MantelβHaenszel risk difference and exclude the final action, because it is the last chance to answer (Appendix C.1). Condition What changes Who stops none base prompt agent permit may answer from memory agent budget permit + budget, counter agent stated rule permit + rule in words agent call cost budget + price per call agent decide running count shown agent enforced rule only finish after the run harness combo budget + enforced rule harness Measure Based on Role Ξ own judgments primary design-label contrast failure onset (design) fallback integration index I failure run (design) confounded hazard slope own judgments exploratory Table 1: Conditions (top) and stopping measures (bottom). Both the stated and the enforced rule prescribe answering after five consecutive results judged useless. Ξ and I are pre-registered; prompts in Appendix B.1. Reading Ξ . History (b) is the natural comparison because its latest result was also judged useless, so the two histories differ in how much useless evidence has accumulated. An agent that holds useful evidence may also reasonably answer, which pushes Ξ down. Hence Ξ is zero for stopping by time or the deadline, positive for stopping on the useless run, and negative when stopping follows earlier useful evidence (Figure 2). A hazard model that separates the useless run from earlier useful evidence confirms this reading (Β§3), and on simulated agents Ξ separates time-, deadline- and evidence-driven policies as intended (Appendix C.1). Where judgments are unavailable, the design-label contrast makes the same comparison between a source failing from the start and one failing from the third result. Conditions. Each condition adds one element to a simpler one (Table 1). Four tell the agent more: permission, the budget (eight actions, a counter, unanswered counts as wrong), a price of 0.05 per call against 1 per correct answer, or the stopping rule in words (stated rule). The decide condition shows the agent its running count but leaves the choice to it. The enforced rule instead lets the harness act after k=5k=5 consecutive useless judgments (k set on development data). Two controls replace these judgments with random or lexical signals. Models and protocol. We run Qwen2.5-7B, Llama-3.1-8B, Qwen3-8B and Qwen3-32B (Qwen Team, 2024; Grattafiori et al., 2024; Yang et al., 2025) locally from their official weights with vLLM (Kwon et al., 2023). All four run without thinking, using the Qwen3 modelsβ recommended sampling and greedy decoding for the others. Section 7 adds a reasoning mode and a search agent trained with reinforcement learning. To limit cost, the Claude models run through the official API in fewer conditions. Both run unaided and with the enforced rule (Claude Sonnet 5 on 100 questions), and Claude Haiku 4.5 also runs with the rule or the cost stated on 200 questions. Every threshold was set on 100 development questions; confirmatory results use a disjoint set of 300 (test300) and the FEVER claims. All hypotheses were registered before the corresponding held-out runs. The amendments that define Ξ followed earlier results on the same questions, so we re-test their predictions on 300 questions never used before (fresh300). Appendix Table 3 marks which claims rest on pre-registered tests and which on exploratory analyses. Results are raw unless marked repaired, and Appendix A lists the deviations. Differences are paired by question with 2,000 bootstrap resamples, and tests across models are Holm-corrected. Measurement validity. Stated judgments have to be read from free text, so two LLM annotators from different model families labeled them, with author adjudication (binary ΞΊ 0.90; Appendix B.2). All other results rest on design labels, side-channel judgments or deterministic action parsing, which had no errors in a 100-point audit. Some answers copied the template finish[answer]; we repaired them with one fixed procedure, and the main conclusions do not change (Appendix C.5). 3 Agents judge correctly but do not act on their judgments Continued search would be reasonable if the agent misjudged the results or believed that continuing pays. We test both explanations and then ask whether actions follow the judgments. Judgments are accurate. Agents do recognize useless results. Through the side channel, every model whose judgments we recorded calls a failing sourceβs results useless 97β100% of the time on HotpotQA (Figure 1) and FEVER, and the judgments Qwen3-8B states in its own reasoning are about as accurate (Appendix B.2). This is not a bias toward βuselessβ, since models call a gold paragraph useful 77β98% of the time when it is first retrieved, except Qwen2.5-7B (57%; Appendix C.3). Beliefs alone do not explain it. A second explanation is that agents know the results are useless but believe that continuing still pays. On a failing source, Qwen3-8B does overestimate the value of continuing, but Qwen2.5-7B overestimates that of answering now and Llama-3.1-8B underestimates it (descriptive probe, Appendix C.4). Beliefs thus point in different directions across models, and actions do not follow them. Where their own reports favor answering now, Llama-3.1-8B answers at only 32 of 595 decision points and Qwen2.5-7B at 267 of 2,443 (persistent and plausible regimes; Β§6). Actions do not follow the judgments. After judging five consecutive results useless, unaided Qwen3-8B answers on only 1 of 292 questions, and the other models in Figure 1 answer on at most 7%. Otherwise they search until the budget runs out. Stopping there would have paid, since answering at that point succeeds 15β17% of the time and continuing at most 7% (7β8B models, exploratory; Appendix D.2). Stopping also does not become more likely as useless judgments accumulate, as the negative Ξ of every open model shows (Figure 3). A negative Ξ could also mean answering after useful evidence, so we fit a discrete-time hazard model that separates the two (exploratory; Appendix C.2). In this model, each further useless judgment changes the unaided 7β8B agentsβ probability of answering by β0.02-0.02 to β0.01-0.01, against +0.15+0.15 to +0.17+0.17 under the enforced rule. The same holds for judgments stated in reasoning. The judgments the open models state in their reasoning agree with the side channel on 87β96% of decision points. When Ξ is computed from these stated judgments instead (Appendix D.4), it stays below zero, and after five stated useless judgments in a row the agents answer on 0β0.4% of questions (Qwen3-32B: 6%). 4 Telling agents more changes when they stop but not what they stop on Figure 3: Only when the harness stops (blue labels) does stopping follow the agentβs judgments for every model. Cells show Ξ from the agentsβ own judgments in test300 (blue >0>0, red <0<0), with the fresh300 replication in parentheses. Decide is exploratory; intervals are in Appendix C.1. If agents keep searching because they lack some information, providing it should make them stop on the evidence. We test the candidates one at a time, mainly on the 7β8B models (Table 2, Figure 3). Condition Qwen2.5 7B Llama-3.1 8B Qwen3 8B Qwen3 32B none .192 .263 .448 .509 permit .213 .269 .423 .517 budget .328 .312 .464 .581 stated rule .206 .267 .439 .526 call cost .326 .317 .458 .575 decide .360 .266 .426 β enforced rule .237 .328 .474 .572 combo .375 .337 .478 .578 Agent none enforced rule Claude Haiku 4.5 .501 .598 Claude Sonnet 5β .632 .653 Search-R1β‘ .403 .427 Table 2: The budget condition matches or exceeds the enforced ruleβs success on several models, so success alone cannot show what stopping responds to. Mean6 success (test300, raw; bold: best per model; β: not run). Bottom: agents run only unaided and with the enforced rule (β first 100 questions; β‘RL-trained, own prompt). Intervals and per-regime results: Appendix D.1. Permission yields indiscriminate stopping. Perhaps agents believe they may not answer without evidence. Telling them they βmay also answer from [their] own knowledgeβ leaves Llama-3.1-8B and Qwen2.5-7B almost unchanged. Permission does make Qwen3-8B stop earlier, but its stops ignore the evidence. The model answers before any real result on 55% of questions whose source recovers after three failures, and its Ξ drops to β0.32-0.32. A stated budget yields stopping at the deadline. Or agents may not know that steps are limited and that an unanswered question counts as wrong. Stating the budget with a counter does make the 7β8B models answer, but on the last action (51β79% of answers on a failing source), and their Ξ stays negative. This condition matches the enforced ruleβs success for Qwen3-8B and Llama-3.1-8B and exceeds it for Qwen2.5-7B, so success alone cannot tell the two apart. A change of deadline separates them. With the budget doubled, budget-cued agents move their answers from the eighth to the sixteenth action and roughly double their tool calls on a failing source without any gain in success (Figure 4). Stating the rule or the cost is not enough. If agents simply did not know which stopping rule to use or what a call costs, writing these into the prompt should work. With the rule stated in words, only Llama-3.1-8B and Qwen3-32B follow it, and only partly (Ξ=+0.09 =+0.09 and +0.08+0.08, against +0.37+0.37 and +0.33+0.33 when enforced). Qwen2.5-7B and Qwen3-8B answer early whatever they judged, and no model reaches the enforced ruleβs success. With a price of 0.05 per call added to the budget, the 7β8B models still answer at the deadline and do no better than with the budget alone; only Qwen3-32B stops partly on its judgments. Claude Haiku 4.5 behaves similarly. With the rule stated, its stopping follows the evidence only partly. With the price stated, it answers on almost every question regardless of the evidence and even rivals the enforced ruleβs success (Appendix D.3). Writing the rule or the price into the prompt thus supplies the integration step only partly. Figure 4: Budget-cued 7β8B models answer at whatever the deadline is, whereas the enforced rule and the combination answer at the same action under both budgets. Bars: median action at which agents answer on a persistently failing source (among answered questions) with an 8-action (light) and a 16-action (dark) budget. Qwen2.5-7B answers early under the enforced rule because even unaided it stops after one useless result. A backup tool does not make leaving evidence-driven. A different reason to stay is that leaving means giving up. We therefore added a second tool, backup_search, that searches an intact copy of the knowledge base. The prompt describes it as slower and costlier and suggests it when search is not useful (Appendix D.5). When the primary search fails throughout, Qwen3-8B and Qwen3-32B reasonably switch at the first useless result. By the fourth action, however, Llama-3.1-8B and Qwen2.5-7B have switched on only 17% and 49% of questions, and Llama-3.1-8Bβs success falls to 0.05 (0.50 with a working primary). Yet no unaided agent becomes more likely to leave as useless results accumulate (Ξβ€+0.02 β€+0.02), whereas disabling the failing tool after five useless judgments makes leaving evidence-driven (Ξ=+0.26 =+0.26 and +0.30+0.30 for Llama-3.1-8B and Qwen3-8B; Appendix D.5). Even the running count is not enough. Agents might also judge each result correctly but lose count. The decide condition therefore writes the agentβs judgment and its running count of useless judgments into the context at every step and asks it to weigh continuing against answering now. Agents still do not stop on the count, and their Ξ is β0.14-0.14 to β0.24-0.24 for the three 7β8B models (exploratory; Figure 3). Only Qwen2.5-7B gains appreciably on mean6 and even beats the enforced rule, but it does so by stopping early indiscriminately. 5 Enforced integration makes stopping follow the evidence The agents judge correctly, and neither information nor the running count makes the 7β8B models stop on their judgments. This leaves the step that turns accumulated judgments into an action, which we test directly by letting the harness take it. An enforced integration step anchors stopping to the evidence. The enforced ruleβs positive Ξ (+0.33+0.33 to +0.40+0.40) is expected by construction. The informative questions are whether this step alone raises success and whether it frees the stopping point from the deadline. Mean6 rises for the four open models and Claude Haiku 4.5 (+0.026+0.026 to +0.097+0.097, Holm-significant). Where the source recovers or never fails, the rule is non-inferior to the unaided agent in 18 of 20 tests (margin β0.05-0.05; Qwen3-8B fails narrowly twice). With the budget doubled, the enforced rule answers at the same action as before (Figure 4). The gain comes from acting on an accurate signal. Against a random signal that says βuselessβ equally often but at random steps, the enforced rule wins by 0.0110.011 to 0.0290.029 (Holm-significant), so the timing of the signal matters. A cheap lexical detector that calls a result useless when it shares few content words with the question does about as well as the agent (exploratory). Deadline and integration combine. The budget prevents premature answers and the enforced rule supplies the evidence-driven stop, so together they should keep both benefits. The combination is indeed non-inferior to both components on every model and task (14 of 14 pre-registered tests), and it beats the enforced rule by +0.138+0.138 for Qwen2.5-7B, whose main failure is answering too early. How good is the stopping point? Against the best fixed cap or run-length policy chosen in hindsight, unaided agents lose 6β13 points of success (exploratory; Appendix D.2), and with a doubled budget the budget-cued 7β8B Qwen models make 61β71% more calls for no gain. The enforced rule also makes three to five judgment calls per question, which would put it below the budget condition if they were charged like tool calls. These calls can be avoided by reading the judgment from the agentβs own thought with a keyword reader, which keeps success within 0.01 of the enforced rule for Llama-3.1-8B and the Qwen3 models. Combined with the budget statement, the reader beats the budget condition alone at 0.05 per call for Llama-3.1-8B and Qwen3-8B (Appendix D.3). The reader does not help Qwen2.5-7B, which rarely states a judgment. 6 The pattern replicates on fresh questions, a second task and harder failures The predictions hold on fresh questions. Because Ξ was defined after earlier results were known, we re-tested its predictions on 300 fresh questions, with Ξ registered as the primary endpoint. All nineteen tests pass (Figure 3, in parentheses; Appendix D.4). Again, Ξ stays below zero for the unaided and budget-cued agents and reaches +0.30+0.30 to +0.35+0.35 under the enforced rule, which beats the unaided agent on mean6 (+0.030+0.030 to +0.061+0.061). After five useless judgments, the unaided 7β8B agents answer on at most 3% of questions. On fact verification the effects are larger. On FEVER a random guess is right half the time, so an agent that stops and answers from memory gains more. With every parameter carried over unchanged, the pattern replicates. Unaided agents answer after five useless judgments on at most 3% of claims, and the enforced rule raises mean6 by +0.129+0.129 to +0.201+0.201. The budget condition again answers mostly on the last action (69β96%; F-H10 in Appendix A). Harder failures weaken judgments but not the dissociation. Unrelated paragraphs are easy to recognize, whereas real failures are often on topic. When the failed observations are the questionβs own distractor paragraphs (plausible) or its own pages with every supporting fact removed (answerless), judgments are less accurate (84β97% and 76β97% useless), but the dissociation remains. After five useless judgments, the unaided agents answer on 2β23% and 5β21% of questions. The enforced rule still raises success in both, although on the answerless source the budget condition does better (Appendices C.1, D.5). Two further runs with new seeds and failure draws reproduce the main contrasts (Appendix D.6). 7 Larger, reasoning and search-trained agents rarely integrate either The 7β8B models might lack the capacity to integrate, so we test a larger open model, a frontier model, a reasoning mode and a search-trained agent. Left alone, larger models still rarely integrate. Qwen3-32B judges failing results as accurately as the smaller models, yet answers after five such judgments on only 19 of 284 questions. Claude Sonnet 5 is the only unaided model whose stopping partly follows the evidence (design-label contrast +0.12+0.12 [+0.05,+0.18+0.05,+0.18], exploratory). Still, unaided Claude Sonnet 5 exhausted its budget on 45% of the questions where its enforced rule fired. The enforced rule raises its success on a failing source from 0.40 to 0.51, although on 100 questions the gain over six regimes is not significant (+0.022+0.022, p=0.051p=0.051). A budget cue makes a larger model stop earlier, but mostly by time. With its budget stated, Qwen3-32B answers around the sixth action and matches the enforced ruleβs success. Its integration index rises to +0.32+0.32, so our pre-registered prediction that it would stay near zero failed. The time-matched contrast shows what drives this rise. At the same step, Qwen3-32B answers as often after earlier evidence it judged useful as after a run it judged useless (Ξ=0.00 =0.00 [β0.06,+0.05-0.06,+0.05]), and with the budget doubled it answers a step later and makes 46% more tool calls for no gain. Training to search does not teach integration. Training to search might teach an agent when to stop, but it does not for Search-R1 (Jin et al., 2025), which was trained with PPO on HotpotQA and NQ and runs here with its own prompt. On a failing source it answers from memory after one or two searches on 27% of questions but searches to the end of its budget on 45%. At the same step, it answers no more often after a longer failure (design-label contrast β0.03-0.03 [β0.07,+0.01-0.07,+0.01]). Because its judgments also call most real results useless (89%), we do not interpret its Ξ ; the enforced rule still raises its mean6 by +0.024+0.024 (Appendix D.4). A reasoning mode makes agents stop earlier regardless of the evidence. Thinking might let an agent integrate within its reasoning. Instead, in thinking mode Qwen3-32B answers from memory on 97% of failing-source questions (11% without thinking). Where the source recovers after two failures, it answers before any real result on 47% of questions (Ξ=+0.01 =+0.01 [β0.18,+0.17-0.18,+0.17]). Qwen3-8B in thinking mode likewise stops no more often after a longer failure. Some of Qwen3-32Bβs stops may follow the evidence only when its budget is stated (design-label contrast +0.32+0.32, from few late decisions; Appendix D.4). 8 Related work Agents and unreliable tools. Agents carry wrong tool output into their answers even when their reasoning notices the conflict (Yang et al., 2026), mostly fail to diagnose recoverable tool faults (Tian et al., 2026), and can be hurt by misleading feedback (Ming et al., 2025). We ask instead whether they stop relying on a source that returns nothing useful. When to stop searching. Search-augmented models over-search (Xie et al., 2026; Liu et al., 2026a). Remedies train the search/answer boundary (Zhang et al., 2026a), verify sufficiency (Roh and Han, 2026), add a controller (Soudani et al., 2026) or guard against unsupported stops (Liu, 2026). Adaptive retrieval decides when to retrieve (Asai et al., 2024; Jiang et al., 2023) or corrects poor retrievals with a learned evaluator (Yan et al., 2024), and RL-trained search agents learn when to stop (Jin et al., 2025). These methods build the decision into the system, whereas we measure whether agents act on their own judgment that a source keeps failing. Abstention and recovery. Benchmarks ask whether agents know when to abstain (Luo et al., 2026) or not to act (Liu et al., 2026b). Luo et al. (2026) use a fixed search budget, which we find sets a deadline. Chen et al. (2026) learn fallback strategies with reinforcement learning, whereas our rule uses only the agentβs own judgments. Knowing and doing. Models largely know what they know (Kadavath et al., 2022). A gap between judging and doing has nonetheless been shown for the decision to call a tool at all (Li et al., 2026), and making the judgment explicit reduces it (Cheng et al., 2026). The gap also appears in bandit tasks, where models state the right strategy but act greedily (Schmied et al., 2026). In our setting, an explicit judgment does not lead to stopping even when its running count is shown. 9 Discussion and conclusion Agents recognize useless results, but their beliefs, what they are told and the running count all fail to make the 7β8B models stop on them, apart from Llama-3.1-8Bβs partial following of the stated rule. The missing piece is integration. In the spirit of sequential testing (Wald, 1945), the enforced rule adds a run-length test that maps the count to an action. The step matters even where success does not show its absence. With a known deadline and free calls, answering at the deadline does not reduce success, but it spends every call and shifts whenever the budget does. Agent designs should therefore make this step explicit rather than expect it from prompting. Outcome-reward reinforcement learning did not teach it to Search-R1 either (Β§7). The step is also cheap, because the harness can read the judgment in the agentβs reasoning instead of asking for it. Evaluations should not infer evidence-driven stopping from success, which a deadline also yields. They should instead test whether stopping at a fixed step rises with the run of results the agent judged useless. An agentβs stated judgment of its evidence does not guarantee that it acts on it, so the two must be measured separately. Limitations. We simulate failures by replacing observations in fixed knowledge bases. Real tools also fail by returning partial, stale or erroneous content, and whether the pattern holds under such natural failures is untested, although on-topic failures leave the dissociation intact (Β§6). Our sources recover after at most three failures, and the ruleβs threshold was chosen for that distribution. When recovery comes later, the rule can give up just before it (Appendix D.5), so the threshold would need tuning for real sources. Most controls ran on open models, and the two Claude models received fewer conditions and questions. Judgments are elicited on a copy of the conversation and beliefs are self-reports, so we compare them only within a model. Stated judgments were labeled by LLM annotators with author adjudication rather than by independent human annotators. Answering from memory is not automatically safe, since parametric knowledge can be stale or wrong. Our results therefore argue for making the decision to abandon a source explicit and evaluating it, rather than for abandoning tools. References Asai et al. [2024] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. In The Twelfth International Conference on Learning Representations, 2024. Chen et al. [2026] Chaoran Chen, Vy Nguyen, Ziji Zhang, Abhinav Gullapalli, Ziyi Wang, Yuxuan Lu, Dakuo Wang, Jing Huang, Zhou Yu, and Jin Lai. Retry, Switch, or Abstain? Learning Strategy-Aware Tool-Use Policies via Controlled Error Injection, 2026. URL https://arxiv.org/abs/2608.11977. Cheng et al. [2026] Yize Cheng, Chenrui Fan, Mahdi JafariRaviz, Keivan Rezaei, and Soheil Feizi. Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use, 2026. URL https://arxiv.org/abs/2605.14038. Grattafiori et al. [2024] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. The Llama 3 Herd of Models, 2024. URL https://arxiv.org/abs/2407.21783. Jiang et al. [2023] Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active Retrieval Augmented Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969β7992, 2023. Jin et al. [2025] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. In Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=Rwhi91ideu. Kadavath et al. [2022] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, et al. Language Models (Mostly) Know What They Know, 2022. URL https://arxiv.org/abs/2207.05221. Kwon et al. [2023] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611β626, 2023. Li et al. [2026] Yifan Li, Shengbin Yue, Boyu Feng, Jinhu Qi, Bo Ke, Zixing Song, Hongru Wang, Zhongyu Wei, and Irwin King. From Knowing to Acting: Benchmarking Self-Awareness Capability of LLM Agents, 2026. URL https://arxiv.org/abs/2606.20661. Liu [2026] Jason Liu. When May an Agent Stop? Evidence-Carrying Termination for Tool-Using LLMs, 2026. URL https://arxiv.org/abs/2608.23623. Liu et al. [2026a] Qi Liu, Jiaxin Mao, Fengbin Zhu, and Tat-Seng Chua. Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents, 2026a. URL https://arxiv.org/abs/2608.01913. Liu et al. [2026b] Xun Liu, Yi Evie Zhang, Vira Kasprova, Parisa Rabbani, Pardis Sadat Zahraei, Tianyu Zhang, Ali Ebrahimpour-Boroojeny, and Varun Chandrasekaran. AgentAbstain: Do LLM Agents Know When Not to Act?, 2026b. URL https://arxiv.org/abs/2607.10059. Luo et al. [2026] Han Luo, Bingbing Wen, and Lucy Lu Wang. Agentic Abstention: Do Agents Know When to Stop Instead of Act?, 2026. URL https://arxiv.org/abs/2606.28733. Ming et al. [2025] Yifei Ming, Zixuan Ke, Xuan-Phi Nguyen, Jiayu Wang, and Shafiq Joty. Helpful Agent Meets Deceptive Judge: Understanding Vulnerabilities in Agentic Workflows, 2025. URL https://arxiv.org/abs/2506.03332. Qwen Team [2024] Qwen Team. Qwen2.5 Technical Report, 2024. URL https://arxiv.org/abs/2412.15115. Roh and Han [2026] Daeyoung Roh and Donghee Han. HALT: Verification-Aware Stopping for Retrieval-Augmented Search Agents, 2026. URL https://arxiv.org/abs/2608.02009. Schmied et al. [2026] Thomas Schmied, JΓΆrg Bornschein, Jordi Grau-Moya, Markus Wulfmeier, and Razvan Pascanu. LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=weUP6H5Ko9. Soudani et al. [2026] Heydar Soudani, Elizabeth Lingg, Faegheh Hasibi, and Navid Rekabsaz. When Deep Research Agents Stagnate: Enhancing Reasoning with Retrieval-Aware Agent Control, 2026. URL https://arxiv.org/abs/2608.15191. Thorne et al. [2018] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 809β819, 2018. URL https://aclanthology.org/N18-1074/. Tian et al. [2026] Yang Tian, Zhengpeng Shi, Yu Zhou, and Bo Zhao. Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability, 2026. URL https://arxiv.org/abs/2606.25819. Wald [1945] Abraham Wald. Sequential Tests of Statistical Hypotheses. The Annals of Mathematical Statistics, 16(2):117β186, 1945. Xie et al. [2026] Roy Xie, Deepak Gopinath, David Qiu, Dong Lin, Haitian Sun, Saloni Potdar, and Bhuwan Dhingra. Over-Searching in Search-Augmented Large Language Models. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7714β7739, 2026. 10.18653/v1/2026.eacl-long.361. URL https://aclanthology.org/2026.eacl-long.361/. Yan et al. [2024] Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. Corrective Retrieval Augmented Generation, 2024. URL https://arxiv.org/abs/2401.15884. Yang et al. [2025] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, et al. Qwen3 Technical Report, 2025. URL https://arxiv.org/abs/2505.09388. Yang et al. [2026] Hoyeol Yang, Woojung Song, Taewon Kim, Jonghyun Song, Seoyeon Park, and Yohan Jo. Agentsβ Overreliance on Unreliable Tools, 2026. URL https://arxiv.org/abs/2609.05587. Yang et al. [2018] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369β2380, 2018. URL https://aclanthology.org/D18-1259/. Yao et al. [2023] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=WE_vluYUL-X. Zhang et al. [2026a] Wenlin Zhang, Kuicai Dong, Junyi Li, Yingyi Zhang, Xiaopeng Li, et al. To Search or Not to Search: Aligning the Decision Boundary of Deep Search Agents via Causal Intervention. In Proceedings of the ACM Web Conference 2026, pages 2049β2059, 2026a. 10.1145/3774904.3792235. Zhang et al. [2026b] Xiangxin Zhang, Zhanwei Zhang, Zhihang Fu, Binbin Lin, and Wenxiao Wang. From Inertia to Objectivity: Improving Deep Research Agents with Noise Isolation, 2026b. URL https://arxiv.org/abs/2608.23045. Appendix Table 3 maps each claim of the main text to the pre-registered tests that bear on it (Appendix A) and to the exploratory analyses, with the appendix where each is reported. Appendix tables use short condition names (rule = enforced rule, policy or stated = stated rule, cost or stated cost = call cost, think = reasoning mode), short regime names (pers. = persistent, reccc = recover_after_c, late = late_onset_from_3) and the test IDs of the pre-registration. Claim (main text) Pre-registered tests (outcome) Exploratory Appendix Agents judge a failing sourceβs results useless (Β§3) 16b: stated and side-channel judgments agree (pass) judgment rates; specificity on real results; accuracy of stated judgments B.2, C.3, D.4 Unaided agents rarely answer after five useless judgments (Β§3) J1 (reported); F5, fresh300 (pass 3/3); T2, stated judgments (pass 2/2); R3, Search-R1 (fail) whether stopping would have paid C.1, D.4, D.2 Without enforcement, stopping does not follow the useless run (Β§3β4) J2: none, permit, budget (pass 12/12), stated rule (fail for Llama-3.1-8B, Qwen3-32B); C, call cost (pass 3/4); P1, stated rule (partial for Llama-3.1-8B only); H1 and H3 of A20, Claude Haiku 4.5 (partial; not on the evidence); F1βF2, fresh300 (pass 8/8); T1 of A18, stated judgments (pass 4/4) hazard model; decide C.1, C.2, D.4, D.3 A stated budget moves stopping to the deadline (Β§4) H9, H10, F-H9, F-H10, D1, D3, D4, Rp3 (pass); L-H9 (fail) and A9 (mixed), Qwen3-32B D.1, D.6 The enforced rule raises success (Β§5) H1, F-H1, L-H1, Rp1, F4, R2, S-H2 (pass); S-H1, Claude Sonnet 5 (fail, p=.051p=.051) D.1, B.4, D.4, D.6 without loss where the source recovers H3 (14/16; 18/20 with Qwen3-32Bβs four under L-H1), F-H3 (12/12); 13c, recovery after four or five failures (mixed) D.1, D.5 because it acts on an accurate signal H2, F-H2 (pass) lexical detector D.1 with a stopping point that does not follow the budget D2 (fail for Qwen2.5-7B); A9x, Qwen3-32B (pass) D.1 Deadline and integration combine (Β§5) C1βC4, L-C1, L-C3 (pass); Rp4 (5/6) D.1, D.6 How good the stopping point is (Β§5) D3, C4: tool calls (pass) best simple policy; cost D.2 with no judgment call, read from the agentβs reasoning E1βE3 of A19 (pass on the 3 applicable models; Qwen2.5-7B not applicable), E4 (2/3) D.3 The pattern replicates (Β§6) fresh300: F1βF5 (pass); FEVER: F-H1βF-H10 (pass); second and third runs: Rp1βRp4, 13d (Rp2 fails for Qwen2.5-7B; Rp4 5/6 in the second run) plausible and answerless sources D.4, D.1, D.6, C.1, D.5 Larger, reasoning and search-trained agents (Β§7) L-H1, L-H7, L-C1, L-C3 (pass), L-H9 (fail); T1 of A15, reasoning mode (pass); V, Search-R1 judgments (fail), R2 (pass), R1 (formally pass, not interpreted), R3 (fail) Claude Sonnet 5 design-label contrast D.1, B.4, D.4 Table 3: Where each claim of the main text is tested. Test IDs and decision rules as in Tables 4 and 5; exploratory analyses were not pre-registered. Appendix A Pre-registration, amendments and deviations Protocol. The pre-registration was written before any held-out run (test300 had seen only a three-question pipeline check). All thresholds were set on the 100 development questions. k=5k=5 maximized development mean6 for Qwen3-8B and Llama-3.1-8B (0.405, 0.268), and Qwen2.5-7Bβs development sweep also peaked at k=5k=5 (0.192). The Qwen2.5-7B result was recorded in a results file before its test runs but was not entered in the document as the protocol required, which we count as a deviation. Claude Haiku 4.5βs k=5k=5 came from its own pilot. Differences are paired by question, with 95% intervals from 2,000 bootstrap resamples (clustered by question for I); tests across models are Holm-corrected; non-inferiority margins are β0.05-0.05. The stop rule (withdraw the intervention claim if H1 failed for two of three 7β8B models) did not trigger. Amendments. Each amendment was frozen before the runs it governs; A3βs H9βH10 were added after the promptsβ development runs and A9 after L-H9 had failed. A fair-coin random control almost never yields five βuselessβ in a row, so A1 replaced it by one matched to each modelβs development βuselessβ rate. A1 also set the lexical threshold by balanced accuracy (0.25) instead of accuracy (0.6). A2 made unparseable judgments reset the ruleβs counter (3.4% of Claude Haiku 4.5βs; none for local models). A6 widened C1βs margin from β0.02-0.02 to β0.05-0.05 at freezing. Registration and multiplicity. The pre-registration and its amendments were kept as one append-only document, and each amendment was frozen before the runs it governs. The document was not deposited with a third-party registry. Holm correction is applied across models within each hypothesis, not across the many hypotheses in Tables 4 and 5; we rest the central claims on effects that replicate across models, two further runs and the fresh questions rather than on any single test. Doc ID Prediction; models; decision rule Outcome Orig. H1 Rule >> none (mean6); 7β8B, Claude Haiku 4.5; one-sided, Holm pass 4/4: +.045+.045 / +.064+.064 / +.026+.026; Claude Haiku 4.5 +.097+.097 H2 Rule >> rule fed a rate-matched random signal (A1); 7β8B; one-sided, Holm pass 3/3: +.011+.011 (p=.0285p=.0285) / +.029+.029 / +.021+.021 H3 Rule β₯ none β0.05-0.05 on rec1, rec2, rec3, clean; 7β8B, Claude Haiku 4.5; 95% LB 14/16; fail Qwen3-8B rec1 (LB β.057-.057), rec3 (LB β.060-.060) H4 Decide << rule (mean6); 7β8B; one-sided rule β- decide β.123-.123 / +.062+.062 / +.048+.048: fail for Qwen2.5-7B H5e Rule vs. rule fed a lexical detector; 7β8B +.012+.012 / β.001-.001 / β.011-.011 H6e Thinking vs. rule, incl. rec3; Qwen3-8B +.003+.003 [β.032-.032, +.039+.039]; rec3: .437 vs. rule .470 A3 H7 Rule >> permit (mean6); 7β8B; one-sided, Holm +.023+.023 (p=.055p=.055) / +.059+.059 / +.051+.051: fail for Qwen2.5-7B H8 Rule vs. budget (mean6); 7β8B; two-sided, no direction β.091-.091 [β.124-.124, β.060-.060] / +.016+.016 / +.009+.009 H9 I upper bound <+0.10<+0.10 for permit and for budget; 7β8B pass 6/6 (largest bound +.064+.064) H10 Budget: over half of persistent answers at action 8; 7β8B pass 3/3: .749 / .791 / .506 A4 F-H1 As H1 on FEVER; 7β8B pass 3/3: +.129+.129 / +.201+.201 / +.142+.142 F-H2 As H2 on FEVER; 7β8B pass 3/3: +.044+.044 / +.123+.123 / +.096+.096 F-H3 As H3 on FEVER; 7β8B pass 12/12 F-H9 As H9 on FEVER; 7β8B pass 6/6 (largest bound +.064+.064) F-H10 As H10 on FEVER; 7β8B pass 3/3: .77 / .96 / .69 A5 D1 16 actions, budget: median answering action on persistent β₯12β₯ 12; 7β8B pass 3/3: 16 / 16 / 16 D2 16 actions, rule: median answering action on persistent stays 6; 7β8B 2 / 6 / 6: fail for Qwen2.5-7B (6 on its 107 fired questions) D3 16 actions: budget makes β₯4β₯ 4 more tool calls than rule on persistent; 7β8B pass 3/3: +10.61+10.61 / +9.23+9.23 / +6.53+6.53 D4 16 actions, budget: I upper bound <+0.10<+0.10 (last action excluded); 7β8B pass 3/3: UB β.012-.012 / +.012+.012 / β.027-.027 (index β.027-.027 / +.003+.003 / β.052-.052) A6 C1 Combo β₯ rule β0.05-0.05 and β₯ budget β0.05-0.05, HotpotQA and FEVER; 7β8B pass 12/12 (lowest LB β.023-.023) C2 Combo >> rule; Qwen2.5-7B; one-sided, each task pass: +.138+.138 [+.104+.104, +.173+.173]; FEVER +.088+.088 [+.059+.059, +.118+.118] C3 Combo: I lower bound >+0.10>+0.10 (HotpotQA, FEVER, 16 actions); 7β8B pass 9/9 (lowest bound +.391+.391) C4 16 actions: combo β₯4β₯ 4 fewer tool calls than budget, median answer β€7β€ 7; 7β8B pass 3/3: 8.66 / 9.41 / 7.62 fewer; median 6 A7 S-H1 Rule >> none (mean6); Claude Sonnet 5, first 100 questions; one-sided fail: +.022+.022 [β.003-.003, +.052+.052], p=.051p=.051 S-H2 Rule >> none on persistent; Claude Sonnet 5; one-sided pass: +.11+.11 [+.05+.05, +.18+.18] A8 L-H1 As H1; Qwen3-32B pass: +.063+.063 [+.045+.045, +.080+.080]; rec1β3, clean: LB β₯β.043β‘β₯-.043 L-H7 As H7; Qwen3-32B pass: +.055+.055 [+.026+.026, +.084+.084] rev. L-H9 As H9; Qwen3-32B fail 2/2: +.172+.172 [+.108+.108, +.234+.234], +.317+.317 [+.277+.277, +.358+.358] L-C1 As C1 (HotpotQA); Qwen3-32B pass 2/2: LB β.022-.022 (vs. rule), β.019-.019 (vs. budget) L-C3 As C3 (HotpotQA); Qwen3-32B pass: +.525+.525 [+.497+.497, +.554+.554] A9 A9 16 actions, budget: evidence-driven (median β€8β€ 8, last-action share <.2<.2, I LB >+.10>+.10), deadline-driven (median β₯12β₯ 12 or share β₯.5β₯.5), else mixed; Qwen3-32B mixed: median 7, share .01, but I +.103+.103 [+.064+.064, +.144+.144] A9x 16 actions, rule and combo: median β€7β€ 7, tool calls within 1 of 8 actions; Qwen3-32B pass 2/2: median 6, 6; calls 4.99β 4.97, 4.19β 4.28 Table 4: Every pre-registered test, part 1 (original pre-registration and amendments A3βA9); notation as in Table 5. Doc ID Prediction; models; decision rule Outcome A10 Rp1 Replicate (new seeds, new draw of failed observations): rule >> none (mean6); 7β8B; one-sided, Holm pass 3/3: +.048+.048 / +.061+.061 / +.042+.042 Rp2 Replicate: rule I LB >+.30>+.30; none I β€+.05β€+.05; 7β8B rule I .19 [.14, .24] / .46 / .46: fail for Qwen2.5-7B; none β.31-.31 / .00.00 / .00.00: pass Rp3 Replicate: budget armβs median answering action on persistent =8=8 and I β€+.10β€+.10; 7β8B pass 3/3: median 8 / 8 / 8; I β.01-.01 / +.04+.04 / β.02-.02 Rp4 Replicate: combo β₯ rule β0.05-0.05 and β₯ budget β0.05-0.05 (95% LB); 7β8B 5/6: vs. rule +.108+.108 / +.007+.007 / β.024-.024 [LB β.054-.054], fail for Qwen3-8B (repaired: pass); vs. budget LB +.033+.033 / +.008+.008 / +.003+.003 A11 P1 Stopping rule stated in the prompt: executed on the evidence if Ξ LB >0>0, persistent median β€7β€ 7 and non-inferior to the rule; not executed if Ξ CI includes or lies below 0 or persistent answer rate <.5<.5; else partial; 7β8B, Qwen3-32B not executed / partial / not executed; Qwen3-32B not executed (design-label Ξ β.05-.05 / +.07+.07 / β.10-.10; β.10-.10) P2 Stated rule vs. external rule (non-inferiority, margin β0.05-0.05) and vs. permit (mean6) fail 4/4 vs. rule: β.031-.031 / β.061-.061 / β.034-.034; Qwen3-32B β.047-.047 (LB β.056-.056 / β.089-.089 / β.061-.061 / β.073-.073; all upper bounds <0<0; repaired: Qwen3-8B β.003-.003 [β.026-.026, +.021+.021], others unchanged in sign); vs. permit β.007-.007 / β.002-.002 / +.016+.016; +.008+.008 A12 J1 Same-trajectory dissociation (unaided; own replayed judgments); 7β8B, Qwen3-32B reported: answers on 3/122, 1/282, 1/292; 19/284 J2 Own-judgment Ξ : CI includes or lies below 0 for none, permit, budget, policy, cost; LB >0>0 for rule rule pass 4/4 (+.33+.33 to +.37+.37); none, permit, budget pass 12/12; fail: policy Llama-3.1-8B (+.09+.09), Qwen3-32B (+.08+.08); cost Qwen3-32B (+.09+.09) C Stated cost: Ξ CI includes or lies below 0; combo no longer called cost-optimal if cost armβs cost-adjusted success LB >β0.02>-0.02 vs. combo Ξ : pass 3/4, fail Qwen3-32B; vs. combo β.095-.095 / β.039-.039 / β.041-.041; Qwen3-32B β.003-.003 [β.027-.027, +.020+.020] (LB <β.02<-.02) A13 S1βS3 Two sources: switched by action 4 β₯.5β₯.5 β βswitchesβ; else βdoes not abandonβ if own-judgment Ξ for leaving includes or lies below 0; 7β8B, Qwen3-32B Qwen2.5-7B, Llama-3.1-8B: does not abandon (.49, .17; Ξ β.54-.54, β.06-.06); Qwen3-8B, Qwen3-32B: switches (.73, .95) S4 Switch rule β- none, mean over six two-source regimes Llama-3.1-8B +.054+.054, Qwen3-8B +.016+.016 (both LB >0>0); Qwen2.5-7B +.011+.011, Qwen3-32B +.007+.007 (n.s.) 13c Long recovery: rule << none in rec5; rule β₯ none β0.05-0.05 in rec4 rec5: holds for Qwen3-8B (β.213-.213), Qwen3-32B (β.137-.137), fails for Qwen2.5-7B, Llama-3.1-8B; rec4: 3/4, fail Qwen3-8B (LB β.050-.050) 13d Third run: Rp1βRp4 Rp1 3/3; Rp2 fail Qwen2.5-7B (as before); Rp3 3/3; Rp4 6/6 A14 Both exits named: backup share of first exits >.6>.6 / <.4<.4; 7β8B, Qwen3-32B prefers evidence: Qwen2.5-7B .93, Llama-3.1-8B .93, Qwen3-32B .90; no clear preference: Qwen3-8B .52 A15 T1 Qwen3-32B thinking mode, four regimes: own-judgment Ξ CI includes or lies below 0 for none and budget; else the claim is narrowed pass 2/2: none +.01+.01 [β.18-.18, +.17+.17]; budget +.25+.25 [β.01-.01, +.50+.50] (few late decisions) Table 5: Every pre-registered test (part 2, amendments A10βA20; part 1 in Table 4), by the document that fixed it (IDs and condition names as registered: policy = stated rule, rule or external rule = enforced rule, cost = call cost; Orig.: original pre-registration; A3βA20: amendments; A1 and A2 are described in the text). Test IDs are those of each document; where a document names none (A12βs C, A15βs T1) or its names would clash (A17βs S1βS3 and validity check), we label them here (R1βR3, V). Paired differences in mean6 unless stated, as Qwen2.5-7B / Llama-3.1-8B / Qwen3-8B where three appear; brackets: 95% intervals; LB: lower bound; 7β8B: Qwen2.5-7B, Llama-3.1-8B, Qwen3-8B; reccc: recover_after_c. Raw analysis. eExploratory by design. β‘A8 adopted the local modelsβ criteria without naming H3; the main text counts these four tests together with H3. A9βs predictions are unnamed in the pre-registration. Doc ID Prediction; models; decision rule Outcome A16 F1βF2 fresh300: own-judgment Ξ UB <+.06<+.06 for none and budget; 7β8B (Qwen3-32B if time allows) pass 8/8: none β.09-.09 / β.03-.03 / β.05-.05, Qwen3-32B β.06-.06; budget β.17-.17 / β.14-.14 / β.19-.19, Qwen3-32B β.05-.05 (UB +.01+.01; others <0<0) F3 fresh300: rule Ξ LB >+.20>+.20 pass 4/4: +.31+.31 [LB +.22+.22] / +.35+.35 [+.33+.33] / +.34+.34 [+.31+.31]; Qwen3-32B +.30+.30 [+.26+.26] F4 fresh300: rule >> none (mean6), one-sided, Holm pass 4/4: +.030+.030 / +.061+.061 / +.036+.036; Qwen3-32B +.057+.057 (all p<.001p<.001) F5 fresh300: unaided agent answers on <10%<10\% of questions after five own useless judgments (persistent) pass 3/3: 3/125, 2/283, 0/289 16b Stated vs. side-channel judgments: annotator binary ΞΊβ₯.80ΞΊβ₯.80 on the adjudicated labels to proceed; agreement β₯.85β₯.85 expected ΞΊ .874 [.798, .939]; agreement .87 / .94 / .94; Qwen3-32B .96 A17 V Search-R1: judgment validity, USELESS after failed β₯.80β₯.80 and after real β€.50β€.50, else Ξ only descriptive fail: 1.00 and .89 R1 Search-R1: own-judgment Ξ UB <+.06<+.06 (none) formally pass (β.26-.26 [β.58-.58, β.06-.06]); not interpreted (V) R2 Search-R1: rule >> none (mean6), one-sided pass: +.024+.024 [+.009+.009, +.041+.041] R3 Search-R1: unaided answers on <10%<10\% after five own useless judgments (persistent) fail: 29/165 (.18) A18 T1 Ξ from stated judgments (unaided, test300): UB <+.06<+.06 pass 4/4: β.08-.08 / β.01-.01 / β.04-.04; Qwen3-32B β.02-.02 (UB +.002+.002) T2 Stated judgments: answers on <10%<10\% after five consecutive stated useless judgments (persistent); descriptive below 20 questions and for Qwen3-32B pass 2/2: Llama-3.1-8B 1/266, Qwen3-8B 1/287; Qwen2.5-7B 0/15, Qwen3-32B 16/252 (descriptive) A19 E1 Stated-judgment rule (keyword reader on the agentβs own thought, no judgment call) >> none (mean6); one-sided, Holm pass 3/3 applicable: +.064+.064 / +.036+.036 / +.064+.064 (Llama-3.1-8B, Qwen3-8B, Qwen3-32B); Qwen2.5-7B +.018+.018, not applicable (fires on 0.05 of persistent questions) E2 Stated-judgment rule β₯ enforced rule β0.05-0.05 (mean6, 95% LB) pass 3/3 applicable (lowest LB β.017-.017); Qwen2.5-7B LB β.041-.041 E3 Utility at Ξ»=0.05Ξ»=0.05, all calls counted: stated-judgment rule >> enforced rule; one-sided, Holm pass 3/3 applicable: +.236+.236 / +.196+.196 / +.190+.190; Qwen2.5-7B +.086+.086 E4 Utility at Ξ»=0.05Ξ»=0.05: stated-judgment combination >> budget; one-sided, Holm 2/3 applicable: +.045+.045 / +.022+.022; fail Qwen3-32B (.000.000); Qwen2.5-7B +.027+.027 A20 H1 Claude Haiku 4.5, rule stated in the prompt (200 questions): executed / partial / not executed, by the rule of P1 partial: design-label contrast +.052+.052 [+.021,+.085+.021,+.085], median action 7, vs. enforced rule β.050-.050 [LB β.086-.086] H2 Claude Haiku 4.5, stated rule vs. none and vs. enforced rule (mean6), reported +.043+.043 / β.050-.050 H3 Claude Haiku 4.5, call cost stated: design-label contrast CI includes or lies below 0; utility at Ξ»=0.05Ξ»=0.05 reported pass: +.013+.013 [β.023,+.048-.023,+.048]; mean6 vs. enforced rule +.053+.053; utility vs. none +.154+.154, vs. enforced rule +.213+.213 Table 5: (continued) Amendments A16βA20. Deviation 1 (during the original runs). One trajectory of Qwen2.5-7Bβs unaided arm exceeded the 8,192-token context, so that arm was run with 16,384 tokens and the same greedy decoding; all other Qwen2.5-7B arms stayed within 8,192 tokens, so the limit could not have changed them. This had no effect. Deviation 2 (found after the runs). Some answers copied the template finish[answer]; they were repaired uniformly (Appendix C.5). Deviation 3 (during A5). Qwen2.5-7Bβs 16-action budget arm exceeded 16,384 tokens in recover_after_2; only that regime was rerun, with 32,768 tokens and greedy decoding. This had no effect. Deviation 4 (executing A15). A15βs third arm (stated cost) was not run; A15βs predictions concern only the unaided and budget arms, which ran as planned. Deviation 5 (executing A16). On fresh300, as in deviation 1, some trajectories exceeded the 8,192-token context, so the Qwen3-8B arms, their side-channel replays and the remaining fresh300 runs used 16,384 tokens (Llama-3.1-8Bβs none and budget arms had completed at 8,192). The limit affects only trajectories that would otherwise exceed it. Deviation 6 (executing A15). The Qwen3-32B thinking runs allowed up to 8,192 new tokens per step instead of the registered 2,048; 25 of 7,141 generations exceeded 2,048. Exploratory analyses. Not pre-registered: I in the first runs for the unaided, decide, thinking, random, lexical, call-cost and rule arms (it was pre-registered for the permission, budget, stated-rule and combination arms, as a descriptive quantity for unaided Qwen3-32B and both Claude Sonnet 5 arms, and for the rule and unaided arms of the second run); the combinationβs superiority; FEVER comparisons of the rule with lexical, decide and prompt arms; thinkingβs token cost; beliefs on test300; Claude Sonnet 5βs non-inferiority; and a noise check. Rerunning unaided Qwen3-8B and Llama-3.1-8B with identical settings reproduced them (mean6 difference +0.001+0.001 each), so run-to-run variation arises when an intervention changes vLLMβs batching. In Qwen3-8Bβs failed H3 tests, questions where the rule fired account for a net β2-2 and β5-5 correct answers of 300 (rec1, rec3), questions it never touched for β6-6 and β4-4. Appendix B Setup details B.1 Prompts and signals Agent. All HotpotQA runs use the system prompt below, except Search-R1, which keeps the prompt it was trained with (Appendix D.4); the two-source runs add the backup tool (Appendix D.5). FEVER replaces the task description, the Thought line and the answer line (βSubmit your verdict. The answer must be exactly one of: SUPPORTS, REFUTESβ), opens with βVerify this claim:β, and says βclaimβ/βverdictβ in the judgment, answer-now and action prompts; the budget and cost statements are unchanged. You are a question-answering agent. You answer questions by interacting with a Wikipedia-like knowledge base. Available actions: - search[query]: Search for a Wikipedia article. Returns the article text. - lookup[keyword]: Find the next sentence containing the keyword in the current article. - finish[answer]: Submit your final answer. The answer should be short (a few words). Format your response as: Thought: <your step-by-step reasoning> Action: <action_name> Action Input: <input> Produce exactly one Thought and one Action per turn. End your response after Action Input and wait for the tool observation. Do not simulate tool results or future actions. Prompt arms. permit appends to the system prompt: If the search results do not give you the information you need, you may also answer from your own knowledge: use finish[answer] with your best guess. budget appends the same sentence and You can take at most 8 actions for each question, including finish. If you have not used finish[answer] by then, the question is scored as wrong. (16 in the doubled-budget runs), and adds [Actions remaining: n] to the question and to every observation. The stated rule appends to the permit prompt Stopping policy: if five consecutive search or lookup results in a row are useless for answering the question, stop consulting the knowledge base and answer from your own knowledge with finish[answer]. and cost appends to the budget prompt Scoring: a correct final answer earns 1 point; each search or lookup costs 0.05 points; a wrong or missing answer earns 0. Maximize your total points. Side-channel judgment. Before each action after the first, the history is copied, the copy receives Before acting: was the LAST observation useful for answering the original question? Reply with exactly one word: USEFUL or USELESS. and the reply (at most 8 tokens) is parsed and discarded; the acting history is not changed. A reply that is not an explicit USELESS resets the streak. Rule. When the streak of USELESS judgments reaches k=5k=5, the system prompt receives βFor this turn only, the search and lookup actions are unavailable; the only available action is finish[answer].β and the latest observation is followed by βDo not make any further tool calls. Give your final answer to the original question using finish[answer].β Otherwise the history is untouched. random replaces the judgment by a seeded draw that says USELESS with the modelβs own development-set rate of USELESS judgments; lexical says USELESS when fewer than 25% of the questionβs content words occur in the first 400 characters of the observation (threshold Ο=0.25Ο=0.25, chosen on development labels; balanced accuracy 0.77), or when the tool returns no result. combo applies the budget prompt and the rule together. Decide. The agentβs judgment is written into the history, followed by Your judgment of the last observation: j. Consecutive observations judged USELESS so far: k. In one or two sentences, weigh how likely you are to succeed if you keep consulting this knowledge base against how likely you are to answer correctly right now from what you already know. Then, on a final line, write exactly: DECISION: CONTINUE or DECISION: ANSWER_NOW. and an instruction to act consistently with the decision. Belief probe (continuation turn). Pause. Before deciding anything, give an honest assessment of your situation. You have an unspecified number of tool interactions left with this knowledge base. Estimate two probabilities, each between 0 and 1: - p_continue: the probability that you would end up answering the original question correctly if you kept consulting this knowledge base with the remaining interactions; - p_answer_now: the probability that you would answer correctly if you gave your best answer right now from what you already know. Reply with a single JSON object and nothing else: "p_continue": number, "p_answer_now": number, "prefer": one of "continue", "answer_now", "unsure", "reason": one short sentence B.2 Annotation of stated judgments On Qwen3-8Bβs development trajectories, two label streams were annotated from a written rubric: the evidence type of each observation (S1: useful new, related but not answering, redundant, off topic, unclear) and the judgment the agent states in its next thought (S2: says useless, partly useful, useful, no explicit judgment, unclear). The reference set below used rubric v0.3.1; v0.3.2, released with the code, adds the rulings from its adjudication. Records were shown blinded, without regime, gold answer or outcome. Full set. Two LLM annotators of different families, Claude Sonnet 5 (primary) and GPT-5.6-luna, labeled all 1,844 observations and 1,675 stated judgments independently through their official APIs. On the 1,792 and 1,672 items both labeled validly they agree at five-class ΞΊ 0.617 (S1) and 0.844 (S2), binary 0.927 (useful new or not) and 0.903 (says useless or not); items of the reference set below take their adjudicated label. Where they agree, their label is used. The first author adjudicated all 59 disagreements that change a consecutive-useless count. For the 328 that do not (mostly off topic versus related), the primary annotatorβs label is used. The first author also adjudicated a random 40 of these and found the primary label correct on 35 (0.875, Wilson 95% [0.74, 0.95]); no count changes. Reference set. Two further LLM annotators, Claude Fable 5.1 and GPT-6-sol, independently labeled a stratified sample of 360 observations and 327 stated judgments through their official APIs, without access to each otherβs labels. They agree at ΞΊ 0.875 and 0.868 (binary 0.972 and 0.898), and the first author adjudicated their disagreements. No human annotators other than the first author were involved. Accuracy of stated judgments. Against these labels, the judgments Qwen3-8B states in its own reasoning call useless evidence useless 93.9% of the time (n=1,463n=1,463) and treat useful or partly useful evidence as such 98.1% of the time (n=212n=212). Stated versus side-channel judgments. For amendment 16b a local Qwen3-32B annotator (non-thinking, temperature 0, the same rubric) labeled the 327 reference stated judgments at five-class ΞΊ 0.842 and binary 0.874 before being applied to four modelsβ test trajectories (Appendix D.4). B.3 Figure 1 selection rule Held-out questions were taken in the order of the released question manifest and the first one was selected where (1) Claude Haiku 4.5 without intervention exhausts its budget in the persistent regime, (2) on the same question the rule armβs side-channel judgments are USELESS five times in a row and the rule fires, and (3) the rule armβs answer is correct. No other candidate was inspected. B.4 Claude Sonnet 5 Setup. Following A7, Claude Sonnet 5 ran through the official API without thinking (it accepts no sampling parameters, so decoding was the service default) on the first 100 test300 questions, in the six mean6 regimes, with the unaided and rule arms. The rule runβs per-step side-channel judgments were not saved for five of its six regimes, so the pre-registered share of βuselessβ judgments and its own-judgment measures cannot be computed. Rule firings are recovered from the trajectories. A firing requires five consecutive USELESS judgments and appends the strict forced-answer suffix (ββ¦the only available action is finish[answer].β) to the system message. The rule fired on 65 of 100 persistent questions. Pre-registration and results. A7 fixed S-H1 (rule β- none, mean6) and S-H2 (persistent), both one-sided; descriptive quantities (the βuselessβ share, unaided answering after five βuselessβ judgments, both armsβ I); and two rules: report no significant gain if S-H1 failed, and partial self-stopping, judged by I, if the unaided agent answered on most persistent questions. S-H2 passed (+0.11+0.11 [+0.05+0.05, +0.18+0.18]); S-H1 failed (+0.022+0.022 [β0.003-0.003, +0.052+0.052], p=0.051p=0.051). The unaided agent answered on 69 of 100 persistent questions, with I +0.18+0.18 [+0.11+0.11, +0.26+0.26] (rule: +0.49+0.49); on the 65 questions where the rule fired, the unaided agent exhausted its budget on 29 (45%; Claude Haiku 4.5: 282 of 284). Non-inferiority (not pre-registered). Rule β- none was +0.00+0.00 [β0.06-0.06, +0.06+0.06], +0.02+0.02 [β0.04-0.04, +0.08+0.08] and β0.03-0.03 [β0.09-0.09, +0.03+0.03] on recover_after_1β3 and β0.01-0.01 [β0.08-0.08, +0.06+0.06] on clean; only recover_after_2 clears β0.05-0.05. Non-inferiority therefore cannot be shown in three of four regimes. This reflects sample size and run-to-run variation rather than harm. Premature answers in recover_after_1β3 occur before the rule can fire, yet they also differed between arms (0.05, 0.12, 0.18 with the rule; 0.00, 0.07, 0.17 without). Appendix C Measures C.1 Time-matched contrast Amendment 12 was frozen before any of its runs. For the unaided, permission, budget, stated-rule and stated-cost arms, the side-channel question of the rule arm was asked again at every decision point of the recorded trajectories (the recorded history verbatim, at most 8 generated tokens, nothing written back); the rule and combination arms use the judgments recorded in their own runs. In the same trajectories, once its own judgments have called five consecutive results useless, the unaided agent answers on 3 of 122 questions (Qwen2.5-7B), 1 of 282 (Llama-3.1-8B), 1 of 292 (Qwen3-8B) and 19 of 284 (Qwen3-32B). Definition. After t results, let s be the current run of consecutive results the agent judged useless (any other reply resets it). At each decision with 3β€tβ€63β€ tβ€ 6 (actions 4β7), group (a) has s=ts=t, every result judged useless, and the reference group (b) has 1β€sβ€tβ21β€ sβ€ t-2: the latest result judged useless and the most recent result not judged useless lying between the second and the second-to-last. Excluding s=tβ1s=t-1 keeps (b) from differing from (a) only in the first result. Ξ is the MantelβHaenszel risk difference in answering between (a) and (b), stratified by t; intervals come from 2,000 question-level bootstrap resamples. Table 6 gives the time-matched contrast on the agentβs own judgments (the measure in Figure 3), the same contrast stratified by question and action so that every comparison is within a question, and the exploratory version on design labels, with 95% question-cluster intervals. Pre-registered predictions: Ξ βs interval includes 0 or lies below it for the unaided and prompt arms, and lies above 0 for the rule. Both hold for the rule and for the unaided, permission and budget arms on all four models; they fail for the stated rule on Llama-3.1-8B and Qwen3-32B and for the stated cost on Qwen3-32B, which the main text reports as partly evidence-driven. Within questions the stated rule stays positive for Llama-3.1-8B but the two Qwen3-32B effects lose significance; within regimes the stated ruleβs contrast for Qwen3-32B grows (+0.17+0.17) and the stated costβs vanishes, and permission reaches +0.11+0.11 [+0.00,+0.20+0.00,+0.20]. Under every stratification, the unaided and budget-cued contrasts of every model have 95% upper bounds below +0.06+0.06. The stated cost (0.05 per call, with the budget) left the 7β8B models answering at the deadline (median action 8; 53β86% of failing-source answers at the final action) and did not change cost-adjusted success relative to the budget prompt (β0.009-0.009 to +0.012+0.012); the combinationβs cost-adjusted success exceeds the stated costβs for the 7β8B models (+0.039+0.039 to +0.095+0.095) and ties it for Qwen3-32B (+0.003+0.003 [β0.020,+0.027-0.020,+0.027]). Charging each one-word judgment call of the rule and combination at the price of a tool call (second number in the last block) makes them the most expensive arms. The ruleβs own-judgment contrast for Claude Haiku 4.5 is +0.40+0.40 [+0.35,+0.46+0.35,+0.46]. Validation on simulated policies. Table 8 applies the three measures to simulated agents in the same five failure regimes and 8-action budget (no model calls; 300 questions, mean of 10 simulations). Corrupted observations are judged useless with probability 0.99 and real ones with probability 0.45, and a real observation judged useful ends the episode with probability 0.3; each policy adds its own stopping rule. The integration index is positive for a clock and for a hazard that rises with time, and undefined when stopping precedes four failures; both contrasts are zero in expectation for every time- or deadline-driven policy and positive for every evidence-driven policy whose threshold falls within the budget. No measure detects a threshold beyond the budget (k=7k=7). Non-inferiority margin. At the β0.02-0.02 margin drafted before amendment 6, the combinationβs non-inferiority to its components holds in 12 of 14 original tests; it fails for Qwen3-8B against the rule on HotpotQA (lower bound β0.023-0.023) and Qwen3-32B against the rule on HotpotQA (lower bound β0.022-0.022). Model none permit budget stated cost enforced combo time-matched contrast Ξ on the agentβs own judgments (amendment 12) Qwen2.5-7B β0.11-0.11 [β0.17,β0.05-0.17,-0.05] β0.16-0.16 [β0.31,β0.06-0.31,-0.06] β0.18-0.18 [β0.25,β0.11-0.25,-0.11] β0.31-0.31 [β0.55,β0.13-0.55,-0.13] β0.18-0.18 [β0.25,β0.11-0.25,-0.11] +0.33+0.33 [+0.25,+0.41+0.25,+0.41] +0.21+0.21 [+0.11,+0.29+0.11,+0.29] Llama-3.1-8B β0.02-0.02 [β0.03,β0.02-0.03,-0.02] β0.19-0.19 [β0.23,β0.16-0.23,-0.16] β0.10-0.10 [β0.12,β0.08-0.12,-0.08] +0.09+0.09 [+0.06,+0.13+0.06,+0.13] β0.17-0.17 [β0.19,β0.14-0.19,-0.14] +0.37+0.37 [+0.36,+0.39+0.36,+0.39] +0.27+0.27 [+0.25,+0.30+0.25,+0.30] Qwen3-8B β0.06-0.06 [β0.10,β0.04-0.10,-0.04] β0.32-0.32 [β0.41,β0.24-0.41,-0.24] β0.20-0.20 [β0.26,β0.15-0.26,-0.15] β0.18-0.18 [β0.25,β0.11-0.25,-0.11] β0.19-0.19 [β0.24,β0.14-0.24,-0.14] +0.33+0.33 [+0.29,+0.37+0.29,+0.37] +0.20+0.20 [+0.13,+0.27+0.13,+0.27] Qwen3-32B β0.07-0.07 [β0.10,β0.04-0.10,-0.04] +0.01+0.01 [β0.08,+0.09-0.08,+0.09] β0.00-0.00 [β0.06,+0.05-0.06,+0.05] +0.08+0.08 [+0.01,+0.16+0.01,+0.16] +0.09+0.09 [+0.04,+0.14+0.04,+0.14] +0.33+0.33 [+0.29,+0.36+0.29,+0.36] +0.19+0.19 [+0.12,+0.25+0.12,+0.25] same, stratified by question Γ action (exploratory) Qwen2.5-7B β0.17-0.17 [β0.23,β0.11-0.23,-0.11] β0.20-0.20 [β0.37,β0.07-0.37,-0.07] β0.20-0.20 [β0.28,β0.13-0.28,-0.13] β0.24-0.24 [β0.55,β0.02-0.55,-0.02] β0.23-0.23 [β0.29,β0.16-0.29,-0.16] +0.31+0.31 [+0.22,+0.40+0.22,+0.40] +0.13+0.13 [+0.02,+0.22+0.02,+0.22] Llama-3.1-8B β0.02-0.02 [β0.03,β0.01-0.03,-0.01] β0.20-0.20 [β0.24,β0.17-0.24,-0.17] β0.12-0.12 [β0.14,β0.10-0.14,-0.10] +0.13+0.13 [+0.09,+0.17+0.09,+0.17] β0.17-0.17 [β0.20,β0.14-0.20,-0.14] +0.37+0.37 [+0.34,+0.39+0.34,+0.39] +0.26+0.26 [+0.23,+0.29+0.23,+0.29] Qwen3-8B β0.08-0.08 [β0.11,β0.05-0.11,-0.05] β0.39-0.39 [β0.48,β0.29-0.48,-0.29] β0.23-0.23 [β0.28,β0.18-0.28,-0.18] β0.20-0.20 [β0.28,β0.12-0.28,-0.12] β0.19-0.19 [β0.25,β0.14-0.25,-0.14] +0.29+0.29 [+0.24,+0.34+0.24,+0.34] +0.16+0.16 [+0.09,+0.24+0.09,+0.24] Qwen3-32B β0.07-0.07 [β0.10,β0.04-0.10,-0.04] β0.01-0.01 [β0.11,+0.09-0.11,+0.09] β0.02-0.02 [β0.08,+0.05-0.08,+0.05] +0.02+0.02 [β0.06,+0.10-0.06,+0.10] +0.07+0.07 [β0.00,+0.15-0.00,+0.15] +0.30+0.30 [+0.26,+0.33+0.26,+0.33] +0.13+0.13 [+0.05,+0.20+0.05,+0.20] same, stratified by regime Γ action (exploratory) Qwen2.5-7B β0.07-0.07 [β0.13,β0.01-0.13,-0.01] β0.08-0.08 [β0.23,+0.03-0.23,+0.03] β0.14-0.14 [β0.21,β0.07-0.21,-0.07] β0.27-0.27 [β0.48,β0.11-0.48,-0.11] β0.13-0.13 [β0.19,β0.06-0.19,-0.06] +0.37+0.37 [+0.29,+0.44+0.29,+0.44] +0.23+0.23 [+0.13,+0.31+0.13,+0.31] Llama-3.1-8B β0.02-0.02 [β0.03,β0.01-0.03,-0.01] β0.05-0.05 [β0.10,β0.00-0.10,-0.00] β0.02-0.02 [β0.06,+0.01-0.06,+0.01] +0.09+0.09 [+0.03,+0.16+0.03,+0.16] β0.06-0.06 [β0.10,β0.01-0.10,-0.01] +0.36+0.36 [+0.33,+0.39+0.33,+0.39] +0.33+0.33 [+0.30,+0.36+0.30,+0.36] Qwen3-8B β0.05-0.05 [β0.08,β0.02-0.08,-0.02] β0.20-0.20 [β0.30,β0.10-0.30,-0.10] β0.11-0.11 [β0.19,β0.04-0.19,-0.04] β0.00-0.00 [β0.10,+0.10-0.10,+0.10] β0.09-0.09 [β0.15,β0.02-0.15,-0.02] +0.37+0.37 [+0.33,+0.41+0.33,+0.41] +0.23+0.23 [+0.16,+0.29+0.16,+0.29] Qwen3-32B β0.07-0.07 [β0.11,β0.04-0.11,-0.04] +0.11+0.11 [+0.00,+0.20+0.00,+0.20] β0.07-0.07 [β0.14,+0.00-0.14,+0.00] +0.17+0.17 [+0.07,+0.26+0.07,+0.26] +0.04+0.04 [β0.05,+0.14-0.05,+0.14] +0.34+0.34 [+0.29,+0.37+0.29,+0.37] +0.18+0.18 [+0.10,+0.26+0.10,+0.26] design-label contrast (persistent β- late_onset_from_3 at the same action; exploratory) Qwen2.5-7B β0.03-0.03 [β0.05,β0.02-0.05,-0.02] β0.11-0.11 [β0.15,β0.08-0.15,-0.08] β0.05-0.05 [β0.07,β0.03-0.07,-0.03] β0.05-0.05 [β0.07,β0.03-0.07,-0.03] β0.06-0.06 [β0.08,β0.03-0.08,-0.03] +0.08+0.08 [+0.04,+0.11+0.04,+0.11] +0.06+0.06 [+0.02,+0.09+0.02,+0.09] Llama-3.1-8B β0.01-0.01 [β0.01,β0.00-0.01,-0.00] β0.03-0.03 [β0.06,β0.01-0.06,-0.01] β0.01-0.01 [β0.03,+0.01-0.03,+0.01] +0.07+0.07 [+0.04,+0.11+0.04,+0.11] β0.03-0.03 [β0.05,β0.02-0.05,-0.02] +0.19+0.19 [+0.16,+0.21+0.16,+0.21] +0.19+0.19 [+0.16,+0.22+0.16,+0.22] Qwen3-8B β0.02-0.02 [β0.03,β0.01-0.03,-0.01] β0.15-0.15 [β0.26,β0.07-0.26,-0.07] β0.11-0.11 [β0.15,β0.07-0.15,-0.07] β0.10-0.10 [β0.16,β0.05-0.16,-0.05] β0.08-0.08 [β0.12,β0.03-0.12,-0.03] +0.16+0.16 [+0.12,+0.20+0.12,+0.20] +0.05+0.05 [β0.00,+0.11-0.00,+0.11] Qwen3-32B β0.02-0.02 [β0.04,β0.00-0.04,-0.00] β0.06-0.06 [β0.15,+0.02-0.15,+0.02] β0.03-0.03 [β0.09,+0.02-0.09,+0.02] β0.10-0.10 [β0.20,β0.01-0.20,-0.01] +0.05+0.05 [β0.01,+0.11-0.01,+0.11] +0.17+0.17 [+0.13,+0.21+0.13,+0.21] +0.05+0.05 [β0.02,+0.11-0.02,+0.11] cost-adjusted success, mean6 β- 0.05 Γ tool calls (/ also charging judgment calls) Qwen2.5-7B 0.010 0.053 0.083 0.014 0.078 0.090 / β-0.053 0.173 / β-0.027 Llama-3.1-8B β-0.043 0.019 0.043 0.016 0.054 0.062 / β-0.192 0.093 / β-0.149 Qwen3-8B 0.208 0.285 0.272 0.252 0.263 0.269 / 0.069 0.304 / 0.130 Qwen3-32B 0.271 0.366 0.399 0.362 0.398 0.367 / 0.165 0.401 / 0.224 Table 6: Stopping measures and cost for the arms with own judgments, HotpotQA test300, raw analysis. Design-label contrast for other arms and models: thinking β0.02-0.02 [β0.13,+0.08-0.13,+0.08] (Qwen3-8B), Claude Sonnet 5 unaided +0.12+0.12 [+0.05,+0.18+0.05,+0.18], Claude Haiku 4.5 unaided β0.02-0.02 [β0.04,β0.01-0.04,-0.01]. Simulated policy I design Ξ own Ξ never stops +0.00 +0.00 +0.00 answers at the deadline +0.00 +0.00 +0.00 clock, action 4 β +0.00 +0.00 clock, action 6 +0.50 +0.00 +0.00 hazard rising with time +0.18 β-0.02 0.00 own-judgment count k=3k=3 +0.39 +0.60 +1.00 own-judgment count k=5k=5 +0.44 +0.21 +0.38 own-judgment count k=7k=7 +0.00 +0.00 +0.00 design-label count k=3k=3 β +1.00 +0.33 design-label count k=5k=5 +0.37 +0.33 +0.20 hazard rising with own count +0.32 +0.18 +0.31 Table 7: The stopping measures on simulated policies with known behavior. judged answers after Ξ (own) success Model useless 5 useless none rule none rule Qwen2.5-7B 97.5% 39/200 β0.02-0.02 +0.16+0.16 0.077 0.160 Llama-3.1-8B 87.9% 5/207 β0.01-0.01 +0.45+0.45 0.047 0.140 Qwen3-8B 86.5% 26/176 β0.03-0.03 +0.35+0.35 0.120 0.190 Qwen3-32B 84.3% 40/174 β0.10-0.10 +0.25+0.25 0.113 0.233 Table 8: Plausible regime, HotpotQA test300, raw analysis. Plausible regime. Here every failed observation is one of the questionβs own distractor paragraphs, which are on topic. For the unaided agent, with judgments replayed on its own trajectories, Table 8 gives the share of these paragraphs its judgments call useless, how often it answers once its judgments reach five consecutive βuselessβ, and the own-judgment contrast within this regime. It also gives the rule armβs contrast and both armsβ success (exploratory). Decide (exploratory). Ξ on the judgments recorded during the decide runs (Figure 3): Qwen2.5-7B β0.24-0.24 [β0.37,β0.13-0.37,-0.13]; Llama-3.1-8B β0.14-0.14 [β0.18,β0.11-0.18,-0.11]; Qwen3-8B β0.15-0.15 [β0.29,β0.01-0.29,-0.01]. Decide was not run for Qwen3-32B. C.2 Stopping hazards and the integration index Figure 5 gives the stopping hazard by the number of consecutive failed observations just seen, and Table 9 the pre-registered integration index for the arms in Table 2. Figure 5: Stopping hazard: probability of answering at a decision point after j consecutive failed observations (final-action decisions excluded; 8-action budget; test300, Claude Sonnet 5 on 100 questions). Unaided agents (gray) do not rise with j, except for a partial rise in Claude Sonnet 5. Prompt cues (purple) stay flat or fall in the 7β8B models and rise only in Qwen3-32B. The enforced rule (blue) steps up at j=5j=5; the reasoning mode (brown, Qwen3-8B) and the budget-cued Qwen3-32B rise with j, but not with the evidence at a fixed step (Qwen3-32B: Ξ=0.00 =0.00, Figure 3; thinking: design-label contrast β0.02-0.02, Appendix C.1). Model none permit budget stated cost decide enforced combo Qwen2.5-7B β.32/β.11-.32/-.11 β.29/β.16-.29/-.16 β.02/β.18-.02/-.18 β.18/β.31-.18/-.31 .00/β.18.00/-.18 β.33/β.24-.33/-.24 .18/.33.18/.33 .47/.21.47/.21 Llama-3.1-8B .00/β.02.00/-.02 .03/β.19.03/-.19 .05/β.10.05/-.10 .31/.09.31/.09 .02/β.17.02/-.17 .05/β.14.05/-.14 .48/.37.48/.37 .49/.27.49/.27 Qwen3-8B β.01/β.06-.01/-.06 β.23/β.32-.23/-.32 .00/β.20.00/-.20 .03/β.18.03/-.18 .00/β.19.00/-.19 β.29/β.15-.29/-.15 .47/.33.47/.33 .43/.20.43/.20 Qwen3-32B .01/β.07.01/-.07 .17/.01.17/.01 .32/.00.32/.00 .54/.08.54/.08 .41/.09.41/.09 β .49/.33.49/.33 .52/.19.52/.19 Claude Haiku 4.5 .00/β.00/-- β β β β β .50/.40.50/.40 β Claude Sonnet 5 β .18/β.18/-- β β β β β .49/β.49/-- β Table 9: Integration index I (pre-registered for some arms, Appendix A) and time-matched contrast Ξ on the agentβs own judgments, test300, raw analysis. β First 100 questions; its per-step judgments were not saved (Appendix B.4). Hazard model. Ξ compares histories with and without an earlier result judged useful, so a negative Ξ could come from answering on sufficient evidence rather than from ignoring useless evidence. Table 10 separates the two with a discrete-time linear probability model on every decision after a result the agent judged useless (decisions after one to six results, the five failure regimes): answering =Ξ±action+Ξ±regime+Ξ²β run+Ξ³β earlier useful+Ξ΅= _action+ _regime+Ξ²Β·run+Ξ³Β·earlier useful+ , where run is the current number of consecutive useless judgments. Ξ²>0Ξ²>0 means stopping rises with accumulated useless evidence; Ξ³>0Ξ³>0 means stopping follows earlier useful evidence. Without an earlier useful judgment, run equals the number of results so far, so Ξ² is identified from histories with one. The second line adds question fixed effects (within-question estimator). Each fit uses 3,064β6,434 decisions, 5β31% of them with an earlier useful judgment. Intervals: question-cluster bootstrap, 1,000 resamples. Exploratory. Model none permit budget stated cost enforced combo test300: Ξ² (run of useless judgments), pooled; second line with question fixed effects Qwen2.5-7B β.01-.01 [β.04,+.03-.04,+.03] +.01+.01 [β.06,+.07-.06,+.07] β.03-.03 [β.08,+.01-.08,+.01] β.09-.09 [β.27,+.01-.27,+.01] β.05-.05 [β.09,β.02-.09,-.02] +.17+.17 [+.14,+.21+.14,+.21] +.13+.13 [+.09,+.17+.09,+.17] β.00-.00 [β.05,+.05-.05,+.05] +.00+.00 [β.06,+.07-.06,+.07] β.05-.05 [β.11,β.00-.11,-.00] β.09-.09 [β.31,+.03-.31,+.03] β.05-.05 [β.09,β.01-.09,-.01] +.18+.18 [+.12,+.23+.12,+.23] +.11+.11 [+.07,+.15+.07,+.15] Llama-3.1-8B β.01-.01 [β.01,+.00-.01,+.00] β.04-.04 [β.07,β.02-.07,-.02] β.03-.03 [β.05,β.01-.05,-.01] +.04+.04 [+.02,+.06+.02,+.06] β.05-.05 [β.07,β.03-.07,-.03] +.15+.15 [+.14,+.16+.14,+.16] +.11+.11 [+.09,+.13+.09,+.13] β.01-.01 [β.02,.00-.02,.00] β.04-.04 [β.06,β.01-.06,-.01] β.02-.02 [β.04,β.01-.04,-.01] +.04+.04 [+.02,+.07+.02,+.07] β.05-.05 [β.07,β.03-.07,-.03] +.14+.14 [+.13,+.16+.13,+.16] +.10+.10 [+.08,+.13+.08,+.13] Qwen3-8B β.02-.02 [β.03,+.00-.03,+.00] β.07-.07 [β.12,+.00-.12,+.00] β.04-.04 [β.07,+.00-.07,+.00] β.03-.03 [β.08,+.02-.08,+.02] β.05-.05 [β.08,β.01-.08,-.01] +.15+.15 [+.13,+.17+.13,+.17] +.11+.11 [+.08,+.15+.08,+.15] β.01-.01 [β.03,+.01-.03,+.01] β.10-.10 [β.19,β.02-.19,-.02] β.01-.01 [β.05,+.03-.05,+.03] β.00-.00 [β.05,+.05-.05,+.05] β.03-.03 [β.07,+.01-.07,+.01] +.16+.16 [+.14,+.18+.14,+.18] +.14+.14 [+.09,+.18+.09,+.18] Qwen3-32B β.04-.04 [β.06,β.02-.06,-.02] +.00+.00 [β.06,+.07-.06,+.07] +.04+.04 [+.01,+.07+.01,+.07] +.07+.07 [+.03,+.12+.03,+.12] +.03+.03 [β.01,+.06-.01,+.06] +.14+.14 [+.13,+.16+.13,+.16] +.09+.09 [+.05,+.14+.05,+.14] β.03-.03 [β.05,β.02-.05,-.02] β.01-.01 [β.09,+.06-.09,+.06] +.05+.05 [+.01,+.09+.01,+.09] +.10+.10 [+.06,+.15+.06,+.15] +.05+.05 [+.01,+.09+.01,+.09] +.15+.15 [+.13,+.18+.13,+.18] +.11+.11 [+.07,+.16+.07,+.16] fresh300: Ξ² (run of useless judgments), pooled; second line with question fixed effects Qwen2.5-7B β.01-.01 [β.05,+.02-.05,+.02] β β.03-.03 [β.07,+.00-.07,+.00] β β +.17+.17 [+.13,+.21+.13,+.21] β β.06-.06 [β.12,β.01-.12,-.01] β.04-.04 [β.08,.00-.08,.00] +.13+.13 [+.07,+.18+.07,+.18] Llama-3.1-8B β.01-.01 [β.02,β.00-.02,-.00] β β.05-.05 [β.06,β.03-.06,-.03] β β +.15+.15 [+.14,+.16+.14,+.16] β β.01-.01 [β.02,β.00-.02,-.00] β.05-.05 [β.07,β.03-.07,-.03] +.14+.14 [+.13,+.16+.13,+.16] Qwen3-8B .00.00 [β.01,+.01-.01,+.01] β +.01+.01 [β.03,+.04-.03,+.04] β β +.15+.15 [+.13,+.17+.13,+.17] β +.00+.00 [β.01,+.01-.01,+.01] +.01+.01 [β.03,+.05-.03,+.05] +.15+.15 [+.13,+.17+.13,+.17] Qwen3-32B β.02-.02 [β.03,.00-.03,.00] β +.00+.00 [β.03,+.04-.03,+.04] β β +.16+.16 [+.14,+.17+.14,+.17] β β.01-.01 [β.03,+.00-.03,+.00] +.01+.01 [β.03,+.04-.03,+.04] +.16+.16 [+.14,+.18+.14,+.18] Ξ³ (an earlier result judged useful), test300, pooled Qwen2.5-7B +.08+.08 [+.02,+.16+.02,+.16] +.19+.19 [+.08,+.33+.08,+.33] +.06+.06 [β.02,+.16-.02,+.16] β.00-.00 [β.20,+.13-.20,+.13] +.02+.02 [β.05,+.09-.05,+.09] +.11+.11 [+.04,+.18+.04,+.18] +.08+.08 [β.01,+.17-.01,+.17] Llama-3.1-8B +.01+.01 [β.01,+.02-.01,+.02] +.04+.04 [β.03,+.10-.03,+.10] +.00+.00 [β.04,+.05-.04,+.05] +.02+.02 [β.04,+.07-.04,+.07] +.01+.01 [β.04,+.06-.04,+.06] +.01+.01 [β.02,+.04-.02,+.04] +.01+.01 [β.03,+.06-.03,+.06] Qwen3-8B +.01+.01 [β.02,+.06-.02,+.06] +.10+.10 [β.05,+.31-.05,+.31] +.05+.05 [β.03,+.15-.03,+.15] +.03+.03 [β.08,+.18-.08,+.18] +.02+.02 [β.06,+.12-.06,+.12] +.06+.06 [+.01,+.11+.01,+.11] +.07+.07 [β.03,+.19-.03,+.19] Qwen3-32B β.04-.04 [β.08,β.01-.08,-.01] β.03-.03 [β.20,+.16-.20,+.16] +.11+.11 [+.03,+.20+.03,+.20] +.11+.11 [β.01,+.23-.01,+.23] β.04-.04 [β.12,+.05-.12,+.05] +.02+.02 [β.02,+.07-.02,+.07] +.01+.01 [β.07,+.12-.07,+.12] Table 10: Hazard model: change in the probability of answering per additional consecutive useless judgment (Ξ²) and after an earlier useful judgment (Ξ³), with 95% intervals. C.3 Specificity of side-channel judgments Table 11 gives the share of side-channel judgments in the rule armβs HotpotQA test300 log that call a real search result useful. It splits them by whether the page is one of the questionβs two gold paragraphs or one of its eight distractor paragraphs, and by whether the agent has already seen it in the trajectory. Real distractor paragraphs are usually useless for the question, and a repeated page adds nothing new, so low rates there are appropriate. gold paragraph distractor paragraph Model first repeat first repeat Qwen2.5-7B 0.57 (1045) 0.03 (76) 0.09 (442) 0.00 (235) Llama-3.1-8B 0.92 (1771) 0.09 (150) 0.41 (689) 0.05 (438) Qwen3-8B 0.96 (1817) 0.17 (155) 0.36 (745) 0.08 (825) Qwen3-32B 0.98 (2026) 0.06 (194) 0.38 (849) 0.03 (703) Claude Haiku 4.5 0.77 (2157) 0.05 (274) 0.08 (884) 0.00 (365) Table 11: Share of side-channel USEFUL judgments on real search results (number of judgments in parentheses). C.4 Beliefs probe Probe. At every decision point of a recorded trajectory we fork the history, append one assessment turn, and discard the reply; nothing is written back. Two turns are asked separately, each on its own fork of the same history: a usefulness turn (five categories aligned with the annotation rubric) and a continuation turn that asks for pcontinuep_continue and panswer_nowp_answer\_now. The continuation turn does not state the remaining budget (βan unspecified number of tool interactionsβ), offers no action, and never mentions the action syntax. Replies that do not parse are recorded as missing, never coerced. Realized values. VcV_c is the success of the recorded trajectory from that decision point on; VfV_f is the success of an answer-now branch: the same history replayed with the ruleβs finish instruction (Appendix B.1), which the agents obey at 89β98% of decision points. Both are computed on the unaided test runs of Qwen2.5-7B, Llama-3.1-8B and Qwen3-8B in four regimes; this analysis is descriptive, not pre-registered. The ideal observer Eβ‘[Vcβ£k]E[V_c k] after k results that all failed is a reference rather than an optimum. It puts an equal prior on the four regimes the probe was run on (clean, persistent, plausible, recover_after_2) and updates it with the rate at which the modelβs own usefulness replies call the result at each position in each regime not newly useful. It then averages the regimesβ realized values of continuing. Figure 6 shows the three quantities for the three models on which the probe was run. Figure 6: The three models err in different directions, and where a modelβs own report favors answering, its action does not follow. Held-out persistent regime, placeholder answers repaired; means over decision points, by the number of failed results so far. Left: self-reported value of continuing against the ideal observer (the realized value is the success on this source, 0.00β0.04). Middle: self-reported value of answering now against the realized value of the answer-now branch. Right: share of decision points at which the agent answers, and at which its own report favors answering. C.5 Answer repair What happened. After the runs we found answers that copied the template finish[answer]: the word βanswerβ (sometimes βguessβ) or no argument, scored wrong. The forced-answer wording and the prompt armsβ βuse finish[answer] with your best guessβ elicited them most. Placeholders can only lower success, and they occurred almost only in arms that stop and answer, so H1 and H2 were conservative. Contrasts between answering arms could move either way, and stopping measures are unaffected. Procedure (post hoc, identical for all arms). For every answered record whose answer was empty or a template string (βanswerβ, βguessβ, βyour best guessβ and six similar), in every arm and model except the call-cost arm, the fresh-question runs, the third run and the runs of amendments 13β20 (which are reported raw only), we rebuilt the exact conversation, kept the modelβs finish turn, and appended one fixed message: βYour Action Input was a placeholder, not an answer. Respond again in the required format (Thought / Action / Action Input) with Action: finish and your actual final answer to the original question as the Action Input.β One attempt, unchanged decoding; a second placeholder stayed wrong. Only answers and scores changed. Raw results are primary; repaired ones are a sensitivity analysis, used only for the realized values of the belief probe (Appendix C.4). Frequency. On HotpotQA test300 (2,100 trajectories per arm), placeholders concentrated in Qwen3-8Bβs stated-rule, budget, combination, rule, lexical and permit arms (297, 231, 226, 188, 146, 114), Llama-3.1-8Bβs stated-rule arm (149) and Qwen2.5-7Bβs rule and random arms (105, 82); the unaided arms had one in total, and the unrepaired call-cost arm 364 for Qwen3-8B. Including 2,628 in the answer-now branch, the local models produced 3,834 (stated-rule and combination arms excluded); 3,254 became real answers, 545 correct; Qwen2.5-7B complied least (rule arm: 42 of 105). Claude Haiku 4.5 had 13, Qwen3-32B 24, Claude Sonnet 5 none; FEVERβs seven original arms had 606, 339 of them empty answers from Llama-3.1-8Bβs decide arm. HotpotQA test300 FEVER raw repaired raw repaired Contrast (mean6) Q2.5 L Q3 Q2.5 L Q3 Q2.5 L Q3 Q2.5 L Q3 rule β- none (H1, F-H1) +.045+.045 +.064+.064 +.026+.026 +.046+.046 +.069+.069 +.040+.040 +.129+.129 +.201+.201 +.142+.142 +.134+.134 +.203+.203 +.142+.142 rule β- random (H2, F-H2) +.011+.011 +.029+.029 +.021+.021 +.011+.011 +.032+.032 +.033+.033 +.044+.044 +.123+.123 +.096+.096 +.045+.045 +.124+.124 +.096+.096 rule β- decide (H4) β.123-.123 +.062+.062 +.048+.048 β.123-.123 +.059+.059 +.062+.062 β.084-.084 +.350+.350 +.005+.005 β.079-.079 +.180+.180 +.005+.005 rule β- permit (H7) +.023+.023 +.059+.059 +.051+.051 +.024+.024 +.061+.061 +.054+.054 +.138+.138 +.138+.138 +.064+.064 +.142+.142 +.108+.108 +.062+.062 rule β- budget (H8) β.091-.091 +.016+.016 +.009+.009 β.091-.091 +.019+.019 β.011-.011 β.024-.024 +.066β +.066 β.021β -.021 β.022-.022 +.022β +.022 β.026β -.026 combo β- rule (C1) +.138+.138 +.009+.009 +.004+.004 +.139+.139 +.007+.007 +.024+.024 +.088+.088 +.026+.026 +.051+.051 +.093+.093 +.064+.064 +.053+.053 combo β- budget (C1) +.047+.047 +.025+.025 +.013β +.013 +.048+.048 +.026+.026 +.013β +.013 +.064+.064 +.091+.091 +.030+.030 +.071+.071 +.086+.086 +.027+.027 Table 12: Key contrasts, raw vs. repaired (IDs refer to HotpotQA; FEVER H4/H7/H8 counterparts are exploratory). Q2.5: Qwen2.5-7B; L: Llama-3.1-8B; Q3: Qwen3-8B. β Significance (95% interval excluding zero) differs between analyses. Also unchanged in decision: H1 for Claude Haiku 4.5 (+.097+.097 both), L-H1 (+.063+.063 both), Qwen3-8Bβs H3 lower bounds on rec1/rec3 (β.057-.057/β.060-.060 raw, β.057-.057/β.057-.057 repaired), H6 (+.003+.003 raw, β.011-.011 repaired, both n.s.). Realized answer-now success on persistent: .12.12/.15.15/.14.14 raw, .14.14/.17.17/.19.19 repaired. Effect. No decision in Table 12 changed (in the second run, Rp4 for Qwen3-8B passes only after repair; Appendix D.6); all failures remain (Qwen2.5-7Bβs H7 at p=0.0505p=0.0505). Three contrasts in the table changed significance, as did Qwen3-8Bβs stated-rule contrast in P2 (Table 5). On FEVER, the ruleβs lead over the budget cue for Llama-3.1-8B fell from +0.066+0.066 [+0.041+0.041, +0.090+0.090] to +0.022+0.022 [β0.005-0.005, +0.047+0.047], and the budget cueβs lead for Qwen3-8B became significant (β0.026-0.026 [β0.048-0.048, β0.003-0.003]). On HotpotQA, the combinationβs lead over the budget cue for Qwen3-8B moved from [β0.000-0.000, +0.028+0.028] to [+0.001+0.001, +0.026+0.026]. Llama-3.1-8Bβs FEVER decide arm rose from 0.401 to 0.574; the unaided arm scores 0.551. Appendix D Additional results D.1 Full result tables All numbers in this appendix are computed by one script from the run logs. HotpotQA results use test300 with the 8-action budget (Claude Sonnet 5: its first 100 questions); FEVER results use all 292 claims, for which no plausible regime was run. Success is the share of questions answered correctly in a regime; mean6 averages the six regimes of Β§2 with equal weight. Unless a column says repaired, numbers come from the raw, pre-registered analysis, in which a template-placeholder answer counts as wrong (Appendix C.5). Intervals are 95% percentile intervals over 2,000 bootstrap resamples of questions (for I, whole questions are resampled with all their decision points); the one-sided p is the share of resampled differences β€0β€ 0, unadjusted for multiplicity. Condition names are as in Table 2, with short forms: rule = enforced rule, policy or stated = stated rule, cost = call cost, think = reasoning mode, random and lexical = the enforced rule on a random or lexical signal; βββ marks a condition or regime that was not run. Table 13 gives per-regime success and the integration index on HotpotQA, Table 14 the same on FEVER, Table 15 the paired contrasts, and Table 16 the budget-scaling runs. success per regime integration index Model Arm pers. rec1 rec2 rec3 late clean plaus. mean6 I 95% CI h4β-β5h_4-5 h1β-β2h_1-2 Qwen2.5-7B none .043 .160 .153 .147 .220 .427 .077 .192 β.32-.32 [β.37,β.27][-.37,-.27] .02 .34 permit .033 .170 .207 .137 .303 .430 .143 .213 β.29-.29 [β.34,β.24][-.34,-.24] .03 .32 budget .090 .407 .377 .327 .333 .433 .183 .328 β.02-.02 [β.04,.00][-.04,.00] .02 .04 decide .183 .370 .300 .277 .517 .513 .230 .360 β.33-.33 [β.37,β.29][-.37,-.29] .03 .36 random .080 .187 .183 .163 .300 .443 .143 .226 β.11-.11 [β.17,β.05][-.17,-.05] .23 .33 lexical .090 .173 .170 .160 .313 .440 .110 .224 .15 [.09,.20][.09,.20] .48 .33 rule .100 .193 .173 .163 .327 .463 .160 .237 .18 [.13,.22][.13,.22] .51 .33 combo .163 .417 .420 .370 .393 .487 .243 .375 .47 [.45,.49][.45,.49] .51 .04 Llama-3.1-8B none .000 .370 .300 .280 .207 .423 .047 .263 .00 [β.01,.01][-.01,.01] .00 .00 permit .017 .363 .310 .270 .263 .390 .090 .269 .03 [.01,.05][.01,.05] .04 .01 budget .057 .390 .353 .307 .323 .440 .137 .312 .05 [.03,.06][.03,.06] .05 .01 decide .033 .343 .303 .223 .287 .403 .113 .266 .05 [.03,.07][.03,.07] .07 .02 random .033 .387 .337 .293 .267 .473 .093 .298 .09 [.07,.11][.07,.11] .09 .00 lexical .160 .370 .333 .260 .377 .470 .060 .328 .44 [.43,.46][.43,.46] .45 .00 rule .143 .373 .327 .273 .377 .473 .140 .328 .48 [.46,.50][.46,.50] .49 .00 combo .177 .390 .330 .303 .400 .420 .160 .337 .49 [.47,.51][.47,.51] .50 .01 Qwen3-8B none .007 .553 .537 .500 .487 .603 .120 .448 β.01-.01 [β.01,.00][-.01,.00] .00 .01 permit .200 .427 .393 .367 .567 .587 .263 .423 β.23-.23 [β.28,β.18][-.28,-.18] .07 .30 budget .203 .503 .497 .470 .537 .577 .230 .464 .00 [β.04,.03][-.04,.03] .08 .09 decide .263 .503 .340 .283 .570 .593 .260 .426 β.29-.29 [β.37,β.19][-.37,-.19] .13 .42 think .297 .533 .493 .437 .543 .560 .273 .477 .26 [.20,.33][.20,.33] .40 .13 random .050 .543 .527 .487 .497 .617 .130 .453 .06 [.04,.08][.04,.08] .07 .01 lexical .167 .557 .510 .510 .553 .610 .140 .484 .44 [.42,.46][.42,.46] .44 .01 rule .153 .527 .527 .470 .547 .620 .190 .474 .47 [.45,.49][.45,.49] .48 .01 combo .270 .530 .487 .453 .543 .583 .263 .478 .43 [.40,.45][.40,.45] .52 .09 Qwen3-32B none .027 .630 .607 .570 .543 .680 .113 .509 .01 [.00,.03][.00,.03] .03 .01 permit .290 .533 .510 .450 .627 .693 .297 .517 .17 [.11,.23][.11,.23] .36 .19 budget .330 .643 .623 .540 .650 .700 .337 .581 .32 [.28,.36][.28,.36] .38 .07 rule .267 .647 .620 .567 .643 .690 .233 .572 .49 [.47,.51][.47,.51] .50 .01 combo .310 .650 .597 .567 .650 .693 .323 .578 .52 [.50,.55][.50,.55] .59 .07 Claude Haiku 4.5 none .000 .633 .623 .590 .523 .637 .057 .501 .00 [.00,.01][.00,.01] .00 .00 rule .380 .667 .643 .577 .653 .667 .363 .598 .50 [.49,.51][.49,.51] .51 .00 Claude Sonnet 5 none .400 .700 .670 .660 .650 .710 β .632 .18 [.11,.26][.11,.26] .23 .04 rule .510 .700 .690 .630 .690 .700 β .653 .49 [.46,.52][.46,.52] .54 .05 Table 13: HotpotQA (test300, 8-action budget), raw analysis: success per regime, mean6, and the integration index with its two components, for every model and arm. n=300n=300 questions per arm (Claude Sonnet 5: the first 100; its plausible regime was not run). pers.: persistent; reccc: recover_after_c; late: late_onset_from_3; plaus.: plausible (not part of mean6). I =h4β-β5βh1β-β2=h_4-5-h_1-2, where haβ-βbh_a-b is the probability of answering at a decision point just after a to b consecutive failed observations, pooled over the five regimes with failures, decisions at the final action excluded; its 95% CI is from a question-cluster bootstrap. success per regime integration index Model Arm pers. rec1 rec2 rec3 late clean mean6 I 95% CI h4β-β5h_4-5 h1β-β2h_1-2 Qwen2.5-7B none .127 .675 .664 .644 .572 .716 .566 β.12-.12 [β.15,β.10][-.15,-.10] .01 .13 permit .151 .579 .541 .538 .729 .805 .557 β.20-.20 [β.25,β.16][-.25,-.16] .02 .23 budget .384 .818 .764 .798 .750 .801 .719 .00 [β.02,.02][-.02,.02] .03 .03 decide .668 .822 .733 .729 .870 .856 .780 β.26-.26 [β.35,β.15][-.35,-.15] .17 .43 random .377 .723 .695 .682 .699 .733 .651 .12 [.08,.16][.08,.16] .25 .14 lexical .445 .702 .705 .685 .709 .740 .664 .34 [.30,.38][.30,.38] .47 .14 rule .490 .729 .726 .729 .743 .753 .695 .37 [.34,.40][.34,.40] .50 .14 combo .568 .832 .805 .812 .849 .832 .783 .47 [.45,.49][.45,.49] .50 .03 Llama-3.1-8B none .007 .712 .695 .664 .524 .702 .551 .00 [.00,.01][.00,.01] .01 .00 permit .017 .777 .760 .719 .640 .767 .614 .00 [.00,.01][.00,.01] .01 .01 budget .195 .812 .805 .805 .699 .801 .686 .01 [.01,.02][.01,.02] .02 .00 decide .116 .394 .548 .507 .387 .455 .401 .02 [.00,.04][.00,.04] .04 .02 random .209 .743 .743 .705 .610 .764 .629 .09 [.07,.11][.07,.11] .09 .00 lexical .613 .760 .726 .729 .777 .736 .724 .45 [.43,.48][.43,.48] .45 .00 rule .610 .812 .719 .726 .812 .832 .752 .46 [.44,.48][.44,.48] .46 .00 combo .692 .805 .774 .784 .784 .825 .777 .51 [.49,.53][.49,.53] .51 .00 Qwen3-8B none .051 .832 .812 .805 .764 .842 .684 β.02-.02 [β.03,.00][-.03,.00] .00 .02 permit .212 .880 .873 .849 .866 .887 .761 .00 [β.03,.03][-.03,.03] .06 .06 budget .613 .887 .897 .904 .887 .894 .847 .04 [.02,.06][.02,.06] .05 .01 decide .736 .849 .788 .771 .894 .890 .821 β.22-.22 [β.37,β.03][-.37,-.03] .31 .53 random .216 .853 .836 .818 .812 .846 .730 .07 [.04,.09][.04,.09] .09 .02 lexical .592 .849 .839 .822 .860 .842 .801 .41 [.38,.43][.38,.43] .43 .02 rule .658 .873 .853 .829 .873 .870 .826 .49 [.47,.50][.47,.50] .51 .02 combo .774 .904 .897 .880 .897 .908 .877 .51 [.49,.52][.49,.52] .52 .01 Table 14: FEVER (292 claims, 8-action budget, every parameter carried over from HotpotQA), raw analysis: success per regime, mean6 and integration index for every arm. n=292n=292 claims per arm; no plausible regime. Columns as in Table 13. HotpotQA FEVER Model Contrast Ξ mean6 95% CI p Ξ mean6 95% CI p Qwen2.5-7B rule β- none +.045+.045 [+.029,+.062][+.029,+.062] <<.001 +.129+.129 [+.104,+.154][+.104,+.154] <<.001 rule β- permit +.023+.023 [β.005,+.051][-.005,+.051] .055 +.138+.138 [+.104,+.172][+.104,+.172] <<.001 rule β- budget β.091-.091 [β.124,β.060][-.124,-.060] 1.000 β.024-.024 [β.055,+.007][-.055,+.007] .938 rule β- random +.011+.011 [.000,+.022][.000,+.022] .029 +.044+.044 [+.032,+.057][+.032,+.057] <<.001 rule β- lexical +.012+.012 [β.002,+.026][-.002,+.026] .043 +.031+.031 [+.019,+.043][+.019,+.043] <<.001 combo β- rule +.138+.138 [+.104,+.173][+.104,+.173] <<.001 +.088+.088 [+.059,+.118][+.059,+.118] <<.001 combo β- budget +.047+.047 [+.028,+.068][+.028,+.068] <<.001 +.064+.064 [+.046,+.083][+.046,+.083] <<.001 Llama-3.1-8B rule β- none +.064+.064 [+.046,+.082][+.046,+.082] <<.001 +.201+.201 [+.178,+.224][+.178,+.224] <<.001 rule β- permit +.059+.059 [+.033,+.086][+.033,+.086] <<.001 +.138+.138 [+.115,+.163][+.115,+.163] <<.001 rule β- budget +.016+.016 [β.013,+.047][-.013,+.047] .157 +.066+.066 [+.041,+.090][+.041,+.090] <<.001 rule β- random +.029+.029 [+.013,+.044][+.013,+.044] <<.001 +.123+.123 [+.103,+.142][+.103,+.142] <<.001 rule β- lexical β.001-.001 [β.014,+.013][-.014,+.013] .539 +.028+.028 [+.012,+.044][+.012,+.044] <<.001 combo β- rule +.009+.009 [β.019,+.037][-.019,+.037] .286 +.026+.026 [+.001,+.050][+.001,+.050] .024 combo β- budget +.025+.025 [+.006,+.043][+.006,+.043] .004 +.091+.091 [+.070,+.112][+.070,+.112] <<.001 Qwen3-8B rule β- none +.026+.026 [+.012,+.041][+.012,+.041] .001 +.142+.142 [+.124,+.160][+.124,+.160] <<.001 rule β- permit +.051+.051 [+.020,+.079][+.020,+.079] .001 +.064+.064 [+.045,+.084][+.045,+.084] <<.001 rule β- budget +.009+.009 [β.018,+.037][-.018,+.037] .249 β.021-.021 [β.043,+.001][-.043,+.001] .970 rule β- random +.021+.021 [+.008,+.033][+.008,+.033] .002 +.096+.096 [+.082,+.110][+.082,+.110] <<.001 rule β- lexical β.011-.011 [β.022,+.001][-.022,+.001] .963 +.025+.025 [+.015,+.036][+.015,+.036] <<.001 combo β- rule +.004+.004 [β.023,+.032][-.023,+.032] .389 +.051+.051 [+.031,+.071][+.031,+.071] <<.001 combo β- budget +.013+.013 [.000,+.028][.000,+.028] .028 +.030+.030 [+.017,+.043][+.017,+.043] <<.001 Qwen3-32B rule β- none +.063+.063 [+.045,+.080][+.045,+.080] <<.001 β rule β- permit +.055+.055 [+.026,+.084][+.026,+.084] .001 β rule β- budget β.009-.009 [β.035,+.017][-.035,+.017] .744 β combo β- rule +.006+.006 [β.022,+.033][-.022,+.033] .346 β combo β- budget β.003-.003 [β.019,+.012][-.019,+.012] .640 β Claude Haiku 4.5 rule β- none +.097+.097 [+.080,+.114][+.080,+.114] <<.001 β Claude Sonnet 5 rule β- none +.022+.022 [β.003,+.052][-.003,+.052] .051 β Table 15: Paired contrasts in mean6 between arms, raw analysis: difference (first arm minus second), 95% interval over 2,000 question-level bootstrap resamples, and one-sided bootstrap p for a positive difference (unadjusted; the main text applies Holm correction across models where pre-registered). n=300n=300 paired questions on HotpotQA (Claude Sonnet 5: 100) and 292 on FEVER; Qwen3-32B, Claude Haiku 4.5 and Claude Sonnet 5 were not run on FEVER. 8-action budget 16-action budget Model Arm mean3 pers. ans. med. final calls I mean3 pers. ans. med. final calls I Qwen2.5-7B none .208 .043 156 2 .00 3.9 β.35-.35 .219 .043 158 2 .01 7.1 β.35-.35 budget .300 .090 175 8 .75 6.8 β.02-.02 .312 .093 232 16 .75 13.4 β.03-.03 rule .246 .100 262 2 .00 2.8 .09 .249 .093 260 2 .00 2.8 .10 combo .357 .163 282 6 .00 4.7 .45 .357 .130 267 6 .00 4.7 .44 Llama-3.1-8B none .241 .000 4 4.5 .25 7.9 .00 .298 .010 16 13 .06 15.6 .00 budget .283 .057 215 8 .79 7.0 .03 .299 .050 183 16 .70 14.4 .00 rule .314 .143 285 6 .01 5.1 .47 .377 .170 296 6 .00 5.2 .47 combo .309 .177 296 6 .03 5.0 .48 .349 .177 294 6 .00 5.0 .49 Qwen3-8B none .382 .007 8 3 .12 7.9 β.01-.01 .393 .013 13 5 .08 15.5 β.01-.01 budget .426 .203 235 8 .51 5.8 β.03-.03 .433 .183 198 16 .56 12.0 β.05-.05 rule .433 .153 284 6 .00 5.1 .46 .442 .160 286 6 .00 5.5 .46 combo .447 .270 300 6 .01 4.3 .42 .436 .220 300 6 .00 4.4 .42 Qwen3-32B none .438 .027 34 6 .18 7.6 .01 β budget .551 .330 298 6 .04 4.6 .29 .551 .347 300 7 .01 6.7 .10 rule .526 .267 294 6 .00 5.0 .48 .543 .290 294 6 .00 5.0 .48 combo .533 .310 298 6 .00 4.2 .51 .546 .323 298 6 .00 4.3 .46 Table 16: Budget scaling on HotpotQA (test300, raw analysis): each arm with the 8- and the 16-action budget. The 16-action runs cover three regimes (persistent, recover_after_2, clean), so mean3 averages these three at both budgets in place of mean6. On the persistent regime: success (pers.), number of the 300 questions on which the agent answers rather than exhausting its budget or failing to produce a well-formed action (ans.), median action at which it answers, among those (med., 1-based), share of those answers given at the final action (final), and mean tool calls (search and lookup) per question (calls). I is computed as in Table 13 but over the persistent and recover_after_2 regimes only, with the final action of the respective budget excluded; its 8-action values therefore differ from Table 13. n=300n=300 per cell; Qwen3-32B without intervention was not run with 16 actions. D.2 Fixed step caps, the best simple policy, and whether stopping pays These analyses are exploratory. They evaluate stopping policies offline on the unaided runs: a policy that stops at a given decision point is credited with the answer-now branch replayed from that point (the same forced-finish instruction the rule uses; Appendix B.1), or with the unaided runβs own outcome if the agent had already answered. Answer-now branches exist for four regimes, so these comparisons average those (mean4). Continue or answer now. Would stopping on the agentβs own judgments have paid? We take the first decision point in each unaided run where the agentβs replayed judgments reach five consecutive USELESS (the four regimes with answer-now branches). There we compare the success of answering right then (the answer-now branch) with that of what the agent actually did; it almost always continued. Answering then succeeds more often for all three models: Qwen2.5-7B 0.171 against 0.065 (difference +0.106+0.106 [+0.065,+0.145+0.065,+0.145]; n=461n=461; answering better on 67 and continuing on 18 decisions); Llama-3.1-8B 0.159 against 0.005 (difference +0.154+0.154 [+0.119,+0.191+0.119,+0.191]; n=610n=610; answering better on 96 and continuing on 2 decisions); Qwen3-8B 0.149 against 0.013 (difference +0.136+0.136 [+0.104,+0.169+0.104,+0.169]; n=544n=544; answering better on 77 and continuing on 3 decisions). Intervals are 95% question-bootstrap. Continuing spent 2.5β2.6 further tool calls on average. The comparison is exploratory and specific to sources that, once failed five times, do not recover. Best simple policy. Table 17 evaluates offline, on the unaided runs, every policy that answers after the c-th observation (cap c), after k consecutive useless side-channel judgments (own k; judgments replayed on the unaided trajectories), or whichever comes first, using the answer-now branch at the decision where it fires (the four regimes with answer-now branches: clean, persistent, plausible and recover_after_2). Utility is success minus Ξ» times tool calls; the best policy is chosen on the same data. Regret is the best policyβs utility minus the armβs (the armsβ own runs; negative means the arm is better). The offline own-5 policy reproduces the rule arm (success .378 vs. .372, .271 vs. .271, .216 vs. .224 for Qwen3-8B, Llama-3.1-8B, Qwen2.5-7B). regret of Model Ξ» best policy utility none permit budget cost rule combo Qwen2.5-7B 0.0 own 7 0.232 +0.057 +0.028 β-0.039 β-0.042 +0.008 β-0.097 0.02 cap 3 0.174 +0.082 +0.041 +0.005 +0.003 +0.014 β-0.073 0.05 cap 3 0.108 +0.140 +0.080 +0.092 +0.090 +0.043 β-0.017 Llama-3.1-8B 0.0 own 3 or cap 5 0.322 +0.129 +0.120 +0.075 +0.074 +0.051 +0.050 0.02 own 3 or cap 5 0.252 +0.188 +0.153 +0.114 +0.113 +0.085 +0.075 0.05 own 3 or cap 4 0.148 +0.277 +0.205 +0.173 +0.171 +0.138 +0.114 Qwen3-8B 0.0 cap 7 0.388 +0.071 +0.027 +0.011 +0.023 +0.015 β-0.013 0.02 own 3 or cap 7 0.321 +0.112 +0.018 +0.025 +0.040 +0.034 β-0.009 0.05 own 3 or cap 5 0.227 +0.180 +0.010 +0.052 +0.071 +0.067 +0.004 Table 17: Distance from the best simple stopping policy (exploratory; mean over clean, persistent, plausible, recover_after_2). cost: the stated-cost arm, whose prompt gives the scoring with Ξ»=0.05Ξ»=0.05. Calls under a doubled budget. Table 18 gives tool calls per trajectory at 8 and 16 actions: the budget cueβs calls grow with the budget, whereas the ruleβs and the combinationβs barely change. Model budget rule combo Qwen2.5-7B 5.09 β 8.69 2.88 β 2.91 4.06 β 4.10 Llama-3.1-8B 5.45 β 8.81 5.11 β 5.39 4.74 β 4.99 Qwen3-8B 4.11 β 6.62 4.10 β 4.27 3.54 β 3.62 Table 18: Tool calls per trajectory with an 8- and a 16-action budget (persistent, recover_after_2 and clean, the regimes run at both budgets; exploratory). D.3 The rule on stated judgments, and Claude Haiku 4.5 with the rule or cost stated Amendment 19 replaces the enforced ruleβs side-channel question with a reading of the judgment the agent already writes in its reasoning; amendment 20 runs the stated-rule and call-cost conditions on Claude Haiku 4.5. Both were frozen before their test runs (Table 5). Reading the stated judgment. After every observation the agent writes its usual thought and action. Before the action is executed, a keyword reader (no model call) classifies the first two sentences of the thought as useless (the agent says the result does not help), useful (it reports something usable) or no judgment, and keeps the run of stated-useless judgments: useless adds one, useful resets it, no judgment leaves it unchanged. When the run reaches five and the proposed action is a search or lookup, that turn is discarded and the agent is asked again with the enforced ruleβs finish instruction, so the rule fires at the same step as with the side channel. The reader was written on development data only: on the 1,675 adjudicated stated judgments of Qwen3-8Bβs development trajectories its useless readings have precision 0.971 and recall 0.913. The stated-judgment combination adds the budget statement. Agents do not always write a judgment: the reader finds none at 0.25, 0.15 and 0.22 of decision points for Llama-3.1-8B, Qwen3-8B and Qwen3-32B, but at 0.74 for Qwen2.5-7B, whose thoughts are mostly empty (0.39 when told its budget). We count every call. The side-channel rule makes one judgment call after each observation, and the stated-judgment rule one extra generation each time it fires. Utility is success minus 0.05 per tool or extra call, which is the price in the call-cost condition. Table 19 gives success, calls and utility for every condition, and Table 20 the pre-registered tests. Model Condition mean6 tool calls extra calls utility (Ξ»=0.05Ξ»=0.05) Qwen2.5-7B none 0.192 3.63 0.00 +0.010+0.010 budget 0.328 4.90 0.00 +0.083+0.083 enforced rule 0.237 2.94 2.86 β0.053-0.053 combo 0.375 4.03 4.00 β0.027-0.027 stated-judgment rule 0.209 3.46 0.08 +0.033+0.033 budget + stated-judgment rule 0.343 4.42 0.24 +0.110+0.110 Llama-3.1-8B none 0.263 6.13 0.00 β0.043-0.043 budget 0.312 5.38 0.00 +0.043+0.043 enforced rule 0.328 5.31 5.09 β0.192-0.192 combo 0.337 4.88 4.85 β0.150-0.150 stated-judgment rule 0.328 5.33 0.35 +0.044+0.044 budget + stated-judgment rule 0.346 4.90 0.27 +0.088+0.088 Qwen3-8B none 0.448 4.79 0.00 +0.208+0.208 budget 0.464 3.85 0.00 +0.272+0.272 enforced rule 0.474 4.09 4.01 +0.069+0.069 combo 0.478 3.48 3.47 +0.130+0.130 stated-judgment rule 0.484 4.07 0.31 +0.265+0.265 budget + stated-judgment rule 0.478 3.50 0.18 +0.294+0.294 Qwen3-32B none 0.509 4.76 0.00 +0.271+0.271 budget 0.581 3.64 0.00 +0.399+0.399 enforced rule 0.572 4.10 4.05 +0.165+0.165 combo 0.578 3.54 3.54 +0.224+0.224 stated-judgment rule 0.573 4.12 0.26 +0.355+0.355 budget + stated-judgment rule 0.580 3.55 0.06 +0.399+0.399 Table 19: Stopping on the judgment stated in the agentβs reasoning (amendment 19; test300, raw, per question averaged over the six mean6 regimes). Extra calls: side-channel judgments for the enforced rule and the combination, discarded turns for the stated-judgment conditions. E1 E2 E3 E4 Model success vs none success vs enforced rule utility vs enforced rule combination utility vs budget fires (pers.) Qwen2.5-7B +0.018β+0.018^* [+0.011,+0.024+0.011,+0.024] β0.027-0.027 [β0.041,β0.012-0.041,-0.012] +0.086β+0.086^* [+0.070,+0.102+0.070,+0.102] +0.027β+0.027^* [+0.018,+0.037+0.018,+0.037] 0.05 Llama-3.1-8B +0.064β+0.064^* [+0.049,+0.079+0.049,+0.079] +0.000+0.000 [β0.017,+0.018-0.017,+0.018] +0.236β+0.236^* [+0.216,+0.257+0.216,+0.257] +0.045β+0.045^* [+0.032,+0.059+0.032,+0.059] 0.91 Qwen3-8B +0.036β+0.036^* [+0.022,+0.051+0.022,+0.051] +0.010+0.010 [β0.002,+0.021-0.002,+0.021] +0.196β+0.196^* [+0.182,+0.210+0.182,+0.210] +0.022β+0.022^* [+0.009,+0.034+0.009,+0.034] 0.97 Qwen3-32B +0.064β+0.064^* [+0.047,+0.081+0.047,+0.081] +0.001+0.001 [β0.014,+0.016-0.014,+0.016] +0.190β+0.190^* [+0.173,+0.205+0.173,+0.205] 0.0000.000 [β0.007,+0.006-0.007,+0.006] 0.89 Table 20: Pre-registered tests of amendment 19 for the stated-judgment rule (E1βE3) and its combination with the budget (E4); mean6 or utility at Ξ»=0.05Ξ»=0.05; question-paired 95% intervals; β one-sided Holm-corrected p<0.05p<0.05; E2 is a non-inferiority test with margin β0.05-0.05, passed by all four models. By the pre-registered applicability rule, a model whose stated-judgment rule fires on fewer than 20% of persistently failing questions does not count as support. Claude Haiku 4.5 with the rule or the cost stated (amendment 20). The first 200 test300 questions, six regimes, compared with Claude Haiku 4.5βs unaided and enforced-rule runs on the same questions (unaided mean6 0.518, enforced rule 0.611). Under these prompts Claude Haiku 4.5 often wrote its native tool-call markup instead of an action line, or answered in prose without the finish action; the runs therefore read the first tool call as the action and send one fixed format reminder when a turn has no readable action. Both fixes were set before the runs. The earlier unaided and enforced-rule runs lacked them and had 2β3% format failures and a further 2% truncated generations on these questions, which favors the new conditions, most of all the call-cost condition with almost no format failures. With the price stated, utility at Ξ»=0.05Ξ»=0.05 exceeds the unaided agentβs by +0.154+0.154 and the enforced ruleβs by +0.213+0.213 (the latter counting its judgment calls). Table 21 gives the results. The stated-rule verdict uses the rule of amendment 11 unchanged: partial. Claude Haiku 4.5 mean6 β- unaided β- enforced rule design-label contrast answered (pers.) median action stated rule 0.561 +0.043+0.043 [+0.007,+0.077+0.007,+0.077] β0.050-0.050 [β0.086,β0.017-0.086,-0.017] +0.05+0.05 [+0.02,+0.09+0.02,+0.09] 0.565 7 call cost 0.663 +0.145+0.145 [+0.113,+0.182+0.113,+0.182] +0.053+0.053 [+0.023,+0.082+0.023,+0.082] +0.01+0.01 [β0.02,+0.05-0.02,+0.05] 0.985 7 Table 21: Claude Haiku 4.5 with the stopping rule or the call cost stated (amendment 20; first 200 test300 questions, raw; question-paired 95% intervals). D.4 Fresh questions, stated judgments, reasoning mode and Search-R1 Amendments 15β17 were frozen before any of their runs (Table 5). Fresh questions (amendment 16a). Before any run on them, we sampled 300 new questions from the HotpotQA distractor development set with a new random seed, disjoint from every question used before (pilot, development and test; the list is in the code release). We ran them with the settings of test300. Table 22 gives mean6 and the own-judgment contrast Ξ (judgments replayed for none and budget, recorded for the rule) with the design-label contrast; Ξ was the primary endpoint. mean6 Ξ own judgments (second line: design labels) answered after Model none budget rule none budget rule 5 useless (none) Qwen2.5-7B 0.175 0.315 0.205 β0.09-0.09 [β0.16,β0.03-0.16,-0.03] β0.17-0.17 [β0.24,β0.11-0.24,-0.11] +0.31+0.31 [+0.22,+0.38+0.22,+0.38] 3/125 β0.02-0.02 [β0.04,0.00-0.04,0.00] β0.06-0.06 [β0.08,β0.04-0.08,-0.04] +0.11+0.11 [+0.07,+0.14+0.07,+0.14] Llama-3.1-8B 0.234 0.281 0.294 β0.03-0.03 [β0.05,β0.02-0.05,-0.02] β0.14-0.14 [β0.16,β0.11-0.16,-0.11] +0.35+0.35 [+0.33,+0.37+0.33,+0.37] 2/283 β0.01-0.01 [β0.02,0.00-0.02,0.00] β0.03-0.03 [β0.05,β0.01-0.05,-0.01] +0.17+0.17 [+0.14,+0.20+0.14,+0.20] Qwen3-8B 0.422 0.453 0.458 β0.05-0.05 [β0.07,β0.03-0.07,-0.03] β0.19-0.19 [β0.24,β0.14-0.24,-0.14] +0.34+0.34 [+0.31,+0.38+0.31,+0.38] 0/289 β0.01-0.01 [β0.02,0.00-0.02,0.00] β0.10-0.10 [β0.14,β0.06-0.14,-0.06] +0.16+0.16 [+0.12,+0.20+0.12,+0.20] Qwen3-32B 0.477 0.528 0.534 β0.06-0.06 [β0.09,β0.04-0.09,-0.04] β0.05-0.05 [β0.10,+0.01-0.10,+0.01] +0.30+0.30 [+0.26,+0.34+0.26,+0.34] 19/284 β0.02-0.02 [β0.04,0.00-0.04,0.00] β0.05-0.05 [β0.11,0.00-0.11,0.00] +0.19+0.19 [+0.14,+0.22+0.14,+0.22] Table 22: Confirmatory replication on fresh300 (amendment 16a), raw analysis, 95% intervals. Stated and side-channel judgments (amendment 16b). A local Qwen3-32B annotator (non-thinking, temperature 0, the rubric of the LLM annotators) labeled the 327 adjudicated stated-judgment items with five-class ΞΊ 0.842 [0.766, 0.910] and binary (useless or not) ΞΊ 0.874 [0.798, 0.939]. We took 160 decision points per model from the unaided test300 runs (80 after a failed and 80 after a real observation). Where the agent states a judgment in its next thought, that judgment agrees with the side-channel judgment replayed at the same point as follows: Qwen2.5-7B 0.87 (ΞΊ 0.60; 99 of 160 explicit); Llama-3.1-8B 0.94 (ΞΊ 0.84; 155 of 160 explicit); Qwen3-8B 0.94 (ΞΊ 0.84; 156 of 160 explicit); Qwen3-32B 0.96 (ΞΊ 0.89; 157 of 160 explicit). Stopping on stated judgments (amendment 18). The same annotator labeled the judgment stated in the next thought at every decision point of the unaided test300 runs (failure regimes; decision index β€6β€ 6, persistent β€7β€ 7). Stated useful or partly useful judgments reset the run; points without an explicit judgment make the run unknown until the next useful one, and decisions with an unknown run are left out. Table 24 gives Ξ and the dissociation computed from these stated judgments next to the side-channel versions. Model explicit Ξ stated Ξ side channel n (all useless / ref.) answered after 5 Qwen2.5-7B 65% β0.08-0.08 [β0.15,β0.03-0.15,-0.03] β0.11-0.11 [β0.17,β0.05-0.17,-0.05] 304 / 136 0/15 Llama-3.1-8B 96% β0.01-0.01 [β0.02,β0.01-0.02,-0.01] β0.02-0.02 [β0.03,β0.02-0.03,-0.02] 1911 / 1029 1/266 Qwen3-8B 99% β0.04-0.04 [β0.06,β0.01-0.06,-0.01] β0.06-0.06 [β0.10,β0.04-0.10,-0.04] 2072 / 380 1/287 Qwen3-32B 98% β0.02-0.02 [β0.05,0.00-0.05,0.00] β0.07-0.07 [β0.10,β0.04-0.10,-0.04] 1787 / 418 16/252 Table 23: Time-matched contrast and dissociation from the judgments agents state in their own reasoning (amendment 18), unaided agents, test300. explicit: share of decision points with an explicit stated judgment. Qwen3-32B mean4 ans. med. prem. Ξ own Ξ design none 0.464 0.11 6 0.02 β0.07-0.07 [β0.10,β0.04-0.10,-0.04] β0.02-0.02 [β0.04,0.00-0.04,0.00] none, thinking 0.540 0.97 3 0.47 +0.01+0.01 [β0.18,+0.17-0.18,+0.17] 0.000.00 [β0.14,+0.12-0.14,+0.12] budget 0.576 0.99 6 0.11 0.000.00 [β0.06,+0.05-0.06,+0.05] β0.03-0.03 [β0.09,+0.02-0.09,+0.02] budget, thinking 0.525 1.00 3 0.77 +0.25+0.25 [β0.01,+0.50-0.01,+0.50] +0.32+0.32 [+0.10,+0.52+0.10,+0.52] Table 24: Qwen3-32B with and without thinking (test300, four regimes). ans.: share answered on a persistently failing source; med.: median answering action there; prem.: share of recover_after_2 questions answered before any real result. Reasoning mode (amendment 15). Qwen3-32B ran with thinking enabled (Qwen3βs recommended thinking sampling, 12,288-token context; up to 8,192 new tokens per step instead of the pre-registered 2,048, deviation 6) on persistent, late_onset_from_3, recover_after_2 and clean; the stated-cost arm was not run (deviation 4). Thinking minus non-thinking, mean over the four regimes: none +0.076+0.076 [+0.038,+0.112+0.038,+0.112], budget β0.051-0.051 [β0.083,β0.018-0.083,-0.018] (Table 24). Search-R1 (amendment 17). We ran the released Search-R1 checkpoint (Qwen2.5-7B-Instruct, PPO, trained on NQ and HotpotQA training questions) with its own prompt, <search>/<information>/<answer> tags and one document per search; knowledge bases, failure regimes, the 8-action budget and test300 as in the main runs; greedy decoding. The side-channel question was asked after every observation on a copy of the conversation. Its judgments call 100% of failed and 89% of real observations useless (pre-registered validity: at most 50% of real ones; the other open modelsβ rule-arm logs on the same six regimes: 35β73%), so its own-judgment contrast (β0.26-0.26 [β0.58,β0.06-0.58,-0.06]) is not interpreted. Unaided, it answers on 53% of persistently failing questions (median action 3); design-label contrast β0.03-0.03 [β0.07,+0.01-0.07,+0.01]. The rule fired 498 times over the six mean6 regimes, and the agent answered after 424 of them; on a persistently failing source it ends without a well-formed answer on 12% of questions (none: 3%). Rule minus none, mean6: +0.024+0.024 [+0.009,+0.041+0.009,+0.041]. D.5 Backup tool and harder failures Amendment 13 was frozen before its runs; all runs are local. Two sources. The prompt adds backup_search[query]: βSearch a second, independent copy of the knowledge base. It is slower and more expensive than search: prefer search, and use backup_search when search does not give you useful results.β The backup searches the intact knowledge base and never fails; the failure schedules apply to the primary search. The switch rule disables the primary search after five consecutive side-channel βuselessβ judgments of its results and appends βThe search tool is no longer available. Use backup_search[query] or finish[answer].β Table 25 reports success, the timing of the first switch on a persistently failing primary, and the own-judgment time-matched contrast for leaving the primary (switching or answering; judgments replayed for the unaided and budget arms, logged for the switch rule). By the pre-registered interpretation, a model βswitchesβ if its share switched by the fourth action is at least 0.5, and otherwise βdoes not abandonβ if the contrastβs interval includes or lies below 0. Qwen2.5-7B and Llama-3.1-8B therefore do not abandon, and Qwen3-8B and Qwen3-32B switch. Switch rule minus none, mean over the six regimes: Qwen2.5-7B +0.011+0.011 [β0.005,+0.026-0.005,+0.026]; Llama-3.1-8B +0.054+0.054 [+0.037,+0.071+0.037,+0.071]; Qwen3-8B +0.016+0.016 [+0.005,+0.026+0.005,+0.026]; Qwen3-32B +0.007+0.007 [β0.008,+0.024-0.008,+0.024]. success persistently failing primary Model Arm clean persistent mean6 switched by 4 ever switched median first answered first Ξ (leave) Qwen2.5-7B none 0.480 0.230 0.383 0.49 0.62 2 0.02 β0.54-0.54 budget 0.490 0.240 0.400 0.50 0.61 3 0.11 β0.38-0.38 switch rule 0.470 0.293 0.393 0.51 0.97 4 0.02 β0.29-0.29 Llama-3.1-8B none 0.497 0.050 0.334 0.17 0.33 4 0.01 β0.06-0.06 budget 0.477 0.130 0.356 0.15 0.54 6 0.31 β0.04-0.04 switch rule 0.540 0.223 0.388 0.19 0.91 6 0.06 +0.26+0.26 Qwen3-8B none 0.593 0.367 0.529 0.73 0.79 2 0.01 β0.09-0.09 budget 0.607 0.347 0.523 0.31 0.37 3 0.44 β0.24-0.24 switch rule 0.600 0.477 0.544 0.73 0.99 2.5 0.01 +0.30+0.30 Qwen3-32B none 0.677 0.597 0.639 0.95 0.99 2 0.01 +0.02+0.02 budget 0.710 0.620 0.672 0.87 0.96 3 0.04 +0.14+0.14 switch rule 0.673 0.603 0.646 0.95 0.99 2 0.01 +0.03+0.03 Table 25: Two-source setting, HotpotQA test300, raw analysis. On a failing primary source: βswitched by 4β, share that switched to the backup by the fourth action; βmedian firstβ, median action of the first switch, among questions where it switched; βanswered firstβ, share that answered before any switch; Ξ (leave), own-judgment contrast for leaving the primary source by either exit. Answerless source and long recovery. In the answerless regime every page lacks the questionβs supporting-fact sentences. The unaided agentβs replayed judgments call these pages useless 97% (Qwen2.5-7B), 81% (Llama-3.1-8B), 86% (Qwen3-8B) and 76% (Qwen3-32B) of the time. After five consecutive useless judgments, it answers on 41 of 202, 9 of 166, 26 of 170 and 26 of 122 questions, respectively. Table 26 gives success and the ruleβs contrast with the unaided agent; the pre-registered prediction that the rule loses in recover_after_5 holds for the Qwen3 models and not for Llama-3.1-8B and Qwen2.5-7B, and non-inferiority in recover_after_4 holds for three models and narrowly fails for Qwen3-8B (lower bound β0.050-0.050). Model Arm answerless rec. after 4 rec. after 5 Qwen2.5-7B none 0.137 0.117 0.087 budget 0.217 0.317 0.200 rule 0.183 0.167 0.093 combo 0.273 0.253 0.163 Llama-3.1-8B none 0.107 0.223 0.117 budget 0.173 0.287 0.187 rule 0.153 0.210 0.153 combo 0.210 0.270 0.167 Qwen3-8B none 0.173 0.473 0.410 budget 0.260 0.447 0.397 rule 0.203 0.453 0.197 combo 0.273 0.420 0.277 Qwen3-32B none 0.227 0.517 0.427 budget 0.353 0.513 0.403 rule 0.253 0.533 0.290 combo 0.347 0.473 0.317 rule β- none Qwen2.5-7B +0.047+0.047 [+0.013,+0.080+0.013,+0.080] +0.050+0.050 [+0.013,+0.087+0.013,+0.087] +0.007+0.007 [β0.020,+0.033-0.020,+0.033] Llama-3.1-8B +0.047+0.047 [+0.003,+0.087+0.003,+0.087] β0.013-0.013 [β0.047,+0.023-0.047,+0.023] +0.037+0.037 [β0.013,+0.090-0.013,+0.090] Qwen3-8B +0.030+0.030 [+0.000,+0.060+0.000,+0.060] β0.020-0.020 [β0.050,+0.013-0.050,+0.013] β0.213-0.213 [β0.273,β0.153-0.273,-0.153] Qwen3-32B +0.027+0.027 [β0.010,+0.063-0.010,+0.063] +0.017+0.017 [β0.030,+0.060-0.030,+0.060] β0.137-0.137 [β0.200,β0.077-0.200,-0.077] Table 26: Answerless source and long recovery, raw analysis. D.6 Second and third runs Amendment 10 repeated the none, budget, rule and combination arms on the three 7β8B models with new sampling seeds (Qwen3) and a new draw of the failed observations (all models), keeping questions, regimes and prompts fixed; it was frozen before the replicate runs. Table 27 gives the replicateβs results and pre-registered tests (raw). Rp1 (rule β- none >0>0, Holm) passes for all three models. Rp2 (rule index lower bound >+0.30>+0.30) fails for Qwen2.5-7B, whose index was 0.180.18 in the original run as well, while the unaided index stays at or below +0.05+0.05. Rp3 (budget arm answers at the deadline, index β€+0.10β€+0.10) passes for all three. Rp4 (combination non-inferior to both components, margin β0.05-0.05) passes five of six tests and fails against the rule for Qwen3-8B (lower bound β0.054-0.054), but passes all six after answer repair (+0.008+0.008 [β0.020,+0.036-0.020,+0.036] for Qwen3-8B). By amendment 10βs interpretation rule, these two conclusions are not stable in the replicate run. All other outcomes are unchanged by the repair. Replicate minus original mean6 is within Β±0.023Β± 0.023 for every model and arm. mean6 (replicate) rule β- none rule I [LB] none I budget median rule design Ξe ^e Model none budget rule combo (Rp1) (Rp2) (Rp2) action (Rp3) Qwen2.5-7B 0.209 0.314 0.257 0.366 +0.048+0.048 [+0.033,+0.063+0.033,+0.063] 0.19 [0.14] β0.31-0.31 8 0.06 Llama-3.1-8B 0.264 0.307 0.325 0.332 +0.061+0.061 [+0.043,+0.078+0.043,+0.078] 0.46 [0.44] 0.000.00 8 0.17 Qwen3-8B 0.455 0.456 0.497 0.473 +0.042+0.042 [+0.029,+0.056+0.029,+0.056] 0.46 [0.44] 0.000.00 8 0.15 replicate β- original mean6 (paired by question) Model none budget rule combo Qwen2.5-7B +0.018+0.018 [β0.006,+0.041-0.006,+0.041] β0.014-0.014 [β0.036,+0.008-0.036,+0.008] +0.021+0.021 [β0.003,+0.044-0.003,+0.044] β0.009-0.009 [β0.031,+0.013-0.031,+0.013] Llama-3.1-8B +0.001+0.001 [β0.017,+0.019-0.017,+0.019] β0.005-0.005 [β0.024,+0.014-0.024,+0.014] β0.003-0.003 [β0.021,+0.014-0.021,+0.014] β0.004-0.004 [β0.024,+0.017-0.024,+0.017] Qwen3-8B +0.007+0.007 [β0.013,+0.026-0.013,+0.026] β0.009-0.009 [β0.036,+0.017-0.036,+0.017] +0.023+0.023 [+0.003,+0.043+0.003,+0.043] β0.004-0.004 [β0.031,+0.021-0.031,+0.021] Table 27: Pre-registered replicate (amendment 10), raw analysis, HotpotQA test300. Top: replicate results and tests; bottom: replicate minus original. eDesign-label contrast, exploratory. Third run. Table 28 repeats amendment 10βs tests on a third run and pools the ruleβs gain over all three runs, with a bootstrap over runs and questions. Model rule β- none (Rp1) rule I [lo] none I budget median / I (Rp3) combo LB vs rule / budget (Rp4) pooled rule β- none Qwen2.5-7B +0.047+0.047 [+0.032,+0.063+0.032,+0.063] 0.17 [0.12] β-0.34 8 / β-0.02 +0.082 / +0.024 +0.047+0.047 [+0.033,+0.061+0.033,+0.061] Llama-3.1-8B +0.068+0.068 [+0.049,+0.086+0.049,+0.086] 0.48 [0.46] +0.00 8 / +0.03 β-0.023 / β-0.001 +0.064+0.064 [+0.048,+0.080+0.048,+0.080] Qwen3-8B +0.044+0.044 [+0.029,+0.059+0.029,+0.059] 0.47 [0.46] 0.00 8 / β-0.01 β-0.011 / +0.004 +0.037+0.037 [+0.022,+0.051+0.022,+0.051] Table 28: Third run with new seeds and failure draws: amendment 10βs tests, raw analysis. Appendix E Licenses and compute HotpotQA [Yang et al., 2018] is distributed under C BY-SA 4.0 and FEVER [Thorne et al., 2018] under C BY-SA 3.0 (Wikipedia text, C BY-SA). Qwen2.5-7B, Qwen3-8B, Qwen3-32B, the Search-R1 framework and the base model of its checkpoint [Jin et al., 2025] are released under Apache 2.0, and Llama-3.1-8B under the Llama 3.1 Community License. Claude Haiku 4.5, Claude Sonnet 5 and the other LLM annotators (Claude Fable 5.1, GPT-5.6-luna and GPT-6-sol) were used through their official APIs under the providersβ terms. All open models ran locally from their official weights, including the Search-R1 checkpoint released by its authors. Local models ran with vLLM on four NVIDIA RTX A5000 GPUs (24 GB; Qwen3-32B with tensor parallelism over all four), about 350 GPU-hours in total including development. No model was trained or fine-tuned. Our code, prompts and question manifests are available under the MIT license at https://github.com/bennidict23/judged-useless-queried-anyway, where the labels will also be released.