Paper deep dive

LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks

Chih-Hsuan Yang, Jingyan Jiang, Cheng-Hau Yang, Vikram Vasudevan, Huihuo Zheng, Venkatram Vishwanath, Rajeev Thakur

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 82

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%

Last extracted: 8/18/2026, 4:54:37 AM

Summary

This paper investigates cost-aware protocol routing for multi-agent Large Language Model (LLM) systems. It compares four collaboration protocols (Baseline, Single, PER, Broadcast) across multiple benchmarks (Omni-MATH 2, JEEBench, SciBench, LAB-Bench) using two solver families (gpt-oss-120b, Gemma-4-31B-it). The study finds that while LLMs can effectively predict failure risk (whether Baseline will fail), they struggle to predict which specific collaboration protocol provides the best value for its cost. Heuristic and learned routers show directional errors (under- or over-escalation), and frozen LLM routers often over-escalate. A self-confidence gate offers a cost-efficient initial escalation decision, but full protocol-specific routing remains an unresolved challenge.

Entities (14)

gpt-oss-120b · model · 99%Gemma-4-31B-it · model · 98%Omni-MATH 2 · benchmark · 98%Baseline · protocol · 97%Broadcast · protocol · 97%PER · protocol · 97%Single · protocol · 97%JEEBench · benchmark · 96%LAB-Bench · benchmark · 96%SciBench · benchmark · 96%AUPRC · metric · 95%AUROC · metric · 95%Fixed-order oracle · method · 95%Self-confidence gate · method · 95%

Relation Signals (13)

Broadcast → istypeof → Protocol

confidence 99% · multi-agent deliberation (Broadcast)

Baseline → istypeof → Protocol

confidence 99% · direct solving (Baseline)

Single → istypeof → Protocol

confidence 99% · iterative self-correction (Single)

PER → istypeof → Protocol

confidence 99% · planner-executor-reviewer collaboration (PER)

gpt-oss-120b → usedin → Omni-MATH 2

confidence 98% · The primary benchmark comprises 4,181 competition-level math problems... using the same gpt-oss-120b solver stack

Self-confidence gate → achievessolverate → 78.0%

confidence 95% · the pre-answer self-confidence gate reaches 78.0% solve at 45K tokens

Fixed-order oracle → achievessolverate → 92.4%

confidence 95% · 92.4% for a retrospective fixed-order oracle

gpt-oss-120b → hasfailureriskpredictionaccuracy → 0.8847 AUROC

confidence 95% · A post-answer, pre-collaboration gpt-oss-120b probe ranks Baseline failures with 0.8847 AUROC

Broadcast → hassolverate →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Multi-agent large language model (LLM) systems can improve reasoning by spending more computation, but deployment requires deciding when extra collaboration is worth its cost. We isolate this decision by running every problem under four protocols while holding the solver fixed within each setting: direct solving (Baseline), iterative self-correction (Single), planner-executor-reviewer collaboration (PER), and multi-agent deliberation (Broadcast). The primary benchmark comprises 4,181 competition-level math problems; paired robustness checks cover four benchmarks spanning competition math, biology, and broader science with two solver families. Across fixed policies, trained routers, and frozen LLM routers, conservative policies under-escalate, whereas higher-solve frozen routers often over-escalate. A post-answer, pre-collaboration gpt-oss-120b probe ranks Baseline failures with 0.8847 AUROC (4,151 parseable cases; 95% CI [0.8732, 0.8955]). The same score remains informative for predicting whether any collaboration helps (0.7683 AUPRC), but is much weaker for identifying PER- or Broadcast-specific value (0.1674 and 0.1041 AUPRC). Separately, the pre-answer self-confidence gate reaches 78.0% solve at 45K tokens, compared with 73.8% at 71.3K for a frozen gpt-oss-120b router and 92.4% for a retrospective fixed-order oracle. Across 10 paired model-condition settings, the oracle adds 23.2-58.3 points of retrospective coverage over Baseline, but protocol profiles vary by task. In the six settings with held-out router evaluations, oracle gaps remain 18.5-28.9 points. Confidence can therefore support initial escalation, while protocol-specific cost-aware routing remains unresolved.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

78,994 characters extracted from source content.

Expand or collapse full text

LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks Chih-Hsuan Yang 1 , Jingyan Jiang 1 , Cheng-Hau Yang 1 , Vikram Vasudevan 2 , Huihuo Zheng 1 , Venkatram Vishwanath 1 , Rajeev Thakur 1 1 Argonne National Laboratory, Lemont, IL, USA 2 Oregon State University, Corvallis, OR, USA bellayang@anl.gov Abstract Multi-agent large language model (LLM) sys- tems can improve reasoning by spending more computation, but deployment requires deciding when extra collaboration is worth its cost. We isolate this decision by running every problem under four protocols while holding the solver fixed within each setting: direct solving (Base- line), iterative self-correction (Single), planner– executor–reviewer collaboration (PER), and multi-agent deliberation (Broadcast). The pri- mary benchmark comprises 4,181 competition- level math problems; paired robustness checks cover four benchmarks spanning competition math, biology, and broader science with two solver families. Across fixed policies, trained routers, and frozen LLM routers, conserva- tive policies under-escalate, whereas higher- solve frozen routers often over-escalate. A post-answer, pre-collaborationgpt-oss-120b probe ranks Baseline failures with 0.8847 AU- ROC (4,151 parseable cases; 95% CI [0.8732, 0.8955]). The same score remains informa- tive for predicting whether any collaboration helps (0.7683 AUPRC), but is much weaker for identifying PER- or Broadcast-specific value (0.1674 and 0.1041 AUPRC). Separately, the pre-answer self-confidence gate reaches 78.0% solve at 45K tokens, compared with 73.8% at 71.3K for a frozengpt-oss-120brouter and 92.4% for a retrospective fixed-order oracle. Across 10 paired model–condition settings, the oracle adds 23.2–58.3 points of retrospective coverage over Baseline, but protocol profiles vary by task. In the six settings with held-out router evaluations, oracle gaps remain 18.5– 28.9 points. Confidence can therefore support initial escalation, while protocol-specific cost- aware routing remains unresolved. 1 Introduction Large language model (LLM) reasoning systems can spend drastically different amounts of com- putation on the same problem. A direct solver may cost tens of thousands of tokens, while self- correction or multi-agent deliberation can cost an order of magnitude more. Stronger protocols are of- ten more accurate, but deploying the strongest pro- tocol everywhere silently multiplies cost. Deploy- ment therefore requires deciding both whether to escalate and which collaboration protocol is worth its marginal cost. We build a matched benchmark of 4,181 competition-level math problems, each evaluated under four protocols with sharply different token costs and solve rates. Observing every proto- col on every problem permits offline evaluation against realized solve and cost outcomes and a retrospective fixed-order oracle. This design re- veals a directional failure hidden by routing ac- curacy: cost-conservative policies under-escalate, whereas higher-solve frozen LLM routers often over-escalate. These observations motivate a sharper distinc- tion. Failure-risk prediction asks whether Baseline will be wrong; collaboration-value prediction asks whether, and which, stronger protocol justifies its added cost. A full-benchmark post-answer probe directly supports the first claim, but its precision de- grades for increasingly protocol-specific value tar- gets. Matched outcome checks extend across four benchmarks spanning competition math, biology, and broader science, using bothgpt-oss-120band Gemma-4-31B-it(Gemma Team, 2026; Laurent et al., 2024; Arora et al., 2023; Wang et al., 2024). The more complete confidence and held-out-router evaluations remain a targeted six-setting subset, so this breadth is evidence of task dependence rather than universality. Several related literatures vary different infer- ence components. Model routing selects among models or cascades with different capabilities and prices (Chen et al., 2024; Ong et al., 2025; Somer- step et al., 2025; Song et al., 2025). Adaptive com- putation and sample routing vary reasoning depth or the number of sampled paths (Wang et al., 2023; 1 arXiv:2608.14927v1 [cs.AI] 14 Aug 2026 Balachandran et al., 2025; Wu et al., 2025). Tool and multi-agent routing vary tools, roles, or col- laboration structures (Yue et al., 2025; Gan et al., 2025; Zhu et al., 2025). Calibration work instead asks whether confidence tracks correctness (Joshi et al., 2025; Li et al., 2025a,b). Our setting holds the solver family fixed within each comparison and changes only the collaboration protocol. This iso- lates protocol value from a change in base model capability; Supplementary Table S1 summarizes these distinctions. Contributions. First, we provide matched four- protocol outcomes and solve–cost evaluation that keep the solver fixed within each setting, with paired coverage checks across four benchmarks. Second, we identify asymmetric under- versus over- escalation across heuristic, learned, and frozen- LLM routers. Third, we empirically separate fail- ure risk from protocol-specific collaboration value: same-model confidence supports an initial stay- or-escalate decision, while substantial fixed-order- oracle gaps and weak protocol-specific precision leave full cost-aware routing open. The source package includes machine-readable aggregate ta- bles, and the companion dataset archive contains anonymized protocol traces and outcome labels for the matched comparisons. 2 Task, Benchmark, and Metrics Protocol routing. A router selects one action before observing protocol outcomes. Fixed policies are degenerate routers; learned and frozen-LLM routers map problem text and/or allowed metadata to BASELINE, SINGLE, PER, BROADCAST, or None. TheNoneaction abstains and can be both an oracle label and a router prediction. Our main benchmark uses the clean exact- answer Omni-MATH 2 subset, a manually revised release derived from Omni-MATH, with 4,181 competition-level math problems (Gao et al., 2024; Ballon et al., 2026). Router-visible metadata in- cludes source, domain path, numeric difficulty, and a ten-level difficulty tier; real examples include source valuescayley,fermat, andpascal; one domain path isMathematics -> Algebra -> Prealgebra -> Simple Equations. For the LAB-Bench robustness study, visible fields are dataset name, domain, slice/subset, original identi- fier, prompt condition, and subtask, e.g.,Biology -> LAB-Bench -> CloningScenarios . Routers and confidence probes never receive gold answers, correctness labels, oracle labels, or protocol out- comes. Protocols. In the main benchmark, every prob- lem is run once under four protocols with the samegpt-oss-120bsolver stack (OpenAI, 2025). BASELINE is one direct attempt without self- correction. SINGLE adds iterative self-correction. PER uses planner, executor, and reviewer roles. BROADCAST uses multi-agent deliberation with shared candidates and peer approval.On the main held-out split, these protocols average 18.2K, 47.6K, 401.9K, and 622.1K tokens and solve 56.3%, 78.5%, 84.9%, and 88.9%, respectively. Runs use temperature 0.0; protocol definitions and compute accounting are detailed in the supplement. Fixed-order oracle.For each problem, the retro- spective oracle is the first successful action in the aggregate cost order BASELINE<SINGLE<PER <BROADCAST; it isNoneif all four fail. This is a matched, single-realization diagnostic upper bound, not a deployable policy, a per-instance minimum- token oracle, or expected success under repeated sampling. On the full benchmark, its labels are 56.8% Baseline, 23.0% Single, 8.8% PER, 4.2% Broadcast, and 7.3% None. Routers and evaluation. The main learned- router comparison uses a stratified 80/10/10 split (seed 42; 3,342/416/423 problems). Tier-majority predicts each difficulty tier’s train-split major- ity fixed-order-oracle label, falling back to the train-split global majority.It is a transparent metadata-only sanity check, not a deployed heuris- tic. Learned routers are five-class logistic regres- sions over allowed metadata, with optional TF– IDF word uni- and bigrams from the problem text; model and hyperparameter selection use the de- velopment split only. Frozen routers receive the same text and metadata in a label-only prompt. The cost-aware ablation also provides numeric average protocol costs and short routing examples. We report solve rate, average tokens, and excess tokens, the mean positive per-problem token over- payment relative to the realized oracle. Under- and over-escalation mean choosing cheaper and costlier actions than the oracle.Nonehas zero protocol cost and ranks below Baseline; router or probe overhead still counts. Hence, choosing a protocol when the oracle isNoneis over-escalation, and all its tokens are excess. 2 12345678910 Difficulty tier (1 easiest → 10 hardest) 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of tier (a) Oracle action composition 050100150200250 Δ avg tokens vs Tier-majority (K) −5 0 5 10 15 20 25 30 Δ solve rate (p) (b) Solve--cost frontier Tier-majority Metadata-only Text+metadata Text+metadata balanced gpt-oss-120b gpt-oss-20b Llama-3.1-70B Gemma-4-E4B Gemma-3-27B gpt-oss + cost Self-conf. gate Oracle BaselineSinglePERBroadcastNone Figure 1: (a) Fixed-order-oracle labels by difficulty tier; fractions sum to 1 within each tier. (b) Solve–cost frontier relative to Tier-majority; upper-left is better. The self-confidence gate improves the low-cost frontier, while larger frozen routers buy additional solves through substantially more token use. Gemma-4-E4B here is a frozen router over the main gpt-oss traces, not the Gemma-4-31B-it solver used in Section 5. ClassPolicySolve (%) [95% CI]Avg. tokens (K) [95% CI]Excess (K) [95% CI] FixedBaseline56.3 [51.5, 61.0]18.2 [17.1, 19.2]1.7 [1.1, 2.4] HeuristicTier-majority65.0 [60.5, 69.5]28.9 [26.0, 31.9]5.7 [3.9, 7.7] Frozen LLM gpt-oss-120b73.8 [69.5, 77.5]71.3 [56.5, 86.7]37.1 [24.2, 50.6] Frozen LLM gpt-oss + cost prompt78.3 [74.2, 82.0]88.6 [71.0, 107.6]51.6 [35.2, 68.9] ConfidenceSelf-confidence gate78.0 [74.0, 81.8]45.0 [41.6, 48.7]14.8 [12.2, 17.4] RetrospectiveFixed-order oracle92.4 [89.8, 95.0]101.1 [80.1, 124.5]0.0 [0.0, 0.0] Table 1: Representative routing policies on the primary held-out test split (n = 423). Intervals use 2,000 problem- level bootstrap resamples. Excess is positive token overpayment above the realized fixed-order oracle. The protocol executions and router/probe calls use deterministic decoding, so the intervals do not measure fresh-run variability. 3 Routing Errors and Cost Tradeoffs We compare fixed and heuristic policies, lightweight trained routers, frozen LLM routers, a cost-aware prompt, and the confidence policy introduced in Section 4. Figure 1 shows the oracle composition by tier and the resulting solve–cost frontier. Table 1 reports representative policies with uncertainty; the supplement gives the full grouped comparison. Cheap references are competitive. Baseline solves 56.3% at 18.2K average tokens.Tier- majority raises solve rate to 65.0% at 28.9K tokens. The selected metadata-only and text+metadata lo- gistic routers solve 60.8% and 61.2%, respectively, below this simple reference. Balanced and embed- ding variants recover more solves only by routing more often to expensive actions; full results and feature ablations appear in the supplement. Frozen routers trade cost for fewer misses.The frozengpt-oss-120brouter reaches 73.8% solve at 71.3K tokens. Showing numeric protocol costs and routing examples raises it to 78.3%, but also to 88.6K tokens and 51.6K excess. Larger cross- family frozen routers reach 80–83% solve while spending 156K–280K tokens on average. No eval- uated router approaches the fixed-order-oracle op- erating point. Errors are directional. Tier-majority under- escalates on 27.4% and over-escalates on 12.5% of test problems.Thegpt-oss-120brouter cuts under-escalation to 18.0% but raises over- escalation to 33.3%. Higher-solve Llama and Gemma frozen routers reduce under-escalation to 6–11% while over-escalating on 63–71%. Thus, router gains are not interchangeable: conserva- tive policies miss recoverable problems; aggressive routers buy solves through costly escalation. 3 4 Failure Risk Is Not Protocol Value The directional errors raise two different questions. Can the solver recognize that its Baseline answer is likely wrong? If so, can it identify which stronger protocol is worth the added cost? Post-answerfailure-riskprobe. After gpt-oss-120bproduces its Baseline answer, we ask forP(Baseline correct). The probe sees only the problem, allowed metadata, and its own Baseline final answer; it sees no reasoning trace, gold answer, correctness label, oracle label, or collaboration outcome. Of 4,181 matched prob- lems, 4,151 produce parseable scores (99.28%). The 30 unparseable outputs are excluded rather than imputed. Using1− P(correct)as failure risk yields 0.8847 AUROC (95% CI [0.8732, 0.8955]), 0.8950 AUPRC, and 0.0852 expected calibration error. This differs from the pre-answer confidence probe, which asks for single-pass solve probabil- ity before the model sees a Baseline answer. That probe has 0.859 AUROC on 329 usable estimates from the primary 423-example test split. The two results correspond to distinct operating points and are not treated as replicates. Protocol-specific precision is much weaker.Ta- ble 2 applies the same no-leakage failure score to increasingly specific collaboration-value targets. It remains useful for the coarse question of whether any protocol improves on Baseline. For PER and Broadcast, however, AUPRC falls to 0.1674 and 0.1041. These exceed the rare target prevalences, so confidence is not devoid of value signal, but it is not a reliable full protocol selector. A binary gate is useful but narrower.The self- confidence gate uses the earlier pre-answer confi- dence score: keep Baseline at confidence≥ 70, oth- erwise choose Single; missing confidence scores TargetPrev. AUROC AUPRC Baseline fails43.40.88470.8950 Any collaboration helps36.00.85440.7683 PER first success8.80.72590.1674 Broadcast-only success4.20.76390.1041 Table 2: The same post-answer failure score on in- creasingly specific targets (n = 4,151 parseable cases). Prevalence is a percentage. “PER first” means Base- line and Single fail but PER succeeds; “Broadcast-only success” means Baseline, Single, and PER fail while Broadcast succeeds in the matched realized runs. also choose Single. The threshold is selected on dev by a knee-point rule (Satopä et al., 2011; Deb and Gupta, 2011). On the full test split it reaches 78.0% solve at 45.0K tokens, versus 73.8% at 71.3K for the frozengpt-ossrouter (Table 1). The over- lapping solve intervals support a cost-efficiency comparison, not a claim of higher solve rate. We reserve self-confidence gate for this binary policy; two-threshold cascade denotes the separate supple- mentary ablation that uses Baseline, Single, and Tier-majority with high/low thresholds of 70/10. The gate does not choose PER, Broadcast, or None. Its 14.4-point gap to the fixed-order oracle and the target-specific results identify the remain- ing problem: estimating the marginal value of each expensive protocol after the initial escalation deci- sion. 5 Matched Breadth and Router Robustness To test whether the matched protocol pattern is confined to OmniMath, we extend the analysis to four benchmarks across three broad task families: OmniMath 2 (competition math), JEEBench (engi- neering entrance-exam STEM), SciBench (college- level science), and LAB-Bench (biology multiple- choice).LAB-Bench contributes two no-tool prompt conditions: “strict” includes relevant ev- idence in the prompt, while “text-no-tool” is a broader text-only slice. Bothgpt-oss-120band Gemma-4-31B-itcover all five conditions, yield- ing 10 paired settings. Stronger collaboration has recoverable value in all 10 paired settings (Figure 2): fixed-order-oracle coverage exceeds Baseline by 23.2–58.3 points. The protocol profile varies, however. Broadcast is the strongest fixed protocol in nine settings, while PER exceeds Broadcast by 2.7 points for Gemma on SciBench. This breadth supports task-dependent protocol value, not universal generalization. The deeper confidence and held-out-router anal- yses use the narrower six-setting subset compris- ing both solvers on OmniMath and the two LAB- Bench conditions. We train the same pre-specified text+metadata logistic router using a stratified 70/15/15 split (seed 20260712), dev selection only, and refitting on train+dev. It improves over Base- line by 7.2–37.5 points in all six settings, yet re- mains 18.5–28.9 points below the fixed-order or- acle. Relative to Tier-majority, gains are mixed. Thus, observable text and metadata contain routing 4 Base Single PERBroad.Oracle OmniMath 2 (n=4,181) JEEBench (n=515) SciBench (n=565) LAB strict (n=741) LAB text-no-tool (n=1,542) 56.878.885.2 89.2 92.7 41.555.391.5 95.0 96.5 62.172.287.4 89.7 91.2 19.030.247.6 74.2 77.3 30.555.659.3 81.3 86.4 (a) gpt-oss-120b Base Single PERBroad.Oracle 69.485.990.7 93.0 95.7 70.581.495.9 98.2 99.2 70.379.3 91.3 88.793.5 44.458.269.4 89.3 90.5 42.074.885.0 91.0 96.4 (b) Gemma-4-31B-it Figure 2: Matched solve/coverage percentages across four benchmarks (five prompt conditions), four protocols, and two solver families. Cell hue identifies the protocol, and deeper tint indicates higher solve/coverage. The outlined cell is the strongest deployable fixed protocol in each row. Oracle is retrospective fixed-order coverage computed from the same four realized outcomes. Unlike Table 1, which reports the primary held-out test split, this figure uses all matched problems available in each setting. Each value is one deterministic realized execution per problem–protocol pair. signal but do not yield uniformly reliable protocol- value routing. The expensive protocols are also not interchange- able. Conditional on Baseline and Single both fail- ing, Broadcast exceeds PER by 9.6–10.7 points on OmniMath and 30.8–44.6 points on LAB-Bench. Nevertheless, PER-only successes remain in every setting (2.4–11.1% of these conditional subsets), so Broadcast does not pointwise dominate PER. 6 Discussion and Conclusion The matched design separates two decisions that aggregate routing accuracy can hide. Failure-risk estimation is useful for deciding whether to remain with a cheap direct solver. Collaboration-value es- timation must additionally predict which expensive workflow provides enough marginal benefit. Con- servative and aggressive routers fail in opposite directions, the binary gate covers only the first de- cision, and the broader held-out routers retain large oracle gaps. Our conclusion is therefore diagnostic rather than a claim to have solved full routing. Same- model confidence can support an initial stay-or- escalate gate, but selecting among Single, PER, Broadcast, and None remains a cost-aware control problem. The benchmark, directional error analy- sis, and explicit failure-risk/protocol-value distinc- tion provide measurable targets for that next step. 7 Limitations Our main router comparison and pre-answer con- fidence gate use one solver family and one math benchmark. The broader paired matched-outcome study covers four benchmarks across three broad task families and two solvers. Post-answer con- fidence and held-out router analyses cover the narrower six settings formed by both solvers on OmniMath and the two LAB-Bench conditions. These are targeted robustness checks, not a claim of broad universality. Each problem–protocol pair has one deterministic realized outcome. The 2,000- resample intervals quantify variation over bench- mark problems, not fresh-run stochastic variability. The fixed-order oracle is retrospective and uses an aggregate cost order; it is neither deployable nor an estimate of per-instance expected utility. We measure cost primarily in logged tokens. La- tency, monetary price, energy, parallelism, and an- swer quality beyond exact correctness can change deployment preferences. The post-answer probe excludes 30 unparseable outputs from its proba- bility metrics, and confidence quality varies sub- stantially by model and domain. Finally, the ob- served PER–Broadcast differences do not identify a causal mechanism; understanding why protocol value changes by task remains open. Acknowledgments This research used resources of the Argonne Lead- ership Computing Facility, a U.S. Department of Energy (DOE) Office of Science user facility at 5 Argonne NationalLaboratory (ANL) operated un- der Contract No. DE-AC02-06CH11357. References Daman Arora, Himanshu Singh, and Mausam. 2023. Have LLMs advanced enough? a challenging prob- lem solving benchmark for large language models. In Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pages 7527–7543. Association for Computational Linguis- tics. Vidhisha Balachandran, Jingya Chen, Lingjiao Chen, Shivam Garg, Neel Joshi, Yash Lara, John Langford, Besmira Nushi, Vibhav Vineet, Yue Wu, and Safoora Yousefi. 2025. Inference-time scaling for complex tasks: Where we stand and what lies ahead. arXiv preprint arXiv:2504.00294. Marthe Ballon, Andres Algaba, Brecht Verbeken, and Vincent Ginis. 2026. Benchmarks saturate when the model gets smarter than the judge. arXiv preprint arXiv:2601.19532. Lingjiao Chen, Matei Zaharia, and James Zou. 2024. FrugalGPT: How to use large language models while reducing cost and improving performance. Transac- tions on Machine Learning Research. Kalyanmoy Deb and Shivam Gupta. 2011. Understand- ing knee points in bicriteria problems and their impli- cations as preferred solution principles. Engineering Optimization, 43(11):1175–1204. Bingzheng Gan, Yufan Zhao, Tianyi Zhang, Jing Huang, Li Yusu, Shu Xian Teo, Changwang Zhang, and Wei Shi. 2025. Master: A multi-agent system with LLM specialized MCTS. In Proceedings of the 2025 Con- ference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 9409–9426, Albuquerque, New Mexico. Asso- ciation for Computational Linguistics. Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. 2024. Omni-MATH: A univer- sal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985. Gemma Team. 2026. Gemma 4 technical report. arXiv preprint arXiv:2607.02770. Abhinav Joshi, Areeb Ahmad, and Ashutosh Modi. 2025. Calibration across layers: Understanding cal- ibration evolution in LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 14686–14714, Suzhou, China. Association for Computational Linguistics. Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Sid- dharth Narayanan, Manvitha Ponnapati, Andrew D. White, and Samuel G. Rodriques. 2024. LAB-Bench: Measuring capabilities of language models for biol- ogy research. arXiv preprint arXiv:2407.10362. Chengzu Li, Han Zhou, Goran Glavaš, Anna Korho- nen, and Ivan Vuli ́ c. 2025a. Large language models are miscalibrated in-context learners. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11575–11596, Vienna, Austria. Associa- tion for Computational Linguistics. Yingjie Li, Yun Luo, Xiaotian Xie, and Yue Zhang. 2025b. Task calibration: Calibrating large language models on inference tasks. In Findings of the Asso- ciation for Computational Linguistics: ACL 2025, pages 6937–6951, Vienna, Austria. Association for Computational Linguistics. Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. 2025. RouteLLM: Learning to route LLMs with preference data. In International Conference on Learning Representations. OpenAI. 2025. gpt-oss-120b model. Accessed: 2026- 05-23. Ville Satopä, Jeannie Albrecht, David Irwin, and Barath Raghavan. 2011. Finding a “kneedle” in a haystack: Detecting knee points in system behav- ior. In 2011 31st International Conference on Dis- tributed Computing Systems Workshops, pages 166– 171. IEEE. Seamus Somerstep, Felipe Maia Polo, Allysson Flavio Melo de Oliveira, Prattyush Mangal, Mírian Silva, Onkar Bhardwaj, Mikhail Yurochkin, and Subha Maity. 2025. CARROT: A cost aware rate optimal router. arXiv preprint arXiv:2502.03261. Wei Song, Zhenya Huang, Cheng Cheng, Weibo Gao, Bihan Xu, GuanHao Zhao, Fei Wang, and Runze Wu. 2025. IRT-router: Effective and interpretable multi-LLM routing via item response theory. In Pro- ceedings of the 63rd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 15629–15644, Vienna, Austria. Asso- ciation for Computational Linguistics. Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. 2024. SciBench: Evaluating college-level scien- tific problem-solving abilities of large language mod- els. In Proceedings of the Forty-First International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 50622–50649. PMLR. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowd- hery, and Denny Zhou. 2023. Self-consistency im- proves chain of thought reasoning in language mod- 6 els. In International Conference on Learning Repre- sentations. Menghua Wu, Cai Zhou, Stephen Bates, and Tommi Jaakkola. 2025. Thought calibration: Efficient and confident test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 14291–14305, Suzhou, China. Association for Computational Linguistics. Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi. 2025. MasRouter: Learning to route LLMs for multi-agent systems. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 15549–15572, Vienna, Austria. Association for Computational Linguistics. Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Xiangru Tang, Heng Ji, and Jiaxuan You. 2025. MultiAgentBench : Evaluating the collabora- tion and competition of LLM agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8580–8622, Vienna, Austria. Association for Computational Linguistics. 7 A Cross-Setting Robustness and Diagnostics This section establishes the scope of the main claims through targeted robustness and diagnos- tic analyses. Unless stated otherwise, confidence intervals use 2,000 percentile bootstrap resamples over problem identifiers. A.1 Positioning Within Routing and Adaptive Inference Table S1 makes explicit which inference compo- nent each nearby line of work changes. Model routing and cascades select among base models (Chen et al., 2024; Ong et al., 2025; Somerstep et al., 2025; Song et al., 2025); adaptive computa- tion and self-consistency allocate steps or samples (Wang et al., 2023; Balachandran et al., 2025; Wu et al., 2025); tool and agent routing changes tools, roles, or collaboration structures (Yue et al., 2025; Gan et al., 2025). Our matched analysis changes the collaboration protocol while holding the solver fixed within a setting and observing every action’s realized outcome. A.2 Matched Protocol Coverage Across Four Benchmarks We evaluate matched four-protocol outcomes on four benchmarks across three broad task families: OmniMath 2 (competition math), JEEBench (engi- neering entrance-exam STEM), SciBench (college- level science), and LAB-Bench (biology multiple- choice) (Ballon et al., 2026; Arora et al., 2023; Wang et al., 2024; Laurent et al., 2024). LAB strict is an LLM-only, no-tool/no-retrieval condition with relevant evidence included in the prompt; LAB text- no-tool is a broader text-only no-tool condition. Bothopenai/gpt-oss-120b(OpenAI, 2025) and google/gemma-4-31B-it(Gemma Team, 2026) cover all five evaluation conditions. Tables S2 and S3 therefore report 10 matched model–condition settings. A.3 Post-Answer Confidence Probes The model-call file and scoring labels are physically separated.The model reads probe_inputs.jsonl,containingproblem text, allowed metadata, and its own Baseline final answer. Correctness and protocol outcomes are stored only inlabels_for_scoring.csvand joined after inference. A schema validator rejects forbidden keys, and the saved leakage checks pass in all six settings. No probe receives a reasoning trace. For the primary gpt-oss OmniMath setting, 4,151 of 4,181 outputs are parseable. Its AUROC inter- val uses seed 20260712; the remaining tables use seed 20260714. Both use 2,000 problem-identifier resamples. The result differs from the 0.859 AU- ROC of the pre-answer confidence probe because that run uses 329 cleaned estimates from the 423- example test split and does not see a Baseline an- swer. The post-answer run uses the full matched setting, includes the Baseline final answer, allows repair attempts, and reaches 99.28% coverage. A.4 Held-Out Text+Metadata Routers For each setting, we stratify 70/15/15 by fixed- order-oracle label using seed 20260712.The pre-specified router family is five-class logistic regression over TF–IDF word unigrams/bigrams (min_df=2, at most 20,000 features) plus a sparse encoding of allowed metadata. The metadata fea- tures include setting/model identifiers and available dataset name, source, domain, difficulty/tier, sub- ject, question type, subset, subtask, and prompt condition. We searchC ∈ 0.25, 1, 4with or without balanced class weights. Selection uses dev macro-F1, then dev solve rate, then oracle gap; the selected configuration is refit on train+dev. No test outcome participates in feature or hyperparameter selection. Protocol-token fields are complete for the gpt- oss outcome files but not for Gemma, so Gemma router costs are not estimated. For gpt-oss, aver- age/excess tokens are 151.7K/105.5K (OmniMath), 556.4K/261.3K (LAB strict), and 266.0K/112.5K (LAB text-no-tool), using the same cost definitions as the main paper. Missing Gemma costs are re- ported as unavailable rather than imputed. A.5 PER–Broadcast Interaction Table S8 focuses on the decision point where both cheaper protocols fail. Broadcast is stronger on average, but PER-only successes occur in every setting. Thus, a global ordering does not imply pointwise dominance. The larger LAB differences are an observed domain dependence; we do not have direct measurements establishing a causal ex- planation. For completeness, in the six confidence/router settings ordered as Gemma OmniMath, gpt-oss OmniMath, Gemma LAB strict, gpt-oss LAB strict, Gemma LAB text-no-tool, and gpt-oss LAB text- 8 FamilyRouted unitFixed solver? Matched outcomes? Main target Model routing/cascadesBase modelNoUsually noQuality–price tradeoff Adaptive computationSteps/depthUsuallyUsually noStop or continue Self-consistency routing Sampled reasoning paths YesUsually noNumber of samples Tool/agent routingTools, roles, topologyVariesUsually noWorkflow selection This workCollaboration protocolYesYesRealized solve–token value Table S1: Scope comparison. “Matched” means that every candidate action is observed for every evaluated problem, enabling paired offline analysis. SettingnBaselineSinglePERBroadcastOracle gpt-oss-120b OmniMath 24,181 56.83 [55.4,58.4] 78.78 [77.6,80.0] 85.17 [84.1,86.2] 89.21 [88.3,90.2] 92.68 [91.9,93.5] JEEBench515 41.55 [37.5,45.8] 55.34 [51.1,59.6] 91.46 [88.9,94.0] 94.95 [93.0,96.7] 96.50 [94.8,98.1] SciBench565 62.12 [58.1,66.0] 72.21 [68.5,75.8] 87.43 [84.6,90.1] 89.73 [87.1,92.2] 91.15 [88.7,93.5] LAB strict741 19.03 [16.1,21.9] 30.23 [27.0,33.7] 47.64 [44.0,51.3] 74.22 [71.1,77.5] 77.33 [74.2,80.3] LAB text-no-tool 1,542 30.54 [28.3,32.9] 55.64 [53.2,58.1] 59.27 [56.7,61.7] 81.26 [79.4,83.2] 86.38 [84.6,88.1] Gemma-4-31B-it OmniMath 24,181 69.39 [68.0,70.8] 85.94 [84.9,87.0] 90.65 [89.8,91.5] 92.99 [92.2,93.8] 95.67 [95.0,96.3] JEEBench515 70.49 [66.6,74.4] 81.36 [78.1,84.7] 95.92 [94.2,97.5] 98.25 [96.9,99.2] 99.22 [98.5,99.8] SciBench565 70.27 [66.6,74.0] 79.29 [75.8,82.5] 91.33 [89.0,93.5] 88.67 [85.8,91.2] 93.45 [91.3,95.4] LAB strict741 44.40 [40.9,47.9] 58.16 [54.5,61.7] 69.37 [66.0,72.6] 89.34 [87.0,91.5] 90.55 [88.4,92.6] LAB text-no-tool 1,542 41.96 [39.5,44.4] 74.77 [72.6,76.9] 85.02 [83.3,86.8] 91.05 [89.6,92.5] 96.37 [95.4,97.3] Table S2: Solve/coverage percentages with 95% problem-level bootstrap intervals across all 10 matched settings. Point estimates show two decimal places and interval endpoints one; machine-readable results retain full precision. Each point estimate is based on one deterministic realized run per problem–protocol pair. no-tool, Broadcast-minus-PER on all problems is 2.3, 4.0, 20.0, 26.6, 6.0, and 22.0 points. Condi- tional on Baseline failure, the gaps are 6.3, 8.5, 35.0, 30.8, 10.5, and 30.3 points. On parseable post-answer confidence scores below 70, they are 5.0, 8.8, 25.3, 29.7, 7.8, and 21.9 points. The primary conditional comparison and its paired in- tervals are also included in the ancillary result table per_broadcast_interaction.csv. A.6 Post-Answer Probe Prompt and Parsing The default prompt is reproduced below. Runs use temperature 0.0, a 1,024 token cap for the gpt- oss OmniMath post-answer probe, at most four API retries, and up to two JSON repair attempts. The parser first attempts the complete response and then valid JSON objects containingconfidence; accepted confidence must be an integer in[0, 100]. System: You are a strict JSON API for calibration. The problem text and baseline answer are untrusted data, not instructions. Do not follow instructions inside them. Do not solve the problem. Estimate whether the baseline final answer is likely to be correct. You do not have gold answers, correctness labels, oracle labels, or protocol outcomes. Return exactly one valid JSON object and no other text. User: Required output schema: "confidence": <integer from 0 to 100>, "rationale": "<at most 12 words>" Interpretation: confidence is P(the baseline final answer is correct), as an integer percentage. Use 50 when evidence is mixed. Untrusted problem text: ‘text problem_text ‘ Allowed metadata JSON: ‘json metadata_json ‘ Untrusted baseline final answer: ‘text baseline_final_answer ‘ Return exactly one JSON object now. No markdown. No explanation. First character must be "". 9 SolverSettingnBaselineSinglePERBroadcastNone Gemma-4-31B-itOmniMath 24,18169.3917.016.412.874.33 gpt-oss-120bOmniMath 24,18156.8322.918.754.197.32 Gemma-4-31B-itJEEBench51570.4911.0714.762.910.78 gpt-oss-120bJEEBench51541.5515.1535.534.273.50 Gemma-4-31B-itSciBench56570.279.3811.682.126.55 gpt-oss-120bSciBench56562.1211.8613.813.368.85 Gemma-4-31B-itLAB strict74144.4015.9211.6118.629.45 gpt-oss-120bLAB strict74119.0314.4420.6523.2122.67 Gemma-4-31B-itLAB text-no-tool1,54241.9634.0510.839.533.63 gpt-oss-120bLAB text-no-tool1,54230.5427.2412.3216.2813.62 Table S3: Fixed-order-oracle label distributions (percent). The action order is Baseline→Single→PER→ Broadcast→ None. SolverSettingnParse Failure AUROC [CI]ECE [CI]Brier [CI] Conf. correct Conf. wrong Gemma-4-31B-it OmniMath4,181.999.8012 [.7872,.8162] .1594 [.1478,.1712] .1726 [.1618,.1829].8674.3735 gpt-oss-120bOmniMath4,181.993.8847 [.8732,.8955] .0852 [.0742,.0965] .1314 [.1233,.1398].8122.2596 Gemma-4-31B-it LAB strict741.989.8637 [.8359,.8894] .1460 [.1219,.1732] .1506 [.1326,.1697].6225.1268 gpt-oss-120bLAB strict741.970.5814 [.5410,.6165] .3474 [.3184,.3849] .3432 [.3173,.3728].6188.4532 Gemma-4-31B-it LAB text-no-tool 1,542 1.000.7337 [.7110,.7562] .1100 [.0891,.1321] .2117 [.1999,.2237].6387.3703 gpt-oss-120bLAB text-no-tool 1,542.979.6069 [.5808,.6347] .2132 [.1919,.2393] .2874 [.2714,.3025].4980.4059 Table S4: Post-answer, pre-collaboration confidence metrics. The AUROC target is Baseline failure and the score is 1− P(Baseline correct). Unparseable outputs are excluded, not imputed. Solver/settingTargetn Prev. AUROC AUPRC Gemma OmniMath Baseline fails4,178.306.8012.5871 Any collaboration helps 4,178.263.7828.5057 PER first success4,178.064.7290.1255 Broadcast-only success4,178.029.7389.0626 gpt-oss OmniMathBaseline fails4,151.434.8847.8950 Any collaboration helps 4,151.360.8544.7683 PER first success4,151.088.7259.1674 Broadcast-only success4,151.042.7639.1041 Gemma LAB strictBaseline fails733.558.8637.8692 Any collaboration helps733.464.7771.7297 PER first success733.116.6932.2186 Broadcast-only success733.188.7875.3370 gpt-oss LAB strictBaseline fails719.819.5814.9040 Any collaboration helps719.590.5611.6589 PER first success719.211.5589.2528 Broadcast-only success719.238.5344.2979 Gemma LAB textBaseline fails1,542.580.7337.8204 Any collaboration helps 1,542.544.6764.7086 PER first success1,542.108.6902.2054 Broadcast-only success1,542.095.8737.3195 gpt-oss LAB textBaseline fails1,510.703.6069.8363 Any collaboration helps 1,510.566.5694.6282 PER first success1,510.126.5541.1630 Broadcast-only success1,510.166.5684.2664 Table S5: The same no-leakage failure score evaluated against coarse and protocol-specific targets. “PER first” means Baseline and Single fail and PER succeeds. “Broadcast-only success” means Baseline, Single, and PER fail while Broadcast succeeds in the matched realized runs. AUPRC should be read relative to target prevalence. SolverSettingn Base Router [95% CI] Oracle Gap Macro-F1 Route B/S/P/R/N (%) Gemma-4-31B-it OmniMath 628.694.766 [.731,.798].957 .191.268 49.8/24.0/12.7/5.6/7.8 gpt-oss-120bOmniMath 628.568.667 [.631,.704].927 .260.284 45.7/19.9/15.6/7.2/11.6 Gemma-4-31B-it LAB strict112.446.705 [.616,.786].911 .205.495 57.1/3.6/6.2/26.8/6.2 gpt-oss-120bLAB strict112.188.562 [.473,.661].768 .205.499 14.3/15.2/29.5/25.0/16.1 Gemma-4-31B-it LAB text232.418.776 [.720,.828].961 .185.485 36.6/43.1/4.3/13.8/2.2 gpt-oss-120bLAB text232.306.573 [.509,.638].862 .289.456 34.9/29.7/3.0/16.4/15.9 Table S6: Held-out text+metadata router evaluation. B/S/P/R/N denote Baseline, Single, PER, Broadcast, and None. Baseline, router, and oracle use identical held-out problem identifiers within each setting. 10 SolverSettingn Router− Tier-majority [95% CI] Router− Baseline [95% CI] Gemma-4-31B-it OmniMath 628+7.2 [4.0,10.2]+7.2 [4.0,10.2] gpt-oss-120bOmniMath 628+3.3 [0.0,6.5]+9.9 [6.5,13.2] Gemma-4-31B-it LAB strict112+25.0 [16.1,33.9]+25.9 [17.9,34.0] gpt-oss-120bLAB strict112−8.0 [−17.0,1.8]+37.5 [28.6,46.4] Gemma-4-31B-it LAB text232+10.8 [5.2,16.4]+35.8 [29.7,42.2] gpt-oss-120bLAB text232+0.9 [−5.2,6.5]+26.7 [20.7,32.8] Table S7: Paired solve-rate differences in percentage points with 2,000 problem-level bootstrap resamples. Perfor- mance relative to Tier-majority is heterogeneous even though every router improves over Baseline. SolverSettingn PER Broad. Broad.−PER [CI] PER-only Broad.-onlyBothNeither Gemma-4-31B-it OmniMath 569 47.157.810.7 [6.0,15.3]59 (10.4)120 (21.1) 209 (36.7) 181 (31.8) gpt-oss-120bOmniMath 847 43.252.89.6 [5.8,13.3]94 (11.1)175 (20.7) 272 (32.1) 306 (36.1) Gemma-4-31B-it LAB strict 294 29.373.844.6 [38.4,51.0]7 (2.4)138 (46.9)79 (26.9)70 (23.8) gpt-oss-120bLAB strict 493 31.062.731.6 [27.0,36.3]16 (3.2)172 (34.9) 137 (27.8) 168 (34.1) Gemma-4-31B-it LAB text370 45.175.930.8 [24.6,37.6]33 (8.9)147 (39.7) 134 (36.2)56 (15.1) gpt-oss-120bLAB text651 29.263.934.7 [30.9,38.9]25 (3.8)251 (38.6) 165 (25.3) 210 (32.3) Table S8: PER versus Broadcast conditional on Baseline and Single both failing. Solve rates, differences, and parenthesized outcome rates are percentages; cells before parentheses give counts. CIs are paired problem-level bootstrap intervals. 11 B Detailed OmniMath Router Analyses This section expands the primary OmniMath 2 anal- ysis. The first subsections report the full routing table, bootstrap intervals, the cost-aware prompt and embedding ablations, marginal-cost account- ing, and router-call token sensitivity. The next sec- tions separate difficulty metadata from lightweight problem-text signal, then show the self-assessment, binary confidence gate, and two-threshold cascade checks. The final sections document benchmark provenance, scope checks, prompts, cleaning, and the dataset archive. For navigation, Table S9 gives the full routing metrics, Table S10 gives bootstrap intervals, Table S11 gives the cost-prompt and em- bedding checks, Table S12 gives the marginal-cost view, Table S13 gives router-call token accounting, Tables S14–S15 give metadata ablations, Figure S2 and Table S16 give self-assessment checks, and Table S17 gives the reduced Gemma-3 actor-stack scope check. Appendix D lists prompts and execu- tion details. Unless a subsection states otherwise, API-based protocol traces, frozen-router calls, cost- aware prompt calls, direct self-assessment probes, and protocol-value probes use deterministic decod- ing with temperature 0.0. B.1 Full Routing Comparison Table S9 expands the main paper’s routing compar- ison. For quick reading, the primary columns are solve rate, tokens, excess cost, under-escalation, and over-escalation. Accuracy and macro-F1 are included to show why flat label quality is a sec- ondary objective for this task. The qualitative pattern matches the main text: cost-conservative baselines are cheap but miss many solvable prob- lems, lightweight text-feature routers do not clearly dominate metadata or tier baselines, and higher- solve frozen LLM routers improve solve rate mainly through over-escalation. The balanced text+metadata variant is especially informative: it improves macro-F1 relative to the early-stopped text router, but spends far more tokens and re- mains behind frozen LLM routers on solve rate. The comparison separates ordinary label-balance improvement from the cost-aware objective of the benchmark.For the twogpt-ossfrozen routers, parse reliability is also part of the interpre- tation:gpt-oss-120bhas a 14.2% fallback rate andgpt-oss-20bhas a 25.5% fallback rate, while the other frozen routers produce parseable labels for this evaluation. Bootstrap intervals and targeted ablations. The confidence intervals in Table S10 use a percentile bootstrap over held-out test prob- lems with 2,000 resamples and seed 42. The sentence-embedding rows in Table S11 use sentence-transformers/all-MiniLM-L6-v2to encode problem text, mean pool the final hid- den states,L 2 -normalize the vectors, and select embedding-only or embedding+metadata kNN/l- ogistic models on the dev split by macro-F1 with lower excess cost as the tie-breaker. Thegpt-oss cost-prompt row is a frozen-router ablation with the same allowed labels as the main frozen-router experiment, but with numeric protocol costs and brief routing examples added to the prompt. The cost-aware prompt improves solve rate over the original frozen prompt, but the prompt still spends nearly twice the tokens of the self-confidence gate at the same solve rate and over-escalates on 45.2% of test problems. We therefore treat the cost-aware prompt as a prompt robustness check rather than the headline router. B.2 Difficulty-Metadata Ablation Table S14 separates calibrated difficulty metadata from lightweight text features. Once explicit dif- ficulty fields are removed, text+metadata remains near the early-stopped full model, while text-only gains solve rate mainly by selecting more costly protocols. Table S15 adds a source-holdout stress 12 LabelSolve and costError direction RouterAcc.F1 Solve Tok. K Excess K Cost/Solve K Missed Under Over Always Baseline56.3 14.456.318.21.732.336.236.27.6 Always Single22.97.578.547.617.360.713.913.263.8 Always PER9.03.384.9401.9308.0473.57.64.386.8 Always Broadcast4.31.688.9622.1523.4699.83.50.095.7 Tier-majority60.0 23.065.028.95.744.529.727.412.5 Tier+source majority59.6 24.064.340.412.762.828.127.912.5 Metadata-only57.2 23.060.844.320.572.934.331.411.3 Text+metadata56.3 23.161.241.618.268.033.831.212.5 Text+metadata balanced48.0 30.166.4141.490.1212.928.124.627.4 gpt-oss-20b50.1 17.160.029.110.948.535.031.918.0 gpt-oss-120b48.7 23.673.871.337.196.720.218.033.3 gpt-oss + cost prompt41.8 23.178.388.651.6113.215.313.045.2 Self-conf. gate44.2 20.078.045.014.857.715.613.742.1 Two-threshold cascade55.1 24.275.241.712.355.518.716.528.4 Llama-3.1-70B25.3 13.080.4155.7114.1193.713.011.163.6 Gemma-4-E4B29.3 19.681.8195.6129.7239.111.57.862.9 Gemma-3-27B22.2 14.082.5279.8196.2339.110.76.471.4 Oracle100.0–92.4101.10.0109.40.00.00.0 Table S9: Extended routing comparison on the held-out test split. Percent-valued columns are percentages; token columns are thousands. Missed is the share of oracle-solvable problems not solved by the selected protocol. Cost/Solve is average tokens per solved problem. RouterSolve (%)Avg tok. (K)Excess (K) Baseline56.3 [51.5, 61.0]18.2 [17.1, 19.2]1.7 [1.1, 2.4] Tier-majority65.0 [60.5, 69.5]28.9 [26.0, 31.9]5.7 [3.9, 7.7] Meta-only60.8 [56.0, 65.0]44.3 [32.3, 58.5]20.5 [10.3, 33.7] Text+meta61.2 [56.5, 66.0]41.6 [30.9, 54.5]18.2 [9.0, 29.5] gpt-oss-120b73.8 [69.5, 77.5]71.3 [56.5, 86.7]37.1 [24.2, 50.6] gpt-oss + cost prompt 78.3 [74.2, 82.0]88.6 [71.0, 107.6]51.6 [35.2, 68.9] Self-conf. gate78.0 [74.0, 81.8]45.0 [41.6, 48.7]14.8 [12.2, 17.4] Llama-3.1-70B80.4 [76.6, 84.2] 155.7 [120.5, 197.6]114.1 [80.6, 153.5] Gemma-4-E4B81.8 [78.0, 85.6] 195.6 [165.5, 225.6] 129.7 [103.0, 157.7] Gemma-3-27B82.5 [78.7, 86.1] 279.8 [248.8, 313.6] 196.2 [167.6, 225.3] Oracle92.4 [89.8, 95.0]101.1 [80.1, 124.5]0.0 [0.0, 0.0] Table S10: Percentile bootstrap confidence intervals for the main comparison, computed with 2,000 problem-level resamples drawn with replacement from the held-out test set. RouterSolveTok. Excess Under Over Fall. (%)(K)(K)(%)(%) Embed-only kNN66.091.854.526.022.90 Embed-only logreg68.6 176.0124.121.733.80 Embed+meta kNN66.052.422.526.515.40 Embed+meta logreg70.9 215.7153.819.634.00 gpt-oss + cost prompt78.388.651.613.045.21 Table S11:Robustness ablations:stronger text features and a cost-aware prompt for the frozen router.Sentence embedding routers recover addi- tional solve signal, but not at the cost-efficiency ofdifficulty_tier_majority. Thegpt-oss-120b numeric-cost/few-shot prompt improves solve rate over the original frozen prompt but still over-escalates sub- stantially. test over the six largest sources. For each held-out source, routers are trained on the original train split excluding that source, selected on the original dev split excluding that source, and evaluated on all examples from the held-out source. No-tier learned routers can improve solve rate on these held-out partitions, but the gain comes with substantially higher token use and over-escalation. Step∆Solve∆Tok. Tok./extra solve (p) (K/problem)(K) Baseline→ Tier-majority8.710.8123 Tier-maj.→ Self-conf.13.016.1124 Self-conf.→ gpt-oss-4.326.4dominated Self-conf.→ Llama2.4110.74,684 Self-conf.→ Gemma-43.8150.63,981 Table S12: Marginal cost of additional solves. The last column divides the increase in average tokens per problem by the increase in solve probability; “domi- nated” means the destination policy is more expensive and solves fewer problems. The marginal-cost view makes the self-confidence gate’s efficiency visible: later frozen-router gains are much more expensive. B.3 Direct Self-Assessment and Confidence Policies The pre-answer confidence probe examines the mechanism behind the directional error pattern. The checks below show four things: self-reported confidence predicts one-shot success, confidence alone does not identify the required collaboration protocol, simple self-confidence gates are useful 13 0 50 100 Rate (%) Tier-majoritygpt-oss-20bgpt-oss-120b 246810 Difficulty tier 0 50 100 Rate (%) Llama-3.1-70B 246810 Difficulty tier Gemma-4-E4B 246810 Difficulty tier Gemma-3-27B Asymmetric escalation errors by tier under-escalationover-escalation Figure S1: Under- and over-escalation by tier for all compared routers. The plot supports the main finding that routing errors are directionally biased rather than uniformly noisy. RouterMain tok. Router tok. Tok. incl. Excess incl. (K)(K)(K)(K) gpt-oss-20b29.10.429.511.3 gpt-oss-120b71.30.471.837.6 gpt-oss + cost prompt88.60.889.352.4 Llama-3.1-70B155.70.4156.1114.4 Gemma-4-E4B195.60.4195.9130.0 Gemma-3-27B279.80.4280.2196.5 Table S13: Router-call token sensitivity for frozen LLM- as-router policies. The main tables count the selected protocol trace cost; the sensitivity table additionally adds each router decision call from the API usage logs. The added router calls are small relative to protocol execution costs, so the solve-cost ordering is unchanged. The self-confidence-gate row in the main tables already includes its self-assessment query cost, averaging 0.34K tokens per problem. but incomplete routers, and the cleaning procedure does not create the hard-tier pattern. Figure S2 shows the two sides of the mechanism. The relia- bility panel explains why confidence is useful for deciding whether a low-cost direct attempt is plau- sible. The oracle-composition panel explains why confidence cannot by itself choose among SINGLE, PER, BROADCAST, andnoneonce the model is unsure. The cross-model rows are confound checks rather than new routers. They show that other capa- ble models can produce parseable estimates for ev- ery test problem, but their estimates remain weaker and less calibrated than same-modelgpt-ossself- assessment. Gemma-3 recovers roughly tier-level RouterSolveTok. ExcessF1 (%)(K)(K) Tier-majority65.028.95.7 23.0 Source-maj. w/o tier62.633.610.8 21.4 Meta-only60.844.320.5 23.0 Meta-only w/o tier60.855.730.2 23.7 Text+meta61.241.618.2 23.1 Text+meta w/o tier61.243.921.5 21.2 Text-only63.680.145.8 23.5 Text+meta bal.66.4 141.490.1 30.1 Table S14: Difficulty-metadata ablation on the held- out test split. Rows marked “w/o tier” remove both difficultyanddifficulty_tier. Learned routers use the same split and dev-selected hyperparameter rule as the main learned-router baselines. discrimination with better calibration than Llama, but it still does not match the same-model signal. We therefore interpret the self-confidence gate as evidence for a same-model self-assessment signal, not as proof that arbitrary LLMs can estimate an- other solver’s capability. B.4 Reduced Gemma-3 Actor-Stack Check To scope the single-solver concern without adding a second full benchmark, we recover a prior tier- sampled Gemma-3 actor sweep from the trace bundle. This reduced subset was sampled for more balanced coverage across difficulty tiers, rather than to match the full benchmark distribu- tion; it therefore stresses the hard-tail problems where escalation is most relevant. The sweep usesgoogle/gemma-3-27b-itas the actor under 14 0.00.20.40.60.81.0 Self-reported confidence 0.0 0.2 0.4 0.6 0.8 1.0 Empirical Baseline success (a) Reliability of self-confidence perfect calibration gpt-oss-120b 0.07–0.300.30–0.380.38–0.580.58–0.850.85–0.98 Confidence bin (gpt-oss-120b) 0.0 0.2 0.4 0.6 0.8 1.0 Fraction (b) Oracle protocol by confidence bin BaselineSinglePERBroadcastNone Figure S2: Direct self-assessment diagnostics for the cleaned pre-answer confidence probe. The left panel compares self-reported single-pass confidence with empirical Baseline success; the right panel shows oracle routing labels within confidence bins. RouterSolve Tok. Excess Under Over (%)(K)(K)(%)(%) Tier-maj. cross-src68.7 21.72.925.39.1 Source-maj. fallback65.7 17.11.228.45.9 Meta-only w/o tier78.0 42.517.715.446.7 Text+meta w/o tier74.9 58.230.818.439.0 Text-only73.0 61.536.220.426.5 Table S15: Source-holdout stress test over the six largest sources (n = 2994). Rows are aggregated across sep- arate held-out-source runs. The source-majority row falls back to the global train-set majority because the held-out source is unseen during training. the same four protocol templates and keeps the gpt-oss-120bequivalence judge fixed. It contains 833 unique problems after joining the four protocol outcomes. Table S17 shows that collaboration still changes outcomes under this actor family: SINGLE and PER improve substantially over one-shot solv- ing, and the fixed-order-oracle labels remain spread across protocol levels. On the same subset, Gemma-3-as-router pro- duces parseable labels for all examples but se- lects PER on 56.8% of problems and never pre- dicts BROADCAST ornone. Evaluated against the Gemma actor oracle, this router solves 63.3% of examples, under-escalates on 6.4%, and over- escalates on 67.0%. Gemma-3 same-model self- confidence is also predictive of Gemma’s one-shot success (AUROC 0.787 versus 0.765 for tier alone), but its calibration is weaker than the maingpt-oss ProbeCoverage AUROC Tier AUROC ECE Brier (%)(p) gpt-oss self77.80.8590.7737.5 0.152 Llama cross-model100.00.7430.756 28.7 0.288 Gemma-3 cross-model100.00.7680.756 15.8 0.217 Table S16: Predictive and calibration summary for the pre-answer confidence probe. The same-model probe is the cleanedgpt-oss-120bself-assessment used by the self-confidence gate. The cross-model checks ask Llama-3.1-70B and Gemma-3-27B to estimate whether gpt-oss-120bwill solve the same problems in one di- rect attempt. Expected calibration error (ECE) uses ten equal-width confidence bins; lower Brier score indicates better probabilistic accuracy. self-assessment (14.7 p ECE). We treat this as a scope check: the qualitative escalation pattern is not unique to the primary solver stack, but the sub- set is smaller and deliberately difficulty-balanced; it should not be read as a full multi-solver replica- tion or as an estimate of the full-benchmark distri- bution. Self-confidence gate and two-threshold cascade. We turn the cleaned pre-answer confidence signal into one main gate and one supporting ablation. The self-confidence gate keeps Baseline when con- fidence is at or above a threshold and otherwise es- calates to Single. The supporting two-threshold cas- cade keeps Baseline above a high threshold, uses Single at intermediate confidence, and falls back to 15 Gemma-3 check n Base Single PER Broad. None Protocol solve833 42.560.4 65.558.6– Oracle share833 42.518.4 10.22.826.2 Gemma router pred. 833 17.625.6 56.80.00.0 Table S17: Reduced Gemma-3 actor-stack check on a tier-sampled subset with more balanced difficulty cov- erage than the full benchmark. Values are percentages. The first row reports realized solve rates for each proto- col with Gemma-3 actors. The second row reports the resulting fixed-order-oracle label share. The third row reports a Gemma-3 frozen-router prompt on the same subset. the tier-majority heuristic at very low confidence. Thresholds are tuned only on the dev split. For the gate, we use the Kneedle rule for increasing con- cave tradeoff curves (Satopä et al., 2011). For the two-threshold policy, we first compute the dev-set Pareto frontier and then apply the same knee-point rule, consistent with the standard knee-solution view of bicriteria tradeoffs (Deb and Gupta, 2011). Figure S3 shows the selection curves. Utility-based dev selection with moderate cost weights reproduces the same thresholds selected by the Kneedle rule: for both confidence policies, utility weightsλ ∈ 0.4, 0.5recover the same held-out operating points. Budget-constrained dev sweeps trace the expected solve-cost frontier and lead to the same qualitative operating-point com- parison. Protocol-value probe.As an exploratory check, we also askgpt-oss-120bto estimate success probability under each protocol and then choose with simple dev-tuned rules. The protocol-value probe tests a stronger version of the same question: whether the model can estimate the marginal value of each escalation level, not only one-shot confi- dence. Table S20 shows that the probe is useful but not a clean replacement for the simpler self- confidence gate. A utility-selected policy matches the self-confidence gate’s solve rate but spends more tokens and over-escalates far more often. We therefore treat the probe as mechanism evidence rather than a new headline router. The probe is run on the dev and test splits with openai/gpt-oss-120bat temperature 0.0. The analysis evaluates two simple policies. The thresh- old policy chooses the first protocol in cost order whose estimated success probability is at leastτ, sweepingτ ∈20, 30,..., 90on dev. The utility policy choosesarg max p ˆs p − λc p , whereˆs p is the model’s estimated success probability andc p is the prompt-visible cost in thousands of tokens. We sweep 13λvalues from 0 to 2 on dev and select un- der a 75K-token dev budget, breaking ties by lower token cost. If the probe is missing or unparsable, the policy falls back to the tier-majority router. The prompt is listed in Appendix D. 16 020406080100 Confidence threshold 62.5 65.0 67.5 70.0 72.5 75.0 77.5 80.0 Solve rate (%) = 202530354045 Avg tokens (K) 60.0 62.5 65.0 67.5 70.0 72.5 75.0 77.5 Solve rate (%)  =  = 25 30 35 40 45 50 Avg tokens (K) (a) Binary gate threshold(b) Two-threshold cascade Figure S3: Dev-set knee-point selection for the confidence policies. Left: for the self-confidence gate, the selected τ = 70lies at the knee where solve gains flatten relative to additional cost. Right: for the two-threshold cascade, the selected (τ low ,τ high ) = (10, 70) lies near the knee of the dev Pareto frontier in solve-cost space. PolicySelected threshold(s)Solve (%)Avg tok. (K)Under (%)Over (%) Self-confidence gate τ = 7078.0 [74.0, 81.8]45.0 [41.5, 48.6]13.7 [10.6, 17.0]42.1 [37.1, 46.6] Two-threshold cascade τ low = 10, τ high = 7075.2 [70.9, 79.2]41.7 [38.1, 45.3]16.5 [13.0, 20.3]28.4 [23.6, 32.6] Table S18: Self-confidence gate and two-threshold cascade on the held-out test split. Thresholds are tuned on the dev split using confidence on a 0–100 scale. Costs include the pre-answer confidence-query tokens. Bracketed ranges are 95% bootstrap confidence intervals. PolicySubsetn Solve (%) Avg tok. (K) Self-conf. gate Full test42378.045.0 Self-conf. gate Confidence available 32975.446.4 Two-thresholdFull test42375.241.7 Two-thresholdConfidence available 32975.446.7 Table S19: Missing-confidence ablation for the confi- dence policies. The full-test rows keep each policy’s built-in behavior on problems without a usable cleaned confidence estimate; the confidence-available rows re- strict to examples with cleaned confidence values. PolicyDev value Solve Tok. Excess Under Over (%)(K)(K)(%)(%) Value threshold3067.4 37.912.325.114.9 Value utility0.178.0 52.922.013.761.5 Table S20: Exploratory protocol-value probe. Threshold selects the first protocol in cost order whose estimated success probability exceeds 30%. Utility selects the protocol maximizing estimated solve probability minus 0.1times prompt-visible token cost in thousands. Probe coverage means that all four requested probabilities were parsed; missing rows fall back to tier-majority. 17 TiernMean conf.Actual pass 11595.0100.0 23885.986.8 3259.0100.0 48761.465.5 58248.351.2 63343.730.3 72336.713.0 83026.816.7 91718.623.5 10210.050.0 Table S21: Cleaned pre-answer self-assessment by dif- ficulty tier. Confidence and pass rates are percentages. Tiers with very smallnshould be interpreted cautiously. Confidence bin n Baseline Single PER Broadcast None 0–202913.827.6 24.110.324.1 20–4010422.140.4 12.511.513.5 40–607048.630.0 17.12.91.4 60–805275.015.45.80.03.8 80–1007497.32.70.00.00.0 Table S22: Oracle label composition within cleaned pre-answer confidence bins. Low confidence does not identify a single next-best protocol; bins withn < 30 should be interpreted cautiously. QuantityValue Total test problems423 Usable cleaned estimates329 Coverage77.8% Original JSON parses310 Recovered direct estimates11 Recovered ranges8 No estimate before truncation72 HTTP 429 failures22 Table S23: Coverage decomposition for the cleaned pre- answer self-assessment data. C Benchmark and Protocol Details C.1 Benchmark Provenance and Metadata The routing benchmark contains 4,181 matched competition-level mathematics problems from the filtered exact-answer Omni-MATH 2 slice (Ballon et al., 2026), which is derived from Omni-MATH (Gao et al., 2024). The slice keeps the upstream problem source, domain path, difficulty score, and ten difficulty-tier fields. The tier variable is ordinal metadata inherited from the benchmark rather than a post-hoc label from our protocol outcomes. The field is useful for diagnosis and for simple base- lines, but the field should not be read as a perfectly calibrated latent difficulty axis. The benchmark contains 64 source labels; the largest sources in- cludeHMMT_2,HMMT_11,fermat,pascal,cayley, imo_shortlist,usamo, andputnam. Tables S14 and S15 isolate how much the routing story de- pends on this curated difficulty metadata. The matched-outcome robustness study addition- ally uses JEEBench (Arora et al., 2023), SciBench (Wang et al., 2024), and LAB-Bench (Laurent et al., 2024). These benchmarks extend the evalua- tion to engineering entrance-exam STEM, college- level science, and biology tasks. We treat the two LAB-Bench prompt conditions as separate evalua- tion conditions, not as separate benchmarks. Both solver families cover all five evaluation conditions. C.2 Splits and Oracle-Label Scope All learned-router, confidence-policy, and held-out- test comparisons use the same deterministic split. We stratify by the oracle routing label using an 80/10/10 train/dev/test split with seed 42, yielding 3,342 train, 416 dev, and 423 test examples. The dev split is used for hyperparameter selection and confidence-policy threshold selection; all main-text values are reported on the held-out test split. Each problem is evaluated under four protocols, ordered by average cost from BASELINE to SIN- GLE, PER, and BROADCAST. The oracle routing label is the first protocol in this fixed order whose final answer is correct. If no protocol solves the problem, the oracle label isnone. The oracle tar- get is therefore an outcome-grounded control label, not a manually assigned difficulty label. The oracle target is also a single-realization target: the oracle is defined from the observed matched runs rather than from repeated-sampling estimates of expected protocol success. The single-realization scope is important. Replaying saved submissions from the same trace family with Llama and Gemma evalu- ators gives low full-benchmark pairwise disagree- ment (3.38–5.76%), rising to 5.25–9.68% on hard PER/Broadcast tiers. The evaluator audit checks sensitivity to the equivalence judge; it does not estimate repeated-run protocol variance. The re- duced Gemma-3 actor sweep in Appendix B.4 is also a scope check, not headline evidence: it uses a smaller tier-sampled subset and a different actor stack, and it shows that structured protocol behav- ior can shift across model families. We therefore use these prior trace audits to scope the benchmark and motivate release, not to claim full cross-family generality. Tables S24, S25, and S26 give the split and held-out test quantities behind the main results. 18 Split n Base Single PER Broad. None Train 334256.823.08.74.27.3 Dev41657.023.18.74.17.2 Test42356.322.99.04.37.6 Table S24: Split sizes and oracle-label percentages. Splits are stratified by the fixed-order-oracle label. ProtocolAvg. tokensSolve rate BASELINE18.2K56.3 SINGLE47.6K78.5 PER401.9K84.9 BROADCAST622.1K88.9 Table S25: Protocol-level cost and solve rate on the held-out test split. C.3 Protocol Execution and Compute Accounting All four protocol traces use the same un- derlying acting and evaluator model family, openai/gpt-oss-120b, through an OpenAI- compatible inference endpoint (OpenAI, 2025). The paper configs use deterministic decoding with temperature 0.0 and a 4096-token per-call gener- ation cap for solver/evaluator calls. BASELINE uses one direct solver attempt; SINGLE adds iter- ative self-correction with question-scoped repair memory; PER uses planner, executor, and reviewer roles; and BROADCAST uses multi-agent delibera- tion with shared candidate state and peer approval. All question-scoped memories and chat histories are reset between problems. Token metrics in the routing tables are read from the matched trace logs. On the full 4,181- problem benchmark, the four protocol runs account for about 4.53B logged tokens in total: 77M for BASELINE, 201M for SINGLE, 1.67B for PER, and 2.58B for BROADCAST. The corresponding logged model-call totals are 40,501, 75,370, 416,037, and 562,939. These quantities are reported to make the compute footprint of constructing the matched benchmark explicit; router-side costs are included separately where applicable, such as in the confi- dence cascade analysis. Logged average wall times per problem are about 44s, 74s, 4.7 minutes, and 4.9 minutes for the four protocols in the same order. C.4 Learned Router Implementation The lightweight learned routers are five-class lo- gistic regressions over the same protocol labels used throughout the paper: Baseline, Single, PER, Broadcast, and None. Text features are TF-IDF Oracle labelCountFraction BASELINE23856.3 SINGLE9722.9 PER389.0 BROADCAST184.3 none327.6 Table S26: Fixed-order-oracle protocol distribution on the test split. word unigrams and bigrams fit on the training problems only (min_df=2,max_features=20,000, lowercasing and Unicode accent stripping). Meta- data features include a one-hot source encoder, a multi-label domain encoder, and standardized difficultyanddifficulty_tierfields. The no-tier ablations remove both numeric difficulty fields; the text-only router removes all metadata features. Forthemaintext+metadataand metadata-onlyrouters,wetrainscikit- learnLogisticRegressionmodelswith solver=saga,random_state=42, and a grid overC ∈ 0.25, 1, 4andclass_weightin None,balanced.Validation training uses warm-start one-epoch increments for at most 30 epochs and selects the epoch with lowest dev log loss. The final configuration is selected on the dev split by macro-F1, breaking ties by lower excess cost. The selected main text+metadata model uses C = 0.25, no class weighting, and 15,124 total features; the selected metadata-only model uses C = 0.25, no class weighting, and 220 features. Sentence-embedding router details. The em- bedding ablation uses only the problem statement text for the embedding-only rows and concate- nates those embeddings with the same source, do- main, difficulty, and difficulty-tier metadata fea- tures for the embedding+metadata rows.We use a small frozen encoder rather than a fine- tuned transformer so that the ablation remains a low-cost semantic-router check rather than a new modeling contribution. Logistic variants sweep C ∈ 0.1, 0.25, 1, 4, 16with and without bal- anced class weights. kNN variants sweep the neigh- bor count and distance weighting on the dev split. All reported embedding-router results use the same train/dev/test split and the same cost-aware evalua- tion code as the other router baselines. 19 D Prompt and Execution Details D.1 Frozen Router Prompt All frozen LLM routers receive the same meta- data fields: problem source, difficulty score, dif- ficulty tier, domain summary, and full problem text. The prompt gives the protocol order ordi- nally, not as exact token costs, so the frozen router must infer the value of escalation from the prob- lem and metadata rather than from numeric token budgets. All frozen router runs use temperature 0.0. If a response cannot be parsed into one of the five allowed labels, the offline evaluator as- signs the fallback predictionbaseline_llm. To reproduce the model call exactly, the listings retain the machine-readable labels used during inference: baseline_llm,single_agent,per,broadcast, andnonecorrespond to Baseline, Single, PER, Broadcast, and None. The exact prompt is repro- duced below. You are a routing classifier, not a solver. Do NOT solve the math problem. Do NOT compute intermediate steps. Your task is only to estimate the cheapest protocol likely to be sufficient. Cost order from cheapest to most expensive: baseline_llm < single_agent < per < broadcast. You may also output none if all protocols are unlikely to succeed. Guidelines: - baseline_llm: short direct calculations or standard manipulations. - single_agent: moderate multi-step problems where one model with self-correction is likely enough. - per: explicit planning and review are likely helpful. - broadcast: especially hard problems that may benefit from multiple independent attempts. Output exactly one label from: baseline_llm, single_agent, per, broadcast, none. Start your reply with the label on the first line. Stop after the label if possible. Problem source: source Difficulty score: difficulty Difficulty tier: difficulty_tier Domain: domain_summary Problem: problem Classify the likely cheapest sufficient protocol for this problem. Do not solve the problem. Reply with exactly one label on the first line: baseline_llm, single_agent, per, broadcast, or none. D.2 Cost-Aware Frozen Router Prompt Ablation The numeric-cost/few-shot ablation in Table S11 usesopenai/gpt-oss-120b, temperature 0.0, and the same test split as main-paper Table 1. The parser accepts one of the five allowed labels after an explicitLABEL:prefix; unparsable responses fall back to Baseline. The final cleaned run has one fallback. The system prompt below differs from the main frozen-router prompt only by adding numeric average costs and short routing examples. You are a routing classifier, not a solver. Do NOT solve the math problem. Do NOT compute intermediate steps. Your task is only to estimate the cheapest protocol likely to be sufficient. Cost order from cheapest to most expensive: baseline_llm < single_agent < per < broadcast. You may also output none if all protocols are unlikely to succeed. Guidelines: - baseline_llm: short direct calculations or standard manipulations. - single_agent: moderate multi-step problems where one model with self-correction is likely enough. - per: explicit planning and review are likely helpful. - broadcast: especially hard problems that may benefit from multiple independent attempts. Output exactly one label from: baseline_llm, single_agent, per, broadcast, none. Start your reply with the label on the first line. Do not include reasoning unless the label already appears on the first line. Approximate average token costs from the benchmark: - baseline_llm: 18K tokens - single_agent: 48K tokens - per: 402K tokens - broadcast: 622K tokens Escalate only when the expected gain in solve probability justifies the extra token cost. Routing examples: - If a problem is routine arithmetic or a direct substitution, output baseline_llm. - If a problem needs several algebraic steps but one solver is likely enough, output single_agent. - If a problem likely needs explicit planning and checking, output per. - If a problem is very hard and independent attempts may add value, output broadcast. - If none of the four protocols is likely to solve it, output none. Problem source: source Difficulty score: difficulty Difficulty tier: difficulty_tier Domain: domain_summary Problem: problem Classify the cheapest protocol whose expected solve gain is worth its token cost. Do not solve the problem. Reply with exactly one line: LABEL: baseline_llm, LABEL: single_agent, LABEL: per, LABEL: broadcast, or LABEL: none. D.3 Direct Self-Assessment Prompt Theheadlineself-confidencegateuses gpt-oss-120bbecause that model produced the benchmark’s Baseline solutions. The probe asks for an explicit verbalized probability on a 0–100 scale in JSON; it does not use token log-probabilities, self-consistency, or repeated 20 sampling. The main run uses the test split, tem- perature 0.0, a 256-token completion limit, and at most 20 concurrent requests. Cross-model checks use the same JSON format but ask Llama-3.1-70B or Gemma-3-27B to estimategpt-oss-120b’s one-shot success; the reduced Gemma actor check asks Gemma-3-27B about its own one-shot success on that subset. Non-gpt-ossprobes use at most seven concurrent requests. You are evaluating mathematical problem difficulty. Do NOT solve the problem. Do NOT compute intermediate steps. If you start solving the problem, you are failing the task. Your task is to estimate, without working through the solution, how likely you are to answer this problem correctly in a single direct attempt -- working alone, with no tools, no self-correction, and no collaboration . Return only valid JSON with exactly this key: "SINGLE_PASS_PROB": <integer 0-100> Do not add any explanation, reasoning, or extra keys. Your first character must be ’ and your last character must be ’. Problem source: source Difficulty score: difficulty Difficulty tier: difficulty_tier Domain: domain_summary Problem: problem Estimate your single-pass solve probability (0-100) without solving the problem. Do not solve the problem. Forcross-modelprobes,thesystem anduserpromptsreplace“you”with “openai/gpt-oss-120b” as the target solver. These runs are diagnostic checks; they are not used to tune the main self-confidence gate. D.4 Protocol-Value Probe Prompt The exploratory protocol-value probe in Table S20 uses the following JSON prompt. You are estimating protocol success probabilities for math problem routing. Do NOT solve the problem and do NOT compute intermediate steps. Estimate how likely each protocol is to solve the problem correctly. Protocol costs are approximately: baseline_llm 18K tokens, single_agent 48K, per 402K, broadcast 622K. Return only valid JSON with exactly these integer keys, each from 0 to 100: "BASELINE_LLM_PROB": <integer>, "SINGLE_AGENT_PROB": <integer>, "PER_PROB": <integer>, "BROADCAST_PROB": <integer> Do not add explanation, reasoning, or extra keys. Problem source: source Difficulty score: difficulty Difficulty tier: difficulty_tier Domain: domain_summary Problem: problem Estimate the probability that each protocol solves this problem correctly. Do not solve the problem. D.5 Pre-Answer Confidence Parsing and Coverage The initial parser recovered usable pre-answer con- fidence estimates for 310 of 423 test problems (73.3%). A manual re-parse of failed rows recov- ered 19 additional estimates, yielding 329 usable rows (77.8%). The remaining missing rows are not treated as parser misses: 72 were truncated before an estimate was stated and 22 were HTTP 429 fail- ures. All pre-answer confidence figures and tables use the cleaned analysis file and filter to rows with a recovered probability. The same parse issue also appears on the dev split used for threshold tuning. Most dev fallbacks were not network failures, but parse failures where the model drifted into solving mode instead of re- turning the requested JSON probability. We there- fore do not treat missing confidence as missing at random. We also tested a completion-cue variant that asked for a single integer. The completion-cue vari- ant was not used for the main analysis: a naive first- integer parser can silently extract numbers copied from the problem statement rather than confidence estimates. We therefore use the cleaned JSON- prompt run throughout the paper. The coverage caveat matters for the interpretation of the mecha- nism results. The cleaning step raises usable cover- age but does not turn the probe into a routing ora- cle: even after cleaning, low-confidence examples remain spread across multiple fixed-order-oracle protocols and true failures. E Additional Diagnostic Figures The final diagnostics support two boundary con- ditions for the main interpretation.The loss- history plot checks that the learned-router result is not just a single unlucky evaluation point: the text+metadata model continues improving on train- ing loss while its development loss worsens, con- sistent with overfitting in this lightweight setup. The source-level heatmap checks that the router 21 051015202530 Epoch 0.95 1.00 1.05 1.10 1.15 1.20 Log-loss Training and dev log-loss text+metadata (train) text+metadata (dev) metadata-only (train) metadata-only (dev) (a) Training and dev log-loss. Text+metadata dev loss diverges from train loss after early epochs, motivating early stopping. HMMT_11 HMMT_2 ToT cayley china_team_selection_test fermat imc imo imo_shortlist pascal putnam usamo Tier-majority Metadata-only Text+meta early-stop gpt-oss-20b gpt-oss-120b Llama-3.1-70B Gemma-4-E4B Gemma-3-27B 0.720.600.290.930.671.000.670.670.380.800.890.71 0.720.600.430.930.441.000.170.420.500.800.670.43 0.720.600.430.930.441.000.170.420.500.800.670.43 0.740.640.431.000.221.000.500.170.120.840.670.43 0.790.730.710.930.561.000.830.750.560.880.890.71 0.830.810.711.000.781.000.830.670.690.880.890.71 0.840.810.711.000.671.000.830.830.690.841.000.71 0.820.830.711.000.671.001.000.830.810.840.890.57 Per-source solve rate by router 0.0 0.2 0.4 0.6 0.8 1.0 Solve rate (b) Per-source solve rates. The aggregate router ranking broadly persists across sources. Figure S4: Additional learned-router diagnostics. (a) The lightweight text+metadata router begins to overfit after early epochs. (b) The aggregate comparison is not explained by a single benchmark source. comparison is not driven by one benchmark source alone. These diagnostics support the narrower inter- pretation that the current routers are miscalibrated about the marginal value of escalation. F Data and Reproducibility This appendix records the exact prompts, feature schemas, split rules, execution set- tings,anduncertaintyproceduresneeded to interpret the results.The compan- ion dataset archive is hosted athttps:// huggingface.co/datasets/ChihHsuan-Yang/ 22 scientific-agent-protocol-traces. For the settings studied here, it contains anonymized four-protocol traces, per-protocol outcome labels, matched outcome tables, and registries that map machine-readable identifiers to the Baseline, Single, PER, and Broadcast names used in the paper. The arXiv source package also includes the aggregate result tables underlying the reported comparisons as ancillary CSV files. The companion archive covers additional ex- periments beyond this paper. The claims in this paper use only the four protocols and the bench- mark/model settings defined in the main text; those boundaries should be used when selecting records from the broader archive. Raw problem redistribution follows each up- stream benchmark’s license; where redistribution is restricted, the release provides identifiers and re- construction instructions rather than raw prompts. 23

Instant research discovery