Paper deep dive

Oracle Gap and Signal Fidelity: A Fixed-Pool Diagnostic for Test-Time Collaboration

Jie Hu

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 48

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%

Last extracted: 7/21/2026, 5:10:57 AM

Summary

The paper introduces 'OracleGap', a fixed-pool diagnostic framework for evaluating test-time collaboration in LLMs. It decomposes selection gains into recoverable mass, signal coverage, conditional quality, and harm. The study finds that gains are bounded by the oracle gap (difference between best candidate and first sample) and signal fidelity (agreement between verifier verdicts and official labels). High-fidelity signals like public-test verifiers yield significant gains, while low-fidelity or high-harm selectors (like some LLM selectors) may fail or damage correct outputs.

Entities (10)

GPQA-Diamond · benchmark · 99%LiveCodeBench · benchmark · 99%MATH · benchmark · 99%OracleGap · framework · 98%Oracle Gap · metric · 97%signal fidelity · metric · 97%Qwen3.6-35B-A3B-BF16 · model · 95%Recoverable Mass · metric · 95%L4-gen · method · 92%L4-public · method · 92%

Relation Signals (8)

L4-public → achievesgainover → sample0

confidence 98% · On LiveCodeBench, a public-test verifier (MCC 0.825) gains +8.14 percentage points (pp) over a first-sample baseline

L4-gen → achievesgainover → sample0

confidence 98% · a generated-test verifier (MCC 0.248) improves by +2.70pp

L4-public → hashigherfidelitythan → L4-gen

confidence 96% · The public-test verifier reaches 92.98% accuracy and MCC 0.825; the generated-test verifier reaches 53.05% and MCC 0.248

GPQA Diamond → haslowrecoverablemass → 3.03%

confidence 95% · On GPQA-Diamond, recoverable mass is only 3.03%

Qwen3.6-35B-A3B-BF16 → usedon → LiveCodeBench

confidence 95% · The main model source is Qwen3.6-35B-A3B-BF16 for LiveCodeBench...

OracleGap → uses → signal fidelity

confidence 95% · ...measure coverage, signal fidelity, and harm before investing in collaboration.

OracleGap → uses → Oracle Gap

confidence 95% · Our framework yields a practical pre-deployment diagnostic: estimate the oracle gap...

L4-gen → haslowerharmthan → L1

confidence 94% · L4-gen... operates at near-zero harm versus the selector's 4.69% harm rate.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Test-time collaboration, including self-consistency, best-of-N selection, critic models, and verifier pipelines, is often credited with broadly improving LLM reasoning, yet its gains are uneven and sometimes negative. We ask when training-free collaboration should be expected to help. For a fixed candidate pool, we decompose a selector or verifier's net gain into measurable factors: recoverable mass, verification-signal coverage, conditional selection quality, and harm to already-correct outputs. This reframes collaboration as a candidate-selection problem rather than as an intrinsic property of a multi-agent topology. Across LiveCodeBench, MATH Level-5 hard subjects, and GPQA-Diamond, gains are bounded first by the oracle gap and then by signal fidelity, which we measure directly as candidate-level agreement between verifier verdicts and official labels. On LiveCodeBench, a public-test verifier (MCC 0.825) gains +8.14 percentage points (pp) over a first-sample baseline; a generated-test verifier (MCC 0.248) improves by +2.70pp and is not statistically distinguishable from an LLM selector, but operates at near-zero harm versus the selector's 4.69% harm rate. On MATH, a symbolic answer-equivalence selector beats self-consistency by +4.67pp, while LLM selectors are negative. On GPQA-Diamond, recoverable mass is only 3.03% and 87.54% of candidate pools are answer-identical; a weaker model's pools shrink both further, suggesting that oracle gap is a joint property of task, model, and sampling configuration. Our framework yields a practical pre-deployment diagnostic: estimate the oracle gap, then measure coverage, signal fidelity, and harm before investing in collaboration.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

46,514 characters extracted from source content.

Expand or collapse full text

Oracle Gap and Signal Fidelity: A Fixed-Pool Diagnostic for Test-Time Collaboration Jie Hu Research Institute of China Telecom hujiesse@gmail.com Abstract Test-time collaboration, including self-consistency, best-of-N selection, critic models, and verifier pipelines, is often credited with broadly improving LLM reasoning, yet its gains are uneven and sometimes negative. We ask when training-free collaboration should be expected to help. For a fixed candidate pool, we decompose a selector or verifier’s net gain into measurable factors: recoverable mass, verification-signal coverage, conditional selection quality, and harm to already-correct outputs. This reframes collaboration as a candidate-selection problem rather than as an intrinsic property of a multi-agent topology. Across LiveCodeBench, MATH Level-5 hard subjects, and GPQA-Diamond, gains are bounded first by the oracle gap and then by signal fidelity, which we measure directly as candidate-level agreement between verifier verdicts and official labels. On LiveCodeBench, a public-test verifier (MCC 0.825) gains +8.14+8.14 percentage points (p) over a first-sample baseline; a generated-test verifier (MCC 0.248) improves by +2.70+2.70p and is not statistically distinguishable from an LLM selector, but operates at near-zero harm versus the selector’s 4.69% harm rate. On MATH, a symbolic answer-equivalence selector beats self-consistency by +4.67+4.67p, while LLM selectors are negative. On GPQA-Diamond, recoverable mass is only 3.03% and 87.54% of candidate pools are answer-identical; a weaker model’s pools shrink both further, suggesting that oracle gap is a joint property of task, model, and sampling configuration. Our framework yields a practical pre-deployment diagnostic: estimate the oracle gap, then measure coverage, signal fidelity, and harm before investing in collaboration. 1 Introduction Test-time collaboration is now a common recipe for improving large language model reasoning: sampling multiple candidates and voting by self-consistency, asking a critic to inspect an answer, filtering code with tests, or routing an input to a stronger worker. These mechanisms are often grouped under one intuition: more computation, more agents, or more judging should improve reliability. Empirically, this intuition is incomplete. A critic may fix some failures while damaging answers that were already correct. Self-consistency is meaningful for math problems with normalized final answers, but exact majority voting over code strings is mostly a fallback rule rather than a real selector. Public tests can be an extremely strong signal for code, while self-generated tests can inherit the model’s own blind spots. A second model may add diversity, or it may simply choose among nearly identical candidates, right or wrong. In short, collaboration is not a property of the topology alone: its value depends on the candidate pool, the task’s evaluation structure, and the fidelity of the signal used to choose among candidates. This paper asks: when should training-free collaboration be expected to help? We refer to the resulting fixed-pool diagnostic framework as OracleGap. We study the selector/verifier setting: for each problem, a fixed candidate pool is generated, and a mechanism chooses one candidate without further training. This covers best-of-N selection, LLM selectors, generated-test filtering, public-test filtering, and symbolic answer-equivalence selection, and deliberately separates selection from repair: a repair mechanism can create new candidates, whereas a selector can only capture the improvement already present in the pool. (a) Fixed-pool diagnostic and gain decompositionInput xxy1y_1y2y_2y3y_3y4y_4y5y_5 Fixed pool C(x)C(x), k=5k=5referenceVerification signalcoverage: signal defined?fidelity: verdict vs. official labelSelect y yfallback →y1→ y_1Outcome accountingfixed (+)(+) / harmed (−)(-) / unchanged (0)(0) ref. wrong ++ correct alternative ⇒ recoverable; oracle gap official-label audit only gain=P(R∧D)q−P(C∧D)hgain=P(R D)\,q-P(C D)\,hRR: recoverable, C: reference correct, D: signal defined(b) Pre-deployment workflow 1. Estimate oracle gap 2. Audit coverage and fidelity 3. Compare capture against harm 4. Deploy only if net-positive small gap ⇒ stop GPQA: 3.03 p low coverage/fidelity limits capture L4-gen: 35.8% active MCC 0.248 harm can erase recovered gains GPQA L1: net −1.68-1.68 p high-fidelity signals pay L4-public: +8.14+8.14 p zero harm Figure 1: OracleGap overview. (a) Official labels expose the audit-only oracle gap and measure the coverage and fidelity of a deployable signal; selection then produces fixes, harms, or no changes. (b) The pre-deployment workflow stops when recoverable mass is small and deploys only signals whose captured gains exceed harm. Our central claim is that selection gains are bounded first by the oracle gap and then by the coverage, fidelity, quality, and harm of the verification signal. We make this concrete with a fixed-pool decomposition: gain=P(recoverable∧defined)q−P(referencecorrect∧defined)h, splitgain=&P(recoverable )\,q\\ &-P(reference\ correct )\,h, split (1) where recoverable means the reference output is wrong but some candidate in the pool is correct; q is conditional selection success on recoverable examples with a defined signal; and h is how often the mechanism changes a reference-correct example into an incorrect one among examples with a defined signal. We also measure verifier fidelity directly as candidate-level agreement between verifier verdicts and official labels, linking the decomposition to imperfect-verifier theory. We evaluate on LiveCodeBench (execution signals available, generated tests imperfect), MATH Level-5 hard subjects (answers normalizable or symbolically checkable), GPQA-Diamond (a low-recoverable, low-diversity boundary), and HumanEval+ (a saturated code anchor) (Jain et al., 2024; Hendrycks et al., 2021; Rein et al., 2024; Liu et al., 2023). Figure 1 summarizes the fixed-pool accounting and turns it into a pre-deployment decision workflow. Our contributions are: 1. A training-free decomposition of selector/verifier gain into recoverable mass, signal coverage, conditional quality, and harm, making collaboration gains measurable at the task level rather than attributed to agent count or topology. 2. A three-seed LiveCodeBench selector ladder comparing a first-sample baseline, LLM selectors, a generated-test verifier, a public-test verifier, and an any-of-k oracle upper bound on a unified task set with hierarchical confidence intervals. 3. Evidence that verifier fidelity is an empirical bottleneck (Figure 2): a public-test verifier with high candidate-level fidelity captures much more of the oracle gap than a generated-test verifier with low fidelity, while the latter has near-zero harm. 4. Cross-task boundary evidence: on MATH, a symbolic answer-equivalence selector improves over self-consistency while natural-language LLM selectors are negative; on GPQA-Diamond, the candidate pool has only a 3.03p oracle gap with high answer homogeneity, and LLM selectors are net-negative on that pool. 5. A routing-η extension (supplementary) showing that worker routing and candidate selection obey different constraints and should not be conflated under a single notion of collaboration. Together, these results yield a practical diagnostic: before deploying a critic, verifier, or multi-agent selector, first estimate the oracle gap of the candidate pool, then measure whether the available signal has enough coverage, fidelity, and low harm to justify the extra computation. 2 Related Work 2.1 Test-Time Selection and Execution-Based Filtering Self-consistency and best-of-N sampling improve reasoning when aggregation or selection is reliable (Wang et al., 2023; Brown et al., 2024; Snell et al., 2024). For code, CodeT filters candidates with generated tests (Chen et al., 2022), AlphaCode combines sampling with test filtering and clustering (Li et al., 2022), and MBR-Exec selects by behavioral agreement (Shi et al., 2022). These are direct predecessors of our generated- and public-test mechanisms. Rather than introduce another selector, we fix candidate pools and decompose realized gain into recoverable mass, signal coverage, conditional quality, and harm. This diagnostic layer directly compares natural-language, generated-test, public-test, and symbolic-equivalence mechanisms that are usually evaluated separately. 2.2 Trained and Imperfect Verifiers Verifier-based selection includes trained outcome and process verifiers (Cobbe et al., 2021; Uesato et al., 2022; Lightman et al., 2024). Multi-Agent Verification introduces BoN-MAV (Lifshitz et al., 2025), while newer general-purpose and multi-sequence verifiers improve granularity, calibration, and budget-aware ranking (Kwok et al., 2026; Kim et al., 2026). These improve the verifier; our training-free setting asks when an available signal converts candidate diversity into net gain at all. A complementary theoretical line derives best-of-N and rejection-sampling behavior from verifier ROC geometry (Dorner et al., 2025), shows that false positives impose accuracy ceilings and can bend scaling curves downward (Stroebl et al., 2024), and establishes asymptotic advantages for verifier-based scaling (Setlur et al., 2025). Compute-matched verification is not uniformly optimal (Singhi et al., 2025); Venktesh et al. (2025) survey the broader design landscape. Most directly, Lu et al. (2025) systematically study when solution verification pays off across self-, within-family, and cross-family solver–verifier pairs. They find stronger cross-family verification as solver–verifier similarity decreases; our L3 instead uses a smaller cross-family selector, so the L1–L3 gap reflects capability rather than family similarity, which our capture/harm decomposition makes explicit. We make verifier quality observable through candidate-level agreement with official labels, including accuracy, error rates, and MCC. Separating fidelity from coverage and harm captures verifiers that are accurate but rarely active, or conservative and low-harm yet miss many correct candidates, complementing imperfect-verifier theory with cross-domain empirical accounting. 2.3 When Multi-Agent Collaboration Helps Recent work directly challenges the assumption that increasing agent count or interaction rounds is intrinsically beneficial. Diversity-based analysis shows that homogeneous agents saturate because their outputs are correlated, whereas heterogeneous agents provide complementary effective channels (Yang et al., 2026). The closest same-question work finds that entropy dynamics vary with task and coordination topology; reducing interaction-induced uncertainty therefore need not by itself establish genuine improvement (Zhao et al., 2026). Uncertainty decomposition in multi-agent debate similarly distinguishes epistemic gain from aleatoric cost (Qiao et al., 2026). System-level studies reach compatible conclusions from different angles. MAST catalogs system-design issues, inter-agent misalignment, and task-verification failures (Cemri et al., 2025); controlled scaling studies find capability saturation and topology-dependent error propagation (Kim et al., 2025); and matched-token experiments show that single-agent reasoning can outperform multi-agent systems when total thinking compute is held fixed (Tran and Kiela, 2026). Earlier work also finds that self-correction is unreliable without external feedback and that debate can improve some factual and reasoning tasks (Huang et al., 2024; Du et al., 2023). Our scope is narrower and more mechanistic than these topology-, entropy-, or system-level analyses. We ask whether a concrete selector can convert a fixed candidate pool into a better output. GPQA-Diamond provides a boundary case: most k=5k=5 pools are answer-identical and the oracle gap is only 3.03p, leaving little room for any selector. LiveCodeBench and MATH provide the complementary case: recoverable candidates exist, but gains depend on whether the selection signal is faithful and low-harm. Thus our oracle-gap and signal-fidelity framework is complementary to diversity and entropy: it quantifies the improvement space and the mechanism’s ability to capture it. 2.4 Generated Tests and Label-Free Fidelity Estimation Generated-test methods are especially close to our deployable code verifier. CoSPlay co-evolves code candidates and self-generated unit tests without ground-truth tests (Hu et al., 2026). UTGen learns error-revealing inputs and expected outputs, while UTDebug spends test-time compute to validate feedback and avoid overfitting (Prasad et al., 2025). UTRL trains test and code generators adversarially so that generated tests better discriminate faulty programs (Lee et al., 2026). These approaches address the limitation surfaced by our results: self-generated tests may be low-harm yet low-fidelity because they inherit the generator’s blind spots or assign incorrect expected outputs. Our contribution is complementary. We use official labels to audit generated-test verdicts directly, identifying fidelity and activation as separate bottlenecks. This creates a bridge to label-free reliability estimation: future generated-test systems should be evaluated not only by final pass@1, but also by whether their internal confidence or reliability scores predict official-label fidelity. 2.5 Positioning Summary Together, these literatures cover candidate generation, verifier training, imperfect-verifier theory, multi-agent scaling, and methods for improving generated tests. Our contribution is the missing fixed-pool measurement layer across benchmarks with different decidability profiles: before attributing gains to collaboration or topology, measure whether recoverable answers exist and whether the deployed signal has enough coverage, fidelity, quality, and low harm to recover them. 3 Method and Experimental Setup 3.1 Problem Setup For each input x, a model produces a fixed candidate pool C(x)=y1,y2,…,yk.C(x)=\y_1,y_2,…,y_k\. (2) The first candidate y1y_1 is the first-sample baseline. A training-free collaboration mechanism then selects one candidate or supplies a verification signal over the pool. This setup covers best-of-N selection, self-consistency, LLM-based selectors, generated-test filtering, public-test filtering, and symbolic answer-equivalence selection. We distinguish fixed-pool selection from repair. A repair mechanism may create a new answer outside the original pool, so its gain depends jointly on detection and generation. A selector can only capture improvement already present among the sampled candidates. Fixing the pool isolates the question studied here: how much available oracle space does a concrete selection mechanism convert into realized gain? 3.2 Oracle Gap and Recoverable Mass For each task, the any-of-k oracle succeeds if at least one candidate passes the official evaluator. Relative to a reference output, a task is recoverable when the reference is wrong but some candidate is correct: recoverable(x)=referencewrong(x)∧any@kcorrect(x).recoverable(x)=reference\ wrong(x) @k\ correct(x). (3) The recoverable mass is P(recoverable)P(recoverable). The oracle gap is the score difference between the any-of-k oracle and the chosen reference. It is an upper bound on every fixed-pool selector. If the gap is small, elaborate collaboration has little absolute room to improve, regardless of selector sophistication. 3.3 Coverage, Quality, Harm, and Fidelity We decompose gain using three conditional quantities: coverage =P(signaldefined), =P(signal\ defined), (4) quality =P(selectedpasses∣recoverable,defined), =P(selected\ passes ,defined), (5) harm =P(selectedfails∣referencepasses,defined). =P(selected\ fails \ passes,defined). (6) Signal defined means the mechanism produced usable evidence: for example, a parseable LLM selector output, executable public tests, generated tests that can be run, or a symbolic answer cluster. These quantities deliberately live on different slices. Quality is neither overall accuracy nor one minus harm; it asks whether the mechanism selects a correct candidate among recoverable rows where evidence is available. Harm asks whether the same mechanism turns a reference-correct row into an incorrect selected output, again conditional on evidence being defined. With these definitions, task-level gain follows Equation 1. For deployed mechanisms with a fallback policy, we additionally report effective capture and effective harm over all recoverable and reference-correct rows, respectively. These effective quantities include fallback and no-op behavior and therefore close the fixed-minus-harmed accounting identity. They must not be confused with the conditional quality and harm terms above. This distinction is important for conservative verifiers: a method may have low effective harm because it frequently falls back to y1y_1, even though its active signal is available on only a small slice. We separately measure verifier fidelity as a property of the evidence itself: candidate-level agreement between verifier verdicts and official candidate labels, summarized by accuracy, false-positive rate, false-negative rate, and MCC. Keeping fidelity separate from coverage and effective behavior prevents a high-fallback mechanism from appearing reliable merely because it seldom changes the reference. 3.4 Selector Ladder We compare the mechanisms in Table 1 on fixed k=5k=5 pools. Self-consistency and majority voting are aggregation baselines rather than ladder tiers, reported where answers are naturally normalizable (MATH and GPQA). Verifier implementation depends on the benchmark: executed tests for code, symbolic or normalized-answer equivalence for MATH, and no natural executable verifier for GPQA, which therefore serves primarily as a low-recoverable boundary case. Table 1: Selector ladder used on fixed k=5k=5 pools. Tier Description sample0 First generated candidate. L1 Same-family LLM selects without editing. L3 Different-family or different-size selector. L4-gen Selection using generated tests. L4-public Selection using public or visible tests. Oracle Any officially correct candidate (upper bound). 3.5 Critic Actions and Routing Boundary The word “critic” can denote scientifically different actions. A verdict critic judges one candidate; a selector critic chooses among candidates; neither creates a new answer. A repair critic rewrites an answer, and test-feedback repair rewrites using an external failure signal. Only the first two are bounded by the fixed-pool oracle gap. Our main tables therefore evaluate selector and verifier mechanisms, not repair. Worker routing is also distinct from candidate selection: it chooses a worker from input-level features before candidate evidence exists. Because routing depends on feature predictability rather than candidate-level verification, its full definition and results are reported as a supplementary boundary analysis (Appendix A.2), not as a selector-ladder tier. 3.6 Benchmarks, Models, and Statistical Reporting Table 2: Benchmarks and their diagnostic roles. Benchmark Size Role LiveCodeBench 1,055 Main code benchmark MATH L5 hard 250/seed Symbolic answer checks GPQA-Diamond 198/seed Low-recoverable boundary HumanEval+ 164 Saturated code anchor The main model source is Qwen3.6-35B-A3B-BF16 for LiveCodeBench, MATH, and the GPQA pool-boundary table; Qwen3.5-9B, Qwen2.5-14B, DeepSeek-R1-Distill-Qwen-14B, and Gemma-3-4B-it appear only in follow-up, robustness, or routing analyses. To avoid provenance mismatches, every horizontally compared number carries a five-part provenance tuple (benchmark,taskset,reference,labelsource,modelsource)(benchmark,task\ set,reference,label\ source,model\ source); numbers with different provenance are never merged into one table, and aggregation and confidence-interval methods are declared per table. The legacy suffix _vllm is a run label mapping to Qwen3.6-35B-A3B-BF16 via the run-label registry. Code tasks use the official LiveCodeBench evaluator; MATH uses trusted regrading or explicitly marked answer-grounded checks; GPQA compares extracted option letters with dataset labels. The LiveCodeBench main table uses a three-seed all-tier common task set with task-cluster hierarchical bootstrap confidence intervals (resampling tasks, keeping seed observations within clusters). Descriptive boundary analyses (GPQA) report per-seed ranges and explicit denominators. Before running the full ladder we preregistered: (i) 8–14% recoverable mass for the Qwen3.6-35B pool; (i) public/execution selection beats generated-test selection; (i) generated-test selection beats natural-language selection; (iv) cross-model natural-language selection does not stably approach execution selection; and (v) all mechanism results report coverage, quality, harm, fixed/harmed counts, confidence intervals, and common denominators. Section 4 reports which expectations held. 4 Results 4.1 LiveCodeBench: Fidelity Explains the Gap between Test-Based Selectors Table 3 shows the three-seed ladder on the all-tier common set (969/960/959 task-seed observations per seed; 2,888 total). Table 3: LiveCodeBench selector ladder on the three-seed common set. Mechanism Pass@1 Gain vs. sample0 95% CI Fixed / harmed First sample 2089/2888 = 72.33% 0 — — L3 cross-model LLM 2146/2888 = 74.31% +1.97+1.97p [0.74,3.20][0.74,3.20] 175 / 118 L4-gen generated tests 2167/2888 = 75.03% +2.70+2.70p [2.02,3.43][2.02,3.43] 80 / 2 L1 same-family LLM 2190/2888 = 75.83% +3.50+3.50p [2.26,4.73][2.26,4.73] 199 / 98 L4-public public tests 2324/2888 = 80.47% +8.14+8.14p [6.99,9.36][6.99,9.36] 235 / 0 Oracle any@5 2428/2888 = 84.07% +11.74+11.74p — 339 / 0 The oracle any@5 row is not deployable; it measures recoverable mass: 339/2888 first-sample failures have a correct candidate. This bound is large enough for selection to matter, but mechanisms capture it unevenly. The strongest contrast is between the two execution-based verifiers: +8.14+8.14p (public tests) versus +2.70+2.70p (generated tests). We therefore measure fidelity at candidate level. On the all-tier common candidate slice (14,440 candidates), the public-test verifier reaches 92.98% accuracy and MCC 0.825; the generated-test verifier reaches 53.05% and MCC 0.248 (same ordering on the full slice). Figure 2 visualizes the ladder and the fidelity/risk profile. The L4-public/L4-gen gap is not simply “tests help”; it is a fidelity gap between two test signals. Figure 2: LiveCodeBench selector performance and verifier fidelity/risk profile on the common slice. Public tests capture substantially more of the oracle gap than generated tests; generated tests remain conservative and low-harm. The hatched, dashed oracle bar is a non-deployable upper bound, visually separated from deployable mechanisms. The natural-language results are more nuanced. L1 is numerically higher than L4-gen by +0.80+0.80p, but the direct paired contrast crosses zero (95% CI [−0.38,2.01][-0.38,2.01]): the preregistered prediction that generated tests beat a same-family LLM selector is not supported. The two mechanisms occupy different risk points, however: L1 captures 199/339 recoverable cases but harms 98/2089 first-sample-correct cases (4.69%); L4-gen captures 79/339 on the clean decomposition slice with 2/2089 harm (0.10%). The low harm of L4-gen is partly structural: generated tests exist for 2444/3165 = 77.22% of task-seed rows, and at least one candidate passes them on only 1132/3165 = 35.77%; a raw trace audit (2705 available rows) records 1736 fallback-to-sample0 decisions, including 618 rows with no generated tests, versus 969 active selections. The conservative fallback protects correct first samples but misses many recoverable cases. Table 4 instantiates the clean-slice accounting with the availability audit. Capture and harm are effective quantities after each mechanism’s fallback policy, not the conditional terms of Section 3.3; the fixed-minus-harmed accounting identity closes exactly on the clean slice (2,887 rows; one task-seed with inconsistent timeout/harness labels is excluded from the decomposition slice only). Table 4: Clean-slice accounting and signal availability. Mechanism Signal availability audit Effective capture Effective harm Accounting Observed L1 3155/3165 = 99.68% parseable 199/339 = 58.70% 98/2089 = 4.69% +3.50+3.50p +3.50+3.50p L3 1671/1671 = 100% available traces 175/339 = 51.62% 118/2089 = 5.65% +1.97+1.97p +1.97+1.97p L4-gen 77.22% tests; 35.77% active 79/339 = 23.30% 2/2089 = 0.10% +2.67+2.67p +2.67+2.67p L4-public 2711/3165 = 85.66% active 235/339 = 69.32% 0/2089 = 0% +8.14+8.14p +8.14+8.14p Signal availability binds, especially for generated tests. The L4-gen gap has two separable sources: coverage/activation loss (rows falling back without active test evidence) and fidelity loss (tests disagreeing with official labels when they exist). This is why the framework keeps coverage, fidelity, effective capture, and harm separate. Table 5 reports the preregistered scorecard; it is mixed, which is exactly why the decomposition is useful. Table 5: Preregistered LiveCodeBench scorecard. Recoverable mass (8–14%) Public >> generated Generated >> natural language L3 not near execution Full reporting Pooled 11.74% Yes +8.14+8.14 vs. +2.70+2.70p Yes Beats L3, not L1 No; CI overlaps 0 +1.97+1.97 vs. +8.14+8.14p Yes Tables 3–4 Yes 4.2 MATH: Symbolic Answer Equivalence Beats Self-Consistency MATH offers a different decidable signal: final answers can be normalized or checked for symbolic equivalence. Table 6 reports 750 task-seed observations (250 tasks × 3 seeds) under trusted per-task regrading. Table 6: MATH selector results under trusted regrading. Mechanism Ref. Net gain 95% CI F/H L4-symbolic SC +4.67+4.67 [2.93,6.40][2.93,6.40] 39/4 L4-symbolic sample0 +6.00+6.00 [3.87,8.13][3.87,8.13] 57/12 L1 SC −3.20-3.20 [−5.20,−1.20][-5.20,-1.20] 17/41 L3 SC −1.87-1.87 [−3.73,0.00][-3.73,0.00] 20/34 L4-symbolic should be read carefully: it is not a problem-grounded verifier that proves candidates from the question. It clusters final answers by math_verify equivalence and selects from the strongest class—518/750 = 69.07% of rows use an equivalence key, while 232/750 = 30.93% fall back to the first sample. The result shows that symbolic answer equivalence is a stronger aggregation baseline than surface normalized-answer SC, not that an arbitrary extra verifier beats SC. This result does not imply that every explicit math verifier beats self-consistency. A separate L4-grounded artifact, evaluated under its own label source, selects 419/750 correct answers while SC selects 420/750. It fixes no SC failures and harms one SC success, for −0.13-0.13p relative to SC under that artifact. We therefore treat L4-grounded as evidence that a weak grounding signal may fail to exceed SC, while the trusted L4-symbolic result is the formal positive result. Together, the MATH results reinforce the framework’s main distinction: an extra natural-language judge is not enough, but a high-fidelity answer-equivalence signal can convert recoverable mass into gain. 4.3 GPQA-Diamond: The Oracle Gap Is Too Small to Pay for Harm GPQA-Diamond answers normalize to option letters, but correctness is not execution-checkable. Table 7 reports the three-seed boundary on the Qwen3.6-35B pool. Table 7: GPQA-Diamond boundary on the 35B pool. Mechanism Accuracy Gain Fixed/harmed First sample 283/594 = 47.64% 0 — SC majority 273/594 = 45.96% −1.68-1.68p — L1 273/594 = 45.96% −1.68-1.68p 5/15 L3 269/594 = 45.29% −2.36-2.36p 6/20 Oracle any@5 301/594 = 50.67% +3.03+3.03p — Note: Across the three seeds, row-wise accuracy ranges are [45.96,48.99][45.96,48.99], [43.94,47.98][43.94,47.98], [43.94,47.98][43.94,47.98], [43.94,46.97][43.94,46.97], and [49.49,52.53][49.49,52.53], respectively. The oracle gap is only 18/594 = 3.03p: a perfect selector could add about three points. The raw pools explain why: 520/594 pools (87.54%) contain five identical answer letters (mean unique letters 1.138). SC is 10/594 below the first sample under the pipeline’s tie-break rule. LLM selectors are net-negative, and predictably so: L1 captures 5/18 recoverable cases but harms 15/283 first-sample-correct cases (5.30% harm; L3: 6/18, 7.07%). The cost side is roughly first-sample-correct mass times harm rate—for L1, 15/594 = 2.53p, already exceeding what five recovered cases can compensate. L1 and SC tie at 273/594 by cancellation, not degeneration: they disagree on 20 observations, each uniquely correct on 10. A weaker Qwen3.5-9B pool shrinks the boundary further (recoverable 0.67%, 94.44% identical pools): the oracle gap is a joint property of task, model, and sampling configuration, not a benchmark constant. 4.4 Summary The same diagnostic pattern appears across benchmarks—and signal availability binds across them as well (35.77% active generated-test rows on LiveCodeBench; 69.07% equivalence-key rows on MATH). On LiveCodeBench, substantial recoverable mass exists and high-fidelity public tests capture much of it; generated tests are positive but limited by low fidelity and low activation. On MATH, symbolic answer equivalence beats SC while LLM selectors are negative for L1 and marginal for L3. On GPQA-Diamond, the pool has too little recoverable mass for LLM selector harm to be worthwhile. Training-free collaboration is not intrinsically useful because it adds another agent; it is useful when the pool contains recoverable answers and the signal has enough coverage, fidelity, quality, and low harm. 5 Discussion and Limitations 5.1 Collaboration Gain Is Not a Topology Property The LiveCodeBench contrast among L4-public, L4-gen, L1, and L3 shows that gains do not arise from adding a judging layer by itself. Public tests are strongest, but the deployable generated-test verifier is not statistically distinguishable from L1 in this setting. The relevant axis is not simply test-based versus natural-language selection; it is the joint effect of signal fidelity, coverage, conditional quality, fallback behavior, and harm. The large L4-public–L4-gen gap identifies generated-test fidelity and activation as the current bottlenecks. 5.2 Best-of-N Supplies Candidate Space Best-of-N and self-consistency are not outside collaboration. They create the candidate pool; selectors and verifiers determine whether that pool is converted into a better final answer. Without an oracle gap, no fixed-pool selector can help. With an oracle gap but weak verification, the available improvement remains uncaptured. This two-stage view separates candidate generation from selection and makes clear why agent count or sampling budget alone is not an explanation of realized gain. 5.3 Scope and Limitations First, LiveCodeBench selected-output evaluation and stored candidate oracle labels contain a small timeout/harness inconsistency. We identify seed43/lcb_abc323_d as the responsible task-seed and exclude it only from the clean decomposition slice, while retaining official selected-output evaluation in the main pass@1 table. Second, candidate-level fidelity is fully measured for L4-public and L4-gen, not for the L1/L3 natural-language selectors. The L3 availability audit also covers only the available trace subset rather than the full selected-output slice. Adding LLM verdict matrices would support a broader fidelity comparison; we leave this extension for future work rather than treating final pass@1 as a substitute for signal fidelity. Third, L4-public uses task-visible tests and should be read as a public-test verifier or partial-oracle diagnostic upper bound, not as a fully deployable mechanism. The current overlap audit is a 50-task sample rather than a complete 1,055-task non-overlap proof. Fourth, MATH contains two mechanism and label-source regimes. The trusted L4-symbolic per-task selector is the formal positive result. L4-grounded uses a separate grounded-summary label source and is interpreted separately rather than merged into the same horizontal comparison. Fifth, GPQA-Diamond establishes a low-recoverable boundary for the studied Qwen3.6-35B pool, not for every natural-language knowledge task. The oracle gap is a joint property of the task, model, prompt, temperature, and sampling configuration; more diverse sampling could change the available recoverable mass. Sixth, this paper studies selector and verifier mechanisms, not repair. Repair critics can create answers outside the fixed pool, but they introduce a distinct detection–generation trade-off and a different harm profile. Seventh, we do not claim compute optimality. All main selector results use k=5k=5 candidate pools, and the LiveCodeBench token-cost proxy was not recomputed on the final all-tier common set. Earlier probes suggest comparable gain per selector token for L4-gen and L1, with lower efficiency for L3, but those are appendix-level diagnostics rather than a compute-matched conclusion. A deployment decision should compare verifier calls with larger-k sampling, normalized self-consistency where applicable, and execution cost (Snell et al., 2024; Singhi et al., 2025). Finally, the diagnostic requires labels on a development set. Estimating oracle gap, fidelity, quality, and harm is not label-free in the form used here; it is intended as a small labeled audit before investing in a collaboration mechanism. GPQA illustrates the potential sample efficiency of that audit: 594 task-seed observations are sufficient to identify a 3.03p oracle gap and predict net-negative LLM selection. Label-free reliability estimation for generated tests is a promising substitute when development labels are unavailable. 6 Conclusion Training-free collaboration is not valuable merely because it adds an agent or critic. In fixed-pool selection, recoverable mass bounds gain; signal coverage, fidelity, quality, and harm determine how much is realized. High-fidelity public tests capture substantial oracle space on LiveCodeBench, symbolic equivalence beats self-consistency on MATH, and LLM selectors become net-negative on homogeneous GPQA-Diamond pools with a 3.03p oracle gap. The deployment rule is therefore: estimate oracle gap first, then audit signal availability, fidelity, and harm before spending additional inference compute. Reproducibility Statement The paper reports task-set denominators, seed counts, model provenance, label sources, selection rules, fixed/harmed counts, and confidence-interval procedures. The supplementary material documents the routing extension and additional audit details. Code and artifacts, including the run-label registry, selected-output records, candidate verdicts, and analysis scripts, are available at https://github.com/AmGarfield/OracleGap. References B. Brown, J. Juravsky, R. Ehrlich, R. Clark, Q. V. Le, C. Ré, and A. Mirhoseini (2024) Large language monkeys: scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787. External Links: Link Cited by: §2.1. M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandran, M. Zaharia, J. E. Gonzalez, and I. Stoica (2025) Why do multi-agent LLM systems fail?. arXiv preprint arXiv:2503.13657. External Links: Link Cited by: §2.3. B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J. Lou, and W. Chen (2022) CodeT: code generation with generated tests. arXiv preprint arXiv:2207.10397. External Links: Link Cited by: §2.1. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. External Links: Link Cited by: §2.2. F. E. Dorner, Y. Chen, A. F. Cruz, and F. Yang (2025) ROC-n-reroll: how verifier imperfection affects test-time scaling. arXiv preprint arXiv:2507.12399. External Links: Link Cited by: §2.2. Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch (2023) Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325. External Links: Link Cited by: §2.3. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Cited by: §1. Z. Hu, C. Liu, T. Huang, J. Li, Y. Yang, J. Wu, Z. Zhong, M. Yang, and Y. Yue (2026) CoSPlay: cooperative self-play at test-time with self-generated code and unit test. arXiv preprint arXiv:2605.23491. External Links: Link Cited by: §2.4. J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou (2024) Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798. External Links: Link Cited by: §2.3. N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica (2024) LiveCodeBench: holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974. External Links: Link Cited by: §1. Y. Kim, S. Lee, C. Jang, H. Lee, and J. Lee (2026) Parallel test-time scaling with multi-sequence verifiers. arXiv preprint arXiv:2603.03417. External Links: Link Cited by: §2.2. Y. Kim, K. Gu, C. Park, C. Park, S. Schmidgall, A. A. Heydari, Y. Yan, Z. Zhang, Y. Zhuang, Y. Liu, M. Malhotra, P. P. Liang, H. W. Park, Y. Yang, X. Xu, Y. Du, S. Patel, T. Althoff, D. McDuff, and X. Liu (2025) Towards a science of scaling agent systems. arXiv preprint arXiv:2512.08296. External Links: Link Cited by: §2.3. J. Kwok, S. Li, P. Atreya, Y. Liu, Y. Jiang, C. Finn, M. Pavone, I. Stoica, and A. Mirhoseini (2026) LLM-as-a-Verifier: a general-purpose verification framework. arXiv preprint arXiv:2607.05391. External Links: Link Cited by: §2.2. D. Lee, C. Hwang, and K. Lee (2026) Learning to generate unit test via adversarial reinforcement learning. In International Conference on Learning Representations, External Links: Link Cited by: §2.4. Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, T. Hubert, P. Choy, C. de Masson d’Autume, I. Babuschkin, X. Chen, P. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. Robson, P. Kohli, N. de Freitas, K. Kavukcuoglu, and O. Vinyals (2022) Competition-level code generation with AlphaCode. Science 378 (6624), p. 1092–1097. External Links: Document Cited by: §2.1. S. Lifshitz, S. A. McIlraith, and Y. Du (2025) Multi-agent verification: scaling test-time compute with multiple verifiers. arXiv preprint arXiv:2502.20379. External Links: Link Cited by: §2.2. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2024) Let’s verify step by step. In International Conference on Learning Representations, External Links: Link Cited by: §2.2. J. Liu, C. S. Xia, Y. Wang, and L. Zhang (2023) Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §1. J. Lu, R. Teehan, J. Jin, and M. Ren (2025) When does verification pay off? a closer look at LLMs as solution verifiers. arXiv preprint arXiv:2512.02304. External Links: Link Cited by: §2.2. A. Prasad, E. Stengel-Eskin, J. C. Chen, Z. Khan, and M. Bansal (2025) Learning to generate unit tests for automated debugging. In Conference on Language Modeling, External Links: Link Cited by: §2.4. D. Qiao, B. Chen, F. Cai, J. Chen, W. Li, F. Jiang, Z. Chen, H. Zha, T. Zhang, and B. Wang (2026) Epistemic gain, aleatoric cost: uncertainty decomposition in multi-agent debate for math reasoning. arXiv preprint arXiv:2603.01221. External Links: Link Cited by: §2.3. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman (2024) GPQA: a graduate-level google-proof Q&A benchmark. In First Conference on Language Modeling, External Links: Link Cited by: §1. A. Setlur, N. Rajaraman, S. Levine, and A. Kumar (2025) Scaling test-time compute without verification or RL is suboptimal. arXiv preprint arXiv:2502.12118. External Links: Link Cited by: §2.2. F. Shi, D. Fried, M. Ghazvininejad, L. Zettlemoyer, and S. I. Wang (2022) Natural language to code translation with execution. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, p. 3533–3546. External Links: Document, Link Cited by: §2.1. N. Singhi, H. Bansal, A. Hosseini, A. Grover, K. Chang, M. Rohrbach, and A. Rohrbach (2025) When to solve, when to verify: compute-optimal problem solving and generative verification for LLM reasoning. arXiv preprint arXiv:2504.01005. External Links: Link Cited by: §2.2, §5.3. C. Snell, J. Lee, K. Xu, and A. Kumar (2024) Scaling LLM test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314. External Links: Link Cited by: §2.1, §5.3. B. Stroebl, S. Kapoor, and A. Narayanan (2024) The limits of inference scaling through resampling. arXiv preprint arXiv:2411.17501. External Links: Link Cited by: §2.2. D. Tran and D. Kiela (2026) Single-agent LLMs outperform multi-agent systems on multi-hop reasoning under equal thinking token budgets. arXiv preprint arXiv:2604.02460. External Links: Link Cited by: §2.3. J. Uesato, N. Kushman, R. Kumar, H. F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins (2022) Solving math word problems with process- and outcome-based feedback. arXiv preprint arXiv:2211.14275. External Links: Link Cited by: §2.2. V. Venktesh, M. Rathee, and A. Anand (2025) Trust but verify! a survey on verification design for test-time scaling. arXiv preprint arXiv:2508.16665. External Links: Link Cited by: §2.2. X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou (2023) Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, External Links: Link Cited by: §2.1. Y. Yang, C. Qu, M. Wen, L. Shi, Y. Wen, W. Zhang, A. Wierman, and S. Gu (2026) Understanding agent scaling in LLM-based multi-agent systems via diversity. arXiv preprint arXiv:2602.03794. External Links: Link Cited by: §2.3. Y. Zhao, S. Chen, and N. Su (2026) On the Uncertainty of Large Language Model-Based Multi-Agent Systems. arXiv preprint arXiv:2602.04234. External Links: Link Cited by: §2.3. Appendix A Supplementary Results A.1 HumanEval+ Saturation Anchor HumanEval+ serves only as a saturated code anchor. In the Qwen3.6-35B V100 follow-up, first-sample pass@1 is 138/164, 139/164, and 138/164 across seeds 43/44/45, while oracle any@5 is 144/164, 144/164, and 142/164. The corresponding recoverable counts are only 6/164, 5/164, and 4/164, so HumanEval+ is not used as the main code selector benchmark. A.2 Routing-η Extension Worker routing chooses which model should answer an input rather than selecting among candidates from a fixed pool. We define η=score(learnedrouter)−score(defaultworker)score(oraclerouter)−score(defaultworker).η= score(learned\ router)-score(default\ worker)score(oracle\ router)-score(default\ worker). (7) An η near one means that the learned router captures most of the oracle routing space; an η near zero means it does not exploit the available worker differences. Table 8 summarizes the strongest-feature runs. These results are descriptive and are not part of the selector-ladder evidence because the default-worker identities and confidence-interval pipeline are not integrated into the paper-level provenance system. Table 8: Routing-η extension. Benchmark Router Default Router Oracle η (95% CI) LCB Logistic 64.93 64.93 69.48 0.000 [0,0][0,0] LCB Random forest 64.93 66.26 69.48 0.292 [.122,.459][.122,.459] MATH Logistic 46.80 58.80 60.00 0.909 [.783,1][.783,1] MATH Random forest 46.80 58.00 60.00 0.848 [.700,.968][.700,.968] GPQA Logistic 43.94 56.57 66.16 0.568 [.368,.750][.368,.750] GPQA Random forest 43.94 57.07 66.16 0.591 [.386,.769][.386,.769] The LCB logistic router degenerates to selecting the default worker, yielding η=0η=0. The remaining results show that input-level routing can recover some oracle-routing space, but this depends on worker differences and feature predictability rather than candidate-level verification. Candidate selection and worker routing are both test-time collaboration mechanisms, but they obey different constraints. A.3 Additional Audit Details One task-seed observation, seed43/lcb_abc323_d, is excluded from the clean decomposition slice because selected-output evaluation marks it as passing while all stored candidate oracle labels fail with timeout or harness codes. The main pass@1 table retains the official selected-output evaluation. The raw generated-test trace audit contains 2,705 available rows, versus 2,887 rows in the clean decomposition slice, because the seed-45 trace artifact is incomplete; final pass@1 uses the official selected-output details. The public-test verifier should be interpreted as a partial-oracle diagnostic upper bound, not a fully deployable mechanism. The current overlap audit covers a 50-task sample rather than proving non-overlap for all 1,055 LiveCodeBench tasks. The MATH grounded artifact uses a separate label source and is therefore not merged with the trusted symbolic-equivalence result. A.4 LLM Usage Disclosure An LLM-based assistant was used for language editing and preparation of the LaTeX manuscript. The author reviewed the resulting text, equations, tables, citations, and numerical claims and retains full responsibility for the submission.

Instant research discovery