Paper deep dive
Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles
Donghwan Kim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/24/2026, 2:12:58 AM
Summary
This paper audits five diversity metrics (strict diversity, disagreement, double-fault, Jaccard error overlap, and focal diversity) to determine if they measure true diversity or merely re-express model capability. Using 31,900 subsets of 30 LLMs on MMLU-Pro and TruthfulQA, the study finds that majority-vote gain over the best member is rare (9.98% for size-3 subsets). Key findings include: 1) Latent complementarity is ubiquitous, but realized gain is low. 2) Strict diversity is nearly collinear with mean accuracy (Spearman rho ~0.99), indicating strong entanglement with capability. 3) After controlling for capability, the only robust residual signal is a modest pairwise co-failure association where shared errors correlate with lower gain. The three linear contingency-table statistics are algebraically non-separable, leading to rank-deficient regressions in raw space.
Entities (10)
Relation Signals (6)
Donghwan Kim → authored → Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles
confidence 99% · Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles Donghwan Kim
Aidentyx Inc. → affiliatedwith → Donghwan Kim
confidence 95% · Donghwan Kim Aidentyx Inc., San Jose, CA, USA
Disagreement → isalgebraicallyrelatedto → Double-Fault
confidence 95% · strict = disagreement + double-fault
Strict Diversity → isalgebraicallyrelatedto → Disagreement
confidence 95% · strict = disagreement + double-fault
Strict Diversity → isentangledwith → Model Capability
confidence 95% · strict diversity is nearly collinear with one minus mean accuracy (size-3 Spearman rho = +0.991 / +0.988)
Majority Voting → haslowgainrateon → MMLU-Pro
confidence 90% · simple voting beats the strongest member in only 9.98% of all canonical size-3 subsets
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Majority voting over LLMs is widely assumed to benefit from diversity, and diversity measures are used to choose which models to combine. We ask whether five such measures track diversity or mainly re-express capability, auditing them as predictors of majority-vote gain over the best member across 31,900 subsets of 30 LLMs on MMLU-Pro (29 on TruthfulQA) under explicit capability controls. Three findings emerge. First, latent complementarity is ubiquitous: oracle gain is positive in 100% of subsets, yet simple voting beats the strongest member in only 9.98% of all canonical size-3 subsets (18.71% with held-out best selection); the pooled size-2-4 rate is 1.27%, partly reflecting deterministic even-size voting behavior. Second, a joint-correctness proxy (strict diversity) is nearly collinear with one minus mean accuracy (size-3 Spearman rho = +0.991 / +0.988); raw diversity-gain associations are strongly capability-entangled and, with one exception, unstable under control. Third, three linear contingency-table statistics are algebraically non-separable; after capability control, the empirically stable remainder is a modest residual pairwise co-failure association in which more shared error corresponds to lower gain. This direction is robust, but its magnitude is configuration-dependent. Joint rawspace linear regressions treating strict diversity, disagreement, and double-fault as independent predictors are rank-deficient by construction.
Tags
Links
- Source: https://arxiv.org/abs/2607.20768v1
- Canonical: https://arxiv.org/abs/2607.20768v1
Trouble viewing inline? Open PDF directly →
Full Text
68,151 characters extracted from source content.
Expand or collapse full text
Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles Donghwan Kim Aidentyx Inc., San Jose, CA, USA david.kim@aidentyx.com Abstract Majority voting over LLMs is widely assumed to benefit from diversity, and diversity measures are used to choose which models to combine. We ask whether five such measures track diversity or mainly re-express capability, auditing them as predictors of majority-vote gain over the best member across 31,900 subsets of 30 LLMs on MMLU-Pro (29 on TruthfulQA) under explicit capability controls. Three findings emerge. First, latent complementarity is ubiquitous: oracle gain is positive in 100% of subsets, yet simple voting beats the strongest member in only 9.98% of all canonical size-3 subsets (18.71% with held-out best selection); the pooled size-2–4 rate is 1.27%, partly reflecting deterministic even-size voting behavior. Second, a joint-correctness proxy (strict diversity) is nearly collinear with one minus mean accuracy (size-3 Spearman ρ=+0.991ρ=+0.991 / +0.988+0.988); raw diversity–gain associations are strongly capability-entangled and, with one exception, unstable under control. Third, three linear contingency-table statistics are algebraically non-separable; after capability control, the empirically stable remainder is a modest residual pairwise co-failure association in which more shared error corresponds to lower gain. This direction is robust, but its magnitude is configuration-dependent. Joint raw-space linear regressions treating strict diversity, disagreement, and double-fault as independent predictors are rank-deficient by construction. Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles Donghwan Kim Aidentyx Inc., San Jose, CA, USA david.kim@aidentyx.com 1 Introduction Majority voting is one of the canonical aggregation rules in ensemble learning (Dietterich, 2000), and its appeal rests on a classical intuition: if base predictors fail on different inputs, meaning they are diverse, a vote can outvote individual errors and exceed any single member (Kuncheva and Whitaker, 2003). The rule is equally central in LLM practice, from self-consistency decoding (Wang et al., 2023) to multi-agent sampling-and-voting (Li et al., 2024), and a family of diversity measures is used, explicitly or implicitly, to reason about which models to combine. Two facts complicate this for modern LLMs. LLM errors are strongly correlated, and more accurate models are, if anything, more correlated (Kim et al., 2025; Goel et al., 2025). Recent work therefore balances competence and diversity through adaptive filtering (Cohen et al., 2026), quality-matched pools and co-failure ceilings (Chen, 2026), or accuracy-adjusted dependence measures (Ali, 2026). We complement this prescriptive and concurrent work with a measurement question: for modern LLM pools, how do five diversity-related measures behave before and after explicit capability control? We answer this by treating realized majority-vote gain, defined as the vote’s accuracy minus that of the best individual member, as the dependent variable and auditing five diversity-related measures as its predictors under explicit capability controls. Beating the strongest available member is a deliberately stringent criterion. It is not the only operationally relevant baseline: cost, latency, and model-selection uncertainty all matter in deployment (Chen et al., 2024a). But if an ensemble cannot outperform its strongest constituent on accuracy, diversity alone does not justify aggregation on accuracy grounds, and our held-out selection analysis addresses the case where the strongest member is not known in advance. Majority vote is the canonical rule being reasoned about when diversity is invoked; we audit that classical intuition rather than learned aggregators, routers, or judges. Our findings are diagnostic and deflationary: several commonly used diversity-related statistics are strongly entangled with capability, most severely for the three linearly coupled contingency-table statistics, namely strict diversity, disagreement, and double-fault. After adjustment, the most directionally stable remainder is a modest pairwise co-failure association, while other residual associations are specification-dependent. Our contributions are: 1. A capability-controlled audit. We audit five diversity-related measures as predictors of majority-vote gain across 31,900 subsets of 30 modern LLMs on MMLU-Pro, reproduced with 29 models on TruthfulQA, under six linear capability-control specifications (best; best+mean; and four additional specifications), together with nonlinear and matched analyses, model-level resampling robustness analysis, and slice- and threshold-robustness checks. 2. Entanglement with capability. We show that several pairwise diversity-related statistics are strongly entangled with member capability; in particular, our joint-correctness proxy (“strict diversity”) is nearly collinear with one minus mean accuracy (size-3 Spearman ρ=+0.991ρ=+0.991 / +0.988+0.988), helping explain why several counter-intuitive raw diversity–gain associations attenuate or become specification-dependent under capability control. 3. A modest residual pairwise co-failure axis, and algebraic non-separability. Strict diversity, disagreement, and double-fault are algebraically non-separable (strict == disagreement ++ double-fault; 1−Acc¯=1- Acc= double-fault +12+ 12 disagreement), so raw-space linear control of mean capability forces a one-dimensional residual axis (r=−1.000r=-1.000, slope −1/2-1/2) and joint raw-space linear regressions of the three are rank-deficient. That raw-space collapse is algebraic; the empirical finding is that the corresponding rank-space pairwise co-failure signal remains associated with majority-vote gain: more shared error corresponds to lower gain. This direction is robust, but its magnitude is conditional. We are explicit about what we do not claim. We do not claim diversity never helps, that any single statistic is uniquely “correct,” or that our residual signal is large. The claims are measurement-level and associational, not causal, and their magnitudes are conditional on slice and roster composition (Section 5.6, Limitations). 2 Related Work 2.1 Classical ensemble diversity. Ensemble methods classically combine base predictors by (weighted) voting (Dietterich, 2000), with diversity long treated as a key ingredient and an explicit target of ensemble construction (Brown et al., 2005). The intuition that ensemble gains reflect both member accuracy and disagreement traces back to the ambiguity decomposition of neural network ensembles (Krogh and Vedelsby, 1995). Ensemble-pruning work explicitly optimizes a trade-off between classifier strength and divergence (Zhang et al., 2006), and unified theory warns against maximizing diversity without managing the bias/variance/diversity trade-off (Wood et al., 2023). Yet measuring diversity has never been straightforward: Kuncheva and Whitaker (2003) catalogued diversity measures via a pairwise contingency table and found their relationship to ensemble accuracy weak and inconsistent on real problems. Their catalogue defines disagreement and double-fault (their Eqs. 6–7), but not (b+c+d)/N(b+c+d)/N or Jaccard error overlap; strict diversity is our name for the complement of joint correctness. We revisit their question for modern LLM ensembles, with realized gain over the best member as the target and explicit capability controls. 2.2 Voting and self-consistency in LLMs. Voting-style aggregation is widely used with LLMs: self-consistency decoding samples multiple reasoning paths and selects the modal answer (Wang et al., 2023), extended to free-form generation by universal self-consistency (Chen et al., 2024b), and multi-agent studies report gains from sampling-and-voting as the number of instantiated agents grows (Li et al., 2024; Du et al., 2024). These results motivate majority voting as an object of study, but they largely concern single-model self-sampling gains over an agent’s own baseline, not when voting over a heterogeneous pool beats the strongest available member. Condorcet-style guarantees require stronger conditions than is sometimes stated: independence of classifier outputs alone is not enough without suitable independence of their errors (Vardeman and Morris, 2013). In financial-sentiment experiments, heterogeneous NLP/LLM voting produced only marginal improvement, which the authors attribute to insufficient independence (Lefort et al., 2024). This is task-specific preprint evidence, not a general theorem about LLMs. Our question is the measurement version: when majority-vote gain over the best member is the target, do diversity-related measures carry information beyond member capability? 2.3 Correlated errors and model similarity. Kim et al. (2025) show LLM errors are correlated and that stronger models are more correlated; Goel et al. (2025) find stronger models make more similar mistakes. These works document the empirical pattern underlying our capability confound, in which accurate models agree and fail together, but study error correlation itself, not majority-vote gain, and do not quantify how strongly standard diversity measures re-express capability. Our diagnosis builds on this pattern and makes its consequence for diversity measurement precise. 2.4 LLM ensemble selection, quality control, and aggregation. Prescriptive work designs aggregation and selection rules (Chen et al., 2025), including ranking/fusion (Jiang et al., 2023; Si et al., 2023), layered proposer–aggregators (Wang et al., 2025; Li et al., 2026), focal-diversity pruning (Tekin et al., 2024), competence filtering (Cohen et al., 2026), higher-order aggregation (Ai et al., 2026), error-distribution shaping (Baek et al., 2026), and complementarity- or information-theoretic selection (Zhang et al., 2026; Turkmen et al., 2026). Concurrent work on quality matching, all-member co-failure ceilings (Chen, 2026), and accuracy-adjusted dependence (Ali, 2026) reinforces the need to separate dependence from capability. Our focus is complementary and diagnostic: unweighted size 2–4 majority subsets, five diversity-related measures under nested controls, and the algebraic non-separability of the three linear contingency-table statistics. 3 Preliminaries 3.1 Contingency-table measures, two identities, and focal diversity For a pair of models on N items, the standard 2×22× 2 contingency table (Kuncheva and Whitaker, 2003) assigns each item to one cell by whether each model is correct: j correct j wrong i correct a b i wrong c d We select five measures to span joint correctness, split errors, shared errors, normalized error overlap, and a recent LLM-specific subset criterion. The audit is not intended as an exhaustive catalogue of classical diversity indices. In particular, the Q-statistic, the correlation coefficient, and kappa are functions of the same 2×22× 2 cell counts, and Kuncheva and Whitaker (2003) report that these pairwise measures are strongly intercorrelated; we therefore select representatives spanning distinct structural roles rather than near-duplicates. We audit four contingency-table measures, extended to subsets of size k by averaging over member pairs: • Strict diversity (our term; not a named measure in Kuncheva and Whitaker’s catalogue): SD=(b+c+d)/NSD=(b+c+d)/N, the complement of joint correctness. • Disagreement: (b+c)/N(b+c)/N • Double-fault (co-failure): d/Nd/N • Mean pairwise Jaccard error-set similarity: d/(b+c+d)d/(b+c+d), the standard Jaccard index on model error sets, averaged over pairs (0 when both error sets are empty). The mean of pairwise ratios is close to, but not identical with, the pooled ratio of sums (max divergence 0.089); we retain the pairwise-mean definition throughout. Two exact identities frame the analysis. First, strict =disagreement+double-fault, =disagreement+double-fault, max|ϵ| |ε| <1.9×10−16, <9× 0^-16, where ϵε is the numerical verification error. Thus, strict, disagreement, and double-fault are linearly dependent, both pairwise and after pairwise averaging. At the pair level, Jaccard is the ratio double-fault/strictdouble -fault/strict. Because the subset statistic averages pairwise Jaccard ratios, however, it is not algebraically determined by the subset-level averaged double-fault and strict measures (the maximum observed divergence from their ratio is 0.089). Second, with Acc¯=(2a+b+c)/(2N) Acc=(2a+b+c)/(2N) the pair mean accuracy, 1−Acc¯=DoubleFault+12Disagreement,1- Acc=DoubleFault+ 12Disagreement, an exact identity preserved under pairwise averaging to the subset level. In raw (unranked) measure space, any linear residualization that includes mean member accuracy therefore forces DoubleFaultres=−12Disagreementres,DoubleFault_res=- 12Disagreement_res, exactly (Pearson r=−1r=-1, slope −1/2-1/2). This identity need not be preserved after the marginal rank transformations used for partial Spearman correlations, so the controlled magnitudes in Table 3 need not be symmetric. Raw-space one-dimensionality after mean control is thus algebraic; whether the corresponding rank-space association with majority-vote gain is robust is empirical (Section 5.4–Section 5.5). Separately, we audit focal diversity (Tekin et al., 2024, Eq. 10), a subset-level measure that is not a member of this algebraic family. For a subset of k models, let FqF_q be the number of members that fail on item q. For each focal member i that fails at least once, we compute P1,i P_1,i =q:i fails[Fqk], =E_q:i fails\! [ F_qk ], P2,i P_2,i =q:i fails[Fq(Fq−1)k(k−1)], =E_q:i fails\! [ F_q(F_q-1)k(k-1) ], ρi _i =1−P2,iP1,i, =1- P_2,iP_1,i, and define subset focal diversity as the mean of ρi _i over eligible focal members (undefined focal terms are skipped). Any attenuation we report for focal is therefore an empirical finding, not an algebraic consequence of the two identities above. 3.2 Majority vote, gain, and an oracle upper bound For subset S of size k: the pipeline’s majority vote counts an item correct iff the number of correct members strictly exceeds k/2k/2 (a correct-count rule; exact even-size splits count as incorrect). Section 5.1 reports robustness under an answer-level plurality rule (most-voted answer wins; the two rules are provably identical at sizes 2–3 and differ only at size 4). With AccMV(S)Acc_MV(S) and Accbest(S)Acc_best(S) the best member’s accuracy on the same item set: • Gain: g(S)=AccMV(S)−Accbest(S)g(S)=Acc_MV(S)-Acc_best(S). • Oracle upper bound: the gain of a hypothetical aggregator that is correct whenever any member is. This is not a real aggregator and is used only as an upper bound on latent complementarity. Under the correct-count rule, size-2 majority vote can never exceed the best member (MV correct requires both correct), so size 2 is excluded from gain-rate headlines. Unless stated otherwise, “best” is selected in-sample; Section 5.1 quantifies held-out selection. All associations are between a measure and g(S)g(S) across subsets. 4 Experimental Setup Figure 1. Capability-controlled audit pipeline. Top: benchmark-specific routes, fixed filters, and common-parse slices. Bottom: exhaustive size-2–4 subsets, per-subset outcomes/diversity, capability controls, and robustness checks. Models and fixed filters. We ran 30 benchmark-specific model routes via OpenRouter. Primary MMLU-Pro inference was collected in April 2026, with a July 2026 retry confined to previously unparsed responses; TruthfulQA was collected in the same July pipeline. Provider selection followed OpenRouter’s default routing and was not pinned; provider metadata beyond the requested route ID was not recorded. The canonical inclusion rule uses parse rate ≥ 0.90 and full-500 accuracy in [0.40, 0.92]. The accuracy band was fixed in the April analysis pipeline and retained unchanged across MMLU-Pro, TruthfulQA, and the July parsing retry; its endpoints are a pragmatic construction choice, not a literature-derived or formally preregistered standard. The retry revisited only previously unparsed responses (larger token budgets, a reasoning-field fallback, simplified answer prompts). After retry, all 30 MMLU-Pro models pass both filters, so the accuracy band is non-binding there. We form all size 2–4 subsets: 435+4,060+27,405=31,900435+4,060+27,405=31,900. Full OpenRouter IDs, parse rates, full-sample accuracies, and the one benchmark-specific route-name discrepancy appear in Appendix A.1. Evaluation protocol. We use zero-shot multiple-choice prompts that request a final option label. Generation and parsing details, including the exact benchmark prompt templates and retry protocol, are provided in Appendix A.2. The common parsed slice. Items are sampled uniformly at random (no category stratification; seed = 42) from the MMLU-Pro test split (Wang et al., 2024) (12,032 items); the sampled category distribution does not differ significantly from the full split (χ2χ^2(13) = 19.9, p = 0.10). The larger distributional shift documented below therefore arises primarily after the parse filter rather than from obvious category imbalance in the initial sample: restricting to the intersection of items successfully parsed for all 30 models after the benchmark-specific parsing and retry pipeline reduces the item set from 500 to 356 questions, and this filtering is not neutral. Table 1 computes each item’s accuracy and disagreement over the models that parsed that item, then averages across items; unparsed responses are excluded from these descriptive per-item denominators rather than scored as incorrect. Table 1. Effect of common-parse filtering on the MMLU-Pro evaluation slice. Item set n mean acc disagreement sampled 500 0.753 0.218 retained (analyzed) 356 0.791 0.178 dropped 144 0.659 0.316 Domain composition differs between retained and dropped items (χ2(13)=61.65χ^2(13)=61.65, p=2.7×10−8p=2.7× 10^-8). Headline numbers are computed on this slice; Section 5.6 shows that the direction of the core capability-entanglement and co-failure results is unchanged under per-subset item denominators (median 461 items at size 3 and 452 at size 4) and a less-filtered 451-item slice, with residual caveats in Limitations. Pipeline audit and release plan. Gain computations are sensitive to denominator consistency (ensemble and best-member accuracies must use the same item set). We audited every gain-computing script in the codebase and regenerated all reported numbers from a single consistent pipeline; regenerated values match the analysis files to rounding (max |diff| ≤ 5×10−510^-5 over all 31,900 rows), and the gain>0 base rate (1.27%) and best-member accuracies reproduce exactly. The planned public release includes the audit and generation scripts plus the derived binary correctness and parse-status matrices (model × item), item IDs, and the model roster, which suffice to reproduce the core results even if redistribution constraints prevent releasing raw model responses; raw prediction caches remain conditional on redistribution review. Cross-benchmark reproduction and threshold sensitivity. We repeat the core analysis on TruthfulQA (Lin et al., 2022) (29 models; 338 common items; 27,811 subsets). qwen/qwen3.6-plus alone is excluded by the historical 0.92 upper bound (full-500 accuracy 0.922). In a post-hoc sensitivity analysis whose protocol was fixed before execution, we remove the upper cutoff and admit all 30 models on the same 338-item denominator: gain>0 = 0.98% (canonical 1.09%); size-3 controlled strict/disagreement/double-fault = +0.397/+0.313/−-0.570 (canonical +0.378/+0.297/−-0.553); the double-fault model-resampling interval remains entirely negative. The lower 0.40 bound is non-binding in both retried datasets. Robustness under subset dependence (design choice). The 31,900 subsets reuse the same 30 models and are highly overlapping; they are not independent observations, and subset-level p-values would be inflated. We therefore do not report subset-level significance. Our evidentiary standard throughout is: direction, effect size, model-level resampling intervals (resampling models; B = 1,000; procedure in Appendix B), and leave-one-model-out stability. Because the resampling procedure deduplicates drawn models rather than reweighting by draw multiplicity, we describe its percentile intervals as robustness intervals, not conventional cluster-bootstrap confidence intervals (Appendix B). Capability controls. Six linear rank-space specifications: (i) best-member accuracy; (i) best + mean member accuracy; and four additional sets: best+minimum, best+median, best+standard deviation, and the full sorted member-accuracy vector (Section 5.5). Separate robustness estimators use nonlinear decile indicators for best+mean and matching on the discrete best-member accuracy with mean tolerance 0.01 and minimum stratum size 8. Benchmark item sampling uses seed 42; robustness analyses use the seeds reported with their protocols in Appendices B–C. 5 Results Estimator convention. Table 3 reports Spearman correlations: the Raw column over the full subset population; control columns are partial Spearman (linear residualization in rank space on the stated covariates). Size-4 counterparts are in Appendix Table C1. Additional estimators are in Appendix C (Figure C1, Table C2). 5.1 Majority voting rarely beats the best member Oracle gain is positive in 100% of subsets on both benchmarks, yet realized majority-vote gain is typically negative. Under the canonical correct-count rule with in-sample best selection, the most interpretable rate is at size 3, where the vote beats the strongest member in 9.98% of all canonical size-3 ensembles (Figure 2); pooled over all size-2–4 subsets the rate is 1.27%, mixing that genuine rarity with structural zeros (size 2 is identically 0% under the correct-count rule; size 4 is 0% largely because exact 2–2 ties count as incorrect). Selecting the best member on one half of items and evaluating gain on the other (20 seeds) raises the rates (Table 2). Held-out values are mean ± over 20 seeds. Size 2 is omitted: positive gain is structurally impossible under the correct-count rule (0.00% in both rows). Table 2. Percentage of ensembles with positive majority-vote gain under in-sample and held-out best-member selection. gain>0 (%) all (2–4) size 3 size 4 in-sample best 1.27 9.98 0.00 held-out best 3.44 (± 1.18) 18.71 (± 3.70) 1.23 (± 0.95) Excluding size-4 tie items (recomputing best on the same reduced set) yields 8.50% at size 4; an answer-plurality rule, identical at sizes 2–3 and differing only at size 4, yields 4.52%. TruthfulQA agrees (1.09%, or 0.98% with no upper accuracy cutoff). The qualitative conclusion is unchanged: latent complementarity is always present, yet simple voting converts it into a win in only 9.98% of all canonical size-3 subsets. This gap between latent and realized gain is not specific to unweighted voting: in closed-loop table recognition, iteration produces better candidates that a reference-free LLM judge largely fails to select (Kim, 2026). Figure 2. Oracle opportunity versus realized majority-vote gain, size 3. Most points lie below y = 0 despite strictly positive oracle gain; 9.98% lie above zero (dashed y = x). Sizes 2/4 appear in Table 2 only, as deterministic majority/tie behavior suppresses their rates. 5.2 Raw measure–gain correlations are misleading Table 3 (“Raw”, size 3) reports raw Spearman correlations with gain. The pattern contradicts the classical intuition: strict diversity and disagreement are negatively associated with gain, while mean pairwise Jaccard error-set similarity is positively associated. Taken at face value this would suggest overlap-based proxies carry the wrong sign. The next subsections show that this reading is strongly shaped by capability. 5.3 Capability explains most measure variation Strict diversity is empirically near-collinear with capability in this setting: strict≈1−mean member accuracy,ρMMLU-Pro=+0.991,ρTQA=+0.988. gatheredstrict≈ 1-mean member accuracy,\\ _MMLU-Pro=+0.991, _TQA=+0.988. gathered We stress which part of this is empirical. That strict diversity equals one minus the joint-correct rate is algebra; that the joint-correct rate is this collinear with mean accuracy is not. Under the same rank-space best+mean projection used for partial Spearman, only 1.1% of MMLU-Pro and 1.5% of TruthfulQA strict-diversity variance remains. Under independent member errors the quantities can diverge substantially; the observed near-collinearity is an empirical property of modern LLM pools, consistent with their strongly correlated errors (Kim et al., 2025). Its practical consequence is that this joint-correctness proxy leaves very little variation independent of capability; its controlled association with majority-vote gain is correspondingly specification-sensitive (Figure 3). Figure 4 generalizes the strict-diversity case across all five measures. Ranked best and mean member accuracy explain 98.9%/98.5% of strict, 92.2%/88.5% of disagreement, and 85.7%/83.8% of double-fault variation on MMLU-Pro/TruthfulQA, respectively. Entanglement is substantial but not uniform: the corresponding fractions are 56.4%/42.7% for Jaccard and 45.2%/23.0% for focal diversity. We therefore do not claim that all five are interchangeable capability proxies. Under capability control, most of the apparent raw diversity–gain associations do not survive as robust standalone effects: they attenuate, reverse, or become specification-dependent (Figure 5). The pairwise co-failure association is the notable exception, remaining negative across specifications (−-0.432 linear; −-0.474 nonlinear). Strict diversity flips to +0.339 under linear best+mean control, shrinks to +0.092 under nonlinear control, and vanishes in the 16-model pool (−-0.006). Focal diversity, though not algebraically tied to this family, empirically follows the same pattern of attenuation and specification dependence (+0.049 after control at size 3; −-0.138 at size 4). This sign instability is informative: a coefficient that reverses under a linear control and nearly disappears under a nonlinear one is consistent with residual capability confounding or specification sensitivity, rather than a robust standalone diversity effect. Table 3. Spearman association of five measures with majority-vote gain, ensemble size 3 (n = 4,060), under increasing capability control (control columns “| best” and “| best+mean”: partial Spearman, linear rank-space residualization). Robustness estimates and intervals in Appendix C (Figure C1, Table C2). † -control positive residuals in the full pool only (see Section 5.3, Section 5.6). Size 2 excluded for strict (there strict = 1 −- Acc_MV, a tautology). Measure Raw | best | best + mean strict diversity −-0.650 −-0.799 +0.339 † disagreement −-0.667 −-0.742 +0.292 † Jaccard (pairwise) +0.537 +0.530 −-0.226 focal (TOPLA) −-0.537 −-0.532 +0.049 double-fault (co-failure) −-0.449 −-0.815 −-0.432 Size-4 associations (Appendix Table C1) show the same qualitative pattern: raw signs reverse or attenuate under best+mean control, and double-fault remains negative (−-0.380). Figure 3. Strict diversity against one minus mean member accuracy, size 3, on MMLU-Pro (left) and TruthfulQA (right), OLS fits, shared axes. Figure 4. Capability entanglement across the five audited measures (size 3). Points show the descriptive in-sample fraction of each measure’s rank variance accounted for by an OLS projection on ranked best and mean member accuracy, the same control space used for partial Spearman rather than a held-out predictive R2R^2. Entanglement is strongest for strict diversity, disagreement, and double-fault, and weaker for Jaccard and focal diversity. Figure 5. Capability control changes apparent diversity–gain associations (size 3; values from Table 3): raw vs. partial Spearman controlling best and best+mean. Only double-fault stays negative throughout. 5.4 A residual pairwise co-failure association remains The contingency-table statistics do not provide several independent linear signals. From the exact identity in Section 3.1, 1−Acc¯=DoubleFault+12Disagreement1- Acc=DoubleFault+ 12Disagreement, any linear residualization of the raw (unranked) measures that includes mean member accuracy forces DoubleFaultres=−12Disagreementres,DoubleFault_res=- 12Disagreement_res, exactly, hence Pearson r=−1.000r=-1.000 with slope −1/2-1/2 (max numerical error ∼3×10−16 3× 10^-16; Appendix B.1). This raw-space one-dimensionality is algebraically inevitable; it is not an empirical discovery. Table 3 and Figure 6 instead use marginal ranks followed by linear residualization. Rank transformation does not preserve the raw linear identity, so the partial-Spearman magnitudes for disagreement and double-fault need not be symmetric (+0.292 vs −-0.432). What is empirical is that rank-residual pairwise co-failure remains negatively associated with majority-vote gain: more shared error, lower gain (Figure 6). A raw-space joint regression of gain on disagreement, double-fault, mean is rank-deficient by construction. Figure 6. Residual pairwise co-failure association (size 3, MMLU-Pro): rank residuals of gain vs. double-fault after best+mean control, with decile-binned means (TruthfulQA: ρ=−0.553ρ=-0.553, Appendix Figure C2). 5.5 Robustness and boundary conditions The residual pairwise co-failure association is directionally robust but bounded: more shared error is associated with less gain (forest plot in Appendix Figure C1; full battery in Appendix Table C2: nonlinear and matched controls, model-level resampling intervals, leave-one-model-out, per-subset denominators, 451-item slice, TruthfulQA). Robust in direction. Negative under all six linear control specifications, nonlinear and matched/stratified controls, plurality voting, per-subset denominators, a less-filtered slice, TruthfulQA, and removal of the upper accuracy cutoff. Model-level resampling intervals exclude zero at size 3 and on TruthfulQA; at size 4 the MMLU-Pro interval crosses zero while leave-one-model-out estimates remain uniformly negative. Conditional in magnitude. Point estimates span approximately −-0.18 to −-0.57 across slice, roster, benchmark, and threshold configurations. Under a non-circular difficulty split (difficulty from 15 held-out models; analysis on the other 15), the association is strong on easy/medium items (≈−-0.75 to −-0.86) but much weaker on hard items (−-0.18 to −-0.22), where gain variation is limited (Appendix Figure C3); the medium band is small (36–42 items) due to ties in definer accuracy. Magnitude is pool-sensitive (near zero in a restricted Chinese-origin pool), and held-out prediction of rare gain>0 events remains weak (size-3 pairwise co-failure AUC 0.597). This is a diagnosis of shortfall, not a recipe for winning ensembles. The pairwise co-failure component carries the association; a strict all-members-wrong rate is much weaker after control, distinguishing our residual axis from the all-member co-failure ceiling of concurrent work (Chen, 2026). 5.6 Is the parsed slice driving the results? A natural objection is that diversity matters most on hard items that a common-parse filter tends to drop. Three cache-only checks argue otherwise. (i) Per-subset denominators (each subset on its own members’ parsed intersection; median 461/452 items at sizes 3/4) leave results essentially unchanged: strict↔ (1−-mean) ρ=+0.991ρ=+0.991; double-fault −-0.418/−-0.358; size-3 gain>0 = 8.05%. (i) A less-filtered 16-model / 451-item slice removes most slice bias and replicates all three core directions (strict↔ (1−-mean) ≈ +0.996; controlled strict vanishes; double-fault −-0.236/−-0.195). (i) Dropped items are harder and show higher disagreement, and 63.2% fall in the hard band, where the controlled pairwise co-failure association is substantially weaker (approximately −-0.18 to −-0.22). Thus, the common-parse filter disproportionately removes items from a regime in which this particular signal is less informative. 6 Discussion Raw correlations that seem to say “diversity hurts and overlap helps” are strongly shaped by capability: the audited measures are entangled with member accuracy and, for strict diversity, nearly collinear with it. This is a measurement-level finding, convergent with competence filtering (Cohen et al., 2026), quality matching / all-member co-failure ceilings (Chen, 2026), and accuracy-adjusted dependence (Ali, 2026). Intervention work such as D-FUSEr shows that reshaping error distributions can still improve majority vote (Baek et al., 2026); our narrower claim is that observational diversity statistics over heterogeneous pools are substantially capability-entangled, especially for the linearly coupled contingency-table trio. Progress may therefore require error-shaping interventions or capability-controlled measurement rather than raw diversity scores alone. The negative direction of the residual co-failure association is itself unsurprising under majority voting: errors shared by multiple members directly reduce the cases a vote can recover. Our contribution is therefore not the sign alone, but the measurement diagnosis that, after capability adjustment, this pairwise co-failure direction is the only contingency-table component that remains directionally stable, while several commonly interpreted diversity signals do not provide separable information. These results should not be read as evidence that error diversity is unimportant for aggregation; rather, the audited observational statistics do not cleanly isolate ensemble-relevant error structure from member capability. Practical recommendations. (i) Evaluate against the strongest member; (i) control for capability level and spread, including nonlinear specifications, before crediting diversity; (i) treat contingency-table measures as algebraically coupled; (iv) validate selection on held-out items (Section 5.1). 7 Conclusion Across 31,900 MMLU-Pro subsets and a 29-model TruthfulQA reproduction, the audited measures are substantially but unevenly entangled with capability, most severely for the three linearly coupled statistics: strict diversity, disagreement, and double-fault (84–99% of rank variance explained). Strict diversity is nearly collinear with one minus mean accuracy, whereas Jaccard and focal diversity retain substantially more residual variation. Controlled measure–gain associations are specification-dependent, with pairwise co-failure providing the most directionally stable residual association: more shared error is associated with lower majority-vote gain, although the magnitude is modest and configuration-dependent. 8 Limitations 1. Slice conditionality (mitigated, not eliminated). Retained items are easier and show higher agreement than dropped ones (Section 4). Section 5.6 shows the direction of the core capability-entanglement and co-failure results is unchanged under per-subset denominators and a less-filtered 451-item slice; however, effect magnitudes attenuate on less-filtered configurations and should not be extrapolated. Separately, the 356-item set introduces item-sampling noise at the subset level, but this noise attenuates rather than inflates the reported association (split-half correction ≈−-0.53 vs. reported −-0.43), and subsampling curves are stable from 150 items onward (Section 5.5). 2. Modest, configuration-dependent effect. Point estimates span approximately −-0.18 to −-0.57 across slice, roster, benchmark, and threshold configurations; the MMLU size-4 resampling interval includes zero (leave-one-model-out remains uniformly negative), the association weakens sharply on hard items, and it is near zero in a restricted Chinese-origin pool. The linear-control positive residuals of strict/disagreement are roster-dependent and should not be interpreted as a robust reversal. 3. Thin residual. Because strict ≈ 1 −- mean, only 1.1%/1.5% of its rank variance survives best+mean control on MMLU-Pro/TruthfulQA; estimates on this residual are inherently less stable (mitigated via nonlinear controls and model-level resampling intervals, but a structural limit). 4. Voting-rule and baseline conditionality. Headline rates use a conservative correct-count vote (even-size ties incorrect) and in-sample best selection; we report plurality-rule and held-out alternatives (Section 5.1), under which the qualitative conclusions are unchanged. The 1.27% all-subset rate mixes structural size-2 zeros and size-4 tie effects with the more informative size-3 rate (9.98%). 5. Correlational, overlapping, exploratory. All analyses are correlational; subsets reuse 30 models (model-level resampling robustness analysis throughout; Appendix B); the analysis was exploratory and iteratively refined. The exact accuracy-band endpoints were not formally preregistered or literature-derived. The no-upper-cutoff analysis was post hoc, with its protocol fixed before execution, and establishes local robustness to one boundary model rather than universal threshold invariance. 6. Prompt-regime and parsing conditionality. All models were evaluated under a common zero-shot chain-of-thought instruction. Different prompting or reasoning regimes may alter both member capability and error dependence, so the measured diversity–capability relationships should not be assumed prompt-invariant. Initial parsing rules also differed slightly across collection batches; the common-slice and alternative-denominator analyses mitigate but do not eliminate parser-induced missingness. The retry also re-queried previously unparsed cases with progressively simplified prompts, so the final correctness matrix mixes a small number of recovered responses with the initial prompting regime. An initial-response-only sensitivity check is not reported: the pre-retry intersection of items parsed by all 30 models is only 18 items, below the 150-item stability floor used elsewhere (Section 5.5). 7. Concurrent work and scope. Concurrent preprints study quality-matched pools, all-model co-failure ceilings, and accuracy-adjusted pair dependence (Chen, 2026; Ali, 2026). Our scope is narrower and diagnostic: unweighted majority voting, multiple-choice benchmarks, and five audited measures. References Ai et al. (2026) Rui Ai, Yuqi Pan, David Simchi-Levi, Milind Tambe, and Haifeng Xu. 2026. Beyond majority voting: LLM aggregation by leveraging higher-order information. In Forty-Third International Conference on Machine Learning. Ali (2026) Junade Ali. 2026. Quantifying diversity of thought: A predictive law of weighted LLM ensemble lift. Preprint, arXiv:2607.17384. Baek et al. (2026) David Baek, Andrew Estornell, Yichi Zhang, Muhammad Faaiz Taufiq, Jean-Francois Ton, Jie Mei, and Tao Wang. 2026. D-FUSEr: Diverse failure, unified success via error-distribution shaping in LLM reasoning. In Forty-Third International Conference on Machine Learning. OpenReview: To2O1ed5cV. Brown et al. (2005) Gavin Brown, Jeremy Wyatt, Rachel Harris, and Xin Yao. 2005. Diversity creation methods: a survey and categorisation. Information Fusion, 6(1):5–20. Chen (2026) Josef Chen. 2026. When does combining language models help? a co-failure ceiling on routing, voting, and mixture-of-agents across 67 frontier models. Preprint, arXiv:2606.27288. Chen et al. (2024a) Lingjiao Chen, Matei Zaharia, and James Zou. 2024a. FrugalGPT: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research. Chen et al. (2024b) Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2024b. Universal self-consistency for large language models. In ICML 2024 Workshop on In-Context Learning. Earlier version titled “Universal Self-Consistency for Large Language Model Generation”, arXiv:2311.17311, 2023. Chen et al. (2025) Zhijun Chen, Xiaodong Lu, Jingzheng Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Ming Li, Likang Xiao, Dingqi Yang, Xiao Huang, Yikun Ban, Hailong Sun, and Philip S. Yu. 2025. Harnessing multiple large language models: A survey on LLM ensemble. Preprint, arXiv:2502.18036. Version 6, updated 22 April 2026. Cohen et al. (2026) Seffi Cohen, Nurit Cohen Inger, Niv Goldshlager, Bracha Shapira, and Lior Rokach. 2026. DFPE: A diverse fingerprint ensemble for enhancing LLM performance. In Findings of the Association for Computational Linguistics: EACL 2026, pages 5326–5336, Rabat, Morocco. Association for Computational Linguistics. Dietterich (2000) Thomas G. Dietterich. 2000. Ensemble methods in machine learning. In Multiple Classifier Systems, volume 1857 of Lecture Notes in Computer Science, pages 1–15, Berlin, Heidelberg. Springer. Du et al. (2024) Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 11733–11763. PMLR. Goel et al. (2025) Shashwat Goel, Joschka Strüber, Ilze Amanda Auzina, Karuna K Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, and Jonas Geiping. 2025. Great models think alike and this undermines AI oversight. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 19621–19678. PMLR. Jiang et al. (2023) Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. LLM-Blender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165–14178, Toronto, Canada. Association for Computational Linguistics. Kim (2026) Donghwan Kim. 2026. LLM-as-a-judge scores are unreliable optimization signals in closed-loop table recognition. Preprint, arXiv:2607.13347. Kim et al. (2025) Elliot Myunghoon Kim, Avi Garg, Kenny Peng, and Nikhil Garg. 2025. Correlated errors in large language models. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 30038–30066. PMLR. Krogh and Vedelsby (1995) Anders Krogh and Jesper Vedelsby. 1995. Neural network ensembles, cross validation, and active learning. In Advances in Neural Information Processing Systems 7, pages 231–238. MIT Press. Proceedings of the 1994 conference. Kuncheva and Whitaker (2003) Ludmila I. Kuncheva and Christopher J. Whitaker. 2003. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine Learning, 51(2):181–207. Lefort et al. (2024) Baptiste Lefort, Eric Benhamou, Jean-Jacques Ohana, Béatrice Guez, David Saltiel, and Thomas Jacquot. 2024. Examining independence in ensemble sentiment analysis: A study on the limits of large language models using the condorcet jury theorem. Preprint, arXiv:2409.00094. Li et al. (2024) Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. 2024. More agents is all you need. Transactions on Machine Learning Research. Li et al. (2026) Wenzhe Li, Yong Lin, Mengzhou Xia, and Chi Jin. 2026. Rethinking mixture-of-agents: Is mixing different large language models beneficial? Transactions on Machine Learning Research. Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics. Si et al. (2023) Chenglei Si, Weijia Shi, Chen Zhao, Luke Zettlemoyer, and Jordan Boyd-Graber. 2023. Getting MoRE out of mixture of language model reasoning experts. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8234–8249, Singapore. Association for Computational Linguistics. Tekin et al. (2024) Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang, Sihao Hu, and Ling Liu. 2024. LLM-TOPLA: Efficient LLM ensemble by maximising diversity. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11951–11966, Miami, Florida, USA. Association for Computational Linguistics. Turkmen et al. (2026) Yigit Turkmen, Baturalp Buyukates, and Melih Bastopcu. 2026. Don’t always pick the highest-performing model: An information theoretic view of LLM ensemble selection. Preprint, arXiv:2602.08003. Vardeman and Morris (2013) Stephen B. Vardeman and Max D. Morris. 2013. Majority voting by independent classifiers can increase error rates. The American Statistician, 67(2):94–96. Wang et al. (2025) Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. 2025. Mixture-of-agents enhances large language model capabilities. In International Conference on Learning Representations. Spotlight. Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations. Wang et al. (2024) Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems 37: Datasets and Benchmarks Track. Wood et al. (2023) Danny Wood, Tingting Mu, Andrew M. Webb, Henry W. J. Reeve, Mikel Luján, and Gavin Brown. 2023. A unified theory of diversity in ensemble learning. Journal of Machine Learning Research, 24(359):1–49. Zhang et al. (2006) Yi Zhang, Samuel Burer, and W. Nick Street. 2006. Ensemble pruning via semi-definite programming. Journal of Machine Learning Research, 7(48):1315–1338. Zhang et al. (2026) Yichi Zhang, Kevin Lu, Yuang Zhang, Jie Gao, Lirong Xia, and Fang-Yi Yu. 2026. Mixture of complementary agents for robust LLM ensemble. Preprint, arXiv:2605.24048. Associated ICLR 2026 submission withdrawn. Appendix A Model roster and parsing audit A.1 Model roster and full-sample filtering statistics Parse rate and accuracy use the full 500-item sample. Full-500 accuracy is the number of correct predictions divided by 500; unparsed responses therefore contribute zero to accuracy. Canonical inclusion requires parse rate ≥ 0.90 and full-500 accuracy in [0.40, 0.92]. MMLU-Pro: 30/30 routes pass the canonical filter. Table A1. MMLU-Pro model roster and full-sample filtering statistics. Alias OpenRouter model ID Parse rate Full-500 accuracy Canonical roster claude_35_haiku anthropic/claude-3.5-haiku 99.8% 63.8% Yes claude_37_sonnet anthropic/claude-3.7-sonnet 99.6% 82.4% Yes claude_haiku anthropic/claude-haiku-4.5 98.0% 79.0% Yes claude_sonnet46 anthropic/claude-sonnet-4.6 95.2% 83.0% Yes ernie45 baidu/ernie-4.5-21b-a3b 97.0% 61.8% Yes seed16 bytedance-seed/seed-1.6 94.2% 78.8% Yes deepseek_v3 deepseek/deepseek-chat-v3-0324 100.0% 79.2% Yes deepseek_r1 deepseek/deepseek-r1 100.0% 84.6% Yes gemini25_flash google/gemini-2.5-flash 91.6% 69.8% Yes gemini25_flash_lite google/gemini-2.5-flash-lite 94.8% 71.4% Yes gemini25_pro google/gemini-2.5-pro 99.8% 82.6% Yes gemma4_31b google/gemma-4-31b-it 93.6% 82.8% Yes llama31_8b meta-llama/llama-3.1-8b-instruct 100.0% 51.0% Yes llama33_70b meta-llama/llama-3.3-70b-instruct 100.0% 72.0% Yes llama4_maverick meta-llama/llama-4-maverick 94.2% 76.8% Yes llama4_scout meta-llama/llama-4-scout 95.0% 72.0% Yes minimax_m25 minimax/minimax-m2.5 96.4% 72.2% Yes mistral_large mistralai/mistral-large 91.6% 74.0% Yes mistral_small mistralai/mistral-small-3.1-24b-instruct 97.2% 64.0% Yes kimi_k2 moonshotai/kimi-k2-thinking 94.8% 80.2% Yes gpt41 openai/gpt-4.1 96.4% 79.4% Yes gpt41_nano openai/gpt-4.1-nano 92.6% 60.8% Yes gpt4o openai/gpt-4o 99.0% 74.4% Yes gpt4o_mini openai/gpt-4o-mini 98.0% 49.4% Yes qwen35_9b qwen/qwen3.5-9b 94.4% 58.6% Yes qwen36_plus qwen/qwen3.6-plus 97.2% 85.4% Yes grok4_fast x-ai/grok-4-fast 100.0% 83.8% Yes grok41_fast x-ai/grok-4.1-fast 99.6% 85.0% Yes glm47 z-ai/glm-4.7 97.2% 70.6% Yes glm51 z-ai/glm-5.1 98.8% 75.0% Yes TruthfulQA: 29/30 routes pass the canonical filter. Table A2. TruthfulQA model roster and full-sample filtering statistics. Alias OpenRouter model ID Parse rate Full-500 accuracy Canonical roster claude_35_haiku anthropic/claude-3.5-haiku 100.0% 79.4% Yes claude_37_sonnet anthropic/claude-3.7-sonnet 99.8% 89.4% Yes claude_haiku45 anthropic/claude-haiku-4-5 99.8% 88.0% Yes claude_sonnet46 anthropic/claude-sonnet-4.6 100.0% 90.0% Yes ernie45 baidu/ernie-4.5-21b-a3b 98.2% 62.4% Yes seed16 bytedance-seed/seed-1.6 97.2% 87.4% Yes deepseek_v3 deepseek/deepseek-chat-v3-0324 99.8% 76.6% Yes deepseek_r1 deepseek/deepseek-r1 96.4% 84.2% Yes gemini25_flash google/gemini-2.5-flash 100.0% 84.2% Yes gemini25_flash_lite google/gemini-2.5-flash-lite 99.8% 79.2% Yes gemini25_pro google/gemini-2.5-pro 100.0% 85.4% Yes gemma4_31b google/gemma-4-31b-it 99.8% 89.2% Yes llama31_8b meta-llama/llama-3.1-8b-instruct 97.8% 54.0% Yes llama33_70b meta-llama/llama-3.3-70b-instruct 99.0% 76.8% Yes llama4_maverick meta-llama/llama-4-maverick 99.8% 77.2% Yes llama4_scout meta-llama/llama-4-scout 99.4% 72.6% Yes minimax_m25 minimax/minimax-m2.5 96.8% 77.0% Yes mistral_large mistralai/mistral-large 100.0% 81.8% Yes mistral_small mistralai/mistral-small-3.1-24b-instruct 93.6% 67.8% Yes kimi_k2 moonshotai/kimi-k2-thinking 90.4% 81.6% Yes gpt41 openai/gpt-4.1 99.6% 87.4% Yes gpt41_nano openai/gpt-4.1-nano 99.6% 67.2% Yes gpt4o openai/gpt-4o 99.0% 82.2% Yes gpt4o_mini openai/gpt-4o-mini 99.0% 78.2% Yes qwen35_9b qwen/qwen3.5-9b 99.8% 77.8% Yes qwen36_plus qwen/qwen3.6-plus 99.8% 92.2% No (accuracy > 0.92) grok4_fast x-ai/grok-4-fast 100.0% 88.2% Yes grok41_fast x-ai/grok-4.1-fast 100.0% 90.4% Yes glm47 z-ai/glm-4.7 99.8% 90.8% Yes glm51 z-ai/glm-5.1 92.2% 86.8% Yes Route-provenance note. The benchmark files each contain 30 aliases, but the recorded Haiku alias/slug differs: MMLU-Pro uses claude_haiku → anthropic/claude-haiku-4.5, whereas TruthfulQA uses claude_haiku45 → anthropic/claude-haiku-4-5. The table preserves the recorded benchmark-specific identifiers and does not silently assume that differently written routes are identical. On TruthfulQA, qwen36_plus passes parsing but is excluded only because its full-500 accuracy is 92.2%, above the historical 92% upper bound. A.2 Prompting, inference, and parsing protocol Benchmarks and sampling. MMLU-Pro is loaded from TIGER-Lab/MMLU-Pro (test split; 12,032 items). We sample 500 row indices uniformly without replacement using Python’s random.sample after setting seed 42; no category stratification is used. TruthfulQA is loaded from truthfulqa/truthful_qa (multiple_choice configuration, validation split), from which 500 items are sampled by the same procedure and seed. We use the mc1_targets choices and the unique choice labeled 1 as the TruthfulQA answer. The released sampled-question files store the original dataset row index, local question ID, question text, options, and answer label; all 500 TruthfulQA mappings were verified against the Hugging Face validation-row order. Initial prompt templates. Both benchmarks use zero-shot chain-of-thought multiple-choice prompts with no demonstrations. No system prompt was used: every request contained one user message only. In the templates below, question is replaced by the benchmark question and lettered_options by one option per line in A. option text, B. option text, … format. Individual benchmark questions are not part of the template. MMLU-Pro: Answer the following multiple choice question. Think step by step, then provide your final answer. At the very end, write your answer in this exact format: ANSWER: X (where X is the letter of the correct option) Question: question Options: lettered_options Let’s think step by step: TruthfulQA: Answer the following question truthfully. Think step by step, then provide your final answer. At the very end, write your answer in this exact format: ANSWER: X (where X is the letter of the correct option) Question: question Options: lettered_options Let’s think step by step: Table A3. Inference and parsing settings. Setting Initial evaluation Unparsed-response retry message roles one user message; no system message one user message; no system message decoding temperature 0.0; top_p, stop sequences, penalties, and request seed unset (provider defaults) same max output tokens MMLU-Pro: 1,024, except 8,192 for five designated reasoning routes listed below; TruthfulQA: 1,024 2,048, except 4,096 for qwen35_9b request timeout 90 s for the original eight-route MMLU-Pro batch; SDK/provider default otherwise SDK/provider default API-error attempts up to 3 per prompt; 3/6-s backoff in the original MMLU-Pro batch and 4/8-s otherwise up to 3 per prompt; 4/8-s backoff parser source response content content, then provider reasoning field as fallback The five MMLU-Pro routes assigned 8,192 initial output tokens were deepseek_r1, kimi_k2, gemini25_pro, gemini25_flash, and gemini25_flash_lite. Initial parser. Matching ignores case. Labels outside the available option range are rejected. Patterns run in order, using the last match from each. MMLU-Pro, original eight-route batch. In order: ANSWER: X, answer: X, The answer is X, Final answer: X, and a response consisting only of one label. MMLU-Pro, later scheduler. First remove complete <think>...</think> blocks; then check ANSWER: X, answer: X, boxed labels, bold Answer: X, bold labels, The [correct] answer is X, the Chinese prefixes encoded by U+9009/U+62E9 (“selection”), U+7B54/U+6848 (“answer”), or U+6B63/U+786E/U+7B54/U+6848 (“correct answer”), a line beginning with X. or X), and a response consisting only of one label. TruthfulQA. First remove complete <think>...</think> blocks; then check ANSWER: X, answer: X, boxed labels, bold labels, The [correct] answer is X, and a response consisting only of one label. If no valid label is found, the record is stored with no prediction and parse_success=False; it is not scored as a parsed prediction. Exact retry prompts and sequence. The retry pass uses the following three benchmark-agnostic templates, in order, for both benchmarks. Options in Retry 1–2 use A. option text; Retry 3 uses (A) option text. Retry 1: Answer the following multiple choice question. Think step by step, then provide your final answer. At the very end, write your answer in this exact format: ANSWER: X (where X is the letter of the correct option) Question: question Options: lettered_options Let’s think step by step: Retry 2: Question: question lettered_options Which option is correct? Reply with just the letter. Your answer (one letter only): Retry 3: Select the correct answer. question parenthesized_options Answer with only one letter like: A For each retry prompt, we first parse response content using this ordered cascade: ANSWER: X, answer: X, Answer: X, bold X, The answer is X, correct answer is X, a standalone X, and a final valid option letter. If content parsing fails, the same parser is applied to the provider reasoning field. A successful parse stops the sequence immediately; otherwise the next prompt is tried. After all three prompts fail, the entry remains unparsed. Retry scope and preservation guarantee. Retries were applied only to previously unparsed responses (parse_success=False) from routes below the parse-rate threshold. Already parsed predictions were never regenerated, re-queried, or replaced. The merge changes only a targeted unparsed record after a successful retry and carries every previously parsed record forward unchanged. The released retry_parsing.py implements this target selection, three-prompt schedule, parser fallback, and merge logic and provides a no-API --dry-run audit. Voting rules. The canonical correct-count rule marks an item correct only when more than half the subset members are correct; exact even-size splits are incorrect. The answer-level plurality robustness check requires the correct option to be the unique most-voted answer; any tie for the highest answer count is marked incorrect. These rules are identical for subset sizes 2–3 and can differ at size 4. Appendix B Model-level resampling procedure Subsets heavily overlap because they reuse the same 30 models. We therefore avoid subset-level p-values and use model-level resampling for robustness intervals on partial Spearman associations (B = 1,000; NumPy seed 0, using one RNG stream across sizes 3 and 4 within each benchmark). Pseudocode: 1. Draw a multiset of model indices of size M uniformly with replacement. 2. Deduplicate to a set U of unique drawn models. If |U|<8|U|<8, discard the replicate. 3. Retain only subsets whose members are all in U (subsets that use any non-drawn model are dropped). If fewer than 50 subsets remain, discard the replicate. 4. On the retained subsets, recompute the partial Spearman of double-fault with gain given best, mean (rank-space residualization, same as the main tables). 5. The 95% interval is the 2.5th–97.5th percentile of valid replicate statistics. Because step 2 deduplicates draws, duplicate multiplicities do not reweight the statistic; the procedure is a random-subpool perturbation of the model roster rather than a conventional cluster bootstrap, and we accordingly report its percentile intervals as model-level resampling robustness intervals, not bootstrap confidence intervals. Leave-one-model-out estimates drop all subsets containing a held-out model and recompute the same partial association. B.1 Residual identity verification Numerically, using the raw (unranked) measures, max|double-faultres+12disagreeres| |double -fault_res+ 12disagree_res| is at machine precision (∼3×10−16 3× 10^-16) after linearly removing best and mean member accuracy at size 3, matching the algebraic prediction of Section 3.1: the residuals are perfectly collinear (Pearson r=−1.000r=-1.000) with slope −1/2-1/2. Table 3 and Figure 6 use rank-space residualization instead; the empirical content of Section 5.4 is the resulting association with majority-vote gain, not the raw-space collinearity itself. Appendix C Full robustness battery This appendix collects the size-4 counterpart to main-text Table 3 and the full robustness battery for the controlled double-fault association with majority-vote gain. Figure C1 summarizes point estimates (and, where available, model-level resampling intervals and leave-one-model-out ranges) across control, denominator, slice, and benchmark specifications at sizes 3 and 4. Figure C2 is the TruthfulQA partial-residual counterpart to main-text Figure 6. Figure C3 shows the non-circular difficulty boundary condition without aggregating its two model splits into an uncertainty interval. Figure C4 decomposes the gain>0 rate by ensemble size and voting or selection rule. Table C1 is the size-4 analogue of main-text Table 3: Spearman associations of the five measures with majority-vote gain under the Raw → ||best → ||best+mean control ladder (n=27,405n=27,405). Table C2 lists the numerical values behind Figure C1 and the remaining robustness checks referenced in Sections 5.4–5.6 (alternative controls, resampling intervals, leave-one-model-out, denominators, slice, and TruthfulQA). Figure C1. Controlled double-fault association across specifications (size 3 circles, size 4 triangles); bars show the MMLU-Pro model-resampling interval and LOMO range. Figure C2. TruthfulQA partial-residual plot (size 3; ρ=−0.553ρ=-0.553), counterpart to Figure 6. Table C1. Size-4 associations (counterpart to main-text Table 3), ensemble size 4 (n = 27,405). Measure Raw | best | best + mean strict diversity −-0.804 −-0.874 +0.333 disagreement −-0.803 −-0.830 +0.261 Jaccard (pairwise) +0.625 +0.636 −-0.110 focal (TOPLA) −-0.658 −-0.664 −-0.138 double-fault (co-failure) −-0.635 −-0.862 −-0.380 Table C2. Full robustness of the co-failure (double-fault) association under best+mean control unless noted. Estimate size 3 size 4 partial Spearman (best) −-0.815 −-0.862 partial Spearman (best+mean) baseline −-0.432 −-0.380 control (best, min) −-0.616 −-0.728 control (best, median) −-0.587 −-0.461 control (best, std) −-0.628 −-0.741 full accuracy vector −-0.338 −-0.234 nonlinear (decile dummies, best+mean) −-0.474 −-0.445 matched / stratified (tol 0.01) −-0.510 −-0.394 MMLU-Pro model-level resampling 95% interval (B=1,000) [−-0.539, −-0.193] [−-0.523, +0.054] leave-one-model-out range (30 models) [−-0.454, −-0.388] [−-0.431, −-0.269] answer-plurality MV (best+mean) identical to Table 3 −-0.456 per-subset denominators (Section 5.6) −-0.418 −-0.358 high-parse 16-model / 451-item slice −-0.236 −-0.195 attenuation-corrected (approx.) ≈−-0.53 — TruthfulQA (best+mean) −-0.553 −-0.438 TruthfulQA model-level resampling 95% interval (B=1,000) [−-0.656, −-0.358] [−-0.612, −-0.095] TruthfulQA, no upper cutoff −-0.570 −-0.464 TruthfulQA, no upper cutoff resampling 95% interval [−-0.655, −-0.376] [−-0.633, −-0.125] strict under nonlinear control +0.092 +0.075 At size 4 under answer-plurality voting, the Raw / |best / |best+mean double-fault associations are −-0.377 / −-0.781 / −-0.456; sizes 2–3 plurality coincides with the correct-count rule and therefore matches Table 3. C.1 Additional robustness definitions Table C3. Protocols and numerical results for additional robustness checks cited in Sections 5.5 and 8. Check Protocol Size-3 result non-circular difficulty Randomly split 30 models into 15 difficulty definers and 15 evaluators; form item terciles from definer accuracy and analyze evaluator-only size-3 subsets; seeds 0–1 hard −-0.219/−-0.182; medium −-0.855/−-0.790; easy −-0.851/−-0.747 Chinese-origin restricted pool Restrict subsets to nine Chinese-origin routes: qwen36_plus, qwen35_9b, glm51, glm47, kimi_k2, minimax_m25, seed16, ernie45, deepseek_r1 double-fault −-0.038; strict +0.001; n=84 held-out gain prediction Split items A/B; compute the oriented score −double-fault-double -fault on A and predict 1[gB>0]1[g_B>0] on B; 20 seeds (2000–2019) AUC 0.597 ± 0.031 all-member failure Replace pairwise double-fault with the fraction of items on which every subset member is wrong; partial Spearman given best+mean all-fail −-0.101 versus pairwise double-fault −-0.432 split-half attenuation Estimate gain and double-fault reliability from 20 random item halves; apply Spearman–Brown full-length correction and the classical attenuation formula reliability(gain) = 0.757, reliability(double-fault) = 0.860; corrected ≈−-0.535 versus observed −-0.432 item subsampling Sample 100, 150, 200, 250, and 300 of 356 items without replacement; 20 seeds per size; recompute gain>0 and double-fault partial Spearman ρ: −-0.391± 0.077, −-0.434± 0.054, −-0.412± 0.052, −-0.429± 0.043, −-0.448± 0.041 Figure C3. Difficulty-conditioned controlled pairwise co-failure association on MMLU-Pro, size 3. For each of the two reported split seeds (0 and 1), 15 models define item-difficulty terciles and the disjoint 15 form 455 evaluated subsets; points are seed-specific partial Spearman coefficients after rank-space best+mean control. Ties yield unequal easy/medium/hard bands: n = 190/36/130 items for seed 0 and 194/42/120 for seed 1. Lines connect bands within a split only; with two splits, no uncertainty interval is estimated. Figure C4. Decomposition of the gain>0 rate by ensemble size and rule. The pooled 1.27% mixes structural size-2 zeros and size-4 tie effects with the size-3 rate. Held-out best selection raises the observed win rate at sizes 3–4, while size 2 remains structurally zero. Control and slice estimators. The six linear control specifications in Table C2 are (best), (best, mean), (best, minimum), (best, median), (best, standard deviation), and the full sorted member-accuracy vector; all use the rank-space partial-Spearman procedure defined in Section 5. The nonlinear specification replaces continuous rank controls for best and mean with decile indicators. The matched estimator exactly matches the discrete best-member accuracy, bins mean accuracy at width 0.01, requires at least eight subsets per stratum, computes within-stratum Spearman correlations, and averages them weighted by stratum size. Matching on best+mean retains 3,551/4,060 size-3 subsets across 134 strata (87.5%) and 27,137/27,405 size-4 subsets across 230 strata (99.0%). Per-subset-denominator results recompute every subset on the intersection parsed by its own members. The 451-item slice greedily removes low-parse routes until a common set of at least 450 items is available, leaving 16 models and 451 items. TruthfulQA repeats the same filtering, subset construction, and estimator on its 338-item common denominator. The no-upper-cutoff intervals come from the separately initialized deterministic RNGs in the pre-fixed threshold-sensitivity protocol. Held-out best selection. For each of 20 seeds (0–19), the 356 common items are randomly permuted and split into halves of 178 items each (A/B; remainder none). Within each subset, the comparison member is the model with maximum accuracy on split A; exact accuracy ties are broken by the lowest model index in the filtered roster order (numpy.argmax). Majority-vote accuracy and the selected member’s accuracy are then evaluated exclusively on split B. The reported 18.71% (± 3.70) is the mean (± ) over seeds of the fraction of size-3 subsets with positive held-out gain. The difficulty result above is deliberately non-circular: the models defining item difficulty never enter the evaluated subsets. Because ties in definer accuracy make the three item bands unequal, we report the two seed-specific coefficients rather than pooling them. The held-out AUC is a predictive diagnostic, not a capability-controlled effect estimate. The attenuation-corrected value is also approximate: the classical correction applies directly to a raw correlation, whereas applying it to the partial coefficient additionally treats the best and mean controls as measured with negligible error. We therefore use it only to assess whether item noise plausibly attenuates, rather than creates, the observed association. Oracle-gain capture ratio. For completeness, define this descriptive ratio as g(S)/goracle(S)g(S)/g_oracle(S) on subsets with positive oracle gain. Its MMLU-Pro mean across subset-level ratios is −-131.7%, while the pooled ratio ∑Sg(S)/∑Sgoracle(S) _Sg(S)/ _Sg_oracle(S) is −-120.6%. These negative values reflect typical underperformance relative to the best member, but the per-subset ratio can be unstable when oracle gain is small; we therefore do not use it as a headline result.