Paper deep dive

Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

Michael Hardy, Ruhana Azam, Anka Reuel, Mykel Kochenderfer, Sanmi Koyejo

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 182

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 10/2/2026, 9:25:11 AM

Summary

This paper introduces a Bayesian variance-decomposition framework based on Generalizability Theory to assess the reliability of agent leaderboards. Analyzing 22 benchmarks from the Holistic Agent Leaderboard and Harbor Index, the study finds that reliability is claim-dependent: fixed model-scaffold systems are ranked reliably (0.935-0.994), while underlying model reliability is much lower (0.148-0.841). The authors demonstrate that scaffold choice significantly impacts rankings and that adding more tasks within the same benchmark yields diminishing returns for model ranking reliability. Conversely, pooling diverse benchmarks improves cross-task ranking reliability at a lower cost.

Entities (7)

Harbor-Index · dataset · 98%Holistic Agent Leaderboard · dataset · 98%Generalizability Theory · methodology · 97%AssistantBench · benchmark · 96%SWE-BENCH · benchmark · 96%Scaffold · concept · 95%Bayesian variance-decomposition · methodology · 94%

Relation Signals (5)

Holistic Agent Leaderboard → contains → AssistantBench

confidence 96% · analyze agent rollouts from all nine benchmarks distributed via HAL... AssistantBench

Generalizability Theory → usedfor → Bayesian variance-decomposition

confidence 95% · Using Generalizability theory, we develop a Bayesian variance-decomposition framework

Pooling diverse benchmarks → improves → Cross-task Ranking Reliability

confidence 94% · Pooling diverse benchmarks can improve cross-task rankings at lower cost... raises projected reliability from 0.44 to 0.75

Scaffold → affects → Model Ranking Reliability

confidence 93% · Scaffold choice can change conclusions... inter-scaffold reliability measures whether scaffolds preserve model rankings

Harbor-Index → contains → SWE-BENCH

confidence 90% · apply it to 22 benchmarks from the Holistic Agent Leaderboard and Harbor Index... SWE-bench

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Agent evaluations are increasingly used to compare LLMs and inform deployment decisions, yet ranks can reflect not only the model but also the effects of the evaluation conditions such as the scaffolds or tasks. This makes reliability claim-dependent: an evaluation that reliably ranks deployed systems may not reliably rank underlying models. We ask which conclusions current agent evaluations reliably support and what additional evaluation would improve them. We develop a Bayesian variance-decomposition framework for sparse, imbalanced agent leaderboards and apply it to 22 benchmarks from the Holistic Agent Leaderboard and Harbor Index. The framework separates signal, performance differences relevant to the intended claim, from noise, irrelevant variation that can still change rankings. We find: (1) Reliability depends on the measurement goal. Fixed model-scaffold systems are ranked reliably (0.935-0.994), while underlying-model reliability is substantially lower (0.148-0.841). (2) Scaffold choice can change conclusions. Inter-scaffold reliability measures whether scaffolds preserve model rankings, showing that scaffold effects vary substantially across evaluations. (3) More tasks cannot resolve all uncertainty. Even infinitely many similarly constructed tasks improve model-ranking reliability of a benchmark by at most 0.097 when uncertainty is dominated by limited scaffold coverage. (4) Pooling diverse benchmarks can improve cross-task rankings at lower cost. For rankings across diverse agentic tasks, pooling benchmarks raises projected reliability from 0.44 to 0.75 at the same task budget and can reduce projected cost by up to 83\%. Evaluation design should follow the intended claim: identify what a score or ranking should mean, diagnose what limits its reliability, and spend evaluation budget on the sources of uncertainty that matter.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

178,483 characters extracted from source content.

Expand or collapse full text

Under review as a conference paper AGENT EVALUATION RELIABILITY: MORE TASKS WON’T (ALWAYS) FIX AN AGENT LEADERBOARD Michael Hardy ♢,∗ Ruhana Azam ♣ Anka Reuel ♢,† Mykel Kochenderfer ♢,† Sanmi Koyejo ♢,† ♢ Stanford University, ∗ hardym[α]stanford·edu ♣ University of Illinois Urbana-Champaign † Senior Author ABSTRACT Agent evaluations are increasingly used to compare models, assess capabilities, and inform deployment decisions, yet observed scores can reflect not only the model but also the effects of the evaluation conditions such as the scaffolds or tasks. This makes reliability claim-dependent: an evaluation that reliably ranks deployed systems may not reliably rank underlying models or produce reliable absolute scores. We ask which conclusions current agent evaluations reliably support and what additional evaluation would actually improve them. Using Generalizability theory, we develop a Bayesian variance-decomposition framework for sparse, imbalanced agent leaderboards and apply it to 22 benchmarks from the Holistic Agent Leaderboard and Harbor Index. The framework separates signal, performance differences relevant to the intended claim, from noise, irrelevant variation that can still change scores or rankings. We find four practical results: (1) Reliability depends on the measurement goal. Fixed model–scaffold systems are ranked reliably (Eρ 2 = 0.935–0.994), while underlying-model reliability is substantially lower (0.148–0.841). (2) Scaffold choice can change evaluation conclusions. We introduce inter-scaffold reliability, measuring whether scaffolds preserve model rankings, and show that scaffold effects vary substantially across evaluations. (3) More tasks cannot resolve all uncertainty. Even infinitely many similarly constructed tasks improve model-ranking reliability of the dataset by at most0.097when uncertainty is dominated by limited scaffold coverage. (4) Pooling diverse benchmarks can improve cross-task rankings at lower cost. For rankings across diverse agentic tasks, pooling benchmarks raises projected reliability from 0.44to0.75at the same task budget and can reduce projected cost by up to83%. Evaluation design should therefore follow the intended claim: practitioners should identify what a score or ranking should mean, diagnose what limits its reliability, and spend evaluation budget on the sources of uncertainty that matter. 1 1INTRODUCTION Agentic AI systems interact with software, tools, users, and environments to accomplish multi-step goals. Agent evaluations such as SWE-bench (Jimenez et al., 2023) andτ-bench (Yao et al., 2024) are increasingly used to test and rank such systems in technical reports, system cards, and policy discussions (Anthropic, 2026; OpenAI, 2026). Yet, a score on such evaluations is only useful if we know what conclusions it can reliably support: Does an observed ranking reflect model performance differences that persist beyond the particular evaluation setup? Would the same score or ranking hold under a different scaffold, a different set of tasks, or a broader range of agentic settings? These questions are particularly important for agent evaluations because performance depends on more than the underlying model. A scaffold 2 manages the state, exposes tools, and translates model outputs into actions; together, the model 3 and scaffold form the system. This creates an 1 Code and data: https://github.com/hardy-education/scaffold_eval 2 In the current literature, the software infrastructure and post-training methods surrounding the LLM allowing it to act is synonymous with “scaffold”, “harness”, and even “agent”. 3 Throughout, “model” denotes a base large language model paired with a reasoning-effort configuration; for example, gpt-5-high and gpt-5-minimal are treated as distinct, following our primary dataset. 1 arXiv:2610.00651v1 [cs.AI] 30 Sep 2026 Under review as a conference paper immediate measurement choice: are we trying to evaluate the complete system as deployed, or isolate performance differences attributable to the underlying model? The answer changes what should count as signal and what should count as evaluation error. Reliability also depends on the conclusion we want to draw. A practitioner may care whether a model ranking remains stable, whether an absolute score would remain similar, or whether performance generalizes across different agentic tasks. These are different measurement goals and need not have the same reliability. Existing agent reliability work primarily studies whether a fixed agent behaves consistently across repeated runs or prompt perturbations of the same task (Rabanser et al., 2026; Razavi et al., 2025; Yao et al., 2024). We instead ask: what claims do current agent evaluations reliably support, and what additional evaluation would make those claims more reliable? We use Generalizability Theory (G-theory) (Cronbach et al., 1972; Brennan, 2001) to separate variation associated with models, tasks, scaffolds, benchmarks, and their interactions. To word it differently, our framework separates signal–performance differences that persist across the conditions an evaluation is meant to generalize over–from noise–variation that is irrelevant to the intended claim but can still change the resulting score or ranking; reliability increases when signal dominates this noise. G-theory lets us identify not only how reliable an evaluation is for a particular claim, but also what limits that reliability and whether adding more tasks, scaffolds, or benchmark coverage would help. We develop a Bayesian variance-decomposition approach for the sparse and imbalanced designs common in agent leaderboards and apply it to30,859rollouts from 9 benchmarks in the Holistic Agent Leaderboard (HAL) (Kapoor et al., 2025) and 13 the Harbor Index (Shi et al., 2026). Our results yield four practical findings: •Reliability depends on the measurement goal. Fixed model–scaffold systems are ranked reliablyEρ 2 MA ∈ [0.94,0.99], while rank reliability for the models is substantially lowerEρ 2 M ∈ [0.148, 0.841]. • Changing the scaffold can change evaluation conclusions. We introduce inter-scaffold reliabil- ity, which measures whether different scaffolds produce similar model rankings when evaluating the same models on the same tasks. Its posterior medians range from0.151to0.852across benchmarks. We further show that scaffold choice can change which tasks a model solves even without improving its overall benchmark score. •Adding more tasks does not always make an evaluation more reliable. Under the observed scaffold coverage, even infinitely many similarly constructed tasks would improve model-ranking reliability by at most approximately 0.10. •When the goal is to rank models across diverse agentic tasks, pooling diverse benchmarks can improve reliability at lower cost. At the same task budget, projected ranking reliability rises from approximately0.44within one benchmark to0.75across the nine-benchmark battery, while reliability-aware allocation can reduce projected evaluation cost by up to 83%. The broader lesson is that more evaluation is not automatically better evaluation. Practitioners should first specify what they want an evaluation result to mean, then identify which sources of uncertainty limit that claim, and allocate evaluation effort accordingly. 2RELATED WORK AND BACKGROUND Agent evaluation. Agent benchmarks span software engineering (Jimenez et al., 2023), web navigation (Zhou et al., 2023; He et al., 2024), general assistance (Mialon et al., 2023), and customer service (Yao et al., 2024). Existing reliability studies primarily test whether a fixed agent behaves consistently across repeated runs, prompt perturbations, tool configurations, or environmental failures (Rabanser et al., 2026; Razavi et al., 2025; Wang et al., 2026; Kumar & Mishra, 2025). These analyses assess robustness of a particular system. We instead study the reliability of the comparative inference: whether the reported ordering of models or systems would persist under new tasks, scaffolds, or benchmarks. This distinction is consequential because a system can be repeatable within one harness while its rank is highly contingent on that harness. Generalizability theory separates signal from conditional advantage. Generalizability theory (G-theory) treats evaluation conditions as measurement facets and decomposes score variation into their main effects and interactions (Cronbach et al., 1972; Shavelson et al., 1989; Brennan, 2001; 2 Under review as a conference paper Cronbach & Shavelson, 2004). Related work shows that AI benchmark conclusions depend on task sampling, evaluation conditions, and the model population (Madaan et al., 2024; Hardy & Kim, 2026; Hardy et al., 2026). The crucial first choice is the object of measurement. Model– scaffold compatibility is signal when selecting a deployable system, but error when ranking models independently of scaffolding. Reliability therefore belongs to an object, a generalization universe, and an evaluation design, not to a benchmark alone. For an objectoand designD, the relative generalizability coefficient and its signal-to-noise ratio are Eρ 2 o (D) = σ 2 o σ 2 o + σ 2 δ (D) ,SNR o (D) = σ 2 o σ 2 δ (D) = Eρ 2 o 1− Eρ 2 o .(1) Hereσ 2 o is the variance of scores averaged over the specified universe, andσ 2 δ contains variation that can change relative standing. UnderX o = Z o + δ o , with uncorrelated universe score and error, Eρ 2 o = Corr 2 (X o ,Z o ). Facet main effects cancel from comparisons only when objects share the same conditions and weights. A uniformly difficult task, for example, does not change relative latent scores under a common allocation. Object-by-facet interactions do: they represent advantages that depend on the chosen task, scaffold, or benchmark. G-theory uses these components in a Decision study (D-study) to project reliability before collecting more data. 3RELIABILITY AND GENERALIZABILITY FOR AGENT LEADERBOARDS Agent leaderboards are often sparse, imbalanced, and partially crossed: models are evaluated through different scaffolds, task counts are unequal, and many combinations are absent. In this section, we discuss several metrics to measure reliability in agent leaderboards given these common constraints. We fit a Bayesian variance decomposition to binary task outcomes, then propagate its uncertainty into reliability, design projections, and model ranks. 3.1OBSERVATIONS AND GENERALIZATION UNIVERSE Letb ∈ Bindex benchmarks,i ∈ I b tasks nested within benchmarkb,m ∈ Mmodels, and a∈Aagent scaffolds. The responsey bima ∈0, 1indicates whether modelm, operated through scaffolda, solves taskifrom benchmarkb. Our primary object is the modelm. Its universe score is the component expected to persist across new tasks, benchmarks, and scaffolds resembling those represented in the leaderboard. We also consider the model–scaffold pair(m,a), the relevant object when selecting a deployable system. We treat tasks as draws from benchmark-specific construction and grading processes; benchmarks as instruments drawn from a battery of contemporary agent evaluations; scaffolds as contemporary evaluation and orchestration systems; and models as the frontier-model population represented in the data. These universes delimit the inference. In particular, generalization to new benchmark-like tasks does not establish that a benchmark represents all real- world uses. 3.2A LATENT VARIANCE DECOMPOSITION FOR BINARY OUTCOMES We use Bayesian Bernoulli–logit mixed models:y bima ∼ Bernoulli(p bima ),logit(p bima ) = η bima . Equivalently,y bima = ⊮z bima > 0, wherez bima = η bima + ε bima andε bima ∼ Logistic(0, 1), giving the standard latent logistic residual varianceπ 2 /3. Modeling the binary responses directly avoids Gaussian approximations that are particularly misleading under floor or ceiling effects. All random effects are mean-zero Gaussian and mutually independent; for example,u (M) m ∼N (0,σ 2 M ). Variance components and reliability coefficients are defined on the common latent log-odds scale (see Appendix G.1). From an Item Response Theory (IRT) perspective, these are random-item, many-facet Rasch models (Wang & Wilson, 2005; Fox & Glas, 2001; Linacre & Wright, 2002). 3.2.1BENCHMARK-LEVEL MODEL AND SYSTEM RELIABILITY For each benchmark b, we fit η (b) ima = β (b) 0 + u (I) i + u (M) m + u (A) a + u (IM) im + u (IA) ia + u (MA) ma .(2) 3 Under review as a conference paper Variance components are benchmark-specific, with the indexbsuppressed for readability. The decomposition separates persistent model differences from task sensitivity, scaffold differences, and model–scaffold compatibility. Because repeated observations of the same(i,m,a)cell are rare, the task–model–scaffold interaction is not separately identifiable from latent response variation; we denote this terminal residual variance byσ 2 IMA,e = π 2 /3. For an equal-allocation design withn i tasks and n a scaffolds, model-ranking reliability is Eρ 2 M(b) (n i ,n a ) = σ 2 M σ 2 M + σ 2 IM /n i + σ 2 MA /n a + σ 2 IMA,e /(n i n a ) .(3) Scaffold main effects do not enter the denominator because shifting every model equally does not change their ordering. In contrast, model–scaffold interactions are rank-relevant: they encode which models benefit from which scaffolds. When the object is the complete model–scaffold system, Eρ 2 MA(b) (n i ) = σ 2 M + σ 2 A + σ 2 MA σ 2 M + σ 2 A + σ 2 MA + (σ 2 IM + σ 2 IA + σ 2 IMA,e )/n i .(4) To measure whether scaffold choice preserves model ordering, we additionally define the inter- scaffold reliability for two independently sampled scaffolds evaluated on the same n i tasks: ρ (b) A ′ (n i ) = σ 2 M + σ 2 IM /n i σ 2 M + σ 2 IM /n i + σ 2 MA + σ 2 IMA,e /n i .(5) This is analogous to inter-rater reliability, with scaffolds acting as alternative measurement procedures. A low value means that changing the scaffold can change which model appears strongest, even when each scaffold yields internally stable scores. 3.2.2PERSISTENT MODEL DIFFERENCES ACROSS POOLED BENCHMARKS Each benchmark contains too few scaffolds to estimate all scaffold-related components precisely in isolation. We therefore also fit a joint model across the nine-benchmark leaderboard: η bima = β 0 + u (B) b + u (I[B]) bi + u (M) m + u (A) a + u (BM) bm + u (BA) ba + u (MA) ma + u (IM[B]) bim + u (IA[B]) bia + u (BMA) bma .(6) Items are nested within benchmarks, while models and scaffolds are crossed with benchmarks wher- ever supported by the observed incidence graph. Shared models and scaffolds connect benchmarks and permit partial separation of their effects. Hierarchical priors can produce estimates for weakly supported contrasts, but cannot supply empirical identification between disconnected components; connectivity and estimability diagnostics are reported in Appendix E. The leaderboard universe score for modelmisu (M) m , the component expected to persist across sampled benchmarks, tasks, and scaffolds. Benchmark-conditioned capabilityu (BM) bm contributes to performance on benchmarkb, but is not assumed to transfer to a new benchmark. For a balanced design with n b benchmarks, n i tasks per benchmark, and n a scaffolds, Eρ 2 M (n b ,n i ,n a ) = σ 2 M σ 2 M + σ 2 BM /n b + σ 2 MA /n a + σ 2 BMA /(n b n a ) + σ 2 IM[B] /(n b n i ) + σ 2 BIMA,e /(n b n i n a ) .(7) Hereσ 2 BIMA,e = π 2 /3is terminal cell-specific variation and the logistic residual (see § D.3.1 for separated variance). Equation 7 maps each design intervention to the uncertainty it can reduce: tasks average task-indexed error, benchmarks average benchmark-conditioned differences, and scaffolds average scaffold-conditioned differences. Table 1 consolidates the estimands used in the main body. 3.3WHAT ADDITIONAL EVALUATIONS CAN RESOLVE Proposition 3.1 (Facet-specific replication and reliability ceilings). For variance components, the reliability in Equations 3 and 7 is non-decreasing in each sample size. Holding n b and n a fixed, lim n i →∞ Eρ 2 M(b) (n i ,n a ) = σ 2 M σ 2 M + σ 2 MA n a , lim n i →∞ Eρ 2 M (n b ,n i ,n a ) = σ 2 M σ 2 M + σ 2 BM n b + σ 2 MA n a + σ 2 BMA n b n a .(8) 4 Under review as a conference paper Table 1: Reliability estimands. Each row appliesEρ 2 o = σ 2 o /(σ 2 o +σ 2 δ ),SNR o = σ 2 o /σ 2 δ , but changes the object whose ordering should generalize. The n f terms denote equal allocation over facet f . EstimandObject of inferenceσ 2 o σ 2 δ (D)Eq. Eρ 2 M(b) , SNR M(b) Does benchmark b preserve model order? σ 2 M σ 2 IM n i + σ 2 MA n a + σ 2 IMA,e n i n a (3) Eρ 2 MA(b) Does benchmark b preserve system order? σ 2 M + σ 2 A + σ 2 MA σ 2 IM + σ 2 IA + σ 2 IMA,e n i (4) ρ (b) A ′ Do scaffolds induce the same model order? σ 2 M + σ 2 IM n i σ 2 MA + σ 2 IMA,e n i (5) Eρ 2 M , SNR M Does the leaderboard preserve model order? σ 2 M σ 2 BM n b + σ 2 MA n a + σ 2 BMA n b n a + σ 2 IM[B] n b n i + σ 2 BIMA,e n b n i n a (7) Proof.Increasingn i monotonically decreases only denominator terms indexed by tasks. Those terms converge to zero, while benchmark- and scaffold-indexed terms remain. Thus, arbitrarily many tasks cannot overcome model–benchmark heterogeneity or model–scaffold coupling. This result motivates evaluating diversity, rather than task count alone, as a design resource. Corollary 3.2 (Benchmark breadth at a fixed task budget). Fixn a and the total numberN = n b n i of tasks per scaffold. Under the balanced, exchangeable-facet model, σ 2 δ = σ 2 MA n a + σ 2 BM + σ 2 BMA /n a n b + σ 2 IM[B] + σ 2 BIMA,e /n a N .(9) Thus, distributing the same task budget across more benchmarks increases reliability whenever σ 2 BM + σ 2 BMA /n a > 0. Proof. Use n i = N/n b in Eq. 7. Only the benchmark-conditioned term changes with n b . Breadth does not create additional persistent model signal; it reduces contamination by condition- specific advantages. The corollary assumes similarly informative benchmark draws, adequate crossing, and no additional benchmark setup cost. It does not imply that arbitrary new benchmarks outperform more tasks, or that benchmarks always offer greater value than scaffolds. 3.4BAYESIAN ESTIMATION AND DESIGN STUDIES We use Bayesian estimation with regularizing priors because sparse crossed designs can yield unstable variance estimates, especially near floor and ceiling performance. For every posterior draw, we compute reliability, SNR, and the task-only ceilings. D-study projections therefore retain uncertainty in the variance decomposition rather than substituting point estimates. Partial pooling allows connected observations to inform common variance components while retaining uncertainty where scaffold or cross-benchmark replication is limited. Primary D-studies use common equal allocations; analyses reproducing the observed imbalance appear in Appendix C. We measure evaluation volume in model–scaffold–task trials and use dashboard prices for dollar-cost projections. Repeated task subsampling checks whether reduced designs retain the ranking information predicted by the D-study (see § F.1). 3.5POSTERIOR CAPABILITY, RANKS, AND TRANSPORTABILITY For benchmarkb, the scaffold-marginalized latent capability of modelmin posterior drawsis θ (s) mb = u (M,s) m + u (BM,s) bm . To test whether the estimated shared model component is transportable rather than an artifact of the nine-benchmark panel, we evaluate b θ m = u (M) m on four contemporaneous benchmarks excluded from model estimation. Among overlapping models, we compare its association with external performance against that of the conventional in-panel mean-accuracy aggregate. This is an out-of-panel test of convergent predictive validity, not evidence of universal deployment validity nor internal construct validity. Finally, we assess sensitivity to the link function, estimation method, 5 Under review as a conference paper Table 2: HAL dataset design summary. Counts denote unique levels of each facet. BenchmarkModelsTasksAgent ScaffoldsModel–Scaffold Pairs AssistantBench1833230 CORE-Bench Hard3445356 GAIA21165234 Online-Mind2Web13300223 SciCode1765337 ScienceAgentBench19102225 SWE-bench Verified Mini2450226 τ -bench Airline2350342 USACO13307214 variance-component estimator, prior specification, and leave-one-benchmark-out refits. The latter analysis also identifies benchmarks that disproportionately contribute signal or connectivity to the leaderboard. Appendices have full computational estimation details (§ C), robustness checks and sensitivity analyses (§ D), and methodological comparisons (§ G). 4DATA We analyze agent rollouts from all nine benchmarks distributed via HAL Kapoor et al. (2024): AssistantBench (Yoran et al., 2024), CoreBench Hard (Siegel et al., 2024), GAIA (Mialon et al., 2023), Online-Mind2Web (Xue et al., 2025), SciCode (Tian et al., 2024), ScienceAgentBench (Chen et al., 2024), SWE-bench Verified Mini (Jimenez et al., 2023),τ-bench Airline (Yao et al., 2024), and USACO (Shi et al., 2024), covering web navigation, scientific programming tasks, multi-step and user assistance, software engineering, customer-service interaction, and competitive programming (details in Appendix A.1). From each of AssistantBench, GAIA, SciCode and SWE-bench Verified Mini, the HAL leaderboard uses a subset of tasks from the full benchmark (Kapoor et al., 2025). HAL dataset covers29,923total agent rollouts across 54 models and 13 scaffolds; 9 models appear on every benchmark each of which span at least 69% of the available scaffolds. Each rollout is over a single task, model, reasoning-effort, and scaffold. Summary statistics are found in Table 2. All outcomes are task-level binary scores. The incidence structure is incomplete at every level, a common problem in leaderboards (Singh et al., 2026). One scaffold is shared across eight benchmarks; AssistantBench connects the remaining benchmark through an additional shared scaffold; all other benchmarks contain at least one additional benchmark-specific scaffold. Models and scaffolds are neither fully crossed nor evenly replicated. Nevertheless, shared models and scaffolds connect the observation graph, permitting partial separation of model, benchmark, and scaffold effects. Additionally, to corroborate our claims and provide external convergent validity, we use two sup- plemental datasets. The Harbor Index dataset is a meta-benchmark containing task level scores from 29 agent benchmarks for 9 LLMs and 4 scaffolds. The second dataset consists of LLM-level scores for four benchmarks that are contemporaneous with and include LLMs specified in the HAL data. Appendices more detail for the datasets (§ A) and data connectivity analysis (§ E). These data represent current practice: because agent evaluations are expensive,many model–scaffold–benchmark combinations absent entirely. 5RESULTS AND PRACTICAL RECOMMENDATIONS The results identify a structural limitation of task-only scaling: current evaluations can distinguish fixed systems precisely while leaving persistent model differences unresolved. The useful response is to change the measurement design, not merely enlarge it. 5.1RELIABILITY DEPENDS ON THE MEASUREMENT GOAL Reliability depends on the measurement claim a practitioner wants to make. In agent evaluations, two choices are relevant here: whether the object of measurement is the underlying model or the model–scaffold system, and whether the goal is relative (rank) or absolute interpretation of scores. 6 Under review as a conference paper Model and system reliability can differ substantially. Across the nine benchmarks, estimated system reliability (i.e., ranking model–scaffold pairs) isEρ 2 MA(b) ∈ [0.935, 0.994], whereas model reliability isEρ 2 M(b) ∈ [0.148, 0.841] (Figure 1). These are not conflicting assessments of the same score. System reliability counts scaffold differences and compatibility as signal; model reliability requires differences that persist after averaging over scaffolds. A leaderboard can therefore be reliable for ranking model-scaffold pairs but unreliable for solely ranking the underlying model. assistantbenchcorebench_hardgaiamind2websciagentbenchscicodeswebenchtaubenchusaco 131030 131030110100 110100110100 1310 30131030131030110100 0.00 0.25 0.50 0.75 1.00 Number of Agentic Tasks Est. Rank Reliability: E ρ 2 Object Model Scaffold-Model Figure 1: Reliability depends on the object being ranked & task replication cannot remove scaffold-dependent model advantages. D-study reliability as tasks are added for base models and fixed model–scaffold systems. Dashed lines are task-only asymptotes for model rankings. Estimates are posterior medians with 68% standard error HDIs. See also Table 16. Recommendation. Specify whether the intended object of measure is the model or a model– scaffold system, especially in leaderboards using multiple agentic benchmarks, and whether the intended interpretation is rank comparison (or whether absolute scores are intended to carry meaning, such as capability thresholds. 4 Provide rankings, calculate scores, and estimate reliability for that specific claim rather than reporting a single generic reliability statistic. 5.2CHANGING THE SCAFFOLD CAN CHANGE EVALUATION CONCLUSIONS How much the scaffold matters depends on the benchmark. Inter-scaffold reliability (Eq. 5) measures whether different scaffolds produce similar rankings when evaluating the same models on the same tasks ranging (with 95% HDI) from 0.151 [0.004,0.577] on OnlineMind2Web to 0.852 [0.631,0.947] on CORE-Bench Hard (Figure 2). At the low end, changing only the scaffold can substantially change which models appear to perform best on a benchmark, even when the models and tasks remain fixed. Scaffolds can change which tasks get solved without improving the overall benchmark score. The pooled decomposition (Eq. 6) distinguishes persistent differences from task-specific sensitivity. In the pooled analysis, the contrastσ 2 M −σ 2 A favors neither direction (posterior directional probability approximately50%) (Makowski et al., 2019). However, task-specific variation across scaffolds is larger than task-specific variation across models with posterior probabilityPr(σ 2 IA[B] > σ 2 IM[B] | y) = 84% (see Figure 10). Benchmark specific decompositions (Eq. 2) show the same pattern (Figure 2, middle and right) This suggests that scaffold choice can strongly affect success on individual tasks, even without making one scaffold producing uniformly better on a benchmark as a whole. Similarly, A scaffold can change which tasks a system solves without producing a uniformly stronger system. Recommendation. If the goal is to evaluate the underlying model, test the same models across multiple scaffolds rather than relying on a single implementation. Report how much conclusions change across scaffolds, and avoid attributing scaffold-specific advantages to the model itself. If the deployed model–scaffold system is the intended object of evaluation, scaffold variation can instead be treated as part of the system being measured. 5.3MORE TASKS CANNOT ALWAYS RESOLVE EVALUATION UNCERTAINTY More tasks only help when the main uncertainty comes from differences across tasks.Adding tasks helps when a model’s measured performance changes substantially depending on which tasks it 4 There may be situations where reliability of actual numeric scores (not just ranking) is needed. We illustrate this separate estimation on the observed scale in § G.5, which shows that score reliability is substantially lower for all benchmarks. Reliable rankings do not imply reliable scores. 7 Under review as a conference paper 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 assistantbench corebench_hard gaia onlinemind2web scicode scienceagentbench swebench_veri fi ed_mini taubench_airline usaco Inter-Scaffold Reliability Inter-Scaffold Reliability Across Benchmark Tasks -0.50 -0.25 0.00 0.25 assistantbench corebench_hard gaia onlinemind2web scicode scienceagentbench swebench_veri fi ed_mini taubench_airline usaco Model-Agent Difference Model-Agent Difference in Rank-relevant Variance -0.6 -0.4 -0.2 0.0 assistantbench corebench_hard gaia onlinemind2web scicode scienceagentbench swebench_veri fi ed_mini taubench_airline usaco Model-Agent Difference Model-Agent Difference in Rank-relevant Task Variance Figure 2: Scaffolds are rank-relevant and result in distinct task-level consequences. Left: posterior inter-scaffold reliabilityρ (b) A ′ . Middle and Right: model-minus-scaffold system rank-relevant variance contrasts (over denominator from Eq. 4) at the benchmark and task levels, respectively; positive values favor the model contribution. Task-interaction contrasts measure task-specific sensitivity. Points and bars denote posterior means and medians, respectively, with 68% standard error HDIs. is tested on (Proposition 3.1). It does not fix uncertainty caused by other facets, such as the choice of scaffold. In our data, even infinitely many similarly constructed tasks would improve model-ranking reliability by at most0.10. Only CORE-Bench Hard and SciCode can exceedEρ 2 = 0.75through task scaling alone; for OnlineMind2Web, reliability increases only from 0.148 to 0.153. A benchmark can run out of useful information before it runs out of tasks. Adding tasks repeatedly measures the same model–scaffold and benchmark-specific effects. Once these sources of uncertainty dominate, more tasks make the existing evaluation setup more precise without making the broader model claim substantially more reliable. Low reliability does not mean that a benchmark measures an unimportant capability; it means that, for the models being compared, its scores do not reliably distinguish the quantity of interest. Thus, reliability should be re-estimated as the competitor population changes. Recommendation.Estimate how much reliability can improve from adding tasks before expanding a benchmark. Add tasks when task sampling is the main source of uncertainty; otherwise, spend evaluation budget on relevant facets that drive uncertainty, such as broader scaffold coverage. 0.2 0.4 0.6 0.8 1030100300 Total Number of Agentic Tasks Estimated Overall Rank Reliability: E ρ 2 Benchmarks 1 bench9 benches (a) Model reliability. HALHarbor Index 0100200300 0100200300 1 2 3 Total Number of Agentic Tasks Estimated Signal-to-Noise Ratio: S/N( δ ) Benchmarks 1 2 3 4 5 6 7 8 9 (b) Rank-relevant signal-to-noise ratio. Figure 3: Broader measurement conditions reduce error that an increase in the number of test items alone cannot. Pooled D-studies compare concentration within one benchmark against allocation across multiple benchmarks. Estimates are posterior medians; panel (a) includes68% standard error HDIs. HAL and Harbor are fitted separately. The reference bands[2, 3]represent minimal detection limits used in laboratory measurement (CDER, 2024; Sheehan & Yost, 2026; Taleuzzaman, 2018), and are descriptive rather than benchmark calibrated thresholds. 8 Under review as a conference paper 5.4POOLING DIVERSE BENCHMARKS CAN MAKE MODEL RANKINGS MORE RELIABLE Agentic leaderboards often consist of multiple benchmarks from which model agentic capability is to be inferred. If the goal is to rank models on their ability to perform on diverse agentic tasks, pooling benchmarks that test different kinds of tasks provides more information about which performance differences persist across settings. At a fixed task budget, benchmark breadth averages condition- specific model advantages that within-benchmark replication leaves untouched (Corollary 3.2). Table 3: Agreement with external agent benchmarks. Kendall’sτfor the unweighted HAL mean and reliability-adjusted latent ( b θ) model effect (median posterior) with [95%] CIs. External BenchmarkMean score b θDifferenceLLM Observations BFCL v40.429 [−0.282, 0.836] 0.714 [0.147, 0.928]+0.2857 Terminal-Bench 2.00.524 [−0.165, 0.869] 0.810 [0.361, 0.954]+0.2867 SWE-bench Verified0.565 [0.306, 0.746] 0.765 [0.595, 0.870]+0.20020 τ 2 -bench Core0.333 [−0.515, 0.852] 0.867 [0.383, 0.977]+0.5346 Mean τ0.4630.789+0.326— Pooling benchmarks can achieve more reliable model rankings at lower cost.At the same task budget, distributing evaluations across the nine-benchmark battery raises projected model-ranking reliability from approximately 0.44 for a single benchmark to 0.75 (Figure 3). A complementary pooled analysis of the Harbor Index (Shi et al., 2026) (Appendix A.2) supports the same qualitative pattern. Moreover, the full HAL battery costs more than ($47,000), while a balanced allocation achieves comparable reliability for approximately ($19,000). For an illustrative target of (SNR = 2.5,Eρ 2 ≈ 0.71), approximately (14) tasks per benchmark cost ($8,144), an estimated (83%) reduction. In Appendix Table 19, we corroborate the diminishing returns using task subsampling. Task diversity leads to better model rank generalization. Model effect estimates ˆ θ also agree more closely with all four held-out agent benchmarks than a simple average of benchmark scores (mean Kendall’sτ:0.789vs.0.463; Table 3; see § A.3 for data and contamination prevention). Together, these results suggest that pooling diverse benchmarks can help separate model differences that recur across settings from advantages specific to a particular benchmark or evaluation condition. These results provide convergent predictive evidence, not proof of a universal one-dimensional agent capability. 5 Breadth is most useful when the design is connected with informative benchmarks.For these data, CORE-Bench Hard contains the greatest SNR (see Figure 7, § D.4), particularly with its own scaffold, CORE Agent (§ D.5). Removing this strongest discriminating signal increases pooled uncertainty more than any other LOO removals. Thus, having a benchmark with high model signal can support anchoring a common scale through sufficient connectivity to other benchmarks. Recommendation.When the intended claim concerns performance across different kinds of agentic tasks, pool benchmarks that represent that range rather than evaluating each benchmark exhaustively. Use the reliability analysis to determine how much evaluation is needed within each benchmark while preserving enough task diversity to support the broader cross-task claim. For imbalanced designs, prioritize identifying informative anchors to bridge conditions. 6CONCLUSION Agent evaluations should not be treated as having a single, intrinsic level of reliability. Reliability depends on what practitioners want to learn from them: the underlying model or a complete model– scaffold system, a ranking or an absolute score, and performance on one benchmark or across a 5 Internally, the pooled estimation strongly outperforms per-benchmark estimations in correlations of posterior rankings with benchmark-wise observed scores, [0.62,0.96] and [0.08,0.25] respectively (Table 10, § D.2). Yet posterior rank intervals remain wide; many neighboring models have ordering probabilities near1/2(Figure 13), with some models moving across rank quartiles between reported and adjusted rankings (Figures 11 and 12). 9 Under review as a conference paper broader range of agentic tasks. Our results show that these distinctions matter in practice and hence that evaluation design should follow the intended claim: Practitioners should first specify what they want a score or ranking to mean, then identify which sources of uncertainty prevent the evaluation from supporting that claim. More evaluation is useful only when it addresses those sources. Reliability analysis can therefore guide not only how evaluation results are interpreted, but also where additional evaluation effort is most valuable for differentiating a target model population. REPRODUCIBILITY STATEMENT To facilitate reproducibility, the code and data are available online, 6 original data sources are linked in Appendix A, and computation and estimation details are found in Appendix C. AI USE STATEMENT AI was used during the writing phase of this study. GPT 5.6 Sol was used after initial drafts to reduce the text length of several sections in the main body, all of which required further editing after use. Paragraphs describing tabular results in Appendices D.4–D.6 were revised using GPT 5.5. Drafts of several sections of Appendix E were created from research notes using GPT 5.5, which were revised and then rewritten using human-edited combinations of text from GPT 5.5, GPT 5.6 Sol, and Claude Opus 4.8. Finally, the 100% human-written code used in this study was refactored with expanded comments and validated against paper findings (under subsampling) to support reproducibility using Claude Opus 4.8 with Claude Code. REFERENCES Anthropic. System Card: Claude Opus 4.7. System card, Anthropic, April 2026. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. $τ^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment, June 2025. URLhttps: //arxiv.org/abs/2506.07982v1. Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software, 67(1), 2015. ISSN 1548-7660. doi: 10.18637/jss.v06 7.i01. URL http://w.jstatsoft.org/v67/i01/. Aidan D. Bindoff. Partial pooling predicts cross-validation reliability: a closed-form triage and Rao-Blackwellised cure for hierarchical LOO, July 2026. URLhttp://arxiv.org/abs/ 2607.18836. arXiv:2607.18836 [stat.ME]. Robert L. Brennan. Generalizability Theory. Springer, New York, NY, 2001. ISBN 978-1-4419- 2938-9 978-1-4757-3456-0. doi: 10.1007/978-1-4757-3456-0. URLhttp://link.springe r.com/10.1007/978-1-4757-3456-0. Paul-Christian Bürkner. Bayesian Item Response Modeling in R with brms and Stan. Journal of Statistical Software, 100:1–54, November 2021. ISSN 1548-7660. doi: 10.18637/jss.v100.i05. URL https://doi.org/10.18637/jss.v100.i05. CDER. Q2(R2) Validation of Analytical Procedures, March 2024. URLhttps://w.regula tions.gov/document/FDA-2022-D-1503-0002. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery. arXiv preprint arXiv:2410.05080, 2024. Lee J. Cronbach and Paul E. Meehl. Construct validity in psychological tests. Psychological Bulletin, 52(4):281–302, 1955. ISSN 1939-1455. doi: 10.1037/h0040957. 6 https://github.com/hardy-education/scaffold_eval 10 Under review as a conference paper Lee J. Cronbach and Richard J. Shavelson. My Current Thoughts on Coefficient Alpha and Successor Procedures. Educational and Psychological Measurement, 64(3):391–418, June 2004. ISSN 0013-1644. doi: 10.1177/0013164404266386. URLhttps://doi.org/10.1177/001316 4404266386. Lee J. Cronbach, Goldine Gleser, Harinder Nanda, and Nageswari Rajaratnam. The Dependability of behavioral measurements: theory of generalizability for scores and profiles. Wiley, New York, 1972. ISBN 978-0-471-18850-6. Jean-Paul Fox and Cees A. W. Glas. Bayesian estimation of a multilevel IRT model using gibbs sampling. Psychometrika, 66(2):271–288, June 2001. ISSN 1860-0980. doi: 10.1007/BF02294839. URL https://doi.org/10.1007/BF02294839. Michael Hardy and Yunsung Kim. Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact, April 2026. URLhttp://arxiv.org/abs/2603.00883. arXiv:2603.00883 [cs]. Michael Hardy, Anka Reuel, Lijin Zhang, Jodi M. Casabianca, Sang Truong, Yash Dave, Hansol Lee, Benjamin Domingue, and Sanmi Koyejo. AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems, May 2026. URLhttp://arxiv.org/abs/2605.25272. arXiv:2605.25272 [cs.AI]. Xavier A. Harrison. A comparison of observation-level random effect and Beta-Binomial models for modelling overdispersion in Binomial data in ecology & evolution. PeerJ, 3:e1114, July 2015. ISSN 2167-8359. doi: 10.7717/peerj.1114. URLhttps://pmc.ncbi.nlm.nih.gov/art icles/PMC4517959/. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 6864–6890, 2024. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The twelfth international conference on learning representations, 2023. Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI Agents That Matter, July 2024. URL https://arxiv.org/abs/2407.01502v1. Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary S. Siegel, Boyi Wei, Tianci Xue, Ziru Chen, Felix Chen, Saiteja Utpala, Franck Ndzomga, Dheeraj Oruganty, Sophie Luskin, Kangheng Liu, Botao Yu, Amit Arora, Dongyoon Hahm, Harsh Trivedi, Huan Sun, Juyong Lee, Tengjun Jin, Yifan Mai, Yifei Zhou, Yuxuan Zhu, Rishi Bommasani, Daniel Kang, Dawn Song, Peter Henderson, Yu Su, Percy Liang, and Arvind Narayanan. Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation, October 2025. URLhttp://arxiv.org/ab s/2510.11977. arXiv:2510.11977 [cs]. Pankaj Kumar and Subhankar Mishra. Robustness in large language models: A survey of mitigation strategies and evaluation metrics. arXiv preprint arXiv:2505.18658, 2025. John Linacre and Benjamin Wright. Understanding Rasch measurement: Construction of measures from many-facet data. Journal of applied measurement, 3:486–512, February 2002. Lovish Madaan, Aaditya K. Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stene- torp, Sharan Narang, and Dieuwke Hupkes. Quantifying Variance in Evaluation Benchmarks, June 2024. URL http://arxiv.org/abs/2406.10229. arXiv:2406.10229. Dominique Makowski, Mattan S. Ben-Shachar, S. H. Annabel Chen, and Daniel Lüdecke. Indices of Effect Existence and Significance in the Bayesian Framework. Frontiers in Psychology, 10, December 2019. ISSN 1664-1078. doi: 10.3389/fpsyg.2019.02767. URLhttps://w.fron tiersin.org/journals/psychology/articles/10.3389/fpsyg.2019.0276 7/full. 11 Under review as a conference paper Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Jesse Hu, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Alex Dimakis, Andy Konwinski, and Ludwig Schmidt. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces, 2026. URLhttps://arxiv. org/abs/2601.11868. Version Number: 1. Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023. OpenAI. GPT-5.5 System Card. System card, OpenAI, April 2026. Topi Paananen, Juho Piironen, Paul-Christian Bürkner, and Aki Vehtari. Implicitly adaptive impor- tance sampling. Statistics and Computing, 31(2):16, February 2021. ISSN 1573-1375. doi: 10.100 7/s11222-020-09982-2. URL https://doi.org/10.1007/s11222-020-09982-2. Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. In Proceedings of the 42nd International Conference on Machine Learning, p. 48371–48392. PMLR, October 2025. URLhttps: //proceedings.mlr.press/v267/patil25a.html. Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, and Arvind Narayanan. Towards a Science of AI Agent Reliability, February 2026. URLhttp://arxiv.org/abs/ 2602.16666. arXiv:2602.16666 [cs]. Amirhossein Razavi, Mina Soltangheis, Negar Arabzadeh, Sara Salamat, Morteza Zihayat, and Ebrahim Bagheri. Benchmarking prompt sensitivity in large language models. ArXiv, abs/2502.06065, 2025. URLhttps://api.semanticscholar.org/CorpusID: 276249536. Maria L. Rizzo and Gábor J. Székely. DISCO analysis: A nonparametric extension of analysis of variance. The Annals of Applied Statistics, 4(2), June 2010. ISSN 1932-6157. doi: 10.1214/09-A OAS245. URL http://arxiv.org/abs/1011.2288. arXiv:1011.2288 [stat]. Olawale Salaudeen, Anka Reuel, Ahmed Ahmed, Suhana Bedi, Zachary Robertson, Sudharsan Sundar, Ben Domingue, Angelina Wang, and Sanmi Koyejo. Measurement to Meaning: A Validity- Centered Framework for AI Evaluation, June 2025. URLhttp://arxiv.org/abs/2505 .10573. arXiv:2505.10573 [cs]. Richard J. Shavelson, Noreen M. Webb, and Glenn L. Rowley. Generalizability theory. American Psychologist, 44(6):922–932, 1989. ISSN 1935-990X. doi: 10.1037/0003-066X.44.6.922. Terry L. Sheehan and Richard A. Yost. What’s the Most Meaningful Standard for Mass Spectrometry: Instrument Detection Limit or Signal-to-Noise Ratio? | Spectroscopy Online, August 2026. URL https://w.spectroscopyonline.com/view/what-s-most-meaningful-s tandard-mass-spectrometry-instrument-detection-limit-or-signa l-noise-ratio-0. Lin Shi, Haowei Lin, Zixuan Zhu, Xiaoyue Zhou, Xiang Li, Xiangning Lin, Yaxuan Deng, Han Xu, Yuangang Li, Shanda Li, Zizhao Chen, Hanwen Xing, Harsh Raj, Bo Chen, Quan Shi, Steven 12 Under review as a conference paper Dillmann, Yipeng Gao, Puneesh Khanna, Ruofan Lu, Chao Beyond Zhou, Michael Yang, Robert Zhang, Siyuan Chai, Jiayu Chang, Yizhao Chen, Xiaokun Chen, Yiwei Dai, Wenting Yang, Hange Liu, Minghao Liu, Zihan Wang, Adnan El Assadi, Benedikt Stroebl, E. Kelly Buchanan, Han Meng, Junwei He, Longxuan Yu, Radin Shayanfar, Yukyung Lee, Zhikang Dong, Allen G Hart, Anjiang Wei, Anurag Kashyap, Arpandeep Khatua, Audrey Jixin Zheng, Chengrui Ma, David Heineman, Dubing Chen, Hai-Anh Trinh, Haishuo Fang, Hefan Zhang, Hui Shen, Issa Sugiura, Jiankai Sun, Jiechao Gao, Junhong Lin, Junnan Li, Kai Yang, Lei Hsiung, Maoyu Wang, Mengze Tang, Nabil Omi, Negin Raoof, Nicholas Edwards, Octavia Guo, Orfeas Menis Mastromichalakis, Pengliang Ji, Przemysław Hejman, Qi Qi, Qunshu Lin, Richard Zhuang, Rui Yang, Ruichen Zheng, Ryan Marten, Shaghayegh Fazliani, Shizheng Hou, Sicong Jiang, Sijie Li, Boqin Yuan, Michael Glass, Song Bian, Terry Yue Zhuo, Tianqing Wu, Tom Tang, Wanjia Zhao, Weihao Xuan, Wenhua Liang, Xian Liu, Xin Lan, Xuan Zhang, Xuandong Zhao, Yanchuan Tang, Yifan Jiang, Yijiang Li, Yitong Guan, Yizhi Li, Yonghui Liu, Yuheng Tang, Yujun, Mao, Yunfei Zhao, Yuxin Wang, Yuxuan Tang, Zhenheng Tang, Zhifei Li, Ziruo Wang, Ziyu She, Kaiyuan Liu, Iheb Chaabane, Yuxin Tang, Xiangyi Li, Satya Sai Srinath Namburi GNVV, Xinyue Zheng, Andy Konwinski, Boxuan Li, Leon Liangyu Chen, Alex Dimakis, Nicholas Carlini, Soroush Vosoughi, Sanmi Koyejo, Di He, Etash Guha, Benjamin Feuer, Mike Merrill, Ludwig Schmidt, and Alex Shaw. Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation, September 2026. URL https://arxiv.org/abs/2609.04298. Version Number: 3. Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. Can language models solve olympiad programming? arXiv preprint arXiv:2404.10952, 2024. Zachary S Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. Core- bench: Fostering the credibility of published research through a computational reproducibility agent benchmark. arXiv preprint arXiv:2409.11363, 2024. Shivalika Singh, Yiyang Nan, Alex Wang, Daniel Dsouza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah Smith, Beyza Ermis, Marzieh Fadaee, and Sara Hooker. The Leaderboard Illusion. Advances in Neural Information Processing Systems, 38, April 2026. doi: 10.52202/085713-2620. URLhttps://proceedings.neurips.c/paper _files/paper/2025/hash/70a93f260a51123b3c0e33ecd1b4de97-Abstrac t-Datasets_and_Benchmarks_Track.html. Gábor J. Székely and Maria L. Rizzo. The Energy of Data. Annual Review of Statistics and Its Application, 4(1):447–479, March 2017. ISSN 2326-8298, 2326-831X. doi: 10.1146/annurev-sta tistics-060116-054026. URLhttps://w.annualreviews.org/doi/10.1146/an nurev-statistics-060116-054026. Mohamad Taleuzzaman. Limit of Blank (LOB), Limit of Detection (LOD), and Limit of Quantifi- cation (LOQ). Organic & Medicinal Chemistry International Journal, 7(5), August 2018. ISSN 24747610. doi: 10.19080/OMCIJ.2018.07.555722. URLhttps://juniperpublishers. com/omcij/OMCIJ.MS.ID.555722.php. Minyang Tian, Luyu Gao, Shizhuo D Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, et al. Scicode: A research coding benchmark curated by scientists. Advances in Neural Information Processing Systems, 37:30624–30650, 2024. Aki Vehtari, Andrew Gelman, and Jonah Gabry. Practical Bayesian model evaluation using leave- one-out cross-validation and WAIC. Statistics and Computing, 27(5):1413–1432, September 2017. ISSN 1573-1375. doi: 10.1007/s11222-016-9696-4. URLhttps://doi.org/10.1007/s1 1222-016-9696-4. Aki Vehtari, Daniel Simpson, Andrew Gelman, Yuling Yao, and Jonah Gabry. Pareto Smoothed Importance Sampling. Journal of Machine Learning Research, 25(72):1–58, 2024. ISSN 1533- 7928. URL http://jmlr.org/papers/v25/19-556.html. Ruipeng Wang, Yuxin Chen, Yukai Wang, Chang Wu, Junfeng Fang, Xiaodong Cai, Qi Gu, Hui Su, An Zhang, Xiang Wang, et al. Agentnoisebench: Benchmarking robustness of tool-using llm agents under noisy condition. arXiv preprint arXiv:2602.11348, 2026. 13 Under review as a conference paper Wen-Chung Wang and Mark Wilson. Exploring Local Item Dependence Using a Random-Effects Facet Model. Applied Psychological Measurement, 29(4):296–318, July 2005. ISSN 0146-6216. doi: 10.1177/0146621605276281. URLhttps://doi.org/10.1177/01466216052762 81. Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An Illusion of Progress? Assessing the Current State of Web Agents, October 2025. URL http://arxiv.org/abs/2504.01382. arXiv:2504.01382 [cs.AI]. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.τ-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024. Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. Assistantbench: Can web agents solve realistic and time-consuming tasks? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 8938–8968, 2024. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023. 14 Under review as a conference paper APPENDIX LIMITATIONS AND FUTURE WORK Reliability is necessary but not sufficient for valid evaluation.The bounds we estimate describe how consistently a benchmark differentiates the objects it ranks, not whether the rankings correspond to underlying capability or to any external ground truth: a benchmark can be reliable but invalid. The object-of-measurement problem we document is in this sense a construct validity problem in disguise, since reliability against an undefined construct cannot be assessed in principle (Cronbach & Meehl, 1955; Salaudeen et al., 2025). Future work should complement the reliability framework with validity studies, including external-criterion validation and cross-benchmark transfer analyses. Scope conditions on the headline ceiling.The reliability bounds we report are joint properties of the benchmark, the scaffold sample, the task sample, and the competitor set being ranked. A clustered frontier-model pool depresses σ 2 M regardless of benchmark quality, lowering reliability not because the benchmark is poorly designed but because it is being used to distinguish systems too close for its resolution; hence, the ceiling we find should be read as a statement about current leaderboards for the current frontier-model competitor set. Scaffolds are selected artifacts, often co-developed with the benchmarks they evaluate, and per-benchmark counts (n a ∈2, 3) reflect standard practice across the field rather than a feature of HAL; pooling across benchmarks in Equation 6 partially addresses but per-benchmark scaffold-variance claims remain weaker, and pooling itself depends on HAL-GENERALIST as the cross-benchmark harness anchor (Appendix E). Future work should construct evaluation datasets that systematically span larger scaffold libraries, and should track how the ceiling shifts as the model pool evolves. Scope of Agent Scaffolds and Systems.These results are conditional on the sampled model pool and scaffold set. A compressed frontier depressesσ 2 M and thus reliability independent of benchmark quality; a richer scaffold sample would sharpen the model–scaffold decomposition. Our claims are therefore about these leaderboards for this competitor set, and generalize to new models and scaffolds only insofar as the sampled conditions represent the intended universe. We view stating this scope explicitly as part of reliable practice rather than a caveat to it. Inference remains conditional on the represented evaluation population; partial pooling does not by itself correct selective evaluation or reporting. Latent-scale reliability.We estimate reliability on the latent logit scale, while leaderboards report observed proportions. The two scales do not translate one-to-one, so our numbers should be read as approximate bounds on observed-scale rank stability. Future work could entail a sensitivity analysis comparing the scales. Raising reliability bounds. The empirical scope of this work is the characterization of reliability bounds rather than their displacement. Validating interventions that raise the ceiling is the subject of subsequent work, since each candidate intervention introduces its own measurement-design tradeoffs. Candidate interventions include task selection using item discrimination methods, scaffold sampling under a defined scaffold family, and task-quality auditing; empirical evaluation of these, alongside methods that reduce the measurement cost of the recommendations, are the most direct extensions. ADATASET DESCRIPTIONS A.1HAL DATASET The Holistic Agent Leaderboard (HAL) dataset 7 is a standardized, cost-aware, and third-party evaluation platform and dataset initiative developed by the SAgE (Science of Agent Evaluation) research group at Princeton University. The formatted and cleaned tasks-level of this dataset is publicly released 8 . 7 https://hal.cs.princeton.edu/ 8 https://huggingface.co/datasets/razam2/hal-response-matrix 15 Under review as a conference paper A.1.1BENCHMARKS AssistantBench Tasks consist of time-consuming, busy-work tasks that an average person may face, seeking information from the web. Which gyms near Tompkins Square Park (<200m) have fitness classes before 7am? CORE-Bench HardTasks ask an agent to computationally reproduce specific quantitative results from a published scientific paper, given a code repository, dataset, and research paper. Run the main.py file three times. First, withconfig/uci.json, the preprocessing task, and the CTGCN- C method. Second, withconfig/uci.json, the embedding task, and the CTGCN-C method. Third, using Python3 with config/uci.json and the link-pred task. GAIA GAIA consists of tasks requiring multi-step tool use — including web browsing, code execution, file reading (PDFs, spreadsheets, audio), and multimodal understanding. (Given an Excel file) The attached Excel file contains the sales of menu items for a local fast-food chain. What were the total sales that the chain made from food (not including drinks)? Express your answer in USD with two decimal places. OnlineMind2WebOnline Mind2Web is the live, online version of Mind2Web. It does not rely on cached pages and allows for real-time testing against dynamic, evolving web interfaces. All tasks are sourced from 136 popular websites to reflect authentic user workflows. Browse used Audi cars made before 2015 and sort by lowest price on KBB. Website: https://w.kbb.com/ SciCode Tasks consist of research-level coding problems decomposed into subproblems drawn from actual scientific work across physics, chemistry, biology, math, and materials science. Main problem: Reproduce the Chern number phase diagram of the Haldane model on a hexagonal lattice. Subproblem 1.1: Write a Haldane model Hamiltonian on a hexagonal lattice, given: wavevector componentsk x andk y , lattice spacinga, nearest-neighbor coupling constantt 1 , next-nearest-neighbor coupling constantt 2 , phaseφfor next-nearest-neighbor hopping, and on-site energym. Output: a2× 2 matrix. Subproblem 1.2: Calculate the Chern number using the Haldane Hamiltonian, given the grid sizeδ for discretizing the Brillouin zone. . . ScienceAgentBench ScienceAgentBench is a benchmark for evaluating the ability of language agents to conduct data-driven scientific discovery. (Given separate training and test datasets) Train a multitask model on the Clintox dataset to predict a drug’s toxicity and FDA approval status. Save the test set predictions, including the SMILES representation of drugs and the probability of positive labels, to “pred_results/clintox_test_pred.csv”. SWE-bench Verified MiniA subset of SWE-bench Verified where each task gives the agent a real GitHub issue description and a full Python repository, and requires the agent to produce a code patch that makes failing unit tests pass. (Given a repo and issue description) Navigate the repository, identify the relevant code path in django/db/models/query.py, and produce a.patchfile that makes the FAIL_TO_PASS unit tests pass without breaking existing PASS_TO_PASS tests. τ-bench AirlineTasks simulate a realistic airline customer service scenario in which a human user (played by another LLM) contacts an agent with requests like rebooking a flight, adding a passenger, 16 Under review as a conference paper or canceling a reservation. The agent must follow a detailed airline policy document and a limited set of callable functions. (Given the airline’s policy and pre-defined functions) Your user id isdaiki_muller_1116. You want to cancel your upcoming flights within reservation IDs XEHM4B and 59X6W. If the agent says either reservation has basic economy flights, ask to upgrade them to economy first and then cancel them. You are very persistent and terse but clear. After the third agent message, also ask whether you have any other upcoming flights and what their total cost is. USACO Tasks are problems from the USA Computing Olympiad, spanning four difficulty tiers (Bronze through Platinum). Farmer John hasNcows (2≤ N ≤ 10 5 ), each liking exactly one type of hayh i . He can host focus groups over contiguous ranges of cows — if more than half the cows in a group prefer the same type, all cows switch to that type. He wants to know which types of hay can become universally liked. GivenTtest cases, each with N and a list of h i values, output all achievable universal hay types in increasing order, or−1. A.1.2SCAFFOLDS The scaffolds included in the HAL benchmark dataset are in Table 7. For each benchmark, the scaffold consist of a contrast between a generalist scaffold (e.g., HAL generalist, Claude Code) and and a specialist scaffold tailored to the specific benchmark (e.g., SWE agent,τ-bench tool-calling). For most benchmarks, the majority of LLMs have scores across multiple scaffolds, with the exceptions of USACO, SWE-bench Verified Mini, and ScienceAgentBench. USACO in particular has very little model–scaffold diversity, with uncertainty visible in its very large HDI intervals when trying to generalize scaffolds. A.2HARBOR INDEX DATASET FOR DECISION STUDY CORROBORATION Harbor-Index 9 is a curated meta-dataset and benchmark containing 82 difficult and diverse tasks designed for evaluating AI language model agents. Distilled from a pool of over 6,000 candidate tasks across 54 benchmarks, the final dataset features 82 high-quality tasks spanning 29 benchmarks and seven domains (including software engineering, scientific research, tool use, mathematics, data analytics, and security). It was built by passing candidate tasks through difficulty filtering, automated AI audits, human reviews, and iterative audit-and-fix loops to weed out structurally broken or flawed tasks. The final dataset lacks sufficient per-benchmark task representation that prevent inclusion in all the analyses of this paper. Descriptive statistics are in Table 4. Thus, to estimate both task and benchmark effects, we take the subset of benchmarks with at least three items. The cross-benchmark model of Equation 6 is fit and produces the sample the posterior draws used in Figure 3b. For our analyses, we use their verified and judged outcome classifications as the measure of task success (i.e., if the verifier marked a reward as a false negative and the judge determined it was a “true solve” it was marked as successful). Because the dataset suffers from a small quantity of total tasks, both the proportion of variance due to persistent differences in LLM and overall reliability begins to drop more sharply if we take the subset of benchmarks with at least four items, decreasing further with the subset that has five items. We conjecture this pattern may be in part due to the nonrandom nature of the item selection process used to represent each benchmark. A.3EXTERNAL VALIDATION DATASETS FOR MODEL RANKINGS We compare the posterior-median model effect b θ m with four contemporaneous external benchmarks containing at least six overlapping LLMs and which report individual task-level performance by model. The main body reports concordance of the observed rankings with b θ m and mean score with these external benchmarks in Table 3; § A.3.1 analyzes the differences between the two HAL ranking mechanisms. To prevent data leakage for the external SWE-bench andτ 2 -bench, the semantically 9 https://harbor-index.org/ 17 Under review as a conference paper Table 4: Descriptive statistics from the complementary Harbor Index dataset. Bolded datasets have at least three tasks and were used in the analysis of benchmark and item effects on agent system signal. BenchmarkModelsTasksScaffoldModel-Scaffold PairMean Score algotune954180.078 arcagi2954180.067 bigcodebench914180.222 bixbench954180.122 codepde914180.000 cybergym924180.1 39 dacode914180.111 featurebench944180.042 gaia934180.185 gaia2954180.051 gpqadiamond914180.056 gso974180.024 hle984180.079 labbench944180.182 omnimath924180.278 qcircuitbench914180.000 replicationbench914180.444 scicode934180.019 skillsbench924180.167 sldbench914180.056 spider2924180.000 swebenchpro944180.139 swebenchverified954180.067 swelancer924180.139 swesmith914180.111 swtbenchverified914180.333 tb934180.241 usaco914180.000 widesearch914180.000 corresponding HAL benchmark is excluded before estimating both b θ m and the raw-score baseline (see § D.4), as described in the details of the data and collection below. •The Berkeley Function Calling Leaderboard (BFCL) V4 (Patil et al., 2025) is a benchmark designed to evaluate the tool use, API execution, and agentic capabilities of large language models. 10 . We use BFCL’s holistic score for function-calling models. •Terminal-Bench 2.0 (Merrill et al., 2026) is an evaluation suite and benchmark designed to test how well AI agents perform complex, multi-step tasks inside a real command-line interface (CLI). 11 . To compare LLMs, we average each model’s score across reported agent runs. •SWE-bench Verified (Jimenez et al., 2023) is the superset of tasks that are associated with the HAL dataset SWE-bench Verified Mini. The data of several third-party external leaderboard providers 12 were collected. Accuracy scores were averaged across leaderboards for overlapping models for the standard Kendall’sτ, and where left separate for the multilevel partial correlation (Appendix A.3.2). When performing analyses with this data, such as in 10 https://gorilla.cs.berkeley.edu/leaderboard.html 11 https://hub.harborframework.com/datasets/terminal-bench/terminal-bench -2/latest?tab=leaderboard&leaderboard=2-0 12 https://w.swebench.com/ ,https://w.vals.ai/benchmarks/swebench, https://llm-stats.com/benchmarks/swe-bench-verified 18 Under review as a conference paper Table 3, the external correlations were calculated with estimates from the HAL dataset after removing SWE-bench Verified Mini from the analysis. The description of this process based on the pooled model (Eq. 6) can be found in Appendix D.4. • τ 2 -bench Core (Barres et al., 2025) is an evaluation framework and benchmark for conversa- tional AI and customer service agents that tests performance in a “dual-control” environment where both the AI agent and the user take active actions. It contains tasks that evaluate LLM agents on retail, airline (of which HAL’sτ-bench Airline is a subset), and telecom customer-service tasks, where the agent and the user both act on the world. 13 We use the overall score across all of these domains and, as with SWE-bench Verified, we only calculate correlations with HAL data after removingτ-bench Airline. The description of this process based on the pooled model (Eq. 6) can be found in Appendix D.4. A.3.1DIFFERENCES IN CONCORDANCE WITH EXTERNAL PERFORMANCE In addition to Table 3, we provide additional analyses to explore the differences between transportable ranking methods. This section supplements the differences in correlations. We compared the rank agreement of the more traditional aggregated mean score ̄xand the latent estimated value ˆ θwith external performance separately for each benchmark. For each model, the external reference was its mean reported benchmark score across the available sources and evaluation conditions. Both scoring mechanisms were evaluated on the same models within each benchmark. We used Kendall’s τ b , defining the paired difference as ∆ b = τ b ( ̄x,Z b )− τ b ( ˆ θ,Z b ), where Z b denotes the aggregated external reference for benchmark b. Thus, ∆ b < 0 favors ˆ θ. To characterize uncertainty, we constructed nominal 95% percentile intervals using a paired bootstrap within each benchmark (with 20,000 bootstraps). Each resampled unit was a complete model-level observation containing mean score, ˆ θ , and the external reference, preserving the dependence between the two estimated correlations. Given the small numbers of models, particularly for benchmarks with six or seven observations, these intervals were interpreted as exploratory rather than as reliably calibrated confidence intervals. We also recomputed each difference after removing each model in turn to assess sensitivity to individual observations. The resulting leave-one-out ranges are influence diagnostics, not confidence intervals. Table 5: Within-benchmark agreement with the aggregated external reference. Negative differences favor ˆ θ. Bootstrap intervals are exploratory, marginal 95% percentile intervals and are not adjusted for multiple comparisons. Benchmarkn τ b ( ̄x,Z) τ b ( ˆ θ,Z)∆ b Bootstrap intervalLOO range BFCL70.4290.714 −0.286[−1.111, 0.556][−0.533, 0.000] SWE-bench200.5330.755 −0.222[−0.503, 0.045][−0.258,−0.129] τ 2 -core60.3330.867 −0.533[−1.231, 0.000][−0.800,−0.400] Terminal-Bench 2.070.5240.810 −0.286[−0.824, 0.000][−0.400,−0.133] The observed correlations consistently favored ˆ θ(Table 5). Its advantage inτ b ranged from 0.222 on SWE-bench to 0.533 onτ 2 -core. Both mechanisms were positively associated with the external reference in every benchmark, but ˆ θ exhibited stronger agreement throughout. The direction of the comparison was also stable under single-model deletion. Every leave-one-out difference remained strictly negative for SWE-bench,τ 2 -core, and Terminal-Bench 2.0. For BFCL, deleting an individual model could reduce the difference to zero, but no deletion reversed its sign. Consequently, the observed direction was not dependent on retaining any single model, although this diagnostic does not establish stability to broader changes in the model sample. As expected, uncertainty remained substantial due to the constraint of having few candidate con- temporaneous benchmarks that correspond with HAL benchmark models. Thus predictably for small sample sizes, none of the bootstrap intervals excluded zero: those for BFCL and SWE-bench 13 https://taubench.com/leaderboard?benchmark=core 19 Under review as a conference paper extended above zero, whereas those forτ 2 -core and Terminal-Bench 2.0 ended at zero. The latter endpoints should not be interpreted as precise significance thresholds, given the discrete rank statistic and small samples. Interval bounds below−1are permissible because a difference between two Kendall correlations lies in[−2, 2]. Undefined bootstrap differences were absent except forτ 2 -core, where they occurred in 0.01% of resamples; its interval was calculated from the defined replicates. These difference analyses provide consistent descriptive evidence that ˆ θranks the observed models more closely to their aggregated external performance than the mean score does. They do not, however, establish a statistically conclusive advantage within individual benchmarks. Moreover, the analysis treats each model’s aggregated external score as its reference value: it does not separately propagate uncertainty in that score or adjust for unequal coverage of sources and evaluation conditions. The results therefore characterize agreement with the observed mean references, rather than with context-adjusted latent performance. A.3.2COMBINED EXTERNAL DATA MULTILEVEL PARTIAL CORRELATION In addition to traditional Kendall’sτ b correlations, a multilevel partial correlation was calculated to clarify the relationships with greater statistical power. For LLMmevaluated on benchmarkband external leaderboard providersℓ, letR mbℓ denote its observed performance rank andP mbℓ the rank induced by the calculated proxy. Benchmark- and leaderboard-provider-level heterogeneity can be modeled through the rank-based mixed-effects specifications R mbℓ = x ⊤ mbℓ β R + u R b + v R ℓ + ε R mbℓ , P mbℓ = x ⊤ mbℓ β P + u P b + v P ℓ + ε P mbℓ , whereu (·) b andv (·) ℓ are random effects for benchmarks and leaderboards, respectively, andx mbℓ contains any control variables. The inclusion of benchmark and leaderboard random effects removes the need to average across the SWE-bench Verified leaderboards, increasing the overall number of observations to 52 for this analysis. The multilevel partial Kendall correlation is then defined as the concordance between the adjusted residual ranks, τ partial = E h sgn ̃ R a − ̃ R a ′ sgn ̃ P a − ̃ P a ′ i , where ̃ R mbℓ = R mbℓ −x ⊤ mbℓ ˆ β R − ˆu R b − ˆv R ℓ and ̃ P mbℓ = P mbℓ −x ⊤ mbℓ ˆ β P − ˆu P b − ˆv P ℓ . Thus,τ partial measures agreement between the LLM rankings and the proxy rankings after accounting for observed covariates and clustering attributable to benchmarks and leaderboards. The results are in Table 6. Table 6: Hierarchical concordance with external agent benchmarks. Kendall’sτfor the un- weighted HAL mean and reliability-adjusted latent ( b θ ) model effect (median posterior) with[95%] CIs. External BenchmarkMean score b θDifferenceLLM Observations Multilevel τ partial 0.430 [0.266, 0.570] 0.632 [0.507, 0.732]+0.20252 BHAL COVERAGE MATRIX Below is the list of agents used to test each of the benchmarks. Each model was not uniformly tested with each benchmark and agent. As shown in Table 7, about nearly all (11/13) agents have only been tested on a single benchmark. Similarly, many models (12/54) are tested with a single agent. The full breakdown of which models, agents and benchmarks have been tested can be seen in Figure 5. 20 Under review as a conference paper Table 7: Agent harness/scaffold by benchmark. BenchmarkAgent Names AssistantBenchhal_generalist, browser-use CORE-Bench Hardhal_generalist, coreagent, claude_code GAIAhal_generalist, hf_open_deep_research OnlineMind2Webbrowser-use, seeact SciCodehal_generalist, tool_calling_agent, scicode_zero ScienceAgentBenchhal_generalist, sab_selfdebug SWE-bench Verified Minihal_generalist, sweagent τ -bench Airlinehal_generalist, taubench_tool_calling, taubench_fewshot usacohal_generalist, usaco_episodic_semantic Figure 4: Number of benchmarks tested (left) and agents tested (right) per model. CMETHODS, ESTIMATION, AND COMPUTATIONAL DETAILS This appendix reports prior distributions, parameterization, sampler settings, convergence diagnostics, posterior predictive checks, and the treatment of repeated observations. It also gives reliability expressions for the observed unbalanced allocation. For each posterior draw, design-specific error is computed using the realized cell weights rather than the equal-allocation approximations used for the main D-studies. We provide the full random-effects structure for each model using standard mixed-model notation. For the main body studies, we estimate all variance components jointly using Bayesian generalized linear mixed models, which provide partial pooling for sparsely observed cells and propagate uncertainty into the resulting generalizability coefficients. C.1ESTIMATION DETAILS C.1.1LEADERBOARD-LEVEL POOLED MODEL Let y bima ∈0, 1 denote success on benchmark b, task i, model m, and scaffold a. We assume y bima | p bima ∼ Bernoulli(p bima ),logit(p bima ) = η bima , 21 Under review as a conference paper Figure 5: Matrix indicating all the variations of benchmarks, scaffold and models (with reasoning-effort) that were included in our dataset. 22 Under review as a conference paper with latent linear predictor (from Equation 6) η bima = μ + u b + u bi + u m + u a + u bim + u bia + u ma + u bm + u ba + u bma . Here bi identifies a task nested within its benchmark. Each term is a mean-zero random intercept, u F,j | σ F ∼N (0,σ 2 F ), F ∈b,bi,m,a,bim,bia,ma,bm,ba,bma, and random-effect families are conditionally independent. This decomposition separates persistent model and scaffold differences from variation attributable to benchmark choice, task composition, and model–scaffold compatibility. In particular,u bm captures benchmark-dependent model performance, whileu bma captures benchmark-specific compatibility between models and scaffolds; both can alter rankings even when average model effects are unchanged. The model was fitted to29,923observations spanning9benchmarks,1,117benchmark–task units, 54models, and13scaffolds. Because the likelihood is Bernoulli-logit, all variance components are defined on the latent log-odds scale. When an observation-level residual is required for a generalizability coefficient, the conventional logistic varianceπ 2 /3is used. Reliability quantities are computed separately for every posterior draw, rather than from ratios of posterior mean variances, thereby preserving uncertainty in nonlinear variance decompositions. brms specification: brm(score ~ 1 + (1|benchmark) + (1|benchmark:task_id) + (1|model_name) + (1|agent_name) + (1|benchmark:task_id:model_name) + (1|benchmark:task_id:agent_name) + (1|model_name:agent_name) + (1|benchmark:model_name) + (1|benchmark:agent_name) + (1|benchmark:model_name:agent_name), family = bernoulli("logit"), ...) C.1.2BENCHMARK-LEVEL DECOMPOSITION MODELS To quantify the signal supplied by each benchmark, we also fit a separate model within every benchmark using Equation 2: y ima ∼ Bernoulli(p ima ), logit(p ima ) = μ + u i + u m + u a + u im + u ia + u ma . These fits distinguish stable model variation from task-, scaffold-, and interaction-driven variation within an instrument. Thus, a benchmark with many observations need not be highly informative for ranking models: its effective signal depends on the posterior magnitude of model-related variance relative to the variance induced by tasks, scaffolds, and their interactions. brms specification: brm(score ~ 1 + (1|task_id) + (1|model_name) + (1|agent_name) + (1|task_id:model_name) + (1|task_id:agent_name) + (1|model_name:agent_name), family = bernoulli("logit"), ...) C.1.3PRIORS AND POSTERIOR COMPUTATION We used the default weakly informative brms priors. For every group-level standard deviation, σ F ∼ Student -t + (3, 0, 2.5), wheret + denotes truncation toσ F ≥ 0. Equivalently, the estimated variance component isσ 2 F . The prior is concentrated near modest latent-scale heterogeneity while retaining sufficiently heavy tails for large benchmark or task effects. The intercept used the corresponding weakly informative 23 Under review as a conference paper Table 8: Full posterior summary. Bayesian fit estimates for the full leaderboard-level model (Eq. 6). Group-level entries are standard deviations of random intercepts on the latent log-odds scale. Intervals are equal-tailed 95% posterior credible intervals. EffectLevels Estimate SD95% CrI b R Bulk ESS Tail ESS Group-level standard deviations Scaffold, σ a 130.79 0.44[0.06, 1.74] 1.0034543360 Benchmark, σ b 92.45 0.73[1.39, 4.21] 1.0037433783 Benchmark× scaffold, σ ba 210.69 0.36[0.06, 1.43] 1.0035803615 Benchmark× model, σ bm 1820.52 0.17[0.12, 0.80] 1.0017712110 Benchmark× model× scaffold, σ bma 2870.55 0.14[0.24, 0.79] 1.0015592062 Task within benchmark, σ bt 11172.41 0.09[2.25, 2.58] 1.0033093350 Task× scaffold in benchmark, σ bta 23941.03 0.05[0.93, 1.14] 1.0034453540 Task× model in benchmark, σ btm 188560.88 0.07[0.75, 1.01] 1.0025133241 Model, σ m 540.78 0.14[0.52, 1.07] 1.0031523550 Model× scaffold, σ ma 2090.36 0.14[0.05, 0.61] 1.0018522376 Population-level coefficient Intercept, μ— −1.95 0.88 [−3.62,−0.17] 1.0036933574 Note. “Levels” denotes the number of observed levels of each grouping factor. Estimate and SD are the posterior mean and posterior standard deviation, respectively. For group-level effects, the estimate summarizes the random-intercept standard deviationσ; for the intercept, it summarizes the coefficient itself. ESS denotes effective sample size. “Scaffold” corresponds to agent_name in the estimation data. Student -t(3, 0, 2.5)prior. These priors regularize components supported by few levels—notably benchmark and scaffold effects—without forcing them toward equality. Posterior sampling used Stan’s Hamiltonian Monte Carlo implementation throughbrms(Bürkner, 2021), with the conservative target average proposal acceptance probability during the adaptation period for the sampleradapt_delta = 0.95without any resultant divergent transitions. The full model used six chains of9,000iterations, including4,000warm-up iterations, with thinning by eight, yielding3,750retained draws. Each benchmark-specific model used four chains of2,000 iterations, including1,000warm-up iterations, with thinning by three. On an Apple M1 Max, the full fit required approximately5.5hours, while a benchmark-specific fit required approximately6.5 minutes on average. For the full model, all reported split- b R values rounded to1.00; bulk effective sample sizes for variance components ranged from1,559to3,743, and tail effective sample sizes ranged from2,062to3,783. Benchmark-specific fits were also inspected individually. For example, the largest split- b Rin the USACO fit was1.03, with uncertainty retained in all downstream posterior summaries rather than suppressed through plug-in estimates. Convergence. All parameters across all models fit and all estimates derived from posterior draws throughout this study achieve ˆ R < 1.05, the standard threshold for adequate mixing. For the 95% of estimates b Rvalues rounded to1.00. For the leaderboard-level model, complete convergence information is in Table 8 C.1.4LEAVE-ONE-BENCHMARK-OUT STABILITY To assess whether the leaderboard-level decomposition was dominated by any single instrument, we refit Equation 6 after removing each benchmark in turn. These are full Bayesian refits, not importance-sampling approximations. Each refit used four chains of4,000iterations, including2,000 warm-up iterations, with thinning by five, yielding1,600retained draws. Stability was evaluated by comparing the posterior distributions of variance components and derived reliability quantities with those from the complete-data fit. This analysis directly tests whether conclusions about model, scaffold, and benchmark contributions persist under changes to the benchmark ecosystem. 24 Under review as a conference paper C.2BAYESIAN ESTIMATION FOR LINEAR MIXED EFFECT MODELS To establish methodological sensitivity, Equation 6 was estimated using a Bayesian linear mixed effect model on the observation scale. Additional detail about the differences in methods and results can be found in Appendix G. All linear estimations of this use the same hyperparameters as above, except that the family is Gaussian with an identity link. C.3FREQUENTIST LINEAR MIXED EFFECTS (LME) We fit a Gaussian identity-link model via restricted maximum likelihood (REML). Although the binary outcome violates normality, LME provides a familiar baseline and is the most commonly used variance-component estimator in G-theory applications. Variance components are extracted directly from the REML fit. C.4FREQUENTIST GENERALIZED LINEAR MIXED EFFECTS (GLME) A logistic mixed-effects model with a Bernoulli likelihood and logit link is fit via Laplace approxima- tion to the marginal likelihood, with parameter estimation by penalized iteratively reweighted least squares (PIRLS). This respects the binary nature of the data but estimates are on the logit scale; we convert variance proportions by computing the share of total variance (including the logistic residual varianceπ 2 /3) attributable to each component. Both the Linear and Generalized Linear models were estimated using lme4 (Bates et al., 2015). C.5NONPARAMETRIC DISTANCE COMPONENTS (DISCO) We estimate a nonparametric variance decomposition (see Appendix G.2) using distance components (Rizzo & Székely, 2010; Székely & Rizzo, 2017) from the energy-statistics literature. DISCO decomposes total dispersion—measured by pairwise Euclidean distances—into between- and within- group components without distributional assumptions. For a single facet with K groups, S total = S between +S within ,(10) whereSdenotes the energy-based dispersion statistic. The G-coefficient analog isEρ 2 DISCO = S p /(S p +S within ) . DISCO captures nonlinear relationships and is robust to the heavy skewness observed in the Bayesian posteriors, particularly for facets with few levels (e.g., agents). However, it does not guarantee a positiveσ 2 ε /S global within term, so its reported “variance”/dispersion shares sum to one across the total explained dispersion. C.6DETAILS ON COMPUTE-USAGE Estimations of the main variance decomposition models used in the body of the paper took 8 total hours on an Apple M1 Max. LOO ablation studies in Appendix D took 57 hours. Methodological contrasts reported in Appendix G took 2 hours for frequentist estimations (both linear mixed effect models and generalized mixed effect models), Bayesian linear mixed effect models took 8 hours, and nonparametric distance components estimates took 18 hours. C.7POSTERIOR ESTIMANDS Table 9 contains the set of estimands found in the paper. DABLATIONS AND SENSITIVITY ANALYSES The benchmark ecosystem is sparse, unbalanced, and only partially connected. Tasks are nested within benchmarks, most model–scaffold combinations are absent, and many interaction levels are observed only once. Consequently, a fully crossed variance decomposition could in principle be driven by a small number of influential observations or by a particularly informative benchmark. We therefore examine robustness at four complementary levels: pointwise predictive stability, stability of latent rankings, leave-one-benchmark-out sensitivity, and leave-one-benchmark–scaffold-out sensitivity. 25 Under review as a conference paper Table 9: Reliability estimands. Each row appliesEρ 2 o = σ 2 o /(σ 2 o +σ 2 δ ),SNR o = σ 2 o /σ 2 δ , but changes the object whose ordering should generalize. The n f terms denote equal allocation over facet f . EstimandObject of inferenceσ 2 o σ 2 δ (D)Eq. Eρ 2 M(b) , SNR M(b) Does benchmark b preserve model order? σ 2 M σ 2 IM n i + σ 2 MA n a + σ 2 IMA,e n i n a (3) Eρ 2 MA(b) Does benchmark b preserve system order? σ 2 M + σ 2 A + σ 2 MA σ 2 IM + σ 2 IA + σ 2 IMA,e n i (4) ρ (b) A ′ Do scaffolds induce the same model order? σ 2 M + σ 2 IM n i σ 2 MA + σ 2 IMA,e n i (5) Eρ 2 M , SNR M Does the leaderboard preserve model order? σ 2 M σ 2 BM n b + σ 2 MA n a + σ 2 BMA n b n a + σ 2 IM[B] n b n i + σ 2 BIMA,e n b n i n a (7) lim n i →∞ Eρ 2 M(·) Does having unlimited tasks preserve model order? σ 2 M σ 2 MA n a or σ 2 BM n b + σ 2 MA n a + σ 2 BMA n b n a (8) ∆ρ 2 (M−A)(b) Do models have more benchmark- relevant signal than scaffolds? σ 2 M − σ 2 A σ 2 MA + σ 2 IM +σ 2 IA +σ 2 IMA,e n i (11) ∆ρ 2 I(M−A)(b) Do models have more task- level signal than scaffolds? σ 2 M + σ 2 IM n i − σ 2 A − σ 2 IA n i σ 2 MA + σ 2 IM +σ 2 IA +σ 2 IMA,e n i (12) ∆ρ 2 M−A Do models have more leaderboard- relevant signal than scaffolds? σ 2 M − σ 2 A σ 2 MA + σ 2 BM n b + σ 2 BA n b + σ 2 BMA n b + σ 2 IM[B] n b n i + σ 2 IA[B] n b n i + σ 2 BIMA,e n b n i (13) ∆ρ 2 B(M−A) Do models have more benchmark- level signal than scaffolds? σ 2 M + σ 2 BM n b − σ 2 A − σ 2 BA n b σ 2 MA + σ 2 BM n b + σ 2 BA n b + σ 2 BMA n b + σ 2 IM[B] n b n i + σ 2 IA[B] n b n i + σ 2 BIMA,e n b n i (14) ∆ρ 2 I[B](M−A) Do models have more task- level signal than scaffolds? σ 2 M + σ 2 BM n b + σ 2 IM n b n i − σ 2 A − σ 2 BA n b − σ 2 IA n b n i σ 2 MA + σ 2 BM n b + σ 2 BA n b + σ 2 BMA n b + σ 2 IM[B] n b n i + σ 2 IA[B] n b n i + σ 2 BIMA,e n b n i (15) D.1REFERENCE INDEXED VARIANCE DECOMPOSITION For observationn, letb[n],i[n],m[n], anda[n]denote its benchmark, task, model, and scaffold. The reference model with additional indexing from equation 6 is Y n ∼ Bernoulli(p n ), logit(p n ) = μ + u B b[n] + u I b[n],i[n] + u M m[n] + u A a[n] + u BM b[n],m[n] + u BA b[n],a[n] + u MA m[n],a[n] + u IM b[n],i[n],m[n] + u IA b[n],i[n],a[n] + u BMA b[n],m[n],a[n] , where every random effect is independently distributed asu F ℓ ∼N (0,σ 2 F ) for facet or interaction F. On the latent logistic scale, the observation-level residual variance isπ 2 /3. The reference fit containsN = 29,923binary observations, 54 models, 13 scaffolds, nine benchmarks, and 1,117 benchmark-specific tasks. The central object of measurement is the model. For a fixed evaluation designD, the relative generalizability coefficient has the schematic form from Equation 1 ρ 2 D = σ 2 M σ 2 M + σ 2 δ,D ,(16) whereσ 2 δ,D contains only those components that change model contrasts under repeated realizations ofD, divided by their effective replication counts. Components that shift all models equally under a shared condition do not enter the relative error variance. Cancellation of common condition effects. For two modelsmandm ′ evaluated under the same benchmark and scaffold, η bima − η bim ′ a eliminates the benchmark main effectu B b , scaffold main effectu A a , benchmark–scaffold effectu BA ba , task–scaffold effectu IA[B] bia , and task main effectu I bi . Thus, these components can affect absolute scores but not the ordering of models evaluated under identical conditions. This distinction is important below: instability in σ 2 A or σ 2 BA need not imply instability in relative model reliability. All posterior chains for the reference model mixed adequately: the reported variance parameters had b R = 1.00, with bulk effective sample sizes between 1,559 and 3,743. The following analyses address robustness to the data and specification rather than only Monte Carlo convergence. 26 Under review as a conference paper Table 10: Association between observed rankings and latent-effect rankings. “Full” uses the ecosys- tem model (Eq. 6); “Per-benchmark” uses an independently fitted model for each benchmark (Eq. 2) EstimatorMetricAssistantBenchCORE-HardGAIAMind2WebSciCodeScienceAgentBenchSWE-mini τ -benchUSACO FulldCor 2 n .404.899.780.872.703.651.833.828.430 Per-benchmarkdCor 2 n -.005.107.046.039.013.007.046.051.032 FullKendall τ.478.859.771.744.731.582.764.787.590 Per-benchmarkKendall τ.081.248.196.151.128.101.167.180.142 FullSpearman ρ.619.961.900.896.867.795.916.930.731 Per-benchmarkSpearman ρ.111.340.271.211.171.137.236.255.199 D.2SENSITIVITY OF MODEL RANKINGS Standard leaderboards rank systems by empirical mean accuracy, 14 ̄y bm = 1 N bm X n:b[n]=b,m[n]=m y n , possibly aggregating over different scaffolds and nonidentical task sets. Such rankings treat all observed variation as evidence about model capability. The mixed models instead estimate latent model effects after separating task difficulty, scaffold effects, and their interactions. For each benchmark, we formed rankings from posterior summaries of the relevant latent model effects. We considered two estimators: 1. Full-model latent ranking, obtained from the joint model in Equation 6, which partially pools information through models and scaffolds appearing elsewhere in the benchmark ecosystem. 2. Per-benchmark latent ranking, obtained by fitting, within each benchmark, logit(p ima ) = μ + u I i + u M m + u A a + u IM im + u IA ia + u MA ma . This estimator uses only the information available within one benchmark. These posterior conditional means are the Bayesian analogue of best linear unbiased predictors (BLUPs). We compared their induced rankings with the rankings from observed mean scores using Kendall’sτ, Spearman’sρ, and bias-corrected squared distance correlationdCor 2 n computed on rank vectors. Kendall’sτmeasures pairwise ordering agreement; Spearman’sρmeasures monotone rank association; and corrected squared distance correlation can detect more general dependence between the rank assignments. Two findings are salient. First, the jointly estimated rankings retain substantial agreement with observed leaderboards for CORE-Bench Hard, GAIA, Online-Mind2Web, SciCode, SWE-bench Verified Mini, andτ-Bench Airline. For example, full-model Spearman correlations are at least0.85 on these benchmarks. Their empirical rankings therefore contain recoverable cross-system signal even after nuisance variation is separated. Second, the independently fitted per-benchmark models exhibit weaker correspondence with raw rankings on every benchmark. This result is not evidence that the per-benchmark models are necessarily “less accurate.” Instead, it exposes an information and estimand mismatch. Within one sparse benchmark, model effects must be separated from model–scaffold and task–model interactions using few connected observations. Strong posterior shrinkage is therefore appropriate, but it compresses distinctions that empirical average accuracy treats as model signal. Moreover, the raw ranking may combine deployable model–scaffold performance, whereas the latent model ranking intentionally removes scaffold-specific contributions. The joint model can recover more stable model effects because shared models and scaffolds connect otherwise isolated benchmark-specific designs. In other words, the combined model, after accounting for sources of variation, provides more stable and reliable estimates by taking advantage of shared facet variation. Accordingly, rank correlation with the observed leaderboard is a sensitivity diagnostic, not a ground- truth accuracy measure. High agreement indicates that adjustment for known nuisance facets preserves 14 e.g., for HAL see “Accuracy” athttps://hal.cs.princeton.edu/reliability/for HELM see “mean score” at https://crfm.stanford.edu/helm/, etc. 27 Under review as a conference paper Table 11: PSIS-LOO diagnostics for the full Bayesian logistic mixed model. QuantityEstimateStandard error elpd loo −11,217.194.2 p loo 3,032.032.0 LOOIC22,434.2188.5 Pareto diagnosticCountPercentage b k ≤ 0.729,902> 99.9% 0.7 < b k ≤ 121< 0.1% b k > 100.0% the reported ordering. Low agreement indicates that the ordering depends on whether task and scaffold variation are treated as capability or as measurement error. It does not establish which ordering is externally valid without an independent criterion. D.3POINTWISE PREDICTIVE STABILITY WITH PSIS-LOO We first assessed whether individual observations exert disproportionate influence on the posterior predictive distribution. Exact leave-one-out cross-validation would requireNrefits. We instead used Pareto-smoothed importance-sampling leave-one-out cross-validation (PSIS-LOO), with moment matching for observations having unstable importance ratios (Vehtari et al., 2017; Paananen et al., 2021; Vehtari et al., 2024). For observation n, PSIS approximates p(y n | y −n ) = Z p(y n | θ)p(θ | y −n )dθ using draws from the full posteriorp(θ | y). The generalized Pareto shape diagnostic b k n measures the tail behavior of the resulting importance ratios. Values b k n ≤ 0.7generally indicate a reliable approximation, whereas larger values identify observations for which deleting the point substantially changes its posterior predictive distribution. The diagnostics in Table 11 are favorable for a model of this complexity. The effective number of parameters is substantially smaller than bothNand the nominal number of coefficients induced by the random effects, demonstrating strong regularization through partial pooling. No observation has b k > 1, and only 21 of 29,923 observations have b k > 0.7. The few warnings are explained by the connectivity of the design rather than by broad model failure. Approximately32%of observed benchmark–task–model groups are singletons, a known point of sensitivity for PSIS-LOO (Bindoff, 2026). Of the 21 observations with b k > 0.7, 19 (90.5%) belong to such singleton groups. The remaining two form the only observations in their group: SciCode task 14 evaluated with GPT-4.1 under two scaffolds. This behavior follows directly from the pointwise LOO target. Ify n is the only observation informing a random-effect levelu ℓ , deletion ofy n makes the leave-one-out distribution ofu ℓ approximately prior-predictive. The full-data posterior, by contrast, has adaptedu ℓ toy n . Importance sampling must therefore bridge two meaningfully different distributions, producing a heavy-tailed importance ratio. A large b k n in this setting primarily says that the outcome of an isolated, data-defined group cannot be predicted after removing its only observation. It does not, by itself, imply that population-level variance components are determined by that point. D.3.1OBSERVATION-LEVEL RANDOM-EFFECT ABLATION. As an additional check, we augmented Equation 6 with u IMA b,i,m,a ∼N (0,σ 2 IMA ). Because benchmark–task–model–scaffold cells generally lack within-cell replication in the HAL dataset, this term acts as an observation-level random effect (OLRE) (Harrison, 2015). In a Bernoulli- logit model, it competes with the fixed latent logistic residual rather than identifying a conventionally 28 Under review as a conference paper Prop. Var. Explained by AI ModelSignal-to-noise Estimate: 1 bench., 1 items each Ep2 Reliability Estimate: 1 bench., 1 items eachPer Task Var(model) - Var(scaffold) 0.000.050.100.150.00.10.20.30.40.5 0.00.10.20.3-0.4-0.20.0 0 5 10 0 3 6 9 12 0 5 10 0 10 20 30 40 Estimated Proportion of Variance Density bench_removed assistantbench corebench_hard gaia none onlinemind2web scicode scienceagentbench swebench_verified_mini taubench_airline usaco Figure 6: Posterior distributions for benchmark-level LOO ablations for the paper’s quantities of interest replicated three-way interaction. Including it did not materially change the reliability estimates: it absorbed variation previously treated as observation-level noise, and its contribution is attenuated by task replication in the D-study. The principal generalizability conclusions are therefore not an artifact of omitting the highest-order cell term. Importantly, this additional term should be used for studies where output stability is of interest where benchmark–task–model–scaffold levels have multiple runs. This variation can support measuring the stochasticity of task scores under the same conditions. D.4LEAVE-ONE-BENCHMARK-OUT SENSITIVITY Pointwise LOO evaluates interpolation within the observed ecosystem. A stronger test removes an entire benchmark and therefore deletes all of its tasks, score distribution, and benchmark-specific interactions. For each b ⋆ ∈B, we refitted the complete variance decomposition to D −b ⋆ =y n : b[n]̸= b ⋆ and recomputed the posterior variance decomposition and D-study reliability curves. This procedure asks whether conclusions about the benchmark battery are broadly distributed across benchmarks or are driven by a single test. The reliability estimations, proportions of variance attributable to models, and the principal interaction facets were generally stable across these refits as shown in Fig. 6. Sensitivity nevertheless depended on which benchmark was withheld. Removing CORE-Bench Hard caused the largest reduction in estimated reliability. This is substantively expected: CORE-Bench Hard supplies the most runs, spans the widest range of observed accuracies, and has the highest estimated signal-to-noise ratio among the included benchmarks. It therefore contributes unusually strong information both for distinguishing models and for connecting model performance to the rest of the ecosystem. Without CORE-Bench Hard, the posterior signal-to-noise ratio from the D-study no longer reaches the prespecified limit-of-detection band. This finding should not be read merely as “more observations are better.” A large but noisy benchmark can contribute little to the reliability of a battery. CORE-Bench Hard is influential because it combines replication with discrimination. Thus, increasing the number of benchmarks is not equivalent to increasing effective measurement information. More generally, for a battery aggregating conditionally independent benchmark-level measurements with signalS b and errorE b , the information supplied by benchmarkbis governed by its discrimination relative to error, not simply its task count. Although the fully crossed design includes interactions and is more complicated than this schematic case, the same principle applies: benchmarks with substantial model variation and controlled task- and harness-dependent error contribute disproportionately to stable ecosystem-level rankings. 29 Under review as a conference paper Table 12: Leave-one-benchmark–scaffold-out proportions of variance. SE is the standard error across refits, and CV is the coefficient of variation. FacetMeanMedianMin.Max.SDSECV Residual.237.235.226.267.008.002.035 Scaffold (A).020.019.008.037.007.002.338 Benchmark (B).258.259.229.298.015.003.060 Benchmark–scaffold (BA).026.028.000.035.009.002.345 Benchmark–model (BM ).020.020.009.031.004.001.208 Benchmark–model–scaffold (BMA).014.013.010.019.003.001.180 Task within benchmark (I ).328.329.282.365.015.003.047 Task–scaffold (IA).052.053.028.066.009.002.168 Task–model (IM ).003.004.000.006.001 < .001.426 Model (M ).033.033.026.044.004.001.110 Model–scaffold (MA).008.009.003.012.002 < .001.249 This ablation also clarifies the intended scope of generalization. The leave-one-benchmark-out results do not claim that the fitted model can predict an arbitrary future benchmark with no shared structure. Rather, they test whether the estimated variance decomposition and design recommendations survive removal of one observed measurement instrument. The sensitivity to CORE-Bench Hard indicates that the present nine-benchmark ecosystem contains limited redundancy at the high-signal end. D.5LEAVE-ONE-BENCHMARK–SCAFFOLD-OUT SENSITIVITY Because scaffold coverage is also unbalanced, we conducted a complementary sensitivity analysis that removed one observed benchmark–scaffold combination at a time and re-estimated the full decomposition. Table 12 summarizes the resulting frequentist proportions of total variance. This ablation is especially stringent for benchmark-specific scaffolds, whose removal can eliminate nearly all direct information about a scaffold level. Table 13: Leave-one-LLM-out proportions of variance. SD is the standard deviation across refits, SE is the standard error across refits, and CV is the coefficient of variation. FacetMeanMedianMin.Max.SDSECV Residual0.2360.2360.2310.2410.00200.008 Scaffold (A)0.0200.0200.0160.0290.00200.098 Benchmark (B)0.2590.2590.2520.2680.00200.009 Benchmark–scaffold (BA)0.0260.0260.0220.0300.00100.056 Benchmark–model (BM )0.0200.0200.0150.0220.00100.065 Benchmark–model–scaffold (BMA)0.0140.0140.0080.0160.00100.092 Task within benchmark (I )0.3280.3280.3200.3340.00200.006 Task–scaffold (IA)0.0520.0520.0490.0530.00100.018 Task–model (IM )0.0040.0040.0010.0040.00100.166 Model (M )0.0330.0330.0280.0360.00200.048 Model–scaffold (MA)0.0090.0090.0060.0120.00100.107 The dominant components are stable. Task difficulty accounts for approximately 32.8% of variance across refits, benchmark differences for25.8%, and residual variation for23.7%. Their coefficients of variation are only0.047,0.060, and0.035, respectively. The model component is also stable in absolute terms, ranging from 0.026 to 0.044. The largest relative variation occurs for the scaffold main effect, the benchmark–scaffold interaction, and the task–model interaction. These cases require different interpretations. The first two are weakly identified because most scaffolds occur in only one benchmark; removing a benchmark–scaffold cell can therefore remove much of the relevant connectivity. However, common scaffold and benchmark– scaffold shifts cancel from same-condition model contrasts and consequently do not enter the relative model reliability estimates used in the primary D-study. 30 Under review as a conference paper The task–model component has the largest coefficient of variation (0.426), but its estimated proportion is always between0and0.006. Its high relative variability is therefore primarily a small-denominator effect: even its maximum is smaller than the typical contribution of every other reported component. Coefficients of variation should not be interpreted without the corresponding absolute scale. Addi- tionally, both this component and the second largest CV value, the benchmark–scaffold component, are the only components that contain minimum values equal to zero; these sensitivities, paired with extreme ablation minima, are likely elevated due to the nature of frequentist estimations, which can result in singular values in highly unbalanced designs (see Appendix G.3). These estimates may be more stable under a full Bayesian ablation where CV estimates would be less likely to decrease the mean in the denominator. The model-related components that directly govern ranking stability remain comparatively well behaved. The model proportion has CV0.110, while the benchmark–model and model–scaffold components remain small across all deletions. Hence, no individual benchmark–scaffold cell ap- pears to create the principal conclusion that bare-model rankings are less reliable than rankings would appear under a decomposition that treats all observed model–scaffold performance as signal. However, it is worth noting that removing AssistantBench during the LOO ablation disconnects the OnlineMind2Web scaffold from the broader network, which means that, for that connection, identifiability conditions found in Proposition E.3 would not be complete. Nevertheless, the estimates remain stable, even with this disconnection. Importantly, the contribution to overall reliability of CORE-Bench Hard discussed in § D.4 can be narrowed further to the contribution of the CORE Agent, which shows the most between-LLM discrimination on its benchmark. Removing this particular benchmark–scaffold accounts for the large increase in uncertainty across the pooled model. As this scaffold is less connected than the HAL Generalist scaffold, this reaffirms that having a strong discriminating signal is critical to pooled reliability, potentially more than large connectivity. D.6LEAVE-ONE-LLM-OUT SENSITIVITY We next test whether the estimated measurement structure is driven by a single LLM and whether the variance decomposition generalizes to new LLMs. For each LLM modelm ⋆ ∈M, we removed all of its observations, D −m ⋆ =y n : m[n]̸= m ⋆ , and refit the variance-decomposition model. This is a stronger perturbation than deleting one response: it simultaneously removes a model main-effect level and every observed benchmark–model, model–scaffold, task–model, and benchmark–model–scaffold cell involving that LLM. Because evaluation coverage differs substantially across models—only nine of the 54 models appear in every benchmark—the ablation also tests sensitivity to the most highly connected models in the design. The LOO ablations for this and those of Appendix D.5 were conducted in a frequentist framework to reduce practical computation costs of hundreds of Bayesian refittings (see Appendix C.6). Table 13 shows that the decomposition is remarkably insensitive to the removal of any LLM. The three largest components are almost invariant: task-within-benchmark variance has mean proportion 0.328, range0.320–0.334, and CV0.006; residual variance has mean0.236, range0.231–0.241, and CV0.008; and benchmark variance has mean0.259, range0.252–0.268, and CV0.009. Thus, the conclusion that variation is dominated by differences among tasks and benchmarks, together with substantial unexplained response variation, is not attributable to the performance profile of a particular model. More importantly for relative reliability, the model variance is also stable. Its proportion remains between0.028and0.036, with mean0.033and CV0.048. The benchmark–model component ranges from0.015to0.022, while the benchmark–model–scaffold component ranges from0.008to 0.016. Removing even a highly connected or unusually capable LLM therefore does not qualitatively change the estimated amount of systematic between-model variation or the extent to which model performance depends on the benchmark and scaffold. As in the benchmark–scaffold ablation, the largest relative variability occurs in small components. The task–model interaction has CV0.166, but its variance share is only0.001–0.004. The model– scaffold interaction has CV0.107and remains between0.006and0.012. Their relative sensitivity 31 Under review as a conference paper should consequently not be confused with a large contribution to total variance. In absolute terms, deletion-induced changes in both components are small. The leave-one-model results are also more stable than the leave-one-benchmark results. This asym- metry reflects the structure of the available evidence. Each LLM contributes another sample from the population of systems, and its outcomes are partially pooled with those of the remaining 53 models. By contrast, removing a benchmark deletes an entire measurement instrument, all of its unique tasks, and its characteristic signal-to-noise ratio. The ecosystem therefore has greater redundancy across models than across high-quality benchmarks. Adding another LLM primarily improves estimation of the distribution of model capability, whereas adding a discriminating benchmark can alter the quality of the measurement battery itself. Implication for reliability. For a fixed D-study design, relative reliability depends on the model variance and on model-dependent error components such asBM,MA,IM, andBMA, after scaling by their effective replication counts. The stability of these components under model deletion implies that the reported reliability conclusions are not generated by one extreme or unusually well-connected LLM. Nevertheless, this ablation evaluates influence on the population variance decomposition, not the ability to predict the performance or rank of a previously unseen model. Generalization to a new LLM additionally requires that it be exchangeable with the sampled model population and evaluated on conditions that connect it to the existing design. No single LLM acts as a leverage point for the principal variance decomposition. The greater sensitivity to removing an informative benchmark than to removing an LLM reinforces a central design recommendation: once a reasonably diverse model sample has been obtained, additional evaluation resources may yield greater reliability gains by improving benchmark quality, task replication, and cross-scaffold connectivity than by adding sparsely evaluated models. D.7METHODOLOGICAL ROBUSTNESS We also estimate each model using five different approaches. A separate Appendix G explains and discusses each method and reports the full results. D.8RANK STABILITY ACROSS POSTERIOR Ranking models within each draw yields posterior rank distributions and pairwise ordering proba- bilitiesPr(θ mb > θ m ′ b | y).These quantities distinguish an estimated ordering from evidence that two models are meaningfully distinguishable. We compare posterior and published rankings using Spearman correlation and ranked unbiased squared distance correlation,dCor 2 n ; the latter remains informative in the presence of extensive ties. We compare posterior ranks with published ranks using Spearman correlation and ranked unbiased squared distance correlation, dCor 2 n (Table 14). The latter is useful for benchmarks with extensive ties and provides a direct diagnostic of whether estimated capability ranking is statistically associated with the reported ordering. Posterior capability estimates differ from ranks obtained by sorting raw percent-correct scores. Raw scores credit the model for every condition with which it happens to be paired. The pooled decomposition instead estimates the portion of performance that persists after averaging over the specified benchmark and scaffold universes. Table 14: Median correlation of reported and estimated ranks across posterior draws Corr.AssistantBenchCORE-BenchGAIAMind2webSciCodeSci.AgentBenchSWEbench τ -benchUSACO Spearman0.4420.8240.790.6210.6620.6110.8080.6830.692 dCor 2 n 0.1410.640.5760.330.360.3230.6330.4080.395 D.9IMPLICATIONS FOR BENCHMARK DESIGN These checks support five practical conclusions. 32 Under review as a conference paper 1.Sparse cells principally limit local prediction. PSIS warnings are almost entirely confined to singleton or doubleton interaction levels. The model cannot predict an isolated cell after its sole observation is removed, but the population-level variance decomposition is stable to these observations. 2.Partial pooling is necessary for ecosystem-level ranking. Per-benchmark data are gener- ally insufficient to cleanly distinguish model capability from scaffold and task interactions. Cross-benchmark connectivity substantially stabilizes latent model rankings, although the resulting rankings remain uncertain on low-signal benchmarks. 3.Benchmark quality is not interchangeable with benchmark quantity. The leave-one- benchmark-out analysis identifies CORE-Bench Hard as a high-information anchor. A useful test battery should include multiple independently constructed benchmarks with high discrimination and controlled error, rather than merely adding more noisy tasks or near-duplicate benchmarks. 4.Absolute-score instability and relative-rank instability are distinct. Scaffold and benchmark–scaffold effects can materially shift reported accuracies while canceling from same-condition model comparisons. Benchmark reports should therefore state whether reliability concerns absolute deployment performance, model ranking, or model–scaffold system ranking; these are different objects of measurement and induce different error terms. 5.Representative LLM panel improves benchmark diagnostics. once a reasonably diverse model sample has been obtained, additional evaluation resources may yield greater reliability gains by improving benchmark quality, task replication, and cross-scaffold connectivity than by adding sparsely evaluated models. The robustness analyses do not imply that the sparse design is harmless. Rather, they localize its con- sequences. The main variance and reliability conclusions are not driven by a handful of observations or benchmark–scaffold cells, but the precision of individual rankings remains strongly dependent on cross-benchmark connectivity and on the inclusion of at least one high-signal measurement instrument. This distinction is essential for designing future agentic benchmark batteries: additional evaluation should be allocated to conditions that improve connectivity and reduce model-dependent error, not only to increasing the nominal number of tasks or sparsely connecting models. EIDENTIFIABILITY AND CONNECTIVITY OF THE LEADERBOARD DECOMPOSITION The leaderboard data are sparse by construction: models are submitted selectively, scaffolds are not used uniformly, and tasks are unique to benchmarks. Such missingness is common in public leaderboards (Singh et al., 2026). A fully crossed experiment would simplify estimation, but it is not necessary for the questions studied here. What is required is sufficient overlap to distinguish persistent model variation from variation associated with benchmarks, scaffolds, and their interactions. This appendix establishes that the observed design provides the connectivity needed for those contrasts. We distinguish three concepts that are often conflated: 1. Connectivity: whether observed model, benchmark, and scaffold levels belong to a common comparison network. 2.Structural identifiability: whether distinct variance components imply distinct distributions over the observed responses. 3. Practical estimability: whether the finite data determine those components precisely. Connectivity and structural identifiability are design properties; practical estimability also depends on sample size, outcome variation, and prior regularization. Our Bayesian estimator can produce a proper posterior for a weakly informed component, but a proper posterior alone is not evidence that the component is strongly data-identified. We therefore use posterior uncertainty and sensitivity analyses, rather than existence of an estimate, to characterize practical estimability. 33 Under review as a conference paper E.1INCIDENCE-GRAPH CONNECTIVITY AND IDENTIFIABILITY Represent the observed design as a multipartite incidence graph whose vertices are benchmarks, mod- els, and scaffolds, with edges induced by observed evaluations. Shared models and scaffolds connect benchmark-specific observations and support estimation of common variance components. Con- trasts within a connected component are informed by observed paths; contrasts across disconnected components are not identified without additional assumptions. We report component membership, articulation vertices, benchmark degrees, and the change in connectivity produced by removing each benchmark. Posterior regularization stabilizes weakly supported components but does not create evidence for disconnected contrasts. E.2OBSERVED INCIDENCE STRUCTURE The full dataset contains nine benchmarks, 54 LLMs, and 13 scaffolds. Nine models appear on every benchmark and span at least69%of the scaffold set. One scaffold is shared by eight benchmarks. AssistantBench connects the remaining benchmark through an additional shared scaffold, and every other benchmark contains at least one additional benchmark-specific scaffold. Models and scaffolds are consequently neither fully crossed nor evenly replicated. Let Ω =(b,i,m,a) : y bima is observed denote the observed response cells. The pooled decomposition is equation 6: η bima = β 0 + u (B) b + u (I[B]) bi + u (M) m + u (A) a + u (BM) bm + u (BA) ba + u (MA) ma + u (IM[B]) bim + u (IA[B]) bia + u (BMA) bma ,(b,i,m,a)∈ Ω. Each random effect has mean zero and a component-specific variance. The Bernoulli–logit likelihood fixes the latent scale through the standard logistic residual variance π 2 /3. It is useful to represent Ω as a multipartite incidence graph. Let G = (V,E), V =B∪M∪A, where an edge joins two levels when they co-occur in at least one observed response. For example, m–bis an edge if modelmis evaluated on benchmarkb, anda–bis an edge if scaffoldais used on benchmarkb. Items need not connect across benchmarks because they are intentionally nested within benchmark. The nine models observed on every benchmark form model-side anchors. The scaffold used on eight benchmarks forms a scaffold-side anchor, while the second shared scaffold connects AssistantBench to the rest of the graph. Benchmark-specific scaffolds are leaves or local branches attached to this connected core. Thus, all benchmarks and their associated observations belong to one comparison network. Lemma 1 (connected additive contrasts). Lemma E.1 (connected additive contrasts). Consider an additive model on an observed incidence graph, η ma = μ + α m + γ a , with one centering constraint per facet. If the model–scaffold incidence graph is connected, all estimable contrasts α m − α m ′ and γ a − γ a ′ are identified. Proof. Suppose two parameterizations produce the same linear predictor on every observed edge. Their differences satisfy∆α m + ∆γ a = 0on each edge. Along any path in a connected bipartite graph, these equalities imply that all model differences equal a common constant and all scaffold differences equal its negative. Centering removes this remaining additive degree of freedom. Lemma E.1 concerns fixed additive effects, whereas Equation 6 uses random effects and interactions. It nevertheless provides the relevant intuition: an observation from a model or scaffold that is disconnected from the remainder of the leaderboard cannot support a common ranking. Shared models and scaffolds create paths along which relative effects can be compared. 34 Under review as a conference paper E.3IDENTIFIABILITY OF RANDOM-EFFECT VARIANCES For observations ordered as a vector, write the latent linear predictor as η = β 0 1 + K X k=1 Z k u k , u k ∼N (0,σ 2 k I), whereZ k is the incidence matrix for componentk. On the latent scale, the random effects induce covariance Cov(η) = K X k=1 σ 2 k K k , K k = Z k Z ⊤ k .(17) The entries ofK k indicate which pairs of observations share a level of facet or interactionk. For example, two observations share the model kernelK M when they use the same model, and share K BM only when they use both the same benchmark and model. Proposition E.2 (variance-component criterion). A collection of latent variance componentsσ 2 k : k ∈Kis structurally identifiable from the observed design if its covariance kernelsK k : k ∈K, restricted toΩ, are linearly independent after removing components that are deterministically confounded with the terminal residual. Proof. If the restricted kernels are linearly independent, equality of two induced covariance matrices implies X k∈K (σ 2 k − eσ 2 k )K k = 0, which has only the trivial solutionσ 2 k = eσ 2 k for everyk. If the kernels are linearly dependent, a nonzero perturbation of their coefficients leaves the induced covariance unchanged, so the corresponding components cannot be separated from the observed design. For a Bernoulli GLMM, Proposition E.2 is most directly interpreted as a design criterion on the latent scale. The nonlinear likelihood can affect the amount of information, especially under floor effects, but it cannot create distinctions absent from the incidence matrices. A practical interpretation is the following: to distinguish two variance components, the design must contain pairs of observations that share the grouping represented by one component without always sharing the grouping represented by the other. Complete crossing is sufficient for this condition but is not necessary. E.4HOW THE OBSERVED DESIGN SEPARATES THE REQUIRED COMPONENTS Table 15 summarizes the principal replication patterns. These conditions concern variance compo- nents, not estimation of every individual random-effect realization. E.4.1MODEL AND BENCHMARK–MODEL VARIATION The distinction between persistent model capability and benchmark-specific performance is central to the leaderboard reliability coefficient. If every model appeared on only one benchmark, thenu (M) m andu (BM) bm would be inseparable: “model” would be nested within benchmark. That failure does not occur here. Nine models appear on every benchmark, producing observations that shareMwhile differing in BM . Consequently, the model kernelK M and benchmark–model kernelK BM have different support. The former links observations from the same model across benchmarks; the latter links them only within a benchmark. This overlap identifies the contrast between globally persistent model variationσ 2 M and benchmark-conditioned model variation σ 2 BM . 35 Under review as a conference paper Table 15: Observed comparisons supporting the pooled variance decomposition. “Separation” identifies the principal pair of components distinguished by each overlap pattern. SeparationRequired comparisonSupport in the observed design M versus BMThe same model appears on multiple benchmarks. Nine models appear on all nine benchmarks; additional models pro- vide partial cross-benchmark replica- tion. A versus BA The same scaffold appears on multiple benchmarks. One scaffold spans eight bench- marks; an additional shared scaffold connects AssistantBench. MA versus BMAThe same model–scaffold pair recurs across benchmarks. Cross-benchmark models evaluated through shared scaffolds create re- peated model–scaffold pairs. M versus MAModels are observed under multiple scaffolds, with overlap across models. The cross-benchmark models span at least nine of the 13 scaffolds, and shared scaffolds provide common comparison conditions. BM versus BMAWithin a benchmark, models are ob- served under more than one scaffold. Each benchmark contains shared or locally replicated scaffold conditions in addition to benchmark-specific scaffolds. I[B] versus IM [B]Each item is attempted by multiple models. Benchmark items are repeatedly scored across leaderboard models. I[B] versus IA[B]Each item is attempted under multiple scaffold conditions. Scaffold replication within bench- marks provides item–scaffold con- trasts where observed. E.4.2SCAFFOLD AND BENCHMARK–SCAFFOLD VARIATION Benchmark-specific scaffolds alone would not distinguish a scaffold main effect from a benchmark– scaffold interaction. For a scaffold used on exactly one benchmark, itsAandBAcolumns coincide. The design avoids complete confounding because one scaffold spans eight benchmarks and a second shared scaffold connects AssistantBench. Observations using a shared scaffold have the sameAlevel but different BA levels, providing the comparisons required to distinguish σ 2 A from σ 2 BA . Benchmark-specific scaffolds remain less individually informed than shared scaffolds. Their effects are estimated through the exchangeability assumptions of the hierarchical model, with uncertainty propagated into the posterior. The shared scaffolds identify the population-level separation; partial pooling regularizes levels with little direct replication. E.4.3MODEL–SCAFFOLD AND BENCHMARK–MODEL–SCAFFOLD VARIATION SeparatingMAfromBMArequires recurrence of a model–scaffold pair across benchmarks. The cross-benchmark models and shared scaffolds provide such recurrence: observations can share a model and scaffold while differing in benchmark. Within-benchmark scaffold variation then supplies the complementary comparisons needed to estimate how model–scaffold compatibility changes by benchmark. This component is more weakly informed than the model and benchmark–model components because repeated model–scaffold pairs are less frequent than repeated models. Our estimands therefore integrate over its posterior uncertainty. The leave-one-benchmark-out and alternative-estimator analyses in Appendix D assess whether substantive conclusions depend on a small number of these bridges. E.4.4ITEM-INDEXED VARIATION Items are nested within benchmarks and are not expected to recur across benchmarks. Their main effects are identified because the same item is attempted by multiple models and scaffold conditions. Item–model variation is informed when a benchmark item is attempted by multiple models, while item–scaffold variation is informed by scaffold replication within benchmark. 36 Under review as a conference paper No assumption is made that an item from one benchmark is exchangeable with an identically labeled item from another benchmark. Pooling occurs at the variance-component level: the model estimates the typical magnitude of item-conditioned interactions across the observed benchmark panel. E.5WHAT IS NOT SEPARATELY IDENTIFIABLE The design does not contain independent repeated executions of every(b,i,m,a)cell. A four-way benchmark–item–model–scaffold interaction is therefore observationally confounded with cell-level execution variation and the Bernoulli residual for most of such interactions. We combine these sources into the terminal component σ 2 BIMA,e . This is not a limitation for the reported relative reliability coefficients: all unresolved cell-specific variation belongs in the error term because it can change model ordering across repeated evaluation conditions. Separating execution stochasticity, grading instability, and four-way interaction would require repeated runs under the same recorded cell and, ideally, explicit grading replicates. More generally, the present design does not support: • an unregularized estimate for every unobserved model–scaffold cell; • precise scaffold-specific effects for scaffolds used on only one benchmark; • causal claims about replacing one scaffold with another; or • generalization to scaffold or benchmark populations unrelated to the connected evaluation panel. These quantities are unnecessary for our research questions. We require population-level variance components and scaffold-marginalized model contrasts, not predictions for every missing factorial cell. E.6IDENTIFIABILITY OF THE RELIABILITY ESTIMANDS The principal leaderboard coefficient is Equation 7: Eρ 2 M (n b ,n i ,n a ) = σ 2 M σ 2 M + σ 2 BM n b + σ 2 MA n a + σ 2 BMA n b n a + σ 2 IM[B] n b n i + σ 2 BIMA,e n b n i n a . This coefficient depends on a small set of variance components and their sums, not on every random effect in Equation 6. The design requirements are correspondingly weaker than those needed to reconstruct the complete model–benchmark–scaffold response tensor. Proposition E.3 (sufficiency for the leaderboard estimand). Suppose that: (i) the benchmark–model incidence graph is connected and contains models observed on multiple benchmarks; (i)the benchmark–scaffold incidence graph is connected and contains scaffolds observed on multiple benchmarks; (i) at least some model–scaffold pairs recur across benchmarks; and (iv) benchmark items are attempted by multiple models and scaffolds. Then the covariance kernels corresponding toM,BM,MA,BMA, andIM [B]have distinct ob- served replication patterns. The variance combinations entering Equation 7 are therefore structurally distinguishable, up to the explicitly combined terminal component σ 2 BIMA,e . Justification Conditions (i)–(iv) provide observation pairs that share, respectively:Mbut notBM; Abut notBA;MAbut notBMA; andIM [B]and not onlyI[B]. These yield distinct covariance kernels under Proposition E.2. The cell-specific remainder is intentionally aggregated because no additional replication distinguishes its constituents. □ The observed leaderboard satisfies these conditions. In particular, the nine cross-benchmark models identify persistent versus benchmark-conditioned model variation, while shared scaffolds and repeated model–scaffold pairs identify the scaffold-related components that determine the task-only reliability ceiling. Corollary E.4 (identifiability of D-study ceilings). Under Proposition E.3, the task-only limit lim n i →∞ Eρ 2 M (n b ,n i ,n a ) = σ 2 M σ 2 M + σ 2 BM /n b + σ 2 MA /n a + σ 2 BMA /(n b n a ) 37 Under review as a conference paper is identified without separately decomposing terminal item-level residual variation. Corollary E.4 is important for the paper’s central design conclusion. The claim that adding tasks can- not eliminate benchmark- and scaffold-conditioned error does not depend on a precise decomposition of every item-level noise source. It depends on the persistent components whose identification is supported by cross-benchmark models and scaffolds. E.7BAYESIAN REGULARIZATION AND EVIDENTIAL SCOPE All variance components are estimated with proper, weakly informative priors. The posterior is therefore proper even when a component is weakly informed near the boundary. We use priors to stabilize finite-sample estimation, not to substitute for disconnected comparisons. Three features constrain interpretation: 1.Data-supported contrasts. Model rankings are inferred through the connected observation graph. We do not report comparisons between disconnected components because none exist in the observed design. 2. Posterior rather than plug-in reliability. Equation 7 is evaluated within each posterior draw. Weak separation among related components therefore appears as uncertainty in reliability and its D-study projection. 3. Sensitivity to bridges. Leave-one-benchmark-out refits test whether conclusions rely on a particular benchmark or scaffold connection. Prior and estimator comparisons test whether weak components are driving the reported variance ordering. These analyses are reported in Appendix D. The distinction between identifiability and precision is especially important for agent leaderboards. Sparse designs can be connected enough to estimate whether scaffold variation is non-negligible while still being too sparse to estimate its exact magnitude narrowly. Large credible intervals in this setting are not model failure; they reveal that the leaderboard contains too few common evaluation conditions to support precise claims. E.8CONNECTIVITY SUMMARY The full design is incomplete but connected. Nine models evaluated across every benchmark anchor the model scale; shared scaffolds connect the benchmark panel; and recurring model–scaffold pairs distinguish general scaffold compatibility from benchmark-specific compatibility. These overlaps are sufficient for the variance combinations underlying model-ranking reliability, signal-to-noise ratios, scaffold-versus-model comparisons, and D-study ceilings. The design does not identify every possible cell effect, nor is that required. Unreplicated terminal variation is assigned to relative error, and uncertainty from sparsely observed scaffold interactions is propagated through the Bayesian posterior. The resulting claims are therefore appropriately scoped: they characterize the reliability supported by this connected leaderboard ecosystem, rather than asserting a complete factorial decomposition of all possible models, scaffolds, tasks, and benchmarks. FADDITIONAL TABLES AND FIGURES The coefficients of Table 1 isolate where a benchmark’s signal lives and which facet dissipates it. Because each coefficient shares a common numerator–denominator structure (signal variance over signal plus generalizing error), differences across coefficients within a single benchmark are directly interpretable: they reveal how much of the apparent capability signal survives when we generalize over tasks, over scaffolds, or over both. Table 17 reports posterior means from the per-benchmark fit of Eq. 2. F.1REPEATED TASK SUBSAMPLING We confirm the cost reduction findings by repeatedly samplingktasks per benchmark without replacement, using the same sampled tasks for all models to ensure a paired comparison. Within 38 Under review as a conference paper Table 16: Reliability depends on the object being ranked. Reliability estimates for each benchmark under model and model–scaffold systems as measurement systems, including the task limit for models as the object of measurement. EstimandASSISTANTCORE-HARDGAIAMIND2WEBSCICODESCIENCEAGENTSWE-MINI τ -AIRLINEUSACO lim n i →∞ Eρ 2 M (8)0.2100.8920.3250.1530.8400.6830.5520.6690.377 Eρ 2 M (3)0.1820.8410.3180.1480.8110.6280.5430.5720.375 Eρ 2 MA (4)0.9350.9720.9900.9740.9750.9870.9930.9440.994 assistantbenchcorebench_hardgaiaonlinemind2webscicodescienceagentbenchswebench_verified_minitaubench_airlineusaco 1310301310301101001101001310 30110100131030131030110100 0 1 2 3 4 0 1 2 3 0 2 4 6 0 2 4 6 0 2 4 6 0 1 2 3 0 1 2 3 0 2 4 6 0 1 2 3 Number of Agentic Tasks Signal-to-Noise Ratio: S/N( δ ) Figure 7: Benchmarks differ sharply in ranking signal. Posterior model-ranking signal-to-noise ratio as tasks are added. Shaded bands give external detection-limit reference values. Points and intervals are posterior medians and 68% standard error HDIs. each subsample, scores are first averaged at the benchmark–model level and then averaged across benchmarks, giving each benchmark equal weight. The resulting model ranking is compared with the ranking obtained from the complete dataset using Spearman’s rank correlation coefficient (ρ) and Kendall’s rank correlation coefficient (τ). Repeating this procedure 500 times across many random subsamples yields a distribution of rank correlations for eachk, quantifying how stable the model rankings are with respect to the number and selection of evaluated tasks. Task subsampling corroborates the diminishing returns. Across500paired subsamples,15tasks per benchmark retain mean Spearman agreement of0.932with the complete-data ranking at an estimated82% cost reduction;30tasks increase agreement to0.976Agreement with the full ranking establishes information retention, but makes not claim to ranking validity. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 assistantbenchcorebench_hardgaiaonlinemind2webscicodescienceagentbench swebench_verified_mini taubench_airlineusaco G-coef fi cient Object of Measurement agentmodelmodel_agent Figure 8: Reliability by Object of Measurement across all tasks in a benchmark. Estimated using variance components of Equation 2, and posteriors of Equations 3 and 4. Intervals are pseudo-standard error at 0.68 HDI, bar estimates are medians and points are means of posterior draw distributions. 39 Under review as a conference paper Table 17: Posterior-mean reliability and generalizability coefficients per benchmark.Eρ 2 M(i) and Eρ 2 A(i) are task reliabilities (σ 2 o /(σ 2 o + σ 2 io ) ) for model and scaffold rankings;ρ (b) A ′ is inter-scaffold reliability (Eq. 5);Eρ 2 M(b) ,Eρ 2 A(b) , andEρ 2 MA(b) are G-coefficients by object of measurement (Eqs. 3–4); the last column is the task-saturated model ceiling lim n i →∞ Eρ 2 M(b) (Eq. 8). BenchmarkEρ 2 M(i) Eρ 2 A(i) ρ (b) A ′ Eρ 2 M(b) Eρ 2 A(b) Eρ 2 MA(b) Eρ 2 M(b) (∞) scicode0.6020.5340.7460.7360.5160.9710.765 gaia0.2630.7050.3330.3280.5790.9900.334 taubench_airline0.5490.5390.5450.5330.7590.9380.619 swebench_verified_mini0.5990.8570.5040.4990.6390.9920.506 usaco0.2380.2530.3920.4290.4780.9930.432 assistantbench0.3620.4450.2870.2600.5210.9110.305 corebench_hard0.6360.9410.8280.8170.8050.9720.866 onlinemind2web0.2960.2280.2050.2030.4300.9720.210 scienceagentbench0.5580.7530.5670.5610.8810.9820.610 Table 18: Internal Score Transportability. Kendall’s tau correlation matrix of benchmarks and summary statistics against each benchmark’s mean accuracy, the unweighted HAL mean score, and reliability-adjusted latent ( b θ) model effect (median posterior) ASSISTANTCORE-HARDGAIAMIND2WEBSCICODESCIENCEAGENTSWE-MINI τ -AIRLINEUSACOMEAN SCORE b θ assistantbench1.00.3000.485-0.0610.1030.215-0.4040.1800.2150.2050.261 corebench_hard0.3001.00.4870.4000.4920.4580.1480.5810.1670.6870.835 gaia0.4850.4871.00.0280.2600.4790.2150.3630.1820.4530.606 onlinemind2web-0.0610.4000.0281.00.1710.0560.4240.315-0.1410.5810.348 scicode0.1030.4920.2600.1711.00.1840.1170.4770.2250.2020.472 scienceagentbench0.2150.4580.4790.0560.1841.00.1290.587-0.0850.3920.499 swebench_verified_mini-0.4040.1480.2150.4240.1170.1291.00.0230.0320.5090.554 taubench_airline0.1800.5810.3630.3150.4770.5870.0231.00.0730.4040.588 usaco0.2150.1670.182-0.1410.225-0.0850.0320.0731.00.3900.338 Table 19: Task subsampling shows rank information preservation. Bootstrapped correlations of subsampled tasks and leaderboard Ranks kmean ρmedian ρminρmaxmean τmedian τminτmaxtop@1 agreementmean abs rank changecost savings 50.7900.8010.5840.9170.6340.6400.4580.7660.6407.2210.94 100.8910.8960.8040.9520.7420.7450.6460.8290.7565.1100.88 150.9320.9360.8700.9660.7990.8020.7200.8580.8344.0520.82 200.9540.9570.9150.9780.8370.8390.7780.8870.9123.3230.76 250.9670.9690.9400.9820.8630.8650.8110.9010.9602.8220.69 300.9760.9770.9570.9880.8860.8890.8420.9230.9902.3850.63 GMETHOD AND SCALE COMPARISON G.1LATENT SPACE ESTIMATIONS Variance components in our generalized linear mixed model (GLMM) decompositions live on the latent (logit) scale, soρ 2 should be interpreted as reliability of the linear predictorη(i.e., of differences in log-odds) rather than as reliability of raw percent-correct. This is standard in binary-response measurement: the latent scale is the scale on which additive random effects and G-theory D-study algebra apply, and it is the scale on which “true score + error” is well-defined without probability- dependent heteroskedasticity. Importantly, a given amount of latent variability implies different variability in percent-correct depending on where a model sits on the sigmoid: nearp≈ 0.5, small changes inηtranslate to large changes inp, while nearp≈ 0or1they compress. For that reason, a latent-scale reliability such asρ 2 ≈ 0.45does correspond to meaningful rank instability on the probability scale. The latent-scaleρ 2 is a mathematically coherent reliability target for binary data, and the associated posterior predictive mapping quantifies how it manifests as practically relevant leaderboard instability in percent-correct. In addition to the interpretation benefits and alignment to measurement theoretic latent modeling, the logistic mixed effect approach also handles unbalanced data and values near zero, both common in leaderboard testing, better than the observation-level linear approach. This can be seen in Figure 40 Under review as a conference paper 0% 25% 50% 75% 100% assistantbenchcorebench_hardgaiaonlinemind2webscicodescienceagentbench swebench_verified_minitaubench_airlineusaco Benchmark Prop. of Lat. Variance Explained Variance Component AgentItemItem-AgentModel-ItemModelModel-AgentResidual Figure 9: Proportion of Variance Explained in the posterior distributions from Eq. 2. “Item” from benchmarking literature represents “task” commonly found in agentic evaluation literature. “Agent” represents “scaffold”. -0.6-0.4-0.20.00.2 Model-Agent Difference in Variance (Leaderboard) Posterior Density -0.4-0.20.00.2 Model-Agent Difference in Variance (Benchmark) Posterior Distributions of Model-Agent Difference in Rank-relevant Variance -0.4-0.20.00.2 Model-Agent Difference in Variance (Task) Figure 10: Posterior density of Model-Agent difference of rank-relevant variance across the leader- board (left), across a sampled benchmark (middle), and across a sampled task (right). Positive numbers suggest LLM variance contribution is greater than that of agent scaffold. 15, which shows the latent and observed modeling estimated reliabilities. Rank order reliabilities are higher for the latent estimations proportionate to the quantity of low scoring models (mean score per model ̄y mb ⪅ 0.1). This is particularly true of SciCode where 100% of models have an accuracy score less than 0.1. G.1.1VARIANCE SHARES AND RELIABILITY ON THE LATENT SCALE For GLMM/Bayesian GLMM, we report variance shares as proportions of total latent variance: π k = σ 2 k P j σ 2 j , where the sum runs over included random effects and (optionally) a latent residual term. For Bayesian fits,π k is computed per posterior draw, yielding credible intervals. These can be found in Tables 20 and 21. D-study scaling.When an interaction term involves a sampled facet, its contribution to the variance of an averaged score shrinks with the number of sampled levels. For example, with tasks averaged (n i tasks), an I × M component contributes σ 2 IM /n i . G.2DISTANCE COMPONENTS (DISCO) QUASI-RELIABILITY The DISCO decomposition (Rizzo & Székely, 2010) generalizes classical ANOVA to arbitrary metric spaces. For K groups with n k observations each, define the within-group dispersion as S W = K X k=1 1 n k X i<j ∥X ki − X kj ∥,(18) and the total dispersion as S T = 1 N X i<j ∥X i − X j ∥,(19) 41 Under review as a conference paper swebench_verified_minitaubench_airlineusaco onlinemind2webscicodescienceagentbench assistantbenchcorebench_hardgaia 0.000.250.500.751.000.000.250.500.751.000.000.250.500.751.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 Original/Published Rank Posterior Rank (Median, 95% CI) claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_3_7_sonnet_20250219low claude_3_7_sonnetdefault claude_3_7_sonnethigh claude_haiku_4_5_20251001default claude_haiku_4_5_20251001high claude_haiku_4_5default claude_opus_4_1_20250805default claude_opus_4_1_20250805high claude_opus_4_1default claude_opus_4_1high claude_opus_4_20250514default claude_opus_4_20250514high claude_opus_4_5_20251101default claude_opus_4_5_20251101high claude_opus_4_5default claude_opus_4default claude_sonnet_4_20250514default claude_sonnet_4_20250514high claude_sonnet_4_5_20250929default claude_sonnet_4_5_20250929high claude_sonnet_4_5default claude_sonnet_4_5high deepseek_chat_v3_0324default deepseek_chat_v3_1default deepseek_chatdefault deepseek_r1_0528default deepseek_r1default deepseek_v3default gemini_2_0_flash_001high gemini_2_0_flashdefault gemini_2_5_pro_preview_03_25default gemini_3_pro_previewdefault gpt_4_1_2025_04_14default gpt_4o_2024_11_20default gpt_5_2025_08_07default gpt_5_2025_08_07medium gpt_5default gpt_5high gpt_5medium gpt_5minimal gpt_oss_120bdefault gpt_oss_120bhigh o3_2025_04_03default o3_2025_04_16default o3_2025_04_16low o3_2025_04_16medium o3_mini_2025_01_31high o3_mini_2025_01_31low o3_mini_2025_01_31medium o4_mini_2025_04_16default o4_mini_2025_04_16high o4_mini_2025_04_16low Figure 11: Published score versus posterior model percentile ranks using model-wise mean published score (x-axis) and draw-wise posterior median rankings with 95% CIs. Vertical intervals are95% posterior rank intervals; jitter separates horizontal ties. claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_3_7_sonnet_20250219low claude_opus_4_1default claude_opus_4_1high claude_sonnet_4_5_20250929default claude_sonnet_4_5high deepseek_r1default deepseek_v3default gemini_2_0_flashdefault gpt_4_1_2025_04_14default gpt_5default gpt_5minimal o3_2025_04_16default o3_2025_04_16medium o4_mini_2025_04_16default o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_sonnet_4_20250514default claude_sonnet_4_20250514high deepseek_r1default deepseek_v3default gemini_2_0_flashdefault gemini_2_5_pro_preview_03_25default gpt_4_1_2025_04_14default gpt_5_2025_08_07medium o3_2025_04_16default o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_haiku_4_5_20251001default claude_haiku_4_5_20251001high claude_haiku_4_5default claude_opus_4_1_20250805default claude_opus_4_1_20250805high claude_opus_4_20250514default claude_opus_4_20250514high claude_sonnet_4_5_20250929default claude_sonnet_4_5_20250929high deepseek_r1default deepseek_v3default gemini_2_0_flashdefault gpt_4_1_2025_04_14default gpt_4o_2024_11_20default gpt_5_2025_08_07default gpt_5_2025_08_07medium o3_2025_04_16default o3_2025_04_16medium o3_mini_2025_01_31high o3_mini_2025_01_31low o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_3_7_sonnet_20250219low claude_3_7_sonnetdefault claude_3_7_sonnethigh claude_haiku_4_5default claude_opus_4_1default claude_opus_4_1high claude_opus_4_5_20251101default claude_opus_4_5_20251101high claude_opus_4_5default claude_sonnet_4_20250514default claude_sonnet_4_20250514high claude_sonnet_4_5_20250929default claude_sonnet_4_5default claude_sonnet_4_5high deepseek_chat_v3_0324default deepseek_chat_v3_1default deepseek_r1_0528default deepseek_r1default deepseek_v3default gemini_2_0_flashdefault gemini_2_5_pro_preview_03_25default gemini_3_pro_previewdefault gpt_4_1_2025_04_14default gpt_5medium gpt_oss_120bdefault gpt_oss_120bhigh o3_2025_04_16default o3_2025_04_16medium o3_mini_2025_01_31low o3_mini_2025_01_31medium o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_haiku_4_5default claude_opus_4_1default claude_opus_4_1high claude_sonnet_4_5_20250929default claude_sonnet_4_5high deepseek_r1default deepseek_v3default gemini_2_0_flashdefault gpt_4_1_2025_04_14default gpt_5default o3_2025_04_16default o3_2025_04_16low o4_mini_2025_04_16default o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_3_7_sonnetdefault claude_3_7_sonnethigh claude_opus_4_1default claude_opus_4_1high claude_opus_4_20250514default claude_opus_4_20250514high deepseek_chatdefault deepseek_r1default deepseek_v3default gemini_2_0_flash_001high gemini_2_0_flashdefault gpt_4_1_2025_04_14default gpt_5_2025_08_07medium gpt_5high gpt_5minimal o3_2025_04_03default o3_2025_04_16default o3_2025_04_16medium o3_mini_2025_01_31high o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_haiku_4_5default claude_opus_4_1default claude_opus_4_1high claude_opus_4_20250514default claude_opus_4_20250514high claude_opus_4default claude_sonnet_4_5default claude_sonnet_4_5high deepseek_chat_v3_0324default deepseek_r1default deepseek_v3default gemini_2_0_flashdefault gpt_4_1_2025_04_14default gpt_5_2025_08_07default o3_2025_04_16default o3_mini_2025_01_31high o3_mini_2025_01_31low o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_3_7_sonnet_20250219low claude_haiku_4_5_20251001default claude_haiku_4_5_20251001high claude_opus_4_1_20250805default claude_opus_4_1_20250805high claude_sonnet_4_5_20250929default claude_sonnet_4_5_20250929high deepseek_r1default deepseek_v3default gemini_2_0_flashdefault gemini_2_5_pro_preview_03_25default gpt_4_1_2025_04_14default gpt_5_2025_08_07medium o3_2025_04_16default o3_2025_04_16medium o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_opus_4_1default deepseek_r1default deepseek_v3default gemini_2_0_flashdefault gpt_4_1_2025_04_14default gpt_5_2025_08_07default o3_2025_04_16default o3_mini_2025_01_31high o3_mini_2025_01_31low o4_mini_2025_04_16high o4_mini_2025_04_16low swebench_verified_minitaubench_airlineusaco onlinemind2webscicodescienceagentbench assistantbenchcorebench_hardgaia Figure 12: Rank order changes by benchmark based on comparison with median posterior rank from full variance decomposition model of Eq. 6. 42 Under review as a conference paper swebench_verified_minitaubench_airlineusaco onlinemind2webscicodescienceagentbench assistantbenchcorebench_hardgaia 0.000.250.500.751.000.000.250.500.751.000.000.250.500.751.00 Average Pr(θ_m > θ_m') Model Ranking by θ _m Model Name claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_3_7_sonnet_20250219low claude_3_7_sonnetdefault claude_3_7_sonnethigh claude_haiku_4_5_20251001default claude_haiku_4_5_20251001high claude_haiku_4_5default claude_opus_4_1_20250805default claude_opus_4_1_20250805high claude_opus_4_1default claude_opus_4_1high claude_opus_4_20250514default claude_opus_4_20250514high claude_opus_4_5_20251101default claude_opus_4_5_20251101high claude_opus_4_5default claude_opus_4default claude_sonnet_4_20250514default claude_sonnet_4_20250514high claude_sonnet_4_5_20250929default claude_sonnet_4_5_20250929high claude_sonnet_4_5default claude_sonnet_4_5high deepseek_chat_v3_0324default deepseek_chat_v3_1default deepseek_chatdefault deepseek_r1_0528default deepseek_r1default deepseek_v3default gemini_2_0_flash_001high gemini_2_0_flashdefault gemini_2_5_pro_preview_03_25default gemini_3_pro_previewdefault gpt_4_1_2025_04_14default gpt_4o_2024_11_20default gpt_5_2025_08_07default gpt_5_2025_08_07medium gpt_5default gpt_5high gpt_5medium gpt_5minimal gpt_oss_120bdefault gpt_oss_120bhigh o3_2025_04_03default o3_2025_04_16default o3_2025_04_16low o3_2025_04_16medium o3_mini_2025_01_31high o3_mini_2025_01_31low o3_mini_2025_01_31medium o4_mini_2025_04_16default o4_mini_2025_04_16high o4_mini_2025_04_16low Figure 13: Cumulative posterior dominance rankings across individuals. The point estimate for each individualmrepresents their global dominance score, defined as the expected probability that their latent parameterθ m = u (M) m + u ( mb MB) is strictly greater than that of a randomly selected peerm ′ from the population, given the observed datay:Global Dominance(m) = 1 M−1 P m ′ ̸=m Pr(θ m > θ m ′ | y)whereMis the total number of individuals, and the pairwise probabilities are empirically derived via Bayesian draws as:Pr(θ m > θ m ′ | y) ≈ 1 D P D d=1 I θ (d) m > θ (d) m ′ withI(·)denoting the indicator function andDrepresenting the total number of posterior draws. Individuals are arranged along the vertical axis in ascending order of their global dominance scores. The vertical dashed line atPr = 0.5denotes the theoretical baseline of a perfectly average individual who exhibits structural parity relative to the rest of the cohort. Deviations toward1.0signify deterministic stochastic dominance over the population, whereas values approaching0.0indicate systematic underperformance relative to the group. 43 Under review as a conference paper swebench_verified_minitaubench_airlineusaco onlinemind2webscicodescienceagentbench assistantbenchcorebench_hardgaia claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_haiku_4_5_20251001default claude_haiku_4_5_20251001high claude_haiku_4_5default claude_opus_4_1_20250805default claude_opus_4_1_20250805highclaude_opus_4_20250514default claude_opus_4_20250514high claude_sonnet_4_5_20250929default claude_sonnet_4_5_20250929high deepseek_r1default deepseek_v3default gemini_2_0_ fl ashdefault gpt_4_1_2025_04_14default gpt_4o_2024_11_20default gpt_5_2025_08_07default gpt_5_2025_08_07medium o3_2025_04_16default o3_2025_04_16medium o3_mini_2025_01_31high o3_mini_2025_01_31low o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_3_7_sonnetdefault claude_3_7_sonnethigh claude_opus_4_1default claude_opus_4_1high claude_opus_4_20250514default claude_opus_4_20250514high deepseek_chatdefault deepseek_r1default deepseek_v3default gemini_2_0_ fl ash_001high gemini_2_0_ fl ashdefault gpt_4_1_2025_04_14default gpt_5_2025_08_07medium gpt_5high gpt_5minimal o3_2025_04_03defaulto3_2025_04_16default o3_2025_04_16medium o3_mini_2025_01_31higho4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_opus_4_1default deepseek_r1default deepseek_v3default gemini_2_0_ fl ashdefault gpt_4_1_2025_04_14default gpt_5_2025_08_07default o3_2025_04_16default o3_mini_2025_01_31high o3_mini_2025_01_31low o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219highclaude_sonnet_4_20250514default claude_sonnet_4_20250514high deepseek_r1default deepseek_v3default gemini_2_0_ fl ashdefault gemini_2_5_pro_preview_03_25default gpt_4_1_2025_04_14default gpt_5_2025_08_07medium o3_2025_04_16default o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_haiku_4_5default claude_opus_4_1default claude_opus_4_1high claude_sonnet_4_5_20250929default claude_sonnet_4_5high deepseek_r1default deepseek_v3default gemini_2_0_ fl ashdefault gpt_4_1_2025_04_14default gpt_5default o3_2025_04_16default o3_2025_04_16low o4_mini_2025_04_16default o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_3_7_sonnet_20250219low claude_haiku_4_5_20251001default claude_haiku_4_5_20251001high claude_opus_4_1_20250805default claude_opus_4_1_20250805high claude_sonnet_4_5_20250929default claude_sonnet_4_5_20250929high deepseek_r1default deepseek_v3default gemini_2_0_ fl ashdefault gemini_2_5_pro_preview_03_25default gpt_4_1_2025_04_14default gpt_5_2025_08_07medium o3_2025_04_16default o3_2025_04_16medium o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_3_7_sonnet_20250219low claude_opus_4_1default claude_opus_4_1high claude_sonnet_4_5_20250929default claude_sonnet_4_5high deepseek_r1default deepseek_v3default gemini_2_0_ fl ashdefault gpt_4_1_2025_04_14default gpt_5default gpt_5minimal o3_2025_04_16default o3_2025_04_16medium o4_mini_2025_04_16default o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_3_7_sonnet_20250219low claude_3_7_sonnetdefault claude_3_7_sonnethigh claude_haiku_4_5default claude_opus_4_1default claude_opus_4_1high claude_opus_4_5_20251101default claude_opus_4_5_20251101high claude_opus_4_5default claude_sonnet_4_20250514default claude_sonnet_4_20250514high claude_sonnet_4_5_20250929default claude_sonnet_4_5default claude_sonnet_4_5high deepseek_chat_v3_0324default deepseek_chat_v3_1default deepseek_r1_0528default deepseek_r1default deepseek_v3default gemini_2_0_ fl ashdefault gemini_2_5_pro_preview_03_25default gemini_3_pro_previewdefault gpt_4_1_2025_04_14default gpt_5medium gpt_oss_120bdefault gpt_oss_120bhigh o3_2025_04_16default o3_2025_04_16medium o3_mini_2025_01_31low o3_mini_2025_01_31medium o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_haiku_4_5default claude_opus_4_1default claude_opus_4_1high claude_opus_4_20250514default claude_opus_4_20250514high claude_opus_4default claude_sonnet_4_5default claude_sonnet_4_5high deepseek_chat_v3_0324default deepseek_r1default deepseek_v3default gemini_2_0_ fl ashdefault gpt_4_1_2025_04_14default gpt_5_2025_08_07default o3_2025_04_16default o3_mini_2025_01_31high o3_mini_2025_01_31low o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_haiku_4_5default claude_opus_4_1default claude_opus_4_1high claude_opus_4_20250514default claude_opus_4_20250514high claude_opus_4default claude_sonnet_4_5default claude_sonnet_4_5high deepseek_chat_v3_0324default deepseek_r1default deepseek_v3default gemini_2_0_flashdefault gpt_4_1_2025_04_14default gpt_5_2025_08_07default o3_2025_04_16default o3_mini_2025_01_31high o3_mini_2025_01_31low o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_3_7_sonnet_20250219low claude_haiku_4_5_20251001default claude_haiku_4_5_20251001high claude_opus_4_1_20250805default claude_opus_4_1_20250805high claude_sonnet_4_5_20250929default claude_sonnet_4_5_20250929high deepseek_r1default deepseek_v3default gemini_2_0_flashdefault gemini_2_5_pro_preview_03_25default gpt_4_1_2025_04_14default gpt_5_2025_08_07medium o3_2025_04_16default o3_2025_04_16medium o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_opus_4_1default deepseek_r1default deepseek_v3default gemini_2_0_flashdefault gpt_4_1_2025_04_14default gpt_5_2025_08_07default o3_2025_04_16default o3_mini_2025_01_31high o3_mini_2025_01_31low o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_3_7_sonnet_20250219low claude_3_7_sonnetdefault claude_3_7_sonnethigh claude_haiku_4_5default claude_opus_4_1default claude_opus_4_1high claude_opus_4_5_20251101default claude_opus_4_5_20251101high claude_opus_4_5default claude_sonnet_4_20250514default claude_sonnet_4_20250514high claude_sonnet_4_5_20250929default claude_sonnet_4_5default claude_sonnet_4_5high deepseek_chat_v3_0324default deepseek_chat_v3_1default deepseek_r1_0528default deepseek_r1default deepseek_v3default gemini_2_0_flashdefault gemini_2_5_pro_preview_03_25default gemini_3_pro_previewdefault gpt_4_1_2025_04_14default gpt_5medium gpt_oss_120bdefault gpt_oss_120bhigh o3_2025_04_16default o3_2025_04_16medium o3_mini_2025_01_31low o3_mini_2025_01_31medium o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_haiku_4_5default claude_opus_4_1default claude_opus_4_1high claude_sonnet_4_5_20250929default claude_sonnet_4_5high deepseek_r1default deepseek_v3default gemini_2_0_flashdefault gpt_4_1_2025_04_14default gpt_5default o3_2025_04_16default o3_2025_04_16low o4_mini_2025_04_16default o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_3_7_sonnetdefault claude_3_7_sonnethigh claude_opus_4_1default claude_opus_4_1high claude_opus_4_20250514default claude_opus_4_20250514high deepseek_chatdefault deepseek_r1default deepseek_v3default gemini_2_0_flash_001high gemini_2_0_flashdefault gpt_4_1_2025_04_14default gpt_5_2025_08_07medium gpt_5high gpt_5minimal o3_2025_04_03default o3_2025_04_16default o3_2025_04_16medium o3_mini_2025_01_31high o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_3_7_sonnet_20250219low claude_opus_4_1default claude_opus_4_1high claude_sonnet_4_5_20250929default claude_sonnet_4_5high deepseek_r1default deepseek_v3default gemini_2_0_flashdefault gpt_4_1_2025_04_14default gpt_5default gpt_5minimal o3_2025_04_16default o3_2025_04_16medium o4_mini_2025_04_16default o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_sonnet_4_20250514default claude_sonnet_4_20250514high deepseek_r1default deepseek_v3default gemini_2_0_flashdefault gemini_2_5_pro_preview_03_25default gpt_4_1_2025_04_14default gpt_5_2025_08_07medium o3_2025_04_16default o4_mini_2025_04_16high o4_mini_2025_04_16low claude_3_7_sonnet_20250219default claude_3_7_sonnet_20250219high claude_haiku_4_5_20251001default claude_haiku_4_5_20251001high claude_haiku_4_5default claude_opus_4_1_20250805default claude_opus_4_1_20250805high claude_opus_4_20250514default claude_opus_4_20250514high claude_sonnet_4_5_20250929default claude_sonnet_4_5_20250929high deepseek_r1default deepseek_v3default gemini_2_0_flashdefault gpt_4_1_2025_04_14default gpt_4o_2024_11_20default gpt_5_2025_08_07default gpt_5_2025_08_07medium o3_2025_04_16default o3_2025_04_16medium o3_mini_2025_01_31high o3_mini_2025_01_31low o4_mini_2025_04_16high o4_mini_2025_04_16low Individual theta_m' Individual theta_m Probability 0.00 0.25 0.50 0.75 1.00 Pairwise Posterior Dominance Matrices: Pr(Row > Column | y) Figure 14: Pairwise Posterior Dominance, where color is Pr(Row > Column| y) across posterior draws. The point estimate for each individualmrepresentsPr(θ m > θ m ′ | y), defined by the expected probability of latent parameter θ m = u (M) m + u ( mb MB), for each benchmark b. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 assistantbenchcorebench_hardgaiaonlinemind2webscicodescienceagentbench swebench_verified_mini taubench_airlineusaco Benchmark G-coef fi cient interaction(obj, type) model.latentmodel.observed Reliability Across Benchmarks Figure 15: Rank order reliability, G-coefficient, per benchmark, for both latent and observed esti- mations with LLM as object of measurement. Bars are medians, points are means, and intervals are pseudo-standard error 0.68 HDI across the posterior using Eq. 3 44 Under review as a conference paper whereN = P k n k . The between-group component isS B =S T −S W , and the proportion attributable to the grouping factor isS B /S T . For multi-facet designs, we compute DISCO sequentially for each facet, using the dispersion ratio as the analog ofη 2 . Because DISCO does not produce a residual term, the facet proportions do not generally sum to one when computed independently; we normalize to aid comparison with the parametric estimates. Remark G.1. DISCO attributes substantially more dispersion to interaction terms than parametric methods. This occurs because pairwise distances capture nonlinear dependencies–such as a model performing anomalously well on a specific cluster of tasks–that additive random-effects models cannot represent. The DISCO interaction estimates should therefore be interpreted as an upper bound on the importance of non-additive effects, complementing the more conservative parametric decomposition. DISCO decomposes dispersion using pairwise distances rather than squared deviations, improving robustness under non-normality and heavy-tailed random effects. For an object groupingg(e.g., model or model–scaffold pair), DISCO yields between-group and within-group dispersion terms (T g ,T within ), from which we define Eρ 2 = T g T g + T within . We use DISCO primarily as a methodological sensitivity analysis to validate that conclusions (domi- nant facets; low model-ranking reliability) are not artifacts of Gaussian random-effect assumptions. G.3WHY ESTIMATION METHOD CHANGES CONCLUSIONS—AND WHAT TO DO ABOUT IT The five estimation methods are not interchangeable, and their disagreements are informative. A notable outcome is that variance shares differ across linear mixed model (LMM), Bayesian LMM, generalized linera mixed model (GLMM), Bayesian GLMM, and DISCO, especially under sparse observations: • LMM tends to allocate a large portion to residualσ 2 , which can understate structured interactions when the link is misspecified for Bernoulli data. •GLMM can collapse some components toward zero (boundary estimates) in unbalanced designs, yielding deceptively “clean” decompositions. •Bayesian LMM allows for estimates of reliability and distributional uncertainty on observed scale, but less equipped for floor or ceiling effects or highly imbalanced data. •Bayesian GLMM exposes skewness and large uncertainty, which is appropriate when the design under-identifies components. •DISCO can attribute more to interaction-like dispersion without parametric assumptions, often aligning with the intuition that agentic pipelines create nonlinear, heteroskedastic effects. Practical guidance. 1. Use Bayesian or nonparametric methods to communicate uncertainty when data are sparse; prefer point-estimate GLMM/LMM only when coverage is dense. 2.Triangulate: when all four methods agree on the ranking of dominant facets (e.g., task vs interactions vs residual), recommendations are robust; when they disagree, the correct conclusion is that the benchmark is under-instrumented for that inference. Linear vs. generalized models.The LME estimates generally attribute the largest share of variance to the residual (with a borderline exception of USACO), with correspondingly compressed named components. The GLME and Bayesian estimates, by contrast, allocate substantially more variance to specific tasks and models. This divergence is expected: the Gaussian identity-link model treats binary0, 1responses as continuous, and its residual absorbs the Bernoulli variance floor (p(1−p)) that the logit-link models separate out via the distributional assumption. The practical implication is that linear G-theory estimates applied to pass/fail benchmarks systematically underestimate the proportion of variance attributable to named facets and overestimate residual noise, leading to artificially compressed—but not necessarily more conservative—reliability estimates. 45 Under review as a conference paper Bayesian vs. frequentist GLME. The Bayesian and GLME estimates agree in broad strokes but diverge in two instructive ways. First, the Bayesian posteriors for agent variance are markedly right-skewed, with posterior means 2–5×larger than posterior medians (e.g., SWE-bench agent mean= 0.27, median= 0.19). The GLME point estimate, which approximates the posterior mode, misses this tail mass and underestimates the expected contribution of scaffold variance. Second, the credible intervals from the Bayesian analysis expose the decision-relevant uncertainty: for several benchmarks, the 95% HDI for model variance includes zero, meaning the data are consistent with no true model differentiation at all. This uncertainty is invisible in frequentist analyses. Nonparametric DISCO. The DISCO estimates depart most dramatically from the parametric methods on the interaction terms, difference likely due to heterogeneous cluster imbalances (and hence our preference for Bayesian estimates). DISCO attributes 40–57% of total dispersion to task×model interactions across benchmarks, compared to 1–11% from the Bayesian estimates. This divergence likely reflects two mechanisms: (i) DISCO’s sensitivity to nonlinear dependencies that the additive random-effects models cannot capture, and (i) the absence of a separate residual term in DISCO, which forces unexplained variation into the named interactions. The practical upshot is that DISCO serves as a useful upper bound on interaction effects and a reminder that the additive decomposition assumed by mixed-effects models may understate the complexity of the task×model relationship. G.4VARIANCE DECOMPOSITION EXPLAINS THE RELIABILITY CEILING AND GENERALIZABILITY Figure 16 reports posterior variance shares from the leaderboard model. Persistent model variation is smaller than the combined variation associated with scaffolds, benchmark-conditioned model performance, and item-conditioned interactions. Most observed item outcomes therefore contain more condition-specific variation than globally transferable model signal. Agent Benchmark Benchmark-Agent Benchmark-Model Benchmark-Model-Agent Item Item-Agent Model-Item Model Model-Agent Residual 0%25%50%75%100% Proportion of Variance Explained Figure 16: Full variance decomposition of Eq. 6 as percentages of total variance. Variance decompo- sitions for individual benchmarks are in Appendix F. Black represents residual variance. This does not imply that model capability is unimportant. Reliability is population-dependent: contemporary frontier models are often similar, whereas tasks and scaffolds are heterogeneous. A benchmark could have ranked a historically broader model population reliably and yet fail to distinguish the current frontier. Likewise, a future benchmark may become uninformative as models saturate it. Reliability must consequently be re-estimated as the evaluated population and agent ecosystem change. For strong generalization to new tasks, we would want to see larger main effects for models and scaffolds, making decisions clearer. However, the reality is that the second-order task-level inter- actions, model–task and scaffold–task, represent larger variation components than their respective main effects. This quantifies that, instead of providing clear architecture choices, some models or scaffolds are better at certain tasks than others, implying that a developer should, for a given task, choose a specific model and scaffold for each new task, rather than for the group of tasks represented by a benchmark or leaderboard (because of a smaller benchmark–model component). This is not ideal for developers, as it means they would need to know about the demands of each task and the appropriateness of architecture selections for it. This partly explains why there is such low measurement reliability for fewer numbers of items in Figures 1 and 3: a different model scaffold might be optimal for each task item. 46 Under review as a conference paper G.5ABSOLUTE RELIABILITY ON LINEAR SCALE Generalizability theory also permits estimation of reliability of numeric scores. Absolute reliability, as it is known, is always less than or equal to the corresponding estimate of relative reliability. But because scores are on the observed scale, all of the analyses in this section are based on the Bayesian linear mixed model (§ C.2). They are illustrative and serve as a framework for practitioners looking to measure reliability of assigned numeric scores. Reliable rankings do not imply reliable scores.Relative reliability (Eρ 2 ) asks whether the ranking of models stays similar across evaluation conditions, while absolute reliability (Φ) asks whether their reported scores stay similar. Figure 18 shows that this distinction matters substantially in practice: absolute reliability is consistently lower across the nine benchmarks, and for several benchmarks remains near zero even as ranking reliability increases with additional tasks. An evaluation can therefore support a stable ranking without supporting equally strong claims about a model’s absolute score or whether it exceeds a fixed threshold. While not recommended for the HAL dataset and benchmarks, there may be some agentic benchmarks where absolute scores are needed for decision-making. Absolute reliability is most interpretably performed on the observed scale, rather than the latent scale used throughout the main body of this study. For the HAL dataset, there is simply not enough signal relative to noise in the benchmarks for any reliable absolute scoring. Nevertheless, the concept of absolute reliability may be important for some agentic measures. G.5.1ABSOLUTE RELIABILITY: NUMERIC VALUES OF SCORES CARRY MEANING Rank-based reliability measures whether an evaluation preserves the ordering of models or systems, but it does not establish whether their score levels are reproducible. Absolute reliability is important when scores are interpreted directly—for example, when assessing whether a model exceeds a deployment threshold, meets a minimum capability requirement, or achieves a specified performance target. In these settings, a task set or scaffold that shifts every model’s score equally can change the decision even without changing the ranking. The absolute reliability, or dependability coefficient,Φ, therefore includes these common shifts in its error variance. Treating tasks and scaffolds as random facets, the absolute reliability of a model score under an equal-allocation design is Φ M(b) (n i ,n a ) = σ 2 M σ 2 M + (σ 2 I + σ 2 IM )/n i + (σ 2 A + σ 2 MA )/n a + (σ 2 IA + σ 2 IMA,e )/(n i n a ) .(20) Relative to model-ranking reliability, the denominator additionally includes task main effects, scaffold main effects, and task–scaffold interactions. These components affect the absolute score of a model averaged over sampled tasks and scaffolds, even when they do not affect its position relative to other models. Increasing the number of tasks reduces task-related error, whereas increasing the number of scaffolds reduces scaffold-related error. When the object of measurement is the complete model–scaffold system, scaffold differences are part of the system’s universe score rather than measurement error. Absolute reliability is then Φ MA(b) (n i ) = σ 2 M + σ 2 A + σ 2 MA σ 2 M + σ 2 A + σ 2 MA + (σ 2 I + σ 2 IM + σ 2 IA + σ 2 IMA,e )/n i .(21) Here, task main effects enter the denominator because sampling an easier or harder task set can shift a system’s score across an absolute decision threshold. In contrast, scaffold main effects and model–scaffold interactions remain signal because they distinguish the systems being evaluated. For either object of measurement, absolute reliability is no greater than the corresponding ranking reliability: Φ M(b) (n i ,n a )≤ Eρ 2 M(b) (n i ,n a ),Φ MA(b) (n i )≤ Eρ 2 MA(b) (n i ).(22) A large gap indicates that rankings may be reproducible even though reported score levels are sensitive to the sampled evaluation conditions. Likewise, high inter-scaffold ranking reliability does not guarantee absolute agreement: two scaffolds can preserve model ordering while producing systematically different scores. 47 Under review as a conference paper 0.00 0.05 0.10 0.15 0.20 0.25 assistantbenchcorebench_hardgaiaonlinemind2webscicodescienceagentbench swebench_verified_mini taubench_airlineusaco Benchmark Inter-Scaffold Reliability Standardized Absolute Inter-Scaffold Reliability Figure 17: Standardized Absolute Inter-scaffold Reliability. Standardized inter-scaffold reliability looks at the reliability for a single fixed task, rather than the composite reliability across all tasks in a benchmark. assistantbenchcorebench_hardgaiaonlinemind2webscicodescienceagentbenchswebench_verified_minitaubench_airlineusaco 131030 131030110100110100131030110100 131030131030110100 0.00 0.25 0.50 0.75 Number of Agentic Tasks Est. Reliability: Φ or E ρ 2 stat ep2 phi Figure 18: Absolute vs Relative Reliability on Observed Scale G.6FULL ESTIMATION TABLES FOR PROPORTION OF LATENT VARIATION EXPLAINED Full variance-proportion tables for all models and all five estimation methods are provided in the following tables including Bayesian posterior summaries and also LME, GLME, and DISCO estimates to enable cross-method comparison. 48 Under review as a conference paper Table 20: Proportion of variation explained per benchmark. For comparability across methods, Bayesian estimates represent posterior means (rather than medians found in most of the study) to preserve proportional relationships. BenchmarkFacetLMMBayes LMMGLMMBayes GLMMDISCO scicodeitem0.2350.2320.7660.7180.209 scicodeagent0.0010.0410.0040.0540.001 scicodemodel0.0100.0110.0530.0570.012 scicodeitem_agent0.0370.0360.0120.0160.252 scicodeitem_model0.1300.1230.0120.0350.509 scicodemodel_agent0.0020.0040.0030.0150.017 scicodesigma0.5840.5530.1500.105NA gaiaitem0.2100.1130.3420.2800.169 gaiaagent0.0410.4710.0290.1940.017 gaiamodel0.0300.0200.0400.0420.046 gaiaitem_agent0.0360.0200.0350.0360.209 gaiaitem_model0.0970.0510.0410.1020.481 gaiamodel_agent0.0590.0390.0830.0790.077 gaiasigma0.5270.2860.4310.267NA taubench_airlineitem0.1660.1400.2770.2440.161 taubench_airlineagent0.0430.2370.0440.1800.021 taubench_airlinemodel0.0270.0240.0390.0350.025 taubench_airlineitem_agent0.0860.0700.1070.1010.250 taubench_airlineitem_model0.0350.0240.0000.0330.484 taubench_airlinemodel_agent0.0120.0120.0160.0200.059 taubench_airlinesigma0.6310.4930.5180.386NA swebench_verified_miniitem0.1530.0700.3490.2770.092 swebench_verified_miniagent0.2460.6450.1030.2600.073 swebench_verified_minimodel0.0560.0270.2720.1420.119 swebench_verified_miniitem_agent0.0680.0340.0240.0260.190 swebench_verified_miniitem_model0.0890.0390.0090.0590.399 swebench_verified_minimodel_agent0.0570.0360.0420.1300.126 swebench_verified_minisigma0.3310.1490.2020.106NA usacoitem0.2570.1410.5510.4500.239 usacoagent0.0720.4690.0000.0900.003 usacomodel0.0620.0220.0000.0490.027 usacoitem_agent0.2000.1140.1300.1720.262 usacoitem_model0.1840.1030.0000.1160.440 usacomodel_agent0.0000.0270.1030.0660.030 usacosigma0.2250.1240.2160.057NA assistantbenchitem0.1160.0740.4370.3310.156 assistantbenchagent0.0000.3770.0000.1810.002 assistantbenchmodel0.0000.0030.0000.0240.017 assistantbenchitem_agent0.0830.0570.1070.1370.219 assistantbenchitem_model0.0470.0220.0000.0570.574 assistantbenchmodel_agent0.0130.0090.0690.0580.032 assistantbenchsigma0.7410.4580.3860.213NA corebench_harditem0.2280.1780.4150.3620.153 corebench_hardagent0.0630.2860.0470.1800.027 corebench_hardmodel0.0710.0560.1340.1130.074 corebench_harditem_agent0.0100.0090.0000.0050.200 corebench_harditem_model0.0990.0740.0290.0630.462 corebench_hardmodel_agent0.0120.0120.0110.0160.083 corebench_hardsigma0.5170.3850.3650.262NA onlinemind2webitem0.2830.2050.4140.3720.239 onlinemind2webagent0.0000.2720.0000.0850.000 onlinemind2webmodel0.0020.0040.0030.0090.007 onlinemind2webitem_agent0.1330.0970.1640.1630.296 onlinemind2webitem_model0.0220.0140.0000.0230.446 onlinemind2webmodel_agent0.0160.0130.0270.0290.012 onlinemind2websigma0.5450.3950.3930.319NA scienceagentbenchitem0.2460.1290.5320.4130.208 scienceagentbenchagent0.0420.5050.0480.2580.011 scienceagentbenchmodel0.0160.0080.0260.0200.016 scienceagentbenchitem_agent0.0860.0450.0490.0460.249 scienceagentbenchitem_model0.0340.0170.0000.0210.493 scienceagentbenchmodel_agent0.0000.0030.0080.0140.023 scienceagentbenchsigma0.5770.2940.3380.227NA 49 Under review as a conference paper Table 21: Proportion of variation explained full leaderboard. For comparability across methods, Bayesian estimates represent posterior means (rather than medians found in most of the study) to preserve proportional relationships. FacetLMMBayes LMMGLMMBayes GLMMDISCO agent0.0160.0390.0200.0400.014 benchmark0.0750.1080.2590.2990.023 benchmark_agent0.0280.0360.0260.0310.030 benchmark_item0.2520.2340.3280.2970.117 benchmark_item_agent0.0700.0650.0520.0550.140 benchmark_item_model0.0470.0430.0040.0400.240 benchmark_model0.0100.0080.0200.0150.040 benchmark_model_agent0.0130.0140.0140.0160.046 model0.0260.0250.0330.0320.015 model_agent0.0070.0060.0090.0080.035 benchmark_task_model_agent + sigma0.4560.4220.2360.1680.302 50

Instant research discovery