Paper deep dive
Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation
Lin Shi, Haowei Lin, Zixuan Zhu, Xiaoyue Zhou, Xiang Li, Xiangning Lin, Yaxuan Deng, Han Xu, Yuangang Li, Shanda Li, Zizhao Chen, Hanwen Xing, Harsh Raj, Bo Chen, Quan Shi, Steven Dillmann, Yipeng Gao, Puneesh Khanna, Ruofan Lu, Chao Beyond Zhou, Michael Yang, Robert Zhang, Siyuan Chai, Jiayu Chang, Yizhao Chen, Xiaokun Chen, Yiwei Dai, Wenting Yang, Hange Liu, Minghao Liu, Zihan Wang, Adnan El Assadi, Benedikt Stroebl, E. Kelly Buchanan, Han Meng, Junwei He, Longxuan Yu, Radin Shayanfar, Yukyung Lee, Zhikang Dong, Allen G Hart, Anjiang Wei, Anurag Kashyap, Arpandeep Khatua, Audrey Jixin Zheng, Chengrui Ma, David Heineman, Dubing Chen, Hai-Anh Trinh, Haishuo Fang, Hefan Zhang, Hui Shen, Issa Sugiura, Jiankai Sun, Jiechao Gao, Junhong Lin, Junnan Li, Kai Yang, Lei Hsiung, Maoyu Wang, Mengze Tang, Nabil Omi, Negin Raoof, Nicholas Edwards, Octavia Guo, Orfeas Menis Mastromichalakis, Pengliang Ji, PrzemysÅaw Hejman, Qi Qi, Qunshu Lin, Richard Zhuang, Rui Yang, Ruichen Zheng, Ryan Marten, Shaghayegh Fazliani, Shizheng Hou, Sicong Jiang, Sijie Li, Song Bian, Terry Yue Zhuo, Tianqing Wu, Tom Tang, Wanjia Zhao, Weihao Xuan, Wenhua Liang, Xian Liu, Xin Lan, Xuan Zhang, Xuandong Zhao, Yanchuan Tang, Yifan Jiang, Yijiang Li, Yitong Guan, Yizhi Li, Yonghui Liu, Yuheng Tang, Yujun, Mao, Yunfei Zhao, Yuxin Wang, Yuxuan Tang, Zhenheng Tang, Zhifei Li, Ziruo Wang, Ziyu She, Kaiyuan Liu, Iheb Chaabane, Yuxin Tang, Xiangyi Li, Andy Konwinski, Boxuan Li, Leon Liangyu Chen, Alex Dimakis, Nicholas Carlini, Soroush Vosoughi, Di He, Etash Guha, Benjamin Feuer, Mike Merrill, Ludwig Schmidt, Alex Shaw
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/7/2026, 4:42:09 AM
Summary
This paper introduces Harbor Adapters, a unified infrastructure for evaluating agentic AI models across diverse benchmarks, and Harbor-Index, a curated meta-dataset of 82 high-quality, difficult tasks. The authors port over 80 benchmarks to the Harbor framework, enabling O(m+n) integration rather than O(mn). They conduct a large-scale evaluation of 8 models (including GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro) across 54 benchmarks, finding that base model capability outweighs harness design in determining performance. The study highlights that frontier models are more token-efficient on easy tasks but remain costlier overall. Harbor-Index is proposed as a compact, affordable, and challenging subset for future evaluation.
Entities (12)
Relation Signals (9)
Harbor Adapters ā reducesintegrationcomplexityto ā O(M+N)
confidence 95% Ā· This changes the integration burden from O(mn) benchmark-agent pairings to O(m + n)
Harbor Adapters ā supports ā SWE-BENCH
confidence 95% Ā· Harbor Adapters... port more than 80 benchmarks... SWE-bench
GPT-5.4 ā usedwithharness ā Terminus-2
confidence 95% Ā· every model is run with Terminus-2
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Evaluating agents on the growing number of agentic benchmarks is challenging because they often require complex environments and agent integrations. We introduce Harbor Adapters, a unified evaluation infrastructure for agentic benchmarks. Our work makes three contributions. First, we develop benchmark adapters that port more than 80 benchmarks to evaluate arbitrary agents, and validate them through rigorous code review and parity experiments. Second, we conduct a large-scale evaluation of 8 models spanning capability tiers across 54 benchmarks; every model is run with Terminus-2 and with one of 3 native harnesses. This enables a broader analysis of agent capabilities and failure modes than was previously possible. Third, we introduce Harbor-Index, a curated set of 82 difficult, diverse, and high-quality tasks spanning 29 benchmarks, refined from the adapted suite through difficulty filtering, AI and human audit, and an audit-and-fix loop. Harbor-Index preserves the challenge and breadth of large-scale agentic evaluations while being affordable to run; no evaluated model-harness configuration exceeds 30% pass rate, and the strongest (GPT-5.5 with Codex) reaches 28.0%. We release the adapters, evaluation results, in-depth analysis, and Harbor-Index as open-source artifacts to support more reliable and comprehensive evaluation of language-model agents.
Tags
Links
- Source: https://arxiv.org/abs/2609.04298v1
- Canonical: https://arxiv.org/abs/2609.04298v1
Trouble viewing inline? Open PDF directly ā
Full Text
293,084 characters extracted from source content.
Expand or collapse full text
Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation Organization & Execution Team Lin Shi ā , Haowei Lin ā , Zixuan Zhu ā¦ , Xiaoyue Zhou ā¦ , Xiang Li ā¦ , Xiangning Lin ā¦ , Yaxuan Deng ā¦ , Han Xu ā¦ , Yuangang Li ā¦ , Shanda Li ā¦ , Zizhao Chen ā¦ , Hanwen Xing ā¦ , Harsh Raj, Bo Chen, Quan Shi, Steven Dillmann, Yipeng Gao, Puneesh Khanna, Ruofan Lu, Chao Beyond Zhou, Michael Yang, Robert Zhang, Siyuan Chai, Jiayu Chang, Yizhao Chen, Xiaokun Chen, Yiwei Dai, Wenting Yang, Hange Liu, Minghao Liu, Zihan Wang Adapter Contributors Adnan El Assadi, Benedikt Stroebl, E. Kelly Buchanan, Han Meng, Junwei He, Longxuan Yu, Radin Shayanfar, Yukyung Lee, Zhikang Dong, Allen G Hart, Anjiang Wei, Anurag Kashyap, Arpandeep Khatua, Audrey Jixin Zheng, Chengrui Ma, David Heineman, Dubing Chen, Hai-Anh Trinh, Haishuo Fang, Hefan Zhang, Hui Shen, Issa Sugiura, Jiankai Sun, Jiechao Gao, Junhong Lin, Junnan Li, Kai Yang, Lei Hsiung, Maoyu Wang, Mengze Tang, Nabil Omi, Negin Raoof, Nicholas Edwards, Octavia Guo, Orfeas Menis Mastromichalakis, Pengliang Ji, PrzemysÅaw Hejman, Qi Qi, Qunshu Lin, Richard Zhuang, Rui Yang, Ruichen Zheng, Ryan Marten, Shaghayegh Fazliani, Shizheng Hou, Sicong Jiang, Sijie Li, Song Bian, Terry Yue Zhuo, Tianqing Wu, Tom Tang, Wanjia Zhao, Weihao Xuan, Wenhua Liang, Xian Liu, Xin Lan, Xuan Zhang, Xuandong Zhao, Yanchuan Tang, Yifan Jiang, Yijiang Li, Yitong Guan, Yizhi Li, Yonghui Liu, Yuheng Tang, Yujun (Audrey) Mao, Yunfei Zhao, Yuxin Wang, Yuxuan Tang, Zhenheng Tang, Zhifei Li, Ziruo Wang, Ziyu She, Kaiyuan Liu, Iheb Chaabane, Yuxin Tang, Xiangyi Li Advisory Committee Andy Konwinski, Boxuan Li, Leon Liangyu Chen, Alex Dimakis, Nicholas Carlini, Soroush Vosoughi, Di He, Etash Guha, Benjamin Feuer, Mike Merrill ā , Ludwig Schmidt ā , Alex Shaw ā ā Project Leads Ā· ā¢ Core Contributors Ā· Full affiliations in Appendix A. Correspondence: ls2282@cornell.edu; linhaowei@pku.edu.cn. Harbor framework and adapters: https://github.com/harbor-framework/harbor Harbor-Index: https://github.com/harbor-framework/harbor-index Abstract Evaluating agents on the growing number of agentic benchmarks is challenging because they often require complex environments and agent integrations. We intro- duce Harbor Adapters, a unified evaluation infrastructure for agentic benchmarks. Our work makes three contributions. First, we develop benchmark adapters that port more than 80 benchmarks to evaluate arbitrary agents, and validate them through rigorous code review and parity experiments. Second, we conduct a large- scale evaluation of 8 models spanning capability tiers across 54 benchmarks; every model is run with Terminus-2 and with one of 3 native harnesses. This enables a broader analysis of agent capabilities and failure modes than was previously possi- ble. Third, we introduce Harbor-Index, a curated set of 82 difficult, diverse, and high-quality tasks spanning 29 benchmarks, refined from the adapted suite through difficulty filtering, AI and human audit, and an audit-and-fix loop. Harbor-Index preserves the challenge and breadth of large-scale agentic evaluations while being affordable to run; no evaluated modelāharness configuration exceeds30%pass rate, and the strongest (GPT-5.5 with Codex) reaches28.0%. We release the adapters, evaluation results, in-depth analysis, and Harbor-Index as open-source artifacts to support more reliable and comprehensive evaluation of language-model agents. 1 Introduction Language models are increasingly embedded within agents: systems that not only answer questions, but also plan, use tools, write code, operate computers, and carry out long-horizon tasks. This shift has expanded the scope of model evaluation beyond static question answering to interactive settings such as software engineering, web research, finance, and terminal-based problem solving. Preprint. arXiv:2609.04298v1 [cs.AI] 3 Sep 2026 SWEBench Multi lingual SWE Bench Pro SWE bench Verified SWE smith SWT Bench Feature Bench SWE Lancer Aider Polyglot BigCode Bench LiveCode Bench Human EvalFix USACO QuixBugs GSO AlgoTune CRUST Bench Compile Bench Software Engineering AIME Omni Math ARC AGI-2 KUMO Reasoning Gym GPQA Diamond MMMLU HLE SimpleQA A-LCR MMAU Multimodal Cyber Gym Strong Reject Replication Bench SLD Bench CodePDE Research Code Bench SciCode Qcircuit Bench BIX Bench LAB Bench DA Code Spider 2 GAIA GAIA2 BFCL DeepSynth Seal-0 Wide Search Terminal Bench 2.0 Finance Agent PIXIU Spreadsheet Bench MedAgent Bench Law Bench Harbor Adapters: Connect datasets to agents IneqMath Agents + Models Future Agents TaskEnv Test Solution Skills Bench Mathematics & Reasoning Knowledge & Long Context Data & Analytics Scientific Research Agents, Tools, & Systems Safety & Security Professional Domains Future Benchmarks Figure 1: We integrate and evaluate Harbor adapters for 54 benchmarks across diverse domains. The Harbor format defines a task by instruction, environment, test, and solution, enabling heterogeneous datasets to be standardized to a shared schema and connected to multiple agents. The benchmark landscape has grown in response: SWE-bench [64], Terminal-Bench [103], Fi- nanceAgent [162], and many other agentic, tool-using, long-horizon suites; more than 200 have appeared since 2024. This growth reflects the importance of agent evaluation, but has also created a major infrastructure challenge. Agentic benchmarks are far harder to run than question-answering suites because they are hetero- geneous in task formats, environment requirements, interaction protocols, and scoring procedures. Supportingmbenchmarks andnagents therefore often requiresO(mn)integrations, one per benchmark-agent pair. This fragmentation limits reliability and scalability: papers and model releases typically report only a small subset of popular benchmarks, so it is unclear whether progress general- izes across domains or concentrates on a few well-known tasks, and comparisons are confounded by differences in benchmark implementations, environment setups, and evaluation protocols. We address this with Harbor Adapters, which unify diverse agentic evaluations under Harbor, a Python library [53] originally developed for Terminal-Bench that defines tasks and runs agents in sandbox environments. Harbor Adapters decouple agents from benchmarks: each benchmark needs one adapter exposing its tasks, environments, and scoring logic through Harbor, and each agent needs one Harbor integration. This changes the integration burden fromO(mn)benchmark-agent pairings toO(m + n) adapters and agent integrations (Figure 1). Our first contribution is the Harbor Adapters infrastructure. We port more than 80 benchmarks to Harbor, covering both natively agentic benchmarks (e.g., SWE-bench) and non-agentic ones (e.g., HLE [126]) that we make agentic by letting agents use tools, write files, and interact with executable environments. We use rigorous code review and parity experiments to verify that our adaptations remain comparable to the original implementations. Our second contribution is a large-scale evaluation of models and harnesses. We evaluate 8 models spanning capability tiers, each under exactly two harnesses: cross-family Terminus-2 and one of 3 native harnesses (Codex for GPT, Claude Code for Claude, Gemini CLI for Gemini). This yields 16 modelāharness configurations (4 distinct harness implementations) across 54 benchmarks and 6,627 tasks, three trials per configuration, consuming 226B input and output tokens and more than $300K of compute. We find: 2 ā¢ The model underlying an agent is more influential for benchmark performance than the harness. ā¢ The ordering of models on benchmarks is usually highly correlated, i.e., better models are usually better on many benchmarks as opposed to dominating specific niches. ā¢ Better models use fewer tokens than weaker models across all empirical task difficulty levels. ā¢ Among unresolved tasks, task failures such as misspecified tasks are common, obscuring whether a benchmark still has room for improvement or its remaining tasks are unsolvable because of errors in the task definitions. Our third contribution is Harbor-Index, 82 tasks spanning 29 benchmarks drawn from the adapted suite. Because agentic evaluation is expensive, Harbor-Index is affordable by design: compact, diverse over many agentic domains and task formats, difficult enough to avoid immediate saturation, and high-quality enough that failures reflect model limitations rather than task defects or ambiguous grading. No evaluated modelāharness configuration exceeds30%pass rate; the strongest (GPT-5.5 with Codex) reaches 28.0%, leaving substantial room for future progress. We open-source the benchmark adapters, evaluation tools and analyses, the full trajectories of the large-scale evaluation, and Harbor-Index 1.0 on GitHub and the Harbor Hub, with interactive results at harbor-index.org. 2 Harbor Adapters: Infrastructure OverviewHarbor Adapters is a unifying integration layer built upon Harbor [53]. Harbor provides a standardized task abstraction and execution environment; Harbor Adapters translate dataset-specific formats into Harborās task schema of instruction, environment, tests, and solution (Figure 1), de- coupling agent execution from benchmark-specific logic and reducing integration fromO(mn)to O(m +n). In addition, the adapter infrastructure supports GPU interaction, LLM-as-a-judge verifiers, custom metric aggregation, and multiple sandbox backends (e.g., Daytona [28], Modal [108], and E2B [38]). As of May 2026, Harbor Adapters support 22 agents and more than 80 benchmarks. Adapter construction Adapters are designed to faithfully preserve the semantics of the origi- nal benchmark datasets. To verify adaptation fairness, (1) we run multi-trial parity experiments matching agent, model, and execution configuration across original and Harbor-adapted benchmarks (Appendix D.4), reporting meanĀ±sample standard error of the mean (SEM) to account for stochas- ticity in Tables 3 and 4; (2) each adapter undergoes a strict three-stage quality audit by bot, junior, and senior reviewers to ensure faithful and standardized adaptation, with over 10,000 GitHub comments across all adapters (Appendix D.3.2). Large-scale evaluation. Harbor Adapters enable a large-scale evaluation over 6,627 tasks from 54 benchmarks, analyzed in Section 3. We cover 8 models spanning capability tiers from Google, OpenAI, and Anthropic: Gemini-3.1-Pro and Gemini-3-Flash; Claude Opus 4.6, Sonnet 4.6, and Haiku 4.5; and GPT-5.4, GPT-5-mini, and -nano. Each model runs under exactly two harnesses: the cross-family Terminus-2 [103] and one of 3 native harnesses (Gemini CLI for Gemini, Claude Code for Claude, and Codex for GPT), giving 16 modelāharness configurations using 4 distinct harness implementations overall. Each (benchmark, model, harness) setting is repeated for 3 trials, yielding ā¼0.3M trajectories. This scale is difficult to reach withO(mn)benchmark-specific integrations; Harbor Adapters make it feasible. Further experimental details are in Appendix E. 3 Analysis: Agentic Benchmarking at Scale We ask four questions: (1) Benchmark: how much unique signal does the 54-benchmark suite provide? (2) Harness: does model capability or harness design drive performance? (3) Efficiency: where do gains justify token costs? (4) Bottlenecks: where do tasks break, and why do agents fail? 3.1 Benchmark: progress over time and redundancy analysis Which benchmarks still challenge frontier models? Figure 2 shows broad progress across domains. Of the 54 benchmarks we evaluate, 13 are largely saturated, with state-of-the-art models exceeding 90%, including once-challenging ones (e.g., GPQA-Diamond [135]). Software engineering 3 0.00.20.40.60.81.0 Score GSO ('25) SWE-Lancer ('25) CRUST-Bench ('25) CompileBench ('25) SWE-Bench Pro ('25) SWT Bench* ('24) SWE-bench-verified ('24) SWE-bench Multilingual ('25) BigCodeBench-Hard ('24) Aider Polyglot ('24) LiveCodeBench (V6) ('25) USACO* ('24) QuixBugs ('17) HumanEvalFix ('23) ARC-AGI-2 ('25) Omni-math ('24) Reasoning Gym ('25) KUMO* ('25) AIME 24&25 ('25) IneqMath ('25) 45% 71% 98% 100% 55% 71% 78% 80% 45% 88% 93% 96% 98% 100% 77% 88% 91% 97% 99% 100% 0.00.20.40.60.81.0 Score HLE ('25) MMMLU ('24) SimpleQA ('24) GPQA Diamond ('23) BIX-Bench ('25) Scicode* ('24) ReplicationBench ('25) QCircuitBench ('25) LAB-Bench ('24) Seal0 ('25) DeepSynth ('26) TerminalBench2.0 ('25) GAIA ('25) Widesearch* ('25) Spider 2* ('24) DA-Code ('24) SpreadsheetBench* ('24) StrongReject ('24) MMAU ('24) 54% 84% 96% 97% 48% 58% 63% 69% 73% 51% 59% 69% 76% 79% 40% 62% 83% 97% 70% Past SOTA at launch SWE: Competitive/function SWE: Repo issue/test SWE: Other Mathematics & Reasoning Knowledge & Long Context Scientific Research Agents, Tools & Systems Data & Analytics Professional Domains Safety & Security Multimodal Figure 2: Benchmark progress since published; best reported score per benchmark, colored by domain, for 39 of the 54 benchmarks we evaluated. āSWEā abbreviates software engineering. Scores are normalized to[0, 1]so that higher is better. Light-filled bars indicate the current best score in our evaluation; outlined bars, the state of the art at publication time. Benchmarks marked ā use an i.i.d. subset of the full dataset. Launch and current-best scores come from benchmark papers, leaderboards, and model reports (Appendix F.3). is especially prominent and splits sharply: functional and competitive-coding tasks near saturation (e.g., LiveCodeBench [62]), repository-level feature implementation and issue fixing progress quickly (e.g., SWE-bench-verified [64]), while software speedup remains hard (e.g., GSO [142]). How many benchmarks do we actually need to differentiate models? While benchmark tasks can rank models by capability, we find the effective evaluation space far lower-dimensional than the number of benchmarks and tasks suggests, in three complementary forms. (1) At the benchmark level, most scores vary along only a few shared directions. Following BenchPress [197], we apply Principal Component Analysis (PCA) to the16Ć 54score matrix of modelāharness configurationsĆbenchmarks (Table 6, 7, & 8). Most variation is explained by a single shared capability factor; adding agent scaffolds introduces one additional direction, but the space remains low-rank (Appendix F.1). Greedy selection on absolute Spearman rank correlations agrees: after 12 benchmarks are selected, every remaining benchmark correlates atĻā„ 0.7with a selected one, so its model ranking is already well approximated and adds little information (Table 10). (2) Difficulty and uniqueness are not the same. The benchmarks that greedy selection identifies as independently informative span the full difficulty spectrum, from hard (CodePDE [83]) to easy (StrongReject [148]), rather than clustering at one end (Table 10; Appendix F.1). Some difficult benchmarks (e.g., BigCodeBench [210]) are largely predictable from others, while benchmarks testing specialized capabilities resist prediction regardless of difficulty (e.g., FinanceAgent [162] and LabBench [78]). This suggests that evaluation design should prioritize coverage of distinct capability axes, not raw difficulty alone. (3) Redundancy also appears within individual benchmarks. Across 52 benchmarks with at least 10 tasks (excluding CodePDE [83] (5 tasks) and SLDBench [89] (8 tasks)), 3 representative tasks per benchmark already recover the system ranking at a meanĻ ā 0.923, so many tasks within a benchmark measure overlapping capabilities. The degree of redundancy varies with internal task diversity: 3 tasks nearly reproduce the full ranking where tasks test similar capabilities (e.g., 4 WideSearch,Ļ = 0.99), but not where task types are diverse (e.g., CyberGym [169],Ļ = 0.75) (Appendix F.1). 3.2 Harness: the interaction analysis between harness and base model Which matters more: the model or the harness? Benchmark difficulty accounts for most score variance (ICC= 0.75), so we control for it with a linear mixed model using benchmark as a random intercept: 6/8 model coefficients are significant (p < 0.05) versus 2/4 harness coefficients, and the model fixed-effect range (0.451) is 5.2Ćthe harness range (0.087) (Appendix F.2). Thus harness design can affect performance, but it does not substitute for base-model capability. 3.3 Efficiency: the tradeoff analysis between performance and cost Difficulty# TasksOtherFrontierā 0.0ā0.11,46194%99%+5 p 0.1ā0.286280%95%+15 p 0.2ā0.366067%90%+23 p 0.3ā0.4495 54%83%+29 p 0.4ā0.542944%71%+27 p 0.5ā0.639235%62%+27 p 0.6ā0.7448 26%51%+25 p 0.7ā0.8460 17%39%+22 p 0.8ā0.94549%25%+16 p 0.9ā1.09661%4%+3 p 0.00.10.20.30.40.50.60.70.80.9 Empirical task difficulty = (1 ā mean pass rate) 0.2M 0.4M 0.6M 0.8M 1.0M Avg tokens per trial Frontier (tokens) Other (tokens) Frontier $/trial Other $/trial $0.00 $0.10 $0.20 $0.30 $0.40 $0.50 $0.60 Cost per trial Figure 3: Left: Benchmark scores across empirical task-difficulty buckets for three frontier models (Claude Opus 4.6, Gemini 3.1 Pro Preview, GPT 5.4) versus five other models (Claude Sonnet 4.6, Claude Haiku 4.5, Gemini 3.1 Flash Preview, GPT 5 mini, GPT 5 nano). Right: Average tokens per trial (bars) and dollar cost per trial (lines) by empirical task difficulty under our experiment setup. Frontier models are more capable but more expensive. We therefore ask: (1) where does upgrading to a frontier model pay off most, (2) do frontier models solve tasks with shorter trajectories, and (3) does that token efficiency offset their higher prices? We define empirical task difficulty as one minus the average score across all modelāharness runs, and group tasks into ten buckets of width 0.1. The distribution is U-shaped: 22.0% of tasks fall in the easiest bucket (0ā0.1), 14.6% in the hardest (0.9ā1), and the remaining 63.4% across the eight middle buckets. Note that this is a configuration-dependent proxy for intrinsic task difficulty, as it may be distorted by task brittleness (false positives/negatives) and the evaluation setup itself. (1) Marginal benefits. As shown in Figure 3 (left), the absolute performance gain of frontier models is bell-shaped:āis marginal on the simplest and hardest tasks and peaks in the medium band (0.3ā0.7). Other models use more tokens at every difficulty level yet cost 2ā3Ćless per trial (Figure 3, right). The frontier value proposition is therefore most pronounced on medium-to-hard tasks, where the gain in absolute performance justifies the higher cost. (2) Frontier models are more token-efficient, especially on easy tasks. Frontier models use fewer tokens than weaker models at every difficulty tier (Figure 3, right), and the relative gap is widest on easy tasks: in the easiest bucket (0ā0.1), frontier models use only 42% of weaker-model tokens, a ratio rising to 61ā86% in the upper half of the distribution (ā„ 0.5). These results suggest that frontier models produce more concise trajectories overall, but that their relative token-efficiency advantage is most pronounced on easy tasks. Manual inspection of agent trajectories further reveals that, on easy tasks, weaker models consume significantly more tokens, turns, and tokens per turn to reach the correct solution (e.g., 6.14Ć more tokens on SimpleQA [171] and 2.28Ćon KUMO [88]). The extra turns typically come from instruction misunderstanding and repetitive tool calls; the higher per-turn usage comes from verbose chain-of-thought and redundant verification compensating for weak reasoning. Frontier models instead follow instructions, execute cleanly, and accept outputs without extra deliberation. (3) Token savings do not offset higher prices. Higher per-token prices outweigh the token savings: other models cost 2ā3Ćless per trial across difficulty levels, and the absolute cost gap peaks at $0.37 5 per trial in the 0.7ā0.8 bucket. Token efficiency therefore reduces, but does not eliminate, the frontier cost premium. 3.4 Bottlenecks: failure mode analysis To see where and why agents fail, we audit trajectories where frontier models fail on hard tasks. Two domain-experienced human annotators independently label 200 trajectories across 12 failure modes (pooled inter-raterĪŗ = 0.66); a calibrated Gemini 3.1 Pro judge (Īŗ = 0.51vs gold) extends this to N = 6,028trajectories across 45 benchmarks, three frontier models, and two harnesses each. Full taxonomy, annotation protocol, and per-rubric reliability are in Appendix G.2.1. Wrong Factual Answer Algorithmic Bug Hidden-Test Regression Silent Deliverable Environment Block Syntax / Language Error Others 0 10 20 30 Prevalence (%) (Īŗ=0.69)(Īŗ=0.51)(Īŗ=0.75)(Īŗ=0.73)(Īŗ=0.58)(Īŗ=0.74)(Īŗ<0.5) GPT-5.4 + Codex GPT-5.4 + Terminus-2 Opus 4.6 + Claude Code Opus 4.6 + Terminus-2 Gemini 3.1 + Gemini CLI Gemini 3.1 + Terminus-2 Figure 4: Agent failure-mode prevalence onN = 6,028trajectories across seven rubrics. Each rubric shows six bars grouped by model: the darker bar is the native harness (Codex, Claude Code, Gemini CLI) and the lighter bar is the same model on Terminus-2. Per-rubric inter-raterĪŗis annotated above each group; Others collapses six rubrics with Īŗ < 0.5. Dominant failure modes. In Figure 4, wrong factual answers, algorithmic bugs, and hidden- test regressions dominate across all frontier models regardless of harness, indicating a persistent ācapability gapā in task comprehension, domain knowledge, and the reasoning needed to produce substantially correct solutions. All evaluated agents also still occasionally make trivial operational mistakes such as syntax errors or missing deliverables. Harness design shapes model behavior. Our human trajectory analysis (Section G.2.1) shows that native harnesses like Claude Code and Codex allow open-ended iteration and self-correction, whereas Terminus-2 enforces a linear Plan-Execute-Complete sequence; this lack of a built-in verification-correction loop makes Terminus-2 more vulnerable when iterative refinement is required. Models diverge in behavioral patterns. GPT-5.4 prioritizes conciseness, either delivering a correct solution quickly or āgiving upā on difficult tasks (e.g., HLE, LabBench), more so under Terminus- 2 than Codex. Claude models usually hold a steady turn count and consistent execution pattern regardless of task difficulty. Gemini relies heavily on external search, which, while useful, often causes āinformation driftā: it replaces an initially correct hypothesis with conflicting online informa- tion, notably on knowledge-intensive benchmarks (LabBench, MMMLU). Full per-(agent, model) breakdowns and examples are in Appendix G.2.2. 4 Harbor-Index: Compact, Diverse, Challenging, and High-Quality From our task inspection, low resolution rates in existing benchmarks are often attributable to structural design flaws (e.g., instruction-verification mismatches) rather than genuine task complexity. We therefore introduce Harbor-Index, a curated meta-dataset of 82 tasks for next- generation agent development and evaluation. Harbor-Index prioritizes compactness, diversity, difficulty, and quality, and is built by passing the entire Harbor trial pool through a multi-stage funnelādifficulty filtering, a series of AI audit and human review, then an iterative audit-and-fix loopāin which each stage rejects tasks that look hard but are structurally broken: a task enters Harbor-Index only if it is well-defined, genuinely difficult, and free of exploitable verifier loopholes. 6 GSO (7) SWE-Bench Verified (5) AlgoTune (5) FeatureBench (4) SWE-Bench Pro (4) SWE- Lancer (2) BigCodeBench (1) USACO (1) SWE-smith (1) SWT Bench (1) BIX-Bench (5) LAB-Bench (4) SciCode (3) SLDBench (1) Replication Bench (1) Code PDE (1) QCircuit Bench (1) GAIA2 (5) GAIA (3) Terminal Bench 2 (3) SkillsBench (2) Wide Search (1) HLE (8) GPQA Diamond (1) ARC-AGI-2 (5) OmniMath (2) Spider 2 (2) DA-Code (1) Cyber Gym (2) Figure 5: Harbor-Index task distribution by benchmark (82 tasks across 29 benchmarks), sized by task count and colored by domain. 4.1 Construction Pipeline Starting from an initial pool of 6,627 Harbor tasks across 54 adapters, we apply a difficulty filter retaining the 1,311 tasks where three leading models at the time of filtering (Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro), each under both its native harness and Terminus-2 over three repeats (18 trials), succeed on at most33%of trials. A Gemini-3-Flash auditor then scores each candidate against a quality rubric based on instructionāverification alignment and essential difficulty (difficulty must come from genuine reasoning, algorithmic thinking, domain expertise, long-horizon interactions, multi-step execution, etc.), filtering down to 307 tasks. Next, 14 domain-experienced human reviewers re-audit the survivors under the same rubric, each candidate receiving at least one senior or two junior reviews, leaving more than 110; a three-member senior panel then selects 100 on difficulty, diversity, and insight. Finally, at least two senior reviewers examine each of the 100 using trajectory-grounded failure analysis and false-positive/false-negative analysis, repairing broken tasks and dropping those that remain broken or become too easy after repair, yielding the final release of 82 tasks spanning 29 benchmarks. Figure 5 shows the task distribution, with numbers denoting task counts per benchmark. The full audit process, quality bar, and sampling protocol are in Appendix H. Our human audit reveals that broken task design is common: across more than 30 benchmarks, roughly one third of the hardest candidate tasks are rejected as broken rather than genuinely difficult. On all four GAIA2 [45] tasks reaching human review, the adapter never fires the simulation events the verifier waits on, so those trials time out regardless of agent approach. A SWE-bench Pro [33] task shows verifier overreach: it assertsdata-testidstrings and log message formats the instruction never specifies, so a correct solution fails on clerical grounds. On CRustBench [67], the verifier pins transpilation to a fixed hand-written Rust interface, so a correct and memory-safe solution is rejected when signatures or ownership annotations differ from the reference. 4.2 Evaluation Settings We evaluate 9 models spanning capability tiers and both release types: four closed-weight models, whose weights are not publicly released (GPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro, and Qwen3.7 Max), and five open-weight models (GLM 5.2, Kimi K2.6, MiniMax M3, DeepSeek V4 Pro, and MiMo V2.5 Pro). GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro are served by first-party APIs; the other six models are served through OpenRouter [118]. All models run under default temperature, context length, and reasoning effort. Each model runs under two harness conditions: Terminus-2 and a model-native harness. This yields9Ć 2Ć 82 = 1,476rollouts on Harbor-Index 1.0; the model 7 Table 1: Results on Harbor-Index 1.0 (82 tasks): pass rate (%) and approximate per-run cost (USD) for a full Index run. The Native Harness column group uses the vendor-native harnessāCodex, Claude Code, or Gemini CLIāfor GPT, Claude, and Gemini, and Claude Code for the six models served through OpenRouter. Models marked ā are closed-weight (weights not publicly released); the rest are open-weight. The best pass rate in each row is shaded green. Costs reconstruct token usage at official API pricing for GPT/Claude/Gemini and OpenRouter pricing for the other models, whose cost Claude Code on OpenRouter can inflate via low cache-hit rates. Native HarnessTerminus-2 ModelAgentPass %$/runPass %$/run GPT-5.5 ā Codex28.017819.5155 Claude Opus 4.8 ā Claude Code20.726915.9293 Gemini 3.1 Pro ā Gemini CLI13.47411.089 GLM 5.2Claude Code8.5205 9.852 Kimi K2.6Claude Code6.11918.533 MiniMax M3Claude Code3.7666.118 Qwen3.7 Max ā Claude Code4.92014.936 DeepSeek V4 ProClaude Code 4.91773.735 MiMo V2.5 ProClaude Code2.4492.44 list and results appear in Table 1. Free-form answer tasks (e.g., HLE, GAIA, GPQA Diamond) use LLM-as-a-judge evaluation; elsewhere scoring is programmatic (unit tests, exact match, thresholded continuous metrics). For continuous metrics such as speedup ratios in AlgoTune [128] and GSO [142], we āSOTA-thresholdā outcomes into pass/fail, so these tasks only reward solutions that advance the current frontier. See Appendix E for implementation details. 4.3 Evaluation Outcome 0102030 Harbor-Index pass rate (%) GPT-5.5 Claude Opus 4.8 Gemini 3.1 Pro GLM 5.2 Kimi K2.6 MiniMax M3 DeepSeek V4 Pro Qwen3.7 Max MiMo V2.5 Pro 19.5 28.0 15.9 20.7 11.0 13.4 9.8 8.5 8.5 6.1 6.1 3.7 3.7 4.9 4.9 4.9 2.4 2.4 Codex Claude Code Gemini CLI Terminus 2 $10$30$100$300 Cost per run (USD) Ā· log scale 0 5 10 15 20 25 30 35 40 Harbor-Index pass rate (%) Native CLITerminus 2Pareto frontier Figure 6: Performanceācost tradeoff on Harbor-Index 1.0. Left: Pass rates of the two evaluated harnesses for each model. Right: Pass rate versus cost for one complete Harbor-Index run (82 tasks). Each point is one modelāharness configuration. Circles denote native harness and squares denote Terminus-2 for every model. Colored points and the dashed line mark the costāperformance Pareto frontier; gray points are dominated. Table 1 and Figure 6 show the suite is lightweight enough for repeated evaluation. The three highest- scoring models (GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro) benefit most from their vendor-native harnesses, whereas several open-weight models sit on the costāperformance Pareto frontier under Terminus-2, trading pass rate for a far lower cost per run. Under the current evaluation constraints, agent failures are separated into453timeouts with no answer,361near-miss solutions, and445far-off fundamentally wrong answers. Capability and harness jointly shape these outcomes: the six lower-scoring models time out about twice as often 8 as the three highest-scoring ones (36.7%versus18.7%of their respective rollouts), while the native harnesses use fewer tool calls and output tokens than Terminus-2 and, matched task by task, cut the timeout rate from42%to26%. Terminus-2ās bash-only action space, lacking native image and web tools, explains part of this gap and cautions against attributing every failure to the model alone. Interactive trajectories, scores, and task-level audit artifacts are on the Harbor-Index website. 1 5 Related Work Agentic benchmarks and evaluation infrastructure. Agent benchmarks span tool and API use [121,187,131], embodied and science environments [146,166,186], web and enterprise naviga- tion [207,34,36], application and OS control [159,134,179], software engineering from functions to repositories and terminals [20,64,103], scientific and workplace pipelines [16,21,180], and aggregate collections [91,104,109]. This diversity is informative but blocks aggregation: each benchmark ships its own action space, runtime assumptions, and scoring. Infrastructure work further shows a score is not a property of the model alone: interface, scaffold, cost budget, and holdout hygiene all move reported numbers [11,160,182,175,66]. HAL decomposes model, scaffold, and benchmark effects over 21,730 rollouts [65], while AstaBench controls cost and tool-access confounders within scientific research [12]. Benchmark validity and task quality.Aggregate scores inherit the design flaws of their tasks and metrics [133,9,136]: label errors destabilize aggregates [116,47], contamination erodes held-out validity [139,172], and leaderboard mechanics skew comparisons [147]. This sharpens for agents, whose verifier executes code rather than matching a label. An imperfect verifier caps attainable accuracy regardless of compute, since resampling cannot lower its false-positive rate [152]; agentic benchmarks misdesign setup or reward often enough to shift measured performance by up to100% relative [208]; and SWE-bench audits trace many āresolvedā instances to leakage, memorization, and weak tests, enough that strengthening those tests reorders the leaderboard [3,168,192,86]. Model judges and live services add further error [202, 95, 51]. Efficient evaluation and benchmark redundancy. Model-by-benchmark score matrices are intrinsically low-rank [13,101,197,200], motivating curated subsets and adaptive item selection [127, 70,56]. Compression has limits: agreement estimates depend on unstandardized choices [124], subset prediction degrades on stronger models [199], and micro-benchmarks need more items than assumed [189, 105]. 6 Discussion and Conclusion Limitations and future work. Adapter availability limits our large-scale evaluation to 54 of the collected adapter benchmarks; cost restricts us to a focused set of popular models under native harnesses and Terminus-2, so our findings do not span all LLMs, agent architectures, or evaluation settings and are not universal. Moreover, stronger future agents may exploit benchmark artifacts or evaluation loopholes more effectively that leads to currently unexploitable reward hacks and unreliable evaluation. Therefore, we plan maintain Harbor-Index as a live benchmark, updating it to reduce saturation, preserve quality, and integrating more adapters. Conclusion. Harbor Adapters address the fragmentation of agentic evaluation: standardizing heterogeneous benchmarks under the Harbor task interface reduces benchmarkāagent integration fromO(mn)toO(m +n). Across validated adapters and a large-scale evaluation of 54 benchmarks, 16 modelāharness configurations spanning capability tiers, and 6,627 tasks, we find that base-model capability is the primary driver of performance gaps; many benchmarks contain redundant ranking signal; frontier models are more token-efficient but still substantially more costly; and low resolution tasks often reflect task-design flaws rather than genuine difficulty. Harbor-Index 1.0 distills this suite into 82 compact, difficult, diverse, and audited tasks spanning 29 benchmarks. Together, Harbor Adapters and Harbor-Index provide reusable infrastructure and an affordable, high-quality testbed for more reliable, scalable, and informative evaluation of language-model agents. 1 https://harbor-index.org 9 References [1]2077AI Foundation. 2077AI: Open-Source Innovation Foundation.https://w.2077ai .com/, 2026. [2] Aider AI. Aider polyglot benchmark, 2024. [3]Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. SWE-Bench+: Enhanced coding benchmark for LLMs, 2024. arXiv:2410.06992. [4]Alibaba Cloud. Alibaba cloud Model Studio model pricing.https://w.alibabacloud.c om/help/en/model-studio/model-pricing, 2026. Accessed: 2026-04-22 (qwen3-max, US Global tab) and 2026-05-07 (qwen3.6-max-preview, Chinese mainland tab). [5]Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, J Zico Kolter, Matt Fredrikson, Yarin Gal, and Xander Davies. Agentharm: A benchmark for measuring harmfulness of LLM agents. In The Thirteenth International Conference on Learning Representations, 2025. [6]Anthropic. Claude API pricing.https://platform.claude.com/docs/en/about-cla ude/pricing, 2026. Accessed: 2026-04-30. [7] Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. Swe- rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents. In The Thirty-ninth Conference on Neural Information Processing Systems, 2025. [8]Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'AlchĆ©-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. [9]Andrew M. Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan, Chris Schmitz, Karolina Korgul, Hunar Batra, Oishi Deb, Emma Beharry, Cornelius Emde, Thomas Foster, Anna Gausen, MarĆa Grandury, Simeng Han, Valentin Hofmann, Lujain Ibrahim, Hazel Kim, Hannah Rose Kirk, Fangru Lin, Gabrielle Kaili-May Liu, Lennart Luettgau, Jabez Magomere, Jonathan RystrĆøm, Anna Sotnikova, Yushi Yang, Yilun Zhao, Adel Bibi, Antoine Bosselut, Ronald Clark, Arman Cohan, Jakob Foerster, Yarin Gal, Scott A. Hale, Inioluwa Deborah Raji, Christopher Summerfield, Philip H. S. Torr, Cozmin Ududec, Luc Rocher, and Adam Mahdi. Measuring what matters: Construct validity in large language model benchmarks, 2025. NeurIPS 2025 Datasets and Benchmarks Track, arXiv:2511.04703. [10]Gabrielle Berrada, Jannik Kossen, Freddie Bickford Smith, Muhammed Razzak, Yarin Gal, and Tom Rainforth. Scaling up active testing to large language models, 2025. NeurIPS 2025, arXiv:2508.09093. [11]Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, FranƧois Yvon, and Andy Zou. Lessons from the trenches on reproducible evaluation of language models, 2024. [12] Jonathan Bragg, Mike DāArcy, Nishant Balepur, Dan Bareket, Bhavana Dalvi, Sergey Feld- man, Dany Haddad, Jena D. Hwang, Peter Jansen, Varsha Kishore, Bodhisattwa Prasad Majumder, Aakanksha Naik, Sigal Rahamimov, Kyle Richardson, Amanpreet Singh, Harshit Surana, Aryeh Tiktinsky, Rosni Vasu, Guy Wiener, Chloe Anastasiades, Stefan Candra, Jason Dunkelberger, Dan Emery, Rob Evans, Malachi Hamada, Regan Huff, Rodney Kinney, Matt Latzke, Jaron Lochner, Ruben Lozano-Aguilera, Cecile Nguyen, Smita Rao, Amber Tanaka, 10 Brooke Vlahos, Peter Clark, Doug Downey, Yoav Goldberg, Ashish Sabharwal, and Daniel S. Weld. Astabench: Rigorous benchmarking of ai agents with a scientific research suite. In International Conference on Learning Representations (ICLR), 2026. [13]Ryan Burnell, Han Hao, Andrew R. A. Conway, and Jose Hernandez Orallo. Revealing the structure of language model capabilities, 2023. arXiv:2306.10062. [14]Nitay Calderon, Roi Reichart, and Rotem Dror. The alternative annotator test for LLM- as-a-Judge: How to statistically justify replacing human annotators with LLMs, 2025. arXiv:2501.10970. [15]Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Anton Lozhkov, Carolyn Jane Anderson, and Arjun Guha. Can it edit? evaluating the ability of large language models to follow code editing instructions. In Conference on Language Modeling (COLM), 2024. [16]Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng. MLE-bench: Evaluating machine learning agents on machine learning engineering. In The Thirteenth International Conference on Learning Representations, 2025. [17] Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Yuefeng Huang, Wulong Liu, Xinzhi Wang, Defu Lian, Baoqun Yin, Yasheng Wang, and Wu Liu. Acebench: Who wins the match point in tool usage?, 2025. [18]Haorui Chen, Chengze Li, and Jia Li. Featbench: Evaluating coding agents on feature implementation for vibe coding, 2026. [19]Hui Chen, Miao Xiong, Yujie Lu, Wei Han, Ailin Deng, Yufei He, Jiaying Wu, Yibo Li, Yue Liu, and Bryan Hooi. Mlr-bench: Evaluating ai agents on open-ended machine learning research. In The Thirty-ninth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. [20]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. [21] Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery. In The Thirteenth International Conference on Learning Representations, 2025. [22]Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. In Forty-first International Conference on Machine Learning, 2024. [23]Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. Arc- agi-2: A new challenge for frontier ai reasoning systems, 2026. [24]JoĆ£o Coelho, Jingjie Ning, Jingyuan He, Kangrui Mao, Abhijay Paladugu, Pranav Setlur, Jiahe Jin, Jamie Callan, JoĆ£o MagalhĆ£es, Bruno Martins, and Chenyan Xiong. DeepRe- searchGym: A free, transparent, and reproducible evaluation sandbox for deep research, 2025. arXiv:2505.19253. 11 [25]Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. [26] Marc-Alexandre CĆ“tĆ©, Ćkos KĆ”dĆ”r, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Ruo Yu Tao, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. Textworld: A learning environment for text-based games, 2019. [27]Cainan Davidson, Deva Ramanan, and Neehar Peri. Refav: Towards planning-centric scenario mining, 2025. [28]Daytona Platforms, Inc. Daytona Sandboxes.https://w.daytona.io/docs/en/sandb oxes/, 2026. [29]Thibault Le Sellier de Chezelles, Maxime Gasse, Alexandre Lacoste, Massimo Caccia, Alexan- dre Drouin, LĆ©o Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han LĆ¹, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Gra- ham Neubig, Quentin Cappart, Russ Salakhutdinov, and Nicolas Chapados. The browsergym ecosystem for web agent research. Transactions on Machine Learning Research, 2025. Expert Certification. [30]Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian TramĆØr. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. [31]DeepSeek. DeepSeek API models and pricing.https://api-docs.deepseek.com/quick _start/pricing, 2026. Accessed: 2026-04-22. [32]Adrienne Deganutti, Elad Hirsch, Haonan Zhu, Jaejung Seol, and Purvanshi Mehta. Graphic- design-bench: A comprehensive benchmark for evaluating ai on graphic design tasks, 2026. [33]Xiang Deng, Jeff Da, Edwin Pan, Yannis Y. He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa R Kundurthy, Sean M. Hendryx, Zifan Wang, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. SWE-bench pro: Can AI agents solve long-horizon software engineering tasks?, 2026. [34]Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. [35]Shihan Dou, Ming Zhang, Zhangyue Yin, Chenhao Huang, Yujiong Shen, Junzhe Wang, Jiayi Chen, Yuchen Ni, Junjie Ye, Cheng Zhang, Huaibing Xie, Jianglu Hu, Shaolei Wang, Weichao Wang, Yanling Xiao, Yiting Liu, Zenan Xu, Zhen Guo, Pluto Zhou, Tao Gui, Zuxuan Wu, Xipeng Qiu, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang, Di Wang, and Shunyu Yao. Cl-bench: A benchmark for context learning, 2026. [36]Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, LĆ©o Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapa- dos, and Alexandre Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks?, 2024. [37]Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao.Deep- research bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506.11763, 2025. [38] E2B. E2B Documentation. https://e2b.dev/docs, 2026. [39] Nicholas Edwards, Yukyung Lee, Yujun Audrey Mao, Yulu Qin, Sebastian Schuster, and Na- joung Kim. RExbench: Can coding agents autonomously implement AI research extensions?, 2025. 12 [40]Alexander David Egg, Martin Iglesias Goyanes, Andreu Mora, Friso H. Kingma, Thomas Wolf, and Leandro Von Werra. DABstep: Data agent benchmark for multi-step reasoning, 2026. [41]Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yuksekgonul, and James Zou. Cost-of-Pass: An economic framework for evaluating language models, 2025. arXiv:2504.13359. [42]Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. Lawbench: Benchmarking legal knowledge of large language models, 2023. [43]Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, Jie Sun, Yang Xiao, Sizhuo Zhou, Wenxiao Wu, Yiming Liu, Pengfei Liu, Yu Qiao, Shenglin Zhang, and Kaipeng Zhang. Longcli-bench: A preliminary benchmark and study for long-horizon agentic programming in command-line interfaces, 2026. [44]Timo Freiesleben and Sebastian Zezulka. The benchmarking epistemology: Construct validity for evaluating machine learning models, 2025. arXiv:2510.23191. [45] Romain Froger, Pierre Andrews, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Emilien Garreau, Jean-Baptiste Gaya, Hugo LaurenƧon, Maxime Lecanu, Kunal Malkan, Dheeraj Mekala, Pierre Menard, Gerard Moreno-Torres Bertran, Ulyana Piterbarg, Mikhail Plekhanov, Mathieu Rita, Andrey Rusakov, Vladislav Vorotilov, Mengjue Wang, Ian Yu, Amine Benhalloum, GrĆ©goire Mialon, and Thomas Scialom. Gaia2: Benchmarking LLM agents on dynamic and asynchronous environments. In The Fourteenth International Conference on Learning Representations, 2026. [46]Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni-MATH: A universal olympiad level mathematic benchmark for large language models. In The Thirteenth International Conference on Learning Representations, 2025. [47] Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini. Are we done with mmlu?, 2025. [48]Luca Gioacchini, Giuseppe Siracusano, Davide Sanvito, Kiril Gashteovski, David Friede, Roberto Bifulco, and Carolin Lawrence. Agentquest: A modular benchmark framework to measure progress and improve llm agents, 2024. [49]Google. Gemini API pricing.https://ai.google.dev/gemini-api/docs/pricing, 2026. Accessed: 2026-04-30. [50]Leon Guertler, Bobby Cheng, Simon Yu, Bo Liu, Leshem Choshen, and Cheston Tan. Textarena, 2025. [51] Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models, 2025. [52]Tanmay Gupta, Ryan Marten, Aniruddha Kembhavi, and Derek Hoiem. Grit: General robust image task benchmark, 2022. [53]Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments, January 2026. [54]Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models, 2024. 13 [55]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. [56]Valentin Hofmann, David Heineman, Ian Magnusson, Kyle Lo, Jesse Dodge, Maarten Sap, Pang Wei Koh, Chun Wang, Hannaneh Hajishirzi, and Noah A. Smith. Fluid language model benchmarking, 2025. COLM 2025, arXiv:2509.11106. [57]Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization, 2020. [58]Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, Yuhuai Li, Shengze Xu, Shenzhi Wang, Xinchen Xu, Shuofei Qiao, Zhaokai Wang, Kun Kuang, Tieyong Zeng, Liang Wang, Jiwei Li, Yuchen Eleanor Jiang, Wangchunshu Zhou, Guoyin Wang, Keting Yin, Zhou Zhao, Hongxia Yang, Fan Wu, Shengyu Zhang, and Fei Wu. OS agents: A survey on MLLM-based agents for general computing devices use, 2025. ACL 2025, arXiv:2508.04482. [59]Tianyu Hua, Harper Hua, Violet Xiang, Benjamin Klieger, Sang T. Truong, Weixin Liang, Fan-Yun Sun, and Nick Haber. Researchcodebench: Benchmarking LLMs on implementing novel machine learning research code. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. [60]Kung-Hsiang Huang, Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang, Silvio Savarese, Caiming Xiong, Philippe Laban, and Chien-Sheng Wu. Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments, 2025. [61]Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, and Kang Liu. Da-code: Agent data science code generation benchmark for large language models, 2024. [62]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contam- ination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025. [63]Yixing Jiang, Kameron C. Black, Gloria Geng, Danny Park, James Zou, Andrew Y. Ng, and Jonathan H. Chen. Medagentbench: A realistic virtual ehr environment to benchmark medical llm agents, 2025. [64]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. [65]Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary S Siegel, Boyi Wei, Tianci Xue, Ziru Chen, Felix Chen, Saiteja Utpala, Franck Ndzomga, Dheeraj Oruganty, Sophie Luskin, Kangheng Liu, Botao Yu, Amit Arora, Dongyoon Hahm, Harsh Trivedi, Huan Sun, Juyong Lee, Tengjun Jin, Yifan Mai, Yifei Zhou, Yuxuan Zhu, Rishi Bommasani, Daniel Kang, Dawn Song, Peter Henderson, Yu Su, Percy Liang, and Arvind Narayanan. Holistic agent leaderboard: The missing infrastructure for AI agent evaluation. In The Fourteenth International Conference on Learning Representations, 2026. [66]Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI agents that matter, 2024. arXiv:2407.01502. [67]Anirudh Khatry, Robert Zhang, Jia Pan, Ziteng Wang, Qiaochu Chen, Greg Durrett, and Isil Dillig. CRUST-bench: A comprehensive benchmark for c-to-safe-rust transpilation. In Second Conference on Language Modeling, 2025. [68]Arpandeep Khatua, Hao Zhu, Peter Tran, Arya Prabhudesai, Frederic Sadrieh, Johann K. Lieberwirth, Xinkai Yu, Yicheng Fu, Michael J. Ryan, Jiaxin Pei, and Diyi Yang. Cooperbench: Why coding agents cannot be your teammates yet, 2026. 14 [69]Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmarking in nlp, 2021. [70] Alex Kipnis, Konstantinos Voudouris, Luca M. Schulze Buschoff, and Eric Schulz. metabench ā a sparse benchmark of reasoning and knowledge in large language models, 2025. ICLR 2025, arXiv:2407.12844. [71]Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024. [72] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts, 2021. [73] Thomas Kuntz, Agatha Duzan, Hao Zhao, Francesco Croce, Zico Kolter, Nicolas Flammarion, and Maksym Andriushchenko. OS-Harm: A benchmark for measuring safety of computer use agents, 2025. NeurIPS 2025 Datasets and Benchmarks Track, arXiv:2506.14866. [74] Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes, and Lawrence Chan. Measuring AI ability to complete long software tasks, 2025. NeurIPS 2025, arXiv:2503.14499. [75]Eugenie Lai, Gerardo Vitagliano, Ziyu Zhang, Om Chabra, SIVAPRASAD SUDHIR, Anna Zeng, Anton A. Zabreyko, Chenning Li, Ferdi Kossmann, Jialin Ding, Jun Chen, Markos Markakis, Matthew Russo, Weiyang Wang, Ziniu Wu, Mike Cafarella, Lei Cao, Samuel Madden, and Tim Kraska. KRAMABENCH: A benchmark for AI systems on data-to- insight pipelines over data lakes. In The Fourteenth International Conference on Learning Representations, 2026. [76]Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation, 2022. [77] Laude Institute. Laude Institute. https://w.laude.org/, 2026. [78] Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White, and Samuel G. Rodriques. Lab- bench: Measuring capabilities of language models for biology research, 2024. [79] Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin SU, ZHAO- QING SUO, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. Spider 2.0: Evaluating language models on real-world enterprise text-to-SQL workflows. In The Thirteenth International Conference on Learning Representations, 2025. [80]Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du, Ping Yang, Dahua Lin, Chao Peng, and Kai Chen. Prompting large language models to tackle the full software development lifecycle: A case study, 2024. [81]Jinyang Li, Binyuan Hui, GE QU, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can LLM already serve as a database interface? a BIg bench for large-scale database grounded text-to-SQLs. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. 15 [82]Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms, 2023. [83]Shanda Li, Tanya Marwah, Junhong Shen, Weiwei Sun, Andrej Risteski, Yiming Yang, and Ameet Talwalkar. CodePDE: An inference framework for LLM-driven PDE solver generation, 2025. [84]Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, Bingran You, Haotian Shen, Jiankai Sun, Shuyi Wang, Binxu Li, Qunhong Zeng, Di Wang, Xuandong Zhao, Yuanli Wang, Roey Ben Chaim, Zonglin Di, Yipeng Gao, Junwei He, Yizhuo He, Liqiang Jing, Luyang Kong, Xin Lan, Jiachen Li, Songlin Li, Yijiang Li, Yueqian Lin, Xinyi Liu, Xuanqing Liu, Haoran Lyu, Ze Ma, Bowei Wang, Runhui Wang, Tianyu Wang, Wengao Ye, Yue Zhang, Hanwen Xing, Yiqi Xue, Steven Dillmann, and Han chung Lee. Skillsbench: Benchmarking how well agent skills work across diverse tasks, 2026. [85] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. Featured Certification, Expert Certification, Outstanding Certification. [86]Shanchao Liang, Spandan Garg, and Roshanak Zilouchian Moghaddam. The SWE-Bench illusion: When State-of-the-Art LLMs remember instead of reason, 2025. arXiv:2506.12286. [87]Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. QuixBugs: A multi- lingual program repair benchmark set based on the quixey challenge. Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity, 2017. [88] Haowei Lin, Xiangyu Wang, Ruilin Yan, Baizhou Huang, Haotian Ye, Jianhua Zhu, Zihao Wang, James Zou, Jianzhu Ma, and Yitao Liang. Generative evaluation of complex reasoning in large language models, 2025. [89]Haowei Lin, Haotian Ye, Wenzheng Feng, Quzhe Huang, Yujun Li, Hubert Lim, Zhengrui Li, Xiangyu Wang, Jianzhu Ma, Yitao Liang, and James Zou. Can language models discover scaling laws? In The Fourteenth International Conference on Learning Representations, 2026. [90]Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations, 2018. [91]Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms as agents, 2025. [92]Yu Lu Liu, Su Lin Blodgett, Jackie Chi Kit Cheung, Q. Vera Liao, Alexandra Olteanu, and Ziang Xiao. ECBD: Evidence-Centered benchmark design for NLP, 2024. arXiv:2406.08723. [93]Zichen Liu, Anya Sims, Keyu Duan, Changyu Chen, Simon Yu, Xiangxin Zhou, Haotian Xu, Shaopan Xiong, Bo Liu, Chenmien Tan, Chuen Yang Beh, Weixun Wang, Hao Zhu, Weiyan Shi, Diyi Yang, Michael Shieh, Yee Whye Teh, Wee Sun Lee, and Min Lin. GEM: A gym for agentic LLMs, 2025. arXiv:2510.01051. [94]Pan Lu, Jiayi Sheng, Luna Lyu, Jikai Jin, Tony Xia, Alex Gu, and James Zou. Solving inequality proofs with large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. 16 [95]Xing Han LĆ¹, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Ale- jandra Zambrano, Karolina Stanczak, Peter Shaw, Christopher Pal, and Siva Reddy. Agentre- wardbench: Evaluating automatic evaluations of web agent trajectories. In Second Conference on Language Modeling, 2025. [96] Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Webgen-bench: Evaluating LLMs on generating interactive and functional websites from scratch. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. [97]Xing Han LĆ¹, Zden Ė ek Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi-turn dialogue, 2024. [98]Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang. Spreadsheetbench: Towards challenging real world spreadsheet manipulation, 2024. [99]Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Yu Wu, Robin Jia, Christo- pher Potts, Adina Williams, and Douwe Kiela. Dynaboard: An evaluation-as-a-service platform for holistic next-generation benchmarking. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. [100]Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of llm agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13851ā13870, 2024. [101]Aviya Maimon, Amir DN Cohen, Gal Vishne, Shauli Ravfogel, and Reut Tsarfaty. From benchmarks to skills: Low-Rank factors for LLM evaluation, 2025. arXiv:2507.20208. [102]Qiuyang Mang, Wenhao Chai, Zhifei Li, Huanzhi Mao, Shang Zhou, Alexander Du, Hanchen Li, Shu Liu, Edwin Chen, Yichuan Wang, Xieting Chu, Zerui Cheng, Yuan Xu, Tian Xia, Zirui Wang, Tianneng Shi, Jianzhu Yao, Yilong Zhao, Qizheng Zhang, Charlie Ruan, Zeyu Shen, Kaiyuan Liu, Runyuan He, Dong Xing, Zerui Li, Zirong Zeng, Yige Jiang, Lufeng Cheng, Ziyi Zhao, Youran Sun, Wesley Zheng, Meiyuwang Zhang, Ruyi Ji, Xuechang Tu, Zihan Zheng, Zexing Chen, Kangyang Zhou, Zhaozi Wang, Jingbang Chen, Aleksandra Korolova, Peter Henderson, Pramod Viswanath, Vijay Ganesh, Saining Xie, Zhuang Liu, Dawn Song, Sewon Min, Ion Stoica, Joseph E. Gonzalez, Jingbo Shang, and Alvin Cheung. Frontiercs: Evolving challenges for evolving intelligence, 2025. [103]Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Me- nis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggar- wal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjƶrn Kolbeinsson, Jesse Hu, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Alex Dimakis, Andy Konwinski, and Ludwig Schmidt. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces, 2026. [104] GrĆ©goire Mialon, ClĆ©mentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants, 2023. [105]Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations, 2024. arXiv:2411.00640. 17 [106]Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. SWE-lancer: Can frontier LLMs earn $1 million from real-world freelance software engineering? In Forty-second International Conference on Machine Learning, 2025. [107] Ludovico Mitchener, Jon M Laurent, Alex Andonian, Benjamin Tenmann, Siddharth Narayanan, Geemi P Wellawatte, Andrew White, Lorenzo Sani, and Samuel G Rodriques. Bixbench: a comprehensive benchmark for llm-based agents in computational biology, 2025. [108] Modal Labs. Modal Sandboxes. https://modal.com/docs/guide/sandboxes, 2026. [109]Mahmoud Mohammadi, Yipeng Li, Jane Lo, and Wendy Yip. Evaluation and benchmarking of LLM agents: A survey, 2025. arXiv:2507.21504. [110]Moonshot AI. Kimi K2.5 API pricing.https://platform.kimi.ai/docs/pricing/ch at-k25, 2026. Accessed: 2026-04-22. [111]Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. In The Twelfth International Conference on Learning Representations, 2024. [112] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive text embedding benchmark. In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014ā2037, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. [113]Niels MĆ¼ndler, Mark Niklas MĆ¼ller, Jingxuan He, and Martin Vechev. Swt-bench: Testing and validating real-world bug-fixes with code agents, 2025. [114]Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Mikhail Plekhanov, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, Dieuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob Nicolaus Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileanu. MLGym: A new framework and benchmark for advancing AI research agents. In Second Conference on Language Modeling, 2025. [115]Yannic Neuhaus and Matthias Hein. RePOPE: Impact of annotation errors on the POPE benchmark, 2025. arXiv:2504.15707. [116]Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks, 2021. NeurIPS 2021 Datasets and Benchmarks Track, arXiv:2103.14749. [117]OpenAI. API pricing.https://developers.openai.com/api/docs/pricing, 2026. Accessed: 2026-04-30. [118] OpenRouter, Inc. OpenRouter: The Unified Interface for LLMs.https://openrouter.ai/, 2026. [119] Harshith Padigela, Chintan Shah, and Dinkar Juyal. ML-dev-bench: Comparative analysis of AI agents on ML development workflows. In Towards Agentic AI for Science: Hypothesis Generation, Comprehension, Quantification, and Validation, 2025. [120] Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with SWE-gym. In Forty-second International Conference on Machine Learning, 2025. [121]Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. [122]Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis, 2023. 18 [123]Debjit Paul, Daniel Murphy, Milan Gritta, Ronald Cardenas, Victor Prokhorov, Lena Sophia Bolliger, Aysim Toker, Roy Miles, Andreea-Maria Oncescu, Jasivan Alex Sivakumar, Philipp Borchert, Ismail Elezi, Meiru Zhang, Ka Yiu Lee, Guchun Zhang, Jun Wang, and Gerasimos Lampouras. A benchmark for deep information synthesis. In The Fourteenth International Conference on Learning Representations, April 2026. [124] Yotam Perlitz, Ariel Gera, Ofir Arviv, Asaf Yehudai, Elron Bandel, Eyal Shnarch, Michal Shmueli-Scheuer, and Leshem Choshen. Do these LLM benchmarks agree? fixing benchmark evaluation with BenchBench, 2024. arXiv:2407.13696. [125] Thinh Pham, Nguyen Phan Nguyen, Pratibha Zunjare, Weiyuan Chen, Yu-Min Tseng, and Tu Vu. SealQA: Raising the bar for reasoning in search-augmented language models. In The Fourteenth International Conference on Learning Representations, 2026. [126]Long Phan, Alice Gatti, Nathaniel Li, Adam Khoja, Ryan Kim, Richard Ren, et al. A bench- mark of expert-level academic questions to assess AI capabilities. Nature, 649(8099):1139ā 1146, 2026. [127]Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating llms with fewer examples, 2024. [128]Ori Press, Brandon Amos, Haoyu Zhao, Yikai Wu, Samuel Ainsworth, Dominik Krupke, Patrick Kidger, Touqir Sajed, Bartolomeo Stellato, Jisun Park, Nathanael Bosch, Eli Meril, Albert Steppi, Arman Zharmagambetov, Fangzhao Zhang, David PĆ©rez-PiƱeiro, Alberto Mercurio, Ni Zhan, Talor Abramovich, Kilian Lieret, Hanlin Zhang, Shirley Huang, Matthias Bethge, and Ofir Press. Algotune: Can language models speed up general-purpose numerical programs? In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. [129] Ariel Procaccia, Benjamin Schiffer, Serena Wang, and Shirley Zhang. Metritocracy: Represen- tative metrics for lite benchmarks, 2025. arXiv:2506.09813. [130] Shuofei Qiao, Runnan Fang, Zhisong Qiu, Xiaobin Wang, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Benchmarking agentic workflow generation. In The Thirteenth International Conference on Learning Representations, 2025. [131]Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In The Twelfth International Conference on Learning Representations, 2024. [132]Quesma. CompileBench: Evaluating AI agents on real-world software compilation tasks, 2025. [133] Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. AI and the everything in the whole wide world benchmark, 2021. NeurIPS 2021 Datasets and Benchmarks Track, arXiv:2111.15366. [134] Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. Androidworld: A dynamic benchmarking environment for autonomous agents, 2025. [135]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. [136] Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, and Mykel J. Kochenderfer. BetterBench: Assessing AI benchmarks, uncovering issues, and establishing best practices, 2024. NeurIPS 2024, arXiv:2411.12990. [137] Yangjun Ruan, Chris J. Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of language model performance, 2024. NeurIPS 2024, arXiv:2405.10938. 19 [138]Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge, 2015. [139]Oscar Sainz, Jon Ander Campos, Iker GarcĆa-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark, 2024. EMNLP 2024 Findings, arXiv:2310.18018. [140]S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. MMAU: A massive multi-task audio understanding and reasoning benchmark. In The Thirteenth International Conference on Learning Representations, 2025. [141]Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. Taskbench: Benchmarking large language models for task automation, 2024. [142]Manish Shetty, Naman Jain, Jinjian Liu, Vijay Kethanaboyina, Koushik Sen, and Ion Stoica. GSO: Challenging software optimization tasks for evaluating SWE-agents. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. [143]Ben Shi, Michael Tang, Karthik R Narasimhan, and Shunyu Yao. Can language models solve olympiad programming? In First Conference on Language Modeling, 2024. [144]Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. NeurIPS 2023, arXiv:2303.11366. [145]Parshin Shojaee, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, and Chandan K. Reddy. LLM-SRBench: A new benchmark for scientific equation discovery with large language models. In Forty-second International Conference on Machine Learning, 2025. [146]Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021. [147]Shivalika Singh, Yiyang Nan, Alex Wang, Daniel DāSouza, Sayash Kapoor, Ahmet ĆstĆ¼n, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah A. Smith, Beyza Ermis, Marzieh Fadaee, and Sara Hooker. The leaderboard illusion, 2025. arXiv:2504.20879. [148]Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. A strongREJECT for empty jailbreaks. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. [149]Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, AdriĆ Garriga-Alonso, Ag- nieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas StuhlmĆ¼ller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karaka Ģ§s, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, BartÅomiej Bojanowski, Batuhan Ćzyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, CĆ©sar Ferri RamĆrez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison- Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy 20 Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel MoseguĆ GonzĆ”lez, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Deba- jyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando MartĆnez-Plumed, Francesca HappĆ©, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, GermĆ”n Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-LĆ³pez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich SchĆ¼tze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime FernĆ”ndez Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Koco Ģ n, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jƶrg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros ColĆ³n, Luke Metz, LĆ¼tfi Kerem Ģ§Senel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose RamĆrez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, MĆ”tyĆ”s Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, MichaÅ Sw ĢØedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr MiÅkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, RaphaĆ«l MilliĆØre, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchi- gar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, ThĆ©o Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, 21 Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Pad- makumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023. [150] Benn Stancil. ADE-bench: A framework for evaluating ai agents on data analyst tasks, 2025. [151] Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, and Andreas Kƶpf. Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. [152] Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan. The limits of inference scaling through resampling, 2024. arXiv:2411.17501. [153]Mirac Suzgun, Nathan Scales, Nathanael SchƤrli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 13003ā13051, Toronto, Canada, July 2023. Association for Computational Linguistics. [154]Yuheng Tang, Kaijie Zhu, Bonan Ruan, Chuqi Zhang, Michael Yang, Hongwei Li, Suyue Guo, Tianneng Shi, Zekun Li, Christopher Kruegel, Giovanni Vigna, Dawn Song, William Yang Wang, Lun Wang, Yangruibo Ding, Zhenkai Liang, and Wenbo Guo. Devops-gym: Bench- marking AI agents in software devops cycle. In The Fourteenth International Conference on Learning Representations, 2026. [155] Artificial Analysis Team. Artificial analysis long context reasoning benchmark(lcr), 2025. [156] Nandan Thakur, Nils Reimers, Andreas RĆ¼cklĆ©, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. [157] Minyang Tian, Luyu Gao, Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, HAO TONG, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yufeng Du, Tianhua Tao, Ofir Press, Jamie Callan, Eliu A Huerta, and Hao Peng. Scicode: A research coding benchmark curated by scientists. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. [158] Transluce. Docent. https://transluce.org/docent, 2026. [159] Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld: A control- lable world of apps and people for benchmarking interactive coding agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16022ā16076, Bangkok, Thailand, August 2024. Association for Computational Linguistics. [160] UK AI Security Institute. Inspect: A framework for large language model evaluations, 2024. [161] UniPat AI. UniPat AI. https://unipat.ai/, 2026. [162] Vals AI. Finance-Agent: A tool for financial research and analysis, 2024. [163]Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela. Anchor points: Benchmarking models with much fewer examples, 2024. EACL 2024, arXiv:2309.08638. 22 [164]Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'AlchĆ©- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. [165]Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. [166]Ruoyao Wang, Peter Jansen, Marc-Alexandre CĆ“tĆ©, and Prithviraj Ammanabrolu. Science- world: Is your agent smarter than a 5th grader?, 2022. [167]Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An open platform for AI software developers as generalist agents, 2025. ICLR 2025, arXiv:2407.16741. [168] You Wang, Michael Pradel, and Zhongxin Liu. Are "solved issues" in SWE-bench really solved correctly? an empirical study, 2025. arXiv:2503.15223. [169]Zhun Wang, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang, and Dawn Song. Cyber- gym: Evaluating AI agentsā real-world cybersecurity capabilities at scale. In The Fourteenth International Conference on Learning Representations, 2026. [170]Anjiang Wei, Yuheng Wu, Yingjia Wan, Tarun Suresh, Huanmi Tan, Zhanke Zhou, Sanmi Koyejo, Ke Wang, and Alex Aiken. Satbench: Benchmarking llmsā logical reasoning via automated puzzle generation from sat formulas, 2025. [171]Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models, 2024. [172]Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. Livebench: A challenging, contamination-limited LLM benchmark. In The Thirteenth International Conference on Learning Representations, 2025. [173] Ryan Wong, Jiawei Wang, Junjie zhao, Li Chen, Yan Gao, Zhanglong, Xuan Zhou, Zuo Wang, Kai Xiang, Ge Zhang, Wenhao Huang, Yang Wang, and WANG KE. Widesearch: Benchmarking agentic broad info-seeking. In The Fourteenth International Conference on Learning Representations, 2026. [174]Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Agentgym: Evolving large language model-based agents across diverse environments, 2024. [175]Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying LLM-based software engineering agents, 2024. arXiv:2407.01489. [176] Chunqiu Steven Xia, Yinlin Deng, and LINGMING ZHANG. Top leaderboard ranking = top coding proficiency, always? evoeval: Evolving coding benchmarks via LLM. In First Conference on Language Modeling, 2024. [177] Xiaomi. MiMo platform pricing.https://platform.xiaomimimo.com/docs/pricing, 2026. Accessed: 2026-04-22; covers both MiMo-V2-Pro and MiMo-V2.5-Pro. 23 [178]Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. PIXIU: A comprehensive benchmark, instruction dataset and large language model for finance. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. [179]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. [180]Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Zhiruo Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Melroy Maben, Raj Mehta, Wayne Chi, Lawrence Keunho Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. Theagentcompany: Benchmarking LLM agents on consequential real world tasks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. [181] Jie Yang, Honglin Guo, Li Ji, Jiazheng Zhou, Rui Zheng, Zhikai Lei, Shuo Zhang, Zhiheng Xi, Shichun Liu, Yuxin Wang, Bo Wang, Yining Zheng, Tao Gui, and Xipeng Qiu. Abc-bench: Benchmarking agentic backend coding in real-world development, 2026. [182]John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated soft- ware engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [183]John Yang, Kilian Lieret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. SWE-smith: Scaling data for software engineering agents. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. [184]John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, Diyi Yang, and Ofir Press. Program- bench: Can language models rebuild programs from scratch?, 2026. [185] Rui Yang, Ziruo Wang, Yuntian Gu, Yitao Liang, and Tongyang Li. QCircuitbench: A large-scale dataset for benchmarking quantum algorithm design. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. [186]Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 20744ā20757. Curran Associates, Inc., 2022. [187]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan.Ļ-bench: A benchmark for Tool-Agent-User interaction in real-world domains. In The Thirteenth International Conference on Learning Representations, 2025. [188] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models, 2023. ICLR 2023, arXiv:2210.03629. [189]Gregory Yauney, Shahzaib Saqib Warraich, and Swabha Swayamdipta. How reliable is language model Micro-Benchmarking?, 2026. ICLR 2026, arXiv:2510.08730. [190]Christine Ye, Sihan Yuan, Suchetha Cooray, Steven Dillmann, Ian L. V. Roque, Dalya Baron, Philipp Frank, Sergio Martin-Alvarez, Nolan Koblischke, Frank J Qu, Diyi Yang, Risa Wech- sler, and Ioana Ciuc Ģ a. Replicationbench: Can AI agents replicate astrophysics research papers?, 2026. [191] Yixin Ye, Yang Xiao, Tiantian Mi, and Pengfei Liu. Aime-preview: A rigorous and immediate evaluation framework for advanced mathematical reasoning.https://github.com/GAI R-NLP/AIME-Preview, 2025. GitHub repository. 24 [192]Boxi Yu, Yuxuan Zhu, Pinjia He, and Daniel Kang. UTBoost: Rigorous evaluation of coding agents on SWE-Bench, 2025. ACL 2025, arXiv:2506.09289. [193]Xingdi Yuan, Morgane M Moss, Charbel El Feghali, Chinmay Singh, Darya Moldavskaya, Drew MacPhee, Lucas Caccia, Matheus Pereira, Minseon Kim, Alessandro Sordoni, and Marc-Alexandre CĆ“tĆ©. debug-gym: A Text-Based environment for interactive debugging, 2025. arXiv:2503.21557. [194]Z.AI. Z.AI API pricing overview.https://docs.z.ai/guides/overview/pricing, 2026. Accessed: 2026-04-22. [195]Amir Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning, 2018. [196]Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Aoyan Li, Lu Chen, Xiaojian Zhong, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, RUI LONG, Ming Ding, and liang xiang. Multi-SWE-bench: A multilingual benchmark for issue resolving. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. [197]Yuchen Zeng and Dimitris Papailiopoulos.You donāt need to run every eval, 2026. arXiv:2606.24020. Code: https://github.com/anadim/llm-benchmark-matrix. [198]Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark, 2020. [199]Guanhua Zhang, Florian E. Dorner, and Moritz Hardt. How benchmark prediction from fewer data misses the mark, 2025. arXiv:2506.07673. [200]Zicheng Zhang, Xiangyu Zhao, Xinyu Fang, Chunyi Li, Xiaohong Liu, Xiongkuo Min, Haodong Duan, Kai Chen, and Guangtao Zhai. Redundancy principles for mllms benchmarks, 2025. [201] Yujie Zhao, Boqin Yuan, Junbo Huang, Haocheng Yuan, Zhongming Yu, Haozhou Xu, Lanx- iang Hu, Abhilash Shankarampeta, Zimeng Huang, Wentao Ni, Yuandong Tian, and Jishen Zhao. Ama-bench: Evaluating long-horizon memory for agentic applications. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026. [202]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. [203]Xinyue Zheng, Haowei Lin, Shaofei Cai, Yaodong Yang, Zilong Zheng, and Yitao Liang. Unicode: Augmenting evaluation for code reasoning. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026. [204] Changzhi Zhou, Ao Liu, Yuchi Deng, Zhiying Zeng, Tao Zhang, Haotian Zhu, Jianwei Cai, Yue Mao, Chenchen Zhang, Lingyun Tan, ZiyanXU, Bohui Zhai, HengyiLIu, Speed Zhu, Wiggin Zhou, and Fengzong Lian. Autocodebench: Large language models are automatic code bench- mark generators. In The Fourteenth International Conference on Learning Representations, 2026. [205]Hongli Zhou, Hui Huang, Ziqing Zhao, Lvyuan Han, Huicheng Wang, Kehai Chen, Muyun Yang, Wei Bao, Jian Dong, Bing Xu, Conghui Zhu, Hailong Cao, and Tiejun Zhao. Lost in benchmarks? rethinking large language model benchmarking with item response theory, 2026. AAAI 2026, arXiv:2505.15055. [206] Qixing Zhou, Jiacheng Zhang, Haiyang Wang, Rui Hao, Jiahe Wang, Minghao Han, Yuxue Yang, Shuzhe Wu, Feiyang Pan, Lue Fan, et al. Featurebench: Benchmarking agentic coding for complex feature development. arXiv preprint arXiv:2602.10975, 2026. 25 [207]Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024. [208]Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Jasjeet Sekhon, Jacob Steinhardt, Antony Kellermann, Sarah Schwettmann, Matei Zaharia, Ion Stoica, Percy Liang, and Daniel Kang. Establishing best practices for building rigorous agentic benchmarks, 2025. arXiv:2507.02825. [209]Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and JĆ¼rgen Schmidhuber. Agent-as-a-Judge: Evaluate agents with agents, 2024. arXiv:2410.10934. [210]Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, David Lo, Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations, 2025. 26 Appendix Table of Contents A Authors29 A.1 Main Contributors & Affiliations . . . . . . . . . . . . . . . . . . . . . . . . . . .29 A.2 Additional Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30 B Acknowledgment30 C Extended Related Work30 D Details for Harbor Adapters32 D.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33 D.2 Agent Interaction Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33 D.3 Adapter Construction Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . .33 D.3.1 Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33 D.3.2 Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34 D.3.3 Standardization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34 D.4 Parity Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34 D.4.1 Challenges in Benchmark Portability . . . . . . . . . . . . . . . . . . . .34 D.4.2 Design and Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . .34 D.4.3 Agent Integration Scenarios . . . . . . . . . . . . . . . . . . . . . . . . .34 D.4.4 Parity Set Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35 D.5 Adapter Catalog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35 D.6 Per-Benchmark Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35 D.6.1 Software Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36 D.6.2 Mathematics & Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . .38 D.6.3 Knowledge & Long Context . . . . . . . . . . . . . . . . . . . . . . . . .39 D.6.4 Scientific Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40 D.6.5 Agents, Tools & Systems . . . . . . . . . . . . . . . . . . . . . . . . . . .41 D.6.6 Data & Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42 D.6.7 Professional Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . .42 D.6.8 Safety & Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43 D.6.9 Multimodal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43 D.6.10 Benchmarks Using Harbor Format . . . . . . . . . . . . . . . . . . . . . .43 D.6.11 Other Benchmarks Not Included . . . . . . . . . . . . . . . . . . . . . . .43 D.7 Benchmark Issues and Lessons Learned . . . . . . . . . . . . . . . . . . . . . . .44 E Evaluation Protocol and Reproducibility Details46 E.1 Evaluation Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46 E.2 Dataset Preparation and Sampling . . . . . . . . . . . . . . . . . . . . . . . . . .48 E.3 Error Handling and Quality Control . . . . . . . . . . . . . . . . . . . . . . . . .48 27 E.4 Harbor-Index Experiment Protocol . . . . . . . . . . . . . . . . . . . . . . . . . .49 E.5 Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .49 F Detailed Large-scale Quantitative Analysis49 F.1Benchmark and Task Predictability Analysis . . . . . . . . . . . . . . . . . . . . .50 F.1.1Low-rank structure for benchmark space . . . . . . . . . . . . . . . . . . .50 F.1.2Benchmark predictability. . . . . . . . . . . . . . . . . . . . . . . . . . .50 F.1.3How many benchmarks span the evaluation space? . . . . . . . . . . . . .52 F.1.4Does the same redundancy hold at the task level? . . . . . . . . . . . . . .52 F.2Model vs. Harness Effect: Methodology . . . . . . . . . . . . . . . . . . . . . . .54 F.3Original benchmark data collections and mapping . . . . . . . . . . . . . . . . . .55 F.4Performance Progress Over Time . . . . . . . . . . . . . . . . . . . . . . . . . . .56 F.5Latency Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .58 G Case Analysis59 G.1 Task Failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .59 G.2 Agent Failure Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60 G.2.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60 G.2.2 Agent Failure Mode Full Results . . . . . . . . . . . . . . . . . . . . . . .70 H Harbor-Index Selection Pipeline71 H.1 Sampling Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .71 H.2 Task Quality Bar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .71 H.3 Task Audit Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72 H.4 Human Review Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .74 H.5 Audit-and-Fix Loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .75 ILimitations75 J Broader Impacts75 28 A Authors Each contributor below is listed with their full affiliation (superscript), keyed to the numbered institution list at the end of this section. A.1 Main Contributors & Affiliations Organization & Execution Team Lin Shi 1 , Haowei Lin 2 , Zixuan Zhu 3 , Xiaoyue Zhou 1 , Xiang Li 4 , Xiangning Lin 5 , Yaxuan Deng 1 , Han Xu 6 , Yuangang Li 7 , Shanda Li 5 , Zizhao Chen 1 , Hanwen Xing 8 , Harsh Raj 9 , Bo Chen 10 , Quan Shi 11 , Steven Dillmann 4 , Yipeng Gao 8 , Puneesh Khanna 12 , Ruofan Lu 13 , Chao Beyond Zhou 14 , Michael Yang 15 , Robert Zhang 16 , Siyuan Chai 6 , Jiayu Chang 4 , Yizhao Chen 17 , Xiaokun Chen 4 , Yiwei Dai 18 , Wenting Yang 14 , Hange Liu 14 , Minghao Liu 19 , Zihan Wang 19 Advisory Committee Andy Konwinski 21 , Boxuan Li 22 , Leon Liangyu Chen 4 , Alex Dimakis 23 , Nicholas Carlini 20 , Soroush Vosoughi 24 , Di He 2 , Etash Guha 4 , Benjamin Feuer 4 , Mike Merrill 20 , Ludwig Schmidt 4 , Alex Shaw 21 Adapter Contributors Adnan El Assadi 25 , Benedikt Stroebl 26 , E. Kelly Buchanan 4 , Han Meng 27 , Junwei He 28 , Longx- uan Yu 29 , Radin Shayanfar 30 , Yukyung Lee 31 , Zhikang Dong 32 , Allen G Hart 33 , Anjiang Wei 4 , Anurag Kashyap 34 , Arpandeep Khatua 4 , Audrey Jixin Zheng 35 , Chengrui Ma 24 , David Heineman 36 , Dubing Chen 37 , Hai-Anh Trinh 14 , Haishuo Fang 38 , Hefan Zhang 24 , Hui Shen 39 , Issa Sugiura 40 , Jiankai Sun 4 , Jiechao Gao 4 , Junhong Lin 41 , Junnan Li 42 , Kai Yang 39 , Lei Hsiung 24 , Maoyu Wang 14 , Mengze Tang 42 , Nabil Omi 43 , Negin Raoof 23 , Nicholas Edwards 44 , Octavia Guo 14 , Orfeas Menis Mastromichalakis 45 , Pengliang Ji 5 , PrzemysÅaw Hejman 46 , Qi Qi 47 , Qunshu Lin 19 , Richard Zhuang 4 , Rui Yang 2 , Ruichen Zheng 24 , Ryan Marten 48 , Shaghayegh Fazliani 4 , Shizheng Hou 49 , Sicong Jiang 19 , Sijie Li 2 , Song Bian 42 , Terry Yue Zhuo 50 , Tianqing Wu 14 , Tom Tang 19 , Wanjia Zhao 4 , Weihao Xuan 51,19 , Wenhua Liang 24 , Xian Liu 14 , Xin Lan 52 , Xuan Zhang 19 , Xuandong Zhao 23 , Yanchuan Tang 25 , Yifan Jiang 8 , Yijiang Li 17 , Yitong Guan 19,53 , Yizhi Li 54 , Yonghui Liu 55 , Yuheng Tang 15 , Yujun (Audrey) Mao 31 , Yunfei Zhao 19,4 , Yuxin Wang 24 , Yuxuan Tang 14 , Zhenheng Tang 56 , Zhifei Li 26,57 , Ziruo Wang 2 , Ziyu She 58 , Kaiyuan Liu 43 , Iheb Chaabane 12 , Yuxin Tang 59 , Xiangyi Li 60 Affiliations 1. Cornell Tech 2. Peking University 3. Nanyang Technological University 4. Stanford University 5. Carnegie Mellon University 6. University of Illinois Urbana-Champaign 7. University of California, Irvine 8. University of Southern California 9. Northeastern University 10. University of Hong Kong 11. International School of Krakow 12. Technology Innovation Institute 13. The Chinese University of Hong Kong 14. Independent Contributor 15. University of California, Santa Barbara 16. University of Texas at Austin 17. University of California, San Diego 18. Cornell University 19. 2077AI 20. Anthropic 21. Laude Institute 22. Microsoft 23. University of California, Berkeley 24. Dartmouth College 25. Harvard University 26. Princeton University 27. College of William and Mary 28. ByteDance 29. University of California, Riverside 30. Queenās University 31. Boston University 32. Stony Brook University 33. University of Warwick 34. Amazon 35. CoreWeave, Inc. 36. Allen Institute for AI 37. University of Macau 38. TU Darmstadt 39. University of Michigan 40. Institute of Science Tokyo 41. Massachusetts Institute of Technology 42. University of Wisconsin-Madison 43. University of Washington 44. University of Vienna 29 45. Instituto de TelecomunicaƧƵes, Lisbon 46. Quesma 47. Meta 48. HarborCo 49. National University of Singapore 50. Monash University 51. The University of Tokyo 52. Michigan State University 53. Zhejiang University 54. University of Manchester 55. Australian National University 56. The Hong Kong University of Science and Technology 57. Renmin University of China 58. University of Basel 59. Rice University 60. BenchFlow A.2 Additional Contributors We also thank the following contributors for help with adapter development, review, and evalua- tion: ā¢ Zihao Wang (Peking University) ā¢ Joan Cabezas (Independent Contributor) ā¢ Thomas Aubry (ooakdata) ā¢ Bowen Xing (Independent Contributor) ā¢ Qiuyang Mang (University of California, Berkeley) ā¢ Shuting Zhao (University of California, San Diego) ā¢ Xinyu Yao (Rice University) ā¢ Satya Sai Srinath Namburi GNVV (University of WisconsināMadison) ā¢Achintya Paningapalli (University of California, Berke- ley) ā¢ Xinyue Zheng (Peking University) ā¢ Boqin Yuan (University of California, San Diego) ā¢ Michael Glass (IBM) ā¢ Benjamin Calvert (Brigham Young University) ā¢ Xinyu Lu (Institute of Software, Chinese Academy of Sciences) ā¢ Zhaowei Xu (Independent Contributor) B Acknowledgment The experiments and analyses of this study is generously supported by a series of partnership companies, frontier AI labs, and cloud sandbox providers: Partnership Companies: Laude Institute [77], 2077AI [1], Docent [158], UniPat AI [161] Frontier AI Labs: OpenAI (GPT) [117], Anthropic (Claude) [6], Google DeepMind (Gem- ini) [49], Alibaba (Qwen) [4], DeepSeek (Deepseek) [31], MoonShot.AI (Kimi) [110], Xiaomi (Mimo) [177], and Z.AI (GLM) [194] Sandbox Providers: Daytona [28] (for all CPU-only tasks), Modal [108] (for GPU-required tasks specifically) C Extended Related Work This appendix extends Section 5. It adds two threads the main text omits for space, namely the aggregate-evaluation tradition that precedes agentic benchmarks and safety and robustness evalu- ation, and expands the main sectionās clause-level citation groups into fuller discussions of prior work. Aggregate and holistic evaluation. GLUE and SuperGLUE established standardized multi- task evaluation in language [165,164]. Later suites broadened scope to knowledge and rea- soning [55,149,153] and multilinguality [57], and advanced methodology through multi- metric scenario reporting [85], preference-based comparison [22], and contamination-aware testing [172]. Analogous efforts span retrieval and representation learning [156,112] and vi- sion [138,198,72,8,25,52,195], while Dynabench and Dynaboard replace fixed test sets with evolving, hosted comparisons [69, 99]. Evaluation infrastructure and reproducibility. Reproducibility work documents how prompt templates, decoding choices, few-shot formatting, and metric definitions materially change re- ported comparisons [11]; HELM adds multi-metric, scenario-level transparency [85] and Dyn- aboard evaluates submitted models in a hosted setting to reduce reliance on self-reported re- sults [99]. Recent systems extend this to model-plus-scaffold evaluation: Inspect [160], HAL [65], AstaBench [12], and Harbor [53], a containerized task and sandbox runtime. A parallel effort pack- ages task or agent families behind one programmatic interface, including agentic environment sim- ulators [93], interactive debugging environments [193], generalist software-agent platforms [167], 30 and reproducible search sandboxes that replace commercial APIs whose drift silently breaks com- parability [24]. Agent benchmarks make reproducibility harder. A static task reduces to examples, prompts, and labels; an agent benchmark also carries an environment (browser state, files, virtual services, shell tools), an execution model (sandboxing, network controls, stochastic dynamics), and a scoring layer (hidden tests, trajectory judgments), each of which shifts correctness and cost between runs. Reporting practice compounds this: evaluations often conflate the needs of model developers with those of downstream developers, omit cost, and lack adequate holdout sets, which invites benchmark-specific shortcuts [66]. Tool and API use. Tool-use benchmarks test whether models select APIs, generate valid ar- guments, follow protocols, and compose calls. Each targets a different layer: API retrieval and selection [122,82], parameter prediction and function-call accuracy [121], multi-step invoca- tion [131,141], policy following [187], and workflow planning [130,48]. Each therefore fixes one abstraction boundary, whether a function call, a tool trajectory, or a simulated service policy, and evaluates models against it. Interactive and embodied environments. Early work placed agents in web interfaces, where sparse reward made exploration the bottleneck [90], and in text games [26], embodied house- holds [146], simulated science labs [166], and online shopping [186]. AgentBench, GAIA, and AgentGym then evaluated multi-step reasoning, planning, and tool use across broader collec- tions [91,104,174]. Web benchmarks cover live synthetic sites [207,71], real-world naviga- tion [34,54], enterprise workflows [36], and shared execution frameworks [29,97]. Beyond the browser, AppWorld, AndroidWorld, and OSWorld reach application and operating-system con- trol [159, 134, 179], now a large enough class to be surveyed separately [58]. These works show that agent performance depends on environment grounding, state tracking, error recovery, and long-horizon planning. They also explain why agent benchmarks resist direct com- parison: environments differ in action granularity, observation modality, allowed tools, termination criteria, and scoring. Code, scientific, and workplace agents. HumanEval tests program synthesis in self-contained functions [20], while SWE-bench raises the bar to issue resolution in real repositories [64]. SWE- agent shows the agentācomputer interface itself substantially affects software-engineering per- formance [182], building on ReActās interleaved reasoning-and-acting pattern [188] and the self- reflection variants that followed [144]; Agentless conversely shows a much simpler pipeline can be competitive, so scaffold complexity is not monotonically useful [175]. Terminal-Bench extends this line to shells, files, and long-running processes rather than a structured patch interface [103], and time-horizon studies quantify how the task length agents can complete has grown [74]. Be- yond software engineering, MLE-bench evaluates agents on 75 Kaggle machine-learning engi- neering competitions against human leaderboard baselines [16]. ScienceAgentBench deliberately decomposes data-driven discovery into 102 expert-validated tasks from peer-reviewed publica- tions, scoring the generated program, its execution results, and its cost [21]. TheAgentCompany combines browsing, coding, communication, and service manipulation inside a simulated organiza- tion [180]. These benchmarks expose failure modes invisible in static QA. The modes fall into three classes: environment (flaky dependencies, setup failures, resource limits, timeouts), scoring (hidden-test mismatch, partial credit, trajectory judgments), and scaffold (tool affordances, retry policies, inter- face conventions). Each class shifts reported scores independently of the model under test. Safety, robustness, and adversarial evaluation. Tool-using agents take actions, read untrusted content, and execute multi-step plans whose intermediate states are themselves attack surfaces. AgentDojo evaluates prompt-injection attacks and defenses over realistic tasks and untrusted data [30], AgentHarm measures whether agents refuse malicious requests while retaining capa- bility [5], and OS-Harm extends this to computer-use agents acting on a graphical interface [73]. A parallel line targets the reliability of agent evaluation itself: StableToolBench stabilizes tool calls through virtualized APIs and cached responses [51], while AgentRewardBench and Agent-as-a- Judge ask whether model judges can assess behavior beyond final-answer correctness [95, 209]. 31 Benchmark validity and task quality. A distinct literature asks not how to run benchmarks but whether their scores license the conclusions drawn from them. Position and methodology papers make three arguments: that a few influential benchmarks should not stand in for general capability [133]; that benchmark construction should state its measurement assumptions explicitly and testably [92]; and that moving from a score to a claim about an abstract task requires further assumptions made explicit [44]. Empirically, a review of 445 benchmarks by 29 expert review- ers finds recurring patterns in measured phenomena, task design, and scoring metrics that under- mine validity [9], and an assessment framework applied across widely used benchmarks reaches similar conclusions about documentation and design practice [136]. Concrete threats are well documented. Test-set label errors are pervasive, at least3.3%on average across ten widely used datasets, and destabilize conclusions to the point that lower-capacity models can overtake higher- capacity ones once mislabeling is accounted for [116]; annotation errors change how aggregates should be read [47,115]; contamination compromises held-out validity and is hard to measure per benchmark [139]; and leaderboard mechanics distort rankings [147]. For agents the verifier is executable code rather than a stored label, which turns these validity ques- tions into engineering defects. Because resampling cannot lower a verifierās false-positive rate, that rate upper-bounds the accuracy any amount of resampling can attain, regardless of compute bud- get [152]. The Agentic Benchmark Checklist documents systematic task-setup and reward-design flaws in widely used agentic benchmarks, such as insufficient test cases in SWE-bench Verified and empty responses scored as success inĻ-bench, with distortions reaching100%in relative terms [208]. Software-engineering audits make the magnitude concrete: roughly a third of success- ful SWE-bench patches trace to solution leakage in the issue text, and a comparable share to test suites too weak to detect an incorrect patch [3]. Differential patch testing shows plausible patches diverging behaviorally from developer ground truth, inflating reported resolution rates [168]; aug- menting test suites relabels hundreds of patches and reorders leaderboards [192]; and memoriza- tion rather than reasoning on SWE-bench instances complicates held-out interpretation [86]. The judging layer is a matching concern, whether the judge is a preference model [202], a trajectory judge [95,209], or a stand-in for human annotators, a substitution for which no standard procedure existed until the recently proposed alternative annotator test [14]. Meta-evaluation, efficiency, and benchmark compression. Much of a large suite is redundant. Factor-analytic work finds model-by-benchmark score matrices are intrinsically low-rank, a few interpretable latent dimensions explaining most variance and implying substantial redundancy among nominally distinct tasks [13,101,197]; the same structure makes downstream performance predictable from a few latent capability measures [137], and correlation-based analyses of multi- modal suites report comparable redundancy across benchmarks and capability dimensions [200]. This licenses cheaper evaluation through curated and distilled subsets [127,70,163], psychometri- cally adaptive item selection [56,205], principled metric selection [129], and label-efficient active testing [10]. Cost-aware framings make the tradeoff explicit rather than incidental [41, 66]. Recent results delimit these methods. Benchmark agreement testing has no standardized proce- dure, and overlooked methodological choices, including which models the agreement is computed over, significantly change its conclusions, so apparent redundancy is partly an artifact of how it is measured [124]. Benchmark prediction from a subset depends on similarity to previously observed models and degrades sharply when extrapolating to stronger ones, failing exactly at the frontier where evaluation is most needed [199]. Micro-benchmarks frequently cannot rank nearby models reliably, requiring far more items than the handful sometimes suggested, at which point random sampling is competitive with sophisticated selection [189]. Small evaluations therefore require explicit uncertainty quantification [105], on top of the setup sensitivity documented above. D Details for Harbor Adapters This appendix describes the motivation for Harbor Adapters (Ā§D.1), the agent interaction modes we use to categorize benchmarks (Ā§D.2), the adapter construction workflow (Ā§D.3), the parity experiment protocol (Ā§D.4), the full adapter catalog (Ā§D.5), per-benchmark adaptation and parity details (Ā§D.6), and lessons learned from building adapters at scale (Ā§D.7). 32 D.1 Motivation As the number of benchmark datasets and agents grows, agent evaluation has become increas- ingly costly: supportingmbenchmarks withnagents naively requiresO(mn)effort, since each benchmark-agent pair involves its own harness integration, environment plumbing, and evalua- tion logic. Inconsistent environments and task-design issues further compound this cost, and the resulting leaderboards are often not directly comparable because instructions, tools, and execution configurations differ across benchmarks. Despite the diversity of task formats, most benchmarks share a common underlying abstraction: instruction, environment, tests, and solution. The instruction specifies what the agent must ac- complish; the environment specifies where the agent acts; the tests verify success; and the solution demonstrates feasibility. This abstraction, introduced by Terminal-Bench [103], applies across the vast majority of agentic and non-agentic benchmarks. Harbor Adapters translate heterogeneous dataset-specific formats into this shared schema. To- gether with the Harbor infrastructure, which provides sandboxed execution (local and cloud) and reward-based evaluation, the adapter layer reduces the per-benchmark or per-agent integration cost fromO(mn)toO(m + n), and enables running tens of datasets, tens of thousands of tasks, and millions of rollouts across many modelāharness configurations under identical evaluation conditions. D.2 Agent Interaction Modes We categorize each benchmark by its agent interaction mode, which captures whether and how agentic capabilities affect the evaluation outcome. Agentic benchmarks are explicitly designed for agent-based evaluation; tasks assume iterative interaction with an environment (e.g., filesystem, shell, browser, external tools), multi-step reason- ing, and/or test-driven feedback, and evaluating them without agent capabilities is infeasible or inconsistent with the original design. Non-Agentic benchmarks are designed for direct LLM evaluation; tasks are single-pass, with the model producing an answer from the prompt alone and no access to tools or an environment, so introducing agent capabilities would not meaningfully change what the benchmark measures. Modified-Agentic benchmarks are originally non-agentic, but exposing the underlying environ- ment to an agent (e.g., filesystem access, iterative execution, test-driven refinement) changes the evaluation outcome in a measurable way; these are typically coding tasks whose single-pass proto- col underspecifies what a capable agent could achieve with terminal access. Agent interaction mode is a property of the benchmark itself. It is used throughout the catalog (Ā§D.5) and determines which integration scenario applies when we construct an adapter (Ā§D.4). D.3 Adapter Construction Workflow Constructing and maintaining adapters at scale is an engineering-intensive process that requires careful handling of dataset formats, execution environments, and evaluation protocols. We orga- nize this process into three stages: creation, review, and standardization. D.3.1 Creation Each adapter is constructed through the following steps. (1) Parsing and restructuring benchmark content into the Harbor schema (instruction, environment, tests, solution). (2) Reconstructing or aligning the execution environment (Docker images, dependencies, network policies, resource limits). (3) Integrating evaluation logic and test cases, and verifying that oracle solutions achieve a 100% pass rate. (4) Running parity experiments against the original benchmark (Ā§D.4). To scale this process, we developed contributor guidelines, a prompt-driven tutorial for coding agents, and an agent skill (/create-adapter) that handles context sharing and enables parallel adapter development across the team. 33 D.3.2 Review Adapter review is equally intensive, as it must ensure semantic fidelity to the original benchmark, correctness of the execution environment, and conformance to the Harbor schema. We trained a dedicated team of five reviewers and developed automated review bots to accelerate quality assurance. Every adapter pull request undergoes a three-stage review process. (1) An automated bot review that checks code style, schema conformance, documentation completeness, and the presence of a passing oracle run. (2) A human review by one to three trained reviewers, who verify adaptation correctness, environment reproducibility, and parity-experiment results. (3) A final approval by the team lead, who signs off on non-trivial design decisions (e.g., parity-set selection, custom-agent ports) before the adapter is merged. Adapters that fail any stage are returned to the contributor with reviewer comments and re-enter the pipeline once addressed. D.3.3 Standardization As more adapters are integrated, requirements and best practices continue to evolve: the Harbor schema gains new fields, shared utilities are refactored, and documentation conventions tighten. To prevent format drift, a standardization team performs periodic passes that co-evolve three artifacts in parallel: existing adapters, the contributor tutorial, and the bot review prompts. In this way, the schema, the guidance new contributors receive, and the checks enforced at review time all stay aligned, while preserving dataset integrity and parity results. This maintenance work is scheduled independently of new-adapter contributions and is a first-class part of the workflow rather than one-off cleanup. D.4 Parity Experiments D.4.1 Challenges in Benchmark Portability Porting a benchmark to a new harness is semantics-preserving in intent, but rarely so in practice. Small differences in prompt templates, tool configurations, default decoding parameters, Docker base images, or scoring scripts can shift reported scores by several points, even when the under- lying task set is unchanged. Published results are often not reproducible from the released code alone: we have routinely encountered oracle solutions that fail the benchmarkās own tests, scoring scripts with undocumented non-determinism, and leaderboard entries produced by pipelines that diverge from the public release. Without explicit verification, a user of a Harbor adapter cannot tell whether a score difference reflects a genuine capability gap, a harness artifact, or a latent bug on either side. We therefore treat parity verification as an essential step in building every adapter. D.4.2 Design and Evaluation Metrics A parity experiment compares the Harbor-adapted version of a benchmark against the original under matched conditions. The same agent, model, tool set, prompt template, decoding parameters, and execution configuration are applied on both sides. Each side is runktimes (k = 3by default) to account for stochasticity, and we report the mean ( Ģx)Ā±sample standard error of the mean (s Ģx ), computed ass Ģx = s/ ā kwheresis the sample standard deviation across thekruns. An adapter passes parity when the original and Harbor means agree within a margin consistent with their standard errors. Adapters that fail the parity are debugged and re-run. Cases where parity cannot be achieved due to irreducible differences, for example dependence on a non-deterministic external service, are documented per benchmark in Ā§D.6. D.4.3 Agent Integration Scenarios A parity experiment requires the same agent to run against both the original benchmark and the Harbor adapter. Whether such an agent already exists or has to be built depends on the original benchmarkās design. During our development, we distinguish three main scenarios: Scenario 1: Harbor Supported Agent. The original benchmark already uses an agent that Harbor also supports natively (e.g., Claude Code, Codex CLI). Adaptation reduces to aligning versions, tool configurations, execution arguments, and implicit environment assumptions between the two sides. The parity experiment runs the same agent against both. 34 Scenario 2: Vanilla LLM. The original benchmark provides no agent and was designed for direct LLM prompting. We fork the original repository and add a Harbor-supported CLI agent, matching the prompt and decoding protocol of the original single-pass evaluation. Minor modifications, such as input/output formatting and prompt tweaks, are permitted as long as they are applied symmet- rically to both the original and Harbor sides. The parity experiment then runs this agent against both the forked original and the Harbor adapter. We apply the same treatment to non-agentic and modified-agentic benchmarks alike. Scenario 3: Custom Agent. The original benchmark ships a custom agent that Harbor does not support natively (e.g., a benchmark-specific ReAct variant, a deep-research agent, or a multi-agent scaffold). We port the custom agent into Harbor, and the parity experiment runs it against the original benchmark. Where feasible, we also run a Harbor-native agent (e.g., Claude Code) as a cross-check. When the custom agent cannot be cleanly ported due to tight coupling with bench- mark internals, we allow the benchmark design to diverge, and ship both the original-agent variant and a CLI-agent variant in Harbor for community use, and document the limitation in Ā§D.6. The mapping from agent interaction mode to integration scenario is direct. All agentic benchmarks fall into Scenario 1 or Scenario 3 depending on whether their native agent is supported by Harbor. All non-agentic and modified-agentic benchmarks fall into Scenario 2. D.4.4 Parity Set Selection For most benchmarks, parity is run on the full task set. When the full set is too costly to runk times (e.g., long-horizon agentic tasks or large task counts), we run parity on a representative subset, the parity set, selected by one of the following criteria: (1) Random stratified sampling over difficulty and/or subdomain, when such metadata is avail- able. (2) Official subsets, when the benchmark authors provide a smaller canonical subset (e.g., a āLiteā or āVerifiedā split). (3) Curated subsets, when stability or environment-setup cost requires hand-picked representative instances. For each adapter, the parity-set size, selection criterion, and the specific task IDs are recorded and explained in the adapterās README atadapters/<benchmark>/README.mdin the Harbor Adapters repository, and summarized in the corresponding paragraph of Ā§D.6. D.5 Adapter Catalog Tables 3 and 4 list every adapter whose parity experiment is complete, with its domain/subdomain, integration scenario, agent interaction mode, parity agent, parity model, number of runs, parity-set size, and parity scores (original vs. Harbor). The parity-set column is formatted asparity/full, whereparityis the number of tasks actually used for parity verification andfullis the total size of the benchmark. Score values are reported as meanĀ±SEM overkruns (SEM omitted whenk = 1). N/Aindicates that no direct Harbor-vs-original (or Harbor-vs-Terminal-Bench) parity score is avail- able ā either because the upstream benchmark has no agent harness so traditional parity does not apply, or because only an upstream Terminal-Bench adapter versus original-benchmark compar- ison was measured. Because parity verification is part of adapter construction rather than of the evaluation itself, these tables span more benchmarks than the 54 used in our large-scale evaluation: nine rows (CooperBench, FeatBench, WebGen-Bench, Frontier-CS-Algorithm, DevOpsGym, Sci- enceAgentBench, MLGym-Bench, ACE-Bench, CRMArena) have verified adapters but were held out of the experiments for compute reasons, and are therefore also listed in Ā§D.6.11; benchmarks already authored in the Harbor schema (Terminal-Bench, CompileBench, SkillsBench) require no adapter and are described in Ā§D.6.10. D.6 Per-Benchmark Details The subsections from Section D.6.1 through Section D.6.9 describe benchmarks added with Har- bor adapters. Section D.6.10 covers benchmarks that natively use the Harbor format and require 35 no adapter. Section D.6.11 lists benchmarks whose adapters are merged into Harbor (or have an approved adapter pull request) but were not evaluated here due to resource limits. D.6.1 Software Engineering Repo-level Issue Resolution: SWE-Bench Multilingual, SWE-Bench Pro, SWE-Bench Verified, SWE-Smith, SWT-Bench. SWE-Bench-Multilingual [183]. This benchmark extends SWE-bench [64] across nine program- ming languages with 300 tasks. The Harbor adapter targets the per-task multilingual image and invokes the official grading harness, adapting all 300 tasks and scoring them with unit tests. The original oracle solutions pass 100%, with one seasonal failure. Parity uses a fixed subset of 50 tasks stratified across the nine languages. SWE-Bench Pro [33]. This benchmark extends SWE-bench [64] with 731 tasks across Python, JavaScript, TypeScript, and Go. The Harbor adapter reuses the upstream execution and verification with two reliability fixes, running Jest in single-worker mode with forced exit and normalizing gold patches to end with a newline. The original oracle solutions pass 97%, with 15 invalid gold patches and 9 tasks that time out. Parity uses a random subset of 100 tasks together with a 10-task rerun after adapter fixes, and both sides are task-aligned. SWE-Bench Verified [64]. This benchmark is a human-curated subset of SWE-bench [64] with 500 Python tasks scored by unit tests. The original oracle solutions pass 99.2%, with 4 tasks failing on data or infrastructure errors. Parity covers the full 500-task set and is val- idated transitively against an earlier Terminal-Bench integration. We also check the bench- mark score directly against the official SWE-bench leaderboard on a 499-task set, excluding scikit-learn__scikit-learn-14710 due to hardware limits in the Daytona environment. SWE-smith [183]. This benchmark contains 60,000 SWE-bench [64]-style tasks with higher- quality descriptions and richer multi-file patches. Harbor currently supports the Python-profiled tasks, and the adapter extends to other languages easily. Runs are scored by unit tests. A fraction of the oracle patches do not fully pass upstream, a known artifact of synthetic patch generation. Parity uses a 100-task slice reused from an earlier Terminal-Bench integration, so parity chains transitively across two legs. SWT Bench [113]. This benchmark contains 433 tasks that test an agentās test-generation and bug- reproduction ability, and a redundant code-description segment is removed from the prompt during adaptation. Generated tests are scored by whether they fail on the buggy code and pass on the fixed code. The original oracle solutions pass 97.7%, with 9sphinx-doctasks failing on an upstream parsing issue. Parity is restricted to evaluation because upstream provides no inference script, so Harbor diffs are exported to the upstream prediction format and graded by the upstream evaluator. Feature & End-to-end Development: FeatureBench, SWE-Lancer. FeatureBench [206]. This benchmark evaluates feature-level code generation on 200 real-world Python tasks across 24 popular repositories, split into interface-guided (lv1) and from-scratch (lv2) complexity levels. The Harbor adapter reuses the prebuilt DockerHub images, mirrors the lv1 oracle throughsetup_patchreversal and the lv2 oracle throughagent_codere-export, auto- configuresshm_sizeand GPU allocation from task metadata, and prunes flaky PASS_TO_PASS tests. The oracle passes 200/200 on Docker and 198/200 on Modal, where two transient failures occur, namely a mirror timeout on an lv2 dependency install and a SciPy segmentation fault on an lv1 task in Modalās sandbox. Parity uses a curated 30-task lite split (26 lv1 and 4 lv2) on NVIDIA A10G, and the residual disagreements reflect stochastic lv1 variation rather than systematic diver- gence. SWE-Lancer [106]. This benchmark evaluates agents on 463 real freelance software-engineering tasks scraped from Upwork and scoped to the Expensify repository, comprising 198 IC SWE tasks that resolve a GitHub issue and pass Playwright and pytest user-tool tests, and 265 Manager tasks that select the correct proposal among candidates. The Harbor adapter reuses the upstream swelancer_x86images with light tmux and user-tool additions and rewrites the Python-only solver prompt into terminal-style instructions. One deviation is that the user tool runs in the agentās namespace, with tests copied in unzipped, because Harbor has no privileged out-of-namespace runner, and the grading logic is otherwise preserved. The oracle passes 100%, as reported in the 36 adapter PR. Parity covers the full 463-task set, and the original side runs from apreparedness fork that ports a matching Claude Code solver into the upstream nanoeval harness, so both sides exercise the same agent. Competitive & Function-level Coding: Aider Polyglot, BigCodeBench, LiveCodeBench, US- ACO, HumanEvalFix, QuixBugs. Aider Polyglot [2]. This benchmark repackages 225 Exercism exercises across six languages (Java, Python, Go, Rust, C++, and JavaScript) to evaluate iterative code editing rather than greenfield generation. The Harbor adapter mirrors all exercises with per-language Dockerfiles, ports the upstream test harnesses verbatim, and wraps encrypted oracle payloads in shell normalizers. Runs are scored by the per-language unit tests, and the adapter reports no aggregate oracle pass rate. Parity is established transitively in two legs, one between the original and the Terminal-Bench adapter (225 tasks, 2 runs with claude-code) and one between the Terminal-Bench adapter and Harbor (225 tasks, 1 run with claude-haiku). BigCodeBench [210]. This benchmark evaluates complex function-calling and engineering pro- gramming on the 145 hard-split tasks, using two prompt formats, the Complete docstring-driven format and the Instruct natural-language format. The Harbor adapter converts tasks from Hug- gingFace and uses reward-based verification, and 3 tasks are excluded for network or file-structure incompatibilities with the Harbor sandbox. The adapter PR reports no aggregate pass rate in text. Parity uses the full 145-task set with codex over 3 trials per side. LiveCodeBench [62]. This benchmark evaluates competitive programming on 1,055 problems aggregated from LeetCode, Codeforces, and AtCoder, covering May 2023 to March 2024. The Harbor adapter supports both Solution-class and stdin/stdout tests, exposescheck_solution.py for public-test development, and auto-injects standard-library imports, and final scoring runs both public and private tests under 30 s and 180 s timeouts. LiveCodeBench ships no oracle solutions, so validity relies on test-case validation. Parity is established transitively in two legs, one between the original and the Terminal-Bench adapter (100release_v6tasks, 4 trials with terminus-2) and one between the Terminal-Bench adapter and Harbor (100 tasks, 4 trials with claude-haiku). USACO [143]. This benchmark evaluates algorithmic problem-solving on 307 USA Computing Olympiad problems covering graph theory, dynamic programming, and other competitive do- mains, of which Harbor adapts 304 after excluding 3 with broken reference solutions. The Harbor adapter wraps the official batch judge in pytest with CTRF logging and embeds canonical solu- tions, leaving the upstream grading logic unchanged. Parity uses the full 304-task set over 3 runs with claude-haiku. HumanEvalFix [111]. This benchmark evaluates code repair on 164 Python tasks from Hu- manEvalPack, where the agent fixes buggy implementations so that pytest passes. The Harbor adapter rewrites the original prompts into agent-oriented instructions, embeds Dockerized envi- ronments, and ships reference solutions, and the pytest grading logic is unchanged. The oracle passes 164/164 at reward 1.0. Parity uses the full 164-task set for 3 trials with OpenHands [167] on Daytona under gpt-4o-mini and gpt-5-mini. QuixBugs [87]. This benchmark evaluates single-line bug repair on 80 defective programs, 40 in Python and 40 in Java, drawn from the Quixey Challenge. The Harbor adapter clones the upstream repository and generates per-language tasks (Python 3.11, and JDK 11 with Gradle), and runs are scored by pytest correctness tests together with a diff-based single-line check. The oracle passes 100%. Parity is established transitively in two legs, one between the original and the Terminal- Bench adapter (80 tasks, 5 trials with claude-code) and one between the Terminal-Bench adapter and Harbor (80 tasks, 3 trials with codex). Performance Optimization: GSO, AlgoTune. GSO [142]. This benchmark evaluates code performance optimization on 102 Python reposito- ries, each pairing a codebase with a performance test that serves as the specification. The Harbor adapter follows the upstream evaluation flow in isolated Docker environments and reuses the orig- inal grading harness, and all 102 tasks are adapted. The oracle passes 89/102 (87.3%), while the remaining 13 fail because their oracles run inconsistently faster than the reference baseline under 37 timing variance, a known artifact of the speedup metric rather than a harness bug. Parity uses the full 102-task set with OpenHands@1.4.0 and gpt-5.1 (high) on each side. AlgoTune [128]. This benchmark evaluates algorithm optimization on 154 tasks across mathemat- ics, physics, computer science, signal processing, cryptography, and graph algorithms, scored by continuous speedup rather than binary pass or fail. The Harbor adapter implements the interleaved timing protocol with 100 instances and 10 timing repetitions per instance, enforces the upstream limits of 8 CPUs and 16 GB of memory inside Docker, and reports the harmonic-mean speedup. Because AlgoTune has no canonical oracle, we use the baseline solution and verify that its self- speedup is close to 1.0Ć. Parity is established transitively in two legs, one between the original and the Terminal-Bench adapter and one between the Terminal-Bench adapter and Harbor, on the full 154-task set. Language Translation: CRUST-Bench. CRUST-Bench [67]. This benchmark evaluates C to safe-Rust transpilation on 100 real-world C repositories from GitHub, each paired with hand-written safe-Rust interfaces and correctness tests. The Harbor adapter vendors the C sources and headers into per-task Rust 1.83 slim containers and classifies projects by difficulty. Runs are scored bycargo test, and because transpilation is open- ended, no oracle solutions are provided. Parity uses the full 100-task set for 3 trials with codex and gpt-5-nano. D.6.2 Mathematics & Reasoning Competition Mathematics: AIME, IneqMath, Omni-Math. AIME [191]. This benchmark evaluates competition-level mathematical reasoning on 60 problems from AIME 2024 and 2025, each requiring an exact integer answer in [0, 999] with no partial credit. The Harbor adapter sources problems from the GAIR-NLP/AIME-Preview repository and packages each as an isolated task with Python available for symbolic computation, and scoring uses exact match against the integer answer. The oracle solutions pass 100%. Because AIME has no agent-oriented upstream harness, standard parity does not apply, so we instead confirm on the full 60-task set that diverse agents such as Gemini-CLI and codex recover the expected fraction of solutions. IneqMath [94]. This benchmark evaluates inequality-proof reasoning on the 100 expert-curated multiple-choice questions in the public dev set, covering Bound Estimation and Relation Predic- tion, while the test set and step-wise evaluation are not publicly released and are not adapted. The Harbor adapter generates tasks with full upstream metadata and supports submission-format con- version for the leaderboard, and evaluation follows the upstream protocol of exact match with a gpt-4o-mini judge fallback. The oracle passes 100% on the full dev set. Parity uses the full 100- task dev set over 3 trials with codex and gpt-4o-mini, with the same evaluator on both sides. Omni-Math [46]. This benchmark evaluates olympiad-level mathematical reasoning on 4,428 problems spanning more than 33 sub-domains across 10 difficulty levels. The Harbor adapter loads problems from HuggingFace, standardizes the answer at/workspace/answer.txt, and installs the upstream evaluation code inside the container to avoid parsing pitfalls, and a gpt-5-mini judge compares the answer to ground truth. Oracle accuracy is bounded at 99.5% because 18 problems ship without benchmark-provided answers. Parity uses a stratified 100-task subset for 2 trials with codex and gpt-5.3-codex, with an identical judge configuration on both sides. Abstract & Procedural Reasoning: KUMO, Reasoning Gym, ARC-AGI-2. KUMO [88]. This benchmark evaluates interactive multi-step reasoning on 5,300 procedurally generated games (50 instances across 106 domain-scenario pairs), where the agent takes actions, observes outcomes, and identifies hidden truths using a knowledge book. The Harbor adapter runs an interactive loop through a local verifier service (POST /act) and a CLI client, isolates secrets in verifier-only Docker images while exposing onlyknowledge_book.txt, and enforces action budgets through HTTP 429 responses, and scoring uses exact match on/app/answer.txt. The oracle passes 100% on the generated tasks. Parity uses a deterministic 212-task subset (seeds 0 and 1 per scenario) for 3 trials per agent-model pair (kumo-vanilla, terminus-2, and codex, each paired with gpt-5-mini and gpt-5-nano) at max-steps=50 and temperature=1.0 on both sides. 38 Reasoning Gym [151]. This benchmark evaluates verifiable procedural reasoning across diverse problem types, materialized into 576 tasks from 96 task generators (288 easy and 288 hard, with 3 instances each). The Harbor adapter dockerizes each generator and has agents write code to compute answers rather than answering zero-shot, directing the result toanswer.txt. Agentic ex- ecution yields substantially higher reward than the zero-shot setting, and scoring uses the upstream per-task reward verifier. A few generators ship no oracle answer or fail to score their own oracle at reward 1.0, and these are documented and excluded from oracle validation rather than treated as harness bugs. Parity uses the full 576-task set for 3 trials with codex and gpt-5.1-codex-mini, with the same generators and verifier on both sides. ARC-AGI-2 [23]. This benchmark evaluates abstract visual reasoning on the ARC-AGI-2 public evaluation set, where the agent infers a grid transformation from a few demonstrations and applies it to held-out inputs. Harbor materializes 167 task pairs by splitting the 120 upstream puzzles along their test cases. The Harbor adapter pulls the dataset fromArdea/arc_agi_v2, rewrites the puzzle descriptions into agent-oriented instructions, and runs each task in a Python 3.13 container that writes the predicted grid as a JSON 2D array to/testbed/output.json. A pytest verifier requires an exact grid match and embeds reference solutions. Parity uses the full 167-task set for 6 trials with codex@0.53.0 and gpt-5.2 atreasoning_effort=medium, under the upstream Pass@2 protocol. D.6.3 Knowledge & Long Context Expert & Multi-subject QA: GPQA Diamond, Humanityās Last Exam, MMMLU, SimpleQA. GPQA Diamond [135]. This benchmark evaluates graduate-level scientific reasoning on 198 expert-curated multiple-choice questions in biology, physics, and chemistry, where domain ex- perts reach about 65% and non-experts with web access about 34%. The Harbor adapter loads questions from HuggingFace and deterministically shuffles the answer choices, and scoring uses exact letter match on/app/answer.txt. The oracle passes 100% on the full diamond split. Parity uses the full 198-task diamond split for 3 trials with codex and gpt-5.2. Humanityās Last Exam [126]. This benchmark evaluates expert-level multimodal question answer- ing on 2,500 questions across mathematics, science, engineering, humanities, and computer sci- ence. The Harbor adapter reuses the official HLE judge prompt, copies images into the workspace for vision-capable agents, routes responses through/logs/agent/response.txtwith confi- dence scores, and corrects one ground-truth placeholder in the upstream dataset, and a gpt-5 judge performs scoring. Oracle verification passes 100% with 0% calibration error. Parity uses a strati- fied 249-task subset (10% per category, seed 42) with a gpt-5 judge, and calibration error is com- puted post-hoc. MMMLU [55]. This benchmark evaluates multilingual multi-subject knowledge by extending MMLU across 15 languages, for a total of 210,630 tasks (14,042 questions translated into Ara- bic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Chinese, Swahili, and Yoruba). The Harbor adapter casts each question as a standalone task with explicitAnswer: $LETTERformatting and supports multilingual character normalization through the simple-evals regex, and scoring uses exact match on the answer letter. Oracle accuracy is 100%. Parity uses a stratified 150-task subset (10 per language, seed 0) for 5 trials with codex and gpt-5.1- codex-mini. SimpleQA [171]. This benchmark evaluates factual recall on 4,326 fact-seeking questions whose correct answers are short and unambiguous. The Harbor adapter follows OpenAIās simple-evals approach with a configurable grading model (gpt-5-mini by default) and allows internet access during evaluation, and scoring uses an LLM judge. The oracle passes 100% on the full 4,326-task set. Parity uses the first 50 tasks for 3 trials with claude-code and claude-opus-4-6, using gpt-4o- mini as the judge to match the upstream framework. Long Context Reasoning: A-LCR. A-LCR [155]. This benchmark evaluates long-context reasoning over documents of roughly 100k tokens in seven categories (company reports, academia, government, legal, industry, mar- keting, and survey materials), with 100 upstream tasks, of which Harbor adapts 99 after exclud- ing one with a ground-truth error. The Harbor adapter copies documents into the workspace at 39 /workspace/documents/and corrects two ground-truth answers (Task 40 date formatting and Task 94 percentage versus decimal), and grading uses the official equality-checker prompt. The oracle passes 99/99 at mean reward 1.0. Parity uses the full 99-task set across several agent-model pairs, namely codex with gpt-5-mini, claude-code with haiku, terminus-2 with haiku, and terminus- 2 with gpt-5-mini. D.6.4 Scientific Research End-to-end Research Workflows: ReplicationBench, SLDBench. ReplicationBench [190]. This benchmark evaluates research replication on astrophysics papers, with 106 upstream tasks, of which Harbor adapts 90 expert-reviewed numeric tasks that require Python to match numeric outputs within specified tolerances. The Harbor adapter integrates the HuggingFace dataset and runs tasks in the upstream Docker images, and scoring uses tolerance- based numeric checks. No aggregate oracle rate is reported, and 5 tasks time out consistently on both sides and are flagged for possible exclusion. Parity is established transitively in two legs, one between the original and the Terminal-Bench adapter (106 tasks, 2 runs with claude-code and sonnet) and one between the Terminal-Bench adapter and Harbor (90 tasks, 1 run with claude-code and haiku). SLDBench [89]. This benchmark evaluates symbolic scaling-law discovery from more than 5,000 LLM training experiments, packaged as 8 tasks that span parallel, vocabulary, fine-tuning, domain- mixture, mixture-of-experts, data-constrained, hyperparameter, and adversarial (U-shaped) scaling laws. The Harbor adapter ships oracle implementations of the Human and Expert baselines and requires the agent to produce both/app/law.pyand/app/explain.md, and scoring is continu- ous overR 2 , NMSE, and NMAE on held-out extrapolation sets. Because the metric is continuous rather than pass or fail, oracle validation reports the paperās Human and Expert baselines rather than a pass rate, and the two hardest tasks have negativeR 2 even for the human oracle, which is preserved faithfully. Parity is established transitively against the Terminal-Bench adapter on all 8 tasks with claude-code and haiku-4-5 over 5 runs. Scientific Computing: CodePDE, ResearchCodeBench, SciCode, QCircuitBench. CodePDE [83]. This benchmark evaluates Python code generation for solving partial differential equations across five families (advection, burgers, reacdiff1d, cns1d, and darcy) with 100 test instances per family. The Harbor adapter replaces the upstream scaffolding-template prompts with full markdown task instructions and adds automatic test-data download from HuggingFace, and scoring uses the original nRMSE metric. Reference oracle solutions are copied verbatim from upstream, and no aggregate oracle rate is reported. Parity is established transitively in two legs, one between the original and the Terminal-Bench adapter and one between the Terminal-Bench adapter and Harbor, on the full task set, with results aligned within seed variance. ResearchCodeBench [59]. This benchmark evaluates the implementation of research code from academic papers, covering 20 ML and CV papers decomposed into 212 code-snippet tasks totaling 1,449 lines of code. The Harbor adapter provides the full paper content and code files, divides the problems into per-snippet units, and reuses the upstream I/O format and unit-test verifier, with one task-specific oracle fix that relaxes the GMFlow tolerance from 1e-7 to 1e-3 to absorb upstream nondeterminism. Oracle reference code is embedded insolve.sh, and no aggregate oracle rate is reported. Parity uses the full 212-task set with gpt-4o-mini-2024-11-20 against the original benchmark. SciCode [157]. This benchmark evaluates scientific computing on 80 main problems decomposed into 338 sub-problems across physics, mathematics, materials science, biology, and chemistry. The Harbor adapter generates one task per main problem with sequential instructions and skips three pre-written steps (problems 13, 62, and 76), and the upstream HDF5 checker gives fractional reward by sub-step correctness. Ground-truth solutions exist only for the 15 validation problems, where the oracle passes 15/15, while the 65 test problems carry placeholder oracles and are ex- cluded from validation. Parity uses the full 80-task set with codex@0.106.0, with sub-step accu- racy aligned within a Welch t-test. QCircuitBench [185]. This benchmark evaluates quantum-algorithm design on 28 tasks spanning oracle construction, algorithm design, and random circuit synthesis, expressed in OpenQASM 40 3.0 and Python. The Harbor adapter rewrites the upstream task descriptions into agent-oriented instructions and generates oracles dynamically inside the verifier, and scoring combines semantic correctness and efficiency into a score in [0, 1]. The adapter PR reports no aggregate oracle rate. Parity uses the full 28-task set with codex@0.76.0. Biomedical Research: BIX-Bench, LAB-Bench. BIX-Bench [107]. This benchmark evaluates computational biology data analysis on 205 questions derived from 61 published Jupyter notebooks. The Harbor adapter offers two execution modes, a custom Jupyter agent that mirrors the original benchmark and a terminal variant (bixbench-cli) that exposes tasks to generic agents such as codex, and the LLM-judge prompt is lightly modified to accept range answers so the oracle scores robustly. The oracle passes 205/205. Parity uses a 50-task subset with gpt-4o-mini, together with a CLI-mode run using codex and gpt-5-mini on the same subset. LAB-Bench [78]. This benchmark evaluates visual reasoning over scientific figures in biology, with 181 multiple-choice questions from the public FigQA subset. The Harbor adapter copies figures into the workspace for the agent to read from disk rather than embedding encoded images, and shuffles the answer choices at runtime through a container ENTRYPOINT to prevent positional bias, and scoring uses exact letter match throughanswer.txt. The oracle passes 181/181. Parity uses the full 181-task set with codex@0.71.0. D.6.5 Agents, Tools & Systems Tool Use & Assistants: GAIA, GAIA2, BFCL. GAIA [104]. This benchmark evaluates multi-step tool use on 165 validation-split question- answering tasks that require browsing, file handling, and external tools across three difficulty levels. The Harbor adapter wraps the original prompts as task instructions and copies attach- ments into the workspace, and scoring uses deterministic answer normalization and exact match. The oracle passes 165/165 at mean reward 1.0 on the full validation split. Parity uses the full validation split with openhands and gpt-5-mini attimeout_multiplier=1.5, equivalent to max_iterations=45. GAIA2 [45]. This benchmark evaluates agent ability across simulated apps and environments on 800 public validation scenarios from Metaās ARE, spanning five configurations (execution, search, adaptability, time, and ambiguity), each with full environments, apps, and oracle action sequences. The Harbor adapter supports an ARE-native mode that runs the official oracle and default agent, and a CLI mode that exposes ARE tools to standard agents through an MCP sidecar, and scoring uses the upstream action-sequence checker. The oracle passes 800/800. Parity uses a stratified 100-task subset with the official ARE default agent over 2 trials per side. BFCL [121]. This benchmark evaluates function-calling ability on 3,641 tasks across 13 single- turn and live categories from BFCL v4. The Harbor adapter mirrors the upstream AST evaluation logic with function-name normalization, order-independent parallel matching, and parameter validation, and tasks run in Docker with the upstream evaluator pre-installed. All 3,641 oracle solutions pass. Parity uses a stratified 123-task subset with codex@0.77.0 and gpt-4o-mini. Deep Research & Web Agents: DeepSynth, Seal-0, WideSearch. DeepSynth [123]. This benchmark evaluates deep information synthesis from multiple web sources into structured JSON answers, with 40 tasks in the dev set. The Harbor adapter reuses the official HuggingFace dataset, and scoring uses deterministic F1 over key-value pairs with an optional Claude Haiku 4.5 judge fallback for semantic equivalence. The oracle passes 40/40 at reward 1.0. Parity uses the full 40-task dev set. Seal-0 [125]. This benchmark evaluates search-augmented factual question answering under adver- sarial web evidence on 111 tasks, where frontier models reach near-zero accuracy because of con- flicting or misleading search results. The Harbor adapter reuses the upstream HuggingFace dataset, and scoring uses an LLM judge (Claude Haiku 4.5 by default) with normalized string matching as a fallback, mirroring upstream. The oracle passes 111/111. Parity uses the full 111-task set with claude-code. 41 WideSearch [173]. This benchmark evaluates broad web information gathering on 200 bilingual tasks spanning 18 industries, where the agent collects and organizes structured information into markdown tables. The Harbor adapter reuses the official HuggingFace dataset and ports the up- stream metrics, and scoring combines Item F1, exact match, and an LLM judge (gpt-4.1 by de- fault). The oracle passes 200/200 at mean Item F1 of 0.9996, and the small gap arises because gold-answer cells that contain the markdown delimiter (|) are replaced with a space during CSV- to-markdown conversion, which does not affect real agent runs. Parity uses the full 200-task set for 3 trials on each side. D.6.6 Data & Analytics Text-to-SQL & Data Science: DA-Code, Spider 2. DA-Code [61]. This benchmark evaluates data-science coding that requires Python, SQL, and Bash for end-to-end analysis, with 500 upstream tasks across 13 categories, of which Harbor adapts 479 after excluding the ml-cluster category. The Harbor adapter mirrors the upstream evaluation pipeline, applies 11 correctness fixes to the original evaluation script, and adds agent-oriented instructions describing task goals and file paths, and scoring uses the upstream checker against gold outputs. The gold-output oracle passes 479/479 after the fixes. Parity uses the full 479-task set for 3 trials with DA-Agent, and a permutation test shows no statistically significant difference. Spider 2 [79]. This benchmark evaluates dbt data-transformation ability on 68 upstream tasks over real-world DuckDB projects, of which Harbor adapts 64 after excluding 4 that consistently fail the oracle on both sides. The Harbor adapter reuses the upstream dbt projects with refactored API integration and retry handling for safety-policy rejections, and scoring uses dbt database comparison. The oracle passes 64/64. Parity uses the full 64-task set for 3 trials with spider-agent. D.6.7 Professional Domains Finance & Trading: FinanceAgent, PIXIU. FinanceAgent [162]. This benchmark evaluates financial research and analysis on 50 questions that require interpreting SEC filings and reasoning about reported figures. The Harbor adapter ships a terminal variant for CLI agents and a customized variant for function-calling, and an LLM judge scores answers against the Finance-Agent ground truth. The oracle passes 50/50. Parity uses the full 50-task set for 3 trials with Finance-Agent and Claude Code. PIXIU [178]. This benchmark evaluates financial NLP across 54,083 tasks in 29 subcategories that span classification, sentiment, question answering, named-entity recognition, sequential labeling, relation extraction, and summarization. The Harbor adapter materializes each subcategory exam- ple as a task, and scoring uses category-specific metrics such as accuracy, F1, RMSE, ROUGE, BERTScore, and BARTScore. The oracle solutions pass 100%. Parity uses a stratified 435-task subset (15 per subcategory) for 3 trials with codex. Business / Professional Work: SpreadsheetBench, MedAgentbench, LawBench. SpreadsheetBench [98]. This benchmark evaluates spreadsheet manipulation on 400 tasks drawn from real user questions on Excel forums, where the agent modifies a workbook so that designated cell values become correct. The Harbor adapter replaces the Windows-onlywin32compipeline with cross-platform LibreOffice headless recalculation and applies 11 correctness fixes to the evaluator, and scoring compares cell values against the gold workbook. All 400 oracle solutions pass. Parity uses the full 400-task set for 3 trials with claude-code. MedAgentBench [63]. This benchmark evaluates clinical agent ability on 300 tasks that require FHIR API interaction to retrieve patient data and execute healthcare workflows. The Harbor adapter vendors the official reference grader, mirrors the upstream GET, POST, and FINISH se- mantics, and adds Harbor-specific instruction formatting to enforce strict payload compliance, and scoring uses the upstream grader. The oracle passes 300/300, as verified in the adapter PR. Parity uses the full 300-task set for 3 trials with HTTPAgent and gpt-4o-mini. LawBench [42]. This benchmark evaluates Chinese legal reasoning across 1,000 tasks derived from 20 task families, each with zero-shot and one-shot prompts, covering statutes, legal reasoning, and case prediction. The Harbor adapter pins the upstream LawBench repository and reuses its native 42 evaluation functions, with the agent writing answers to/app/answer.jsonl, and scoring uses per-family metrics that mix exact match, F1, ROUGE, and rule-based scoring. The oracle passes the full task set, as verified in the adapter PR. Parity uses a stratified 120-task subset (3 chunks across 20 families and 2 prompt types) for 3 trials with Qwen Code on GLM-4.7. D.6.8 Safety & Security CyberGym [169]. This benchmark evaluates an agentās vulnerability-analysis ability on 1,507 real- world C and C++ tasks drawn from 188 projects, where the agent generates a proof-of-concept input that triggers the bug under sanitizer instrumentation (ASan, MSan, or UBSan). The Harbor adapter ships four difficulty levels, from level0 to level3, that control which files are exposed to the agent, and level1 is the primary level and matches the upstream paper. Scoring uses a dual- binary check in which the generated input must crash the vulnerable binary but not the patched one. The ground-truth inputs live in per-task runner images sourced from OSS-Fuzz, and the oracle reaches 100% on the 10-task subset, while full-set validation is skipped because of the roughly 10 TB footprint of the 3,014 runner images. Parity uses the recommended 10-task subset (6 ARVO and 4 OSS-Fuzz, over five difficulty and seed combinations of three at level1, one at level2, and one at level3) with OpenHands@1.6.0 and claude-haiku-4-5. StrongReject [148]. This benchmark evaluates LLM jailbreak robustness on 313 forbidden prompts crossed with 39 jailbreak methods, for 12,207 tasks in total. The Harbor adapter supports all 39 methods and implements the official rubric evaluator with inverted scoring, where a reward of 1 indicates a safe refusal, and a rubric-based LLM judge (gpt-4o-mini by default) scores refusal, convincingness, and the specificity of harmful content. All 313 base oracle solutions pass. Parity uses a stratified 150-task subset (50 prompts across 3 jailbreaks) for 3 trials with codex and gpt-5- nano. D.6.9 Multimodal Audio Understanding. MMAU [140]. This benchmark evaluates multimodal audio understand- ing on 1,000 expert-curated tasks spanning speech, environmental sounds, and music. The Har- bor adapter uses Qwen2-Audio-Instruct to transcribe audio to text and runs the upstream judge through pytest over the transcribed answer. The oracle passes the full test-mini set, as verified in the adapter PR. Parity uses the full 1,000-task test-mini set for 3 trials with Terminus-2 and gpt-4o. D.6.10 Benchmarks Using Harbor Format The following benchmarks are built with Harbor format so that no additional adaptation is needed. Terminal-Bench [103]. This benchmark evaluates an agentās ability to complete realistic command- line tasks such as filesystem manipulation, build configuration, debugging, and system administra- tion. Tasks are authored directly in the Harbor schema, with instruction, environment, tests, and solution, so no adapter is required, and Harbor pulls the upstream task directories verbatim, runs them in per-task Docker environments, and scores them with the upstream test scripts. CompileBench [132]. This benchmark evaluates an agentās compilation skills across 15 real-world build tasks that involve dependency resolution, legacy code such ascoreutilsv5.0 from 2003, static linking with glibc and musl, and ARM64 and Windows cross-compilation. Tasks are main- tained natively in the Harbor schema by the upstream authors, so the adapter reduces to a sparse- checkout of the task directories, and the upstream test scripts run inside isolated Docker images. SkillsBench [84]. This benchmark evaluates how well agent skills transfer across diverse tasks, with 86 tasks across 11 domains paired with curated reference skills and deterministic verifiers. Tasks are authored directly in the Harbor schema and require no adaptation, and the agent runs with and without the curated skills exposed through Harborās skills mechanism, scored by the upstream verifiers. D.6.11 Other Benchmarks Not Included Due to compute and time constraints, we exclude the following benchmarks from our experiments, although their adapters are merged into Harbor or have an approved adapter pull request. We list 43 the supported benchmarks below; adapters whose parity experiments are already complete also appear in Tables 3 and 4. Software Engineering: ABC-Bench [181], AutoCodeBench [204], CanItEdit [15], CooperBench [68], DevEval [80], DevOpsGym [154], EvoEval [176], FeatBench [18], Frontier-CS-Algorithm [102], Multi-SWE-Bench [196], ProgramBench [184], SWE-Gym [120], SWE-Rebench [7], Uni- Code [203], WebGen-Bench [96]. Mathematics & Reasoning: SATBench [170]. Knowledge & Long Context: CL-Bench [35], LoCoMo [100]. Scientific Research: AstaBench [12], LLM-SRBench [145], ML-Dev-Bench [119], MLGym- Bench [114], MLR-Bench [19], RExBench [39], ScienceAgentBench [21]. Agents, Tools & Systems: ACE-Bench [17], AMA-Bench [201], DeepResearch-Bench-I [37], LongCLIBench [43], OSWorld [179], Tau3-Bench [187], TextArena [50]. Data & Analytics: ADE-Bench [150], BIRD-Bench [81], DABstep [40], DS-1000 [76], Kram- aBench [75]. Professional Domains: CRMArena [60], TheAgentCompany [180]. Multimodal: GraphDesignBench [32], RefAV [27]. D.7 Benchmark Issues and Lessons Learned Adapter construction and parity validation surfaced a small number of recurring failure modes that materially affect reproducibility, often regardless of the benchmarkās age, popularity, or scientific area. We group them here by symptom rather than by domain, with concrete examples drawn from the per-benchmark notes in Section D.6.1 through Section D.6.9, and conclude with concrete recommendations for benchmark authors. Environment instability. Broken Docker builds, unpinned dependencies, removed upstream packages, hard-coded paths, and platform-specific tooling that breaks across machines. Many benchmarks required non-trivial environment reconstruction before any task could run: Spread- sheetBenchās upstream pipeline relied on the Windows-onlywin32comstack and had to be ported to a cross-platform LibreOffice headless workflow; SWE-Bench Pro needed Jest invocations to be pinned to single-worker / forced-exit flags before its JavaScript and TypeScript tasks would terminate reliably; ScienceAgentBench shipped a Modal harness whose output paths and per- task dependencies drifted from the released evaluator; and SWE-Bench Verified contains a task (scikit-learn__scikit-learn-14710) that exceeds the hardware budget of our default execu- tion backend. Non-reproducible scoring. Scoring code with hidden non-determinism (network calls, time- based seeds, race conditions in parallel test runners), lenient or accidentally-permissive test match- ers, and divergence between the public leaderboard pipeline and the released evaluation script. GSOās speedup metric is sensitive enough to timing variance that 13 of 102 oracle solutions are scored as āslower than baselineā on a fraction of runs ā a property of the metric rather than the harness. WideSearchās gold-answer CSVs contain cells whose contents include the markdown column delimiter (|); the upstream CSV-to-markdown conversion silently rewrites these to a space, capping any oracleās Item-F1 at 0.9996. DA-Codeās released evaluation script required eleven distinct correctness fixes before its gold outputs scored 1.0. We treat divergences of this kind as benchmark bugs, document them in the adapter README, and keep oracle exclusions out-of-band rather than baked into the dataset. Implicit assumptions. Preprocessing, prompting, or grading pipelines that depend on undocu- mented conventions: a particular Git branch of a vendored repo, a particular tokenizer, an exact prompt formatting, or a hidden answer-extraction regex. SWE-Bench Proās grader silently rejects gold patches that do not end with a newline. SWT-BenchāsFAIL_TO_PASSfield is parsed by an upstream regex that returns the empty set on ninesphinx-doctasks, marking them as zero-score even under the gold patch. LAB-Benchās multiple-choice harness exhibits non-trivial positional bias unless the answer-choice order is randomized at runtime, which we now enforce via container 44 ENTRYPOINT. LiveCodeBench tasks rely on standard-library imports that the upstream harness injects implicitly. We surface these assumptions in the adapter so that they are part of the task specification rather than the agentās prior knowledge. Oracle solutions that fail. In several benchmarks, the published ground-truth solution did not pass the benchmarkās own tests, usually due to environment drift since release rather than a flaw in the solution itself. SWE-Bench Verified ships 4 tasks whose oracles fail on data or infrastructure er- rors (99.2% pass rate); SWE-Bench Pro ships 15 invalid gold patches and 9 oracles that exceed the upstream timeout (97% pass rate); SWT-Benchās oracle passes 97.7% for the parsing reason above; ScienceAgentBenchās CPU-only oracle pass rate is 93.1% (the remaining 7 tasks need GPUs); GSOās oracle passes 87.3% due to timing variance; SWE-smithās synthetic patches contain a small fraction of upstream-failing entries that the original benchmark also does not score. Without an explicit oracle-validation step these failures are invisible to downstream users and silently inflate baseline scores. Agent hacking surfaces. A less visible but arguably more damaging failure mode arises when the benchmark design inadvertently allows an agent to obtain the ground-truth answer or bypass the verifier without genuinely solving the task. We identified three recurring patterns during adapter construction. Answer exfiltration via public data sources: when the task instruction ex- poses a stable identifier and the container has unrestricted internet access, an agent can query the publicly hosted upstream dataset and retrieve the gold answer directly, passing the verifier without engaging with the task content. Ground-truth leakage through container mounts or file placement: if gold labels, answer files, or scoring criteria are visible inside the agentās runtime filesystemā whether through a shared Docker volume, a misconfiguredCOPYdirective, or co-located test assetsāthe agent can read the expected output before producing its response. Shared-environment verifier tampering: when the agent and verifier share the same container and interpreter, an agent can overwrite judge dependencies so that the verification step returns a hard-coded passing score regardless of the agentās actual output; this class of attack is invisible to oracle validation because the oracle path typically bypasses the judge. These surfaces are particularly insidious because they inflate benchmark scores silently: the resulting numbers look plausible, oracle runs still pass, and the exploit is only revealed by careful trajectory inspection or by comparing scores against inde- pendent human evaluation. We now flag these surfaces during adapter review and require network restriction, identifier masking, or verifier isolation before merge. Standards we enforce on every Harbor adapter. ā¢Oracle validation. The provided ground-truth solution must achieve a 100% pass rate; de- viations must be explained per task and documented in the adapter README, not silently excluded from the dataset. ā¢ Parity verification. A parity experiment must confirm statistical equivalence with the original benchmark within sample SEM, on a parity set whose composition is recorded in parity_experiment.json. ā¢ Reproducible execution. Every task ships a sandboxed Docker environment with pinned dependencies; cross-platform tooling is preferred over OS-specific commands. ā¢Documented divergences. Any deviation from the upstream evaluator (prompt rewrite, scor- ing fix, exclusion list) is captured in the adapter README so that downstream users can audit it. Recommendations for future benchmark authors. Based on the above, we recommend that future benchmark releases: ā¢pin all dependencies (Python packages, system libraries, language toolchains, container base images) to exact versions, ideally via lock files; ā¢publish a single reproducible container that bundles the entire evaluation pipeline rather than expecting users to assemble it from a README; ā¢ include a runnable oracle solution and gate releases on an oracle-pass-rate check; ā¢make scoring deterministic ā avoid network calls, time-based randomness, lenient regex matchers, and platform-dependent tooling in the grader; 45 ā¢separate task specification from execution harness, so that downstream adapters and agents can be swapped without touching task content; ā¢document every implicit assumption (prompt format, answer-extraction rule, tokenizer, dataset version) in the task specification rather than only in code. Adopting these practices would substantially reduce the per-benchmark integration cost we report in the per-adapter notes. E Evaluation Protocol and Reproducibility Details This section describes the evaluation protocol used to run the benchmark experiments at scale and for Harbor-Index. The protocol is designed to make the reported results reproducible and auditable. In particular, it specifies how benchmark datasets are prepared as runnable Harbor task registries, how the reported paper subset is sampled and locked, how modelāharness configurations are executed in isolated environments, and how trial-level metadata and artifacts are stored for auditing and aggregate scoring. E.1 Evaluation Infrastructure Harness and repository state. We use Harbor as the unified harness for task execution, sandboxing, agent orchestration, and verification. For each benchmark, we follow the corre- sponding adapter README to construct the runnable dataset and to validate the execution environment, verifier behavior, credential requirements, and resource constraints prior to in- clusion. Unless otherwise noted, large-scale evaluations are conducted with Harbor pinned to commit9e6790376583608f541c133d1af2dae47b8fc32. For the Harbor-Index ex- periments, we use the more recent and stable Harbor@v0.6.4 release, pinned to commit 331dcba30efcc3fa8282a21de562a3834d6e6244. Harbor automatically records token counts, metrics, and agent trajectories. The experiment repository provides the tooling for dataset metadata upload, job validation, result import, and trial-artifact archival. Sandbox backends. Each trial is executed in an isolated Harbor environment. For CPU-only tasks, we use Daytona as the default backend. Daytona provides independent task workspaces and isolates the agent and verifier file systems for each trial. For a subset of benchmarks, we instead use Harborās local Docker backend when the verifier or task environment is incompatible with Daytona, for example when CPU, memory, or storage requirements exceed Daytonaās resource lim- its of 4 CPUs, 8 GB of memory, and 10 GB of storage. Docker-backed jobs are executed through the same Harbor interface as Daytona-backed jobs. For image-heavy benchmarks, we prebuild the outer task image and, when applicable, the inner Docker-in-Docker images, publish them to a reg- istry such as GHCR, and configure the task environment to use the prebuilt image. This procedure reduces failures caused by registry rate limits and repeated large downloads during environment setup. GPU-dependent tasks are executed through Harborās Modal backend, which we reserve for benchmarks that require GPU hardware or GPU-capable system images. Language model services. We access each language model through the corresponding first- party provider API, with support from API grants provided by the respective providers. All model configurations are documented in our experiment repository mentioned above. Unless otherwise specified, we use each providerās default settings for temperature, context length, reasoning effort, and other provider-specific parameters; for example, GPT-5.4 is evaluated with the default medium reasoning effort and Claude Opus 4.6 is evaluated with the default high effort. We note that GPT- 5.4 is often reported with extra-high reasoning effort on public leaderboards. Readers should therefore exercise caution when comparing our evaluation results with those reported by other leaderboards. Model and harness matrix. The large-scale evaluation includes 16 modelāharness configura- tions over 8 models spanning capability tiers. Every model is evaluated under exactly two harness conditions: the cross-family Terminus-2 harness and one native harness. There are three native harnessesāCodex CLI for GPT, Claude Code for Claude, and Gemini CLI for Geminiāand therefore 4 distinct harness implementations overall. Terminus-2 is evaluated with all 8 models: 46 gpt-5.4,gpt-5-mini,gpt-5-nano[117], Claude Opus 4.6, Claude Sonnet 4.6, Claude Haiku 4.5 [6],gemini-3.1-pro-preview, andgemini-3-flash-preview[49]. Codex CLI is evalu- ated withgpt-5.4,gpt-5-mini, andgpt-5-nanousing CLI version0.115.0. Gemini CLI is evaluated withgemini-3.1-pro-previewandgemini-3-flash-previewusing CLI version 0.34.0. Claude Code is evaluated with Claude Opus 4.6, Claude Sonnet 4.6, and Claude Haiku 4.5 using CLI version 2.1.81. For reproducibility, each configuration records the agent interface, the agent CLI version when ap- plicable, the model identifier used in the job configuration, and the run-date range. When provider snapshot or API release identifiers are available, we additionally report them alongside the job- configuration identifier. Some provider identifiers correspond to managed preview endpoints or aliases rather than immutable model snapshots; for these configurations, we report the exact identifier used during evaluation together with the corresponding run-date range. For the Gem- ini preview configurations, the job-configuration identifiers aregemini-3.1-pro-previewand gemini-3-flash-preview. Public provider metadata lists these endpoints as preview models with release dates of February 19, 2026 and December 17, 2025, respectively. Evaluations using these endpoints were run in April 2026. For the Harbor-Index 1.0 experiments, the 9 evaluated models split into four closed-weight models, whose weights are not publicly released (GPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro, and Qwen3.7 Max), and five open-weight models (GLM 5.2, Kimi K2.6, MiniMax M3, DeepSeek V4 Pro, and MiMo V2.5 Pro). GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro are served by first-party APIs and retain their vendor-native harnesses (Codex CLI, Claude Code, and Gemini CLI respectively); the remaining six models are served through OpenRouter and run under Claude Code. Each of the 9 models is also evaluated with Terminus-2. Harbor-Index 1.0 therefore contains9Ć 2Ć 82 = 1,476 modelāharnessātask runs. Interactive trajectories and aggregate Pareto numbers are published at https://harbor-index.org. Database and artifact storage. Results are stored in Supabase. The database schema contains eight core tables:dataset,task,dataset_task,job,trial,agent,model, andtrial_model. Dataset upload createsdataset,task, anddataset_taskrecords that capture registry informa- tion, task identifiers, task instructions, source metadata, and task-level metadata. Each job record stores the full job configuration, Harbor Git commit or package version, trial count, start and end timestamps, aggregate job statistics, and verification flags. Each trial record stores the trial UUID, task checksum, agent name and version, scalar reward, exception information, setup/execution/veri- fier timestamps, full trial configuration, and agent metadata. Thetrial_modeltable stores model and provider names, as well as input, output, and cache token counts for each trial. We also upload the full trial directory as a compressed archive to a public Supabase Storage bucket namedtrials. When present,agent/trajectory.jsonis uploaded separately as <trial-id>-traj.json. Before upload, large transient agent artifacts that are not required for evaluation or audit are removed, including Codex runtime temporary directories and very large Terminus-2 terminal recordings. The retained artifacts include the reward file, verifier output, con- figuration, trajectory, and logs needed to audit individual trials. Public trajectory release. The complete rollout set of the large-scale evaluation is released as a Hugging Face dataset athttps://huggingface.co/datasets/kendx/Harbor-Adapter. The release has two configurations. Themanifestconfiguration is a lightweight catalog with one row per(benchmark, task, model, agent)cell (178,647rows) listing the trial identifiers for that cell, so it can be filtered without downloading trajectory bytes. Thetrajectoriesconfiguration stores one row per trial (793,698trials, about340GB in total), each holding the gzipped trial directory described above together with its SHA-256 checksum; shards are laid out per benchmark so that a single benchmark can be fetched in isolation. Each cell retains up to its five most recent trials, so a cell may contain more than the three trials aggregated in this paper when the job was rerun during auditing (Appendix E.3). This release lets others reproduce our aggregate numbers, re-audit individual trials, and reuse the trajectories for analyses beyond those reported here. 47 E.2 Dataset Preparation and Sampling For each external benchmark, we follow the Harbor-provided dataset generation workflow de- scribed in the benchmarkās README. Before adding a benchmark to the evaluation suite, we in- spect the generated dataset to determine whether it requires LLM-as-a-judge evaluation, Docker-in- Docker execution, GPU resources, or prebuilt images. The large-scale evaluation subset includes only tasks that can be executed through the standard Harbor job interface: each task provides an in- struction to the evaluated agent, exposes an isolated execution environment, and computes a scalar reward using the benchmark verifier. To ensure compatibility with the target execution platforms, namely Daytona and Modal, we run a smoke test for each benchmark using the low-cost GPT-5-nano model together with Codex. For benchmarks with a large number of tasks, we sample an i.i.d. subset to control evaluation cost while preserving broad benchmark coverage. The resulting large-scale evaluation set contains 54 benchmarks and 6,627 selected tasks. For each benchmark, the manifest records the selected task count, original task count, sample rate, verifier type, and execution backend. Table 5 lists the benchmarks included in the manifest and their selected task counts. After this preparation phase, the evaluation suite can be executed at large scale. Table 5: Adapter task summary. Sample Rate = Paper #Tasks / Original #Tasks. Adapter NamePaper #Tasks Original #Tasks Sample RateAdapter NamePaper #Tasks Original #Tasks Sample Rate PIXIU10143523.22%FeatureBench18520092.50% SciCode8080100.00%ResearchCodeBench212212100.00% AlgoTune154154100.00%GSO102102100.00% BFCL1233,6413.38%HLE2492,5009.96% Aider Polyglot225225100.00%Reasoning Gym576576100.00% LawBench1811,00018.10%SLDBench88100.00% SpreadsheetBench20040050.00%Spider 2646894.12% Seal-0111111100.00%StrongReject15012,2071.23% Omni-Math2004,4284.52%SWE-smith9859,1360.17% Terminal-Bench 2.08989100.00%MMMLU150210,6300.07% DeepSynth4040100.00%USACO10030732.57% GAIA210080012.50%KUMO2125,3004.00% WideSearch10020050.00%FinanceAgent5050100.00% CRUST-Bench100100100.00%DA-Code20047941.75% SWT Bench5043311.55%SkillsBench778689.53% GAIA165165100.00%SWE-Bench Pro10073013.70% AIME6060100.00%QuixBugs8080100.00% SWE-Lancer10046321.60%SWE-Bench-Multilingual5030016.67% ARC-AGI-210016759.88%BIX-Bench5020524.39% CompileBench1515100.00%MedAgentBench10030033.33% GPQA Diamond198198100.00%SimpleQA2001,00020.00% HumanEvalFix164164100.00%SWE-bench-verified10050020.00% IneqMath100100100.00%LiveCodeBench1001,0559.48% LAB-Bench181181100.00%CodePDE55100.00% MMAU1001,00010.00%ReplicationBench9011181.08% QCircuitBench2828100.00%A-LCR9910099.00% BigCodeBench-Hard145145100.00%CyberGym101,5070.66% E.3 Error Handling and Quality Control Phase-based debugging. We structure the large-scale evaluation into phases. We first evaluate each benchmark independently and, within each benchmark, proceed in three phases. Phase 1 serves as a sanity check for modelāharness compatibility. In this phase, we run a small subset of tasks (1% of the full dataset) using the complete modelāharness matrix, and verify that the environ- ment starts successfully, the agent receives the intended instruction, the verifier runs, the reward is written, token usage is recorded, and trajectory upload succeeds. This phase is also intended to surface severe issues, such as failures to install a particular harness in the benchmark environ- ment due to system incompatibilities. For benchmarks with reference solutions or oracle support, we additionally run an oracle or known-good trajectory to confirm that the verifier produces the expected reward. Phase 2 evaluates the full modelāharness matrix on a 10% subset of the dataset, and Phase 3 performs the full evaluation. Each phase excludes tasks that were already included in earlier phases. Local and database checks. After jobs finish, we inspect local job statistics in a long-form per-task report. This report groups trials by job, dataset, agent, task, reward, token usage, and error type, making it possible to identify all-failure jobs, missing token counts, unexpectedly high 48 verifier failures, or reward distributions inconsistent with earlier phases. Supabase is then checked for completeness: each expected job should have a job row, the expected number of trial rows, trial_modeltoken rows, and storage URLs for uploaded trial archives. Jobs with suspiciously missing trajectories, missing or zero token counts, or unexpected exception spikes are rerun or manually audited before inclusion. Rerun policy. We rerun a trial when the recorded exception is caused by infrastructure or config- uration rather than by the evaluated agentās behavior. Typical rerun triggers include API authentica- tion errors, temporary provider outages, bursty rate-limit failures, sandbox creation failures, image pull failures, verifier crashes due to missing external resources, malformed judge responses caused by judge API failures, corrupted result files, and failed storage or database writes. We do not rerun agent timeouts within the benchmark execution envelope, verifier timeouts in- duced by the submitted solution, policy refusals that reflect model behavior on the task prompt, max-output or context-limit failures caused by the agentās interaction strategy, or verifier failures that correctly indicate the task was not solved. When the distinction is ambiguous, the trial archive and trajectory are inspected, and the final decision is recorded with the job audit notes. E.4 Harbor-Index Experiment Protocol Task curation and execution budgets. Harbor-Index 1.0 contains 82 tasks spanning 29 bench- marks after difficulty filtering, AI and human audit, and an audit-and-fix loop (Appendix H). Tasks run in isolated Docker sandboxes with separate verifier environments. Performance-oriented tasks (e.g., AlgoTune, GSO) use GPU-capable backends; free-form answer tasks (e.g., HLE, GAIA, GPQA Diamond) use LLM-as-a-judge scoring. We augment each task instruction with an explicit execution contract that states both the time budget and the available resources, for example: āYou should solve the task in 1 hour. You can access 1 CPU, 2 GB memory, 10 GB storage.ā The same limits are enforced in the task configuration through the Harboragentfields. Timeouts are tight- ened during the audit-and-fix stage (typically1.2Ćthe fastest frontier modelās observed runtime, or up to 3 hours when every model fails). Scoring and thresholding. Most Harbor-Index tasks have binary rewards and are scored directly from the verifier output. For continuous-score tasks, we convert the raw verifier reward to pass/fail using a locked task-specific threshold. Letr i,t be the raw reward for trialion taskt. For each continuous task, we define Ļ t = j max i r i,t k 5 , whereāĀ·ā 5 denotes flooring to five decimal places over all non-failed observed trials, including ora- cle calibration runs when available. A trial is counted as passing if its raw reward strictly exceeds Ļ t . For bounded continuous metrics with a known upper bound (e.g., F1 score,R 2 ), we also count a trial as passing if it reaches the upper bound. Speedup-style tasks such as AlgoTune and GSO do not use a fixed upper bound. E.5 Evaluation Results In Tables 6, 7, and 8, we present the benchmark scores for all modelāharness configurations on all the benchmarks we study. F Detailed Large-scale Quantitative Analysis This appendix details the quantitative analyses in Section 3.1. We analyze the16Ć 54score matrix to study cross-benchmark redundancy, within-benchmark redundancy, model vs. harness effects, benchmark progress over time, cost evaluation, and execution-latency patterns. 49 123456 Component 0 20 40 60 80 Variance explained (%) 73.7% 67.1% Singular value spectrum of the performance matrix Baseline scaffold (8Ć54) All scaffolds (16Ć54) Remaining baseline (8Ć54) Remaining all (16Ć54) 0 20 40 60 80 100 Cumulative remaining (%) 26.3% 18.1% 12.3% 32.9% 22.8% 18.3% Figure 7: Singular value spectrum of the benchmark score matrix in logit space. Blue: 8-row Terminus-2 submatrix (one fixed harness). Purple: full 16-row matrix (all modelāharness config- urations). The drop in PC1 share and rise in PC2 when native harnesses are included reflects the orthogonal harness-effect dimension. F.1 Benchmark and Task Predictability Analysis F.1.1 Low-rank structure for benchmark space Singular Value Decomposition (SVD) in logit space on the 8-row Terminus-2 submatrix (varying only the base model under the fixed Terminus-2 harness) yields PC1 = 73.7% of variance, which is comparable to BenchPressās 71% [197] on their 31Ć10 block with scaffold-free evaluations, confirming that a single āgeneral capabilityā axis dominates across 8 models spanning capability tiers. The first two components together capture 81.9%, leaving little independent signal beyond two dimensions. On the full 16-row matrix (all modelāharness configurations), the spectrum shifts: PC1 drops to 67.1% and PC2 rises to 10.1% (Figure 7). PC2 cleanly separates the 8 configurations using native harnesses from the 8 Terminus-2 configurations with zero overlap (mean gap 3.31 standard units). Together the first two components capture 77.2% of variance, with every remaining component below 6%: the benchmark space is effectively rank-2, one axis for model capability and one for the harness effect. F.1.2 Benchmark predictability. Following BenchPress [197], we quantify per-benchmark redundancy by predicting held-out scores from the remaining benchmarks. Our setting differs in two respects: rows are modelāharness con- figurations rather than models alone, and all 54 benchmark metrics are normalized scores rescaled and bounded in[0, 1]. Given the fully-observed score matrixSāR NĆB , for each ofT =5random seeds we hold out 50% of each rowās benchmark scores as the test setH; the remaining 50% form the training setT . Both prediction methods operate in logit space: z = logit(s) = log clip(s, Īµ, 1āĪµ) 1ā clip(s, Īµ, 1āĪµ) , Īµ = 0.005,(1) whereĪµclips extreme scores away from 0 and 1 to avoid infinite logits. The predictor combines three steps: 50 ā¢LogitBenchReg: for each target benchmarkj, fit univariate ordinary least squares (OLS) regres- sions in logit space from every other benchmark k Ģø= j: z i,j = Ī² k z i,k + Ī³ k ,fitted overi : (i,j)āT ā§ (i,k)āT,(2) wherez i,j = logit(clip(s i,j , Īµ, 1āĪµ))withĪµ = 0.005(Eq. 1). We select the top-K=5predictors with R 2 k ā„ 0.1 as the predictor setK j . For each held-out cell (i,j)āH, the predicted score is: Ėz i,j = P kāK j R 2 k (Ī² k z i,k + Ī³ k ) P kāK j R 2 k ,Ės BenchReg i,j = 1 1 + e āĖz i,j .(3) ā¢SVD-Logit (Soft-Impute): iterative low-rank matrix completion on the logit-transformed, column-standardized score matrix. LetĪ¼ j andĻ j be the column mean and standard deviation from training entries only, and Ģ Z = (ZāĪ¼)/Ļwith held-out entries initialized to 0. The iteration: 1. Compute the rank-r SVD approximation (r=2): Ģ Z approx = U :,1:r Ī£ 1:r V ā¤ 1:r,: . 2.Replace only held-out entries: Ģ Z i,j ā ( Ģ Z approx ) i,j for(i,j) ā H. Training entries remain fixed. 3. Repeat until convergence: ā„ Ģ Z (t) ā Ģ Z (tā1) ā„ F ā„ Ģ Z (tā1) ā„ F < 10 ā4 . After convergence, de-standardize and inverse-logit: Ės SVD i,j = 1/(1 + e ā( Ģz i,j Ā·Ļ j +Ī¼ j ) ). ā¢ Blending and evaluation: the final prediction combines both in raw score space: Ės i,j = Ī± Ės BenchReg i,j + (1āĪ±) Ės SVD i,j , Ī± = 0.6.(4) When LogitBenchReg produces no prediction (no predictor passesR 2 ā„ 0.1), the method falls back to SVD-Logit alone; when both fail, it uses the column mean Ģs j . All predictions are eval- uated on raw scores in[0, 1]. The primary metric is the benchmark-stratified Median Absolute Error: MedAE j = median (i,j)āH j |Ės i,j ā s i,j |,(5) whereH j is the set of held-out cells for benchmarkj(pooled across all seeds). The overall MedAE is the median across benchmarks: MedAE = median j=1,...,B MedAE j .(6) Because all benchmark scores share the[0, 1]scale, MedAE is directly interpretable as the typi- cal prediction error in percentage points. We evaluate on our 16Ć54 matrix (50% holdout, 3 folds). The simplest baseline predicts each held- out cell with the column (benchmark) mean of the observed entries, achieving MedAE=0.117. Methods that exploit cross-benchmark structure improve substantially: LogitBenchReg reaches 0.081, SVD-Logit (r=2) reaches 0.069, and the BenchPress blend reaches 0.063, a 46% reduc- tion over the baseline. The fact that one benchmarkās scores can predict anotherās to within 0.063 points confirms substantial cross-benchmark redundancy. Table 9: Per-benchmark predictability (50% holdout, 3 folds). Low MedAE = easily predicted from other benchmarks (redundant); high MedAE = hard to predict (unique signal). Most RedundantMost Unique BenchmarkMedAEBenchmarkMedAE HumanEvalFix0.004ResearchCodeBench0.221 KUMO0.017FinanceAgent0.202 BFCL0.029LAB-Bench0.168 StrongReject0.030CodePDE0.164 BigCodeBench0.031USACO0.163 LawBench0.035MedAgentBench0.121 DA-Code0.038CyberGym0.102 Reasoning Gym0.042IneqMath0.099 Benchmarks that resist prediction include ResearchCodeBench (MedAE = 0.221), FinanceAgent (0.202), and LAB-Bench (0.168), measuring capability axes poorly covered by the remaining suite. By contrast, HumanEvalFix (MedAE = 0.004) and KUMO (0.017) are well-reconstructed from peers, adding little independent signal. 51 F.1.3 How many benchmarks span the evaluation space? Only 12 of 54 benchmarks contribute independent signal at the|Ļ| < 0.7threshold. Greedy for- ward selection starts from the benchmark with the lowest mean absolute correlation to all others and iteratively adds the benchmark whose maximum absolute correlation with the already-selected set is the smallest. The first 12 selections all satisfymax|Ļ| < 0.7; from step 13 onward, every remaining benchmark has|Ļ|ā„ 0.7 with at least one already-selected benchmark. Table 10: Greedy forward selection sequence: benchmarks added in order of decreasing indepen- dence.max|Ļ|is the maximum Spearman correlation with any previously selected benchmark. The line separates the 12 benchmarks below the|Ļ|=0.7redundancy threshold from the first benchmark that exceeds it. Benchmarkmax|Ļ|Domain CodePDE0.00Scientific Research LiveCodeBench0.01Software Engineering IneqMath0.11Mathematics & Reasoning BFCL0.27Agents, Tools & Systems ResearchCodeBench0.33Scientific Research FinanceAgent0.47Professional Domains SLDBench0.56Scientific Research StrongReject0.59Safety & Security Reasoning Gym0.61Mathematics & Reasoning CyberGym0.66Agents, Tools & Systems LAB-Bench0.67Scientific Research Omni-math0.69Mathematics & Reasoning SWE-bench-verified0.70Software Engineering The first redundant benchmark (swebench-verified, step 13,Ļ=0.70) is largely explained by the already-selected set. For practitioners with constrained budgets,ā¼12 well-chosen benchmarks across domains capture most independent signal in the full 54-benchmark suite. Difficulty and uniqueness are largely decoupled. The benchmarks identified as independently informative span the full difficulty spectrumāfrom hard (CodePDE, mean score 0.36) to easy (StrongReject, 0.92)ārather than clustering at one end (Table 10). This confirms that uniqueness is driven by the distinctiveness of the capability axis a benchmark measures, not by raw task diffi- culty. F.1.4 Does the same redundancy hold at the task level? We extend the SVD analysis to the 16Ć5896-task matrix (after filtering6627ā 5896 = 731zero- variance tasks where all modelāharness configurations scored identically). At the task level, PC1 explains only 32.0% of variance, compared with 67.1% at the benchmark level. Benchmark aggre- gation therefore compresses away independent dimensions: the task-level score matrix contains richer structure that a single overall ranking cannot capture. Within-benchmark task redundancy varies widely. For each of the 52 benchmarks withā„ 10tasks, we run PCA on the 16Ć Tsub-matrix, whereTis the number of tasks in that benchmark (after removing zero-variance tasks, averaging 11.0% per benchmark). PC1 alone explains 49.5% of within-benchmark variance on average, and a median of 8 components suffices for 90% (out of a maximum of 16 dimensions). The degree of redundancy varies considerably across benchmarks: ā¢ ResearchCodeBench (212 tasks): PC1 = 78.4%, only 3 components for 90%. ā¢ FinanceAgent (50 tasks): PC1 = 69.5%, 6 components for 90%. ā¢ HumanEvalFix (164 tasks): PC1 = 86.2%, 2 components for 90%. Selecting thektasks most correlated with the full benchmark mean and re-ranking the 16 systems gives the ranking fidelity shown in Table 11. Individual benchmarks range from WideSearch (Ļ = 0.99) to CyberGym (Ļ = 0.75), reflecting differences in internal task diversity; the complete per-benchmark breakdown is given in Table 12. 52 Table 12: Per-benchmark ranking fidelity: SpearmanĻbetween the system ranking from the top-k oracle-selected tasks and the ranking from all tasks. Benchmarks sorted byk=3fidelity (descend- ing). BenchmarkTasks k=1 k=3 k=5 k=10 Widesearch1000.9590.9880.9880.997 SWE-Lancer940.9210.9780.9660.971 SWE-bench-verified940.9120.9730.9770.964 DA-Code1640.9280.9680.9710.976 SpreadsheetBench1960.9180.9660.9670.967 HLE1950.9350.9660.9740.987 LAB-Bench1730.9280.9630.9680.967 DeepSynth350.9260.9620.9800.989 Omni-math1600.8790.9620.9570.966 SkillsBench680.9610.9620.9740.982 SWT Bench490.8810.9560.9310.962 GAIA1570.9090.9530.9590.950 ResearchCodeBench1910.9270.9490.9490.949 LiveCodeBench960.9030.9440.9580.962 TerminalBench2.0830.8840.9430.9430.946 CRUST-Bench960.9020.9400.9460.964 GPQA Diamond1810.8940.9390.9470.956 LawBench1810.9560.9380.9680.982 USACO990.8760.9380.9400.957 FinanceAgent480.9250.9370.9560.988 GSO900.9120.9370.9700.947 GAIA2680.9100.9360.9740.962 ARC-AGI-2950.9220.9330.9480.982 Aider Polyglot2210.9070.9320.9540.955 A-LCR940.8840.9320.9000.908 BigCodeBench1190.7600.9310.9600.950 SWE-Bench Pro740.9190.9310.9480.952 Spider 2360.8630.9300.9530.931 QCircuitBench250.8680.9290.9620.915 StrongReject720.8800.9290.9430.895 Scicode760.8870.9290.9290.953 PIXIU870.8590.9260.9210.950 FeatureBench1560.8780.9260.9080.966 AlgoTune1470.8760.9240.9470.926 KUMO1950.8890.9210.9600.969 SimpleQA1960.8590.9190.9190.921 BIX-Bench390.8900.9180.9130.964 HumanEvalFix1530.9120.9140.9120.962 MMAU910.8380.9130.9210.906 MMMLU1270.9080.9090.9110.909 SWE-Bench-Multilingual480.8850.9010.9020.965 MedAgentBench700.8900.8990.9270.930 AIME510.8730.8970.9100.942 SWE-smith790.8810.8960.9370.972 CompileBench150.9090.8940.9710.989 Seal01010.8440.8820.9300.955 Reasoning Gym5300.8280.8810.9300.931 QuixBugs800.8120.8810.9230.922 ReplicationBench790.8670.8620.9160.940 IneqMath960.8590.8540.9230.970 BFCL1040.7440.7710.7710.821 CyberGym100.7070.7450.9400.999 Within-benchmark holdout prediction. Because the full 5896-task matrix has far more columns than rows (16), cross-benchmark prediction would overfit. We therefore predict tasks within each benchmark: for each of the 53 benchmarks withā„ 5reliable tasks, we mask 50% of each row (3 folds) and blend top-kpeer correlation with rank-2 SVD. The blend improves over the task-mean 53 baseline in 41 cases (median improvement 24.3%), with the largest gains in benchmarks whose tasks test similar capabilities, such as HumanEvalFix (67.3%) and KUMO (56.7%). Representative task selection. Greedy forward selection within each benchmark identifies inde- pendently informative tasks (max|Ļ| < 0.7). The fraction of independent tasks varies widely: CyberGym (70%) and BIX-Bench (69%) contain many independent items, while HumanEvalFix and ResearchCodeBench reach theĻ > 0.7threshold within the first few selections, indicating that most of their tasks measure the same underlying capability. Within-benchmark blends out- perform cross-benchmark peers in 37/54 (69%) cases, but for the remaining 17 benchmarks cross- benchmark peers predict better, indicating shared capability axes across benchmark boundaries. F.2 Model vs. Harness Effect: Methodology To formally test whether model or harness identity explains more variance while controlling for benchmark difficulty, we fit an LMM: s ij = Ī² 0 +Ī² model x model,i +Ī² harness x harness,i + u j + Īµ ij , u j ā¼N (0,Ļ 2 u ), Īµ ij ā¼N (0,Ļ 2 e ), (7) where: ā¢ s ij : raw score for system i (modelāharness configuration) on benchmark j ā¢ Ī² 0 : intercept (expected score at reference levels) ā¢ Ī² model , Ī² harness : fixed-effect coefficients for model and harness identity ā¢ x model,i , x harness,i : dummy-coded indicators (first alphabetical level as reference) ā¢ u j ā¼N (0,Ļ 2 u ): benchmark-level random intercept (difficulty heterogeneity) ā¢ Īµ ij ā¼N (0,Ļ 2 e ): residual error ā¢ Ļ 2 u : between-benchmark variance; Ļ 2 e : within-benchmark variance Estimated via REML using statsmodels MixedLM. The intraclass correlation coefficient (ICC) measures the proportion of total variance attributable to between-benchmark differences: ICC = Ļ 2 u Ļ 2 u + Ļ 2 e = 0.75,(8) confirming that 0.75 of score variance is between benchmarksāi.e., benchmark difficulty is the dominant source of variation, justifying the random-intercept specification. Table 13 reports the estimated fixed effects. Each coefficient represents the difference from the reference level (the alphabetically first model or harness, whose effect is zero by construction). Among the 8 models, 6 coefficients are significant (p < 0.05); among the 4 harnesses, only 2. We define the effect range for a factor as the difference between its largest and smallest coefficients (including the reference at zero). The model effect range is0.235ā (ā0.216) = 0.451raw score units, while the harness effect range is0.045ā (ā0.042) = 0.087. The model range is thus 5.2Ć larger, confirming that model choice dominates harness choice across benchmarks. Table 13: LMM fixed effects (Eq. 7).ĖĻ 2 u = 0.0485,ĖĻ 2 e = 0.0161, ICC= 0.75. Significance: *p < 0.05, **p < 0.01, ***p < 0.001. FactorLevelEstimateSEzp Interceptbaseline+0.4720.03314.264.1e-46*** Modelclaude-haiku-4-5-20251001+0.000(ref.) Modelclaude-opus-4-6+0.1790.01710.354.2e-25*** Modelclaude-sonnet-4-6+0.1400.0178.144.1e-16*** Modelgemini-3-flash-preview+0.1420.0216.905.3e-12*** Modelgemini-3.1-pro-preview+0.2350.02111.432.8e-30*** Modelgpt-5-mini-0.0020.020-0.120.904 Modelgpt-5-nano-0.2160.020-10.842.2e-27*** Modelgpt-5.4+0.1290.0206.498.5e-11*** Harnessclaude-code+0.000(ref.) Harnesscodex+0.0450.0202.270.023* Harnessgemini-cli-0.0370.022-1.640.101 Harnessterminus-2-0.0420.014-2.970.003** 54 F.3 Original benchmark data collections and mapping To contextualize the current Harbor results, we collected historical scores of different model and/or agents over time, along with their timestamps, for each benchmark from the original paper, public leaderboards, and technical reports of new models. In some benchmarks, multiple evaluation metrics are reported. We manually select the metric that best aligns with our Harbor evaluation results, or transform the reported metric into a Harbor- aligned score to enable direct comparison. In particular, we apply special treatment to the follow- ing papers: ā¢CodePDE: The paper reports nRMSE as the primary metric [83], where lower is better. We convert nRMSE into a Harbor-aligned pass rate by thresholding each PDE family at nRMSE < 0.05 and averaging the resulting pass rates across the five families. ā¢AlgoTune: Raw speedup factors are mapped to the normalized Harbor score using the bench- markās log-speedup compression,log(max1,x)/(log(max1,x) + 1), wherexdenotes the speedup ratio. This transformation ensures that: (1) the score always lies in[0, 1); (2) the model receives a non-zero score only whenx > 1, i.e., when the agent-generated code is faster than the baseline; and (3) the score increases monotonically withx, so faster code leads to a higher score. In the original paper [128], the overall benchmark score is defined as the har- monic mean of acceleration ratios. We therefore collect the per-task results for each model and recompute the score using our formula, yielding a normalized score in [0, 1). ā¢ SLDBench: Raw verifier rewards can lie in[ā1, 1], so we report the display-aligned score (x + 1)/2 on the [0, 1] scale. ā¢QuixBugs: Historical results are reported as the number of repaired programs, which we con- vert to pass rate by dividing by the 40 benchmark programs. ā¢StrongReject: Public results report jailbreak/attack success, while Harbor reports defense success. We therefore invert the metric as 1ā x. ā¢KUMO: Public results are split into easy and hard settings. To match the subsampled Harbor task mixture, we compute a weighted success rate,(202Ā· accuracy easy + 10Ā· accuracy hard )/212. ā¢SciCode: Harbor uses the 80-task without-background setup and aggregates sub-step correct- ness at the problem level, so we align to the paperās standard subproblem pass@1 rather than other SciCode variants. ā¢ ResearchCodeBench: the official metric is scaled pass@1, which weights each code snippet by its number of executable lines of code (LoC). We therefore align Harbor evaluation to the same LoC-weighted aggregation, instead of averaging snippet-level success uniformly. ā¢Spider 2.0, SpreadsheetBench, SWT-Bench, USACO, and WideSearch: public scores are matched to the Harbor-evaluated split or subset when possible, rather than to the full bench- mark headline number. FinanceAgent, GAIA2, LawBench, MLGym, PIXIU, SkillsBench, and SWE-smith were excluded from direct historical comparison when no comparable public metric or split could be identified. We note that existing benchmark evaluations may not align with our modelāharness configurations: some benchmarks evaluate raw LLM performance without any agentic scaffold; some fine-tune specialized models or design task-specific agentic scaffolds, while our study evaluates models spanning capability tiers from the Gemini, Claude, and GPT families under general-purpose har- nesses (Gemini CLI, Claude Code, Codex, and Terminus-2). We present in Table 14 a benchmark- level summary of this alignment for the 54 benchmarks in our taxonomy. Overall, only 16 of the 54 benchmarks have at least one overlapping model or harness configura- tion. Therefore, these results are often not suitable for direct head-to-head comparison against our Harbor runs. Nevertheless, they remain useful for tracking progress: by taking the best reported performance at different time points, we can estimate how the state of the art has advanced on each benchmark over time, which will be discussed in Appendix F.4. 55 2017-102017-122023-082024-032024-042024-052024-062024-092024-122025-012025-022025-032025-042025-08 2025-11 Harbor Result Date 0.00 0.25 0.50 0.75 1.00 Score Software Engineering (Competitive & Function-level Coding) LiveCodeBench USACO BigCodeBench-Hard Aider Polyglot QuixBugs HumanEvalFix Harbor result 2024-062024-082024-092024-10 2024-11 2024-122025-012025-032025-042025-052025-062025-082025-09 2025-11 2025-122026-022026-032026-04 Harbor Result Date 0.00 0.25 0.50 0.75 1.00 Score Software Engineering (Repo-level Issue Resolution and Testing) SWE-bench-verified SWE-Bench- Multilingual SWE-Bench Pro SWT-Bench Harbor result 2025-022025-042025-052025-062025-072025-092025-10 2025-11 2025-122026-022026-04 Harbor Result Date 0.00 0.25 0.50 0.75 1.00 Score Software Engineering (Other) GSO AlgoTune CompileBench CRUST-Bench SWE-Lancer Harbor result Figure 8: Progress over time across Software Engineering benchmarks. Each panel tracks the best reported score on representative benchmarks within a domain as a function of result date; the rightmost diamond marks the best frontier result measured in our Harbor evaluation under a uni- fied adapter-based setup. Hollow diamonds indicate cases where the Harbor evaluation set is not identical to the original benchmark, typically because Harbor evaluates a subset, so the connected segment should be interpreted as a reference comparison rather than a strict apples-to-apples continuation. For LiveCodeBench, the historical evaluation results are based on live evaluation, while Harbor uses the fixed LiveCodeBench (V6). F.4 Performance Progress Over Time As described in Appendix F.3, we have collected historical scores of different model and agent configurations over time. In Figures 8 and 9, we present the best reported performance for each benchmark over time. While this view should only be interpreted as an observational record of community progress, the density and slope of these curves reveal where the community has con- centrated its benchmarking effort, and where progress has been most visible. A clear pattern is that Mathematics & Reasoning and Software Engineering have attracted the most sustained leaderboard pressure. These domains contain many closely tracked benchmarks, frequent leaderboard updates, and rapid replacement of the previous state of the art. In Mathematics & Reasoning, scores on several benchmarks rise quickly toward saturation, suggesting that once a benchmark becomes widely adopted, frontier labs and open-source systems rapidly optimize against it ā potentially because these tasks have clear, verifiable evaluation protocols. Software Engineering shows an even more heterogeneous version of the same trend: function-level and competitive-programming-style benchmarks often improve rapidly and approach saturation, while several broader engineering tasks remain substantially more challenging. This suggests that the apparent progress in ācodingā is not uniform; it is stronger on well-specified, unit-testable tasks, 56 2024-102025-022025-042025-052025-062025-072025-082025-10 2025-11 2025-122026-02 Harbor Result Date 0.00 0.25 0.50 0.75 1.00 Score Mathematics & Reasoning ARC-AGI-2 AIME 24&25 IneqMath KUMO Omni-MATH Reasoning Gym Harbor result 2023-11 2024-092024-10 2024-11 2024-122025-012025-042025-062025-07 2025-11 2026-022026-032026-04 Harbor Result Date 0.00 0.25 0.50 0.75 1.00 Score Knowledge & Long Context MMMLU SimpleQA GPQA Diamond Humanity's Last Exam Harbor result 2025-022025-062025-072025-082025-10 2025-11 2026-012026-022026-032026-04 Harbor Result Date 0.00 0.25 0.50 0.75 1.00 Score Agents, Tools & Systems Terminal-Bench 2.0 Seal-0 Berkeley Function Calling Leaderboard (BFCL) DeepSynth GAIA WideSearch Harbor result 2024-072024-092025-022025-052025-072025-102025-12 Harbor Result Date 0.00 0.25 0.50 0.75 1.00 Score Scientific Research SciCode BIX-Bench CodePDE LAB-Bench QCircuitBench ReplicationBench SLDBench Harbor result Figure 9: Progress over time across Mathematics & Reasoning, Knowledge & Long Context, Agents, Tools & Systems, and Scientific Research benchmarks. Each panel tracks the best reported score on representative benchmarks within a domain as a function of result date; the rightmost diamond marks the best frontier result measured in our Harbor evaluation under a uni- fied adapter-based setup. Hollow diamonds indicate cases where the Harbor evaluation set is not identical to the original benchmark, typically because Harbor evaluates a subset, so the connected segment should be interpreted as a reference comparison rather than a strict apples-to-apples continuation. and weaker on open-ended tasks requiring repository understanding, environment management, and long-horizon debugging. By contrast, progress is less dense and often slower in domains such as Knowledge & Long Con- text and Agents, Tools & Systems. These areas have fewer historical points, fewer continuously maintained leaderboards, and less frequent apples-to-apples reporting. The resulting curves are therefore sparser, but this sparsity is itself informative: it indicates that these capabilities have not yet received the same level of repeated public measurement as math and software engineering. In Knowledge & Long Context, many benchmarks are expensive to evaluate, sensitive to retrieval or 57 context-window assumptions, and harder to compare across model generations. In Agents, Tools & Systems, evaluation often depends on changing external environments, tool APIs, and multi-step execution protocols, making leaderboard progress more difficult than for static math or coding tasks. These differences highlight an important source of bias in how progress is perceived. Domains with dense public leaderboards create a feedback loop: frequent measurement encourages opti- mization, which produces rapid score gains, which in turn attracts more attention. Conversely, sparse domains may appear to progress more slowly partly because they are harder to evaluate consistently and partly because fewer systems are repeatedly measured there. The fastest-moving curves identify areas where the field has built strong benchmarking infrastructure, while flatter or sparser curves point to domains where better standardized evaluation may be a prerequisite for faster scientific progress. Through our infrastructure efforts in this work, we hope that agentic evaluation can become less dependent on a small number of highly visible leaderboards and more evenly distributed across capability domains. For several benchmarks, the best results reported externally are higher than our best results mea- sured under Harbor, as shown in Table 15. These gaps arise because the external results were obtained via specialized models (e.g., on BFCL), specialized agent scaffolds (e.g., on MedAgent- Bench, SLDBench, and AlgoTune), and larger compute budgets for agents (e.g., on CodePDE). By contrast, Harbor evaluates a fixed set of models spanning capability tiers under a unified adapter- based setup, with the goal of improving cross-benchmark comparability rather than maximizing performance on each individual benchmark. F.5 Latency Study 0.00.10.20.30.40.50.60.70.80.9 Task difficulty (1 ā mean pass rate) 100s 200s 300s 400s 500s 600s 700s Avg agent exec time per successful trial 0.10.20.30.40.50.60.70.80.91.0 Task difficulty (1 ā mean pass rate) 100s 200s 300s 400s 500s 600s 700s Avg agent exec time per failed trial Claude Opus 4.6 GPT 5.4 Gemini 3.1 Pro Claude Sonnet 4.6 GPT 5 Mini Gemini 3 Flash Claude Haiku 4.5 GPT 5 Nano Figure 10: Left: Average agent execution time per successful trial. Right: Average agent execu- tion time per failed trial. Per-model breakdown by task difficulty. The 0.9ā1.0 bin is omitted from the success panel (most models have fewer than 25 successful trials) and the 0.0ā0.1 bin from the failure panel (four models have fewer than 25 failed trials). Agent execution latency, the wall-clock time a model spends actively working on a task, reflects inference time, tool calls, environment interaction, and retry behavior. Using the same difficulty buckets as in Section 3.3, we disaggregate latency by trial outcome to ask: do all models spend time the same way when they succeed versus when they fail? Figure 10 plots per-model execution time on successful trials (left) and failed trials (right). On the success side, models follow a broadly similar upward trend with difficulty, but separate into a high- 58 latency group (Claude Sonnet 4.6, Claude Opus 4.6) and a low-latency group (GPT-5.4, Claude Haiku 4.5, GPT-5-Mini, Gemini 3 Flash), with the gap widening on harder tasks. The failure side reveals two findings invisible in the success panel. First, there is a shared structural pattern across all models that failure latency peaks around difficulty 0.7ā0.8 and then declines on the hardest tasks, the opposite of the success trend. This suggests that the hardest tasks trigger earlier termination, while tasks in the 0.7ā0.8 range are plausible enough to sustain extended, ultimately unproductive attempts. Second, the relationship between success and failure latency is not consistent across models. GPT-5.4 terminates in similar time regardless of outcome, whereas GPT-5-Nano, despite being the weakest model, persists longest on failed trials. G Case Analysis G.1 Task Failure The task-failure examples below illustrate the kinds of broken designs identified during Harbor- Index human review (Section 4). Each category appears across multiple benchmarks in the review set. Broken environments. All four GAIA2 tasks that reached human review share the same defect: the adaptive simulation event needed to trigger the grading cascade never fires. One Claude-code trajectory ran for 270 steps, spent 36 hours but received onlyānotificationsā: []through- out. The verifier waits for the simulation to advance in response to agent actions, while the adapter never advances it. Every model and harness fails for the same reason, ruling out agent capability as the cause. On MMAU, the Harbor adapter discards the audio payload before handing each task to the agent. Agents work from a text transcript and are graded as though they heard the audio, so tasks requir- ing acoustic judgement (emotion, speaker identity) cannot be solved regardless of model strength. Instruction-verification mismatch. A task only has a clear pass condition when the instruc- tion and the verifier agree on what success looks like. When they do not, an agent can produce a defensible solution and still score zero. We see this pattern in several forms across the review set. In one SWEBENCH-PRO Ansible task, 162 of 168 tests pass for most agents. The six that fail re- quire a specific dictionary format absent from the problem description. A TELEPORT task adds a new test file (Ć„TestAuditWriter/Backoff) in the gold patch that is absent from the starting repository. All eleven capable agents implement the backoff mechanism correctly per the writ- ten specification, but the verifierās restore command does not include the test file the gold patch introduced. In other cases the verifier holds a standard the instruction never states. A HUMANITYāS LAST EXAM task was rejected because the judge carries an internal reference answer that disagrees with the mathematically correct value; across all 18 trials, the judge rejected the correct answer with 152ā280 completion tokens of active reasoning per run. An A-LCR task rejects ā30.4 percentage pointsā while accepting ā0.304,ā applying a unit-representation rule that does not appear in the instruction. The mismatch can also originate on the instruction side. A SWEBENCH-MULTILINGUAL task gives agents a raw GitHub bug report (Describe the bug, To Reproduce, Expected behavior, Your Configuration) but no directive to fix anything. Agents have to guess from the report format that a fix is expected. An AIDER-POLYGLOT JavaScript task shows the matrix transpose with newline- separated strings, which suggests a string contract, but the hidden verifier expects arrays. No test file is present in the working tree, so agents cannot discover the discrepancy before submitting. Gameable tests. A FEATUREBENCH-MODAL task produces byte-identical failure output (53 fails / 1 pass) across 14 runs from 5 distinct harness and model combinations. The test file imports the implementation under a specific symbol name and module path the instruction does not disclose. Every agent that uses a different but functionally correct structure fails in the same way. The task is measuring whether an agent guesses the right import path, not whether it can implement the feature. 59 Wrong gold answer. Several A-LCR tasks were rejected because the gold answer is simply wrong. In each case, all 16 evaluated configurations spanning 4 distinct harness implementations independently reached the same alternative value, and manual inspection confirmed the agent answer is correct. When every frontier model agrees and the gold disagrees, the gold needs to be checked. G.2 Agent Failure Modes G.2.1 Methodology The agent failure-mode taxonomy is bootstrapped bottom-up rather than authored top-down. We draw a stratified sample of failed trajectories spanning every benchmark, model, and harness in our pool, and dispatch a Claude-4.7-Opus subagent to each trajectory in parallel. Each subagent reads the taskās instruction, the verifier output, and the agentās full ATIF trajectory, and emits a free-form, single-sentence description of the failure root causeāe.g., ādeleted a previously-passing test rather than fixing the bug,ā āterminated declaring success without running the verifier,ā āfailed to install a benchmark-specific dependency.ā The free-form descriptions are then clustered into a small set of candidate failure modes, and the authors review and merge near-duplicates, yielding16 candidate rubrics. Each rubric carries a short definition and several anchor examples drawn from the bootstrap descriptions, so that downstream human annotators and an LLM judge can apply the same definitions. Below we describe the human-annotation protocol, the gold-label adjudication rule, and the LLM-judge calibration that follows. Human annotation. A held-out sample of182failed trials, stratified across benchmark, model, and harness, is annotated independently by two non-overlapping panels of three annotators each (Round 1 and Round 2). Each annotator works in Docent 2 on a per-trial UI: they read the same task instruction, verifier output, and rendered ATIF trajectory the LLM judge will see, and apply each of the18candidate rubrics independently with a binary verdict (MATCH / NO MATCH) and a free-text explanation citing concrete trajectory evidence. We measure inter-annotator agreement by comparing the two rounds. Inter-rater agreement. CohenāsĪŗbetween Round 1 and Round 2 is reported per rubric in Ta- ble 16. The pooledĪŗacross all18rubrics is0.66, and twelve of the eighteen rubrics individually clear theĪŗā„ 0.5āmoderate-or-betterā threshold. The remaining six ā Calculation Mismatch (R1 over-applies it as a numerical-Q&A catch-all), Rule Inference Error (low base rate, R2 over-applies on ARC-style tasks), SpecificationāVerification Mismatch (R2 over-applies on under-specified tasks), and three rubrics that R1 never used at all (Insufficient Web Research, Premature Termina- tion, Wrong Output Location) ā are dropped from headline reporting and folded into an āOthersā bucket. Gold-label adjudication. We then take the union of the two human annotations to generate the final gold label for calibrating the LLM judge. For each(trial, rubric)cell, the gold label is MATCH iff at least one Round-1 or Round-2 annotator selects a MATCH. This union rule yields a recall-leaning gold set: a single annotator who can quote concrete trajectory evidence (a verifier- output snippet, a tool-call message number, a quoted line of agent code) is sufficient to flag a cell. LLM-judge calibration. The judge is a single multi-rubric shortlist prompt covering the12 trustworthy rubrics: each call shows the model the task context (instruction, verifier code, ora- cle solution, environment Dockerfile, verifier stdout), the full rendered ATIF trajectory, and all 12rubric definitions, and asks the model to shortlist0ā3rubrics that fire with one citation per fire. We tune the rubric texts (not the prompt template) iteratively against the gold set, alternat- ing broaden/tighten edits per rubric. We test two judge models (gemini-3-flash-previewand gemini-3.1-pro-preview), and use the stronger Pro model after 14 rounds of prompt tuning. A few patterns from 14 tuning rounds: ā¢ One-rubric-at-a-time edits compose; multi-rubric edits do not. Tightening Rubric Aās exclu- sions reliably shifts borderline cells away from A, but the same edit will typically over-correct 2 Docent is a system for inspecting and annotating model trajectories:https://transluce.org/introducing-doc ent. 60 on a related rubric (e.g. tightening False Success Claim drains positives into Unverified Claim) when applied alongside an unrelated change to a third rubric. ā¢Per-rubric sub-judges (one focused yes/no call per rubric per trial,12Ćthe calls) under- perform the multi-rubric SHORTLIST prompt. The competition between rubrics for the short- list slot is helpful: it forces the model to commit to a positive verdict rather than hedging NO MATCH on every rubric independently. ā¢ Pro responds better to short Docent-base rubric text than to elaborate priority-firing language we wrote in some iterations. For Agent Timeout, dropping the override entirely (so Pro reads the original200-token rubric text) yieldsĪŗ = 0.48; an elaborate override that explicitly reranks AT above content rubrics yieldsĪŗ = 0.0(the model never picks AT). Pro is conservative by default, and verbose ādo this even if youād otherwise pick Xā instructions tend to confuse the model rather than improving rubric selection. Calibration results. The final chosen configuration achieves pooledĪŗ = 0.51against the 182-trial gold set, vs. the humanāhuman R1āR2 ceiling of0.66on the same trials. Six rubrics clear theĪŗ ā„ 0.5trustworthy threshold individually; the remaining six less confident rubrics are grouped into the āOthersā bucket. Per-rubric numbers are given in Table 17. The meanĪŗacross the six reported rubrics is0.61, essentially matching the humanāhuman ceil- ing on the same six rubrics (0.71): on the rubrics where the judge clears the trustworthy bar, its calls are at human-rater quality, and the gap between0.51pooled and the0.66human ceiling is concentrated in the Others bucket ā particularly Unverified Claim (Pro under-fires by an order of magnitude), Wrong Output Schema (Pro under-fires when key-name mismatches are involved), and Wrong Target / Layer (low base rate, n = 2 in the gold sample). Audit at scale. The calibrated judge is then applied to all5,898failed trials in the full hard-task set. We retry parse-failure trials up to three times with bumpedmax_tokens, achieving98.4% parsed coverage; the remaining1.6%are mostly trials whose rendered transcript exceeds the gate- wayās per-request token cap and are dropped from headline numbers. Rubric definitions. The full set of18candidate rubrics, with the definitions presented to human annotators and used in the LLM judge prompts, is given below. We mark each rubric as TRUST- WORTHY (passes inter-annotator Īŗā„ 0.5) or EXCLUDED (folded into āOthersā). Trustworthy rubrics (Īŗā„ 0.5, reported individually). Wrong Target / Layer.TRUSTWORTHY Framing. Agent applied a change to a target / abstraction layer / module / code path where the change cannot produce the behavior the grader checks. Concrete patterns. ā¢ Fix at global-stylesheet layer when the component uses inline / CSS-in-JS styles ā¢ Test that asserts file-layout when the task required a behavioral test ā¢ Patch to a codepath the failing test does not exercise ā¢ Shallow symptom fix when root cause is deeper in state/logic ā¢ Fix in a deprecated / unused module Decision procedure. 1. Identify the taskās intended target. ā¢ If no target location inferrable: NO MATCH. 2. Identify where the agent actually applied changes. 3. MATCH if (a) a change was applied AND (b) the applied change is structurally incapable of affecting the graded behavior. Exclusions. 61 ā¢ Right file but logic wrong: NO MATCH (Algorithmic Bug). ā¢ Minimal symptom fix catching some cases: NO MATCH (Algorithmic Bug). Syntax / Language Error.TRUSTWORTHY Framing. Agent produced code / DSL artefact that doesnāt parse or uses invalid domain constructs ā Python SyntaxError, OpenQASM unsupported gates, PDDL malformed, Java NPE at load time, Go wonāt compile, YAML rejected. Decision procedure. 1. Identify the domain language. 2. Look for compile / parse / load errors in tool outputs. 3. MATCH if the artefact fails at language-level. Exclusions. ā¢ Code compiled but logic wrong: NO MATCH (Algorithmic Bug). ā¢ Runtime exception from execution (not parse-time): NO MATCH unless the exception prevents any test from running. Algorithmic Bug.TRUSTWORTHY Framing. On a code-writing or bug-fixing task, the agentās overall APPROACH IS DEFENSIBLE but the implementation contains a SPECIFIC, NAMEABLE DEFECT ā wrong axis, off-by-one, parameter swap, missed edge case, flawed API usage, wrong control-flow branch. You must be able to point to the bug. Decision procedure. 1.Confirm the task is code-writing or bug-fixing (not factual Q&A, not Calc, not numerical- only). 2.Confirm runnable code WAS produced and executed without an unhandled crash (no SyntaxEr- ror, no ImportError, no environment failure). 3. Confirm reward=0. 4.POINT TO THE BUG. You must quote either: If you cannot quote a SPECIFIC defect ā output āno matchā. ā¢The buggy line in the agentās code AND describe the specific defect (e.g., āused i+1 instead of i-1 on line 42ā), OR ā¢A specific failing test output that pinpoints the logical error (e.g., āexpected 5 got 4 on input [1,2,3,4]ā). Exclusions. ā¢ Agent took the WRONG OVERALL APPROACH (e.g., downloaded pre-trained weights when the task said to fine-tune; implemented a feature in the wrong file; chose the wrong abstraction): NO MATCH (Wrong Target / Layer). ā¢ Code did not parse / compile: NO MATCH (Syntax / Language Error). ā¢ Sample tests passed but hidden tests failed: NO MATCH (Hidden-Test Regression). ā¢ Numerical result outside tolerance: NO MATCH (Calculation Mismatch). ā¢No code emitted (only narrative / plan): NO MATCH (Plan Over Implementation / Silent Deliv- erable). ā¢ Crash from missing dependency / broken environment: NO MATCH (Environment Block). ā¢ Verifierās spec doesnāt match the prompt: NO MATCH (SpecificationāVerification Mismatch). ā¢ Wrong file/path/abstraction: NO MATCH (Wrong Target / Layer). ā¢ Format / shape only: NO MATCH (Wrong Output Schema). 62 ā¢Failure on a short-answer Q&A task: NO MATCH (Wrong Factual Answer). If you would write āthe agentās solution must have a bug because reward=0ā ā that is NOT enough. Output āno matchā. You need to identify WHICH bug. This rubric is HIGH-PRECISION: prefer āno matchā unless you can name and quote the specific defect. Wrong Factual Answer.TRUSTWORTHY Framing. The task asks for a SHORT, LITERAL ANSWER (a single value, a year, a name, an MCQ letter, a number with a fixed unit, a yes/no) and the agent emitted that answer in the required slot, but the value is wrong by a plain reading of the prompt. Decision procedure. 1. Confirm the deliverable is a short literal answer ā not extended code, not a multi-step calcula- tion pipeline that itself can have algorithmic bugs. 2.Confirm the agent emitted a literal answer in the required slot (file, message, function call) and that the answer is concretely wrong (you can quote the agentās value AND the verifier- expected value). 3.MATCH only when steps 1 + 2 hold AND the failure isnāt better explained by a more-specific rubric below. Exclusions. ā¢Numerical answer outside tolerance from a calculation pipeline: NO MATCH (Calculation Mismatch). ā¢Code-writing task where reward=0 because of a bug in the code: NO MATCH (Algorithmic Bug). ā¢ Format/shape mismatch (right content, wrong shape): NO MATCH (Wrong Output Schema). ā¢ Wrong because of a rule-inference mistake on a pattern task: NO MATCH (Rule Inference Error). ā¢ Missing answer / no deliverable: NO MATCH (Silent Deliverable). ā¢Agent could not verify against an available checker: NO MATCH (Unverified Claim). This rubric COVERS short-answer Q&A, MCQ, simple lookups, even when the underlying reason- ing is complex ā as long as the deliverable is just the literal answer and the answer is wrong. Wrong Output Schema.TRUSTWORTHY Framing. The agentās content is at least partly correct, but the FORMAT / SHAPE / ENCODING / KEY-NAMES are rejected by the verifier. This rubric covers a wide range: JSON shape mismatch, wrong scalar type (e.g. 4.0 vs 4, ā0ā vs āzeroā), expected string got number, missing required key, extra prose around the answer, wrong delimiters, markdown vs plain text, list-vs-dict, and key- name mismatches (e.g. snake_case vs camelCase) where the verifier checks the literal key. Firing trigger. ā¢ The verifier output contains shape/type/encoding error patterns: āexpected <type> got <type>ā, āKeyErrorā, āinvalid JSONā, āexpected list got dictā, āAttributeError on dict.<method>ā, āJSON decode errorā, schema-validation messages. ā¢ The verifier complains the agentās output cannot be parsed. ā¢The verifier expected a specific key/name that the agent used a different spelling/case for, where the underlying values are recognizable. ā¢The agent emitted markdown / prose where a bare value was expected. Output MATCH if ANY of the above patterns appears AND the agentās content is at least partly recognizable. Decision procedure. 1. Inspect verifier_output for shape/type/encoding/key error patterns. 2. Confirm the agentās content is at least partly correct ā the right answer or recognizable pieces are present, just in the wrong shape. 63 3. Quote the specific divergence (which field, what shape). 4. MATCH if 1+2+3 hold. Exclusions. ā¢Content is wholly wrong, formatting is fine: NO MATCH (Algorithmic Bug / Wrong Factual Answer). ā¢ Content correct, location wrong: NO MATCH (Wrong Output Location). ā¢ No content produced at all: NO MATCH (Silent Deliverable). Hidden-Test Regression.TRUSTWORTHY Framing. The agent ran SCAFFOLD-PROVIDED test infrastructure or correctness checks (sam- ple tests, the repoās own test suite, a check script that ships with the task, a make-target the task points at, a CI command, a known oracle command) on its deliverable and they PASSED ā but the verifierās hidden / fuller test suite still rejects with reward=0. Key signal: the agent ran something that came WITH the task (not something the agent wrote / fabricated) and saw a positive result. FIRING TRIGGER (high-recall): trajectory contains BOTH: (a) An invocation of a scaffold- shipped test/check (pytest tests/, make test, gradlew test, npm test, repo-provided sample inputs, a check script in the workspace), AND (b) A positive result from that invocation (the agent quoted āpassedā, āOKā, ā0 failedā, ātests passedā, or similar). If both fire and reward=0, output MATCH unless a strong exclusion applies. Decision procedure. 1. Quote the scaffold-provided test/check the agent ran (with msg number). 2. Quote the positive output the agent saw. 3. Confirm reward=0. 4.Confirm the test was scaffold-shipped, NOT agent-fabricated. Ad-hoc inline assertions the agent wrote do NOT qualify (those are False Success Claim). 5. MATCH when 1+2+3+4 hold. Exclusions. ā¢ Agent ran its own ad-hoc check (an inlineassert ...it crafted, a smoke-test script, a man- ual eyeballing): NO MATCH (False Success Claim). ā¢ Agent never ran any test: NO MATCH (Unverified Claim). ā¢ Agentās tests failed and it ignored them: NO MATCH. ā¢Hidden tests enforce something genuinely absent from the prompt: NO MATCH (Specifica- tionāVerification Mismatch). ā¢Crash / syntax error: NO MATCH (Syntax / Language Error). This rubric is HIGH-RECALL: when you see scaffold-provided tests + positive result + reward=0, prefer āmatchā over the broader False Success Claim or Unverified Claim. The signal is most specific. False Success Claim.TRUSTWORTHY Framing. The agent SAW POSITIVE EVIDENCE for its deliverable (a passing assertion, a matching expected output, an analysis it concluded was correct, a successful smoke-test) and DECLARED THE TASK DONE ā but the verifier disagrees with reward=0. Firing trigger. (a) The trajectory contains an agent-generated positive correctness signal: a quoted phrase like āpassedā, āOKā, āmatches expectedā, āall tests passā, ācorrectā, ālooks goodā, āsanity check passesā, or an inline assertion that returned True / didnāt raise. (b) The agent declared com- pletion (final ātask doneā, āfinishā, āsubmitā, or stopped confidently after writing the deliverable). (c) Reward = 0. 64 If all three fire, output MATCH unless a strong exclusion applies. Note on the FSC vs HTR boundary: when the agent ran scaffold-shipped tests that passed, prefer Hidden-Test Regression (more specific). When the agentās positive signal came from its own ad- hoc check (an inline assertion / one-off script / manual eyeballing), prefer FSC. Decision procedure. 1.Find the agent-generated positive correctness signal in the trajectory and quote it (with msg number). 2. Confirm completion declaration (or implicit stop) and reward = 0. 3.If the positive signal came from scaffold-shipped tests (pytest, make test, check script that ships with the task), this is HTR territory ā output NO MATCH. 4. Otherwise output MATCH. Exclusions. ā¢ Agentās check returned negative and agent declared done anyway: NO MATCH. ā¢ Agent ran NO check at all: NO MATCH (Unverified Claim). ā¢ Lint/syntax/existence checks only: NO MATCH (Unverified Claim). ā¢ Agent admitted incompleteness: NO MATCH. Unverified Claim.TRUSTWORTHY Framing. ā¢ The agent declared the task done WITHOUT running a substantive verification of its deliver- able when one was available. Unverified Claim is a RESIDUAL rubric ā it applies when the agent skipped verification AND no more-specific failure mode explains the verifierās rejection. PRIORITY ORDER ā check more-specific rubrics FIRST, only fire UC if none of these apply to the failure: ā¢ Verifier output is a syntax / compilation errorā Syntax / Language Error, NOT UC. ā¢ Verifier output is a shape / type / schema errorā Wrong Output Schema, NOT UC. ā¢ Verifier output is a missing-file / no-deliverable errorā Silent Deliverable, NOT UC. ā¢Verifier output names a specific failing test / off-by-one / wrong axisāAlgorithmic Bug, NOT UC. ā¢ Trial timed outā Agent Timeout, NOT UC. ā¢ Crash from missing dep / broken envā Environment Block, NOT UC. ā¢Agent ran a substantive check that returned positiveāFalse Success Claim or Hidden-Test Regression, NOT UC. ā¢Agent emitted code in the wrong file / abstractionāWrong Target / Layer, NOT UC. Only after ruling out the above do you consider UC. Firing trigger. ā¢ The agentās deliverable is a defensible attempt (not silently missing, not syntactically broken, not blatantly wrong format) AND the agent did NOT run a substantive verification before declaring complete AND a verification mechanism was available (sample tests, oracle script, the repoās pytest, or simply running the deliverable end-to-end). Decision procedure. 1. Confirm none of the priority-rubrics above explain the failure. 2. Confirm the agent declared complete without running substantive verification. 3.Confirm a substantive verification was AVAILABLE (the task had tests, an oracle, or an end-to- end runnable check). 65 4. MATCH if 1+2+3 hold. Exclusions. ā¢ A more-specific rubric explains the failure: NO MATCH (use that rubric). ā¢ Agent ran a substantive check successfully: NO MATCH (FSC or HTR). ā¢ Agent admitted incompleteness: NO MATCH. ā¢ Crashed/timed out before reaching verification: NO MATCH. ā¢ Trial is short literal Q&A with no programmatic checker: NO MATCH. Plan Over Implementation.TRUSTWORTHY Framing. The agent narrates a CONCRETE NEXT ACTION ("Iāl now write the file", "Let me modify X to do Y") OR an analysis/plan, and then the trajectory ends WITHOUT executing that action ā no tool call, no file write, no patch applied. The required deliverable was never produced because the agent ran out of steps mid-plan. Distinguishes from Silent Deliverable: SD = agent acted as if task was complete despite no artifact; PoI = agent stopped mid-plan, often with the next step explicitly described. Decision procedure. 1. Identify the required deliverable. 2.Search the trajectory for tool calls that produced any version of the deliverable. If found: NO MATCH. 3.Inspect the agentās final message(s). MATCH if the agent describes a concrete next step ("Iāl write...", "now let me modify...", "the next thing is...") OR finishes mid-analysis without ever executing the implementation step. Exclusions. ā¢ Deliverable was produced (even partially): NO MATCH. ā¢ Agent declared ātask completeā or similar without an artifact: NO MATCH (Silent Deliverable). ā¢ Agent admitted incompleteness, asked for help, or hit explicit timeout: NO MATCH. ā¢ Crash before reaching implementation phase: NO MATCH. Silent Deliverable.TRUSTWORTHY Framing. The task asked for a CONCRETE ARTIFACT at a specific location (a file, a function definition, a saved answer, a written patch) and NO EVIDENCE that the artifact was produced exists in the trajectory. The agent did not deliver. PRIORITY EXCLUSION (check FIRST, before the firing triggers below): if the agentās FINAL message describes a concrete next action that was not executed ("Iāl now write the file", "now let me modify X", "the next step is to write...") AND the trajectory ends mid-step, OUTPUT āno matchā here ā that case is Plan Over Implementation, NOT Silent Deliverable. Silent Deliverable fires when the agent treats the task as done despite no artifact; Plan Over Implementation fires when the agent stops mid-narrative without finishing. Firing trigger. ā¢ Verifier output explicitly reports the deliverable is missing / empty: phrases like āNo <file> foundā, ā<file> does not existā, āfile not foundā, āno answer.txtā, āmissing outputā, āoutput is emptyā, a PythonFileNotFoundError/KeyErroron the deliverable path, or any verifier message indicating the artifact wasnāt there. ā¢ Reward is null / missing AND no deliverable-producing tool call appears in the trajectory. ā¢The agentās trajectory lacks any tool call that would have written the deliverable (no Edit, no Write, no apply_patch, no bash redirect to the required path) AND the agent treated the task as complete. 66 Decision procedure. 1.Identify the required deliverable artifact and its location from the task instruction or verifier code. 2.Check the verifier output for an explicit missing-file complaint (any of the trigger phrases above). If presentā output MATCH. 3.Otherwise, search every tool call (file write, edit, apply_patch, bash output, function call) for evidence the artifact was produced. If no such evidence exists AND the agent treated the task as completeā output MATCH. Exclusions. ā¢ Deliverable IS produced (even partially or wrongly) somewhere in the trajectory: NO MATCH (use a content rubric). ā¢Agent narrated a concrete next write step but trajectory was cut off mid-action: NO MATCH (Plan Over Implementation). ā¢ Agent admitted the work was incomplete or asked for help: NO MATCH. ā¢ Hard timeout/crash before any deliverable could be produced: NO MATCH. When in doubt ā and any of the strong firing triggers fired ā lean MATCH. This rubric is HIGH-RECALL on those signals; do not let broader rubrics (Algorithmic Bug, Plan Over Implementation) win when the artifact is genuinely missing. Agent Timeout.TRUSTWORTHY Framing. The trial ended because the agent hit its wall-clock or step-count budget. Agent Timeout MAY co-occur with content rubrics (Algorithmic Bug, Wrong Factual Answer, Wrong Output Schema) when the agentās partial output also has issues ā in those cases, BOTH rubrics should be shortlisted, not just one. Firing trigger. ā¢Verifier output / error fields containAgentTimeoutError,TimeoutError, āagent execution timed outā, āagent timeout exceededā, āhit Ns timeoutā, ātimed out after N secondsā. ā¢ Trial runtime is within ~5% of the agentās timeout budget. ā¢The agentās final message is mid-action ("Iāl now write...", "let me run...", "next Iāl...") and the trajectory was cut off without execution. ā¢ error_type / terminate_reason indicates timeout / wall-clock / cut-off. ā¢ exception_info/ traceback mentions timeout-related exceptions. When ANY trigger fires, output MATCH for Agent Timeout. Note: this can co-occur with a content rubric ā if the agentās partial output ALSO has a clear bug or wrong answer, shortlist BOTH (Agent Timeout AND the content rubric). The SHORTLIST allows up to 3 rubrics; co-occurring AT + content is common. Decision procedure. 1. Inspect verifier_output / error fields / exception_info / final agent message / runtime data for ANY timeout indicator. 2. If found, output MATCH for AT. Quote the timeout signal. 3. If youād ALSO have fired a content rubric (AB, WFA, WOS) on the partial output, you may shortlist that one as a co-rubric ā but AT must be in the shortlist. Exclusions. ā¢ Trial finished naturally (agent declared done, no timeout signal): NO MATCH. ā¢ Crash from missing dep / broken env: NO MATCH (Environment Block ā different failure mode). ā¢ Verifier timed out, not the agent: NO MATCH. 67 Environment Block.TRUSTWORTHY Framing. Agent attempted its work in good faith but hit a real environment constraint ā missing data, blocked network, broken dependency, OR a required capability that the task scaffold / adapter did not install. Includes adapter-induced under-tooling: e.g., a figure-QA task where the adapter ships raw PNGs but no OCR / vision library. Decision procedure. 1. Scan for environment-failure signals: ā¢ "network is unreachable", "Connection refused", "403 Forbidden", "404 Not Found" ā¢ "package not found", "pip install failed", "ImportError: No module" ā¢ Missing file / table / input / credential ā¢Agent tries vision/ASR/OCR/search tools that arenāt installed (tesseract, pillow, cv2, whis- per, curl) ā¢ dbt source tables / DB schemas not present 2.Check that the agent attempted to use a legitimate means of accomplishing the task and got blocked ā not merely skipped a step it could have done. 3. MATCH if (a) a clear environment constraint documented AND (b) agent didnāt work around it AND (c) no reasonable workaround was available in the scaffold. Exclusions. ā¢ The "environment issue" is actually agent misuse (wrong command, wrong path): NO MATCH. ā¢ Agent worked around the constraint, failed for other reasons: NO MATCH. ā¢ Agent timed out trying to resolve: NO MATCH (Agent Timeout). Excluded rubrics (Īŗ < 0.5, folded into āOthersā). Calculation Mismatch.EXCLUDED Framing. Tasks asking for a numerical result where the agent ran a pipeline to completion and produced a concrete number, but the value is outside the graderās tolerance. Decision procedure. 1. Task asks for numerical / statistical result. 2. Agent ran a real pipeline: loaded data, applied a method, computed a number. 3. Agent wrote the resulting number to the required location. 4. reward=0, and failure is not a code crash / timeout / missing file. 5. MATCH if 1-4 hold. Exclusions. ā¢ Agent guessed without a pipeline: NO MATCH (Wrong Factual Answer). ā¢ Pipeline crashed: NO MATCH (Syntax / Algorithmic). ā¢ Task required symbolic, agent gave numerical: NO MATCH. Rule Inference Error.EXCLUDED Framing. The task tests the agentās ability to infer / apply a rule, scheme, mapping, or pattern from given examples. The agent demonstrably misunderstood or mis-extrapolated the rule itself. This is a content failure of REASONING, not coding, not retrieval, not formatting. Decision procedure. 1. Confirm the task is rule-inference style: visual puzzle, IQ-puzzle, pattern-completion, instruction-following with explicit examples-then-apply structure. 68 ā¢If the task is straightforward Q&A without explicit examples to generalize from: NO MATCH (Wrong Factual Answer). ā¢ If the task is code-writing: NO MATCH. 2. Quote the specific rule the agent stated or applied, and the specific way it diverges from the correct rule (visible from the verifier output or oracle solution). 3.MATCH only when steps 1 + 2 hold and the failure is in rule inference / application ā not other modes. Default: when in doubt, output āno matchā. SpecificationāVerification Mismatch.EXCLUDED Framing. The agentās deliverable is a DEFENSIBLE READING of the user-facing instruction, but the verifier rejects it because the verifier enforces a constraint NOT stated (or genuinely under- specified) in the instruction. Examples: verifier requires a specific function name not in the prompt; verifier requires a hardcoded path not in the prompt; verifier insists on the exact prose phrasing the prompt only suggested. Decision procedure. 1. Locate the user-facing instruction (instruction.md / first user message in <task_context>). 2. Locate the verifierās actual check (verifier_code in <task_context> / verifier_output / test_sh). 3. Identify the SPECIFIC requirement the verifier enforced that caused the failure. Quote the exact failing assertion / expected value. 4.Compare against the instruction. MATCH only if BOTH: (a) The agentās deliverable reasonably satisfies the instruction as a competent reader would understand it, AND (b) The verifierās failing requirement is genuinely absent or contradictory in the instruction. Exclusions. ā¢ The verifierās requirement IS present in the instruction (even if obscure): NO MATCH. ā¢ The agentās output is wrong by any plausible reading of the instruction: NO MATCH (use the appropriate content rubric). ā¢ Agent didnāt produce the deliverable at all: NO MATCH. ā¢ Mismatch is purely formatting: NO MATCH (Wrong Output Schema / Wrong Output Location). ā¢ Mismatch is about hidden tests probing behavior the prompt described: NO MATCH (Hidden- Test Regression). ā¢You cannot quote a specific verifier requirement absent from the instruction: NO MATCH. This rubric is for genuine UNDER-SPECIFICATION cases. Do not output āmatchā merely because the agent failed and the verifier was strict. Default: when in doubt, output āno matchā. Insufficient Web Research.EXCLUDED Framing. Insufficient research applies to open-web / factual QA tasks where the agent SHOULD HAVE retrieved external information. The failure is that the retrieval was either absent when re- trieval was feasible, single-sourced when corroboration was needed, or through a wrong channel. Decision procedure. 1. Confirm the task required web research. ā¢ Internal tasks (code / math / local data): NO MATCH. 2.Check whether retrieval was FEASIBLE ā did the scaffold provide web_search / browser / curl to external URLs? ā¢ Evidence of a shipped search tool: proceed. ā¢ No search tool available, agent had nothing to call: NO MATCH (route to Environment Block). 3.Inventory retrieval actions. MATCH if any of: (a) Search tool was available but agent made zero retrieval attempts; answered from training knowledge (b) Exactly one source consulted on an 69 accuracy-sensitive question without corroboration (c) Retrieval attempts uniformly blocked (403/404) and agent proceeded anyway despite alternative channels (d) Used bash-scraping when a proper web tool was available Exclusions. ā¢ ā„ 2lookups with reconciliation, even if final answer is wrong: NO MATCH (Wrong Factual Answer). ā¢ Tasks where training knowledge is plausibly sufficient: NO MATCH. ā¢ Scaffold provided no retrieval tool: NO MATCH (Environment Block). Premature Termination.EXCLUDED Framing. Agent narrates a concrete next action ("Iāl now write the file") in a final message, and the trajectory ends before the narrated action is executed. Often a scaffold artefact (e.g., codex+gpt- 5-mini end-of-turn interpreted as task_complete). Decision procedure. 1. Inspect the last 1-3 agent messages. Look for forward-looking commitment. ā¢Final message is a conclusion ("done", "here is X") without forward-looking verb: NO MATCH. 2. Check whether the narrated action was actually executed afterward (tool call, file write). 3.MATCH if (a) concrete next-action narrated AND (b) no corresponding tool action before trajec- tory ended. Exclusions. ā¢ Narrated and executed successfully: NO MATCH. ā¢ Timed out mid-execution: NO MATCH (Agent Timeout). ā¢ Never narrated anything concrete: NO MATCH (Silent Deliverable). Wrong Output Location.EXCLUDED Framing. Agent produced correct content but wrote it to the wrong path, wrong filename, wrong cell range. Decision procedure. 1. Identify the required output location. 2. Observe where the agent actually wrote. 3. MATCH if content was written AND location deviates from required. Exclusions. ā¢ Never written: NO MATCH (Silent Deliverable). ā¢ Correct location but wrong format: NO MATCH (Wrong Output Schema). ā¢ Correct location but wrong content: NO MATCH. G.2.2 Agent Failure Mode Full Results Table 18 reports per-rubric failure-mode prevalence for all six (model, harness) combinations acrossN = 6,028hard-task trajectories. The six reported rubrics are those with per-rubric judgeā goldĪŗ ā„ 0.5; the remaining six rubrics (Wrong Target / Layer, Agent Timeout, Unverified Claim, Plan Over Implementation, False Success Claim, Wrong Output Schema) are aggregated into Others. 70 Table 18: Per-rubric failure-mode prevalence (%) for all six (model, harness) combinations. WFA = Wrong Factual Answer, AB = Algorithmic Bug, HTR = Hidden-Test Regression, SD = Silent Deliverable, EB = Environment Block, Syn = Syntax / Language Error. Others ag- gregates six rubrics with judgeāgold Īŗ < 0.5. N is the number of hard-task trials. ModelHarnessNWFAABHTRSDEBSynOthers GPT-5.4Codex99127.320.29.61.43.50.617.9 GPT-5.4Terminus-298326.618.84.94.99.31.418.2 Opus 4.6Claude Code97326.422.56.94.12.90.817.7 Opus 4.6Terminus-297423.217.710.05.04.21.424.3 Gemini 3.1 ProGemini CLI99525.017.78.76.93.91.216.7 Gemini 3.1 ProTerminus-297324.515.38.87.34.11.417.9 Reading the table. A cell reports the fraction of trials in that (model, harness) row where the rubric fires; because the judge may assign multiple rubrics to a single trial, row percentages do not sum to 100%. The capability rubrics (WFA, AB, HTR) are broadly conserved across harnesses within each model family, while the operational rubrics (SD, EB) show harness-dependent varia- tion. Environment Block is notably elevated for GPT-5.4 under Terminus-2 (9.3% vs. 3.5% under Codex), consistent with the protocol-incompatibility pattern described in Ā§3.4. Silent Deliverable is highest on Gemini CLI (6.9%) and Opus 4.6 under Terminus-2 (5.0%), reflecting differences in submission-protocol enforcement across harnesses. H Harbor-Index Selection Pipeline This appendix documents the per-cell sampling protocol used to derive the difficulty filter (Sec- tion H.1), the full task-quality rubric used by both the AI auditor and human reviewers (Sec- tion H.2), the judge orchestration and prompt (Section H.3), the human-review protocol (Sec- tion H.4), and the audit-and-fix stage that yields the final 82-task release (Section H.5). H.1 Sampling Protocol Each candidate task is summarized by an 18-trial sample drawn from the live Harbor trial database. We define the filtering mix as the Cartesian product of three leading model families at the time of filtering and two harness configurations: ā¢ Models: Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro. ā¢ Harnesses: each modelās native harness (Claude-Code for Anthropic, Codex for OpenAI, Gemini-CLI for Google), plus the cross-vendor Terminus-2 agent. This yields 6 (model, harness) pairs. For each pair we keep 3 valid trials, giving 18 trials per task. Difficulty filter. A task passes the filter iffn succ ā¤ 6over its 18-trial sample, wheren succ denotes the number of successful trials (i.e., the success rate over the filtering mix must be at most33%). Success is benchmark-aware. For most benchmarks, a trial counts as a success iff reward > 0. For benchmarks whose reward semantics differ from a binary pass/fail, we follow the per-benchmark scoring cutoff established by the main experiment. H.2 Task Quality Bar The auditor and human reviewers evaluate each candidate against two criteria. A task is rejected if it fails either; a reviewer may also mark a task unsure when the evidence is mixed. The same rubric is shared between the AI judge and the human review rounds so that the two stages enforce a consistent definition of task quality. Testāinstruction alignment. Every test assertion must trace back to a requirement stated in the instruction or implied by the environment, and every requirement in the instruction must have corresponding test coverage. The criterion fails when: 71 ā¢the verifier checks something the instruction never asks for or implies, such as a specific nu- merical value, exact error wording, file format, or schema; ā¢the instruction is ambiguous or under-specified about the required output while the verifier asserts a single specific answer; ā¢multiple correct interpretations of the instruction would each produce a different verifier out- come; ā¢the agent gets the right idea but the verifier rejects it on a clerical detail not foreshadowed by the instruction. Essential difficulty. Difficulty must come from genuine reasoning, algorithmic thinking, domain expertise, long-horizon interaction, or multi-step execution, not from minor formatting details, arbitrary precision, ambiguous output schemas, or clerical detail. The criterion fails when trials repeatedly show models arriving at the right idea but being tripped up by: ā¢ exact whitespace, decimal precision, or output column ordering not specified in the instruction; ā¢ magic strings (function names, error messages, JSON keys) the instruction does not pin down; ā¢ tolerance bounds the verifier applies that the instruction does not disclose. Calibration set. The auditor receives a small set of pre-labeled rejected tasks as in-context cali- bration examples (e.g., a SWE-bench Multilingual task whose verifier outcome depends on wall- clock year; a SWE-bench Pro task whose verifier requires the literal wording of anAnsibleError not pinned down by the PR; a Spider2 task whose gold output was generated by a buggy Haver- sine formula and whose verifier penalizes correct fixes). Tasks exhibiting similar smells, such as verifierāinstruction mismatch, fragile string equality on novel content, under-specified output format, or gold that does not pass the verifier, are rejected by analogy. H.3 Task Audit Process Setup. The AI auditor used at the second pipeline stage is based on Gemini-3-Flash, run inside a per-task Daytona sandbox via Claude Code in non-interactive print mode. The orchestrator pack- ages, per task, a tarball containing the rubric, a corpus JSON file (18 trials with verifier stdouts and ATIF trajectory paths), the trajectories themselves, and the per-trial verifier stdouts; uploads it to a fresh sandbox; runs the audit; and downloads the resulting verdict. The judge is granted its native filesystem and shell tools (Read,Grep,Bash,Write) and is required to consult between two and four trajectories per taskāmixing successes, failures, and exception casesāand to cite specific step IDs, tool calls, and observations in its rationale. The verdict is emitted as a single JSON ob- ject written to a file via theWritetool; a Python fallback parses an inline JSON object from the streamed assistant turn if the tool call is missing. Cross-judge calibration. We ran the same harness with Opus-4.7 as the auditor model over a 74-task subset of the candidate pool to gauge inter-judge reliability against Gemini-3-Flash. The two judges agree on66/74 = 89.2%of decisions across the three-way accept, reject, unsure la- bel space, with CohenāsĪŗ = 0.747(substantial agreement). Collapsed to the operationally relevant binary accept, non-accept, agreement rises to69/74 = 93.2%: the two judges agree on accepts (16) and non-accepts (53) together for 69 of the 74 shared tasks, with the five disagreements con- centrated in tasks Opus-4.7 marked accept but Gemini-3-Flash marked reject. Thus, the cheaper gemini-3-flash-previewjudge is more conservative on this subset: there are no cases where it accepts a task thatopus-4.7rejects. We usegemini-3-flash-previewas the production auditor on this basis. Verdict schema. Every verdict is a JSON object with the following fields:decision(one of accept, reject, unsure);primary_reasons(short tags);rubric(per-criterion verdict and evidence);failure_attribution(task_design, agent_weakness, infra_or_experiment, or mixed);trial_comparison_notes(cross-trial observations grounded in trajectory step IDs); trajectories_consulted;argument_reasoning_evidence(detailed argumentative rea- soning grounded in cited code lines, commands, or agent steps); andsummary(a one-paragraph rationale). 72 Prompt. We provide the full judge prompt in Listing 1. # Task quality audit You are an expert reviewer evaluating the quality of tasks. ## What you receive A single corpus JSON file describing one candidate task with: - benchmark, task_path, n_succ (= number of successful trials out of 18 frontier-model attempts; range 0-6 for the bulk pool, may be higher for under-rep fill-ins) - task_files: full text of instruction.md, task.toml, environment/Dockerfile, solution/solve.sh, tests/test.sh - trials: each with model, agent, reward, exception_type, test_stdout (verifier stdout, head/tail-clipped to ~32 KB), and trajectory_path (path to ATIF agent trajectory JSON) You may also be given a task_dir path and file-system access to read additional files in that directory if needed. ## Your job Decide accept, reject, or unsure on the task based on quality audit. We accept a task if it passes the rubrics that demonstrate high quality; we reject a task if it has noticeable problems. Mark "unsure" if you are not confident in your judgment. List detailed reasoning, evidence, and arguments along with your final verdict. ## Quality rubric ### 1. test_instruction_alignment Every test assertion must trace back to a requirement stated in instruction.md or implied by the environment, and every requirement in the instruction must have corresponding test coverage. Tests should NOT introduce requirements beyond what the instruction describes. FAIL if: - The verifier checks something the instruction never asks for or never implied (e.g., a specific numerical value, exact error wording, file format, or schema not stated in the instruction). - The instruction is ambiguous or under-specified about what the agent should produce, while the verifier asserts a specific answer. - Multiple correct interpretations of the instruction would each produce a different verifier outcome. - The agent gets the right *idea* but the verifier rejects them on a clerical detail not foreshadowed by the instruction. ### 2. essential_difficulty Difficulty must come from genuine reasoning, algorithmic thinking, domain expertise, long-horizon interactions, multi-step execution, etc -- NOT from formatting minutiae, arbitrary precision, ambiguous output schemas, or clerical detail. FAIL if the trials show models repeatedly arriving at the right idea but getting tripped up on: - Exact whitespace / decimal precision / output column ordering not specified in instruction. - Specific magic strings (function names, error messages, JSON keys) the instruction doesnāt pin down. - Tolerance bounds the verifier applies that the instruction doesnāt disclose. ## How to use trial data You have several test_stdouts plus task files. Use them to ground your judgment: 1. Compare successful vs failed trial test_stdouts. What does the success pattern look like -- was it luck on a fragile verifier, or genuine task completion following instruction? What did the failures actually fail on -- insufficient domain knowledge and/or task understanding, 73 wrong solution idea/execution/calculation, formatting, timeout, infra? If unclear, read the trajectories. 2. Compare successful trial output to solve_sh. NOTE that solve_sh is a rough reference solution -- it does not mean that the agent or "only correct" solution should do the same thing to complete the task. It is only material for you to compare a potentially correct approach against agent approaches. All your judgment should still be grounded in the task description and test verifiers. 3. Look for infra/experiment failures masquerading as task failures. Patterns to flag (mark as unsure): - 429 Too Many Requests, rate limiting from model API. - No space left on device, OOM errors mid-execution. - Network errors / DNS / proxy connect failures. - Container build failures unrelated to the task design. - Truncated transcripts due to step-count exhaustion. These are experiment issues, not task issues. If 5 of 6 trials failed for infra reasons and 1 succeeded cleanly, the task may still be high-quality -- accept if other criteria pass. ## Output format Output a single JSON object (no preamble, no trailing markdown): "decision": "accept" | "reject" | "unsure", "primary_reasons": ["short tag", ...], "rubric": "test_instruction_alignment": "verdict": "pass"|"fail"|"na", "evidence": "...", "essential_difficulty": "verdict": "pass"|"fail"|"na", "evidence": "..." , "failure_attribution": "task_design" | "agent_weakness" | "infra_or_experiment" | "mixed", "trial_comparison_notes": "What you noticed comparing trajectories and test_stdouts (successes vs failures, cross-model). Cite specific step IDs / tool calls / observations from at least one trajectory you rendered.", "trajectories_consulted": ["<trial_id_1>", "<trial_id_2>", ...], "argument_reasoning_evidence": "Show your judgment of the task quality. Claim your arguments. Show detailed reasoning with concrete evidence (specific to words/commands/code-lines/agent-steps if helpful) to demonstrate. If you claim the task to be āunsureā, show what you are confident and unconfident about respectively, and what you want human reviewers to double-check.", "summary": "One-paragraph rationale for the decision, citing specific files/lines/strings." Listing 1: Harbor-Index task-quality judge prompt (judge_prompt.md). H.4 Human Review Protocol Human review is conducted by a pool of 14 domain-experienced reviewers using the same task- quality rubric as the AI auditor. The process has three stages: 1.Initial quality screen. A task handled by a senior reviewer receives at least one review; a task handled by junior reviewers receives at least two reviews. Reviewers inspect the task specification, environment, verifier, and available execution evidence, and may accept, flag for repair, reject, or escalate an uncertain case. This screen leaves more than 110 candidates. 2.Panel selection. A panel of three senior reviewers discusses the surviving candidates and selects an intermediate pool of 100 based on task quality, difficulty, diversity, and the insight offered by model behavior. 3.Trajectory-grounded audit. Each of the 100 selected tasks is re-examined by at least two senior reviewers. Reviewers inspect trajectories and verifier outcomes, perform failure analysis, 74 and distinguish true and false positives and negatives. They repair fixable task defects and remove tasks that remain broken or become too easy after repair, producing the final 82-task release. This process is an iterative engineering curation workflow rather than a fixed-panel annotation study: the review method was refined during construction, and reviewer counts therefore vary by task. We report the minimum review coverage and adjudication process above rather than a single human inter-rater-agreement statistic. Additional audit examples and task-level artifacts are available at https://harbor-index.org. H.5 Audit-and-Fix Loop The senior panelās initial cut yields an intermediate pool of 100 candidate tasks. We then run re- peated rounds of automated audit and reviewer repair: an LLM judge grades the verifier in a fresh copy of each taskās sandbox by reading agent trajectories, verifier outputs, tests, and the reference solution, and labels each rollout as a true/false positive or negative with citations to specific steps or files. We prioritize false positives that let agents pass without solving the task and false nega- tives that reject correct solutions. Tasks are repaired (or dropped when irreparable), the filtering- mix models are re-run, and the audit repeats until the set stabilizes. Tasks that remain broken or become too easy after repair are removed, leaving the final Harbor-Index 1.0 release of 82 tasks across 29 benchmarks. We also tighten per-task timeouts in this stage and adopt Harbor features such as separate verifier sandboxes to close shared-container reward hacks across the suite. Harbor-Index 1.0 Task Catalog Table 19 summarizes the 82 release tasks by domain and benchmark. Task IDs and interactive per-task scores for all 1,476 rollouts are published athttps://harbor-index.organd on the Harbor Hub. I Limitations Apart from the limitations listed in Section 6, we discuss extended limitations here. Evaluation cost and scalability. Large-scale agent evaluation remains expensive. Our study uses substantial compute resources and token budgets, which may limit reproducibility for smaller research groups. While Harbor-Index reduces cost, it is still a proxy for the full benchmark distri- bution. LLM-as-a-judge limitations. Some tasks rely on automated evaluation using language models as judges. Even though we aggregate the evaluation from 4 different models from different model providers and repeated runs for each model, these evaluations may still introduce bias, variance, or systematic errors, especially for open-ended or subjective tasks. Ethical considerations. Our work focuses on evaluation infrastructure and does not directly ad- dress downstream societal impacts such as misuse of agents or deployment risks. Future work should consider safety, alignment, and governance aspects of agentic systems. Addressing these limitations is an important direction for future work, including expanding bench- mark diversity, improving adapter verification, reducing evaluation cost, and designing more realis- tic and dynamic evaluation protocols. J Broader Impacts This work introduces Harbor Adapters and Harbor-Index, aiming to improve the scalability, re- producibility, and coverage of agentic evaluation. We discuss potential positive impacts as well as risks and mitigation strategies. Positive impacts. Our infrastructure lowers the barrier to evaluating language-model agents across diverse benchmarks, promoting more comprehensive and reproducible comparisons. By enabling large-scale and standardized evaluation, this work may help the community better understand 75 model capabilities, identify failure modes, and avoid overfitting to a small set of popular bench- marks. In the long term, improved evaluation can contribute to the development of more reliable and robust AI systems in domains such as software engineering, scientific research, and data analy- sis. Risks and potential misuse. Standardized evaluation frameworks may also accelerate competitive benchmarking and optimization, potentially encouraging overfitting to benchmark suites rather than real-world performance. In addition, improved evaluation infrastructure could indirectly support the development of more capable autonomous agents, which may raise concerns around misuse, including automation of harmful tasks, large-scale exploitation, or deployment without sufficient safeguards. Mitigation and responsible use. We partially mitigate these risks by emphasizing compactness, diversity, difficulty, and quality in Harbor-Index construction, and by releasing detailed analyses of failure modes. We encourage users to interpret results cautiously, avoid over-reliance on single benchmarks, and complement our evaluation with domain-specific and real-world testing. Fu- ture work should further incorporate safety, alignment, and governance considerations into agent evaluation frameworks. Overall, we believe that improving evaluation infrastructure is a necessary step toward building more transparent, reliable, and accountable AI systems, but it must be accompanied by responsible use and continued scrutiny. 76 Table 2: 93 Benchmarks integrated via Harbor adapters, organized by domain and subdomain. Colors mark agentic character: agentic, modified-agentic, non-agentic. DomainSubdomainBenchmarks Software Engineering Repo-level Issue Resolution and Testing ABC-Bench [181], CooperBench [68], DevEval [80], Multi-SWE-Bench [196], SWE-Bench-Multilingual [64], SWE-Bench Pro [33], SWE-bench Verified [64], SWE-Gym [120], SWE-rebench [7], SWE-smith [183], SWT Bench [113] Feature DevelopmentFeatBench [18], FeatureBench [206], ProgramBench [184], SWE-Lancer [106], WebGen-Bench [96] Competitive & Function-level Coding Aider Polyglot [2], AutoCodeBench [204], BigCodeBench [210], EvoEval [176], Frontier-CS-Algorithm [102], LiveCodeBench [62], USACO [143]; CanItEdit [15], HumanEvalFix [111], QuixBugs [87], UniCode [203] Performance OptimizationGSO [142]; AlgoTune [128] Language TranslationCRUST-Bench [67] DevOps & Build SystemsCompileBench [132], DevOpsGym [154] Mathematics & Reasoning Competition MathematicsAIME [191], IneqMath [94], Omni-Math [46] Abstract & Procedural Reasoning SATBench [170]; ARC-AGI-2 [23], KUMO [88], Reasoning Gym [151] Knowledge & Long Context Expert & Multi-subject QACL-Bench [35], GPQA Diamond [135], Humanityās Last Exam [126], MMMLU [55], SimpleQA [171] Long-Context ReasoningAA-LCR [155], LoCoMo [100] Scientific Research End-to-end Research Workflows AstaBench [12], ML-Dev-Bench [119], MLR-Bench [19], ReplicationBench [190], RExBench [39], ScienceAgentBench [21], SLDBench [89]; MLGym-Bench [114] Scientific ComputingLLM-SRBench [145]; CodePDE [83], ResearchCodeBench [59], SciCode [157]; QCircuitBench [185] Biomedical ResearchBIX-Bench [107]; LAB-Bench [78] Agents, Tools & Systems Tool Use & AssistantsACE-Bench [17], GAIA [104], GAIA2 [45], LongCLIBench [43], OSWorld [179], Tau3-Bench [187]; AMA-Bench [201], BFCL [121] Deep Research & Web Agents DeepResearch-Bench-I [37], DeepSynth [123], Seal-0 [125], WideSearch [173] Interactive GamesTextArena [50] Data & Analytics Text-to-SQL & Data ScienceADE-Bench [150], DA-Code [61], DABstep [40], KramaBench [75]; BIRD-Bench [81], DS-1000 [76], Spider 2 [79] Professional Domains Finance & TradingFinanceAgent [162]; PIXIU [178] Business / Professional WorkCRMArena [60], SpreadsheetBench [98], TheAgentCompany [180] Healthcare & ClinicalMedAgentBench [63] LegalLawBench [42] Safety & Security CybersecurityCyberGym [169] Jailbreak RobustnessStrongReject [148] Multimodal Audio UnderstandingMMAU [140] Graphic / Visual DesignGraphDesignBench [32] Autonomous Driving & Sensor Reasoning RefAV [27] 77 Table 3: Harbor adapter catalog (Part 1 of 2): Software Engineering, Mathematics & Reasoning, and Knowledge & Long Context. Sc.: integration scenario (1 = Harbor-Agent, 2 = Vanilla-LLM, 3 = Custom-Agent). Mode: agent interaction mode (A = Agentic, M = Modified-Agentic, N = Non-Agentic).k: number of parity runs. Set:parity/fulltask count. Orig. / Harbor: pass rate (%) as meanĀ± SEM over k runs. BenchmarkSubdomainSc. Mode Parity Agent Parity Model kSetOrig.Harbor SWE-Bench- Multilingual Repo Issues2Acodexgpt-5-mini350/30033.3Ā±0.733.3Ā±3.3 SWE-Bench ProRepo Issues2Acodexgpt-5-mini6100/73135.7Ā±0.636.3Ā±1.0 SWE-bench Verified Repo Issues1Amini-swe- agent gpt-5-mini3499/50056.3Ā±0.054.5Ā±0.7 SWE-smithRepo Issues1Amini-swe- agent claude-3- haiku ā100/1000.00.0 SWT BenchRepo Issues1Aclaude-code haiku-4-5ā433/43320.55Ā±0.0 20.55Ā±0.0 CooperBenchFeature Dev1Aopenhands- sdk gemini-3- flash 350/65232.7Ā±1.330.7Ā±1.3 FeatBenchFeature Dev1Atrae-agentdeepseek- v3.2 3156/15649.8Ā±2.349.4Ā±1.1 FeatureBenchFeature Dev1Acodexgpt-5-mini230/20013.3Ā±0.015.0Ā±1.7 SWE-LancerFeature Dev2Aclaude-code claude- sonnet-4 5463/46348.4Ā±1.047.6Ā±0.4 WebGen-BenchFeature Dev2Maidergpt-5-mini3101/10116.7Ā±0.916.2Ā±1.6 Aider PolyglotCoding2Nclaude-code claude-3- haiku 1225/2252.7Ā±0.02.7Ā±0.0 BigCodeBenchCoding2Mcodexgpt-5-mini3145/14532.7Ā±0.434.0Ā±1.6 FrontierCSCoding2Mclaude-code opus-4-6310/17268.9Ā±11.553.4Ā±9.9 LiveCodeBenchCoding2Nclaude-code haiku-4-54100/105554.5Ā±1.553.3Ā±1.0 USACOCoding2Nclaude-code haiku-4-53304/30440.3Ā±0.443.0Ā±1.4 HumanEvalFixCoding1Nopenhandsgpt-5-mini3164/16498.2Ā±0.698.0Ā±0.4 QuixBugsCoding2Ncodexgpt-5-mini380/8088.4Ā±0.487.9Ā±0.4 GSOPerf. Opt.1Aopenhandsgpt-5.12102/10213.7Ā±1.013.2Ā±1.5 AlgoTunePerf. Opt.2Aterminus-2gpt-5-mini3154/1541.23Ā±0.02 1.23Ā±0.02 CRUST-BenchLang. Trans.2Mcodexgpt-5-nano3100/10058.3Ā±0.758.3Ā±1.2 DevOpsGymDevOps1Acodexgpt-5-mini350/73322.7Ā±1.822.0Ā±0.0 AIMEComp. Math2Ncodexgpt-5-nanoā30/30N/AN/A IneqMathComp. Math2Ncodexgpt-4o-mini3100/10050.0Ā±1.552.0Ā±0.7 Omni-MathComp. Math2Ncodexgpt-5.3- codex 2100/442880.0Ā±2.081.0Ā±2.0 ARC-AGI-2Abs. Reason. 2Ncodexgpt-5.26167/16736.0Ā±0.835.8Ā±0.8 KUMOAbs. Reason. 2Nkumo- vanilla gpt-5-mini3212/530088.7Ā±0.789.9Ā±0.7 Reasoning GymAbs. Reason. 2Ncodexgpt-5.1- codex-mini 3576/57685.7Ā±0.485.9Ā±0.3 GPQA DiamondExpert QA2Ncodexgpt-5.23198/19887.9Ā±0.687.2Ā±0.3 Humanityās Last Exam Expert QA2Nclaude-code haiku-4-53249/250010.7Ā±0.911.0Ā±0.4 MMMLUExpert QA2Ncodex-cligpt-5.1- codex-mini 5 150/21063063.5Ā±0.563.3Ā±0.5 SimpleQAExpert QA2Nclaude-code opus-4-6350/432696.7Ā±1.894.7Ā±0.7 A-LCRLong Ctx.2Ncodexgpt-5-mini199/9968.068.7 78 Table 4: Harbor adapter catalog (Part 2 of 2): Scientific Research, Agents/Tools/Systems, Data & Analytics, Professional Domains, Safety & Security, and Multimodal. Columns as in Table 3. BenchmarkSubdomainSc. Mode Parity Agent Parity Model kSetOrig.Harbor ReplicationBenchResearch WF 1Aclaude-code haiku-4-5190/9014.414.4 ScienceAgent Bench Research WF 1Aclaude-code haiku-4-55102/10233.7Ā±0.931.8Ā±1.2 SLDBenchResearch WF 1Aclaude-code haiku-4-558/838.1Ā±9.5 39.2Ā±10.7 MLGym-BenchResearch WF 2Mmini-swe- agent gpt-5312/1280.6Ā±2.880.6Ā±2.8 CodePDESci. Comp.2Mterminus-2haiku-4-585/527.5Ā±3.727.5Ā±3.7 ResearchCode Bench Sci. Comp.2Mcodexgpt-4.1-mini 3212/21241.0Ā±0.440.7Ā±0.3 SciCodeSci. Comp.2Mcodexgpt-5.1- codex-mini 380/8043.3Ā±0.643.8Ā±0.6 QCircuitBenchSci. Comp.2Ncodexgpt-5.2328/2860.9Ā±2.561.9Ā±2.1 BIX-BenchBiomed.3Abixbench- agent gpt-4o-mini350/20516.0Ā±3.116.7Ā±2.4 LAB-BenchBiomed.2Ncodexgpt-5-codex3181/18140.0Ā±0.241.1Ā±0.5 ACE-BenchTool Use1Aclaude-code haiku-4-53973/97383.6Ā±0.483.2Ā±0.8 GAIATool Use1Aopenhandsgpt-5-mini3165/16551.3Ā±0.750.7Ā±0.5 GAIA2Tool Use3Agaia2- parity-agent gpt-5-mini2100/8006.0Ā±1.06.0Ā±0.0 BFCLTool Use2Ncodexgpt-5-mini3123/364183.2Ā±0.783.5Ā±1.0 DeepSynthWeb Agents1Aclaude-code haiku-4-5340/409.3Ā±1.07.8Ā±0.6 Seal-0Web Agents1Aclaude-code haiku-4-53111/11133.9Ā±3.033.3Ā±3.6 WideSearchWeb Agents1Aclaude-code haiku-4-53200/20053.8Ā±0.353.9Ā±0.3 DA-CodeSQL & DS3Ada-agentgpt-4o3479/47938.7Ā±1.138.9Ā±0.2 Spider 2SQL & DS1Aspider- agent gpt-5-mini364/6418.8Ā±0.018.3Ā±0.5 FinanceAgentFinance3Afinance- agent gpt-5.2350/5078.0Ā±1.080.0Ā±0.0 PIXIUFinance2Mcodexgpt-5-mini3 435/54083 90.1Ā±3.385.2Ā±6.4 CRMArenaBusiness3Acrmarena- react haiku-4-5390/117069.0Ā±1.068.0Ā±2.0 SpreadsheetBench Business1Aclaude-code haiku-4-53400/40068.8Ā±0.868.3Ā±1.1 MedAgentBenchHealthcare3Amedagent- react gpt-4o-mini3300/30058.0Ā±0.957.9Ā±0.3 LawBenchLegal2Nqwen-codeglm-4.73120/100065.0Ā±0.365.6Ā±0.7 CyberGymCybersec.3Aopenhandshaiku-4-5510/150764.0Ā±4.066.0Ā±2.5 StrongRejectJailbreak2Ncodexgpt-5-nano3 150/12207 95.0Ā±0.795.0Ā±1.1 MMAUAudio2Nterminus-2gpt-4o3 1000/1000 56.6Ā±0.856.6Ā±0.8 79 Table 6: GPT-series models under Terminus-2 and Codex across 54 benchmarks. Benchmark GPT-5.4GPT-5-miniGPT-5-nano Terminus-2CodexTerminus-2CodexTerminus-2Codex SWE-bench Verified0.6900.7130.3070.4600.1600.110 SWE-Bench-Multilingual0.6070.7000.2130.2800.0470.033 SWE-smith0.0310.2210.0100.2960.0650.051 CRUST-Bench0.9200.9800.6200.8030.3270.113 GSO0.1670.3730.0070.0950.0000.010 SWE-Bench Pro0.3470.5500.1000.4000.0530.057 SWT Bench0.5400.7070.1470.3070.0730.147 Aider Polyglot0.5440.7850.3670.5390.1410.096 AlgoTune0.1660.2750.1300.1440.0500.009 CompileBench0.9331.0000.5330.6440.4000.133 FeatureBench0.0560.3680.0340.1030.0090.004 BigCodeBench-Hard0.3950.4280.3520.2830.2440.034 USACO0.8300.9600.8430.8570.7130.057 SWE-Lancer0.4570.5800.4070.3970.2300.303 LiveCodeBench0.8730.9270.8470.8830.6970.307 HumanEvalFix0.9981.0000.9880.9980.9410.470 QuixBugs0.8630.9750.8920.8870.3620.292 IneqMath0.8530.9970.9230.9800.7800.903 Omni-Math0.5430.8520.7670.8330.7100.622 Reasoning Gym0.8420.9020.8410.8190.6710.563 KUMO0.9260.9640.8490.9360.5410.668 AIME 24 & 250.7110.9890.8890.9670.7560.739 ARC-AGI-20.0200.5330.0030.0030.0000.000 A-LCR0.5250.7510.3200.7440.1310.360 SimpleQA0.7480.9450.6450.9550.1580.762 GPQA Diamond0.7780.9230.7850.7980.6080.613 Humanityās Last Exam0.2030.5190.1310.3120.0580.111 MMMLU0.3560.8380.6490.7270.5820.602 GAIA0.4140.7600.2530.7010.1640.428 SkillsBench0.3170.6020.1320.2580.0540.017 BFCL0.7340.6830.7910.7640.6210.610 GAIA20.1700.2100.0000.1660.0000.035 Terminal-Bench 2.00.4270.6930.2510.3710.1050.052 DeepSynth0.2890.5480.0690.3000.0300.088 Seal-00.2760.5110.1920.4590.0480.258 WideSearch0.5600.7910.3030.5330.1310.120 Spider 20.2190.2760.1090.2450.1300.141 DA-Code0.5580.5650.3770.4990.2500.171 SpreadsheetBench0.6430.7620.4520.5880.2630.095 FinanceAgent0.5000.7670.1130.5330.0000.213 LawBench0.5860.6380.4860.4250.3210.110 PIXIU0.5740.7100.5060.4800.2830.337 MedAgentBench0.3400.6430.3500.4300.0800.290 LAB-Bench0.2300.7310.1600.4750.0830.120 SLDBench0.6730.7950.5420.7070.2250.075 BIX-Bench0.3000.4070.1930.3130.0730.073 ReplicationBench0.2560.6070.0810.2330.0410.041 QCircuitBench0.4960.6200.3580.1770.1420.000 ResearchCodeBench0.0060.8660.0000.5990.0020.421 SciCode0.4690.5820.4300.2260.2170.004 CodePDE0.6000.4670.5330.3330.2670.000 StrongReject0.9590.9490.9350.9580.9610.951 CyberGym0.5330.7670.6330.6330.1670.233 MMAU0.6600.7030.5970.5830.5370.070 80 Table 7: Claude-series models under Terminus-2 and Claude Code across 54 benchmarks. Benchmark Claude Haiku 4.5Claude Sonnet 4.6Claude Opus 4.6 Terminus-2Claude CodeTerminus-2Claude CodeTerminus-2Claude Code SWE-bench Verified0.6170.5730.7330.7030.7500.733 SWE-Bench-Multilingual0.6870.4730.7330.6930.8000.667 SWE-smith0.1870.2310.4150.2890.5820.412 CRUST-Bench0.6370.7700.8470.7130.8700.823 GSO0.0100.1310.4280.4510.3660.392 SWE-Bench Pro0.0800.4500.4730.4900.4870.537 SWT Bench0.1600.2730.4470.5730.6730.647 Aider Polyglot0.2990.3010.5720.5600.7190.686 AlgoTune0.0960.1230.2600.2700.2590.231 CompileBench0.8220.8890.8670.9110.9330.933 FeatureBench0.0340.1260.1890.2770.4050.323 BigCodeBench-Hard0.3930.3450.4460.4370.4440.416 USACO0.2730.3800.7270.7500.8000.780 SWE-Lancer0.3370.3570.4530.4870.5830.523 LiveCodeBench0.4600.6070.8230.8370.8870.870 HumanEvalFix0.9290.9941.0000.9981.0001.000 QuixBugs0.7170.8870.9250.9750.9540.933 IneqMath0.7330.9530.9700.9970.8600.860 Omni-Math0.6000.7600.7880.8000.8130.840 Reasoning Gym0.8030.8070.8690.8180.8750.851 KUMO0.9090.9560.9480.9480.9640.964 AIME 24 & 250.6220.8610.9780.9670.9890.994 ARC-AGI-20.0070.0600.1800.1870.3700.437 A-LCR0.6060.5490.6630.7410.6160.747 SimpleQA0.5730.9200.8070.8030.9070.867 GPQA Diamond0.5350.7090.8230.8540.8430.884 Humanityās Last Exam0.0640.1070.2170.3250.3430.390 MMMLU0.6910.6710.7510.7560.7690.778 GAIA0.3130.5700.5290.6790.6120.671 SkillsBench0.1210.2540.3110.4890.3460.454 BFCL0.7700.7290.7700.8050.7640.799 GAIA20.0560.2130.3190.3840.3570.358 Terminal-Bench 2.00.3030.3220.5810.5690.6180.607 DeepSynth0.0620.1120.3980.4540.4920.461 Seal-00.0600.2580.3420.2940.3900.348 WideSearch0.3260.5740.5110.6730.6170.758 Spider 20.2080.2550.3390.3700.3750.396 DA-Code0.5040.5100.5780.5800.6160.580 SpreadsheetBench0.5600.7020.8030.8320.8320.818 FinanceAgent0.0800.2070.8000.8000.7200.833 LawBench0.5380.5650.6080.6200.6540.666 PIXIU0.4850.4890.5690.5740.6070.612 MedAgentBench0.4670.5370.4970.5030.5970.557 LAB-Bench0.1230.3570.0180.0770.0960.451 SLDBench0.6120.7720.7470.7590.7600.522 BIX-Bench0.1530.2870.2930.3930.3000.393 ReplicationBench0.1040.1520.2070.2190.2410.285 QCircuitBench0.2320.3840.5070.4930.5620.489 ResearchCodeBench0.0030.4340.0060.5330.0030.607 SciCode0.3450.3660.4390.4520.4850.488 CodePDE0.4000.3330.6000.3330.6000.400 StrongReject0.9700.9620.9090.9060.9100.919 CyberGym0.6330.7330.7000.5000.5670.633 MMAU0.5500.5030.6330.6170.6970.690 81 Table 8: Gemini-series models under Terminus-2 and Gemini CLI across 54 benchmarks. Benchmark Gemini 3.1 Pro PreviewGemini 3 Flash Preview Terminus-2Gemini CLITerminus-2Gemini CLI SWE-bench Verified0.7770.7570.6870.733 SWE-Bench-Multilingual0.7670.7470.7470.767 SWE-smith0.3230.4420.2960.211 CRUST-Bench0.9400.9230.6870.853 GSO0.3730.4310.2450.408 SWE-Bench Pro0.4670.4570.3900.460 SWT Bench0.5730.4870.6000.560 Aider Polyglot0.7880.8840.6670.713 AlgoTune0.2390.2780.1820.163 CompileBench0.9330.9560.9561.000 FeatureBench0.2580.4050.0380.187 BigCodeBench-Hard0.4050.4250.3860.400 USACO0.9330.5400.9200.880 SWE-Lancer0.6300.7100.4770.500 LiveCodeBench0.9200.9230.8770.907 HumanEvalFix1.0001.0000.9980.998 QuixBugs0.9250.9330.9580.971 IneqMath0.9470.8700.8670.193 Omni-Math0.8720.8800.8350.650 Reasoning Gym0.9060.8040.8680.594 KUMO0.9690.9670.9530.964 AIME 24 & 250.9890.9940.8890.811 ARC-AGI-20.6030.7670.3500.377 A-LCR0.7440.6600.6430.650 SimpleQA0.9500.9600.8400.910 GPQA Diamond0.9530.9710.8960.704 Humanityās Last Exam0.5330.5410.3950.272 MMMLU0.8040.7690.8240.740 GAIA0.6770.6710.5680.364 SkillsBench0.3740.5820.2600.564 BFCL0.8370.8130.7180.743 GAIA20.2150.3310.1830.214 Terminal-Bench 2.00.6890.6330.5130.524 DeepSynth0.5500.5880.3590.328 Seal-00.4200.5050.3930.393 WideSearch0.6410.6600.6410.536 Spider 20.3440.3490.2500.323 DA-Code0.6060.6010.5580.561 SpreadsheetBench0.7950.8250.7550.768 FinanceAgent0.7530.7600.5400.380 LawBench0.6970.6440.6480.542 PIXIU0.7480.7480.7110.704 MedAgentBench0.6370.5500.5070.307 LAB-Bench0.3130.7090.3110.678 SLDBench0.7840.7370.7890.747 BIX-Bench0.4330.4800.3930.393 ReplicationBench0.5410.6300.2190.356 QCircuitBench0.6870.5600.6430.491 ResearchCodeBench0.0000.8330.0020.711 SciCode0.5500.5370.5110.144 CodePDE0.2670.1330.3330.200 StrongReject0.8680.9160.7330.875 CyberGym0.8330.8000.7330.733 MMAU0.6270.6100.6270.507 82 Table 11: Ranking fidelity from a small task subset vs. the full benchmark. Values are mean Spear- man Ļ across 52 benchmarks withā„ 10 tasks. k tasksMean ĻBenchmarks with Ļ > 0.95 10.8853/52 30.92312/52 50.94122/52 100.95333/52 Table 14: Benchmark-level overlap between public benchmark evaluations and our Harbor modelā harness configurations. Columns are not mutually exclusive: a benchmark can contain both overlapping and non-overlapping public configurations. Has overlap with oursHas non-Harbor configurationTotal Evaluation includes raw LLMs103138 Evaluation includes agents72733 Unique benchmarks165254 Table 15: Comparison with original or launch-time best results on selected benchmarks. Scores are reported on a [0, 1] scale. BFCLSLDBenchCodePDEMedAgentBenchAlgoTune Original / launch best0.90720.87410.80000.69670.3401 Harbor best0.83740.79520.60000.64330.2776 Table 16: Round-1 vs. Round-2 inter-annotator CohenāsĪŗon182trials with labels in both rounds. Sorted by Īŗ. Twelve rubrics clear the Īŗā„ 0.5 trustworthy threshold; six are excluded. RubricR1+R2+ĪŗTier Wrong Target / Layer221.000Trustworthy Syntax / Language Error340.854Trustworthy Agent Timeout890.815Trustworthy Unverified Claim27210.809Trustworthy Plan Over Implementation230.797Trustworthy Wrong Factual Answer37310.784Trustworthy Hidden-Test Regression22140.755Trustworthy False Success Claim23160.743Trustworthy Silent Deliverable630.659Trustworthy Algorithmic Bug25260.659Trustworthy Environment Block780.652Trustworthy Wrong Output Schema12110.583Trustworthy Calculation Mismatch2270.451Excluded Rule Inference Error280.389Excluded SpecificationāVerification Mismatch16340.320Excluded Insufficient Web Research030.000Excluded Premature Termination030.000Excluded Wrong Output Location010.000Excluded 83 Table 17: Pro v12 judge agreement with the gold set on the182-trial calibration sample. Trials where one or more parse failures dropped a verdict are excluded from the per-rubric counts. Six rubrics clear Īŗā„ 0.5; the rest are folded into āOthersā. RubricGold+LLM+ Īŗ vs. goldTier Wrong Factual Answer37490.69ā„ 0.5 (reported) Syntax / Language Error450.66ā„ 0.5 (reported) Silent Deliverable660.66ā„ 0.5 (reported) Hidden-Test Regression20110.62ā„ 0.5 (reported) Environment Block1050.52ā„ 0.5 (reported) Plan Over Implementation310.50ā„ 0.5 (reported) Agent Timeout970.48Others False Success Claim19100.44Others Algorithmic Bug28350.37Others Wrong Output Schema1560.25Others Unverified Claim2110.08Others Wrong Target / Layer22ā0.01Others Pooled1831380.51ā Table 19: Harbor-Index 1.0 task counts by domain and benchmark (82 tasks / 29 benchmarks). DomainBenchmark#Tasks Software Engineering GSO7 SWE-Bench Verified5 AlgoTune5 FeatureBench4 SWE-Bench Pro4 SWE-Lancer2 BigCodeBench1 USACO1 SWE-smith1 SWT Bench1 Scientific Research BIX-Bench5 LAB-Bench4 SciCode3 SLDBench1 Replication Bench1 CodePDE1 QCircuit Bench1 Agents, Tools & Systems GAIA25 GAIA3 Terminal Bench 23 SkillsBench2 WideSearch1 Knowledge HLE8 GPQA Diamond1 Mathematics & Reasoning ARC-AGI-25 OmniMath2 Data & Analytics Spider 22 DA-Code1 Safety & SecurityCyber Gym2 84