Paper deep dive

An Executable Benchmarking Suite for Tool-Using Agents

Zhiqing Zhong, Zhijing Ye, Jiamin Wang, Xiaodong Yu

Year: 2026Venue: arXiv preprintArea: cs.SEType: PreprintEmbeddings: 65

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%

Last extracted: 7/14/2026, 4:08:16 AM

Summary

This paper introduces an executable benchmarking suite for closed-loop tool-using agents that establishes an explicit evidence-admission contract. The suite connects WebArena Verified, a SWE-Gym slice, and MiniWoB++ through shared workload adapters, task manifests, event schemas, and replay policies. It distinguishes between workloads, action-generating drivers (e.g., Qwen-backed LLMs), and admitted evidence, separating paper-facing claims from diagnostic or smoke-test artifacts. The release provides an auditable contract for latency, invalid-action behavior, and patch-generation costs, demonstrating that evidence admission gates can influence systems-facing decisions, such as controller selection under different stress conditions.

Entities (9)

Stevens Institute of Technology · affiliation · 99%Zhiqing Zhong · author · 99%WebArena Verified · benchmark-environment · 98%Evidence-Admission Contract · methodology · 97%MiniWoB++ · benchmark-environment · 95%SWE-Gym · benchmark-environment · 95%Qwen · model-family · 92%SGLang · backend-engine · 90%vLLM · backend-engine · 90%

Relation Signals (7)

Zhiqing Zhong → affiliatedwith → Stevens Institute of Technology

confidence 99% · Affiliation: Stevens Institute of Technology

Zhiqing Zhong → authored → An Executable Benchmarking Suite for Tool-Using Agents

confidence 99% · An Executable Benchmarking Suite for Tool-Using Agents Zhiqing Zhong

SWE-Gym → isconnectedto → Executable Benchmarking Suite

confidence 95% · The suite connects... a SWE-Gym slice... through common workload adapters

MiniWoB++ → isconnectedto → Executable Benchmarking Suite

confidence 95% · The suite connects... MiniWoB++ through common workload adapters

WebArena Verified → isconnectedto → Executable Benchmarking Suite

confidence 95% · The suite connects WebArena Verified... through common workload adapters

Evidence-Admission Contract → separates → Paper-Facing Evidence

confidence 95% · The evidence gate separates paper-facing evidence from preflight, fixture, smoke, and diagnostic rows

Qwen → isusedby → Executable Benchmarking Suite

confidence 90% · local Qwen-family LLM-driver traffic

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Closed-loop tool-using agents are increasingly evaluated in executable web, code, and micro-task environments, but benchmark reports often conflate workloads, action-generating drivers, and the evidence admitted for systems-facing claims. We present an executable benchmarking suite that makes these objects explicit under a shared evidence-admission contract. The suite connects WebArena Verified, a SWE-Gym slice with SWE-bench-compatible verification, and MiniWoB++ through common workload adapters, task manifests, event schemas, replay/freeze policy, declared drivers, and reporting pipelines. In the canonical release, the gate separates paper-facing evidence from preflight, fixture, smoke, and diagnostic rows while preserving non-admitted artifacts for audit and onboarding. The admitted evidence records latency, invalid-action behavior, patch-generation cost, verifier metadata, replay bindings, and provenance under one auditable contract. The gate is decision-relevant rather than merely clerical: in a separate WebArena Verified controller study, clean-baseline and medium live-stressed evaluation select different fixed controller variants under the same workload and admission contract. The release is scoped as a benchmarking suite and admitted evidence, not a new agent policy, model leaderboard, backend comparison, or autonomous SWE-bench solver.

PDF

Open source PDF →

PDF not stored locally. Use the link above to view on the source site.

Full Text

64,602 characters extracted from source content.

Expand or collapse full text

An Executable Benchmarking Suite for Tool-Using Agents Zhiqing Zhong Zhijing Ye Jiamin Wang Xiaodong Yu Affiliation: Stevens Institute of Technology Affiliation: zzhong9,zye25,jwang259,xyu38@stevens.edu Abstract Closed-loop tool-using agents are increasingly evaluated in executable web, code, and micro-task environments, but benchmark reports often conflate three objects: the workload being evaluated, the driver generating actions, and the evidence admitted for systems-facing claims. We present an executable benchmarking suite with an explicit evidence-admission contract that keeps these objects distinct under declared drivers. The suite connects WebArena Verified, a SWE-Gym slice with SWE-bench-compatible verification, and MiniWoB++ through shared workload adapters, task manifests, event schemas, replay classes, release-time freeze policy, and evidence-admission checks. In the canonical release, the evidence gate admits a paper-facing evidence surface from a single versioned release root and excludes non-admitted rows such as preflight, fixture, smoke, diagnostic-only, and unsupported stress rows, with no missing planned evidence strata or validation failures. The admitted evidence contains real-task anchors, local Qwen-family LLM-driver traffic, and bounded extension or diagnostic rows. It reports model-call latency, invalid-action behavior, patch-generation cost, verifier metadata, replay bindings, and provenance under one auditable contract. The evidence gate is decision-relevant rather than merely a bookkeeping filter. In a separately admitted WebArena Verified controller study, clean-baseline evaluation and the promoted medium live-stressed operating setting select different fixed controller variants under the same workload, driver, and admission contract. The controller ordering reverses across both shipped rollout backends and every directly comparable backend–seed–budget cell in the tested grid. We report this decision evidence as a separate controller study rather than using it to backfill the canonical paper-facing evidence surface. The release is scoped as a benchmarking suite and admitted evidence: it defines which workload–driver–setting evidence can support which systems-facing claims, not a new agent policy, model leaderboard, backend comparison, full RL training result, or autonomous SWE-bench solver. 1 Introduction Closed-loop tool-using agents are increasingly evaluated in executable environments: browsers, software repositories, and interactive web tasks. We use this term descriptively for language-agent systems that iteratively observe environment state, call a model or external tool, take an action, and receive environment or verifier feedback, following prior work on reasoning-and-acting agents, tool-using language models, and interactive agent benchmarks (Yao et al., 2023; Schick et al., 2023; Liu et al., 2023; Zhou et al., 2024). In these environments, an evaluation run is no longer just a prompt and an answer. It is an execution trace: the agent observes state, calls a model, takes an action, waits for a browser or repository to respond, and eventually receives an evaluator or verifier outcome. This changes what a benchmark must make explicit. Two papers can use the same named environment while differing in the driver that generated actions, the logging and replay boundary, the backend used for model calls, or the release snapshot used for evaluation. Existing environments provide strong pieces of this picture. WebArena and WorkArena expose long-horizon interaction, rendering, screenshots, DOM access, and evaluator behavior (Zhou et al., 2024; Drouin et al., 2024). BrowserGym and WebArena Verified further emphasize reusable infrastructure for web-agent evaluation (de Chezelles et al., 2025; ServiceNow Research, 2026). Software-engineering benchmarks expose repository state, patch application, build/test execution, and verifier delay (Jimenez et al., 2024). SWE-Gym extends that setting to agent and verifier training under executable repository workloads (Pan et al., 2025). Micro-task environments provide short-horizon, high-throughput interaction that is useful for profiling and instrumentation (Liu et al., 2018). Other recent executable-agent environments and evaluation platforms, including AgentLab, OSWorld, AppWorld, and Tau-bench, further expand the design space for interactive agent benchmarking (ServiceNow Research, 2024; Xie and others, 2024; Agashe and others, 2024; Sierra Research, 2025). Recent web-agent work also studies safety and end-to-end training (Levy et al., 2024; Tur et al., 2025; Wei et al., 2025). These systems are valuable, and our goal is not to replace them. The missing layer is a common contract for running them, recording them, replaying what can be replayed, and deciding which evidence is allowed to support paper claims. We present an executable benchmarking suite for closed-loop tool-using agent evaluation. The v1 suite connects WebArena Verified, a SWE-Gym slice with SWE-bench-compatible verification, and MiniWoB++ through shared adapters, manifests, event schemas, replay/freeze policy, declared drivers, and reporting pipelines. The evidence gate separates rows used for paper claims from smoke tests, fixtures, preflight checks, and diagnostic artifacts, while preserving those artifacts for onboarding and audit. Evaluation role. The contribution is an evidence-admission contract: it specifies when a workload–driver–setting run may support a paper-facing systems claim, when it must remain diagnostic evidence, and how admitted rows join to release roots, replay metadata, and claim maps. This boundary is decision-relevant rather than merely exclusionary. In a separately admitted WebArena Verified controller study, clean-baseline and medium live-stressed evidence select different fixed controller variants under the same workload contract, showing that clean-only evaluation can miss decision-relevant behavior without claiming a new controller algorithm or backend superiority. Our contributions are sixfold. First, we provide an evidence-admission contract for executable benchmark evidence: real web, code, and micro-task workloads run under declared drivers, and only admitted rows support paper-facing systems claims. Second, we define a cross-family adapter, manifest, event-schema, replay/freeze, and reporting contract across WebArena Verified, a SWE-Gym slice, and MiniWoB++. Third, we separate workloads from declared drivers, including Qwen-backed LLM drivers, controller drivers, and calibration controls. Fourth, we report systems-facing outputs: latency decomposition, queueing, replay behavior, trace completeness, invalid-action behavior, patch-generation cost, verifier metadata, and bounded sanity checks. Fifth, we show that admitted evidence can change a systems-facing decision: clean and medium live-stressed WebArena Verified evidence select different controller variants within the tested grid. Sixth, we release manifests, validation reports, redaction manifests, registries, evidence-gate reports, figure inventories, paper-facing evidence summaries, exclusion tables, and reproduction documentation. Takeaway. The release should be interpreted by its admitted workload–driver–setting evidence surfaces, not by workload names alone. 2 Scope and Positioning 2.1 What is under test This is a benchmark-suite and benchmarking-toolchain paper for closed-loop tool-using agent evaluation. The object under test is the execution and evidence substrate: the machinery that runs workloads, records closed-loop interaction, binds each run to a declared driver, and decides which evidence may support paper claims. It is not a new agent policy, prompting strategy, runtime optimization, or model/backend leaderboard. We use three terms throughout. A workload is the task family and environment being evaluated, including its reset rule, evaluator or verifier, and replay boundary. A driver is the component that generates actions for that workload, such as a Qwen-backed LLM driver, a controller driver, a scripted calibration driver, or another contract-conforming agent. An operating setting is the evaluation condition under which a workload–driver pair is run, such as clean-baseline or medium live-stressed execution. The same workload can be paired with different drivers and evaluated under different operating settings. The evidence gate connects these objects to reporting. A paper-facing row must be tied to a workload manifest, declared driver, event-schema version, release root, replay/freeze metadata, and report-generation path. Smoke tests, fixture-backed examples, preflight checks, and diagnostic probes may appear in the artifact for onboarding or validation, but they are not counted as quantitative paper evidence. This scope is deliberate. The v1 release optimizes for an executable, replayable, observable, and version-frozen substrate over an intentionally scoped set of workload families rather than for maximum environment count or leaderboard breadth. 2.2 Why v1 includes these workload families The v1 suite uses three workload families because each stresses a distinct part of the closed-loop evaluation path under the same evidence contract. WebArena Verified (de Chezelles et al., 2025; ServiceNow Research, 2026) is the web anchor: it preserves browser interaction, rendering, screenshots, DOM access, evaluator checks, and replay-oriented web execution, and is used for browser-side execution behavior, bounded live/stressed settings, and the separate controller-choice study. The SWE-Gym slice with SWE-bench-compatible verification (Pan et al., 2025; Jimenez et al., 2024) is the repository/verifier anchor: it exposes checkout, patch application, build/test execution, verifier service time, verifier queueing, and failure-sensitive behavior. MiniWoB++ (Liu et al., 2018) is the lightweight profiling anchor: it provides short-horizon browser interaction for profiling, instrumentation, and bounded sanity checks. Together, these families cover heterogeneous browser I/O, executable verification with repository state, and low-cost micro-task profiling without forcing heterogeneous tasks into a single success metric. 2.3 Claim scope The paper makes three scoped claims. First, heterogeneous web, code, and micro-task workloads can be bound to a common manifest, adapter, event-schema, replay/freeze, driver, and reporting contract. Second, paper-facing evidence can be admitted by an explicit gate that separates real workload rows, LLM-driver rows, bounded extension cells, controlled diagnostics, adapter rows, preflight checks, fixture-backed rows, and non-paper-facing rows; the final canonical surface admits 930 rows and excludes 1,184 rows. Third, admitted workload–driver–setting evidence can change a systems-facing evaluation conclusion relative to a clean baseline alone; the separately admitted controller study admits 56/56 fixed-budget rows and 336/336 tested-grid rows, with all 12 comparable vLLM/SGLang × seed × budget cells reversing controller ordering between clean and medium live-stressed settings. These claims are intentionally bounded: the release does not claim a new agent policy, model/backend superiority, policy state of the art, full RL training, autonomous SWE-bench solving, or robustness outside the admitted workload families, drivers, operating settings, and tested controller grid. The corresponding evaluation criteria are whether the substrate makes workload, driver, setting, replay boundary, release root, and evidence status explicit; whether paper-facing claims join to admitted evidence rather than mixed run directories; and whether the admission boundary constrains or changes a systems-facing conclusion. 3 Unified Benchmark Substrate 3.1 Workload adapters and task manifests The substrate separates the workload being evaluated from the component that drives the interaction. A workload adapter connects one workload family to the shared runner by resolving a task manifest, initializing the environment, executing interaction steps, invoking validation or replay hooks, and emitting structured artifacts. The adapter boundary is intentionally narrow: it does not force web, code, and micro-task environments to share the same observations, actions, or evaluators. Instead, it records those family-specific choices under one execution contract. Task manifests are the unit of release-time binding. A manifest records the workload family, task identifier, environment image or snapshot reference, reset contract, verifier or evaluator identifier, adapter version, replay class, schema version, and release binding. Code workloads add repository state, patch-application semantics, build/test commands, and verifier version. Web workloads add browser/session configuration, evaluator semantics, and live or replay execution mode. Micro-task workloads add lightweight environment state and collector configuration. A run is therefore not identified by a task alone. It is identified by a task manifest, a declared driver, an operating setting, and a release root. This distinction is necessary because the same workload can be exercised by different drivers, and the same driver can be evaluated under clean, stressed, or diagnostic settings. Appendix A records the shared benchmark layers and replay bindings, while Appendix D describes bounded model/backend and framework-export extensions. 3.2 Declared drivers A driver is the component that generates actions for a workload. In v1, declared drivers include Qwen-backed LLM drivers (Qwen Team, 2024), controller drivers, scripted or calibration drivers, bounded sanity drivers, and fixture-backed smoke drivers. These drivers are not introduced as new agent policies. They are declared sources of interaction used to exercise the benchmark substrate. The driver contract records the metadata needed to audit a run: driver identifier and type, driver version, model family when applicable, model backend, backend engine, model identifier or hosted-model reference, revision or path hash when available, tokenizer hash when available, policy version, prompt-template hash, action-parser version, budget, seed, operating setting, implementation source, and evidence-validation status. Action-level records additionally carry observation hashes, prompt hashes, raw-output hashes, parsed-action hashes, parse status, invalid-action markers, token counts, model latency, backend engine, and policy version. The contract separates traffic from claims. Qwen-backed drivers show that local LLM-backed interaction can be bound to real workloads and audited under the shared schema. Controller drivers show how telemetry can be consumed by downstream decision logic. Calibration and noop drivers test whether evaluators and verifiers discriminate intended outcomes. Bounded sanity drivers support diagnostic or sanity claims only when explicitly admitted; fixture-backed smoke drivers validate onboarding and contract plumbing and remain non-paper-facing. We also separate backend engines from framework-export adapters. A backend engine is a model-serving system used to execute driver traffic, such as vLLM or SGLang (Kwon et al., 2023; Zheng et al., 2024). A framework-export adapter is a downstream format bridge for exporting trajectory and evidence records to external agent-training stacks, such as veRL or OpenRLHF (veRL Project, 2025; Hu and others, 2024). Backend-engine rows test bounded serving compatibility under declared drivers. Framework-export rows test export-format compatibility only; they are not evidence of full RL training, policy improvement, or broad framework generalization. 3.3 Event schema and evidence gate All workload families emit typed event records. Required records include run and episode boundaries, model request start/end, action parsing, environment step start/end, tool calls where applicable, verifier or evaluator outcomes, retries, errors, and terminal episode results. Each record carries run, episode, step, trace, and span identifiers; timing fields such as queue wait, service time, model latency, tool latency, and verifier latency; and provenance fields such as manifest hash, driver identifier, model/backend identifier, schema version, replay class, environment snapshot or image digest, verifier version, and seed. Where applicable, trace propagation follows W3C Trace Context semantics (W3C, 2021), and the emitted fields are designed to interoperate with common LLM observability conventions (OpenTelemetry, 2025; Arize AI, 2024). The evidence gate decides whether a run can support paper-facing quantitative claims. A run is admitted only if it has a resolved manifest, a declared driver, complete trace boundaries, a terminal verifier or evaluator outcome, release-binding metadata, schema-version metadata, replay/freeze metadata, and no fixture-only flag. A run is rejected from paper-facing evidence or quarantined for diagnostic review if driver metadata is missing, the terminal outcome is absent, the trace is incomplete, the manifest is unresolved, or the row is marked smoke-only or fixture-backed. The gate records the rejection reason, such as missing terminal outcome, invalid sample, version mismatch, snapshot mismatch, retry-budget violation, or fixture-only provenance. Smoke and fixture paths remain in the artifact for onboarding and validation, but they are separated from paper-facing evidence. Reported figures and tables are derived only from admitted rows that bind workload, driver, operating setting, telemetry, replay/freeze metadata, and release metadata. 3.4 Replay classes and release-time freeze Replay is family-specific rather than a universal determinization layer. The suite uses three replay classes. R0 denotes summary or statistical replay for lightweight and profiling-oriented workloads. R1 denotes event-trace replay with evaluator freeze, which is appropriate for replay-oriented web workloads. R2 denotes snapshot or manifest replay, which is appropriate for code workloads that require repository state, environment image, and verifier bindings. These classes describe each family’s reproducibility boundary; they are not a hierarchy of task difficulty. At release time, each reported run is bound to a freeze record. The record contains the suite version, task-manifest hash, driver identifier and version, model/backend identifier, prompt-template hash when applicable, action-parser version, environment snapshot or image digest, repository commit where relevant, verifier or evaluator version, schema version, replay-harness version, operating-setting label, and seed policy. This binding allows clean, stressed, diagnostic, and decision-study rows to be compared without losing track of the version boundary that made them admissible. 3.5 Claim-scoped reporting The reporting layer turns admitted event records into claim-scoped outputs rather than a flat run directory. Substrate reports include evidence admission, traceability, declared-driver coverage, model-call latency, token cost, invalid actions, parser outcomes, verifier queue/service fields, replay behavior, and trace completeness. For LLM-backed drivers, the reports also include model latency, token counts, invalid-action rate, action-parse outcomes, and driver-level provenance. Reports are organized by claim scope. The canonical substrate evidence surface establishes the common workload–driver–replay–reporting contract and the evidence-admission claim. Qwen-family driver rows support traffic-cost, token-cost, parse, and invalid-action analyses. Bounded extension and diagnostic rows support model/backend compatibility, failure attribution, verifier calibration, and sanity checks only. Separately admitted controller-study rows use the same telemetry fields as controller inputs or sample-validity signals, but they are not merged into the 930-row canonical substrate evidence surface. This structure distinguishes the suite from a profiler: the release does more than record where time is spent in a single run; it defines which workload, driver, operating setting, replay boundary, and release binding may support each paper claim. Appendix A reports the full compatibility and replay-contract tables. 4 Experimental Protocol 4.1 Workload subsets and execution units The benchmark is evaluated over versioned subsets of the three workload families. Exact task identifiers are bound in manifests rather than prose. Runs are organized as episodes and steps with manifest-resolved environment state, declared-driver metadata, operating setting, backend configuration, replay/freeze metadata, and release binding. This organization makes the execution unit explicit. A reported row is not just a task result: it is a workload–driver–setting run with a manifest, trace boundary, terminal outcome, schema version, replay class, and release root. This allows controlled reruns under different telemetry settings, replay modes, drivers, and operating settings while keeping the evidence status of each row auditable. 4.2 Execution modes Where supported, the suite distinguishes live and replay execution. Live execution preserves environment-side variability and is used for operating realism, bounded sanity checks, or decision-study operating settings. Replay constrains family-supported variation and is used for controlled systems comparisons. In web tasks, replay-oriented WebArena Verified is the clean baseline and replay anchor; live or stressed web operating settings enter paper-facing evidence only when they pass the same manifest, driver, schema, replay/freeze, terminal-outcome, and release-binding checks. This policy keeps execution mode separate from evidence status. A live row is not admitted because it is live, and a replay row is not admitted because it is replayed. Both must pass the same evidence gate before they support paper-facing claims. 4.3 Driver, backend, and observability configurations The protocol is backend-agnostic at the driver-contract level and supports both remote and self-hosted model-serving paths when available. The final admitted evidence is organized around declared driver rows rather than bare model calls. Local Qwen-family drivers cover 1.5B, 7B, 14B, and 30B sizes across the admitted MiniWoB++ and WebArena Verified evidence. Bounded external-model rows include Gemma and Ministral, with paired diagnostic Llama rows where noted. MiniWoB++ contains the widest admitted LLM-driver matrix. Rows from the SWE code slice are interpreted more narrowly. The SWE code slice contains a five-instance patch-provider path for local and external LLM drivers plus oracle/noop calibration rows. The oracle/noop rows validate the verifier path and are not counted as agent patch-provider success. Patch-provider rows therefore support declared-driver traffic and cost evidence, not autonomous SWE-solving claims. Observability is recorded through evidence-stratified telemetry configurations in the release manifests. Real-task anchor rows, Qwen-family driver rows, bounded extension rows, diagnostic rows, and decision-study rows remain separated by their evidence-stratum tags, manifest identifiers, and admission gates. The serving-backend contrast is intentionally bounded: matched vLLM/SGLang rows are used to show that the substrate preserves backend provenance and comparable systems fields, not to claim that one serving engine is universally better. The artifact also contains framework-export adapter rows showing that trajectory and evidence records can be exported to external agent-training stacks. These rows are interpreted as export-format compatibility only, not as full RL training, policy improvement, or broad framework generalization evidence. 4.4 Metrics We report seven metric groups: evidence admission and traceability; declared-driver coverage; model-call and token cost; invalid-action and parse behavior; family-specific bottleneck attribution; replay and telemetry behavior; and focused diagnostic or decision-study outputs. Evidence-admission metrics record whether a row has declared-driver metadata, resolved manifests, complete trace boundaries, terminal outcomes, schema-version metadata, replay/freeze metadata, and release-binding metadata. Driver-aware metrics include model latency, prompt and completion token counts, total tokens, action count, parse status, invalid-action rate, terminal outcome, trace completeness, backend engine, and policy version. Family-specific metrics are interpreted by claim scope. Web rows report browser-side execution and replay/stress behavior. SWE rows report patch-provider status, patch-apply outcome, build/test/verifier time, verifier queue wait, and normalized failure category. MiniWoB++ rows report lightweight interaction, profiling, invalid-action, and sanity behavior. Decision-study rows report controller input telemetry and reward AUC over wall-clock for the admitted WebArena Verified controller grid. Each reported claim is tied to the corresponding admitted evidence surface rather than to a raw run directory. 4.5 Reporting and reproducibility policy All released comparisons are bound to one canonical release root. Report scripts operate on manifest-linked runs and produce family-level tables, plots, evidence-gate reports, and claim matrices from the shared telemetry substrate. Replay class, version bindings, telemetry configuration, repetition counts, operating-setting labels, and evidence-gate decisions are part of the protocol, not hidden implementation details. The 930-row canonical substrate evidence surface and the WebArena Verified controller study are reported separately. Decision-study rows have separate gate reports and claim matrices; the final gates admit 56/56 fixed-budget rows and 336/336 tested-grid rows, with 12/12 comparable vLLM/SGLang × seed × budget cells supporting the clean-versus-medium live-stressed ordering reversal. These decision-study rows are not used to backfill the 930-row canonical substrate evidence surface. 5 Results 5.1 Canonical substrate evidence surface The canonical substrate evidence surface contains the rows used to support the benchmark-substrate claims. From 2,114 indexed rows, the evidence gate admits 930 paper-facing rows and excludes 1,184 rows from quantitative paper evidence. The exclusions are explicit: 913 rows are not marked paper-facing, and 271 generic stress rows are rejected from promoted stress/decision evidence because they do not carry the required decision labels. All three planned evidence strata are present, and the validation report records zero failures. Figure 1 keeps this boundary visible. The 930 admitted rows evaluate the benchmark substrate, declared drivers, and evidence-admission path for the main release. They are separate from the WebArena Verified controller study reported in Section 5.6. The 56-row fixed-budget decision slice and the 336-row budget-grid controller study are admitted by separate decision gates and are not used to backfill the canonical substrate evidence surface or the generic stress rows. The canonical substrate evidence surface contains 224 real-task anchor rows, 640 Qwen-family driver rows, and 66 bounded extension or diagnostic rows. By workload family, it contains 804 MiniWoB++ rows, 81 WebArena Verified substrate rows, 43 rows from the SWE code slice, and 2 framework-export adapter rows. These counts are not used as family weights; they reflect the v1 role split: MiniWoB++ is the high-throughput profiling anchor, WebArena Verified and the SWE code slice provide scoped web, repository, and verifier evidence, and the separate WebArena Verified controller study provides decision-study evidence. Appendix B reports the full model/backend inventory and claim-support matrix. The main claim surfaces are intentionally separated. The canonical 930-row surface supports the substrate and evidence-admission contract, not a leaderboard. The 640 Qwen-family rows support declared-driver traffic, latency, token-cost, parse, and invalid-action evidence, not model capability. SWE controls and recomputed driver rows support verifier validation and traffic/cost evidence, not autonomous SWE-bench solving. The 56-row fixed-budget and 336-row budget-grid controller-study surfaces support the tested-grid clean-versus-medium reversal, not controller universality or backend superiority. Appendix B Claim-support and study evidence tabulates the full claim-support matrix. Figure 1: Canonical evidence admitted by the gate across workload families and evidence strata. 5.2 Declared LLM-driver traffic and cost The 640 admitted Qwen-family rows support declared-driver traffic and cost evidence under the shared contract. They show that contract-conforming LLM-driver traffic can be bound to driver metadata, model-call latency, invalid-action behavior, token cost, patch-generation latency, and evidence-admission status. Terminal outcomes remain in the artifact tables for auditability, while the promoted claims are limited to traffic, cost, parser, invalid-action, and provenance evidence. SWE patch-provider rows are also kept separate from the gold/noop verifier controls: the controls validate the repository and verifier path, whereas the patch-provider rows measure declared-driver traffic and cost. Appendix C reports the per-family Qwen cost tables and visualization. 5.3 Bounded backend, model, and framework compatibility The bounded compatibility rows test whether the same driver/evidence contract can admit additional serving backends, external model families, and framework-export formats. vLLM (Kwon et al., 2023) and SGLang (Zheng et al., 2024) are serving backend engines. Gemma, Ministral, and Llama rows provide bounded external-model admission and diagnosis. veRL and TRL rows are framework-export adapter rows. These rows are interpreted as bounded compatibility and provenance evidence under declared drivers. The vLLM/SGLang rows show serving-backend-compatible traffic under admitted metadata. The external-model rows show bounded model-family admission and diagnosis. The veRL/TRL rows show export-format compatibility only; they are not evidence of full RL training, policy improvement, or broad framework generalization. Appendix D contains the full extension-cell table. 5.4 Failure attribution and stronger-driver sanity We next ask whether low natural success rates hide a substrate or evidence-chain failure. In the audited natural non-success evidence, we do not observe such a failure. MiniWoB++ non-success is locally attributable to loop/no-progress and parser/interface patterns, while WebArena Verified non-success labels remain too coarse-grained to support a promoted semantic-failure prevalence claim. A bounded stronger-driver sanity check outside the canonical substrate evidence surface produces successful MiniWoB++ and WebArena Verified trajectories with complete evidence validation. This is not a paired agent-comparison or statistically significant success-improvement claim; it only shows that the substrate can carry successful trajectories under a declared stronger driver. Appendices E and G report the audit and per-family counts. 5.5 Verifier and stress diagnostics The final diagnostics bound the interpretation without expanding the claim scope. For SWE, the official-harness controls validate the verifier-calibration path: gold/oracle controls pass all five checked instances and noop controls fail all five. The resulting SWE driver aggregates remain traffic/cost evidence only, not agent patch-success or autonomous SWE-bench-solving evidence. The WebArena stress diagnostic is a bounded operating-setting check, not broad web generalization, backend superiority, or successful stress-setting task performance. Controller-choice evidence is admitted by separate decision gates and reported as decision-study evidence in Section 5.6, not merged into the 930-row canonical substrate evidence surface. 5.6 Decision-study evidence changes the systems decision Finally, we test whether admitted telemetry can change a controller-choice systems decision. The WebArena Verified controller study is a downstream telemetry-consumption study over a fixed two-variant controller grid. For traceability, Table 1 retains the artifact identifiers hook_a_only and hook_b_only; in prose, we refer to them as Hook A and Hook B. Hook A is a sample-validity and staleness filter: a sample is dropped if it lacks a terminal verifier outcome, carries an invalid-sample marker from the harness, has a version or snapshot mismatch when such metadata are present, or exceeds the configured retry budget. Hook B is an adaptive concurrency and verifier-scheduling hook: it monitors a rolling telemetry window and adjusts actor concurrency when verifier-side queue pressure crosses the configured threshold. The study does not introduce either hook as a new controller algorithm. It asks a narrower evaluation question: do admitted clean and medium live-stressed evidence surfaces make the same selection over a fixed controller-variant set? Table 1 reports the exact reward-AUC values for the fixed-budget slice. All 56 merged rows are admitted and none are blocked. At budget=7 and seed=0, clean-baseline evaluation selects Hook A under both backends, while the promoted medium live-stressed operating setting selects Hook B under both backends. The budget-grid study admits all 336 planned rows and blocks none; all 12 comparable vLLM/SGLang × seed × budget cells reverse the clean-versus-medium ordering. Thus, within the released contract and admitted decision-study evidence, clean-only evaluation can mis-rank controller choice. Mechanistically, the medium live-stressed surface gives greater weight to queue- and tail-sensitive operating costs, whereas the clean surface favors the sample-validity signal. The scope remains tested-grid only: this is not a new controller algorithm, policy-SOTA result, backend-superiority claim, or universal controller-robustness claim. Table 1: Separately admitted WebArena Verified decision-study evidence. The fixed-budget slice admits 56/56 rows; the budget-grid study admits 336/336 planned rows and has 12/12 clean-versus-medium-live-stressed reversals over vLLM/SGLang × seeds 0/1 × budgets 5/7/9. Backend Setting hook_a_only hook_b_only Selected vLLM clean 0.051875 0.045106 hook_a_only vLLM medium 0.035093 0.044943 hook_b_only SGLang clean 0.052251 0.045242 hook_a_only SGLang medium 0.035086 0.045132 hook_b_only 6 Artifact Release, Responsible Use, and Limitations The release provides code, adapters, manifests, validators, telemetry/report scripts, evidence-gate reports, figure/table inputs, claim maps, and release metadata for inspecting admitted evidence. Public artifacts follow a minimum-privilege, redaction-by-default policy: plaintext credentials, API keys, access tokens, cookies, raw browser sessions, storage-state exports, external-service headers, unredacted request/response logs, and credential-bearing remote-service records are excluded or represented by hashes, manifests, validation status, and redaction metadata. The v1 release is scoped to the admitted substrate and separately admitted WebArena Verified decision-study evidence; it does not establish a new agent policy, model/backend leaderboard, broad task generalization, full RL training, autonomous SWE-bench solving, or robustness outside the evaluated task slice, controller variants, operating settings, backends, seeds, and budgets. References R. Agashe et al. (2024) AppWorld: A Controllable World of Apps and People. Note: https://aclanthology.org/2024.acl-long.850.pdfProject: https://appworld.dev/ Cited by: G Stronger-driver sanity check, §1. Arize AI (2024) OpenInference Specification. Note: https://arize-ai.github.io/openinference/spec/ Cited by: G Stronger-driver sanity check, §3.3. T. L. S. de Chezelles, M. Gasse, A. Drouin, M. Caccia, L. Boisvert, M. Thakkar, T. Marty, R. Assouel, S. O. Shayegan, L. K. Jang, X. H. Lù, O. Yoran, D. Kong, F. F. Xu, S. Reddy, G. Neubig, Q. Cappart, R. Salakhutdinov, N. Chapados, and A. Lacoste (2025) The browsergym ecosystem for web agent research. Transactions on Machine Learning Research. Cited by: G Stronger-driver sanity check, §1, §2.2. A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. Del Verme, T. Marty, L. Boisvert, M. Thakkar, Q. Cappart, D. Vazquez, N. Chapados, and A. Lacoste (2024) WorkArena: how capable are web agents at solving common knowledge work tasks?. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 11282–11314. Cited by: G Stronger-driver sanity check, §1. H. Hu et al. (2024) OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework. Note: https://arxiv.org/abs/2405.11143Code: https://github.com/openrlhf/openrlhf Cited by: G Stronger-driver sanity check, §3.2. C. E. Jimenez, J. Yang, A. Wettig, K. Lieret, S. Yao, K. Pei, O. Press, and K. R. Narasimhan (2024) SWE-bench: can language models resolve real-world github issues?. In The Twelfth International Conference on Learning Representations, Cited by: G Stronger-driver sanity check, §1, §2.2. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with pagedattention. arXiv preprint arXiv:2309.06180. Note: SOSP 2023 Cited by: D Model/backend extension, G Stronger-driver sanity check, §3.2, §5.3. I. Levy, B. Wiesel, S. Marreed, A. Oved, A. Yaeli, and S. Shlomov (2024) ST-webagentbench: a benchmark for evaluating safety and trustworthiness in web agents. arXiv preprint arXiv:2410.06703. Cited by: G Stronger-driver sanity check, §1. E. Z. Liu, K. Guu, P. Pasupat, T. Shi, P. Shah, M. Lewis, D. Marcu, and P. Liang (2018) Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations, Cited by: G Stronger-driver sanity check, §1, §2.2. X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, et al. (2023) AgentBench: evaluating LLMs as agents. Note: https://arxiv.org/abs/2308.03688 Cited by: G Stronger-driver sanity check, §1. OpenTelemetry (2025) OpenTelemetry Semantic Conventions for Generative AI Systems. Note: https://opentelemetry.io/docs/specs/semconv/gen-ai/ Cited by: G Stronger-driver sanity check, §3.3. A. Q. Pan, Z. Tang, S. Liu, Y. Wang, P. Pasupat, E. Z. Liu, A. Gu, G. Neubig, W. Yih, H. Ji, W. Y. Wang, Y. Su, and Y. Wang (2025) Training software engineering agents and verifiers with SWE-Gym. In Proceedings of the 42nd International Conference on Machine Learning, Cited by: G Stronger-driver sanity check, §1, §2.2. Qwen Team (2024) Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Cited by: G Stronger-driver sanity check, §3.2. T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom (2023) Toolformer: language models can teach themselves to use tools. Advances in Neural Information Processing Systems 36, p. 68539–68551. Cited by: G Stronger-driver sanity check, §1. ServiceNow Research (2024) AgentLab. Note: https://github.com/ServiceNow/AgentLab Cited by: G Stronger-driver sanity check, §1. ServiceNow Research (2026) WebArena Verified. Note: https://github.com/ServiceNow/webarena-verifiedVerified release of the WebArena benchmark with captured traces and deterministic evaluators Cited by: G Stronger-driver sanity check, §1, §2.2. Sierra Research (2025) Tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Note: https://sierra.ai/-/cdn/document?src=https%3A%2F%2Fcdn.sanity.io%2Ffiles%2Fca4jck6w%2Fproduction%2F24362e7e6f296c9b160287f46bba258223ab555.pdf Cited by: G Stronger-driver sanity check, §1. A. D. Tur, N. Meade, X. H. Lù, A. Zambrano, A. Patel, E. Durmus, S. Gella, K. Stańczak, and S. Reddy (2025) SafeArena: evaluating the safety of autonomous web agents. arXiv preprint arXiv:2503.04957. Cited by: G Stronger-driver sanity check, §1. veRL Project (2025) veRL. Note: https://github.com/verl-project/verlMonitoring docs: https://verl.readthedocs.io/en/latest/advance/grafana_prometheus.html Cited by: G Stronger-driver sanity check, §3.2. W3C (2021) W3C Trace Context. Note: https://w.w3.org/TR/trace-context/ Cited by: G Stronger-driver sanity check, §3.3. Z. Wei, W. Yao, Y. Liu, W. Zhang, Q. Lu, L. Qiu, C. Yu, P. Xu, C. Zhang, B. Yin, H. Yun, and L. Li (2025) WebAgent-r1: training web agents via end-to-end multi-turn reinforcement learning. arXiv preprint arXiv:2505.16421. Cited by: G Stronger-driver sanity check, §1. T. Xie et al. (2024) OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. Note: https://arxiv.org/abs/2404.07972Project: https://os-world.github.io/; Code: https://github.com/xlang-ai/OSWorld Cited by: G Stronger-driver sanity check, §1. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2023) ReAct: synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, Cited by: G Stronger-driver sanity check, §1. L. Zheng, L. Yin, Z. Xie, C. L. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y. Sheng (2024) SGLang: efficient execution of structured language model programs. arXiv preprint arXiv:2312.07104. Note: NeurIPS 2024 Cited by: D Model/backend extension, G Stronger-driver sanity check, §3.2, §5.3. S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, S. Sridhar, X. Cheng, Y. Bisk, D. Fried, and U. Alon (2024) WebArena: a realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, Cited by: G Stronger-driver sanity check, §1, §1. A Implementation and reproducibility Release binding. This paper is bound to a versioned release and its manifest-linked runs. The exact repository commit is recorded in the release manifest rather than repeated in the PDF. All figures and tables in the paper derive from this release root and its manifest-linked runs. Version metadata. Manifest version: v1. Schema version: 1.0.0. Suite version: 0.1.0. Replay-harness version: 0.1.0. Bound metadata. In addition to the identifiers above, the release records the task-manifest hash, declared driver identifier, driver type and version, model-backend identifier where applicable, prompt-template hash where applicable, action-parser version, environment image digest or snapshot identifier, repository commit where relevant, browser/container version where relevant, verifier/evaluator version, operating-setting label, and seed policy. Evidence-gate metadata. Paper-facing rows are admitted only when the run has a resolved manifest, declared driver, complete trace boundaries, terminal verifier or evaluator outcome, schema-version metadata, replay/freeze metadata, and no fixture-only flag. Smoke-only and fixture-backed artifacts are retained for onboarding and contract validation, but are tracked separately from the paper-facing evidence. Table A1: Workload families, declared drivers, and claim scope in v1. The SWE code slice uses SWE-Gym-style executable repository tasks with SWE-bench-compatible test-based verification. Family Drivers Paper-facing role Not claimed WebArena Verified Qwen-backed LLM drivers; controller drivers; bounded sanity/extension drivers Web anchor; browser-side execution evidence; bounded stressed settings; separate controller-choice study web-agent leaderboard; broad web generalization; backend superiority; universal controller robustness SWE code slice Calibration controls; Qwen patch-provider drivers; bounded extension drivers Verifier/repository anchor; patch-generation traffic and cost evidence; verifier-control calibration Autonomous SWE-bench solving; broad SWE failure diversity; patch-success leaderboard MiniWoB++ Reference/scripted drivers; Qwen-backed LLM drivers; bounded sanity/extension drivers Lightweight profiling anchor; high-throughput browser interaction; non-degenerate sanity trajectories Task-solving leaderboard; broad MiniWoB++ generalization; LLM web-agent ceiling Repeated-run summaries. The appendix retains the repeated clean-baseline and replay-anchor summaries that support reproducibility, variance inspection, and report regeneration checks. Table A2: Repeated concurrency summary for the clean full rerun. Family Conc. Throughput (eps/s) p50 (ms) p95 (ms) p99 (ms) Pass rate MiniWoB++ 1 57.2 ± 26.6 32.5 32.5 32.5 1.00 MiniWoB++ 4 73.9 ± 2.2 13.3 14.0 14.0 1.00 MiniWoB++ 8 68.4 ± 5.2 14.1 18.0 18.0 1.00 SWE code slice 1 2.29 ± 0.08 194.8 194.8 194.8 1.00 SWE code slice 4 8.51 ± 1.58 230.4 244.2 244.2 1.00 SWE code slice 8 17.49 ± 1.88 176.8 196.3 196.3 1.00 WebArena Verified 1 3.23 ± 0.25 71.6 71.6 71.6 1.00 WebArena Verified 4 12.33 ± 1.94 76.8 80.2 80.2 1.00 WebArena Verified 8 22.26 ± 4.73 74.9 85.9 85.9 1.00 Table A3: Replay-anchor summary from the current paper-facing release evidence. Family Replay class Live mean ± std (ms) Replay mean ± std (ms) Mean reduction WebArena Verified R1 7038.49 ± 436.08 6.45 ± 0.25 99.91% A.1 Compatibility and replay contracts Table A4: Common benchmark contract and family-specific freedom. Layer Required contract Family-specific freedom Task manifest family, task id, environment or snapshot reference, reset contract, verifier/evaluator id, adapter version, replay class, schema version task-specific parameters and environment metadata Adapter API resolve manifest, initialize environment, execute step, validate or replay, emit artifacts observation/action spaces and evaluator implementations Driver driver id/type/version, model/backend id when applicable, prompt-template hash, parser version, budget, seed, operating-setting label LLM, controller, calibration, sanity, or scripted implementation Event records run/episode/step ids, model/tool/env/verifier events, timing, queue wait, retry, error, provenance, version tags additional family-specific telemetry fields Replay/freeze R0/R1/R2 replay class, manifest hash, snapshot/image/repository binding, verifier version, schema version, replay-harness version family-specific replay boundary Report outputs executability, latency decomposition, replay behavior, telemetry overhead, outcome/failure normalization, trace completeness family-specific attribution of bottlenecks and errors Table A5: Replay boundary by workload family. Family Primary replay class Frozen input material MiniWoB++ R0 summary statistics and profiling-oriented collector bindings WebArena Verified R1 event trace, evaluator freeze, and bound web-session configuration SWE code slice R2 manifest metadata, repository or snapshot binding, and verifier freeze B Claim-support and study evidence Appendix B is the claim-audit index for the final release. It records the admitted model/backend inventory, the paper-facing claim-support matrix, and the separately admitted WebArena Verified decision-study evidence. The tables in this appendix are not additional benchmark claims; they define which admitted evidence surfaces support which statements in the main text, and which surfaces remain bounded, caveated, appendix-only, separated, or outside scope. Table B1: Appendix claim map for admitted evidence surfaces and scoped paper claims. Admitted evidence Size / cells Admission / validation Main quantitative finding Paper claim Canonical substrate evidence 930 admitted / 2,114 indexed 1,184 excluded; 0 validation failures 224 anchors, 640 Qwen driver rows, 66 bounded extension or diagnostic rows evidence-gated executable substrate Family coverage 804 MiniWoB++, 81 WebArena, 43 SWE code-slice rows, 2 adapters all paper-facing rows admitted by gate cross-family workload / driver / replay / reporting contract substrate coverage, not leaderboard breadth Qwen-family driver traffic 640 rows declared-driver metadata present latency, invalid actions, token / patch cost traffic/cost evidence, not capability Failure-attribution audit 685 non-success rows + audited slices no substrate/evidence-chain failure; WebArena labels caveated MiniWoB++ loop/no-progress and parser/interface claims supported caveated attribution, not taxonomy Stronger-driver sanity 45 episodes complete validation; separate evidence successful MiniWoB++ / WebArena trajectories sanity only, not leaderboard SWE verifier-control study 10 controls + 31 SWE rows gold 5/5 pass, noop 5/5 fail; no substrate/evaluator bug SWE rows recomputed as traffic/cost evidence verifier path validated, not autonomous SWE solving WebArena Verified controller study 56 fixed-budget + 336 tested-grid rows 56/56 and 336/336 admitted; 0 blocked 12/12 vLLM/SGLang × seeds 0/1 × budgets 5/7/9 cells reverse ordering supports the tested-grid decision-study claim B.1 Model/backend inventory and claim-support matrix Table B2: Model/backend inventory over the admitted paper-facing evidence. Qwen inventory counts include bounded extension cells; the promoted Qwen-traffic claim surface is the 640-row local-driver subset reported in Table B3. Model family Backend Paper-facing rows Families covered Role Qwen vLLM 652 MiniWoB++, SWE code slice, WebArena main local LLM traffic + bounded stress Qwen SGLang 9 MiniWoB++, SWE code slice, WebArena bounded backend-compatibility cells Gemma vLLM 15 MiniWoB++, SWE code slice, WebArena external-model compatibility Ministral vLLM 15 MiniWoB++, SWE code slice, WebArena external-model / parser-stress Llama vLLM 3 WebArena paired-stress diagnostic non-model/control unknown 236 MiniWoB++, SWE code slice, WebArena real-task anchors + controls Table B3: Claim-support matrix. Claim Status Rows Scope Evidence-gated executable substrate supported 930 final canonical paper-facing evidence Real-task anchors supported 224 real WebArena / SWE code slice / MiniWoB++ anchors Qwen-family traffic supported 640 local Qwen-family LLM driver rows Model/backend extension supported, bounded 45 external model / backend compatibility Framework export adapters appendix only 2 export-format demos only; not RL-training, policy-improvement, or framework-generalization evidence SWE verifier controls supported 10 controls official-harness study: gold 5/5 pass, noop 5/5 fail; no confirmed substrate/evaluator bug SWE driver aggregates recomputed, traffic-only 31 SWE rows 20 patch-provider traffic/cost, 10 verifier-control guardrails, 1 aggregate summary; not SWE-solving evidence WebArena stress supported, bounded 9 p99 operating-point diagnostic only Separate controller study: fixed-budget slice supported, fixed-budget 56 merged rows separate admitted controller-study rows; 56/56 admitted, 0 blocked; budget=7, seed=0; clean versus medium live-stressed reversal across vLLM and SGLang Separate controller study: tested-grid study (5/7/9) supported, tested-grid 336 planned rows separate admitted controller-study rows; 336/336 admitted, 0 blocked; frozen task_ids 105, 106, 124, 125, 142, 143, 149, 156, 163; 12/12 comparable backend × seed × budget cells over vLLM/SGLang, seeds 0/1, and budgets 5/7/9 show the same reversal Failure attribution audit caveated 685 non-success + audited slices no observed substrate/evidence failure; WebArena semantic labels remain caveated Stronger-driver sanity supported, appendix 45 episodes separate evidence of successful trajectories B.2 Decision-sensitive admission audit The WebArena Verified controller studies are admitted by decision-specific gates and reported separately from the 930-row canonical substrate evidence surface. Their purpose is to support a tested-grid systems conclusion about controller choice, not to characterize the whole benchmark substrate. The two artifact identifiers, hook_a_only and hook_b_only, denote fixed single-hook controller variants over the same telemetry substrate: Hook A filters by sample-validity and staleness conditions, while Hook B adapts concurrency using verifier-side queue-pressure telemetry. Accordingly, decision rows are not counted in the canonical substrate evidence surface and are not used to backfill generic stress rows. Table B4: Decision-sensitive admission status. Audit item Fixed-budget slice Budget-grid study (5/7/9) Merged or planned rows 56 merged rows 336 planned rows Admitted rows 56 336 Blocked rows 0 0 Supported claim level supports the claim supports the tested-grid controller-study claim Task slice released fixed-budget WebArena Verified controller slice frozen WebArena Verified nine-task slice: 105, 106, 124, 125, 142, 143, 149, 156, 163 Budgets budget=7 budgets 5, 7, and 9, as recorded in the study claim matrix Seeds seed=0 seeds 0 and 1, as recorded in the study claim matrix Backends vLLM, SGLang vLLM, SGLang Operating settings clean, medium live-stressed clean, medium live-stressed Comparable cells 2 backend cells 12 backend × seed × budget cells over vLLM/SGLang × seeds 0/1 × budgets 5/7/9 Ordering result clean selects hook_a_only; medium live-stressed selects hook_b_only same reversal in 12/12 comparable cells Table B5: Secondary controller-study clauses and support status. Clause Status Evidence basis Fixed-budget reversal Supported 56/56 admitted, 0 blocked; budget=7, seed=0; vLLM and SGLang both reverse controller ordering between the clean setting and the medium live-stressed setting Tested-grid reversal Supported 336/336 admitted, 0 blocked; all 12 comparable backend × seed × budget cells over vLLM/SGLang, seeds 0/1, and budgets 5/7/9 reverse controller ordering Frozen nine-task study Supported task_ids 105, 106, 124, 125, 142, 143, 149, 156, and 163 Untested budgets, seeds, models, task slices, or stress settings Not claimed the controller-study claim is limited to the admitted fixed-budget slice and tested-grid study over budgets 5, 7, and 9 Universal controller robustness Not claimed the evidence supports a tested-grid systems decision claim, not a universal controller theorem Accordingly, the main text claims a tested-grid controller-choice reversal on the admitted WebArena Verified controller-study rows for backends vLLM and SGLang, seeds 0 and 1, and budgets 5, 7, and 9. It does not claim controller universality, backend superiority, cross-model robustness, or robustness outside the admitted budgets, seeds, task slices, backends, and operating settings. B.3 Diagnostic evidence index Table B6: Diagnostic evidence index for bounded, caveated, appendix-only, and separated evidence surfaces. Diagnostic/study item Status Rows Interpretation Failure attribution audit caveated 685 non-success + audited slices no observed substrate/evidence failure; WebArena semantic labels remain caveated Stronger-driver sanity supported, appendix 45 episodes successful MiniWoB++ and WebArena trajectories with complete validation; separate evidence SWE verifier controls supported 10 controls official-harness study: gold 5/5 pass, noop 5/5 fail; no confirmed substrate/evaluator bug SWE driver evidence set recomputed, traffic-only 31 SWE rows 20 patch-provider traffic/cost, 10 verifier-control guardrails, 1 aggregate summary WebArena stress supported, bounded 9 p99 operating-point diagnostic only Separate controller study: fixed-budget slice supported, fixed-budget 56 merged rows separate admitted controller-study rows; 56/56 admitted, 0 blocked; budget=7, seed=0; clean versus medium live-stressed controller reversal across vLLM and SGLang Separate controller study: tested-grid study (5/7/9) supported, tested-grid 336 planned rows separate admitted controller-study rows; 336/336 admitted, 0 blocked; frozen nine-task WebArena Verified slice; same reversal in 12/12 comparable cells over vLLM/SGLang × seeds 0/1 × budgets 5/7/9 Framework export adapters appendix only 2 export-format demos only; not RL-training, policy-improvement, or framework-generalization evidence C Qwen traffic details This appendix records the declared Qwen-family traffic used for traffic-cost and evidence-admission auditing. The tables retain terminal outcomes for provenance, but those outcomes are not interpreted as a model leaderboard, WebArena-solving result, or autonomous SWE-solving result. The supported quantities are model-call latency, invalid-action behavior, patch-generation latency, token cost, and evidence admission. Figure C1: Qwen-family driver cost summary. Table C1: MiniWoB++ Qwen-family traffic retained for traffic-cost auditing. Model Rows Mean model latency (ms) P95 model latency (ms) Invalid action rate Terminal success (audit only) Qwen2.5-1.5B 144 65.56 108.88 0.000 0 Qwen2.5-7B 144 173.69 271.32 0.000 0 Qwen2.5-14B 144 322.71 785.89 0.076 0 Qwen3-30B-A3B 144 118.64 191.16 0.000 0 Table C2: WebArena Verified Qwen-family traffic retained for traffic-cost auditing. Model Rows Mean model latency (ms) P95 model latency (ms) Invalid action rate Terminal success (audit only) Qwen2.5-1.5B 11 299.44 674.80 0.545 0 Qwen2.5-7B 11 610.03 762.69 0.000 0 Qwen2.5-14B 11 1236.06 1919.00 0.000 0 Qwen3-30B-A3B 11 393.12 782.50 0.091 0 Table C3: Qwen-family patch-provider traffic from the SWE code slice retained for traffic-cost auditing. Model Rows Mean patch-gen. latency (ms) P95 patch-gen. latency (ms) Mean total tokens Patch apply success (audit only) Passed (audit only) Qwen2.5-1.5B 5 1307.12 1857.75 346.6 0 0 Qwen2.5-7B 5 6842.03 13400.93 409.6 0 0 Qwen2.5-14B 5 7146.85 8031.86 359.0 0 0 Qwen3-30B-A3B 5 3967.22 6107.48 474.6 0 0 D Model/backend extension This appendix records bounded model/backend extension cells and framework-export adapter rows. These rows test whether the declared driver/evidence contract can admit and diagnose additional serving backends, external model families, and export formats. They are not a matched backend benchmark, a backend-superiority claim, a cross-model robustness claim, or evidence of RL training. Table D1: Bounded backend and external-model extension records. Evidence set Rows Main numbers Interpretation MiniWoB++ Qwen2.5-1.5B vLLM extension 6 mean / p95 latency 129.79 / 134.17 ms; invalid 0 compatibility reference MiniWoB++ Qwen2.5-1.5B SGLang extension 6 mean / p95 latency 104.75 / 119.55 ms; invalid 0 bounded SGLang-compatible cell MiniWoB++ Gemma vLLM 12 mean / p95 latency 246.04 / 306.57 ms; invalid 0 external-model compatibility MiniWoB++ Ministral vLLM 12 mean / p95 latency 441.57 / 516.46 ms; invalid 1.0 parser-stress / external-model diagnostic WebArena Qwen SGLang 2 mean / p95 latency 1099.20 / 1125.39 ms; success 0 bounded SGLang-compatible web rows WebArena Gemma vLLM 2 mean / p95 latency 2092.92 / 3347.15 ms; success 0 bounded external web rows WebArena Ministral vLLM 2 mean / p95 latency 888.37 / 1050.59 ms; invalid 1.0 parser-stress diagnostic SWE code slice Qwen SGLang 1 patch latency 15805.82 ms; patch apply true; passed false bounded backend-compatible patch row SWE code slice Gemma vLLM 1 patch latency 4926.52 ms; patch apply true; passed false external patch-provider row SWE code slice Ministral vLLM 1 patch latency 1580.95 ms; patch apply false; passed false external patch-provider row Table D2: Framework-export adapter rows for export-format compatibility auditing. Adapter row Rows Interpretation veRL export demo 1 Export-format compatibility only; not RL or policy-improvement evidence TRL export demo 1 Export-format compatibility only; not RL or policy-improvement evidence These rows are intentionally separated from backend-engine compatibility cells. vLLM [Kwon et al., 2023] and SGLang [Zheng et al., 2024] are serving backend engines used to execute traffic under declared drivers; veRL and TRL are downstream framework export adapters. The two rows in Table D2 only test whether admitted trajectory/evidence records can be exported in formats consumable by those downstream frameworks. They do not run RL training, do not measure policy improvement, and do not establish broad compatibility across RL or agent-training stacks. E Failure attribution QC This appendix records the targeted non-success attribution audit. The audit asks whether observed non-success is explained by a hidden substrate or evidence-chain failure. It is not a complete natural-failure taxonomy and does not estimate global semantic-failure prevalence. Table E1: Failure-attribution QC audit summary. Audited slice Rows Observed status Interpretation MiniWoB++ audited non-success slices audited slices loop/no-progress and parser/interface failures supported; no substrate/evidence-chain failure observed supported local attribution slice WebArena Verified non-success slice 50 9 parser/interface rows; 37 semantic label-granularity caveats; 4 insufficient-trace rows; no substrate/evidence-chain failure observed global attribution remains caveated rather than promoted as a complete taxonomy Interpretation boundary. The supported conclusion is that the audited surface contains no observed substrate/evidence-chain failure. MiniWoB++ loop/no-progress and parser/interface patterns are locally supported in audited slices. WebArena Verified semantic labels remain too coarse-grained to support a promoted natural semantic-failure prevalence claim. F SWE verifier-control repair and driver evidence recompute This appendix records the SWE verifier-control repair and the resulting driver evidence recompute. The repair stages the official test patch before applying the oracle or noop patch. Under the repaired official harness, gold/oracle controls pass on all five checked instances and noop controls fail on all five instances. This validates the control-calibration path and attributes the earlier control failure to a harness-invocation bug in the control path, not to a confirmed substrate/evaluator bug. The affected SWE driver evidence is recomputed as traffic/cost evidence: 20 patch-provider rows, 10 verifier-control guardrail rows, and one aggregate summary row. Control rows are calibration-only and not eligible for the canonical substrate evidence surface. This supports the SWE-Gym slice with SWE-bench-compatible verification as an executable verifier/repository anchor, but it does not convert LLM patch-provider rows into autonomous SWE-solving evidence. Table F1: SWE verifier-control repair and driver evidence recompute record. Item Current status after verifier-control repair and driver evidence recompute Interpretation Official harness availability yes official-compatible verifier path available Gold/oracle positive controls 5/5 pass positive-control path supported Noop negative controls 5/5 fail as expected negative-control path supported Root cause of previous control failures harness_invocation_bug repaired by staging official test_patch before oracle/noop patch Confirmed substrate/evaluator bug no no substrate/evaluator bug confirmed Affected rows recomputed 31 recomputed under repaired invocation path Affected rows excluded 0 no exclusion-based study remains Recomputed SWE-only evidence set 31 rows 20 patch-provider traffic/cost, 10 guardrails, 1 aggregate summary Control rows calibration-only; not eligible for the canonical substrate evidence surface guardrails, not task-solving evidence SWE patch-provider rows traffic/cost evidence not autonomous SWE-bench solving G Stronger-driver sanity check This appendix reports the bounded stronger-driver sanity check. The check is not part of the canonical surface and is not a paired agent-comparison experiment. Its only role is to show that the same evidence-gated substrate can carry non-degenerate successful MiniWoB++ and WebArena Verified trajectories with complete evidence validation. This sanity slice is not part of the canonical surface and is non-comparative. It is not a paired comparison between the admitted Qwen driver evidence set and another agent, and it is not used to estimate statistical significance of success-rate improvement. Its purpose is narrower: to rule out the interpretation that low natural success under the admitted weak-driver traces is caused by an inability of the substrate, validators, or evidence gate to admit successful trajectories. Table G1: Stronger-driver sanity-check summary. Family Episodes Successes Unique tasks solved Evidence pass Canonical-surface eligible MiniWoB++ 36 18 6 36/36 0 WebArena Verified 9 9 3 9/9 0 SWE code slice skipped — — — 0

Instant research discovery