Paper deep dive

ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents

Tianchen Guan, Xinlei Lin, Royce Cheng-Yue, Xiangjun Wang, Shuyan Zhou

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 82

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 8/20/2026, 4:20:17 AM

Summary

The paper introduces ComponentBench, a benchmark and diagnostic pipeline for evaluating computer-use agents at the component level on modern web UIs. It addresses the gap between long-horizon workflow benchmarks and atomic GUI-grounding tests by providing 2,910 programmatically verified tasks across 97 canonical UI components. The study evaluates seven models (including GPT-5.4, Gemini 3, Qwen3-VL, and UI-TARS) across four observation/action spaces (AX-tree, Set-of-Marks, Pixel, Browser-Use), revealing that interface design significantly impacts performance, with pass rates shifting by over 30% for the same model. Key findings include that agents often fail to match human efficiency, taking up to 3.7x longer, and struggle with spatial manipulations like sliders and drag-and-drop.

Entities (13)

ComponentBench · benchmark · 100%COLM 2026 · conference · 95%Gemini 3 Flash · model · 95%GPT-5-mini · model · 95%Qwen3-VL-235B · model · 95%UI-TARS-1.5-7B · model · 95%Ant Design · ui-library · 90%AX-tree · observation-space · 90%Browser-Use · observation-space · 90%Mantine · ui-library · 90%MUI · ui-library · 90%Pixel · observation-space · 90%Set-of-Marks · observation-space · 90%

Relation Signals (9)

ComponentBench → contains → 2,910 tasks

confidence 95% · ComponentBench is organized around a library-agnostic ontology of 97 canonical UI components instantiated as 2,910 programmatically verified tasks

ComponentBench → evaluates → Computer Use Agents

confidence 95% · ComponentBench is a benchmark and diagnostic pipeline for component-level evaluation of computer-use agents on modern web UIs.

ComponentBench → publishedat → COLM 2026

confidence 95% · Published as a conference paper at COLM 2026

UI-TARS-1.5-7B → evaluatedin → Pixel

confidence 90% · UI-TARS-1.5-7B... evaluated through its own native pixel-only harness

GPT-5-mini → performsbetterin → AX-tree

confidence 90% · GPT-5 mini falls from 83.1% with accessibility-tree observations to 48.9% with coordinate-only Pixel control.

GPT-5-mini → performsworsein → Pixel

confidence 90% · GPT-5 mini falls from 83.1% with accessibility-tree observations to 48.9% with coordinate-only Pixel control.

ComponentBench → uses → Ant Design

confidence 90% · For implementation we select Ant Design, MUI, and Mantine

ComponentBench → uses → MUI

confidence 90% · For implementation we select Ant Design, MUI, and Mantine

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Current evaluation of computer-use agents is split between long-horizon workflow benchmarks and atomic GUI-grounding tests. This leaves an under-instrumented middle layer: realistic component-centered interactions (e.g., toggle a button set) that are short enough to diagnose and rich enough to capture the burdens of modern interfaces. We present ComponentBench, a benchmark and diagnostic pipeline for component-level evaluation of computer-use agents on modern web UIs. ComponentBench is organized around a library-agnostic ontology of 97 canonical UI components instantiated as 2,910 programmatically verified tasks across widely used component libraries, paired with cleaned human reference trajectories that enable evaluation of both task success and interaction efficiency. Beyond task collection, we introduce a scalable pipeline for auditing realized structural difficulty after implementation and synthesizing structured failure analyses across tasks and component families. Evaluating seven models -- GPT-5.4, Gemini 3 Flash, GPT-5.4 mini, GPT-5 mini, Gemini 3.1 Flash-Lite, Qwen3-VL-235B, and UI-TARS-1.5-7B -- across four observation and action spaces, we show that these design choices critically impact performance. Within a single shared harness, changing only the observation and action space shifts task success by more than 30% for the same model: GPT-5 mini falls from 83.1% with accessibility-tree observations to 48.9% with coordinate-only Pixel control. Moreover, even the fastest configuration takes 3.7x as long as the matched human reference, and spatial manipulations that are trivial for humans continue to challenge current agents.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

78,532 characters extracted from source content.

Expand or collapse full text

Published as a conference paper at COLM 2026 ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents Tianchen Guan 1 Xinlei Lin 2 Royce Cheng-Yue 2 Xiangjun Wang 2 Shuyan Zhou 1 tianchen.guan, shuyan.zhou@duke.edu 1 Duke University 2 Amazon AGI SF Lab Abstract Current evaluation of computer-use agents is split between long-horizon workflow benchmarks and atomic GUI-grounding tests. This leaves an under-instrumented middle layer: realistic component-centered interac- tions (e.g., toggle a button set) that are short enough to diagnose and rich enough to capture the burdens of modern interfaces. We present Com- ponentBench, a benchmark and diagnostic pipeline for component-level evaluation of computer-use agents on modern web UIs. ComponentBench is organized around a library-agnostic ontology of 97 canonical UI compo- nents instantiated as 2,910 programmatically verified tasks across widely used component libraries, paired with cleaned human reference trajecto- ries that enable evaluation of both task success and interaction efficiency. Beyond task collection, we introduce a scalable pipeline for auditing re- alized structural difficulty after implementation and synthesizing struc- tured failure analyses across tasks and component families. Evaluating seven models—GPT-5.4, Gemini 3 Flash, GPT-5.4 mini, GPT-5 mini, Gem- ini 3.1 Flash-Lite, Qwen3-VL-235B, and UI-TARS-1.5-7B—across four ob- servation and action spaces, we show that these design choices critically impact performance. Within a single shared harness, changing only the observation and action space shifts task success by more than 30% for the same model: GPT-5 mini falls from 83.1% with accessibility-tree obser- vations to 48.9% with coordinate-only Pixel control. Moreover, even the fastest configuration takes 3.7×as long as the matched human reference, and spatial manipulations that are trivial for humans continue to challenge current agents. 1 Introduction Computer-use agents are moving from research prototypes toward user-facing systems that act on websites and software through the same interfaces people use. OpenAI’s Operator (OpenAI, 2025b) and Computer Use API (OpenAI, 2026a), along with Anthropic’s computer- use tool (Anthropic, 2026a), make the rendered interface itself—typically screenshots plus mouse and keyboard actions—a first-class control surface. This makes full-visual evaluation increasingly central. As screenshot-native agents become stronger, the key question is no longer simply whether an agent can occasionally complete a browser task, but which rendered UI components still prevent reliable and efficient use in deployment. Current evaluation paradigms still emphasize two extremes. Long-horizon benchmarks such as WebArena (Zhou et al., 2023), VisualWebArena (Koh et al., 2024), OSWorld (Xie et al., 2024), WebVoyager (He et al., 2024), and Online-Mind2Web (Xue et al., 2025) measure end-to-end competence on realistic tasks, but make failure attribution difficult: when an agent misses a workflow, the root cause may be planning, state tracking, grounding, or one brittle interaction buried inside a larger task. At the other extreme, grounding benchmarks such as ScreenSpot (Cheng et al., 2024) and ScreenSpot-Pro (Li et al., 2025) isolate local- ization ability, but stop before the short multi-step interactions that modern widgets often require. Efforts between the two—MiniWoB++’s synthetic micro-environments (Liu et al., 1 arXiv:2608.18307v1 [cs.AI] 18 Aug 2026 Published as a conference paper at COLM 2026 Figure 1: ComponentBench evaluates computer-use agents on 97 canonical UI component types organized into 14 families. Left: Tasks span diverse interaction types implemented across Ant Design, MUI, and Mantine. Right: The ontology covers the breadth of modern web UI interaction. 2018), Mind2Web’s offline real-site traces (Deng et al., 2023), and the web-action taxonomies, archived GUI subtasks, and appearance variations of WebSuite, WARC-Bench, and Ope- nApps (Li & Waldo, 2024; Srivastava et al., 2025; Ullrich et al., 2025)—remain partial for our purpose: to our knowledge, none are organized around a broad cross-library ontology of modern UI components with programmatic end-state verification, human reference traces, and post-render difficulty auditing. This missing layer matters because modern web software is composed from recurring component primitives exposed at scale by production UI libraries. A long workflow can therefore fail not because the agent misunderstood the user ’s goal, but because it mishandled one date picker, multiselect, splitter, or drag target. Long tasks are only as reliable as the component interactions they contain: under a simple independence approximation, five critical interactions at 80% reliability imply only about a 33% end-to-end ceiling. And pass rate alone is insufficient. Recent work shows that even strong computer-use agents often take far more steps than humans, with major latency and cost implications for real deployment (Abhyankar et al., 2025). We introduce ComponentBench, 1 a benchmark and diagnostic pipeline for evaluating computer-use agents through component-centered tasks on modern web UIs. Component- Bench organizes evaluation around 97 canonical component types, 14 interaction families, and 24 task templates, instantiated as 2,910 programmatically verified tasks primarily across Ant Design, MUI, and Mantine (thirty markdown-editor tasks use an external implemen- tation; Appendix A.5). Each task is anchored to a single primary component, even when realistic carrier context is required. The benchmark evaluates agents under four observation and action spaces—AX-tree, Set-of-Marks, Pixel, and Browser-Use—and distinguishes in- tended difficulty from the difficulty the rendered UI actually presents through replay-based audits using human reference traces. ComponentBench is also designed to evaluate efficiency, not only eventual completion. Because we collect cleaned human reference traces for all tasks, we can ask not just whether a component is solvable, but whether it is solved directly enough to be usable—critical for full-visual agents, where every extra step implies more latency, more token cost, and another opportunity to drift. To support faster stress-testing, we further derive ComponentBench- Core, a distilled 912-task hard-only suite of newly generated tasks that concentrates on the unresolved regions of the full suite. Our experiments on ComponentBench-Full across seven models and four observa- tion/action spaces reveal four main findings. First, observation/action space can shift pass rates by more than 30% within a single model, and the benefit of Set-of-Marks is model- dependent rather than universal. Second, efficiency remains a major deployment bottleneck: even the fastest configuration takes 3.7×as long as the matched human reference, and the 1 Website:https://componentbench.com;code:https://github.com/TianchenGuan/ ComponentBench; data and runs: https://huggingface.co/datasets/TianchenGuan/ComponentBench. 2 Published as a conference paper at COLM 2026 Figure 2: A concrete ComponentBench task (datatablefilterable-mantine-T10). (a) The agent sees a rendered page containing three visually similar mini-tables. (b) The instruction, programmatic verifier, and structured metadata. The agent must disambiguate the correct table instance (Invoices), apply two filters, and commit via the instance-local Apply button. strongest model solves many more tasks eventually than within the human step budget. Third, several spatial manipulation components that humans finish in 1–2 steps—including sliders, drag-and-drop lists, and splitters—remain below 60% mean pass rate across all agents tested. Finally, difficulty is strongly conditioned on visual context, with substan- tially wider AX-tree–Pixel gaps under clutter and compact spacing. A trace-grounded failure taxonomy (Section 4.4) ties these findings to concrete mechanisms. Together, these results show that component-level evaluation exposes failure modes largely invisible in both long-horizon task scores and single-step grounding benchmarks. 2 Benchmark construction A worked example. Before introducing the benchmark schema, we start with a concrete task. Figure 2 showsdatatablefilterable-mantine-T10. The page contains three visually similar mini-tables labeled Orders, Invoices, and Payouts. The agent must operate only the Invoices instance: set Payment status to Late, set Currency to EUR, and then click the local Apply button. This single example already illustrates several recurring design choices in ComponentBench: each task targets one primary component type; the page may include surrounding carrier context that adds realism without changing what is being tested; nearby instances can create disambiguation burden; and success is defined by a committed end state, not by a draft selection. 2.1 Component inventory and task specification To cover modern web interaction systematically, we build a structured inventory by combin- ing the WAI-ARIA Authoring Practices Guide (APG), which documents common widget patterns and behaviors, with the production component inventories of major React UI libraries—Ant Design, MUI, Mantine, Fluent UI, Chakra UI, and Headless UI (World Wide Web Consortium, 2026; Ant Group, 2026; MUI, 2026; Mantine, 2026; Microsoft, 2026a; Chakra UI, 2026; Tailwind Labs, 2026). The resulting component ontology is a library-agnostic set of 97 canonical component types (e.g., date picker) grouped into 14 families (e.g., Drag/Drop & Workspace). For implementation we select Ant Design, MUI, and Mantine for their compre- hensive coverage and stylistic diversity, which also lets cross-library variation serve as a controlled factor in the analysis. Each task evaluates one primary component type, even when the rendered page contains many other controls: in Figure 2, a filterable data table, with the enclosing summary table and neighboring mini-tables serving as carrier context. Tasks are specified in YAML. Each specification includes a canonical type, an implemen- tation source, a task template (a reusable action pattern such as open-and-select or drag 3 Published as a conference paper at COLM 2026 operation), a scene context (eight controlled factors: theme, spacing, layout, placement, scale, instances, guidance, clutter), an intended difficulty block (seven conceptual axes: precision requirement, target acquisition, density/choice interference, depth/layering, feedback dy- namics, semantic observability, disambiguation load), a success trigger, and negative cases. The full suite contains 24 canonical task templates (plus one ad-hoc variant used by a single task) and 2,910 tasks; Appendix A.5 summarizes the realized diversity across all of these dimensions. 2.2 Task generation, implementation, and human verification ComponentBench was built through a structured, LLM-assisted pipeline. First, GPT-5.2 Pro generated 30 task specifications per canonical type under the shared YAML schema, includ- ing task templates, scene factors, intended difficulty labels, success triggers, and negative cases. Second, Claude Code implemented these specifications as real interactive Next.js pages rather than static mockups. Third, each implemented task was executed twice by a human operator, recorded as low-level actions (clicks, drags, keyboard input, scrolls) with timestamps, cleaned—merging consecutive keystrokes and removing accidental resets—and the shorter pass was kept as the reference trajectory. The human recordings play two roles. During construction, they are the decisive validity check that the task is solvable and faithful to the intended specification; tasks that were broken, miswired, or no longer matched their specification were corrected before inclusion. Later, the same cleaned traces serve as reference trajectories for efficiency analysis and replay-based difficulty auditing (a validation study with two additional annotators appears in Appendix K). Across all 2,910 tasks, the cleaned traces have a mean of 2.7 normalized steps (median 2), with 97.8% of tasks solvable in 10 or fewer steps and a mean completion time of 4.8 seconds. 2.3 Programmatic verification Every task is paired with a programmatic verifier that checks the committed end state. The benchmark therefore does not ask whether the agent briefly opened the right menu or drafted the right intermediate selection; it asks whether the underlying task predicate is satisfied after the relevant interaction has actually been committed. For some tasks, the live state is sufficient. For others, success requires an explicit local control such as Apply, Save, OK, or Confirm. For the running example in Figure 2, success requires that the Invoices mini-table—and not the neighboring Orders or Payouts tables—has Payment status = Late and Currency = EUR, and that these selections have been committed via the instance-local Apply button. Each YAML task also enumerates negative cases so that nearby but incorrect states do not count as success. Concretely, the YAML specifies a canonical success predicate, and the page implementation provides a JavaScript checker that evaluates this predicate against the live component state. At the environment level, termination is intentionally simple and deterministic: success is signaled by revealing a benchmark banner (#cb-success-banner). This gives all observation/action spaces the same termination condition while leaving the checker logic task-specific underneath. Importantly, verifier state is isolated from agent observation. Target values and success predicates reside in React component closures and are never exposed as DOM attributes, accessible-tree labels, or page text; the success banner appears only after the correct state is already reached and therefore cannot be used to shortcut the task. In benchmark mode, aMutationObserveradditionally strips all test-only DOM attributes (data-testid,data-cy, etc.) from every element in real time, and a CI-ready scanner verifies that no such attributes survive across all 2,910 pages. 2.4 Observation and action spaces A core goal of ComponentBench is to evaluate the same underlying task under different observation and action spaces. The benchmark therefore supports four regimes. 4 Published as a conference paper at COLM 2026 AX-tree. The agent receives a screenshot plus accessibility-tree text and acts through element IDs. Set-of-Marks (SoM). The agent receives a screenshot with numbered overlays on inter- actable elements and also acts through element IDs. Pixel. The agent receives only the screenshot and must act through coordinates. Browser-Use. The same tasks are executed through the separatebrowser-useframework, which provides screenshot interaction together with a richer tool surface including serialized DOM information and grounded elements (M ̈ uller & Zuß, 2024). A Browser-Use “step” is a model turn that can execute up to 4 primitive actions, effectively giving it up to 80 actions within the 20-step budget, compared to exactly 1 action per step in the other three modes. The first three regimes share the same BrowserGym-based harness and differ only in what the agent sees and how it refers to targets (de Chezelles et al., 2024). Browser-Use is therefore not a strictly like-for-like fourth mode but a tool-rich reference regime—a distinction that lets us separate gains from stronger underlying models from gains that come from exposing additional structure and control affordances. 2.5 Intended difficulty and realized difficulty ComponentBench distinguishes between the difficulty a task is designed to have and the difficulty the rendered page actually presents. Intended difficulty is assigned during task generation: the seven conceptual axes and coarse tier labels record why we expect the task to be easy or hard (for the running example, mainly the disambiguation among similar instances). Because implementation can change difficulty through spacing, overlay structure, or clutter, we also measure realized difficulty by replaying the cleaned human trace in Playwright (Microsoft, 2026b) and extracting a 24-feature structural vector—including target size, spacing, overlay depth, option density, and metadata-derived properties such as control continuity and reset cost. The human trace serves as a traversal oracle, reaching hidden states (popovers, nested panels, internal scroll regions) that a static snapshot would miss. Features are aggregated into the same seven axes via weighted averaging with frozen cutpoints. In short, intended difficulty says what we meant to build; realized difficulty says what the implemented task actually is. 2.6 Diagnostic pipeline and Core distillation Beyond the executable benchmark itself, ComponentBench includes a three-stage diag- nostic pipeline that turns tasks, traces, and model runs into reusable evidence: Layer 1 deterministically assembles per-task packets joining the YAML specification, task metadata, realized-difficulty outputs, human reference traces, and selected agent evidence; Layer 2 pro- duces a structured task-level observation for each task; Layer 3 aggregates those records into one component-level report per canonical type. This pipeline is what makes the benchmark diagnostic rather than purely leaderboard-oriented. The full suite is intentionally coverage-oriented. Once broad evidence exists, we use realized- difficulty audits, Layer 2/3 reports, and scene-factor analyses to distill a sharper pressure test, ComponentBench-Core: 19 generation units, 45 canonical components, and 912 hard-only tasks, preserving the same pipeline while concentrating benchmark mass on unresolved interaction families. 3 Experiments This section focuses on ComponentBench-Full (v1), the broad coverage suite, because it provides the cleanest setting for comparing models across observation and action regimes. All tasks are run with a maximum budget of 20 agent steps. Our goal is not only to rank models, but to isolate how much of component-level performance comes from the underlying model versus the interface exposed to it. 5 Published as a conference paper at COLM 2026 ModelBrowser-UseAX-treeSoMPixel Gemini 3 Flash95.289.687.185.4 GPT-5.490.481.577.083.8 Gemini 3.1 Flash-Lite87.477.773.563.3 GPT-5 mini87.083.178.548.9 GPT-5.4 mini85.879.174.777.1 Qwen3-VL-235B ‡ 78.877.054.450.5 UI-TARS-1.5-7B † —12.6 Table 1: ComponentBench-Full pass rates (%) by model and observation/action space. All runs use a 20-step budget. Browser-Use is a tool-rich reference regime with DOM access; AX-tree, SoM, and Pixel share the same harness and differ only in observation and action space. Task-resampling bootstrap 95% CIs (B=10,000) have half-width≤1.9% per cell. † Native pixel-only harness with coordinate prediction. ‡ Browser-Use served via Amazon Bedrock (precision unspecified); other Qwen cells use local vLLM FP8 (Appendix M). 3.1 Models and regimes We evaluate seven models on v1 in up to four observation/action spaces. Gemini 3 Flash (Google DeepMind, 2025) and Gemini 3.1 Flash-Lite (Google, 2026) are Google’s multimodal models at different capability tiers. GPT-5.4 (OpenAI, 2026b), GPT-5.4 mini (OpenAI, 2026c), and GPT-5 mini (OpenAI, 2025a) are OpenAI models spanning full-scale and compact tiers. Qwen3-VL-235B (Bai et al., 2025) is an open-weight vision-language model. UI-TARS- 1.5-7B (Qin et al., 2025) is a 7B open-weight model trained for GUI interaction, evaluated through its own native pixel-only harness rather than the shared BrowserGym harness. The four observation/action spaces are defined in Section 2.4: AX-tree, SoM, and Pixel share the same benchmark harness and differ only in what the agent sees and how it refers to targets, while Browser-Use is the separate tool-rich reference regime (M ̈ uller & Zuß, 2024). The five Gemini and GPT models and Qwen3-VL-235B are evaluated in all four spaces; UI-TARS-1.5-7B in its native pixel-only mode. 3.2 Main results Table 1 presents the main v1 results. We highlight three observations. Observation/action space can shift pass rates by over 30% within a single model. The most dramatic example is GPT-5 mini, which drops from 83.1% in AX-tree to 48.9% in Pixel—a 34.2% swing within the same harness, where only the observation and action space changes (87.0% to 48.9% including the tool-rich Browser-Use regime). Every model evaluated in multiple spaces shows a substantial gap between its best and worst space, confirming that the interface through which an agent encounters a component is not merely a convenience choice but a major determinant of success. The benefit of structured aids is model-dependent.For models that are weaker at direct screenshot-based interaction, SoM provides a clear advantage over Pixel: GPT-5 mini gains 29.6% and Gemini 3.1 Flash-Lite gains 10.2%. But for two of the six models evaluated in both SoM and Pixel—GPT-5.4 mini and GPT-5.4—the ordering reverses: Pixel outperforms SoM by 2.4% and 6.8% (paired bootstrap 95% CIs [1.0, 3.9] and [5.3, 8.3]). For GPT-5.4, Pixel (83.8%) also outperforms AX-tree (81.5%). Gemini 3 Flash sits near the crossover, with only a 1.7% SoM advantage. This pattern suggests that structured overlays are not uniformly beneficial: they help weaker models substantially, but for models already strong at direct visual interaction, the added clutter and indirection of SoM can become neutral or slightly harmful. Both model capability and interface regime have large effects, and the benchmark is far from saturated. Gemini 3 Flash achieves 85.4% even in Pixel—the most restrictive observation space—exceeding several weaker models’ Browser-Use performance. At the 6 Published as a conference paper at COLM 2026 Browser-UseAX-treeSoMPixel ModelPass ≤ H ≤2H Pass ≤ H ≤2H Pass ≤ H ≤2H Pass ≤ H ≤2H Gemini 3 Flash95.2 56.185.3 89.6 74.782.5 87.1 73.281.4 85.4 65.077.8 GPT-5.490.4 40.675.3 81.5 56.572.4 77.0 49.668.2 83.8 48.169.5 Gemini 3.1 FL87.4 50.277.3 77.7 68.673.9 73.5 64.870.9 63.3 47.356.2 GPT-5 mini87.0 51.778.5 83.1 66.676.5 78.5 63.273.4 48.9 19.028.2 GPT-5.4 mini85.8 51.077.9 79.1 69.073.8 74.7 64.070.6 77.1 60.268.9 Qwen3-VL-235B 78.8 39.770.1 77.0 66.173.2 54.4 41.148.4 50.5 29.338.1 UI-TARS † — 12.69.010.5 Table 2: Step efficiency (% of 2,910 tasks).≤ H/≤2H: solved within 1/2×human steps, whereH = max(human steps, 1); hover-only references count asH=1. Browser-Use “steps” are model turns of up to 4 primitive actions (page-load turn excluded), so its efficiency columns are optimistic. † Native pixel-only harness. Full≤3H/5Hbreakdown in Ap- pendix C. other end, UI-TARS-1.5-7B, a 7B model trained specifically for GUI interaction, achieves 12.6% in its native pixel-only mode, with 87% of tasks hitting the 20-step ceiling. 4 Analysis 4.1 Agents solve most tasks but rarely match human efficiency Table 2 reports not only whether tasks are solved, but whether they are solved efficiently. The gap between Pass and≤ His the efficiency gap: tasks solvable but requiring more effort than a human. Because a Browser-Use step can bundle up to four primitive actions, its≤ H columns are an optimistic turn-level reference rather than a like-for-like action comparison. AX-tree and SoM are the most step-efficient modes (median ratio 1.0×). Browser-Use has the largest gap—Gemini 3 Flash reaches 95.2% pass but only 56.1% within human steps—and the gap is most severe for weak model–hard mode combinations: GPT-5 mini Pixel passes 48.9% but only 19.0% within human steps (mean ratio 3.6×, p90 = 9.0×). Wall-clock time reinforces this: even the fastest configuration (GPT-5.4 mini SoM) takes 3.7×as long as the matched human reference, and the slowest (GPT-5 mini Pixel) takes 21.5×as long, averaging 71.8 seconds per successful task (full time data in the appendix). 4.2 Browser-Use is a tool-rich reference regime with family-dependent trade-offs Browser-Use provides an aggregate pass-rate advantage (+7.8% to+18.2% over the mean of other modes), largest for weaker models, but at the cost of lower step efficiency. Part of this advantage comes from its ability to execute up to 4 actions per model turn (up to 80 actions versus 20 for the other modes). The advantage is not uniform, however: across the six models it is strongly family-dependent,+29.0% on Advanced Editors but−24.0% on Drag/Drop. The Layer 3 diagnostic reports explain why: Browser-Use’s DOM-level action surface cannot replicate genuine spatial drag operations on tasks like Kanban boards; on this family, GPT-5.4 falls from 70% in AX-tree and 63% in Pixel to 55% in Browser-Use. The CUA gap also scales with intended difficulty (+7.2% on easy tasks, +18.2% on hard). 4.3 Spatial manipulation components are trivial for humans but hard for agents Table 3 summarizes family-level pass rates. Families span from Command & Navigation (91.6% average) to Drag/Drop (47.7%), and no single observation/action space is universally best: Drag/Drop favors Pixel over Browser-Use, while Advanced Editors strongly favor Browser-Use. The full per-model heatmap is in the appendix (Figure 3). The Layer 3 diagnostic reports reveal distinct mechanistic patterns behind these family-level differences: on editable data grids agents identify the correct row yet fail to enter the widget’s editor mode or commit through its expected control (an interaction knowledge failure rather 7 Published as a conference paper at COLM 2026 FamilyBU AX SoM Pix Avg Drag/Drop & Workspace 29.7 72.131.8 57.4 47.7 Continuous Precision74.0 53.853.9 56.2 59.4 Advanced Editors83.6 67.849.7 46.4 61.9 Date & Time83.4 76.265.8 59.4 71.2 Disclosure & Progressive 80.1 76.064.1 68.3 72.1 Discrete Choice97.2 85.783.3 79.0 86.3 Overlays & Transient UI96.4 91.088.6 80.7 89.2 Command & Navigation 97.3 96.189.9 83.1 91.6 Table 3: Pass rate (%) by family and mode, averaged across all six models (excluding UI-TARS). Top: five hardest families; bottom: three easiest. Full breakdown in Figure 3 (Appendix). than a grounding failure); on rich text editors they find the toolbar but cannot establish and hold transient contenteditable selections; on context menus the difficulty is not locating the target but controlling the transient overlay after the right-click. Section 4.4 quantifies these mechanisms across all failed traces. A striking difficulty inversion emerges when comparing human and agent difficulty: nine canonical types require≤2 human steps yet achieve<60% agent pass rate— resizablecolumns (24.4%), windowsplitter (38.3%), sliderrange (39.9%), and six other spatial manipulation tasks (see Figure 4 in the appendix and Table 13). These are trivial mouse gestures for humans but among the hardest interactions for agents. The Layer 3 report for window splitters explains that the difficulty is “compressing perception, motor control, and verification into one thin affordance”: every task has maximal precision re- quirement, but outcomes depend on whether the separator is even exposed as a real control in the agent’s observation space. No component type shows the reverse pattern (hard for humans, easy for agents). 4.4 A trace-grounded failure taxonomy To move from family-level pass rates to failure mechanisms, we assign every failed trace a primary diagnostic category supported by trace evidence, using two complementary labelings (Table 4): a deterministic trace-feature taxonomy over all 8,864 failed Pixel/SoM/AX- tree traces across the five models with complete BrowserGym trace logs (each trace is parsed into its action sequence and mapped from a component-driven prior refined by trace evidence), and the Layer-2 diagnostic labels (an LLM reading of all 2,752 failed runs of one model) mapped onto the same categories. Methodological details and adversarially reviewed case studies appear in Appendix J. The two labelings surface overlapping high-frequency mechanism families, though their percentages are not directly comparable: continuous calibration errors (the agent engages the right slider or meter but cannot map pointer movement to the required value, overshooting and undershooting until timeout), transient state loss (an opened popover, editor mode, or selection is lost before commit), target acquisition / wrong-instance errors, missing commit actions, and missing widget-specific procedures. Two cross-cutting observations sharpen the earlier findings. First, slider and meter failures are usually not instruction-understanding failures: agents locate the correct control but cannot calibrate the continuous value, which explains why these tasks are trivial for humans (one drag) yet resistant to added reasoning. Second, a no-progress loop is a symptom rather than a root cause: 55.8% of failed traces end in a repeated-action loop, but it is distributed across every mechanism (≈48–67% within each category)—agents loop because they are stuck on the underlying mechanism. 4.5 Clutter and spacing disproportionately burden visual agents Scene factors create differential burdens across observation/action spaces (Figure 5, Ap- pendix). Averaged across models, medium clutter drops Pixel by 14.3% but AX-tree by only 0.2%; compact spacing drops Pixel by 13.6% but AX-tree by 4.5%. The failure taxonomy 8 Published as a conference paper at COLM 2026 Failure categoryDeterministicLayer-2 (5 models, n=8,864) (1 model, n=2,752) Continuous calibration error20.2%— Transient state loss19.9%15.0% Missing commit or confirmation11.6%20.0% Target acquisition / wrong instance11.2%31.6% Repetition / no-progress loop11.2%5.6% Widget-specific procedure missing9.6%24.1% Drag execution failure4.3%— Semantic value error3.6%3.3% Other / unclear8.4%0.3% Table 4: Failure taxonomy over all failed BrowserGym-mode traces, under two comple- mentary labelings: a deterministic trace-feature pass over five models, and Layer-2 LLM diagnostic labels for Gemini 3.1 Flash-Lite mapped onto the same categories. The two labelings surface overlapping high-frequency mechanism families, though their percentages are not directly comparable (different model sets and schemas). Continuous-calibration and drag-execution failures are not represented as separate categories in the original Layer-2 schema, hence absent from that column. shows this is mechanistic: clutter manifests as target-acquisition and wrong-instance failures, stressing visual grounding rather than task semantics—which is why Browser-Use, acting on DOM elements, is essentially clutter-immune (medium clutter costs it 0.8%). The interaction between model capability and scene factors is especially clear for GPT-5.4: at clutter=none, GPT-5.4 Pixel (86.1%) outperforms AX-tree (81.6%) by 4.5%, but at clutter=medium the gap reverses and AX-tree (82.3%) leads Pixel (72.6%) by 9.7%—the preferred observation space flips on a single scene factor. The intended difficulty axes also validate empirically: precision requirement is the most predictive axis (r = +0.44 with failure rate), and the prediction is mode-dependent (r = +0.41 for Pixel vs.r = +0.05 for Browser-Use on depth/layering). Difficulty tiers decline monotonically: L0 = 87.3%, L3 = 65.2%, with the AX-tree–Pixel gap widening from 4.6% (L0) to 21.9% (L3). Across all models, a task-level asymmetry also emerges: 123 tasks pass reliably in AX-tree (mean pass rate>0.7) while failing in Pixel (mean<0.3), whereas only 52 show the reverse—the asymmetry between structured and visual observation is large and directional. 4.6 ComponentBench-Core as a pressure test ComponentBench-Core is derived from the full suite using the evidence described in Sec- tion 2.6: realized-difficulty audits, Layer 2/3 diagnostic reports, and scene-factor interactions merge overlapping canonical types into 19 interaction-centered generation units, each con- tributing 48 regenerated hard-only tasks (912 total). Core is not a subset of Full: its tasks are newly generated, dropping saturated easy families and concentrating benchmark mass on the interaction patterns that still separate current agents. ModelModePass≤H≤ 2H≤ 3H Gemini 3 Flash Browser-Use84.551.571.578.5 Gemini 3 Flash Pixel60.930.551.056.0 GPT-5.4 miniBrowser-Use57.836.851.255.3 GPT-5.4 miniPixel37.722.032.134.1 Opus 4.6Pixel65.434.153.859.4 Table 5: ComponentBench-Core results (% of 912 tasks). Same metrics (and Browser-Use turn-level caveat) as Table 2, on the hard-only CORE suite. Pass rates drop 10–39% from FULL, confirming that CORE concentrates diagnostic mass on unresolved interaction families. Opus 4.6 is evaluated only on CORE Pixel. 9 Published as a conference paper at COLM 2026 Table 5 presents the results on CORE. Pass rates drop substantially relative to FULL: Gem- ini 3 Flash Browser-Use falls from 95.2% to 84.5% (−10.7%), Gemini 3 Flash Pixel from 85.4% to 60.9% (−24.5%), GPT-5.4 mini Browser-Use from 85.8% to 57.8% (−28.0%), and GPT-5.4 mini Pixel from 77.1% to 37.7% (−39.4%). The drop is largest for the weakest combination: CORE disproportionately stresses the modes and models already borderline on FULL. To test whether CORE remains diagnostic for frontier models not evaluated on FULL, we additionally run Opus 4.6 (Anthropic, 2026b) on CORE Pixel. It achieves 65.4%—the highest pixel-only result on the hard suite, surpassing Gemini 3 Flash Pixel (60.9%)—but still leaves over a third of tasks unsolved, with only 34.1% solved within the human step budget and a 9.5×time-to-human ratio (67.4 vs. 7.1 mean seconds per successful task). CORE is therefore not an artifact of weaker models: it stays challenging even for a frontier computer-use model. Human traces for CORE average 5.2 normalized steps versus 2.7 for FULL, reflecting the CORE tasks’ structural complexity. Where FULL suits broad exploration, CORE is a compact stress test for the hardest families. 5 Limitations and scope ComponentBench targets recurring component families from WAI-ARIA patterns and three major production libraries, not every bespoke widget in the wild; CAPTCHA-like tasks are excluded. The benchmark is web-first (Next.js on Chromium), though the ontology is designed to port to desktop and mobile. It measures component-level competence, not long- horizon planning; validating how component-level scores predict end-to-end workflow success is future work. The realized-difficulty audit is a hybrid system: some quantities are measured from the rendered DOM, others approximated from type-level metadata. The primary human reference traces come from two passes by a single annotator—a practical reference, not a proof of near-optimality; a validation study with two additional annotators (Appendix K) shows inter-annotator variation (1.05×per-task) is small relative to the agent–human gap (1.27–3.02×), though agents with tree- or DOM-level tools may admit shorter non-visual paths. Both should be treated as structured operationalizations, not oracles. Our experiments cover seven models on the full suite and one additional frontier model (Opus 4.6) on Core—a broad but not exhaustive snapshot. The main tables report a single run per model–mode combination; in a repeated-run study on a 278-task subset (Appendix L) the observed run-to-run deviation was at most 1.4% with no ordering changes, though task-level outcomes churn on borderline tasks. Because task specifications were generated with GPT-5.2 Pro and pages implemented with Claude Code, the benchmark may contain generator-specific regularities. Three design choices mitigate this risk: every task is scored by a deterministic programmatic verifier rather than LLM judgment, the generation prompt enforces combinatorial diversity over a fixed ontology (Appendix A.5), and the human recording passes (including the annotators of Appendix K) served as quality checks that surfaced no broken, ambiguous, or miswired tasks. We do not, however, audit exhaustively for contamination. Additionally, GPT models were accessed via the chat completions API rather than OpenAI’s Operator or Computer Use API, Opus 4.6 through a custom harness rather than Anthropic’s full computer-use environment, and BrowserGym renders no visible cursor—so our results may underestimate native-interface performance. 6 Conclusion We presented ComponentBench, a component-level benchmark for diagnosing where computer-use agents fail on modern web UIs. Across 97 canonical component types, 2,910 tasks, and four observation regimes, representation, efficiency, and visual context all critically shape agent performance, and a trace-grounded failure taxonomy ties these effects to concrete mechanisms—helping localize the component-level causes of workflow failures. 10 Published as a conference paper at COLM 2026 Acknowledgments Model API access for the experiments in this paper was provided by the Amazon AGI SF Lab. Experiments were run on the Duke Computer Science cluster. We thank Jiacheng Sang and Xunjian Yin for contributing human reference annotations and for helpful advice. References Reyna Abhyankar, Qi Qi, and Yiying Zhang. OSWorld-Human: Benchmarking the efficiency of computer-use agents, 2025. URL https://arxiv.org/abs/2506.16042. Ant Group. Ant design components overview.https://ant.design/components/overview/, 2026. Accessed 2026-03-25. Anthropic. Computer use tool — Claude API documentation.https://docs.anthropic. com/en/docs/agents-and-tools/computer-use, 2026a. Accessed 2026-03-25. Anthropic.Introducing Claude Opus 4.6.https://w.anthropic.com/news/ claude-opus-4-6, 2026b. Accessed 2026-07-24. Anthropic.Introducing Claude Opus 4.8.https://w.anthropic.com/news/ claude-opus-4-8, 2026c. Accessed 2026-07-24. Shuai Bai et al. Qwen3-VL technical report, 2025. URLhttps://arxiv.org/abs/2511.21631. Chakra UI.Chakra UI components.https://w.chakra-ui.com/docs/components/ concepts/overview, 2026. Accessed 2026-07-11. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024. URL https://arxiv.org/abs/2401.10935. Thibault Le Sellier de Chezelles, Maxime Gasse, Alexandre Drouin, Massimo Caccia, L ́ eo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Ke- unho Jang, Xing Han L ` u, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Quentin Cappart, Graham Neubig, Ruslan Salakhutdinov, Nicolas Chapados, and Alexandre Lacoste.The browsergym ecosystem for web agent research, 2024.URLhttps: //arxiv.org/abs/2412.05467. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023. URLhttps: //arxiv.org/abs/2306.06070. Paul M. Fitts. The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47(6):381–391, 1954. doi: 10.1037/h0055392. Google. Gemini 3.1 Flash-Lite: Built for intelligence at scale.https://blog.google/ innovation-and-ai/models-and-research/gemini-models/gemini-3-1-flash-lite/, 2026. Accessed 2026-07-24. Google DeepMind.Gemini 3 flash model card.https://storage.googleapis.com/ deepmind-media/Model-Cards/Gemini-3-Flash-Model-Card.pdf, 2025. Accessed 2026- 03-30. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models, 2024. URL https://arxiv.org/abs/2401.13919. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024. URLhttps://arxiv. org/abs/2401.13649. 11 Published as a conference paper at COLM 2026 Eric Li and Jim Waldo. Websuite: Systematically evaluating why web agents fail, 2024. URL https://arxiv.org/abs/2406.01623. Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. ScreenSpot-Pro: Gui grounding for professional high-resolution computer use, 2025. URL https://arxiv.org/abs/2504.07981. Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforce- ment learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations, 2018. URL https://arxiv.org/abs/1802.08802. Mantine. Mantine core components.https://mantine.dev/core/package/, 2026. Accessed 2026-03-25. Microsoft. Fluent UI React components.https://react.fluentui.dev/, 2026a. Accessed 2026-03-25. Microsoft. Evaluating javascript — playwright documentation.https://playwright.dev/ docs/evaluating, 2026b. Accessed 2026-03-25. MUI. Mui components documentation.https://mui.com/components/, 2026. Accessed 2026-03-25. Magnus M ̈ uller and Gregor Zuß. Browser-use: Make websites accessible for AI agents. https://github.com/browser-use/browser-use, 2024. MIT License, v0.12. OpenAI. OpenAI GPT-5 system card, 2025a. URL https://arxiv.org/abs/2601.03267. OpenAI. Introducing operator.https://openai.com/index/introducing-operator/, 2025b. Accessed 2026-03-25. OpenAI. Computer use — OpenAI API documentation.https://platform.openai.com/ docs/guides/tools-computer-use, 2026a. Accessed 2026-07-11. OpenAI. Introducing GPT-5.4.https://openai.com/index/introducing-gpt-5-4/, 2026b. Accessed 2026-03-30. OpenAI.Introducing GPT-5.4 mini and nano.https://openai.com/index/ introducing-gpt-5-4-mini-and-nano/, 2026c. Accessed 2026-07-24. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, et al. UI-TARS: Pioneering automated GUI interaction with native agents, 2025. URLhttps://arxiv.org/abs/2501. 12326. Sanjari Srivastava, Gang Li, Cheng Chang, Rishu Garg, Manpreet Kaur, Charlene Y. Lee, Yuezhang Li, Yining Mao, Ignacio Cases, Yanan Xie, and Peng Qi. Warc-bench: Web archive based benchmark for gui subtask executions, 2025. URLhttps://arxiv.org/abs/ 2510.09872. Tailwind Labs. Headless UI components.https://headlessui.com/, 2026. Accessed 2026- 03-25. Karen Ullrich, Jingtong Su, Claudia Shi, Arjun Subramonian, Amir Bar, Ivan Evtimov, Nikolaos Tsilivis, Randall Balestriero, Julia Kempe, and Mark Ibrahim. OpenApps: Simulating environment variations to measure UI-agent reliability, 2025. URLhttps: //arxiv.org/abs/2511.20766. World Wide Web Consortium. Aria authoring practices guide (apg).https://w.w3.org/ WAI/ARIA/apg/, 2026. Accessed 2026-03-25. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. URL https://arxiv.org/abs/2404.07972. 12 Published as a conference paper at COLM 2026 Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents, 2025. URL https://arxiv.org/abs/2504.01382. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2023. URLhttps://arxiv. org/abs/2307.13854. A Benchmark construction details A.1 Task specification schema Each task is specified in YAML with the following fields:id,canonicaltype, implementationsource(antd/mui/mantine/external),tasktemplate,browsergymgoal (natural-language instruction),scenecontext(8 controlled factors),difficulty(bucket, tier, and 7 axis ratings with justification),successtrigger(human-readable criteria and canonical predicate),negativecases, andexpectedinteractionpath. Scene factors and their realized levels are:theme(light/dark),spacing(comfortable/compact),layout(8 levels; isolatedcard, formsection, dashboard, settingspanel, . . . ),placement(center plus four off-center corners),scale(default/small/large),instances(1–10),guidance (text/visual/mixed), andclutter(none/low/medium/high). Table 6 reports the real- ized distribution over these levels. A.2 LLM-assisted construction pipeline GPT-5.2 Pro generated 30 task specifications per canonical type, including templates, in- tended difficulty labels, scene factors, success triggers, and negative cases. Claude Code then implemented those specifications as real interactive Next.js pages. Each task was subsequently executed by a human and recorded as a reference trajectory. This recording stage also served as the final quality-control pass: tasks that were broken, miswired, or did not match their intended specification after implementation were corrected before inclusion. A.3 Task generation prompt For each canonical component type, GPT-5.2 Pro received a structured prompt together with a research CSV file mapping the component to its available library implementations, supported interaction patterns, and difficulty considerations. The prompt specified the following constraints: Single-component focus and realistic intents.Each task must target exactly one primary component whose state determines success. Instructions must read as plausible micro- user intents (e.g., “Set the Price range slider to $20–$80”), not benchmark jargon. Internal tolerances, checker rules, and implementation details are excluded from the agent-facing instruction. Structured output schema. Each task specification includes: a stable ID, the canonical type, implementation source and variant, a primary task template (from the 24 defined templates), the complete scene context (all 8 factors), difficulty ratings (bucket, tier, and all 7 axis ratings with justification), a detailed setup description of the rendered page, an explicit success trigger with both human-readable conditions and a machine-friendly canonical predicate (including predicate type, target state, tolerance, confirmation requirements, and correct-instance requirements), a list of negative cases, and an expected interaction path for debugging. Coverage and balance constraints. The prompt requires exactly 30 tasks per component type, with a fixed difficulty distribution of 10 easy, 10 medium, and 10 hard tasks. When a 13 Published as a conference paper at COLM 2026 component is supported by all three primary libraries, tasks are split 10/10/10 across Ant Design, MUI, and Mantine with balanced per-library difficulty. The prompt also enforces minimum coverage of scene-factor variations: at least 3 tasks with dark theme, at least 3 with compact spacing, at least 4 with clutter, at least 4 with multiple instances (if meaningful for the component), and at least 7 distinct task templates per component type. Difficulty calibration.The prompt defines a “default-first” principle: easy tasks use library defaults and simple contexts, while hardness comes from realistic variations—scene factors (compact spacing, clutter, multiple instances), component feature variants (toggling sub- controls, enabling search, restricting input), and within-component depth (nested overlays, multi-step navigation). Adversarial tricks such as invisible elements or overlapping click traps are explicitly prohibited. A.4 Programmatic verification details At the environment level, task termination is deterministic: the page’s programmatic verifier checks whether the committed end state satisfies the task-specific predicate, and success is signaled by presenting a DOM element (#cb-success-banner). For tasks requiring explicit commit actions (Apply, Save, Confirm), the verifier checks only the post-commit state. The YAML enumerates negative cases so that nearby but incorrect states do not count as success. A.5 Task diversity summary Because the tasks are authored with LLM assistance, a natural concern is that they could collapse onto a few repeated patterns. The generation pipeline explicitly enforces com- binatorial coverage (Appendix A.3); Table 6 summarizes the realized distribution of the 2,910 Full tasks over libraries, templates, difficulty, and scene factors. All 97 canonical types contribute exactly 30 tasks; all 24 templates and all levels of every scene factor are exercised, with deliberately skewed marginals (e.g., most tasks use the default clean context, while ∼40% carry at least one added burden such as clutter, compact spacing, dark theme, or multiple instances). DimensionLevels (count)Coverage Canonical type97 types× 30 tasks100% Family14 families (120–300 tasks each)100% Libraryantd 1,000 / mui 910 / mantine 970 / external 304 Task template24 templates; most-used match reference (351),24/24 least-used file manage (10) Difficulty bucket easy 972 / mid 985 / hard 9533/3 Difficulty tierL0 789 / L1 1,050 / L2 818 / L3 2534/4 Themelight 2,600 / dark 3102/2 Spacingcomfortable 2,595 / compact 3152/2 Scaledefault 2,634 / small 272 / large 43/3 Clutternone 1,751 / low 666 / medium 339 / high 1544/4 Instances1: 2,266 / 2: 398 / 3: 206 /≥4: 40full Layout8 layouts; isolatedcard 1,866, formsection 244,8/8 dashboard 210, settings panel 207, others 383 Placementcenter 2,456 / off-center (4 corners) 4545/5 Guidancetext 2,476 / visual 250 / mixed 1843/3 Table 6: Realized diversity of the 2,910 ComponentBench-Full tasks over libraries, templates, difficulty, and the eight controlled scene factors. The 30 external tasks are the markdown- editor tasks, which use the third-party@uiw/react-md-editorbecause none of the three primary libraries ships a core markdown editor. One additional ad-hoc template variant (replacecode) is used by a single OTP-input task. The realized difficulty buckets deviate slightly from the prompted 970/970/970 split because two later-added types (breadcrumb, pagination) were generated by a run that did not enforce the per-type balance quota. 14 Published as a conference paper at COLM 2026 B Realized difficulty details B.1 Replay-based measurement The realized audit replays a cleaned human trajectory in Playwright (Microsoft, 2026b), usingpage.evaluate()to run measurement logic inside the browser page. Features are extracted before and after each action, capturing both the initial state and states revealed through interaction (popovers, nested panels, internal scroll regions). The audit covers all 2,910 Full tasks and all 912 Core tasks. B.2 Feature list The 24 canonical features are: 1. Minimum target size (px) 2. Target spacing (px) 3. Fitts-style acquisition difficulty (Fitts, 1954) 4. Interactable element density 5. Option/choice count 6. Overlay depth 7. Reference step burden 8. Scroll requirement (boolean + depth) 9. Feedback persistence 10. Reset cost 11. Visible state fraction 12. ARIA richness score 13. Contrast ratio 14. Clutter level 15. Placement offset 16. Instance count 17. Control continuity (metadata-derived) 18. State dimensionality (metadata-derived) 19. Approximate granularity (metadata-derived) 20. Tolerance (metadata-derived) 21. Feedback modality (metadata-derived) 22. Confirmation requirement (metadata-derived) 23. Scroll region depth (metadata-derived) 24. Precision surface type (metadata-derived) Features 1–16 are measured from the rendered DOM and layout; features 17–24 are derived from type-level metadata. B.3 Axis aggregation The seven realized axes are computed by weighted averaging of normalized features: s a (t) = ∑ f∈F a w f ˆ f (t) ∑ f∈F a w f ,(1) where ˆ f (t)is the min-max normalized feature value and the current implementation uses unit weights (w f =1). Continuous scores are discretized with frozen, versioned cutpoints into 1–5 ratings. C Full step efficiency table 15 Published as a conference paper at COLM 2026 ModelModePass≤H≤ 2H≤ 3H≤ 5H Gemini 3 FlashBrowser-Use95.256.185.389.191.9 AX-tree89.674.782.584.787.4 SoM87.173.281.483.685.2 Pixel85.465.077.880.983.6 GPT-5.4Browser-Use90.440.675.382.986.8 AX-tree81.556.572.476.379.4 SoM77.049.668.272.975.5 Pixel83.848.169.575.780.0 Gemini 3.1 Flash-Lite Browser-Use87.450.277.381.183.8 AX-tree77.768.673.975.376.5 SoM73.564.870.972.272.8 Pixel63.347.356.259.061.0 GPT-5 miniBrowser-Use87.051.778.582.184.8 AX-tree83.166.676.578.880.6 SoM78.563.273.475.977.3 Pixel48.919.028.232.738.7 GPT-5.4 miniBrowser-Use85.851.077.981.683.7 AX-tree79.169.073.875.476.9 SoM74.764.070.672.273.5 Pixel77.160.268.972.174.9 Qwen3-VL-235BBrowser-Use78.839.770.174.277.0 AX-tree77.066.173.274.875.7 SoM54.441.148.451.353.0 Pixel50.529.338.142.346.3 UI-TARS-1.5-7B † Pixel12.69.010.511.211.9 Table 7: Full step efficiency on ComponentBench-Full (% of 2,910 tasks). Pass: solved within 20 steps.≤ H/≤2H/≤3H/≤5H: solved within 1/2/3/5×the human step count. Bold: best per model. Browser-Use steps are model turns of up to 4 primitive actions (page-load turn excluded), so its efficiency columns are optimistic. † Native pixel-only interface. D Full time efficiency table 16 Published as a conference paper at COLM 2026 ModelModeHuman AgentRatio Med. Agent Gemini 3 FlashAX-tree4.5s21.5s4.7×13.5s SoM4.6s22.0s4.8×14.4s Pixel4.3s27.6s6.4×17.1s Browser-Use4.6s32.6s7.1×21.2s Gemini 3.1 Flash-Lite AX-tree4.1s14.5s3.6×11.2s SoM3.8s14.4s3.8×11.6s Pixel3.1s15.8s5.1×11.8s Browser-Use4.5s23.1s5.1×17.7s GPT-5.4AX-tree4.0s31.9s7.9×19.2s SoM3.7s35.0s9.4×22.3s Pixel4.2s23.0s5.5×14.2s Browser-Use4.6s36.2s7.8×24.3s GPT-5 miniAX-tree4.5s32.1s7.2×18.9s SoM4.3s32.7s7.7×19.9s Pixel3.3s71.8s 21.5×41.4s Browser-Use4.6s36.3s8.0×24.2s GPT-5.4 miniAX-tree3.9s14.7s3.8×11.4s SoM3.9s14.2s3.7×11.4s Pixel3.6s14.2s3.9×10.7s Browser-Use4.2s17.6s4.2×13.7s Qwen3-VL-235BAX-tree3.9s21.6s5.6×13.8s SoM3.4s24.4s7.1×14.4s Pixel2.9s26.8s9.3×16.0s Browser-Use4.1s45.0s 10.9×28.1s UI-TARS-1.5-7BNative Pixel2.4s15.5s6.5×10.1s Table 8: Full time efficiency on successful ComponentBench-Full tasks. Human and Agent columns report mean wall-clock duration over the same task set: the tasks each model–mode configuration solved (hence the Human column varies by row). Ratio is agent-to-human mean time. E Representation trade-off tables ModelSoMPixel∆ (SoM−Pixel) GPT-5 mini78.548.9+29.6 Gemini 3.1 Flash-Lite73.563.3+10.2 Qwen3-VL-235B54.450.5+3.9 Gemini 3 Flash87.185.4+1.7 GPT-5.4 mini74.777.1−2.4 GPT-5.477.083.8−6.8 Table 9: SoM–Pixel delta on ComponentBench-Full (%). The SoM advantage is model- dependent, ranging from +29.6% to−6.8%. 17 Published as a conference paper at COLM 2026 ModelBrowser-UseMean(AX,SoM,Pix)∆ Qwen3-VL-235B78.860.6 +18.2 GPT-5 mini87.070.2 +16.8 Gemini 3.1 Flash-Lite87.471.5 +15.9 GPT-5.490.480.8 +9.6 GPT-5.4 mini85.877.0 +8.8 Gemini 3 Flash95.287.4 +7.8 Table 10: Browser-Use advantage over mean non-Browser-Use pass rate (%). The advantage is largest for weaker models. F Additional analysis figures Figures 3–10 provide the full per-model family heatmap, difficulty-inversion scatter, scene- factor effects, and per-template, efficiency, and step-curve breakdowns referenced in the main text. 18 Published as a conference paper at COLM 2026 BUAXSoMPixBUAXSoMPixBUAXSoMPix Drag/Drop & Workspace Continuous & High-Precision Advanced Editors Date & Time Disclosure & Progressive Structured Data Display List-based Selection (Flat) Hierarchical Selection & Nav. Combobox & Autocomplete Text Entry & Structured Field Files, Clipboard, Downloads Discrete Choice Overlays & Transient UI Command & Navigation 637838865570316328683154 936369808755576873545455 937863608970536782654840 898785788177697985716046 908475919077658080766467 999291899380737688706650 1009294769681747297677852 989493889584819192907947 1008898919681919294718272 1009898709387878994888362 919396957987939678878773 1009793969787859397768278 999496959691909396909084 999995969796939598958985 Gemini 3 FlashGPT-5.4Gemini 3.1 FL ComponentBench-Full: Pass rate (%) by family, model, and observation/action space BUAXSoMPixBUAXSoMPixBUAXSoMPix Drag/Drop & Workspace Continuous & High-Precision Advanced Editors Date & Time Disclosure & Progressive Structured Data Display List-based Selection (Flat) Hierarchical Selection & Nav. Combobox & Autocomplete Text Entry & Structured Field Files, Clipboard, Downloads Discrete Choice Overlays & Transient UI Command & Navigation 12743328127132618722652 625554337252556657453535 886957258665485563602832 858173488570626376724642 787770557879637565634941 908071338578686579754236 958281449274686784674442 928982439189818780855947 927884518972828286705760 949287619588877889855955 918994517789949285886356 998688599678848693906762 989392599692909293867362 999794639894929194957768 GPT-5 miniGPT-5.4 miniQwen3-VL 020406080100 Pass rate (%) Figure 3: Pass rate (%) by component family, model, and observation mode on ComponentBench-Full. Families sorted by difficulty (hardest at top). The heatmap re- veals where mode ordering inverts: Drag/Drop favors Pixel over Browser-Use; Advanced Editors strongly favor Browser-Use. 19 Published as a conference paper at COLM 2026 246810 Mean human reference steps 30 40 50 60 70 80 90 100 Mean agent pass rate (%) Human-agent difficulty inversion resizable columns window splitter slider range alpha slider kanban board drag drop rich text editor datetime picker range Advanced Editors Combobox Command & Nav. Continuous Precision Date & Time Disclosure Discrete Choice Drag/Drop & Workspace Files & Clipboard Hierarchical Nav. List-based Selection Overlays Structured Data Text Entry Figure 4: Human-agent difficulty inversion. Each point is one canonical type. The lower-left quadrant contains components trivial for humans (≤2 steps) but hard for agents (<60% pass). nonelowmediumhigh 0 20 40 60 80 100 Pass rate (%) 81 83 81 78 76 76 66 65 71 70 57 54 Clutter AX-tree SoM Pixel comfortablecompact 0 20 40 60 80 100 Pass rate (%) 82 77 75 65 70 56 Spacing Scene factor effects on pass rate by observation mode Figure 5: Clutter and spacing disproportionately hurt Pixel mode, widening the AX-tree– Pixel gap from 10% to 24% (clutter) and 12% to 21% (spacing). 20 Published as a conference paper at COLM 2026 050100150200250300 Number of tasks Files/Clipboard/Downloads Command & Navigation List Selection (Flat) Combobox & Autocomplete Text Entry & Struct. Fields Overlays & Transient UI Discrete Choice Structured Data Display Advanced Editors Hierarchical Sel. & Nav. Date & Time Disclosure & Progressive Drag/Drop & Workspace Continuous / High-Precision GPT-5.4 mini: SoM vs Pixel task outcomes by family Both passSoM onlyPixel onlyBoth fail Figure 6: GPT-5.4 mini SoM vs. Pixel task outcomes by component family. For each family, tasks are classified as both-pass, SoM-only, Pixel-only, or both-fail. Drag/Drop and Continu- ous Precision families show the strongest Pixel-over-SoM advantage. 21 Published as a conference paper at COLM 2026 −20−100102030 Browser-Use advantage over mean(AX-tree, SoM, Pixel) (%) Drag/Drop & Workspace Interactions Files, Clipboard, Downloads Command & Navigation Overlays & Transient UI Disclosure & Progressive Hierarchical Selection & Navigation Text Entry & Structured Field Input Discrete Choice Combobox & Autocomplete Date & Time Continuous & High-Precision Input Structured Data Display List-based Selection (Flat) Advanced Editors CUA (Browser-Use) gap by component family Figure 7: Browser-Use (CUA) advantage by component family, averaged across the six models evaluated in all four regimes. The advantage ranges from+29% (Advanced Editors) to−24% (Drag/Drop), confirming that DOM-level tool access is not uniformly beneficial. 22 Published as a conference paper at COLM 2026 020406080100 Pass rate (%) activate disclose navigate_to file_manage open_overlay clear_reset toggle_state select_one enter_text hierarchical_path_select table_operation enter_formatted open_and_select search_and_select select_many confirm_cancel file_upload match_reference set_scalar transfer_move scroll_find editor_operation drag_operation set_range Pass rate by task template and observation mode Browser-UseAX-treeSoMPixel Figure 8: Pass rate by task template and observation mode. Templates requiring spatial control (setrange, dragoperation) or complex editing (editoroperation) are substantially harder than simple activation or disclosure tasks. 23 Published as a conference paper at COLM 2026 01020304050607080 Mean time per successful task (seconds) 40 50 60 70 80 90 100 Pass rate (%) Efficiency frontier: Pass rate vs. time cost G3F G3F G3F G3F G3.1FL G3.1FL G3.1FL G3.1FL G5m G5m G5m G5m G5.4 G5.4 G5.4 G5.4 G5.4m G5.4m G5.4m G5.4m Q3VL Q3VL Q3VL Q3VL Human Mode Browser-Use AX-tree SoM Pixel Model Gemini 3 Flash Gemini 3.1 FL GPT-5 mini GPT-5.4 GPT-5.4 mini Qwen3-VL Figure 9: Efficiency frontier: pass rate vs. mean time per successful task. Each point is one model-mode combination among the six models evaluated in the shared and Browser-Use regimes; the native UI-TARS configuration is omitted. The human reference point (100% pass, 4.8s) is shown for comparison. 2468101214161820 Step number 0 5 10 15 20 25 30 35 40 Marginal tasks solved (%) Gemini 3 Flash: Marginal gain per step by mode Browser-Use AX-tree SoM Pixel Figure 10: Marginal tasks solved per additional step (Gemini 3 Flash). Most diagnostic value is concentrated in the first 5 steps; steps 6–20 contribute diminishing returns. 24 Published as a conference paper at COLM 2026 G Difficulty axis validation Difficulty axisOverall AX-tree Pixel Browser-Use Precision requirement +0.44 +0.30 +0.41+0.33 Target acquisition+0.31 +0.17 +0.36+0.19 Density / choice interf. +0.23 +0.10 +0.31+0.12 Feedback dynamics+0.23 +0.12 +0.29+0.13 Depth / layering+0.20 +0.11 +0.29+0.05 Semantic observability +0.13 +0.08 +0.17+0.05 Disambiguation load+0.09 +0.04 +0.15+0.02 Table 11: Pearson correlation between intended difficulty axes and agent failure rate. Pre- cision requirement is the most predictive axis overall; all axes predict Pixel failure more strongly than Browser-Use failure. TierAX-treeSoMPixelBrowser-Use L0 (easy)87.7%85.9%83.0%92.8% L1 (medium)82.0%75.1%69.8%87.6% L2 (hard)77.4%66.4%57.6%84.3% L3 (hard+)71.4%59.4%49.5%80.4% Table 12: Pass rate by intended difficulty tier and observation mode, averaged across all six models (excluding UI-TARS). The AX-tree–Pixel gap widens monotonically from 4.6% (L0) to 21.9% (L3). H Ontology: 97 canonical component types The 14 families and their canonical types (with component count per family): Command & Navigation (10):button, iconbutton, link, menubutton, splitbutton, tool- bar, breadcrumb, pagination, stepper, tabs Disclosure & Progressive (5):accordion,collapsibledisclosure,carousel, feedinfinitescroll, windowsplitter Text Entry & Structured Field Input (10):textinput, textarea, passwordinput, num- berinputspinbutton, maskedinput, pininputotp, tagsinput, mentionsinput, searchinput, inlineeditabletext Discrete Choice (9):checkbox, checkboxgroup, checkboxtristate, radiogroup, switch, togglebutton, togglebuttongroupmulti, segmentedcontrol, rating List-based Selection (Flat) (7): selectnative, selectcustomsingle, selectcustommulti, se- lectwithsearch, listboxsingle, listboxmulti, transferlist Combobox & Autocomplete (4):comboboxeditablesingle, comboboxeditablemulti, au- tocompletefreeform, autocompleterestricted Hierarchical Selection & Navigation (7): menu,menubar,contextmenu,treeview, treeselect, treegrid, cascader Continuous & High-Precision Input (8):slidersingle, sliderrange, alphaslider, meter, progressbar, colorpicker2d, colorswatchpicker, colortextinput Date & Time (8): datepickersingle,datepickerrange,datetimepickersingle, datetimepickerrange,timepicker,calendarembedded,dateinputtext, timeinputtext Overlays & Transient UI (9):dialogmodal,drawer,popover,tooltip,hovercard, toastsnackbar, alertdialogconfirm, notificationcenter, tourteachingtip Structured Data Display (7): tablestatic,datatablesortable,datatablepaginated, datatablefilterable, datagrideditable, datagridrowselection, virtuallist 25 Published as a conference paper at COLM 2026 Files, Clipboard, Downloads (5):fileuploadbutton, filedropzone, filelistmanager, clip- boardcopy, downloadtrigger Drag/Drop & Workspace Interactions (4):dragdropsortablelist, dragdropbetweenlists, kanbanboarddragdrop, resizablecolumns Advanced Editors (4): richtexteditor, markdowneditor, codeeditor, jsoneditor I Hardest canonical component types Component typeAgent % Human steps Family resizablecolumns24.41.7 Drag/Drop windowsplitter38.31.3 Disclosure sliderrange39.91.9 Continuous richtexteditor40.74.9 Adv. Editors meter46.51.6 Continuous datetime pickerrange48.910.5 Date/Time alphaslider50.01.5 Continuous kanbanboarddragdrop52.11.4 Drag/Drop selectnative53.11.4 List Selection feedinfinitescroll53.34.3 Disclosure colorpicker2d55.72.9 Continuous virtual list56.73.3 Structured Data dragdropbetweenlists56.91.7 Drag/Drop dragdropsortablelist57.51.8 Drag/Drop codeeditor59.03.9 Adv. Editors Table 13: The 15 hardest canonical types by mean agent pass rate (averaged across all models and modes, excluding UI-TARS). Human steps column shows the mean normalized human reference steps. Components with≤2 human steps but<60% agent pass rate represent the human-agent difficulty inversion. J Failure taxonomy details and case studies J.1 Method The deterministic labeling in Table 4 parses every failed Pixel/SoM/AX-tree trace (8,864 traces across Gemini 3 Flash, Gemini 3.1 Flash-Lite, GPT-5 mini, GPT-5.4, and GPT-5.4 mini) into its action sequence and extracts feature flags: whether a drag was emitted, whether the target control was ever interacted with, whether coordinates repeat across steps, and whether a value was typed. Each trace starts from a component-driven category prior (e.g., slider tasks default toward continuous calibration) and is refined or overridden by the trace evidence;repetitionornoprogressloopis retained as a residual category only when the trace does not support a more specific mechanism, withotherorunclearas the final fallback. The Layer-2 column maps each failed run’s LLM-assignedprimaryfailurefamily and secondary tags onto the same nine categories; these labels are produced by GPT-5.4 (high reasoning effort) reading each failed run’s full action log and screenshots under a fixed JSON output schema. Categories that require low-level pointer evidence (continuous calibration, drag execution) are not separable in the Layer-2 labels and are folded into their nearest semantic categories there. J.2 Adversarially reviewed case studies We selected 20 representative failed traces spanning the major categories, wrote a mechanism narrative for each by reading the full action log and screenshots alongside the Layer-2 diagnosis, and then had an independent adversarial reviewer—a separately prompted Claude Opus 4.8 (claude-opus-4-8; Anthropic, 2026c) pass instructed to re-open the same evidence with skepticism and refute each narrative—re-check every case. Each case received a single review pass, issued through the Claude Code agent harness at its default sampling 26 Published as a conference paper at COLM 2026 and reasoning settings (i.e., not a deterministic decode); the review script ships with the benchmark. Fifteen cases were confirmed as labelled; five were relabelled by the reviewer, and we report the reviewer-adjusted category. Three abbreviated examples: Continuous calibration (meter-mui-T09, GPT-5.4 mini, Pixel). Instruction: drag the Server B load meter to 42% in a three-row table. The agent targets the correct row from the first step and successfully drags the bar (15%→28% across the run), but the bar spans only∼100px for 0–100%, so each coarse drag overshoots or undershoots; the agent issues progressively smaller nudges and times out at∼28%. The failure is value calibration, not grounding or instance selection. Target acquisition in drag (dragdropbetweenlists-antd-T01, GPT-5 mini, Pixel). In- struction: drag Editor from Available to Assigned roles. The drag primitive works—items visibly move—but the very first drag grabs the row above the intended one (aimed aty=350 for Editor; thatyis the Admin row), placing Admin into Assigned. The agent later also moves Editor but never removes the mis-dragged Admin, so the committed set fails exact-set verification. An off-by-one-row grab, not a drag-execution failure. Perception error masquerading as a loop (kanbanboarddragdrop-antd-T04, GPT- 5.4 mini, Pixel). Instruction: reorder the Review column to match a reference panel. The agent misreads the reference (quoting the column’s own current order back as the tar- get), concludes the board already matches, and spends all 20 steps re-asserting completion without ever issuing a drag. The root cause is a wrong belief about the target state; the loop is only the symptom. The five reviewer adjustments are themselves instructive: in each, a plausible category (e.g., “calibration error ”) was overturned by finer trace reading (e.g., the agent was monotonically ratcheting a spinner in the wrong direction—a no-progress loop, not calibration). All 20 case studies with trace pointers are released alongside the benchmark. K Human reference validation with additional annotators The human reference traces in the main experiments come from two passes by a single annotator (Section 2.2). To test whether the efficiency comparisons depend on that anno- tator’s idiosyncrasies, we collected recordings from two additional annotators on a fixed, pre-specified 278-task stratified validation subset: one task per realized (canonical type, library) pair, chosen by a fixed-seed randomized search (seed 2026) that minimizes deviation from the global difficulty distribution before inspecting any new results. The subset covers all 97 canonical types and all 14 families (bucket distribution easy 93 / mid 94 / hard 91), and contains 278 rather than 97×3=291 tasks because not every canonical type is implemented in every library. Each additional annotator followed the same protocol as the original reference: two recorded passes per task with the shorter successful pass kept, cleaned with the same normalization pipeline. All three annotators completed all 278 tasks successfully. Table 14 summarizes agreement on action tasks (258–265 per annotator; the remainder are hover-only tasks with zero normalized actions). Pairwise statistics are computed over the tasks where both annotators in the pair recorded at least one normalized action; hover-only tasks are excluded. On a like-for-like per-task basis, the human-to-human action-count ratio is 1.05×(each annotator versus the median of the others), whereas agents on the same subset take 1.27– 3.02×the human action count on tasks they solve (Appendix L). The agent–human efficiency gap is therefore several times larger than inter-annotator variation. We accordingly present the reference traces as practical successful references rather than claims of human optimality; the efficiency conclusions in Section 4.1 are robust to the choice of annotator. The recording sessions also served as an independent task-quality spot check: annotators could flag broken, ambiguous, or miswired tasks during recording, and no task in the subset was flagged—evidence against pervasive implementation or specification defects in this stratified subset. 27 Published as a conference paper at COLM 2026 MetricValue Mean normalized steps (Original / A1 / A2)2.92 / 2.90 / 3.14 Median normalized steps (all annotators)2 Pairwise step-count Pearson correlation0.79–0.94 Exact step-count agreement71%–84% Mean absolute step-count difference0.32–0.64 Table 14: Agreement between the original reference annotator and two additional annotators (A1, A2) on the 278-task validation subset (action tasks only). Human-to-human variation is small at the aggregate level. L Repeated-run stability Each model–mode combination in the main tables was evaluated in a single deterministic run. To quantify run-to-run variation, we ran each selected model–mode cell twice in total on the same 278-task validation subset (Appendix K) at an identical harness commit and model endpoint, for four models from two providers (Gemini 3 Flash, Gemini 3.1 Flash-Lite, GPT-5 mini, GPT-5.4 mini) under the two regimes that stress different variance sources: pure- visual Pixel and tool-rich Browser-Use. We report nonparametric bootstrap 95% confidence intervals (resampling tasks, B=10,000, seed 2026). ModelModeMean pass (%) Max dev Agree% Jaccard95% CI Gemini 3 FlashBrowser-Use95.11.197.50.974 [92.6, 97.3] Gemini 3 FlashPixel87.40.790.60.898 [84.0, 90.8] Gemini 3.1 Flash-Lite Browser-Use87.10.794.20.936 [83.3, 90.6] Gemini 3.1 Flash-Lite Pixel77.00.086.30.837 [72.5, 81.3] GPT-5 miniBrowser-Use86.71.492.10.913 [82.9, 90.3] GPT-5 miniPixel52.50.082.70.718 [47.1, 57.9] GPT-5.4 miniBrowser-Use85.41.191.00.900 [81.5, 89.0] GPT-5.4 miniPixel80.61.487.80.859 [76.3, 84.7] Table 15: Repeated-run stability on the 278-task subset (2 runs per cell, same harness commit and endpoint). Max dev: run-to-run pass-rate deviation (%). Agree%: task-level exact pass/fail agreement. Jaccard: overlap of passed-task sets. CI: bootstrap 95% interval on the mean pass rate. Three observations. (1) Aggregate pass rates are stable: run-to-run deviation is at most 1.4% (median 0.9%), far smaller than the mode and model effects in the main results, and the headline orderings (Gemini 3 Flash strongest; Browser-Use>Pixel for every model; GPT- 5 mini Pixel weakest) are identical in both runs. (2) Aggregate stability can hide task-level churn: GPT-5 mini Pixel has an identical pass rate in both runs (52.5%) yet 17.3% of tasks flip outcome, with pass→fail and fail→pass flips canceling in the aggregate. We therefore report task-level agreement and Jaccard alongside pass rates; conclusions about individual borderline tasks should be read with this churn in mind. Flips concentrate on mid/hard tasks and on the drag/slider components already identified as brittle. (3) Interaction counts are far more stable than wall-clock time: mean successful-task action counts change by at most 0.40 steps between runs, while mean durations shift by up to 11 seconds—so we base efficiency claims on action counts rather than latency-confounded durations. On the same subset, the per-task action-count ratio of each successful agent run to the human median ranges from 1.27×(GPT-5.4 mini and Gemini 3.1 Flash-Lite, Pixel) to 3.02×(GPT-5 mini, Pixel), the like-for-like comparison quoted against the 1.05×human–human baseline in Appendix K. M Experimental setup details This section documents the model access, coordinate handling, and framework configuration used in our experiments. 28 Published as a conference paper at COLM 2026 M.1 BrowserGym-based evaluation (AX-tree, SoM, Pixel) All models except UI-TARS-1.5-7B and Opus 4.6 are evaluated through a shared BrowserGym-based harness (de Chezelles et al., 2024) with a 1280×720 viewport, a maxi- mum of 20 steps per task, and a 600-second per-task wall-clock budget; the BrowserGym modes additionally enforce a 300-second per-step timeout. The three observation/action spaces (AX-tree, SoM, Pixel) differ only in what the agent sees and how it refers to targets; the underlying page, verifier, and termination logic are identical. In Pixel mode, different model families use different coordinate conventions: •Gemini 3 Flash and Gemini 3.1 Flash-Lite are accessed via the Google AI Studio API. These models output coordinates in a normalized 0–1000 space; the harness linearly maps them to screen pixels. •Qwen3-VL-235B-FP8 is served locally via vLLM (FP8 quantization, tensor-parallel). It also outputs 0–1000 normalized coordinates, mapped to pixels by the harness. •GPT-5.4 and GPT-5.4 mini are accessed via the OpenAI API directly. These models output raw pixel coordinates; no coordinate transformation is applied. • GPT-5 mini is accessed via a Duke LiteLLM proxy (OpenAI-compatible endpoint). It also outputs raw pixel coordinates with no transformation. All models receive a screenshot as a base64-encoded image. In AX-tree mode, the accessibility-tree text is appended to the prompt. In SoM mode, numbered bounding-box overlays are rendered onto the screenshot. M.2 UI-TARS-1.5-7B (native pixel mode) UI-TARS-1.5-7B (Qin et al., 2025) is evaluated through its own native harness rather than the shared BrowserGym pipeline, to match its training-time interface. The model is served via vLLM (bfloat16, single GPU, 32K context window,gpumemoryutilization=0.90). Screen- shots are rescaled usingsmartresize(from the Qwen-VL codebase) before being sent to the model. The model predicts coordinates in the resized-image space; the agent maps them back to the original 1280×720 screen pixels. The conversation uses a multi-turn format with a 4-turn sliding window for history, matching the model’s training setup. Temperature is set to 0.0, with up to 3 retry attempts per step for invalid actions. M.3 Opus 4.6 (Core Pixel only) Opus 4.6 is evaluated only on ComponentBench-Core (912 tasks) in Pixel mode. Screenshots are captured at the native 1280×720 viewport and anisotropically resized to 1024×768 (16:9 to 4:3, no padding or cropping) before being sent to Claude via Anthropic’s computer-use tool interface (Anthropic, 2026a). Claude outputs coordinates in 1024×768 space; the agent maps them back to 1280×720 using separate horizontal and vertical scale factors. The target resolution follows Anthropic’s XGA recommendation for optimal model accuracy. M.4 Browser-Use mode The Browser-Use observation/action space uses thebrowser-useframework, which pro- vides the agent with screenshot-based interaction together with serialized DOM information and grounded element references. Key parameters: • Viewport: 1280×720, headless Chromium • usevision=True, usethinking=True, flashmode=False • max actionsperstep=4, maxfailures=3, steptimeout=120s • highlight elements=False (no visual element highlighting) • Maximum 20 steps per task; the initial page-load step is excluded from step-count analysis 29 Published as a conference paper at COLM 2026 Gemini models are accessed via the Google AI Studio API; GPT models are accessed via the OpenAI API or Duke LiteLLM proxy. Qwen3-VL-235B’s Browser-Use runs are served through Amazon Bedrock’s OpenAI-compatible endpoint (qwen.qwen3-vl-235b-a22b); Bedrock does not document its serving precision, so these runs may differ slightly from the locally served FP8 configuration used for its AX-tree, SoM, and Pixel runs. 30

Instant research discovery