Paper deep dive
ClawBench: Can AI Agents Complete Everyday Online Tasks?
Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, Junwen Miao, Xuan Lu, Wendong Xu, Yunzhuo Hao, Songcheng Cai, Xiaochen Wang, Huaisong Zhang, Xian Wu, Yi Lu, Minyi Lei, Kai Zou, Huifeng Yin, Ping Nie, Liang Chen, Dongfu Jiang, Wenhu Chen, Kelsey R. Allen
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 99%
Last extracted: 4/10/2026, 4:32:48 AM
Summary
ClawBench is a new evaluation framework for AI agents consisting of 153 everyday online tasks across 144 live platforms. Unlike existing benchmarks that rely on static sandboxes, ClawBench operates on production websites using a lightweight interception layer to block final submissions, ensuring safety while maintaining real-world complexity. Evaluations of 7 frontier models reveal significant performance gaps, with top models achieving only 33.3% success, highlighting the difficulty of real-world, write-heavy web interactions.
Entities (5)
Relation Signals (3)
ClawBench â evaluates â AI Agents
confidence 100% ¡ ClawBench, an evaluation framework of 153 simple tasks that people need to accomplish regularly in their lives and work
Claude Sonnet 4.6 â performedon â ClawBench
confidence 100% ¡ Our evaluations of 7 frontier models show that both proprietary and open-source models can complete only a small portion of these tasks. For example, Claude Sonnet 4.6 achieves only 33.3%.
Agentic Evaluator â scores â AI Agents
confidence 100% ¡ Agent trajectories are then scored by comparing against these human references. We evaluate each recorded trajectory using an Agentic Evaluator
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:AI agents may be able to automate your inbox, but can they automate other routine aspects of your life? Everyday online tasks offer a realistic yet unsolved testbed for evaluating the next generation of AI agents. To this end, we introduce ClawBench, an evaluation framework of 153 simple tasks that people need to accomplish regularly in their lives and work, spanning 144 live platforms across 15 categories, from completing purchases and booking appointments to submitting job applications. These tasks require demanding capabilities beyond existing benchmarks, such as obtaining relevant information from user-provided documents, navigating multi-step workflows across diverse platforms, and write-heavy operations like filling in many detailed forms correctly. Unlike existing benchmarks that evaluate agents in offline sandboxes with static pages, ClawBench operates on production websites, preserving the full complexity, dynamic nature, and challenges of real-world web interaction. A lightweight interception layer captures and blocks only the final submission request, ensuring safe evaluation without real-world side effects. Our evaluations of 7 frontier models show that both proprietary and open-source models can complete only a small portion of these tasks. For example, Claude Sonnet 4.6 achieves only 33.3%. Progress on ClawBench brings us closer to AI agents that can function as reliable general-purpose assistants.
Tags
Links
- Source: https://arxiv.org/abs/2604.08523v1
- Canonical: https://arxiv.org/abs/2604.08523v1
Trouble viewing inline? Open PDF directly â
Full Text
41,257 characters extracted from source content.
Expand or collapse full text
ClawBench: Can AI Agents Complete Everyday Online Tasks? Yuxuan Zhang 1,2,3,§ , Yubo Wang 2,5 , Yipeng Zhu 1 , Penghui Du 3 , Junwen Miao 4 , Xuan Lu 6 , Wendong Xu 7,§ , Yunzhuo Hao 8 , Songcheng Cai 5 , Xiaochen Wang 9 , Huaisong Zhang 10 , Xian Wu 3 , Yi Lu 5 , Minyi Lei 5 , Kai Zou 11 , Huifeng Yin 7 , Ping Nie 5,§ , Liang Chen 7,â , Dongfu Jiang 2,5,â , Wenhu Chen 2,5,â , Kelsey R. Allen 1,2,â 1 University of British Columbia 2 Vector Institute 3 Etude AI 4 Carnegie Mellon University 5 University of Waterloo 6 Shanghai Jiao Tong University 7 UniPat AI 8 Zhejiang University 9 HKUST 10 Tsinghua University 11 Netmind.ai https://claw-bench.com Comprehensive Categories Daily Life (21) Academia (5) Job Search (8) Travel (13) Shopping (16) Social (8) Entertainment (15) Sandbox Evaluation Task Offline / Sandbox - Static HTML only - Fixed DOM structure - No login / auth - No dynamic content - No real-world challenge fake-shop.test Submit Controlled ¡ Oversimplified ¡ Unrealistic Check final page No explanation Real-World Evaluation (ClawBench - ours) Task - Cookie consent popups - Dynamic JS rendering - Complex and multi-step interaction Challenging ¡ Dynamic ¡ Realistic google.com/flights Search Flights Live Session Replay Action Screenshots HTTP Traffic Agentic Evaluator Rich Verdict score + justification + per-evidence references Traceable Analysis Performance Comparison (Traditional Benchmarks) Task Completion Rate % 0 20 40 60 80 Claude-sonnet-4.6GPT-5.4 6.5 33.3 67.3 66.4 75.0 72.5 OSWorld-VerifiedWebArena-Verified ClawBench Agent Messages Browser Actions Rating (10) Office (9) Personal Mgmt(4) Dev & Tech (15) Automation (3) Education (9) Pets (11) Finance (6) Figure 1: CLAWBENCH overview. Left: 153 tasks across 15 life categories. Middle: existing benchmarks evaluate agents in offline sandboxes with static HTML and fixed DOM struc- tures; CLAWBENCH evaluates on live websites with real-world complexity and provides rich, traceable verdicts via an agentic evaluator. Right: Claude-Sonnet-4.6 and GPT-5.4 achieve 65-75% task completion on established benchmarks such as OSWorld and WebArena, but only 33.3% and 6.5%, respectively, on CLAWBENCH, highlighting the difficulty of real-world everyday web tasks. 010203040 Claude Sonnet 4.6 GLM-5 Gemini 3 Flash Claude Haiku 4.5 GPT-5.4 Gemini 3.1 Flash Lite Kimi K2.5 33.3% 24.2% 19.0% 18.3% 6.5% 3.3% 0.7% Figure 2: Main results: success rate on CLAWBENCH for 7 frontier models. Even the strongest model (Claude Sonnet 4.6) completes only 33.3% of tasks, while two of seven models score below 5%. See Table 2 for the per-category breakdown. § Project Lead. â Advisors. 1 arXiv:2604.08523v1 [cs.CL] 9 Apr 2026 Abstract AI agents may be able to automate your inbox, but can they automate other routine aspects of your life? Everyday online tasks offer a realistic yet un- solved testbed for evaluating the next generation of AI agents. To this end, we introduce CLAWBENCH, an evaluation framework of 153 simple tasks that people need to accomplish regularly in their lives and work, spanning 144 live platforms across 15 categories, from completing purchases and booking appointments to submitting job applications. These tasks require demanding capabilities beyond existing benchmarks, such as obtaining relevant information from user-provided documents, navigating multi-step workflows across diverse platforms, and write-heavy operations like filling in many detailed forms correctly. Unlike existing benchmarks that evaluate agents in offline sandboxes with static pages, CLAWBENCH operates on production websites, preserving the full complexity, dynamic nature, and challenges of real-world web interaction. A lightweight interception layer captures and blocks only the final submission request, ensuring safe evalu- ation without real-world side effects. Our evaluations of 7 frontier models show that both proprietary and open-source models can complete only a small portion of these tasks. For example, Claude Sonnet 4.6 achieves only 33.3%. Progress on CLAWBENCH brings us closer to AI agents that can function as reliable general-purpose assistants. 1 Introduction AI agents powered by large language models can now navigate graphical interfaces, fill forms, and execute multi-step workflows autonomously (Yao et al., 2023; Wang et al., 2024), as demonstrated by commercial systems such as OpenAI Operator (OpenAI, 2025) and Anthropic Computer Use (Anthropic, 2025a), and the open-source agent OpenClaw (Stein- berger, 2025). However, the extent to which they can serve as truly general online assistants is unknown. To be general assistants, agents need to do more than summarize emails. They need to reliably handle the everyday online tasks that people depend on, for example book- ing flights, ordering groceries, and submitting job applications. These tasks are individually straightforward for a human, typically requiring under thirty minutes, yet they involve production websites with dynamic content, authentication flows, anti-bot defenses, and constantly evolving layouts. Evaluating agents on such tasks is challenging precisely because real websites are unpredictable and consequential. To avoid safety risks, most existing benchmarksâ WebArena (Zhou et al., 2024), VisualWebArena (Koh et al., 2024), OSWorld (Xie et al., 2024), TheAgentCompany (Xu et al., 2025)âevaluate agents in offline sandboxes with static HTML, fixed DOM structures, no authentication, and no dynamic content (Figure 1, middle top). This controlled setting simplifies evaluation but removes the very complexities that make real-world web interaction difficult: cookie consent pop-ups, dynamic JavaScript ren- dering, complex and multi-step interaction. Benchmarks that do operate on real websitesâ WebVoyager (He et al., 2024), AssistantBench (Yoran et al., 2024), Online-Mind2Web (Xue et al., 2025), Claw-Eval (Ye et al., 2026)âare limited to read-only information retrieval or mock APIs for testing simple write operations. As a result, write-heavy task completion on live platformsâthe category most directly relevant to peopleâs daily livesâremains unevaluated. We have no reliable picture of how well agents can actually âget things doneâ on the real web. We present CLAWBENCH, a benchmark of 153 everyday online tasks spanning 15 life categories across 144 live platforms (Figure 1, left). Rather than recreating websites in sandboxes, we let the agent operate on production sites and address the safety concern with a single targeted mechanism: a lightweight Chrome extension that records low-level browser actions, paired with a CDP-based instrumentation server that monitors outgoing network traffic and intercepts the final submission requestâ 2 the single HTTP call that would commit an irreversible transaction (Figure 1, middle bottom). During each task, five layers of behavioral data are captured (Figure 3): session replay via Xvfb virtual display and FFmpeg, per-step action screenshots, HTTP traffic, agent messages (reasoning traces and tool calls), and low-level browser actions (clicks, keystrokes, scrolls). Human annotators complete every task under the same setup to produce ground-truth trajectories. Agent trajectories are then scored by comparing against these human references. We evaluate each recorded trajectory using an Agentic Evaluator, implemented by invoking a Claude Code sub-agent under a fixed evaluation rubric. The evaluator consumes the task instruction together with the human reference actions and payloads and the agentâs executed actions and payloads, and judges whether the task was completed correctly. Instead of relying on a single final-state check, it applies predefined evaluation criteria to compare the agent trajectory against the human reference and produces a binary verdict with a structured justification grounded in the request schema and step-level evidence. As a result, the evaluation is not only outcome-based but also traceable: it reveals not just whether the agent failed, but which required fields or steps were incorrect and where the agent diverged from the reference trajectory. Human Task Task: Book a one-way economy-class flight from JFK to LAX on Dec 31 this year for me. ⢠Departure (JFK) ⢠Arrival (LAX) ⢠Class (Economy) ⢠Date (Dec 31) Verification Conditions Setup Execution Agent Recorded Trajectory Session Replay Action Screenshots HTTP Traffic Agent Messages (thinking, tool-call) Browser Actions Record Evaluation Evaluation Module Binary Score 1 / 1 Final step reached + all verify conditions met One way1Economy New York JFK Los Angeles LAX Dec 31 All filters StopsAirlinesBagsPrice Agentic Evaluator Figure 3: The CLAWBENCH evaluation pipeline. Setup: a human-authored task with explicit verification conditions. Execution: the agent operates in a real browser while five layers of behavioral data are recorded. Evaluation: the recorded trajectory is scored against a human ground-truth trajectory via an Agentic Evaluator, producing a binary pass/fail verdict with step-level justification. We evaluate 7 frontier models on CLAWBENCH: Claude Sonnet 4.6 (Anthropic, 2026), GPT-5.4 (OpenAI, 2026), Gemini 3.1 Flash Lite (Google DeepMind, 2026), Claude Haiku 4.5 (Anthropic, 2025b), Gemini 3 Flash (Google DeepMind, 2025), GLM-5 (Zeng et al., 2026), and Kimi K2.5 (Team et al., 2026). As shown in Figure 1 (right), Claude Sonnet 4.6 (An- thropic, 2026) and GPT-5.4 (OpenAI, 2026) score 65â75% on traditional web benchmarks (OSWorld (Xie et al., 2024), WebArena (Zhou et al., 2024)) but only 33.3% and 6.5%, respec- tively, on CLAWBENCH. The five-layer recording makes each failure traceable to specific steps, providing concrete diagnostic signals for future agent development. Our contributions are as follows: (1) We introduce CLAWBENCH, a benchmark of 153 everyday online tasks across 15 life categories and 144 live platforms, targeting write-heavy, consequential web interactions that existing benchmarks do not cover. 3 (2) We design a Chrome extension and CDP-based instrumentation mechanism that records browser actions and intercepts only the final submission request, enabling safe evaluation on production websites without sacrificing ecological validity. (3) We develop a five-layer recording infrastructure and an agentic evaluator that pro- vides step-level alignment against human ground truth, along with fully traceable failure diagnostics. (4) We open-source the complete data collection and evaluation pipeline to support community-driven benchmark maintenance and expansion. (5) We benchmark 7 frontier models and provide step-level failure analysis, documenting a large gap between agent performance on structured coding tasks and real-world everyday web tasks. 2 Benchmark We present CLAWBENCH, an evaluation infrastructure for measuring how well AI agents can complete everyday online tasks on the real web. Unlike prior benchmarks that retreat to sandboxes or restrict evaluation to read-only information retrieval, CLAWBENCH operates on 144 live platforms, targets write-heavy transactions (purchases, reservations, applica- tions), and verifies outcomes by comparing the agentâs full behavioral trajectory against a human ground-truth reference. Table 1 positions CLAWBENCH among existing web-agent benchmarks across four axes: real-world environment, write-heavy task coverage, evalua- tion reliability, and diagnostic traceability. Figure 3 illustrates the three-stage architecture: task definition, agent execution with five-layer recording, and automated evaluation via the Agentic Evaluator. Table 1: Comparison of CLAWBENCH with existing web agent benchmarks across four axes: real-world environment fidelity, write-heavy task coverage, evaluation reliability, and diagnostic traceability. CLAWBENCH is the only benchmark that combines live-website execution, write-heavy state-changing tasks, human-grounded comparative evaluation, and multi-layer trajectory recording. Environment: Offline = static cached pages; Sandbox = self-hosted or VM-based replicas; Real Web = live production websites. Task Type: Read-only = information retrieval / QA; Mixed = includes some write operations in controlled set- tings; Write-heavy = majority of tasks involve state-changing submissions on live platforms. Recording: None = no behavioral trace beyond the final output; Screenshot = per-step or final screenshots only; 5-Layer = session replay, action screenshots, HTTP traffic, agent messages, and browser actions. Human Traj.: whether full human ground-truth trajectories (recorded under the same infrastructure) are available for evaluation. BenchmarkEnvironment# Tasks# SitesTask TypeVerificationRecordingHuman Traj. Mind2Web (Deng et al., 2023)Offline (static traces)2,350137Read-onlyAction seq. matchNonePartial (action seq.) WebArena (Zhou et al., 2024)Sandbox (self-hosted)8125MixedScript-basedNoneâ VisualWebArena (Koh et al., 2024)Sandbox (self-hosted)9103MixedScript-basedNoneâ OSWorld (Xie et al., 2024)Sandbox (VM)3699MixedScript + screenshotScreenshotâ WebVoyager (He et al., 2024)Real Web64315Read-onlyLLM-as-judgeScreenshotâ TheAgentCompany (Xu et al., 2025)Sandbox (self-hosted)1756MixedCheckpoint-basedNoneâ Online-Mind2Web (Xue et al., 2025)Real Web300136Read-onlyHuman + LLM judgeScreenshotâ EconWebArena (Liu & Quan, 2025)Real Web36082Read-onlyExact numeric + URLNoneâ Claw-Eval (Ye et al., 2026)Sandbox (Docker + FastAPI)13915MixedAPI state checkCLI logsâ ClawBench (Ours)Real Web153144Write-heavyAgentic Evaluator5-Layerâ (all tasks) 2.1 Task Design and Collection CLAWBENCH focuses on write-heavy web tasks: actions that modify server-side state through form submissions, reservations, purchases, applications, and similar state-changing op- erations. We target these tasks because (i) they are the category most directly relevant to peopleâs daily livesâeach task is something an ordinary person might need to accomplish in 4 Table 2: Main results on CLAWBENCH. Success rate (%) of seven AI agents on the 153-task CLAWBENCH benchmark, reported overall and for each of the 8 high-level task category groups. Models are ordered by overall success rate. Bold marks the best result in each column;underlinemarks the second best.â denotes a text-only model without vision capability. RankModelOverall Task Categories DailyFinanceWorkDevAcademic TravelSocialPets 1Claude Sonnet 4.633.344.250.019.011.150.023.138.918.2 2GLM-5 â 24.230.816.738.116.728.60.016.718.2 3Gemini 3 Flash19.015.433.323.822.228.630.811.10.0 4Claude Haiku 4.5 18.315.433.319.027.821.47.716.718.2 5GPT-5.46.59.60.00.011.17.17.70.09.1 6Gemini 3.1 Flash Lite3.31.90.00.05.614.30.00.09.1 7Kimi K2.50.71.90.00.00.00.00.00.00.0 under thirty minutes, and (i) they produce observable HTTP payloads that enable objective verification. Candidate generation. We construct CLAWBENCH by curating realistic everyday online tasks on live websites and retaining 153 tasks in the final benchmark. Each task is defined by three elements: a natural-language user instruction, a starting URL, and a terminal submission target specified at the HTTP-request level. Human annotators survey represen- tative platforms across life categories, instantiate realistic user goals, and complete each task end-to-end within our recording framework. For every task, this yields a human reference trajectory together with the corresponding intercepted submission payload, which later serves as the basis for evaluation. Multi-stage filtering.We apply a rigorous filtering pipeline that removes tasks requiring paid subscriptions, geographically restricted services, or websites that have gone offline. At each stage, independent annotators verify that the task remains completable and that the ground-truth trajectory is reproducible. The final dataset comprises 153 tasks across 144 unique platforms. Interception signal annotation.A distinguishing aspect of our data collection is that every taskâs interception signalâthe specific HTTP endpoint, request method, and payload schema that identifies the dangerous, inreversable submissionâis manually annotated by a human expert. This annotator inspects the browserâs network traffic during the ground-truth execution, identifies the exact request that would commit the irreversible transaction, and records a declarative specification (URL pattern, HTTP method, and required payload field names). This human-in-the-loop design ensures that the ClawBench framework intercepts precisely the intended request, avoiding both false positives (blocking benign navigation requests) and false negatives (allowing the submission to reach the server). The result is a safe, targeted interception that preserves the full complexity of live-site interaction while guaranteeing zero real-world side effectsâno orders are placed, no applications are submitted, and no critical server-side state is modified. 2.2 Task Taxonomy To support fine-grained analysis, we organize the 153 tasks into a two-level taxonomy (Figure 4). The top level consists of 8 high-level category groupsâDaily, Work, Dev, Social, Academic, Travel, Pets, and Financeâthat capture broad domains of everyday web activity. Each group is further divided into 15 fine-grained categories (e.g., âDaily Life & Shopping & Entertainment,â âJob Search & Office & Personal Management,â âAcademic & Educationâ). 5 Daily [52] Work [21] Dev [18] Social [18] Academic [14] Travel [13] Pets [11] Finance [6] Daily Life [21] Shopping [16] Entertainment [15] Office [9] Job Search [8] Personal Mgmt [4] Dev & Tech [15] Automation [3] Rating [10] Social [8] Academia [5] Education [9] Travel [13] Pets [11] Finance [6] Tasks 153 Figure 4: Task taxonomy of CLAWBENCH. Inner ring: 8 high-level category groups; outer ring: 15 fine-grained categories. The dataset spans 153 tasks across diverse real-world domains. Score % (Claude-Sonnet-4.6) 0 20 40 60 80 100 33.3 51.1 66.4 72.5 77.6 88.0 PinchBenchClaw-Eval OSWorld-VerifiedWebArena-Verified WildClawBenchClawBench Benchmark Saturation Top models are saturating existing benchmarks ClawBench remains challenging Figure 5:Benchmark saturation comparison. Claude-Sonnet-4.6 per- forms substantially better on exist- ing web-agent benchmarks than on CLAWBENCH, indicating that CLAW- BENCH remains challenging for fron- tier agents. 2.3 Interception Mechanism The central design insight of CLAWBENCH is that evaluating agents on real websites does not require preventing them from interacting with real websitesâit only requires intercepting the final request. We implement this via a lightweight Chrome extension and a CDP server that is loaded alongside the agentâs browser session. How it works. The instrumentation server connects to the browser through CDP to monitor all outgoing HTTP requests against the human-annotated interception specifications (Section 2.1). When the agentâs action triggers a request matching the declared URL pattern and HTTP method, the system (i) captures the full request body, including all form fields, payloads, headers, and query parameters; (i) blocks the request before it leaves the browser, preventing it from reaching the server; and (i) logs the captured payload to a local file alongside a timestamp and the originating tab URL. All other requestsâpage loads, AJAX calls for dynamic content, image fetches, analytics pingsâpass through unmodified, so the agent experiences the website exactly as a human user would. Safety guarantees.Because interception signals are human-annotated at the endpoint level rather than inferred by heuristics, the mechanism achieves high precision: in a validation study over all 153 tasks, the extension correctly intercepted the terminal request in 100% of the human ground-truth runs with zero false positives on navigation traffic. This targeted approach avoids the risks associated with letting agents operate freely on production servers (e.g., accidentally placing orders or submitting real applications) while also avoiding the ecological validity loss inherent in sandbox-based alternatives. 2.4 Five-Layer Recording Infrastructure A distinctive feature of CLAWBENCH is that every agent run produces five synchronized layers of behavioral data, enabling both automated evaluation and deep post-hoc diagnosis. 6 (1) Session Recording. The Chrome browser is running on a Xvfb virtual display which is monitored by ffmpeg, producing a full-session video recording of the browser window. (2) Action Screenshots. A per-step screenshot is captured immediately after each agent browser action (click, type, scroll), providing a chronological visual record of the agentâs observations and the resulting page states. (3) HTTP Traffic. All HTTP requests are logged via the Chrome DevTools Protocol, including request bodies, payloads and timing information. The intercepted terminal payload is a special case of this layer. (4) Agent Messages. The full chain of reasoning traces, tool calls, and intermediate outputs produced by the agent framework is recorded in a structured JSON format, preserving the agentâs âthought processâ at each decision point. (5) Browser Actions.Low-level browser eventsâmouse clicks (with coordinates), keystrokes, scroll offsets, tab switches, and navigation eventsâare captured via the Chrome extension, providing a fine-grained behavioral log independent of the agentâs self-reported actions. Human annotators produce ground-truth recordings under the same five-layer setup for every task. The parallel structure between agent and human recordings is what enables the Agentic Evaluator (Section 2.5): both trajectories are represented in the same multi-modal format, making step-level alignment and comparison feasible. The five layers are complementary by design. Session recording and screenshots capture what the agent saw; agent messages capture what the agent thought; browser actions capture what the agent did; and HTTP traffic captures what effect the agentâs actions had on the network. Together, they enable a level of failure diagnosis that goes far beyond a binary pass/fail score: when an agent fails, a developer can trace the failure to the exact step, inspect the page state the agent observed, read the reasoning that led to the wrong decision, and compare the agentâs action against the human annotatorâs action at the corresponding step. Human Reference Trajectory Session Replay Action Screenshots HTTP Traffic Browser Actions Session Replay Action Screenshots HTTP Traffic Browser Actions Agent Messages Agent Trajectory Claude Code Cross-reference & verification Role: Judge task success from full trajectory Key Rules: 1.Must use provided personal info correctly 2.Must complete forms and submit 3.Interceptor / phone verification block: PASS if all prior actions are correct 4.CAPTCHA: must attempt, else FAIL Output Pass Fail Agent Prompt Figure 6: Agentic Evaluator Inference Pipeline. The evaluator determines whether a browser agent successfully completes a real-world web task by comparing the agent trajectory against a reference trajectory across five evidence layers: session replay, action screenshots, HTTP traffic, browser actions, and agent messages. A Claude Code sub-agent applies a struc- tured evaluation prompt that enforces key behavioral rules. The evaluator performs cross- referencing and verification over these multimodal traces to produce a final decision. PASS indicates the agent has effectively executed the task (including cases where completion is blocked but all prior actions are correct), while FAIL indicates unsuccessful execution or violation of required behaviors. 7 2.5 Evaluation Protocol Given the five-layer recordings from both the agent and the human annotator, we need an evaluation method that can reliably determine whether the agent completed the task correctly. We investigate three evaluator designs that operate on different subsets of the recorded data, each representing a different modality and comparison strategy. As illustrated in Figure 6, our evaluation is grounded in a multi-layer comparison between the agent trajectory and a human reference trajectory. The evaluator operates over synchronized evidence streamsâincluding session replay, screenshots, HTTP traffic, browser actions, and agent messagesâand applies a structured rubric to determine task success. Agentic Evaluator. We evaluate each task using a single Agentic Evaluator, implemented by invoking a Claude Code sub-agent under a fixed evaluation rubric. Given the task instruction, the agent trajectory, and the human reference trajectory, the evaluator performs an explicit alignment between the two executions: it identifies corresponding steps, detects divergences, checks whether the required fields and actions are correct, and determines whether the agent reaches a terminal state equivalent to the human reference. This design leverages the full multi-layer recordings from both runs and grounds evaluation in a concrete human demonstration rather than in the task instruction alone. Figure 7 provides a schematic view of this evaluation process, where the evaluator takes both the agent and human trajectories as input and produces a binary verdict with structured justification. Scoring.For each taskt â T, letq (t) denote the task instruction,T (t) a the recorded agent tra- jectory, andT (t) h the recorded human reference trajectory. We denote the Agentic Evaluator by A, which maps these inputs to a binary task-level verdict: Score(t) = A q (t) , T (t) a , T (t) h ,(1) whereScore(t) â 0, 1, with 1 indicating successful task completion and 0 indicating failure. The overall success rate over a task set T is then defined as SR = 1 |T| â tâT Score(t),(2) where |T| is the number of evaluated tasks. Unlike evaluators that judge an agent trajectory in isolation,Aperforms an explicit com- parison between the agent trajectory and the human reference trajectory under a fixed evaluation rubric. This comparative signal provides a concrete specification of successful task completion, including platform-specific details such as field bindings, interaction order, and terminal submission structure, which may be difficult to infer reliably from the task instruction alone. 3 Experiments We evaluate 7 frontier AI models on CLAWBENCH, spanning both proprietary and open- source systems. This section describes the experimental setup, presents the main results, analyzes failure modes, and reports ablation studies on the observation modality. 3.1 Experimental Setup Models. We evaluate 5 proprietary models (Claude Sonnet 4.6 (Anthropic, 2026), GPT- 5.4 (OpenAI, 2026), Gemini 3.1 Flash Lite (Google DeepMind, 2026), Claude Haiku 4.5 (An- thropic, 2025b), and Gemini 3 Flash (Google DeepMind, 2025)) and 2 open-source models (GLM-5 (Zeng et al., 2026) and Kimi K2.5 (Team et al., 2026)). 8 Input Evaluator Output (0 / 1) Human Actions Human Action JSON payloads Agent Actions Agent Action JSON payloads Agentic Evaluator Claude Code sub- agent invoked under a fixed evaluation rubric Justification (from schema) âdeparture = Toronto arrival = New York dep_date = Aug 01 return_date = Aug 07 type = direct flight â Each schema field checked against request payload Agent Input Task Instruction Figure 7: Evaluation protocol. The evaluator takes as input the task instruction together with the human reference actions and payloads and the agentâs executed actions and payloads. It then invokes a Claude Code sub-agent under a fixed evaluation rubric to determine whether the task was completed correctly, producing a binary verdict and a structured justification grounded in schema-level checks over the request payload. Infrastructure. Each model is given control of a Chromium browser instance via the OpenClaw agent framework. The OpenClaw agent is connected to a managed isolated browser. It then can use the OpenClaw frameworkâs browser toolset to interact with the browser (click, type, scroll, navigate, etc.). The CLAWBENCH Chrome extension and CDP instrumentation server runs in the background throughout the session, intercepting HTTP requests and agent actions for post-hoc evaluation. Reproducibility controls.Each benchmark run uses an encapsulated container, ensuring environment isolation and cross-run consistency. Chrome is launched with flags that disable UI prompts, sync, and irrelevant extensions to minimize environmental variance. Metrics. Our primary metric is success rate (SR): the percentage of tasks for which the agent receives a binary score of 1 (Section 2). We report SR both overall and broken down by the 8 high-level category groups. 3.2 Main Results Table 2 reveals three main findings. First, Claude Sonnet 4.6 is the strongest model overall, achieving 33.3% success rate, followed by GLM-5 at 24.2%. This gap suggests that frontier models differ substantially in their ability to handle realistic everyday web workflows. Second, model performance varies considerably across task categories. Claude Sonnet 4.6 leads on Daily, Finance, Academic, and Social, while GLM-5 performs best on Work, Gemini 3 Flash on Travel, and Claude Haiku 4.5 on Dev. These patterns indicate that current agents do not yet exhibit uniform competence across domains, but instead show category-specific strengths and weaknesses. At the same time, even the best category-level results remain far from saturation, underscoring the overall difficulty of CLAWBENCH. 4 Related Work Web Agent Benchmarks.Early web agent benchmarks such as MiniWoB (Shi et al., 2017) evaluated agents on simplified, synthetic web interfaces with short action sequences. We- bArena (Zhou et al., 2024) introduced self-hosted, realistic web environments with 812 tasks across 5 domains, using URL and element matching for evaluation. VisualWebArena (Koh et al., 2024) extended this to visually grounded tasks on 3 self-hosted sites. Mind2Web (Deng et al., 2023) scaled to 2,350 tasks on 137 real-world domains but evaluated action sequences rather than end-to-end task completion. OSWorld (Xie et al., 2024) broadened 9 the scope to full operating system tasks across 9 applications. More recently, REAL Bench (Garg et al., 2025) evaluated agents on live websites but relied on manual rating for scoring. CLAWBENCH differs from all prior work by (i) operating on 144 live platforms rather than self-hosted sandboxes, (i) focusing on write-heavy, state-changing tasks, and (i) providing traceable, comparative evaluation against human reference trajectories through an agentic evaluator. LLM-Based Web Agents. The emergence of large language models has driven rapid progress in autonomous web agents. Systems such as WebGPT (Nakano et al., 2021), We- bAgent (Gur et al., 2023), and SeeAct (Zheng et al., 2024) demonstrated that LLMs can interpret web pages and execute multi-step browsing tasks when given appropriate observa- tion and action interfaces. Recent approaches combine visual perception (screenshots) with structured page representations (accessibility trees, HTML) to improve grounding accuracy. Agent frameworks including AgentGPT, AutoGPT, and OpenClaw provide standardized interfaces for deploying LLMs as web agents with tool use and action execution capabili- ties. CLAWBENCH is designed to evaluate any agent system that can control a Chromium browser, independent of the underlying model or framework. Evaluation Methods for Agent Systems. Evaluating autonomous agents remains chal- lenging due to the diversity of possible action trajectories and the difficulty of defining success criteria. Prior work has used action sequence matching, URL-based success detec- tion, screenshot comparison, and human judgement. Action-level metrics suffer from the problem of multiple valid paths: an agent may complete a task correctly through a different sequence of actions than the reference trajectory. Screenshot-based methods require visual similarity thresholds that introduce non-determinism. Human evaluation, while flexible, is expensive and non-reproducible. CLAWBENCH sidesteps these issues by combining intercepted submission payloads with an agentic evaluator that performs explicit step-level alignment between the agent trajectory and a human reference trajectory, producing a binary verdict together with a structured justification grounded in the recorded evidence. Concurrent and Complementary Work. Several recent benchmarks address related but distinct aspects of web agent evaluation. TheAgentCompany (Xu et al., 2025) provides a self-hosted sandbox simulating a software company with 175 tasks and checkpoint-based partial credit; CLAWBENCH trades environmental control for real-world breadth across 144 live platforms. EconWebArena (Liu & Quan, 2025) is a live-web benchmark for economic research tasks featuring 360 read-only tasks with exact numeric matching and URL prove- nance; CLAWBENCH extends the live-web paradigm to write-heavy, state-changing tasks. MCP-Bench (Wang et al., 2025) evaluates LLM agents on tool invocation via the Model Context Protocol with strict schema validation; CLAWBENCH targets browser-based web interaction rather than structured API calls. TrickyArena (Ersoy et al., 2025) studies dark pattern susceptibility in web agents across 4 controlled applicationsâan orthogonal safety concern that highlights the importance of evaluating on real websites where dark patterns occur naturally. AssistantBench (Yoran et al., 2024) defines 214 realistic open-web tasks with automated evaluation, focusing on information retrieval; CLAWBENCH complements this with write-heavy tasks. WebCanvas (Pan et al., 2024) proposes key-node evaluation for 542 tasks on dynamic websites in a similar live-web setting but without HTTP payload verification. Taken together, these efforts illustrate a fundamental realism-vs-reproducibility trade-off: sandboxed benchmarks offer perfect reproducibility but may not reflect the com- plexity of real websites, while live-web benchmarks expose agents to authentic challenges at the cost of environmental variability. CLAWBENCH deliberately chooses realism and mitigates reproducibility concerns through human-grounded comparative evaluation and full multi-layer trajectory recording. 5 Conclusion We introduce CLAWBENCH, a benchmark of 153 real-world everyday web tasks spanning 144 live platforms across 8 high-level category groups. By evaluating agents on live pro- duction websites and focusing on write-heavy, state-changing workflows, CLAWBENCH 10 provides a substantially more realistic testbed than prior benchmarks built on static pages or sandboxed environments. Our framework combines final-request interception, five-layer trajectory recording, and an agentic evaluator. Experiments on 7 frontier models show that strong performance on existing web-agent benchmarks does not transfer to CLAWBENCH, underscoring the gap between controlled benchmark success and real-world everyday web competence. We release the benchmark, evaluation toolkit, and supporting infrastructure to support future research on realistic web-agent evaluation. References Anthropic.Introducingcomputeruse.https://w.anthropic.com/news/ 3-5-models-and-computer-use, 2025a. Accessed: 2026-03-20. Anthropic. Claude haiku 4.5.https://w.anthropic.com/news/claude-haiku-4-5, 2025b. Anthropic. Claude sonnet 4.6.https://w.anthropic.com/news/claude-sonnet-4-6, 2026. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Devin Ersoy, Brandon Lee, Ananth Shreekumar, Arjun Arunasalam, Muhammad Ibrahim, Antonio Bianchi, and Z Berkay Celik. Investigating the impact of dark patterns on llm-based web agents. arXiv preprint arXiv:2510.18113, 2025. Divyansh Garg, Shaun VanWeelden, Diego Caples, Andis Draguns, Nikil Ravi, Pranav Putta, Naman Garg, Tomas Abraham, Michael Lara, Federico Lopez, et al. Real: Bench- marking autonomous agents on deterministic simulations of real websites. arXiv preprint arXiv:2504.11543, 2025. Google DeepMind. Gemini 3 flash.https://deepmind.google/models/gemini/flash/, 2025. Google DeepMind.Gemini 3.1 flash-lite.https://deepmind.google/models/gemini/ flash-lite/, 2026. Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Saber, Yutaka Matsuo, Douglas Eck, and Aleksandra Fishi. A real-world WebAgent with planning, long context understanding, and program synthesis. arXiv preprint, 2023. arXiv:2307.12856. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. In Annual Meeting of the Association for Computational Linguistics (ACL), 2024. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. In Annual Meeting of the Association for Computational Linguistics (ACL), 2024. Zefang Liu and Yinzhu Quan. Econwebarena: Benchmarking autonomous agents on economic tasks in realistic web environments. arXiv preprint arXiv:2506.08136, 2025. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. WebGPT: Browser-assisted question-answering with human feedback. arXiv preprint, 2021. arXiv:2112.09332. OpenAI. Introducing operator. https://openai.com/index/introducing-operator/, 2025. OpenAI. Gpt-5.4. https://platform.openai.com/docs/models/gpt-5.4, 2026. 11 Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. Webcanvas: Benchmarking web agents in online environments. arXiv preprint arXiv:2406.12373, 2024. Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In Proceedings of the 34th International Conference on Machine Learning, p. 3135â3144. PMLR, 2017. Peter Steinberger. Openclaw: Your own personal AI assistant.https://github.com/ openclaw/openclaw, 2025. Accessed: 2026-03-20. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024. Zhenting Wang, Qi Chang, Hemani Patel, Shashank Biju, Cheng-En Wu, Quan Liu, Aolin Ding, Alireza Rezazadeh, Ankit Shah, Yujia Bao, et al. Mcp-bench: Benchmarking tool-using llm agents with complex real-world tasks via mcp servers. arXiv preprint arXiv:2508.20453, 2025. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems (NeurIPS), 2024. Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: Benchmarking LLM agents on consequential real world tasks. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2025. Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents. In Conference on Language Modeling (COLM), 2025. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. Bowen Ye, Rang Li, Qibin Yang, and Lei Li. Claw-eval: A transparent benchmark for real-world agents.https://github.com/claw-eval/claw-eval, 2026. Peking University & University of Hong Kong. Accessed: 2026-03-20. Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. Assistantbench: Can web agents solve realistic and time-consuming tasks? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 8938â8968, 2024. Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chengxing Xie, Cunxiang Wang, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. SeeAct: GPT-4V(ision) as a generalist web agent, if grounded. arXiv preprint, 2024. arXiv:2401.01614. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations (ICLR), 2024. 12