Paper deep dive

WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing

Fanheng Kong, Jingyuan Zhang, Yang Yue, Chenxi Sun, Yang Tian, Shi Feng, Xiaocui Yang, Daling Wang, Yu Tian, Jun Du, Wenchong Zeng, Han Li, Kun Gai

Year: 2026Venue: arXiv preprintArea: cs.SEType: PreprintEmbeddings: 90

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%

Last extracted: 3/27/2026, 1:34:16 AM

Summary

WebTestBench is a benchmark designed to evaluate end-to-end automated web testing capabilities of computer-use agents (CUAs). It addresses the limitations of existing benchmarks by focusing on latent logical constraints and long-horizon interactions. The authors introduce WebTester, a two-stage framework (checklist generation and defect detection) to evaluate LLMs, revealing significant performance gaps in industrial-grade web testing.

Entities (5)

WebTestBench · benchmark · 100%WebTester · framework · 100%Computer-Use Agents · technology · 95%Large Language Models · technology · 95%Lovable.dev · platform · 90%

Relation Signals (3)

WebTester → evaluates → WebTestBench

confidence 95% · propose WebTester, a baseline framework for this task [on WebTestBench]

WebTestBench → evaluates → Computer-Use Agents

confidence 95% · WebTestBench, a benchmark for evaluating end-to-end automated web testing [by computer-use agents]

WebTestBench → uses → Lovable.dev

confidence 90% · we utilize Lovable.dev... to synthesize web application projects [for WebTestBench]

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:The emergence of Large Language Models (LLMs) has catalyzed a paradigm shift in programming, giving rise to "vibe coding", where users can build complete projects and even control computers using natural language instructions. This paradigm has driven automated webpage development, but it introduces a new requirement about how to automatically verify whether the web functionalities are reliably implemented. Existing works struggle to adapt, relying on static visual similarity or predefined checklists that constrain their utility in open-ended environments. Furthermore, they overlook a vital aspect of software quality, namely latent logical constraints. To address these gaps, we introduce WebTestBench, a benchmark for evaluating end-to-end automated web testing. WebTestBench encompasses comprehensive dimensions across diverse web application categories. We decompose the testing process into two cascaded sub-tasks, checklist generation and defect detection, and propose WebTester, a baseline framework for this task. Evaluating popular LLMs with WebTester reveals severe challenges, including insufficient test completeness, detection bottlenecks, and long-horizon interaction unreliability. These findings expose a substantial gap between current computer-use agent capabilities and industrial-grade deployment demands. We hope that WebTestBench provides valuable insights and guidance for advancing end-to-end automated web testing. Our dataset and code are available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

89,351 characters extracted from source content.

Expand or collapse full text

WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing Fanheng Kong 1,2 * Jingyuan Zhang 2 Yang Yue 2 Chenxi Sun 2 Yang Tian 2 Shi Feng 1 † Xiaocui Yang 1 Daling Wang 1 Yu Tian 2 Jun Du 2 Wenchong Zeng 2 Han Li 2 Kun Gai 2 1 Northeastern University 2 Kuaishou Technology Abstract The emergence of Large Language Models (LLMs) has catalyzed a paradigm shift in pro- gramming, giving rise to “vibe coding”, where users can build complete projects and even con- trol computers using natural language instruc- tions. This paradigm has driven automated web- page development, but it introduces a new re- quirement about how to automatically verify whether the web functionalities are reliably im- plemented. Existing works struggle to adapt, relying on static visual similarity or predefined checklists that constrain their utility in open- ended environments. Furthermore, they over- look a vital aspect of software quality, namely latent logical constraints. To address these gaps, we introduce WebTestBench, a benchmark for evaluating end-to-end automated web testing. WebTestBench encompasses comprehensive di- mensions across diverse web application cate- gories. We decompose the testing process into two cascaded sub-tasks, checklist generation and defect detection, and propose WebTester, a baseline framework for this task. Evaluating popular LLMs with WebTester reveals severe challenges, including insufficient test complete- ness, detection bottlenecks, and long-horizon interaction unreliability. These findings expose a substantial gap between current computer-use agent capabilities and industrial-grade deploy- ment demands. We hope that WebTestBench provides valuable insights and guidance for advancing end-to-end automated web testing. Our dataset and code are available athttps:// github.com/friedrichor/WebTestBench. 1 Introduction Web applications, serving as foundational plat- forms supporting a wide range of daily activities, provide rich services and content, becoming an in- dispensable pillar of modern society (Foundation, * Work done during an internship at Kuaishou Technology. † Corresponding Author. 2026). As user demand grows, web pages prolifer- ate rapidly, with increasing functionality and com- plexity. However, developing a web application is complicated and time-consuming, requiring devel- opers to master diverse technical skills (e.g., CSS, JavaScript), and adapt to evolving frameworks (e.g., React, Vue) (Borges and de Souza, 2024). More- over, a complete development lifecycle is collabo- rative, spanning specialized roles such as coding, testing, and documentation (Liang et al., 2022). The rise of LLMs has provided support for ef- ficient coding (Fried et al., 2023; Li et al., 2023; Jiang et al., 2024). Agentic coding products, exem- plified by Claude Code (Anthropic, 2026), Gem- ini (Google, 2026), and Codex (Openai, 2026), are rapidly gaining traction, enabling non-expert users to build complete web applications from scratch via natural language instructions, effectively compress- ing the traditional multi-person development team into a single human-AI collaborative unit. How- ever, these products rarely generate flawless out- comes in a single attempt, and the synthesized ap- plications frequently exhibit omissions or defects in both appearance and functionality (Lu et al., 2025b; Wan et al., 2025). For creators lacking software engineering expertise, verifying the quality and re- liability of these generated applications remains a barrier. As code generation becomes highly auto- mated, the critical bottleneck in modern web de- velopment has decisively shifted toward automated web testing (Le et al., 2025; Ye et al., 2025). Despite this shift, current research struggles to provide sufficient evaluation frameworks for this emerging paradigm. Existing efforts often emerge as byproducts of web generation tasks. Research that reconstructs web designs from UI screenshots assess generation quality through static visual sim- ilarity (Si et al., 2025; Xiao et al., 2025) or iso- lated interactive components (Xiao et al., 2024). A more realistic approach synthesizes target web- pages from text instructions and employs computer- 1 arXiv:2603.25226v1 [cs.SE] 26 Mar 2026 Benchmark#SampleTaskEnvChecklistInte.L.H.Func.Cons.Test Method Application Quality Evaluation Methods Originate from Web Generation Tasks Web2Code (Yun et al., 2024)60KGenerationWeb✗Visual Interaction2Code (Xiao et al., 2024)127GenerationWeb✗✓✗✓✗Visual+Rule Design2Code (Si et al., 2025)484GenerationWeb✗Visual DesignBench (Xiao et al., 2025)900GenerationWeb✗Visual+Rule+MLLM WebGen-Bench (Lu et al., 2025b)101GenerationWeb✗✓✗Agent ArtifactsBench (Zhang et al., 2025)1,825GenerationWeb✗✓✗Visual+Script+MLLM WebCoderBench (Liu et al., 2026)1,572GenerationWeb✗✓✗✓✗Rule+LLM Application Testing Benchmarks GTArena (Zhao et al., 2024a)10,844TestingMobile✓✗✓✗Agent GUITestBench (Gao et al., 2026)143TestingMobile✗✓✗Agent PlanATA (Chevrot et al., 2025)113TestingWeb✗✓✗Agent WebTestPilot (Teoh et al., 2026)100TestingWeb✗✓✗Agent WebTestBench100TestingWeb✓Agent Table 1: Comparison with representative related benchmarks across several aspects: the number of samples (#Sample); the primary task (Task); the app environment (Env); whether the evaluation supports end-to-end testing without manually predefined test items (Checklist); whether testing requires dynamic interaction with the app (Inte.); whether it involves long-horizon interactions (L.H.); whether functional correctness is evaluated (Func.); whether latent logical constraints are evaluated (Cons.); and the approach used to execute tests (Test Method). use agents (CUAs) to simulate human behavior for dynamic functional evaluation (Lu et al., 2025b; Zhang et al., 2025). Nevertheless, these methods heavily rely on predefined, rigid checklists, which constrain their utility in open-ended development environments. Furthermore, the unreliability of CUAs in complex web environments (Zhou et al., 2024; Xie et al., 2024) introduces execution bias into the functional evaluation process. On the other hand, recent testing application testing methods are similarly constrained by human-written checklists. Crucially, existing works overlook a fundamental aspect of software quality: potential logical con- straints, e.g., in a meeting room reservation system, the same room cannot be double-booked within an overlapping time slot. To address these gaps, we introduce WebTest- Bench, a benchmark for end-to-end long-horizon automated web testing, requiring agents to gen- erate checklist and perform defect detection. To ground evaluation in realistic AI-driven web devel- opment scenarios, we synthesize 100 web applica- tions spanning seven categories. We evaluate CUA performance along four quality dimensions: func- tionality, constraint, interaction, and content. Be- yond standard functional verification and content relevance, WebTestBench also emphasis on latent logical constraints, a critical yet overlooked dimen- sion in existing works. To evaluate LLM perfor- mance on WebTestBench, we design WebTester, a two-stage framework for end-to-end web testing comprising a Checklist Generation Agent and a De- fect Detection Agent. Building on this framework, we conduct a comprehensive evaluation of several powerful LLMs. The results are sobering: all eval- uated models score below30%F1 on WebTest- Bench, revealing a substantial gap between current CUA capabilities and the demands of industrial- grade web testing deployment. In summary, our contributions are: •We introduce WebTestBench, a meticulously an- notated web testing benchmark that simulates end-to-end automated testing scenarios for mod- ern AI-driven web development, accompanied by a suite of evaluation metrics. •We design WebTester, a baseline framework for end-to-end automated web testing built upon the WebTestBench evaluation protocol. •We conduct a comprehensive evaluation of sev- eral popular LLMs, uncovering their strengths and weaknesses across various dimensions. Hopefully, these findings provide solid guidance for advancing automated web testing in the era of vibe coding. 2 Related Work Computer-Use Agents. Beyond pure code genera- tion, CUAs have emerged as a tool for interacting with web environments. Through screenshots (Lin et al., 2025a; Nayak et al., 2025; Wang et al., 2025) or accessible code (e.g., DOM trees) (Deng et al., 2023; de Chezelles et al., 2025), CUAs can execute multi-step tasks involving navigation, clicking, and typing. Current benchmarks (Zhou et al., 2024; Xie et al., 2024; Miyai et al., 2025) primarily focus on operational success in realistic web scenarios, such as online shopping and discussion forums. While CUAs have demonstrated potential in general web 2 Instruction Collection 1 Web App Idea Collection Filter & Standardization Developer Communities, Blogs Highly Interactive, Explicit Perspective, Function-focused 2 Perspective: User, Manager Web Development Instructions 7 Category: Presentation, Search, Tool, Commerce, Data Management, Workflow, User-Generated Content Web App Synthesis 2 Instructions Instruction Checklist (4 dimensions) I need a webpage where I can create and manage employee shift schedules. Each employee should have a profile listing their skills ... Operate Webpage (e.g., Navigate, Click, Type) • Users can create employee profiles that include relevant skills. (Functionality) • The same employee cannot be assigned overlapping shifts within ... (Constraint) • The interface provides a visual distinction between draft schedules... (Interaction) • The weekly schedule must clearly show the date, time, and ... (Content) ... Checklist and Result Annotation 3 Result • Users can create employee profiles that include relevant skills. * Status: PASS • The same employee cannot be assigned overlapping shifts within the same time period. * Status: FAIL * Bug: The same employee can be scheduled for overlapping shifts within the same time period. Synthetic Web Application Iterative Refinement Annotation Check & Feedback Human Annoators Web Expert Figure 1: Overview of WebTestBench construction. activities, their effectiveness in software quality assurance, particularly defect detection, remains insufficiently explored. Web Testing Benchmarks. Driven by the remark- able capabilities of CUAs and GUI agents in web task automation, recent works have introduced benchmarks to evaluate automated application test- ing. GTArena (Zhao et al., 2024a) proposes a com- prehensive evaluation framework for automated GUI testing, decomposing the process into test in- tention generation, test task execution, and GUI defect detection. However, its defect detection is limited to atomic-level bugs inferred from single- step before-and-after state comparisons. GUITest- Bench (Gao et al., 2026) focuses on exploratory GUI defect discovery. Chevrot et al. (2025) in- vestigates the feasibility of adapting automated web agents as automated test agents, introducing a benchmark to assess their ability to execute human- written test cases. Teoh et al. (2026) advances this further by additionally examining whether agents can accurately determine defect verdicts. Despite this progress, these works rely on human-written checklists, which presents a barrier for non-expert developers who lack the expertise to write compre- hensive test cases. Furthermore, none of them ac- count for latent logical constraints. WebTestBench aims to bridge these gaps, targeting the realistic capability of CUAs in checklist-free end-to-end automated web testing, and filling a critical miss- ing piece in the application development lifecycle enabled by human-AI collaborative units. 3 WebTestBench The advent of AI-driven web development enable users to build web applications via natural lan- guage, yet introduces a new requirement, where novice creators lack the software engineering exper- tise to verify whether functions are reliably imple- mented. To systematically evaluate testing agents within this paradigm, we introduce WebTestBench. In this section, we define the end-to-end automated web testing task, followed by a detailed description of our data construction and evaluation protocol. 3.1 Task Definition Given a user instruction and a corresponding web application, the target is to evaluate the end-to-end automated web testing capability of computer-use agents. Specifically, each agent is required to com- plete a two-stage task: (1) Checklist Generation. The agent decomposes the instruction into a set of verifiable test items,T pred =p 1 ,...,p m , where each test itemp j describes a concrete behavior or property of the web application that can be vali- dated through interaction. (2) Defect Detection. The agent plans and executes all test items by sim- ulating human interactions (e.g., clicking, typing) within the web environment, and determines an ex- ecution statuss j ∈Pass, Failfor each item. The final output of the agent is a set of predicted test item-result pairsP =(p j ,s j ) m j=1 , which jointly represent the evaluation of the web application. 3.2 Data Construction Figure 1 illustrates the workflow for WebTestBench construction, primarily consisting of instruction collection, web application synthesis, and checklist and result annotation. Instruction Collection. To ensure the diversity and practical relevance of our benchmark, we ini- tially collect 451 trending web application ideas from diverse developer communities and blogs. We eliminate highly similar duplicates and filter these raw ideas to isolate those aligned with our eval- uation objectives, prioritizing attributes such as interactivity, practicality, and testability. These ideas range from brief design sketches to overly verbose descriptions. Sketches are often too vague, while detailed descriptions include irrele- vant content such as background narratives, refer- 3 I want a pet adoption website where I can search and filter adoptable animals from multiple shelters, using options like species, age range, size, and whether they’re good with kids or other pets. I should see a clear list of matching pets, sort them by relevance or time listed, open a detailed profile for each pet with photos and a description, save favorites to a simple list, and send an adoption inquiry through a form on the pet’s page. 1. Users can filter animals for adoption based on species, age range, size, and compatibility (whether they are suitable for children/other pets). ... 1. Before submitting the adoption consultation form, the required fields (such as name and contact information) must be validated to prevent sending empty information or incorrectly formatted requests. • Bug: You can submit the form without filling in your phone number. ... Development Instruction Checklist 1. When a user clicks the "Save" button, the icon should immediately change visually (e.g., from hollow to solid). ... 1. All pet photos shown must be related to the specific pet shown (placeholders or generic images may not be used in place of the actual pet). • Bug: The same pet picture does not correspond to the same breed. ... Functionality Content Constraint Interaction Webpage Screenshots Figure 2: An illustrative example from WebTestBench. The web application is built from a given development instruction, typically comprising multiple pages with rich interactive functionality. The benchmark provides a gold checklist across four quality dimensions: Functionality, Constraint, Interaction, and Content. Each test case is annotated with a binary Pass/Fail verdict. ences to commercial websites, and advertisements. We therefore rewrite each idea into a well-formed instruction usingGPT-5.1. The model assigns each idea a specific perspective (i.e., User or Administra- tor) to disambiguate permission logic, categorizes it into one of seven functional themes (i.e., Pre- sentation, Search, Tool, Commerce, Data Manage- ment, Workflow, and User-Generated Content), and produces a formal development instruction. Subse- quently, human annotators review and refine these instructions to ensure quality and standardization. Finally, we sample 100 instructions and maintain a balanced distribution across all categories. Web Application Synthesis. Evaluating web test- ing agents requires environments that are ecolog- ically valid and contain diverse and non-trivial defects. However, standard web resources of- ten present limitations. Commercial websites are typically well tested and continuously updated, which makes them unsuitable as benchmark sam- ples. Open-source projects often feature simple de- signs with shallow structures or limited interactive functionality, and therefore fail to reflect realistic user interactions. To bridge this gap, we utilize Lovable.dev 1 , an AI-powered web development platform that generates complete websites from user instructions, to synthesize web application projects. Through this process, we obtain an initial web application for each instruction, providing a realistic webpages for defect detection. Gold Checklist and Result Annotation. Given an development instruction and its application, human annotators construct a testable checklist. Inspired by software quality and evaluation standards (e.g., ISO/IEC 25010 (ISO/IEC, 2023)), and adapting 1 https://lovable.dev/ them to the features of our benchmark, we catego- rize test cases into four dimensions: Functionality, Constraint, Interaction, and Content. Beyond iso- lated atomic interaction tests, we emphasize long- horizon interactions and latent logical constraints. For example, in a “meeting room booking” system, a functional test item would not merely be “click a button”, but a complete sequence: “select a specific room, complete the reservation form, and verify the successful status update.” Additionally, potential logical checks are also necessary, such as “the same meeting room cannot be reserved by two different users for the overlapping time slots.” Annotators derive a checklist from the instruc- tion, and then interact with the actual application to align test cases with the implemented components and interaction flows. Finally, annotators execute the checklist by interacting with the website and document the Pass/Fail status of each item and pro- vide concise bug reports for failures. Iterative Refinement. Initial synthesis often pro- duces applications with few defects, which limits the effectiveness in discriminatively evaluating web testing capabilities. To address this, annotators per- form a iterative refinement. This involves revising the instruction for re-generate app or continuing the conversation withlovable.devto add new fea- tures. Throughout this iteration, the checklist and results are updated synchronously until the samples contain sufficient defects for evaluation. Quality Control. To ensure the quality of the dataset, all annotators undergo related training and conduct cross-validation during the annotation pro- cess. Finally, a senior annotation leader (non- authors) performs a final scan of the entire dataset, providing feedback and guiding annotators to op- 4 timize the annotations. This process ensures the standardization and high quality of the annotations. 3.3 Evaluation Metrics LetG = (g i ,r i ) n i=1 denote the gold outcome set, whereg i represents a gold test item and r i ∈ Pass, Failis its ground-truth status. To quantify performance, we compare the agent’s outcome setPagainst an annotated gold out- come setG. Specifically, for a generated check- listT pred = p 1 ,...,p m and a gold checklist T gold = g 1 ,...,g n , each predicted itemp j is mapped tog i if they are semantically equivalent in testing intent. We define a matching function φ : T pred → T gold ∪∅, whereφ(p j ) = g i in- dicates thatp j is matched tog i . We define the matched set of a gold itemg i asΦ(g i ) = p j ∈ T pred | φ(p j ) = g i . We employQwen3.5-27Bas a semantic judge to perform this matching step. Coverage. Coverage measures the completeness of the generated checklist, focusing on whether all necessary test items have been listed. It is defined as the proportion of gold test items successfully covered by the agent’s predictions: Coverage = g i ∈T gold | Φ(g i )̸=∅ |T gold | (1) Defect Detection Performance. We treat defect detection as a binary classification task where “Fail” (a real defect) is the positive class. For each gold itemg i , we aggregate its matched predictions into a single verdict: D(g i ) = ( Fail, if∃p j ∈ Φ(g i ), s j = Fail Pass, otherwise (2) whereD(g i ) = PasswhenΦ(g i ) =∅. The confusion matrix entries are formalized as follows: •True Positive (TP): The agent correctly identifies a ground-truth bug. TP = n X i=1 1(r i = Fail∧ D(g i ) = Fail)(3) • False Positive (FP): The agent reports a defect for a gold item that is actually correct. FP = n X i=1 1(r i = Pass∧ D(g i ) = Fail)(4) •False Negative (FN): The agent fails to detect a real bug, either due to misjudgment during detec- tion or omission in checklist generation. FN = n X i=1 1(r i = Fail∧ D(g i ) = Pass)(5) •True Negative (TN): The agent correctly raises no alarm for a passing gold item, either by correctly judging it as “Pass” or by not covering the item. TN = n X i=1 1(r i = Pass∧ D(g i ) = Pass) (6) Based on these formal definitions, we can calcu- late Precision (P ), Recall (R), and F1 score: P = TP TP + FP , R = TP TP + FN , F1 = 2× P× R P + R (7) 3.4 WebTester Given the current absence of an established frame- work for end-to-end automated web testing, we in- troduce WebTester, a minimal yet functional base- line designed for this task. Specifically, WebTester consists of two components: a checklist generation agent and a defect detection agent. This framework enables autonomous testing of a web application based on the user instruction and the web appli- cation, ultimately producing structured test results accompanied by bug reports. Checklist Generation Agent. Given a develop- ment instructionIused to synthesize the target web application, the checklist generation Agent A C is responsible for decomposingIinto a struc- tured, executable test checklistT pred . Formally, we define a test item p as a triple: p = (d, a, e),(8) wheredis a textual description of what is being tested,ais the action to be performed on the web interface (e.g., clicking the “Edit” button), andeis the expected outcome that determines the Pass/Fail verdict. The full checklist generated byA C is a typed collection of such triples: T pred = A C (I) =p j m j=1 ,(9) Defect Detection Agent. Given the development instructionI, the generated checklistT pred , and the target web application W , the defect detection agentA D is responsible for executing each test itemp j ∈T pred by simulating human interactions with the webpages, and determining a binary ex- ecution status for each item. Formally,A D maps 5 Model#Turns#Tokens FunctionalityConstraintInteractionContentOverall Cov.F1Cov.F1Cov.F1Cov.F1Cov.PRF1 Open-Source LLMs Minimax-M2.141.73.58M77.912.340.415.842.219.947.17.760.122.314.615.2 Qwen3-Coder-Next63.46.24M77.614.148.323.842.711.435.94.360.427.815.817.3 GLM-4.7 41.63.47M79.916.547.320.536.017.248.94.361.126.716.618.1 GLM-541.33.71M79.711.950.126.941.420.950.63.463.130.415.619.0 Step-3.5-Flash57.03.37M79.820.153.627.948.521.260.62.666.034.620.823.4 MiMo-V2-Flash59.87.26M80.021.948.729.248.020.350.37.363.534.824.625.1 Closed-Source LLMs Claude Opus 4.542.92.60M83.318.850.321.240.814.742.16.863.233.016.520.2 Claude Sonnet 4.537.61.90M81.022.247.722.546.719.951.61.763.732.119.721.9 GPT-5.269.57.43M76.925.351.921.543.023.246.16.261.024.725.222.9 GPT-5.130.30.87M76.430.951.226.949.722.057.515.363.125.833.326.4 Table 2: Web testing performance of representative LLM on WebTestBench under the WebTester framework. We report the Coverage metric (Cov.) for checklist generation, and the average number of iteration turns (#Turns), average context tokens (#Tokens) per instance, and Precision/Recall/F1 metrics for defect detection. Results in bold and underlinedenote the best and second-best performances. each test item together with the instruction and the web application to a verdict: s j = A D (p j , I, W )∈Pass, Fail,(10) For each failed item,A D additionally produces a bug reportb j describing the observed erroneous behavior. The final output of WebTester is thus a set of predicted outcome triples: S = A D (T pred ,I,W ) = p j ,s j [,b j ] m j=1 , (11) 4 Experiments 4.1 Settings We build WebTester on Claude Code as the ba- sic agent framework. Specifically, we adopt the Claude Agent SDK 2 to provide a unified evalua- tion interface across different LLMs. To interact with the web application, we leverage Playwright MCP 3 , a Model Context Protocol server that pro- vides browser automation capabilities (e.g., naviga- tion, clicking, typing) to the agent. We evaluate several closed-source models and open-source models on WebTester, including: Claude Opus/Sonnet 4.5 (Anthropic, 2025a,b), GPT-5.2/5.1 (Openai, 2025b,a), GLM-5/4.7 (Zeng et al., 2026, 2025), Step-3.5-Flash (Huang et al., 2026), Qwen3-Coder-Next (Qwen Team, 2026), MiMo-V2-Flash (Xiao et al., 2026), and Minimax- M2.1 (MiniMax, 2025). More details are available in Appendix B.1. 2 https://github.com/anthropics/ claude-agent-sdk-python 3 https://github.com/microsoft/playwright-mcp 4.2 Main Results Overall Performance.Table 2 demonstrates the performance of the tested LLMs on end-to-end au- tomated web testing. Despite advanced LLMs have made progress in agentic coding tasks and general web browsing tasks, they still exhibit notable per- formance bottlenecks when confronted with highly dynamic web testing scenarios involving complex interactions and constraints. All evaluated mod- els failed to achieve an F1 score exceeding30%. Among all evaluated models, GPT-5.1 attains the highest F1 score of 26.4%, a standing driven pri- marily by its leading recall of 33.3%. MiMo-V2- Flash ranks second with an F1 of 25.1%, achieving the highest precision (34.8%). Insufficient Test Completeness. Reliable defect detection relies on a comprehensive test checklist. However, across all tested models, their coverage consistently remains below 70%. This indicates that a substantial portion of test items are excluded from subsequent detection and introduces false neg- atives. Even the two strongest models, GPT-5.1 and MiMo-V2-Flash, omit at least one-third of the test cases. This reflects a fundamental limitation in how LLMs decompose development instructions into verifiable test cases, particularly for implicit cases not explicitly stated in the instructions. Detection Bottleneck. CUAs exhibit severe de- ficiencies in web defect detection, taking in two primary failure modes. The first is a high false- positive rate.Most models achieve precision around 30%, indicating that CUAs often misclas- sify benign behaviors as defects. For instance, tran- sient UI rendering delays and asynchronous state 6 ModelPresentationSearchToolCommerceDMWorkflowUGC Minimax M2.125.6/10.3/14.318.9/6.9/9.412.3/12.6/11.710.9/10.1/8.636.4/23.4/25.318.4/12.3/12.827.2/20.3/17.8 Qwen3 Coder Next 16.4/16.4/14.321.5/11.1/13.733.8/12.2/16.626.8/16.8/19.142.6/20.1/23.218.4/15.8/15.027.0/15.9/16.4 GLM 4.722.4/12.7/15.619.4/26.5/20.017.8/7.8/9.835.9/19.1/24.141.8/22.1/24.722.6/15.7/16.321.7/16.2/16.3 GLM 530.8/16.7/21.111.1/4.6/6.524.7/11.4/14.033.6/15.4/19.638.0/20.7/24.640.1/23.3/26.823.6/9.9/12.7 Step 3.5 Flash38.6/13.2/18.824.1/13.7/17.137.2/22.2/25.334.3/22.5/25.647.9/33.8/35.627.2/19.2/19.024.3/12.0/14.6 MiMo-V2-Flash 23.7/14.2/16.526.6/34.1/26.042.1/28.1/30.735.5/20.2/21.644.1/33.2/34.144.1/24.0/28.413.8/16.1/10.6 Claude Opus 4.541.2/25.0/30.57.4/4.6/5.625.5/15.7/16.255.1/16.2/24.228.2/13.4/17.042.8/24.5/29.023.6/11.0/13.7 Claude Sonnet 4.543.5/19.1/23.821.1/20.9/19.732.6/22.4/24.622.7/13.0/14.043.8/23.4/26.931.4/22.8/24.720.1/12.6/14.3 GPT 5.233.7/19.3/22.913.9/13.0/13.231.0/35.1/31.920.8/19.9/18.228.4/24.4/23.921.8/28.8/23.616.6/29.0/19.9 GPT 5.1 27.6/27.6/25.723.1/31.3/23.425.0/33.4/26.428.7/28.8/27.229.2/39.9/29.428.0/38.2/29.515.5/28.5/19.5 Avg. F1 (±Std.)20.4 ±5.4 15.5 ±7.0 20.9 ±7.8 20.2 ±5.7 26.5 ±5.4 22.5 ±6.3 15.6 ±3.6 Table 3: Category-wise evaluation results on WebTestBench. Each cell contains “Precision/Recall/F1”. Results in bold and underlinedenote the best and second-best performances. updates are easily mistaken as functional failures, reflecting insufficient model understanding of dy- namic web behavior. On the other hand, the low recall poses a greater risk. Most models fail to ex- ceed a 25% recall, meaning the major real defects remain undetected. Beyond limited defect cogni- tion, these false-negatives are partly from a default- correctness bias, where models default to a pass judgment when no explicit evidence is observed. Additionally, we observe a strategic divergence in CUA behavior: they either employ an aggressive detection strategy that favors recall at the cost of precision (e.g., GPT-5.1 achieves 33.3% recall but only 25.8% precision) or adopt a conservative one that prioritizes precision while overlooking numer- ous real defects (e.g., MiMo-V2-Flash achieves 34.8% precision but only 24.6% recall). Long-horizon Interaction Unreliability. Com- pleting a comprehensive web defect detection pro- cess typically requires dozens of interaction turns and millions of tokens. For example, Step-3.5- Flash requires an average of 57.0 turns and 3.37M tokens per sample. Such long-horizon tasks de- mand rigorous long-context memory and planning stability. Specifically, as the interaction history accumulates, models become increasingly suscepti- ble to tracking failures, resulting in the loss of prior states or the execution of redundant operations. ModelKendall’s τSpearman’s ρPearson r GPT-5-Mini68.875.078.9 GPT-576.682.785.7 Qwen3.5-27B78.583.287.1 Table 4: Human judgment correlation scores for our automatic evaluation. All p-values < 0.05. Correlation with Human Judgments. To validate the effectiveness and robustness of our automatic evaluation pipeline, we perform a correlation anal- ysis between the automated evaluation results and human judgments, reporting Kendall’sτ, Spear- man’sρ, and Pearsonrcorrelation coefficients. As shown in Table 4, all judge models achieve strong correlation with human judgments and Qwen3.5- 27B attains the best correlation. Moreover, as an open-source model, it is not subject to depreca- tion or silent updates over time, ensuring long-term reproducibility and stability. We therefore adopt Qwen3.5-27B as the judge model in this work. More details are available in Appendix B.2. 4.3 Fine-grained Analysis Quality Dimensions. While recent efforts have progressed toward comprehensive functional test- ing (Lu et al., 2025b; Zhang et al., 2025), functional correctness alone is insufficient to capture overall application quality. To better understand where CUAs struggle in web testing, we report results across four quality dimensions. As shown in Ta- ble 2, Functionality consistently achieves the high coverage across all models. This occurs because functional items are typically explicit and can be directly extracted from the development instruc- tion. Conversely, coverage drops substantially for the other three types, which encompass implicit re- quirements demanding deeper reasoning. Despite lower coverage, Constraint type achieves a rela- tively higher F1, revealing that once a constraint item is covered, defect detection becomes straight- forward, as constraint violations typically produce clear, observable signals. However, Content type performs worst by a marked margin. Error anal- ysis reveals two primary failure modes: (1) mod- els often fail to generate items targeting the actual defects, resulting in zero true positives and conse- quently zero F1 on many samples despite non-zero coverage. (2) models struggle to judge whether the displayed content is semantically aligned with the 7 GPT-5.1Sonnet 4.5Mimo-V2-FStep-3.5-F 0 20 40 60 80 25.8 32.1 34.8 34.6 47.0 61.0 44.3 53.4 Precision (%) E2E Setting Oracle Setting GPT-5.1Sonnet 4.5Mimo-V2-FStep-3.5-F 0 20 40 60 80 33.3 19.7 24.6 20.8 63.4 47.4 30.4 38.7 Recall (%) E2E Setting Oracle Setting GPT-5.1Sonnet 4.5Mimo-V2-FStep-3.5-F 0 20 40 60 80 26.4 21.9 25.1 23.4 46.8 49.2 31.9 40.9 F1 Score (%) E2E Setting Oracle Setting Figure 3: Comparison of perfermance between the end-to-end (E2E) setting and the oracle setting for representative models. In the oracle setting, the gold checklist is directly provided to the defect detection agent, decoupling detection performance from checklist generation quality. theme of the development instruction. These find- ings indicate that the primary bottleneck in check- list generation lies in the insufficient surfacing of implicit requirements, while content-level semantic alignment remains a formidable challenge. Category-wise Results. Table 3 reveals that per- formance varies across categories, driven by the na- ture of verification required. Models excel at Data Management, where correctness reduces to track- ing explicit, structurally observable state transitions (e.g., record creation/deletion/updates, or numeri- cal changes). Performance degrades markedly in Search and User-Generated Content, exposing a semantic verification bottleneck: evaluating search relevance or dynamic content requires assessing abstract alignment with user intent, rather than ob- serving discrete component states. Detailed analy- sis of each category are available in Appendix B.3. 4.4 Oracle Setting In the end-to-end setting, checklist generation and defect detection are executed sequentially, so final performance is jointly influenced by both sub-tasks. To isolate detection capability, we evaluate models in an oracle setting, where the defect detection agent receives the human-written gold checklist directly, thereby decoupling the detection phase from the upstream checklist generation. As shown in Figure 3, all models achieve no- table performance improvements in the oracle set- ting, confirming that test incompleteness is one of the primary bottlenecks in the end-to-end set- ting. The detection strategy preference observed earlier maintains consistent. GPT-5.1 remains ag- gressive (recall 63.4%), while Claude Sonnet 4.5 remains conservative (precision 61.0%). Notably, in oracle settings, closed-source models substan- tially outperform open-source models. This gap is not present in end-to-end setting, but becomes pro- nounced in oracle setting, indicating that for closed- source models, checklist generation is a more criti- cal bottleneck in end-to-end testing. In comparison, Open-source models face challenges in both sub- tasks, and improving them requires advances in checklist coverage and detection reliability. More details and analysis are available in Appendix B.5. < 200200~300300~400400~500≥ 500 20 30 40 50 60 DOM Node Count F1 Score (%) < 1010~1515~2020~25≥ 25 20 30 40 50 60 DOM Interactive Elements F1 Score (%) < 1010~2020~3030~40≥ 40 0 10 20 30 40 Number of Interactive Elements F1 Score (%) GPT-5.1Claude Sonnet 4.5Mimo-V2-FlashStep-3.5-Flash Figure 4: Performance comparison across different web complexity in the oracle setting. Web Complexity. As shown in Figure 4, model performance generally degrades as web complexity increases, measured by DOM node count and the number of interactive elements per page. GPT-5.1 and Claude Sonnet 4.5 are comparatively robust, likely due to their more efficient interaction behav- ior (fewer turns, lower token usage), which reduces cascading errors in dense web environments. 5 Conclusion In this paper, we introduce WebTestBench, a chal- lenging benchmark for end-to-end automated web testing without relying on human-written test cases, grounded in the context of AI-driven web develop- ment. Beyond standard functional testing, WebTest- Bench incorporates latent logical constraints, en- abling a comprehensive assessment of application quality. We further propose WebTester, a simple yet effective end-to-end web testing framework, and evaluate representative LLMs on WebTest- Bench. The results reveal a substantial gap be- tween current CUA capabilities and the demands of industrial-grade deployment. Through compre- 8 hensive analyses, we identify the key pain points and multifaceted challenges. We hope WebTest- Bench serves as a foundation for future research in web testing during the vibe coding era, and encour- ages developing more reliable web testing agents. Limitations Although WebTestBench provides fine-grained an- notations across four quality dimensions, the con- struction process is inherently labor-intensive and demands domain expertise. Annotators must rea- son beyond explicit requirements to identify latent logical constraints. This specialized nature makes it costly to scale or replicate our methodology for building similar benchmarks. Second, our evalu- ation protocol also carries an inherent limitation. The scoring system relies on aligning predicted test items against the gold checklist. Thus, it can- not assess the validity of generated test cases that fall outside the gold checklist coverage, even if such cases reflect genuine quality concerns. Fi- nally, while WebTestBench spans seven web ap- plication categories, it does not fully capture the diversity of real-world applications. Certain appli- cation types, such as “games” and “complex multi- user real-time interactive systems”, are deliberately excluded, as they demand high-frequency capture and interpretation of highly dynamic content that exceeds the perceptual latency of both DOM-based and screenshot-based approaches, and thus lies be- yond the current capability boundary of CUAs. Ethics Policy The web development instructions collected for WebTestBench may contain background narratives or promotional content originating from real-world sources. Following our collection procedure, we re- move or rewrite all such material, with human anno- tators verifying that no real-world entities remain. To support defect detection, we useLovable.dev to populate each synthesized web application with sample data such as placeholder accounts and con- tent. This data is not intended to represent actual in- dividuals or organizations, and any resemblance is coincidental. Despite these precautions, the dataset may reflect latent social biases or stereotypes inher- ent in generative models, including those related to gender, race, age, and socioeconomic status. We therefore advise users to exercise caution when interpreting results or building upon this content. References Anthropic. 2025a. Introducing claude opus 4.5.https: //w.anthropic.com/news/claude-opus-4-5. Anthropic. 2025b.Introducing claude sonnet 4.5.https://w.anthropic.com/news/ claude-sonnet-4-5. Anthropic. 2026. Claude code - ai coding agent for terminal & ide.https://claude.com/product/ claude-code. Giovana Giardini Borges and Rogéria Cristiane Gratão de Souza. 2024. Skills development for software engineers: Systematic literature review. Information and Software Technology, 168:107395. Sacha Brisset, Romain Rouvoy, Lionel Seinturier, and Renaud Pawlak. 2022. Erratum: Leveraging flexi- ble tree matching to repair broken locators in web automation scripts. Information and Software Tech- nology, 144:106754. Antoine Chevrot, Alexandre Vernotte, Jean-Rémy Fal- leri, Xavier Blanc, Bruno Legeard, and Aymeric Cretin. 2025. Are autonomous web agents good testers? Proceedings of the ACM on Software Engi- neering, 2(ISSTA):206–228. Thibault Le Sellier de Chezelles, Maxime Gasse, Alexandre Lacoste, Massimo Caccia, Alexandre Drouin, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Ke- unho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Graham Neubig, Quentin Cappart, Russ Salakhutdinov, and Nicolas Chapados. 2025. The browsergym ecosystem for web agent re- search. Transactions on Machine Learning Research. Expert Certification. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Common Crawl Foundation. 2026. Common crawl - open repository of web crawl data.https:// commoncrawl.org/. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023. Incoder: A generative model for code infilling and synthesis. In International Conference on Learning Representa- tions (ICLR). Pengfei Gao, Zhao Tian, Xiangxin Meng, Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, and 1 others. 2025. Trae agent: An llm-based agent for software en- gineering with test-time scaling.arXiv preprint arXiv:2507.23370. Yifei Gao, Jiang Wu, Xiaoyi Chen, Yifan Yang, Zhe Cui, Tianyi Ma, Jiaming Zhang, and Jitao Sang. 2026. Guitester: Enabling gui agents for exploratory defect discovery. arXiv preprint arXiv:2601.04500. 9 Google. 2026. Gemini - google.https://gemini. google.com. Yongxiang Hu, Yu Zhang, Xuan Wang, Yingjie Liu, Shiyu Guo, Chaoyi Chen, Xin Wang, and Yangfan Zhou. 2025. Kuitest: Leveraging knowledge in the wild as gui testing oracle for mobile apps. In Pro- ceedings of the International Conference on Soft- ware Engineering: Software Engineering in Practice (ICSE-SEIP). Ailin Huang, Ang Li, Aobo Kong, Bin Wang, Binxing Jiao, Bo Dong, Bojun Wang, Boyu Chen, Brian Li, Buyun Ma, and 1 others. 2026. Step 3.5 flash: Open frontier-level intelligence with 11b active parameters. arXiv preprint arXiv:2602.10604. ISO/IEC. 2023.ISO/IEC 25010:2023 — Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality model.https://w.iso.org/ standard/78176.html. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A survey on large lan- guage models for code generation. arXiv preprint arXiv:2406.00515. Yuanhong Lan, Yifei Lu, Zhong Li, Minxue Pan, Wen- hua Yang, Tian Zhang, and Xuandong Li. 2024. Deeply reinforcing android gui testing with deep reinforcement learning.In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE). Nguyen-Khang Le, Quan Minh Bui, Minh Ngoc Nguyen, Hiep Nguyen, Trung Vo, Son T Luu, Shoshin Nomura, and Minh Le Nguyen. 2025. Auto- mated web application testing: End-to-end test case generation with large language models and screen transition graphs. arXiv preprint arXiv:2506.02529. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, and 1 others. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161. Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2019. Humanoid: A deep learning-based ap- proach to automated black-box android app testing. In the IEEE/ACM International Conference on Auto- mated Software Engineering (ASE). Jenny T Liang, Thomas Zimmermann, and Denae Ford. 2022. Understanding skills for oss communities on github. In Proceedings of the 30th ACM Joint Eu- ropean Software Engineering Conference and Sym- posium on the Foundations of Software Engineering (ESEC/FSE), pages 170–182. Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan Weixian Lei, Li- juan Wang, and Mike Zheng Shou. 2025a. Showui: One vision-language-action model for gui visual agent. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR). Zhiyu Lin, Zhengda Zhou, Zhiyuan Zhao, Tianrui Wan, Yilun Ma, Junyu Gao, and Xuelong Li. 2025b. We- buibench: A comprehensive benchmark for evaluat- ing multimodal large language models in webui-to- code. arXiv preprint arXiv:2506.07818. Chenxu Liu, Yingjie Fu, Wei Yang, Ying Zhang, and Tao Xie. 2026. Webcoderbench: Benchmarking web application generation with comprehensive and interpretable evaluation metrics.arXiv preprint arXiv:2601.02430. Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Xing Che, Dandan Wang, and Qing Wang. 2024. Make llm a testing expert: Bring- ing human-like interaction to mobile gui testing via functionality-aware decisions. In Proceedings of the International Conference on Software Engineering (ICSE). Yuxuan Lu, Bingsheng Yao, Hansu Gu, Jing Huang, Zheshen Jessie Wang, Yang Li, Jiri Gesi, Qi He, Toby Jia-Jun Li, and Dakuo Wang. 2025a. Uxagent: An llm agent-based usability testing framework for web design. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. 2025b. Webgen- bench: Evaluating llms on generating interactive and functional websites from scratch. arXiv preprint arXiv:2505.03733. MiniMax. 2025. Minimax m2.1: Significantly en- hanced multi-language programming, built for real- world complex tasks.https://w.minimax.io/ news/minimax-m21. Atsuyuki Miyai, Zaiying Zhao, Kazuki Egashira, Atsuki Sato, Tatsumi Sunada, Shota Onohara, Hiromasa Ya- manishi, Mashiro Toyooka, Kunato Nishina, Ryoma Maeda, Kiyoharu Aizawa, and Toshihiko Yamasaki. 2025. Webchorearena: Evaluating web browsing agents on realistic tedious web tasks. arXiv preprint arXiv:2506.01952. Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A Rodriguez, Montek Kalsi, Nicolas Chapados, M Tamer Özsu, Aishwarya Agrawal, David Vazquez, Christopher Pal, and 1 others. 2025. Ui-vision: A desktop-centric gui benchmark for visual perception and interaction. In International Conference on Ma- chine Learning (ICML). Openai. 2025a.Gpt-5.1: A smarter, more con- versational chatgpt.https://openai.com/index/ gpt-5-1/. Openai. 2025b. Introducing gpt-5.2.https://openai. com/index/introducing-gpt-5-2/. Openai. 2026. Codex | openai.https://openai.com/ codex. 10 Qwen Team. 2026. Qwen3-coder-next technical report. Technical report. Accessed: 2026-02-03. Chenglei Si, Yanzhe Zhang, Ryan Li, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. 2025. Design2Code: Benchmarking multimodal code generation for au- tomated front-end engineering. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), pages 3956–3974, Albuquerque, New Mexico. Xiwen Teoh, Yun Lin, Duc-Minh Nguyen, Ruofei Ren, Wenjie Zhang, and Jin Song Dong. 2026. Webtestpilot: Agentic end-to-end web testing against natural language specification by inferring oracles with symbolized gui elements.arXiv preprint arXiv:2602.11724. Yuxuan Wan, Tingshuo Liang, Jiakai Xu, Jingyu Xiao, Yintong Huo, and Michael R Lyu. 2025. Auto- matically generating web applications from require- ments via multi-agent test-driven development. arXiv preprint arXiv:2509.25297. Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, and 1 others. 2025. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning. arXiv preprint arXiv:2509.02544. Jason Wu, Eldon Schoop, Alan Leung, Titus Barik, Jef- frey Bigham, and Jeffrey Nichols. 2024. UICoder: Finetuning large language models to generate user interface code through automated feedback. In Pro- ceedings of the 2024 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7511–7525, Mexico City, Mexico. Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, and 1 others. 2026. Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780. Jingyu Xiao, Yuxuan Wan, Yintong Huo, Zixin Wang, Xinyi Xu, Wenxuan Wang, Zhiyao Xu, Yuhang Wang, and Michael R Lyu. 2024. Interaction2code: Bench- marking mllm-based interactive webpage code gen- eration from interactive prototyping. arXiv preprint arXiv:2411.03292. Jingyu Xiao, Ming Wang, Man Ho Lam, Yuxuan Wan, Junliang Liu, Yintong Huo, and Michael R Lyu. 2025. Designbench: A comprehensive benchmark for mllm- based front-end code generation. arXiv preprint arXiv:2506.06251. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, and 1 others. 2024. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Naimeng Ye, Xiao Yu, Ruize Xu, Tianyi Peng, and Zhou Yu. 2025. Ai agents for web testing: A case study in the wild. arXiv preprint arXiv:2509.05197. Sukmin Yun, Rusiru Thushara, Mohammad Bhat, Yongxin Wang, Mingkai Deng, Jinhong Wang, Tian- hua Tao, Junbo Li, Haonan Li, Preslav Nakov, and 1 others. 2024. Web2code: A large-scale webpage- to-code dataset and evaluation framework for multi- modal llms. Daoguang Zan, Ailun Yu, Wei Liu, Dong Chen, Bo Shen, Yafen Yao, Wei Li, Xiaolin Chen, Yong- shun Gong, Bei Guan, and 1 others. 2024. Codes: Natural language to code repository via multi-layer sketch. ACM Transactions on Software Engineering and Methodology (TOSEM). Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chengxing Xie, Cunxiang Wang, and 1 others. 2026. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763. Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, and 1 others. 2025. Glm-4.5: Agentic, reasoning, and coding (arc) foundation mod- els. arXiv preprint arXiv:2508.06471. Chenchen Zhang, Yuhang Li, Can Xu, Jiaheng Liu, Ao Liu, Changzhi Zhou, Ken Deng, Dengpeng Wu, Guanhua Huang, Kejiao Li, and 1 others. 2025. Artifactsbench: Bridging the visual-interactive gap in llm code generation evaluation. arXiv preprint arXiv:2507.04952. Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024. CodeAgent: Enhancing code generation with tool-integrated agent systems for real-world repo- level coding challenges. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13643– 13658, Bangkok, Thailand. Kangjia Zhao, Jiahui Song, Leigang Sha, Haozhan Shen, Zhi Chen, Tiancheng Zhao, Xiubo Liang, and Jianwei Yin. 2024a. Gui testing arena: A unified benchmark for advancing autonomous gui testing agent. arXiv preprint arXiv:2412.18426. Wenting Zhao, Nan Jiang, Celine Lee, Justin T Chiu, Claire Cardie, Matthias Gallé, and Alexander M Rush. 2024b. Commit0: Library generation from scratch. arXiv preprint arXiv:2412.01769. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, and 1 others. 2024. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations (ICLR). 11 A WebTestBench A.1 Dataset Statistics Table 5 presents the statistics of WebTestBench. The benchmark comprises 100 carefully curated de- velopment instructions spanning seven categories, each paired with an AI-generated web application that exhibits naturally occurring defects. The appli- cations are structurally complex and highly interac- tive, reflecting the characteristics of real-world web development. On average, each instance is asso- ciated with 17.5 gold test items distributed across four dimensions: functionality, constraint, interac- tion, and content. The relevant descriptions and typical examples for the seven web application cat- egories and four test case types are provided in Tables 7 and 8, respectively. A.2 Annotation A.2.1 Annotation Document To standardize the annotation process, we prepare a detailed annotation document for human anno- tators covering four components: (1) a webpage category taxonomy, (2) a test case dimension tax- onomy, (3) annotation content and procedures, and (4) expected outputs. Webpage Category Taxonomy. We categorize web application development instructions into seven types. The annotation document provides annotators with the description, core functionali- ties, and typical examples of each category so that classification can be applied consistently. Detailed definitions are provided in Table 7. Test Case Dimension Taxonomy. To enable fine- grained evaluation of CUA performance in end-to- end web testing, each test case is assigned to one of four dimensions. This design allows our experi- ments to analyze challenges arising from different aspects of the task. Notably, annotators should not merely extract explicitly mentioned requirements from development specifications. They must also apply additional understanding and reasoning to identify implicit test cases, such as potential corner cases. The annotation document provides descrip- tions, evaluation focus, and typical examples for each dimension. Detailed definitions are provided in Table 8. Annotation Procedure. Human annotators par- ticipate in the entire annotation pipeline. During instruction collection, annotators review and refine the development instructions rewritten byGPT-5.1, ensuring that the descriptions focus on application StatisticValue Instructions & Web Applications Total Samples100 Presentation13 Search9 Tool17 Commerce13 Data Management19 Workflow17 User-Generated Content12 Avg. Instruction Tokens127.8 Avg. Pages5.3 Avg. DOM Average Depth9.4 Avg. DOM Node Count243.5 Avg. DOM Interactive Elements18.9 Avg. DOM Unique Tag Count31.1 Test Cases Total items1750 Functionality854 Constraint398 Interaction247 Content251 Total Pass / Fail items1302/448 Functionality (Pass / Fail)653/201 Constraint (Pass / Fail)270/128 Interaction (Pass / Fail)176/71 Content (Pass / Fail)203/48 Table 5: Overall statistics of WebTestBench. implementation and avoid references to real indi- viduals, companies, or brands. They also verify and correct the category label assigned to each instruc- tion. During web application synthesis, annotators useLovable.devto generate a web application from each instruction. During gold checklist and result annotation, annotators construct the required test cases by combining direct information from the instruction with additional reasoning about po- tential behaviors. Each test case is then assigned to one of the four dimensions. Annotators interact with the synthesized application to align test cases with the corresponding interface components and interaction flows, and perform defect detection for each item. This process produces a tuple consisting of the test case description, a verdict (Pass or Fail), and an optional bug report for failed items. If an in- stance contains fewer than three defects, annotators either continue interacting withLovable.devto introduce additional features or modify the instruc- tion directly. The checklist and results are updated accordingly until the instance contains a sufficient number of defects, ensuring that each example is informative for defect detection evaluation. Finally, all annotation results are cross-validated among annotators. 12 Expected Output. Following this pipeline, each instance in the WebTestBench includes a devel- opment instruction, an web application contain- ing naturally occurring defects, a manually written checklist, a Pass/Fail verdict for each test item, and an optional bug report for failed items. A.2.2 Human Annotators We employ crowdsourcing for data annotation. All annotators have prior experience evaluating the quality of AI-generated web applications in simi- lar annotation tasks. Before the main annotation stage, candidates are required to study the annota- tion document and complete a trial annotation on three example samples. This stage serve both as training and as a qualification assessment. Only candidates who demonstrate high consistency, ac- curacy, and a clear understanding of the guidelines are selected for the main annotation task. A senior annotation lead, familiar with the task requirements and expected data quality, review all annotations and resolved disagreements identified during cross- validation. A.2.3 Reproducibility The code for each web application is available and can be deployed locally with a minimal setup com- mand (npm install && npm run dev) and does not rely onLovable.devduring inference. This design ensures that the benchmark remains fully accessible and that experimental results can be re- produced independently of any future changes to the Lovable.dev platform. BMore Experimental Details and Results B.1 More Experimental Settings In this work, we employ Claude Code v2.1.25 and the Claude Agent SDK v0.1.0, with the maximum iteration turns limited to 150. To enable Claude Code to support non-Claude models, we integrate OpenRouter 4 with Claude Code as a unified invo- cation interface. For the checklist generation agent, Playwright MCP is not employed, as the target task involves test case generation rather than web in- teraction. For the defect detection agent, we use Playwright MCP v0.0.41 with the viewport fixed at 1280×720 during evaluation. The prompt tem- plates for the two agents are shown in Figures 6 and 7. When scoring, we set the temperature to 0.1 4 https://openrouter.ai/ to ensure reproducibility. All other configurations follow the default settings. While various LLMs can be deployed within Claude Code, the high complexity of the task presents substantial practical challenges. Models that fail to complete the testing process, such as inability to utilize Playwright MCP, output format errors, or unexpected termination, will be excluded from our evaluation. The matching between predicted and gold test items described in Section 3.3 is performed by Qwen3.5-27Bas a semantic judge. The prompt template used for this step is provided in Figure 8. B.2 Correlation with Human Judgments As described in Section 3.3, our evaluation protocol incorporates an LLM only for the relatively simple task of matching predicted test items to gold test items, rather than asking the LLM to directly assign scores. This design choice makes the evaluation more reliable compared to direct LLM-based scor- ing approaches. To further validate the alignment between our automated evaluation protocol and hu- man judgment, we conduct a correlation analysis on results aggregated from three models, includ- ing GLM-4.7, Step-3.5-Flash, and Claude Sonnet 4.5. Given the high cost of manual annotation, we randomly sample a subset of 20 instances with a balanced category distribution, comprising 352 gold test items and 1,144 predicted test items in total across the three models (GLM-4.7: 379, Step- 3.5-Flash: 398, Claude Sonnet 4.5: 367). The per- category sample counts are 2/3/3/3/3/3/3 across the seven categories. Human annotators are asked to perform the item matching task manually, and the resulting evaluation metrics are compared against those produced by our automated pipeline. We report Kendall’sτ, Spearman’sρ, and Pearsonr measures. The results demonstrate that our auto- matic evaluation achieves strong alignment with human judgments across all correlation metrics. B.3 Detailed Category-wise Analysis As shown in Table 10, we present detailed category- wise results, along with the cross-model mean and standard deviation for each category and metric, to reflect both the average performance and the variability across models. We further analyze these results in conjunction with the category definitions provided in Table 7. Data Management achieves the highest overall performance, with an average F1 score of 26.5%. 13 GPT-5.1 Avg. Playwright Calls: 23.23 ToolCalls browser_click14.69 browser_fill_form2.14 browser_snapshot1.81 browser_wait_for1.41 browser_navigate1.23 browser_evaluate0.68 browser_type0.55 browser_press_key0.20 GPT-5.2 Avg. Playwright Calls: 61.96 ToolCalls browser_click35.71 browser_snapshot6.47 browser_evaluate5.86 browser_fill_form4.72 browser_navigate2.13 browser_wait_for2.08 browser_type1.56 browser_press_key1.36 Claude Sonnet 4.5 Avg. Playwright Calls: 34.74 ToolCalls browser_click18.74 browser_snapshot4.73 browser_type3.39 browser_evaluate2.24 browser_navigate2.02 browser_fill_form1.48 browser_wait_for1.13 browser_press_key0.58 MiMo-V2-Flash Avg. Playwright Calls: 49.37 ToolCalls browser_click25.23 browser_evaluate8.29 browser_navigate4.00 browser_snapshot3.57 browser_fill_form2.96 browser_type2.47 browser_wait_for0.86 browser_console_messages0.46 Step-3.5-Flash Avg. Playwright Calls: 48.83 ToolCalls browser_click26.01 browser_snapshot6.98 browser_type4.89 browser_evaluate3.30 browser_navigate2.59 browser_wait_for1.78 browser_fill_form1.47 browser_press_key0.73 GLM-5 Avg. Playwright Calls: 37.73 ToolCalls browser_click23.84 browser_snapshot4.33 browser_type3.90 browser_navigate1.74 browser_fill_form1.63 browser_evaluate0.69 browser_press_key0.62 browser_wait_for0.37 Table 6: Statistics of Playwright tool calls per instance for representative models. Applications in this category involve explicit state transitions, such as record creation, deletion, and numerical updates, where Pass/Fail judgments can be directly inferred from observable changes in the interface state before and after an operation. Work- flow applications share a similar state-transition structure but present additional challenges due to multi-step processes and the need for cross-stage context tracking. Commerce and Tool applications occupy the middle tier, with average F1 scores of 20.2% and 20.9%, respectively.Commerce applica- tions involve structured transaction flows that can generally be verified, though complex busi- ness logic, such as inventory constraints and pay- ment validation, introduces additional reasoning requirements. Tool applications follow an input- processing-output paradigm, where correctness ver- ification requires models to infer expected outputs based on specific inputs. Notably, Tool shows the largest cross-model variance in F1 (7.8%), suggest- ing that the reasoning demands in this category are unevenly handled across models. Presentation performance is dominated by closed-source models, with Claude Opus 4.5 achieving the highest F1 score of 30.5%, while open-source models generally perform much worse. This gap suggests that evaluating content layout and semantic structure benefits disproportionately from stronger model capacity. Search shows sim- ilar large cross-model variance (7.0%), with per- formance ranging from 5.6% for Claude Opus 4.5 to 26.0% for MiMo-V2-Flash. Both categories require assessing semantic alignment between ren- dered content and user intent, rather than simply ob- serving discrete component states, which presents a verification challenge that no model currently handles consistently. User-Generated Content has the worst perfor- mance, with an average F1 score of only 15.6% and the smallest cross-model variance (S3.0%), in- dicating that all models struggle equally in this domain. UGC applications involve unstructured, dynamically generated content, whose correctness cannot be determined from fixed component states, necessitating abstract semantic judgment that con- sistently exceeds current CUA capabilities. Together, these disparities emphasize the funda- mental distinction between operation-driven veri- fication, where correctness is grounded in observ- able state transitions, and understanding-driven verification, where correctness relies on semantic alignment with user intent. Overcoming this gap presents a central challenge for the future develop- ment of automated web testing. 14 B.4 Analysis of Tool Usage To better understand the interaction workflows of CUAs during automated web testing, we analyze the playwright tool usage of representative models, as shown in Table 6. We observe substantial variation in interaction volume across models. GPT-5.2 produces the high- est number of tool calls (61.96 per instance), in- dicating a preference for aggressive exploration through frequent web interactions. In contrast, GPT-5.1 uses fewer calls (23.23 per instance), sug- gesting a more efficiency-oriented strategy, likely supported by stronger planning capabilities. Across all models,browser_clickis the most frequently used tool, accounting for approximately 50% to 63% of total calls. This is consistent with real-world web interaction patterns, where click- based actions constitute the most fundamental and pervasive form of engagement. In addition, notable differences emerge in how models leverage auxil- iary tools. MiMo-V2-Flash and GPT-5.2 rely more heavily onbrowser_evaluate(8.29 and 5.86 per instance respectively), indicating a tendency to in- spect underlying page structures such as the DOM structure or JavaScript execution results. While this approach can enable more precise state verification, it also incurs considerable computational overhead, as reflected in the significantly higher context token consumption of both models (7.26M and 7.43M respectively). Notably, these models exhibit rela- tively low usage ofbrowser_wait_for, suggest- ing that asynchronous behaviors are not adequately handled. This may cause models to misinterpret transient UI rendering delays and asynchronous state updates as genuine defects, contributing to elevated false-positive rates. These findings highlight that future CUA sys- tems would benefit from more deliberate interac- tion planning and more selective use of tools, both of which are important for improving testing relia- bility in dynamic web environments. In addition, Figure 5 illustrates the web inter- action process of CUAs during defect detection. We present the invoked Playwright tools, the corre- sponding actions, and the resulting webpage screen- shots after each interaction. This visualization is based on the output of GPT-5.1. B.5 Detailed Results and Analysis in the Oracle Setting Table 11 reports detailed results under the oracle setting, where the gold checklist is directly pro- vided to the defect detection agent. Compared to the end-to-end setting, all models show clear improvements, indicating that incomplete check- lists are the major bottleneck in end-to-end test- ing. In this setting, the gap between closed-source and open-source models also becomes more pro- nounced. Claude Sonnet 4.5 and GPT-5.1 achieve overall F1 scores of 49.2% and 46.8%, respectively, clearly outperforming MiMo-V2-Flash (31.9%) and Step-3.5-Flash (40.9%). This pattern indicates that checklist generation is the primary limiting fac- tor for closed-source models in end-to-end setting, while open-source models still face difficulties in both checklist generation and detection. Across all models, Constraint consistently achieves the highest F1 scores, with Claude Sonnet 4.5 reaching 66.7% and GPT-5.1 reaching 59.2%. This is consistent with the nature of constraint vi- olations, which often appear as explicit and ob- servable state changes that are easy to verify once the relevant test item is given. In contrast, Func- tionality remains difficult for open-source models even in the oracle setting, suggesting limitations in handling multi-step interactions and reasoning over execution processes. Notably, Content shows the lowest performance across all models. This in- dicates that verifying semantic alignment between displayed content and development intent or test query remains challenging. C More Related Work Web Application Generation. LLMs have cat- alyzed a paradigm shift in programming, evolving from basic code completion (Fried et al., 2023; Li et al., 2023) to coding agents (Yang et al., 2024; Zhang et al., 2024; Gao et al., 2025) and project synthesis from scratch (Zan et al., 2024; Zhao et al., 2024b). These advances lay a crucial foundation for the application of LLM in web application gen- eration. Early studies focus on the UI-to-Code paradigm (Yun et al., 2024; Wu et al., 2024), where models replicate web designs from visual screen- shots. However, corresponding benchmarks focus on static visual fidelity, such as visual similarity or component matching (Si et al., 2025; Xiao et al., 2025; Lin et al., 2025b), neglecting dynamic inter- actions (Xiao et al., 2024). In contrast, Language- 15 to-Code is closer to practical development: users specify desired functionality and appearance, and an agent synthesizes interactive applications (Wan et al., 2025). ArtifactsBench (Zhang et al., 2025) introduce interaction-oriented evaluation via inter- action screenshots, while WebGen-Bench (Lu et al., 2025b) used web agents to test functionality. How- ever, these evaluations rely on predefined check- lists, limiting their ability to handle open-end de- velopment scenarios. Automated Web Testing. Web application test- ing is a critical yet time-consuming phase in soft- ware development, motivating the pursuit of auto- mated solutions. Early automation methods like Selenium use script tools to execute user operations. While simplifying manual tasks, they tightly bind to the website’s DOM structure and event triggers, leading to limited scalability and high maintenance costs (Brisset et al., 2022). Training-based meth- ods (Li et al., 2019; Lan et al., 2024) can learn testing strategies, but they suffer from high train- ing costs and limited generalization to emerging web frameworks. Recent works (Liu et al., 2024; Hu et al., 2025) investigate multimodal LLMs for testing mobile applications. For web applications, Le et al. (2025) focus on automating site naviga- tion and form filling. UXAgent (Lu et al., 2025a) introduces personalized agents for testing web ap- plications. In this paper, we focus on the automated web application testing. Beyond basic functionality, we explore agents’ performance on long-horizon tasks and their ability to detect latent bugs. 16 CategoryDetails Presentation Description: Presents information in a clear, structured form, , enabling users to browse, read, and access content efficiently. Core Functionalities: Content layout and typography, navigation structure, pagination and indexing, media rendering, basic search and filtering. Typical Examples: Blog and article websites, personal homepages, corporate websites, documentation and wikis, portfolios, and news portals. Search Description: Supports search, filtering, ranking, and recommendation over heterogeneous datasets, emphasizing information discovery and matching. Core Functionalities: Search input and query syntax, filter and sort controls, result list presentation, recommendation-based content delivery, result detail pages. Typical Examples: Job platforms, real estate listing websites, academic literature search engines, content recommendation systems. Tool Description: Provides a structured input–process–output workflow to help users perform transformation, analysis, or generation tasks, typically emphasizing accuracy and clear feedback. Core Functionalities: Input collection (forms), algorithmic or logical processing, result presentation and visualization, and editors for text, graphics, or tables. Typical Examples: Data calculation and conversion tools, visualization generators, online document editors, format converters, simulators and configurators. Commerce Description: Supports complete transaction flows, from product or service selection, often involving business logic related to inventory, pricing, orders, and payment verification. Core Functionalities: Product and service browsing, shopping cart and checkout pro- cesses, payment and confirmation mechanisms, order management, inventory and pricing management, reservation and scheduling features. Typical Examples: E-commerce websites, booking systems for hotels, flights, restaurants, or meeting rooms, subscription platforms, second-hand marketplaces. Data Management Description: Provides management, monitoring, and operational capabilities over struc- tured data, typically in internal or back-office settings, emphasizing organization, editing, and visualization. Core Functionalities: Data tables with CRUD operations, access control mechanisms, dashboards and audit logs, statistical charts, and system configuration. Typical Examples: User, order, and inventory management systems, enterprise back-office platforms, visual analytics dashboards. Workflow Description: Organizes user tasks through predefined steps or stages, emphasizing process control, state management, approval flows, and cross-role collaboration. Core Functionalities: Multi-step forms, tracking and state transitions, task and ticket management, approval workflows, content management modules, cross-role collaboration. Typical Examples: Content management systems (CMS), project management systems, task and approval systems, complex registration or data collection pipelines. User-Generated Content Description: Core content is created, edited, shared, and interacted with by users, empha- sizing community dynamics, creation toolchains, and social networks. Core Functionalities: Content publishing and editors, comments and reactions, user profiles and social graphs, content moderation, notifications and real-time updates. Typical Examples: Online forums, commenting systems, social platforms, creative plat- forms for writing, images, or video content. Table 7: Taxonomy of web application categories used in WebTestBench. 17 DimensionDetails Functionality Description: Focuses on whether the application can successfully complete the pri- mary tasks and business flows specified in the development instruction under normal usage conditions. Evaluation Focus: Successful completion of core operations (create, view, edit, delete, submit); correct responses to normal user requests (search, filter, navigate, form submission); coherent cross-module flows and reachable navigation. Typical Examples: A user can submit a search query and view the returned results; a user can create a new project and see it reflected in the list. Constraint Description: Focuses on whether the application is robust under abnormal or bound- ary usage, including input validation, error handling, and prevention of invalid or conflicting states. Evaluation Focus: Effective validation of required fields with clear error messages; blocking of conflicting or illegal operations (e.g., double-booking, editing non-existent records); graceful handling of boundary conditions (empty lists, excessively long inputs, invalid formats); consistency between interface state and underlying data. Typical Examples: An already-reserved meeting room cannot be booked again; sub- mitting a form with empty required fields is blocked with a clear prompt. Interaction Description: Focuses on the dynamic feedback and visual or state changes the ap- plication provides in response to user actions, reflecting the experiential quality of functionality rather than its correctness. Evaluation Focus: Clear feedback after click, input, or toggle actions (e.g., button state changes, toast notifications, success or failure messages); smooth state transitions such as tab switching, expand/collapse, and toggle; visible and interpretable action outcomes (e.g., redirect or confirmation after submission). Typical Examples: A success notification appears after a booking is created; hovering over a button triggers an appropriate visual change. Content Description: Focuses on whether the information displayed by the application is accurate, complete, and semantically consistent with the development instruction, covering all front-end content elements including text, data, images, and icons. Evaluation Focus: Thematic consistency of text, field labels; complete and well- formatted content in lists, detail pages, and cards; media assets (images, icons) rendered correctly and relevant to the theme. Typical Examples: All images on an iPhone showcase page are iPhone-related; each blog card includes the title, summary, and date fields required by the instruction. Table 8: Taxonomy of test case dimensions used in WebTestBench. 18 Model FunctionalityConstraintInteractionContent Cov.PRF1Cov.PRF1Cov.PRF1Cov.PRF1 Minimax M2.177.915.914.012.340.419.914.415.842.222.620.319.947.18.67.37.7 Qwen3 Coder Next77.617.514.914.148.327.523.123.842.712.412.811.435.95.13.94.3 GLM 4.779.921.016.816.547.326.518.420.536.018.917.617.248.95.13.94.3 GLM 579.716.811.011.950.132.426.526.941.424.220.320.950.65.12.63.4 Step 3.5 Flash79.828.118.720.153.632.627.427.948.524.221.221.260.62.62.62.6 MiMo-V2-Flash80.024.824.821.948.734.228.629.248.022.619.320.350.37.39.07.3 Claude Opus 4.5 83.323.218.118.850.327.418.321.240.818.613.414.742.17.76.46.8 Claude Sonnet 4.581.026.123.322.247.729.020.322.546.722.620.319.951.62.61.31.7 GPT 5.276.925.431.925.351.923.123.121.543.027.521.923.246.17.75.86.2 GPT 5.176.428.742.130.951.229.627.526.949.723.523.522.057.514.516.715.3 Table 9: Detailed quality dimension-wise evaluation results on WebTestBench. Model PresentationSearchToolCommerce Cov.PRF1Cov.PRF1Cov.PRF1Cov.PRF1 Minimax M2.156.925.610.314.352.018.96.99.458.715.213.713.457.110.910.18.6 Qwen3 Coder Next62.516.416.414.362.721.511.113.756.133.812.216.666.126.816.819.1 GLM 4.759.522.412.715.663.919.426.520.062.417.87.89.863.435.919.124.1 GLM 564.530.816.721.161.611.14.66.564.024.711.414.064.933.615.419.6 Step 3.5 Flash65.738.613.218.868.424.113.717.161.437.222.225.372.834.322.525.6 MiMo-V2-Flash58.523.714.216.565.026.634.126.059.842.128.130.766.235.520.221.6 Claude Opus 4.5 64.641.225.030.559.57.44.65.662.125.515.716.263.655.116.224.2 Claude Sonnet 4.562.143.519.123.864.621.120.919.761.932.622.424.663.522.713.014.0 GPT 5.2 53.433.719.322.959.413.913.013.260.731.035.131.961.520.819.918.2 GPT 5.1 55.627.627.625.762.823.131.323.460.025.033.426.470.128.728.827.2 Avg.60.330.417.520.462.018.716.715.560.728.520.220.964.930.418.220.2 Std.4.28.85.55.44.46.110.97.02.28.49.67.84.411.75.25.7 Model Data ManagementWorkflowUser-Generated ContentOverall Cov.PRF1Cov.PRF1Cov.PRF1Cov.PRF1 Minimax M2.163.836.423.425.365.618.412.312.860.927.220.317.860.122.314.615.2 Qwen3 Coder Next59.342.620.123.259.418.415.815.059.427.015.916.460.427.815.817.3 GLM 4.759.141.822.124.761.122.615.716.359.721.716.216.361.126.716.618.1 GLM 561.338.020.724.665.540.123.326.858.823.69.912.763.130.415.619.0 Step 3.5 Flash 67.347.933.835.665.027.219.219.063.524.312.014.666.034.620.823.4 MiMo-V2-Flash66.444.133.234.164.944.124.028.463.713.816.110.663.534.824.625.1 Claude Opus 4.563.028.213.417.066.442.824.529.061.023.611.013.763.233.016.520.2 Claude Sonnet 4.5 64.743.823.426.967.231.422.824.760.820.112.614.363.732.119.721.9 GPT 5.2 62.428.424.423.965.321.828.823.662.016.629.019.961.024.725.222.9 GPT 5.1 66.629.239.929.464.428.038.229.560.615.528.519.563.125.833.326.4 Avg.63.438.025.426.564.529.522.522.561.021.317.215.662.529.220.321.0 Std.2.97.27.95.42.49.87.46.31.64.76.83.01.84.45.93.6 Table 10: Detailed category-wise evaluation results on WebTestBench. ModelFunctionalityConstraintInteractionContentOverall MiMo-V2-Flash22.2/20.1/19.350.5/49.7/46.633.3/31.1/30.316.2/16.7/15.844.3/30.4/31.9 Step-3.5-Flash33.5/25.6/26.461.4/56.9/55.639.2/36.9/35.930.3/33.3/31.253.4/38.7/40.9 GPT-5.1 42.8/54.8/42.057.0/76.5/59.255.9/65.7/56.036.5/46.2/38.247.0/63.4/46.8 Claude Sonnet 4.5 45.0/32.3/34.569.1/72.6/66.744.8/42.8/41.432.3/32.7/30.761.0/47.4/49.2 Table 11: Detailed results on WebTestBench in the oracle setting. 19 Tool: browser_navigate Action: Navigate to web_url Tool: browser_click Action: Click “Book Repair” button 1 Tool: browser_click Action: Click “Refrigerator Cooling issues,” button 2 3 Tool: browser_click Action: Click “Continue” button Tool: browser_click Action: Click “Not cooling properly” button 4 Tool: browser_type Action: Type “Refrigerator not ...” on “Additional details ...” textbox 56 Tool: browser_click Action: Click “Continue” button Tool: browser_click Action: Click “Not cooling properly” button 7 Tool: browser_click Action: Click “Continue to Payment” button 89 Tool: browser_click Action: Click “Confirm & Pay $” button Tool: browser_fill_form Action: Click “Not cooling properly” button 10 Tool: browser_click Action: Click “Confirm & Pay $” button 1112 Tool: browser_click Action: Click “Cancel Booking” button Tool: browser_click Action: Click “Book Repair” button 1314 ● Development Instruction: I want a webpage where users can book home appliance repair services. Users should be able to select the type of appliance that needs repair (like refrigerator, washing machine, air conditioner, or microwave), describe the issue they're experiencing, and choose from available appointment time slots. After selecting a time, they can proceed to a payment page where they enter payment details and confirm the booking. Once the booking is confirmed, users can view their appointments in an order history page and have the option to cancel an appointment if needed. Figure 5: An example illustrating how CUA interacts with a web application during defect detection. 20 Prompt for Checklist Generation Agent # Role You are a Senior Software Quality Assurance Engineer who can read a user instruction and immediately produce a complete, executable UI/UX test checklist. Your focus is strictly on ’what’ the application should do (the features), not ’how’ it should be built (the technical implementation). # Task Directly generate executable test checklist. Decompose the user instruction into structured, testable items. Each item must be: - **Specific**: Clear action and expected outcome. - **Binary**: PASS or FAIL (no ambiguity). - **Debuggable**: Failure indicates exactly what’s missing. Your checklist will be used to: 1. Test web applications. Produce PASS/FAIL results for each test item. 2. Generate detailed bug reports identifying which requirement failed and why. ## Checklist Item Category 1. Functionality (FT) * Focus: Core user tasks and workflows that must succeed when inputs are valid. * Scope: What happens when everything goes right? * Example: "User can submit a search query", "User can add an item to the cart". 2. Constraint (CS) * Focus: Rules, validations, state invariants, and conflict-prevention logic that prevent the system from entering invalid or contradictory states. * Scope: What prevents the user from doing the wrong thing? What happens with conflicting data? * Examples: "Meeting room cannot be booked if already occupied.", "Cannot submit form with empty required fields." 3. Interaction (IX) * Focus: Dynamic behaviors and system responses to user actions (non-functional visual/state changes, user experience). * Scope: How does the interface respond to events like clicks, hover? * Examples: "Show success toast after reservation is created." 4. Content (CT) * Focus: The relevance and integrity of text, data, and media (images, icons, videos). Content must strictly align with the instruction’s theme/purpose. * Scope: Is the displayed information relevant, and fully functional? * Examples: "All displayed images must be directly relevant to the theme of ’iPhone’." ## Default Data Assume the application has default data (e.g., pre-existing products in a store). Do not create new data for testing; use the default data already present in the application. # Unified Checklist Item Template ```markdown - [ ] [ID]: [Test description] - Action: [What to do] - Expected: [What should happen] ``` # Output Format (Markdown) ```markdown # Test Checklist ## Functionality - [ ] FT-01: [use unified template] - [ ] FT-02: [use unified template] ## Constraint - [ ] CS-01: [use unified template] ## Interaction - [ ] IX-01: [use unified template] ## Content 21 - [ ] CT-01: [use unified template] ``` # Rules 1. Testable: Every item must produce a clear Pass/Fail result. 2. Executable: Quality assurance tester should know exactly what to do. 3. Specific for action/expected: Include exact element names, button text, expected messages, etc. 4. Concise for description: Test description should be 1-2 lines, action/expected should be brief. 5. No Implementation: Specify what the app does, not how it’s built (no framework details). 6. Desktop Only: Ignore responsive design requirements. 7. Max 20 items total: Prioritize core requirements. Keep only what is necessary to satisfy the instruction. 8. No Redundancy: Avoid duplicating content or behavior that is covered by other categories (e.g., "success messages" should be included only once). Each checklist item MUST be assigned exactly one primary category (FT / CS / IX / CT), even if it has secondary implications. # Input ## User Instruction $instruction # Output (Markdown) Figure 6: Prompt for Checklist Generation Agent. Prompt for Defect Detection Agent # Role You are an expert Quality Assurance Test Engineer specializing in automated UI/UX testing. Your task is to validate a web application against a provided checklist. You must systematically execute actions, verify results, and update the checklist status. # Execution Standards ## 1. Interaction Strategy - Tool Use: Use **Playwright tools** to interact with the DOM. Disallow the use of `Bash`, `Read`, and `Write` tools to operate web pages. - DOM-Only: Do NOT use screenshots or visual validation. Rely on DOM attributes (text, id, class, accessibility roles) for verification. - Integrity: Execute all items; never skip. If an item cannot be done, mark FAIL with a concrete reason (no hallucination). - Batching: For pure data entry (e.g., filling a form), you may combine multiple `fill/select` actions into a single code block to save time. - Limited Budget: The entire execution process must operate within a limited budget of turn/tool-call (max 150 times total). Plan first, and execute with as few operations as possible. - Navigation: Only navigate if the checklist item explicitly requires it. Disable page refresh operations unless the page crashes. ## 2. Verification Logic - Strict Verification: Compare the `‘Actual` behavior of the page against the `Expected` field in the checklist. - Pass: The feature works exactly as described. - Fail: Any deviation (missing element, wrong text, no response, error message) is a FAIL. ## 3. Workflow 1. Initialize: Navigate to the Target URL. 2. Iterate: Go through the Checklist items. 3. Execute: Perform the ‘Action‘ defined in the item. 4. Verify: Check if the ‘Expected‘ result is met. 5. Record: Update the item’s status immediately in your internal memory. # Output Format (Markdown) You must output the Full Checklist with updated statuses. Do not summarize; return the complete list. ## Unified Result Item Template If PASS: Change `- [ ]` to `- [X]` to mark the test as passed. ```markdown 22 - [X] TEST-ID: [original Description] - Action: [original Action] - Expected: [original Expected] ``` If FAIL: Keep `- [ ]` and append a ‘Bug Report‘ block immediately after the test item. ```markdown - [ ] TEST-ID: [original Description] - Action: [original Action] - Expected: [original Expected] - Bug Report: - Issue: [Specific problem type: e.g., Unresponsive Button, Incorrect Form Submission, Element Occlusion] - Actual: [Quote the observed deviation: e.g., Button does not trigger the expected modal, Button text overlaps with icon] ``` ## Output Template ```markdown # Test Result ## Functionality [use unified result item template for each FT-x] [use unified result item template for each FT-x] ## Constraint [use unified result item template for each CS-x] ## Interaction [use unified result item template for each IX-x] ## Content [use unified result item template for each CT-x] ``` # Input ## User Instruction $instruction ## Application URL $server_url ## Test Checklist ```markdown $checklist ``` # Output Figure 7: Prompt for Defect Detection Agent. 23 Prompt for Test Item Matching Given a list of predicted checklist items and a list of gold checklist items. You are required to align predicted (model-generated) test items to gold (human-labeled) test items for the same web instruction. Instruction: ``` $instruction ``` Gold Test Items (`"gold_id": "description"`): ``` $gold_items ``` Predicted Test Items (`"pred_id": "description"`): ``` $pred_items ``` Goal: For each predicted item, decide if it corresponds to exactly one gold item describing the same requirement/behavior. Produce a one-to-one mapping; unmatched predictions should map to None. Matching rules: 1. Mapping constraint: each predicted item maps to AT MOST ONE gold item; each gold item MAY be assigned to MULTIPLE predicted items. 2. Prioritize intent over wording: if a predicted item is more specific/less specific but clearly covers the same user requirement, match it; otherwise, leave it unmatched. 3. Do NOT force matches: if no gold item cleanly aligns, use None. 4. Preserve predicted order: output tuples follow the input predicted sequence; length of output list equals number of predicted items. Output Format (Markdown) [("pred_id_1", "gold_id" or None), ("pred_id_2", "gold_id" or None), ...] DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only output the List. Output: Figure 8: Prompt for Test Item Matching. 24

Instant research discovery