Paper deep dive
WebUIProof: Benchmarking WebUI Code Generators with UI-Agent Execution Harness
Yun-Yun Tsai, Yuning Mao, Shiqi Wang, Junfeng Yang, Sinong Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/5/2026, 6:35:32 AM
Summary
The paper introduces WebUIProof, an execution-oriented benchmark for evaluating WebUI code generation that uses structured specifications and executable interaction tests via a UI-agent harness. It highlights that existing benchmarks often miss functional correctness in interactive scenarios. The authors evaluate eight commercial LLMs, finding frequent failures in interaction-based requirements, particularly for 3D simulations. Additionally, they propose VisRL, a reinforcement learning approach using outcome-level training signals from the UI-agent harness, which significantly improves functional completion and reduces build failures for compact models like Qwen2.5 14B and MIMO 7B.
Entities (8)
Relation Signals (6)
WebUIProof → uses → UI-agent harness
confidence 95% · WebUIProof includes a UI-agent harness that runs executable interaction tests in a headless browser
WebUIProof → covers → 3D simulation
confidence 92% · task families: general WebUIs... and 3D interactive simulation
VisRL → improves → Qwen2.5-14B
confidence 90% · Training compact models (e.g., Qwen2.5 14B... with RL rewards... improves functional completion
VisRL → improves → MiMo-7B
confidence 90% · Training compact models (e.g., ... MIMO 7B) with RL rewards... improves functional completion
UI-agent harness → implements → Playwright
confidence 88% · We evaluate functional correctness using a fixed UI-agent execution harness implemented with Playwright webdriver
WebUIProof → evaluates → Claude-Sonnet-4
confidence 85% · We evaluate across eight commercial LLMs... Claude Sonnet 4... observe frequent failures
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Evaluating WebUI code generation at scale is difficult: outputs may compile and look plausible yet fail under user interaction, and prior benchmarks largely rely on free-form prompts with static checks (build success, screenshots) that miss functional correctness. We introduce WebUIProof, an execution-oriented benchmark that provides structured specifications and dense, executable interaction tests for WebUI generation across two task families: general WebUIs (e.g., dashboards, game, interactive tools) and 3D interactive simulation (e.g., particle/galaxy systems, physics dynamics). WebUIProof includes a UI-agent harness that runs executable interaction tests in a headless browser using an iterative plan--act--observe loop: it locates DOM elements, performs actions, observes resulting UI/DOM changes, and checks the specified assertions. We evaluate across eight commercial LLMs and observe frequent failures on interaction-based requirements even when pages render successfully, especially on 3D simulation interfaces. Finally, we show the UI-agent harness can provide outcome-level training signals. Training compact models (e.g., Qwen2.5 14B and MIMO 7B) with RL rewards derived from executable interaction tests improves functional completion while reducing build failures.
Tags
Links
- Source: https://arxiv.org/abs/2610.02617v1
- Canonical: https://arxiv.org/abs/2610.02617v1
Trouble viewing inline? Open PDF directly →
Full Text
97,101 characters extracted from source content.
Expand or collapse full text
WebUIProof: Benchmarking WebUI Code Generators with UI-Agent Execution Harness Yun-Yun Tsai †thanks: Work done in internship at Meta Affiliation: Columbia University Affiliation: yunyuntsai,junfeng@cs.columbia.edu, Yuning Mao Affiliation: Meta SuperIntelligence Labs Affiliation: yuningm,tcwangshiqi,sinongwang@meta.com Shiqi Wang Affiliation: Meta SuperIntelligence Labs Affiliation: yuningm,tcwangshiqi,sinongwang@meta.com Junfeng Yang Affiliation: Columbia University Affiliation: yunyuntsai,junfeng@cs.columbia.edu, Sinong Wang Affiliation: Meta SuperIntelligence Labs Affiliation: yuningm,tcwangshiqi,sinongwang@meta.com Abstract Evaluating WebUI code generation at scale is difficult: outputs may compile and look plausible yet fail under user interaction, and prior benchmarks largely rely on free-form prompts with static checks (build success, screenshots) that miss functional correctness. We introduce WebUIProof, an execution-oriented benchmark that provides structured specifications and dense, executable interaction tests for WebUI generation across two task families: general WebUIs (e.g., dashboards, game, interactive tools) and 3D interactive simulation (e.g., particle/galaxy systems, physics dynamics). WebUIProof includes a UI-agent harness that runs executable interaction tests in a headless browser using an iterative plan–act–observe loop: it locates DOM elements, performs actions, observes resulting UI/DOM changes, and checks the specified assertions. We evaluate across eight commercial LLMs and observe frequent failures on interaction-based requirements even when pages render successfully, especially on 3D simulation interfaces. Finally, we show the UI-agent harness can provide outcome-level training signals. Training compact models (e.g., Qwen2.5 14B and MIMO 7B) with RL rewards derived from executable interaction tests improves functional completion while reducing build failures. Project link: https://github.com/yunyuntsai/webuiproof-benchmark 1 Introduction With LLMs increasingly integrate into the software development cycles, reliable evaluation for code generation becomes critical, motivating the benchmarks for real-world GitHub issues Jimenez et al. (2023), programming competitions Li et al. (2022), multilingual coding tasks Zheng et al. (2023), and interview coding problems Hendrycks et al. (2021). Frontend code generation (WebUI generation) differs fundamentally from backend code synthesis and remains comparatively underexplored. The correctness depends not only on whether code compiles, but on what the UI renders and how it behaves under interaction. Existing WebUI benchmarks only partially address this challenge. WebDev-Arena Zheng et al. (2024) provides a large set of web prompts, but instructions are largely free-form and evaluation relies on human preference rather than executable functional verification. WebGen-Bench Liu et al. (2024b) introduces test cases for web prompts, but many requirements stated in instructions are left unchecked, resulting in incomplete coverage of instruction-following. More broadly, many benchmarks Xiao et al. (2025); Zhang et al. (2025); Team (2024) depend on static checks such as build success, lint/type checks, or screenshots, which are insufficient to validate interactive behavior. We introduce WebUIProof, an execution-oriented benchmark designed to measure functional, interactive correctness of WebUI code generation. WebUIProof replaces free-form prompts with structured specifications (website summary and UI element requirements) and pairs them with executable interaction tests that drive the UI agent to verify expected behavior. WebUIProof contains 219 tasks (149 general + 70 3D) paired with 3k executable interaction tests, enabling functional evaluation at much higher granularity than prompt-only benchmarks. The tasks span two families that cover standard UI workflows and richer interactive behaviors, including dashboards/games/tools and interactive 3D simulations such as physics dynamics and 3D scenes (see Fig. 1). Our evaluation protocol is automated and reproducible, with a fixed UI-agent execution harness that runs these tests in a headless browser by iteratively planning the next interaction, executing browser actions (e.g., click, select, type, scroll), observing DOM/visual state changes, and verifying test assertions. While WebUIProof is primarily a benchmark contribution, the harness also provides outcome-level training signals by self-observing the rendered UI in training rollouts and deriving interactive tests from it. We therefore include VisRL (Visual-guided Reinforcement Learning), an execution-grounded RL approach that optimizes a compact model using composite rewards from build/runtime execution, visual quality, and UI-agent execution test success. In addition, we adopt several training strategies, such as multi-temperature sampling and fast-gate filtering, to improve rollout efficiency and stabilize learning under costly executable evaluation. Our contributions are as follows: 1. We introduce WebUIProof, an execution-oriented benchmark for WebUI code generation that pairs structured specifications with executable interaction tests and a fixed UI-agent evaluation protocol for functional correctness. It contains 219 tasks spanning (i) general WebUIs (e.g., dashboards, games, interactive tools) and (i) 3D simulation WebUIs (e.g., physics-driven interactive scenes). 2. We provide an agent-assisted construction pipeline for producing structured specs. and element-aligned executable tests at scale (Sec. 3). 3. With our executable UI-agent protocol (DOM grounding + interaction execution + assertions), we benchmark eight commercial LLMs and show that many outputs build/render yet fail interaction requirements, especially on 3D simulation interfaces. (Sec. 4). 4. We include VisRL as an execution-grounded RL approach enabled by the execution harness with composite rewards. Training compact model such as Qwen2.5 14B Instruct and MIMO 7B yield significant improvement of accuracy on general WebUI (+15.7% gain) and 3D simulation tasks (+4.9% gain) over SFT. We provide comprehensive ablations on evaluator robustness, benchmark quality analysis, and learning strategies (Sec. 5). Figure 1: Examples of WebUI outputs in WebUIProof, spanning general applications (landing pages, code IDE, social/profile pages, games) and 3D simulation interfaces (physics dynamics, planetary scenes). These examples illustrate the benchmark’s emphasis on both visual fidelity and interaction-driven functionality. Each task is evaluated with executable interaction tests that check user actions (click/type/drag) and the resulting UI behavior. 2 Related Works Benchmark for Frontend Code Generation. Prior agentic SWE systems (e.g., SWE-Agent He et al. (2024b), SWE-bench Jimenez et al. (2023), SWE-Gym Yang et al. (2024b), StackEval Zhang et al. (2024)) focus on repository-level editing and bug fixing, whereas we study end-to-end WebUI generation evaluated through buildability, visual fidelity, and UI-agent-tested interactivity. FrontendBench Zhu et al. (2025) first standardizes the pipeline for frontend code generation as: (1.) LLMs receive instructions for generating code, (2.) render the code in a controlled sandbox environment, (3.) automate test scripts, and let the agent provide feedback loops. WebDev-Arena Zheng et al. (2024) is a large-scale benchmark that contains over 10k instruction prompts collected from online sources. Models are evaluated through pairwise comparisons with human votes determining preference. While valuable for capturing human preference, the instructions are loosely structured and lack automated test cases. DesignBench Xiao et al. (2025) evaluates framework-based web development (e.g., React/Vue/Angular) across generation, editing, and repair. The key difference in their setting is the evaluation signal, which scores outputs based on screenshot-level visual similarity and code modification similarity via Jaccard similarity. In particular, it does not employ a UI agent to locate DOM elements and verify behavioral assertions via scripted test cases. WebGen-Bench Liu et al. (2024b) introduces web agents that take prompt instructions to generate a frontend project with full repository, and evaluate the website with executable web tasks in the sandbox build environment In contrast, our WebUIProof provides structured specifications (website summary, functional descriptions, and required UI elements) paired with executable test cases, and evaluates generations by a UI agent that interacts with the rendered page via DOM-level element localization and action execution (click/type/navigate) to verify test assertions. Table 1 provides a detailed comparison of benchmarks for WebUI code generation. Figure 2: Benchmark instance format of WebUIProof. We show a structured spec. with paired test cases. Our benchmark construction pipeline first lets the UI agent collect a sequence of visual snapshots and HTML/DOM structure from the given seed webpage. Based on observations from webpages, the UI agent provides a website summary and test case recommendations. The red and blue boxes illustrate element-to-test alignment: a subset of entries in "element_requirement" is grouped and paired with executable test cases, making element-level coverage explicit. Web-based Testing with UI Agents. WebUI evaluation commonly relies on headless browsers such as Playwright11 1 https://playwright.dev and Selenium WebDriver22 2 https://w.selenium.dev/, which supports reliable automated testing via DOM inspection, event simulation, and screenshot capture. Building on this, recent UI agents, such as WebVoyager He et al. (2024a), combine HTML and screenshots for end-to-end reasoning and action. In addition, WebArena Zhou et al. (2023) and BrowserGym Raman et al. (2023) provide standardized environments for reproducible web-agent evaluation. Related benchmarks such as WebBench Halluminate and Skyvern (2025) and VisualWebBench Liu et al. (2024a) assesses UI-agent capability on webpage browsing, understanding, and grounding. In our work, we implement our evaluator based on WebVoyager for both general WebUI generation and 3D simulation. Table 1: Benchmarks comparison for WebUI code generation. We compare the benchmark based on instruction number, form, type, length, and output scope. UI Topic indicates general WebUIs (dashboards/tools/games) or additionally includes 3D simulation interfaces (classical mechanics/particles simulation). UI Agent Eval. denotes whether the evaluation involves a UI agent that executes test suites. Test coverage rate (TCR) measures element-level coverage. Our WebUIProof provides diverse UI topics (general+3D) with rich test suites and high TCR. Benchmark #Instr. Instr. Form Instr. Type Instr. Length Output Scope UI Topic UI Agent Eval. #Tests TCR(%) DesignBench Xiao et al. (2025) 900 NL prompts Gen./Edit/Repair ∼ 80-150 Single file (HTML) General ✗ 0 0% WebDev-Arena Zheng et al. (2024) >>10k NL prompts Gen. ∼ 60 Repo (App) General ✗ 0 0% FrontendBench Zhu et al. (2025) 148 NL prompts Gen. ∼ 50-100 Single file (HTML) General ✓ 148 ∼ 90% WebGen-Bench Liu et al. (2024b) 101 NL prompts Gen. ∼ 40-70 Repo (App) General ✓ ∼ 600 49.24% WebUIProof (Ours) 219 Structured Spec. Gen. ∼ 270-550 Repo (App) General+3D ✓ ∼ 3k 97.23% 3 Method This section describes WebUIProof, including (1) the benchmark overview, (2) the UI-agent evaluation protocol used for scoring, and (3) the agent-assisted benchmark construction pipeline. Separately, we also present VisRL as a training approach to illustrate how the executable harness can provide outcome-level learning signals. 3.1 WebUIProof Benchmark Overview Task Families and Scope. WebUIProof contains 219 tasks spanning two families: (i) General WebUIs (149) and (i) 3D simulation WebUIs (70). General tasks cover five types of common web applications, such as dashboards and data analytics tools, social profile interfaces, and interactive games; 3D simulation tasks cover six types of interactive Three.js-style scenes, such as physics dynamics, fluid waves simulation, and astro particles visualizations. (Appendix B.2 Fig. 10 shows the category distribution). Data Sources. We collect seed prompts from WebGen-Bench Liu et al. (2024b) and WebDev-Arena Zheng et al. (2024), and we generate seed webpages with these prompts to drive agent-assisted structured specification and test generation. The 3D simulation WebUIs are built from Three.js33 3 https://threejs.org/ example pages and similar public demo scenes, focusing on browser-based WebGL simulations with interactive user controls (See Appendix B.1). Benchmark Instance Format. Fig. 2 illustrates the benchmark instance format of WebUIProof. Each task is packaged as (website_summary, element_requirement, test_cases). The website_summary describes appearance and functionality; element_requirement lists required UI elements with stable identifiers and locator hints; and test_cases are executable interaction tests with preconditions, procedure, and assertions. We group tests into three semantic types: Functionality, Data-display, and Design-validation tests. (Appendix B.2 Fig. 11 shows the test case distribution). 3.2 UI-Agent Evaluation Protocol WebUIProof uses a fixed UI-agent execution harness to score functional correctness via executable interaction tests. For each test case, the agent follows an iterative Plan→ → loop. Specifically, it observes DOM and visual state changes, thinks and plans the interaction for the next step, executes browser actions (e.g., click/type/hover/drag/scroll, etc), and checks the test assertions. The final score aggregates test outcomes into Full/Partial/No completion based on passed assertions. This protocol is applied identically to all models and tasks during the evaluation. 3.3 Benchmark Construction Pipeline We construct WebUIProof with an agent-assisted pipeline that produces structured specifications paired with executable test cases. The pipeline is designed to (i) scale to diverse webpages, (i) maximize test coverage over UI elements and behaviors, and (i) minimize manual annotation. Each benchmark instance is packaged as: (website_summary, element_requirement, test_cases). Phase I: Seeding from static webpages. Given a target webpage, we first collect (1) a set of visual snapshots (multiple views if available) and (2) the HTML/DOM structure of the page. We prompt a UI agent to produce a website summary that includes appearance and functionality, and to propose a small set of seed test cases that cover common workflows (e.g., search/filter, navigation, form submission). This stage provides basic test cases for UI agent to start the interactive exploration. Phase I: Interactive exploration and UI elements retrieval. The dynamic states, such as hidden panels, modal dialogs, pagination, and hover states, may exist but are underexplored. We therefore run the agent on the webpage with basic test cases for exploration. Specifically, the agent executes an iterative Plan→ → loop: (1.) Plan: The agent receives the test cases and plans the execution (e.g., open settings, trigger sorting, expand menus). (2.) Action: The agent makes decisions on the next action (e.g., click/type/hover/drag/scroll). (3.) Observation: The agent records state changes and extracts structured information: (i) newly discovered UI elements, (i) updated DOM attributes, (i) textual/UI feedback, (iv) intermediate task progress, and (v) test cases recommendation. We ask the LLMs to provide the test cases in three perspectives, including function accessibility, data display correctness, and UI design or style consistency. From these loop trajectories, we incrementally update our structured spec.: (i) element_requirement, a canonical list of required UI elements with stable identifiers and locator hints (text, role, class/id, hierarchical path); and (i) test_cases, expanded to reflect newly discovered states. Phase I: Test case synthesis and post-processing. After each agent loop, we parse the interaction trajectory and extract candidate test cases from step-wise observations. Each case is formatted with: test_name, coverage_element, preconditions, procedure, and assertions. We then perform element-level coverage checking to identify UI elements that never appear in interactions or assertions. For any uncovered elements, we prompt the UI agent to generate additional test cases. Finally, we de-duplicate near-identical cases by merging variants that differ only in constants (e.g., slider values or dropdown options). The resulting instance is stored as a JSON file containing the structured specification and executable test suite. In Appendix B.3 and B.4, we provide more detail about collection pipeline and samples for structured specification. Figure 3: VisRL Overview: Given instruction prompts, the Policy LLM generates UI code candidates, which are built and rendered to produce screenshots; rewards combination (i) code validity with error fixing loop, (i) vision-based screenshot scoring, and (i) UI-agent interaction tests (element detection, task proposal, and execution success), and are used to update the policy via PPO. 3.4 VisRL: RL Training with Executable Harness While WebUIProof is primarily an evaluation benchmark, the executable UI-agent harness can provide outcome-level training signals. We include VisRL (Visual-guided Reinforcement Learning) as a new training approach that optimizes a compact model using rewards computed from multiple outcomes: (i) build/runtime execution, (i) rendered visual quality, and (i) UI-agent interaction-test success. Concretely, given generated code c and a task specification p, we build and render the project and compute a composite reward r(c,p)=αvrv(I(c))+αuru(s(c),(p))+αmrm(c)r(c,p)= _vr_v(I(c))+ _ur_u(s(c),T(p))+ _mr_m(c), where rvr_v scores rendered screenshots, rur_u is the interaction-test pass rate, and rmr_m is a lightweight analyzer penalty/bonus for common failure patterns. Full PPO details are provided in Appendix C.2 C.3, C.4. Design Insight. We observe that single-signal rewards are insufficient in WebUI generation. Optimizing interaction tests only tends to produce minimal but functional UIs, while optimizing visual scores only improves visual appearance quality without guaranteeing correct behavior. Thus, combining build/runtime, visual, and interaction signals yields the most balanced improvements across functionality and UI appearance, We provide comprehensive studies on reward decomposition and weight sensitivity in Appendix D.2, D.3. Training Stability. Because rewards are only observed after build/render/test, we adopt two lightweight strategies to stabilize and reduce rollout cost: multi-temperature sampling to increase candidate diversity under a fixed best-of-N budget, and fast-gate filtering to skip obviously invalid generations before expensive execution. Ablations in Sec. 5.3 show both components matter for accuracy and build failure reduction. 4 Experiment Table 2: General WebUI generation on WebUIProof. We report the build failure rate (% of generations that fail to build/render), and the UI-agent verified task outcome among build-successful generations: No/Partial/Full completion rate (%). Overall Accuracy is computed as Acc=#Full+0.5⋅#PartialN×100Acc= \#Full+0.5·\#PartialN× 100, where build failures are counted as No completion. Rows are grouped into eight proprietary (API) models and two self-trained models, including Qwen2.5 14B Instruct and MIMO 7B under three training variants: SFT, PPO w/ vision agent, and VisRL (ours). Model Source Model Name / Training Variant Build Failure Rate (%) Build Success – UI-Agent Verified Task Outcome (%) Accuracy (%) (Full + 0.5⋅·Partial) No Completion Partial Completion Full Completion Proprietary (API) Claude Sonnet 4 Anthropic (2025) 30.37 12.59 22.10 35.05 46.10 Gemini 2.5 Pro DeepMind (2025) 32.21 15.02 20.38 32.67 42.86 Qwen 3 Coder 480B Alibaba Cloud (2025) 33.05 13.92 20.76 31.62 41.99 DeepSeek R1 DeepSeek (2025) 32.89 14.91 20.51 31.65 41.90 GPT 4.1 OpenAI (2025) 36.58 11.67 20.30 31.25 41.40 Llama 4 Maverick Meta (2025) 46.64 2.53 18.72 31.41 40.77 Kimi K2 Moonshot (2025) 46.31 8.70 11.94 33.33 39.30 Llama 3 405B Meta (2024) 47.32 10.13 14.18 29.10 36.19 Self-Trained Qwen 2.5 14B Alibaba Cloud and Team (2024) SFT 56.25 10.97 11.84 20.66 26.08 + PPO w/ Vision agent 41.42 (-14.83) 13.96 11.65 32.97 38.79 (+12.71) + VisRL (Ours) 36.10 (-20.15) 13.71 15.13 34.23 41.80 (+15.72) MIMO 7B Xiaomi (2025) SFT 58.71 12.15 7.33 20.2 23.8 + PPO w/ Vision agent 51.44 (-7.27) 15.33 11.52 21.3 27.06 (+3.26) + VisRL (Ours) 40.22 (-18.49) 17.43 15.71 25.63 33.43 (+9.63) 4.1 Experimental Setting Model. We evaluate eight state-of-the-art LLMs spanning different architectures, scales, and training paradigms. Claude Sonnet 4 Anthropic (2025) and Gemini 2.5 Pro DeepMind (2025) represent strong closed-source reasoning-oriented models with multimodal capabilities. Qwen 3 Coder 480B Instruct44 4 https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct Alibaba Cloud (2025) and Llama 4 Maverick Meta (2025) / Llama 3 405B Meta (2024) are open-source instruction-tuned models optimized for code synthesis and general reasoning. DeepSeek R1 DeepSeek (2025) is an open-weight model emphasizing step-by-step “reasoning-first” training. Moonshot Kimi K2 Moonshot (2025) is a high-efficiency proprietary model designed for long-context understanding and structured output. Finally, GPT 4.1 OpenAI (2025) serves as a strong general-purpose baseline. Furthermore, we use two compact models Qwen 2.5 14B Instruct55 5 https://huggingface.co/Qwen/Qwen2.5-14B-Instruct Alibaba Cloud and Team (2024); Yang et al. (2024a) and MIMO 7B Xiaomi (2025) as the base model for training. We compare the results of VisRL with two training variants (SFT baseline and PPO with Vision agent). UI-agent Evaluator Setup. We evaluate functional correctness using a fixed UI-agent execution harness implemented with Playwright webdriver and a vision-language model (Qwen2.5 VL 72B) as the evaluator. The evaluator runs in a headless browser with a 1280×800 viewport, a per-task step budget of 15, and supports the actions click, type, select, scroll, drag, hover, wait. Each task is scored by executing its test suite and aggregating outcomes into Full/Partial/No completion. We keep this evaluator frozen and identical across all models and methods. Details of the execution environment, prompts, and failure handling are provided in Appendix A. We additionally validate evaluator robustness via expanded human–agent agreement (N=100 across multiple model families) and step-budget sensitivity analysis (5/10/15 steps) with failure-mode attribution (see Sec. 5.1). Training Details of VisRL. We use Qwen 2.5 14B Instruct Alibaba Cloud and Team (2024) as our base model. We first launch the SFT training to establish a cold-start checkpoint. After that, we launch the PPO learning with LoRA adapters (r=16r=16, α=32α=32 for general WebUI task), batch size 24, learning rate 2×10−72× 10^-7, and vLLM for off-policy generation with periodic adapter reloads. We use 4 H100-80 GB NVIDIA GPUs for training. We compare VisRL with two training variant, including pure SFT and PPO with vision agent. The baseline, PPO with vision agent, leverages the agent to score the WebUI screenshot with vision rubric (see Appendix C.7). We provide implementation details, including training hyperparameters, hardware resources, system configuration, and training infrastructure ( see Appendix C.1). In Appendix D.1, we show the training curve analysis. Training Data. For general WebUI task, we leverage the 6000 training set from webgen-bench Liu et al. (2024b) to do the SFT and PPO training. We de-duplicate the training data and also avoid using data that requires a login page first to facilitate our training process. For the 3D simulation task, we construct our own Three.js training dataset by scraping the official threejs.org examples. Using the files.json index, we crawl each demo and parse its HTML to extract metadata (title, description, tags), which we convert into instruction prompts. The resulting dataset covers a wide range of real-world Three.js patterns (e.g., animation, interactive controls, external assets, visual effects) and contains 1600+ prompts. Table 3: 3D simulation WebUI generation on WebUIProof. Same protocol as Table 2: we report build failure rate and UI-agent verified task outcomes (%), including No/Partial/Full completion rate, and compute Accuracy, counting build failures as No completion. Results compare eight proprietary (API) models and self-trained models, including Qwen2.5 14B Instruct and MIMO 7B under three training variants: SFT baseline, PPO w/ vision agent, and VisRL (ours). Model Source Model Name / Training Variant Build Failure Rate (%) Build Success – UI-Agent Verified Task Outcome (%) Accuracy (%) (Full + 0.5⋅·Partial) No Completion Partial Completion Full Completion Proprietary (API) Claude Sonnet 4 Anthropic (2025) 0.00 49.66 20.21 30.14 40.24 Qwen 3 Coder 480B Alibaba Cloud (2025) 1.43 51.37 17.12 29.11 37.67 Gemini 2.5 Pro DeepMind (2025) 0.00 54.79 18.49 26.71 35.96 Kimi K2 Moonshot (2025) 0.00 54.79 20.21 25.00 35.10 Llama 4 Maverick Meta (2025) 0.00 59.93 15.75 24.43 32.19 DeepSeek R1 DeepSeek (2025) 11.43 49.66 17.12 21.23 29.79 GPT 4.1 OpenAI (2025) 2.85 65.41 15.75 15.07 22.95 Llama 3 405B Meta (2024) 2.86 69.86 13.70 12.67 19.52 Self-Trained Qwen 2.5 14B Alibaba Cloud and Team (2024) SFT 0.00 55.68 22.94 20.97 32.44 + PPO w/ Vision agent 0.00 51.00 26.37 23.28 36.47 (+4.03) + VisRL (Ours) 0.00 52.24 19.31 27.73 37.38 (+4.94) MIMO 7B Xiaomi (2025) SFT 4.77 71.24 13.31 10.54 17.20 + PPO w/ Vision agent 2.86 (-1.91) 70.30 14.86 11.32 18.75 (+1.55) + VisRL (Ours) 0.00 (-4.77) 69.20 18.86 10.67 20.10 (+2.9) 4.2 Experimental Results Tables 2 and 3 report WebUI generation performance on WebUIProof under the same fixed, executable UI-agent evaluation protocol across eight commercial models. We report (i) build failure rate (cannot build/render), and (i) UI-agent verified task outcomes among build-successful generations (No/Partial/Full), summarized as Acc=(#Full+0.5⋅#Partial)/NAcc=(\# Full+0.5·\# Partial)/N with build failures counted as No. On General WebUI (Table 2), Claude Sonnet 4 performs best (46.10% acc.) but still fails to build in 30.37% of cases. The second-best models, Gemini 2.5 Pro (42.86% acc. and 32.21% build failure rate) and Qwen 3 Coder 480B (41.99% acc. and 33.05% build failure rate), form a tight cluster around 42% accuracy with similarly high build failures. On 3D simulation WebUIs (Table 3), Claude Sonnet 4 achieves 40.24% accuracy with 0% build failure, with Qwen 3 Coder 480B close behind at 37.67% (1.43% build failure), indicating that even frontier models leave substantial headroom on both of these interactive interfaces under executable UI-agent tests. We also report the results for compact open-sourced models trained under VisRL. On General WebUI, VisRL improves Qwen 2.5 14B Instruct from 26.08% to 41.80% accuracy while reducing build failures drop by 20.15%, and improves MIMO 7B from 23.80% to 33.43% while reducing build failures by 18.49% (Table 2). On 3D simulation WebUIs, VisRL improves Qwen 2.5 14B from 32.44% ro 37.38% (build failures remain 0%), and improves MIMO 7B from 17.12% to 20.10% while reducing build failures by 4.77 points (Table 3). These results illustrate that the UI-agent harness provides a learnable signal for training compact baselines. Appendix E.1 provides per-category accuracy breakdowns (Tables 10, 11, 12, 13). 5 Ablation Studies 5.1 Evaluator Reliability and Failure Modes Since WebUIProof relies on a UI-agent execution harness to run interaction tests, we quantify evaluator reliability by reporting (i) human-agent agreement on the Full/Partial/No labels across multiple model families, and (i) sensitivity to the evaluator step budget with failure attribution. ∙ Human–agent agreement across model families. We annotate N=100N=100 General WebUI samples and compare the UI agent’s labels against human judgments for three model families (Claude Sonnet 4, Gemini 2.5 Pro, and self-trained Qwen 2.5 14B with VisRL). Table 4 reports agreement rates and confusion matrices. Overall agreement is consistently above ∼ 80%, suggesting that the evaluator provides a reasonably stable signal across different generation styles. ∙ Step budget sensitivity and failure attribution. Evaluator errors can arise from limited interaction budget or brittle element grounding. To characterize this, we vary the per-task step budget (5/10/15) on the same N=100N=100 samples and attribute disagreements to concrete failure modes: timeout/step-limit, interaction failure (action executed but UI does not respond as expected), and element-locator failure (target cannot be grounded). Table 5 shows agreement increases with larger budgets while timeouts decrease, indicating that a meaningful portion of disagreement is due to resource/grounding limitations rather than systematic scoring bias. Full Agree. Disagree. Model Families Total Agree. N–N P–P Y–Y Adj. Far Claude Sonnet 4 86 18 30 38 13 1 Gemini 2.5 Pro 82 14 31 37 15 3 Qwen2.5 14B (VisRL) 85 18 32 35 10 5 Table 4: Human–agent agreement (N=100 on General WebUI). We report overall agreement rate, full agreement breakdown (N–N, P–P, Y–Y), and disagreement breakdown (Adjacent v.s. Far). "Adjacent" (Adj.) counts No↔ and Partial↔ confusions; "Far" counts No↔ confusions. Evaluator Errors # Step Total Agree. Timeout Interaction fail. Element Loc. fail. 5 80.0 6.7 10.0 3.3 10 84.0 3.3 6.7 6.7 15 86.7 0.0 4.3 9.0 Table 5: Step budget sensitivity of the UI-agent evaluator (N=100, General WebUI). Increasing the step budget improves human–agent agreement and reduces timeouts; remaining errors are primarily interaction and element-locator failures. 5.2 Analysis on Benchmark Quality ∙ Sensitivity to instruction complexity. To quantify how functional requirements scale with instruction difficulty, we define an instruction complexity index (ICI) that captures more than prompt length. ICI aggregates two factors, including Size S (number of required elements and tests), Interactivity I (interaction verbs/actions): ICI=0.5S+0.5IICI=0.5S+0.5I. While increasing the step, we increment the instruction with more requirements. The generation of each step is built sequentially; that is, the model will generate the code for the current step based on the code generated from prior steps. In Fig 5 (left side subgraph), we show the commercial models’ performance. When increasing the step from 1 to 9, the ICI score increases up to 68 (red dotted line), and the accuracy of the three frontier models gradually drops. On the right side subgraph, we show the accuracy of our self-trained model, including baseline SFT and VisRL (PPO with UI agent). The trend is similar to the frontier models. ∙ Test coverage rate of WebUIProof. We assess benchmark quality using TCR, the fraction of required UI elements exercised by the test suite. For each task, a UI element is counted as covered if it is referenced by any test case as an interaction target or assertion. We compare TCR between free-form instruction prompts collected from WebGen-Bench Liu et al. (2024b) and stuctured specs. from WebUIProof (ours). We report two metrics: model accuracy and build failure rate. High TCR paired with low accuracy indicates that the instruction provides dense, informative tests that remain challenging. Fig. 5 shows TCR increases from 30.71% (free-form) to 97.23% (structured spec.); the resulting drop in accuracy suggests that WebUIProof is more challenging. Figure 4: Sensitivity analysis of model robustness vs. instruction complexity. Commercial models exhibit superior accuracy retention (Claude Sonnet 4: 100% to 40%) compared to our trained models Qwen 2.5 14B (VisRL), which drops from 100% to 21.4% across 9 steps, with ICI scores increasing from 19.5 to 68.4 (red margin with dotted line). Figure 5: Benchmark quality analysis via test coverage. We compare free-form vs. structured specifications on General WebUI. Dashed lines indicate the average TCR. Structured specs. increase coverage (30.71%→ 97.23%) and yield a more discriminative evaluation. Table 6: VisRL learning-strategy ablations. The upper table shows that MTS and FGF improve UI stability/efficiency; Lower table shows that multi-temperature outperforms single-temperature under the same best-of-N. Learning strategy Build fail. (%) Acc. (%) VisRL (Ours) 36.10 41.80 w/o MTS (τ=0.7τ=0.7) 46.31 36.76 w/o FGF 49.53 39.00 w/o MTS & FGF 50.32 32.12 Best-of-N τ Acc. (%) N=3N=3 0.3,0.45,0.7\0.3,0.45,0.7\ 41.80 N=2N=2 0.3,0.7\0.3,0.7\ 41.53 N=2N=2 0.30.3 39.00 N=1N=1 0.70.7 36.76 N=1N=1 0.30.3 35.28 Figure 6: UI-agent execution examples on WebUIProof. Top: general WebUIs (e.g., interactive dashboards and widgets). Bottom: 3D simulation WebUIs (e.g., physics-driven interactive scenes). In all cases, the UI agent executes interaction tests by grounding to DOM/UI elements and performing actions (click/type/drag) to validate functional behavior. 5.3 Analysis of Learning Strategies in VisRL We study learning strategies to improve rollout efficiency in VisRL. Table 6 provides two analyses: (i) the effect of multi-temperature candidate sampling (MTS) and fast-gate filtering (FGF). (i) the effect of sampling budget and temperature. The upper sub-table shows that removing either MTS or FGF increases build failures and reduces overall accuracy, showing both are important for stable and efficient RL training. The lower sub-table analyzes the budget of best-of-N sampling under different sampling numbers (N=1,2,3N=1,2,3), with multi-or single temperature (τ indicates temperature set). For the same N, multi-temperature achieves higher accuracy than the same temperature, indicating that the gain comes from diversity, not just more sampling attempts. 5.4 Visualization of Tasks and UI-Agent Execution Traces Figure 6 visualizes six WebUIProof tasks and the corresponding UI-agent execution traces. The top row shows general WebUI applications (e.g., dashboards and interactive widgets), while the bottom row shows 3D simulation interfaces (e.g., object collisions). The UI agent localizes task-relevant elements by cropping their bounding regions from the rendered page and marking them with colored boxes and index labels, making the executed interaction targets explicit. Additional visualization of UI-agent execution traces and task samples for both domains is provided in Appendix E.2. 6 Conclusion We introduce WebUIProof, an execution-oriented benchmark for WebUI code generation that pairs structured specifications with executable interaction tests spanning general WebUIs and 3D simulation interfaces. WebUIProof includes a reproducible UI-agent harness for automated evaluation, enabling functional verification beyond build success or static visual checks. Using this protocol, we benchmark eight frontier models and find that interactive WebUI generation remains challenging in both domains, with substantial headroom on 3D tasks. We also show that the same harness can provide outcome-level training signals by including VisRL, an execution-grounded RL appraoch that optimizes compact models with composite rewards from build/runtime execution, rendered visual quality, and UI-agent test success. VisRL improves both functional completion and reliability compared to SFT and vision-only baselines. Our ablation suggests that performance remains bottlenecked by complex instruction and multi-step requirements, motivating future work on reward assignment, richer executable signals, and robust evaluation agents for dynamic 3D WebUI. Overall, our work shed light on the importance of outcome-grounded, executable evaluation and training signals for reliable code generation beyond static correctness and visual-only metrics. References Alibaba Cloud and Team (2024) Q. T. Alibaba Cloud and Q. Team Qwen2.5: a party of foundation models. External Links: Link Cited by: §4.1, §4.1, Table 2, Table 3. Alibaba Cloud (2025) Q. T. Alibaba Cloud Qwen 3 Coder: agentic code-generation model. Note: https://github.com/QwenLM/Qwen3-CoderAccessed 2025-10-29 Cited by: Table 10, Table 12, §4.1, Table 2, Table 3. Anthropic (2025) Anthropic Claude Sonnet 4: advanced reasoning & coding model. Note: https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude/sonnet-4Accessed 2025-10-29 Cited by: Table 10, Table 12, §4.1, Table 2, Table 3. DeepMind (2025) G. DeepMind Gemini 2.5 Pro: multimodal reasoning & coding model. Note: https://ai.google.dev/gemini-api/docs/modelsAccessed 2025-10-29 Cited by: Table 10, Table 12, §4.1, Table 2, Table 3. DeepSeek (2025) DeepSeek DeepSeek-R1: reasoning-optimized open-weight llm. Note: https://github.com/deepseek-ai/DeepSeek-R1Accessed 2025-10-29 Cited by: Table 10, Table 12, §4.1, Table 2, Table 3. Halluminate and Skyvern (2025) Halluminate and Skyvern WebBench: ai web browsing agent benchmark. Note: https://webbench.ai/ Cited by: §2. He et al. (2024a) H. He, W. Yao, K. Ma, W. Yu, Y. Dai, H. Zhang, Z. Lan, and D. Yu WebVoyager: building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Vol. 1, p. 6864–6890. Note: Also arXiv preprint arXiv:2401.13919 External Links: Link Cited by: §A.2, §2. He et al. (2024b) H. He, Y. Tang, Y. Zheng, H. Zhang, Y. Huang, Y. Wang, J. Chen, X. Wang, D. S. Zhou, and M. Li Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §2. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring coding challenge competence with apps. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1. Jimenez et al. (2023) C. Jimenez, T. Nguyen, X. Wang, Q. Le, and et al. SWE-bench: can language models resolve real-world github issues?. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2. Li et al. (2022) Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, and et al. Competition-level code generation with alphacode. Science 378 (6624), p. 1092–1097. Cited by: §1. Liu et al. (2024a) J. Liu, Y. Song, B. Y. Lin, W. Lam, G. Neubig, Y. Li, and X. Yue VisualWebBench: how far have multimodal llms evolved in web page understanding and grounding?. External Links: 2404.05955 Cited by: §2. Liu et al. (2024b) Y. Liu, C. Huang, W. Li, J. Wang, and H. Xu WebGen-bench: benchmarking webui code generation with structured test cases. arXiv preprint arXiv:2408.01839. Cited by: §B.1, §1, §2, Table 1, §3.1, §4.1, §5.2. Meta (2024) Meta Llama 3.1 405B: large-scale open foundation model. Note: https://developer.nvidia.com/blog/creating-synthetic-data-using-llama-3-1-405b/Accessed 2025-10-29 Cited by: Table 10, Table 12, §4.1, Table 2, Table 3. Meta (2025) Meta Llama 4 Maverick: next-gen multimodal/multilingual foundation model. Note: https://llama.com/models/llama-4/Accessed 2025-10-29 Cited by: Table 10, Table 12, §4.1, Table 2, Table 3. Moonshot (2025) Moonshot Kimi K2: trillion-parameter moe model for agentic tasks. Note: https://moonshotai.github.io/Kimi-K2/Accessed 2025-10-29 Cited by: Table 10, Table 12, §4.1, Table 2, Table 3. OpenAI (2025) OpenAI GPT 4.1: enhanced context & instruction-following model. Note: https://platform.openai.com/docs/models/gpt-4.1Accessed 2025-10-29 Cited by: Table 10, Table 12, §4.1, Table 2, Table 3. Raman et al. (2023) P. Raman, Z. Wang, Y. Deng, K. Narasimhan, and A. Beirami BrowserGym: a browser interaction simulation environment for training and evaluating web agents. arXiv preprint arXiv:2311.05657. External Links: Link Cited by: §2. Team (2024) E. Team ReactEval: a benchmark for evaluating llms on react front-end development. Note: https://e2b.dev/blog/reacteval-building-llm-benchmark-for-frontendAccessed: 2025-10-05 Cited by: §1. Xiao et al. (2025) J. Xiao, M. Wang, M. H. Lam, Y. Wan, J. Liu, Y. Huo, and M. R. Lyu DesignBench: a comprehensive benchmark for mllm-based front-end code generation. arXiv preprint arXiv:2506.06251. Note: multi-framework, multi-task front-end benchmark Cited by: §1, §2, Table 1. Xiaomi (2025) L. Xiaomi MiMo: unlocking the reasoning potential of language model – from pretraining to posttraining. External Links: 2505.07608, Link Cited by: §4.1, Table 2, Table 3. Yang et al. (2024a) A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, Y. Fan, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Y. Liu, Z. Cui, Z. Zhang, and Z. Fan Qwen2 technical report. arXiv preprint arXiv:2407.10671. Cited by: §4.1. Yang et al. (2024b) Y. Yang, Y. Sun, J. Wang, X. Guo, H. Zhou, J. Chen, and D. He SWE-gym: training software engineering agents and verifiers. arXiv preprint arXiv:2412.21139. Cited by: §2. Zhang et al. (2025) C. Zhang, Y. Li, C. Xu, J. Liu, A. Liu, S. Hu, D. Wu, G. Huang, K. Li, Q. Yi, et al. ArtifactsBench: bridging the visual-interactive gap in llm code generation evaluation. arXiv preprint arXiv:2507.04952. Cited by: §1. Zhang et al. (2024) R. Zhang, Y. Chen, K. Zhao, W. Zhou, and M. Li StackEval: benchmarking llms in coding assistance. arXiv preprint arXiv:2412.05288. Cited by: §2. Zheng et al. (2024) K. Zheng, R. Fang, W. Qian, X. Wang, T. Zhang, S. Zhao, Y. Chen, D. Song, H. Xu, and et al. WebDev-arena: benchmarking llm agents for web development. arXiv preprint arXiv:2407.15066. Cited by: §B.1, §1, §2, Table 1, §3.1. Zheng et al. (2023) Q. Zheng, R. Wang, Y. Xu, Y. Chen, S. Zhang, D. He, and Z. Liu CodeGeeX: a pre-trained model for code generation with multilingual evaluations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §1. Zhou et al. (2023) Y. Zhou, Y. Deng, Y. Chen, T. Zhang, T. Gao, K. Narasimhan, A. Beirami, and M. Li WebArena: a realistic web environment for building autonomous agents. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §2. Zhu et al. (2025) H. Zhu, Y. Zhang, B. Zhao, J. Ding, S. Liu, T. Liu, D. Wang, Y. Liu, and Z. Li FrontendBench: a benchmark for evaluating llms on front-end development via automatic evaluation. arXiv preprint arXiv:2506.13832. Note: 148 prompt–test pairs, automatic sandboxed evaluation Cited by: §2, Table 1. Appendices Appendix Outline A. Reproducibility and Evaluation Protocol • A.1 Execution Environment (web framework and build process) • A.2 UI-agent Evaluator Setup (action space, step budget) • A.3 System Prompt for Code Generation (general + 3D WebUI) • A.4 Evaluation Metrics and Failure Handling B. WebUIProof Benchmark Construction • B.1 Data sources and Task Scope (general + 3D WebUI) • B.2 WebUI Categories and Test Case Distribution • B.3 Data Collection Pipeline and Benchmark Collection Prompt • B.4 Sample of Structured specification C. VisRL: Training Details • C.1 Implementation Details (training hyperparameters configuration, resource usage, training infrastructure) • C.2 Sequence-level Rollout and Probability • C.3 Reward definition • C.4 Additional learning strategies (multi-temperature sampling, fast gate filtering) • C.5 Pseudo Algorithm of VisRL • C.6 UI Agent Task Execution Process • C.7 Vision Agent - Evaluation Rubric D. Reward and Hyperparameter Analyses • D.1 Training Curve for Different PPO Strategy • D.2 Reward Decomposition Analysis • D.3 Reward Weight Analysis - α Sensitivity and Trade-off Discussion E. Extended Results and Diagnostics • E.1 Additional Experiments (per-test category results of general WebUI and 3D WebUI) • E.2 Visualization of UI Agent Execution Traces • E.3 More Visualization for general WebUI and 3D WebUI • E.4 Build Failure Taxonomy (error categories and examples) Appendix A Reproducibility and Evaluation Protocol A.1 Execution Environment In the main paper Sec. 4, we introduce the evaluation setup for WebUIProof, including model details, UI-agent evaluator setup. Here, we provide additional detail on the execution environment for evaluation. Web Framework and Build Process We use React, Next.js, and Node.js to build the web app during the evaluation. We transform the React codes generated from LLMs into a fully functional Next.js web application via a specific build process: (1.) JSON code files parsing: At the beginning, the LLM generates codes and saves them in the JSON files. We extract TSX/React code from the JSON payload (e.g., payload.get("metadata", ).get("code")) Repository Structure repo/ ├─ next.config.mjs ├─ next.env.d.ts ├─ tsconfig.json ├─ tailwind.config.js ├─ style/ │ └─ globals.css ├─ pages/ │ ├─ _app.tsx │ ├─ index.tsx │ └─ 404.tsx └─ scripts/ └─ after-export.cjs Figure 7: Next.js project structure (2.) Code Sanitization: we find valid TSX code with export default using a regex search. (3) Web Project Scaffolding: we create a completely new Next.js project with npm build commands, such as npm init -y for project initialization, npm install react@18.3.1 next@15.5.4 for react/next.js installation, npm install -D typescript @types/react @types/node tailwindcss@3 postcss to setup libraries for UI style. (4) Files Generation: After all the dependency installation, we create a repo structure for web app with several files (as shown in Fig. 7). The Next.js web app includes one file for dynamically inserting the LLM-generated code (index.tsx), and several fixed structure files with pre-built templates to ensure static export for deployment (5) Build the Repo: after constructing the repo structure, we run npm run build, which triggers Next.js static export. The build script will finally output an out/ folder with index.html, 404.html, _next/ that contain all the Next.js static build assets (JavaScript bundles, compiled css files, build manifest files) for us to directly render in the browser using the command npx serve out. A.2 UI-agent Evaluator Setup Our evaluation pipeline is built upon the existing UI agent evaluator Webvoyager He et al. (2024a). a vision-language web agent that automate browser interaction through a combination of visual percpetion and DOM analysis. The web agent uses Selenium as a Webdriver with our own configured Chrome/Chromium browser (Chrome driver version 119.0.6045.105). The agent operates through a number of iteration steps, where each step follows a thinking loop: (1.) Selenium takes UI screenshots and parses the DOM interactive web elements via executable JavaScript code snippets. (2.) LLMs reason from the visual information in the UI screenshots and provide "Thought" and "Action" responses. The actions are predefined under an action space (e.g., click/type/scroll/select/hover…) (see Table 7). LLMs will choose one action for each step, and the action is executed by Selenium. The UI state will update for the next iteration, and at the final step, we ask LLMs to assess task completion, including three answer options: "FULL", "NO", and "PARTIAL" completion. In our experiment, we set the maximum iteration step as 15. (Mentioned in Sec. 3.1 as well) For visual understanding, the agent supports multiple vision-language models (VLMs), either via the API (e.g., GPT-4O from OpenAI), HuggingFace endpoints, or open-source VLMs hosted locally. In our evaluation, we chose to use the Qwen2.5-VL-72B66 6 https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct, a powerful open-source VLM. Table 7: Action space of UI agent Action Type Detail Click Click any web element identified by its numerical label (e.g., Click [0]) Type Clear textbox and input content (automatically hits ENTER) (e.g., Type [0]) Scroll Scroll specific element or entire window (e,g, Scroll WINDOW up / down) Hover Hover for tooltips/effects, with precise coordinate positioning for charts (e.g., Hover [0:75,50] for 75% from left, 50% from top) Wait Pause for 5 seconds (for loading processes) GoBack Navigate to previous webpage Answer Provide final task result (terminates interaction) (e.g., FULL/PARTIAL/NO) A.3 System Prompt for Code Generation (General + 3D WebUI) To ensure reproducibility and make our evaluation pipeline fully auditable, we disclose the exact system prompts used for our two task families: (i) General WebUI generation and (i) 3D WebUI simulation (As shown in Fig. 8 and 9). These two settings require different code formats and runtime assumptions, and we therefore use task-specific prompts to standardize model outputs. For General WebUI tasks (Fig. 8), models are instructed to generate a single-file React/TypeScript component compatible with a Next.js (pages router) scaffold, with strict output constraints (default export only, no surrounding text) and styling constraints (Tailwind classes without arbitrary values). In contrast, 3D simulation tasks (Fig. 9) require a self-contained HTML document that renders an interactive Three.js scene. The prompt fixes the Three.js version and enforces verified CDN imports to avoid dependency drift, and additionally emphasizes runtime robustness (error handling, interactivity checks) and output formatting (start from <!DOCTYPE html> and return HTML only). By explicitly specifying the target artifact (pages/index.tsx React code vs. standalone HTML), these prompts reduce formatting variance across models and ensure that downstream building, rendering, and UI-agent testing are applied consistently. General WebUI Generation You are an expert frontend React engineer who is also a great UI/UX designer. Follow the instructions carefully. Think carefully step by step. • Create a React component for whatever the user asked you to create and make sure it can run by itself by using a default export. • Make sure the React app is interactive and functional by creating state when needed and having no required props. • If you use any imports from React like useState or useEffect, import them directly. Use TypeScript as the language for the React component. • Use Tailwind classes for styling. DO NOT USE ARBITRARY VALUES (e.g., h-[600px]). Use a consistent color palette. • Specify and install ALL additional dependencies. • Include all necessary code in one file. Do not touch project dependency files like package.json, package-lock.json, requirements.txt, etc. • Use Tailwind margin and padding classes so components are spaced nicely. • ONLY return the full React code starting with imports, nothing else. DO NOT start with ‘typescript, ‘javascript, ‘tsx, or ‘. • ONLY if the user asks for a dashboard/graph/chart, you may import recharts (e.g., import LineChart, XAxis, ... from "recharts"). You may also use shadcn/ui charts (e.g., ChartConfig, ChartContainer from @/components/ui/chart). • For placeholder images, use <div className="bg-gray-200 border-2 border-dashed rounded-xl w-16 h-16" />. You can use the following templates: nextjs-developer: "A Next.js 13+ app that reloads automatically. Using the pages router. All components must be included one file.". File: pages/index.tsx. Dependencies installed: nextjs@15.5.4, typescript, @types/node, @types/react, @types/react-dom, postcss, tailwindcss, shadcn. Port: 3000. Figure 8: System prompt for General WebUI task. Models generate React code generation with strict styling and output constraints. 3D WebUI Simulation You are an expert web developer specializing in creating interactive Three.js HTML simulations and visualizations. Follow these instructions carefully to ensure proper rendering: • ALWAYS use the same Three.js version consistently across all imports. • Use only verified CDN URLs. For example: ⬇ <script src="https://cdn.jsdelivr.net/npm/three@0.140.0/build/three.min.js"></script> <script src="https://cdn.jsdelivr.net/npm/three@0.140.0/examples/js/controls/ OrbitControls.js"></script> • NEVER mix different Three.js versions in the same file. • ALWAYS include comprehensive error handling with JavaScript loader.load(). OUTPUT FORMAT: • Return ONLY the complete HTML code. • Do not include explanatory text before or after. • Start directly with <!DOCTYPE html>. • Ensure the simulation is functional and interactive. • Test all interactive elements work properly. Figure 9: System prompt for Three.js HTML simulation generation. A.4 Evaluation Metrics and Failure Handling For each task, we categorize the outcome into four mutually exclusive cases: (i) Full Success, (i) Partial Success, (i) No Success, and (iv) Build Failure. Build Failure covers build-time or launch-time failures (e.g., compilation errors, runtime crashes, or timeouts) where the generated project cannot be executed for UI-agent testing. Importantly, we average Full/Partial/No Success over all tasks by treating build failure with no credit. As a result, the corresponding rates satisfy Full+Partial+No Success+Build failure≈100%Full+Partial+No Success+Build failure≈ 100\%. Appendix B WebUIProof: Benchmark Construction B.1 Data Source and Task Scope General WebUI In the main paper, Sec. 3, we mention that the data collection process for structured specification requires seed pages. Here, we describe how we collect the seed pages. For the general WebUI task, we collect seed instructions mainly from two data sources: WebGen Bench Liu et al. (2024b) and WebDev-Arena Zheng et al. (2024). We use 101 instructions from the WebGenBench testset and 48 instructions from WebDev-Arena-10k as our seed instructions and generate seed webpages using DeepSeek R1 model. This collection encompasses five main application domains, including creative media, business data, game sports, social interaction, and learning style. For example, social media platforms (reddit clone, social profile like Twitter, dating app), interactive gaming platforms (chess applications, 2048 games, wheel of fortune), chat interface (mail box with chat function) sports analytics systems (basketball betting analysis, baseball simulation), business applications (credit repair lead generation, clinical office websites), data visualization tools (regulatory policy dashboards, stock report generators), and complex strategy games (text-based browser games with resource management). Our collected instructions emphasize both frontend user experience elements and backend functionality, requiring implementations that support real-time interactions, data visualization via charts and dashboards, user authentication systems, and responsive design patterns compatible with modern web development frameworks. We show more sample visualization in E.3 3D WebUI Simulation The 3D simulation test instruction dataset features advanced Three.js-based interactive visualizations and simulations derived from Three.js opensource website examples and community contributions. This specialized collection contains 70 sophisticated 3D programming challenges that span astronomical simulations (particle galaxy formations with spiral arms, black hole visualizations with event horizon effects), geometric animations (tesseract rotations with collision detection, merry-go-round mechanics with adjustable parameters), physics-based demonstrations (bouncing ball dynamics within complex 3D shapes), and interactive exploration systems that allow users to navigate through immersive 3D environments using JavaScript controls. Each instruction demands implementation of complex mathematical concepts including 3D transformations, particle systems, collision detection algorithms, and real-time rendering optimizations, while maintaining smooth 60fps performance and responsive user interaction capabilities. The tasks require understanding of WebGL fundamentals, Three.js scene graph management, camera controls, lighting systems, material properties, and animation loops, with many instructions specifically requesting single HTML file implementations that demonstrate advanced computer graphics concepts through browser-based 3D experiences. These instructions evaluate both technical proficiency in 3D programming and creative problem-solving abilities in translating complex visual concepts into interactive web-based simulations. We show more sample visualization in E.3 B.2 WebUI Categories and Test Case Distribution In Fig. 10, we provide the category distribution of two distinct WebUI tasks in WebUIProof: General WebUI applications (149 samples) and 3D Simulation applications (70 samples). General WebUI is dominated by Creative Media applications (41.6%), followed by Data Management (20.1%) and Social Interaction (16.1%). The remaining categories include Learning & Lifestyle (12.8%) and Games & Sports (9.4%). 3D Simulation shows a more balanced distribution with Classical Mechanics leading at 42.9%, Fluid Waves at 24.3%, and equal representation of Astro Particles and Dynamics Chaos (both 12.9%). Structural Physics (4.3%) and Game Physics (2.9%) represent smaller segments. In Fig. 11, we further provide the test cases distribution on two main tasks, we categorize the test cases into three categories, including functionality testing, data display testing, and design validation testing. In the later section E.1, we show additional experimental results on category accuracy for each test case across multiple proprietary models. Figure 10: WebUI Category Distribution for WebUIProof. Figure 11: Test Case Distribution for WebUIProof. B.3 Data Collection Pipeline and Detail Prompt Although we have already described how we collect the data in the main paper, we would like to provide more detail, including the data collection flow, and the system prompt we provide to UI agent for WebUI exploration. As Fig 12 shows, the UI agent receives the webpages screenshots and html file as input at the first step. Then it summarizes the UI functionality, and provides the basic test cases for later exploration. At the second step, it starts the thinking loop, and each loop it will provide the UI observation, newly discovered element, and test case recommendations based on the updated UI state at every iteration. Figure 12: Benchmark Collection Pipeline. Benchmark Collection Prompt In Fig. 13, we show the detail prompt that we provide to WebUI agent to let them explore the seed webpages. As the prompt shows, we give the agent three inputs, including website seed instruction, screenshots of webpages, and an HTML file. Model will provide a JSON file with certain schema. WebUI Test Generation Prompt You are an expert test engineer. Follow the instructions carefully. Input context: • Website generation instructions: website_seed_instructions • N screenshots showing different views and interactions • HTML structure of the webpage Task: Given the instructions, screenshots, and HTML structure, analyze the website using both visual evidence and DOM structure. • First, summarize the website with: – a short name (e.g., Stock Report Generator) – a one-sentence description of the overall look and purpose – a category from: Content Presentation / User Interaction / Data Management / Animation Simulation / Game Design • Then, write two detailed descriptions: – How the website looks: concrete visual layout and design elements (e.g., charts, cards, tables, navigation). – What functions it has: core interactions and workflows (e.g., search, filter, submit, download). Generate test cases across three dimensions: • Functionality testing (≥ 2 cases): core user interactions • Data display testing (≥ 1 case): verify displayed values and updates • Design validation testing (≥ 1 case): confirm key UI design requirements Output format (MUST follow): • Return a single JSON object (no markdown, no extra text). • Use the exact schema fields: website_summary, test_cases, element_requirement. Figure 13: Multimodal prompt (instructions + screenshots + HTML) used for WebUI test-case generation. B.4 Sample of Structured Specification in WebUIProof We show two samples of structured specifications in our WebUIProof benchmark (the first one for General WebUI and the second one for 3D WebUI). We totally have 219 structured specifications. During the evaluation, we provide the model with the specification and let it follow the structured instructions to generate WebUI code. ⬇ 1 2 "website_summary": 3 "name": "Stock Report Generator", 4 5 "description": "The website displays dynamic stock information and offers analytical tools with a clean, professional design.", 6 7 "category": "Data Management", 8 9 "appearance": "The website has a clear and organized layout with distinct sections. It features a navigation header with \"Stock Report Generator\" prominently displayed. Below this, there’s a section for stock information which includes a live price update, volume, and market cap. The \"Historical Data\" section showcases price and volume over time with a line graph. Additionally, the \"Financial Metrics\" and \"Financial Summary\" tables present key financial indicators and revenue details in a tabular format. The \"Ownership Distribution\" offers a pie chart for visualizing investor distribution, while the \"Stock Performance\" shows a line graph of price trends. The footer includes a customizable report section with checkboxes for content selection (e.g., Overview, Financials) and a \"Download Report\" button. The background is white, and various components utilize navy blue for text and buttons, ensuring a consistent color scheme and modern appearance." 10 , 11 "test_cases": 12 "functionality_testing": [ 13 14 "id": "TC001", 15 "name": "Download Button Test", 16 "objective": "To verify that the Download Button triggers the report download process, providing complete standardized or customized report data.", 17 "coverage_element": "Download Button", 18 "preconditions": "The page is loaded, and a report is ready to be downloaded.", 19 "expected_results": "The downloading process visually starts, leading to successfully downloading a report file reflecting data on the UI.", 20 "procedure": [ 21 "Click on the Download Button and confirm a download concern initiating accordingly." 22 ] 23 // ... (the rest of test cases)] 24 "element_requirement": 25 "1. Company Selector : Button for selecting different companies, currently set to \"Apple Inc.\"", 26 "2. Summary Tab : Button that likely switches the display to a summary view", 27 "3. Download Button : Button likely for downloading the current or customized report", 28 "4. Stock Information : Card displaying current price, volume, and market cap data for Apple Inc.", 29 "5. Historical Data Card : Contains a historical data chart with price and volume over time", 30 "6. Financial Metrics Card : Displays key financial indicators", 31 "7. Financial Summary : Card with revenue and income data" 32 33 ⬇ 1 2 "website_summary": 3 "name": "3D Spiral Galaxy Simulation", 4 5 "description": "The website features a dark-themed background to emphasize the 3D spiral galaxy simulation.", 6 7 "category": "Astro Particles", 8 9 "appearance": "At the top of the page, there is a centered text informing the user how to interact with the simulation: \"Spiral Galaxy Simulation - Use mouse to rotate, scroll to zoom\". The galaxy is displayed in full screen within a canvas, creating a visually immersive experience with thousands of small, colored particles representing stars forming spiral arms around a central core. The website allows users to interact with a 3D spiral galaxy simulation using mouse controls. Users can: - Rotate the view of the galaxy by clicking and dragging the mouse. - Zoom in and out using the mouse scroll wheel. - Experience a visualization of spiral galaxy formation with realistic star positioning and colors." 10 , 11 "test_cases": 12 "functionality_testing": [ 13 14 "id": "TC001", 15 "name": "Rotate Galaxy View", 16 "objective": "To verify that the user can rotate the galaxy view using the mouse.", 17 "preconditions": "The website is loaded and the galaxy is visible.", 18 "expected_results": "The galaxy rotates smoothly and interactively in real-time with the mouse movements, allowing the user to explore different angles.", 19 "procedure": [ 20 "Open the website in a supported web browser.", 21 "Click and hold the left mouse button on the galaxy display.", 22 "Drag the mouse to the left and right." 23 ] 24 25 // ... (the rest of test cases) 26 ] 27 28 "element_requirement": 29 "1. Canvas Element : Full-screen 3D rendering canvas displaying the spiral galaxy simulation with thousands of star particles", 30 "2. Instructional Text : Centered white text at the top reading \"Spiral Galaxy Simulation - Use mouse to rotate, scroll to zoom\"", 31 "3. Mouse Rotation Control : Interactive mouse click-and-drag functionality for rotating the galaxy view in real-time", 32 "4. Mouse Scroll Zoom Control : Mouse wheel scroll functionality for zooming in and out of the galaxy simulation", 33 "5. Star Particles : Thousands of colored particles representing stars with varying colors (blue-white, yellow-white, redder) and sizes", 34 "6. Spiral Galaxy Structure : 3D particle system forming realistic spiral arms around a central galactic core", 35 "7. Real-time Animation : Smooth, interactive rendering system maintaining 60fps performance during user interactions", 36 "8. Camera Controls : Three.js camera system enabling 360-degree exploration of the galaxy from different angles", 37 "9. Particle Color System : Randomized star coloring system creating realistic stellar color distribution throughout the galaxy" 38 39 Appendix C VisRL: Training Detail We provide algorithm details and implementation-level details for our sequence-level PPO training, including reward computation, candidate selection, and efficiency heuristics. C.1 Implementation Detail ∙ Cold Start from Supervised Fine-Tuning (SFT) We finetune Qwen2.5-14B-Instruct on a curated dataset of 6000 high-quality webUI generation samples. We use the 6k training set from WebGen-Bench as our seed instruction and generate code snippets using DeepSeek-R1. Our model is trained with distributed training across 4 NVIDIA H100 GPUs (80 GB each) with full parameter tuning. To handle long code sequences, we employ dynamic RoPE scaling with a factor of 2.0, extend the context window to 65536 tokens, and apply a chunking strategy that segments sequences into 1792-token windows with 128-token overlap. The model is trained under 8 epochs with a learning rate 8×10−68× 10^-6 and an effective batch size of 256 (per-device batch size of 1 with 64 gradient accumulation steps across 4 GPUs). Other training hyperparameters, hardware configuration are shown in Table 8. We apply chat template formatting where only assistant responses are supervised (prompt tokens masked with -100). Training converges in approximately 8 hours after eight training epochs. Table 8: Supervised Fine-Tuning Configuration and Resource Usage Training Hyperparameters Base Model Qwen2.5-14B-Instruct Training Samples 6,000 Train/Val Split 95% / 5% Epochs 8 Learning Rate 8×10−68× 10^-6 LR Schedule Cosine (5% warmup) Optimizer AdamW Batch Size (per-device) 1 Gradient Accumulation 64 Effective Batch Size 256 Weight Decay 0.07 Gradient Clipping 1.0 Max Sequence Length 65,536 Chunk Length 1,792 Chunk Overlap 128 RoPE Scaling Dynamic (2.0×) Save Frequency Every 100 steps Hardware & Resources GPUs 4× H100 (80GB) per GPU Memory Usage ∼ 60GB System RAM 256GB Storage 1TB Training Time ∼ 8 hours System Configuration Training Framework HuggingFace TRL Precision BF16 mixed Parallelization DeepSpeed ZeRO-2 Attention Flash Attention 2 Gradient Ckpt. Enabled Distributed Training DDP (4 GPUs) Logging WandB ∙ Training Curve of SFT In Fig. 14, we show the training curve of SFT for general WebUI. We finetune a Qwen 2.5 VL Instruct with the hyperparameter mentioned in Table 8. Figure 14: Visualization of SFT Training Curve ∙ VisRL Training and Implementaiton Detail After using SFT to train a cold-starting model, we start to use PPO with a hybrid on/off-policy approach that leverages vLLM for fast generation while maintaining near-on-policy learning through frequent adapter reloading. Unlike prior SFT training, which distributes the model across 4 GPUs, we allocate 4 H100 GPUs: GPU-0 fully loads the model to serve as the vLLM inference server (generating code snippets), and other three GPUs run distributed PPO optimization via DeepSpeed ZeRO-2 for policy learning. Due to the memory constraint, we apply LoRA adapters to all attention and MLP projection layers for PPO tuning, where we set the rank-8 and α-16 for general WebUI task. Since the 3D Simulation requires a higher learning capacity, we further set a larger rank-128 with alphaalpha-256 for the 3D simulation task. To make the model loaded on GPU-0 synchronize with the on-policy model, we reload the latest adapter every training step to minimize off-policy lag. Training proceeds for 100 steps with a conservative learning rate of 2×10−72× 10^-7, batch size is 8 per device (effective batch of 24 across 3 GPUs). In Fig. 15, we illustrate the training infrastructure of VisRL. In addition, we show the detailed training hyperparameters of VisRL in Table 9. Training completes in approximately 24-48 hours with checkpoints saved every step for reproducibility. In the later Sec. D.1, we provide the training curve analysis for different PPO strategies. Table 9: PPO Training Configuration and Resource Usage Training Hyperparameters Base Model SFT Qwen2.5-14B Training Steps 100 steps LoRA Rank (General WebUI) 8 LoRA Alpha (General WebUI) 16 Trainable Params (General WebUI) ∼ 4.2M (0.03%) LoRA Rank (3D WebUI) 128 LoRA Alpha (3D WebUI) 256 Trainable Params (3D WebUI) ∼ 598M (4.3%) LoRA Dropout 0.05 LoRA Targets q,k,v,o,gate,up,down Learning Rate 2×10−72× 10^-7 Batch Size (per-device) 8 Effective Batch Size 24 (3 GPUs) Mini-batch Size 8 PPO Clip Range 0.05 Entropy Coefficient 0.005 Value Loss Coefficient 1.0 Gradient Clipping 1.0 Advantage Clipping ± 2.0 Reward Normalization Enabled Reward Components Vision Weight (αv _v) 0.6 UI-Agent Weight (αu _u) 0.3 Analyzer Weight (αm _m) 0.1 Invalid Penalty −0.2-0.2 External Image Penalty −0.05-0.05 Keyword Bonus 0.20 Auto-Fix Bonus 0.3 Sampling Strategy Num Candidates 3 (best-of-3) Keep Top-K K=1 Generation Temp. [0.7, 0.45, 0.3] Max Gen Tokens 7,000 Max Context Tokens 2,048 Hardware & Resources Total GPUs 4× H100 (80GB) Training GPUs 3× H100 (GPUs 1-3) Inference GPU 1× H100 (GPU 0) GPU Memory (training) ∼ 70GB/device GPU Memory (vLLM) ∼ 75GB System RAM 256GB Storage 1TB Training Time ∼ 24-48 hours Time per Step 15-30 minutes System Configuration RL Framework TRL + Custom PPO Inference Engine vLLM (version 0.6+) Precision BF16 mixed Parallelization DeepSpeed ZeRO-2 Distributed Strategy DDP (3 GPUs) Adapter Method LoRA Adapter Reload Freq. Every 1 step Timeouts & Limits for Execution Harness npm Install 300s Next.js Build 360s Dev Server Load 60s vLLM Request 600s Step Timeout 7,200s (2 hours) ∙ Training Infrastructure of VisRL We provide the implementation flow of the training infrastructure to ensure reproducibility in Fig. 15. Figure 15: Training Infrastructure of VisRL. C.2 Sequence-Level Rollout and Probability Given a prompt p, the policy generates a complete code sequence c=ctt=1Tc=\c_t\_t=1^T. We compute sequence log-probability via the chain rule: logπθ(c∣p)=∑t=1Tlogπθ(ct∣c<t,p). _θ(c p)\;=\; _t=1^T _θ(c_t c_<t,p). (1) The PPO importance ratio is computed at the sequence level: ρ(θ)=exp(logπθ(c∣p)−logπθold(c∣p)).ρ(θ)= \! ( _θ(c p)- _ _old(c p) ). (2) Advantage and Loss Terms Because the environment returns a single outcome reward after execution, we use a prompt-conditioned baseline Vϕ(c,p)V_φ(c,p) and compute one advantage per sequence: A(p,c)=r(c,p)−Vϕ(c,p).A(p,c)=r(c,p)-V_φ(c,p). (3) We normalize advantages within each batch ℬB: A^(c,p)=A(c,p)−μℬσℬ+ϵnorm. A(c,p)= A(c,p)- _B _B+ _norm. (4) The clipped PPO objective is applied once per sequence: ℒCLIP(θ)=(c,p)∼ℬ[min(ρ(θ)A^(c,p),clip(ρ(θ),1−ϵ,1+ϵ)A^(c,p))].L^CLIP(θ)=E_(c,p) [ (ρ(θ) A(c,p),\;clip(ρ(θ),1-ε,1+ε) A(c,p) ) ]. (5) We optimize the standard combined loss: ℒ(θ,ϕ)=−ℒCLIP(θ)+cv(c,p)∼ℬ[(Vϕ(c,p)−r(c,p))2]−ceℋ[πθ],L(θ,φ)=-L^CLIP(θ)+c_v\,E_(c,p) [(V_φ(c,p)-r(c,p))^2 ]-c_e\,H[ _θ], (6) where ℋ[πθ]H[ _θ] is token-entropy computed from the model’s next-token distributions to encourage exploration. C.3 Reward Definition We compute the outcome reward only after generating c, building/running it, and rendering a screenshot ℐ(c)I(c). The total reward aggregates complementary signals: r(c,p)=αmrmetrics(c)+αvrvision(ℐ(c))+αurUI agent(c,ℐ(c))+rpenalty(c).r(c,p)\;=\; _m\,r_metrics(c)\;+\; _v\,r_vision(I(c))\;+\; _u\,r_UI agent(c,I(c))\;+\;r_penalty(c). (7) Vision score rvision∈[0,1]r_vision∈[0,1]. A vision evaluator scores the rendered UI for overall layout quality and visual polish (e.g., hierarchy, spacing, typography, color coherence). This provides dense feedback even when task success is hard to verify. UI-agent interaction reward rUI agent≥0r_UI agent≥ 0. A UI agent performs lightweight interaction tests (e.g., form filling, clicking, navigation) on the rendered page and returns a success rate. This signal directly captures functional correctness beyond static appearance. Code metrics bonus rmetrics≥0r_metrics≥ 0. We award small bonuses for richer, more realistic implementations, such as non-trivial component structure and interactive behaviors (state/event handling). This discourages overly minimal outputs that “look okay” but are not functional. C.4 Additional Learning Strategies Multi-Temperature Candidate Sampling and Selection (Best-of-N) For each prompt, we generate a small set of candidates with different temperatures to balance exploration and quality: ci∼πθold(⋅∣p,τi),τi∈τlow,τhigh,i=1,…,N.c_i _ _old(· p, _i), _i∈\ _low, _high\, i=1,…,N. (8) We execute all candidates and keep the best one under a minimum-quality constraint: c∗=argmaxcir(ci,p)s.t. r(ci,p)≥rmin.c^*= _c_ir(c_i,p) .t. r(c_i,p)≥ r_ . (9) This increases the diversity of UI Generation, and also preserves the good enough samples such that the model can learn to generate more advanced UI templates without forgetting the old format of template it already learn before. Fast-gate pre-build filter. Before expensive build/render, we skip candidates that are almost surely invalid (e.g., missing export default, clearly malformed structure, excessive length of code that highly possible truncated). This saves compute by avoiding costly failures. C.5 Pseudo Algorithm for Training Procedure Algorithm 1 Sequence-Level PPO Training for WebUI Code Generation 0: Prompts D, initial policy πθ _θ, value VϕV_φ, temperatures τii=1N\ _i\_i=1^N 1: for each PPO iteration do 2: Sample a batch of prompts ℬ⊂B 3: for each prompt p∈ℬp do 4: Sample candidates cii=1N\c_i\_i=1^N with temperatures τi\ _i\ 5: Apply fast-gate filter; build, run, and render remaining candidates 6: Compute rewards r(ci,p)r(c_i,p) 7: Select c∗=argmaxir(ci,p)c^*= _ir(c_i,p) (optionally thresholded) 8: Store (p,c∗,r∗)(p,c^*,r^*) for PPO update 9: end for 10: Compute advantages A(p,c)=r(c,p)−Vϕ(p)A(p,c)=r(c,p)-V_φ(p) and normalize within batch 11: Update θ,ϕθ,φ using clipped PPO objective and value regression 12: end for C.6 UI Agent Task Execution in VisRL We integrate a vision-grounded UI agent that tests functional correctness by proposing and executing interactions (clicks, form fills) on rendered UIs, computing a success rate that contributes to the composite reward with weight αu=0.3 _u=0.3 alongside vision quality and code metrics. We describe every step as follows: UI-Agent Task Execution (Playwright + Vision) Step 1 — Element Detection (DOM Scanning). Enumerate interactive elements via JS injection and assign stable element IDs. • Candidate tags: INPUT, TEXTAREA, SELECT, BUTTON, A • Also include nodes with onclick or cursor:pointer • Filtering: remove nested clickables; enforce minimum area threshold • Visualization: render a labeled screenshot with red dashed borders and IDs Scanning output (example): ⬇ 1 [0]: <input> "Product Name" 2 [1]: <input> "Price" 3 [2]: <button> "Submit" Step 2 — Vision-Guided Task Proposal. Send the labeled screenshot + element list to the vision model to propose executable test actions. Prompt to vision model: ⬇ 1 Available elements: 2 [0]: <input> "Product Name" 3 [1]: <input> "Price" 4 [2]: <button> "Submit" 5 6 Suggest actions to test UI functionality. 7 Return a JSON action list with element_id and (optional) value. Returned task proposal (example JSON): ⬇ 1 2 "actions": [ 3 "action": "fill", "element_id": 0, "value": "Test Product", 4 "action": "fill", "element_id": 1, "value": "19.99", 5 "action": "click", "element_id": 2 6 ] 7 Step 3 — Action Execution by Element ID. Execute the proposed actions using stored DOM references: • Fill: INPUT/TEXTAREA (type() + input event) • Select: SELECT (match by text/value + change event) • Click: BUTTON/A (check for navigation / DOM change) Step 4 — Multi-Strategy Verification. Determine success/failure using observable UI signals: • Success if expected text appears, key elements change, or DOM count updates • Failure if error keywords appear, exceptions occur, or no observable change is detected Step 5 — UI-Agent Scoring. Compute per-task success rate as: score=successful_actionstotal_actions.score= successful\_actionstotal\_actions. C.7 Vision Agent – Evaluation Rubric The vision evaluator assesses UI quality and the visual presence of interactive UI components (buttons, forms, charts, modals) based on screenshot analysis, providing a proxy signal for interface complexity. Complementary to this, the UI-Agent performs functional verification by executing interactions and measuring actual success rates, ensuring the model learns both visual appeal and operational correctness." We provide the rubric detail as follows: Vision-Agent Rubric (0–5) Step 0: Hard-gate failures (Score = 0). Assign 0 if any of the following occur: • Build/runtime error messages are visible. • Completely blank/white page (no rendered UI). • Browser/system error page (e.g., navigation failure). Step 1: Base score (0.0--4.0) = Design (0--2) + Interactivity (0--2). A) Visual design & complexity (0--2). • 0.0--0.5: Plain/text-only layout; minimal styling. • 0.5--1.0: Basic styled components (color, spacing) with simple layout. • 1.0--1.5: Clear hierarchy, multiple sections/cards, consistent typography and spacing. • 1.5--2.0: Polished UI with advanced styling (shadows/gradients), cohesive theme, animations where appropriate. B) Visual Presence of UI Components.(0--2). • 0.0--0.5: Static content only; no executable UI components interaction. • 0.5--1.0: Basic UI components(e.g., 1--3 buttons/inputs) with simple layout. • 1.0--1.5: Multiple UI sections (4-8 components) with clear visual organization and variety. • 1.5--2.0: Rich component ecosystem (tables, charts, tabs, filters, modals, cards) with sophisticated visual structure. Step 2: Excellence tier (4.0--5.0). After computing the base score, assign a bonus within [0, 1] based on overall polish: • 4.0--4.3: Good UI --- professional palette, consistent spacing, feature-complete. • 4.3--4.6: Outstanding UI --- smooth transitions, advanced components, production-ready details. • 4.6--5.0: Near-perfect UI (rare) --- exceptional hierarchy, cohesive design language, flawless execution. Step 3: Mandatory deductions (apply after Steps 1--2). • Missing buttons/links: -0.3 • Inconsistent spacing or misalignment: -0.3 • Broken images / missing assets: -0.3 Appendix D VisRL: Training Reward and Hyparameter Analyses D.1 Training Curve for Different PPO Strategies Fig. 16 shows that incorporating the UI agent substantially improves PPO training stability and efficiency. In Fig. 16 a, we show the training curve for PPO learning w/ vision agent only. The average reward increases only gradually and remains noisy; the vision score fluctuates with no clear upward trend, and the invalid action rate stays relatively high throughout training. In contrast, VisRL adopts the UI agent (Fig. 16 b), the reward rises sharply in the early phase and maintains a consistently higher level. This improvement is accompanied by a strong increase in vision score and a steady reduction in invalid actions, indicating that agent-based interaction feedback provides a cleaner learning signal and guides the policy toward more executable, UI-consistent code. (a) (b) Figure 16: Comparison of PPO training dynamics with and without a UI agent. Top: average reward (with 7-step moving average). Bottom: vision score and invalid action rate. D.2 Reward Decomposition Analysis To better understand the effectiveness of each reward component, we present a reward decomposition analysis. Figure 17 shows a radar plot analysis comparing reward components across different training strategies for both General WebUI and 3D WebUI Simulation tasks. We evaluate five metrics: Build Success (compilation rate), Average Vision Score (aesthetic quality from GPT-4o-mini), UI Agent Test Full Success (all proposed interactions succeed), UI Agent Test Half Success (≥50% interactions succeed). The analysis compares: (1) SFT baseline without RL (gray), (2) PPO with analyzer-only reward (rmr_m, orange), (3) PPO with vision-only reward (rvr_v, blue), (4) PPO with UI-agent-only reward (rur_u, green), and (5) VisRL with the full composite reward (rm+rv+rur_m+r_v+r_u,red). Results demonstrate that single-component rewards lead to unbalanced optimization: vision-only PPO achieves high aesthetic scores but poor functional correctness, UI-agent-only PPO maximizes interaction success but produces visually simplistic interfaces, and analyzer-only PPO maintains build success but lacks both visual appeal and interactivity. In contrast, VisRL with the composite reward (αv=0.6,αu=0.3,αm=0.1 _v=0.6, _u=0.3, _m=0.1) achieves the most balanced performance, excelling across all dimensions simultaneously. Notably, for 3D simulation tasks, the balanced reward maintains high build success (>0.9) while significantly improving both vision scores and UI agent test success rates compared to single-component baselines, validating the necessity of multi-component reward design for holistic web UI generation quality. Figure 17: Radar plot for reward decompsition analysis D.3 Reward Weight Analysis – α Sensitivity In Fig. 18, we study the sensitivity of VisRL to the composite reward weights (αv,αu,αm)( _v, _u, _m) by sweeping αv _v (vision) and αu _u (UI-agent). We keep the analyzer weight fixed (αm=0.1 _m=0.1), and evaluate four outcome metrics, including average vision score, build success, UI-agent full success, and UI-agent half success. The different combination of α shows a clear visual quality-functionality trade-off. When placing more weight on the vision term yields higher visual scores (e.g., ∼ 0.70 at (0.8,0.1,0.1)(0.8,0.1,0.1)), and shifting weight toward the UI-agent term increases functional correctness (UI-agent full success rises from ∼ 0.28 to ∼ 0.37). Notably, build success also improves with larger αu _u (from ∼ 0.58 to ∼ 0.64), suggesting that emphasizing executable, interaction-grounded rewards encourages more compilable implementations rather than just changing surface-level UI appearance. As a result, we set the weight at the point α=[0.6,0.3,0.1]α=[0.6,0.3,0.1] that preserves strong visual quality while maintaining build success and performance gains in UI-agent-testing correctness. Figure 18: Sensitivity analysis of α Configuration Appendix E Extended Results and Diagnostics E.1 Additional Experimental Results (per-Test Category Results) In Sec. B.2, we provide the test case distribution of WebUIProof , including functionality testing, data display testing, UI design validation testing. Here, we present the per-test category results for both proprietary API models and our self-trained models. For each model, we show the No/Partial/Full completion scores and accuracy by test category (test-category distribution shown in Fig. 11). For example, the accuracy of functionality testing (FT) for a certain model is computed as: #Full completion in FT+0.5∗#Partial completion in FTTotal#FT \#Full completion in FT+0.5*\#Partial completion in FTTotal\#FT (10) In Table 10 and 11, we show the category accuracy on the general WebUI task for proprietary API and self-trained models. In Table 12 and 13, we show the category accuracy on the 3D WebUI task. Table 10: Category accuracy of General WebUI task on Proprietary API. Model Name Build Failure Rate (%) Test Category Build Sucess UI-Agent Verified Task (%) Accuracy (Cat. %) No Completion Partial Completion Full Completion Claude Sonnet 4 Anthropic (2025) Functionality Testing 10.15 23.86 30.28 42.21 30.37 Data Display Testing 15.04 15.76 43.41 51.29 UI Design Validation 18.52 29.63 37.03 51.85 Gemini 2.5 Pro DeepMind (2025) Functionality Testing 16.15 19.44 22.47 32.19 32.21 Data Display Testing 12.89 17.19 51.58 60.17 UI Design Validation 14.81 33.33 35.19 51.85 Qwen 3 Coder 480B Alibaba Cloud (2025) Functionality Testing 15.78 21.57 21.77 32.56 33.05 Data Display Testing 12.89 14.32 50.14 57.31 UI Design Validation 16.67 33.33 33.33 50.00 DeepSeek R1 DeepSeek (2025) Functionality Testing 15.96 21.14 26.24 36.81 32.89 Data Display Testing 14.32 17.19 42.98 51.58 UI Design Validation 11.11 25.93 29.63 42.59 GPT 4.1 OpenAI (2025) Functionality Testing 7.48 17.84 29.96 38.88 36.58 Data Display Testing 18.62 21.49 35.82 46.56 UI Design Validation 14.81 29.63 25.93 40.74 Llama 4 Maverick Meta (2025) Functionality Testing 1.39 15.87 30.97 38.91 46.64 Data Display Testing 4.30 24.36 34.38 46.56 UI Design Validation 3.70 18.52 25.93 35.19 Kimi K2 Moonshot (2025) Functionality Testing 2.39 8.67 34.99 39.33 46.31 Data Display Testing 10.02 8.59 35.82 40.11 UI Design Validation 37.04 37.04 18.52 37.04 Llama 3 405B Meta (2024) Functionality Testing 3.37 15.45 32.17 39.89 47.32 Data Display Testing 5.73 14.33 27.22 34.38 UI Design Validation 55.56 7.40 18.51 22.22 Table 11: Category accuracy of General WebUI task on Self-Trained Model. Training Variants of Qwen 2.5 14B Instruct Build Failure Rate (%) Test Category Build Sucess UI-Agent Verified Task (%) Accuracy (Cat. %) No Completion Partial Completion Full Completion Baseline SFT Functionality Testing 7.75 10.71 20.66 26.01 56.25 Data Display Testing 14.33 14.33 21.49 28.65 UI Design Validation 18.52 11.11 18.52 24.07 PPO W/ Vision agent Functionality Testing 12.87 11.05 34.38 39.90 41.42 Data Display Testing 14.33 7.30 28.65 32.31 UI Design Validation 18.52 25.93 37.04 50.00 VisRL W/ UI agent Functionality Testing 11.70 17.08 35.80 44.34 36.10 Data Display Testing 15.76 12.89 28.65 35.10 UI Design Validation 18.52 11.11 40.74 46.30 Table 12: Category accuracy of 3D WebUI task on Proprietary API. Model Name Build Failure Rate (%) Test Category Build Sucess UI-Agent Verified Task (%) Accuracy (Cat. %) No Completion Partial Completion Full Completion Claude Sonnet 4 Anthropic (2025) Functionality Testing 53.15 16.79 30.08 38.47 0.00 Data Display Testing 21.43 42.86 35.71 57.14 UI Design Validation 68.35 6.33 25.32 Qwen 3 Coder 480B Alibaba Cloud (2025) Functionality Testing 55.25 18.87 27.27 36.71 1.43 Data Display Testing 40.00 25.71 34.29 47.14 UI Design Validation 54.43 6.33 27.85 31.01 Gemini 2.5 Pro DeepMind (2025) Functionality Testing 62.93 15.38 21.67 29.36 0.00 Data Display Testing 35.71 28.57 35.71 50.00 UI Design Validation 56.96 15.19 27.85 35.44 Kimi K2 Moonshot (2025) Functionality Testing 56.54 14.69 28.67 36.02 0.00 Data Display Testing 42.86 40.00 17.14 37.14 UI Design Validation 62.03 12.66 25.32 31.65 Llama 4 Maverick Meta (2025) Functionality Testing 67.13 7.69 25.41 29.25 0.00 Data Display Testing 50.00 28.57 21.43 35.71 UI Design Validation 55.70 18.99 25.32 34.81 DeepSeek R1 DeepSeek (2025) Functionality Testing 34.27 18.87 22.37 31.81 11.43 Data Display Testing 57.14 21.43 21.43 32.14 UI Design Validation 70.89 10.13 18.99 24.05 GPT 4.1 OpenAI (2025) Functionality Testing 67.83 12.58 11.89 18.18 2.85 Data Display Testing 50.00 28.57 21.43 35.71 UI Design Validation 74.68 10.13 15.19 20.25 Llama 3 405B Meta (2024) Functionality Testing 65.03 13.99 13.28 20.28 2.86 Data Display Testing 74.29 8.57 17.14 21.43 UI Design Validation 74.68 17.72 7.59 16.46 Table 13: Category accuracy of 3D WebUI task on Self-Trained Model. Training Variants of Qwen 2.5 14B Instruct Build Failure Rate (%) Test Category Build Sucess UI-Agent Verified Task (%) Accuracy (Cat. %) No Completion Partial Completion Full Completion Baseline SFT Functionality Testing 52.16 12.82 19.74 26.16 0.00 Data Display Testing 48.57 25.71 25.71 38.57 UI Design Validation 68.35 12.66 18.99 25.32 PPO w/ Vision agent Functionality Testing 59.38 27.97 20.96 34.95 0.00 Data Display Testing 21.43 35.71 28.57 46.43 UI Design Validation 62.03 15.19 22.78 30.38 VisRL w/ UI agent Functionality Testing 53.53 14.95 30.05 37.53 0.00 Data Display Testing 42.86 28.57 28.57 42.86 UI Design Validation 58.23 18.99 22.78 32.28 E.2 Visualization of UI Agent Traces Figure 19: WebUI interaction examples (sample 1 and 2). Each row shows a full UI state transition with agent-labeled elements and executed actions. Figure 20: WebUI interaction examples (sample 3 to 5). Each row shows a full UI state transition with agent-labeled elements and executed actions. (a) WebGPU Compute (b) Jolt Physics Instancing (c) CSS3D Mixed (d) Inverse Kinematics (e) Dynamic Lighting Figure 21: Six simulation samples. Each subfigure shows the UI state and interaction annotation for one sample. E.3 More Visualization Sample Visualization for general WebUI Fig. 22, 23, 24, 25, 26 show the general WebUI samples under different UI categories, including creative media, data management, games, social interaction, and e-commerce platforms. Figure 22: We present three webpages generated from self-trained model Qwen 2.5 14B w/ VisRL. The left-most webpage is a music band showcase portal. The upper-right webpage is a codecraft editor. The lower-right webpage is a AI website showdown. Figure 23: Two webpages generated from self-trained model Qwen 2.5 14B w/ VisRL. Both of them are the social media viewer. Figure 24: Two webpages generated from self-trained model Qwen 2.5 14B w/ VisRL. The left side is a 2048 game interface and the right side is a chess game simulator. Figure 25: Two webpages generated from self-trained model Qwen 2.5 14B w/ VisRL. Both of them are the stock report generator. Figure 26: Three webpages generated from self-trained model Qwen 2.5 14B w/ VisRL. The left side is a travel accommodation finder, the upper-right side is an AI agent workflow manager, and the lower-right side is a landing page for tech. company. Sample Visualization for 3D WebUI Simulation Fig. 27, 28, and 29 show the 3D WebUI samples under different categories. Figure 27: Four webpage demo generated from self-trained model Qwen 2.5 14B w/ VisRL. We provide simple prompt for each to showcase how they are generated. (1.) The upper-left demo: Create a 3D animation of a circular island that has multiple trees on it and a river flowing within. (2.) The upper-right demo: Create a 3D simulation of jelly cubes bouncing in a box. (3.) The lower-left demo: Create a 3D aquarium with fishes that swim around and avoid collisions with the tank walls. (4.) The lower-right demo: Create a 3D animation of a sail boat floating in the sea. Figure 28: Same as prior figure, we provide simple prompt for each to showcase how they are generated. (1.) The upper-left demo: Create a 3D simulation of a mars rover moving on the surface of Mars. Users should be able to accelerate and control the direction of the car. Incorporate accurate suspension dynamics as the car moves on the rugged Mars surface. (2.) The upper-right demo: Create a visually engaging animation of a double pendulum using JavaScript. The animation should demonstrate the chaotic motion of two pendulums connected together. (3.) The lower-left demo: Simulate 3D paper lanterns rising into the night sky during a festival, with buoyancy and wind physics, above a reflective lake. (4.) The lower-right demo: Create a realistic 3D ocean scene with: dynamic water movement, accurate wave simulations, and reflective surface effects. Figure 29: Same as prior figure, we provide simple prompt for each to showcase how they are generated. (1.) The upper-left demo: Create a 3D solar system simulation using Three.js with realistic planet orbits, textures, and lighting effects. (2.) The upper-right demo: Create a breakout-style game where a paddle at the bottom of the screen reflects a bouncing ball toward a grid of colored blocks above. The paddle moves left and right with arrow keys, and blocks disappear when hit by the ball. Each block is worth different points based on its color, and the game ends when all blocks are cleared or the ball falls below the paddle. (3.) The lower-left demo: Create a 3D animation of a top spinning. Use accurate physics to demonstrate its movements. (4.) The lower-right demo: Create a dinosaur running game where: (i) A dinosaur character can jump using spacebar/up arrow, (i) Cacti and birds appear as obstacles from the right side, (i) Ground moves continuously to create running effect (iv) Game ends when dinosaur hits an obstacle, (v) Simple pixel art style, and (vi) Day/night cycle every 500 points. E.4 Analysis of Build Failure Taxonomy To investigate the build failure reason, we categorize build-time errors into four primary types based on failure patterns observed during Next.js execution logs. Module Not Found errors occur when Webpack cannot resolve imported packages (e.g., @heroicons/react, react-icons, or react-dnd), typically due to missing installations, incorrect import paths, or renamed files. Syntax errors arise when the TypeScript/JavaScript parser encounters malformed code (e.g., missing braces, parentheses, or template literal errors), preventing the compilation stage from completing. Default Export errors (happen when a Next.js page compiles but fails during the static generation stage because some certain file do not contain a valid default React component export (e.g., pages/modelOutputPage.tsx). Finally, Reference errors occur at runtime during server-side rendering or static generation, often due to accessing variables before initialization or circular imports. In Table 30, we show the four dominant failure counts on eight models, providing insight into the generation ability of each model. Figure 30: Breakdown of build-time errors across models in our WebUI generation benchmark. Bars show the number of failures by error type, including missing dependencies, syntax mistakes, reference issues, and invalid default exports. Higher stacks indicate more frequent build failures.