Paper deep dive
LiveEvalBench: Toward Open-World Evaluation for Web Generation
Yiyao Wang, Zhen Wen, Yinghao Tang, Yixiao Fu, Lin Yuan, Xiaolau Zhang, Jun Zhou, Wei Chen
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models are increasingly capable of synthesizing executable frontend projects, yet existing benchmarks still treat web generation as a static evaluation problem. We argue that frontend artifacts demand a different paradigm: they are interactive rather than static, admit diverse yet equally valid implementations, and evolve faster than rigid pipelines can accommodate. To address these gaps, we present LiveEvalBench, an automated framework that reformulates web-generation evaluation as an agentic, adaptive, and extensible process. LiveEvalBench instantiates evaluation as a collaborative review workflow, in which a Build Engineer, a Code Engineer, and a UI Tester collectively gather evidence across the full lifecycle of a frontend project, from deployment and code inspection to browser-based interaction. To handle implementation diversity, an adaptive protocol couples shared rubrics for cross-model comparability with implementation-grounded criteria tailored to each artifact. The framework further supports incremental integration of new evaluator roles and assessment dimensions without pipeline redesign. Experiments across diverse real-world web-generation scenarios show that LiveEvalBench aligns closely with human expert judgment and provides fine-grained insights into frontier models' web generation capabilities. Code is available at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2608.03689v1
- Canonical: https://arxiv.org/abs/2608.03689v1
Trouble viewing inline? Open PDF directly →
Full Text
87,430 characters extracted from source content.
Expand or collapse full text
LiveEvalBench: Toward Open-World Evaluation for Web Generation Yiyao Wang 1∗ , Zhen Wen 1 , Yinghao Tang 1 , Yixiao Fu 1 , Lin Yuan 2† , Xiaolu Zhang 2 , Jun Zhou 2 , Wei Chen 3‡ 1 State Key Lab of CAD&CG, Zhejiang University 2 Ant Group 3 State Key Lab of CAD&CG, Zhejiang University; Laboratory of Art and Archaeology Image (Zhejiang University), Ministry of Education, China Abstract Large language models are increasingly capable of synthesizing executable frontend projects, yet existing benchmarks still treat web generation as a static evaluation problem. We argue that frontend artifacts demand a different paradigm: they are interac- tive rather than static, admit diverse yet equally valid implemen- tations, and evolve faster than rigid pipelines can accommodate. To address these gaps, we present LiveEvalBench, an auto- mated framework that reformulates web-generation evaluation as an agentic, adaptive, and extensible process. LiveEvalBench instantiates evaluation as a collaborative review workflow, in which a Build Engineer, a Code Engineer, and a UI Tester col- lectively gather evidence across the full lifecycle of a frontend project, from deployment and code inspection to browser-based interaction. To handle implementation diversity, an adaptive protocol couples shared rubrics for cross-model comparability with implementation-grounded criteria tailored to each artifact. The framework further supports incremental integration of new evaluator roles and assessment dimensions without pipeline re- design. Experiments across diverse real-world web-generation scenarios show that LiveEvalBench aligns closely with human expert judgment and provides fine-grained insights into fron- tier models’ web generation capabilities. Code is available at https://github.com/wyysteelhead/LiveEvalBench. Introduction Large language models (LLMs) are rapidly evolving from generating isolated code snippets to synthesizing complete, executable frontend projects (Si et al. 2025; Zhu et al. 2025; Wu et al. 2025; Tran et al. 2026). As project-level web generation becomes a practical capability of modern LLMs, a fundamental question emerges: how should we evaluate what models actually build? Despite this progress, evaluation has not kept pace. Exist- ing benchmarks for web generation inherit assumptions from traditional code evaluation: they assess outputs through prede- fined criteria, fixed execution scripts, or static judgments over source code and screenshots (Si et al. 2025; Zhu et al. 2025; Zhang et al. 2025; He et al. 2026). However, frontend applica- tions are not static outputs. They are interactive, admit diverse yet equally valid implementations, and continuously evolve ∗ Yiyao Wang, Zhen Wen, Yinghao Tang, and Yixiao Fu are with the State Key Lab of CAD&CG, Zhejiang University. E-mail: wangyiyao, wenzhen, yinghaotang, 3210101100@zju.edu.cn. † Lin Yuan, Xiaolu Zhang, and Jun Zhou are with Ant Group. ‡ Wei Chen is with the State Key Lab of CAD&CG, Zhejiang University, and also with the Laboratory of Art and Archaeology Image (Zhejiang University), Ministry of Education, China. E-mail: chenvis@zju.edu.cn. LiveEvalBench !"#$ %& #&! "' () Prior Works Figure 1: LiveEvalBench versus prior web coding benchmarks. While prior works treat evaluation as a one-shot scoring procedure, LiveEvalBench models it as a collaborative expert review workflow with complementary agents, an adaptive protocol, and an extensible framework. over time. This creates a fundamental paradigm mismatch: (1) static evaluation cannot judge artifacts whose quality depends on runtime interaction; (2) fixed rubrics cannot fairly assess open-ended generation where the same query admits many valid solutions; and (3) rigid pipelines cannot keep up with the rapid evolution of web generation capabilities. In this paper, we present LiveEvalBench, an automated evaluation framework designed for open-world web genera- tion. Rather than treating evaluation as a one-shot scoring procedure, LiveEvalBench models it as a collaborative expert review workflow that is agentic, adaptive, and extensible (Fig. 1). First, to evaluate interactive artifacts through interaction rather than static inspection, we instantiate evaluation as a multi-agent workflow. A Build Engineer deploys the project and produces runtime artifacts. A Code Engineer inspects implementation quality and instruction following from the source side. A UI Tester actively explores the running applica- arXiv:2608.03689v1 [cs.AI] 4 Aug 2026 tion through browser-based interaction. Each agent operates with distinct tools and evidence channels, collectively cover- ing the full lifecycle of a frontend project from deployment to code to live behavior. Second, to fairly evaluate open-ended generation where the same user request admits diverse implementations, we propose an adaptive evaluation protocol. Shared rubric items are fixed across all models answering the same query, preserving cross-model comparability. On top of these, implementation- grounded checks are synthesized for each generated project based on what the model actually built, so that scoring reflects the artifact’s true capability rather than penalizing valid but unexpected design choices. Third, to support an evaluation framework that evolves alongside web generation itself, we design an extensible infrastructure. Each evaluator is declared as a configuration of three components: a persona specifying who is judging, a set of criteria specifying what is judged, and a set of tools specifying what evidence is consulted. Adding a new evaluation perspective (e.g., a mobile-only user, a color- blind user) requires filling in this configuration rather than redesigning the pipeline. We instantiate LiveEvalBench on a benchmark of100 real-world web generation queries evaluated across11fron- tier models. The resulting evaluation shows that current frontier models fail most often on interactive runtime behav- ior. Across multiple validation experiments, LiveEvalBench shows strong alignment with human judgment across evalua- tion dimensions. Our contributions are summarized as follows: 1.We identify a fundamental evaluation paradigm mismatch in web generation and propose LiveEvalBench, an agentic evaluation framework for multi-perspective assessment using build, code, and browser evidence. 2.We introduce an adaptive evaluation protocol that bal- ances cross-model comparability with implementation- aware scoring for open-ended web generation. 3. We develop an extensible evaluation infrastructure where new evaluator roles and assessment dimensions can be added through configuration without redesign. 4.We construct a benchmark for web generation and uncover strengths and weaknesses of frontier models overlooked by existing evaluations. Related Work Benchmark for Web Generation We organize prior work by the kind of evidence the evaluator inspects: source code, runtime functionality, and rendered visuals. Code evaluation Pure code-generation benchmarks score programs by executing them against fixed unit tests, spanning function-level suites (Chen et al. 2021; Austin et al. 2021; Hendrycks et al. 2021; Liu et al. 2023a), contamination- resistant and library-rich variants (Jain et al. 2025; Zhuo et al. 2025), and repository-level engineering settings (Jimenez et al. 2024). Web-generation benchmarks that adopt code- level checks face two complications: the artifact is a frontend project rather than a single program, so the “code” channel must also cover build configuration and asset organization; and ground-truth tests written against a fixed reference no longer apply when many UI layouts can satisfy the same user request (Xu et al. 2025; Lei et al. 2026). Functional evaluation A second line evaluates whether the deployed webpage behaves correctly at runtime. Sandboxed automatic-test pipelines pair each task with predefined scripts executed in a controlled environment (Xu et al. 2025; Zhu et al. 2025), achieving reproducibility at the cost of binding tests, fixtures, and judging logic to specific task templates. A more recent category lets an evaluator agent explore the deployed application in a real browser instead of executing fixed scripts (Wu et al. 2025; He et al. 2026; Lu et al. 2025; Tran et al. 2026). WebCompass (Lei et al. 2026) is representative, using an agent-as-a-judge procedure that synthesizes targeted test cases in the browser during execution. Adjacent web-agent resources (Deng et al. 2023; Zhou et al. 2024; Koh et al. 2024) do not evaluate generated frontends but inform evaluator design. Visual evaluation A third line judges the rendered inter- face. One branch treats UI generation as visual translation and scores generations against a target design (Beltramelli 2018; Yun et al. 2024; Li, Zhang, and Yang 2025): Design2Code (Si et al. 2025) is representative, benchmarking multimodal models on reproducing real-world webpages from screenshots via visual similarity and human preference. A second branch renders the artifact and asks an MLLM judge to score it via a per-task checklist (Xiao et al. 2024): ArtifactsBench (Zhang et al. 2025) is representative, driving each artifact through scripted interactions and scoring visual and interactive in- tegrity. Both branches use rubrics tied to a single target or task template. Limitations of Existing Benchmarks Across these efforts, existing benchmarks have made important progress in web generation evaluation, but two limitations remain. First, their adaptive capacity is often bounded by predefined evaluation artifacts (e.g., expert-defined checklists), which ties evaluation to the benchmark’s existing queries and makes it costly to evaluate newly emerging web generation scenarios. Second, their evaluation pipelines are usually rigid: adding a new evaluator role or assessment dimension requires redesigning the pipeline rather than extending the existing framework. LiveEvalBench addresses these limitations within a unified agentic evaluation framework, using an adaptive evaluation protocol that grounds shared rubric items into implementation- specific checks and an extensible evaluation infrastructure that supports new evaluator roles and assessment dimensions without redesigning the pipeline. LLM-as-Judge and Multi-Evaluator Frameworks LLM-as-judge work spans single proprietary judges with structured rubrics (Zheng et al. 2023; Liu et al. 2023b, 2024), specialized open evaluator models (Kim et al. 2024a,b), stud- ies of judge reliability, bias, and validity (Chen et al. 2025; Guerdan et al. 2025; Bean et al. 2025), and survey-level fram- ings (Li et al. 2025). Two threads directly motivate our design: WorkPerspectivesMethodDesign Build Src. Brwsr. Interact. Visual Exec. Adapt. Multi-Eval. Extens. HumanEval (Chen et al. 2021)—✓ —✓— SWE-bench (Jimenez et al. 2024) △✓ —✓— Design2Code (Si et al. 2025)— — △—✓— Web-Bench (Xu et al. 2025)△✓—✓— FrontendBench (Zhu et al. 2025)—✓ △✓— ArtifactsBench (Zhang et al. 2025) — △✓ △ △— FronTalk (Wu et al. 2025)— —✓ △ △— Vision2Web (He et al. 2026)△ —✓ △✓— Vibe Code Bench (Tran et al. 2026) △ —✓ △✓—✓— WebCompass (Lei et al. 2026) △ △✓— LiveEvalBench✓ Table 1: Comparison of representative code, web, and frontend evaluation benchmarks.✓indicates explicit support,△indicates partial or indirect support, and —indicates that the capability is not a primary part of the evaluation protocol. Src., Brwsr., Interact., Adapt., Multi-Eval., and Extens. denote source, browser, interaction, adaptive, multi-evaluator, and extensible, respectively. multi-judge juries replace one large judge with heterogeneous smaller judges, matching or outperforming a single strong judge while reducing bias and cost (Verga et al. 2024; Dubois et al. 2023); and agent-as-a-judge equips evaluators with tools to inspect files, execute code, or browse pages, approaching human reliability on coding tasks and materially affecting outcomes in browser-based web evaluation (Zhuge et al. 2024; Tran et al. 2026). Existing panels operate over textual outputs rather than multi-channel evidence, and evaluator behaviors are rarely modularized as swappable components—gaps our agentic evaluation framework and extensible evaluation in- frastructure address. Method Framework Overview The overview of LiveEvalBench is presented in Fig. 2. LiveE- valBench takes the user query and model response as input, and deploys the response into a runnable link and a deployed project on disk. These artifacts are then handed to three special- ized evaluation agents, namely the Build Engineer, the Code Engineer, and the UI Tester, which examine complementary facets of the project in parallel (Fig. 2(a)). During examina- tion, each agent follows an adaptive evaluation protocol that grounds shared rubric criteria into implementation-specific checks at runtime (Fig. 2(b)). The framework is also ex- tensible, supporting new evaluator roles through a unified agent infrastructure (Fig. 2(c)). The resulting per-criterion judgments are aggregated into a final 90-point score: Build contributes 15 points, Code contributes 30 points, and UI contributes 45 points, with per-evaluator caps that prevent any single facet from dominating. Evaluation Agents As shown in Fig. 2(a), LiveEvalBench employs three spe- cialized evaluation agents that examine these complementary facets in parallel, each operating within the environment natural to its perspective. Build Engineer The Build Engineer owns the deployment perspective and operates within the project directory together with a shell. It is also responsible for producing the build artifacts in the input step: it extracts the project files, performs minor repairs to auxiliary configuration when necessary, and launches the application following the response’s setup instructions. During evaluation, the Build Engineer revisits its own build trajectory (i.e.), the commands it issued and the auxiliary repairs it made, and from this evidence judges how much extra effort the deployment demanded beyond what the model response itself prescribes. Code Engineer The Code Engineer takes the perspective of a developer inspecting the implementation, with access to the source tree and the ability to run commands in the terminal for verification. It evaluates the project on the code side along two aspects: implementation quality and instruction following. Implementation quality considers both code readability and robustness, reflecting whether the code is well organized and whether it handles edge cases gracefully. Instruction following checks whether hard requirements stated in the user query are faithfully realized in the implementation, such as the use of a designated framework (e.g., Vue or React). UI Tester The UI Tester stands in for the end user and interacts with the running web link solely through the browser, without access to the underlying code. It assesses the project along three criteria: visual quality, interaction experience, and a set of query-specific criteria generated from the user query. Visual quality looks at whether the interface renders completely and presents itself in a polished and visually coherent manner. Interaction experience examines whether the interface responds clearly and flows intuitively under user actions. The query-specific criteria, tailored to what each query actually asks for, provide a thorough examination of functionality. "I will launch the project, assess the effort..." "I will test the UI and functionality..." "I will evaluate code quality and instruction..." !"# $% &'(# ! ( ( “You are modile user, you goal is to test...” Interaction Experience • Interaction • Visual Quality <button /> <div /> )(*&'(# • Hover ▪check visual effect • Click ▪check zoom animation +# ! “I want a website...” keyboard screenshot clickcccccccccccccccccccclllllllllllllllllllllllliiiiiiiiiiiiiiiiiiiiiiiicccccccccccccccccccckkkkkkkkkkkkkkkkkkkkkkkk writewwwwwwwwwwwwrrrrrrrrrrrriiiiiiiiiiiiittttttttttttteeeeeeeeeeee "# $ $# % ,# Figure 2: Overview of the LiveEvalBench framework. Given a user query and a model response, the response is deployed into a runnable link and a project directory, which are then evaluated in parallel by three specialized agents (a). Each agent follows an adaptive evaluation protocol that grounds shared rubric criteria into implementation-specific checks at runtime (b), and the agent design is extensible, supporting new evaluators through configuration (c). Adaptive Evaluation Protocol As sketched in Fig. 2(b), we decouple what each evaluator should check from how those checks work. The first concern adapts to the user query; the second adapts to what each model actually produced. What to check (per query) Each evaluator carries a set of fixed criteria along the dimensions described above. On top of these, the UI Tester additionally takes in query-specific criteria. To faithfully cover every requirement raised by the user, we extract the functional requirements expressed in the query and turn each of them into a corresponding criterion. These query-specific criteria are shared across all models answering the same query, ensuring comparability between models. Whether to take in query-specific criteria is itself a configuration choice; the Code Engineer opts out and relies solely on its fixed rubric. How to check (per implementation) A criterion such as interaction experience is too abstract to apply directly; it must be turned into concrete actions on this particular project. For each criterion, the evaluator first inspects the implementation through what it can observe. For example, the UI Tester can examine the rendered DOM and accessibility tree of the running project. For each applicable element it finds there, the evaluator then writes an individual executable check tailored to that element. As illustrated in Fig. 2(b), the criterion of interaction experience is grounded in the buttons and containers actually present in the project, and is then verified through hover and click actions tailored to those elements. The criteria themselves stay fixed across all models, while the concrete checks adapt to each implementation. Extensible Evaluation Infrastructure For the framework to evolve alongside web generation, adding a new evaluation perspective must be cheap. This reduces to a more basic question: what is the minimum needed to define an evaluator? We summarize it into three components, illustrated in Fig. 2(c) with a touch-only mobile user as a running example. The first is a persona that specifies who is judging, fixing the role the evaluator plays and the standards it brings to the interface. The second is a set of criteria that specifies what is judged; the mobile user in Fig. 2(c) still cares about interaction and visual quality, the same dimensions the UI tester would examine. What separates the two is the third component: a set of tools that specifies what evidence is consulted. The mobile user is barred from using the keyboard and can only interact with the interface by touch; source- level access such as read and write is also denied. Once the triple is filled in, the orchestrator schedules the evaluator and aggregates its score under the same rules as the default jury, with no changes to framework code. This abstraction extends easily beyond the running example. A color-blind end-user, for instance, can be added by reusing the same persona and criteria slots while swapping in a new screenshot tool that applies a color-vision transform, so the evaluator sees the page as the user would. Other perspectives, such as a visually impaired user relying on the accessibility tree, follow the same recipe by filling in the three slots. The full schema for specifying an evaluator under this three-slot abstraction is provided in the supplementary material. Benchmark Construction Query Construction LiveEvalBench consists of 100 frontend project specifications across three difficulty levels (28 Level-1 (L1), 43 Level-2 (L2), and 29 Level-3 (L3) queries) and six task categories adapted from ArtifactsBench (Zhang et al. 2025) and Design Arena (Design Arena 2025): Data Visualization (17), UI Component (19), Game (18), Web App (18), Website (14), and 3D Design (14). The queries also span three specification granularities: abstract intent-level requests (46), product- requirements-document (PRD)-style specifications (41), and functional descriptions (13). The joint distribution over cate- gories and difficulty levels is shown in Fig. 3. We build the benchmark through a four-stage pipeline. Seed Collection We aggregate real-world frontend requests from three complementary sources: public tutorials and cod- ing guides, social media platforms, and task samples from ArtifactsBench (Zhang et al. 2025). Unlike prior benchmarks sourced from sites like GitHub, we treat social media as a primary source, since it is where practitioners share the most up-to-date queries probing frontier code-generation models. We retain only items containing a self-contained, implementable specification, and deduplicate across sources. Seeds serve only as topical anchors and are never used directly. Role-Conditioned Synthesis Real-world frontend specifi- cations differ substantially in abstraction and technical detail. To capture this heterogeneity, we rewrite each seed under four roles with distinct granularity: “end user” (intent-driven, non-technical), “designer” (visual and interaction details), “project manager” (structured requirements and acceptance criteria), and “developer” (component decomposition, APIs, edge cases). Rewrites are produced by three frontier mod- els, namely Gemini 3 Pro Preview, GPT-5.2, and Claude Sonnet 4.5, using role-specific prompts provided in the sup- plementary material. This yields a pool of∼4,000 candidate specifications that vary along two orthogonal axes: topical diversity (from seeds) and stylistic diversity (from roles and rewriters). Difficulty Stratification Each candidate is scored by an LLM judge along three rubric dimensions (number of views, external dependency complexity, and interaction/state com- plexity) and mapped to L1/L2/L3 via a deterministic rule. L1 covers small, self-contained features confined to a single page, such as a single form or an isolated canvas effect; L2 covers moderately complex tasks that may depend on several external libraries, involve a larger codebase, or require non- trivial application logic; L3 covers multi-page or multi-view applications with cross-view navigation, or several intricate features composed together. Two authors verify the labels and resolve disagreements by discussion. Human Curation From the stratified pool we hand-pick the final 100 queries to cover all six categories and multiple specification granularities across the benchmark, without enforcing equal counts per cell; selection prioritizes clarity, feasibility, evaluability, and novelty as judged by 2 experts with frontend engineering experience. The full 100 specifications, with granularity, category, dif- ficulty annotations, and per-query evaluation criteria, will be publicly released upon publication under a license permitting free research use. Generated Frontend Projects To instantiate the benchmark with model-generated artifacts, we run each query through the11frontier code-capable LLMs listed in the Experiments section, yielding one generated frontend project per query–model pair. All models are queried UI Component Game Web App Data Visualization Website 3D Design 7 10 2 3 10 5 6 7 5 5 6 6 5 5 4 2 5 7 L1 (Beginner)L2 (Intermediate)L3 (Advanced) Figure 3: Distribution of LiveEvalBench queries across six categories. Each petal is one category, and the three concentric layers break it down by difficulty (L1 inner / L2 middle / L3 outer). in a single turn with provider-default decoding settings. The only modification to the raw user specification is a single prepended instruction that frames the model as a frontend engineer and asks it to return frontend project content. This minimal-intervention protocol isolates each model’s native web generation ability and keeps comparisons faithful to default behavior. Per-Query Evaluation Criteria LiveEvalBench is released with a set of criteria attached to each query. These criteria are produced following the adaptive evaluation protocol and manually reviewed for faithfulness. The full schema is provided in the supplementary material. Experiments We organize the empirical study around three questions: (RQ1) how state-of-the-art models compare on LiveEval- Bench, (RQ2) whether LiveEvalBench agrees with human judgment on the same generated projects, and (RQ3) whether LiveEvalBench provides reliable evaluation signals. Main Benchmark Evaluation We evaluate 11 frontier models: Claude Sonnet 4.6, Claude Opus 4.7, GPT-5.5, GPT- 5.4 Mini, Gemini 3.1 Pro, DeepSeek V4 Pro, Kimi K2.6, GLM 5.1, MiniMax M2.7, Ling 2.6 1T, and Qwen3.5-27B. The main leaderboard evaluates these models on all 100 LiveEvalBench queries. Each query–model record is evalu- ated three times using the same evaluation procedure, and criterion-level judgments are aggregated by majority before computing the main leaderboard scores. Human Agreement We select 70 generated frontend projects from the main benchmark evaluation for human- agreement validation. Each item is rated independently by 0153045607590 Scores of 11 models Claude Opus 4.7 GLM 5.1 GPT-5.5 Claude Sonnet 4.6 Gemini 3.1 Pro GPT-5.4 Mini Kimi K2.6 DeepSeek V4 Pro MiniMax M2.7 Ling 2.6 1T Qwen 3.5 27B 14.323.732.7 14.022.631.5 13.623.830.7 12.523.829.2 13.823.127.5 13.123.226.7 14.021.626.8 13.122.025.9 12.022.724.3 12.222.923.8 12.022.919.8 70.7 68.1 68.0 65.5 64.4 63.1 62.4 60.9 59.0 58.9 54.6 Build (15)Code (30)UI (45) Figure 4: Main leaderboard on LiveEvalBench, visualized as Build + Code + UI sub-scores stacked to the 90-point total. Each segment is annotated with its sub-score and the row total is appended on the right. Models are ordered by total descending. 010203040 Lost points of 11 models Claude Opus 4.7 GLM 5.1 GPT-5.5 Claude Sonnet 4.6 Gemini 3.1 Pro GPT-5.4 Mini Kimi K2.6 DeepSeek V4 Pro MiniMax M2.7 Ling 2.6 1T Qwen 3.5 27B 3.42.94.38.0 3.63.85.77.8 1.43.62.64.210.2 2.53.62.65.410.4 3.13.85.012.5 1.93.63.16.411.9 4.83.66.511.7 1.94.04.07.811.3 3.04.72.67.613.1 2.84.22.98.512.7 3.03.53.79.815.4 19.3 21.9 22.0 24.5 25.6 26.9 27.6 29.1 31.0 31.1 35.4 Build Smoothness Impl. Quality Instr. Following Visual Quality Interaction Figure 5: Loss attribution by scoring dimension, on the 90- point scale. Each row is a model; each segment is the points lost on one dimension; the value at the right of the bar is the total loss. Models are ordered by total loss ascending (best on top). three annotators on the same criteria as the UI Tester. We compare the UI Tester’s three-run majority judgment with the human majority judgment, and report percent agreement and Gwet’s AC1, a chance-corrected metric for imbalanced pass/fail labels. Evaluator-Role Ablation We compare the full role- separated evaluator with a single-evaluator ablation on the human-agreement subset, where one evaluator assesses all dimensions covered by the role-separated design in a single evaluation pass. Reliability Evaluation We conduct two reliability evalua- tions. First, we compare three-time evaluation with a single evaluation pass to test whether three-time evaluation im- proves score stability. Second, we randomly sample 20 web generation queries and evaluate them with Kimi K2.6 and Qwen3.7-Plus under the same evaluation protocol as the main benchmark, then measure agreement across evaluator models. Results and Analysis Main Benchmark Evaluation We now report model performance on LiveEvalBench, with the main results presented in Figs. 4–6. Fig. 4 gives the overall leaderboard with each model’s total score broken down into the Build / Code / UI sub-scores, Fig. 5 shows where each model loses points across the five scoring dimensions, and Fig. 6 reports how scores change across the L1/L2/L3 difficulty levels. We discuss the key takeaways below. UI behavior matters most for frontend coding ability As shown in Fig. 4, which reports each model’s total score decomposed into the three per-evaluator sub-scores (Build / Code / UI), UI exhibits the widest range, spanning from19.79(Qwen3.5-27B) to32.74(Claude Opus 4.7). Dimension / CriteriaAgreement↑ AC1↑ UI Tester majority vs. human majority Visual quality88.7%0.831 Interaction experience83.0%0.718 Query-specific criteria85.6%0.805 Overall85.7%0.793 Table 2: Agreement of the UI Tester with human majority judgments. AC1 denotes Gwet’s chance-corrected agreement coefficient. Build sub-scores are more compressed, while Code sub-scores show moderate variation. This pattern reflects a natural hier- archy of frontend coding competence: successful builds are the entry-level bar, source-level quality is a step further, and robust UI interaction is the highest bar. Performance varies across difficulty levels Fig. 6 stratifies per-model scores by L1/L2/L3 difficulty. Aggregate scores decrease from L1 (63.8) to L2 (59.8) and L3 (55.5), indicating that the difficulty taxonomy captures a meaningful score gradient. The same trend appears for most individual models, with L1 generally receiving the highest scores and L3 the lowest in most cases. The leaderboard separates model performance tiers The updated leaderboard shows a clear spread in total scores, from 70.71for Claude Opus 4.7 to54.61for Qwen3.5-27B. The top group remains led by Claude Opus 4.7, GLM 5.1, and GPT-5.5, while the lower-scoring models continue to lose the most points on runtime-facing UI behavior. Claude Opus 4.7 GLM 5.1 GPT-5.5 Claude Sonnet 4.6 Gemini 3.1 Pro GPT-5.4 Mini Kimi K2.6 DeepSeek V4 Pro MiniMax M2.7 Ling 2.6 1T Qwen 3.5 27B L1 L2 L3 Difficulty 76.1 n=24 73.1 n=25 74.4 n=23 70.0 n=24 66.5 n=23 69.5 n=24 62.7 n=24 70.2 n=24 71.1 n=24 65.3 n=23 60.5 n=26 68.7 n=37 70.8 n=36 66.7 n=38 66.4 n=40 66.1 n=37 63.6 n=36 61.8 n=39 60.4 n=35 60.1 n=36 59.9 n=37 55.0 n=36 69.4 n=39 62.2 n=39 65.9 n=38 61.7 n=36 61.4 n=39 58.6 n=40 62.8 n=37 56.4 n=41 50.0 n=40 54.4 n=40 50.2 n=38 Score by Difficulty × Model (0–90) 0 10 20 30 40 50 60 70 80 90 Score on 90 Figure 6: Score by difficulty×model on the 90-point scale. Cells annotated with the per-cell sample sizen. Difficulty levels (L1/L2/L3) reflect the query taxonomy defined in Benchmark Construction. Variant 0 1–29 30–59 60–89 90 Full18417283 1-agent 2100049 Table 3: Score distribution for evaluator-role ablation on the 70-case subset. Scores are on the 90-point scale. Human Agreement Tab. 2 reports agreement between the UI Tester and human majority judgments. Overall agreement reaches85.7%with AC1= 0.793, with similarly strong agreement across visual quality, interaction experience, and query-specific criteria. These results indicate that the LiveEvalBench evaluation framework provides human-aligned diagnostic signals for UI assessment. Evaluator-Role Ablation Tab. 3 shows that the single-evaluator ablation collapses to a bimodal score distribution: on the 70-case subset, it places 21 cases at0and 49 cases at90, with no intermediate scores. By contrast, the full role-separated design distributes scores across the scale. This indicates that separating evaluator roles provides finer diagnostic granularity rather than merely shifting average scores. Reliability Evaluation Repeated Evaluation Improves Stability As shown in Tab. 4, repeated evaluation yields lower per-query score variability than a single evaluation pass. The mean per-query SD decreases from23.81to17.62, and the mean CV decreases from 0.591 to 0.395. Agreement Across Evaluator Models Across evaluator- model configurations, Kimi K2.6 and Qwen3.7-Plus reach Evaluation protocol Mean per-query SD↓ Mean CV↓ Single evaluation pass23.810.591 Repeated evaluation 17.62 0.395 Table 4: Repeated-evaluation reliability check. Repeated eval- uation reduces per-query score variability compared with a single evaluation pass. 90.1%agreement, with substantial chance-corrected agree- ment (Cohen’sκ = 0.641). This suggests that LiveEvalBench generalizes across evaluator models. Conclusion We presented LiveEvalBench, a multi-agent framework for evaluating LLM-generated frontend projects. The framework integrates complementary agents that evaluate a generated project from build, source, and runtime perspectives, an adap- tive evaluation protocol that combines predefined criteria with implementation-grounded checks synthesized per project, and an extensible orchestration design that abstracts each evaluator into a configurable role, allowing new evaluation perspectives to be added with minimal effort. Building on this framework, we constructed a diverse benchmark of 100 real-world web generation queries across difficulty tiers and categories, and evaluate 11 frontier LLMs on this benchmark. The benchmark results show that runtime interaction re- mains the largest source of lost points for current web genera- tion models. Complementary validation studies further show that LiveEvalBench provides reliable and human-aligned evaluation signals. We hope LiveEvalBench offers a new perspective on evaluating generative coding systems and a foundation for the community to build richer, more faithful evaluations. References Austin, J.; Odena, A.; Nye, M.; Bosma, M.; Michalewski, H.; Dohan, D.; Jiang, E.; Cai, C.; Terry, M.; Le, Q.; et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Bean, A. M.; Kearns, R. O.; Romanou, A.; Hafner, F. S.; Mayne, H.; Batzner, J.; Foroutan Eghlidi, N.; Schmitz, C.; Korgul, K.; Batra, H.; Deb, O.; Beharry, E.; Emde, C.; Foster, T.; Gausen, A.; Grandury, M.; Han, S.; Hofmann, V.; Ibrahim, L.; Kim, H.; Kirk, H. R.; Lin, F.; Liu, G.; Luettgau, L.; Magomere, J.; Rystrø m, J.; Sotnikova, A.; Yang, Y.; Zhao, Y.; Bibi, A.; Bosselut, A.; Clark, R.; Cohan, A.; Foerster, J.; Gal, Y.; Hale, S.; Raji, D.; Summerfield, C.; Torr, P.; Ududec, C.; Rocher, L.; and Mahdi, A. 2025. Measuring what Matters: Construct Validity in Large Language Model Benchmarks. In Belgrave, D.; Zhang, C.; Lin, H.; Pascanu, R.; Koniusz, P.; Ghassemi, M.; and Chen, N., eds., Advances in Neural Information Processing Systems, volume 38. Curran Associates, Inc. Beltramelli, T. 2018. pix2code: Generating code from a graph- ical user interface screenshot. In Proceedings of the ACM SIGCHI symposium on engineering interactive computing systems, 1–6. Chen, L.; Zhang, Z.; Tan, H.; Dai, Q.; Hao, Y.; Dong, Z.; and Chen, X. 2025. Distributional LLM-as-a-Judge. In Belgrave, D.; Zhang, C.; Lin, H.; Pascanu, R.; Koniusz, P.; Ghassemi, M.; and Chen, N., eds., Advances in Neural Information Processing Systems, volume 38, 30038–30067. Curran Associates, Inc. Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. D. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Deng, X.; Gu, Y.; Zheng, B.; Chen, S.; Stevens, S.; Wang, B.; Sun, H.; and Su, Y. 2023. Mind2Web: Towards a Generalist Agent for the Web. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., Advances in Neural Information Processing Systems, volume 36, 28091– 28114. Curran Associates, Inc. Design Arena. 2025. Design Arena. https://w.designarena. ai/. Accessed: 2026-07-27. Dubois, Y.; Li, X.; Taori, R.; Zhang, T.; Gulrajani, I.; Ba, J.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Al- pacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. In Advances in Neural Information Processing Systems. Guerdan, L.; Barocas, S.; Holstein, K.; Wallach, H.; Wu, S.; and Chouldechova, A. 2025. Validating LLM-as-a-Judge Systems under Rating Indeterminacy. In Belgrave, D.; Zhang, C.; Lin, H.; Pascanu, R.; Koniusz, P.; Ghassemi, M.; and Chen, N., eds., Advances in Neural Information Processing Systems, volume 38, 112282–112350. Curran Associates, Inc. He, Z.; Hong, W.; Yang, Z.; Pan, Z.; Liu, M.; Gu, X.; and Tang, J. 2026. Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification. arXiv preprint arXiv:2603.26648. Hendrycks, D.; Basart, S.; Kadavath, S.; Mazeika, M.; Arora, A.; Guo, E.; Burns, C.; Puranik, S.; He, H.; Song, D.; et al. 2021. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938. Jain, N.; Han; Gu, A.; Li, W.-D.; Yan, F.; Zhang, T.; Wang, S.; Solar-Lezama, A.; Sen, K.; and Stoica, I. 2025. Live- CodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In Yue, Y.; Garg, A.; Peng, N.; Sha, F.; and Yu, R., eds., International Conference on Learning Representations, volume 2025, 58791–58831. Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues? In Kim, B.; Yue, Y.; Chaudhuri, S.; Fragkiadaki, K.; Khan, M.; and Sun, Y., eds., International Conference on Learning Representations, volume 2024, 54107–54157. Kim, S.; Shin, J.; Cho, Y.; Jang, J.; Longpre, S.; Lee, H.; Yun, S.; Shin, S.; Kim, S.; Thorne, J.; and Seo, M. 2024a. Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models. In International Conference on Learning Representations. Kim, S.; Suk, J.; Longpre, S.; Lin, B. Y.; Shin, J.; Welleck, S.; Neubig, G.; Lee, M.; Lee, K.; and Seo, M. 2024b. Prometheus 2: An Open Source Language Model Specialized in Evalu- ating Other Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pro- cessing, 4334–4353. Miami, Florida, USA: Association for Computational Linguistics. Koh, J. Y.; Lo, R.; Jang, L.; Duvvur, V.; Lim, M.; Huang, P.-Y.; Neubig, G.; Zhou, S.; Salakhutdinov, R.; and Fried, D. 2024. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 881–905. Bangkok, Thailand: Association for Computational Linguistics. Lei, X.; Che, X.; Xiong, J.; Zhang, C.; Huang, Y.; Zhou, C.; Huang, H.; Liu, M.; Zhu, L.; Ye, H.; et al. 2026. WebCom- pass: Towards Multimodal Web Coding Evaluation for Code Language Models. arXiv preprint arXiv:2604.18224. Li, D.; Jiang, B.; Huang, L.; Beigi, A.; Zhao, C.; Tan, Z.; Bhattacharjee, A.; Jiang, Y.; Chen, C.; Wu, T.; Shu, K.; Cheng, L.; and Liu, H. 2025. From Generation to Judg- ment: Opportunities and Challenges of LLM-as-a-judge. In Christodoulopoulos, C.; Chakraborty, T.; Rose, C.; and Peng, V., eds., Proceedings of the 2025 Conference on Empiri- cal Methods in Natural Language Processing, 2757–2791. Suzhou, China: Association for Computational Linguistics. ISBN 979-8-89176-332-6. Li, R.; Zhang, Y.; and Yang, D. 2025. Sketch2Code: Evalu- ating Vision-Language Models for Interactive Web Design Prototyping. In Chiruzzo, L.; Ritter, A.; and Wang, L., eds., Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), 3921–3955. Albuquerque, New Mexico: Association for Computational Linguistics. ISBN 979-8-89176-189-6. Liu, J.; Xia, C. S.; Wang, Y.; and Zhang, L. 2023a. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in neural information processing systems, 36: 21558–21572. Liu, X.; Lei, X.; Wang, S.; Huang, Y.; Feng, A.; Wen, B.; Cheng, J.; Ke, P.; Xu, Y.; Tam, W. L.; Zhang, X.; Sun, L.; Gu, X.; Wang, H.; Zhang, J.; Huang, M.; Dong, Y.; and Tang, J. 2024. AlignBench: Benchmarking Chinese Alignment of Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 11621–11640. Bangkok, Thailand: Association for Computational Linguistics. Liu, Y.; Iter, D.; Xu, Y.; Wang, S.; Xu, R.; and Zhu, C. 2023b. G-Eval: NLG Evaluation Using GPT-4 with Better Human Alignment. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2511–2522. Singapore: Association for Computational Linguistics. Lu, Z.; Yang, Y.; Ren, H.; Hou, H.; Xiao, H.; Wang, K.; Shi, W.; Zhou, A.; Zhan, M.; and Li, H. 2025. WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch. arXiv:2505.03733. Si, C.; Zhang, Y.; Li, R.; Yang, Z.; Liu, R.; and Yang, D. 2025. Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering. In Chiruzzo, L.; Ritter, A.; and Wang, L., eds., Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 3956–3974. Albuquerque, New Mexico: Association for Computational Linguistics. ISBN 979-8-89176-189-6. Tran, H.; Nashold, L.; Krishnan, R.; Bigeard, A.; and Gu, A. 2026. Vibe Code Bench: Evaluating AI models on end-to-end web application development. arXiv preprint arXiv:2603.04601. Verga, P.; Hofstatter, S.; Althammer, S.; Su, Y.; Piktus, A.; Arkhangorodsky, A.; Xu, M.; White, N.; and Lewis, P. 2024. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. arXiv:2404.18796. Wu, X.; Xue, Z.; Yin, D.; Zhou, S.; Chang, K.-W.; Peng, N.; and Wen, Y. 2025. FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi- Modal Feedback. arXiv preprint arXiv:2601.04203. Xiao, J.; Wan, Y.; Huo, Y.; Wang, Z.; Xu, X.; Wang, W.; Xu, Z.; Wang, Y.; and Lyu, M. R. 2024. Interaction2Code: Benchmark- ing MLLM-based Interactive Webpage Code Generation from Interactive Prototyping. arXiv preprint arXiv:2411.03292. Xu, K.; Mao, Y.; Guan, X.; and Feng, Z. 2025. Web-bench: A llm code benchmark based on web standards and frameworks. arXiv preprint arXiv:2505.07473. Yun, S.; Lin, H.; Thushara, R.; Bhat, M. Q.; Wang, Y.; Jiang, Z.; Deng, M.; Wang, J.; Tao, T.; Li, J.; Li, H.; Nakov, P.; Baldwin, T.; Liu, Z.; Xing, E. P.; Liang, X.; and Shen, Z. 2024. Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs. arXiv preprint arXiv:2406.20098. Zhang, C.; Li, Y.; Xu, C.; Liu, J.; Liu, A.; Zhou, C.; Deng, K.; Wu, D.; Huang, G.; Li, K.; et al. 2025. Artifactsbench: Bridging the visual-interactive gap in llm code generation evaluation. arXiv preprint arXiv:2507.04952. Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., Advances in Neural Information Processing Systems, volume 36, 46595–46623. Curran Associates, Inc. Zhou, S.; Xu, F. F.; Zhu, H.; Zhou, X.; Lo, R.; Sridhar, A.; Cheng, X.; Ou, T.; Bisk, Y.; Fried, D.; Alon, U.; and Neubig, G. 2024. WebArena: A Realistic Web Environment for Build- ing Autonomous Agents. In Kim, B.; Yue, Y.; Chaudhuri, S.; Fragkiadaki, K.; Khan, M.; and Sun, Y., eds., Interna- tional Conference on Learning Representations, volume 2024, 15585–15606. Zhu, H.; Zhang, Y.; Zhao, B.; Ding, J.; Liu, S.; Liu, T.; Wang, D.; Liu, Y.; and Li, Z. 2025. Frontendbench: A benchmark for evaluating llms on front-end development via automatic evaluation. arXiv preprint arXiv:2506.13832. Zhuge, M.; Zhao, C.; Ashley, D.; Wang, W.; Khizbullin, D.; Xiong, Y.; Liu, Z.; Chang, E.; Krishnamoorthi, R.; Tian, Y.; Shi, Y.; Chandra, V.; and Schmidhuber, J. 2024. Agent-as-a- Judge: Evaluate Agents with Agents. arXiv:2410.10934. Zhuo, T. Y.; Vu, M. C.; Chim, J.; Hu, H.; Yu, W.; Widyasari, R.; Yusuf, I. N. B.; Zhan, H.; He, J.; Paul, I.; Brunner, S.; GONG, C.; Hoang, J.; Zebaze, A.; Hong, X.; Li, W.-D.; Kaddour, J.; Xu, M.; Zhang, Z.; Yadav, P.; Jain, N.; Gu, A.; Cheng, Z.; Liu, J.; Liu, Q.; Wang, Z.; Hui, B.; Muennighoff, N.; Lo, D.; Fried, D.; Du, X.; de Vries, H.; and Von Werra, L. 2025. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. In Yue, Y.; Garg, A.; Peng, N.; Sha, F.; and Yu, R., eds., International Conference on Learning Representations, volume 2025, 66602–66656. Supplementary Material Overview. Section A provides implementation details for the LiveEvalBench evaluation framework, including evaluator roles, configuration, scoring, and execution environment. Section B documents benchmark- construction details, including the query/checklist schema and role-conditioned synthesis prompts. Section C reports supplementary result analyses for score patterns, evaluator- model agreement, and evaluator-role ablation. Section D presents qualitative case analyses from real evaluation traces. Section E discusses current limitations and future directions. A Implementation Details This section documents the implementation details behind the LiveEvalBench evaluation framework. It describes the default evaluator agents, the configuration schema used by the extensible evaluation infrastructure, the score aggregation procedure, and the compute environment used for the reported runs. The evaluation framework, evaluator configurations, bench- mark construction scripts, and analysis scripts required to reproduce the reported experiments will be publicly released upon publication under a license permitting free research use. A.1 Evaluator Specifications and Prompts Agent Configurations Table 5 provides information on the evaluator agents, detailing their persona, evaluation criteria, and allowed tools. Prompts The prompt templates for the evaluation process are illustrated in two parts. The adaptive evaluation protocol is detailed in Figures 12–14; in practice, to optimize costs, we integrate the synthesis of the adaptive protocol with the synthesis of query-specific criteria, which allows the system to dynamically select evaluation actions based on the project structure. Furthermore, the system prompts for the Build Engineer, Code Engineer, and UI Tester during task execution are provided in Figures 15–17. A.2 Evaluator Configuration Schema Table 6 lists the JSON fields that constitute an evaluator configuration in the released framework. Adding a new eval- uator amounts to writing one such JSON file and dropping it into the agent registry; the orchestrator reads these fields and routes the pipeline accordingly. If a new tool is required, additional code for the new tool needs to be written, but no other engineering work is necessary. A.3 Scoring and Aggregation Details The reported score is normalized to a 90-point scale over five scoring dimensions. The Build Engineer contributes 15 points for build smoothness. The Code Engineer contributes 30 points, split evenly between implementation quality and instruction following. The UI Tester contributes 45 points, split evenly between visual quality and interaction experi- ence. Query-specific criteria are used by the UI Tester as implementation-grounded evidence for the structured report, but they are not reported as a separate top-level score dimen- sion in the current leaderboard. In the adaptive evaluation protocol, each criterion may be decomposed into one or more concrete tasks. The scoring rule from tasks to a criterion is that if any single task fails, the entire criterion is considered failed, and all points assigned to that criterion are forfeited. A.4 Computational Resources and Evaluation Environment For the reported three-run leaderboard, all evaluation agents (Build Engineer, Code Engineer, and UI Tester) use the same evaluator-model configuration across runs; the released evaluation artifacts record the concrete model settings for each run. Compute infrastructure All experiments were conducted on machines with 64 GB RAM and 32 CPU cores. Evaluation was parallelized at the granularity of a single query–model pair, with up to 8 such pairs processed concurrently. Evaluation environment The evaluation pipeline runs on Ubuntu 22.04 LTS with Node.js 20.x, using Playwright 1.40 for browser automation (Chromium 120.0), npm 10.x for pack- age management, and standard build tools (Vite 5.x, webpack 5.x) as detected from each project’s configuration. To reduce setup latency, each project directory is pre-populated with commonly used packages (react,vue,tailwindcss, etc.); only missing dependencies trigger additionalnpm install calls during evaluation. B Benchmark Construction Details This section provides additional benchmark-construction ma- terial. It specifies the released query and checklist schema and includes the role-conditioned synthesis prompts used to rewrite seed requests into benchmark queries. B.1 Query Specification and Checklist Schema To make the format of a LiveEvalBench query and its ac- companying evaluation checklist concrete, we reproduce one representative pair below. The remaining59queries follow the same structure: a short title, a self-contained natural- language description of the desired frontend project, and an enumeration of the visible features expected from any conforming implementation. No framework, library, or file layout is prescribed. Each query is released together with a checklist of evalu- ation criteria. The checklist is stored as a structured record with the following fields: • query_id –Stable identifier of the query, matching the entry in the released query list. • fixed_criteria –The evaluator-scored criteria shared across all queries: build smoothness, implementation quality, instruction following, visual quality, and interaction experience. • query_specific_criteria – A list of records, each containing: EvaluatorPersonaCriteriaTools Build EngineerDeployment engineer who extracts, config- ures, and launches the project Build smoothness: whether the build completed without manual repairs, missing dependencies, or configura- tion errors file-system read/write, shell exec, runtime probe, port man- agement, build trace inspection Code EngineerDeveloper inspecting implementation qual- ity and instruction adherence Implementation quality (readability + robustness), instruction following file-system read/write, one-off shell exec UI TesterEnd user interacting with the running ap- plication through the browser Visual quality, interaction experi- ence, query-specific functional re- quirements browser navigation, DOM in- spection, element interaction (click, type, hover, scroll), screenshot capture Table 5: Default jury of evaluators: persona, evaluation criteria, and allowed tools. Each evaluator is configured through the extensible evaluation infrastructure. ∗ name — a short label for the criterion. ∗ description — a one- or two-sentence explana- tion of the feature being checked. The query-specific criteria are predefined per query and held fixed across all models, ensuring that every implementa- tion of the same query is evaluated against a consistent set of criteria. B.2 Role-Conditioned Synthesis Prompts The role-specific prompt templates used for query construc- tion are given in Figures 18–23. C Supplementary Result Analysis This section reports supplementary analyses that support the main empirical findings. It includes category-level score patterns, dimension correlations, ranking-significance diag- nostics, evaluator-model agreement details, and evaluator-role ablation details. C.1 Score Pattern Analysis The result reveals two insights. First, several runtime-facing criteria exhibit strong mutual dependence: Build Smoothness correlates strongly with Visual Quality (r ≈0.74) and In- teraction Experience (r ≈0.74), while Visual Quality and Interaction Experience are also correlated (r ≈0.76). This suggests that fragile deployment often co-occurs with weaker rendered and interactive behavior. Second, the code-side di- mensions remain much less redundant with the runtime-facing ones: Implementation Quality and Instruction Following are only moderately correlated (r ≈0.40), and most cross-family pairs are lower than the runtime-facing correlations. This overall pattern provides quantitative justification for retaining all five dimensions rather than collapsing them into a single weighted scalar: each dimension captures a distinct facet of implementation quality that the others do not. Figure 7 examines how model performance varies across query categories. While all models exhibit some category- dependent variation in their scores, the overall pattern reveals that , , and Data Visualization (56.5), UI Component (57.7), and Web App (57.8) form the lowest-scoring group on average, while Game is the highest aggregate category (62.9), suggest- ing that compact but interaction- or data-heavy frontend tasks remain challenging even for strong models. Other categories show more model-specific strengths and weaknesses, with certain models performing markedly better in specific cate- gories (e.g., layout-heavy or interaction-heavy queries) while struggling in others. C.2 Evaluator-Model Agreement Details Table 7 reports a compact evaluator-model agreement check, showing that Kimi K2.6 and Qwen3.7-Plus produce closely matched judgments under the same evaluation setup. C.3 Evaluator-Role Ablation Details This section provides additional diagnostics for the evaluator- role ablation. It extends the main-text score-distribution com- parison with summary statistics for the same 70-case subset. The detailed diagnostics in Table 8 show that the single- evaluator ablation loses graded diagnostic behavior. On the 70-case subset, it assigns 21 cases a score of 0 and 49 cases a score of 90, with no intermediate scores. By contrast, the full role-separated design distributes scores across the full scale. D Cases Analysis This section provides qualitative examples from real evalu- ation traces. The cases illustrate both a successful browser- based verification trajectory and a representative failure mode caused by missing baseline observations. To better understand the behavior of our evaluation agent, we present two representative cases drawn from real evaluation traces: a successful case (Figure 10) in which the agent correctly verifies a non-trivial canvas-based interaction, and a failure case (Figure 11) in which the agent reaches an incorrect verdict due to a missing baseline observation. These two cases jointly illustrate both the strengths of the agent in handling complex frontend scenarios and the typical hallucination patterns that arise when its observation strategy is incomplete. Successful case: Verifying canvas-based interaction on a gomoku board The first case (Figure 10) corresponds to a gomoku (five-in-a-row) board, where the criterion under FieldPurpose id, name, role, stageEvaluator identity, abstract role (builder / evaluator), and pipeline stage (build / evaluate). depends_onUpstream evaluators that must succeed first; enforces the build-gated short-circuit. description, system_promptPersona and detailed behavioral instructions, including hard role boundary, mandatory action sequences, and bounded-repair rules. allowed_toolsWhitelisted tool ids; any tool call outside this list is denied at runtime. criterion.scoring_modee.g., pass_fail_na. criterion.dimensions[]Each dimension declares an id, instruction, weight (recorded but not used in current scoring), and ascoringblock listing per-criterion and per-subcriterion criterion items used by the Planner. check_decomposition_policyA hard override on the planner’s generic heuristic; for example, “one check per criterion” for the Code Engineer or “one check per interactive element, capped at five” for the UI Tester. runtimePer-evaluator budgets: max_steps, max_step_seconds, max_total_seconds. outputOutput requirements: require_evidence, allow_not_applicable. scoring.max_scoreMaximum score contribution to the jury total; uniformly partitioned across the evaluator’s criterion items at scoring time. scoring.task_aggregation_modee.g., strict (any failing check fails the criterion item). enabled, tags, versionRegistry-level metadata. Table 6: JSON fields of the evaluator configuration used by the extensible evaluation infrastructure. The full schema instances for the three default evaluators are included in the released evaluator-configuration files. Evaluator-model pairAgreement↑ Cohen’s κ↑ Kimi K2.6 vs. Qwen3.7-Plus90.1%0.641 Table 7: Evaluator-model agreement for Kimi K2.6 and Qwen3.7-Plus. Agreement denotes the percentage of matched judgments; Cohen’sκmeasures chance-corrected agreement. Variant 0 1–29 30–59 60–89 90 Mean SD Median Full1841728 3 43.6 31.449.5 1-agent 21000 49 63.0 41.590.0 Table 8: Score-distribution diagnostics for the evaluator-role ablation on the 70-case subset. Bins denote total scores on the 90-point scale. evaluation is whether a black piece and a white piece are rendered with clearly distinct colors after being placed on the board. This is a particularly challenging scenario because the board and the pieces are drawn on an HTML<canvas> element, meaning that the agent cannot rely on DOM inspec- tion or accessibility-tree traversal to locate or verify game pieces—two of the primary tools used in most other evaluation tasks. Instead, the agent must reason directly over rendered pixels and interact with the page via spatial clicks, which represents one of the more complex frontend scenarios our framework needs to support. The agent first navigates to the target page and callsget_page_contexttogether with inspect_last_screenshotto build an initial under- standing of the layout, identifying the grid structure and the existing white piece on the board. Crucially, before issuing any real click, the agent invokespreview_click_atto estimate where on the canvas a given coordinate would land, effectively simulating the click in advance to confirm that it falls on a valid grid intersection. Only after this preview step does the agent perform the actualclick_at, followed by anotherinspect_last_screenshotcall to verify that the click indeed produced a visible black piece at the expected location. The same preview-then-click-then-verify pattern is repeated for the second piece, after which the agent compares both pieces in the final screenshot and concludes that they are visible, distinct in color, and correctly rendered. This case demonstrates that, even in canvas-only scenarios where DOM-level signals are unavailable, the combination of preview clicks and post-action visual inspection allows the agent to reach a reliable verdict. Failure Case: Misjudging hover feedback due to miss- ing baseline The second case (Figure 11) involves veri- fying whether the “Add Segment” button on a daily pres- ence timeline page exhibits visible hover feedback (e.g., color, shadow, or border change). After navigating to the page and retrieving its context, the agent immediately in- vokeshover_elementon the button and then calls inspect_last_screenshotto look for a visual change. Seeing no clear difference, it repeats the hover-and-inspect cycle once more before submitting a Failed verdict. The underlying mistake is methodological rather than perceptual: the agent never captured a baseline of the button in its non- hovered state—neither an initial screenshot nor its pre-hover styling viaget_page_context—so when examining the post-hover screenshots it has no reference point for compari- son, rendering any subtle color or shadow change effectively invisible. A hover-feedback check fundamentally requires Figure 7: Score by query category×model on the 90-point scale. Categories are sorted top-to-bottom by aggregate score ascending. Cells are annotated with the per-cell sample size n. Figure 8: Dimension–dimension correlation, Pearson across models. Each dimension is treated as an 11-dimensional vector indexed by model. a before/after comparison, and without an explicit baseline the agent is structurally unable to verify the criterion. We acknowledge that such occasional hallucinations remain a current limitation of our system, where the agent may reach a verdict on a state change without having observed the original state. This limitation could be further alleviated in future work by introducing more fine-grained rule-based checks that enforce baseline capture prior to any state-altering action, as well as incorporating a dedicated reviewer agent responsible for double-checking the verdict. E Discussion and Limitations In this section, we reflect on the current scope of LiveEval- Bench and outline several directions in which we plan to extend it. Our human-agreement study shows that LiveEvalBench aligns well with human majority judgments overall, while agreement on interaction experience still leaves room for improvement. We attribute this to the difficulty of judging multi-step interactive behavior. This can be mitigated through prompt engineering, self-consistency decoding, or stronger reasoning backbones, all of which are drop-in replacements under our pipeline. Looking ahead, we see several promising directions for extending this work. First, as web-generation capabilities continue to evolve, the task suite and category coverage of LiveEvalBench can be continually expanded, so that it remains challenging and avoids the saturation that has limited prior static benchmarks. Second, multi-agent judging schemes are worth exploring as a future direction, where several judge agents deliberate or vote on each criterion, and it remains an open question how much such designs can further improve alignment with human ratings. Finally, as agentic generation becomes increasingly common in real-world web- development workflows, a natural extension is to apply our framework to such settings, where models iteratively refine their outputs or invoke external tools during generation. Query Act as a senior frontend developer. Implement the following requirements. All output must be frontend code (Vue, React, or static HTML). When multiple files are generated, make sure every necessary configuration file (such as package.json, tsconfig. json, etc.) is included without omission. For each file, clearly mark the filename and path. The task requirements are: I need a high-impact data visualization dashboard layout. The centerpiece should be an interactive 3D map---think smooth rotation, zoom capabilities , and glowing markers for data points. Complement this with 3D bar charts that feature distinct depth and perspective. Use a dark, futuristic color palette (deep blues or blacks) with vibrant neon accents to make the charts stand out. Ensure the animations are fluid, specifically adding smooth entrance transitions and responsive hover effects for a polished user experience. Category: data_visualization Difficulty: L3 Granularity: functional Fixed Criteria (shared across all 100 queries) • Build Smoothness • Implementation Quality • Instruction Following • Visual Quality • Interaction Experience Query-Specific Criteria 1. Interactive 3D map functionality and marker visualiza- tion. The 3D map must support smooth rotation and zoom ca- pabilities, and data points must be represented by glowing markers. 2. Fluid animation and responsive hover effects. The dashboard must feature fluid animations, including smooth entrance transitions and responsive hover effects for interactive elements. 3. 3D bar chart depth and perspective rendering. The 3D bar charts must be rendered with distinct depth and perspective to provide a three-dimensional appearance. 4. Dark futuristic color palette with neon accents. The dashboard must utilize a dark, futuristic color palette consisting of deep blues or blacks, complemented by vibrant neon accents. Figure 9: Example query (left) and its accompanying evaluation checklist (right). The checklist contains fixed criteria shared across all 100 queries and query-specific criteria tailored to each query’s functional requirements. This example belongs to the data_visualization category (difficulty: L3). !"# $" #% #%"! &'!#(!")'*+,"'( +( !")')-)'(!"# .(!")'/#!0 1#2"3#!!)%)(#%0)!45666 7 89"!2:"(!4 .(!##'")-)'!': :2%),;;;<0!#= > "9'!#4 &?92'@!)!0"A8 !&@#'! !)9#=#"9,%@8,#3 !0#!0)@#B)CB)9)= D 8)#:;&"9#3"'#@0"!8)#: @"!08%#(=#':@0"!,"( !0#!) (#')'!0('; &?9')! 0)@"!0) %:8 8 "%!A&E !@#'!)9!0"'3 8#"(!0#!@)=)&(#'!"! ) !#':%#'; F#"(B)9)= 8)#:- '(!")'#%"9,%9'!#!")' G0"!F#"(B)9)= 8)#::",%# $"-8%#(=#':@0"!3#9,"(:",%# $"-8%#(=#':@0"!,"(':"'3 H /%#(#8%#(=,"(#':#@0"!,"( )'!08)#:#':2"-!0"()%) #:"!"'(!#':()(!%'::; 3!I,#3I()'!+! J!"!%J4JB)9)= CK"2"'#L)@DJA J %J4J0!!,4M%)(#%0)!4NO6PMJA J#(("8"%"!I!J4Q;;;R "',(!I%#!I('0)! <03#98)#:"%)(#!:"'!0('!)- !0('A:"(!%8%)@!0!+!J '! < '4F%#(=;J&!"#> #3":()'"!"'3)- ST()% 9'#':ST)@)-"'!(!"'3%"'; <0"( '!%)'@0"!,"(C:("(%D %)(#!:#!!0P!0()% 9'-)9!0%U#': !0N!0)@-)9!0!),; ,2"@I(%"(=I#! (%"(=I#! "',(!I%#!I('0)! VA#8%#(=,"("2""8%)'!08)#:;&!" %)(#!:)'!0%U":)-!03":A ,("W(#%%)'!0-) !02!"(#%%"'-)9 !0%U#':!0"+!00)"X)'!#%%"'-)9!0 !),; ,2"@I(%"(=I#! ) () %:(%"(=)'!0"'!(!")' "99:"#!%!)!0"30!)-!0+"!"'38%#(= ,"(C#!!0#90)"X)'!#%%2%D)#' )!09,!> #)'!08)#:!)()'!"' !03#9; (%"(=I#! "',(!I%#!I('0)! VA8)!08%#(=#':@0"!,"(#2""8% )'!08)#:#':#(%#%:"!"'3 "0#8% 8!0"()%)#':0#:"'3; &0#2 ((- %%,%#(:8)!0# 8%#(=,"(#':#@0"!,"()'!03#9 8)#:;<0('0)!()'W9!0#!8)!0 ,"(#2""8%A:"!"'(!"'()%)A#': ()(!%'::)'!08)#:; Figure 10: Successful case of evaluation agent judging web function. !"# $" #% #%"! &'!#(!")'*+,"'( +( !")')-)'(!"# .(!")'/#!0 1#2"3#!!)%)(#%0)!45666 7 89"!2:"(!4 .(!##'")-)'!': :2%),;;;;<0!#= > "9'!#4 &?#'!!)9#=)9!0"'3!0#! 0)?#!"9%"'-)!0:# ?"!0()%);@"=A:"B'! ,#!)-!0:#0) %:0#2 :"B'!()%)!)0)??0' )9)'?#!0)')!; #' ) 0%,99#=!0"C&!0"'= "!0) %:82""8%)'# ?8,#3; D#"%!"9%"'2" #%"E#!")')'?8,#3 )%)F()::!"9%"'39'!-),'( G)2-:8#(=)'.::739'!8 !!)' H /%#(#8%#(=,"(#':#?0"!,"( )'!08)#:#':2"-!0"()%) #:"!"'(!#':()(!%'::; 3!I,#3I()'!+! J!"!%J4JD#"%/'(<"9%"'JA J %J4J0!!,4K%)(#%0)!4LMMMKJA J#(("8"%"!I!J4N;;;O "',(!I%#!I('0)! P#:)'!0,)2"::('0)!A"!"Q')! ,)"8%!):!9"'Q"-!0J.::739'!J 8 !!)'0)?#2" #%(0#'3 ,)'0)2; <0"9#3"#!#!"(,'!#!")')-!0 &A#':!0"')2" #%"':"(#!)R (0## ( ))0)2!#!!%"'3S!) 33!0)? !08 !!)'#(!!) "'!#(!")'; &0#2#!!9,!:!)0)2)2!0 T.::739'!T8 !!)'9 %!",%!"9;U0"% !08 !!)'"(%#%"'!#(!"2R"!"#8 !!)' %9'!SA!0"'))82#8%2" #% (0#'3R (0##()%)0"VA0#:)?A) 8):(0#'3S"'!0,)2"::('0)! ?0'!00)2#(!")'"#,,%":;.!0 > "9'!,("W(#%%#=-)#2" #% !#!(0#'3"':"(#!"'3"'!#(!"2"!A#': ')'")82:A&9 !()'(% :!0 -#! "9""'3)')!"9,%9'!:# +,(!:; %"(=:"()2#8"%"!)-!#! W%!8 !!)' 0)2I%9'! 0)2I%9'! "',(!I%#!I('0)! P#:)'!0,)2"::('0)!A"!"Q')! ,)"8%!):!9"'Q"-!0J.::739'!J 8 !!)'0##0)2B(!;<0( )"')! 2""8%)2!08 !!)'A#':!0"')2" #% "':"(#!")'R (0##()%)(0#'3) 0#:)?S!0#!?) %: 33!#'#(!"20)2 !#!; 3!I,#3I()'!+! J!"!%J4JD#"%/'(<"9%"'JA J %J4J0!!,4K%)(#%0)!4LMMMKJA J#(("8"%"!I!J4N;;;O Figure 11: Failure case of evaluation agent judging web function. You are refining top-level test goals into executable sub tasks for an autonomous browser evaluator. agent_identity_block taskability_block User query: user_query All fixed main tasks metadata: fixed_main_tasks_block query_generation_block Functional requirements: requirements_block Page accessibility context / visible regions: page_context_block Relevant source code excerpt: ‘ source_snippet ‘ Your goal: for each main task below, generate a small set of executable sub tasks. Priority rules: - Preserve the current agent’s role, stage, and persona while decomposing each task. - Every sub task must stay inside the current agent’s allowed tool and capability boundary. - For review-stage or implementation-quality agents, derive sub tasks from implementation quality, maintainability, robustness, and engineering risk questions that the agent can actually investigate. - For review-stage or implementation-quality agents, do NOT decompose the task into literal PRD/spec compliance checks or word-for-word audits against the user query. - Use source code to understand implementation structure and likely risk areas, not to produce line-by-line specification conformance checks. - Determine sub task count from the actually rendered UI structure in page_context first. Count only clearly separate, user-meaningful modules or regions that can be judged independently. - Default to ONE sub task per main task when a single focused pass can answer it. Only return 2 or more when page_context shows multiple distinct rendered modules and each module needs its own evidence collection. - Do NOT create extra sub tasks just because a goal is broad. Breadth alone is not a reason to split. - For build-path, log-driven, source-review, or implementation-review tasks, prefer a single sub task unless there are clearly separate root-cause areas that cannot be judged together. Do not force a page-region split for evidence-centric review tasks. Figure 12: Prompt template for the adaptive evaluation protocol that dynamically selects evaluation actions based on project structure.(1/3) - ** Decomposition policy precedence: ** if the agent identity block above contains a ‘ Subtask decomposition policy‘ section, follow that policy verbatim and IGNORE the per- area defaults below for fixed_dimension tasks. The agent-specific policy is a hard override, not a hint. - ** Default for fixed_dimension origin tasks ("origin": "fixed_dimension") WHEN no agent policy is provided: ** These wrap a rubric dimension with specific objective checks ( detailed in the goal text). Split sub tasks by ** domain-specific implementation areas ** , not by individual checks. For each domain area, aggregate all relevant checks into that subtask’s ‘success_criteria‘. A dimension with 6-8 checks should typically decompose into 2-3 domain-area subtasks, each bundling multiple checks. (Skip this default entirely if an agent policy specifies otherwise.) - Every check in the dimension must be covered by at least one subtask’s ‘ success_criteria‘, regardless of decomposition strategy. Whether a subtask owns 1 check or many is determined by the active decomposition policy (agent override vs. default). - ** Default split heuristic WHEN no agent policy is provided: ** typical splits are by major UI region (header, main content, sidebar, footer), functional flow (search, checkout, form submission), or behavioral aspect (navigation responsiveness, data display correctness, form validation). An agent policy may instead require splitting per rubric check, per interactive DOM element, or per unit test --- in those cases, follow the agent policy. - ALL sub tasks MUST be independent --- they will run in parallel. Never create dependencies between sub tasks. - Every sub task must target a different functional area, visual region, or aspect of the page. No two sub tasks should test the same thing. - Keep the total number of sub tasks small and high value across ALL main tasks. Usually 1-2 independent sub tasks per query-specific main task, or 2-4 sub tasks per fixed_dimension main task when the dimension has multiple distinct checks. - Each sub task must be executable by this agent and should describe what area/feature to exercise and what to verify. - All returned fields must be written in English, even if the user query, requirement text, source code comments, or visible page labels are in another language. - title, task_text, area_label, expected_signals, preconditions, and success_criteria must all be English-only. Figure 13: Prompt template for the adaptive evaluation protocol that dynamically selects evaluation actions based on project structure.(2/3) Return ONLY a JSON object (dict) where each key is a main_task_id and the value is an array of sub tasks for that main task. Every main_task_id from the input must appear as a key. Example format: "dimension::layout": [ "title": "Verify header layout", "task_text": "...", "kind": "independent", "area_label": "header region", "expected_signals": ["..."], "preconditions": [], "success_criteria": ["..."], "needs_clean_state": true, "can_run_parallel": true, "depends_on_indexes": [] ], "query::01": [ ... ] Each sub task must have: title: short title task_text: 2-5 sentence executable description kind: always "independent" --- staged_flow is not allowed area_label: short label for the region/feature this sub task targets expected_signals: array of 1-4 observable outcomes preconditions: array of 0-3 setup conditions success_criteria: array of 1-4 checks needs_clean_state: true can_run_parallel: true depends_on_indexes: [] --- sub tasks have no dependencies Return only valid JSON object, no explanation. Figure 14: Prompt template for the adaptive evaluation protocol that dynamically selects evaluation actions based on project structure.(3/3) You are ** agent_name ** --- a specialised frontend evaluation agent. ## Persona system_prompt ## Application Under Test app_url ## Rubric Scoring mode: ** scoring_mode ** na_policy You must evaluate the application along the following dimensions: dimensions_block ## Allowed Tools You may ONLY use the tools listed below. Any other tool call will be denied. tools_list ## Tool Usage Guidelines tool_usage_guidelines ## Submission When you have gathered enough evidence, call ** submit_group_verdict ** with one entry per dimension. Each entry must include: - ‘standard_id‘: the dimension ** id ** from the rubric above - ‘verdict‘: one of ‘passed‘, ‘failed‘na_verdict_option - ‘reason‘: concise, evidence-based explanation - ‘evidence‘: array of concrete evidence items (tool output snippets, observed state changes) - ‘checks‘: required whenever the dimension defines objective checks - ‘subcriteria‘: required whenever the dimension defines subjective subcriteria Every dimension must have an explicit ‘verdict‘. Do not omit it, even in score-first mode. Before you submit, perform a consistency check over your own prior tool results. If a previous tool result already confirmed a fact, you MUST NOT contradict it in the final verdict unless you explicitly explain why the earlier evidence was insufficient or superseded by newer evidence. For better downstream analysis, also include these optional fields per dimension when available: - ‘confidence‘: ‘high‘ | ‘medium‘ | ‘low‘ - ‘severity‘: ‘critical‘ | ‘major‘ | ‘minor‘ | ‘none‘ - ‘observations‘: short bullet-style strings of what you observed - ‘recommendation‘: one short fix suggestion Figure 15: Prompt template for evaluation agents (Build Engineer, Code Engineer, UI Tester) during task execution. (1/3) Required item shapes when present: - ‘checks‘: ‘"check_id":"...","status":"passed|failed|not_applicable","reason":"..."," evidence":[]‘ - ‘subcriteria‘: ‘"subcriterion_id":"...","rating":"good|ok|poor","reason":"..."," evidence":[]‘ Correct submission examples: submission_examples_block You have a budget of ** max_steps steps ** . Use them wisely. Begin your evaluation now. ============================================================================= REAL AGENT EXAMPLE (software_tester): ============================================================================= When rendered for software_tester agent, the persona block becomes: "You are ** Software Tester ** --- a specialised frontend evaluation agent. ## Persona You are a software test engineer. Your goal is to uncover the largest number of real defects with the fewest steps. You must cover the main flow, edge scenarios, and recovery from failure. Do not give conclusions based only on inspection without testing. Prioritize recording reproducible paths as action sequences plus observed results, and evaluate failures separately from recovery capability. When possible, provide check/subcriteria-level outputs to help the system score objectively." ============================================================================= TOOL USAGE GUIDELINES (agentic section): ============================================================================= - Be thorough but efficient (max max_iterations steps) - Prefer text-based or role selectors (e.g. ‘text=Submit‘, ‘role=button[name=’OK’]‘). - After any page-mutating action, call ‘get_page_context()‘ to verify state. - Do NOT navigate to external URLs --- stay within the application. - Do NOT retry the same action more than 3 times if it fails or produces no change. - Treat an identical tool call with identical parameters as a retry unless you gathered new page evidence first. - Before submitting, reconcile your final verdict with your prior tool results; do not contradict an earlier confirmed observation without explicitly explaining why it no longer applies. - If ‘inspect_last_screenshot‘ or another tool confirmed a visual fact, carry that fact into your verdict reasoning instead of restating visual uncertainty. - Do not make broad visual failure claims from a narrow screenshot question. Use concrete evidence tied to the specific claim. - For visual failures, cite the exact observed issue in ‘reason‘ and include the supporting screenshot analysis or page-context evidence in ‘evidence‘. Figure 16: Prompt template for evaluation agents (Build Engineer, Code Engineer, UI Tester) during task execution. (2/3) Coordinate click guidance (when available): - For coordinate clicks, ALWAYS preview before ‘click_at‘ or ‘dblclick_at‘. Treat ‘ preview_click_at‘ as an annotated-image inspection step: inspect the attached preview image yourself. You may provide a concrete visual question, but if you omit it the tool will still ask a default question about the likely reaction to clicking the marked point. Use ‘dblclick_at‘ only when the intended gesture is explicitly a double- click. If the intended gesture is a double-click, use ‘dblclick_element‘ or ‘ dblclick_at‘ instead of sending the same click twice. After a successful coordinate click, do not repeat the identical action unless you first gathered fresh page evidence. Figure 17: Prompt template for evaluation agents (Build Engineer, Code Engineer, UI Tester) during task execution. (3/3) # Role You are an AI assistant designed to rewrite user queries into the persona of a ** Beginner ** with ** Abstract Granularity ** . # Persona Definition - ** Persona: ** Beginner (Learning-focused, simple examples, unsure of terminology). - ** Granularity: ** Abstract (States a high-level goal but provides almost no details. Relies on the model to infer features and styles). # Task Rewrite the user’s input query to sound like a complete beginner. The rewritten query should: 1. Avoid technical jargon (e.g., instead of "database," use "place to save info"). 2. Be vague about specific features or layout. 3. Express a simple, high-level desire or goal. 4. Sound enthusiastic but unsure of * how * to achieve the result. # Format Start your response strictly with ‘#answer‘. # Examples ** Input: ** "Create a responsive React navigation bar with dropdowns." ** Output: ** #answer I want to make a website menu that looks good on my phone. It needs those little lists that pop down when you click something. I’m new to this, so can you show me a simple way to do it? ** Input: ** "Build a Python script to scrape stock prices from Yahoo Finance." ** Output: ** #answer I want to make a program that automatically gets stock numbers from the internet, maybe from Yahoo? I just want to see the prices without looking them up manually. ** Input: ** [USER INPUT HERE] Figure 18: The role-conditioned synthesis prompt (Beginner). # Role You are an AI assistant designed to rewrite user queries into the persona of a ** Designer ** with ** Functional Granularity ** . # Persona Definition - ** Persona: ** Designer (Visual-focused, cares about UX/UI, layout, and aesthetics, but less about specific code stacks). - ** Granularity: ** Functional (Lists specific features, content, or rough style preferences, but implementation details are flexible). # Task Rewrite the user’s input query to sound like a UI/UX Designer. The rewritten query should : 1. Focus heavily on visual style (colors, spacing, typography) and user experience ( animations, hover states). 2. Describe specific functional elements (e.g., "needs a hero section," "masonry grid," " sticky header"). 3. Leave the specific coding logic (e.g., state management, backend) flexible. 4. Use descriptive, visual language (e.g., "modern," "sleek," "glassmorphism"). # Format Start your response strictly with ‘#answer‘. # Examples ** Input: ** "Create a login form." ** Output: ** #answer I need a stylish, modern login interface. It should be centered on the screen with a subtle drop shadow and rounded corners (approx 16px). Include input fields for email and password with floating labels. The primary button should use a gradient purple color, and I want a smooth fade-in animation when the page loads. ** Input: ** "Make a blog home page." ** Output: ** #answer Please design a blog landing page with a heavy focus on typography. The top needs a large , bold Hero section with a background image. Below that, I want a grid layout for articles that uses whitespace generously. The cards should lift up slightly on hover. Keep the color palette minimal---black, white, and one accent color like teal. ** Input: ** [USER INPUT HERE] Figure 19: The role-conditioned synthesis prompt (Designer). # Role You are an AI assistant designed to rewrite user queries into the persona of a ** Product Manager ** with ** PRD Granularity ** . # Persona Definition - ** Persona: ** Product Manager (Feature-focused, thinks in terms of MVP, business logic, and user stories). - ** Granularity: ** PRD (Strict constraints on UI layout, specific data structures, exact logic flows. Little room for interpretation). # Task Rewrite the user’s input query to sound like a Product Manager writing a spec. The rewritten query should: 1. Define clear requirements and acceptance criteria. 2. Specify exact fields, buttons, and text copy required. 3. Outline the business logic (e.g., "If X is empty, show error Y"). 4. Be structured, professional, and directive. # Format Start your response strictly with ‘#answer‘. # Examples ** Input: ** "Make a todo list app." ** Output: ** #answer ** Feature Request: Task Management Module (MVP) ** Figure 20: The role-conditioned synthesis prompt (Product Manager) 1 / 2. ** Requirements: ** 1. ** Header: ** Title "My Tasks" with a "Add New" button aligned right. 2. ** Task List: ** Display a list of task items. Each item must show: * Checkbox (left) * Task Title (truncated after 50 chars) * Delete Icon (red trash can, right) 3. ** Empty State: ** If list is empty, display text "No tasks pending." 4. ** Logic: ** * Clicking checkbox toggles ‘strikethrough‘ style. * New tasks are added to the top of the list. * Data must persist on page refresh (Local Storage). ** Input: ** "Create a user profile card." ** Output: ** #answer ** Component Spec: User Profile Widget ** ** Layout Constraints: ** * Fixed width: 300px. * Border: 1px solid #E0E0E0. ** Content Fields: ** 1. ** Avatar: ** Circular, 64x64px. Fallback image required if null. 2. ** Name: ** Bold, 18px font size. 3. ** Role: ** Grey text, displayed below name. 4. ** Status Indicator: ** Green dot if ‘isOnline=true‘, Grey dot if ‘false‘. ** Action: ** * "View Profile" button at the bottom, full width. ** Input: ** [USER INPUT HERE] Figure 21: The role-conditioned synthesis prompt (Product Manager) 2 / 2.. # Role You are an AI assistant designed to rewrite user queries into the persona of a ** Developer ** with ** PRD Granularity ** . # Persona Definition - ** Persona: ** Developer (Technical request, specific stack/component, expects clean code ). - ** Granularity: ** PRD (Pixel-perfect instructions, specific hex codes, exact data structures, library mandates). # Task Rewrite the user’s input query to sound like a Senior Developer defining a task. The rewritten query should: 1. Specify the exact tech stack (e.g., React, TypeScript, Tailwind CSS). 2. Define specific interfaces, props, or database schemas. 3. Dictate specific styling classes or Hex codes (e.g., "bg-slate-900", "#1a202c"). 4. Focus on implementation details like component structure and state management. # Format Start your response strictly with ‘#answer‘. # Examples ** Input: ** "Make a gallery with images." ** Output: ** #answer Create a ‘PhotoGrid‘ component using ** React ** and ** Tailwind CSS ** . Figure 22: The role-conditioned synthesis prompt (Developer) 1 / 2. ** Props Interface: ** ‘typescript interface Photo id: number; url: string; caption?: string; interface Props items: Photo[]; ‘ ** Implementation Details: ** 1. Use CSS Grid: ‘grid-cols-1 md:grid-cols-3 gap-4‘. 2. Images must use ‘object-cover‘ and have a ‘rounded-lg‘ class. 3. Implement lazy loading for performance. 4. On click, open a modal (implementation details for modal omitted, just add the handler ). 5. Background color strictly ‘#f8f9fa‘. ** Input: ** "Create a dark mode toggle." ** Output: ** #answer Implement a ‘ThemeSwitcher‘ hook and button using React Context API. ** Technical Specs: ** 1. State: Use ‘localStorage‘ to persist preference key ‘’theme’‘ (’dark’ | ’light’). 2. Logic: Update the ‘html‘ class list by adding/removing ‘’dark’‘. 3. ** UI Component: ** * Use a lucide-react icon (‘Moon‘ / ‘Sun‘). * Button styles: ‘p-2 rounded-full hover:bg-gray-200 dark:hover:bg-gray-700‘. * Transition: ‘transition-colors duration-200‘. ** Input: ** [USER INPUT HERE] Figure 23: The role-conditioned synthesis prompt (Developer).