Paper deep dive
SynWeaver: Website-Prior Task and Trajectory Co-Synthesis for Web Agents
Ruitao Wang, Yuwen Hao, Menglin Yang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/14/2026, 4:19:44 AM
Summary
The paper introduces SynWeaver, a framework for synthesizing task and trajectory data for web agents by constructing website maps and performing collaborative task-trajectory refinement to ensure semantic alignment and executability.
Entities (10)
Relation Signals (8)
SynWeaver → evaluatedon → WebVoyager
confidence 95% · Experiments on WebArena and WebVoyager demonstrate that SynWeaver consistently outperforms
SynWeaver → evaluatedon → WebArena
confidence 95% · Experiments on WebArena and WebVoyager demonstrate that SynWeaver consistently outperforms
SynWeaver → uses → website map
confidence 95% · SynWeaver first performs structured website exploration and constructs a website map
SynWeaver → finetunes → Qwen3-VL-8B-Instruct
confidence 90% · SynWeaver achieves the best overall WebArena success rate on both Qwen3-VL-8B-Instruct
SynWeaver → finetunes → InternVL3-8B
confidence 90% · SynWeaver achieves the best overall WebArena success rate on both Qwen3-VL-8B-Instruct and InternVL3-8B
SynWeaver → outperforms → SynthAgent
confidence 90% · SynWeaver consistently outperforms strong synthesis baselines
Gemini 3.1 Pro → servesas → teacher_model_M2
confidence 85% · and Gemini-3.1-Pro as the stronger teacher M2
Gemini 3 Flash → servesas → teacher_model_M1
confidence 85% · We use Gemini-3-Flash as the base teacher M1
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Web agents often struggle to generalize to unseen websites because they lack website-specific supervision. Recent exploration-based data synthesis methods reduce manual annotation, but they still face two key limitations: they often fail to cover the full functionality of a website, and without sufficient website prior knowledge, they tend to propose hallucinated tasks, which in turn limits the diversity and efficiency of downstream trajectory synthesis. We present \textbf{SynWeaver}, a website-prior task-trajectory co-synthesis framework designed to address these challenges. SynWeaver first performs structured website exploration and constructs a website map that covers a broad set of functionally distinct page states and executable interactions on the target website. It then derives page-level and transition-level supervision from this map to train a UI-aware model with website-specific priors, enabling more grounded task proposals. Finally, SynWeaver performs collaborative task-trajectory synthesis, jointly updating the task and execution trajectory when they become inconsistent, and then verifies and repairs the collected results to produce executable, semantically aligned supervision. Experiments on WebArena and WebVoyager demonstrate that SynWeaver consistently outperforms strong synthesis baselines and yields more effective supervision for both in-domain and out-of-domain generalization.
Tags
Links
- Source: https://arxiv.org/abs/2608.12429v1
- Canonical: https://arxiv.org/abs/2608.12429v1
Trouble viewing inline? Open PDF directly →
Full Text
114,087 characters extracted from source content.
Expand or collapse full text
SynWeaver: Website-Prior Task and Trajectory Co-Synthesis for Web Agents Ruitao Wang and Yuwen Hao and Menglin Yang † Hong Kong University of Science and Technology (Guangzhou) rwang356, yhao481@connect.hkust-gz.edu.cn menglinyang@hkust-gz.edu.cn † Corresponding author. Abstract Web agents often struggle to generalize to unseen websites because they lack website- specific supervision. Recent exploration-based data synthesis methods reduce manual anno- tation, but they still face two key limitations: they often fail to cover the full functionality of a website, and without sufficient website prior knowledge, they tend to propose hallucinated tasks, which in turn limits the diversity and efficiency of downstream trajectory synthesis. We present SynWeaver, a website-prior task- trajectory co-synthesis framework designed to address these challenges. SynWeaver first per- forms structured website exploration and con- structs a website map that covers a broad set of functionally distinct page states and exe- cutable interactions on the target website. It then derives page-level and transition-level su- pervision from this map to train a UI-aware model with website-specific priors, enabling more grounded task proposals. Finally, Syn- Weaver performs collaborative task-trajectory synthesis, jointly updating the task and ex- ecution trajectory when they become incon- sistent, and then verifies and repairs the col- lected results to produce executable, semanti- cally aligned supervision. Experiments on We- bArena and WebVoyager demonstrate that Syn- Weaver consistently outperforms strong synthe- sis baselines and yields more effective super- vision for both in-domain and out-of-domain generalization. The code is publicly available at https://github.com/Eilok/SynWeaver. 1 Introduction End-to-end web agents have shown strong poten- tial on realistic tasks, but training effective agents still requires substantial task-trajectory supervision (Qin et al., 2025; Wang et al., 2025a). Obtaining such data at scale is challenging because manual trajectory annotation is costly and time-consuming (Deng et al., 2023). Moreover, even with inter- action data, agents often transfer poorly to un- seen websites, as multimodal models lack website- specific priors about page structure, interaction af- fordances, and action consequences (Tang et al., 2025). Therefore, web-agent data synthesis is im- portant not only for alleviating the supervision bot- tleneck, but also for injecting target-website knowl- edge that improves generalization. Existing web agent data synthesis methods can be broadly divided into two families. The first fam- ily leverages external resources, such as tutorials or documentation, to instantiate tasks and replay tra- jectories (Zhang et al., 2026a; Xu et al., 2025b; Su et al., 2025; Shen et al., 2024). These methods can provide useful supervision when such resources are abundant, but their coverage is fundamentally bounded by what already exists in the internet. For many websites, relevant materials are sparse, out- dated, or entirely unavailable (Mereu et al., 2026). In this work, we instead focus on the second family, which mines tasks directly from website interaction and is therefore applicable even when external arti- facts are missing (Sun et al., 2025a; Pahuja et al., 2025; Gandhi and Neubig, 2025). Within this family, a central challenge is gener- ating tasks that are both realistic and executable. Some methods synthesize tasks from local inter- actions, which is efficient but often short-sighted: lacking sufficient website-specific knowledge (Shi et al., 2025), the model may propose tasks that appear plausible on the current page yet become infeasible during subsequent interaction. NNetNav (Murty et al., 2024) alleviates this issue through multi-step LLM exploration, but at the cost of sub- stantial redundant interaction. SynthAgent (Wang et al., 2025b) further introduces task refinement during execution and trajectory refinement after ex- ecution, partially mitigating infeasible initial tasks. However, because these refinements are decoupled, the task is often rewritten to fit the collected trajec- tory, producing step-heavy descriptions that read like procedural instructions rather than natural user 1 arXiv:2608.12429v1 [cs.SE] 12 Aug 2026 requests. To address these limitations, we propose SynWeaver, a website-prior task and trajectory co-synthesis framework. SynWeaver places explicit website knowledge at the center of synthesis and proceeds in three stages. First, it constructs a website map of functionally distinct states and executable transitions, improv- ing coverage while reducing redundant exploration. Second, it derives user interface (UI) supervision from the map to fine-tune a UI-aware model, en- abling task proposal grounded in website-specific priors rather than local observations alone. Third, it performs collaborative task-trajectory synthesis, where teacher model jointly refines the task and execution prefix whenever intent and execution be- come inconsistent. This collaborative refinement preserves the natural form of user requests while maintaining execution feasibility. Experiments on WebArena and WebVoyager confirm the effectiveness of SynWeaver. Using only 822 validated task-trajectory pairs synthesized from website maps, SynWeaver achieves the best overall WebArena success rate on both Qwen3-VL- 8B-Instruct and InternVL3-8B, and reaches 27.06 success rate on WebVoyager, outperforming the strongest baseline by 4.64 points. Our contributions are threefold: 1.We introduce a structured website exploration mechanism that reduces redundant interac- tions while preserving coverage of function- ally distinct states, compared to random-walk exploration. 2.We propose SynWeaver, a website-prior- centered framework for collaborative task- trajectory synthesis that couples grounded task proposal with joint refinement of task intent and execution traces. 3.Extensive experiments and ablations show that SynWeaver produces more effective and data-efficient supervision for web agents, yielding strong in-domain and out-of-domain gains over prior synthesis methods. 2 Related Work Executable Task Synthesis. Existing synthesis pipelines usually acquire environment knowledge before generating grounded tasks. Some meth- ods mine executable tasks from online tutorials (Zhang et al., 2026a; Xu et al., 2025b; Shen et al., 2024; Zhou et al., 2025); Learn-by-interact fur- ther augments such seeds with Self-Instruct (Wang et al., 2023; Su et al., 2025). Recent web-centric methods instead interact with target websites: OS- Genesis samples random-walk actions (Sun et al., 2025a), SynthAgent reduces redundancy via func- tion categorization (Wang et al., 2025b), and Go- Browse builds an exploration graph to discover pages and induce tasks (Gandhi and Neubig, 2025). However, these approaches still provide limited guarantees of functional coverage. Graph-based systematic exploration has addressed similar cov- erage issues in apps, where KG-RAG and GUI- Xplore use DroidBot for exhaustive exploration (Guan et al., 2025; Sun et al., 2025b; Li et al., 2017); earlier web research also uses Crawljax to traverse website states with depth-first search (DFS) (Mesbah et al., 2012; Peng et al., 2012). In- spired by graph-structured web exploration (Chen et al., 2025; Zhang et al., 2026b), we construct a website map covering functionally distinct in- teractions and use it as an LLM fine-tuning prior, enabling task synthesis with richer website context. Trajectory Data Synthesis for Web Agents. Web agent training requires trajectory-level super- vision (Nguyen et al., 2025), and synthetic trajecto- ries often rely on stronger LLMs for labeling (Tang et al., 2025). Existing methods filter tutorials with FastText before LLM annotation (Xu et al., 2025a; Bojanowski et al., 2017), roll out trajectories with learned web-world models (Gao et al., 2025), or create verifiable synthetic websites to collect inter- action traces (Wu et al., 2026). For LLM demon- strations on real websites, early approaches retain only trajectories passing binary filters (Lin et al., 2026; Lù and Reddy, 2026), while OS-Genesis scores trajectories for weighted training instead of discarding low-quality data (Sun et al., 2025a). Ex- plorer refines tasks during execution (Pahuja et al., 2025), and SynthAgent adds post-execution trajec- tory refinement (Wang et al., 2025b). Yet decou- pled task and trajectory refinement cannot reliably repair execution-time failures, reducing data effi- ciency and task-trajectory alignment. We therefore introduce collaborative refinement, which jointly optimizes tasks and trajectories throughout synthe- sis to preserve their semantic consistency. 3 Method Problem Formulation. We model a website as a partially observable web environmentW. At interaction stept, the agent receives an observa- tiono t ∈ O(e.g., textual and visual content), 2 makes a reasoningr t ∈ Rand performs an ac- tiona t ∈ A, yielding an execution historyh = (o 1 ,r 1 ,a 1 ,...,o H ,r H ,a H )of lengthH. We for- mulate web agent data synthesis as two coupled generation processes. Task synthesis generates a task instructionxfrom a partial contexth c , i.e., x∼ p θ (x| h c ), whereh c may be a local page state or an exploration prefix ending at the current state. Trajectory synthesis then generates an execution trajectoryh e conditioned onxby sequentially se- lecting actions according toa t ∼ π(a| o t ,x,h <t ) until termination. Our goal is to synthesize a datasetD = (x i ,h e i ) N i=1 ofNexecutable and semantically aligned task-trajectory pairs, rather than trajectories in isolation. In other words, the task and the trajectory are treated as co-equal ob- jects that must express the same underlying intent. As shown in Figure 1, our SynWeaver frame- work proceeds in three stages: website map construction, website-prior learning, and task- trajectory collaborative synthesis. We describe each stage in detail in the following. 3.1 Website Map Construction Existing works often explore websites via random walks (Sun et al., 2025a) or LLM-guided prob- ing (Murty et al., 2024), but such strategies may repeatedly traverse elements with the same func- tionality while missing interactions with distinct effects, yielding redundant and incomplete traces that limit downstream task diversity. We therefore introduce a DFS-based crawler that constructs a website mapG = (S,T )for each target website, where each nodes∈Sdenotes a functionally dis- tinct state together with its page screenshot and accessibility tree, and each edgeτ ∈Tdenotes an executable interaction and records the correspond- ing action. Rather than enumerating superficial page variations,Gcaptures the website’s functional topology, reducing redundant exploration and pro- viding the foundation for subsequent website-prior learning and task-trajectory synthesis. We constructGthrough depth-first exploration from a seed URL. At each visited state, the crawler enumerates executable candidates, performs the corresponding interactions, and records the result- ing transitions. To avoid over-exploring superficial page variations, we introduce a unified state proces- sor with two functions: progressive state compari- son, which determines whether a post-action page should be treated as a new state, and duplicate- trigger detection, which suppresses interactions that have already been triggered. DFS recurses only when a new state is confirmed; once a branch is exhausted, the crawler restores the previous con- text and resumes the remaining unexplored inter- actions. Full construction details are provided in Appendix A. 3.2 Website-Prior Learning Existing works often propose tasks from local states (Sun et al., 2025a; Wang et al., 2025b; Pahuja et al., 2025) or a short interaction prefix (Murty et al., 2024). Without sufficient website knowl- edge, however, the model may overfit local cues and propose tasks that are overly narrow or even infeasible. We therefore learn website-specific UI priors from the website map before task synthesis. Supervision Signals.We design five supervision formats, grouped into page-level and transition- level supervision. Letq,img,a11y,img som , op, and(img ′ ,a11y ′ )denote the question, cur- rent screenshot, accessibility tree, SoM-annotated screenshot (Yang et al., 2023), executed action, and post-interaction state, respectively. Page-level supervision contains two forms: page descrip- tion, which takes(img,a11y)as input and pre- dicts a detailed description of the page content, layout, and style to provide a holistic understand- ing of the page; and page question answering (QA), which takes(img,a11y,q)as input and predicts the answer to an information-seeking question, thereby encouraging targeted informa- tion extraction. Transition-level supervision con- tains three forms: element description, which takes(img som ,a11y)as input and predicts how the marked element should be interacted with and what function it serves, thereby grounding element- level affordances; forward transition description, which takes(img som ,a11y,op)as input and pre- dicts how the page changes after executingop, thereby modeling action-conditioned state changes; and inverse transition description, which takes (img,a11y,img ′ ,a11y ′ )as input and predicts a description of the observed change together with the action most likely to have caused it, thereby modeling change summarization and action infer- ence. The data examples are shown in Appendix G. Data Construction. We construct the supervi- sion data by querying a teacher modelM 1 over page states and recorded transitions in the web- site map. For each node, one call toM 1 with (img,a11y)yields a page description and one 3 Website ... DFS Construct Website Map <My Account> Click < M a n a g e A d d r e s s e s > C l i c k < E d i t > C l i c k Backtrack <General Subscription> Click duplicate with ... State Processor ... Select <Product Name> Page Description Question & Answer Element Description Difference Analysis 1 2 Page-level 1 2 Transition-level 3 Fine-tune . . . Extract Generate Reorganize Environment Tasks Generate Observation Action A s k R e f i n e Execution 1 S t a l l e d 2 Update reasoning Delete step Swap step Refine trajectory ? Refine task ? Trajectory 3 Generate Verifier 4 Verify Refine 6 F a i l Pass Post-verification 5 Data . . . Stage 1: Website Map Construction Stage 3: Task-Trajectory Collaborative Synthesis Stage 2: Website-Prior Learning Figure 1: Overview of SynWeaver. The framework first constructs a website map, then learns website-specific UI priors from the collected states and transitions, and finally performs task-trajectory collaborative synthesis to produce executable and semantically aligned task-trajectory pairs. QA pair, which are then reorganized into page- description data and page-QA instances.For each recorded transition, one teacher call with (img som ,a11y,op,img ′ ,a11y ′ )yields an element function description and a transition change anal- ysis, which are then recombined into element de- scription, forward transition, and inverse transition supervision. Training. We aggregate all supervision sources into a unified datasetD ui and shuffle it. We then fine-tune the target modelMwith LoRA (Hu et al., 2022) onD ui to obtain a UI-aware modelM ui that captures website-specific UI knowledge. 3.3 Task-Trajectory Collaborative Synthesis Prior work such as SynthAgent (Wang et al., 2025b) largely treats task refinement and trajectory refine- ment as separate optimization problems. This de- coupling can be problematic: if one only optimizes the task, the task description may gradually align to the collected trace, yielding procedure-heavy phras- ing instead of a natural user request; if one only op- timizes the trajectory against a fixed task, partially useful traces may be discarded once they no longer fit the specification, reducing data efficiency. We therefore synthesize tasks and trajectories collabo- ratively so that intent alignment and executability are maintained throughout data collection. Reverse Task Synthesis. We follow the reverse task synthesis strategy of OS-Genesis (Sun et al., 2025a) by extracting transition triplets(s,op,s ′ ) from the website map and promptingM ui to in- fer a high-level taskx 0 for which the observed transition is a plausible intermediate consequence. Unlike OS-Genesis, however,M ui already encodes website-specific UI priors, allowing it to propose tasks that are better grounded in verified website behavior and less prone to hallucinated affordances. The synthesized taskx 0 then serves as the initial task specification. Collaborative Trajectory Synthesis. Letx t de- note the task specification at stept, and leth t = (o 1 ,r 1 ,a 1 ,...,o t ,r t ,a t )denote the executed pre- fix up to that point. Starting fromx 0 , a teacher modelM 1 interacts with the website online to collect a trajectory. We trigger collaborative re- finement whenever the current task lacks essen- tial details, becomes incompatible with the ob- served website state, or execution stalls after mul- tiple unsuccessful attempts. In such cases, we in- 4 Find a one-star review of Q Mixers Premium Ginger Ale and summarize its key points. The ... is in the 2 nd position from the left on the 1 st row. Open it first. Click ‘12 Reviews’ to view all reviews and find the first one-star review. Click 1 Click 2 Scroll down 3 Scroll down 4 Scroll up 5 Stop 6 Perform several scrolls and check the reviews twice, but no one-star reviews were found. The first review is five-star rated. Its key comments are: 1..., 2..., 3... Done 3 Step 2: Click ‘12 Reviews’ to check all reviews and find the first five-star. Joint refinement: (1) Refine task: Find the first five-star review of Q Mixers Premium Ginger Ale and summarize its key points. (2) Update step 2 reasoning. (3) Delete steps 3-6. 3456 Figure 2: Case study of collaborative trajectory synthesis. When the target review is non-existent, SynWeaver adopts joint mode to refine the task and trajectory, ensuring the task can be successfully completed. voke a stronger teacherM 2 to co-optimize the task and the collected prefix. We consider two refine- ment modes. In the task-only mode,M 2 applies Update(x t )while keepingh t fixed, producing a revised task that better matches the current obser- vations and can be completed within the next 2- 3 steps. In the joint mode, used when task-only refinement would preserve an excessively noisy prefix,M 2 edits bothx t andh t . Following SynthA- gent (Wang et al., 2025b), we allow two trajectory edits over step indicesiandj:Delete(i)removes an obsolete or redundant step, andReorder(i,j) swaps two locally commutable steps. In addition, we introduceUpdate(r t )to revise affected reason- ing so that the refined execution remains logically coherent. Execution then resumes under the re- fined pair until termination, yielding a candidate trajectory h e . A case is shown in Figure 2. Post-Verification.Although co-optimization im- proves alignment during collection, some candi- date trajectories may still contain failed attempts, redundant operations, or improper termination. We therefore perform post-verification after execution with access to the final task and the full trajec- tory. The first stage is heuristic verification, which applies three programmatic checks: (1) termina- tion effectiveness, requiring an explicit completion action; (2) trajectory validity, checking success- ful completion within the predefined step budget; and (3) trajectory consistency, detecting invalid repetitive action patterns. Additional implemen- tation details are deferred to Appendix B. Pairs that pass are accepted directly. Otherwise, they en- ter a reconstruction stage, whereM 2 performs tar- geted repair under the same joint mode as above by applyingUpdate(x t ),Delete(i),Reorder(i,j), andUpdate(r t )as needed; we additionally allow Add(none)when an explicit completion action is missing. The repaired trajectory is then returned to heuristic verification, and the reconstruction- verification loop repeats until the pair passes. IfM 2 cannot reconstruct the data, the pair is filtered out. Validated pairs are retained inDas semantically aligned task-trajectory supervision. Dataset Expansion. After generating an initial set of tasks from all extracted transitions, we resam- ple transitions and prepend previously synthesized tasks associated with the same transition to the prompt, then askM ui to propose an additional task that is distinct from the existing ones. Repeating the above synthesis and verification process further expands the final datasetD. Further details are shown in Appendix C. 4 Experiments 4.1 Experimental Setup We evaluate SynWeaver on standard web agent benchmarks under matched fine-tuning settings. Benchmarks. We evaluate on WebArena (Zhou et al., 2024), a self-hostable benchmark spanning five websites, and WebVoyager (He et al., 2024), an online benchmark covering diverse real-world web- sites. Following prior work (Wang et al., 2025b), we sample one task per WebArena template and remove cross-website tasks, yielding 226 evalua- tion tasks. For WebVoyager, we apply executability and overlap filtering, leaving 388 evaluation tasks. We report task success rate (SR) under the official protocols. SynWeaver synthesizes data on the five WebArena websites and is evaluated in-domain on WebArena and out-of-domain on WebVoyager. Baselines & Models.We compare against NNet- Nav (Murty et al., 2024), OS-Genesis (Sun et al., 2025a), and SynthAgent (Wang et al., 2025b), which represent self-exploration, random-walk syn- thesis, and categorized exploration with online re- 5 StatisticsShopping CMS Reddit GitLab Maps States9016913115316 Transitions14821320321416 Table 1: Website map statistics retained for SynWeaver data synthesis on WebArena. finement, respectively. We fine-tune two open- source multimodal backbones: Qwen3-VL-8B- Instruct (Bai et al., 2025) and InternVL3-8B (Zhu et al., 2025). We use Gemini-3-Flash as the base teacherM 1 and Gemini-3.1-Pro as the stronger teacher M 2 . Data Synthesis Statistics.Rather than relying on random exploration, SynWeaver first constructs a structured website map for each WebArena website. Table 1 reports the retained states and transitions used for synthesis, totaling 559 states and 794 tran- sitions. From this topology, we synthesize 3,500 UI samples and 822 validated task-trajectory pairs comprising 4,500 action steps. For fair comparison, each baseline is trained on 1,000 public trajectories, so SynWeaver uses a smaller but more structured and validated training set. Training Details.All methods are trained under the same two-stage LoRA pipeline for fair com- parison. We first adapt each backbone with Syn- Weaver’s UI data to obtain a shared UI-aware ini- tializationM ui , and then fine-tuneM ui using the synthesized supervision of each method. Detailed benchmark filtering, environment setup, data sam- pling rules, and hyperparameters are provided in Appendix D. 4.2 Main Results In-domain Results on WebArena.As shown in Table 2, SynWeaver achieves the best overall SR on both model backbones, despite using only 822 validated task-trajectory pairs after filtering, versus the 1,000 trajectories used for each baseline. On Qwen3-VL-8B-Instruct, our method reaches 19.91, outperforming the strongest baseline, SynthAgent, by 3.10 points and the vanilla model by 7.96 points. On InternVL3-8B, it attains 14.16, improving over the strongest baseline, SynthAgent, by 1.33 points and over the vanilla model by 5.31 points. These gains are noteworthy because, under our matched training setup, all methods are fine-tuned from the same UI-aware initializationM ui . This indicates that the advantage mainly comes from the quality of the synthesized task-trajectory pairs rather than from differences in model initialization. Performance varies across websites and model backbones.For Qwen3-VL-8B-Instruct, our method achieves the best SR on Shopping, Reddit, and GitLab, while SynthAgent remains strongest on CMS and Maps. For InternVL3-8B, our method performs best on Shopping, CMS, and GitLab and ties with SynthAgent for the best result on Reddit. A notable exception is Maps, where our method un- derperforms SynthAgent across both backbones. A likely reason is that Maps contributes only 48 tran- sition instances for data synthesis, which is substan- tially fewer than the other websites and therefore provides a much smaller amount of supervision. Out-of-domain Results on WebVoyager. Ta- ble 3 shows that SynWeaver also generalizes well to unseen websites. Using Qwen3-VL-8B-Instruct, our method achieves an overall SR of 27.06 on the evaluated WebVoyager websites, outperforming SynthAgent by 4.64 points and the vanilla model by 13.66 points. It attains the best SR on seven of the nine websites, namely Amazon, Apple, ArXiv, BBC, ESPN, Hugging Face, and WA. However, it does not outperform the strongest baseline on All- recipes and GitHub, indicating that cross-domain transfer remains uneven on these websites. Overall, the results on both benchmarks support our central hypothesis that high-quality web agent data should couple task intent with executable tra- jectories. Compared with baselines that rely on self-exploration, random walks, or decoupled task- trajectory refinement, SynWeaver consistently de- livers stronger overall performance. Notably, this improvement is achieved with 822 validated task- trajectory pairs after filtering, which is fewer than the 1,000 trajectories used for each baseline. These findings suggest that our gains come not only from better alignment, but also from better data effi- ciency. 5 Analysis 5.1 Ablation Study We conduct an ablation study on Qwen3-VL-8B- Instruct over the WebArena subset, as shown in Table 4. The upper block ablates components of the full SynWeaver pipeline, while the lower block compares the Qwen baseline with training on UI data only (+UI) or trajectory data only (+Traj). Rel- ative to Ours, replacing website map sampling with random walk (−Map) reduces the overall SR by 6 ModelMethodShoppingCMSRedditGitLabMapsOverall Qwen3-VL-8B-Instruct Vanilla18.1812.287.693.5718.7511.95 NNetNav20.0014.0411.545.3621.8814.16 OS-Genesis20.0012.2811.5410.7128.1315.93 SynthAgent18.1815.7915.385.3637.5016.81 Ours23.6414.0423.0812.5034.3819.91 InternVL3-8B Vanilla9.0910.537.695.3612.508.85 NNetNav16.3610.530.003.5721.8810.62 OS-Genesis14.5510.5311.545.3615.6311.06 SynthAgent16.367.0215.385.3628.1312.83 Ours20.0012.2815.388.9315.6314.16 Table 2: Task success rate (SR, %) on WebArena. Bold numbers indicate the best result. MethodAllrecipes Amazon Apple ArXivBBCESPN GitHub Hugging FaceWAOverall Qwen4.444.8820.9311.6319.054.5531.7113.9510.8713.40 +NNetNav2.229.7613.9523.2628.579.0934.1516.2821.7417.53 +OS-Genesis6.6721.9527.9118.6021.43 11.3629.2723.2615.2219.33 +SynthAgent4.4424.3925.5823.2635.71 18.1836.5920.9315.2222.42 +Ours2.2229.2739.5327.9140.4820.4529.2730.2326.0927.06 Table 3: Qwen3-VL-8B-Instruct task success rate (SR, %) on WebVoyager. The abbreviations in the header are BBC = BBC News and WA = Wolfram Alpha. Bold numbers denote the best result in each column. Method Shopping CMS Reddit GitLab Maps Overall Ours23.6414.0423.0812.5034.3819.91 −Map20.008.7715.388.9331.2515.49 −CR18.1810.5315.385.3634.3815.04 −WP27.2712.2819.238.9325.0017.70 −PV21.827.0219.237.1425.0014.60 Qwen18.1812.287.693.5718.7511.95 +UI16.3610.5311.545.3634.3814.16 +Traj25.4514.0419.238.9331.2518.58 Table 4: Ablation results (SR, %) on WebArena us- ing Qwen3-VL-8B-Instruct. Ours denotes the full Syn- Weaver pipeline.−Map replaces website map with random walk;−CR replaces collaborative refinement with decoupled refinement;−WP replaces the website- prior proposer with a general-purpose model; and−PV removes post-verification. UI and Traj use only UI data and trajectory data, respectively, for training. 4.42 points, demonstrating the value of structured website exploration. Replacing collaborative re- finement with decoupled refinement (−CR) causes a 4.87-point drop, with the largest losses on Reddit (7.70) and GitLab (7.14), confirming the benefit of jointly refining tasks and trajectories. Replac- ing the website-prior proposer with Gemini-3.1- Pro (−WP) lowers the overall SR by 2.21 points and particularly hurts Maps (9.38), while removing post-verification (−PV) produces the largest over- all drop of 5.31 points. In the lower block, +UI and +Traj improve the overall SR over Qwen by 2.21 and 6.63 points, respectively. Overall, the full pipeline performs best. 5.2Impact of Website Maps on Data Diversity To verify whether tasks synthesized from website maps are more diverse, we compare the synthe- sized task sets in Table 5. We use NovelSum (Yang et al., 2025), which summarizes set-level novelty using both semantic difference and local density; here,K,α, andβcontrol the neighbor- hood size, distance weighting, and density effect, respectively. For evaluation, we use the 1,009 human-annotated training tasks from Mind2Web (Deng et al., 2023) as the reference set, encode task texts with BGE-M3 (Chen et al., 2024), and report three settings to ensure robust results: the stan- dard setting(K=10,β=0.5,α=1), the density- free variant(K=10,β=0,α=1), and the variant with different neighbors(K=5,β=0.5,α=1). We also report cosine distance as a complementary measure. SynWeaver achieves the best NovelSum in all three settings (0.3805, 0.3731, and 0.3854), while remaining essentially tied with the strongest baseline on cosine distance. This shows that web- site maps improve task diversity without sacrificing overall semantic spread. 5.3 Impact of Website-Prior on Task Quality To assess the effect of website-prior on task pro- posal, we compare the UI-aware model with Gemini-3.1-Pro on the same transition triplets as 7 Method NovelSum Cosine Distance Setting 1 Setting 2 Setting 3 NNetNav0.35850.33590.36400.5517 OS-Genesis0.36240.35280.36730.5606 SynthAgent0.37250.36810.37700.5825 Ours0.38050.37310.38540.5820 Table 5: Data diversity comparison across differ- ent synthesis methods. Settings 1, 2, and 3 corre- spond to(K=10,β=0.5,α=1),(K=10,β=0,α=1), and(K=5,β=0.5,α=1), respectively. Higher values indicate greater diversity. Figure 3: Effect of website-prior on task proposal qual- ity across WebArena websites. Table 1, changing only the proposer. Tasks from both models undergo the same Collaborative Tra- jectory Synthesis procedure. We report final task success rate and the average number of task refine- ment, where higher success and fewer refinements indicate better-grounded proposals. Figure 3 shows that the UI-aware proposer achieves higher task success rates with fewer op- timizations across all WebArena websites. We do not control for task difficulty, so simpler tasks may naturally score better on both metrics. Neverthe- less, Table 4 shows that replacing the website-prior proposer with a general-purpose model (−WP) re- duces downstream SR from 19.91 to 17.70, indicat- ing that website-prior proposals and their trajecto- ries provide more effective training supervision. 5.4 Impact of Collaborative Refinement on Data Efficiency and Cost To examine how collaborative refinement affects data efficiency and synthesis cost, we compare it with decoupled refinement on the same task set. De- coupled refinement updates tasks during trajectory collection and revises trajectories after execution, MethodTeacher ModelTask SR (%) Retention (%) Cost ($) SynthAgent GPT-4.1–0.130 DR Gemini-3-Flash + Gemini-3.1-Pro 71.8990.590.091 CR74.7999.520.085 Table 6: Comparison between decoupled (DR) and col- laborative (CR) refinement on the same task set. Syn- thAgent is included as a cost reference. whereas collaborative refinement jointly updates the task specification and executed prefix when in- consistencies arise. We report task success rate, trajectory retention, and estimated synthesis cost, with SynthAgent included as a cost reference. As shown in Table 6, collaborative refinement improves task success rate from 71.89% to 74.79% and raises trajectory retention from 90.59% to 99.52%. The main gain comes from retention: far fewer trajectories are discarded after verification, so more collected data can be retained as supervi- sion. This higher retention directly lowers synthe- sis cost, reducing it from $0.091 to $0.085 despite better task success and data quality. Cost is further reduced by the hierarchical teacher design, where the lightweight teacherM 1 handles routine synthe- sis and the stronger teacherM 2 is used only for difficult refinements and repairs. As a result, Syn- Weaver is also cheaper than SynthAgent ($0.13). Overall, collaborative refinement improves both data efficiency and economic efficiency. 6 Conclusion In this paper, we present SynWeaver, a website- prior task-trajectory co-synthesis framework for web agents. SynWeaver addresses a key limitation of existing exploration-based synthesis methods: they often lack sufficient website-specific priors to propose realistic and executable tasks, leading to limited trajectory quality. To overcome this is- sue, SynWeaver combines exploration-driven web- site map construction with website-prior learning and collaborative task-trajectory refinement, pro- ducing supervision that is more faithfully aligned with real website interactions. Experiments on WebArena and WebVoyager demonstrate that Syn- Weaver consistently outperforms prior synthesis baselines while also improving data diversity and synthesis efficiency. These results highlight the im- portance of explicit website knowledge in scalable web agent data synthesis and provide a practical direction for improving web agent adaptation and generalization. 8 Limitations SynWeaver has limitations in both deployment scope and training paradigm. Its website maps are constructed through automatic exploration; al- though this process is effective in self-hostable and moderately constrained environments, it be- comes less reliable when websites adopt strong anti-bot or human-verification mechanisms, such as CAPTCHA systems, aggressive rate limiting, or dynamic authentication pipelines. In addition, Syn- Weaver mainly improves web agents through super- vised fine-tuning on teacher-generated data, mak- ing the resulting student model inherently bounded by the reasoning and interaction capabilities of the proprietary teacher models used during synthesis. The current framework also does not explore more autonomous self-improving paradigms, such as on- line reinforcement learning, which may further en- hance agents beyond the supervision provided by the teachers. We leave these directions to future work. Ethical Considerations All experiments in this work are conducted in con- trolled, compliant, and authorized web environ- ments, such as benchmarks and self-hosted web- sites. We do not crawl third-party websites with- out permission, bypass access controls, collect pri- vate user data, or disrupt online services. Because automated website exploration could be misused for unauthorized crawling, large-scale scraping, or other harmful activities, we strongly oppose such uses and call on researchers and practitioners to obtain proper authorization, respect website poli- cies and rate limits, and use these methods only for lawful and responsible purposes. Use of AI Assistants AI assistants were used only for grammar checking and language polishing during manuscript prepa- ration. All scientific ideas, methods, experiments, results, analyses, and conclusions were developed and verified by the authors. The authors manually reviewed and edited all AI-assisted text and take full responsibility for the final manuscript. References Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, and 1 others. 2025. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the associa- tion for computational linguistics, 5:135–146. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024.M3- embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self- knowledge distillation. In Findings of the associ- ation for computational linguistics: ACL 2024, pages 2318–2335. Weizhi Chen, Ziwei Wang, Leyang Yang, Sheng Zhou, Xiaoxuan Tang, Jiajun Bu, Yong Li, and Wei Jiang. 2025. Pg-agent: An agent powered by page graph. In Proceedings of the 33rd ACM International Con- ference on Multimedia, pages 6878–6887. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114. Apurva Gandhi and Graham Neubig. 2025. Go-browse: Training web agents with structured exploration. arXiv preprint arXiv:2506.03533. Yifei Gao, Junhong Ye, Jiaqi Wang, and Jitao Sang. 2025. Websynthesis: World-model-guided mcts for efficient webui-trajectory synthesis. arXiv preprint arXiv:2507.04370. Ziyi Guan, Jason Chun Lok Li, Zhijian Hou, Ping- ping Zhang, Donglai Xu, Yuzhi Zhao, Mengyang Wu, Jinpeng Chen, Thanh-Toan Nguyen, Pengfei Xian, and 1 others. 2025. Kg-rag: Enhancing gui agent decision-making via knowledge graph-driven retrieval-augmented generation. In Proceedings of the 2025 Conference on Empirical Methods in Natu- ral Language Processing, pages 5396–5405. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-to- end web agent with large multimodal models. arXiv preprint arXiv:2401.13919. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3. Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017. Droidbot: a lightweight ui-guided test input generator for android. In 2017 IEEE/ACM 39th international conference on software engineer- ing companion (ICSE-C), pages 23–26. IEEE. Zichuan Lin, Feiyu Liu, Yijun Yang, Jiafei Lyu, Yiming Gao, Yicheng Liu, Zhicong Lu, Yangbin Yu, Mingyu Yang, Junyou Li, and 1 others. 2026. Ui-voyager: A 9 self-evolving gui agent learning via failed experience. arXiv preprint arXiv:2603.24533. Xing Han Lù and Siva Reddy. 2026. Structured distilla- tion of web agent capabilities enables generalization. arXiv preprint arXiv:2604.07776. Jacopo Mereu, Alessandro Carcangiu, Valentino Ar- tizzu, Federico Maria Cau, and Lucio Davide Spano. 2026. Specializing llms to low-documented domains with rag: An analysis across models and retrieval depths. SN Computer Science, 7(4):320. Ali Mesbah, Arie Van Deursen, and Stefan Lenselink. 2012. Crawling ajax-based web applications through dynamic analysis of user interface state changes. ACM Transactions on the Web (TWEB), 6(1):1–30. Shikhar Murty, Hao Zhu, Dzmitry Bahdanau, and Christopher D Manning. 2024. Nnetnav: Unsu- pervised learning of browser agents through envi- ronment interaction in the wild.arXiv preprint arXiv:2410.02907. Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namy- ong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, and 1 others. 2025. Gui agents: A survey. In Findings of the Association for Compu- tational Linguistics: ACL 2025, pages 22522–22538. Vardaan Pahuja, Yadong Lu, Corby Rosset, Boyu Gou, Arindam Mitra, Spencer Whitehead, Yu Su, and Ahmed Hassan. 2025. Explorer: Scaling exploration- driven web trajectory synthesis for multimodal web agents. In Findings of the Association for Computa- tional Linguistics: ACL 2025, pages 6300–6323. Zhaomeng Peng, Nengqiang He, Chunxiao Jiang, Zhi- hua Li, Lei Xu, Yipeng Li, and Yong Ren. 2012. Graph-based ajax crawl: Mining data from rich inter- net applications. In 2012 International Conference on Computer Science and Electronics Engineering, volume 3, pages 590–594. IEEE. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, and 1 others. 2025. Ui- tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326. Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Am- lekar, Mouad Hadji, Aaron Podolny, and Ameet Tal- walkar. 2024. Scribeagent: Towards specialized web agents using production-scale workflow data. arXiv preprint arXiv:2411.15004. Chenrui Shi, Zedong Yu, Zhi Gao, Ruining Feng, Enqi Liu, Yuwei Wu, Yunde Jia, Liuyu Xiang, Zhaofeng He, and Qing Li. 2025. Gui knowledge bench: Re- vealing the knowledge gap behind vlm failures in gui tasks. arXiv preprint arXiv:2510.26098. Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Arik. 2025. Learn-by-interact: A data-centric framework for self-adaptive agents in realistic environments. In International Conference on Learning Representations, volume 2025, pages 89044–89097. Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, and 1 others. 2025a. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. In Proceed- ings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 5555–5579. Yuchen Sun, Shanhui Zhao, Tao Yu, Hao Wen, Samith Va, Mengwei Xu, Yuanchun Li, and Chongyang Zhang. 2025b. Gui-xplore: Empowering general- izable gui agents with one exploration. In Proceed- ings of the computer vision and pattern recognition conference, pages 19477–19486. Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, and 1 others. 2025. A sur- vey on (m) llm-based gui agents. arXiv preprint arXiv:2504.13865. Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, and 1 others. 2025a. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning. arXiv preprint arXiv:2509.02544. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceed- ings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pages 13484–13508. Zhaoyang Wang, Yiming Liang, Xuchao Zhang, Qian- hui Wu, Siwei Han, Anson Bastos, Rujia Wang, Chetan Bansal, Baolin Peng, Jianfeng Gao, and 1 others. 2025b. Adapting web agents with synthetic supervision. arXiv preprint arXiv:2511.06101. Yifan Wu, Yiran Peng, Yiyu Chen, Jianhao Ruan, Zi- jie Zhuang, Cheng Yang, Jiayi Zhang, Man Chen, Yenchi Tseng, Zhaoyang Yu, and 1 others. 2026. Au- towebworld: Synthesizing infinite verifiable web en- vironments via finite state machines. arXiv preprint arXiv:2602.14296. Ran Xu, Kaixin Ma, Wenhao Yu, Hongming Zhang, Joyce C Ho, Carl Yang, and Dong Yu. 2025a. Retrieval-augmented gui agents with generative guidelines. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Process- ing, pages 17877–17886. Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. 2025b. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. In International Conference on Learning Representations, volume 2025, pages 79822–79843. 10 Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chun- yuan Li, and Jianfeng Gao. 2023.Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441. Yuming Yang, Yang Nan, Junjie Ye, Shihan Dou, Xiao Wang, Shuo Li, Huijie Lv, Tao Gui, Qi Zhang, and Xuan-Jing Huang. 2025. Measuring data diversity for instruction tuning: A systematic analysis and a reli- able metric. In Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18530–18549. Bofei Zhang, Zirui Shang, Zhi Gao, Wang Zhang, Rui Xie, Xiaojian Ma, Tao Yuan, Xinxiao Wu, Song- Chun Zhu, and Qing Li. 2026a. Tongui: Internet- scale trajectories from multimodal web tutorials for generalized gui agents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 12367–12375. Xuanwang Zhang, Yuteng Han, Jinnan Qi, Mulong Xie, Zhen Wu, and Xinyu Dai. 2026b. Webnavigator: Global web navigation via interaction graph retrieval. arXiv preprint arXiv:2603.20366. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, and 1 others. 2024. Webarena: A realistic web environment for build- ing autonomous agents. In International Conference on Learning Representations, volume 2024, pages 15585–15606. Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang, Sergey Levine, and Li Erran Li. 2025. Proposer-agent-evaluator (pae): Autonomous skill discovery for foundation model internet agents. In Forty-second International Con- ference on Machine Learning. Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, and 1 others. 2025. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479. A Website Crawling Details This appendix describes the construction of the website mapG. Starting from a seed URL, the crawler incrementally explores reachable pages with depth-first search (DFS) and records both states and transitions in a persistent graph. Al- gorithm 1 is the main controller: it acquires the current state, inserts the state and transition into G, filters candidate actions, and recursively ex- plores newly discovered states. Algorithm 2 is called inside this DFS loop to prune redundant lists, forms, and atomic elements before execution. Al- gorithm 3 is applied after each interaction to deter- mine whether the resulting page should be treated as a new state. Algorithm 4 is executed after re- cursive exploration returns, so that the crawler can recover the previous action context and continue ex- ploring the remaining actions from the correct state. Table 8 summarizes the notation shared by these algorithms, and Table 9 defines the principal graph and browser operations used in their pseudocode. Why we adopt DFS?We adopt DFS as the outer exploration strategy because it is substantially more efficient than breadth-first search (BFS) in an in- teractive browser environment. Under BFS, explo- ration proceeds level by level. After expanding one state, the crawler must repeatedly return to earlier states and then restore the browser to another fron- tier state in order to continue expansion. In practice, this requires frequent rollback and replay opera- tions, such as navigating back, reopening pages, or deterministically replaying prior actions, which introduces considerable browser overhead. By con- trast, DFS continues along the current branch and performs recovery mainly when the branch has been exhausted, thereby greatly reducing the num- ber of restoration operations and the associated resource cost. State.A page states d = (u,E,L,F,a11y,img) is a structured snapshot of the current web interface. Specifically,uis the current URL,Eis the set of ac- tionable elements detected on the page,Lis the set of extracted list containers,Fis the set of extracted form containers, and(a11y,img)denote the ac- cessibility tree and screenshot, respectively. Here, a list container is detected from DOM containers such asul,ol,table/tbody, or list-like ARIA con- tainers (e.g.,role="list",role="listbox", and role="menu"). A container is treated as a valid list only if it contains over three actionable list items 11 overall. A form container is detected only from a genuine ancestor<form>element. Elements are grouped by their nearest form ancestor, and a con- tainer is retained as a valid form only if it contains at least one input field and one submit button. Transition. A transition is defined asτ d = (s d ,s d+1 ,newPageOpened,op d ), wheres d is the pre-action state,s d+1 is the resulting state after ex- ecution,newPageOpenedindicates whether the action creates a new page, andop d records the ex- ecuted operation. This representation makes each edge in the website graph self-contained: it cap- tures not only how the crawler moves from one state to another, but also the information needed for later recovery and replay. In practice, transi- tions are produced by three kinds of execution rou- tines: form execution, atomic element execution, and page-level scrolling. Among them, atomic exe- cution is rule-based: the crawler chooses an action according to the target element’s tag, type, text, and accessibility attributes. Table 7 summarizes the action categories used in our implementation, together with their high-level semantics and the tag patterns to which they apply. If the element does not match any action,clickis executed by default. Duplicate-trigger Detection. Duplicate-trigger detection is required to ensure that the web- site graph covers each distinct functional state while avoiding repeated exploration of equiva- lent functionality.To support this goal, the crawler maintains an exploration registryM = (V e ,V l ,X l ,X f ,D u )across recursive calls. The setV e records element signatures so that once an atomic element with the same functional profile has been triggered, the crawler does not need to trigger it again on later pages. However, element signatures alone are not sufficient, because two elements may serve the same function while car- rying different local text. A typical example is a content feed such as Reddit. There are many posts sharing the same structure and affording the same interactions, but their texts differ, so they may still be treated as different elements by signature matching. To address this case,D u records vis- ited element XPaths under the same URL template, allowing the crawler to deduplicate elements that appear at the same structural position on template- equivalent pages. The same design principle is used for lists. The setV l stores list signatures to suppress obviously identical list containers, but lists on structurally similar pages may still pro- vide the same functionality even when their item texts or item counts differ. Therefore,X l addition- ally records visited list-container XPaths so that functionally equivalent lists are not re-explored solely because their contents change. For forms, re- peated functionality is less likely to reappear across different page structures, so the crawler adopts a lighter strategy and records only XPaths of form containers inX f , which saves memory while re- maining sufficient in practice. In Algorithm 2, sig- natures are computed byGetSignas lightweight identifiers rather than full semantic hashes. For an atomic element, the signature is formed by con- catenating its tag, name, aria-label, and text, i.e., sig(e) = tag | name | aria_label | text. For a list container, the signature is computed by concate- nating the container tag with the texts of its list items, equivalently the container tag plus the string obtained by prefixing each item text with a separa- tor and concatenating them in order. This design keeps signature computation simple and efficient, while the combination of signature and XPath pre- vents redundant exploration more robustly than any single mechanism alone. Constrained Exploration. To ensure finite and effective website exploration, the crawler relies not only on the duplicate-trigger detection mecha- nism described above, but also on three additional constraints applied during each recursive step: ShouldStop,IsExitGuarded, andSampleItems. First,ShouldStopdetermines whether the current branch should terminate immediately. In practice, this happens when the crawler fails to obtain the current state, when the graphGdetects that the state is already known, when the current URL leaves the target root domain and enters an external domain, when the current URL matches a manually config- ured blocked URL, or when the predefined explo- ration budget, such as the maximum depth or the maximum number of states, has been reached. Sec- ond,IsExitGuardedchecks whether a candidate element belongs to a predefined set of forbidden interactions before the crawler executes it. This mechanism is mainly used to block elements that may prematurely terminate or disrupt the session, such as logout, sign out, or account deletion con- trols, and thus prevents the crawler from leaving the intended exploration workflow through unsafe ac- tions. Third,SampleItemsconstrains exploration within list containers by selecting only represen- tative elements rather than exhaustively triggering 12 ActionDescriptionApplicable Tags / Targets clickClicks the element and then checks whether the in- teraction opens a new tab. Direct click targets:<button>,<a>,<summary>, <details>,<img>,<span>,<video>,<audio>, <iframe>,<label>;<input>with clickable types (checkbox, radio, button, submit, reset, image) fillFills an input control with a default value based on the field type and textual attributes. <textarea>;<input>with input types (text, email,password,tel,url,number, anddate), ex- cluding search-like fields selectSelects an option from a dropdown menu. <select> hoverHovers over the target el- ement to reveal hidden menus. <li> with submenu cues in text oraria-label; fall- back elements whosearia-labelindicates menu or dropdown behavior scrollScrolls the page by one viewport. N/A form_submissionExecutesacomposite form action: fills fields and then submits form. Extracted<form>containers, typically with<input>, <textarea>, or<select>fields and a submit control such as <button> or <input type="submit"> Table 7: Browser actions for website map construction and their applicable tag patterns. every repeated item. The key observation is that many lists contain multiple items with the same functional structure, so exploring all of them would add cost without revealing new behavior. In our framework, sampling is primarily guided by an LLM: if the list items are judged to provide differ- ent functions, all of them are retained; if they are judged to contain repeated functional groups, only one representative group is kept. For example, in a product-card list, each card may contain the same interaction pattern, such as product image, prod- uct name, add-to-cart, and add-to-wishlist, while differing only in product content, in which case exploring a single representative card is sufficient. Backtrack. After exploring the subtree induced by one action, the crawler must restore the pre- action context before it can continue with the re- maining unexplored actions. Algorithm 4 therefore uses three recovery strategies in order. The first strategy closes a newly opened tab, which directly handles actions that spawn a separate browsing page. The second strategy uses browser history navigation when the interaction changes the cur- rent page within the same tab. The third strategy replays the shortest path from the root state, which is more expensive but more robust when local roll- back is unavailable or unreliable. This prioritized design keeps common recovery cases lightweight while still preserving a fallback for dynamic or non-reversible interactions. B Heuristic Verification Details Heuristic verification is the first stage of post- verification and is applied to a completed candidate trajectoryh e = (o 1 ,r 1 ,a 1 ,...,o H ,r H ,a H ). The procedure is purely offline: it does not re-execute the browser or query the environment again, but instead diagnoses trajectory quality from the fin- ished interaction record. For non-empty trajecto- ries, it applies three deterministic checks in order: (1) termination effectiveness, (2) trajectory valid- ity, and (3) trajectory consistency. A candidate passes heuristic verification if and only if no issue is detected. Termination Effectiveness. A valid trajectory is required to end with an explicit completion ac- tion, denoted bynone. This convention provides a canonical marker that the task objective has been achieved and aligns the trajectory with a final com- pletion statement rather than a mere cessation of interaction. Therefore, if the last step is notnone, the trajectory is judged to be missing an explicit ending signal and fails termination effectiveness. Trajectory Validity.Beyond the explicit ending marker, heuristic verification checks whether the 13 SymbolDescription ΓExploration configuration BBrowser interaction agent that exe- cutes browser operations GWebsite graph M Exploration registry,M= (V e ,V l ,X l ,X f ,D u ) dCurrent exploration depth s d State at depthd, wheres d = (u,E,L,F,a11y,img) τ d Transitiongeneratedat depthd,whereτ d = (s d ,s d+1 ,newPageOpened,op d ) ω l ∈ LList container in the current state ω f ∈ Ω f Form in the current state ω e ∈ Ω e Atomic element in the current state uURL E Set of all actionable elements on the webpage LSet of all list containers on the web- page FSet of all form containers on the webpage a11yAccessibility tree imgScreenshot of the webpage V e Set of visited element signatures V l Set of visited list signatures X l Set of visited list XPaths X f Set of visited form XPaths D u Map from a URL template to visited element XPaths Table 8: Notation Description. completed trajectory is a plausible successful exe- cution artifact. First, an empty trajectory is rejected immediately, since no executable evidence of task completion is present. Second, letn = |h e |and letBdenote the predefined step budget. When n≥ B, the trajectory is flagged with an error. Ex- cessive length often reflects program vulnerabilities because we have set the step limit before execution. Third, the program will detect the state of the envi- ronment during execution. If the execution result is labelled as failure, it fails the test. This often happens when some behaviors of the agent cause environmental errors. Trajectory Consistency.The consistency check is designed to detect cyclic or stalled behavior. We first maph e to an action-signature sequenceΣ = (σ 1 ,...,σ n ) , where eachσ i preserves only the in- FunctionDescription NavigateNavigates to the specified URL u. GetCurrentStateReturns the current web- page state s d . AddStateAdds the states d to the graphGand returns whether it is a duplicate dup. AddTransition Adds the transitionτ d to the graphG. ShouldStopDetermines whether cur- rent exploration branch should be stopped. IsExitGuardedChecks whether the ele- mentω e is in the block list. ExecuteForm Replays a recorded form operation when available; otherwise, it fills and sub- mits the form with the LLM, and falls back to default values if needed. ExecuteAtomic Replays a recorded opera- tion when available; Oth- erwise, it triggers the ele- ment according to its tag. GetSignComputes the signature of the object. GetItems Returns all elements con- tained in the given con- tainer. GetShortestPathComputes the shortest path from the root state to the target state using bidirectional breadth-first search. Table 9: Function Description. formation that determines functional behavior: the action type, the identity of the target when applica- ble, and the action value when the semantics of that action depend on it (e.g.type:[4480]:pencil). This abstraction suppresses incidental observation differences while preserving materially distinct op- erations. The loop search is activated only when n≥ 4. It then enumerates candidate pattern lengths ℓ ∈ [1, min(⌊n/2⌋, 10)]and, for eachℓ, enumer- ates feasible start positionsj. For each candidate 14 pair (j,ℓ), define p j,ℓ = (σ j ,...,σ j+ℓ−1 ). Starting from the next block of lengthℓ, the veri- fier checks whether subsequent blocks matchp j,ℓ exactly and contiguously. If no such repetition ex- ists, it moves to the next candidate; if repetition is observed, it continues counting until the first mis- match, at which point the current candidate stops expanding. Letr j,ℓ denote the number of contigu- ous repeats observed after the initial block. The repeated block therefore satisfies (σ j+kℓ ,...,σ j+(k+1)ℓ−1 ) = p j,ℓ , for k = 1,...,r j,ℓ . Only candidates with at least two contiguous re- peats are retained. For each retained candidate, the repeated-steps coverage is defined as c j,ℓ = ℓr j,ℓ n , and its priority score is s j,ℓ = c j,ℓ 1 + 1 ℓ , which favors both larger coverage and shorter re- peated blocks. Among all retained candidates, the verifier selects the one with the highest score as the dominant loop pattern. A trajectory is then judged inconsistent when this dominant loop cov- ers more than40%of the trajectory, or when its repeat count is at least four even if the coverage is lower. This criterion captures both short repetitive failures, such as repeated clicking or scrolling, and longer multi-step cycles, while remaining restricted to contiguous repetition rather than scattered recur- rence. C Dataset Expansion Details After the initial synthesis round, we continue ex- pansion by revisiting transition triplets(s,op,s ′ ) extracted from the website mapG. The user spec- ifies the number of additional tasks to synthesize, after which the system samples the same number of triplets for a second proposal pass. For each sam- pled triplet, previously accepted tasks associated with it are prepended to the prompt so thatM ui is encouraged to propose a new task that remains com- patible with the observed transition while avoiding near-duplicate outputs. Each newly proposed task is then processed by the same collaborative synthe- sis and post-verification pipeline described in the main text. The resampling order follows the structural ac- cessibility ofG. Letdepth(s)denote the depth of the source statesfrom the root, and letdeg(s)de- note its degree in the website map. We rank triplets lexicographically by depth(s), − deg(s) , so that shallower triplets are revisited first, and among triplets at the same depth, those attached to higher-degree states are preferred. This bias is use- ful because shallow, high-degree regions usually offer larger interaction space and more stable exe- cution prefixes, making them more likely to yield diverse yet completable tasks. D Experimental Details D.1 Environment Benchmarks and Filtering.We evaluate on We- bArena (Zhou et al., 2024), which contains 812 tasks derived from 241 templates across five web- sites, and WebVoyager (He et al., 2024), which contains 643 task queries across 15 websites. For WebArena, we sample one task from each tem- plate and remove cross-website tasks following prior work (Wang et al., 2025b), leaving 226 eval- uation tasks. For WebVoyager, we exclude tasks from Google Flights and Booking because they are no longer executable, remove Cambridge Dic- tionary, Coursera, and Google Search because of strict robot verification, and exclude Google Maps to avoid overlap with the websites used for training in WebArena. This leaves 388 evaluation tasks. Environment Configuration. We host the on- line web environment on an Amazon EC2 instance (t3a.xlarge) with a 1000GB EBS root volume. Both benchmarks provide screenshots and accessi- bility trees as observations. Action Space.Table 10 lists all actions available in the trajectories. Thestopaction terminates ex- ecution when a task is infeasible or the agent is stalled. Thenoneaction denotes the final execu- tion step and returns the task answer. D.2 Baselines We compare against NNetNav (Murty et al., 2024), OS-Genesis (Sun et al., 2025a), and SynthAgent 15 ActionDescription click [id]Clicks the element tagged with the given id. type [id] [content]Types the provided content into the element tagged with the given id. hover [id]Hovers over the element tagged with the given id. press [key comb]Presses a key combination (e.g., Cmd+V). scroll [up|down]Scrolls the page up or down. new_tabOpens a new browser tab. tab_focus [tab_index] Switches focus to the browser tab with the specified index. close_tabCloses the current active tab. goto [url]Navigates to the specified URL. go_backNavigates to the previously viewed page. go_forwardNavigates to the next page. stop [reason] Stops execution and records the reason. none [summary]Terminates execution with the final answer. Table 10: Available browsing actions. (Wang et al., 2025b), and construct matched train- ing sets from their released data. OS-Genesis provides 1,000 trajectories, which we use in full. For NNetNav and SynthAgent, we sample 1,000 trajectories from each method. To balance the data across websites, we sample 200 trajectories per website. For NNetNav, we first merge multi-step records into single trajec- tory records according to data indices, and then use regular-expression matching on the first-step text of each trajectory to identify the correspond- ing website. For SynthAgent, we perform indexed sampling over the task list, whose keys follow the formattask@website. Specifically, we select the first 200 task indices for each website and then ex- tract the corresponding trajectories to construct the training dataset. Because NNetNav and SynthAgent release more trajectories than the 1,000 examples used in our balanced main comparison, we further conduct a supplemental full-data experiment for these two baselines.This experiment fine-tunes the UI- aware model (Qwen3-VL-8B-Instruct) on the full released NNetNav or SynthAgent data using the same LoRA configuration as the main experiments, and evaluates the resulting models on the same WebArena benchmark. Table 11 reports the re- sulting website-level and overall success rates. Using the full released data, NNetNav and Syn- thAgent achieve overall success rates of 17.26% MethodShopping CMS Reddit GitLab Maps Overall NNetNav (full)20.0015.7915.387.1434.3817.26 SynthAgent (full)23.6414.0419.237.1437.5018.58 Table 11: Supplemental full-data baseline results for UI- aware model (Qwen3-VL-8B-Instruct) on WebArena. StatisticsShopping CMS Reddit GitLab Maps States9064613142616 Transitions14875020360616 Table 12: Original website map statistics for WebArena. and 18.58%, respectively, improving upon their balanced 1,000-trajectory results of 14.16% and 16.81%. Nevertheless, both remain below Syn- Weaver’s 19.91%, indicating that data quality and structural coverage remain important beyond scale alone. D.3 SynWeaver Data Synthesis Website Map Construction and Filtering. For the five WebArena websites, we construct a website map using Qwen3-VL-235B-A22B-Instruct to sam- ple representative elements within list containers and Qwen3-VL-Plus as a fallback for state equiva- lence judgment. Table 12 reports the original graph statistics. CMS and GitLab exhibit substantially higher graph complexity, so we apply controlled sampling to balance the final UI and trajectory data. For page-level UI synthesis on these two websites, we retain one state per URL. For transition-level synthesis, we keep one transition every two hops and cap the number of scrolling operations at 25, discarding additional scrolling transitions after the cap is reached. The filtered statistics used for data synthesis are reported in Table 1 in the main paper. Synthesized Data.Using the retained states, we synthesize two page-level datasets, page descrip- tion and page QA, with 559 instances each. Us- ing the retained transitions, we synthesize three transition-level datasets, element description, for- ward transition description, and inverse transition description, with 794 instances each. After shuf- fling these five subsets, we obtain a UI dataset containing 3,500 entries. We then synthesize one task-trajectory pair for each retained transition. To mitigate sample imbalance, we perform task sam- pling three times for the Maps website; additional details are provided in Appendix C. This process yields 826 candidate task-trajectory pairs. Dur- ing post-verification, four pairs fail validation and 16 are removed, leaving 822 validated task-trajectory pairs for training. D.4 Training We train on a cluster of eight Ascend 910C NPUs. We use Gemini-3-Flash 1 as the base teacherM 1 and Gemini-3.1-Pro 2 as the stronger teacherM 2 . In the first stage,M 1 synthesizes page-level UI data for each retained state and transition-level UI data for each retained transition. We then LoRA fine- tune each backbone to obtainM ui using a learning rate of 5e-5, batch size 16, rank 8, alpha 16, and 3 epochs. In the second stage, we LoRA fine-tune M ui into the final experimental model using the synthesized data of each method, with a learning rate of 1e-4, batch size 32, rank 16, alpha 32, and 3 epochs. E Data Scaling Experimental Setup. We investigate how Syn- Weaver scales with the amount of synthesized task- trajectory supervision. Because each sampled tran- sition in the website map serves as an anchor for task-trajectory synthesis, the number of sampled or resampled transitions directly determines the tra- jectory budget. We construct four operational data scales, summarized in Table 13. The small setting (S) uses half of the standard sample, whereas the medium setting (M) uses the standard sampling configuration. To test whether broader graph cov- erage provides further gains, the large setting (L) starts from M and exhaustively samples the tran- sitions of Shopping Admin (CMS) and GitLab. Finally, the extra-large setting (XL) augments L with one additional sampling pass whose budget equals that of M. This last setting increases the number of trajectories generated from the available website-map structure and tests whether additional task diversity remains beneficial after expanding transition coverage. Relative to M, the four settings contain0.50×,1.00×,2.12×, and3.12×as many trajectories, respectively. These points character- ize practical scaling behavior under the available website maps rather than fitting a parametric power law. 1 https://storage.googleapis.com/deepmind-media/Model- Cards/Gemini-3-Flash-Model-Card.pdf 2 https://deepmind.google/models/model-cards/gemini-3- 1-pro/ Scale Data construction# Traj.Rel. SHalf of the standard sam- ple 413 0.50× MStandard sampling config- uration 826 1.00× L M with full CMS and Git- Lab sampling 1,755 2.12× XL L plus one additional M- sized sampling pass 2,581 3.12× Table 13: Construction of the four trajectory scales. For all four settings, we start from the same UI-aware Qwen3-VL-8B-Instruct initialization and keep the synthesis, collaborative refinement, post- verification, fine-tuning hyperparameters, and We- bArena evaluation split unchanged. Thus, the com- parison varies only the amount and coverage of the task-trajectory supervision induced by the sampling configurations above. We report task success rate (SR) on each website and over all 226 evaluation tasks. SMLXL Training data scale 0 10 20 30 40 Success rate (%) Shopping CMS Reddit GitLab Maps Overall 14.16 19.91 20.80 23.89 Figure 4: Performance of SynWeaver across different websites with varying data amounts. Results. Figure 4 shows a consistently positive aggregate scaling trend. Increasing the training set from 413 trajectories (S) to 826 trajectories (M) raises the overall SR from 14.16 to 19.91, a gain of 5.75 percentage points. Expanding transition cov- erage in L further improves the overall SR to 20.80, and the additional sampling pass in XL yields the best result of 23.89. Overall, scaling from S to XL improves SR by 9.73 points, corresponding to a relative improvement of 68.7%. The website-level results show a broadly positive but heterogeneous scaling pattern. Although per- formance on individual websites fluctuates across the intermediate scales, most websites benefit when the data are expanded from S to XL, and the largest setting achieves the strongest or tied-strongest per- formance on four of the five websites. The non- monotonic changes at intermediate scales likely 17 reflect shifts in data composition: L primarily expands transition coverage for CMS and Git- Lab, whereas XL further increases task diversity through an additional sampling pass. Taken to- gether with the steadily increasing overall SR, these results indicate that SynWeaver can effectively ex- ploit additional website-map coverage and repeated transition-conditioned sampling, while the distri- bution of the synthesis budget across websites re- mains important for domain-specific gains. F UI Test Evaluation Data Construction. To determine whether website-prior learning improves the model’s understanding of website-specific UI knowledge, we construct a held-out UI question- answering benchmark from online interactions in WebArena. We first run the teacher model, Gemini- 3-Flash, on all 226 WebArena evaluation tasks and record each interaction step in real time. For every step, we retain the screenshots before and after the interaction, the executed action, and the target el- ement on which the action is performed. We then randomly sample 200 unique interaction steps, sub- ject to the constraint that none of them appears in the website maps used to construct the UI train- ing data. This separation prevents direct overlap between the UI training supervision and the evalu- ation examples. Following the evaluation protocol of GUI Knowl- edge Bench (Shi et al., 2025), we use the sam- pled interactions to construct four categories of UI knowledge questions: Widget Function, which evaluates whether the model understands the pur- pose of a UI element; Layout Semantics, which evaluates its understanding of the element’s role and spatial relationship within the page layout; In- teraction Effect, which asks the model to predict or identify the state change caused by an interac- tion; and Interaction Type, which tests whether the model can infer the appropriate operation for a state change. Each category contains 50 ques- tions generated from a disjoint subset of 50 interac- tions. Consequently, the benchmark contains 200 questions in total, with each sampled interaction contributing to exactly one question. Models and Metric. We evaluate the base Qwen3-VL-8B-Instruct model, its UI-aware coun- terpart obtained after website-prior learning (+UI), and Gemini-3.1-Pro. We report question-answering accuracy over all 200 examples. ModelAccuracy (%) Qwen3-VL-8B-Instruct74.5 +UI89.0 Gemini-3.1-Pro87.5 Table 14: Accuracy on the held-out UI knowledge test. Results. As shown in Table 14, website-prior learning improves Qwen3-VL-8B-Instruct from 74.5% to 89.0%, an absolute gain of 14.5 percent- age points. The resulting UI-aware model also exceeds Gemini-3.1-Pro by 1.5 points on this test. Because all evaluation interactions are unique and excluded from the website maps used for UI data construction, the improvement cannot be attributed to direct memorization of the tested transitions. In- stead, the results indicate that the UI supervision enables the model to acquire transferable knowl- edge on the target websites. G UI Dataset Cases This appendix shows five types of UI data: page de- scription, page QA, element description, forward transition description, and inverse transition de- scription. 18 Algorithm 1 DFS-based Website Crawling Require: Seed URL u 0 , configuration Γ Ensure: Persistent graphG 1: InitializeB ← InitAgent(Γ) 2: InitializeG ← InitGraph(Γ) 3: InitializeM← InitRegistry() 4: B.Navigate(u 0 ) 5: Explore(⊥,⊥, 0,B,G,M, Γ) 6: returnG 7: procedure EXPLORE(s d−1 ,τ d−1 ,d,B,G,M, Γ) 8: s d ←B.GetCurrentState() 9:(u d , _, _, _, _, _)← s d 10: dup←G.AddState(s d ) wheredupindicates whethers d has already been visited. 11:if s d−1 ̸=⊥∧τ d−1 ̸=⊥ then 12: G.AddTransition(τ d−1 ) 13:end if 14:if ShouldStop(dup,s d ,d,G, Γ) then 15:return 16:end if 17:(Ω f , Ω e )← FilterState(s d ,M, Γ) 18:for ω f ∈ Ω f do 19:(s d+1 ,τ d )←B.ExecuteForm(ω f ,s d ) 20: eq ← StateEquiv(s d ,s d+1 , Γ) 21:if¬eq then 22: Explore(s d ,τ d ,d + 1,B,G,M, Γ) 23: Backtrack(τ d ,B,G, Γ) 24:end if 25:end for 26:for ω e ∈ Ω e do 27:if IsExitGuarded(ω e , Γ) then 28:continue 29:end if 30:(s d+1 ,τ d )←B.ExecuteAtomic(ω e ,s d ) 31: eq ← StateEquiv(s d ,s d+1 , Γ) 32:if¬eq then 33: Explore(s d ,τ d ,d + 1,B,G,M, Γ) 34: Backtrack(τ d ,B,G, Γ) 35:end if 36:end for 37:(s d+1 ,τ d )←B.ExecuteScroll(s d ) ▷ Scroll the page 38: eq ← StateEquiv(s d ,s d+1 , Γ) 39:if¬eq then 40: Explore(s d ,τ d ,d + 1,B,G,M, Γ) 41: B.Navigate(u d ) 42:end if 43: end procedure Algorithm 2 Duplicate-trigger Detection Require:Current states d , registryM, configura- tion Γ Ensure: Forms Ω f , raw elements Ω e 1: procedure FILTERSTATE(s d ,M, Γ) 2:Ω f ←∅, Ω e ←∅, ∆←∅ where∆stores elements pruned from fur- ther exploration. 3:(u d ,E,L,F, _, _)← s d 4:(V e ,V l ,X l ,X f ,D u )←M 5: t← GetTemplate(u d ) ▷ Get the URL template // List-level pruning 6:for ω l ∈ L do 7:ifGetSign(ω l ) ∈ V l ∨ ω l .xpath ∈ X l then 8:∆← ∆∪ GetItems(ω l ) 9:else 10:I ω l ← SampleItems(ω l , Γ) ▷ Sample representative items 11:∆← ∆∪ (GetItems(ω l )\ I ω l ) 12:end if 13: V l ← V l ∪GetSign(ω l ) 14: X l ← X l ∪ω l .xpath 15:end for // Form-level pruning 16:for f ∈ F do 17: E f ← GetItems(f ) 18:if f.xpath /∈ X f then 19:Ω f ← Ω f ∪f 20:end if 21:∆← ∆∪ E f 22: X f ← X f ∪f.xpath 23:end for // Element-level deduplication 24:for e∈ E do 25: ife ∈ ∆ ∨ GetSign(e) ∈ V e ∨ e.xpath∈ D u [t] then 26:continue 27:end if 28:Ω e ← Ω e ∪e 29: V e ← V e ∪GetSign(e) 30: D u [t]← D u [t]∪e.xpath 31:end for 32:return (Ω f , Ω e ) 33: end procedure 19 Algorithm 3 Progressive State Comparison Require:Current states d , candidate states d+1 , configuration Γ Ensure: eq 1: procedure STATEEQUIV(s d ,s d+1 , Γ) 2:(u d , _, _, _,a11y d ,img d )← s d 3:(u d+1 , _, _, _,a11y d+1 ,img d+1 )← s d+1 4:(a11yTolerance,useV LM )← Γ // Level 1: URL consistency 5:if u d ̸= u d+1 then 6:return false 7:end if // Level 2: Accessibility-tree similarity 8: C d ← [ ], C d+1 ← [ ] 9:for all row r in a11y d do 10:append the normalized row r to C d 11:end for 12:for all row r in a11y d+1 do 13:append the normalized row r to C d+1 14:end for 15: N d ← |C d |, N d+1 ← |C d+1 |, N ← max(N d ,N d+1 ) 16: if|N d −N d+1 | < N· (1−a11yTolerance) then 17: m← 0 18:for i = 1 to min(N d ,N d+1 ) do 19: remove all digits fromC d [i]and C d+1 [i] 20:if the transformed rows are equal then 21:m← m + 1 22:end if 23:end for 24: sim← m/N 25:if sim≥ a11yTolerance then 26:return true 27:end if 28:end if // Level 3: Visual fallback 29:if useV LM then 30: ifCompareByVLM(img d ,img d+1 )= true then 31:return true 32:else 33:return false 34:end if 35:end if 36:return false 37: end procedure Algorithm 4 Backtracking to a Target State Require:Current transitionτ d , agentB, graphG, configuration Γ Ensure: Backtrack success flag 1: procedure BACKTRACK(τ d ,B,G, Γ) 2:(s d ,s d+1 ,newPageOpened,op d )← τ d // Strategy 1: close newly opened page 3: ifnewPageOpened=true ∧ B.BackToPreviousPage() then ▷ Close the new page and back to the previous page 4: s ′ ←B.GetCurrentState() 5:if StateEquiv(s d ,s ′ , Γ) then 6:return true 7:end if 8:end if // Strategy 2: browser history back 9:(u d , _, _, _, _, _)← s d 10:(u d+1 , _, _, _, _, _)← s d+1 11:if u d ̸= u d+1 ∧B.NavigateBack() then ▷ Back to the previous page. 12: s ′ ←B.GetCurrentState() 13:if StateEquiv(s d ,s ′ , Γ) then 14:return true 15:end if 16:end if // Strategy 3: deterministic replay from root 17: P ←G.GetShortestPath(s d ) 18: u 0 ←B.GetSeedURL() 19: B.ResetContext() 20: B.Navigate(u 0 ) 21:for τ ∈ P do 22:(_, _, _,op)← τ 23:if op.isForm then 24: B.ExecuteForm(op) 25:else 26: B.ExecuteAtomic(op) 27:end if 28:end for 29: s ′ ←B.GetCurrentState() 30:return StateEquiv(s d ,s ′ , Γ) 31: end procedure 20 The screenshot displays a web page titled 'List of forums' from a site called Postmill. The header is a solid red bar containing the site name 'Postmill', links for 'Forums' and 'Wiki', a central search bar, and user-related icons for notifications, 'Submit', and a user profile dropdown for 'spiderman'. Below the header, a white main content area includes a sub-navigation bar with tabs for 'Forums', 'Tags', and 'Alphabetical', along with a 'Submissions' sort dropdown. A prominent red 'Create forum' button is located on the right. The main content is a vertical list of forums (e.g., AskReddit, relationship_advice, worldnews), each showing its '/f/' path, title, a short description, a 'Subscribe' button with a subscriber count (currently 0 for all), and the total number of submissions. The typography is clean and sans-serif, with forum titles in bold black and paths in red. Describe this web page in detail based on the screenshot and accessibility tree. Accessibility Tree: ``` accessibility_tree ``` Include information about: - Visual layout and structure (header, navigation, main content, sidebar, footer) - All visible UI elements and their apparent functions - Visual styling (colors, typography, spacing) Input Output Page Description Figure 5: Example of page description data in the UI dataset. 21 17 comments Look at this web page screenshot and accessibility tree, then answer the following question: Accessibility Tree: ``` accessibility_tree ``` Question: How many comments are listed for this post according to the text link below the deleted post content? Input Output Page Question & Answering Figure 6: Example of page QA data in the UI dataset. 22 The 'Post' button is a submit element for the comment form, allowing users to publish their typed text as a new comment on the discussion thread. The highlighted element in this screenshot is about to be interacted with. Current page accessibility tree: ``` accessibility_tree ``` Target element: <button>Post</button> Describe the functionality of the interacted element and how users would typically interact with it in the interface. Input Output Element Description Figure 7: Example of element description data in the UI dataset. 23 The page has transitioned from an alphabetical directory of all forums to the specific '/f/allentown' forum page. The new view displays a list of submissions and comments specific to that forum, a sidebar with a 'Subscribe' button, forum statistics, and a 'Toolbox' section for moderators. Given the current page state and the action to be performed, predict how the page will change. Current page accessibility tree: ``` accessibility_tree ``` Action to perform: Click on [1298] link 'allentown' url: http://localhost:9999/f/allentown Describe only what changes you expect to see on the page after this action. Input Output Forward Transition Description Figure 8: Example of forward transition description data in the UI dataset. 24 Changes observed: The user profile popover menu for 'BzhizhkMard' has disappeared from the screen. In the comment section below, a new reply interface has appeared under the first comment, consisting of a text input area, a 'Formatting help' link, and a 'Post' button. Additionally, sorting tabs for 'Nested' and 'Linear' views have become visible above the comment thread. Predicted action: Click on [3048] strong ′ Compare these two screenshots showing a web page before and after a user action. Accessibility Tree (Before): ``` accessibility_tree_before ``` Accessibility Tree (After): ``` accessibility_tree_after ``` Based on the visual changes and accessibility tree differences, predict what action the user took. Your answer should include: 1. A description of what changed 2. The predicted action including the action and the node (e.g., 'Click on [576] link 'add the blog') Input Output Inverse Transition Description Figure 9: Example of inverse transition description data in the UI dataset. 25 H Prompts This appendix provides the main prompts used in SynWeaver. You are a webpage structure analysis model. Your task is to determine whether the elements inside an HTML list (such as`<ul>` or`<ol>`) represent **repeated functionality** (only one element or one group of elements needs to be clicked) or **distinct functionality** (all elements must be clicked). You will be given the textual structure of a list, including each element's index and text content. Your goal is to analyze the semantics and structural patterns to decide whether the list items lead to **similar ** or **different** functional outcomes. ## Objective 1. **If the list items represent repeated or symmetric functionality**: Only **one** representative item or **one group of items** should be clicked. 2. **If the list items represent distinct functionality**: All items must be clicked. ## Classification Criteria ### **1. Treat the items as *functionally identical* ( decision = "single") if:** All or most items exhibit these characteristics: - Text content represents members of the same category (e.g ., product categories, product titles, tags, results) - Elements have a highly repetitive structure (e.g., repeated`<li>`, repeated`<a>`, repeated`<div>` templates) - Items correspond to the same type of resource (e.g., products, categories, search results) - Clicking each item typically leads to pages of the **same structure but different content** (e.g., different product detail pages) - The list visually or semantically presents a "set of options of the same type" If these conditions are met, treat the list as **repeated functionality**. ### **2. Treat the items as *functionally different* ( decision = "all") if:** Any of the following are true: - Text content clearly indicates different action targets ( e.g., "new communication", "Sign In", "search communication ", "profile") - Items represent different task categories (e.g., login, registration, account management) - The list acts as a navigation menu with distinct entry points - Clicking each item evidently results in **different functionality**, not symmetrical options If any such condition applies, treat the list as **distinct functionality**. ## Output Format You must output a JSON object in the following format: ```json "decision": "single" or "all", "reason": "Explanation of the decision", "selected_indices": when the decision is "single", return [list of indices]. Otherwise, return an empty list. ``` ## Example ### Example 1 Input: ``` <ul> [21]<a> Black Friday </a> [22]<a> Amazon Haul </a> [23]<span> Medical Care </span> [24]<button> </button> [25]<a> Best Sellers </a> [26]<a> Amazon Basics </a> [27]<a> New Releases </a> [28]<a> Books </a> [29]<a> Registry </a> [30]<span> Groceries </span> [31]<button> </button> [32]<span> Gift Cards </span> </ul> ``` Output: ```json "decision": "all", "reason": "The list items represent distinct functional entry points rather than repeated or symmetric options. Items such as'Black Friday','Amazon Haul','Medical Care ','Best Sellers' all lead to different content categories or features. They do not share a repeated structure or identical functional behavior. Therefore, each item must be treated as a unique function.", "selected_indices": [] ``` ### Example 2 Input: ``` <ul> [12]<img> </img> [13]<span> Outstanding Fiction of the past 25 years </span> [14]<img> </img> [15]<span> Unforgettable Nonfiction of the past 25 years </ span> [16]<img> </img> [17]<span> Most buzzworthy books of the past 25 years </ span> [18]<img> </img> [19]<span> Editors' personal favorites </span> </ul> ``` Output: ``` "decision": "single", "reason": "The list items represent a set of thematically similar options: each entry pairs an image with a descriptive label for a specific curated book collection. The structure is repetitive, and clicking any item is expected to lead to a page of the same functional type (a book collection page), differing only in content. Therefore , these items exhibit repeated functionality.", "selected_indices": [12, 13] ``` Prompt 1: Prompt for sampling representative items from the list container. You are an intelligent form filling assistant. Your task is to generate reasonable test values for each input field in a web form based on the form context and field semantics. ## Context You will receive: 1. **Page URL**: The current page URL to help understand the context 2. **Accessibility Tree**: A structured representation of the page content around the form 3. **Form Fields**: A list of input fields in the form, each with: 26 - Index number (1, 2, 3, ...) - Tag type (input/textarea/select) - Input type (for input elements: text, email, password, etc.) - Name attribute - Aria-label - Placeholder text - Current text content ## Your Task Generate appropriate test values for each form field. The values should be: 1. **Semantically appropriate**: Match the expected content type (email for email fields, phone for phone fields, etc .) 2. **Realistic**: Look like real user input, not obviously fake 3. **Valid**: Pass basic validation (proper email format, valid phone format, etc.) 4. **Contextual**: Consider the form's purpose based on URL and accessibility tree ## Output Format Return a JSON object with a "values" array containing the fill value for each field **in order**: ```json "values": ["value_for_field_1", "value_for_field_2", " value_for_field_3", ...], "reasoning": "Brief explanation of the values chosen" ``` **Important**: - The array length MUST match the number of input fields - Each value corresponds to the field at that index position - For select elements, provide the option text to select - For password fields, use a strong test password like " TestPass123!" - For date fields, use ISO format (Y-M-D) ## Examples ### Example 1: Login Form Input: - URL: https://example.com/login - Fields: 1. input[type=email] name="email" placeholder="Enter your email" 2. input[type=password] name="password" placeholder=" Password" Output: ```json "values": ["testuser@example.com", "TestPass123!"], "reasoning": "Standard login form with email and password fields" ``` ### Example 2: Registration Form Input: - URL: https://shop.example.com/register - Fields: 1. input[type=text] name="first_name" aria-label="First Name" 2. input[type=text] name="last_name" aria-label="Last Name" 3. input[type=email] name="email" 4. input[type=tel] name="phone" placeholder="Phone number " Output: ```json "values": ["John", "Smith", "john.smith@example.com", "555-123-4567"], "reasoning": "E-commerce registration form requiring personal contact information" ``` Prompt 2: Prompt for filling the form container. You are an expert web page analyzer. Your task is to determine whether two screenshots represent the **same functional state** of a web page, even if some minor visual elements differ. # Background During web crawling and state replay, certain transient UI elements (like toast notifications, temporary alerts, or dynamic content) may appear or disappear. However, the underlying page state remains the same. We need to identify whether two screenshots represent the same functional page state while ignoring these superficial differences. # Task Compare the two provided screenshots and determine if they represent the **same web page state**. # What to Consider as the SAME State Two screenshots should be considered the **same state** if they share: 1. **Core Page Structure**: - Same page layout and main content areas - Same navigation elements (header, footer, sidebar) - Same primary content sections 2. **Functional Elements**: - Same interactive elements (buttons, links, forms, input fields) - Same product listings, article content, or data tables - Same menu items and navigation options 3. **Page Identity**: - Same URL path (if visible) - Same page title or heading - Same main purpose/function # What to IGNORE (Acceptable Differences) The following differences should be **ignored** when comparing states: 1. **Transient Notifications**: - Toast messages (success/error/warning notifications) - Temporary alert boxes - Popup hints or tooltips - Tutorial/onboarding overlays - Cookie consent banners 2. **Dynamic Content**: - Timestamps (e.g., "2 minutes ago" vs "5 minutes ago") - Real-time counters (shopping cart quantity, notification badges) - Live data feeds (stock prices, weather updates) - Advertisement content - Personalized recommendations 3. **Visual State Changes**: - Hover effects or focus states - Loading spinners or progress indicators - Scroll position - Animation states - Dropdown menu open/closed states 4. **Session-Specific Elements**: - Session IDs or tokens in URLs - CSRF tokens - Temporary promotional banners - A/B testing variations (minor UI tweaks) # Decision Criteria - **SAME**: If the core page structure, main content, and functional elements are identical (ignoring the acceptable differences listed above) 27 - **DIFFERENT**: If there are substantial differences in layout, content, navigation, or functionality # Output Format Provide your analysis in the following JSON format: ```json "decision": "same" or "different", "confidence": 0.0 to 1.0, "reasoning": "Simple explanation of your decision", ``` Prompt 3: Prompt for state comparison. You are an expert UI analyst tasked with analyzing web page screenshots and accessibility trees. Your analysis will be used to train AI models to understand web interfaces. You will receive: 1. A screenshot of a web page 2. The page's accessibility tree Your task is to provide TWO types of analysis: ## Part 1: Page Description Provide a comprehensive description of the page that includes: - Visual layout and structure (header, navigation, main content, sidebar, footer) - All visible UI elements and their apparent functions - Visual styling (colors, typography, spacing) ## Part 2: Page QA Generate ONE informative question-answer pair about the page. The question should: - Ask about specific information visible on the page - Be answerable ONLY by looking at the screenshot - Focus on extracting factual information (not opinions) - Examples: "What is the main heading?", "How many products are displayed?", "What navigation options are available?" ## Output Format You MUST respond in the following JSON format: ```json "page_description": "Your detailed page description here...", "question": "Your question about the page content...", "answer": "The answer based on the screenshot..." Analyze the following web page: URL: url Accessibility Tree: ``` accessibility_tree ``` Please provide: 1. A detailed description of this page 2. One question-answer pair about specific information visible on the page Remember to respond in the specified JSON format. Prompt 4: System prompt and user prompt for page- level UI data generation. You are an expert UI analyst tasked with analyzing web page interactions. Your analysis will be used to train AI models to understand how UI elements work and how pages change after user actions. You will receive: 1. A "before" screenshot showing the page with a marked element 2. An "after" screenshot showing the page after the action was performed 3. Information about the action taken Your task is to provide TWO types of analysis: ## Part 1: Element Description Describe the functionality of the interacted element and how users would typically interact with it in the interface. ## Part 2: State Transition Description Describe the changes between the before and after states. IMPORTANT: In the state transition description, describe ONLY what changed on the page. Do NOT mention the action taken (click, fill, etc.) - focus purely on the observable differences between the two screenshots. ## Output Format You MUST respond in the following JSON format: ```json "element_description": "Description of the element's functionality...", "state_transition": "Description of how the page changed (without mentioning the action)..." Analyze this web page interaction: Action Performed: action Element: element URL Before: from_url URL After: to_url The first image shows the page BEFORE the action (with the target element highlighted). The second image shows the page AFTER the action was performed. Please provide: 1. A description of the element's functionality 2. A description of how the page changed (without mentioning what action was taken) Remember to respond in the specified JSON format. Prompt 5: System prompt and user prompt for transition- level UI data generation. You are a GUI (Graphical User Interface) expert capable of analyzing interface changes and envisioning executable tasks or instructions. Given a GUI interface change caused by an action (e.g., clicking or typing) and the corresponding element highlighted in red boxes, you are required to analyze the interface and generate related tasks. Your task is to envision tasks based on the current action and the resulting changes in the screenshots. The output should include three components: 1. Sub-Instruction: Create a natural language instruction for the current action based on the interface changes it caused. The instruction should be concise, clear, and actionable, incorporating specific details critical to the task, such as elements, file names, timestamps, or other relevant content visible in the screenshots. For example: - "Click on the'Add to Cart' button next to the product to add it to your shopping cart." - "Type'OpenAI' into the search bar to find relevant articles." - "Scroll down to view the latest blog posts on the homepage." 2. Analysis: Carefully analyze the before-and-after 28 screenshots step by step, focusing on the changes caused by the action. Then, examine key elements in both screenshots and consider possible operations based on these elements. For example: "The previous screen displayed the main interface of a shopping website, featuring multiple product categories and several showcased items. After clicking the 'Sign Up' button, the interface transitioned to a login page where an email and password can be entered to log into an account. The login page also provides other options, such as recovering a password, creating a new account, or logging in with a Google account." 3. High-Level Instruction: Based on the before-and-after screenshots, the action, and the analysis, generate a high- level task that you believe can be completed within the current interface. There are three types of tasks: - Information seeking: The user wants to obtain certain information from the webpage, such as product details, reviews, map information, or route comparisons. Please propose clear and specific questions that need an explicit answer, and avoid asking for summary-type questions, such as "summarize the information about a product." - Site navigation: The user wants to navigate to a specific page or state. - Content modification: The user wants to modify the content of a webpage or its settings. The high-level instruction should be creative. You need to deeply analyze the elements and executable actions on the interface to generate realistic, valuable, and executable tasks that can be completed within the current GUI. The instruction should be specific, actionable, and goal- oriented, ensuring the task can be completed on the current GUI by including all critical specifics such as file names , relevant timings, or required details. Below is a brief description of the current website: website_intro Here are some examples of High-Level Instruction for reference: task_examples Current Action: current_action_str Website Name: website_name Before-action Screenshot: <image is provided in the first attachment> (the action's target element is highlighted in red box if applicable) After-action Screenshot: <image is provided in the second attachment> Please generate tasks that can be completed on the current platform, and avoid tasks that are unrelated to the current website. You MUST respond in the following JSON format: (no extra commentary): ```json "Sub-Instruction": "x", "Analysis": "x", "High-Level-Instruction": "x" ``` RETURN ONLY THE JSON I ASKED FOR. Prompt 6: Prompt for task synthesis, adopted from OS- Genesis (Sun et al., 2025a) and SynthAgent (Wang et al., 2025b). You are a GUI (Graphical User Interface) Web Agent expert capable of long-horizon planning and executing high-level tasks on a website. Based on the observations and the high- level task to complete, generate the next low-level instruction. **Information** 1. High-Level Task (your ultimate goal to finish): "high_level_task" 2. History of Actions (hint_for_history): previous_actions 3. Current Page (only current view, not full page, you may need to scroll to see more): - URL: url - Accessibility Tree (Page Context): page_context - Screenshot (only current view, not full page): img_info **Available Low-Level Actions (exact JSON formats)** Page Operation Actions: - CLICK: "type": "CLICK", "element_id": <int>, "value": "" - TYPE (default behavior is to press Enter after typing unless you explicitly set press_enter_after to 0): "type": "TYPE", "element_id": <int>, "value": "text to type <string>" - HOVER: "type": "HOVER", "element_id": <int>, "value": "" - PRESS (keyboard shortcut. The value MUST be either exactly ONE key, e.g., ArrowDown, or ONE key chord joined by "+", e.g., Ctrl+V / Cmd+V): "type": "PRESS", "element_id": "", "value": "key_comb < string>" - SCROLL: "type": "SCROLL", "element_id": "", "value": "up" or " down" Tab Management Actions: - NEW_TAB (open a new, empty browser tab): "type": "NEW_TAB", "element_id": "", "value": "" - TAB_FOCUS (switch focus to a specific tab by its index, starting from 0): "type": "TAB_FOCUS", "element_id": "", "value": "< tab_index as int>" - CLOSE_TAB (close the currently active tab): "type": "CLOSE_TAB", "element_id": "", "value": "" URL Navigation Actions: - GOTO (navigate directly to a URL): "type": "GOTO", "element_id": "", "value": "url <string >" - GO_BACK: "type": "GO_BACK", "element_id": "", "value": "" - GO_FORWARD: "type": "GO_FORWARD", "element_id": "", "value": "" Completion Actions: - NONE (use ONLY when the task is completed and you have the final answer): "type": "NONE", "element_id": "", "value": "summary content or final answer <string>" - STOP (use ONLY when the task is impossible to complete): "type": "STOP", "element_id": "", "value": "the reason for the STOP <string>" **Critical Rules for Success** 1. You must issue only actions that are valid given the current observation (accessibility tree and screenshot). 2. Only propose ONE atomic action; actions must be executable independently. 3. Prefer actions grounded by element IDs present in the accessibility tree when using CLICK/TYPE/HOVER. 4. You MUST provide meaningful and non-empty value if the action type is in TYPE, SCROLL, GOTO, NONE, STOP, TAB_FOCUS. 5. When you believe the task is complete (e.g., you have the answer), use NONE with the final answer in value. 6. Be concise and avoid redundant or risky actions; ensure 29 each action clearly advances the task. 7. Use STOP only when: - The task lacks necessary information - The target does not exist (hallucination) - The task is harmful or inappropriate - Multiple attempts (>=3) have failed to make progress 8. You MUST first generate a "state_observation_summary" to observe the current environment, then take a step-by-step "reasoning" to decide the next action. 9. The high-level task often requires MULTIPLE steps to complete. Do NOT expect to finish in a single action. 10. You MUST actively decide the next step. Do NOT choose " NONE" or "STOP" unless you are sure the task is finished or impossible. 11. Choose element IDs from the accessibility tree and use them directly in CLICK/TYPE/HOVER actions. 12. If the page does not change after an action, try scrolling to see more elements. 13. When typing dates, use the format "M/D/Y". **Output Requirements** You MUST return a JSON dictionary with the following format (no extra commentary): "state_observation_summary": "your 1-3 sentence summary of the current state relevant to the task", "reasoning": "your step-by-step reasoning to decide the next action", "next_action": "action": "type": "X", "element_id": <int or "">, "value": <string or ""> RETURN ONLY THE DICTIONARY WITHOUT ANY COMMENTARY. Prompt 7: Prompt for task execution on the website. You are a GUI Web Agent expert specializing in task and trajectory optimization. The execution agent has encountered a STOP action, indicating it cannot complete the current task with the existing trajectory. **Your Primary Goal**: Refine the task description so the execution agent can overcome its current obstacle and complete the task within 2-3 additional steps. ## Current Situation **High-Level Task**: "current_task" **Previous High-Level Tasks Attempted**: previous_tasks **Execution Trajectory** (steps 1-total_steps): previous_actions **Current Page URL**: current_url **Current Page Context (Accessibility Tree)**: page_context **Screenshot**: img_info **Important Note**: The last step is a STOP action which produced no actual operation and will be automatically discarded. Focus your optimization on making the trajectory effective before that point. --- ## Optimization Strategies Your goal is to create a refined task that enables the agent to complete its objective within 2-3 more steps. Choose between: ### Strategy 1: Task-Only Optimization **Use when**: The executed trajectory represents meaningful progress, but the original task was too ambiguous or unachievable given the current context. **Goal**: **Clarify user intent and align with the current trajectory, BUT DO NOT describe steps.** The refined task should focus on *WHAT* to achieve (the goal state), not *HOW* to do it (the actions). **Example**: - Bad (Instructional): "Click the blue'Add to Cart' button and then proceed to checkout." (Too specific, loses intent ) - Good (Intent-Driven): "Add the selected item to the shopping cart." (Clear goal, lets agent decide actions) **How to apply**: - Change the original intent if necessary to fit the trajectory's progress. - Clarify the *target entity* or *specific condition* required for success. - Keep`step_order` as the original order [original_order ] (excluding the STOP step) - Set`modified_reasonings` to empty dict ### Strategy 2: Task-Trajectory Co-Optimization **Use when**: The trajectory contains redundant, incorrect, or out-of-order steps that prevent task completion, even with intent clarification. **Example scenario**: Agent clicked wrong tabs, navigated back and forth, or executed steps in illogical order. **How to apply**: - Refine the task description to strictly reflect the core user intent. - Change the original intent if necessary to fit the trajectory's progress and make it achievable in 2-3 steps. - Remove or reorder trajectory steps to create a coherent path: - **Delete**: Omit step numbers (e.g., [1, 3, 5] removes steps 2, 4) - **Reorder**: Change sequence (e.g., [1, 4, 3, 5] swaps steps 3 and 4) - **Combined**: [1, 5, 6] means delete steps 2-4 and keep rest - Update reasoning for affected steps in` modified_reasonings` to ensure logical flow **Critical**: After modifications, the trajectory must be coherent: - Each step should logically follow the previous one - The reasoning should reflect the new context (e.g., " Since we removed the navigation step, now proceed directly to...") - The modified trajectory should lead naturally to task completion in 2-3 more steps --- ## Decision Framework Ask yourself these questions in order: 1. **Root Cause Analysis**: Why did the agent issue a STOP action? - Task was impossible/invalid? - Task was unclear/ambiguous? - Trajectory led to wrong page/state? - Agent got stuck in a loop? 2. **Can Task-Only Optimization Succeed?** - Does the current page state enable task completion? - Will clarifying or changing the **goal state** guide the agent to success in 2-3 steps? - Are the executed steps (before STOP) generally correct ? - If YES, use Strategy 1 (Task-Only). - If NO, proceed to Question 3. 3. **What Trajectory Issues Exist?** - Are there redundant/unnecessary steps? - Are steps in the wrong order? - Did early mistakes prevent success? - If YES, use Strategy 2 (Co-Optimization). 30 4. **Coherence Check** (for Strategy 2): - After reordering/deleting, does the trajectory tell a logical story? - Do you need to update reasoning to maintain coherence? - Will the modified trajectory naturally lead to completion in 2-3 steps? --- ## Output Requirements Return a JSON object with EXACTLY this structure: "analysis": "Detailed analysis: (1) Why STOP occurred (2) Root cause (3) Which strategy is appropriate (4) How refined task enables 2-3 step completion", "strategy": "task_only" or "co_optimize", "need_refine": true or false, "refined_task": "The optimized task description that enables completion in 2-3 more steps (or empty string if no refinement needed)", "step_order": [1, 2, ...], "modified_reasonings": // ONLY include steps whose reasoning needs updates to maintain trajectory coherence after reordering/deletion // Key: step number (as string), Value: updated reasoning text // Example: "3": "Now that we've navigated back, proceed directly to click the search button" , "completion_estimate": "Specific description of the 1-3 steps needed to complete the refined task from current state" **Critical Rules**: 1. The refined task MUST be achievable within 2-3 more steps from the current page state. 2. **INTENT over INSTRUCTION**: The refined task must describe the **Goal State** (e.g., "Find the pricing page") , NOT a list of actions (e.g., "Click menu, then click pricing"). 3.`step_order` must only contain integers from 1 to total_steps (the STOP step should be removed). 4. Each step number can appear at most once in`step_order `. 5. If`strategy` is "task_only",`step_order` must be [ original_order]. 6. For "co_optimize" strategy, ensure modified trajectory is logically coherent. 7. Update`modified_reasonings` for any step whose context changed due to reordering/deletion. **Focus**: Prioritize enabling the agent to complete the task quickly (2-3 steps) over preserving all trajectory steps. Ensure the task remains a high-level goal, not a low -level script. RETURN ONLY THE JSON OBJECT WITHOUT ANY COMMENTARY. Prompt 8: Prompt for task-trajectory collaborative re- finement. You are an expert at analyzing and reconstructing web agent trajectories. A trajectory has failed validation and needs to be reconstructed. ## Original High-Level Task "high_level_task" ## Trajectory (total_steps steps) trajectory_history ## Validation Issues Detected validation_issues --- ## Your Task Analyze the trajectory and validation issues, then reconstruct a valid task-trajectory pair by: 1. **Understanding the Context**: What was the agent trying to accomplish? What meaningful progress was made before the issues? 2. **Extracting Valid Steps**: Identify which steps represent meaningful progress and should be kept. You may: - **Delete** steps that are redundant, erroneous, or part of a loop - **Reorder** steps if needed for logical coherence - **Update reasoning** for steps whose context changes due to deletion/reordering - You **CANNOT** add new steps - only select from existing ones 3. **Handling Different Issues**: - **MISSING_END**: The trajectory lacks a proper NONE ( completion) action. If the existing steps accomplish a meaningful sub-goal, rewrite the task to match what was achieved, then indicate we need to append a NONE action. - **LOOP_DETECTED**: Remove the redundant loop iterations, keeping only the first occurrence of each unique action in the pattern. - **Failed/STOP trajectories**: Extract the meaningful portion of the trajectory that accomplishes a coherent sub- task. Rewrite the task description to match this extracted trajectory. 4. **Ensuring Coherence**: The final trajectory must: - Have steps that logically follow each other - Match the (possibly rewritten) task description - End with a completion state (we'l add NONE if needed) --- ## Output Requirements Return a JSON object with EXACTLY this structure: "analysis": "Detailed analysis of the trajectory issues and your reconstruction strategy", "can_reconstruct": true or false, "reconstructed_task": "The task description (original or rewritten to match extracted trajectory)", "step_order": [list of step numbers to keep, in order, e.g., [1, 2, 4, 5] means delete step 3], "modified_reasonings": // Only include steps whose reasoning needs update // Key: original step number (as string), Value: new reasoning // Example: "4": "After clicking the search button, now viewing results..." , "needs_none_action": true or false, "none_action_value": "Summary/answer for NONE action if needs_none_action is true", "reconstruction_summary": "Brief summary of what was changed and why" **Critical Rules**: 1.`step_order` must only contain integers from 1 to total_steps 2. Each step number can appear at most once 3. You can only delete or reorder existing steps, NOT add new ones 4. If`can_reconstruct` is false, explain why in`analysis` 5. The reconstructed trajectory should represent a coherent , completable task 6. If the trajectory is too broken to salvage, set` can_reconstruct` to false **Example for LOOP_DETECTED**: If steps 3-4-5-6 form a loop pattern (3-4 repeated), keep only [1, 2, 3, 4] and update reasoning if needed. **Example for Failed Trajectory**: If a 10-step trajectory ended in STOP but steps 1-6 accomplished "navigating to product page", rewrite task to "Navigate to the product page" and keep [1, 2, 3, 4, 5, 6]. RETURN ONLY THE JSON OBJECT WITHOUT ANY COMMENTARY. Prompt 9: Prompt for task-trajectory reconstruction in the post-verification. 31