Paper deep dive
Enhancing Web Agents with a Hierarchical Memory Tree
Yunteng Tan, Zhi Gao, Xinxiao Wu
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%
Last extracted: 3/13/2026, 12:29:57 AM
Summary
The paper introduces the Hierarchical Memory Tree (HMT), a framework for web agents that decouples high-level logical planning from site-specific action execution. By organizing interaction trajectories into a three-level hierarchy (Intent, Stage, and Action), HMT mitigates 'intention-execution entanglement,' allowing agents to generalize better across unseen websites by using semantic element descriptions instead of brittle, site-specific identifiers.
Entities (5)
Relation Signals (3)
HMT â improvesgeneralizationon â Mind2Web
confidence 95% ¡ Experimental results on Mind2Web and WebArena show that HMT significantly outperforms flat-memory methods
HMT â comprises â Planner
confidence 90% ¡ we develop a stage-aware inference mechanism comprising a Planner and an Actor.
HMT â comprises â Actor
confidence 90% ¡ we develop a stage-aware inference mechanism comprising a Planner and an Actor.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language model-based web agents have shown strong potential in automating web interactions through advanced reasoning and instruction following. While retrieval-based memory derived from historical trajectories enables these agents to handle complex, long-horizon tasks, current methods struggle to generalize across unseen websites. We identify that this challenge arises from the flat memory structures that entangle high-level task logic with site-specific action details. This entanglement induces a workflow mismatch in new environments, where retrieved contents are conflated with current web, leading to logically inconsistent execution. To address this, we propose Hierarchical Memory Tree (HMT), a structured framework designed to explicitly decouple logical planning from action execution. HMT constructs a three-level hierarchy from raw trajectories via an automated abstraction pipeline: the Intent level maps diverse user instructions to standardized task goals; the Stage level defines reusable semantic subgoals characterized by observable pre-conditions and post-conditions; and the Action level stores action patterns paired with transferable semantic element descriptions. Leveraging this structure, we develop a stage-aware inference mechanism comprising a Planner and an Actor. By explicitly validating pre-conditions, the Planner aligns the current state with the correct logical subgoal to prevent workflow mismatch, while the Actor grounds actions by matching the stored semantic descriptions to the target page. Experimental results on Mind2Web and WebArena show that HMT significantly outperforms flat-memory methods, particularly in cross-website and cross-domain scenarios, highlighting the necessity of structured memory for robust generalization of web agents.
Tags
Links
- Source: https://arxiv.org/abs/2603.07024v1
- Canonical: https://arxiv.org/abs/2603.07024v1
Trouble viewing inline? Open PDF directly â
Full Text
60,654 characters extracted from source content.
Expand or collapse full text
1 Enhancing Web Agents with a Hierarchical Memory Tree Yunteng Tan, Zhi Gao, and Xinxiao Wu AbstractâLarge language model-based web agents have shown strong potential in automating web interactions through advanced reasoning and instruction following. While retrieval-based memory derived from historical trajectories enables these agents to handle complex, long-horizon tasks, current methods struggle to generalize across unseen websites. We identify that this challenge arises from the flat memory structures that entangle high-level task logic with site-specific action details. This entanglement induces a workflow mismatch in new environments, where retrieved contents are conflated with current web, leading to logically inconsistent execution. To address this, we propose Hierarchical Memory Tree (HMT), a structured framework designed to explicitly decouple logical planning from action execution. HMT constructs a three-level hierarchy from raw trajectories via an automated abstraction pipeline: the Intent level maps diverse user instructions to standardized task goals; the Stage level defines reusable semantic subgoals characterized by observable pre- conditions and post-conditions; and the Action level stores action patterns paired with transferable semantic element descriptions. Leveraging this structure, we develop a stage-aware inference mechanism comprising a Planner and an Actor. By explicitly validating pre-conditions, the Planner aligns the current state with the correct logical subgoal to prevent workflow mismatch, while the Actor grounds actions by matching the stored semantic descriptions to the target page. Experimental results on Mind2Web and WebArena show that HMT significantly outperforms flat- memory methods, particularly in cross-website and cross-domain scenarios, highlighting the necessity of structured memory for robust generalization of web agents. Index TermsâLLM-based Web Agents; Hierarchical Memory; Cross-Website Generalization I. INTRODUCTION Web agents are interactive systems designed to fulfill user- specified goals on websites by perceiving the page state and executing interface actions such as clicking, typing, and selecting. Unlike traditional automation scripts relying on fixed website structures and heuristic rules [1], [2], modern web agents leverage the reasoning and language understanding capabilities of Large Language Models (LLMs) to operate across diverse page layouts, dynamic content, and open-ended natural language instructions [3], [4], [5]. This capability is instrumental for practical applications including information seeking, product comparison, service booking, and content management. In this paradigm, the agent transforms raw page observations such as textual accessibility trees or screenshots into representations processable by LLMs that subsequently predict the next optimal action. The authors are with the Beijing Institute of Technology, Beijing 100081, China (e-mail:yunteng,gaozhibit,wuxinxiao@bit.edu.cn). Corresponding author: Zhi Gao. Real-world web tasks are frequently complex and long- horizon, which require the agent to maintain the userâs goal across multiple pages, handle navigation detours, and recover from local errors to ensure continued progress. To manage this complexity, handle recurrent task patterns, and learn from past experiences, such as booking round-trip flights and cross- referencing products, agents need to maintain goal consistency over extended interaction histories. Consequently, memory mechanisms have become a cornerstone of modern agent archi- tecture. A standard design involves augmenting the agent with a retrieval-based memory that stores past successful interaction trajectories [6], [7]. By retrieving relevant trajectories based on the current instruction, the agent can utilize successful strategies as in-context demonstrations. This capability reduces redundant exploration and provides a reference for decision- making, theoretically improving efficiency for tasks with similar procedural logic. Despite the reasoning power of LLMs, existing memory mechanisms often fail to generalize to unseen websites due to a critical bottleneck: the high-level user intents are transferable, but the specific action details (e.g., element identifiers) are not. Most existing methods store trajectories in a flat format represented as linear sequences of observations and actions. This approach entangles transferable task logic with site- specific action details. When such a trajectory is retrieved in a novel environment, it introduces an issue where the related high-level intent is paired with invalid low-level action details. For example, the agent may attempt to click a button with an ID that does not exist on the new site. We refer to this phenomenon as intention-execution entanglement. This entanglement leads to workflow mismatch and context pollution, where the agent retrieves actions that are functionally correct for the source task but sequentially invalid on the target page (e.g., skipping necessary navigation steps), as shown in Fig. 1(a). This context mismatch constitutes a primary obstacle to achieving robust generalization because the memory is retrieved based on intent similarity but impedes execution due to brittle grounding details. To address this challenge, we propose Hierarchical Memory Tree (HMT), a structured framework designed to explicitly de- couple logical planning from action execution. HMT constructs a three-level hierarchy from raw trajectories via an automated abstraction pipeline. First, the Intent Level maps diverse user instructions to standardized intents to stabilize retrieval against phrasing variations. Second, the Stage Level identifies reusable semantic subgoals characterized by observable pre-conditions and post-conditions. These conditions allow the agent to align retrieval with its current progress based on observable page arXiv:2603.07024v1 [cs.AI] 7 Mar 2026 2 Flat MemoryHierarchical Memory Tree (HMT) Flat Interaction Trajectories (Instruction, Trajectory) Instruction: âBook flight to NYCâ [(Obs_1, Act_1: click #btn-123), (Obs_2, Act_2: type #input-456âNYCâ), ...] Instruction: âOrder query: one 15" [(Obs_1, Act_1: c (Obs_2, Act_2: c ...] Instruction: "Check weather in Tokyo" [(Obs_1, Act_1: type #search-box "Tok.."), (Obs_2, Act_2: click #icon-search-glass), ...] Instruction: "Reserve hotel in Paris" [(Obs_1, Act_1: click #menu-hotel-tab), (Obs_2, Act_2: click #date-picker-start), ...] Instruction: "Update profile name" [(Obs_1, Act_1: click #avatar-icon-sm), (Obs_2, Act_2: type #field-username â.."), ...] Instruction ďźBook flight) Observation Retriever Retrieved Trajectory (with linear sequences & Raw IDs) Policy Target Website Workflow Mismatch & Context Pollution Error: Planned step wrong & element grounding error. Structured Memory Hierarchy IntentďźâBook Flightâ Constraints: âDest: âNYCâ Intent Level ďźNormalized Instructionďź StageďźâSearch Flightsâ Pre-cond: âSearch form visibleâ Post-cond: âResults list visibleâ Pattern: âClick Searchâ Semantic Desc: Role: âbuttonâ, Text: âcontains âSearchââ, Position: âbottom-right of formâ Stage Level ďźSubgoals & Condsďź Action Level ďźAbstracted Patternsďź Storedraw IDs & linear sequences w/o subgoals Planner Instruction Observation Observation Actor Select Stage "Search Flights" Selected Stage Re-grounding via Semantic Matching Successful Planning & Grounding Success: Planned step correct & grounding success. Storedstage-level subgoals &transferable semantic descriptions Fig. 1. Comparison between Flat Memory and Hierarchical Memory Tree (HMT). (a) Flat memory methods retrieve interaction trajectories mixed with original workflows and source-specific implementation details w/o subgoals, leading to workflow mismatch and context pollution when applied to unseen websites. (b) HMT decouples intent from execution using a tree structure. It retrieves stage-aligned subgoals and abstract element descriptions, enabling the agent to plan the correct step and ground actions on the target interface effectively. states rather than just the initial instruction. Third, the Action Level stores action patterns paired with transferable semantic element descriptions. These descriptions capture transferable features such as the role, label, relative position, and structural context of a target element rather than site-specific identifiers. This hierarchical design prevents invalid execution details from propagating to new environments while preserving the procedural logic necessary for task completion. Leveraging this memory structure, we develop a stage-aware inference mechanism comprising a Planner and an Actor for inference. The Planner identifies the appropriate high-level functional stage by matching the current observation against the stored pre-conditions and post-conditions. Once the stage is identified, the Actor grounds abstract step patterns onto the current page by matching the stored semantic descriptions to candidate elements in the current page. To further enhance robustness, this process is augmented with a confidence- aware fallback mechanism to handle uncertain retrieval or stage selection (see Fig. 1(b)). We evaluate HMT under two realistic protocols, including an offline setting using Mind2Web to test transferability from pre-built memory, and an online setting using WebArena to test the accumulation of procedural knowledge during deployment. Experimental results demonstrate that HMT effectively mitigates intention- execution entanglement, yielding consistent improvements in success rates under cross-website and cross-domain distribution shifts. In summary, this paper makes the following contributions: â˘We propose HMT, a hierarchical memory architecture for web agents, which mitigates intention-execution entangle- ment by organizing stored interaction trajectories into a hierarchy of intent, stage, and action levels. ⢠We introduce a step-level memory abstraction method that stores action patterns together with transferable semantic element descriptions, enabling element grounding on new websites without relying on raw element identifiers. ⢠We develop a stage-aware inference mechanism based on top-down retrieval and a Planner-Actor decomposition for inference, augmented with a confidence-aware fallback to handle uncertain retrieval or stage selection. ⢠We empirically evaluate HMT on Mind2Web and We- bArena, demonstrating that it significantly outperforms representative flat-memory methods in cross-website set- tings. I. RELATED WORK A. LLM-based Web Agents The development of web agents has evolved through distinct phases. Early approaches primarily relied on reinforcement learning or imitation learning trained on hand-crafted features derived from Document Object Model (DOM) trees or accessi- bility trees. Methods such as World of Bits [1], DOMNet [8], and early DQN-based approaches [9] utilize graph neural networks to encode element relationships. While effective in constrained environments, these methods struggle with the 3 diversity of real-world websites due to sparse rewards and high exploration costs, a challenge also observed in mobile device control datasets like Android in the Wild [10]. The advent of LLMs shifts the paradigm toward agents that leverage pre- trained knowledge for instruction following and reasoning [11], [12], [13], [14]. Foundational frameworks such as ReAct [3] and Tree of Thoughts [15] enable agents to interleave reasoning traces with action execution. Specific to the web domain, systems like WebGPT [16] demonstrate browser-assisted question answering. Following this, general-purpose agents like Min- dAct [17] and SeeAct [18] focus on grounding actions in HTML or screenshots. Nevertheless, directly processing raw HTML often leads to information overload. Recent works like Prune4Web [19] and HtmlRAG [20] demonstrate that dynamically pruning the DOM tree based on current sub- tasks significantly improves grounding accuracy. This validates our design choice of using a hierarchical structure to filter context. Furthermore, research on multimodal agents, such as CogAgent [21] and AppAgent [22], explores visual grounding on mobile and desktop GUIs. Techniques like Set-of-Mark (SoM) prompting [23] are also proposed to overlay visual markers for better element identification. Complementing these architectures, our work introduces a structured memory backbone that leverages semantic element descriptors to actively filter observation noise, thereby addressing the grounding fragility inherent in general-purpose agents. B. Memory and Experience Reuse To support long-horizon autonomy, augmenting agents with external memory has become a critical research direction. Retrieval-augmented generation (RAG) [24], [25] allows agents to access relevant textual knowledge [26], while episodic mem- ory modules enable the retrieval of past interaction trajectories to guide future actions. For instance, Reflexion [6] persists verbal reinforcement to prevent repeated errors. In the context of open-ended exploration, systems like Voyager [27] maintain a skill library to store complex behaviors. More advanced memory architectures, such as MemGPT [28], manage context via tiered storage systems similar to operating systems. Specific to web navigation, Agent Workflow Memory (AWM) [7] induces reusable workflows from successful trials to guide agents in both offline and online settings, and EchoTrail [29] further refines this by utilizing critic-guided exploration to filter high-quality trajectories. Typically, these retrieval-based methods employ a flat memory structure that stores trajectories as linear sequences. While effective in fixed environments, this rigid format risks entangling high-level intent with site-specific action details, leading to context pollution when applied to new environments. This challenge mirrors problems in long-term user behavior modeling and cold-start recommendation, where recent litera- ture suggests solutions like offline trajectory clustering [30] and meta-learning for domain adaptation [31]. Inspired by these directions, our automated abstraction pipeline can be viewed as a form of explicit, task-driven clustering of behavioral sequences. Although recent works like SkillEvo [32] and AriGraph [33] have begun exploring graph-based methods for structuring procedural knowledge, HMT uniquely differentiates itself by restructuring memory into a specific hierarchy designed to explicitly decouple logical planning from execution. This design effectively mitigates the context mismatch problem found in flat retrieval baselines. C. Hierarchical Planning and Abstraction Hierarchical structures have long been utilized to manage complexity in planning and robotics. Hierarchical reinforcement learning decomposes tasks into sub-policies with initiation and termination criteria to handle long time horizons [34]. Similarly, decomposition-based prompting strategies guide models to solve problems via intermediate steps [35], [36], building upon the foundations of Chain-of-Thought (CoT) prompting [37] and Self-Consistency [38]. More recently, advanced planning algorithms like Language Agent Tree Search (LATS) [39], CRITIC [40], and multi-agent frameworks like AutoGen [41] integrate Monte Carlo Tree Search and self-verification to enhance reasoning robustness. To support such reasoning in long-horizon contexts, recent literature explores structured memory representations. For example, MemTree [42] organizes dialogue history into a dynamic tree for efficient retrieval, while HiAgent [43] employs hierarchical subgoal chunks to manage working memory. Such hierarchical paradigms have also proven effective in decoupling complex reasoning tasks across various domains. For instance, PR4SR [44] decouples session-level context setting from path-level reasoning in recommendation, while HSMH [45] fuses local and global knowledge for multi-hop reasoning on graphs. Furthermore, for handling rare or unseen interactions, HMLS [46] demonstrates that hierarchical structures enable knowledge sharing in few-shot scenarios. Tool-use frameworks like ToolLLM [47] also emphasize the importance of structured API calls. In the agent domain, LGRL [48] and LDSC [49] validate the âPlanner-Actorâ paradigm, where a high-level Planner dictates semantic subgoals and a low-level Actor handles local execution. WebOperator [50] further incorporates action-aware tree search to handle reversible actions. While these methods focus on real-time planning, they often lack a mechanism to persist successful plans for future reuse. HMT bridges this gap by crystallizing the Planner-Actor decomposition into a permanent memory structure. Uniquely, HMT introduces a dedicated abstraction pipeline at the leaf level, allowing the agent to not only decompose tasks logically via subgoals but also structurally decouple the action pattern from its specific realization on the target webpage. I. METHOD This section presents HMT, a hierarchical memory frame- work designed to decouple reusable task logic from website- specific action details. We first formalize the web agent setting and the memory representation, and then introduce the memory construction pipeline and the stage-aware inference process. An overview of the entire framework is illustrated in Fig. 2. 4 Memory Construction Pipeline Hierarchical Memory Tree (HMT) Stage-Aware Inference Trajectories 1. Instruction Normalization 2. Subgoal Segmentation 3. Step Abstraction Intent Level (Normalized Task Goals) Stage Level (Semantic Subgoals) Action Level (Action Patterns, Semantic Descriptions) Hierarchical Retrieval (Task â Subgoal) Planner Stage Selection Actor Action Grounding Instruction History Observation Fig. 2. Overview of HMT. The framework consists of a construction pipeline that abstracts raw trajectories into a hierarchical memory tree, and a stage-aware inference mechanism where a Planner selects the logical stage and an Actor grounds the action level description to the target page. A. Problem Formulation We consider a web task where an agent receives a natural- language instructionqand interacts with the browser over discrete time stepst = 1,...,T. At stept, the agent receives an observationo t such as a simplified DOM tree, accessibility tree, or screenshot, and maintains the interaction historyh tâ1 = (o 1 ,a 1 ),..., (o tâ1 ,a tâ1 ), and a denotes actions. Following common benchmark interfaces [17], [51], we assume thato t can be processed into a set of candidate UI elementsE t = e t,1 ,...,e t,K , where each UI elemente t,k contains attributes such as visible text, role, and bounding box. An action is formally represented as a tuplea t = (op t , target t , arg t ) , whereop t denotes the operation type like CLICK or TYPE,target t refers to a unique identifier for an element inE t such as a backend node ID, andarg t is an optional argument. To facilitate long-horizon planning, we augment the agent with a hierarchical memory treeT mem . We define a re- trieval functionRthat returns a relevant memory path P = R(q,h tâ1 ,o t ;T mem ). The policy is decomposed into two stages: a PlannerĎ plan that generates a logical subgoalg t , and an Actor Ď act that produces the grounded action a t : g t = Ď plan (q,h tâ1 ,o t ,P),(1) a t = Ď act (q,h tâ1 ,o t ,g t ,P).(2) This separation ensures that planning operates on transferable logic, while action generation handles local grounding. B. Hierarchical Memory Tree The architecture of HMT is guided by three principles essential for generalization: transferability by abstracting away site-specific details, stage-awareness by aligning retrieval with execution progress, and compactness to fit within context win- dows. Formally,T mem is a rooted tree (see Fig. 3) comprising three levels of abstraction designed to avoid mismatch at different granularities. At the root, the Intent Level consists of task nodes that map diverse user instructions to standardized task goals paired with constraints. For example, a raw request like âI want to fly to New Yorkâ is normalized to âIntent: Book a flight; Constraints: to NYCâ. This standardization strips away phrasing variations from raw interaction trajectories, ensuring that diverse user queries map to the same canonical task. At the second layer, the Stage Level branches into reusable semantic subgoals representing functional stages of the work- flow such as âFilter Resultsâ. Crucially, to enable stage-aware retrieval, each node is characterized by explicit pre-conditions and post-conditions described in terms of observable UI states. For instance, a pre-condition might specify âSearch results visibleâ, while the corresponding post-condition might state âPrice filter appliedâ. This allows the agent to identify its current progress based on visual evidence rather than solely on potentially noisy history. Finally, the leaf layer constitutes the Action Level, providing executable guidance. To prevent brittle transfer caused by site- specific action details, nodes in this level exclude raw element identifiers or coordinates. Instead, they encode an action pattern representing the semantic operation paired with a transferable semantic element description containing attributes like role, label, and relative position. By storing directives such as âClick the âsearchâ buttonâ rather than âClick #btn-123,â HMT ensures that the retrieved memory remains valid across different webs. 5 Instruction: Search for a one-way flight from NYC... Normalized: Search for a one-way flight. Constraints: [src: NYC, dst: London, date: 2023-12-01] Subgoal 1Subgoal 2 Name: Search Flight Name: Select Flight Pre-cond: (landing_page) VISIBLE Pre-cond: (result_list) VISIBLE AND (price_filter) VISIBLE Post-cond: (result_list) VISIBLEPost-cond: (summary_page) VISIBLE Step 1.1Step 1.2Step 2.1Step 2.2 Action Pattern: TYPE â[DST]âAction Pattern: CLICKAction Pattern: CLICKAction Pattern: CLICK Semantic Desc: Role: âtextboxâ, Label: âDestinationâ OR âToâ, Context: âFlight Formâ Semantic Desc: Role: âbuttonâ, Text: âSearchâ OR âFindâ, Pos: âBottom of formâ Semantic Desc: Role: âbuttonâ, Text: âNon-stopâ, Pos: âFilter Sectionâ Semantic Desc: Role: âlinkâ, Text: âSelectâ OR âBook Nowâ, Pos: âTop flight resultâ Temporal consistency alignment Intent Level (Task Nodes) Stage Level (Subgoal Nodes) Action Level (Step Nodes) Fig. 3. Structure of the Hierarchical Memory Tree. Unlike flat lists, HMT organizes memory into intent, stage, and action levels. C. Memory Construction We build HMT from successful interaction trajectoriesD = (q i ,Ď i ), whereĎ i =(o i,t ,a i,t ) T i t=1 represents the sequence of observations and actions. The construction process involves a unified pipeline applied under different settings. 1) Construction Pipeline: The pipeline begins with Instruc- tion Normalization, where we employ an LLM to rewrite the raw instructionq i into a normalized intent and set of constraints. We enforce a structured JSON output format to ensure determinism and to cluster semantically identical requests into a single task node. Following normalization, we perform Subgoal Segmenta- tion to partition the trajectoryĎ i into contiguous segments G i,k using an LLM. We generate semantic names and pre- conditions/post-conditions for each subgoal, enforcing strict temporal consistency checks including coverage, ordering, contiguity, and minimum length to ensure structural validity. If any check fails, a fallback mechanism triggers a coarser segmentation by iteratively merging adjacent problematic segments until structural integrity is restored. Once the hierarchy is established, we proceed to Step Abstraction. For each actiona i,t within a segment, we generate an abstract representation by deriving a semantic element description from the target elementâs attributes such as its accessibility label or surrounding text. This step explicitly discards the specific âraw element identifierâ used inĎ i , creating a transferable step node. Finally, we compute embeddings for all new nodes and update the index to support efficient retrieval. This process is summarized in Algorithm 1. Algorithm 1 Memory Construction Pipeline Require: Successful trajectories D =(q i ,Ď i ) Ensure: Hierarchical Memory Tree T mem 1: Initialize T mem ââ 2: for each trajectory (q i ,Ď i ) in D do # Level 1: Instruction Normalization 3:(I i ,C i )â Normalize(q i ) 4: v task â GetOrCreateNode(T mem ,I i ,C i ) # Level 2: Subgoal Segmentation 5: G i,k â Segment(Ď i ) 6:while ÂŹConsistencyCheck(G i,k ) do 7: G i,k â MergeSegments(G i,k ) 8:end while 9:for each segment G i,k do 10:(name,conds)â Describe(G i,k ) 11:v sub â CreateNode(v task ,name,conds) # Level 3: Step Abstraction 12:for each step (o i,t ,a i,t ) in G i,k do 13:(p i,t ,desc i,t )â AbstractStep(o i,t ,a i,t ) 14:CreateLeafNode(v sub ,p i,t ,desc i,t ) 15:end for 16:end for 17: end for 18: return T mem 2) Induction Settings: This pipeline supports two distinct induction settings. In the offline setting, we populateT mem once using a pre-existing dataset of successful trajectories such as the training split of Mind2Web. In the online setting, the agent starts with an empty memory and incrementally builds it. As it performs tasks in the evaluation stream, we monitor the success status of each episode; if an episode is successful, we immediately apply the abstraction pipeline to that trajectory and insert the new nodes intoT mem , allowing the agent to 6 Algorithm 2 Stage-Aware Inference Require: Instruction q, History h tâ1 , Obs o t , Memory T mem Ensure: Next action a t # Hierarchical Retrieval 1: V T â RetrieveTasks(T mem ,q) 2: G â RetrieveSubgoals(V T ,h tâ1 ,o t ) # Planner: State Abstraction & Stage Selection 3: s t â AbstractState(o t ) 4: (Ëg t ,c t )â Ď plan (q,s t ,G) 5: if IsLowConfidence(G, Ëg t ,c t ) then 6: G Ⲡâ ExpandSearch(G) 7:(Ëg t ,c t )â Ď plan (q,s t ,G Ⲡ) 8:if IsLowConfidence(G Ⲡ, Ëg t ,c t ) then 9:return Ď base (q,h tâ1 ,o t ) 10:end if 11: end if # Actor: Action Grounding 12: S â RetrieveSteps(Ëg t ,o t ) 13: a t â Ď act (q,o t , Ëg t ,S,E t ) 14: return a t accumulate procedural knowledge during deployment. D. Stage-Aware Inference At test time, the agent determines the next action through a unified Stage-Aware Inference process, as summarized in Algorithm 2. This process begins with a top-down retrieval strategy. First, we perform Task Retrieval by queryingT mem with the current instructionqto isolate workflows relevant to the userâs high-level intent, yielding the top-K T task nodesV T . Next, for Subgoal Retrieval, we query the subgoals attached to these tasks. The query combines the task instructionqwith a summary of the historyh tâ1 (recent actions) and an observation o t (salient elements). To prevent workflow mismatch caused by retrieving temporally distinct steps, we align the current page state with the appropriate workflow stage. We compute a combined score for each candidate subgoal g: Score(g) = (1â Îť) cos(q G , emb(g)) + Ν¡ CondMatch(g,o t ),(3) whereq G is the vector embedding of the combined query string,emb(g)denotes the embedding of the candidate sub- goalg,Îť â [0, 1]is a hyperparameter balancing semantic similarity and condition matching, andCondMatchmeasures the lexical overlap (Jaccard similarity) between the subgoalâs pre-conditions/post-conditions and the current observation summary. Based on the retrieved candidates, we employ a Planner and an Actor to perform the following actions. The Planner performs a state abstraction and verification process to ensure temporal consistency. It abstracts the raw observationo t to identify the current logical stage by matching it with the pre- conditions and post-conditions of the retrieved subgoals in G. This stage-aware selection effectively filters out invalid future or past steps, outputting the selected subgoal indexËg t and a confidence scorec t , wherec t represents the predicted probabilityP (Ëg t ) â [0, 1]that the selected stage aligns with the current state. To mitigate error propagation, we implement a Confidence- Aware Fallback mechanism. We compute a robustness metric â = P (Ëg top1 )â P (Ëg top2 )as the probability margin between the top-1 and top-2 ranked subgoals. Ifâfalls below a margin thresholdδor if the absolute confidencec t is lower than a thresholdĎ, we declare a low-confidence state. In such cases, the agent broadens the retrieval scope by increasing the number of candidate subgoalsK G . If uncertainty persists after expansion, the agent reverts to a baseline policy without memory conditioning, preventing it from being misled by irrelevant retrieval. Once a valid stage is confirmed, we retrieve the top-K S step nodes based on similarity to the current observation. These nodes serve as few-shot exemplarsS, providing the Actor with action patterns and semantic element descriptions relevant to the current stage. Then, the Actor generates the concrete action. It receives the instruction, observation, selected subgoal, and retrieved step exemplarsS. Crucially, the Actor does not directly copy IDs from the exemplars. Instead, it uses the retrieved semantic element descriptions to locate the corresponding element in the current candidate setE t . This grounding process involves querying the LLM to select the elementeâE t that best matches the semantic description, such as âbutton labeled âSearchâ â, rather than the raw identifier. This ensures the action is valid on the current page even if the underlying DOM IDs have changed. E. Generalization While presented in a text-based DOM context, HMT is adaptable to other modalities and environments. For environ- ments utilizing screenshots, the semantic element description in step nodes can be augmented with visual embeddings of the elementâs cropped region, allowing retrieval to operate in a joint text-visual embedding space. Furthermore, since different benchmarks require different action formats, such as IDs versus coordinates, HMT maintains memory in an abstract form. A lightweight resolver module maps the Actorâs semantic output to the specific format required by the environment. For ID-based environments like WebArena, it selects the element maximizing the match with the predicted description, and for coordinate-based environments, it invokes a grounding model to predict (x,y) coordinates. IV. EXPERIMENTS This section evaluates HMT under two distinct memory- induction settings. First, we examine an offline setting where memory is constructed from a pre-existing corpus. Second, we explore an online setting where memory is incrementally built during deployment. A. Settings 1) Benchmarks and Datasets: We utilize Mind2Web [17] to evaluate generalization capabilities in the offline setting. This benchmark uses a large-scale dataset containing over 2,000 open-ended web tasks collected from 137 real-world websites. Following the original protocol [17], we evaluate on three distinct test splits to assess different levels of generalization: ⢠Cross-Task: This split contains unseen tasks from websites that were seen during training. In this setting, the agent 7 TABLE I MIND2WEB RESULTS. THE BEST RESULTS ARE BOLDED, AND THE SECOND-BEST ARE UNDERLINED. HMT SHOWS DISTINCT ADVANTAGES IN CROSS-WEBSITE GENERALIZATION. Cross-TaskCross-WebsiteCross-Domain MethodEAAF1StepSRTaskSREAAF1StepSRTaskSREAAF1StepSRTaskSR MindAct [17]6660.636.22.035.851.130.12.021.652.826.42.0 AWM offline [7]50.6 57.345.14.841.446.233.72.336.441.632.60.7 AWM online [7]50.056.443.64.042.145.133.91.640.946.335.51.7 HMT (Ours)54.263.548.54.648.854.539.73.242.348.837.11.9 TABLE I WEBARENA RESULTS. HMT IMPROVES PERFORMANCE ON LOGIC-HEAVY DOMAINS (GITLAB, CMS), BUT SHOWS LIMITATIONS IN SPATIAL-HEAVY DOMAINS (MAPS). THE TOTAL TASKSR IS A WEIGHTED AVERAGE. BEST IN BOLD, SECOND BEST UNDERLINED. MethodTotal TaskSRShoppingCMSRedditGitLabMaps#Steps (Avg) WebArena baseline [48]14.914.011.06.015.016.012.4 SteP [49]33.037.024.059.032.030.08.5 AutoEval [50]20.225.518.125.428.631.946.7 AWM [5]35.530.829.150.931.843.35.9 Flat Retrieval (Baseline)32.128.525.847.028.539.06.5 HMT (Ours)38.7 33.834.152.537.642.25.2 has likely encountered similar page layouts and DOM structures. ⢠Cross-Website: This split includes tasks from websites that were never seen during training, although the domains (e.g., Travel, Shopping) are familiar. This setup assesses the agentâs ability to adapt to entirely new website structures within a known task context. ⢠Cross-Domain: This split holds out entire top-level do- mains (e.g., Information, Service) from training. The agent is expected to generalize to completely new domains without prior exposure to their specific websites or task logic. To assess interactive execution in the online setting, we em- ploy WebArena [51]. This environment spans multiple domains including Shopping, CMS, GitLab, and Maps, requiring the agent to navigate multi-turn tasks to achieve a goal. Success is determined by execution-based validators. 2) Comparison Methods: We compare our proposed HMT with several state-of-the-art methods. These methods include general-purpose agents (MindAct [17] and WebArena Base- line [51]) and hierarchical or retrieval-augmented agents (SteP [52], AutoEval [53], and Agent Workflow Memory (AWM) [7]). AWM induces linear workflows from successful trials and serves as a direct comparison for our memory-based approach. To rigorously evaluate the contribution of our hierarchical structure, we construct a degraded variant of our method as a baseline Flat Retrieval: This variant stores successful trajectories as a flat list of steps and retrieves the top-Ksteps based on embedding similarity, without utilizing the proposed stage-aware planning or semantic element descriptions. 3) Implementation Details: We use GPT-4 as the backbone model for all components. The decoding temperature is set to zero. Based on preliminary validation, we set the retrieval widths for tasks, subgoals, and step exemplars toK T = 5, K G = 8, andK S = 5, respectively. The interaction history is truncated toN h = 6actions, and the observation is summarized to the topN e = 30salient elements. In Eq. (3), the weighting factor isÎť = 0.3. The fallback thresholds are set toδ = 0.1 and Ď = 0.15. B. Main Results The comparison results on Mind2Web and WebArena are shown in Table I and Table I, respectively. From the results, we make several observations. First, on the Mind2Web benchmark (Table I), HMT achieves superior performance across all splits. In the Cross-Task split, HMT performs comparably to AWM (48.5 vs. 45.1 in StepSR and 54.2 vs. 50.6 in EA), indicating that when the DOM structure remains static, flat memory with raw identifiers is sufficient. However, in the critical Cross-Website split, HMT significantly outperforms existing methods, improving StepSR by 6.0% compared to AWM. This validates that when site- specific IDs fail due to layout changes, our semantic element descriptions successfully bridge the grounding gap. Second, on the WebArena benchmark (Table I), HMT achieves the highest total success rate of 38.7%. We observe substantial improvements in logically complex domains like GitLab (+5.8%) and CMS (+5.0%). This suggests that the Intent- Stage hierarchy effectively prevents the agent from losing track of the workflow in long-horizon tasks. Third, regarding the Maps domain in WebArena, HMT performs slightly worse than AWM (42.2% vs. 43.3%). We attribute this to the fact that map interactions often rely on spatial coordinates, where our semantic text-based abstraction offers less benefit than raw trajectory replay. Furthermore, the average number of steps is reduced (5.2 vs. 5.9), indicating that hierarchical planning reduces redundant exploration. C. Ablation Studies To evaluate the effectiveness of individual components, we design four variants of our method for comparison: (1) w/ Flat 8 Recall@1Recall@3Recall@5 0 20 40 60 80 100 Recall (%) 42.5 58.2 65.8 61.3 78.5 84.2 (a) Retrieval Quality (Mind2Web Cross-Website) Flat Retrieval HMT (Ours) Cross-TaskCross-WebsiteCross-Domain 0 20 40 60 80 100 Grounding Success Rate (%) Context Mismatch (Gap: ~64%) (b) Grounding Robustness Raw Element Identifier Semantic Description (Ours) Fig. 4. Mechanism Analysis. (a) Retrieval recall comparisons show that HMT provides more accurate context. (b) Grounding success rate across generalization splits shows that raw identifiers fail in cross-website and cross-domain settings, while the semantic descriptions (ours) remain robust. Memory: we remove the task and subgoal hierarchy to retrieve steps directly from a flat pool, which is equivalent to the Flat Retrieval baseline. This variant evaluates the importance of the hierarchical structure; (2) w/o Pre/Post-conditions: we remove the explicit pre-conditions and post-conditions, causing the Planner to rely solely on textual similarity for stage selection. This variant evaluates the effectiveness of state-aware planning; (3) w/ Raw Element Identifiers: we replace the semantic element descriptions with raw element identifiers extracted from the source website, in order to evaluate the effectiveness of semantic grounding; (4) w/o Planner: we remove the explicit Planner to test the single Actor setup where the model implicitly selects steps; (5) w/o Confidence Fallback: we disable the confidence check to assess the importance of the fallback mechanism. The results of ablation studies are shown in Table I. We have the following observations: (1) Our method substantially outperforms âw/ Flat Memoryâ, which indicates that the hierarchical structure reduces context pollution in long-horizon tasks. For example, a gain of 6.6% is achieved on WebArena. (2) With the removal of pre/post-conditions, the performance drops by 2.5% on WebArena. This demonstrates the importance of considering observable state changes for stage alignment. (3) Most critically, replacing semantic descriptions with raw identifiers (âw/ Raw Element Identifiersâ) proves catastrophic for cross-website generalization. On the Mind2Web Cross- Website split, the StepSR plummets from 39.7% to 12.4%, verifying that raw identifiers are non-transferable when the DOM structure changes. (4) Removing the Planner (âw/o Plannerâ) degrades performance, highlighting the necessity of decomposing planning and execution. D. Mechanism Analysis To provide deeper insights into why HMT works, we conduct focused quantitative analyses regarding retrieval quality and grounding robustness. TABLE I ABLATION STUDY RESULTS. WE MEASURE THE IMPACT OF KEY COMPONENTS ON MIND2WEB STEPSR AND WEBARENA TASKSR. Method Variant Mind2Web (Cross-Website) StepSR (%) WebArena Total TaskSR (%) Full HMT39.738.7 w/ Flat Memory33.232.1 w/o Pre/Post-conditions37.136.2 w/ Raw Element Identifiers12.434.5 w/o Planner35.833.5 w/o Confidence Fallback38.937.8 1) Retrieval Quality: We assess whether the hierarchical structure improves the relevance of retrieved memories by measuring the Recall@5 of the ground-truth action step on the Mind2Web Cross-Website split. As shown in Fig. 4(a), HMT achieves a recall of 84.2%, significantly outperforming the 65.8% obtained by the flat retrieval baseline. This improvement indicates that the stage-level constraints effectively filter out temporally irrelevant noise, ensuring the agent focuses on contextually valid actions. To illustrate this difference concretely, Table IV presents a real-world retrieval case on the TripAdvisor search result page. The user intent is to âFind a flight.â The flat retrieval baseline is misled by semantic overlap, retrieving checkout actions from a previous Expedia task. In contrast, the HMT Planner aligns the state with the âBrowse & Selectâ stage, ensuring the top candidates are all functionally relevant to the current page. 2) Grounding Robustness: We further quantify the transfer- ability of our action abstraction by comparing the success rate of locating the target element using raw identifiers versus our semantic descriptions. As illustrated in Fig. 4(b), while raw identifiers perform acceptably in cross-task settings (92.1%), their performance collapses to 12.4% in cross-website settings. In contrast, semantic descriptors maintain a robust success rate of 76.8%. This contrast confirms that semantic matching is the key factor in resolving intention-execution entanglement. 9 Task Instruction: âBook the cheapest flight to Tokyo.â Intent Level: Book Flight Stage Level: Select Flight Action Level: Click Result Pattern: Click <target> Semantic Description: role: âlinkâ, text_contains: âSelectâ, context: âprice listâ (a) Memory Retrieval(b) Planner Verification & Decision (c) Actor Grounding & Execution Current Observation: URL:trip.com/flights Content:Multiple flight options State Abstraction Page:Flight Search Results URL:trip.com/flights State:Flight list visible Verify Pre-conditions Required:âFlight list visible" Current:Satisfied Evidence:DOM contains flights Proceed to Actor Pass semantic description to Actor for grounding [Planner] Verified Stage "Select Flight". Pre-conditions met (List visible). [Planner] Authorized Actor execution. Passing semantic descriptor. [Actor] Received query: match role:'link', text_contains:'Select'. [Actor] Scanning DOM... Found 2 candidate elements. [Actor] Evaluating #ad-promo-banner -> REJECT (Role mismatch: âad' != 'link'). [Actor] Evaluating #new-btn-119 -> MATCH (Score: 0.98). [Actor] Output Action: Click(#new-btn- 119). Fig. 5. Visual analysis of a successful cross-website grounding trace demonstrated by HMT. (a) Memory Retrieval: The agent retrieves an abstract action pattern and a semantic descriptor from the hierarchical memory, explicitly discarding the site-specific raw identifier (#btn-sfo-136) from the source trace. (b) Planner Verification & Decision: To prevent workflow mismatch, the Planner verifies that the current page satisfies the stage pre-conditions (e.g., âFlight list visibleâ) before authorizing the Actor, ensuring actions are only executed in the correct context. (c) Actor Grounding & Execution: Guided by the semantic descriptor, the Actor scans the target DOM on Trip.com. It successfully distinguishes between a distractor advertisement and the correct flight selection button, executing the correct action despite the layout shift. TABLE IV COMPARISON OF TOP-3 RETRIEVED MEMORIES ON TRIPADVISOR. THE TASK IS âFIND A FLIGHT.â FLAT RETRIEVAL FETCHES TEMPORALLY MISALIGNED ACTIONS (E.G., CHECKOUT) DUE TO SEMANTIC OVERLAP, WHILE HMT RESTRICTS RETRIEVAL TO THE CURRENT WORKFLOW STAGE. RankMethodRetrieved Action Content (Simplified)Valid?Reasoning Flat Retrieval 1Action: Click âPlace Orderâ (from Expedia Checkout)âStage Mismatch: Attempting to checkout before selecting item. 2Action: Click âAdd to Cartâ (from Expedia Detail Page)âContext Pollution: Element does not exist on search page. 3Action: Click Product Title (from Expedia Search)âCorrect action, but ranked low due to noise. HMT (Ours) 1Pattern: Click ItemDescriptor: Role: link, Text: match queryâStage Aligned: Correctly identifies browsing behavior. 2Pattern: Sort PriceDescriptor: Role: dropdown, Text: âSortââValid Option: Contextually relevant for price constraints. 3Pattern: Next PageDescriptor: Role: button, Text: âNextââValid Option: Standard navigation on search pages. E. Efficiency Analysis Beyond accuracy, we evaluate the computational efficiency of HMT. Table V compares the average token consumption and latency per step on WebArena. By abstracting raw HTML trajectories into compact semantic descriptions, HMT reduces the average context length by approximately 72.7% compared to standard retrieval. Despite the two-stage inference process (Planner then Actor), the significant reduction in input tokens results in a lower overall latency (3.5s vs. 5.2s) and a 71.0% reduction in inference cost per task. F. Qualitative Analysis To provide a granular understanding of the proposed frame- work, we conduct a qualitative trace analysis of the decision- making process in both success and failure scenarios. This TABLE V EFFICIENCY COMPARISON ON WEBARENA. HMT REDUCES TOKEN CONSUMPTION AND LATENCY BY COMPRESSING RAW TRAJECTORIES INTO SEMANTIC NODES. Method Avg. Context Tokens Latency (s/step) Cost ($/Task) Flat Retrieval (Raw HTML)11,4505.20.38 HMT (Ours)3,1203.50.11 analysis highlights how the hierarchical structure facilitates knowledge transfer and identifies boundary conditions where current abstractions may fall short. We first visualize a successful cross-website knowledge transfer trace in Fig. 5, where the agent is tasked with booking a flight on Trip.com using memory derived from Expedia. 10 Action: "Load more items" Target: Click Button Semantic Descriptor: role: "button", text_contains: "more", context: "thread" <-- Missing hierarchical constraint Planner Status: Ambiguous candidates detected. Unable to resolve scope. Action: "Confirm Booking" Target: Click #btn-confirm Success Criteria (Retrieved): expected_outcome: "NAVIGATION", signals: ["url_change", "page_reload"] Planner Status: Expectation Mismatch. Visual update detected but insufficient to satisfy "NAVIGATION". [Actor] Query: role:'button', text_contains:'more' [Actor] Scanning DOM... Found 2 candidates. [Actor] Eval #btn-reply-sub (Distractor) -> Score 0.92 [Actor] Eval #btn-feed-load (Target) -> Score 0.89 [Actor] Selecting highest score: #btn-reply-sub [Actor] Output Action: Click(#btn-reply-sub) -> INCORRECT #btn-reply-sub (Distractor) -> Score 0.92 #btn-feed-load (Target) -> Score 0.89 [Actor] Action Executed: Click(#btn-confirm). [Planner] Validating outcome against expectation: NAVIGATION... [Planner] Signal: url_change = False [Planner] Signal: page_reload= False [Planner] Observation: Visual change detected (Modal), but insufficient. [Planner] Conclusion: Verification error. Retry triggered. -> LOOP DETECTED. (a) Failure Case 1: Ambiguous Grounding (b) Failure Case 2: State Verification Error Fig. 6. Analysis of two representative failure modes in cross-website grounding. (a) Ambiguous Grounding: The agent retrieves a generic descriptor (text_contains: more) lacking hierarchical constraints. Consequently, the Actor assigns a higher similarity (score 0.92) to the distractor candidate than to the intended target (score 0.89), leading to an execution error. (b) State Verification Error: The Planner retrieves a strict success criterion expecting a NAVIGATIONevent (e.g., URL change or page reload). Although the action triggers a visual update (modal popup), the URL remains static. The Plannerâs rigid verification logic (url_change = Falseandpage_reload = False) fails to recognize the visual progress, incorrectly marking the step as incomplete and initiating a retry loop. The process begins with the retrieval of a relevant action node. Crucially, the memory construction phase has already discarded the site-specific raw identifier from the source website, retaining only a transferable semantic description consisting of the elementâs role and textual content. Upon retrieval, the Planner initiates the inference process. It performs a state abstraction on the raw page observation and confirms that the pre-condition, specifically the visibility of a flight list, is satisfied. This step prevents the execution of actions in incorrect contexts. Subsequently, the Planner delegates execution to the Actor, passing down the semantic description. The Actor then scans the current DOM tree for matching elements. As illustrated in the execution log, the Actor encounters a distractor element, an advertisement with an ID of#ad-promo-banner, which semantically resembles the target but possesses a conflicting role attribute. The Actor correctly rejects this candidate based on the role mismatch and successfully grounds the action to the correct flight selection button identified as#new-btn-119, which exhibits a high semantic match score. This example demonstrates how the decomposition of planning and grounding effectively resolves the intention-execution entanglement. Despite these capabilities, the system exhibits limitations in scenarios involving Ambiguous Grounding or subtle state transitions, as detailed in Fig. 6. The first failure mode, Ambiguous Grounding, occurs when the retrieved semantic description lacks sufficient hierarchical context. In the social media example shown in Fig. 6(a), the agent intends to load more posts at the bottom of the feed. However, the interface contains a distractor button âShow more repliesâ within a specific comment thread. Since both buttons match the generic description of containing the text âmoreâ, the Actor identifies multiple high-scoring candidates. Lacking a precise parent-child structural constraint in the memory, the Actor assigns a higher similarity score to the distractor candidate than to the intended target âLoad more postsâ, leading to an execution error. The second failure mode, State Verification Error, is observed in Single Page Applications (SPAs) where visual updates do not trigger URL changes. As shown in Fig. 6(b), the agent successfully clicks the confirmation button, triggering a modal popup. However, the Plannerâs rigorous post-condition check requires a URL transition or a page reload to confirm stage completion. Since the URL remains static, the Planner incorrectly perceives the action as failed and initiates a retry loop, repeatedly executing the same action despite the visual progress. These cases suggest that future iterations of hierarchical memory should incorporate richer structural context descriptors and more versatile state verification mechanisms beyond DOM and URL signals. V. CONCLUSION In this work, we focus on the challenge of cross-website generalization by proposing the Hierarchical Memory Tree 11 (HMT), a structured framework that explicitly decouples logical planning from site-specific action execution. By autonomously constructing a hierarchy of standardized intents, stage-aware subgoals defined by observable pre-conditions and post- conditions, and semantic element descriptions, HMT effectively mitigates the intention-execution entanglement inherent in flat memory systems. Our stage-aware inference mechanism, comprising a Planner and an Actor, not only ensures robust grounding in novel environments but also significantly improves computational efficiency by compressing raw trajectories. Extensive evaluations on Mind2Web and WebArena confirm that HMT establishes a new standard for transferable web agents, paving the way for more scalable and autonomous systems capable of lifelong learning. REFERENCES [1] T. Shi, A. Karpathy, L. Fan, J. Hernandez, and P. Liang, âWorld of bits: An open-domain platform for web-based agents,â in International Conference on Machine Learning (ICML), 2017. [2] C. Shi, Z. Yu, Z. Gao, R. Feng, E. Liu, Y. Wu, Y. Jia, L. Xiang, Z. He, and Q. Li, âGui knowledge bench: Revealing the knowledge gap behind vlm failures in gui tasks,â arXiv preprint arXiv:2510.26098, 2025. [3]S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, âReAct: Synergizing reasoning and acting in language models,â in International Conference on Learning Representations (ICLR), 2023. [4]S. Hu, M. Ouyang, D. Gao, and M. Z. Shou, âThe dawn of GUI agent: A preliminary case study with Claude 3.5 computer use,â arXiv preprint arXiv:2411.10323, 2024. [5] B. Zhang, Z. Shang, Z. Gao, W. Zhang, R. Xie, X. Ma, T. Yuan, X. Wu, S.-C. Zhu, and Q. Li, âTongui: Internet-scale trajectories from multimodal web tutorials for generalized gui agents,â in AAAI Conference on Artificial Intelligence (AAAI), 2026. [6]N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, âReflexion: Language agents with verbal reinforcement learning,â in Advances in Neural Information Processing Systems (NeurIPS), 2023. [7] Z. Z. Wang, J. Mao, D. Fried, and G. Neubig, âAgent workflow memory,â in International Conference on Learning Representations (ICLR), 2025. [8]E. Z. Liu, K. Guu, P. Pasupat, T. Shi, and P. Liang, âReinforcement learning on web interfaces using workflow-guided exploration,â in International Conference on Learning Representations (ICLR), 2018. [9]V. Mnih et al., âHuman-level control through deep reinforcement learning,â Nature, vol. 518, no. 7540, p. 529â533, 2015. [10]C. Rawles et al., âAndroid in the wild: A large-scale dataset for Android device control,â in Advances in Neural Information Processing Systems (NeurIPS), 2024. [11]P. Li, Z. Hu, Z. Shang, J. Wu, Y. Liu, H. Liu, Z. Gao, C. Shi, B. Zhang, Z. Zhang et al., âEfficient multi-turn rl for gui agents via decoupled training and adaptive data curation,â arXiv preprint arXiv:2509.23866, 2025. [12] S. Yao, H. Chen, J. Yang, and K. Narasimhan, âWebShop: Towards scalable real-world web interaction with grounded language agents,â in Advances in Neural Information Processing Systems (NeurIPS), 2022. [13]L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang et al., âA survey on large language model based autonomous agents,â Frontiers of Computer Science, 2024. [14] OpenAI, âGPT-4 technical report,â arXiv preprint arXiv:2303.08774, 2023. [15]S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan, âTree of Thoughts: Deliberate problem solving with large language models,â in Advances in Neural Information Processing Systems (NeurIPS), 2023. [16]R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju et al., âWebGPT: Browser-assisted question- answering with human feedback,â arXiv preprint arXiv:2112.09332, 2021. [17] X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su, âMind2Web: Towards a generalist agent for the web,â in Advances in Neural Information Processing Systems (NeurIPS), 2023. [18]B. Zheng, B. Gou, J. Kil, H. Sun, and Y. Su, âGPT-4V(ision) is a generalist web agent, if grounded,â in International Conference on Machine Learning (ICML), 2024. [19]J. Zhang et al., âPrune4Web: DOM tree pruning programming for web agent,â in AAAI Conference on Artificial Intelligence (AAAI), 2026. [20]W. Zhao et al., âHtmlRAG: HTML is better than plain text for modeling retrieved knowledge in RAG systems,â in The Web Conference (W), 2025. [21]W. Hong, W. Wang, Q. Xu, X. Tang et al., âCogAgent: A visual language model for GUI agents,â in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. [22] C. Yang, X. Ma et al., âAppAgent: Multimodal agents as smartphone users,â in International Conference on Learning Representations (ICLR), 2024. [23]J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao, âSet-of-Mark prompting unleashes extraordinary visual grounding in GPT-4V,â in arXiv preprint arXiv:2310.11441, 2023. [24]P. Lewis et al., âRetrieval-augmented generation for knowledge-intensive NLP tasks,â in Advances in Neural Information Processing Systems (NeurIPS), 2020. [25]V. Karpukhin et al., âDense passage retrieval for open-domain question answering,â in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. [26]A. Asai, Z. Wu, Y. Wang et al., âSelf-RAG: Learning to retrieve, generate, and critique through self-reflection,â in International Conference on Learning Representations (ICLR), 2024. [27]G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar, âVoyager: An open-ended embodied agent with large language models,â Transactions on Machine Learning Research (TMLR), 2024. [28]C. Packer, V. Fang, S. G. Patil et al., âMemGPT: Towards LLMs as oper- ating systems,â in International Conference on Learning Representations (ICLR), 2024. [29]M. Pan et al., âEchoTrail-GUI: Building actionable memory for GUI agents via critic-guided self-exploration,â arXiv preprint arXiv:2512.19396, 2025. [30]W.-J. Zhou, Y. Zheng, Y. Feng, Y. Ye, R. Xiao, L. Chen, X. Yang, and J. Xiao, âENCODE: Breaking the trade-off between performance and efficiency in long-term user behavior modeling,â IEEE Transactions on Knowledge and Data Engineering (TKDE), vol. 37, no. 1, p. 265â277, 2025. [31]R. Guan, H. Pang, F. Giunchiglia, Y. Liang, and X. Feng, âCross- domain meta-learner for cold-start recommendation (MetaCDR),â IEEE Transactions on Knowledge and Data Engineering (TKDE), vol. 35, no. 8, p. 7829â7843, 2023. [32]Y. Guo et al., âSkillEvo: An experience learning framework with reinforcement learning for skill evolution,â in International Conference on Learning Representations (ICLR), 2026. [33]P. Anokhin et al., âAriGraph: Learning knowledge graph world models with episodic memory,â in Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2025. [34] R. S. Sutton, D. Precup, and S. Singh, âBetween MDPs and semi- MDPs: A framework for temporal abstraction in reinforcement learning,â Artificial Intelligence, 1999. [35]D. Zhou, N. Sch Ě arli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. Le, and E. Chi, âLeast-to-Most prompting enables complex reasoning in large language models,â in International Conference on Learning Representations (ICLR), 2023. [36]L. Wang, W. Xu, Y. Lan, Z. Hu, Y. Lan, R. K.-W. Lee, and E.-P. Lim, âPlan-and-Solve prompting: Improving zero-shot chain-of-thought reasoning by large language models,â in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023. [37]J. Wei et al., âChain-of-thought prompting elicits reasoning in large language models,â in Advances in Neural Information Processing Systems (NeurIPS), 2022. [38]X. Wang et al., âSelf-consistency improves chain of thought reasoning in language models,â in International Conference on Learning Represen- tations (ICLR), 2023. [39]A. Zhou, K. Yan et al., âLanguage agent tree search unifies reasoning acting and planning,â in International Conference on Machine Learning (ICML), 2024. [40]Z. Gou et al., âCRITIC: Large language models can self-correct with tool-interactive critiquing,â in International Conference on Learning Representations (ICLR), 2024. [41]Q. Wu et al., âAutoGen: Enabling next-gen LLM applications via multi- agent conversation,â arXiv preprint arXiv:2308.08155, 2023. [42] A. Rezazadeh, Z. Li, W. Wei, and Y. Bao, âFrom isolated conversations to hierarchical schemas: Dynamic tree memory representation for LLMs,â in International Conference on Learning Representations (ICLR), 2025. 12 [43]M. Hu, T. Chen, Q. Chen, Y. Mu, W. Shao, and P. Luo, âHiAgent: Hierarchical working memory management for solving long-horizon agent tasks with large language model,â in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025. [44]Y. Cao, S. Shang, J. Wang, and W. Zhang, âExplainable session-based recommendation via path reasoning,â IEEE Transactions on Knowledge and Data Engineering (TKDE), vol. 37, no. 1, p. 278â290, 2025. [45]D. Wang, B. Li, B. Song, C. Chen, and F. R. Yu, âHSMH: A hierarchical sequence multi-hop reasoning model with reinforcement learning,â IEEE Transactions on Knowledge and Data Engineering (TKDE), vol. 36, no. 4, p. 1638â1649, 2024. [46]S. Zheng, W. Chen, W. Wang, P. Zhao, H. Yin, and L. Zhao, âMulti-hop knowledge graph reasoning in few-shot scenarios,â IEEE Transactions on Knowledge and Data Engineering (TKDE), vol. 36, no. 4, p. 1713â1727, 2024. [47]Y. Qin, S. Liang et al., âToolLLM: Facilitating large language models to master 16000+ real-world APIs,â in International Conference on Learning Representations (ICLR), 2024. [48] F. Yang et al., âLLM-guided reinforcement learning for interactive environments,â MDPI Mathematics, vol. 13, no. 12, 2025. [49]A. Shek and P. Tokekar, âOption discovery using LLM-guided semantic hierarchical reinforcement learning,â arXiv preprint arXiv:2503.19007, 2025. [50]M. L. Dihan, T. Hashem, M. E. Ali, and M. R. Parvez, âWeboperator: Action-aware tree search for autonomous agents in web environment,â arXiv preprint arXiv:2512.12692, 2025. [51]S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, Y. Bisk, D. Fried, U. Alon, and G. Neubig, âWebArena: A realistic web environment for building autonomous agents,â in International Conference on Learning Representations (ICLR), 2024. [52]P. Sodhi et al., âHierarchical step-by-step prompting for web navigation,â in arXiv preprint arXiv:2305.14257, 2023. [53]Y. Pan et al., âAutonomous evaluation and refinement of web agents,â in arXiv preprint arXiv:2404.06456, 2024.