Paper deep dive
IntentCUA: Learning Intent-level Representations for Skill Abstraction and Multi-Agent Planning in Computer-Use Agents
Seoyoung Lee, Seobin Yoon, Seongbeen Lee, Yoojung Chun, Dayoung Park, Doyeon Kim, Joo Yong Sim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/21/2026, 12:38:26 AM
Summary
The paper introduces IntentCUA, a multi-agent framework for computer-use agents that stabilizes long-horizon execution through intent-aligned plan memory. It abstracts raw interaction traces into multi-view intent representations and reusable skills, organized hierarchically. A Planner, Plan-Optimizer, and Critic coordinate to retrieve subgroup-aligned skills, reducing redundant re-planning and mitigating error propagation. IntentCUA achieves a 74.83% task success rate and a Step Efficiency Ratio of 0.91, outperforming RL-based and trajectory-centric baselines.
Entities (12)
Relation Signals (10)
IntentCUA → containsagent → Planner
confidence 95% · A Planner, Plan-Optimizer, and Critic coordinate over shared memory...
IntentCUA → containsagent → Plan-Optimizer
confidence 95% · A Planner, Plan-Optimizer, and Critic coordinate over shared memory...
IntentCUA → containsagent → Critic
confidence 95% · A Planner, Plan-Optimizer, and Critic coordinate over shared memory...
IntentCUA → usescomponent → Plan Memory
confidence 92% · IntentCUA, a multi-agent computer-use framework designed to stabilize long-horizon execution through intent-aligned plan memory.
IntentCUA → achievesmetric → Step Efficiency Ratio
confidence 90% · achieved a 74.83% task success rate with a Step Efficiency Ratio of 0.91
IntentCUA → achievesmetric → Task Success Rate
confidence 90% · achieved a 74.83% task success rate
Skill Hints → derivedfrom → Subgroups (SG)
confidence 88% · SG action patterns are converted into parameterized skill schemas (“skill hints”) and stored with their SG
Subgroups (SG) → partof → Intent Groups (IG)
confidence 88% · Within each IG, a second HDBSCAN partitions units into finer subgroups (SG)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Computer-use agents operate over long horizons under noisy perception, multi-window contexts, evolving environment states. Existing approaches, from RL-based planners to trajectory retrieval, often drift from user intent and repeatedly solve routine subproblems, leading to error accumulation and inefficiency. We present IntentCUA, a multi-agent computer-use framework designed to stabilize long-horizon execution through intent-aligned plan memory. A Planner, Plan-Optimizer, and Critic coordinate over shared memory that abstracts raw interaction traces into multi-view intent representations and reusable skills. At runtime, intent prototypes retrieve subgroup-aligned skills and inject them into partial plans, reducing redundant re-planning and mitigating error propagation across desktop applications. In end-to-end evaluations, IntentCUA achieved a 74.83% task success rate with a Step Efficiency Ratio of 0.91, outperforming RL-based and trajectory-centric baselines. Ablations show that multi-view intent abstraction and shared plan memory jointly improve execution stability, with the cooperative multi-agent loop providing the largest gains on long-horizon tasks. These results highlight that system-level intent abstraction and memory-grounded coordination are key to reliable and efficient desktop automation in large, dynamic environments.
Tags
Links
- Source: https://arxiv.org/abs/2602.17049v2
- Canonical: https://arxiv.org/abs/2602.17049v2
Trouble viewing inline? Open PDF directly →
Full Text
57,010 characters extracted from source content.
Expand or collapse full text
IntentCUA: Learning Intent-level Representations for Skill Abstraction and Multi-Agent Planning in Computer-Use Agents Seoyoung Lee ∗ Sookmyung Women’s University Seoul, Republic of Korea leesy3891@gmail.com Seobin Yoon ∗ Sookmyung Women’s University Seoul, Republic of Korea binsong2@sookmyung.ac.kr Seongbeen Lee † Sookmyung Women’s University Seoul, Republic of Korea seongbeen@sookmyung.ac.kr Yoojung Chun Sookmyung Women’s University Seoul, Republic of Korea yj.chun@sookmyung.ac.kr Dayoung Park Sookmyung Women’s University Seoul, Republic of Korea pdysicist@sookmyung.ac.kr Doyeon Kim Sookmyung Women’s University Seoul, Republic of Korea ehdus@sookmyung.ac.kr Joo Yong Sim † Sookmyung Women’s University Seoul, Republic of Korea jysim@sookmyung.ac.kr ABSTRACT Computer-use agents operate over long horizons under noisy per- ception, multi-window contexts, evolving environment states. Ex- isting approaches, from RL-based planners to trajectory retrieval, often drift from user intent and repeatedly solve routine subprob- lems, leading to error accumulation and inefficiency. We present IntentCUA, a multi-agent computer-use framework designed to stabilize long-horizon execution through intent-aligned plan memory. A Planner, Plan-Optimizer, and Critic coordinate over shared memory that abstracts raw interaction traces into multi- view intent representations and reusable skills. At runtime, intent prototypes retrieve subgroup-aligned skills and inject them into partial plans, reducing redundant re-planning and mitigating error propagation across desktop applications. In end-to-end evaluations, IntentCUA achieved a 74.83% task success rate with a Step Efficiency Ratio of 0.91, outperforming RL- based and trajectory-centric baselines. Ablations show that multi- view intent abstraction and shared plan memory jointly improve execution stability, with the cooperative multi-agent loop providing the largest gains on long-horizon tasks. These results highlight that system-level intent abstraction and memory-grounded coordina- tion are key to reliable and efficient desktop automation in large, dynamic environments. KEYWORDS Computer-use agents, Long-horizon automation, Noisy perception, Multi-window context, Multi Agent Planning ∗ Both authors contributed equally to this research. † Corresponding author. This work is licensed under a Creative Commons Attribution Inter- national 4.0 License. Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026), C. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.), May 25 – 29, 2026, Paphos, Cyprus.© 2026 International Foundation for Autonomous Agents and Multiagent Systems (w.ifaamas.org). https://doi.org/10.65109/BRAG3288 ACM Reference Format: Seoyoung Lee, Seobin Yoon, Seongbeen Lee, Yoojung Chun, Dayoung Park, Doyeon Kim, and Joo Yong Sim. 2026. IntentCUA: Learning Intent-level Rep- resentations for Skill Abstraction and Multi-Agent Planning in Computer- Use Agents. In Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026), Paphos, Cyprus, May 25 – 29, 2026, IFAAMAS, 12 pages. https://doi.org/10.65109/BRAG3288 1 INTRODUCTION Rule-based macros and RPA systems enabled early forms of computer- use automation. However, they lack adaptability [11,33] when compared to recent GUI agents powered by large language models (LLMs) that can interpret screens and generate actions dynamically. Research on GUI agents has rapidly expanded, spanning web, mobile, and increasingly desktop environments [26,40]. As high- lighted by Tang et al. [32], automation across all desktop envi- ronments remains particularly challenging due to multi-window operations, OS-level shortcuts and APIs, and the need to adapt to frequent updates and complex software ecosystems. Within such environments, achieving robust long-horizon planning and man- aging multi-context workflows emerge as central challenges that current systems have yet to overcome. Recent multi-modal agents attempt to address these challenges by perceiving screens and generating actions with large models [1, 38]. However, robust long-horizon automation across heteroge- neous desktop applications remains unresolved [25,32]. We identify two recurring failure modes: (i) plans spanning multiple substeps often drift from the original intent and redundantly re-solve already completed routines [23,24], (i) local perception errors accumulate and lead to cascading retries [8,15,19,44]. These factors collec- tively hinder robust long-horizon planning, as agents frequently fall into inefficient and repetitive re-planning cycles. Actions are often retried or nullified when context drifts, leading to prolonged latency and unstable completion rates. To address these limitations, we bridge user interaction and multi-agent planning. Rather than simply replaying trajectories or storing textual reflections [28], we transform interaction traces into arXiv:2602.17049v2 [cs.AI] 27 Feb 2026 Figure 1: Overview of IntentCUA. Offline: raw user traces are multi-view labeled, embedded into a shared intent space, and clustered into intent groups (IG) and subgroups (SG); SG action patterns are converted into parameterized skill schemas (“skill hints”) and stored with their SG in the IG/SG index, while plan memory stores only user-approved global plans (G). Online: the Planner/Plan-Optimizer/Critic query and reuse skills; cache-first reuse and template-based gap filling reduce re-planning on long-horizon desktop tasks. labeled units, induce generalized skills from sub-intent clusters, and learn multi-view representations across environment, action, keyword, and description. These skills are organized hierarchically in a plan memory and retrieved via semantic search during planning, which supports cross-application transfer and helps stabilize long roll-outs. At run- time, intent prototypes are projected into a shared embedding space, where centroid-based retrieval augments partial plans with relevant skills. In end-to-end evaluations, IntentCUA achieves a 74.83% task suc- cess rate with a Step Efficiency Ratio (SER) of 0.91, outperforming both RL-based (UI-TARS-1.5 [27]) and trajectory-centric (UFO 2 [41]) baselines in success rate, efficiency, and latency. Ablation studies confirm that multi-view intent abstraction and shared plan mem- ory jointly improve execution stability, with the cooperative multi- agent loop providing the largest gains on long-horizon tasks. These results indicate that system-level intent abstraction and memory- grounded coordination are central to reliable desktop automation. Our contributions are summarized as follows: (1) We propose IntentCUA, a multi-agent computer-use frame- work that stabilizes long-horizon execution through intent- aligned plan memory and coordinated planning. (2) We introduce a trace-to-skill abstraction pipeline that learns multi-view intent representations and induces hierarchical, reusable skills from raw user interaction traces. (3) We design a planning-time memory mechanism that re- trieves subgroup-aligned skills to augment partial plans, re- ducing intent drift and redundant re-planning in dynamic desktop environments. (4)We demonstrate through extensive ablations and end-to-end evaluations that intent abstraction and memory-grounded coordination significantly improve execution stability, ef- ficiency (SER 0.91), and task success (74.83%) on complex desktop workflows. In summary, IntentCUA shows that intent-level abstraction and memory-grounded multi-agent coordination are key to stabilizing long-horizon desktop automation in large, dynamic environments. This automation is made possible by a robust planning policy that maintains coherence and efficiency across extended sequences. 2 RELATED WORK 2.1 Desktop and GUI Automation Agents GUI automation spans web, mobile, and desktop domains. Web agents such as WebArena and WebVoyager operate under struc- tured DOM feedback [7,45], but real desktop environments lack such schema-level constraints and require cross-application coordi- nation. Desktop benchmarks like OSWorld [36] tasks are typically long- horizon, requiring stable execution 10–20 sequential steps. This makes execution latency-sensitive and error-prone. As step count increases, local perception errors compound and agents often enter loops of repeated or failed actions. Recent desktop agents such as UI-TARS [22], UFO [42], ScreenA- gent [20] extend vision-language models with planner–critic loops. However, surveys highlight a persistent challenge: determining actions that align with specific user contexts and preferences in dynamic, interruption-prone interfaces [32]. Even with improved GUI grounding [8,10,19], intent drift and redundant re-planning remain common in long-horizon workflows. These findings indicate that stable long-horizon planning, rather than perception alone, remains the key bottleneck for reliable desk- top automation. 2.2 Agents Leveraging Interaction Traces One approach to addressing long-horizon instability is to learn directly from large-scale interaction traces. Macro-mining and process-mining techniques cluster demonstra- tions into recurrent procedures or labeled schemas [4,5,9]. Large- scale corpora such as OS-ATLAS support perception pretraining across millions of GUI elements [35]. Offline reinforcement learning has also been explored for device agents [29], while systems such as AppAgentv2 [14], AgentBank [31], and UI-TARS-1.5 [27] leverage hierarchical feedback or large-scale trajectory tuning to improve control robustness. These works demonstrate that interaction traces improve policy generalization and low-level stability. However, most approaches operate at the trajectory or action level, emphasizing replay or large-scale tuning rather than structured intent abstraction. As a result, redundancy and error accumulation often persist in long- horizon execution [24], and reliance on controlled environments or explicit reward signals limits applicability to open-ended desktop workflows [26]. 2.3 Plan Memory, Intent Identification, and Skill Abstraction A complementary direction enhances robustness through memory retrieval and skill abstraction. Memory-based methods such as Reflexion [28], Conversational Memory [34], and Contextual Experience Replay [18] retrieve prior trajectories, manuals, or reflections to guide future decisions [2]. Skill-level prompting approaches such as SkillAct [17] show that abstracted routines can improve interactive performance, while UFO 2 [41] manages app-specific demonstrations as reusable refer- ences. Figure 2: Multi-view intent representation. Control traces use [E,A,D], browsing traces [E,K,D]. A multi-view encoder aligns views into a shared space, inducing environment-centric IG and finer SG. SG centroids enable retrieval, and SG action pat- terns are converted into parameterized skill schemas (“skill hints”) with verb–argument structure for planning. Parallel work investigates intent recognition from UI logs [13, 23] and representation learning of screens (e.g., Screen2Vec [12], Aria-UI [37]), while GUI grounding methods reduce perceptual ambiguity [15,19]. More recent systems explore adaptive planning and dependency modeling from demonstrations [39, 43]. Despite these advances, structured and hierarchical skill abstrac- tions that remain transferable across heterogeneous desktop work- flows are still relatively underexplored. As a result, maintaining stable long-horizon execution under dynamic user contexts contin- ues to be an active area of research. Our approach complements these directions by learning multi-view intent representations that integrate environment, action, and description signals. Skills are stored as hierarchical intent prototypes in plan memory and re- trieved to augment partial plans, supporting stable long-horizon execution [6, 30]. 3 INTENT-LEVEL REPRESENTATION LEARNING & SKILL ABSTRACTION 3.1 Intent-level Representation Learning In this section, we describe how raw user traces are transformed into unified intent-level representations that can be clustered and later abstracted into reusable skills. As shown in Figure 2, each user trace is first labeled across four views: environment (퐸with instances푒 푖 ), action (퐴with instances 푎 푖 ), keyword (퐾with instances푘 푖 ), and description (퐷with in- stances푑 푖 ), where푖indexes the sequential intent units that together compose a user’s interaction trace. Each view푣 ∈ 퐸,퐴,퐾,퐷is rep- resented as an embedded textual vector, capturing its semantic con- tent. Control traces produce intent units,푢 푖 , of the form[푒 푖 ,푎 푖 ,푑 푖 ], while browsing traces yield[푒 푖 ,푘 푖 ,푑 푖 ]. Formally, let푥 (푣) 푖 denote the feature representation of intent unit푢 푖 in view푣 ∈ 퐸,퐴,퐾,퐷. A multi-view encoder휙(푥 (푣) )maps these view-specific features into a single shared representation 푧 푖 : 푧 푖 = 휙 (푥 (푣) 푖 ) 푣∈푉 ∈R 푑 , 푉 ⊆ 퐸,퐴,퐾,퐷.(1) Building on prior multi-view clustering objectives [16], we train the model to ensure that representations from heterogeneous views are (i) contradistinctively aligned, (i) cross-view predictive, and (i) reconstructible. The overall loss is defined as the weighted sum of these three components, as shown in Equation 2: L=L con + 휆 pred L pred + 휆 rec L rec (2) whereL con ,L pred ,L rec are cross-view contrastive loss, dual pre- diction loss, and within-view reconstruction loss, respectively. L con , enforces consistency between embeddings from different views of the same intent unit while separating embeddings from different instances: L con = 1 |푃(푉)| ∑︁ (푝,푞)∈푃(푉) " − 1 푁 푁 ∑︁ 푖=1 log exp ⟨푧 (푝) 푖 , 푧 (푞) 푖 ⟩/휏 Í 푗≠푖 exp ⟨푧 (푝) 푖 , 푧 (푞) 푗 ⟩/휏 # (3) where,푃(푉)denotes the set of all ordered positive view pairs (푝,푞)within the selected view set푉. The temperature parameter휏 controls the sharpness of the contrastive distribution, whereas푁 denotes the number of intent units sampled in a minibatch. L pred introduces two projection heads퐺 푝→푞 and퐺 푞→푝 that learn to predict the embedding of one view from another. Their aver- aged mapping퐺=(퐺 푝→푞 +퐺 푞→푝 )/2 acts as a symmetric predictor encouraging cross-view consistency—ensuring that one view can reconstruct another within the latent space. The scalar coefficient 휆 pred balances this term with the others in Equation 2. L pred = 1 |푃(푉)| ∑︁ (푝,푞)∈푃(푉) 1 2푁 푁 ∑︁ 푖=1 h ∥퐺 푝→푞 (푧 (푝) 푖 )−푧 (푞) 푖 ∥ 2 2 +∥퐺 푞→푝 (푧 (푝) 푖 )−푧 (푞) 푖 ∥ 2 2 i (4) Finally,L rec ensures that the shared embedding푧 푖 retains view- specific semantics by reconstructing each original feature푥 (푢) 푣 through a decoder푔 푣 (·). The weight휆 rec determines the relative strength of this reconstruction constraint within the total loss in Equation 2. L rec = 1 |푉| 푁 ∑︁ 푣∈푉 푁 ∑︁ 푖=1 ∥푔 푣 (푧 (푣) 푖 )− 푥 (푣) 푖 ∥ 2 2 (5) Together, these three objectives jointly align, predict, and recon- struct heterogeneous views, yielding a coherent embedding space where intent-level semantics are preserved. The resulting represen- tation푧 푖 compactly captures user intent across multiple modalities. We then organize푢 푖 hierarchically by푧 푖 : first into higher-level intent groups (퐼퐺) driven by environment context, and then into subgroups (푆퐺) based on action/keyword and description. These 푆퐺representations provide the foundation for extracting recurrent action patterns and constructing abstract skills, as described in Section 3.2. Details of the encoder are in Appendix A. 3.2 Skill abstraction based on Intent Subgroups Given the per-unit embeddings푧 (푢) from Section 3.1, we organize intents into a two-level index for planning. First, we run HDB- SCAN [3] over푧 (푢) to obtain higher-level intent groups (퐼퐺) driven primarily by environment/context. Within each퐼퐺, a sec- ond HDBSCAN partitions units into finer subgroups (푆퐺) using action/keyword and description signals. For every푆퐺, we compute and store its centroid푐 푆퐺 in the same embedding space. At retrieval time, we rank subgroups by the cosine similarity between푐 푆퐺 and a query intent prototype. Retrieval index: for each푆퐺we store (i) the centroid푐 푆퐺 (for cosine-based ranking); (i) top-푘representative traces preselected by proximity to푐 푆퐺 and reranked at query time by similarity to the intent prototype; and (i) a support count for 푆퐺 (defined below) used to prefer stable patterns during planning. A detailed ablation on the representation loss and퐼퐺/푆퐺Gating are reported in Appendix C. We then consolidate the low-level action traces inside each푆퐺 into a reusable skill. Each intent unit푢 푖 in a subgroup is linked to a low-level action sequence:푀 푖 = (푎 1 ,푎 2 , . . .,푎 푚 ) 푖 .. To make traces comparable, we map every atomic action푎 푡 to a pair [verb predicate, typed argument fields] by applying an alias mapΦthat collapses surface variants to a fixed predicate and a fixed set of typed fields. For example, "focus URL bar" and "open web site" →verb=press, arg=address_bar, verb=text_input, arg=address_bar, text:"https://example.com". Here, a verb signature is the ordered list of canonical predicates in a trace, and an typed argument field is a placeholder (e.g.,<url>,<query>,<file_path>) that will be bound at runtime. We collect candidatessg skills,푘 =푀 푖 | 푢 푖 ∈ 푆퐺 푘 and induce a skill prototype as the medoid under a signature-level dissimilarity 푑sigover verb-predicate sequences. The function푑 sig is computed on the canonicalized verb-predicate sequences (after applyingΦ), comparing action patterns at the predicate level while deferring literal-argument handling to the parameterization stage. LetA denote the verb-predicate alphabet. Thus the subgroup’s skill pro- totype is defined as: S 푆퐺 = arg min 푎∈퐴 ∗ ∑︁ 푠∈푠푔 푠푘푖푙푠 d sig (푎,푠)(6) Next we convertS 푆퐺 into a reusable, parameterized schema, skill hint: a verb-predicate sequence together with a typed argument structure. This conversion (i) replaces literal values with typed parameters (the runtime-filled fields), (i) removes incidental or recovery-specific steps that do not affect goal attainment, and (i) enforces canonical predicate and field names viaΦ. We refer to this parameterized schema as a skill hint. Both the skill hintsS푆퐺and the representative traces are stored in plan memory and retrieved at planning time. If multiple predicate sequences are well supported, we keep several schemas ranked by their support, where support counts sub- group members whose similarity toS 푆퐺 exceeds a fixed threshold 휏. Representative traces are the top-푘members minimizing푑sigto S푆퐺and serve as concrete exemplars. At planning time, when a retrieved plan is only a partial match, we perform gap filling: we instantiate the selected skill hintS푆퐺with current-context bind- ings and insert the resulting steps to complete the missing plan units/steps (Section 4.1). 4 INTENT-AWARE PLANNING & FEEDBACK MEMORY Building on the intent-level DB introduced in Section 3, we now focus on how these abstractions are leveraged during planning and execution. This section details the end-to-end workflow in which the Planner, Plan-Optimizer, and Critic cooperate through plan memory to compose, refine, and verify long-horizon automation. The Planning-Automation part in figure 1 represents the overall automation process after the user request. 4.1Planning with Plan Memory and Skill Hints This section explains how the Planner agent composes a high-level plan퐺for a given command; Figure 3 summarizes three pathways: cache miss synthesis (Case 1), direct reuse on exact hit (Case 2), and reuse-with-injection on partial hit (Case 3). At a high level, Planner consults퐼퐺/푆퐺centroids in the shared embedding space, plan- memory entries, and푀(푢)sequences; missing spans are completed with normalized 푆 푆퐺 templates. When no suitable plan exists in plan memory (Figure 3, Case 1), the Planner embeds the command into the shared space, gates by the active퐼퐺, and ranks candidate푆퐺by centroid similarity. Figure 3: Cache-first planning with plan memory. A query intent is gated by IG and ranked over SG. Case 1 (miss): syn- thesize a plan from retrieved skill templates. Case 2 (hit): reuse the stored plan. Case 3 (partial): align to the nearest plan and fill gaps with SG-derived skill hints, reducing re- tries. Plan memory stores previously synthesized global plans퐺that received binary user approval. Plans not approved are discarded. For each plan-unit slot in the intent prototype, it selects the nearest 푆퐺, retrieves the top-2 intent units (푢 푖 ) from that푆퐺, and uses the GPT-4o model [21] to generate the slot’s steps conditioned on the retrieved푀 푖 sequences. A plan unit is a contiguous block of low-level steps in the global plan퐺that achieves one subgoal. A plan-unit slot is the placeholder for such a block in the intent prototype that the Planner must populate with executable steps. Concatenating all slots yields a high-level plan퐺=푔 1 , . . .,푔 푛 , where each푔 푗 expands to a contiguous, execution-ordered list of low-level actions. After the user reviews the generated plan and provides optional feedback, we incorporate the edits and then store 퐺in plan memory for future reuse. Concretely,퐺is materialized as plan units—intent-prototype–level chunks of the퐺derived from the initial user command:Plan Unit 1: [step_1, step_2, ...] Plan Unit 2: [step_푘, ..., step_ℓ] ... When a high-similarity plan is found in active퐼퐺(Figure 3, Case 2), the stored퐺is retrieved and its slots are bound to the current context; because no gaps remain, the planner skips the synthesis and executes the plan as-is. When only a partial match is found (Figure 3, Case 3), the closest stored plan퐺is aligned with the intent prototype. Insert missing plan units or steps by injecting the matched subgroup’s푆 푆퐺 with current-context bindings (gap filling), after which the finalized 퐺is executed. This cache-first pipeline reduces re-planning and stabilizes long-horizon execution by combining centroid gating, plan reuse, and hint-based gap filling. 4.2 Optimizing steps by memory & feedback loop Given a finalized plan퐺, execution shifts into a cooperative loop between the Plan-Optimizer and the Critic Agent, utilizing the Plan Memory. The Plan-Optimizer refines each plan unit by referenc- ing similar traces stored in the memory, dynamically adapting its substeps to current screen contexts. The Critic, in turn, monitors the execution and provides immediate feedback signals—success, retryable, or blocked—to correct local deviations or trigger partial replanning when necessary. For each plan unit푝푢, we compute its representation푧 (푝푢) ∈R 푑 (Section 3.1) and compare it with subgroup centroids푐 푆퐺 from plan memory (Section 3.2). If a subgroup is relevant, its traces are injected as hints into the Plan-Optimizer to guide step execution. After each unit, the Critic evaluates the post-execution state and returns푞 ∈ success, retryable, blockedwith an observation표. If q is retryable, the Plan-Optimizer is re-invoked on the latest state푠 after with observation observation of the current GUI context (표) to produce an adjusted subplan푔 ′ new , which updates퐺before re-execution. As shown in Algorithm 1, the Planner hands over the plan units to the Plan-Optimizer, which integrates hints from prior traces to refine step execution. The Critic then decides whether to proceed, request an adjustment, or terminate. Through this memory-guided collaboration, specialized agents coordinate to minimize redun- dant re-planning and improve robustness by reusing traces that previously led to success. Table 1: Component-wise ablation on planning. We report task success (%) and averaged plan completion (%). MethodSuccess (%)↑Completion (%)↑ 퐵22.7333.78 퐵+푇 푔 46.4357.41 퐵+푇 푆퐺 + 푍54.6477.56 퐵+푇 푆퐺 + 푆 푆퐺 + 푃푀53.8581.23 퐵+푇 푆퐺 + 푍 + 푃푀62.5185.00 퐵+푇 푆퐺 + 푍 + 푆 푆퐺 + 푃푀74.8391.14 Algorithm 1 Execution of the Plan utilizing Memory and Feedback Loop Require:Final global plan퐺with plan units푃푈=푝푢 1 , . . .,푝푢 푀 , where each푝푢is an ordered list of steps; subgroup collection푆퐺 with centroids푐 푆퐺 ∈R 푑 (representation space from Section 3.1); for every푝푢 ∈ 푃푈, its representation푧 (푝푢) ∈R 푑 (precomputed via the encoder in Section 3.1); action space 퐴. Ensure: Execution outcome 1: for each plan unit 푝푢 in 푃푈 do 2:hint← InjectHint(pu, 푆퐺 , 푧 (푝푢) ) 3: for each step 푔 in 푝푢 do 4:푠 ← GUI Grounding of the current screen 5: (푎,푔 ′ ,표) ← Plan-Optimizer(푠,푔,퐺,푝푢,표, hint) 6:Execute action 푎 in the current GUI context 7: 푠 after ← GUI Grounding of the screen after finishing 푝푢 8: (푞,표) ← Critic(푝푢,퐺,푠 after ) 9: if 푞== success then 10:continue to next 푝푢 11: else if 푞== retryable then 12: (푎,푔 ′ new ,표) ← Plan-Optimizer(푠 after ,푔,퐺,푝푢,표, hint) 13:Apply 푔 ′ new to adjust the prior 푔 ′ 14: else 15:return(퐺, BLOCKED) 16: return(퐺, SUCCESS) GUI Grounding refers to the process of enumerating all action- able GUI components on the current screen, similar to the screen parsing method used in UFO [42]. The resulting state푠includes such component data together with summary metadata, composed of window title, panel names and component counts captured from the environment. Each step푔denotes an individual operation in the global plan퐺, composed of an action푎(e.g., click, text input, open) and its corresponding object targets; thus푎specifies the interaction primitive, whereas푔represents the full executable tuple(푎, object). The InjectHint function searches the plan memory for previous plan units whose representations푧 (푝푢) are most similar to the current one, and uses their traces as contextual hints guiding the next execution steps. Example of Planner-Plan-Optimizer-Critic interactions is in Appendix B. 5 ABLATION & CASE STUDIES 5.1 Evaluation Setup We evaluate our design on 286 real-world GUI tasks: 100 in-house, 116 from WebVoyager [7] (643 total), and 70 from ScreenAgent [20] (70 sessions). Tasks span local applications, web platforms, produc- tivity tools, and cross-application workflows. For task mining, we collect 30 active hours of interaction traces across 18 sessions, yielding 113 trace files. The mined corpus is inten- tionally distribution-shifted from the test suite: traces skew toward Local/App, while the evaluation set contains more Web/Crossover tasks. The traces cover 36 domains, whereas the 286-task suite spans 63 domains; only 22 overlap (34.92%), leaving 41 unseen test domains (65.07%). This setup stresses generalization of mining and retrieval rather than memorization. Domain distributions are detailed in Appendix D. All agents use the same atomic GUI action interface and iden- tical timeout policies. We report task success (74.83%), average completion ratio (91.14%), Step Efficiency Ratio(successful steps / actual execution steps; higher is better). Differences are reported in percentage points (p). 5.2 End-to-End Execution of Ablated Models We ablate each component to analyze its impact on long-horizon planning stability and execution depth. For Table 1, Completion denotes averaged plan completion (exe- cuted steps / synthesized plan steps), measuring execution progress beyond binary success. We denote components as:퐵(baseline planner–executor),푇 푔 , 푇 푆퐺 (greedy vs.퐼퐺/푆퐺-gated trace retrieval),푍(intent-level repre- sentation), 푆 푆퐺 (subgroup skill hints), 푃푀 (plan memory reuse) Starting from퐵(22.73% success), adding greedy trace retrieval (퐵+푇 푔 ) improves success by +23.7, p showing that user traces substantially reduce cold-start errors but still induce drift in long horizons. Replacing greedy retrieval with퐼퐺/푆퐺gating +푍brings a further +8.21, p success gain and +20.15, p completion gain, showing that organizing traces into representation-learned intent subgroups significantly stabilizes long-horizon execution. Even without푍, combining푆 푆퐺 +푃푀achieves a high task completion rate (81.23%), indicating that skill hints and plan memory alone substantially improve execution depth. The full system (퐵+푇 푆퐺 +푍+푆 푆퐺 +푃푀) achieves the best success and completion overall, demonstrating that representation learning, intent-level gating, skill abstraction, and plan reuse act complemen- tarily. Evaluation on step-wise planning consistency of the full system is in Appendix E. 5.3 Case Study To illustrate the framework, we consider a task where the user asks the system to summarize a previously viewed lightweight-ML video and record the result in a personal workspace. This requires retrieval, reasoning, and coordination across multiple applications. Figure 4 shows the execution process. The Planner retrieves rel- evant traces from plan memory, including prior interactions with the video platform, AI chatbot, and workspace application, and re- constructs the video source from the user’s history. It decomposes the request into structured intent-level plan units, each grounded into executable GUI actions by the Plan-Optimizer, while the Critic monitors progress and handles local inconsistencies. Through hier- archical reasoning and memory-guided skill retrieval, the system Figure 4: IntentCUA in action: the system recalls intent units from memory and decomposes a multi-application command into intent-level plan units, each executed through learned skills and recomposed into an end-to-end automation plan. completes the multi-application task while maintaining intent co- herence. A failure case occurs when an unexpected pop-up appears during execution. Because underlying components become occluded, the grounding module fails to detect them, leading to incorrect retries. This highlights a limitation of script-based GUI grounding under transient interface changes. 6 EXPERIMENTS We compare IntentCUA against two representative desktop GUI agents chosen for methodological diversity: UI-TARS-1.5 [27], an RL-based visual planner–executor with self-evolving policies and screen grounding, and UFO 2 [41], a trajectory-centric Windows automation agent that organizes demonstrations as executable se- quences. Together these baselines span reinforcement learning–driven automation versus demonstration-driven planning, and both op- erate at the level of atomic GUI actions, ensuring comparability with our interface. We evaluate 286 tasks (the same evaluation suite described in Section 5.1) and report task success rate, Step Efficiency Ratio(SER), and Latency, further analyzing robustness by step-length bins, each step defined as a atomic action performed by the agent. SER is defined as the ratio of successful steps to total steps, ranging from 0 to 1. Latency is measured as the execution time per task, reflecting not only the number of steps but also the overhead of perception and planning. 6.1 Robust Long-Horizon Planning Efficiency We evaluate how each agent sustains task completion as sequence length increases, focusing on the robustness of long-horizon plan- ning. Table 2 and Figure 5 summarize overall and step-wise success trends across 286 evaluation tasks. IntentCUA achieves the highest overall success rate of 74.8%, compared to 51.2% for UFO 2 and 38.8% for UI-TARS-1.5, yielding relative improvements of about +23.6 and +36 percentage points, respectively. Notably, IntentCUA performs consistently well across all datasets, achieving 71.6% on the web-based WebVoyager, 77.1% on the cross- application ScreenAgent, and 78.0% on our in-house local suite, Table 2: End-to-end success rate comparison across datasets (%). Columns show WebVoyager, ScreenAgent, our in-house suite, and overall average. MethodWebVoyagerScreenAgentOursTotal(%) UI-TARS-1.5 [27]35.942.946.038.8 UFO 2 [41]69.041.438.051.2 IntentCUA (ours)71.677.178.0 74.8 Figure 5: Success rate by step length (bin size = 5 steps). The x- axis shows step-length bins and the y-axis shows task success rate (%). demonstrating that its advantage is not confined to a specific bench- mark. While agents like UFO 2 specialize in narrow domains such as web navigation, IntentCUA generalizes effectively to heteroge- neous desktop environments that include both online and offline contexts, confirming its versatility and domain robustness. As shown in Figure 5, IntentCUA maintains stable performance even as task length grows: 85.9% at 10–15 steps, 72.5% at 15–20, and 65.0% at 20–25, while still retaining 42.9% beyond 30 steps. Both baselines, in contrast, decline sharply after 20 steps, dropping below 20%. This gradual degradation indicates that IntentCUA’s planning remains consistent and resistant to drift even in extended workflows spanning multiple windows and applications. The stability across longer horizons can be attributed to its intent- aware retrieval and plan memory reuse, which enable the planner to recall previously successful subplans aligned with the current intent embedding rather than regenerating them from scratch. To- gether, these results confirm that IntentCUA achieves robust and generalizable long-horizon planning efficiency, effectively preserv- ing goal coherence and minimizing redundant re-planning under complex, real-world desktop environments. Figure 6: Performance by task length (bin size = 5 steps). Left: Step Efficiency Ratio (SER). Right: Average latency per task (minutes) 6.2 Stable & Scalable Planning Efficiency and Latency We examine efficiency using two complementary metrics: the Step Efficiency Ratio (SER; Left) and the average latency per task (Right), as shown in Figure 6. IntentCUA achieves the highest SER of 0.91, exceeding UI-TARS (0.85) and UFO 2 (0.82). While SER in IntentCUA decreases moderately from 0.93 at 10–15 steps to 0.88 at 20–25, it remains consistently above 0.85 even for the longest tasks, indicat- ing that most actions continue to contribute effectively to progress. In contrast, both baselines show sharper declines across similar ranges, suggesting increased redundancy or re-planning. Latency patterns further highlight scalability. IntentCUA’s av- erage execution time is 1.46 minutes, approximately 4.5× lower than the baselines (UFO 2 : 6.63 min, UI-TARS: 9.82 min). Its latency increases smoothly with task length—for instance, from 0.95 min at 10–15 steps to 2.01 min at 20–25—showing near-linear growth. By comparison, UI-TARS exhibits irregular delays that expand sharply with step count, and UFO 2 shows unstable spikes on shorter tasks due to looped retries. These results demonstrate that IntentCUA sustains high planning efficiency and low, predictable latency as task complexity increases. Its memory-guided retrieval and feedback design minimize redun- dant computation, yielding a scalable and robust planning policy suitable for real desktop automation. 7 CONCLUSION We presented IntentCUA, a framework that transforms raw inter- action traces into multi-view intent representations, abstracts them into reusable skills, and integrates these with plan memory to sup- port stable long-horizon desktop automation. The system combines representation learning, hierarchical skill induction, and memory- guided planning to reduce re-planning and improve stability across complex workflows. In experiments, IntentCUA achieved a 74.8% task success rate with a step efficiency ratio of 0.91, outperforming both UI-TARS-1.5 (RL-based) and UFO 2 (trajectory-centric) by 4.5×times reduced latency. It also maintained over 40% success on long-horizon tasks exceeding 30 steps. Ablation studies show that each component con- tributes to robustness and efficiency, with the full design providing the greatest improvements on longer tasks. While IntentCUA shows consistent reasoning and cross-application generalization, several aspects remain open for refinement. Retrieval efficiency may fluc- tuate as the plan memory grows, though this mainly affects latency rather than accuracy. Graph-based retrieval and lightweight vision cues could further enhance robustness, allowing the system to adapt more smoothly to dynamic and visually changing interfaces. ACKNOWLEDGMENTS This work was supported by the National Research Foundation of Korea(NRF) grant (No. RS-2022-NR066631, No. RS-2025-02216282) and Institute of Information & communications Technology Plan- ning & Evaluation (IITP) grant (No.RS-2022-I220025) funded by the Korea government(MSIT) and Ministry of Trade, Industry and Energy of Korea (MOTIE RS 2023 00258591). REFERENCES [1]Anthropic. 2024. Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku. https://w.anthropic.com/news/3-5-models-and-computer-use. Published Oct 22, 2024. [2]Yuzhe Cai, Shaoguang Mao, Wenshan Wu, Zehua Wang, Yaobo Liang, Tao Ge, Chenfei Wu, WangYou WangYou, Ting Song, Yan Xia, Nan Duan, and Furu Wei. 2024. Low-code LLM: Graphical User Interface over Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations). Association for Computational Linguistics, Mexico City, Mexico, 12–25. https://doi.org/10.18653/v1/2024.naacl-demo.2 [3]Ricardo J. G. B. Campello, Davoud Moulavi, Arthur Zimek, and Jörg Sander. 2015. Hierarchical density estimates for data clustering, visualization, and outlier detection. ACM Transactions on Knowledge Discovery from Data (TKDD) 10, 1 (2015), 5. [4]Daehyoun Choi, Hind R’bigui, and Chiwoon Cho. 2022. Enabling the gab between RPA and process mining: User interface interactions recorder. IEEE Access 10 (2022), 39604–39612. [5]Mohammadreza Fani Sani, Michal Sroka, and Andrea Burattin. 2023. LLMs and Process Mining: Challenges in RPA: Task Grouping, Labelling and Connector Recommendation. In International Conference on Process Mining. Springer, 379– 391. [6]Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Donzxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, Hengxu Wang, Luowei Zhou, and Mike Zheng Shou. 2024. AssistGUI: Task-Oriented PC Graphical User Inter- face Automation. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Los Alamitos, CA, USA, 13289–13298. https://doi.org/10.1109/CVPR52733.2024.01262 [7]Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Bangkok, Thailand. https://doi.org/ 10.18653/v1/2024.acl-long.371 [8]Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al.2024. Cogagent: A visual language model for GUI agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 14281–14290. [9] Forrest Huang, Gang Li, Tao Li, and Yang Li. 2024. Automatic macro mining from interaction traces at scale. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–16. [10]Yue Jiang, Eldon Schoop, Amanda Swearngin, and Jeffrey Nichols. 2025. Iluvui: Instruction-tuned language-vision modeling of UIs from machine conversations. In Proceedings of the 30th International Conference on Intelligent User Interfaces. 861–877. [11]Rebecca Krosnick and Steve Oney. 2022. ParamMacros: Creating UI Automation Leveraging End-User Natural Language Parameterization. In 2022 IEEE Sympo- sium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, Liver- pool, United Kingdom, 1–10. https://doi.org/10.1109/VL/HCC53370.2022.9833005 [12]Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad A Myers. 2021. Screen2vec: Semantic embedding of gui screens and gui components. In Pro- ceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15. [13]Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020. Map- ping Natural Language Instructions to Mobile UI Action Sequences. In Pro- ceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics. Association for Computational Linguistics, Online, 8198–8210. https: //doi.org/10.18653/v1/2020.acl-main.729 [14]Yanda Li, Chi Zhang, Wenjia Jiang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. 2025. AppAgent v2: Advanced Agent for Flexible Mobile Interactions. arXiv:2408.11824 [cs.HC] https://arxiv.org/abs/2408.11824 [15]Zhangheng Li, Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li, Mohana Prasad Sathya Moorthy, Jeff Nichols, Yinfei Yang, and Zhe Gan. 2024. Ferret-UI 2: Mastering universal user interface understanding across platforms. arXiv preprint arXiv:2410.18967 (2024). [16]Yijie Lin, Yuanbiao Gou, Zitao Liu, Boyun Li, Jiancheng Lv, and Xi Peng. 2021. COMPLETER: Incomplete Multi-view Clustering via Contrastive Prediction. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 11169–11178. https://doi.org/10.1109/CVPR46437.2021.01102 [17]Anthony Z. Liu, Jongwook Choi, Sungryull Sohn, Yao Fu, Jaekyeom Kim, Dong-ki Kim, Xinhe Wang, Jaewon Yoo, and Honglak Lee. 2024. SkillAct: Using Skill Abstractions Improves LLM Agents. In Proceedings of the 41st International Con- ference on Machine Learning. [18] Yitao Liu, Chenglei Si, Karthik R Narasimhan, and Shunyu Yao. 2025. Contextual Experience Replay for Self-Improvement of Language Agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Moham- mad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 14179–14198. https://doi.org/10.18653/v1/2025.acl-long.694 [19]Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. 2024. Omniparser for pure vision based gui agent. arXiv preprint arXiv:2408.00203 (2024). [20] Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. 2024. Screenagent: A vision language model-driven computer control agent. arXiv preprint arXiv:2402.07945 (2024). [21] OpenAI. 2024. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/. [22]Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al.2025. UI-TARS: Pioneering Automated GUI Interaction with Native Agents. arXiv preprint arXiv:2501.12326 (2025). [23] Adrian Rebmann and Han van der Aa. 2024. Recognizing task-level events from user interaction data. Information Systems 124 (2024), 102404. [24] Andrei Cosmin Redis, Mohammadreza Fani Sani, Bahram Zarrin, and Andrea Burattin. 2024. Skill Learning Using Process Mining for Large Language Model Plan Generation. In International Conference on Process Mining. Springer, 650–662. [25]Pascal J. Sager, Benjamin Meyer, Peng Yan, Rebekka von Wartburg-Kottler, Layan Etaiwi, Aref Enayati, Gabriel Nobel, Ahmed Abdulkadir, Benjamin F. Grewe, and Thilo Stadelmann. 2025. AI Agents for Computer Use: A Review of Instruction-based Computer Control, GUI Automation, and Operator Assistants. arXiv:2501.16150 [cs.AI] https://arxiv.org/abs/2501.16150 [26]Pascal J. Sager, Benjamin Meyer, Peng Yan, Rebekka von Wartburg-Kottler, Layan Etaiwi, Aref Enayati, Gabriel Nobel, Ahmed Abdulkadir, Benjamin F. Grewe, and Thilo Stadelmann. 2025. A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directions. arXiv:2501.16150 [cs.AI] https://arxiv.org/abs/2501.16150 [27] ByteDance Seed. 2025. UI-TARS-1.5. https://seed-tars.com/1.5. [28] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems. Curran Associates Inc., New Orleans, LA, USA, Article 377. [29]Yunpeng Song, Yiheng Bian, Yongtao Tang, and Zhongmin Cai. 2023. Navigating interfaces with AI for enhanced user interaction. arXiv preprint arXiv:2312.11190 (2023). [30] Yunpeng Song, Yiheng Bian, Yongtao Tang, Guiyu Ma, and Zhongmin Cai. 2024. Visiontasker: Mobile task automation using vision based ui understanding and llm task planning. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. 1–17. [31]Yifan Song, Weimin Xiong, Xiutian Zhao, Dawei Zhu, Wenhao Wu, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. 2024. AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories. arXiv preprint arXiv:2410.07706 (2024). [32] Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, Kaitao Song, Jian Shao, Weiming Lu, Jun Xiao, and Yueting Zhuang. 2025. A Survey on (M)LLM-Based GUI Agents. arXiv:2504.13865 [cs.HC] https://arxiv.org/abs/2504.13865 [33]Alok Mani Tripathi. 2018. Learning Robotic Process Automation: Create Software robots and automate business processes with the leading RPA tool–UiPath. Packt Publishing Ltd. [34]Bryan Wang and Yang Li. 2023. Enabling Conversational Interaction on Mobile with LLMs. https://research.google/blog/enabling-conversational-interaction- on-mobile-with-llms/. Google Research Blog. [35]Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. 2025. OS- ATLAS: Foundation Action Model for Generalist GUI Agents. In The Thirteenth International Conference on Learning Representations. https://openreview.net/ forum?id=n9PDaFNi8t [36]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. arXiv:2404.07972 [cs.AI] https://arxiv.org/abs/ 2404.07972 [37]Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. 2024. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.16256 (2024). [38]Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023. M-REACT: Prompting ChatGPT for Multimodal Reasoning and Action. arXiv:2303.11381 [cs.CV] https://arxiv.org/abs/2303.11381 [39]Yiwen Yin et al.2025. Automatic Modeling Cognitive Dependencies from User Demonstrations for GUI Task Automation. In Proceedings of ACM/SIGCHI (or appropriate venue). [40] Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. 2025. Large Language Model-Brained GUI Agents: A Survey. arXiv:2411.18279 [cs.AI] https://arxiv.org/abs/2411.18279 [41] Chaoyun Zhang, He Huang, Chiming Ni, Jian Mu, Si Qin, Shilin He, Lu Wang, Fangkai Yang, Pu Zhao, Chao Du, et al. 2025. Ufo2: The desktop agentos. arXiv preprint arXiv:2504.14603 (2025). [42]Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. 2024. Table 3: Representation loss ablation on intent embedding quality. We report size-weighted density separation (in- ter/intra) over all HDBSCAN subgroups. Loss VariantSeparation↑ Baseline Embedding5.60 InfoNCE only5.64 InfoNCE + Prediction6.92 InfoNCE + Reconstruction23.17 Full (Con + Pred + Rec)7.74 UFO: A UI-Focused Agent for Windows OS Interaction. arXiv:2402.07939 [cs.HC] https://arxiv.org/abs/2402.07939 [43]S. Zhang et al.2024. Dynamic Planning for LLM-based Graphical User Interface Automation. In Findings of EMNLP (2024). [44]Shaoqing Zhang, Zhuosheng Zhang, Kehai Chen, Xinbei Ma, Muyun Yang, Tiejun Zhao, and Min Zhang. 2024. Dynamic planning for llm-based graphical user interface automation. arXiv preprint arXiv:2410.00467 (2024). [45] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al.2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854 (2023). A ENCODER DETAILS This appendix summarizes the implementation details of the multi- view encoder described in Section 3.1. Encoder Architecture. Each view (ENV, ACT/KEY, DES) is embed- ded using OpenAItext-embedding-3-large, producing a 3072- dimensional vector. Each embedding is mapped to a shared latent space via a view-specific 2-layer MLP projection head: 3072→ 256→ 256, with GeLU activation, dropout (푝=0.05), and LayerNorm, yielding 푧 (푣) 푖 ∈R 256 . Cross-view consistency is enforced using six symmetric dual predictors, one for each ordered view pair. Each predictor is a lightweight MLP (256→128→256) used only during training. Additionally, a linear decoder (256→3072) is applied per view to reconstruct the original embedding for reconstruction regulariza- tion. Shared Representation and Fusion Weights. The final shared intent representation is computed as a weighted fusion: 푧 푖 = 0.4푧 (퐸) 푖 + 0.3푧 (퐴) 푖 + 0.3푧 (퐷) 푖 . The environment view is assigned the largest weight because exe- cution environment provides the most stable contextual signal in desktop automation and serves as the primary driver for upper- level intent group (IG) formation. This environment-centric fusion improves the stability of hierarchical clustering while still preserv- ing action- and description-level variability for finer subgroups (SG). For model training, we used a learning rate of 1×10 −3 , with 휆 pred = 0.1, 휆 rec = 0.05, and a contrastive temperature 휏= 0.1. Tensor Shapes. For a minibatch of size푁, the encoder operates on: 푥 (푣) ∈R 푁×3072 , 푧 (푣) ∈R 푁×256 , 푧 ∈R 푁×256 . B FRAMEWORK DETAILS (PLANNER-PLAN-OPTIMIZER-CRITIC) Planner. At inference time, the Planner maps a natural-language command푐into the same structured format used for log label- ing by prompting the LLM to produce task units of the form Task Unit: ENV[...] ACT[...], Task k: ENV[...] ACT[...] with short de- scriptions (e.g., “search dog at a browser”→ENV[local/Windows, web/searching browser], ACT[open browser, search]). From these views, it builds an intent prototype and retrieves candidate plans from memory. A cached plan is reused only if its action coverage with respect to the current breakdown is high: in practice, we require that the plan already contains most of the required ACTs (allowing at most 2 missing ACTs per command). For each such missing ACT, we retrieve the corresponding intent subgroup, select the skill template most frequently observed in the logs, instantiate its placeholders from푐(e.g., query = "dog"), and splice the resulting steps into the cached plan. If no cached plan satisfies this condition, the Planner falls back to RAG using representative logs as examples. The final output is a global plan G =푔 푖 , where each step푔 푖 = (action, object) uses one of a fixed set of 17 low-level GUI actions (e.g., text input, click, doubleclick, press, switch focus, save, copy. . . ). Execution(Plan-Optimizer→Critic interaction). Execution con- sists of a Plan-Optimizer that grounds each푔into concrete GUI actions, and a Critic that validates the post-state and triggers lo- cal recovery. Each step푔from the global plan is expanded into an actionable sequence푔 ′ using a fixed library of default action templates (e.g., open : doubleclick icon or click taskbar→type target→press enter), ensuring grounding into atomic GUI actions. The Plan-Optimizer conditions on (1) task-unit context, (2) the parsed screen state푠, and (3) a retrieved plan hint by matching 푧 (푝푢) to the nearest subgroup centroid. Hints contain historical ENV/ACT/DES tuples and action-object traces, biasing execution toward stable patterns rather than free-form generation. After each plan unit, the Critic inspects the post-state푠 after via a structured prompt that checks window focus, component availability, and compatibility with the next expected step and returns retryable=success, retryable, blocked which triggers a lo- calized re-optimization from푠 after , avoiding global re-planning, while ‘blocked’ indicates that neither template-based execution nor exemplar-guided adjustment provides a safe continuation. A step-by-step example of this interaction is illustrated in Figure 7. C ABLATION ON THE REPRESENTATION LOSS We analyze density separation and cluster purity with respect to ENV and ACT tags for further ablation. To address concerns about the representation objective, we com- pare four variants: InfoNCE-only, InfoNCE + cross-view prediction, InfoNCE + reconstruction, and the full loss. Sep w = Í 퐼퐺 Í 푆퐺∈퐼퐺 |푆퐺| Inter(푆퐺) Intra(푆퐺) Í 퐼퐺 Í 푆퐺∈퐼퐺 |푆퐺| (7) Figure 7: Planner–Plan-Optimizer–Critic interaction. The Planner decomposes a command into structured Plan Units and retrieves or synthesizes a global plan. The Plan-Optimizer grounds each unit into executable GUI actions conditioned on the current state, while the Critic validates the post-state 푆 after and triggers local re-optimization if needed. Table 4: ENV/ACT purity under different representation loss variants. We report mean purity with standard deviation in parentheses. Loss VariantENV purityACT purity Baseline Embedding0.83(0.19)0.48(0.25) InfoNCE only0.82 (0.22)0.37 (0.22) InfoNCE + Prediction0.83 (0.21)0.42 (0.26) InfoNCE + Reconstruction0.86 (0.19)0.42 (0.26) Full (Con + Pred + Rec) 0.84 (0.20) 0.42 (0.23) Intra(푆퐺)= 1 |푆퐺| ∑︁ 푥∈푆퐺 푑(푥,c 푆퐺 ) Inter(푆퐺)= min 푆퐺 ′ ≠푆퐺 푑(c 푆퐺 ,c 푆퐺 ′ ) (8) 휇 푋 = 1 |SG| ∑︁ 퐼퐺 ∑︁ 푆퐺∈퐼퐺 Purity 푋 (푆퐺) 휎 푋 = √︄ 1 |SG| ∑︁ 퐼퐺 ∑︁ 푆퐺∈퐼퐺 Purity 푋 (푆퐺)− 휇 푋 2 (9) Purity 푋 (푆퐺)= max 푐 푡 ∈ 푆퐺 : tag 푋 (푡)=푐 |푆퐺| (10) We report (1) size-weighted density separation (inter/intra: Eq 8) in Eq 7 over all HDBSCAN subgroups in Table 3 and (2) semantic purity(Eq 9) measured as majority ENV/ACT ratios(Eq 10) within each subgroup in Table 4. The full loss improves the separation ratio from 5.64 (InfoNCE- only) to 7.74, while InfoNCE + reconstruction produces an inflated separation score of 23.17 due to extreme micro-clusters (size=2), indicating over-fragmentation rather than robust intent abstraction. In terms of semantic consistency, ENV purity increases from 0.82 (InfoNCE-only) to 0.84 (Full), and ACT purity improves from 0.37 to 0.42. Reconstruction yields the highest ENV purity (0.86) but with higher fragmentation, while Prediction consistently improves ACT purity (0.42 vs. 0.37 in InfoNCE-only). The full objective maintains balanced ENV/ACT purity (0.84 / 0.42) with reduced variance (ENV std 0.20, ACT std 0.23), suggesting more stable and semantically coherent intent embeddings. D DOMAIN LEVEL DISTRIBUTIONS OF DATASET/TESTCASES Figure 8 shows the domain distribution of the collected trace cor- pus. The trace data are skewed toward Local/App environments, with several long-tail domains having only a few interaction ses- sions. In contrast, the evaluation suite (Figure 9) contains a broader Web/Crossover share and substantially more domains overall. This asymmetry reflects the intentional distribution shift described in the main text. Importantly, domain-level success rates in Figure 9 indicate that performance does not strictly correlate with trace frequency. Sev- eral domains with very limited or no traces still achieve non-trivial Table 5: Step consistency (%) across five repeated planning runs per testcase (cosine threshold = 0.93). WebVoyagerScreenAgentOursTotal Mean (%)70.381.186.278.5 STD (%)8.25.37.97.8 success rates, suggesting that the planner generalizes beyond mem- orized trajectories. While trace sparsity and bias remain limitations, these statistics provide additional transparency regarding domain coverage and generalization behavior. E PLANNING CONSISTENCY To further assess stability under system complexity, we introduce step consistency as a quantitative measure of plan repeatability. For each of the 286 testcases, we execute planning five times and com- pare the resulting plans. For each plan unit푝푢, we examine whether the generated step sequence is consistently reproduced across all Figure 8: Domain distribution of collected trace data. Each slice indicates a domain category and its proportion within the trace corpus (%). Figure 9: Domain distribution of the 286 evaluation testcases. Each slice shows a domain category and the success rate achieved within that domain (%). five runs. A푝푢is counted as consistent if the pairwise cosine simi- larity between corresponding step embeddings exceeds a threshold of 0.93 in all comparisons. Step consistency is defined as the pro- portion of such consistent푝푢instances within each benchmark split. Table 5 reports the mean and standard deviation across domains. The results indicate stable planning behavior across datasets, includ- ing unseen domains. Despite distribution shift and sparse traces, the Planner–Plan-Optimizer loop maintains high repeatability, sug- gesting that structural constraints (e.g., ENV derived from window structure and ACT from UI semantics) effectively reduce LLM drift and labeling variance.