Paper deep dive
Beyond Success and Failure: Length-Aware Contrastive Learning for GUI Agents
Chengyang Gu, Le Zhang, Jingbo Zhou, Yize Chen, Yu Shi, Siqi Bao, Zheng-Fan Wu, Hua Wu, Hui Xiong
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/25/2026, 7:26:24 AM
Summary
The paper introduces LACL-GUI, a Length-Aware Contrastive Learning framework for GUI agents powered by Multimodal Large Language Models (MLLMs). It addresses the reward-gradient misalignment in Group Relative Policy Optimization (GRPO) by reformulating Reinforcement Learning with Verifiable Rewards (RLVR) as a contrastive objective. Unlike existing methods that rely on binary outcome supervision, LACL-GUI incorporates trajectory-level quality signals: it encourages concise successful executions via length-aware positive preference and differentiates failure quality based on divergence from successful trajectories. Experiments on the OSWorld benchmark demonstrate that LACL-GUI consistently improves agent performance over baselines like GRPO and REAL.
Entities (9)
Relation Signals (8)
Chengyang Gu → authored → LACL-GUI
confidence 95% · Beyond Success and Failure: Length-Aware Contrastive Learning for GUI Agents Chengyang Gu
Hui Xiong → authored → LACL-GUI
confidence 95% · Beyond Success and Failure: Length-Aware Contrastive Learning for GUI Agents Chengyang Gu†& Hui Xiong
LACL-GUI → improves → GUI Agents
confidence 95% · LACL-GUI provides more effective learning signals and consistently improves agent performance over prior methods
GRPO → suffersfrom → reward-gradient misalignment
confidence 95% · widely used methods such as Group Relative Policy Optimization (GRPO) suffer from reward-gradient misalignment
LACL-GUI → addresses → reward-gradient misalignment
confidence 92% · LACL-GUI introduces structured preferences... while preserving optimization stability
LACL-GUI → buildsupon → REAL
confidence 90% · LACL-GUI work on RLVR-based objectives... on top of REAL contrastive objective
LACL-GUI → evaluatedon → OSWorld
confidence 90% · We evaluate LACL-GUI on OSWorld benchmark
LACL-GUI → usesbackbone → Qwen3-VL
confidence 90% · using Qwen3-VL-Thinking (4B and 8B) as backbone models
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) have shown strong potential for automating tasks across diverse digital environments, where reinforcement learning (RL) has become a dominant training paradigm. However, widely used methods such as Group Relative Policy Optimization (GRPO) suffer from reward-gradient misalignment, leading to inefficient and unstable optimization. Recent work addresses this issue by reformulating RL with verifiable rewards (RLVR) as contrastive or classification-based objectives, which improve stability by eliminating problematic gradient behaviors. Despite this progress, existing contrastive RLVR methods rely primarily on outcome-level supervision and fail to capture fine-grained differences in trajectory quality within the same outcome category. In this paper, we propose Length-Aware Contrastive Learning for GUI Agents (LACL-GUI), a contrastive RLVR framework that incorporates trajectory-level quality signals into policy optimization. LACL-GUI introduces structured preferences within both successful and failed trajectories, encouraging concise successful executions and differentiating failure quality based on divergence from successful trajectories, while preserving optimization stability. Experiments on GUI agent benchmarks show that LACL-GUI provides more effective learning signals and consistently improves agent performance over prior methods, highlighting the value of trajectory-level supervision in contrastive RLVR.
Tags
Links
- Source: https://arxiv.org/abs/2608.21830v1
- Canonical: https://arxiv.org/abs/2608.21830v1
Trouble viewing inline? Open PDF directly →
Full Text
68,170 characters extracted from source content.
Expand or collapse full text
Beyond Success and Failure: Length-Aware Contrastive Learning for GUI Agents Chengyang Gu†& Hui Xiong Thanks: Correspondence to: xionghui@hkust-gz.edu.cn Affiliation: Artificial Intelligence Thrust, Information Hub Affiliation: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China Email: †cgu893@connect.hkust-gz.edu.cn Le Zhang Jingbo Zhou Yu Shi Siqi Bao Zheng-Fan Wu & Hua Wu Affiliation: Baidu Inc. Beijing, China Yize Chen Affiliation: Department of Electrical and Computer Engineering Affiliation: University of Alberta, Edmonton, Alberta, Canada Abstract Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) have shown strong potential for automating tasks across diverse digital environments, where reinforcement learning (RL) has become a dominant training paradigm. However, widely used methods such as Group Relative Policy Optimization (GRPO) suffer from reward–gradient misalignment, leading to inefficient and unstable optimization. Recent work addresses this issue by reformulating RL with verifiable rewards (RLVR) as contrastive or classification-based objectives, which improve stability by eliminating problematic gradient behaviors. Despite this progress, existing contrastive RLVR methods rely primarily on outcome-level supervision and fail to capture fine-grained differences in trajectory quality within the same outcome category. In this paper, we propose Length-Aware Contrastive Learning for GUI Agents (LACL-GUI), a contrastive RLVR framework that incorporates trajectory-level quality signals into policy optimization. LACL-GUI introduces structured preferences within both successful and failed trajectories, encouraging concise successful executions and differentiating failure quality based on divergence from successful trajectories, while preserving optimization stability. Experiments on GUI agent benchmarks show that LACL-GUI provides more effective learning signals and consistently improves agent performance over prior methods, highlighting the value of trajectory-level supervision in contrastive RLVR. 1 Introduction Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) (3; 33; 48) have demonstrated remarkable potential in automating tasks across web, mobile, and desktop platforms. Prior work typically relies on supervised fine-tuning (SFT) (40; 26), which demands substantial high-quality synthetic data and often struggles to generalize to unseen interfaces (21; 19). Unlike SFT, reinforcement learning (RL) enables agents to learn from trial-and-error interaction with environmental feedback, continuously refining policies without exhaustive labeled demonstrations (51; 13; 44). As a result, RL has emerged as a dominant paradigm for training advanced GUI agents (20; 13). Among RL-based approaches, Group Relative Policy Optimization (GRPO) is one of the dominant methods due to its strong empirical performance and computational efficiency (29; 11). By leveraging group-wise reward normalization, GRPO eliminates the need for a separate critic network while maintaining stable policy optimization. However, recent studies reveal a misalignment between reward signals and actual gradient allocation, where optimization pressure is disproportionately influenced by policy likelihood rather than trajectory quality (6; 47). (45) theoretically identify two such gradient pathologies in GRPO: Gradient Misassignment in Positives, where high-probability tokens receive disproportionately large updates while low-probability tokens receive weak gradients despite belonging to successful trajectories; and Gradient Domination in Negatives, where gradient magnitudes concentrate excessively on a small number of high-probability negative tokens, allowing them to dominate optimization. Consequently, verifiable rewards are not faithfully reflected in policy updates, limiting both optimization efficiency and training stability. To address reward-gradient mismatch in GRPO-style optimization, recent work has increasingly reformulated RLVR as contrastive or classification-based objectives rather than scalar reward-weighted optimization (23). Representative examples include Rewards as Labels (REAL) (45), CLIPO (5), and Harmony (43), which improve optimization stability, robustness, and learning effectiveness through comparison-based objectives, with REAL in particular showing that classification-based optimization induces monotonic and bounded gradient weighting that mitigates the pathologies of GRPO. However, this line of work still reduces every trajectory to a single binary label, leaving the resulting classification-based objectives blind to any structure within positive or negative groups themselves. Despite these optimization advantages, existing contrastive RLVR methods still derive supervision primarily from outcome-level labels. Consequently, trajectories sharing the same outcome are often treated equivalently, despite potentially large differences in execution quality, efficiency, and proximity to success (41; 46; 30; 50). This coarse-grained supervision limits RLVR methods’ capabilities in capturing fine-grained trajectory preferences (47). This limitation is particularly pronounced for GUI agents, where successful trajectories may vary significantly in efficiency, while failed trajectories can differ substantially in how close they come to completing the task. The central challenge is therefore how to incorporate trajectory-level quality indicators into contrastive RLVR, while preserving reliable policy optimization. To address this challenge, we propose Length-Aware Contrastive Learning for GUI Agents (LACL-GUI), a length-aware contrastive RLVR framework for GUI agent training. LACL-GUI work on RLVR-based objectives and introduces structured preferences within both successful and failed trajectory groups: for successful trajectories, we encourage shorter solutions by assigning stronger optimization pressure to concise executions while down-weighting redundant ones; for failed trajectories, we introduce divergence length, which measures how long a failed trajectory remains aligned with a reference successful execution before deviating, treating later-diverging failures as more informative negative samples and applying weaker suppression. These preferences are incorporated through outcome-preserving, zero-mean logit modulations, so that optimization is refined within each outcome category without altering the underlying success–failure decision boundary. As a result, LACL-GUI successfully provides fine-grained efficiency and failure-quality guidance on top of classification-based RLVR, while preserving its optimization stability and gradient safety. We evaluate LACL-GUI on OSWorld benchmark (39) using Qwen3-VL-Thinking (4B and 8B) as backbone models. Experiments show LACL-GUI consistently outperforms baselines at both scales, improving task success rate by 2.8% over GRPO and 2.7% over REAL on 8B backbone within 50 training iterations. 2 Related Works 2.1 GUI Agents GUI Agents aim to execute tasks across web, mobile, and desktop environments by mapping high-level instructions to grounded interface actions. Early approaches rely on structured representations, such as HTML (7) and DOM trees (36). However, these methods depend heavily on the coverage and quality of such representations, which are often brittle and inconsistent across platforms (37; 31). Recent approaches leverage Multimodal Large Language Models (MLLMs) to enable more general and visually grounded interactions. Given natural language instructions and interface observations, MLLM-based GUI agents such as UI-TARS (26), OS-Atlas (38), and UGround (10) parse model responses into executable actions through predefined prompting and action schemas. In this work, we follow the MLLM-based paradigm and adopt the Qwen3-VL series model (3) with its action parser as the backbone for policy optimization. Figure 1: Overview of LACL-GUI framework. A decoupled architecture collects informative trajectory groups and maintains a Success Buffer for reference trajectories. The proposed objective introduces length-aware positive preference and divergence-aware negative refinement on top of REAL contrastive objective. 2.2 Reinforcement Learning Reinforcement learning (RL) has become a promising paradigm for enhancing the reasoning and decision-making capabilities of LLM and MLLM-based agents (11; 14; 32), particularly in multi-turn interactive environments. Early approaches largely rely on reinforcement learning from human feedback (RLHF), including methods such as Direct Preference Optimization (DPO) (27), which derive reward signals from offline human preference data. However, RLHF suffers from high annotation costs and noisy reward estimation (15). Recent work therefore shifts toward Reinforcement Learning with Verifiable Rewards (RLVR), where rewards are directly obtained from task outcomes. Group Relative Policy Optimization (GRPO) (29) is a representative RLVR approach that removes the need for a critic model through group-wise advantage estimation. Subsequent variants, including Dr.GRPO (18), DAPO (42), and GSPO (49), further improve training stability and efficiency. Nevertheless, recent studies such as REAL (45) reveal that GRPO-style objectives suffer from gradient misalignment, where policy updates can be dominated by token likelihood rather than trajectory quality. These findings motivate exploration of alternative RLVR objectives beyond conventional reward-weighted operations. Recent works also explore finer-grained credit assignment for GUI agents beyond sparse outcome rewards. GiGPO (8) introduces milestone-based rewards through manually designed environment-specific rules, while other approaches estimate intermediate progress using pretrained representation models (50; 35). However, these approaches either require task-specific supervision or additional models, limiting their scalability across diverse GUI environments. In contrast, our approach derives trajectory-level quality signals directly from successful and failed rollouts within a unified contrastive optimization framework. 2.3 Contrastive Learning for RLVR Contrastive learning is a widely used representation learning paradigm that encourages positive samples to be closer while separating negative samples (28). Since RLVR methods typically rely on binary outcome rewards (25), recent studies show that objectives such as GRPO can be interpreted from a classification perspective (22). This perspective has inspired methods that replace reward-weighted optimization with discriminative or contrastive objectives. REAL (45) formulates RLVR as a binary classification problem using BCE loss, providing a stable optimization alternative by contrasting successful and unsuccessful trajectories. Subsequent works, including CLIPO (5) and ConSPO (47), further introduce explicit contrastive objectives based on InfoNCE (23). However, existing approaches mainly operate at the trajectory outcome level, where trajectories within the same outcome category receive identical supervision. Our LACL-GUI addresses this limitation by introducing trajectory-level quality preferences into contrastive RLVR. 3 Methodology 3.1 Problem Formulation We formulate GUI task completion as a sequential decision-making problem under a Partially Observable Markov Decision Process (POMDP). Given a natural language instruction q, an agent interacts with the environment over multiple steps to complete the task. At each timestep t, the agent receives a visual observation ot∈o_t , represented as a screenshot of the interface, instead of accessing the full underlying state. To mitigate partial observability, the agent conditions on a finite history window ht=(ot−k,zt−k,at−k),…,(ot−1,zt−1,at−1)h_t=\(o_t-k,z_t-k,a_t-k),…,(o_t-1,z_t-1,a_t-1)\ of the most recent k interactions, where ztz_t denotes an intermediate reasoning trace and at∈a_t denotes a structured executable action (e.g., parameterized tool calls such as clicking at specific coordinates). A policy πθ _θ, parameterized by a vision-language model, generates a reasoning-action pair: (zt,at)∼πθ(⋅∣ot,ht,q).(z_t,a_t) _θ(· o_t,h_t,q). In practice, ztz_t serves as auxiliary reasoning tokens, while ata_t is the executable action applied to the environment. Executing action ata_t leads to the next observation ot+1o_t+1 according to the environment dynamics. An episode produces a trajectory τ=(o0,z0,a0),(o1,z1,a1),…,(oT,zT,aT),τ=\(o_0,z_0,a_0),(o_1,z_1,a_1),…,(o_T,z_T,a_T)\, which terminates either when the task is completed or when a maximum horizon is reached. We consider a sparse outcome-based reward setting, where supervision is provided only at the trajectory level:R(τ)∈0,1,R(τ)∈\0,1\, indicating task success or failure. All steps within a trajectory share the same outcome-level supervision, without any intermediate rewards for individual actions. The objective is to learn a policy πθ _θ that maximizes the expected success rate θ∗=argmaxθτ∼πθ[R(τ)].θ^*= _θE_τ _θ[R(τ)]. 3.2 Architecture Following DART (16), we adopt an asynchronous training architecture for proposed LACL-GUI framework. The system consists of four modules: Rollout Service, Data Manager, Success Buffer, and Trainer. The Rollout Service collects trajectories from multiple GUI environments in parallel, which are filtered and organized by the Data Manager. Successful trajectories are stored in the Success Buffer for reuse and divergence estimation, while filtered trajectory groups are sent to the Trainer for policy optimization with the LACL-GUI objective. Updated parameters are periodically synchronized back to the rollout workers, forming an asynchronous training loop. Rollout Service Interacting with real-world environments to solve GUI tasks using vision-language models is time-consuming. To address this, we employ asynchronous rollout-wise sampling following DART (16), where each environment independently starts a new trajectory after completion without waiting for other workers. This design improves environment utilization and enables efficient trajectory collection. For each task instruction, we collect N trajectories under the same initial condition, forming trajectory groups for contrastive optimization. Data Manager The Data Manager filters sampled trajectories to construct informative training groups. Since LACL-GUI relies on contrastive optimization between successful and failed trajectories, groups where all N trajectories share the same outcome (i.e., all success or all failure) provide no discriminative supervision and are discarded. For each training iteration, the Data Manager collects a batch of M tasks whose trajectory groups contain both successful and failed samples, and then forwards them to the Trainer. Following DART, policy updates are performed after all trajectories in the batch are collected, enabling meaningful within-task comparisons among sampled trajectories. Success Buffer Sparse rewards in GUI tasks often lead to insufficient successful trajectories during training. To improve sample efficiency, we maintain a Success Buffer that stores successful trajectories for each task. The buffer serves two purposes: first, it provides additional positive samples when current rollouts contain limited successes; second, it provides reference successful trajectories for divergence-aware failure analysis. Specifically, for each failed trajectory (R(τ)=0R(τ)=0), we estimate its quality by comparing it with a successful reference trajectory from the same task. The reference is selected as the shortest successful trajectory from the union of the Success Buffer and current-batch successful trajectories, while only current-batch successful trajectories are considered when the buffer is unavailable. Trainer The Trainer is decoupled from rollout workers and asynchronously optimizes the policy using filtered trajectory groups to eliminate blocking between data collection and policy optimization. After each update, the latest parameters are synchronized back to the rollout service, allowing trajectory collection and policy optimization to proceed concurrently. Overall, this architecture provides an efficient infrastructure for collecting structured trajectory groups and maintaining high-quality successful references, which are essential for the trajectory-level contrastive optimization in LACL-GUI. We next introduce our proposed objective in detail. 3.3 LACL-GUI Objective 3.3.1 Revisiting REAL Objective We first revisit the REAL (45) objective, which formulates RLVR from a classification perspective. Given an instruction q, a set of N trajectories τ1,…,τN\τ^1,…,τ^N\ are sampled from the same policy. Each trajectory τjτ^j is assigned a binary reward rj∈0,1r^j∈\0,1\ by a verifier indicating task success. Based on these labels, trajectories are partitioned into two disjoint sets: +=τj|rj=1,−=τj|rj=0.T^+=\τ^j|r^j=1\, ^-=\τ^j|r^j=0\. For each trajectory τjτ^j in both +T^+ and −T^-, REAL defines a length-normalized relative log probability score over its actions as logits: s¯j=1|τj|∑t=1|τj|(logπθ(at|ot,ht,q)πold(at|ot,ht,q)). s^j= 1|τ^j| _t=1^|τ^j|( _θ(a_t|o_t,h_t,q) _old(a_t|o_t,h_t,q)). where |τj||τ^j| denotes trajectory length, and ata_t, oto_t, and hth_t represent the action, observation, and history at step t, respectively. The score measures the relative change of trajectory probability under the updated policy compared to the old policy. Specifically, s¯j>0 s^j>0 indicates that the trajectory is promoted, whereas s¯j<0 s^j<0 indicates suppression. The binary reward determines the optimization direction by separating desirable and undesirable trajectories, while the logits reflect the magnitude of optimization updates. Based on logits s¯j s^j, REAL optimizes the following contrastive-style objective: ℒREAL=log(1+∑+e−s¯j/λ)+log(1+∑−es¯j/λ).L_REAL= (1+ _T^+e^- s^j/λ)+ (1+ _T^-e s^j/λ).\ (1) where λ is a temperature parameter. This objective encourages positive trajectories and suppresses negative ones, with adaptive gradient allocation determined by s¯j s^j. Specifically, under-optimized positive trajectories with smaller logits receive stronger updates, while over-confident positives are down-weighted; similarly, negative trajectories with larger logits receive stronger penalties. Such adaptive and bounded gradient allocation alleviates the gradient misalignment issues observed in prior RLVR methods (45). However, Eq. equation 1 only distinguishes trajectories at the task-outcome level and does not capture fine-grained differences within +T^+ or −T^-. Trajectories with identical outcomes can still exhibit substantially different qualities. For example, two successful GUI trajectories may complete the same task, while one requires only necessary interactions and the other contains redundant exploratory actions. Similarly, among failed trajectories, one may follow the correct procedure for most steps before making a final mistake, whereas another may diverge immediately from the desired behavior. Although these trajectories receive identical binary rewards, they provide different optimization signals. Since REAL does not explicitly model such intra-group preferences, it cannot encourage efficient successful executions or distinguish informative failures from fundamentally incorrect attempts. This limitation motivates our proposed LACL-GUI objective. 3.3.2 Positive Group Logits Perturbation For fine-grained comparison within successful trajectories, we observe that GUI task execution naturally exhibits an efficiency preference. Among trajectories starting from the same initial state and achieving the same goal, shorter executions are generally preferred, as longer successful trajectories often contain redundant interactions caused by unnecessary exploration, incorrect actions followed by recovery, or repeated operations. For example, in GUI environments, an agent may perform unnecessary exploratory movements, repeatedly click irrelevant regions, or revisit previously completed steps before eventually reaching the correct state. Such inefficiency is particularly undesirable in GUI environments, where each interaction introduces additional inference and execution cost while increasing the possibility of future errors. Based on this observation, we introduce a length-aware perturbation within the positive group in the logit space. Rather than modifying binary success supervision, the proposed perturbation preserves REAL’s success preference while introducing additional efficiency-aware guidance among successful trajectories. For the positive set +T^+, we define the softmax weighting based on logits s¯θj s_θ^j under current policy πθ _θ: p+(τj)=e−s¯θ(τj)/λ∑τj∈+e−s¯θ(τj)/λ.p^+(τ^j)= e^- s_θ(τ^j)/λ _τ^j ^+e^- s_θ(τ^j)/λ. (2) We then compute the weighted mean trajectory length: μw=∑τj∈+p+(τj)|τj|. _w= _τ^j ^+p^+(τ^j)\,|τ^j|. (3) Based on this efficiency preference, we perturb the positive logits as: s~θ(τj)=s¯θ(τj)+ϵ(|τj|−μw),τj∈+. s_θ(τ^j)= s_θ(τ^j)+ε (|τ^j|- _w ), τ^j ^+. (4) where ϵ>0ε>0 is a small perturbation coefficient controlling the magnitude of the logit adjustment. This perturbation introduces a relative efficiency preference within the positive group. Under the exponential weighting p(τj)∝exp(−s~θ(τj)/λ)p(τ^j) (- s_θ(τ^j)/λ), shorter successful trajectories receive relatively larger optimization weights, whereas longer trajectories are down-weighted. Therefore, LACL-GUI encourages efficient successful executions without changing the original success preference defined by REAL. 3.3.3 Negative Group Logits Perturbation While REAL suppresses all failed trajectories uniformly, failures may contain different levels of learning value. In GUI tasks, a trajectory that follows successful behaviors for most steps before deviating is generally closer to task completion than one that fails at an early stage. We therefore introduce a divergence-aware perturbation to distinguish failed trajectories according to their proximity to successful executions. For each task, given a negative trajectory τj∈−τ^j ^-, we compare it with a reference successful trajectory τref+ _ref^+: τj τ^j =(o1j,a1j),…,(o|τj|j,a|τj|j); =\(o_1^j,a_1^j),…,(o_|τ^j|^j,a_|τ^j|^j)\; (5) τref+ τ^+_ref =(o1ref,a1ref),…,(o|τref+|ref,a|τref+|ref). =\(o_1^ref,a_1^ref),…,(o_| _ref^+|^ref,a_|τ^+_ref|^ref)\. The reference trajectory τref+ _ref^+ is selected as the shortest successful trajectory among the success buffer and the current rollout group. Since trajectories of the same task share an identical initial state, their observations can be compared step by step to identify the first divergence point. We compute the observation similarity SIM(otref,otj)SIM(o_t^ref,o_t^j) at each step using Structural Similarity Index (SSIM) (4), following (17). Given a threshold Θ , the divergence length is defined as: dj=mint|SIM(otref,otj)<Θ−1.d^j= \t|SIM(o_t^ref,o_t^j)< \-1. (6) If no divergence is detected, we set dj=|τj|d^j=|τ^j|. Based on the divergence length, we define the number of erroneous steps Ej=|τj|−dj+1.E^j=|τ^j|-d^j+1. (7) which measures the remaining execution length after deviating from the successful trajectory. Analogous to the positive group, we compute the weighted average erroneous steps: p−(τj) p^-(τ^j) =es¯θ(τj)/λ∑τj∈−es¯θ(τj)/λ; = e s_θ(τ^j)/λ _τ^j ^-e s_θ(τ^j)/λ; (8) μe _e =∑τj∈−p−(τj)|Ej|. = _τ^j ^-p^-(τ^j)\,|E^j|. The perturbed logits are then given by s^θ(τj)=s¯θ(τj)+ϵ(Ej−μe),τj∈−. s_θ(τ^j)= s_θ(τ^j)+ε (E^j- _e ), τ^j ^-. (9) This perturbation introduces a relative error-severity preference within the negative group. Failures with larger erroneous-step counts, corresponding to earlier divergence from successful executions, receive larger logits and therefore stronger suppression under the negative loss. In contrast, near-success trajectories with fewer post-divergence steps are penalized more gently. Compared with the uniform negative suppression in REAL, LACL-GUI preserves informative near-success failures while still reducing the influence of fundamentally incorrect execution patterns. Model Max Steps gimp calc impress writer os thunderbird vlc vs_code Overall Closed-source Models OpenAI CUA o3 (24) 100 38.5 10.6 10.6 30.4 62.5 26.7 39.2 39.1 27.8 OpenAI CUA (24) 50 34.6 14.9 29.7 26.1 70.8 66.7 11.6 69.6 36.5 Claude 3.7 Sonnet (1) 50 38.5 31.9 36.1 43.5 50.0 53.3 23.5 56.5 40.1 Seed1.5-VL-250717 (12) 100 50.0 34.8 48.9 56.5 39.1 73.3 35.3 56.5 45.1 Claude 4 Sonnet (2) 50 50.0 31.9 46.7 60.9 45.8 73.3 41.2 60.9 47.3 DeepMiner-Mano-7B (9) 100 69.2 27.7 42.5 56.5 50.0 73.3 35.3 78.3 47.9 Open-source Models OpenCUA-7B (34) 50 43.6 13.2 32.6 33.3 43.5 42.2 28.3 47.1 32.9 UI-TARS-1.5-7B (26) 100 51.9 9.6 38.2 39.1 31.3 40.0 22.4 47.8 33.0 UI-TARS-72B-dpo (26) 100 73.1 6.4 23.8 34.8 37.5 60.0 17.7 52.2 33.4 OpenCUA-32B (34) 100 66.7 18.4 37.6 36.2 55.1 46.7 33.3 63.3 41.6 DART-GUI-7B-0924 (16) 30 80.8 14.9 44.7 47.8 54.2 66.7 27.2 69.6 46.7 Baselines Qwen3-VL-4B-Thinking (3) 50 69.2 25.3 19.1 43.5 56.5 45.8 60.0 56.3 41.2 Qwen3-VL-4B-GRPO 50 73.1 10.6 31.7 56.5 54.2 66.7 46.1 47.8 42.2 Qwen3-VL-4B-REAL 50 76.9 17.0 27.5 52.2 50.0 66.7 66.4 47.8 43.0 Qwen3-VL-8B-Thinking (3) 50 76.9 12.8 29.6 52.2 62.5 60.0 23.5 65.2 42.7 Qwen3-VL-8B-GRPO 50 65.4 25.5 33.8 56.5 66.7 73.3 34.6 60.9 47.2 Qwen3-VL-8B-REAL 50 76.9 21.3 33.8 69.6 66.7 60.0 41.2 47.8 47.3 Proposed LACL-GUI-4B 50 69.2 21.3 31.7 52.2 58.3 66.7 41.2 56.5 44.5 LACL-GUI-8B 50 76.9 27.7 36.0 65.2 70.8 66.7 47.1 47.8 50.0 Table 1: Task success rate (%) on the OSWorld benchmark across different application domains. Results are reported following the official OSWorld evaluation protocol. Qwen3-VL-GRPO and Qwen3-VL-REAL denote models trained with GRPO and REAL objectives, respectively, based on the corresponding Qwen3-VL backbones. 3.3.4 Overall Loss Combining the positive and negative logit perturbations, the LACL-GUI objective is formulated as ℒLACL-GUI=log(1+∑τj∈+e−s~θ(τj)/λ)+log(1+∑τj∈−es^θ(τj)/λ),L_LACL-GUI=~ (1+ _τ^j ^+e^- s_θ(τ^j)/λ )+ (1+ _τ^j ^-e s_θ(τ^j)/λ ), (10) where s~θ(τ) s_θ(τ) and s^θ(τ) s_θ(τ) denote the perturbed logits defined for the positive and negative groups, respectively. In practice, the group-wise statistics μw _w and μe _e are detached from the computational graph using the stop-gradient operator. Consequently, gradients are propagated only through the perturbed logits, while the group statistics are treated as constants during optimization. Theorem 1. (First-order Invariance). Assume that group-wise statistics μw _w and μe _e hold as constants via the stop-gradient operation. Then LACL-GUI objective satisfies: ℒLACL−GUI=ℒREAL+(ϵ2).L_LACL-GUI=L_REAL+O(ε^2). (11) Theorem 11 (See proof in Appendix) shows that LACL-GUI preserves the first-order optimization objective of REAL while introducing higher-order logit adjustments. Therefore, proposed perturbations maintain the original success-failure preference encoded by REAL at the first order, while reshaping gradient allocation and local optimization geometry within each outcome group. Consequently, LACL-GUI refines optimization without changing the underlying success-failure classification boundary. Unlike explicit length penalties that modify the optimization target, LACL-GUI introduces efficiency and error-severity preferences effectively through gradient modulation. LACL-GUI augments trajectory-level classification with fine-grained within-group preferences. By exploiting relative efficiency and divergence information, LACL-GUI provides more informative optimization signals while retaining the stability benefits of classification-based RLVR. 4 Experiments Settings of Benchmark: We evaluate LACL-GUI on OSWorld (39), a comprehensive benchmark for evaluating MLLM-based GUI agents in realistic computer environments. Following (35), we select 222 tasks from the original 369-task benchmark for training and evaluation (detailed settings are provided in the Appendix). This subset removes tasks affected by external instability, such as unreliable network conditions and corrupted files, which may introduce evaluation noise. The selected tasks cover four application domains: (i) Operating System (OS), (i) Office (LibreOffice Calc, Impress, Writer), (i) Daily-use software (VLC Player, Thunderbird), and (iv) Professional tools (VS Code, Gimp). This diversity ensures a comprehensive coverage of realistic user scenarios. We follow the standard OSWorld evaluation protocol (39), where execution-based validation scripts determine task success and assign trajectory rewards R(τ)∈0,1R(τ)∈\0,1\. Implementation: We adopt Qwen3-VL-8B-Thinking and Qwen3-VL-4B-Thinking (3) as backbone models for LACL-GUI and baselines training. Each trajectory is limited to a maximum of 50 interaction steps. At each iteration, we sample 6 tasks and collect N=8N=8 trajectories per task after filtering by the Data Manager. The temperature parameter λ and perturbation coefficient ϵε are fixed to 1.0 and 0.03, respectively, while the history horizon of hth_t is set to 5 steps following DART (16). All models are trained with 8 NVIDIA H100 GPUs for up to 50 iterations. We report the best checkpoint selected according to evaluation performance across training iterations. 4.1 Main Results Overall Performance. Table 1 presents the performance comparison on the OSWorld benchmark across 8 applications. Unless otherwise specified, LACL-GUI models are trained with a similarity threshold Θ=0.85 =0.85. LACL-GUI achieves competitive performance against both open-source and closed-source baselines. In particular, LACL-GUI-8B achieves an overall success rate of 50.0%, improving the backbone model Qwen3-VL-8B-Thinking by 7.3%. It also surpasses several strong open-source and closed-source models, including DART-GUI-7B (46.7%) and DeepMiner-Mano-7B (47.9%). Moreover, under the same backbone architecture, LACL-GUI consistently outperforms RLVR baselines, achieving improvements over GRPO (47.2%) and REAL (47.3%). These results demonstrate the effectiveness of incorporating trajectory-level quality information into contrastive RLVR optimization. Effect over RL Baselines. Compared with existing RL-based objectives, LACL-GUI consistently improves performance across different model scales. On the 8B backbone, LACL-GUI improves over REAL and GRPO by 2.7 and 2.8 percentage points, respectively. Similarly, on the 4B backbone, LACL-GUI achieves a higher success rate than REAL and GRPO (44.5% vs. 43.0%, 42.2%). These gains indicate that explicitly modeling preferences within successful and failed trajectory groups provides more informative optimization signals than outcome-only supervision. Domain-wise Improvements. LACL-GUI achieves substantial improvements across several application domains. In particular, LACL-GUI-8B obtains strong performance on OS (70.8%) and Calc (27.7%), while also achieving competitive results on Gimp (76.9%), Writer (65.2%), and Thunderbird (66.7%). These improvements demonstrate the effectiveness of LACL-GUI in handling diverse GUI scenarios. Summary. Overall, LACL-GUI provides more informative trajectory-level supervision than existing RL objectives, leading to consistent improvements in GUI agent performance across varying scales and application domains. 4.2 Ablation Studies Configuration Success Rate (%) Gain (%) Prompt Baseline 42.7 – + Decoupled Architecture 45.4 +2.7 + Success Buffer 46.5 +1.1 + Trajectory Filter 47.2 +0.7 + LACL-GUI Objective 50.0 +2.8 Table 2: Ablation study of different components in the LACL-GUI framework. Each row incrementally introduces one additional component over the previous configuration. The prompt-based baseline uses Qwen3-VL-8B-Thinking, while GRPO is adopted for all training configurations except the proposed LACL-GUI objective. 4.2.1 Effect of Architecture Components We first investigate the contribution of each component in LACL-GUI by progressively adding the decoupled architecture, success buffer, trajectory filtering, and finally the LACL-GUI objective to the prompt-based Qwen3-VL-8B-Thinking baseline. Results are shown in Table 2. The architecture components improve performance from 42.7% to 47.2%, demonstrating the benefits of efficient rollout management, successful trajectory reuse, and informative contrastive groups under sparse rewards. Introducing the LACL-GUI objective achieves the largest single gain (+2.8%), further improving performance to 50.0%. Since this improvement is obtained on top of the complete GRPO-based framework, the gain mainly comes from the proposed length-aware preference modeling and divergence-aware failure refinement rather than architectural changes alone. 4.2.2 Effect of Length-aware Preference Modeling Figure 2: Effect of length-aware preference modeling in LACL-GUI. We compare average trajectory lengths on tasks where all three methods achieve the same outcome. We further evaluate whether LACL-GUI objective improves trajectory efficiency beyond outcome-based optimization. To isolate trajectory preference from success-rate differences, we only analyze tasks where all methods achieve the same outcome. Results are shown in Figure 2. For successful trajectories, LACL-GUI reduces the average length from 9.13 steps (REAL) to 8.19 steps, achieving a 10.3% reduction, while the base model requires 19.84 steps. This verifies our motivation that, under the same initial state and goal, shorter successful executions generally indicate fewer redundant interactions and higher-quality solutions. For failed trajectories, LACL-GUI similarly reduces the average length from 29.87 to 26.72 steps (10.5% reduction). Unlike global length penalties that may favor short failures over successful behaviors, LACL-GUI performs length modulation within the negative group through divergence-aware erroneous-step modeling, suppressing prolonged ineffective failed trajectories while preserving near-success trajectories as informative signals. 4.2.3 Sensitivity Analysis of Divergence Estimation Figure 3: Sensitivity analysis of the similarity threshold Θ for divergence estimation in LACL-GUI. The default setting (Θ=0.85 =0.85) is highlighted, and results are reported across different domains. The divergence estimation in LACL-GUI relies on a similarity threshold Θ to determine the deviation point between failed and successful trajectories. We evaluate its sensitivity by varying Θ from 0.80 to 0.90 and report the resulting performance across different domains in Figure 3. LACL-GUI maintains stable performance across different threshold settings. The overall success rate changes only marginally (49.5%, 50.0%, and 49.9%), despite larger variations in individual domains. This demonstrates that the effectiveness of divergence-aware failure refinement does not rely on precise threshold selection. Instead, the relative ordering of failed trajectories according to their divergence behaviors remains stable, enabling reliable identification of near-successful failures. 5 Conclusion In this work, we present LACL-GUI, a length-aware contrastive learning framework for training GUI agents under sparse trajectory-level supervision. LACL-GUI introduces fine-grained preference modeling within both successful and failed trajectory groups. It encourages efficient successful behaviors through length-aware positive trajectory refinement and distinguishes informative near-successful failures from early-diverging attempts through divergence-aware failure modeling. To support optimization in long-horizon GUI environments, a decoupled asynchronous training architecture is employed with trajectory filtering and a successful trajectory buffer, enabling efficient rollouts collection and providing successful references for divergence estimation. Extensive experiments on the OSWorld benchmark demonstrate that LACL-GUI consistently improves GUI task completion performance over existing RL-based baselines. Future work will explore more adaptive trajectory quality estimation strategies and extend to broader interactive environments. References Anthropic (2025a) Anthropic Claude 3.7 sonnet system card. Note: https://w-cdn.anthropic.com/9f93dfa8f445c932415d335c88852ef47f1201e/claude-3-7-sonnet-system-card.pdf Cited by: Table 1. Anthropic (2025b) Anthropic System card: claude opus 4 & claude sonnet 4. System Card Anthropic. External Links: Link Cited by: Table 1. Bai et al. (2025) S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cited by: §1, §2.1, Table 1, Table 1, §4. Brunet et al. (2012) D. Brunet, E. R. Vrscay, and Z. Wang On the mathematical properties of the structural similarity index.. IEEE Trans. Image Process. 21 (4), p. 1488–1499. Cited by: §3.3.3. Cui et al. (2026) S. Cui, P. Cheng, J. Song, Y. Gai, G. Zhang, Z. Yu, J. Lin, X. Jiang, and G. Jiang Clipo: contrastive learning in policy optimization generalizes rlvr. arXiv preprint arXiv:2603.10101. Cited by: §1, §2.3. Deng et al. (2026) W. Deng, Y. Li, B. Gong, Y. Ren, C. Thrampoulidis, and X. Li On group relative policy optimization collapse in agent search: the lazy likelihood-displacement. In ICLR 2026 Workshop on Lifelong Agents: Learning, Aligning, Evolving, Cited by: §1. Deng et al. (2023) X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su Mind2web: towards a generalist agent for the web. Advances in Neural Information Processing Systems 36, p. 28091–28114. Cited by: §2.1. Feng et al. (2026) L. Feng, Z. Xue, T. Liu, and B. An Group-in-group policy optimization for llm agent training. Advances in Neural Information Processing Systems 38, p. 46375–46408. Cited by: §2.2. Fu et al. (2025) T. Fu, A. Su, C. Zhao, H. Wang, M. Wu, Z. Yu, F. Hu, M. Shi, W. Dong, J. Wang, et al. Mano technical report. arXiv preprint arXiv:2509.17336. Cited by: Table 1. Gou et al. (2025) B. Gou, D. R. Wang, B. Zheng, Y. Xie, C. Chang, Y. Shu, H. Sun, and Y. Su Navigating the digital world as humans do: universal visual grounding for gui agents. In International Conference on Learning Representations, Vol. 2025, p. 30851–30883. Cited by: §2.1. Guo et al. (2025a) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1, §2.2. Guo et al. (2025b) D. Guo, F. Wu, F. Zhu, F. Leng, G. Shi, H. Chen, H. Fan, J. Wang, J. Jiang, J. Wang, et al. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062. Cited by: Table 1. Hu et al. (2026) J. Hu, J. Liu, J. Lai, J. Hu, Y. Sheng, S. Chen, J. Li, D. Du, and S. Guo GUI agents with reinforcement learning: toward digital inhabitants. arXiv preprint arXiv:2604.27955. Cited by: §1. Jaech et al. (2024) A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720. Cited by: §2.2. Kaufmann et al. (2023) T. Kaufmann, P. Weng, V. Bengs, and E. Hüllermeier A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312.14925. Cited by: §2.2. Li et al. (2025) P. Li, Z. Hu, Z. Shang, J. Wu, Y. Liu, H. Liu, Z. Gao, C. Shi, B. Zhang, Z. Zhang, et al. Efficient multi-turn rl for gui agents via decoupled training and adaptive data curation. arXiv preprint arXiv:2509.23866. Cited by: §3.2, §3.2, Table 1, §4. Lin et al. (2026) Z. Lin, F. Liu, Y. Yang, J. Lyu, Y. Gao, Y. Liu, Z. Lu, Y. Yu, M. Yang, J. Li, et al. Ui-voyager: a self-evolving gui agent learning via failed experience. arXiv preprint arXiv:2603.24533. Cited by: §3.3.3. Liu et al. (2025) Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin Understanding r1-zero-like training: a critical perspective. arXiv preprint arXiv:2503.20783. Cited by: §2.2. Lu et al. (2026) Z. Lu, Y. Chai, Y. Guo, X. Yin, L. Liu, H. Wang, H. Xiao, S. Ren, P. Zhao, G. Liu, et al. Ui-r1: enhancing efficient action prediction of gui agents by reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 17608–17616. Cited by: §1. Lu et al. (2025) Z. Lu, J. Ye, F. Tang, Y. Shen, H. Xu, Z. Zheng, W. Lu, M. Yan, F. Huang, J. Xiao, et al. Ui-s1: advancing gui automation via semi-online reinforcement learning. arXiv preprint arXiv:2509.11543. Cited by: §1. Luo et al. (2025) R. Luo, L. Wang, W. He, L. Chen, J. Li, and X. Xia Gui-r1: a generalist r1-style vision-language action model for gui agents. arXiv preprint arXiv:2504.10458. Cited by: §1. Mroueh (2025) Y. Mroueh Reinforcement learning with verifiable rewards: grpo’s effective loss, dynamics, and success amplification. arXiv preprint arXiv:2503.06639. Cited by: §2.3. Oord et al. (2018) A. v. d. Oord, Y. Li, and O. Vinyals Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Cited by: §1, §2.3. OpenAI (2025) OpenAI OpenAI o3 and o4-mini system card. Technical report OpenAI. External Links: Link Cited by: Table 1, Table 1. Parthasarathi et al. (2025) P. Parthasarathi, M. Reymond, B. Chen, Y. Cui, and S. Chandar GRPO-lambda: credit assignment improves llm reasoning. arXiv preprint arXiv:2510.00194. Cited by: §2.3. Qin et al. (2025) Y. Qin, Y. Ye, J. Fang, H. Wang, S. Liang, S. Tian, J. Zhang, J. Li, Y. Li, S. Huang, et al. Ui-tars: pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326. Cited by: §1, §2.1, Table 1, Table 1. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. Cited by: §2.2. Sammani et al. (2023) F. Sammani, B. Joukovsky, and N. Deligiannis Visualizing and understanding contrastive learning. IEEE Transactions on Image Processing 33, p. 541–555. Cited by: §2.3. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §2.2. Step, Prompt Intermediate (2026) Step, Prompt Intermediate BranPO: scalable contrastive branch sampling for long-horizon agentic reinforcement learning. Cited by: §1. Tang et al. (2026) J. Tang, Y. Xia, Y. Wu, Y. Hu, C. Yuhui, Q. Chen, X. Xu, X. Wu, H. Lu, Y. Ma, et al. Lpo: towards accurate gui agent interaction via location preference optimization. In Findings of the Association for Computational Linguistics: ACL 2026, p. 14617–14628. Cited by: §2.1. Team et al. (2025) K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. Kimi k1. 5: scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. Cited by: §2.2. Wang et al. (2025) H. Wang, H. Zou, H. Song, J. Feng, J. Fang, J. Lu, L. Liu, Q. Luo, S. Liang, S. Huang, et al. Ui-tars-2 technical report: advancing gui agent with multi-turn reinforcement learning. arXiv preprint arXiv:2509.02544. Cited by: §1. Wang et al. (2026a) X. Wang, B. Wang, D. Lu, J. Yang, T. Xie, J. Wang, J. Deng, X. Guo, Y. Xu, C. Wu, et al. Opencua: open foundations for computer-use agents. Advances in Neural Information Processing Systems 38, p. 139756–139806. Cited by: Table 1, Table 1. Wang et al. (2026b) Y. Wang, T. Xie, K. Shen, M. Wang, and L. Yang RLAnything: forge environment, policy, and reward model in completely dynamic rl system. arXiv preprint arXiv:2602.02488. Cited by: §2.2, §4. Wen et al. (2024) H. Wen, Y. Li, G. Liu, S. Zhao, T. Yu, T. J. Li, S. Jiang, Y. Liu, Y. Zhang, and Y. Liu Autodroid: llm-powered task automation in android. In Proceedings of the 30th annual international conference on Mobile computing and networking, p. 543–557. Cited by: §2.1. Wu et al. (2026) Q. Wu, K. Cheng, R. Yang, C. Zhang, J. Yang, H. Jiang, J. Mu, B. Peng, B. Qiao, R. Tan, et al. Gui-actor: coordinate-free visual grounding for gui agents. Advances in Neural Information Processing Systems 38, p. 15101–15128. Cited by: §2.1. Wu et al. (2025) Z. Wu, Z. Wu, F. Xu, Y. Wang, Q. Sun, C. Jia, K. Cheng, Z. Ding, L. Chen, P. P. Liang, et al. OS-atlas: foundation action model for generalist gui agents. In International Conference on Learning Representations, Vol. 2025, p. 5090–5108. Cited by: §2.1. Xie et al. (2024) T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, et al. Osworld: benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems 37, p. 52040–52094. Cited by: §1, §4. Xu et al. (2024) Y. Xu, Z. Wang, J. Wang, D. Lu, T. Xie, A. Saha, D. Sahoo, T. Yu, and C. Xiong Aguvis: unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454. Cited by: §1. Yu et al. (2026a) Q. Yu, A. Tartaglini, P. Hase, C. Guestrin, and C. Potts Outcome rewards do not guarantee verifiable or causally important reasoning. arXiv preprint arXiv:2604.22074. Cited by: §1. Yu et al. (2026b) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. Dapo: an open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems 38, p. 113222–113244. Cited by: §2.2. Yu et al. (2026c) Z. Yu, W. Luo, Y. Sun, H. Fang, R. Cong, C. Lu, and C. Gong Harmony in diversity: multi-domain contrastive policy optimization for large reasoning models. arXiv preprint arXiv:2605.25443. Cited by: §1. Yuan et al. (2026) X. Yuan, J. Zhang, K. Li, Z. Cai, L. Yao, J. Chen, E. Wang, Q. Hou, J. Chen, P. Jiang, et al. Se-gui: enhancing visual grounding for gui agents via self-evolutionary reinforcement learning. Advances in Neural Information Processing Systems 38, p. 127658–127679. Cited by: §1. Zhai et al. (2026) Z. Zhai, M. Chen, J. Zhao, J. Qian, L. Shen, and Y. Lu Rewards as labels: revisiting rlvr from a classification perspective. arXiv preprint arXiv:2602.05630. Cited by: §1, §1, §2.2, §2.3, §3.3.1, §3.3.1. Zhan et al. (2026) Y. Zhan, X. Tang, J. Li, D. Zheng, W. Chai, J. Chen, J. Zhou, G. Wu, W. Tang, and H. Sun GraphPO: graph-based policy optimization for reasoning models. External Links: 2606.18954, Link Cited by: §1. Zhang et al. (2026) F. Zhang, X. Ma, Z. Dong, X. Leng, J. Zhao, X. Sun, Y. Yang, and G. Jiang Revisiting reinforcement learning with verifiable rewards from a contrastive perspective. arXiv preprint arXiv:2605.12969. Cited by: §1, §1, §2.3. Zhang et al. (2024) J. Zhang, J. Wu, T. Yihua, M. Liao, N. Xu, X. Xiao, Z. Wei, and D. Tang Android in the zoo: chain-of-action-thought for gui agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 12016–12031. Cited by: §1. Zheng et al. (2025) C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071. Cited by: §2.2. Zheng et al. (2026) C. Zheng, X. Mo, X. Ma, Q. Lin, Y. Zhao, J. Zhu, X. Lou, J. Wang, Z. Wang, W. Liu, et al. Adaptive milestone reward for gui agents. arXiv preprint arXiv:2602.11524. Cited by: §1, §2.2. Zhou et al. (2026) Y. Zhou, S. Dai, S. Wang, K. Zhou, Q. Jia, and J. Xu Gui-g1: understanding r1-zero-like training for visual grounding in gui agents. Advances in Neural Information Processing Systems 38, p. 95683–95705. Cited by: §1. Appendix A Appendix A.1 Proof of Theorem 1 We first prove Lemma 1: Lemma 1 (Weighted Zero-Mean Property). For the proposed logit perturbations in LACL-GUI, the weighted perturbations satisfy ∑τj∈+e−s¯θ(τj)/λ(|τj|−μw)=0 _τ^j ^+e^- s_θ(τ^j)/λ (|τ^j|- _w )=0 (12) and ∑τj∈−es¯θ(τj)/λ(Ej−μe)=0. _τ^j ^-e s_θ(τ^j)/λ (E^j- _e )=0. (13) Proof. For the positive group, the weighted mean trajectory length is defined as μw=∑τj∈+p+(τj)|τj|, _w= _τ^j ^+p^+(τ^j)\,|τ^j|, (14) where p+(τj)=e−s¯θ(τj)/λ∑τj∈+e−s¯θ(τj)/λ.p^+(τ^j)= e^- s_θ(τ^j)/λ _τ^j ^+e^- s_θ(τ^j)/λ. (15) Therefore, ∑τj∈+e−s¯θ(τj)/λ(|τj|−μw) _τ^j ^+e^- s_θ(τ^j)/λ (|τ^j|- _w ) =∑τj∈+e−s¯θ(τj)/λ|τj|−μw∑τj∈+e−s¯θ(τj)/λ = _τ^j ^+e^- s_θ(τ^j)/λ\,|τ^j|- _w _τ^j ^+e^- s_θ(τ^j)/λ =μw∑τj∈+e−s¯θ(τj)/λ−μw∑τj∈+e−s¯θ(τj)/λ = _w _τ^j ^+e^- s_θ(τ^j)/λ- _w _τ^j ^+e^- s_θ(τ^j)/λ =0. =0. The negative-group identity follows by the same argument with the corresponding weighted mean μe _e and weights es¯θ(τj)/λe s_θ(τ^j)/λ. ∎ Theorem 1. (First-order Invariance). Assume that group-wise statistics μw _w and μe _e hold as constants via the stop-gradient operation. Then LACL-GUI objective satisfies: ℒLACL−GUI=ℒREAL+(ϵ2).L_LACL-GUI=L_REAL+O(ε^2). (16) Proof. We denote the positive-group loss of REAL and LACL-GUI by ℒREAL+L^+_REAL and ℒ~+ L^+, respectively, and the negative-group loss of REAL and LACL-GUI by ℒREAL−L^-_REAL and ℒ~− L^-, respectively. We first consider the positive group. The original REAL positive loss (in log-sum-exp form) is ℒREAL+=log(1+∑τj∈+e−s¯θ(τj)/λ).L^+_REAL= (1+ _τ^j ^+e^- s_θ(τ^j)/λ ). (17) After perturbation it becomes ℒ~+=log(1+∑τj∈+e−(s¯θ(τj)+ϵg(τj))/λ). L^+= (1+ _τ^j ^+e^-( s_θ(τ^j)+ε g(τ^j))/λ ). (18) where OPENg(τj))=|τj|−μwg(τ^j))=|τ^j|- _w. Using the first-order Taylor expansion e−ϵg(τj)/λ=1−ϵg(τj)λ+(ϵ2),e^-ε g(τ^j)/λ=1- ε g(τ^j)λ+O(ε^2), (19) we can expand the sum as ∑τj∈+e−(s¯θ(τj)+ϵg(τj))/λ _τ^j ^+e^-( s_θ(τ^j)+ε g(τ^j))/λ =∑τj∈+e−s¯θ(τj)/λ⋅e−ϵg(τj)/λ = _τ^j ^+e^- s_θ(τ^j)/λ· e^-ε g(τ^j)/λ =∑τj∈+e−s¯θ(τj)/λ(1−ϵg(τj)λ+(ϵ2)) = _τ^j ^+e^- s_θ(τ^j)/λ (1- ε g(τ^j)λ+O(ε^2) ) =∑τj∈+e−s¯θ(τj)/λ−ϵλ∑τj∈+e−s¯θ(τj)/λg(τj) = _τ^j ^+e^- s_θ(τ^j)/λ- ελ _τ^j ^+e^- s_θ(τ^j)/λ\,g(τ^j) +(ϵ2). +O(ε^2). The key observation is that the coefficient of the first-order term vanishes. By Lemma 1, we have the weighted zero-mean property ∑τj∈+e−s¯θ(τj)/λg(τj)=0, _τ^j ^+e^- s_θ(τ^j)/λ\,g(τ^j)=0, (20) Consequently, the linear term in ϵε is exactly zero, and we obtain ∑τj∈+e−(s¯θ(τj)+ϵg(τj))/λ=∑τj∈+e−s¯θ(τj)/λ+(ϵ2). _τ^j ^+e^-( s_θ(τ^j)+ε g(τ^j))/λ= _τ^j ^+e^- s_θ(τ^j)/λ+O(ε^2). (21) Substituting this back into the log-sum-exp loss immediately yields ℒ~+=ℒREAL++(ϵ2). L^+=L^+_REAL+O(ε^2). (22) We now turn to the negative group. The original REAL negative loss is ℒREAL−=log(1+∑τj∈−es¯θ(τj)/λ).L^-_REAL= (1+ _τ^j ^-e s_θ(τ^j)/λ ). (23) After applying the perturbation s^θ(τj)=s¯θ(τj)+ϵy(τj) s_θ(τ^j)= s_θ(τ^j)+ε y(τ^j), where y(τj)=E(τj)−μey(τ^j)=E(τ^j)- _e, it becomes ℒ~−=log(1+∑τj∈−e(s¯θ(τj)+ϵy(τj))/λ). L^-= (1+ _τ^j ^-e^( s_θ(τ^j)+ε y(τ^j))/λ ). (24) Using the first-order expansion eϵy(τj)/λ=1+ϵy(τj)λ+(ϵ2)e^ε y(τ^j)/λ=1+ ε y(τ^j)λ+O(ε^2) (25) and following the same algebraic steps as in the positive group, we obtain ∑τj∈−e(s¯θ(τj)+ϵy(τj))/λ _τ^j ^-e^( s_θ(τ^j)+ε y(τ^j))/λ =∑τj∈−es¯θ(τj)/λ+ϵλ∑τj∈−es¯θ(τj)/λy(τj)+(ϵ2). = _τ^j ^-e s_θ(τ^j)/λ+ ελ _τ^j ^-e s_θ(τ^j)/λ\,y(τ^j)+O(ε^2). By the weighted zero-mean property in Lemma 1 , the coefficient of the linear term vanishes. Therefore, ℒ~−=ℒREAL−+(ϵ2). L^-=L^-_REAL+O(ε^2). (26) Combining the results for both groups, we conclude that ℒLACL-GUI=ℒREAL+(ϵ2).L_LACL -GUI=L_REAL+O(ε^2). (27) Thus the proposed perturbations preserve the REAL objective up to first order while redistributing optimization emphasis in higher-order within each outcome group. ∎ A.2 Experiment Details A.2.1 OSWorld Benchmark Settings We evaluate on a subset of 222 tasks selected from the full OSWorld benchmark of 369 tasks. All experiments are conducted in Baidu’s RLE virtual machine environment. We exclude Chrome and Multi-app tasks. This task filtering is motivated by the need for stable training and evaluation conditions. First, Chrome tasks require reliable external network connectivity, which is unstable in our RLE environment and frequently leads to non-reproducible outcomes. Second, Multi-app tasks often depend on external asset files hosted on the official huggingface cache (for example, task 3f05f3b9-29ba-4b6b-95a-2204697ffc06, which requires the audio file “Zhou Xuan - Nights in Shanghai.mp3”). These assets were frequently missing or incomplete in our setup, resulting in systematic environment failures rather than genuine agent errors. Both issues introduce significant instability into both training and testing, and the corresponding tasks are therefore excluded. After removing the affected Chrome and Multi-app tasks, the remaining 222 tasks span eight application domains. Their distribution is summarized in Table 3. Domain Number of Tasks Gimp 26 LibreOffice Calc 47 LibreOffice Impress 47 LibreOffice Writer 23 OS 24 Thunderbird 15 VLC 17 VS Code 23 Total 222 Table 3: Distribution of the 222 selected OSWorld tasks across application domains. A.2.2 Effect of Length-aware Preference Modeling To further examine the effect of length-aware preference modeling, we conduct a fine-grained analysis of average trajectory lengths on tasks where the base model Qwen3-VL-8B-Thinking, REAL, and LACL-GUI all achieve the same outcome. This isolates the impact of trajectory preference from differences in task success rate. Results are reported separately for successful and failed trajectories. Table 4 presents the average trajectory lengths on the 70 tasks that are successfully completed by all three methods. LACL-GUI reduces the overall average length from 9.13 steps (REAL) to 8.19 steps (a 10.3% reduction). The reduction holds in the majority of application domains (Gimp, LibreOffice Calc, OS, Thunderbird, and VS Code), although the effect is not uniform: LibreOffice Writer shows a slight increase and VLC exhibits a clear increase. These results largely support the design motivation that, under identical initial states and goals, shorter successful executions generally correspond to higher-quality solutions with fewer redundant interactions. Table 5 reports the corresponding statistics on the 93 tasks where all three methods fail. LACL-GUI again yields a shorter average trajectory (26.72 steps) compared with REAL (29.87 steps), corresponding to a 10.5% reduction. Length reduction is observed in six out of eight domains, with particularly large improvements in Gimp and OS. Exceptions occur in LibreOffice Writer and VS Code, where LACL-GUI produces longer failed trajectories. Because length modulation is applied only within the negative group via divergence-aware erroneous-step modeling, LACL-GUI tends to suppress prolonged ineffective failures while still preserving near-success trajectories as informative learning signals. Domain Base LACL-GUI REAL Rel. to REAL Gimp 36.46 12.08 13.38 −9.7%-9.7\% Calc 17.50 10.17 16.00 −36.4%-36.4\% Impress 4.25 4.25 4.25 0.0%0.0\% Writer 10.73 5.36 5.27 +1.7%+1.7\% OS 21.67 4.25 5.50 −22.7%-22.7\% Thunderbird 7.86 6.71 7.29 −8.0%-8.0\% VLC 16.75 10.75 5.00 +115.0%+115.0\% VS Code 30.67 13.44 15.56 −13.6%-13.6\% Average 19.84 8.19 9.13 −10.3%-10.3\% Table 4: Average trajectory lengths on the 70 tasks successfully completed by all three methods. The last column shows the relative change of LACL-GUI with respect to REAL. Domain Base LACL-GUI REAL Rel. to REAL Gimp 33.80 5.20 25.00 −79.2%-79.2\% Calc 39.91 30.12 33.97 −11.3%-11.3\% Impress 30.48 27.86 31.24 −10.8%-10.8\% Writer 37.83 46.67 42.67 +9.4%+9.4\% OS 35.67 10.67 22.67 −52.9%-52.9\% Thunderbird 50.00 26.75 27.75 −3.6%-3.6\% VLC 33.14 13.14 15.57 −15.6%-15.6\% VS Code 36.86 25.14 16.57 +51.7%+51.7\% Average 36.06 26.72 29.87 −10.5%-10.5\% Table 5: Average trajectory lengths on the 93 tasks where all three methods fail. The last column shows the relative change of LACL-GUI with respect to REAL. A.3 Real-world Trajectories To provide an intuitive illustration of why length-aware preference within successful trajectories and divergence-aware modeling within failed trajectories are beneficial, we present concrete trajectory examples from the same Gimp task: Please help change GIMP’s theme from dark to light. (task ID: 7767eef2-56a3-4cea-8c9f-48c070c7d65b). This task requires navigating Gimp’s preference menus to switch the interface theme. Although the goal is relatively straightforward, different rollouts can exhibit substantial variation in efficiency and in how closely failed attempts approach a successful solution. We therefore examine both successful and failed trajectories produced for this task to highlight the fine-grained quality differences that outcome-level supervision alone cannot capture. Positive Group: Efficient vs. Redundant Successful Trajectories. Although both trajectories receive the same success reward, successful executions can exhibit substantially different levels of efficiency. We illustrate this phenomenon using the Gimp theme-switching task. The following example compares a concise successful trajectory generated by LACL-GUI-8B (6 steps) with a longer successful trajectory generated by REAL (27 steps). The 6-step trajectory follows the optimal interaction path: it directly opens the Edit menu, enters Preferences, selects the Theme option, switches to Light mode, and confirms the change. No exploratory or recovery actions are involved. Step 1: Click on the "Edit" menu in the top menu bar. pyautogui.click(124, 72) Step 2: Click on the "Preferences" option in the Edit menu. pyautogui.click(180, 640) Step 3: Click on the "Theme" option under the Interface section. pyautogui.click(157, 667) Step 4: Select the "Light" theme option. pyautogui.click(363, 622) Step 5: Click the "OK" button to apply the theme. pyautogui.click(832, 867) Step 6: Terminate (task completed successfully). DONE Figure 4: LACL-GUI Step 1 (Success) Figure 5: LACL-GUI Step 2 (Success) Figure 6: LACL-GUI Step 3 (Success) Figure 7: LACL-GUI Step 4 (Success) Figure 8: LACL-GUI Step 5 (Success) Figure 9: LACL-GUI Step 6 (Success) In contrast, the 27-step trajectory produced by Qwen3-VL-8B-REAL also successfully completes the task but contains extensive unnecessary interactions: Step 1-4: Navigate to the Theme panel correctly. Step 5-15: Repeatedly click the "Theme" item in the left sidebar. pyautogui.click(163, 667) (repeated) Step 16: Click the "Theme" item again. Step 17-24: Continue clicking the same screen location after the UI state has changed. pyautogui.click(167, 667) (repeated) Step 25: Select the "Light" theme option. pyautogui.click(359, 620) Step 26: Click the "OK" button to apply the change. pyautogui.click(830, 859) Step 27: Terminate (task completed successfully). DONE The long trajectory reveals two types of inefficiency. First, from Step 5 to Step 15, the agent repeatedly executes a valid action without making progress, producing redundant interactions. Second, from Step 17 to Step 24, the agent continues clicking the previous target location even after the interface state has changed, resulting in erroneous actions caused by outdated state understanding. These additional steps do not contribute to task completion and are not desirable behaviors to reinforce, despite the trajectory receiving the same binary success reward. Therefore, length-aware positive preference in LACL-GUI assigns stronger preference to concise successful trajectories, encouraging the policy to learn efficient solutions rather than merely optimizing for eventual task completion. The screenshots below highlight representative failure patterns in the long successful trajectory. Step 5 demonstrates redundant interactions where the agent repeatedly selects an already active option, while Step 17 illustrates incorrect repeated clicking after the UI state has changed. Figure 10: A redundant interaction in the long successful trajectory (Step 5). Figure 11: An erroneous repeated action after the UI state changes (Step 17). Negative Group: Different Failure Patterns Beyond Binary Rewards. While successful trajectories can differ in efficiency, failed trajectories can also exhibit substantially different qualities despite receiving the same binary reward. We illustrate this phenomenon using two failed trajectories generated by Qwen3-VL-8B-Thinking on the same Gimp theme-switching task. Both trajectories obtain a reward of 0, but they exhibit different failure behaviors after deviating from the successful execution path. For reference, the successful trajectory generated by LACL-GUI-8B changes the theme by selecting the “Theme” option at Step 4, followed by choosing the “Light” theme and confirming the change. Both failed trajectories diverge at this critical step: although the agent reaches the Preferences dialog and interacts with the Theme option, it fails to correctly complete the theme switch. The first failure terminates prematurely after only four steps. The agent correctly navigates to the Theme option but incorrectly assumes that the task has been completed without verifying whether the desired state has been achieved. Step 1: Click the "Edit" menu in the top menu bar. pyautogui.click(124, 72) Step 2: Click the "Preferences" option in the Edit menu. pyautogui.click(180, 640) Step 3: Click the "Theme" option under the Interface section. pyautogui.click(157, 669) Step 4: Terminate (task incorrectly considered completed). DONE In contrast, the second failure continues interacting after the same divergence point. Although the agent performs additional actions, these steps do not recover the correct execution path and only introduce ineffective exploration before termination. Step 1-3: Navigate to the Theme option following the correct menu hierarchy. Step 4: Click the "Theme" option again in the Preferences dialog. pyautogui.click(167, 667) Step 5: Click the "OK" button without successfully applying the Light theme. pyautogui.click(830, 859) Step 6: Terminate (task incorrectly considered completed). DONE These examples reveal two common failure modes that are indistinguishable under binary rewards. The first corresponds to premature termination (result in early divergence from reference successful trajectory) after an incomplete state transition, while the second corresponds to unnecessary post-divergence interactions that increase trajectory length without improving task completion. Therefore, evaluating failed trajectories requires finer-grained preference modeling beyond success or failure labels. By incorporating trajectory structure and ineffective actions, LACL-GUI assigns more informative preferences within negative groups, encouraging the policy to avoid both early mistakes and unproductive exploration. The screenshots below show representative failure cases. The first example demonstrates premature termination after the agent reaches the Theme option, while the second illustrates unnecessary post-divergence interactions. Figure 12: Premature termination after an incomplete state transition (Step 4 in 4-step Failed Trajectory). Figure 13: Additional ineffective actions after divergence from the successful execution (Step 5 in 6-step Failed Trajectory). These failure cases also demonstrate the motivation of the negative-group modulation in LACL-GUI. For each failed trajectory, LACL-GUI first identifies the divergence point from the reference successful trajectory using observation similarity and then estimates the number of erroneous steps after divergence. Although both trajectories deviate at the same decision point in this example, they exhibit different error severities: the 4-step trajectory terminates immediately after divergence, whereas the 6-step trajectory continues with additional ineffective interactions. Consequently, the latter receives a larger erroneous-step count and stronger suppression through the perturbed logits in Eq. (9), preventing the policy from reinforcing long unsuccessful exploration. Meanwhile, this fine-grained modeling preserves informative failures that remain closer to successful execution, instead of treating all failed trajectories uniformly as in REAL.