Paper deep dive

Can Agents Work for Everyone? Cross-User Reliability for Mobile GUI Agents in Personalized User Interfaces

Yeji Park, Jaeyun Shim, Taesik Gong

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 71

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Mobile GUI agents increasingly operate on interfaces influenced by users' histories and preferences, but their reliability across different users remains underexplored. We introduce PAIR (Personalized Application-state Instantiation and Rendering), a pipeline for constructing user-conditioned application states that enables controlled evaluation of the same task across different users. We further introduce RePAIR (Reinforcement learning with Personalization-Aware Interaction Rewards), a training approach that learns from cross-user differences in subgoal outcomes to improve reliability across user-conditioned mobile environments. Across six agents, we find substantial variation in task success across users and consistently lower subgoal achievement in user-conditioned UI contexts (6.98 to 15.4 pp). This gap further increases for personal targets drawn from each user's own content (8.77 to 22.0 pp). Failures in these contexts frequently involve selecting another item instead of the intended target, particularly before target exposure. Finally, RePAIR improves user-conditioned SAR (+5.87 pp), all-success (+7.50 pp), and overall Task SR (+9.42 pp) over its supervised fine-tuning parent on unseen users, providing initial evidence that explicitly learning from cross-user variation can improve GUI-agent reliability.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

67,732 characters extracted from source content.

Expand or collapse full text

CAN AGENTS WORK FOR EVERYONE? CROSS-USER RELIABILITY FOR MOBILE GUI AGENTS IN PERSONALIZED USER INTERFACES Yeji Park, Jaeyun Shim, Taesik Gong UNIST yejipark, tlawodbs1170, taesik.gong@unist.ac.kr ABSTRACT Mobile GUI agents increasingly operate on interfaces influenced by users’ histories and preferences, but their reliability across different users remains underexplored. We introduce PAIR (Personalized Application-state Instantiation and Rendering), a pipeline for constructing user-conditioned application states that enables con- trolled evaluation of the same task across different users. We further introduce RePAIR (Reinforcement learning with Personalization-Aware Interaction Rewards), a training approach that learns from cross-user differences in subgoal outcomes to improve reliability across user-conditioned mobile environments. Across six agents, we find substantial variation in task success across users and consistently lower subgoal achievement in user-conditioned UI contexts (6.98 to 15.4 p). This gap further increases for personal targets drawn from each user’s own content (8.77 to 22.0 p). Failures in these contexts frequently involve selecting another item instead of the intended target, particularly before target exposure. Finally, RePAIR improves user-conditioned SAR (+5.87 p), all-success (+7.50 p), and overall Task SR (+9.42 p) over its supervised fine-tuning parent on unseen users, providing initial evidence that explicitly learning from cross-user variation can improve GUI-agent reliability. 1INTRODUCTION Recent advances in mobile GUI agents have enabled them to interpret natural-language requests, interpret graphical user interfaces, and perform actions in a human-like manner (Zhou et al., 2025; Qin et al., 2025; Liu et al., 2024; Zhao et al., 2025a). However, GUI agents’ action predictions are sensitive to interface presentation, including layout and element positioning (Yu et al., 2026; Wang et al., 2026b; Chen et al., 2025; Yang et al., 2026c). In deployed mobile GUI agents, interface variations naturally arise from users’ prior interactions and activity histories. Such user-dependent interface differences can lead to cross-user reliability failures, where an agent that succeeds for one user may fail on the same request for another. Figure 1 shows how user- dependent interfaces can lead an agent to different types of failure, for instance, selecting a visually similar but incorrect target or prematurely concluding that the requested target is unavailable. Our analysis of users’ real mobile interactions from FingerTip 20K (Yang et al., 2026b) likewise indicates a 6.41% (SD 2.38) lower success rate (24.71% vs. 31.12%) on labeled target-bearing screen segments containing user-conditioned elements. However, because FingerTip 20K varies both tasks and screen states, it does not support controlled evaluation of the same request across user-conditioned interfaces. To address this problem, we introduce PAIR (Personalized Application-state Instantiation and Ren- dering), a pipeline for constructing user-conditioned application states from user profiles. PAIR extracts application-relevant user preferences, matches them to candidate content for each application based on semantic similarity, selects the highest-ranked content, and renders it in each application’s UI. By varying user-conditioned content while keeping the application structure, PAIR enables controlled cross-user evaluation of the same task. Additionally, we introduce RePAIR (Reinforcement learning with Personalization-Aware Interaction Rewards), a training-based approach to improve cross-user reliability by explicitly learning from differences in agent performance across users. Re- 1 arXiv:2610.07972v1 [cs.AI] 6 Oct 2026 SoulR&BGospel Ladysmith Black Mambazo South African Classics SoulJazz High Audio Quality Afro-Soul Grooves house FailFail SuccessSuccess User1 UI Context User2 UI Context R&B Yes, that's the right song! ! You completed the tas k exactly as I asked. PerfectCase #2 "I can see 'Chill (Piano Bar)' by This is the song I need to add to 'Piano Jazz Collection' in the list. the new playlist." I said Chill Piano, not Chill Piano Bar. You stopped too early. Look further down my playlists. Find the song 'Chill Piano' in the liked songs and add it to the 'Rainy Night' playlist. Fail Success "There is no playlist named 'Rainy Night' here. The closest which might be the intended one." match is 'Rainy Day Piano', Figure 1: The same request can require navigating different user-conditioned interfaces. We evaluate whether GUI agents remain reliable across users and trace how user-conditioned content disrupts task progress and target selection. PAIR compares the outcomes of subgoals, goals associated with benchmark-defined UI contexts, across different users’ environments under the same task condition and reinforces successful behavior in user-conditioned contexts, where the visible interface content varies across users. By leveraging these cross-user differences as a training signal, RePAIR aims to improve consistent task execution across user-conditioned application states. Using the user-conditioned application states generated by PAIR, we evaluate whether agents complete the same task consistently across users and where cross-user reliability breaks down. Our evaluation reveals four main findings. First, task success varies substantially across users, with a maximum gap of 9.31 p between the best- and worst-performing users. Second, subgoal achievement is consistently lower in user-conditioned contexts than in user-agnostic contexts, where the visible UI does not depend on the user (+12.0 p). This gap becomes larger for personal targets where the agent must identify a target drawn from the user’s own content (+14.4 p). Third, in user-conditioned contexts, subgoal failures involve selecting another item instead of the intended target. These errors occur more frequently for personal targets, particularly before target exposure, when additional navigation is required to reach the intended target. Finally, RePAIR improves user-conditioned SAR by 5.87 points, the all-success rate by 7.50 points, and overall Task SR by 9.42 points over its SFT parent on unseen users, suggesting that explicitly training on cross-user differences in user-conditioned subgoals can improve reliability across unseen users. We summarize our key contributions as follows: • We introduce PAIR, a user profile-based pipeline for constructing user-conditioned GUI environments that enables controlled evaluation of the same task across different users. •We provide an empirical study of cross-user reliability, showing (i) substantial variation in task success across users, (i) lower subgoal achievement in user-conditioned contexts, and (i) more target-selection errors for personal targets, especially before target exposure. • We introduce RePAIR, a training approach that leverages cross-user differences in subgoal outcomes to improve reliable task execution across user-conditioned environments. 2RELATED WORK 2.1GUI AGENT ROBUSTNESS UNDER INTERFACE VARIATION GUI agents have evolved from grounding instructions in structured GUI representations to perceiving and acting on rendered interfaces (Deng et al., 2023; Cheng et al., 2024; Wu et al., 2025). Re- cent systems integrate visual grounding, reasoning, and action prediction in a unified interaction 2 Table 1: Comparison of GUI-agent evaluation settings. The four dimensions are UI Variation, whether UI variations are explicitly evaluated; User Context, whether user preferences or histories are used; User-Conditioned GUI, whether personal records shape the app state; and Cross-User Comparison, whether the same task is evaluated across user-conditioned application states. Evaluation settingUI VariationUser Context User-Conditioned GUI Cross-User Comparison GUI-Perturbed (Wang et al., 2026b)✓✗ GUI-Robust (Yang et al., 2026a)✓✗ D-GARA (Chen et al., 2026a)✓✗ PSPA-Bench (Nie et al., 2026)✗✓✗ Persona2Web (Kim et al., 2026)✗✓✗ KnowU-Bench (Chen et al., 2026b)✗✓✗ FingerTip 20K (Yang et al., 2026b)✗✓✗ iOSWorld (Jang et al., 2026)✗✓✗ PAIR (Ours)✓ loop (Zheng et al., 2024; Qin et al., 2025). This shift highlights the need for robustness to variations in interfaces. Visual attributes affect web-agent selections (Yu et al., 2026), while zoom perturbations reduce grounding accuracy (Wang et al., 2026b). Prior methods address GUI complexity through context simplification (Chen et al., 2025) and adaptive visual focus (Ge et al., 2025). Meanwhile, benchmarks evaluate robustness using static anomaly datasets (Yang et al., 2026a) and dynamically injected disruptions (Chen et al., 2026a). Together, these studies show that reliable GUI interaction requires robustness to dynamic conditions, not just target recognition on fixed screens. Personalization, however, creates persistent, user-conditioned variation beyond temporary or synthetic perturbations. HCI research shows that usage history and personalized content shape interface presentation (Findlater & McGrenere, 2004; Eslami et al., 2015), while user-specific records shape the content and context of modern applications (Jang et al., 2026). Existing robustness studies nevertheless focus on synthetic or transient changes, leaving reliability across persistent, personalized application states underexplored. 2.2PERSONALIZATION IN MOBILE GUI AGENTS Existing personalization methods for mobile GUI agents use user preferences, routines, and histories to infer intended actions or provide proactive assistance (Wang et al., 2026a; Lyu et al., 2026). In contrast, we study whether an agent can reliably execute a fixed instruction when the mobile interface itself varies with user-specific content and state. Prior mobile evaluations examine either robustness to runtime interface disruptions (Chen et al., 2026a) or personalized interaction based on user context (Nie et al., 2026; Yang et al., 2026b; Jang et al., 2026; Chen et al., 2026b). However, they do not directly test whether the same instruction and target remain reliable across different user- conditioned mobile application states. We address this gap through controlled cross-user evaluation. As summarized in Table 1. 3MOTIVATION: REAL-WORLD MOBILE GUI AGENTS ENCOUNTER USER-CONDITIONED INTERFACES To investigate how user-conditioned content influences mobile GUI agents in everyday interfaces, we evaluate six widely used open-source agent models on Fingertip 20K (Yang et al., 2026b), which captures screenshots and action sequences from users’ real mobile interactions. We analyze 103,199 screenshot–action steps spanning 12,016 episodes from 102 apps, collected from 83 users. We group consecutive interactions on the same screen into 60,383 screen segments, and success requires every scored action in a segment to be correct. Across all evaluated agents, segments containing user- conditioned elements, labels unique to a given user within each application, show a mean decrease in success of 4.88 p (SD 1.50) relative to those without them (Figure 2a). 3 GUI-Owl UI-TARS AutoGLM MAI-UI 30B 8B 0 10 20 30 Segment success (%) -6.5 -3.2 -3.5 -4.6 -6.8 -4.5 Qwen3-VL User-conditioned elements WithoutWith (a) Success with/without user-conditioned elements. Ground Truth User-conditioned element Model Prediction AllTo payTo shipmentReceivingRefunds/After-sales Buyer paid Product title Quantity / product option 7-day no-reason return / Fast refund Xiha General Store To shipment Track package Urge shipmentEdit address Ship before 17:18, the day after tomorrow (b) Mistaking user-conditioned order content for the task target. Ground Truth Likes & Favorites Messages New FollowersMentions User-conditioned element Model Prediction Discover group chats (c) Overlooking the visible user-conditioned unread chat. Figure 2: GUI-agent behavior on FingerTip 20K. (a) Success by the presence of user-conditioned content. (b,c) Example failures: the agent (b) selects a user-specific shop name instead of Track package and (c) overlooks a user-specific unread chat in favor of Discover group chats. Figures 2b and 2c illustrate two types of errors influenced by user-conditioned elements. In the first case, the user asks to check the delivery status of the first order. The user intends to select the order’s Track package button, but the agent selects the shop name instead, incorrectly assuming it leads to the order details, where delivery information can be found. In the second case, the user asks to view new messages in a group chat. The intended action is to open the visible group conversation with an unread badge, but the agent selects the shared Discover group chats entry instead, incorrectly assuming that it provides access to new messages in the user’s group chats. These examples highlight the need to interpret user-conditioned content in relation to the instruction and identify the appropriate target in the current application state. See more details in Appendix A. These observations suggest that user-conditioned content can interfere with reliable action selection. However, because FingerTip varies tasks and screen states together, it cannot determine whether the same request is completed consistently across different users’ application states. We refer to this consistency as cross-user reliability. We therefore suggest PAIR to control for the instruction and target while varying user-conditioned application states and study the following questions: • RQ1: How robustly do mobile GUI agents complete the same request across user- conditioned interface states? •RQ2: Where does task progress break down in user-conditioned interfaces, and how do failure patterns vary with the semantic alignment between the target and the interface? 4PAIR: PERSONALIZED APPLICATION ENVIRONMENT GENERATION We introduce PAIR (Personalized Application-state Instantiation and Rendering), a pipeline that turns user profiles into personalized, executable GUI environments. PAIR provides a controlled cross-user evaluation environment by pairing user-conditioned application states with matched task templates and success criteria. This section describes the environment generation pipeline, task design, and evaluation protocol. Figure 3 illustrates the pipeline overview. 4.1CROSS-USER RELIABILITY FORMULATION Our goal is to assess cross-user reliability, defined as the ability of a mobile GUI agent to execute the same request consistently across different users’ interfaces, and to identify where task progress breaks down during execution. Given an instructionqand userp’s application state, the agent observes a screenshot o t and selects an action a t at each interaction step t: a t = π(q,h t ), s t = (o t ,a t ),(1) whereπdenotes the GUI agent,h t = (o 0 ,a 0 ,...,o t )is the interaction history available to the agent, ands t denotes the interaction step. Each action updates the application state, and repeating this process yields the interaction trajectory τ p = (s 0 ,s 1 ,...,s T−1 ). 4 Figure 3: Overview of PAIR. (i) Given a user and app profile, PAIR extracts user preferences and filters them according to their relevance to each application. (i) It then scores candidate application entities using exact matching for explicit entity preferences and semantic similarity for general preferences. (i) The highest-ranked entities are selected and instantiated in the corresponding application state, which is rendered as a user-conditioned interface. For each task, letg(q) = (g 1 ,...,g K )denote the ordered subgoals along a reference execution path. Each subgoalg k specifies an objective associated with a predefined UI context in which that objective can be completed, which we call its subgoal context. For userpand subgoalg k , letI p,k denote the set of step indices, ordered by time, that occur in the corresponding subgoal context. We define the context-specific interaction sequence asτ p,k = (s t ) t∈I p,k . Because an agent may revisit the same context multiple times,I p,k may contain non-consecutive indices, andτ p,k therefore aggregates interactions across repeated visits to the same subgoal context. In our setting, the visible interface state varies across users while the underlying application structure and interaction logic remain fixed. We distinguish user-conditioned (UC) contexts, where relevant visible content varies across users, from user-agnostic (UA) contexts, where the relevant interface content is shared across users, such as common navigation controls. Accordingly, each subgoal context receives a labelc p,k ∈ UC, UA: UC when it contains relevant visible user-conditioned elements, and UA otherwise. This labeling groups the corresponding subtrajectories by context type. 4.2ENVIRONMENT GENERATION PIPELINE PAIR constructs user-conditioned application states from interests and attributes derived from user profiles, as illustrated in Figure 3. The framework proceeds in three stages: (i) profile encoding, which extracts application-relevant preferences from the user profile and candidate entities, content items that can appear in the application state, from the application profile; (i) user-app entity similarity, which measures their correspondence through exact matching or semantic similarity; and (i) entity selection, which selects the highest-ranked entities to instantiate the application state. We first apply user interest filtering to retain preferences associated with userp’s active interests, followed by app relevance filtering to retain those relevant to applicationA. LetP p,A =u i denote the filtered preference values for userpand applicationA, and letE A =e j denote the candidate entity pool for application A. For each candidate application entity e j , we define its relevance as r(e j ) = ( 1,if e j directly matches any preference value u i ∈P p,A , max i cos(f(u i ),f(e j )), otherwise. (2) wheref(·)is a multilingual sentence encoder. We rank candidate entities byr(e j )and use the highest- ranked entities to populate application-specific collections. To instantiate executable environments, we build on MobileGym Wu et al. (2026), injecting the selected entities into user-specific runtime states according to each application’s native schema. Structured records are created in their respective schema. The resulting states are rendered by the original application UI. (See Appendix B.1) 5 4.3TASK DESIGN PAIR constructs task instances by pairing generated user-conditioned states with task templates. Each task template defines the required operation, target UI context, and success criterion, while the target and instruction are varied across conditions. For each user and template, we instantiate tasks along two axes: target type and instruction type. For more information, see Appendix B.2, B.3. Target type. Targets are either common or personal. A common target is shared across users; its instruction wording is fixed. This enables comparison of the same task across user-conditioned states. In contrast, a personal target is selected from the user-conditioned content generated by PAIR, testing whether the agent can resolve a target specific to that user. Instruction type. An explicit instruction refers to the task target by its visible label, whereas an implicit instruction refers to the same target using a contextual cue. For example, an explicit instruction requests, “Find the post ‘Looking for interesting flowers for a fashion project!’”, whereas an implicit instruction describes the same target as “a gardening post seeking flower ideas.” 4.4EVALUATION METRICS Task success rate (Task SR) measures end-to-end task completion asTask SR = E s /E, whereEis the total number of evaluated tasks and E s is the number of successfully completed tasks. All-success rate measures cross-user consistency for each template-condition pair, defined as a task template under a fixed target type and instruction type. The all-success rate is defined as AllSuccess = M a /M, whereMdenotes the number of template-condition pairs andM a denotes the number of pairs that achieve all-success; the agent succeeds for every user. Subgoal achievement rate (SAR) measures the proportion of these subgoals that are successfully completed. We define it asSAR = S s /S, whereSis the number of evaluated subgoals andS s is the number successfully completed. SAR enables trajectory-level localization of failures and comparison of subgoal performance between user-conditioned and user-agnostic contexts. 5REPAIR: PERSONALIZATION-AWARE INTERACTION REWARDS To investigate whether rewarding subgoal success in user-conditioned contexts can mitigate perfor- mance degradation in mobile GUI agents, we introduce RePAIR (Reinforcement learning with Personalization-Aware Interaction Rewards), which uses relative subgoal success across user- conditioned rollouts as the policy optimization signal; see Appendix C for more detail. Rollout Group Construction.For each combination of task template, target type, and instruction type, we pair training userspandqwith compatible subgoal structures. We collect two full-task rollouts from each user’s task instance, forming the groupG p,q = (τ 1 ,τ 2 ,τ 3 ,τ 4 ), whereτ 1 andτ 2 come from p and τ 3 and τ 4 from q. Each rollout starts from the corresponding initial state. Personalization-Aware Rewards. For rollout groupG p,q , letPdenote its shared set of subgoals in user-conditioned contexts; subgoals in user-agnostic contexts are excluded. For rolloutτ j and subgoal k ∈P , we assign a binary success reward: r jk = 1, if τ j satisfies the criteria for subgoal k, 0, otherwise. (3) We determine subgoal success from recorded trajectory evidence, including entry-screen observations, visited screens, clicks, and task-judge outcomes. Group-Relative Advantages.For each subgoal, we normalize its rewards across the four rollouts and then average the normalized rewards across user-conditioned subgoals to obtain one rollout-level advantage. Letμ k andσ k be the mean and population standard deviation of the rewards for subgoal k. For nonemptyP , we compute A j = 1 |P| X k∈P:σ k >0 r jk − μ k σ k .(4) 6 Table 2: Overall performance of base mobile GUI agents and RePAIR on PAIR. Task SR measures task success, all-success measures the rate of tasks completed successfully by all user profiles, and SAR measures subgoal achievement over reached contexts. Parentheses show RePAIR’s percentage- point gains over SFT. ModelParams. Task SR (%) All-success (%) User-agnostic SAR (%) User-conditioned SAR (%) UI-TARS-1.5 (Qin et al., 2025)7B48.522.579.765.4 Qwen3-VL-Instruct (Bai et al., 2025)30B-A3B47.619.279.367.2 Qwen3-VL-Instruct (Bai et al., 2025)8B47.522.574.667.6 AutoGLM-Phone (Xu et al., 2026b)9B53.721.785.369.9 MAI-UI (Zhou et al., 2025)8B53.425.886.271.0 GUI-Owl-1.5-Think (Xu et al., 2026a)8B49.516.777.469.1 Qwen3-VL-Instruct8B47.522.574.667.6 + SFT8B + 15.3M55.120.092.371.9 + SFT + RePAIR (ours)8B + 15.3M 64.5 (+9.4) 27.5 (+7.5)94.3 (+2.0)77.7 (+5.9) For example, supposePcontains two subgoals. Across the four rollouts, the first subgoal has reward vector(1, 1, 0, 0)and thus normalized reward vector(1, 1,−1,−1). The second has reward vector (1, 1, 1, 1)and thus contributes zero to every rollout. Averaging the two subgoal-level values for each rollout gives (A 1 ,A 2 ,A 3 ,A 4 ) = (0.5, 0.5,−0.5,−0.5). IfP is empty, we set A j = 0. Policy Optimization. We optimize an SFT-initialized policyπ θ with GRPO (Shao et al., 2024) using the rollout groups and advantages defined above, while keeping the SFT policy fixed as the referenceπ ref . Letπ old denote the rollout-generating policy. For rolloutτ j , letN j be the number of generated output tokens. At each token positiont∈1,...,N j ,w jt denotes the generated token and h jt the instruction and observation–action history available before generating it. We maximize J (θ) =E G p,q ∼π old 1 4 4 X j=1 1 N j N j X t=1 min π θ (w jt | h jt ) π old (w jt | h jt ) A j , clip π θ (w jt | h jt ) π old (w jt | h jt ) , 1− ε, 1 + ε A j − βD KL,jt . (5) Hereεsets the clipping range, whileβcontrols the KL penalty from the SFT reference. For each sampled token, the penalty is D KL,jt = π ref (w jt | h jt ) π θ (w jt | h jt ) − log π ref (w jt | h jt ) π θ (w jt | h jt ) − 1.(6) Each rollout is weighted equally, and its advantage applies to all response tokens. 6EXPERIMENTS We evaluate cross-user reliability at both the task and trajectory levels using PAIR. Specifically, we examine (1) how task success and (2) how subgoal achievement are affected across user-conditioned application states, (3) when incorrect-item selections occur relative to target visibility, and (4) whether RePAIR improves performance by rewarding subgoal success in user-conditioned contexts. 6.1EXPERIMENTAL SETUP Using user profiles from PersonaLens (Zhao et al., 2025b), we evaluate PAIR by instantiating ten profiles (p0 to p9) with PAIR as the test set for our main evaluation. We then instantiate ten additional profiles (p10 to p19) for the RePAIR study, using eight for training and two for validation. For each user profile, PAIR instantiates three task templates across ten applications under two target types and two instruction types, yielding3× 10× 2× 2 = 120instantiated tasks per persona. This results in 1,200 test tasks, 960 training tasks, and 240 validation instances. 7 p0p1p2p3p4p5p6p7p8p9 40 50 60 Task SR (%) Max 53.1 Min 43.8 Δ 9.3 p Figure 4: User-level task success rate(Task SR). Error bars show standard deviation across agents. GUI-Owl UI-TARS AutoGLM MAI-UI 30B 8B 0 25 50 75 100 SAR -8.3 -14.4 -15.4 -15.2 -12.0 -7.0 Qwen3-VL User-agnosticUser-conditioned (a) UA vs. UC contexts. GUI-Owl UI-TARS AutoGLM MAI-UI 30B 8B 0 25 50 75 100 SAR -0.9 -4.9 -13.6 -1.8 -1.7 -6.5 Qwen3-VL CommonPersonal (b) Common vs. personal targets. Figure 5: Subgoal achievement rate(SAR) across agents by (a) context type and (b) target type within user-conditioned contexts. 050100150 # of subgoal failures Common target Personal target 33 7 123 22 ×3.73 Before exposureAfter exposure Figure 6: Incorrect item selec- tions among subgoal failures by target exposure and target type. GUI-Owl AutoGLM MAI-UI 30B 8B UI-TARS SFT Pass 5 Pass 10 0 20 40 60 80 100 Task sets (%) 16.7 21.7 25.8 19.2 22.522.5 20.0 27.5 31.7 54.2 59.2 57.5 54.2 50.0 55.8 65.0 61.7 59.2 29.2 19.2 16.7 26.7 27.5 21.7 15.0 10.8 9.2 All-successPartial-successAll-failure Baseline / SFTRePAIR (a) Task SR composition. 1258 Persona pool size K 50 55 60 65 70 Task SR (%) 55.9 59.3 62.7 64.5 56.6 57.0 60.3 66.3 Training iteration:510 (b) Task SR by user pool size. Figure 7: RePAIR results. (a) Task success consistency across users. (b) Task SR by training user pool size and training iteration. Models.We evaluate Qwen3-VL-8B, GUI-Owl-1.5-8B-Think (Xu et al., 2026a), MAI-UI-8B (Zhou et al., 2025), AutoGLM-Phone-9B-Multilingual (Liu et al., 2024; Xu et al., 2026b), and UI-TARS- 1.5-7B (Qin et al., 2025), which fall within a deployment-relevant model scale for mobile settings, together with Qwen3-VL-30B-A3B (Bai et al., 2025) as a larger-scale comparison, on the same frozen test set. For the RePAIR, we first fine-tune Qwen3-VL-8B with LoRA (Hu et al., 2021) on explicit-instruction demonstrations from the training personas. We then apply RePAIR to tasks selected based on the preceding cross-persona analysis. More details are provided in Appendix D. 6.2RESULTS To assess overall task completion, cross-user consistency, and subgoal progress, we examine the task success rate (Task SR), the all-success rate, and the subgoal achievement rate (SAR) (see Table 2). Across the six agents, mean Task SR is 50.0% (SD 2.82). As we expected, task success decreases when the instruction is implicit rather than explicit (40.7% vs. 59.3%;p < .001). Personal targets also show lower task success than common targets (46.6% vs. 53.5%). Neither this difference (p = .0642) nor the instruction–target interaction (p = .283) reached statistical significance. Task success varies substantially across personas.We therefore examine how task outcomes vary across personas. Across the evaluated agents, as illustrated in Figure 4, persona-level Task SR ranges from 43.8% to 53.1%, with a gap of up to 9.31 p between the best- and worst-performing personas. This cross-persona variation occurs for both common targets with identical instructions and targets across users (6.94 p) and personal targets drawn from each user’s own records (13.6 p). Thus, the same task can produce different outcomes depending on the persona-conditioned interface in which it is executed. Goal achievement drops in user-conditioned contexts, with difficulty for personal targets. Across all six complete agents, the SAR is consistently lower in user-conditioned than in user- agnostic contexts, with an average drop of 12.1 (SD 3.64) p (Figure 5(a)). Within user-conditioned contexts, personal targets yield lower SAR than common targets across all six agents (Figure 5(b)). These results suggest task progress is more difficult in interface regions containing user-conditioned content, particularly when the target is resolved from semantically related items. Beyond the aggregate SAR gap, the trajectory-level analysis reveals localized peaks in failure at specific subgoals (see Figure 11 in Appendix E). High failure rates frequently occur at subgoals 8 conducted in user-conditioned contexts. This pattern appears across multiple applications and task types, suggesting that the difficulty is not confined to a single interface or operation. From a trajectory- level perspective, user-conditioned contexts can therefore act as major bottlenecks, disrupting task progress even when other stages are completed successfully. Target contextGT resultAgent resultTarget contextGT resultAgent result Agent actionGT action (a) Before target exposure. Target contextGT resultAgent resultTarget contextGT resultAgent result Agent actionGT action (b) After target exposure. Figure 8: Personal-target failures in user-conditioned contexts: the agent selects an incorrect item (a) before target exposure and (b) after the tar- get becomes visible. Red: agent ac- tion; green: ground truth. Personal targets increase competing-item selection errors. To better understand failures in user-conditioned contexts, we further analyze the error patterns that arise when agents resolve targets. For both common and personal targets, incorrect-item selection failures occur more frequently before target exposure, when further navigation is required, than after exposure, with personal targets exhibiting more failures overall (Figure 6). A closer inspection of these failures reveals two recurring patterns depending on whether the intended target has been exposed. Figure 8 illustrates representative personal target failures under explicit instructions. In the first case, the agent opens a similarly titled user-specific post before the intended target appears, rather than continuing to swipe. In the second case, the intended video is already visible, yet the agent selects a different video from the same personalized list. These exam- ples indicate that agents may commit to a competing item be- fore reaching the target, or fail to distinguish it from surround- ing items even after it becomes visible. Notably, both failures occur despite the target being named explicitly, suggesting that exact target specification alone does not eliminate interference from competing user-specific context (Appendix F). RePAIR improves cross-user reliability on unseen personas. We first adapt the base model to the benchmark tasks with supervised fine-tuning. This substantially improves SAR in user-agnostic contexts (+17.7 p) but yields smaller gains in user-conditioned contexts (+4.24 p) and no improvement in all-success, leaving difficulties in user-conditioned contexts and cross-user inconsistency unresolved. We then apply Re- PAIR to examine whether learning from cross-user differences improves reliability in user-conditioned contexts. RePAIR improves user-conditioned SAR (+5.87 p), the all-success rate (+7.50 p), and overall Task SR (+9.42 p) on unseen users. The gain in user-conditioned SAR indicates greater robustness to user-conditioned interface variation, while the increase in the all-success rate reflects more consistent task completion across users (see Figure 7(a)). The concurrent improvement in task success further shows that these gains translate to better end- to-end performance. To this end, we vary the training user pool sizen p and the number of training iterations. Increasingn p from 1 to 8 improves overall Task SR after both 5 (+8.6 p) and 10 (+9.7 p) iterations (Figure 7(b)). However, extending training from 5 to 10 iterations decreases Task SR forn p = 2(−2.3p) andn p = 5(−2.4p), while improving it forn p = 8(+1.8 p). These results suggest that the benefit of additional RePAIR training depends on user-pool diversity, with repeated comparisons improving unseen-user performance only when training spans a broad set of users. 7CONCLUSION AND FUTURE WORK We systematically study cross-user reliability in mobile GUI agents: whether the same request is completed consistently across user-conditioned application states. We introduce PAIR, which constructs such states for controlled evaluation, and RePAIR, which trains from cross-user differences in user-conditioned subgoal outcomes. Experiments reveal cross-user performance variation, lower subgoal achievement in user-conditioned contexts, especially for personal targets, and incorrect selections before and after target exposure; RePAIR improves performance and consistency on unseen personas, providing initial evidence that cross-user differences can serve as a useful training signal. 9 These findings suggest several directions for future work. 1) Develop training methods that leverage cross-user differences in subgoal outcomes to improve reliability in user-conditioned contexts. 2) Strengthen target grounding through uncertainty-aware search, target verification, and recovery from incorrect-item selections before and after target exposure. 3) Determine the optimal balance between persona pool size and the number of training iterations to improve generalization to unseen users. REFERENCES Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. Gongwei Chen, Xurui Zhou, Rui Shao, Yibo Lyu, Kaiwen Zhou, Shuai Wang, Wentao Li, Yinchuan Li, Zhongang Qi, and Liqiang Nie. Less is more: Empowering gui agent with context-aware simplification. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), p. 5901–5911. IEEE, 2025. Sen Chen, Tong Zhao, Yi Bin, Fei Ma, Wenqi Shao, and Zheng Wang. D-gara: A dynamic benchmarking framework for gui agent robustness in real-world anomalies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, p. 17419–17426, 2026a. Tongbo Chen, Zhengxi Lu, Zhan Xu, Guocheng Shao, Shaohan Zhao, Fei Tang, Yong Du, Kaitao Song, Yizhou Liu, Yuchen Yan, et al. Knowu-bench: Towards interactive, proactive, and personal- ized mobile agent evaluation. arXiv preprint arXiv:2604.08455, 2026b. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9313–9332, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.acl-long.505. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114, 2023. Motahhare Eslami, Aimee Rickman, Kristen Vaccaro, Amirhossein Aleyasen, Andy Vuong, Karrie Karahalios, Kevin Hamilton, and Christian Sandvig. ” i always assumed that i wasn’t really that close to [her]” reasoning about invisible algorithms in news feeds. In Proceedings of the 33rd annual ACM conference on human factors in computing systems, p. 153–162, 2015. Leah Findlater and Joanna McGrenere. A comparison of static, adaptive, and adaptable menus. In Proceedings of the SIGCHI conference on Human factors in computing systems, p. 89–96, 2004. Zhiqi Ge, Juncheng Li, Xinglei Pang, Minghe Gao, Kaihang Pan, Wang Lin, Hao Fei, Wenqiao Zhang, Siliang Tang, and Yueting Zhuang. Iris: Breaking gui complexity with adaptive focus and self-refining. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), p. 1–10. IEEE, 2025. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. Lawrence Keunho Jang, Mareks Woodside, Geronimo Carom, Andrew Keunwoo Jang, Jing Yu Koh, and Ruslan Salakhutdinov. iosworld: A benchmark for personally intelligent phone agents. arXiv preprint arXiv:2606.09764, 2026. Serin Kim, Sangam Lee, and Dongha Lee. Persona2web: Benchmarking personalized web agents for contextual reasoning with user history. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=qvvD9hgHoX. 10 Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, et al. Autoglm: Autonomous foundation agents for guis. arXiv preprint arXiv:2411.00820, 2024. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URLhttps://openreview.net/forum?id= Bkg6RiCqY7. Yibo Lyu, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Personalalign: Hierarchical implicit intent alignment for personalized gui agent with long-term user-centric records. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 36074–36089, 2026. Hongyi Nie, Xunyuan Liu, Yudong Bai, Yaqing Wang, Yang Liu, Quanming Yao, and Zhen Wang. Pspa-bench: A personalized benchmark for smartphone gui agent. arXiv preprint arXiv:2603.29318, 2026. OpenAI. gpt-oss-120b & gpt-oss-20b model card, 2025. URLhttps://arxiv.org/abs/ 2508.10925. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URLhttp://arxiv.org/abs/1908. 10084. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Shuoxin Wang, Chang Liu, Gowen Loo, Lifan Zheng, Kaiwen Wei, Xinyi Zeng, Jingyuan Zhang, and Yu Tian. Me-Agent: A personalized mobile agent with two-level user habit learning for enhanced interaction, 2026a. URL https://arxiv.org/abs/2601.20162. Yangyue Wang, Harshvardhan Sikka, Yash Mathur, Tony Zhou, Jinu Nyachhyon, and Pranav Gu- ruprasad. Gui-perturbed: Domain randomization reveals systematic brittleness in gui grounding models. arXiv preprint arXiv:2604.14262, 2026b. Dingbang Wu, Rui Hao, Haiyang Wang, Shuzhe Wu, Han Xiao, Zhenghong Li, Bojiang Zhou, Zheng Ju, Zichen Liu, Lue Fan, and Zhaoxiang Zhang. MobileGym: A verifiable and highly parallel simulation platform for mobile GUI agent research. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2026. URL https://arxiv.org/abs/2605.26114. To appear. Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, Si Qin, Lars Liden, Qingwei Lin, Huan Zhang, Tong Zhang, Jianbing Zhang, Dongmei Zhang, and Jianfeng Gao. GUI-actor: Coordinate-free visual grounding for GUI agents. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=5fSkinHw7w. Haiyang Xu, Xi Zhang, Haowei Liu, Junyang Wang, Zhaozai Zhu, Shengjie Zhou, Xuhao Hu, Feiyu Gao, Junjie Cao, Zihua Wang, et al. Mobile-agent-v3. 5: Multi-platform fundamental gui agents. arXiv preprint arXiv:2602.16855, 2026a. 11 Yifan Xu, Xiao Liu, Xinghan Liu, Jiaqi Fu, Jiayu Huang, Hanchen Zhang, Bohao Jing, Shudan Zhang, Yuting Wang, Yuxiao Dong, et al. Mobilerl: Online agentic reinforcement learning for mobile gui agents. In International Conference on Learning Representations, volume 2026, p. 35282–35315, 2026b. Jingqi Yang, Zhilong Song, Jiawei Chen, Mingli Song, Sheng Zhou, Linjun Sun, Xiaogang Ouyang, Chun Chen, and Can Wang. Gui-robust: A comprehensive dataset for testing gui agent robustness in real-world anomalies. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, p. 2842–2853, 2026a. Qinglong Yang, Haoming Li, Haotian Zhao, Xiaokai Yan, Jingtao Ding, Fengli Xu, and Yong Li. Fingertip 20k: A benchmark for proactive and personalized mobile LLM agents. In The Fourteenth International Conference on Learning Representations, 2026b. URL https://openreview. net/forum?id=n3iFV0gLMc. Wenkui Yang, Chao Jin, Haisu Zhu, Weilin Luo, Derek Yuen, Kun Shao, Junxian Duan, Huaibo Huang, Jie Cao, and Ran He. Are gui agents focused enough? automated distraction via semantic- level ui element injection. In European Conference on Computer Vision, p. 115–132. Springer, 2026c. Kuai Yu, Naicheng Yu, Han Wang, Rui Yang, and Huan Zhang. How do visual attributes influence web agents? a comprehensive evaluation of user interface design factors. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (eds.), Findings of the Association for Computational Linguistics: ACL 2026, p. 34520–34536, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-395-1. doi: 10.18653/v1/2026. findings-acl.1723. URLhttps://aclanthology.org/2026.findings-acl.1723/. Yuyang Zhao, Wentao Shi, Fuli Feng, and Xiangnan He. AppAgent-Pro: A Proactive GUI Agent System for Multidomain Information Integration and User Assistance, August 2025a. URL http://arxiv.org/abs/2508.18689. arXiv:2508.18689 [cs]. Zheng Zhao, Clara Vania, Subhradeep Kayal, Naila Khan, Shay B Cohen, and Emine Yilmaz. Personalens: A benchmark for personalization evaluation in conversational ai assistants. In Findings of the Association for Computational Linguistics: ACL 2025, p. 18023–18055, 2025b. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. GPT-4V(ision) is a generalist web agent, if grounded. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research. PMLR, 2024. URLhttps:// proceedings.mlr.press/v235/zheng24e.html. Hanzhang Zhou, Xu Zhang, Panrong Tong, Jianan Zhang, Liangyu Chen, Quyu Kong, Chenglin Cai, Chen Liu, Yue Wang, Jingren Zhou, et al. Mai-ui technical report: Real-world centric foundation gui agents. arXiv preprint arXiv:2512.22047, 2025. 12 AFINGERTIP-20K ANALYSIS Data.All six models are evaluated on the same 103,199 steps from the FingerTip-20K (Yang et al., 2026b) training split: 12,016 episodes, 83 users and 102 app labels. We retain supported actions with available screenshots from apps used by at least five users. Each prediction uses the recorded screenshot, instruction and human action history. Later screenshots follow the human demonstration. Screen labels and segments. A text label is user-specific if its normalized form appears for only one user within an app. A screen is user-conditioned (UC) if it contains such a label. Numeric and time labels are included; icons and unknown labels are excluded. Labels are not filtered by pixel visibility. Consecutive screens form a segment until the Jaccard similarity of labels outside scrollable regions falls below 0.475; two empty sets have similarity one. This gives 60,383 segments: 51,761 UC and 8,622 without UC elements. Scoring. An action must have the correct type. For clicks and long clicks, the pixel distance from the reference position, divided by the image diagonal and rounded to four decimals, must be at most 0.14. Scroll direction must match; text is compared after removing whitespace and lowercasing. Other supported actions are scored by type, and parsing failures are incorrect. A segment succeeds only if all its actions are correct. All 60,383 segments are included. Target exposure. The exposure analysis uses 29,640 segments with a labeled exit target. A clickable element matches the target label if character-bigram Jaccard similarity is at least 0.85, target-bigram coverage is at least 0.8, the shared prefix has at least six characters, or the matching substring has at least eight. This text-based measure does not establish visual attention. Differences between screen groups are descriptive; tasks, apps and segment lengths are not controlled. 13 BPAIR IMPLEMENTATION, SCREENS AND DATASET STATISTICS B.1USER-CONDITIONED APPLICATION STATE GENERATION PAIRconvertsPersonaLens(Zhaoetal.,2025b)interestsintoapp-specific queries.Content is ranked by name matches (score 1) or cosine similarity using paraphrase-multilingual-MiniLM-L12-v2(Reimers & Gurevych, 2019) with mean pooling, L2 normalization and a 96-token limit. Duplicate IDs/names and reserved Common items are removed before selection. Common items are then added, and lists are shuffled using seed-0 hashes. Most lists contain 20 items. App-specific generators also create transactions, chats and meetings. The records keep their IDs and media links and are rendered by MobileGym (Wu et al., 2026). Alipay · p0Bilibili · p0Maps · p0RedBook · p0Reddit · p0 Alipay · p1Bilibili · p1Maps · p1RedBook · p1Reddit · p1 Figure 9: Generated screens for p0 (top) and p1 (bottom). Each column shows the same app and screen location for both personas. 14 Spotify · p0Tencent Meeting · p0WeChat · p0WeChat Reading · p0X · p0 Spotify · p1Tencent Meeting · p1WeChat · p1WeChat Reading · p1X · p1 Figure 10: Persona comparison for the remaining five apps: p0 (top), p1 (bottom). 15 B.2TASK GENERATION Common targets are shared across personas; Personal targets come from each persona’s data. Explicit and Implicit instructions refer to the same target. All four conditions use the same initial state for a template/persona pair, reset before each episode.The instruction writer (openai/gpt-oss-120b(OpenAI, 2025)) and reviewer use the candidate records. Agents receive only screenshots, instructions and action history. All 1,200 test instances are evaluated against their designated targets, including cases with unre- solved review flags. Automatic review does not guarantee a unique interpretation of every Implicit instruction. Table 3: Apps and target screens. RedBook is also called Xiaohongshu. ApplicationTarget UI contexts AlipayBilling history; message list BilibiliSubscriptions; home recommendations; liked videos MapsRecent searches RedBookSaved notes; liked notes; discover feed RedditHome feed SpotifyLiked songs; recently played tracks; home playlists Tencent MeetingMeeting history; scheduled meetings WeChatChat list; Moments WeChat ReadingHome reading list; Add Favorite Book; recommendations XFor You feed Table 4: All 30 task templates. Each operation applies to the designated Common/Personal target under Explicit/Implicit instructions. ApplicationTask IDTargetTask Template Alipay G-ALI-01TransactionOpen the transaction in billing history. Alipay G-ALI-02Conversation Open the conversation in the message list. Alipay G-ALI-03Conversation Open the conversation; reply exactly “Understood”. Bilibili G-BIL-01VideoOpen the subscription video and like it. Bilibili G-BIL-02VideoOpen the home recommendation video. Bilibili G-BIL-03VideoOpen the video in Liked Videos. Maps G-MAP-01PlaceOpen place details from recent searches. Maps G-MAP-02PlaceGet directions to the place in recent searches. Maps G-MAP-03PlaceOpen place details from recent searches. RedBook G-RBK-01NoteOpen the saved note. RedBook G-RBK-02NoteOpen the liked note. RedBook G-RBK-03NoteOpen the discover-feed note and save it. Reddit G-RDT-01PostOpen the home-feed post. Reddit G-RDT-02PostOpen the post; comment exactly “Thank you”. Reddit G-RDT-03PostOpen the home-feed post. Spotify G-SPT-01SongAdd the liked song to a new “Rainy Night” playlist. Spotify G-SPT-02SongPlay the recently played song. Spotify G-SPT-03PlaylistOpen the home-screen playlist. Tencent Meeting G-MTG-01MeetingOpen the meeting in history. Tencent Meeting G-MTG-02MeetingOpen the scheduled meeting. Tencent Meeting G-MTG-03MeetingOpen the meeting in history. WeChat G-WCT-01ChatOpen the chat. WeChat G-WCT-02ChatOpen the chat; reply exactly “Understood”. WeChat G-WCT-03AuthorOpen the author’s profile from Moments. 16 ApplicationTask IDTargetTask Template WeChat Reading G-WRE-01BookOpen the book in the home reading list. WeChat Reading G-WRE-02BookAdd the book using Add Favorite Book. WeChat Reading G-WRE-03BookOpen the recommended book. X G-X-01PostOpen the For You post. X G-X-02PostOpen the For You post. X G-X-03PostOpen the For You post. B.3REFERENCE SUBGOALS BY TASK Table 5 expands the task templates in Table 4 into their ordered reference subgoals. Each subgoal spec- ifies an objective associated with a UI context and may involve multiple actions, including scrolling, typing, and clicking. The sequence is shared across Common/Personal targets and Explicit/Implicit instructions; the designated target is determined by the task instance. These descriptions specify the reference objectives; the RePAIR training reward is defined separately in Appendix C. Table 5: The 67 reference subgoal positions across all 30 task templates included in our analysis. The optional library-to-history navigation step in G-SPT-02 is excluded, and the remaining positions are numbered consecutively within each task. Task IDOrdered subgoals G-ALI-01(1) Open My from the home screen; (2) Open billing history; (3) Open the designated transaction details. G-ALI-02(1) Open the message list from the home screen; (2) Open the designated conversation. G-ALI-03(1) Open the message list from the home screen; (2) Open the designated conversation; (3) Enter and send “Understood”. G-BIL-01(1) Open Following from the recommendation home screen; (2) Switch to the video tab; (3) Open the designated video; (4) Like the designated video. G-BIL-02(1) Open the designated video from the home recommendation feed. G-BIL-03(1) Open Me from the recommendation home screen; (2) Open the personal space; (3) Open the liked-video list; (4) Open the designated video. G-MAP-01(1) Open search from the map home screen; (2) Open the designated place details from recent searches. G-MAP-02(1) Open search from the map home screen; (2) Select the designated place from recent searches; (3) Open directions to the designated place. G-MAP-03(1) Open search from the map home screen; (2) Open the designated place details from recent searches. G-RBK-01(1) Open Me from the discovery home screen; (2) Open the saved-notes tab; (3) Open the designated note. G-RBK-02(1) Open Me from the discovery home screen; (2) Open the liked-notes tab; (3) Open the designated note. G-RBK-03(1) Open the designated note from the discovery feed; (2) Save the designated note. G-RDT-01(1) Open the designated post from the home feed. G-RDT-02 (1) Open the designated post from the home feed; (2) Enter and submit the comment “Thank you”. G-RDT-03(1) Open the designated post from the home feed. G-SPT-01(1) Open the library from the home screen; (2) Open Liked Songs; (3) Open the menu for the designated track; (4) Select Add to playlist; (5) Select the option to create a new playlist; (6) Create the playlist named “Rainy Night” and complete the track addition. G-SPT-02(1) Open the library from the home screen; (2) Expand the recent-play group; (3) Play the designated track within the expanded group. G-SPT-03(1) Open the designated playlist from the home screen. Continued on next page 17 Task IDOrdered subgoals (continued) G-MTG-01(1) Open meeting history from the home screen; (2) Open the designated meeting details. G-MTG-02(1) Open the designated scheduled meeting details from the home screen. G-MTG-03(1) Open meeting history from the home screen; (2) Open the designated meeting details. G-WCT-01(1) Open the designated conversation from the chat list. G-WCT-02(1) Open the designated conversation from the chat list; (2) Enter and send “Understood”. G-WCT-03(1) Open Discover from the chat list; (2) Open Moments; (3) Open the designated author’s profile. G-WRE-01(1) Open the designated book from the home reading list. G-WRE-02(1) Open Me from the reading home screen; (2) Open the book-list screen; (3) Open the list editor; (4) Open Add Favorite Book; (5) Select the designated book and confirm its addition. G-WRE-03(1) Open the designated book from the home recommendations. G-X-01(1) Open the designated post from the For You feed. G-X-02(1) Open the designated post from the For You feed. G-X-03(1) Open the designated post from the For You feed. CTRAINING REWARDS AND ALGORITHM Reward calculation.We compute a binary reward for each subgoal from the recorded trajectory. A subgoal first requires evidence that its entry screen was observed before an action; appearing only in the final screen is insufficient. For a nonterminal subgoal, reward one additionally requires either a recorded route corresponding to a later subgoal in the reference path or a click on an expected item or control. For the terminal subgoal, defined as the last subgoal in the reference path, reward one instead requires the task judge to report success. Reward labels and matching criteria are used only for scoring and are not provided to the agent. For rolloutiand subgoalk, letE ik indicate the required entry observation,L ik a matching later- reference route in any observation or the final screen, andC ik an expected click anywhere in the log. Let J i be the binary task-judge outcome. The reward is r ik =        0, E ik = 0, J i , E ik = 1 and k is terminal, 1, E ik = 1, k nonterminal, and (L ik ∨ C ik ), 0,otherwise. (7) Matching rules. Route matching compares URL paths after removing query parameters; app identity and screen state are not used. Expected items and controls are specified by the task template. A target-item click matches when its recorded IDs include the designated target ID or identity. Other controls match by trigger ID or by a substring match in either direction between whitespace- normalized labels. The scorer searches the entire log without enforcing visit order. Here, “later” refers to the order of subgoals in the reference path, not the order of observations in the rollout. Policy update. Each iteration collects 124 rollouts with a fixed policy and performs one LoRA update per four-rollout group. Rollout-policy and SFT-reference token log probabilities are cached as specified in the pseudocode below. 18 Table 6: RePAIR training pseudocode (K = 8;N ∈5, 10training iterations).θdenotes trainable LoRA parameters;h it is the recorded token prefix andy it a response token.T i counts response tokens over the entire trajectory, across all actions. 1 θ ← SFT(θ base ,D Explicit ) for one epoch; freeze θ ref ← θ. 2 Build fixed persona pools and a repeating five-iteration pair schedule. 3 for training iteration j = 1,...,N do 4Fix rollout policy θ old ← θ for this iteration. 5for each of the 31 task–condition pairs q do 6Select the scheduled pair; collect two full rollouts per slot with vLLM. 7Cache the four trajectories and their rollout-policy token log probabilities. 8Cache SFT-reference token log probabilities; reset AdamW and the 31-update cosine schedule. 9for each cached group in fixed task order do 10Compute subgoal rewards r ik using Section C. 11a ik ← (r ik − μ k )/ p σ 2 k + 10 −8 , using the mean and population variance over i = 1,..., 4. 12A i ←|U q | −1 P k∈U q a ik ;U q is the UC-subgoal set (use zero if empty). 13Recompute current-policy token log probabilities; reuse cached SFT-reference scores. 14ρ it ← π θ (y it | h it )/π old (y it | h it ); d it ← logπ ref (y it | h it )− logπ θ (y it | h it ). 15ℓ it ←− minρ it A i , clip(ρ it , 0.8, 1.2)A i + 0.01(e d it − d it − 1). 16 Take one AdamW step on 1 4 P 4 i=1 1 T i P T i t=1 ℓ it ; clip gradient norm to 1.0 and advance the scheduler. 17Save the iteration checkpoint. 18 return the iteration-N adapter (124N rollouts; 31N optimizer updates). 19 DEXPERIMENTAL AND TRAINING SETUP Evaluation. We evaluate Qwen3-VL-8B, Qwen3-VL-30B-A3B, GUI-Owl-1.5-8B-Think, MAI- UI-8B, AutoGLM-Phone-9B-Multilingual and UI-TARS-1.5-7B on the same 1,200 test instances. Decoding uses temperature 0, top-p 1 and seed 0, with each model’s action format. The action limit is 20 for 840 instances and 30 for 360. Output limits are 1,024 tokens for Qwen/SFT/RePAIR, 2,048 for MAI-UI and 3,000 for AutoGLM; GUI-Owl/UI-TARS limits are not specified in the saved request configs. Task SR uses all instances; all-success uses 120 ten-persona groups; SAR uses reached subgoals at the 67 reference positions. SFT.Qwen3-VL-8B-Instruct is fine-tuned on successful Explicit-instruction oracle demonstrations across all 30 task templates. Of the 480 Explicit task instances from the eight training personas, three failed oracle executions are excluded, leaving 477 demonstrations containing 1,876 actions. Validation uses 119 successful demonstrations containing 469 actions, after excluding one failed execution from 120 instances. All four excluded executions are from X and failed to reach the designated target route. Implicit instructions, designated targets, and initial states are available, but separate Implicit oracle demonstrations were not collected. SFT therefore uses only Explicit demonstrations. LoRA (Hu et al., 2021) updates the language model’s q/k/v/o attention projections; the vision encoder is frozen. One epoch gives 235 updates using assistant-response loss. Training uses bf16, SDPA and gradient checkpointing on an RTX 6000 Ada (PyTorch 2.13.0+cu130, Transformers 4.57.6, PEFT 0.18.1). Table 7: Core optimization settings. RePAIR inherits the SFT LoRA adapter. SettingSFTRePAIR LoRA rank / alpha16 / 3216 / 32 Dropout0.05Disabled during updates OptimizerAdamW (Loshchilov & Hutter, 2019)AdamW Learning rate10 −4 10 −5 Weight decay / norm clipping0 / 1.00 / 1.0 Scheduler / warmupCosine / 3%Iteration-wise cosine / 3% Update unit8 action samples4 full-task rollouts Policy ratio clip—0.2 Reference KL coefficient—0.01 (fixed SFT) RePAIR task selection.A task specification is a combination of task template, target type (Common or Personal), and instruction type (Explicit or Implicit). The 30 templates yield 120 specifications, each evaluated across ten personas, p0 to p9. For each model, a specification is classified as mixed when its final-goal outcomes include both successes and failures across these personas. We select specifications classified as mixed by all five baselines in the frozen September 16, 2026 analysis: Qwen3-VL-8B, Qwen3-VL-30B-A3B, GUI-Owl, MAI-UI, and AutoGLM. This yields 31 specifications spanning 18 templates, comprising 12 Common and 19 Personal specifications, or 8 Explicit and 23 Implicit specifications. Selection uses the historical AutoGLM results preceding its greedy rerun and excludes UI-TARS. The selected list is held fixed across subsequent training experiments rather than recomputed from updated baseline results. RePAIR training. The 31 selected specifications are instantiated for the eight training personas p10 to p17, yielding 248 distinct task instances. Each training iteration collects four full-task rollouts per specification, totaling 124 rollouts and 31 optimizer updates. Both Explicit and Implicit tasks are included: the policy generates trajectories that are scored using recorded routes, actions, and the task judge, without requiring oracle demonstrations. Seeded selection forms fixed pools ofK ∈1, 2, 5, 8personas. K1 uses one persona, K2 a fixed pair, and K5/K8 cycle pairs; four Reddit task–condition pairs allow same-persona sampling to keep subgoal labels consistent. Group compatibility requires matching ordered subgoal IDs and UA/UC 20 labels across the full reference path. For each scheduled pair, we collect two rollouts per persona slot, resetting the corresponding task instance to its stored initial state before every rollout. Only UC-subgoal rewards are used to compute the trajectory advantage. Rollouts use temperature 0.8, top-p 1, top-k−1, min-p 0 and a 1,024-token output limit. Five/ten iterations give 620/1,240 rollouts and 155/310 updates. Iterations 6 to 10 repeat the persona schedule with new rollouts, continuing from iteration 5. The main result uses K8 at iteration 5; validation is not used to select it. SFT uses seed 0; RePAIR uses seed j− 1 at iteration j with derived update seeds. Selection and review scope.Training rollouts use personas p10 to p17, disjoint from the evaluation personas p0 to p9. However, task specifications were selected using earlier baseline outcomes on p0 to p9. Thus, evaluation personas are held out from policy optimization, but their baseline results informed training-task selection. We interpret the RePAIR study as an exploratory intervention informed by the preceding analysis. The p10 to p19 inputs have not completed semantic and browser review. Successful oracle execution and subsequent model rollouts do not establish completion of that review. vLLM. We use vLLM (Kwon et al., 2023) for inference, with server seed 0 and GPU memory utilization 0.9. Qwen baselines use an 8,192-token batch budget and image bounds of 3,136 to 12,845,056 pixels; AutoGLM uses at most 5,000,000 pixels. K8 rollout logs for iterations 1 to 5 record vLLM 0.19.0, bf16, prefix caching and chunked prefill. These settings are not verified for all earlier FingerTip or recovery runs. Table 8: vLLM settings from evaluation configs and training logs. TP: tensor parallelism; Context: token limit; Sequences/Images: per-server and per-request limits, respectively. Model / useTPContextSequencesImages Qwen3-VL-8B baseline116,38481 Qwen3-VL-30B-A3B216,38481 GUI-Owl132,76885 MAI-UI132,76884 UI-TARS132,76886 AutoGLM (PAIR / FingerTip)125,48016 / 3210 SFT/RePAIR evaluation132,76882 RePAIR rollout collection132,76882 ESUBGOAL FAILURE PATTERNS ACROSS TASK TRAJECTORIES To examine where task progress breaks down, we report subgoal-level failure rates along the reference trajectory for all 30 task templates. For each task, subgoals are ordered according to their position in the reference execution path. We compute the failure rate at each subgoal as100− SARafter the agent reaches the corresponding subgoal context. Thus, failure rates at later positions are conditional on the agent having progressed far enough to reach those contexts. Figure 11 shows that failures are not distributed uniformly across task trajectories. Instead, many tasks exhibit localized failure peaks at particular subgoals. Several pronounced failures occur in user- conditioned contexts, including target-related stages for Bilibili, Maps, Tencent Meeting, WeChat, and WeChat Reading. For example, the failure rate reaches 100% at the third subgoal of the Bilibili subscribed anime task, 90% at the first subgoal of scheduled meeting, and 94% at the first subgoal of liked book. These patterns indicate that particular stages involving user-specific application content can form substantial bottlenecks within otherwise multi-step tasks. At the same time, high failure rates are not exclusive to user-conditioned contexts. Some user- agnostic stages also show large failure rates, such as the second subgoal of the Bilibili liked video and subscribed anime tasks. Thus, the trajectory-level analysis does not imply that all failures originate from personalization. Rather, it distinguishes general GUI interaction difficulties from localized failures that arise at user-conditioned stages. 21 123456 Subgoal bill refund chat service chat home video liked video subscribed anime history category directions history place discover note liked note saved note hobby post comment post home playlist liked song recent track host meeting meeting scheduled meeting open chat chat author profile liked book book recommended interest post news post topic post 25040 END 023 END 813 09 END 10 END 095920 END 128297 –0 END 380 END 065 00 END 458 END 28 50 END 222561 END 17948 END 36 END 19 105 END 30 END 59 END 15150035 14911 END 155 END 1637 END 91 END 3 END 5 3014 END 45274 END 952725033 END 33 END 81 END 29 END 27 END 32 END Alipay Bilibili Map RedBook Reddit Spotify Tencent WeChat WeChat Read X 050100 user-agnostic 050100 user-conditioned Failure rate (%) Figure 11: Subgoal failure rates across task trajectories. Each cell reports the failure rate (100−SAR, %) after entering the corresponding subgoal context, pooled across six complete agents. Blue and red denote user-agnostic and user-conditioned contexts, respectively; darker cells indicate higher failure rates. A dash denotes no entry, and END marks task termination. Later-position rates are conditional on reaching the corresponding context. Overall, this analysis complements the aggregate SAR comparison by showing where the observed performance differences emerge within individual task trajectories. The results suggest that difficulty is often concentrated at specific interaction stages rather than being uniformly distributed throughout an entire task. 22 FCOMPLETE PAIR FAILURE TRAJECTORIES The two failures from the main paper use Personal targets and Explicit instructions. Each row shows all three recorded steps, including termination. Screenshots precede the labeled actions; no post-termination screenshot is available. Reddit · 1 · SWIPEReddit · 2 · CLICKReddit · 3 · COMPLETE Bilibili · 1 · SWIPEBilibili · 2 · CLICKBilibili · 3 · COMPLETE Qwen3-VL-30B Open the post titled “Greta Gerwig's 'Narnia' Wraps Filming”. Target: post_1qs7klq Opened: post_1regcil Wrong post before target exposure. Task failed. GUI-Owl Open the designated video in the home recommendation feed. Target: BVmg00008393 Opened: BVmg00000356 Wrong video after target exposure. Task failed. Figure 12: Reddit (top, Qwen3-VL-30B) selects the wrong post before target exposure. Bilibili (bottom, GUI-Owl) selects the wrong video after target exposure. Both fail the task’s success check; target and selected IDs are shown beside the trajectories. The selected item is confirmed by the next screen’s route. These examples illustrate the two error types, not their overall frequency. 23

Instant research discovery