Paper deep dive
Latent World Action Model for GUI Agents
Seungjun Yu, Hojun Choi, Seojeong Park, Jaeyo Shin, Hyunjung Shim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 10/14/2026, 9:42:56 AM
Summary
The paper introduces a Latent World Action Model for GUI agents to address the high computational cost and identity shortcut problems associated with pixel-space world models. By shifting prediction targets to latent space using a Large Multimodal Model (LMM), the method jointly optimizes action selection and predictive transition modeling via an action-aware contrastive objective. This approach significantly improves success rates and reduces inference latency compared to existing pixel-space methods.
Entities (7)
Relation Signals (5)
Latent World Action Model → developedby → KAIST
confidence 99% · Korea Advanced Institute of Science and Technology (KAIST) {seungjunyu,hchoi256,seojeong.park,jaeyoshin,kateshim}@kaist.ac.kr
Latent World Action Model → uses → Large Multimodal Model (LMM)
confidence 96% · we leverage a large multimodal model as the latent world model
Latent World Action Model → outperforms → Pixel-space world models
confidence 95% · our method achieves up to 31.9% and 40.6% relative gains in success rate over pixel-space world models
Latent World Action Model → employs → Action-aware contrastive objective
confidence 94% · we further introduce an action-aware contrastive objective that encourages the transition representations to capture discriminative action-induced changes.
Latent World Action Model → addresses → Identity Shortcut
confidence 92% · To mitigate the identity shortcut, we further introduce an action-aware contrastive objective
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Pixel-space world models have recently emerged as a promising approach for GUI agents, predicting future screens prior to execution and subsequently re-perceiving them for action selection. However, this two-stage paradigm faces two key challenges in GUI environments: (L1) the high computational cost of pixel reconstruction and (L2) the need to reconstruct large and diverse appearance-level details beyond what is required for action selection. In practice, this screen-generation objective can induce an identity shortcut problem, where future-state prediction largely preserves the current state rather than capturing action-induced changes. In this work, we reformulate GUI world modeling by shifting the prediction target from full-screen reconstruction to action-induced future information in latent space. Concretely, we leverage a large multimodal model as the latent world model, whose shared representations support two dedicated heads for both action selection and predictive transition modeling. The action head directly translates these latent representations into action selection, while the transition head captures action-conditioned future information. To mitigate the identity shortcut, we further introduce an action-aware contrastive objective that encourages the transition representations to capture discriminative action-induced changes. Extensive experiments across offline and online GUI settings show that our method achieves up to 31.9% and 40.6% relative gains in success rate over pixel-space world models, respectively, while substantially reducing inference latency by up to 98.9%. Our code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.14228v1
- Canonical: https://arxiv.org/abs/2610.14228v1
Trouble viewing inline? Open PDF directly →
Full Text
96,528 characters extracted from source content.
Expand or collapse full text
arXiv Preprint LATENT WORLD ACTION MODEL FOR GUI AGENTS Seungjun Yu ∗ , Hojun Choi ∗ , Seojeong Park, Jaeyo Shin, Hyunjung Shim † ∗ Equal contribution. † Corresponding author. Korea Advanced Institute of Science and Technology (KAIST) seungjunyu,hchoi256,seojeong.park,jaeyoshin,kateshim@kaist.ac.kr ABSTRACT Pixel-space world models have recently emerged as a promising approach for GUI agents, predicting future screens prior to execution and subsequently re-perceiving them for action selection. However, this two-stage paradigm faces two key chal- lenges in GUI environments: (L1) the high computational cost of pixel reconstruc- tion and (L2) the need to reconstruct large and diverse appearance-level details beyond what is required for action selection. In practice, this screen-generation objective can induce an identity shortcut problem, where future-state prediction largely preserves the current state rather than capturing action-induced changes. In this work, we reformulate GUI world modeling by shifting the prediction tar- get from full-screen reconstruction to action-induced future information in latent space. Concretely, we leverage a large multimodal model as the latent world model, whose shared representations support two dedicated heads for both action selection and predictive transition modeling. The action head directly translates these latent representations into action selection, while the transition head captures action-conditioned future information. To mitigate the identity shortcut, we fur- ther introduce an action-aware contrastive objective that encourages the transition representations to capture discriminative action-induced changes. Extensive ex- periments across offline and online GUI settings show that our method achieves up to 31.9% and 40.6% relative gains in success rate over pixel-space world models, respectively, while substantially reducing inference latency by up to 98.9%. Our code is available at https://github.com/sjyu001/latent2world. 1INTRODUCTION Recent advancements in large multimodal models (LMMs) have accelerated the development of graphical user interface (GUI) agents (Hong et al., 2024; Rawles et al., 2023; Lee et al., 2024; Gou et al., 2024)—an increasingly indispensable technology in the modern digital era. These agents are designed to complete complex, user-specified tasks on mobile platforms by automatically executing sequences of actions. By reducing manual interaction, they facilitate mobile app use across diverse real-world domains, ranging from commercial services such as e-commerce (Zhang et al., 2025) to accessibility support for people with visual or motor impairments and older adults (Huq et al., 2026). Among such efforts, recent works (Xu et al., 2026) have explored world models that simulate the consequences of actions proposed by GUI agents (e.g., button clicks and scrolling) prior to direct ex- ecution. Such foresight enables GUI agents to anticipate potential failures and adjust their strategies accordingly, improving robustness in long-tail and irreversible scenarios (e.g., payments and account deletion), as well as in long-horizon tasks involving extended action sequences. One promising ap- proach is visual-space prediction (Zheng et al., 2026; Luo et al., 2026), where GUI world models forecast a corresponding next GUI state for each candidate action in a visually renderable form (e.g., a screenshot or renderable HTML). These predicted future states are then re-perceived by a separate action-scoring model, which selects the candidate that best advances the task goal. This two-stage “Render-and-Re-Perceive” pipeline provides explicit spatial and visual information, such as element locations and layouts, that can support grounded GUI action planning beyond relatively concise textual descriptions (Gu et al., 2025). Project page: https://hchoi256.github.io/projects/latent2world/ 1 arXiv:2610.14228v1 [cs.CV] 9 Oct 2026 arXiv Preprint Figure 1: Our method, leveraging latent future-state representations, achieves the best GUI navigation per- formance with substantially improved efficiency. However, explicitly reconstructing future screens introduces an expensive and un- necessarily broad intermediate task into action selection.(L1) Autoregressive or diffusion-based screen synthesis incurs substantial computational latency, posing a bottleneck for responsive GUI interac- tion (Figure 1). Moreover, (L2) GUI tran- sitions exhibit large and varied appearance changes (Figure 2), requiring pixel-space world models to capture diverse future ap- pearances. Yet, action selection does not require reconstructing all such visual de- tails, but rather understanding the future consequence of each action. In practice, pixel-space world models may retain the current screen despite substantial next-screen changes (Figure 4), a behavior we term an identity shortcut. We hypothesize that full-screen reconstruc- tion encourages reusing current-screen information instead of modeling action-induced changes. This distinction raises the question: how can a GUI world model learn future information useful for action selection without reconstructing the future screen? Figure 2: Natural videos exhibit strong temporal continu- ity, whereas GUI transitions show lower visual similarity and large variance, increasing the range of future appear- ance variations that must be modeled in pixel space. Our key insight is that GUI world mod- eling should focus on future information that helps distinguish which candidate action best advances the task goal. This perspective requires connecting what an action will cause with whether that out- come is useful for the task. Training on annotated actions teaches a model which action to select, but does not ex- plicitly require it to anticipate the result- ing state. Conversely, learning to predict observed next states provides informa- tion about action consequences, but does not directly teach the model how to use that information for action selection. We therefore jointly train shared representa- tions for both candidate-action scoring and action-conditioned future-state prediction. The predictive objective encourages these represen- tations to capture action-induced future information, while the action-selection objective connects them to task-relevant decisions. Accordingly, we implement the predictive objective in latent space, enabling the model to learn from future states without reconstructing their full visual appearance. Specifically, given the task goal, current GUI screenshot, action history, and candidate actions pro- posed by a frozen GUI agent, we employ a LoRA-adapted LMM to produce action-specific hidden representations. A lightweight action head maps each representation to a scalar score, and the re- sulting candidate distribution is trained against the ground-truth one-hot distribution identifying the annotated action. In parallel, a transition head predicts a next-state visual embedding from the shared representation of the observed action, using an embedding of the actual next screenshot as its target. However, simply minimizing the distance to this target may allow the previously observed identity shortcut. We therefore train the transition head with an action-aware contrastive objective, aligning its prediction with the next-state embedding while separating it from the current-state embedding and embeddings of unrelated transitions. Both objectives jointly update the shared backbone, allow- ing future-state supervision to shape the representations used for action scoring. At inference time, the transition head is discarded, and the shared backbone and action head evaluate candidate actions in a single forward pass. This design bypasses the conventional “Render-and-Re-Perceive” pipeline while using representations trained to support both action selection and future-state prediction. Through extensive experiments in complementary offline and online settings (Li et al., 2024; Rawles et al., 2025), our method improves success rates over the corresponding base agents that do not use 2 arXiv Preprint world modeling by up to 11.6% and 47.0%, respectively. It further outperforms pixel-space world models by up to 31.9% and 40.6%, while reducing world-model inference latency by up to 98.9%. • We revisit GUI world modeling by shifting the prediction target from full-screen recon- struction to action-induced future information for effective action selection in latent space. • We present a latent GUI world-action model that jointly learns action selection and action- conditioned future prediction with action-aware contrastive learning over latent transitions. • Experiments show performance gains over both pixel-space world models and base GUI agents, while substantially reducing inference latency relative to pixel-space world models. 2RELATED WORKS 2.1MOBILE GUI AGENTS Recent advances in LMMs have substantially improved mobile GUI agents, shifting interaction from metadata- or text-centric interfaces toward direct visual perception and grounded control (Hong et al., 2024; Lin et al., 2025; Qin et al., 2025). Given a user instruction and the current GUI observa- tion, these agents perform human-like operations such as tapping, typing, and swiping to accomplish complex tasks across mobile applications (Wang et al., 2024). Subsequent studies have further en- hanced their capabilities through improved visual grounding (Nayak et al., 2025), interaction-history modeling and structured reasoning (Liu et al., 2025), self-reflection and error correction (Wu et al., 2025), and reinforcement-learning-based policy optimization (Shi et al., 2025; Lu et al., 2025a). Building on this progress, mobile GUI agents have demonstrated broad practical value by automat- ing routine interactions in commercial applications such as e-commerce (Zhang et al., 2025) and reducing interaction barriers for users with visual or motor impairments and older adults (Zhang et al., 2021; Huq et al., 2026). However, these advances primarily improve the action-selection ca- pabilities of GUI agents through better perception, reasoning, and policy optimization. Our work instead focuses on the complementary question of how predictive world-model information should be represented and integrated within GUI agents to explicitly better support action selection. 2.2GUI WORLD MODEL FOR APP AGENTS GUI world models provide agents with predictive knowledge of environment dynamics by simulat- ing action-conditioned future states before direct execution. Early approaches primarily operate in textual space, predicting concise natural-language descriptions of state changes (Chae et al., 2025) or imagined outcomes of candidate actions for model-based planning (Gu et al., 2025). Such textual abstractions efficiently capture semantic transitions but provide limited spatial and visual details. To retain more task-relevant GUI structure, subsequent work represents future states as textual sketches containing semantic and spatial information (Cao et al., 2026). A complementary line moves toward explicit visual reconstruction. Diffusion-based approaches (Luo et al., 2026; Xu et al., 2026) synthe- size future GUI screenshots with textual information, while renderable-code-based methods (Zheng et al., 2026) predict HTML combining structured interface information with deterministic visual rendering. However, explicitly materializing future GUI states requires modeling broad appearance- level details, incurring computational cost beyond what action selection requires. Latent predic- tive modeling provides an alternative and has been explored for action modeling in embodied set- tings (Chen et al., 2026). Motivated by this mismatch, our work brings latent future prediction to GUI action selection. We shift the prediction target from full-state reconstruction to action-induced future information, enabling more efficient use of predictive knowledge for action selection. 3METHODOLOGY In this work, we reformulate GUI world modeling by shifting the prediction target from full-screen reconstruction to action-induced future information in latent space. Our approach learns shared rep- resentations that jointly support complementary objectives: action selection identifies which can- didate action best advances the task goal, while predictive world modeling captures future conse- quence induced by that action. This formulation directly integrates future-state predictive capability into action selection while avoiding unnecessary visual reconstruction and computational overhead. 3 arXiv Preprint (b) Inference Candidate ac*ons (with confidence) GUI Agent Current State Click “Wi-fi”0 Scroll Down1 Press Back2 ⚖ Gate High confidence Low confidence Top-1 ac)on Execute GUI World Model 00.15 10.80 20.05 Ac)on scores 0Click “Wi-fi” 1Scroll Down Qwen3-VL Candidate scores 퐿 !"#$%& (Cross-Entropy) 퐿 '()&*$#$%& (InfoNCE) Click “Wi-fi”0 Scroll Down1 Press Back2 Ac*on candidates History Previous acBons “Turn on Wi-fi” Task goal predicted next state ̂푧 !"# Current State GUI Agent AcBon Head TransiBon Head hidden state ℎ ! ... 0.80푎 $ 0.15푎 # 0.05푎 % 푧 & (memory bank, negative) 푧 ! (nega)ve) 푧 !"# (posi)ve) Vision Encoder 푧 ! 푧 !"# current state 푠 ! next state 푠 !"# (a) Training Figure 3: Overview of our latent world-action model for GUI agents. During training, we lever- age a large multimodal model to learn action-specific shared representations through action scoring and contrastive future prediction. At inference, the transition head is removed for single-pass action scoring, while a confidence gate on the agent determines whether to invoke world-model rescoring. We first introduce the latent world-action model, which employs a LoRA-adapted large multimodal model (LMM) to produce action-specific latent representations shared across world modeling and action selection. These representations are directly mapped to candidate-action scores for task-level decision making (Section 3.1). We then formulate predictive latent modeling of GUI transitions, where a transition head predicts action-conditioned future representations from the shared latent features without explicit screen reconstruction. To capture action-induced changes in these predic- tive representations, we further present an action-aware contrastive objective (Section 3.2). The resulting world-action model is jointly optimized during training while enabling efficient inference. 3.1LATENT WORLD-ACTION MODEL FOR GUI AGENTS Our formulation is motivated by the observation that action selection does not require reconstructing the full visual appearance of a future GUI state. Instead, a world model should preserve the future information necessary to evaluate the consequence of each action. This consideration is particularly relevant in GUI environments, where even a single atomic action can induce substantial appearance changes, while action evaluation may depend on only a small subset of the resulting information. Such a setting naturally motivates latent representations, which provide compact semantic abstrac- tions and have been successfully adopted for predictive modeling across robotics (Bu et al., 2025), autonomous driving (Tan et al., 2026; Jia et al., 2026), and video generation (Xia et al., 2026). For example, if selecting the “Send” button leads to a login page, recognizing the resulting authentica- tion requirement can be sufficient for evaluating the action without reproducing the entire screen. Building on this insight, we introduce a latent world-action model that uses the hidden representa- tions of an LMM as a shared representation space for world modeling and action selection (Figure 3). Given the current GUI context and candidate actions, the model produces an action-specific latent representation for each candidate. A lightweight action head then maps these representations to candidate-action scores, while a transition head operates on the same latent space to predict action- conditioned future representations. This shared formulation allows predictive supervision to directly shape the representations used for action selection without explicit screen reconstruction. 3.1.1LATENT REPRESENTATIONS FOR GUI ACTION SELECTION We first instantiate the action-selection branch of our latent world-action model by directly scoring candidate actions from LMM hidden representations, avoiding explicit visual decoding. At each interaction step, given the current GUI screenshot s t , task goal g, interaction historyH t , and a set of K candidate actions A t = a 1 t ,...,a K t proposed by a frozen GUI agent, our LoRA-adapted 4 arXiv Preprint (a)Identity-ShortcutExample(b)Shortcut Severity & SR By Transition Size Current StateNext State (GT)Code2WorldDiffusion Code2World Diffusion All-Steps Weak Transition Medium Transition Strong Transition Latent (Ours) Base Agent Shortcut Severity (%) Figure 4: Identity shortcut and SR across GUI transition magnitudes. (a) Example of identity- preserving future prediction. (b) Downstream SR on a random 2.5K-sample AndroidControl subset. Shortcut severity is measured in each method’s native prediction space—pixel space for baselines and latent space for ours—as required by each formulation. In practice, larger transitions coincide with higher shortcut severity and lower SR, while ours substantially attenuates these trends. modelW θ (Team, 2025) encodes all candidates jointly into a shared latent representation: h t =W θ s t ,g,H t ,A t ,(1) where the candidates are placed in a single prompt in a randomly permuted order and h t is the last-layer hidden state at the final prompt token. These representations encode candidate-specific information for downstream action selection directly within the shared predictive latent space. A lightweight action head then maps each h k t to a scalar score, yielding the candidate-action distribu- tion p t . We supervise this distribution with the one-hot ground-truth target y t using cross-entropy: p t = softmax [ActionHead(h t ) k ] K k=1 , L Action =− K X k=1 y k t logp k t ,(2) where ActionHead(h t ) k =w ⊤ k h t andw k is the output embedding of the k-th index token. This formulation directly maps shared representations to action decisions without reconstructing future screens. We next extend this latent space to predictive GUI transition modeling in Section 3.2. 3.2PREDICTIVE LATENT MODELING OF GUI TRANSITIONS While the action-selection objective introduced above trains the shared representations to distin- guish among candidate actions, it does not explicitly require them to capture the future consequence induced by each action. We therefore augment action supervision with predictive modeling of GUI transitions as an auxiliary training objective within the same LMM latent space. Rather than re- constructing the full visual appearance of the next screen, we train the model to capture action- conditioned future information that can directly shape the representations used for action selection. Concretely, a lightweight transition head maps the same shared representation h t to the latent rep- resentation of the next state reached by the annotated action within this shared space to provide predictive supervision, while the actual next screenshot s t+1 is encoded by a target encoder ̄ φ: ˆz t+1 = TransitionHead(h t ), z t+1 = ̄ φ(s t+1 ).(3) Because h t is also the representation used to score the candidates, and their order is permuted in ev- ery example, predicting the next state of the annotated action requires h t to encode which candidate is correct and what it leads to. This predictive branch therefore provides complementary future-state supervision directly in the shared latent space without explicitly reconstructing the next screen. 3.2.1IDENTITY SHORTCUT IN GUI WORLD MODELING However, this latent transition formulation alone does not explicitly ensure that the model cap- tures the change induced by the action. That is, a point-wise objective independently pulls each prediction toward its paired next-state latent, without explicitly contrasting it against the current- state latent. Consequently, the model may preserve current-state information rather than capture the action-induced change, a behavior we refer to as the identity shortcut. This issue is particularly 5 arXiv Preprint relevant in GUI environments, where larger transitions can introduce substantial visual content that is weakly constrained by the current screen. Under such uncertainty, reusing current-state informa- tion can become an attractive shortcut, even under substantial visual changes, leading the model to preserve the existing interface rather than capture newly induced changes. We therefore examine whether this shortcut becomes more pronounced as the magnitude of GUI transitions increases. As shown in Figure 4(a), pixel-space world models using either renderable HTML prediction or diffusion-based screenshot generation can produce future screens visually close to the current GUI despite substantial changes in next state. Figure 4(b) analyzes this behavior across transition magni- tudes. We quantify transition magnitude by average absolute pixel change between grayscale 64×64 current and next screenshots, and group transitions into weak, medium, and strong regimes accord- ingly. We identify an identity shortcut when the predicted future remains close to the current screen by this pixel-change measure, despite a substantially larger current-to-next transition. Notably, the identity shortcut becomes increasingly pronounced as transition magnitude grows for pixel-space world models. Their downstream success rates also decrease for larger transitions, further motivat- ing an objective that explicitly distinguishes the action-induced future state from the current state. 3.2.2ACTION-AWARE CONTRASTIVE LEARNING FOR LATENT GUI WORLD MODELING We introduce an action-aware contrastive objective that encourages the predicted future represen- tation to capture action-induced changes. Rather than supervising each prediction solely through point-wise matching to its target, we contrast it against alternative states that should remain dis- tinguishable from the action-induced future. Specifically, we treat the predicted future latent ˆz k t+1 as the anchor and the next-state latent z t+1 = ̄ φ(s t+1 ) as the positive. To directly discourage the identity shortcut, we encode the current screenshot as z t = ̄ φ(s t ) and use it as an explicit negative. However, contrasting only against the current state provides limited discrimination among plausible action outcomes. For example, in the filter interface illustrated in Figure 4(a), selecting “Most pop- ular” or “Closest” produces similar next screens that differ mainly in the activated sorting option. Using only the current screen as a negative encourages either prediction to depart from the source state, but does not distinguish these action-specific outcomes from each other. We therefore main- tain a first-in-first-out memory bank B of latents from other transitions, exposing each prediction to competing future states. These negatives encourage the predicted representation to encode the consequence specific to the selected action rather than merely differ from the current state. To formulate the previously introduced contrastive objective, we ℓ 2 -normalize all latent representa- tions and denote their normalized forms with a tilde. Let e B denote the normalized memory-bank entries. With the negative setN = ̃z t ∪ e B and temperature τ , we define the transition objective: L Transition =− log exp ̃ ˆz k t+1 · ̃z t+1 /τ exp ̃ ˆz k t+1 · ̃z t+1 /τ + P n∈N exp ̃ ˆz k t+1 · n/τ .(4) This objective aligns predictions with the observed next state while separating them from the current state and competing futures. Thus, the transition objective discourages identity preservation and promotes action-specific transition information. Consistent with this objective, Figure 4(b) shows that our latent model substantially reduces shortcut severity between z t and ˆz t+1 directly in the shared latent space and maintains stable downstream SR across transition magnitudes. Finally, with λ balancing action selection and predictive transition learning within the shared space, we jointly optimize the two objectives to couple action discrimination with predictive transition supervision: L =L Action + λL Transition .(5) 4EXPERIMENTS Datasets and Evaluation Metrics. We evaluate our approach in both offline and online GUI inter- action settings. For offline evaluation, we use AndroidControl (Li et al., 2024), a large-scale dataset of Android interaction trajectories across diverse applications and tasks. We also use AITZ (Zhang et al., 2024), which provides mobile interaction trajectories with detailed action annotations. To assess out-of-distribution generalization, we further evaluate on GUI-Odyssey (Lu et al., 2025b), a benchmark specifically designed for cross-app GUI navigation over longer interaction trajectories. 6 arXiv Preprint Table 1: Offline in-distribution (ID) evaluation on AndroidControl and AITZ across GUI agents. ModelSettingsAndroidControl (ID)AITZ (ID)Efficiency TypeGroundingSRTypeGroundingSRTokens/TaskRun Time GPT-5.6 Base71.9260.4659.8459.8447.9941.303585.571.14 Code2World70.3459.6158.2258.6047.0339.2412634.219.31 Diffusion2Image 68.3155.0157.9456.4046.8038.59-135.20 Latent (Ours)75.5758.7558.5264.7848.2146.104260.901.46 Gemini-3.6-Flash Base 76.0260.7857.2565.1449.6745.192001.718.03 Code2World 75.8057.6155.5064.3748.0844.159557.1821.20 Diffusion2Image74.7559.8955.6964.0847.6143.51-194.51 Latent (Ours)80.3660.5061.1366.1945.8047.252658.468.49 GUI-Owl-7B Base73.2341.6540.4060.7335.5634.781120.642.14 Code2World68.5236.0831.8060.0333.5933.453890.489.00 Diffusion2Image66.3937.7233.1159.0134.7633.20-100.88 Latent (Ours)74.1643.5541.9660.9935.2335.061181.682.22 Qwen3-VL-8B Base75.9458.0755.5667.5749.2344.521391.274.70 Code2World75.2156.8054.0467.2148.7243.908170.4614.88 Diffusion2Image74.2956.7354.9667.2649.3043.31-171.68 Latent (Ours)77.1558.9557.0968.3550.4045.621702.174.83 For online evaluation, we use AndroidWorld (Rawles et al., 2025), an interactive Android bench- mark in which agents execute actions in real applications and are evaluated by task completion. For offline evaluation, we report three step-level metrics. Type measures whether the predicted ac- tion type matches the annotation. Grounding, evaluated only on click actions, measures whether the predicted coordinate falls within a fixed radius (14% of the screen width) of the annotated point. SR requires both the action type and argument to be correct. Specifically, a click must fall within the target element’s bounding box, a text input must match the annotated content, and a scroll must match the annotated direction. For online evaluation, we report task success rate together with To- kens/Task, Mean Steps, and Run Time to measure both task performance and interaction efficiency. Implementation Details. We evaluate our framework with GPT-5.6, Gemini 3.6 Flash, GUI-Owl- 7B (Ye et al., 2025), Qwen3-VL-8B, and Qwen3-VL-32B (Team, 2025) as agent backbones. For world-model baselines, we compare against Code2World (Zheng et al., 2026), which predicts ren- derable code for future GUI states, and a diffusion-based world model (Xu et al., 2026), which generates future screenshots in pixel space. For comparison, each GUI agent provides the same candidate-action set to all world-model methods, which differ only in how these candidates are eval- uated using predicted future information. All base agents follow the M3A (Rawles et al., 2025; Xu et al., 2026) setup, receiving the screenshot and accessibility tree and executing an action at each interaction step. For our latent world-action model, we fine-tune Qwen3-VL-8B-Instruct using LoRA (Hu et al., 2022) with rank r=16, scaling factor α=32, and dropout 0.05, applied to all at- tention and MLP projections. We optimize the model with AdamW using a learning rate of 4×10 −5 and an effective batch size of 8 for 3 epochs. We set the transition-loss weight to λ=1.0 and the contrastive temperature to τ =0.07. Training takes 8 hours on an NVIDIA A6000 GPU. Unless specified, the best and second-best results are highlighted in bold and underline , respectively. 4.1OFFLINE EVALUATION Table 1 summarizes the offline results on AndroidControl and AITZ. Our latent world-action model consistently improves action-type prediction across all evaluated backbones, with gains of +0.9 to +4.3 points on AndroidControl and +0.3 to +4.9 points on AITZ over the corresponding base agents. These improvements translate to higher step-level success rates in seven of eight backbone– dataset settings, with gains of up to +4.8 points. Improvements are more consistent for Type than for Grounding, as our rescoring mechanism changes which candidate action is selected but does not refine the grounding coordinates of individual candidates. Consequently, correcting the selected action type does not necessarily improve coordinate-level grounding performance across settings. In contrast, pixel-space world-model baselines reduce SR relative to their base agents in every of- fline setting, with decreases of up to 8.6 points. This pattern highlights the difficulty of using future- screen reconstruction as an intermediate step for action selection, whereas our approach incorporates 7 arXiv Preprint Table 2: Online evaluation on AndroidWorld across GUI agent backbones. ModelSettingsAndroidWorldEfficiency Easy SRMedium SRHard SROverall SRTokens/TaskMean StepsRun Time GPT-5.6 Base68.3350.0016.6754.46363.8K13.12127.4 Code2World68.3352.9422.2256.25525.4K13.71370.9 Diffusion2Image68.3347.0616.6753.57-13.212320.3 Latent (Ours)73.3358.8227.7861.61329.4K12.20141.4 Gemini-3.6-Flash Base70.0052.9433.3358.93204.6K11.42168.6 Code2World71.6752.9422.2258.04310.0K12.55482.3 Diffusion2Image 68.3352.9422.2255.75-11.682490.3 Latent (Ours)73.3358.8238.8963.39200.4K11.12228.1 GUI-Owl-7B Base25.005.880.0015.18226.1K16.67159.5 Code2World26.675.885.5616.69421.64K18.12659.4 Diffusion2Image26.675.880.0016.07-16.811260.2 Latent (Ours)33.3311.765.5622.32229.0K16.99205.3 Qwen3-VL-8B Base41.6714.715.5627.68105.9K11.21125.8 Code2World38.3320.5911.1128.57288.4K14.10519.9 Diffusion2Image 53.3314.715.5633.93-13.59789.9 Latent (Ours)55.0026.4716.6740.18129.3K13.40223.8 Qwen3-VL-32B Base56.6732.3511.1141.96320.8K13.29703.6 Code2World58.3335.2911.1143.75597.3K16.841453.6 Diffusion2Image55.0026.475.5638.39-17.813154.6 Latent (Ours)60.0038.2316.6746.42385.4K16.26960.0 future-state supervision into the latent representations used for candidate scoring. Compared with the base agents, our method increases Run Time by 0.08–0.46 and Tokens/Task by 61–675, while Code2World requires roughly 3.5–5.9× more tokens and substantially longer inference. Also, Dif- fusion2Image incurs a larger Run Time due to iterative screenshot generation. Overall, latent predic- tive supervision consistently improves step-level action selection with modest inference overhead. 4.2ONLINE EVALUATION Table 2 presents the online evaluation results on the interactive AndroidWorld benchmark. Unlike offline step-level evaluation, AndroidWorld rolls out the agent’s own actions, allowing improve- ments in action selection to influence subsequent states throughout the trajectory. Consistent with this intuition, our latent world-action model achieves the highest Overall SR across all five evaluated backbones, improving the corresponding base agents by +4.5 to +12.5 points. Notably, these gains persist across all task difficulty levels, with improvements of +5.6 to +11.1 points on Hard tasks. Complementing these online results, Figure 4(b) shows that our latent model reduces shortcut sever- ity relative to pixel-space world models while maintaining higher SR across transition magnitudes. In contrast, pixel-space world models yield mixed changes in Overall SR while requiring substan- tially greater inference cost. Code2World provides only modest improvements across most back- bones and requires approximately 2.1–4.1× the Run Time of the base agents, while Diffusion2Image requires roughly 4.5–18.2× longer execution. Our latent approach consistently achieves higher Overall SR than both pixel-space baselines while requiring substantially lower Run Time across all backbones. On the API-based backbones, GPT-5.6 and Gemini-3.6-Flash, our method further reduces both Mean Steps and total Tokens/Task relative to the base agents. Overall, our approach retains the benefits of world-model-based action evaluation while approaching the computational efficiency of base agents, bringing predictive GUI agents closer to practical real-time deployment. 4.3ABLATION STUDIES Out-of-Distribution Generalization. Table 3 evaluates generalization to the out-of-distribution GUI-Odyssey benchmark. Our latent approach achieves the highest SR on both GUI-Owl-7B and Qwen3-VL-8B, improving over the corresponding base agents under the cross-app and longer- horizon distribution shift. In contrast, both pixel-space world models reduce SR below the base agents across the two backbones. These results suggest that latent predictive supervision transfers more reliably to unseen GUI environments, preserving the benefits of world-model-based action evaluation across distinct interaction distributions without explicit future-screen reconstruction. 8 arXiv Preprint Table 3: OOD evaluation on GUI-Odyssey. ModelSettingsGUI-Odyssey (OOD) TypeGroundingSR GUI-Owl-7B Base69.0938.9733.73 Code2World66.7732.1327.78 Diffusion2Image67.5731.1126.83 Latent (Ours)70.1539.0734.73 Qwen3-VL-8B Base77.7262.4552.07 Code2World77.2859.7049.94 Diffusion2Image77.9362.9851.44 Latent (Ours)77.9162.2652.13 Table 4: Module ablation of the proposed method on AndroidWorld. Checkmarks in- dicate the inclusion of the components. GroupComponentsSR Action Head Contrastive Learning Transition Head I✗27.68 I✓✗36.36 I✓✗✓38.18 IV✓40.18 Base Click Scroll Type Centroid DiffusionCode2WorldLatent (Ours) Figure 5: t-SNE visualization of action-induced future representations. Our model better aligns its predicted futures with action-specific clusters than the base agent and pixel-space world models. Module Ablation. Table 4 isolates each component’s contribution on AndroidWorld. Group I cor- responds to the base agent without proposed modules. Adding the action head (Group I) improves SR from 27.68 to 36.36, establishing a latent candidate-rescoring baseline before world-model su- pervision. Group I incorporates the transition head to learn future-state information. Without contrastive learning, we train it with point-wise cosine regression,L GroupIII = 1− cos(ˆz k t+1 ,z t+1 ), between predicted and ground-truth next-state latents. This predictive supervision raises SR by +1.82 points to 38.18 over Group I, demonstrating additional benefit beyond action rescoring. Fi- nally, Group IV replaces point-wise matching with our action-aware contrastive objective, raising SR by another +2.00 points to 40.18. This gain isolates the benefit of explicitly distinguishing the action-induced future from the current state and competing future states, supporting our contrastive formulation. Overall, the model improves SR by +12.50 points over the base agent. Importantly, these components operate in latent space, preserving efficiency without future-screen generation. Qualitative Analysis of Action-Specific Future Representations. We qualitatively examine fu- ture representations from AndroidControl to assess whether they capture action-specific transition information useful for action selection beyond current screen. Given the identity-preserving ten- dency observed earlier, we seek to disentangle action-induced structure from current-state informa- tion. A visualization can be misleading because similar GUI states often admit similar action types, such that apparent action-specific clustering may arise from screen–action correlations rather than modeled action-induced transitions. To reduce this confounding, we cluster current states by their latent representations z t using k-means. For each sample, we subtract the mean predicted represen- tation of its current-state cluster and apply t-SNE to the residual. This procedure suppresses infor- mation shared within similar current-state contexts, allowing us to examine whether the remaining variation is structured by the executed action. As shown in Figure 5, the Base and Diffusion rep- resentations remain mixed across action types, while Code2World exhibits only partial separation. In contrast, our latent model forms the clearest action-specific structure around the centroids. This result provides qualitative evidence that our predictive representations capture how different actions induce distinct future-state changes beyond correlations attributable to the current screen. 5CONCLUSION We revisit GUI world modeling by asking which future information matters for action selection. Rather than reconstructing future screens, we introduce a latent world-action formulation that jointly learns action evaluation and predictive transitions with action-aware contrastive learning in a 9 arXiv Preprint shared space, capturing action-induced future information without costly screen generation and re- perception. Empirically, we observe consistent performance gains over base agents and pixel-space world models, with substantially improved inference speed over pixel-space world models. These results suggest a direction for GUI world modeling: future prediction need not reproduce what the future looks like, but should instead capture the decision-relevant consequences of agent actions. REFERENCES Qingwen Bu, Yanting Yang, Jisong Cai, Shenyuan Gao, Guanghui Ren, Maoqing Yao, Ping Luo, and Hongyang Li. Univla: Learning to act anywhere with task-centric latent actions. CoRR, abs/2505.06111, 2025. Yilin Cao, Yufeng Zhong, Zhixiong Zeng, Liming Zheng, Jing Huang, Haibo Qiu, Peng Shi, Wenji Mao, and Wan Guanglu. Mobiledreamer: Generative sketch world model for GUI agent. CoRR, abs/2601.04035, 2026. Hyungjoo Chae, Namyoung Kim, Kai Tzu iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. Web agents with world models: Learning and leveraging environment dynamics in web navigation. In The Thirteenth International Conference on Learning Representations, 2025. Jialei Chen, Kai Wang, Kang Chen, Shuaihang Chen, Feng Gao, Wenhao Tang, Zhiyuan Li, Weilin Liu, Zhuyu Yao, Boxun Li, Yuanbo Xu, and Chao Yu. Lawam: Latent world action models for efficient dynamics-aware robot policies. CoRR, abs/2606.15768, 2026. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, Huan Sun, and Yu Su. Is your LLM secretly a world model of the internet? model-based planning for web agents. Trans. Mach. Learn. Res., 2025, 2025. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for GUI agents. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. IEEE, 2024. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022. Syed Fatiul Huq, Ziyao He, Yirui He, and Sam Malek. Bridging the gap between automated in- tervention and actual user experience: A mixed-methods study on mobile accessibility issues for screen reader users. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, CHI 2026, Barcelona, Spain, April 13-17, 2026. ACM, 2026. Feiyang Jia, Lin Liu, Ziying Song, Caiyan Jia, Hangjun Ye, Xiaoshuai Hao, and Long Chen. Driveworld-VLA: Unified latent-space world modeling with vision–language–action for au- tonomous driving. In Forty-third International Conference on Machine Learning, 2026. Sunjae Lee, Junyoung Choi, Jungjae Lee, Munim Hasan Wasi, Hojun Choi, Steven Y. Ko, Sangeun Oh, and Insik Shin. Mobilegpt: Augmenting LLM with human-like app memory for mobile task automation. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, ACM MobiCom 2024, Washington D.C., DC, USA, November 18-22, 2024. ACM, 2024. Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L. Berg. TVQA: localized, compositional video question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, p. 1369–1379. Asso- ciation for Computational Linguistics, 2018. 10 arXiv Preprint Wei Li, William E. Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyama- gundlu, and Oriana Riva. On the effects of data scale on UI control agents. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Sys- tems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision-language-action model for GUI visual agent. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025, p. 19498–19508, 2025. Tao Liu, Chongyu Wang, Rongjie Li, Yingchen Yu, Xuming He, and Song Bai. Gui-rise: Structured reasoning and history summarization for GUI navigation. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025, 2025. Fanbin Lu, Zhisheng Zhong, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Arpo:end-to-end policy opti- mization for GUI agents with experience replay. CoRR, abs/2505.16282, 2025a. Quanfeng Lu, Wenqi Shao, Zitao Liu, Lingxiao Du, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, and Ping Luo. Guiodyssey: A comprehensive dataset for cross- app gui navigation on mobile devices. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), p. 22404–22414. IEEE, 2025b. Dezhao Luo, Bohan Tang, Kang Li, Georgios Papoudakis, Jifei Song, Shaogang Gong, Jianye HAO, Jun Wang, and Kun Shao. Vimo: A generative visual GUI world model for app agents. In The Fourteenth International Conference on Learning Representations, 2026. Seungjun Nah, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee. NTIRE 2019 challenge on video deblurring and super-resolution: Dataset and study. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, June 16-20, 2019, p. 1996–2005. Computer Vision Foundation / IEEE, 2019. Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A. Rodr ́ ıguez, Montek Kalsi, Nicolas Cha- pados, M. Tamer ̈ Ozsu, Aishwarya Agrawal, David V ́ azquez, Christopher Pal, Perouz Taslakian, Spandana Gella, and Sai Rajeswar. Ui-vision: A desktop-centric GUI benchmark for visual per- ception and interaction. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, volume 267, 2025. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jia- hao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. UI-TARS: pioneering automated GUI interaction with native agents. CoRR, abs/2501.12326, 2025. Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy P. Lillicrap. An- droidinthewild: A large-scale dataset for android device control. In Advances in Neural Infor- mation Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William E. Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Kenji Toyama, Robert James Berry, Divya Tyamagundlu, Timothy P. Lillicrap, and Oriana Riva. Androidworld: A dynamic benchmarking environment for autonomous agents. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenRe- view.net, 2025. Yucheng Shi, Wenhao Yu, Zaitang Li, Yonglin Wang, Hongming Zhang, Ninghao Liu, Haitao Mi, and Dong Yu. Mobilegui-rl: Advancing mobile GUI agent through reinforcement learning in online environment. CoRR, abs/2507.05720, 2025. 11 arXiv Preprint Shuhan Tan, Kashyap Chitta, Yuxiao Chen, Ran Tian, Yurong You, Yan Wang, Wenjie Luo, Yulong Cao, Philipp Kr ̈ ahenb ̈ uhl, Marco Pavone, and Boris Ivanovic. Latent chain-of-thought world modeling for end-to-end autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 39724–39733, 2026. Qwen Team. Qwen3-vl technical report. CoRR, abs/2511.21631, 2025. Zachary Teed and Jia Deng. RAFT: recurrent all-pairs field transforms for optical flow. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part I, volume 12347, p. 402–419. Springer, 2020. Zizhao Tong, Yeying Jin, Hongfeng Lai, Zeqing Wang, Zhaohu Xing, Kexu Cheng, Haoran Xu, Zhao Pu, Shangwen Zhu, Ruili Feng, Jian Zhao, Yan Zhang, Hao Tang, and Ling Shao. SCOPE: simulating cross-game operations in playable environments for FPS world models.CoRR, abs/2605.23345, 2026. Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Ji- tao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. CoRR, 2024. Penghao Wu, Shengnan Ma, Bo Wang, Jiaheng Yu, Lewei Lu, and Ziwei Liu. Gui-reflection: Em- powering multimodal GUI models with self-reflection behavior. In Advances in Neural Informa- tion Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025, 2025. Tianze Xia, Yongkang Li, Lijun Zhou, Jingfeng Yao, Kaixin Xiong, Haiyang Sun, Bing Wang, Kun Ma, Guang Chen, Hangjun Ye, Wenyu Liu, and Xinggang Wang. Drivelaw: Unifying planning and video generation in a latent driving world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 39701–39712, 2026. Weikai Xu, Kun Huang, Yunren Feng, Jiaxing Li, Yuhan Chen, Yuxuan Liu, Zhizheng Jiang, Heng Qu, Pengzhi Gao, Wei Liu, Jian Luan, Xiaolin Hu, and Bo An. How mobile world model guides gui agents?, 2026. URL https://arxiv.org/abs/2605.10347. Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, Jitong Liao, Qi Zheng, Fei Huang, Jingren Zhou, and Ming Yan. Mobile-agent-v3: Fundamental agents for GUI automation. CoRR, abs/2508.15144, 2025. Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI 2025, YokohamaJapan, 26 April 2025- 1 May 2025. ACM, 2025. Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-action-thought for GUI agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, volume EMNLP 2024, p. 12016–12031, 2024. Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle I. Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, and Jeffrey P. Bigham. Screen recognition: Creating accessibility metadata for mobile applications from pixels. In CHI ’21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yokohama, Japan, May 8-13, 2021. ACM, 2021. Yuhao Zheng, Li’an Zhong, Yi Wang, Rui Dai, Kaikui Liu, Xiangxiang Chu, Linyuan Lv, Philip Torr, and Kevin Qinghong Lin. Code2world: A GUI world model via renderable code generation. CoRR, abs/2602.09856, 2026. Luowei Zhou, Chenliang Xu, and Jason J. Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the Thirty-Second AAAI Conference on Artificial In- telligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Or- leans, Louisiana, USA, February 2-7, 2018, p. 7590–7598. AAAI Press, 2018. 12 arXiv Preprint APPENDIX CONTENTS A Limitations and Future Work13 B Experimental Details14 B.1 Device Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14 B.2 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14 B.3 Dataset Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 B.4 Evaluation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 B.5 Pseudocode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 C Additional Qualitative Results21 C.1 GUI Navigation Trajectories . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 C.2 Additional Identity-Shortcut Examples . . . . . . . . . . . . . . . . . . . . . . . .24 D Prompts28 D.1 GUI-Agent Prompt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28 D.2 Candidate-Generation Prompt . . . . . . . . . . . . . . . . . . . . . . . . . . . .29 D.3 World-Model Baseline Prompts . . . . . . . . . . . . . . . . . . . . . . . . . . . .30 D.4 Evaluation and Scoring Prompts . . . . . . . . . . . . . . . . . . . . . . . . . . .31 ALIMITATIONS AND FUTURE WORK Dependence on the Base Agent’s Candidates. Our latent world-action model re-ranks candidate actions proposed by the frozen base GUI agent and does not generate new actions. When the correct action is absent from the candidate set, the model cannot recover it. Its performance is therefore bounded by the quality and coverage of the base agent’s proposals. Expanding candidate coverage through more diverse sampling, adaptive candidate-set sizes, or improved candidate generation is a direction for reducing this dependence, when candidate proposals are narrow or redundant. Selective Invocation. Our model is invoked only when the confidence-based gate identifies a low- confidence decision. Consequently, errors made by the base agent with high confidence can bypass latent world-model evaluation, and terminal decisions such as completing or abandoning a task are not rescored in our current protocol. Replacing the fixed confidence threshold with a learned, uncertainty-aware invocation policy could improve coverage while retaining the computational ben- efit of selective rescoring and accounting for the expected value of additional evaluation. Single-Step Latent Prediction. The transition objective models only the immediate action- conditioned next state during training, and the transition head is discarded at inference. This design enables efficient single-pass action scoring, but it does not explicitly reason over longer-horizon con- sequences or provide interpretable reconstructions of predicted future states. Extending the objective to multi-step latent rollouts could support longer-term action evaluation, while lightweight decoders or probing modules could improve interpretability without requiring full-screen generation. Future Work. These limitations suggest several complementary extensions. Future work can combine latent action evaluation with stronger candidate generation, replace fixed confidence gat- ing with learned invocation policies, and extend one-step transition learning to multi-step latent 13 arXiv Preprint dynamics for longer-horizon decision-making. Finally, training with candidates from multiple base agents and evaluating across web and desktop interfaces would help assess whether latent action- conditioned transition modeling generalizes beyond the mobile GUI environments considered here. BEXPERIMENTAL DETAILS This section provides additional details on the experimental setup, including implementation, datasets, evaluation protocols, and analysis settings. B.1DEVICE INFORMATION All experiments were conducted on Ubuntu 22.04 LTS using eight NVIDIA RTX A6000 GPUs and dual AMD EPYC 7413 CPUs. Training used approximately 31 GiB of GPU memory per device, while evaluation used approximately 22 GiB for the judge and 26 GiB for the Qwen3-VL-8B agent. We used Python 3.11.15, PyTorch 2.6.0 with CUDA 12.4, and a random seed of 3 for training and 30 for AndroidWorld evaluation. B.2IMPLEMENTATION DETAILS This section provides additional implementation details of the baseline protocols, model architec- ture, training and inference procedure, hyperparameters, and analysis settings. B.2.1WORLD-MODEL BASELINE PROTOCOL FOR GUI AGENTS In this section, we describe how GUI world models support action selection through the “Render- and-Re-Perceive” pipeline (Zheng et al., 2026). This common protocol consists of two stages: (1) explicit future-state prediction by a world model and (2) world-model-guided GUI action selection. Stage 1: Future-State GUI World Modeling. Recent works on GUI agents have increasingly explored future-state world modeling, enabling more robust action decision-making by anticipating the consequences of candidate actions. Given the current GUI state s t , a user-specified goal g, and interaction history h t , a GUI agentG first proposes K candidate actions, after which a world model W predicts the corresponding future state ˆs k t+1 for each candidate as follows: A t =G(s t ,g,h t ) =a k t K k=1 ,ˆs k t+1 =W(s t ,a k t ,g).(A1) The resulting future state can take either a visual form (i.e., a GUI screenshot) or a textual form (i.e., renderable HTML), typically generated through diffusion-based image synthesis (Luo et al., 2026) or autoregressive code generation (Zheng et al., 2026), respectively. These forms offer com- plementary strengths: the former directly captures visual layout and spatial cues, whereas the latter additionally preserves structured information (e.g., element semantics, attributes, and coordinates). Stage 2: World-Model-Guided Action Selection. For action selection, the simulated future states ˆs k t+1 predicted by the world model are subsequently fed to a separate scorer π to assess the antic- ipated outcome of each candidate action with respect to the task goal g. Specifically, the scorer assigns a score q k t to each candidate action a k t and selects the optimal action a ∗ t at GUI state s t : q k t = π(s t ,g,a k t , ˆs k t+1 ), a ∗ t = arg max a k t ∈A t q k t .(A2) This straightforward “Render-and-Re-Perceive” paradigm provides the agent with explicit foresight by comparing the predicted pixel-space outcomes of candidate actions before direct execution. B.2.2MODEL ARCHITECTURE OF THE PROPOSED METHOD GUI-Agent Candidate Generation. At each interaction step, we first obtain a set of K candidate actions from the frozen base GUI agent. Given the current screenshot s t , task goal g, and interaction history H t , the agent produces A t = a 1 t ,...,a K t , which serves as the common input to the subsequent world-model-based action evaluation. We use K = 3 candidates proposed by the base 14 arXiv Preprint GUI agent. This preserves the original candidate distribution. During training, the ground-truth action is not inserted into the candidate set; the target is the index of the candidate that matches the ground-truth action under the step-level matcher, and samples in which no candidate matches are retained with the agent’s top-ranked candidate as the target, so that the model also learns to keep the agent’s decision when no alternative is better. Duplicate candidates are not removed during training; at inference, the model is invoked only when the two top-ranked candidates differ. The same candidate-generation protocol is used at training and evaluation. For fair comparison, for a given base GUI agent and interaction step, the identical candidate-action set is provided to all compared world-model methods; no method performs method-specific candidate regeneration. Shared Latent Representations. Our latent world-action model is instantiated with Qwen3-VL- 8B-Instruct as the shared multimodal backbone. The screenshot, goal, the last six actions of the history, and all K candidate actions are placed in a single prompt, and the model encodes them in one forward pass. We take the last-layer hidden state at the final prompt token as the shared repre- sentation h t , yielding a representation of dimension 4096. The same representation is provided to both the action-selection and transition-prediction branches, allowing predictive transition supervi- sion to shape the features used for candidate evaluation. The backbone is adapted with LoRA (rank 16) on all attention and MLP projections, while the original model parameters remain frozen. Action Head. The action head maps h t to one score per candidate. We implement the head as the backbone’s language-modeling output layer restricted to the K index tokens (“0”, “1”, “2”), which amounts to a linear map with input dimension 4096 and output dimension K. The scores over the K candidates are normalized with a softmax and supervised using the one-hot target corresponding to the annotated action; the candidate order is randomly permuted for every training example. At inference, this head provides the candidate scores used for rescoring and action selection. Transition Head. The transition head operates on the same representation h t used by the action head and predicts the latent representation of the next state produced by the executed action, ˆz t+1 . We implement the transition head as a single linear layer (4096→ 4096), with an output dimension matched to that of the target state representation. The transition head is used only during training to provide predictive supervision for latent dynamics and is removed at inference. Target Encoder. To construct supervision for latent transition prediction, a target encoder ̄ φ maps the observed GUI screenshots to latent state representations. Specifically, the next screenshot is encoded as z t+1 = ̄ φ(s t+1 ), which serves as the positive target for ˆz t+1 , while the current screenshot is encoded as z t = ̄ φ(s t ) and used as an explicit negative in the contrastive objective. We use the original Qwen3-VL-8B-Instruct model without LoRA as ̄ φ, feeding the screenshot alone at up to 448× 896 pixels, and extract its representation by mean-pooling the last-layer hidden states over non-padding tokens (4096 dimensions). The target encoder is frozen to ensure stable predictive supervision, and all target representations are precomputed once before training. Before computing the contrastive objective, all target and predicted representations are ℓ 2 -normalized. Contrastive Memory Bank. In addition to the current-state negative z t , we maintain a first-in- first-out (FIFO) memory bankB containing latent representations from other observed transitions. The bank provides competing future states as additional negatives, preventing the transition objective from being satisfied merely by moving the prediction away from the current state. The memory bank stores the (z t+1 ,z t ) pairs of the 192 most recent training examples (384 representations) and is updated after every training example by enqueuing that example’s pair and dequeuing the oldest pair once the bank exceeds its capacity. At each optimization step, all bank entries are included as contrastive negatives. The bank is initialized empty and fills during the first 192 examples, and no deduplication is applied to near-duplicate transitions during joint training. B.2.3JOINT TRAINING AND INFERENCE Training. We jointly optimize the action-selection and contrastive transition objectives as follows: L =L Action + λL Transition ,(A3) where λ controls the contribution of the transition objective. Both objectives update the shared LoRA-adapted backbone, while each objective supervises its head. Accordingly, L Action encour- 15 arXiv Preprint ages the shared representations to discriminate among candidate actions, whereasL Transition incor- porates action-conditioned future information into the representation space. Their joint optimization connects predictive transition modeling with the representations used for action selection. Inference. The transition head serves only as a training-time branch and is discarded at inference. Candidate-action scores are obtained directly from the shared backbone and action head, without generating or re-perceiving GUI screens. Thus, predictive transition supervision shapes the action- scoring representations during training without requiring transition prediction at inference. We further adopt the confidence-based gating strategy used in prior GUI world models (Zheng et al., 2026; Luo et al., 2026; Xu et al., 2026). When the base GUI agent assigns sufficiently high con- fidence to its proposed action, the latent world-action model is bypassed; otherwise, it is invoked to rescore the candidate actions. This selective invocation reduces inference overhead by applying latent world-model rescoring only to low-confidence decisions. B.2.4HYPERPARAMETERS Table A1 lists the hyperparameters used for training and inference in all settings. Computing Infrastructure1 NVIDIA RTX A6000 48GB GPU HyperparameterAssignment backboneQwen3-VL-8B-Instruct precisionbfloat16 LoRA rank / α / dropout16 / 32 / 0.05 LoRA target modulesq, k, v, o, gate, up, down trainable parameters43.6M (0.50%) max image pixels200,704 number of candidates K3 action history length6 transition headlinear, 4096→ 4096 target encoderfrozen Qwen3-VL-8B (mean-pooled) contrastive temperature τ0.07 transition loss weight λ1.0 memory bank size192 optimizerAdamW learning rate4e–5 learning rate schedulerconstant number of warmup steps0 weight decay0.0 effective batch size8 number of epochs3 training samples14,276 training time8 h on one GPU confidence threshold c hi 0.95 (online) / 1.0 (offline) random seed3 Table A1: Hyperparameters for training and inference in all settings. B.2.5ANALYSIS SETTINGS GUI Transition Analysis. To further characterize the appearance changes observed in GUI tran- sitions, we analyze the extent to which consecutive states can be explained by transporting existing visual content. This analysis complements Figure 2 by quantifying both the magnitude and spatial extent of pixel-level changes across GUI transitions relative to other visual domains. This perspec- tive is particularly relevant to GUI world modeling because a single action can introduce substantial changes in screen appearance, while action selection may depend on only a small subset of the resulting information. For example, in Figure A1, selecting the “Send” button redirects the user to a login page; recognizing the resulting authentication requirement can be sufficient for action 16 arXiv Preprint (a)V1:Post-Warp Residual Magnitude(b)V2: Post-Warp Residual Coveragec) Cross-Domain ResidualComparison Larger Residual Changes 2.4×Higher GUI Median 2.8×Higher Coverage AUC Figure A2: (a) GUI transitions exhibit stronger discontinuities than natural videos. (b) These GUI discontinuities also extend over broader spatial regions. (c) Across diverse visual domains, GUI dynamics show the highest degree of discontinuity, motivating latent-space GUI state forecasting. evaluation without reconstructing the full visual appearance of the next screen. Such transitions re- quire explicit future-screen prediction to accommodate a broad range of appearance-level variations, motivating our focus on action-relevant future information in latent space. Current StateWarpedTarget StateResidual Natural Video GUI State Transition 0.25 second V1: 0.009 V2: 0.003 V1: 0.349 V2: 0.676 Figure A1: Natural video transitions are largely ex- plained by pixel transport, resulting in small post-warp residuals. In contrast, GUI transitions induced by ac- tions exhibit substantially larger residuals, revealing the inherently discontinuous nature of GUI dynamics. We quantify these appearance changes using two complementary measures: (V1) Post-Warp Residual Magnitude, which measures the remaining appear- ance difference after transporting pixels from the current state, and (V2) Post- Warp Residual Coverage, which mea- sures how spatially extensive these resid- ual differences are. We estimate back- ward optical flow from the next state to the current state using RAFT-Large (Teed & Deng, 2020) with its default pretrained weights and preprocessing, and warp the current RGB image into the next-state co- ordinate frame using bilinear interpola- tion. Images are resized while preserv- ing aspect ratio such that the longer side is at most 512 pixels and are padded to dimensions divisible by eight. For each valid warped pixel, we compute the RGB L1 residual by averaging the absolute dif- ference across the three color channels. V1 is the mean residual over valid pixels for each transition, with each video or episode summarized by the median across its transitions. V2 is the fraction of valid pixels whose residual exceeds threshold τ , evaluated at 15 uniformly spaced thresholds from 0.02 to 0.30; transition-level coverage curves are averaged within each video or episode and summarized by their normalized area under the threshold curve. For the comparison with natural video, we use the complete AndroidControl validation split con- taining 137 episodes and 690 action-conditioned transitions, together with all 30 REDS validation videos. REDS frame pairs are sampled at 0.25-second intervals, whereas GUI pairs correspond directly to observed current- and next-state transitions. As shown in Figure A2(a) and (b), GUI tran- sitions exhibit 2.4× larger residual magnitude and 2.8× higher residual-coverage AUC than natural videos (Nah et al., 2019). For the cross-domain comparison in Figure A2(c), we evaluate REDS, YouCook2 (Zhou et al., 2018), TVQA (Lei et al., 2018), CrossFPS (Tong et al., 2026), and Android- Control using a common protocol of 30 units per domain and three transitions per unit. Video pairs are separated by one-third of a second and sampled at three positions distributed across each clip, while three GUI transitions are selected evenly within each episode. Dataset selection is determin- istic with seed 42. Overall, GUI transitions exhibit larger and more spatially extensive appearance changes than the other evaluated domains. These observations support the main-paper motivation 17 arXiv Preprint that pixel-space GUI world models must accommodate a broad range of future appearances, even though action selection may require only the action-relevant consequence of those transitions. Identity-Shortcut Analysis. For the analysis in Figure 4, we randomly sample 2.5K transitions from AndroidControl and evaluate the pixel-space world models according to the magnitude of the ground-truth GUI transition. We first convert the current and next screenshots to grayscale and re- size them to 64× 64. For a transition (s t ,s t+1 ), we define its transition magnitude as the mean absolute pixel difference between the two processed screenshots. We divide the samples into weak, medium, and strong transitions using thresholds of 0.08 and 0.24: weak transitions have magnitude below 0.08, medium transitions fall between 0.08 and 0.24, and strong transitions have magni- tude at least 0.24. To identify an identity shortcut, we additionally compare the predicted future screen ˆs t+1 with the current screen s t using the same pixel-change measure. A prediction is clas- sified as an identity shortcut when the ground-truth transition is larger than the predicted transition, d(s t ,s t+1 ) > d(s t , ˆs t+1 ), and the prediction remains close to the current screen, d(s t , ˆs t+1 ) < 0.08, where d(·,·) denotes the same mean absolute pixel difference. Shortcut severity is computed as the fraction of predictions satisfying this criterion within each transition-magnitude group. We report the downstream step-level success rate on the same samples across methods to jointly examine shortcut behavior and action-selection performance across transition magnitudes. Action-Specific Representation Analysis. For the representation analysis in Figure 5, we sample 900 transitions from the AndroidControl test set, stratified over action types (300 each for click, scroll, and text input). Because current GUI states can exhibit correlated action distributions, apply- ing dimensionality reduction to the predicted representations may confound current-state similarity with action-induced structure. We therefore cluster the current-state representations z t using k- means with 100 clusters, and keep the 52 clusters that contain at least six samples and at least two action types. For each sample, we compute a residual representation by subtracting the mean pre- dicted representation of samples belonging to the same current-state cluster from its predicted future representation. We then apply t-SNE to these residual representations and visualize the embeddings by action type. We use the frozen Qwen3-VL-8B target encoder (last-layer states mean-pooled over non-padding tokens) to embed the current screen for the base agent and the rendered next screen for the pixel-space world models, and the transition-head output at the last token of the ground-truth candidate in a three-candidate prompt for our model, with t-SNE perplexity 30, learning rate auto, 1,000 iterations, and random seed 0. Residual representations are ℓ 2 -normalized before t-SNE. This probe reads the representation at the ground-truth candidate’s tokens rather than at the final prompt token used for training and inference; the final-token representation, which predicts the next state of the annotated action, does not separate action types under the same residual analysis. B.3DATASET DETAILS AndroidControl. AndroidControl (Li et al., 2024) is a large-scale mobile GUI interaction dataset containing 15,283 human demonstrations over 14,548 unique tasks and 833 Android applications. Each trajectory provides screenshots, accessibility-tree observations, natural-language instructions, and corresponding interaction actions. A distinctive feature of AndroidControl is that each task includes both high-level goal instructions and low-level step-wise instructions, supporting evaluation of GUI agents under different levels of task abstraction. In our experiments, we use AndroidControl for offline in-distribution evaluation of step-level action selection. AITZ. Android-In-The-Zoo (AITZ) (Zhang et al., 2024) is a mobile GUI interaction dataset con- structed from Android trajectories with detailed action-level annotations. The dataset contains 18,643 screen–action pairs and augments interaction steps with chain-of-action-thought annotations describing the reasoning associated with GUI actions. It spans diverse mobile interaction scenarios, including general application use, Google applications, installation tasks, single-app interactions, and web shopping. We use AITZ as an additional offline in-distribution benchmark to evaluate whether the proposed action-selection framework generalizes across mobile interaction datasets. AndroidWorld. AndroidWorld (Rawles et al., 2025) is an interactive Android environment de- signed for evaluating autonomous GUI agents through online execution. It contains 116 parame- terized tasks across 20 real-world Android applications and instantiates task configurations rather 18 arXiv Preprint than relying solely on a fixed set of recorded trajectories. Unlike the offline benchmarks, the agent’s predicted action is executed in the environment and determines subsequent GUI states, allowing er- rors and improvements in action selection to propagate throughout the trajectory. We therefore use AndroidWorld for online evaluation and measure performance based on end-to-end task completion. GUI-Odyssey. GUI-Odyssey (Lu et al., 2025b) is a mobile GUI navigation dataset designed to evaluate long-horizon cross-application interaction. Its trajectories require agents to execute se- quences of actions that may span multiple applications, providing a substantially different interac- tion distribution from the predominantly single-app settings used in our in-distribution evaluation. We use GUI-Odyssey exclusively for out-of-distribution evaluation, without adapting the proposed model to its evaluation trajectories, to assess whether the learned action-selection representations transfer to unseen cross-app navigation settings under a pronounced distribution shift. B.4EVALUATION DETAILS This section provides additional details on the evaluation protocols and metrics used for offline, online, out-of-distribution, and efficiency evaluation. B.4.1OFFLINE EVALUATION Step-Level Evaluation Metrics. For offline evaluation on AndroidControl and AITZ, we evalu- ate each interaction step independently using three metrics: action type accuracy (Type), grounding accuracy (Grounding), and step-level success rate (SR). Type measures whether the predicted action type matches the annotated action type. Grounding is evaluated only for click actions and is consid- ered correct when the predicted coordinate falls within a radius corresponding to 14% of the screen width from the annotated target point. SR requires both the action type and its associated argument to be correct. Specifically, a click action is successful when the predicted coordinate lies within the target element’s bounding box, a text-input action when the predicted text matches the annotated content, and a scroll action when the predicted direction matches the annotation. All methods are evaluated on the same examples and candidate-action sets for each base GUI agent. B.4.2ONLINE EVALUATION Task-Level Evaluation Protocol. For online evaluation, we use AndroidWorld, where the agent’s selected actions are executed directly in the interactive environment and therefore determine subse- quent GUI states throughout the trajectory. We report task success rate (SR) according to the bench- mark’s task-completion criterion, separately for Easy, Medium, and Hard tasks as well as overall. To characterize interaction efficiency, we additionally report the average number of interaction steps (Mean Steps), total token usage per task (Tokens/Task), and execution time (Run Time). Each task is evaluated using a single task instance per task template (116 templates), with task parameters sampled from a fixed seed (30) so that every compared method receives identical goals, and with a maximum interaction budget of 10× the task’s complexity score as defined by AndroidWorld. Be- fore each rollout, the environment is reset by the task’s own initialization routine, which restores the required app states and returns the device to the home screen, and each rollout is followed by the corresponding tear-down routine, ensuring consistent initial conditions across compared methods. B.4.3OUT-OF-DISTRIBUTION EVALUATION Evaluation Protocol. We evaluate out-of-distribution generalization on GUI-Odyssey using the same step-level Type, Grounding, and SR metrics defined for the offline evaluation.GUI- Odyssey contains cross-application and longer-horizon interaction trajectories that differ from the in- distribution AndroidControl and AITZ settings, allowing us to examine whether the learned action- selection representations transfer beyond the training distribution under long-horizon cross-app nav- igation conditions. We evaluate GUI-Odyssey using 160 episodes (1,598 steps) drawn from the test set of its official random split: the test episodes are shuffled with a fixed seed, episodes with 4 to 12 steps whose actions map to our action space are kept, and the first 160 are used. We apply the same candidate-generation and rescoring protocol across all compared methods. The model is evaluated without any GUI-Odyssey fine-tuning or adaptation; it is trained only on AndroidControl and AITZ, and no GUI-Odyssey episode is used for training, validation, or hyperparameter selection. 19 arXiv Preprint B.4.4EFFICIENCY EVALUATION We evaluate computational efficiency using Tokens/Task, Mean Steps, and Run Time, depending on the evaluation setting. Tokens/Task measures the total number of input and output tokens, in- cluding image tokens, consumed by all model calls made for a task: every action-selection and summary call of the base GUI agent, the generation calls of language-model-based world models (the Code2World prompt and generated HTML), and the prompt of the scoring model (our judge or the world-model selector) on the steps where it is invoked; for API backbones, the token counts reported by the provider are used, and in the offline setting the same quantity is reported per step. Mean Steps, reported for online evaluation, denotes the average number of environment interactions taken per task. Run Time measures the end-to-end wall-clock time in seconds per task, measured from task initialization to the end of the episode and including agent inference, world-model gener- ation or rendering, scoring, and environment execution; in the offline setting, it is the per-step sum of the proposer, world-model, and scorer latencies. For diffusion-based world models, token us- age associated with image generation is not applicable and is therefore omitted. For methods using confidence-based gating, the reported efficiency reflects the same selective world-model invocation policy used for the corresponding performance evaluation under deployment conditions. B.5PSEUDOCODE Algorithm 1 Training the latent transition judge 1: Require: frozen VLM f θ with LoRA φ, transition head A, frozen target encoder E, datasetD, tempera- ture τ , weight λ, bank size M 2: Initialise FIFO bankB ←∅ 3: while not converge do 4:Choose (s t ,g,H t ,a k K k=1 ,y,s t+1 ) fromD 5:Shuffle candidate order; relabel y accordingly 6:# Action selection 7: x← Prompt(s t ,g,H t ,a k K k=1 ) 8: ℓ 0:K−1 , h← f θ,φ (x)(index-token logits, last-token state) 9: L pick ←− log softmax(ℓ) y 10:# Transition contrast 11: e← normalise(A(h)) 12: z t+1 ← E(s t+1 ), z t ← E(s t ) 13: N ←z t ∪B(current screen and bank as negatives) 14: L trans ←− log exp(sim(e,z t+1 )/τ) exp(sim(e,z t+1 )/τ) + P z ′ ∈N exp(sim(e,z ′ )/τ) 15:Take gradient step on∇ φ,A (L pick + λL trans ) 16:Push (z t+1 ,z t ) intoB; drop the oldest pair if|B| > M 17: end while Algorithm 2 Gated action selection at inference 1: Require: frozen agent π, trained judge f θ,φ , confidence threshold c hi 2: while task not finished do 3: (a k ,c k ) K k=1 ← π(s t ,g,H t )(candidates with confidences, most likely first) 4:# Gate 5:if fewer than K candidates or a 1 is terminal or a 1 = a 2 or|k : c k ≥ c hi | = 1 then 6:ˆa← a 1 7:else 8:# Judge (one forward pass, transition head unused) 9:x← Prompt(s t ,g,H t ,a k K k=1 ) 10:ℓ 0:K−1 ← f θ,φ (x) 11:ˆa← a arg max k ℓ k +1 12:end if 13:Execute ˆa; H t+1 ← H t ∪ˆa; observe s t+1 14: end while 20 arXiv Preprint CADDITIONAL QUALITATIVE RESULTS C.1GUI NAVIGATION TRAJECTORIES Figures A3–A5 show complete AndroidWorld episodes of the frozen Qwen3-VL-8B agent with our latent world-action model (Ours, top row) and without it (Base, bottom row), on the same task instance and seed. Every screen is the state actually reached in the emulator. The orange frame marks the step at which the gate opened and our model replaced the agent’s top-ranked candidate; the numbered boxes on that screen are the agent’s three candidates, and the bottom panel lists them with the agent’s own confidences to facilitate direct step-by-step qualitative comparison. Goal Delete the following recipes from Broccoli app: Chicken Alfredo Pasta, Tomato Basil Bruschetta, Grilled... Ours success 21 steps t=1 click "Search" t=2 click "Broccoli" t=3 click "Search" t=4 type "Chicken Alfredo... t=5 click "Chicken Alfred... t=6 click "More options" t=7 click "Delete" +14 more steps Base fail 2 steps t=1 open Broccoli t=2 status: infeasible Judge intervention at t=3 the agent's three candidates (boxes on the highlighted screen) scroll down conf 0.90 agent top-1 click "Search" conf 0.80 judge pick click "Categories" conf 0.60 012 Figure A3: Long-horizon recovery. At step 3 the agent prefers to scroll the full recipe list; our model selects the search button instead. The agent then finds and deletes each recipe and completes the 21-step task, whereas Base declares the task infeasible at step 2. 21 arXiv Preprint Goal Turn brightness to the max value. Ours success 5 steps t=1 open Settings t=2 (invalid action) t=3 scroll down t=4 click "Display" t=5 status: complete Base fail 10 steps t=1 open Settings t=2 scroll down t=3 scroll down t=4 click "System" t=5 scroll down t=6 scroll down t=7 scroll down +3 more steps Judge intervention at t=1 the agent's three candidates (boxes on the highlighted screen) scroll down conf 0.95 agent top-1 open Settings conf 0.85 judge pick scroll down conf 0.75 012 Figure A4: Choosing a reliable route. The agent’s top candidate is a swipe that opens quick settings; our model selects open Settings, and the agent reaches the brightness control in five steps. Base keeps scrolling within the System page until its step budget runs out. 22 arXiv Preprint Goal Turn wifi off. Ours success 6 steps t=1 open Settings t=2 click "Network & inte... t=3 click "Internet" t=4 click "switch_widget" t=5 click "switch_widget" t=6 status: complete Base fail 7 steps t=1 scroll down t=2 scroll down t=3 click "Settings" t=4 click "Network & inte... t=5 click "Internet" t=6 click "switch_widget" t=7 status: complete Judge intervention at t=1 the agent's three candidates (boxes on the highlighted screen) swipe conf 0.95 agent top-1 open Settings conf 0.75 judge pick swipe conf 0.85 012 Figure A5: Same intervention, different task. After the same first-step correction, Ours turns Wi-Fi off and verifies the toggle state. Base reaches the same toggle but leaves Wi-Fi on before declaring completion. 23 arXiv Preprint C.2ADDITIONAL IDENTITY-SHORTCUT EXAMPLES Diffusion-Based World Model. Across all transition magnitudes, the diffusion model reproduces the current screen with only local texture changes, even when the real next screen belongs to a different page or application (Figures A6–A7). Send a hello message to my friend Amelia Miller on the Snapchat app. current s t click real next s t + 1 Diffusion ŝ In the video audio app, learn origami folds. current s t click real next s t + 1 Diffusion ŝ In the BBC News app , Turn ON the news alert notification for the BBC News app. current s t click real next s t + 1 Diffusion ŝ With my son Evan Kramer, I would like to share the updates on the Israel Hamas War, as reported by The... current s t click real next s t + 1 Diffusion ŝ I'd liked the United states museum Art very much and wanted to share the art with cole at... current s t click real next s t + 1 Diffusion ŝ To make the background more eye-catching and appealing, I want to create a black-and-white... current s t scroll left real next s t + 1 Diffusion ŝ Figure A6: Identity shortcut of the diffusion-based world model (1/2). 24 arXiv Preprint Convert the 1 Year unit into all other time units in Time Section , Skip Ads if any . current s t click real next s t + 1 Diffusion ŝ Open the CNN News app and share the article Trump pleads not guilt to 4 felonies in 2020 election case... current s t click real next s t + 1 Diffusion ŝ With my son Evan Kramer, I would like to share the updates on the Israel Hamas War, as reported by the... current s t click real next s t + 1 Diffusion ŝ Open the ArtStation app and find an artwork inspired by sonya agafonova current s t click real next s t + 1 Diffusion ŝ The street art that I post, so I want to share it with Naomi Rousseau on her Gmail ID,... current s t click real next s t + 1 Diffusion ŝ Open the Keep Notes app and share the swimming class note to dbwscratch.test.id3@gmail.com through gmail current s t click real next s t + 1 Diffusion ŝ Figure A7: Identity shortcut of the diffusion-based world model (2/2). 25 arXiv Preprint Code2World. Because Code2World regenerates the page as HTML, its predictions rarely copy the current screen pixel for pixel. Its failures are of a different kind: the rendered page is well formed and plausible for the application, but it does not correspond to the real outcome of the action (Figures A8–A9). In the video audio app, learn origami folds. current s t click real next s t + 1 Code2World ŝ Send a hello message to my friend Amelia Miller on the Snapchat app. current s t click real next s t + 1 Code2World ŝ In the BBC News app , Turn ON the news alert notification for the BBC News app. current s t click real next s t + 1 Code2World ŝ Convert the 1 Year unit into all other time units in Time Section , Skip Ads if any . current s t click real next s t + 1 Code2World ŝ I have some images of an unidentified plant and I want to know which plant is this, so identify this... current s t click real next s t + 1 Code2World ŝ To make the background more eye-catching and appealing, I want to create a black-and-white... current s t scroll left real next s t + 1 Code2World ŝ Figure A8: Code2World predictions on transitions with large real screen changes (1/2). 26 arXiv Preprint I want to add an entrance animation to the presentation's first slide in order to make it look... current s t click real next s t + 1 Code2World ŝ Open the Keep Notes app and share the swimming class note to dbwscratch.test.id3@gmail.com through gmail current s t click real next s t + 1 Code2World ŝ Open the CNN News app and share the article Trump pleads not guilt to 4 felonies in 2020 election case... current s t click real next s t + 1 Code2World ŝ In the Google News app , save the Article of Thane crane girder launcher crashes on samruddhi . current s t click real next s t + 1 Code2World ŝ I would want to delete the Detroit Free Press Marathon Reminder from the Reminders app as it has... current s t click real next s t + 1 Code2World ŝ In the Letter Templates App, create a template of a reimbursement letter, to write the same type of... current s t scroll left real next s t + 1 Code2World ŝ Figure A9: Code2World predictions on transitions with large real screen changes (2/2). 27 arXiv Preprint DPROMPTS All prompts are reproduced from the released code; curly braces mark fields filled at run time. The agent prompt is the unmodified M3A prompt of AndroidWorld. Our method only appends the candidate-generation suffix (Prompt D.2) and consults the judge with the scoring prompt (Prompt D.4). D.1GUI-AGENT PROMPT M3A Action-Selection Prompt You are an agent who can operate an Android phone on behalf of a user. Based on user’s goal/request, you may - Answer back if the request/goal is a question (or a chat message), like user asks “What is my schedule for today?”. - Complete some tasks described in the requests/goals by performing actions (step by step) on the phone. When given a user request, you will try to complete it step by step. At each step, you will be given the current screenshot (including the original screenshot and the same screenshot with bounding boxes and numeric indexes added to some UI elements) and a history of what you have done (in text). Based on these pieces of information and the goal, you must choose to perform one of the action in the following list (action description followed by the JSON format) by outputing the action in the correct JSON format. ACTIONS: - Task completed:“action type”: “status”, “goalstatus”: “complete” - Task infeasible:“actiontype”: “status”, “goalstatus”: “infeasible” - Answer user’s question:“action type”: “answer”, “text”: <answertext> - Click/tap on an element (use the numeric index on the bounding box):“actiontype”: “click”, “index”: <targetindex> - Long press on an element:“action type”: “longpress”, “index”: <targetindex> - Type text into a text field (includes clicking the field and pressing enter):“actiontype”: “inputtext”, “text”: <textinput>, “index”: <target index> - Press the Enter key:“actiontype”: “keyboardenter” - Navigate to the home screen:“action type”: “navigatehome” - Navigate back:“actiontype”: “navigateback” - Scroll the screen or a scrollable UI element: “actiontype”:“scroll”, “direction”: <up, down, left, right>, “index”: <optional targetindex> - Open an app:“actiontype”: “openapp”, “appname”: <name> - Wait for the screen to update:“action type”: “wait” INPUT: The current user goal/request is:goal Here is a history of what you have done so far:history The current screenshot and the same screenshot with bounding boxes and labels added are also given to you. Here is a list of detailed information for some of the UI elements (notice that some elements in this list may not be visible in the current screen and so you can not interact with it, can try to scroll the screen to reveal it first), the numeric indexes are consistent with the ones in the labeled screenshot:ui elements GUIDELINES: General / Action Related / Text Related Operations (the unmodified M3A guidelines; omitted for space). OUTPUT: Now output an action from the above list in the correct JSON format, following the reason why you do that. Your answer should look like: Reason: ... Action:“action type”:... 28 arXiv Preprint D.2CANDIDATE-GENERATION PROMPT Online Candidate Suffix (appended to the M3A prompt) IMPORTANT — output format for THIS step: Propose 3 DISTINCT candidate actions to try next (each a possible single next action for the CURRENT step, not a sequence), each exactly as: CANDIDATE 0 Reason: <one line> Action:json Confidence: <0.1–1.0> CANDIDATE 1 Reason: <one line> Action:json Confidence: <0.1–1.0> CANDIDATE 2 Reason: <one line> Action:json Confidence: <0.1–1.0> Order them best-first. Confidence = how likely that action is the correct next one (0.8–0.9 strong evidence, 0.6–0.7 some ambiguity, 0.1–0.3 unlikely). Use the same JSON action schema as instructed above. Offline Candidate Prompt (AndroidControl / AITZ / GUI-Odyssey) Now propose exactly 3 DISTINCT candidate actions for the NEXT step — three alternatives for this single step, NOT a sequence — ordered most-likely first. Do NOT output a <toolcall>. Instead output ONLY one JSON object of exactly this form (all three keys required): “1”: “Confidence”: <number in [0.1, 1.0]>, “Action”: “action”: “...”, ..., “2”: “Confidence”: ..., “Action”: ..., “3”: “Confidence”: ..., “Action”:... Each “Action” object uses the mobile use argument schema above (action, coordinate, coordinate2, text, button, status, time). CONFIDENCE SEMANTICS: 1.0 = certain; 0.8–0.9 high confidence with strong evidence; 0.6–0.7 moderate with some ambiguity; 0.4–0.5 low due to significant uncertainty; 0.1–0.3 very low. If you feel there are no other reasonable candidates for this step, set the confidence of the remaining candidates to 0.1. Begin your reply with “”. 29 arXiv Preprint D.3WORLD-MODEL BASELINE PROMPTS Code2World System Prompt You are an expert UI State Transition Simulator and Frontend Developer. Your task is to predict the NEXT UI STATE based on a screenshot of the current state and a user interaction. IMAGE INTERPRETATION RULES: - Red Circle: Indicates a Click or Long Press target at that location. - Red Arrow: Indicates a Scroll or Swipe. The arrow points in the direction of finger movement. - These cues exist ONLY to show the action. DO NOT render these red circles or arrows in your output HTML. CRITICAL STRUCTURAL RULES: - Output ONLY raw HTML. Start with <!DOCTYPE html> and end with </html>. - All visible content MUST be wrapped in <div id=“render-target”> with width 1080px, height 2400px, position relative, overflow hidden. [...] CONTENT GENERATION LOGIC: - Transition: Analyze the action. If the user clicks a button, show the result (e.g., a menu opens, a checkbox checks, page navigates). - Images: Use semantic text placeholders. DO NOT use real URLs. - Icons: Use simple inline SVG paths or Unicode. OUTPUT REQUIREMENT: - Do NOT generate Markdown blocks. Do NOT provide explanations or conversational text. Output the code directly. Code2World User Prompt and Diffusion Conditioning Text CODE2WORLD USER PROMPT: <image> (current screenshot with the action drawn as a red circle or arrow) INPUT CONTEXT 1. User Intent: “goal” 2. Interaction Details: Description:semantic desc; Action Data:actionjson COMMAND Based on the visual cues in the image and the interaction data above, generate the HTML for the RESULTING UI STATE (what the screen looks like after this action). semantic desc BY ACTION TYPE: - click: User performed a CLICK at coordinates (x, y). Expect the button/element at this location to trigger. - long press: User performed a LONG PRESS at coordinates (x, y). Expect a context menu or selection state. - scroll: User SCROLLED <DIRECTION>. The content should move, revealing new items from the <direction> direction. - input text: User is TYPING the text: ‘<text>’. The focused input field MUST now contain this text. - openapp: System Context Switch: The user opened the app ‘<app>’. Show the home screen of this app. - navigate back: System Navigation: The user pressed BACK. Return to the previous screen. - navigatehome: System Navigation: The user pressed HOME. Show the Desktop. - wait: Action: WAIT. Keep the UI mostly unchanged unless loading completes. DIFFUSION CONDITIONING TEXT: Predict the next page state via image from this current screenshot using action description “desc” and action target “tgt” and relative coordinates “[x,y]”. - click / long press: (“click” / “long press”, “point(x, y)”) - scroll: (“scroll <direction>”, “screen”) - input text: (“input text: <text>”, “text field”) - openapp: (“open app: <app>”, “app icon”) - other: (<actiontype>, “screen”) 30 arXiv Preprint D.4EVALUATION AND SCORING PROMPTS The same prompt text is shared by all selectors so that they differ only in the images that follow each candidate. For the Code2World and diffusion selectors, these are the rendered next screens. For our model, they are fixed blank placeholders: no world model is run and no future screen is generated or perceived at inference, and the candidate scores are read directly from the index-token logits. The phrase “each followed by a world model’s prediction of the resulting screen” is therefore kept only to keep the prompt identical across selectors. Scoring Prompt IMAGES: [image: current screenshot] [image: prediction for candidate 0] [image: prediction for candidate 1] [image: prediction for candidate 2] (blank placeholder for our judge; rendered next screen for the Code2World and diffusion selectors) TEXT: This is the current mobile-GUI screen (first image). Goal: “goal”. Steps already completed: hist Candidate next actions, each followed by a world model’s prediction of the resulting screen: 0.candidate 0 1.candidate 1 2.candidate 2 Which candidate best advances the goal? Answer with ONLY the number (0–2). HISTORY FORMAT: -hist lists the last six own actions as “Step j: <action text>”, or “(none - this is the first step)”. - The logits of the index tokens are read at the last position; no text is generated. 31