Paper deep dive

AgentLens: Adaptive Visual Modalities for Human-Agent Interaction in Mobile GUI Agents

Jeonghyeon Kim, Byeongjun Joung, Junwon Lee, Joohyung Lee, Taehoon Min, Sunjae Lee

Year: 2026Venue: arXiv preprintArea: cs.HCType: PreprintEmbeddings: 96

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%

Last extracted: 4/26/2026, 5:27:18 PM

Summary

AgentLens is a mobile GUI agent system designed to improve human-agent interaction by addressing the limitations of foreground (high transparency, low multitasking) and background (high multitasking, low awareness) execution modes. Through iterative formative studies, the researchers identified a need for a hybrid model with just-in-time visual interaction. AgentLens implements this using three adaptive visual modalities: Full UI (complete app screen for context), Partial UI (task-relevant region for authenticity), and GenUI (LLM-generated interface for conciseness). The system leverages Android's Virtual Display to enable background execution with non-invasive overlays. A controlled study showed that AgentLens was preferred by 85.7% of participants and achieved high usability and adoption-intent scores.

Entities (8)

AgentLens · systemagent · 100%Android · platform · 100%Full UI · visual-modality · 100%GenUI · visual-modality · 100%GPT-5.4 · large-language-model · 100%M3A · base-system · 100%Partial UI · visual-modality · 100%Virtual Display · technologyabstraction · 100%

Relation Signals (6)

AgentLens → builtontopof → M3A

confidence 100% · implementing our agent prototype on top of the M3A system

AgentLens → leverages → Virtual Display

confidence 100% · uses Virtual Display to enable background execution

GenUI → poweredby → GPT-5.4

confidence 100% · The GUI overlay was pre-generated in HTML using GPT-5.4

AgentLens → usesmodality → Full UI

confidence 100% · AgentLens... adaptively uses three visual modalities... Full UI, Partial UI, and GenUI.

AgentLens → usesmodality → Partial UI

confidence 100% · AgentLens... adaptively uses three visual modalities... Full UI, Partial UI, and GenUI.

AgentLens → usesmodality → GenUI

confidence 100% · AgentLens... adaptively uses three visual modalities... Full UI, Partial UI, and GenUI.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Mobile GUI agents can automate smartphone tasks by interacting directly with app interfaces, but how they should communicate with users during execution remains underexplored. Existing systems rely on two extremes: foreground execution, which maximizes transparency but prevents multitasking, and background execution, which supports multitasking but provides little visual awareness. Through iterative formative studies, we found that users prefer a hybrid model with just-in-time visual interaction, but the most effective visualization modality depends on the task. Motivated by this, we present AgentLens, a mobile GUI agent that adaptively uses three visual modalities during human-agent interaction: Full UI, Partial UI, and GenUI. AgentLens extends a standard mobile agent with adaptive communication actions and uses Virtual Display to enable background execution with selective visual overlays. In a controlled study with 21 participants, AgentLens was preferred by 85.7% of participants and achieved the highest usability (1.94 Overall PSSUQ) and adoption-intent (6.43/7).

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

95,777 characters extracted from source content.

Expand or collapse full text

AgentLens: Adaptive Visual Modalities for Human–Agent Interaction in Mobile GUI Agents Jeonghyeon Kim Sungkyunkwan University (SKKU) Suwon, Republic of Korea jeonghyeon12@skku.edu Byeongjun Joung Sungkyunkwan University (SKKU) Suwon, Republic of Korea bjbj2580@skku.edu Junwon Lee Sungkyunkwan University (SKKU) Suwon, Republic of Korea wwwnsdnjs@skku.edu Joohyung Lee Sungkyunkwan University (SKKU) Suwon, Republic of Korea jhl72e@skku.edu Taehoon Min Sungkyunkwan University (SKKU) Suwon, Republic of Korea mth9428@skku.edu Sunjae Lee ∗ Sungkyunkwan University (SKKU) Suwon, Republic of Korea sunjae.lee@skku.edu A B C D Full UI: Show complete app screen for broader context Partial UI:Display only the relevant region for concise and authentic interaction Background Figure 1: Overview of AgentLens. Given a user request (“I’m hungry!”), AgentLensAoperates a delivery app in the background and adaptively selects among three visual modalities when user interaction is needed.BGenUI presents an LLM-generated interface when a concise, reformatted interaction is most effective.CPartial UI presents only the task-relevant region of the real app screen when authentic app content is needed but full-screen presentation would be unnecessarily intrusive.D Full UI presents the complete original app screen when broader visual or spatial context is required, such as for verifying high-stakes actions at checkout. All overlays appear non-invasively over the user’s ongoing activity (e.g., web browsing). Abstract Mobile GUI agents can automate smartphone tasks by interacting directly with app interfaces, but how they should communicate with users during execution remains underexplored. Existing systems ∗ Corresponding author. rely on two extremes: foreground execution, which maximizes transparency but prevents multitasking, and background execution, which supports multitasking but provides little visual awareness. Through iterative formative studies, we found that users prefer a hybrid model with just-in-time visual interaction, but the most effective visualization modality depends on the task. Motivated by arXiv:2604.20279v2 [cs.HC] 23 Apr 2026 Preprint. this, we present AgentLens, a mobile GUI agent that adaptively uses three visual modalities during human–agent interaction: Full UI, Partial UI, and GenUI. AgentLens extends a standard mobile agent with adaptive communication actions and uses Virtual Display to enable background execution with selective visual overlays. In a controlled study with 21 participants, AgentLens was preferred by 85.7% of participants and achieved the highest usability (1.94 Overall PSSUQ) and adoption-intent (6.43/7). 1 Introduction Mobile GUI agents—AI agents that operate smartphone applica- tions on behalf of users by interacting with graphical user inter- faces themselves—are rapidly emerging as a new mobile interaction paradigm. By perceiving on-screen content and executing touch- based actions such as tapping, scrolling, and typing, these agents can automate tasks across arbitrary mobile apps without requiring dedicated APIs or platform support. While mobile GUI agents hold the promise of transforming human-computer interaction, alleviating users from tedious and cumbersome smartphone operations, recent progress in this area has focused primarily on the agent-side of this pipeline–enhancing perception accuracy, augmenting reasoning capabilities, and boost- ing end-to-end task completion rates [10,26,27,32,37,40,47– 49,54–56]. However, comparatively little attention has been paid to the user-side of the interaction: how should a GUI agent commu- nicate its progress, intentions, and queries to the user who delegated the task? Although a small body of HCI research has begun to explore when agents should involve users—identifying points of interven- tion [24], modeling delegation boundaries [42], and studying human– agent turn-taking—the question of how (i.e., with what modality) the agent should present its intent to the user remains unexplored. Today, mobile GUI agents typically use one of two extremes in how they communicate their activity to the user. They either run entirely in the foreground [48], occupying the full screen and visually exposing every step of their app interaction, or operate completely in the background [2], hiding the target app from the screen and communicating with the user solely through voice or text. The foreground mode provides full transparency—users can observe every action the agent takes—but it prevents multitasking and forces users to passively watch a slow, sequential automation process. The background mode, on the other hand, supports mul- titasking and parallel activity, but lacks visual feedback, leaving users unaware of what the agent is doing. One promising direction to address this gap is to leverage Gener- ative UI (GenUI), a paradigm in which LLMs dynamically generate graphical user interface (GUI) to visually represent its intent to the user [30]. Applied to mobile GUI agents, GenUI could offer a middle ground between foreground and background modes by selectively generating and displaying a lightweight, non-invasive overlay UI only at critical moments, without occupying the full screen. However, our iterative Wizard-of-Oz formative studies revealed that while users found GenUI’s a hybrid model—background ex- ecution with minimally invasive visual overlays at critical deci- sion points—satisfying, many expressed trust concerns about LLM- generated interfaces. In particular, they worried that GenUI could hallucinate critical information such as prices, payment details, or menu options, especially in high-stakes tasks. In addition, partici- pants wanted the agent to adapt how it visualizes its intent based on the task context. These findings led us to a broader design insight: the key question is not simply whether to provide visual feedback, but what kind of visual representation is most appropriate for the task at hand. Guided by these findings, we present AgentLens, a mobile GUI agent system that adaptively employs three complementary visual modalities—Full UI, Partial UI, and GenUI—when human–agent interaction is needed. Full UI presents the live original app screen when broad visual or spatial context is needed; Partial UI shows only the task-relevant region of the real app to preserve authen- ticity while reducing distraction; and GenUI provides a generated interface when concise, reformatted interaction is more effective. Architecturally, AgentLens extends a conventional mobile GUI agent with an adaptive visual interaction layer. Specifically, we augment the agent’s action space with user-facingspeakandask actions, each paired with a four visualization option—voice only, full UI, partial UI, and generative UI. In addition, to enable back- ground execution of unmodified third-party apps and partial UI extraction from their interfaces, AgentLens leverages Android’s Virtual Display [19] abstraction to operate the target app on an invisible surface, while selectively mirroring its cropped interface regions onto AgentLens companion app’s popup overlays. We demonstrate the effectiveness of AgentLens by implementing its agent prototype on top of the M3A system [40], a simple yet effec- tive mobile GUI agent powered by GPT-5.4 [35], and its companion app on the Android platform 1 . We evaluated AgentLens through a controlled user study with 21 participants, comparing it against the two existing interaction extremes, Foreground and Background. The results show that AgentLens is highly usable and practical in real-world: 18 of 21 participants (85.7%) selected AgentLens as their first choice for daily use, and AgentLens achieved a high overall usability score, with 1.94 overall PSSUQ score (lower the better) and 6.43/7. adoption-intent score To the best of our knowledge, AgentLens is the first work to explore the design space of visual interaction modalities between users and mobile GUI agents, and to propose technical solutions for realizing this design in a real mobile environment. Specifically, this paper makes the following three key contributions to the HCI community: (1)Through iterative formative studies, we empirically uncover the underexplored design space of how mobile GUI agents should visually interact with users during task execution. (2)We derive three design principles for visual interaction in mo- bile GUI agents and instantiate them in AgentLens, a system that adaptively selects among Full UI, Partial UI, and GenUI to present task-critical information in a non-invasive, just-in-time manner. (3)Through a user evaluation with푁participants, we validate the effectiveness of adaptive visual feedback in mobile GUI agents and distill implications for designing future mobile human–AI interaction. 1 We will open-source our prototype implementation in the camera-ready version. 2 Preprint. 2 Background and Related Work 2.1 Mobile GUI Agents Mobile GUI Agents have evolved from simple rule-based automa- tion to sophisticated LLM-powered systems capable of understand- ing natural language instructions and interacting with complex mobile interfaces. Modern GUI agents typically operate through a perception–reasoning–action loop that translates user instructions into executable UI actions (e.g., clicks, scrolls, text inputs) across multiple steps: •Perception: The agent converts the current mobile screen into a structured representation that the underlying model can rea- son over. Depending on the system, this representation may be derived from visual markers [52], OCR [46], accessibility meta- data [15], or combinations of these signals. •Reasoning: Given the user instruction, and the perceived current screen, the model predicts the next UI action to execute. This stage is typically prompt-based and often incorporates systematic reasoning strategies such as ReAct [53] or few-shot prompting [4] to improve action grounding and task completion. •Action: Finally, the inferred action is translated into an actual input event on the device. This is commonly implemented either through ADB commands [40,49,50] or through accessibility service events [26, 27]. While this line of work has made rapid progress in improving execution accuracy, it has focused primarily on how agents interact with the GUI itself. By contrast, our work investigates how such agents should interact with the user during execution, particularly when the agent needs to inform, query, or request confirmation from the user while operating in the background. 2.2 Human–Agent Interaction in Mobile Environment Mobile GUI agents operate within a uniquely constrained environ- ment that demands distinct design considerations for human–agent interaction [20,21]. Current approaches either occupy the full dis- play in foreground mode [48,49,54] or run invisibly in background mode without visual feedback [2]. Research on supervisory control has shown that both extremes impose cognitive costs. Foreground execution reduces users to passively watching each action [11], while background execution without awareness cues forces users to tolerate suboptimal behavior or revert to active checking [36]. Recent industry efforts, such as Google’s Gemini assistant [17], have attempted to address this issue by combining background execution with lightweight status notifications. However, whenever the assistant needs to interact with the user (e.g., asking the user for input or confirmation), it still falls back to a full-screen takeover, switching between two interaction extremes rather than bridging the gap between them. To address this challenge, we present, to the best of our knowl- edge, the first non-invasive human–agent interaction model for mobile GUI agents that combines the strengths of foreground and background interaction. By operating in the background by default and surfacing minimally invasive visual overlays only at critical decision points, our approach enables proactive agent–user inter- action without disrupting the user’s ongoing activity. 2.3 Visual Modalities for Human–Agent Interaction In the broader domain of AI agent design, various approaches have been explored for the communication between agents and users. The most prevalent modality remains the text-based chat inter- face [1,5,34]. However, text-based interfaces face fundamental limitations when agents must convey complex action states or collect multiple user input [30]. To address these shortcomings, a growing body of work has turned to Generative UI (GenUI), which leverages LLM to dynamically produce tailored interface compo- nents at runtime [7,14,45]. GenUI has since been explored across diverse HCI contexts, from malleable task-oriented workspaces [6] to ephemeral scaffolding within coding workflows [9]. Despite this promise, applying GenUI to more privileged, high- stakes agents such as mobile GUI agents raises various concerns, as LLM-generated interfaces remain susceptible to hallucinations and inconsistent adherence to user instructions [8,39,51]. Our work mitigates these risks by retrieving and displaying UI elements directly from the underlying app, eliminating the need for the generation in the first place, and restricting GenUI to low-stakes tasks. 2.4 Partial UI Migration The idea of selectively displaying or migrating partial UI elements from one application to another has been explored in prior interface system researches. Early desktop and web systems demonstrated this idea by extracting and migrating atomic UI components out- side their original application context [12,43,44,57], and subse- quent work in mobile environments, such as FLUID [29,33] and A-Mash [28], extended this paradigm to small-screen devices. A substantial body of empirical work further supports the benefits of this approach: studies on peripheral displays have shown that partial information satisfies most awareness needs without requir- ing a full context switch [23,38]; research on adaptive interfaces confirms that selectively surfacing relevant functionality improves user satisfaction [13,41]. In light of these findings, AgentLens lever- ages Partial UI display as one of the three primary communication channels between the agent and the user. 3 Formative Studies To design an effective human–agent interaction model for mo- bile GUI agents, we conducted iterative formative studies using a Wizard-of-Oz (WoZ) methodology. Our initial hypothesis was that leveraging Generative UI (GenUI) could close the gap between the two dominant extremes (i.e., fully foreground and fully back- ground), generating appropriate visual feedback at critical decision moments on the fly. However, as we detail below, the studies did not converge on GenUI as a single solution. Instead, they shifted the core design question from whether such visual feedback is effective to how such feedback should be represented. To investigate this design space, we conducted three formative studies addressing the following research questions: •RQ1 (Formative Study 1): What interaction model do users want from a mobile GUI agent? • RQ2 (Formative Study 2): Does a GenUI prototype satisfy those needs with sufficient trust? 3 Preprint. •RQ3 (Formative Study 3): What visual presentation modality is most appropriate under different task conditions? 3.1 Study 1: Exploring Agent Interaction Modes 3.1.1 Procedure. To understand how users prefer to interact with a mobile GUI agent while delegating a task, we recruited 10 partici- pants (푃1–푃10, aged 18–27) for a within-subject WoZ study. The study involved two experimenters: one remained in the same room as the participant and conducted the interview, while the other stayed in a separate room and remotely controlled the participant’s smartphone to simulate the agent’s behavior. To minimize participant bias and avoid anchoring them to our de- sign hypothesis, participants first experienced two existing interac- tion modes—Foreground (FG) and Background (BG)—in counterbal- anced order within a food-delivery ordering scenario. In Foreground (FG) mode, the agent occupied the full screen and visually exposed its entire app manipulation process. In Background (BG) mode, the agent operated invisibly behind the screen. In both conditions, par- ticipants interacted with the agent via voice only. To reflect the latency of current LLM-based mobile GUI agents, the WoZ operator inserted an approximately 30-second delay at each step transition, based on reported per-step inference times of around 20–30 sec- onds. After each condition, we conducted a semi-structured interview about the participant’s experience. After both conditions, we asked an open-ended question “What would your ideal agent look like?” to elicit participants’ own interaction visions before presenting our GenUI solution. 3.1.2 Findings. Our thematic analysis revealed that neither exist- ing mode was satisfactory on its own. More broadly, users expe- rienced a fundamental dilemma between trust and usability. At the same time, participants spontaneously converged on a hybrid interaction model that combines the strengths of FG and BG. F1-1. Foreground builds trust but nobody would use it in everyday life. All 10 participants reported trusting the agent in FG mode because they could visually verify its actions. Yet not a single participant chose FG as a desirable mode for everyday use. Nine participants found the full-screen takeover unacceptable because it blocked all other phone activity, and seven reported frustration at having to watch the agent’s slow, step-by-step operation. As P10 put it, “I could just tap through it myself faster. Watching the agent go through each voice command, then process, then act... that loop takes too long.” This reveals a key paradox in FG-based GUI agents: fully taking over the user’s phone may increase transparency, but it can also make phone use itself more cumbersome, which it intends to alleviate in the first place. F1-2. Background is convenient, but introduces anxiety and cognitive overload. BG mode was overwhelmingly preferred for its multitasking support (n=9), but it also induced anxiety along two distinct dimensions. Most acutely, nine participants expressed concern over irreversible tasks—particularly payments, where au- ditory confirmation alone felt insufficient; P9 stated, “Messing up an email is fine, but a wrong payment is very hard to undo—I need to at least see the final screen before it goes through.” At the same time, six participants reported cognitive overload from voice-only information delivery: when the agent read out multiple options sequentially, participants struggled to retain and compare them, with P4 and P5 noting, “When there are lots of options, I just can’t remember them.” This demonstrates that the voice is not the most suitable form of human–AI interaction when it comes to mobile GUI Agent. F1-3. Users spontaneously converge on a hybrid model. When asked to describe their ideal agent, without any alternative be- ing suggested by the researchers, all 10 participants independently proposed a middle ground that combines FG’s visual trust with BG’s multitasking convenience. Their responses converged along three core requirements: i) minimally invasive UI : non-intrusive overlays such as small popups (P2, P3), a Dynamic-Island-style element (P4), or toast notifications (P10) that do not interrupt ongoing activity; i) selective visibility: surfacing only relevant information rather than the full app screen(P1, P3, P4, P5, P8, P10); and i) just-in-time intervention: visual UI appearing only at critical moments requiring user input (P2, P3, P6, P7, P8, P10) 3.2 Study 2: Validating GenUI approach 3.2.1 Procedure. Immediately after Study 1, the same 10 partici- pants experienced a WoZ prototype implementing our initial design hypothesis: Generative UI based mobile GUI agent. In this condi- tion, the agent ran in the background by default, but whenever user involvement was needed, it generated and presented a lightweight GUI overlay accompanied by a short voice notification. The GUI overlay was pre-generated in HTML using GPT-5.4, following a common Generative UI workflow [30]. Participants could respond either by tapping the overlay or through voice. After the session, we conducted semi-structured interviews eval- uating the prototype’s acceptability, and their perception of the generated interface. We then explicitly disclosed that the interface had been generated by an LLM, and probed how this knowledge affected their assessment. 3.2.2 Findings. F2-1. The hybrid interaction model is satisfactory. 9 of 10 participants selected GenUI as their preferred mode for daily use over both FG and BG. Participants repeatedly described the GenUI mode as combining the best aspects of the earlier two extremes: it preserves BG’s multitasking support while recovering FG’s visual reassurance at critical moments. P9 noted that the overlay “cut the long voice listings down to a quick glance and a tap,” and P2 valued being able to “see the actual options and choose while still doing something else.” F2-2. GenUI introduces a hallucination problem. However, when we disclosed that the overlay UIs were dynamically generated by an LLM rather than taken directly from the original app, reac- tions diverged sharply. A majority of participants (푛=7) expressed concern. Three participants expressed strong distrust “What if it makes up a menu item that doesn’t exist? Then I’d be charged for the wrong thing” (P5); “If I experienced a hallucination even once, I’d never trust it again” (P4). Four expressed conditional concern, finding GenUI acceptable for simple tasks but not for critical con- texts such as finance. In contrast, three participants with relatively higher technical understanding (P3, P8, P9) expressed compara- tively low concern. P8 explained, “If it is generating the UI with the data extracted from the app, that should be reliable enough.” These 4 Preprint. findings suggest that while the hybrid interaction structure itself was well received, the generative nature of GenUI raised a distinct trust problem. F2-3. GenUI has unique strengths, but the question of what to show remains open. Although participants acknowledged that the Hybrid + GenUI approach has clear strengths, hallucination concerns led most to want actual app content available as an alter- native. Yet when asked what they would prefer instead of GenUI, opinions did not converge: four said that the system should show only the relevant region of the real app, two remained with GenUI, and two said it depends on the context. These results left us with a new design question: in what way should the agent visually interact with the user? This question directly motivated our next study. 3.3 Study 3: Comparing UI Modalities 3.3.1 Procedure. To resolve the open question left by Study 2, we designed three visual modalities that could each be presented within the same minimally invasive overlay at decision-critical moments: Full UI, Partial UI, and GenUI. Full UI displays the app’s entire screen on the overlay; Partial UI shows only the task-relevant region from the app screen; and Gen UI reconstructs the interface based on the information need to be. The complete screenshots for all scenarios and modalities are provided in Appendix B. We recruited a new cohort of 10 participants (푃11–푃20, aged 18–27) to directly compare these three modalities across six task scenarios: food delivery, music streaming, online shopping, weather, financial account inquiry, and taxi hailing. For each scenario, par- ticipants ranked the three methods and explained their reasoning in semi-structured interviews. The order of scenarios and the order of the three presentation methods within each scenario were fully counterbalanced across participants. 3.3.2 Findings. F3-1. No single modality dominates. Across 60 total ratings (10 participants×6 scenarios), no single modality dominated. First- place preferences were distributed relatively evenly: Full UI, 31.7% (19/60); Partial UI, 36.7% (22/60); and GenUI, 31.7% (19/60). These results suggest that there is no one-size-fits-all modality strategy. F3-2. Task characteristics determine modality preference. Instead, preferences varied systematically by scenario. Full UI was preferred when broad app context or spatial layout mattered, such as checking the weather or calling a taxi. P6 explained: “When I call a taxi, I need to check the map, price, and available options before making a decision.” Partial UI was favored when users needed trustworthy, precise information but not the entire app. P2 noted: “I like how it shows only the necessary information directly from the app. I don’t feel comfortable with AI regenerating my financial information.” GenUI was preferred when the task was low-risk and the original app’s UI was seen as suboptimal. P3 commented (under music app scenario): “I always thought that album covers are unnecessary. The clean list of titles is much better.” F3-3. Agent needs to adaptively switch between modalities. At the same time, the clearest consensus across participants (푛=8) was that agents should adaptively switch presentation modality depending on the task, rather than relying on a single fixed UI form; “I would want the agent to select the best modality for me” (P7), “I hope the agent handles it (modality) on its own based on the task, preference, and context” (P6). 4 AgentLens Our formative studies revealed that users want a mobile GUI agent that operates in the background, surfaces visual information only at decision-critical moments, and adapts the form of that information to the task at hand. To the best of our knowledge, no existing mobile GUI agent supports such adaptive, multi-modal visual interaction with the user. In this section, we first present the design principles distilled from our formative findings, then describe the architecture of AgentLens. 4.1 Design Principles DP1. Background execution with non-invasive visual intervention. The agent should run in the background by default. When visual information is needed, it should be presented through a lightweight overlay rather than a full-screen takeover (F1-1, F1-3, F2-1). DP2. Just-in-time intervention at decision-critical moments. Visual overlays should appear only when the user’s input, confirmation, or awareness is required—such as when selecting among multiple options, approving an irreversible action, or checking a task result (F1-2, F1-3, F2-1). DP3. Adaptive selection of UI presentation modality. No single visual presentation method suits all tasks. The agent should dynam- ically select among Full UI, Partial UI, and GenUI based on the task context (F2-3, F3-1, F3-2). 4.2 Agent Overview Building on the standard pipeline architecture of mobile GUI agents introduced in § 2, AgentLens extends a conventional GUI agent with an adaptive visual interaction layer (see Figure 2). In partic- ular, while the core perception–reasoning–action loop remains un- changed, AgentLens augments the agent’s action space with explicit mechanisms for interacting with the user during task execution. Specifically, in addition to standard device-control actions such as taps, scrolls, and text input, AgentLens introduces two user- facing action types: speak and ask. The speak action is used when the agent needs to notify, summarize, or explain information to the user without requiring a response. The ask action is used when the agent requires user input, confirmation, or selection before it can proceed. Both speak and ask are associated with a visualization option parameter that determines how the relevant information should be presented. The agent can choose among the following four visualization methods: • none: The message is delivered through voice only. • show_app: Presents the full app screen to the user (Full UI). • show_element : Extracts and displays only the task-relevant re- gion of the target app screen (Partial UI). • generate_ui: Presents an LLM-generated interface tailored to the current interaction need (GenUI). When the agent outputs a speak or ask action with a visual- ization option other thannone, the AgentLens companion mobile application renders the corresponding UI as a popup overlay, as shown in Figure 3. We engineered the agent’s prompt to reflect the design principles, so that the agent invokes these actions only at 5 Preprint. Figure 2: AgentLens System Architecture. AgentLens uses Virtual Display to operate third-party apps in the background, while displaying the visual feedback (i.e., overlay) on the physical display. decision-critical moments and selects an appropriate visualization mode based on the findings from our formative study (see Appen- dix A for the full prompt). 4.3 AgentLens System Design Realizing the agent described above requires addressing two tech- nical challenges. First, the system must be able to operate arbitrary third-party mobile app in the background while keeping them fully functional. Second, it must extract and present only specific parts of an application’s interface in order to support Partial UI. C1. Background App Execution via Virtual Display. A key design goal of AgentLens is to operate apps without taking over the user’s screen. However, this is non-trivial in mobile operating systems, as applications moved to the background are typically deprioritized—stop rendering, become idle, or cease to receive input events—for resource efficiency, The conventional approach to enable such non-invasive assistant interaction is to integrate a target app with a vendor-provided system-level assistant (e.g., Google Assistant, Siri, Bixby) through dedicated APIs. However, this requires modifying the app’s source code, which contradicts the fundamental premise of GUI agents: automating any third-party app transparently. Worse yet, such integration is limited to a few pre-defined categories of apps such as messaging, alarms, or navigation [16, 18]. To address this challenge, AgentLens leverages the Virtual Dis- play abstraction that exists on mobile platforms under different names (e.g., VirtualDisplay on Android [19], UIWindow on iOS [3]). A Virtual Display is a software-created display surface that behaves like a secondary screen that exists only within the system. For our purposes, they can be utilized as an invisible and isolated execution environment in which an app can be launched without interfering with the user’s physical display. Specifically, when AgentLens receives a request to launch a target app, rather than opening it on the physical display and taking over the screen, it creates a Virtual Display and launches the app within it. The agent’s perception–action loop then operates over this virtualized environment: screenshots are captured from the Virtual Display, and input events (e.g., taps, scrolls) are injected by forwarding them to the Virtual Display. This gives AgentLens a fully functional, isolated execution environment that operates entirely in the background (see Figure 2). We note that this mechanism may appear to pose a security con- cern, as it could in principle allow a malicious agent to silently ma- nipulate a user’s apps. In practice, however, launching a third-party app on a Virtual Display currently requires ADB (Android Debug Bridge) commands, which are accessible only when the device is explicitly connected to a desktop computer with developer options enabled. While our implementation is a research prototype, a pro- duction deployment could be realized through a system-privileged mobile assistant app with appropriate platform permissions. C2. Partial UI Visualization through Cropped Mirroring. To support the Partial UI modality, the system must be able to extract and present only a specific UI element from the target app. Although prior work on UI migration has explored splitting or relocating individual UI elements across devices or applications [28,29,33], these approaches require either modifications to the app or to the Android operating system itself. AgentLens addresses this by cropping the task-relevant region from the Virtual Display and mirroring it onto the overlay surface. To specify which region to display, AgentLens follows the com- mon practice of feeding the LLM the app’s accessibility node tree alongside the screenshot and indexing each element with a numeric identifier, so that the agent can specify UI elements by index rather than by pixel coordinates [22, 40, 49]. Doing so requires preprocessing the accessibility node tree into a LLM-readable form [22,27,40,49]. The most common strategy is to flatten the accessibility tree into a list of leaf nodes, leaving only the visible GUI elements (e.g., buttons, text fields). For AgentLens, however, this is problematic, as it discards hierarchical information essential for AgentLens to identify semantically coherent groups. In many cases, a group of sibling UIs collectively serves a single purpose. For example, Partial UI in Figure 3 encompasses multiple buttons, text labels, and icon images. Without hierarchical struc- ture, AgentLens cannot determine which elements are semantically grouped. To preserve this grouping, we adopt the DOM-based parsing ap- proach from MobileGPT [27], which translates the raw accessibility 6 Preprint. Figure 3: Example screenshots of AgentLens using Full UI, Partial UI, and Gen UI visual interaction tree into an HTML-like representation that retains the hierarchi- cal relationships among elements using layout containers such as <div>. This allows AgentLens to select not only individual elements but also semantically meaningful groups. The agent can also spec- ify multiple non-contiguous element indices when the relevant information spans disparate regions of the screen. 4.4 Visualization of Each Modality Full UI (show_app). For the Full UI modality, AgentLens mirrors the entire content of the Virtual Display onto a popup overlay on the user’s physical screen. When the user interacts with this overlay (e.g., via touch), the touch coordinates are forwarded to the corresponding location on the Virtual Display, enabling direct interaction with the live app. Partial UI (show_element). Upon receiving the agent’s element selection, the AgentLens companion app retrieves the correspond- ing bounding rectangles from the accessibility tree. It then crops the relevant region(s) of the Virtual Display and mirrors them onto the popup overlay with appropriate size scaling, producing the effect of a focused, partial GUI overlay. As with Full UI, touch events on the overlay are forwarded to the corresponding coordinates on the Virtual Display. GenUI (generate_ui). For the GenUI modality, AgentLens takes one additional parameter from the agent: a natural-language speci- fication of the information to be included in the generated interface. This instruction is forwarded to a dedicated GenUI Agent—a sepa- rate LLM instance that produces an HTML-based interface tailored to the specified content. We deliberately decoupled the GUI gen- eration process from the main agent because we found that when the GenUI Agent is exposed to the original app screen, it tends to reproduce the existing design rather than creatively restructure the information. The generated HTML is then rendered onto the popup overlay with a disclosure “This UI has been generated by AI.” 4.5 Implementation We implemented AgentLens on top of M3A [40], a simple yet ef- fective mobile GUI agent. We extended M3A’s action space by aug- menting its system prompt with the speak and ask actions, and modified its backend Python server to parse and dispatch these new action types. The AgentLens companion mobile application was built in Kotlin on Android. All communication between the agent server and the mobile application is conducted through ADB commands. All LLM inference—for both the main GUI agent and the GenUI Agent—was performed using GPT-5.4. 5 Evaluation 5.1 Performance Evaluation of Adaptive Modality Selection To evaluate how well AgentLens selects appropriate visualization modalities when integrated with a live LLM (gpt-5.4), we conducted two complementary studies. 5.1.1 Study 1: Alignment with Human Judgment. Our first study ex- amined whether the LLM’s visualization choices align with human judgment. Specifically, we evaluated whether the agent’s selection ofshow_app,show_element, andgenerate_uiataskandspeak steps aligns with that of human annotators. Dataset construction. Existing mobile GUI agent benchmarks are designed primarily to evaluate GUI interaction (e.g., click, scroll, type) and rarely include user-facing actions needed for AgentLens. Therefore, to construct an evaluation set, we examined multiple existing datasets and extracted tasks that involve explicit interaction with the user. Among existing resources, the MobileGPT dataset [27] includes askactions. Forspeakbehaviors, we usedanswer_useractions from AndroidWorld [40], andfinish_taskactions with user-directed messages from MobiBench [22]. After filtering out tasks that were incorrectly annotated or not reproducible in our environment, we obtained a total of 43 tasks across the three benchmarks, each con- taining at least one ask or speak step. Procedure. For each task instance, we ran AgentLens with a live LLM agent and recorded its output at everyaskorspeakstep, including both the generated message and the selected visualiza- tion option (none,show_app,show_element, orgenerate_ui). We then recruited three independent annotators (A1–A3) from a local community board to generate human judgments for the same task instances. Each annotator was compensated at approximately $15 USD per hour. Annotators were provided with a brief guideline describing each visualization option and the contexts in which it might be most effective. Then, they were asked to select the option they considered most appropriate for each ask or speak step given the surround- ing task context. We compared the LLM’s selections against these human judgments. Results. The LLM’s visualization choices did not align strongly with those of the human annotators. However, inter-annotator agreement was also notably low. In fact, the mean Cohen’s kappa(휅) between the LLM and each human annotator (휅=0.285) was com- parable to, or even slightly higher than, the mean pairwise휅among the three human annotators themselves (휅=0.238). Interestingly, 7 Preprint. Table 1: Distribution of preferred visualization option se- lected by each annotator and the gpt-5.4 across 44 tasks. show_element show_app generate_ui A112 (27%)19 (43%)13 (30%) A224 (55%)5 (11%)15 (34%) A329 (66%)0 (0%)15 (34%) LLM15 (34%)6 (14%)22 (52%) among the three annotators, A1 and A2 exhibited similar tenden- cies, with their preference distributed relatively evenly across the three visual modalities. In contrast, A3 and the LLM shared a simi- lar pattern (with highest휅=0.334), rarely selectingshow_appand favoring show_element. Overall, these results indicate that the LLM’s behavior was not simply incorrect, but reflected one plausible preference among many. Adaptive visualization selection is better understood as a preference-driven decision than as a single ground-truth prediction problem. We also observed that the LLM’s preference can be read- ily adjusted through prompt engineering, suggesting that future systems should adapt modality selection to individual users rather than enforce one fixed policy. 5.1.2 Study 2: User Satisfaction with LLM-Selected Visual Feed- back. Given that the choice of visual modality is not a single-label question, our second study measured whether the resulting visual feedback is nevertheless perceived as appropriate and satisfactory in its context. Procedure. We recruited five additional participants to evaluate the visual overlays generated by AgentLens on the same 43 task instances. For each task, participants were shown the task descrip- tion, a screenshot of the AgentLens overlay as rendered by the LLM’s selected visualization option, and the message generated alongside. Participants were instructed to imagine that they had delegated the task to the agent and were presented with this overlay during execution. They then rated their perceived satisfaction on a 7-point scale: 1 (very bad), 2 (bad), 3 (somewhat bad), 4 (neutral), 5 (acceptable), 6 (good), 7 (very good). Results. On average, bothshow_elementandshow_appreceived a mean score of 5.03 (std=1.31 and 1.47, respectively), correspond- ing to acceptable satisfaction.generate_uireceived a substantially higher mean score of 6.39 (std=0.91). These results suggest that, although the AgentLens did not always select the optimal choices, its visual feedback was generally perceived as appropriate. The strong performance of GenUI can be partly attributed to the composition of our dataset. A substantial portion of our dataset was derived by convertingfinish_task(“final_message”)ac- tions intospeak(“message”)actions. In these cases, the agents are typically required to summarize the progress of the task, which GenUI is specialized to, as it can freely compose and restructure information from multiple steps into a single coherent interface. The results also revealed clear room for improvement. We ob- served that when selectingshow_elementoption, the LLM tended to pick a minimal set of UI elements. However, users generally preferred an abundance of information over a scarce presentation, which led to several low-rated cases. While there are multiple ap- proaches to address such suboptimal behavior—including prompt engineering and model fine-tuning—improving the LLM’s intrinsic Table 2: Task scenarios across four mobile applications. AppTask User RequestSteps Vis Modalities Delivery S1Order a Crispy Fried Chicken63 P, P, F S2Show trending stores21 G Banking S1Send $3 to Alice63 P, F, P S2Check account balance21 P To-Do S1Show today’s tasks21 G S2Summarize weekly exercise91 G Mail S1Summarize unread emails51 G S2Send an email to Bob21 F Steps = # of atomic actions in the task; Vis = # of steps that involves visual feedback F = Full UI, P = Partial UI, G = GenUI. performance is outside the main scope of this paper. We therefore leave such optimization to future work (see § 6). 5.2 User Study To evaluate the user experience of AgentLens and its proposed design, we conducted a controlled user study comparing AgentLens against the two existing interaction extremes: Foreground (FG) and Background (BG). The study was designed to address the follow- ing research questions: (RQ1) Is AgentLens’s interaction design effective and usable in practice? (RQ2) Does AgentLens’s adaptive visual feedback resolve the trust–usability trade-off identified in our formative studies? and (RQ3) How well does AgentLens’s technical design realize the design principles derived from our formative studies? 5.2.1 Study Setup. Our goal in this study was to evaluate the pro- posed interaction paradigm. To isolate its effect from confounding factors such as model failures, app-side nondeterminism, and dy- namic content changes, we constructed a tightly controlled experi- mental environment in three ways. First, we implemented replica versions of the target apps to eliminate unpredictable external factors such as pop-ups, ads, and network delays. Second, we pre- configured the GUI agent to follow predefined action paths rather than generating actions online, to prevent task execution failure. Third, we fixed the visualization option for each interaction point in advance to eliminate nondeterminism in modality selection. To- gether, these controls ensured that participants’ responses reflected the interaction design itself rather than incidental failures of the underlying model. 5.2.2 Scenarios. We designed eight task scenarios spanning four mobile applications—To-Do, Mail, Delivery, and Banking—with two scenarios per app ( Table 2). The set was constructed to cover a range of everyday mobile tasks with varying levels of risk, reversibility, and information complexity. In particular, the scenarios included both low-risk tasks (e.g., checking and organizing information) and high-risk tasks involving irreversible outcomes such as payments and money transfers. 5.2.3 Participants. We recruited 21 participants (P1–P21; 14 male, 7 female; aged 18–27,푀=20.7) through an online university com- munity. A pre-study questionnaire (7-point Likert scale) indicated that participants had moderate familiarity with AI automation tools (푀= 4.95, 푆퐷= 2.20) but limited experience with voice assistants 8 Preprint. ForegroundBackgroundAgentLens 0 5 10 15 20 Count 0 3 18 (a) 1st-place ranking ForegroundBackgroundAgentLens 1 2 3 4 5 6 7 7-point Likert *** *** (b) Adoption intent 0255075100% To-Do Mail Banking Delivery 14% 43% 14% 24% 81% 62% 57% 90% (c) Domain preference ForegroundBackgroundAgentLens Figure 4: (a) First-choice ranking for daily use. (b) Self-reported adoption intent for personal smartphone use (c) Preferred condition per app. (7-point Likert; error bars indicate±1 SD). (푀=2.52,푆퐷=2.09). Each participant was compensated∼$15 USD per hour. 5.2.4 Procedure. Each session began with a brief introduction and a pre-study questionnaire. Participants then received a short tuto- rial on the study interface and the three experimental conditions. After the tutorial, participants completed all eight scenarios under each of the three conditions—Foreground (FG), Background (BG), and AgentLens—in counterbalanced order. In FG, the agent operated the target app directly on the user’s physical screen; participants could observe every action step but could not perform any other activity. In BG, the agent ran entirely behind the screen and commu- nicated results through voice only, while users multi-tasked freely. In AgentLens, the agent ran in the background and surfaced popup overlays only at decision-critical moments, adaptively selecting among Full UI, Partial UI, and GenUI based on task context. To evaluate non-intrusiveness under realistic multitasking con- ditions, participants performed free web browsing as a secondary activity during the BG and AgentLens conditions. After completing each condition, participants filled out the Post-Study System Us- ability Questionnaire(PSSUQ) [31] and four custom 7-point Likert scale questions assessing awareness, non-intrusiveness, perceived control, and trust (Table 3). To maintain consistency with other Likert-scale measures used in our study, we reversed the PSSUQ response scale so that 1 = strongly disagree and 7 = strongly agree, with higher scores indicating greater satisfaction. At the end of the session, we conducted a semi-structured interview probing participants’ overall preferences, trust perceptions, and reactions to the different visual presentation strategies. Each session lasted approximately 70 minutes. 5.2.5 Results and Findings. We organize the results around our three research questions. RQ1: AgentLens’s Interaction Design Is Effective in Practice. Our re- sults on overall usability indicate that participants found AgentLens to be highly usable in practice. In the overall preference ranking (Figure 4a), 18 of 21 participants (85.7%) selected AgentLens as their first choice for daily use, with only 3 selecting BG and none selecting FG. This preference was consistent across task domains (Figure 4c): AgentLens was the dominant choice for Delivery (90%), To-Do (81%), and Mail (62%), while Banking (high-risk tasks) was the only do- main where Foreground (43%) competed against AgentLens (57%). Even so, AgentLens maintained the majority (57%) despite operating in the background, suggesting that just-in-time visual intervention System Usefulness Information Quality Interface Quality Overall 1 2 3 4 5 6 7 Mean score (7-point Likert) ForegroundBackgroundAgentLens Figure 5: Post-Study System Usability Questionnaire (PSSUQ) scores across three conditions (in reverted scale;±1 SD). can address trust concerns even in high-stakes contexts—as P10 put it, “AgentLens only shows what’s needed, it is basically a better version of Foreground.” Adoption intent (Figure 4b) showed a similar pattern. Participants’ willingness to use AgentLens in daily life sig- nificantly exceeded that of both baselines (Friedman휒 2 (2)=29.04, 푝< .001,푊= .691; AgentLens vs. FG:푝< .001; AgentLens vs. BG: 푝< .001), indicating that AgentLens holds practical potential for real-world adoption as a primary interaction paradigm for mobile GUI agents. Participants’ qualitative feedback further explains this strong preference. Thirteen participants explicitly described AgentLens as combining the key advantages of FG and BG while eliminating their respective drawbacks; “AgentLens was the best because it resolved all the drawbacks of both FG and BG” (P4). Participants also directly commented on its high-usability; “Across all scenarios, AgentLens was dominantly most the comfortable one” (P15), “Overall, AgentLens was the most convenient.” (P16). Beyond general impressions, participants also pointed out spe- cific interaction mechanics that they found effective. Twelve par- ticipants highlighted the value of just-in-time visual intervention. As P1 explained, “seeing the actual app screen right before sending money or an email made me feel fully in control.” Participants also valued being able to interact with the app’s GUI rather than dele- gating the task completely to the agent. As P4 noted, AgentLens felt “trustworthy enough, since it lets me handle the sensitive parts myself.” Taken together, these responses suggest that participants did not simply prefer AgentLens over FG and BG, but found its novel design effective and satisfying in actual use. RQ2: AgentLens Resolves the Trust–Usability Trade-off. To un- derstand what drives AgentLens’s strong usability, we examine how it performs on the specific dimensions where FG and BG 9 Preprint. each fall short. In line with our overall usability results, AgentLens achieved the highest overall PSSUQ score ( Figure 5). More no- tably, AgentLens scored highest across all subscales, simultane- ously surpassing Foreground on transparency-related dimensions (Information Quality and Interface Quality) and Background on usability-related dimensions (System Usefulness). Our custom 7-point Likert-scale measures (Table 3) show the same pattern. Across all four criteria—awareness, non-intrusiveness, perceived control, and trust—AgentLens achieved the highest or near-highest score on every item. While FG and BG each showed a clear trade-off between transparency-related qualities (Awareness, Control, Trust) and usability (Non-intrusiveness), AgentLens exhib- ited no such tension, maintaining consistently high scores across all dimensions. Taken together, these quantitative results show that AgentLens not only resolves the trust–usability trade-off, but outperforms each baseline on its own strongest dimension, suggesting that AgentLens is not simply a middle ground between two extremes, but a clear improvement over both existing approaches. RQ3: AgentLens’s System Design Successfully Realizes All Three Design Principles. Participant comments across all eight scenarios confirm that our AgentLens prototype faithfully delivers each of the three design principles derived from our formative studies. DP1. Background execution with non-invasive visual inter- vention. Participants repeatedly valued that AgentLens allowed them to continue their ongoing activity while selectively surfacing visual UI only when needed: “AgentLens showed the screen only when necessary, while not interfering with what I was originally do- ing,” (P4); “I could keep doing my own thing while the AI handled its side of things” (P8). DP2. Just-in-time intervention at decision-critical moments. Participants strongly endorsed AgentLens’s ability to display visual cues at decision-critical moments. “Showing the transfer completion screen and letting me confirm before sending the email is what made me trust it” (P1); “It shows password entry, the transfer button, account selection, and makes you do all those yourself. That’s what makes it trustworthy enough” (P4). DP3. Adaptive selection of visual modality. Participants also appreciated how AgentLens can change its UI form dynamically across contexts. “It adjusted what it showed depending on the app and the situation, which made it convenient” (P19); “Instead of showing the full screen every time, it showed just the relevant part—that’s what kept it from getting in the way” (P21). 5.2.6 Additional Findings. Beyond the three research questions, our post-study interviews revealed several additional findings with implications for future system design. Preference reflects users’ mental models of the agent, not just us- ability scores. The three participants who ultimately preferred BG did not necessarily reject AgentLens; rather, they held different mental models of what an ideal agent should be. P3 and P12 priori- tized uninterrupted multitasking above all else: “When the overlay occupies half the screen, I feel interrupted” (P3), “If a popup suddenly covers what I was doing, it would be inconvenient” (P12). By contrast, P5 was skeptical about using agents in general: “I would not delegate tasks to an agent, but if I have to, I would only do for simple tasks” (P5). Yet all three still acknowledged the value of AgentLens: “it Table 3: Custom Likert scale results (7-point) measuring five dimensions of the trust–disruption trade-off. Dimension Foreground BackgroundAgentLens Awareness6.574.956.10 Non-intrusive3.766.105.71 Control5.294.435.57 Trust6.195.056.24 (AgentLens) does feel more reliable since I see the visuals” (P3), “If I had to use an agent in daily life, I would choose AgentLens because it feels safer” (P5), and “If the task involves money or finance, I would prefer AgentLens” (P12). These cases suggest that user’s preference for agent design is shaped not only by design itself, but also by users’ broader expectations of the AI agent. Trust depends not only on what is shown, but on who acts. Par- ticipants consistently reported greater trust when visual overlay enabled direct user manipulation. They trusted the system more when they themselves entered passwords, selected accounts, or confirmed final actions through visualized UI (Full UI and Partial UI), rather than having the agent complete those steps for them (P1, P4, P8, P10, P19). As P1 noted, “I felt in control because I was the one who pressed the final send button.” This suggests that visual interaction between the agent and the user should consider not only what information to surface, but also which actions to reserve for the user. With this respect, AgentLens’s visual overlay can also function as a handoff mechanism that reallocates the control back to the user at critical steps. Device form factor as a design consideration. Several participants noted that AgentLens’s overlay occluded too much of their ongoing activity on the smartphone’s limited screen. P3 framed this not as a design flaw but as a physical constraint: “On a phone, I’d prefer BG, but if the screen were bigger, I’d switch to AgentLens.” This implies that the optimal form of visual feedback depends not only on task context but also on the physical display environment. 6 Discussion 6.1 Limitations Dependence on structured screen representations. AgentLens’s Par- tial UI modality relies on the accessibility tree to identify and crop task-relevant UI elements by index. In image-only environments where such structured screen information is unavailable (e.g., iOS, Unity) the system must fall back to inferring pixel coordinates di- rectly from screenshots. In our testing, state-of-the-art multimodal LLMs (e.g., GPT-5.4) can approximate element boundaries from screenshots, but their accuracy remains substantially lower than index-based selection. A promising mitigation is to incorporate dedicated object detection models [25,46] as a preprocessing step, then expose those regions to the agent as indexed candidates. Security implications of background app control. AgentLens’s abil- ity to operate third-party apps in the background can, in principle, be abused for silently manipulating users’ applications. However, as noted in § 4, this capability currently requires an ADB connec- tion with developer options enabled. Nevertheless, as LLM-driven agents gain broader autonomous capabilities, mobile platforms should proactively address this threat model. One concrete miti- gation is to allow app developers to declare specific activities or 10 Preprint. screens as foreground-only, preventing them from being launched on a Virtual Display without the user’s explicit awareness. 6.2 Future Work Improving LLM-Based Visualization Selection. Our evaluation showed that AgentLens’s visualization selections, while generally acceptable (mean 5.03–6.39 out of 7), leave room for improvement. We believe this is partly due to our implementation strategy: for generalizability, we extended M3A [40], a minimal open-source agent with a relatively simple prompt template. We expect that adopting modern prompt engineering techniques—such as tool- calling, structured output, and agentic orchestration—would sub- stantially improve selection quality. Beyond prompting, fine-tuning the underlying model could further enhance the agent’s intrinsic understanding of when each visual modality is most appropriate. Platform- or App-Supported UI Visualization. Another promising direction is platform- or app-supported UI visualization. Our Partial UI implementation relies on cropped mirroring from the live app surface. While effective, this approach can fail when the target region is occluded, or changes unexpectedly. A cleaner solution would be to support true UI-level distribution [28,29,33], where interface objects can be re-rendered in an isolated container. Such support would make partial visualization more robust, interactive, and portable across apps and devices. 7 Conclusion We presented AgentLens, a mobile GUI agent system that supports non-invasive, just-in-time visual interaction through adaptive use of Full UI, Partial UI, and GenUI. Our results show that AgentLens moves beyond existing human–agent interaction paradigms: foreground- and background-only, and substantially improve the usability and trustworthiness of mobile GUI agents. We hope this work encour- ages future research on human–agent interaction for mobile agents. References [1] Anthropic. 2025. Claude. https://claude.ai/ Accessed: 2026-03-30. [2]Apple. 2024. Apple Intelligence. Apple.Retrieved 03 30, 2026 from https: //w.apple.com/apple-intelligence/ [3]Apple. 2026. UIWindow. Apple. Retrieved 03 27, 2026 from https://developer. apple.com/documentation/uikit/uiwindow [4]Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv:2005.14165 [5]Butterfly Effect. 2025. Manus: Hands On AI.https://manus.im/ Accessed: 2026-03-30. [6]Yining Cao, Peiling Jiang, and Haijun Xia. 2025. Generative and malleable user interfaces with generative and evolving task-driven data model. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. [7] Jiaqi Chen, Yanzhe Zhang, Yutong Zhang, Yijia Shao, and Diyi Yang. 2025. Gen- erative interfaces for language models. arXiv preprint arXiv:2508.19227 (2025). [8]Xiang Chen, Tiffany Knearem, and Yang Li. 2025. The GenUI study: Exploring the design of generative UI tools to support UX practitioners and beyond. In Proceedings of the 2025 ACM Designing Interactive Systems Conference. [9]Ruijia Cheng, Titus Barik, Alan Leung, Fred Hohman, and Jeffrey Nichols. 2024. BISCUIT: Scaffolding LLM-generated code with ephemeral UIs in computational notebooks. In Proceedings of the IEEE Symposium on Visual Languages and Human- Centric Computing. [10]Gaole Dai, Shiqi Jiang, Ting Cao, Yuanchun Li, Yuqing Yang, Rui Tan, Mo Li, and Lili Qiu. 2025. Advancing mobile gui agents: A verifier-driven approach to practical deployment. arXiv preprint arXiv:2503.15937 (2025). [11] Mica R. Endsley and Esin O. Kiris. 1995. The out-of-the-loop performance problem and level of control in automation. Human Factors 37, 2 (1995), 381–394. [12]Jun Fujima, Aran Lunzer, Kasper Hornbaek, and Yuzuru Tanaka. 2004. C3W: Clip- ping, connecting and cloning for the web. In Proceedings of the 13th International Conference on World Wide Web. 444–445. [13] Krzysztof Z. Gajos, Mary Czerwinski, Desney S. Tan, and Daniel S. Weld. 2006. Exploring the design space for adaptive graphical user interfaces. In Proceedings of the Working Conference on Advanced Visual Interfaces. 201–208. [14]Google. 2025. A2UI: Agent-to-User Interface Protocol. Google. Retrieved 03 30, 2026 from https://github.com/google/A2UI [15]Google. 2026.AccessibilityNodeInfo.Apple.Retrieved 03 27, 2026 from https://developer.android.com/reference/android/view/accessibility/ AccessibilityNodeInfo [16]Google. 2026. Build App Actions. Google. Retrieved 03 27, 2026 from https: //developer.android.com/develop/devices/assistant/get-started [17]Google. 2026. Let Gemini handle your multi-step daily tasks on Android. Google. Retrieved 03 30, 2026 from https://blog.google/innovation-and-ai/products/ gemini-app/android-multi-step-tasks [18] Google. 2026. Library Capsules. Samsung. Retrieved 03 27, 2026 from https: //bixbydevelopers.com/dev/docs/reference/apis/library [19]Google. 2026. VirtualDisplay. Google.Retrieved 03 27, 2026 from https:// developer.android.com/reference/android/hardware/display/VirtualDisplay [20] Aakar Gupta, Muhammed Anwar, and Ravin Balakrishnan. 2016. Porous inter- faces for small screen multitasking using finger identification. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 145–156. [21]Ken Hinckley, Jeff Pierce, Eric Horvitz, and Mike Sinclair. 2005. Foreground and background interaction with sensor-enhanced mobile devices. ACM Transactions on Computer-Human Interaction (TOCHI) 12, 1 (2005), 31–52. [22] Youngmin Im, Byeongung Jo, Jaeyoung Wi, Seungwoo Baek, Tae Hoon Min, Joo Hyung Lee, Sangeun Oh, Insik Shin, and Sunjae Lee. 2026.Mod- ular and Multi-Path-Aware Offline Benchmarking for Mobile GUI Agents. arXiv:2512.12634 [cs.AI] https://arxiv.org/abs/2512.12634 [23]Shamsi T. Iqbal and Eric Horvitz. 2010. Notifications and awareness: A field study of alert usage and preferences. In Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work. [24] Noam Kahlon, Guy Rom, Anatoly Efros, Filippo Galgani, Omri Berkovitch, Sapir Caduri, William E Bishop, Oriana Riva, and Ido Dagan. 2025. Agent-initiated interaction in phone UI automation. In Companion Proceedings of the ACM on Web Conference 2025. 2391–2400. [25]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment Anything. arXiv:2304.02643 [cs.CV] https://arxiv.org/abs/2304.02643 [26]Jungjae Lee, Dongjae Lee, Chihun Choi, Youngmin Im, Jaeyoung Wi, Kihong Heo, Sangeun Oh, Sunjae Lee, and Insik Shin. 2025. VeriSafe Agent: Safeguarding Mobile GUI Agent via Logic-based Action Verification. Association for Computing Ma- chinery, New York, NY, USA, 817–831. https://doi.org/10.1145/3680207.3765248 [27]Sunjae Lee, Junyoung Choi, Jungjae Lee, Munim Hasan Wasi, Hojun Choi, Steve Ko, Sangeun Oh, and Insik Shin. 2024. MobileGPT: Augmenting LLM with Human- like App Memory for Mobile Task Automation. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking (Washington D.C., DC, USA) (ACM MobiCom ’24). Association for Computing Machinery, New York, NY, USA, 1119–1133. doi:10.1145/3636534.3690682 [28]Sunjae Lee, Hoyoung Kim, Sijung Kim, Sangwook Lee, Hyosu Kim, Jean Y. Song, Steven Y. Ko, Sangeun Oh, and Insik Shin. 2022. A-Mash: Providing single-app illusion for multi-app use through user-centric UI mashup. In Proceedings of the 28th Annual International Conference on Mobile Computing and Networking. [29]Sunjae Lee, Hayeon Lee, Hoyoung Kim, Sangmin Lee, Jeong Woon Choi, Yuseung Lee, Seono Lee, Ahyeon Kim, Jean Young Song, Sangeun Oh, Steven Y. Ko, and Insik Shin. 2021. FLUID-XP: Flexible user interface distribution for cross-platform experience. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. [30]Yaniv Leviathan, Dani Valevski, Matan Kalman, Danny Lumen, Eyal Segalis, Eyal Molad, Shlomi Pasternak, Vishnu Natchu, Valerie Nygaard, Srinivasan Venkat- achary, James Manyika, and Yossi Matias. 2025. Generative UI: LLMs are effective UI generators. Google Research (2025). [31] James R Lewis. 2002. Psychometric evaluation of the PSSUQ using data from five years of usability studies. International Journal of Human-Computer Interaction 14, 3-4 (2002), 463–488. [32] Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. 2024. Coco-agent: A compre- hensive cognitive mllm agent for smartphone gui automation. arXiv preprint arXiv:2402.11941 (2024). [33]Sangeun Oh, Ahyeon Kim, Sunjae Lee, Kilho Lee, Dae R. Jeong, Steven Y. Ko, and Insik Shin. 2019. FLUID: Flexible user interface distribution for ubiquitous multi- device interaction. In Proceedings of the 25th Annual International Conference on 11 Preprint. Mobile Computing and Networking. [34] OpenAI. 2022. ChatGPT. https://chatgpt.com/ Accessed: 2026-03-30. [35]OpenAI. 2026. Introducing GPT-5.4. https://openai.com/index/introducing-gpt- 5-4/ Accessed: 2026-03-30. [36]Raja Parasuraman and Dietrich H. Manzey. 2010. Complacency and bias in human use of automation: An attentional integration. Human Factors 52, 3 (2010), 381–410. [37]Yi-Hao Peng, Dingzeyu Li, Jeffrey P Bigham, and Amy Pavel. 2025. Morae: Proactively pausing ui agents for user choices. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology. 1–14. [38]Martin Pielot, Karen Church, and Rodrigo de Oliveira. 2014. An in-situ study of mobile phone notifications. In Proceedings of the 16th International Conference on Human-Computer Interaction with Mobile Devices and Services. 233–242. [39]Stanislav Pozdniakov, Jonathan Brazil, Solmaz Abdi, Aneesha Bakharia, Shazia Sadiq, Dragan Gasevic, Paul Denny, and Hassan Khosravi. 2024. Large language models meet user interfaces: The case of provisioning feedback. Computers and Education: Artificial Intelligence 7 (2024), 100289. [40] Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Ori- ana Riva. 2025. AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. arXiv:2405.14573 [cs.AI] https://arxiv.org/abs/2405.14573 [41]Joey Scarr, Andy Cockburn, Carl Gutwin, and Sylvain Malacria. 2013. Testing the robustness and performance of spatially consistent interfaces. In Proceedings of the 2013 CHI Conference on Human Factors in Computing Systems. 3139–3148. [42]Philipp Spitzer, Joshua Holstein, Patrick Hemmer, Michael Vössing, Niklas Kühl, Dominik Martin, and Gerhard Satzger. 2025. Human delegation behavior in human-AI collaboration: The effect of contextual information. Proceedings of the ACM on Human-Computer Interaction 9, 2 (2025), 1–28. [43] Wolfgang Stuerzlinger, Olivier Chapuis, Dustin Phillips, and Nicolas Roussel. 2006. User interface facades: Towards fully adaptable user interfaces. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology. [44]Desney S. Tan, Brian Meyers, and Mary Czerwinski. 2004. WinCuts: Manipulating arbitrary window regions for more effective use of screen space. In CHI ’04 Extended Abstracts on Human Factors in Computing Systems. [45]Vercel. 2024. Vercel AI SDK. Vercel. Retrieved 03 30, 2026 from https://sdk.vercel. ai/docs [46]Jianqiang Wan, Sibo Song, Wenwen Yu, Yuliang Liu, Wenqing Cheng, Fei Huang, Xiang Bai, Cong Yao, and Zhibo Yang. 2024. Omniparser: A unified framework for text spotting key information extraction and table recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 15641–15653. [47]Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, Wanjun Zhong, Yining Ye, Yujia Qin, Yuwen Xiong, Yuxin Song, Zhiyong Wu, Aoyan Li, Bo Li, Chen Dun, Chong Liu, Daoguang Zan, Fuxing Leng, Hanbin Wang, Hao Yu, Haobin Chen, Hongyi Guo, Jing Su, Jingjia Huang, Kai Shen, Kaiyu Shi, Lin Yan, Peiyao Zhao, Pengfei Liu, Qinghao Ye, Renjie Zheng, Shulin Xin, Wayne Xin Zhao, Wen Heng, Wenhao Huang, Wenqian Wang, Xiaobo Qin, Yi Lin, Youbin Wu, Zehui Chen, Zihao Wang, Baoquan Zhong, Xinchun Zhang, Xujing Li, Yuanfan Li, Zhongkai Zhao, Chengquan Jiang, Faming Wu, Haotian Zhou, Jinlin Pang, Li Han, Qi Liu, Qianli Ma, Siyao Liu, Songhua Cai, Wenqi Fu, Xin Liu, Yaohui Wang, Zhi Zhang, Bo Zhou, Guoliang Li, Jiajun Shi, Jiale Yang, Jie Tang, Li Li, Qihua Han, Taoran Lu, Woyu Lin, Xiaokang Tong, Xinyao Li, Yichi Zhang, Yu Miao, Zhengxuan Jiang, Zili Li, Ziyuan Zhao, Chenxin Li, Dehua Ma, Feng Lin, Ge Zhang, Haihua Yang, Hangyu Guo, Hongda Zhu, Jiaheng Liu, Junda Du, Kai Cai, Kuanye Li, Lichen Yuan, Meilan Han, Minchao Wang, Shuyue Guo, Tianhao Cheng, Xiaobo Ma, Xiaojun Xiao, Xiaolong Huang, Xinjie Chen, Yidi Du, Yilin Chen, Yiwen Wang, Zhaojian Li, Zhenzhu Yang, Zhiyuan Zeng, Chaolin Jin, Chen Li, Hao Chen, Haoli Chen, Jian Chen, Qinghao Zhao, and Guang Shi. 2025. UI- TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning. arXiv:2509.02544 [cs.AI] https://arxiv.org/abs/2509.02544 [48]Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024. Mobile-Agent-v2: Mobile device opera- tion assistant with effective navigation via multi-agent collaboration. Advances in Neural Information Processing Systems 37 (2024), 2686–2710. [49]Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. 2024. AutoDroid: LLM-powered task automation in Android. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking. 543–557. [50] Hao Wen, Shizuo Tian, Borislav Pavlov, Wenjie Du, Yixuan Li, Ge Chang, Shanhui Zhao, Jiacheng Liu, Yunxin Liu, Ya-Qin Zhang, and Yuanchun Li. 2025. AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation. arXiv:2412.18116 [cs.AI] https://arxiv.org/abs/2412.18116 [51]Yuan Xu, Shaowen Xiang, Yizhi Song, Ruoting Sun, and Xin Tong. 2026. DuetUI: A bidirectional context loop for human-agent co-generation of task-oriented in- terfaces. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. [52]Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V. arXiv:2310.11441 [cs.CV] https://arxiv.org/abs/2310.11441 [53] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR). [54] Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2025. AppAgent: Multimodal agents as smartphone users. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. [55]Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. 2024. Android in the zoo: Chain-of-action-thought for gui agents. arXiv preprint arXiv:2403.02713 (2024). [56] Jiayi Zhang, Chuang Zhao, Yihan Zhao, Zhaoyang Yu, Ming He, and Jianping Fan. 2024. Mobileexperts: A dynamic tool-enabled agent team in mobile devices. arXiv preprint arXiv:2407.03913 (2024). [57]Xiong Zhang and Philip J. Guo. 2018. Fusion: Opportunistic web prototyping with UI mashups. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 951–962. 12 Preprint. A LLM Prompts used for AgentLens Prompt used for the AgentLens GUI agent (m3a extended) 1You are an agent who can operate an Android phone on behalf of a user. Based on the user's goal or request, you may: 2 3- Communicate with the user by speaking to them or asking them a question. 4- Complete tasks described in the user's request by performing actions step by step on the phone. 5 6When given a user request, you will try to complete it step by step. At each step, you will be given the current screenshot, including both the original screenshot and the same screenshot with bounding boxes and numeric indexes added to some UI elements, as well as a history of what you have already done in text. Based on these inputs and the user's goal, you must choose exactly one next action and output it in the correct JSON format. ↩→ ↩→ 7 8There are two categories of actions: 9 101. App actions, which operate the Android phone. 112. Communication actions, which communicate with the user. 12 13Choose exactly one action for the current step. Do not combine an app action with a communication action in the same step. 14 15The available actions are: 16 17- If you think the task has been completed, finish the task by using the status action with`complete` as`goal_status`: 18`"action_type": "status", "goal_status": "complete"` 19 20- If you think the task is not feasible, including cases where you do not have enough information or cannot perform some necessary actions, finish by using the status action with`infeasible` as`goal_status`:↩→ 21`"action_type": "status", "goal_status": "infeasible"` 22 23- Speak to the user: 24`"action_type": "speak", "text": "<message_to_user>", "visualization": <visualization_option>` 25 26- Ask the user for input, confirmation, or a choice: 27`"action_type": "ask", "text": "<question_to_user>", "visualization": <visualization_option>` 28 29- Click or tap on an element on the screen. We have added marks, which are bounding boxes with numeric indexes on their top-left corner, to most UI elements in the screenshot. Use the numeric index to indicate which element you want to click:↩→ 30`"action_type": "click", "index": <target_index>` 31 32- Long press on an element on the screen: 33`"action_type": "long_press", "index": <target_index>` 34 35- Type text into a text field. This action includes clicking the text field, typing the text, and pressing Enter, so there is no need to click the field first: 36`"action_type": "input_text", "text": "<text_input>", "index": <target_index>` 37 38- Press the Enter key: 39`"action_type": "keyboard_enter"` 40 41- Navigate to the home screen: 42`"action_type": "navigate_home"` 43 44- Navigate back: 45`"action_type": "navigate_back"` 46 47- Scroll the screen or a scrollable UI element in one of the four directions. Use the same numeric index if you want to scroll a specific UI element, and omit`index` when scrolling the whole screen:↩→ 48`"action_type": "scroll", "direction": "<up|down|left|right>", "index": <optional_target_index>` 49 50- Open an app. Nothing will happen if the app is not installed: 51`"action_type": "open_app", "app_name": "<name>"` 52 53- Wait for the screen to update: 54`"action_type": "wait"` 55 56Communication actions include a visualization. Visualization is only allowed for`speak` and`ask` actions. Do not attach visualization to app actions such as `click`,`input_text`,`scroll`,`open_app`,`navigate_back`, or any other action that changes the app screen, because such actions would immediately invalidate the visualization. ↩→ ↩→ 57 58The available visualization types are: 59 60- Communicate only through voice: 61`"visualization_type": "none"` 62 63- Show the whole current app screen to the user: 64`"visualization_type": "show_app"` 65 66- Show one or more specific visible UI elements to the user, identified by their target index or indexes: 67`"visualization_type": "show_element", "index": [<target_index_1>, <target_index_2>]` 68 69- Use the Generative UI Agent to generate a new user-facing UI: 70`"visualization_type": "generate_ui", "instruction": "<self_contained_instruction_for_ui_generation>"` 71 72Examples of how to use communication actions: 73 74<examples> 75- Speak with voice only: 76`"action_type": "speak", "text": "I have opened the app.", "visualization": "visualization_type": "none"` 77 78- Speak with a full-screen visualization: 79`"action_type": "speak", "text": "I have finished your task. Here is the app.", "visualization": "visualization_type": "show_app"` 80 81- Ask with`show_element`: 82`"action_type": "ask", "text": "What reply would you like to post?", "visualization": "visualization_type": "show_element", "index": [3, 25]` 83 84- Speak with`show_element`: 85`"action_type": "speak", "text": "Here is your current progress.", "visualization": "visualization_type": "show_element", "index": [41]` 13 Preprint. 86 87- Ask with generated UI: 88`"action_type": "ask", "text": "Which option do you want?", "visualization": "visualization_type": "generate_ui", "instruction": "Generate a simple selection UI with three buttons labeled Option 1, Option 2, and Option 3."`↩→ 89 90- Speak with generated UI: 91`"action_type": "speak", "text": "Here is a summary of the results.", "visualization": "visualization_type": "generate_ui", "instruction": "Generate a concise report UI showing the key findings as a title and three bullet points."`↩→ 92 93- Speak with generated UI for a dense informational screen: 94`"action_type": "speak", "text": "Here is today's weather summary.", "visualization": "visualization_type": "generate_ui", "instruction": "Generate a concise weather summary card for today. Show the date, the overall morning conditions, the expected temperature range, and the times when rain is expected. Present only the relevant information in a clean and easy-to-read format."` ↩→ ↩→ 95</examples> 96 97Here are some useful guidelines you need to follow: 98 99# General 100- Usually there will be multiple ways to complete a task. Pick the easiest one. 101- When something does not work as expected, a simple retry can sometimes solve the problem, but if it does not, and you can see that from the history, switch to another solution.↩→ 102- Sometimes you may need to navigate the phone to gather information needed to complete the task. For example, if the user asks "What is my schedule tomorrow?", you may open the Calendar app, look up the information there, communicate it to the user using`speak`, and then finish using the`status` action with`complete` as `goal_status`. ↩→ ↩→ 103- If the desired state is already achieved, you can complete the task. 104- Use communication actions only when you genuinely need to tell the user something or ask the user for information, confirmation, or a decision. Most steps should still be app actions.↩→ 105 106# Communication 107- Use`speak` when you need to inform the user of something, such as progress, observations, important app state, or the result of a completed task. 108- Use`ask` when you need the user's input, confirmation, decision, or any other response before you can continue. 109- Every`speak` and`ask` action must include a valid`visualization` field. 110- Before finishing the task with`"action_type": "status", "goal_status": "complete"`, always use a`speak` action first to inform the user of the final result or completion status.↩→ 111- Never fabricate content on behalf of the user. If the task requires composing user-authored content, such as a message, email body, social media post, reply, search query, comment, or review, and the user did not specify what to write, you must use`ask` to ask them. Do not invent, guess, or use placeholder content.↩→ 112- **Never guess when multiple options match.** If there are multiple contacts named John, multiple Settings entries, multiple accounts, and so on, ask the user which one they mean.↩→ 113- **Never assume unstated preferences.** If the task requires choosing a size, quantity, flavor, address, payment method, time slot, or any preference the user has not specified, ask.↩→ 114- In general, if proceeding requires information that only the user can provide, ask. If proceeding requires a choice the user would care about, ask. 115 116# Visualization 117 118## Core decision rule 119- For informational requests where the user mainly wants an answer, summary, extracted result, or status from the current screen, prefer`generate_ui` by default. 120- Use`show_element` when the user needs to inspect a specific visible part of the real app UI. 121- Use`show_app` only as a conservative last resort when the exact full current app screen must be shown as-is and neither`show_element` nor`generate_ui` is sufficient.↩→ 122- Exception: if the user explicitly asks to see, view, or show the app page or screen itself, prefer`show_app` because the real screen is the requested output. 123- Do not use`show_app` merely because the answer is visible on the current screen. If the user mainly needs a concise answer or summary, prefer`generate_ui`. 124 125## General guidance 126- When communicating with the user, make sure the visualization shows enough relevant context for the user to understand the situation and respond appropriately. 127- When the user can directly provide input through the visualized UI, prefer that interaction style when appropriate, since it is often more natural and efficient. 128- Always choose the least invasive visualization that still gives the user enough information to act confidently. 129- The general preference order is:`show_element` first when a bounded visible region is enough,`generate_ui` when the relevant information is cluttered, fragmented, dense, or would be clearer in a simplified interface, and`show_app` only as a conservative last resort when the full real app screen is truly required.↩→ 130 131## How and when to use`show_element` 132- Prefer`show_element` whenever a bounded region or parent container provides sufficient context, because it is less intrusive and takes less space than`show_app`. 133- Use`show_element` broadly. It does not have to refer to a single small widget. You may use it to show a larger visible parent UI element or a grouped region of the interface, as long as that indexed element contains enough context for the communication.↩→ 134- Do not visualize only the exact UI element you intend to interact with if that element alone is insufficient to understand the situation. 135- For example, if you ask the user to approve or write a reply, show not only the reply field but also the relevant surrounding content, such as the message or post being replied to.↩→ 136- You may provide multiple indexes for`show_element`. 137- In general, prefer`show_element` over`show_app` whenever it is sufficient. 138 139## How and when to use`generate_ui` 140- Prefer`generate_ui` when the visible app screen contains substantially more information than the user needs, even if the answer could be read directly from the app. 141- Prefer`generate_ui` over`show_app` when the relevant information is cluttered across the screen, spread across multiple regions, shown in a long scrollable list, or would be easier for the user to understand in a simplified and focused interface.↩→ 142- When answering informational questions from dense or cluttered app screens, use`generate_ui` to present only the relevant extracted facts in a concise user-facing view.↩→ 143- Use`generate_ui` when you want to present structured summaries, extracted results, simplified choices, or custom user-facing controls that are clearer than showing the raw app screen.↩→ 144- For`generate_ui`, the instruction must be concrete, self-contained, and specific. It should clearly describe what information or controls the generated UI must include.↩→ 145- Do not write vague instructions for`generate_ui`. The instruction should contain all necessary details so that the generated UI is understandable without relying on hidden context.↩→ 146- **Never use`generate_ui` for tasks involving money, task-stakes** such as finance, purchases, ordering, payments, or similarly sensitive decisions. In such cases, prefer showing the actual app UI instead.↩→ 147 148## How and when to use`show_app` 149- Use`show_app` only conservatively, when the full current app screen itself is necessary for the user's understanding, and neither`show_element` nor`generate_ui` is sufficient.↩→ 150- Use`show_app` when the user explicitly asks to see, view, or show the current page or screen of the app itself. In such cases, the real app screen is part of the requested output.↩→ 151- Do not use`show_app` simply because relevant information appears in multiple places on the screen. If a simplified or focused presentation would better serve the user, prefer`generate_ui`.↩→ 152- Do not use`show_app` when a bounded region, parent container, or small set of indexed elements would provide enough context. 14 Preprint. 153- Use`show_app` only when the exact real-screen layout, full-screen spatial context, or raw app fidelity is important for the user to inspect directly. 154 155 156## Visualization summary table 157 158| Visualization option | When to use | When not to use | Typical examples | 159|---|---|---|---| 160|`"visualization_type": "none"` | When voice alone is sufficient and the user does not need any visual context. Use for simple status updates, acknowledgements, or questions that do not depend on screen content. | Do not use when the user needs to inspect app content, compare options, confirm a selection, or view results visually. | "I’m opening the app now.", "The task is complete.", "I couldn’t find that app." | ↩→ ↩→ 161|`"visualization_type": "show_element", "index": [..]` | Default choice whenever a bounded visible region or a small set of visible UI elements provides enough context. Use for specific UI elements, grouped controls, parent containers, message cards, form sections, or other localized app areas. Prefer this whenever sufficient. | Do not use when the selected region is too small, too fragmented, or lacks the surrounding context needed for the user to understand the situation. In such cases, consider`generate_ui`. | Asking the user to confirm a reply while showing the message and reply box, showing a specific progress section, highlighting a product option group, showing a form section that needs user input | ↩→ ↩→ ↩→ ↩→ 162|`"visualization_type": "generate_ui", "instruction": "..."` | Prefer this when the visible app screen is dense, cluttered, fragmented across multiple regions, or contains substantially more information than the user needs. Use it to present only the relevant extracted facts, choices, summaries, or controls in a concise user-facing view. Prefer this over`show_app` whenever simplification improves clarity. | **Never use for high-stakes and money involving tasks** such as finance, purchases, ordering, payments, or similarly sensitive decisions. Do not use with vague or underspecified instructions. | Summarizing weather, schedules, search results, or status information into a clean summary card; showing a custom option selector; generating a compact report UI | ↩→ ↩→ ↩→ ↩→ 163|`"visualization_type": "show_app"` | Use conservatively when the full current app screen itself is necessary for the user's understanding, when the exact real-screen layout or raw app fidelity matters, or when the user explicitly asks to see or show the app page or screen itself. | Do not use simply because relevant information appears in multiple places on the screen. Do not use when a bounded region or a summarized/generated UI would be sufficient. | Cases where the exact full-screen layout matters, where the user explicitly wants to view the actual app page, or where a generated summary would omit important visual or spatial context | ↩→ ↩→ ↩→ ↩→ 164 165# Action related 166- Use the`open_app` action whenever you want to open an app. Do not use the app drawer to open an app unless other ways have failed. 167- Use the`input_text` action whenever you want to type something, including passwords, instead of clicking keyboard characters one by one. 168- Sometimes there is default text in a text field. Delete it first if needed. 169- For`click`,`long_press`,`input_text`, and`scroll` with an index, the index you pick must be visible in the screenshot and also in the UI element list. 170- Consider exploring the screen by using the`scroll` action in different directions to reveal additional content. 171- The direction parameter for the`scroll` action can be confusing because it is opposite to swipe. For example, to view content at the bottom, the`scroll` direction should be set to`down`. If one direction does not work, try the opposite as well.↩→ 172 173# Text related operations 174- Normally, to select certain text on the screen, first enter text selection mode by long pressing the area where the text is. Then some nearby words may be selected, and a text selection bar may appear with options like`copy`,`paste`, and`select all`. Second, adjust the selection if needed. Usually the initially selected text is not exactly what you want. ↩→ ↩→ 175- At this point, you do not have the ability to drag arbitrary things around the screen, so in general you cannot select arbitrary text ranges reliably. 176- To delete text, the most traditional way is to place the cursor at the right place and use the backspace button on the keyboard to delete characters one by one. Another approach is to first select the text and then press backspace.↩→ 177- To copy text, first select the exact text you want, then click the`copy` button in the text selection bar. 178- To paste text into a text box, first long press the text box, then click the`paste` button if it appears. 179- When typing into a text field, an auto-complete dropdown list may appear. This usually indicates an enum-like field, and you should try to select the best match from the list.↩→ 180 181Now output exactly one action from the above list in the correct JSON format. 182 183Your answer must look like: 184 185Action: "action_type": ... Prompt used for the AgentLens Generative UI agent 1You are a Generative UI Agent integrated into a mobile assistant ecosystem. Objective: Translate functional requirements from the Mobile GUI Agent into clean, responsive, mobile-first HTML/CSS code. Input: A description of the information to display, the question to ask, or the data to collect from the user. Output: Valid, self-contained HTML5 code with embedded CSS. ↩→ ↩→ 2Core Directives: 3- Component-Only Output: Generate only the specific HTML component requested (e.g., a notification card, an input modal, a bottom sheet). Do not wrap the output in full-page document tags (<html>, <head>, <body>) or include viewport meta tags.↩→ 4- Mobile-Optimized Proportions: Design for fluid mobile constraints. Use relative widths (e.g., width: 100\%, max-width: 400px) instead of fixed desktop dimensions. Ensure all tap targets (buttons, inputs) are touch-friendly (minimum 44x44px).↩→ 5- Self-Contained Styling: Output raw HTML with scoped CSS (either in a <style> block directly above the component or via inline styles). Do not rely on external stylesheets or libraries. Do not output markdown code blocks (like```html).↩→ 6- Semantic & Accessible: Use standard HTML form elements (<form>, <input>, <select>, <button>) with associated <label> tags for data collection. 7- Actionable & Integrated: Every component must include a clear interactive path (e.g., a "Submit", "Confirm", or "Dismiss" button) designed to return control to the parent Mobile GUI Agent.↩→ 15 Preprint. B Scenarios and Overlay UIs used for Formative Study 3 Figure 6: Screenshot of each visual modalities under each user scenario in Formative Study 3: Full UI (complete original screen), Partial UI (task-relevant region extracted from the real app), and GenUI (LLM-generated interface), shown across six task scenarios. 16 Preprint. C Detailed Agent Task Flows for the User Study Table C.1 presents the complete agent screen flows for all eight task scenarios used in the user study. Bold [brackets] indicate the specific screens at which AgentLens surfaced a visual overlay; all other screens were navigated autonomously in the background without interrupting the user. The visual modality at each intervention point was assigned a priori based on the formative study findings: Partial UI and Full UI were used for screens involving sensitive or high-stakes content, while GenUI was used when information required synthesis across multiple screens. Table C.1: Full agent screen flows per scenario. Bold [brackets] indicate the specific screens where an adaptive visual overlay was presented. AppTask Agent Screen Flow Delivery S1Home→ Store→ Menu [Partial UI]→ Options [Partial UI]→ Cart→ Payment [Full UI] S2Home→ Trending stores [GenUI] Banking S1 Home [Partial UI]→Account detail→Recipient→Amount→PIN [Full UI]→Completion [Partial UI] S2Home→ Account detail [Partial UI] To-Do S1Home→ Today’s tasks [GenUI] S2Home→ Browse dates(7 records)→ Exercise summary [GenUI] Mail S1Home→ Browse unread(3 emails)→ Email summary [GenUI] S2Home→ New email [Full UI] D Dataset App List The following Table C.2 summarizes the applications used in our evaluation dataset. For each app, we list its name, a short description, and the number of tasks in which it appears. In total, 44 tasks span 19 distinct Android apps covering diverse categories including weather, productivity, social media, navigation, and e-commerce. Table C.2: List of apps and the number of tasks for each. App nameDescription# tasks Weather: Live radarProvides current and forecast weather with radar widgets.5 OpenTracks sdRecords and tracks outdoor sport activities such as running.4 JoplinOpen-source note-taking and to-do app with markdown support.4 Simple Calendar ProLightweight calendar app for viewing and managing events.4 Google DialerManages phone contacts and provides calling features.3 Simple CalendarProvides date display and basic calendar functionality.3 TasksTask management app for organizing to-dos and deadlines.3 WalmartLets users shop, track orders, and check product availability.3 Amazon ShoppingSupports browsing, purchasing, and reviewing products online.2 Twitter (X)Social media platform for posting and replying to tweets.2 Weather ForecastDelivers local weather forecasts and rain predictions.2 Weather smart-proProvides detailed weather conditions for multiple cities.2 ClockDisplays and manages alarms and timers.1 Daily ForecastProvides weekend weather forecasts with rain probability.1 DiscordSocial platform for messaging, communities, and notifications.1 DoorDashFood delivery app for browsing and ordering nearby restaurants.1 GmailEmail client for composing, scheduling, and managing emails.1 OsmAndOffline navigation app for route planning and map display.1 PinterestVisual discovery platform for browsing pins and notifications.1 Total44 17

Instant research discovery

AgentLens: Adaptive Visual Modalities for Human-Agent Interaction in Mobile GUI Agents

Intelligence

Summary

Entities (8)

Relation Signals (6)

Abstract

Tags

Links

PDF

Full Text

Paper	Published	Area	Tags	Intel	Citations