Paper deep dive
Towards Trustworthy GUI Agents: A Survey
Yucheng Shi, Wenhao Yu, Wenlin Yao, Wenhu Chen, Ninghao Liu
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%
Last extracted: 3/12/2026, 6:55:53 PM
Summary
This survey explores the trustworthiness of GUI agents, which extend LLMs to perform irreversible actions in digital environments. It identifies an 'execution gap'âthe misalignment between perception, reasoning, and interactionâas the core challenge. The paper introduces a taxonomy of trust (Perception, Reasoning, and Interaction), reviews failure modes and adversarial attacks, and discusses defense mechanisms and evaluation metrics for safer deployment.
Entities (5)
Relation Signals (3)
GUI Agents â faceschallenge â Execution Gap
confidence 95% · This survey identifies the execution gap as a key challenge in building trustworthy GUI agents.
GUI Agents â performs â Irreversible Operations
confidence 95% · GUI agents perform irreversible operations such as submitting forms, granting permissions, or deleting data.
Perception Trust â iscomponentof â Trustworthiness
confidence 90% · We introduce a workflow-aligned taxonomy that decomposes trust into Perception Trust, Reasoning Trust, and Interaction Trust.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Graphical User Interface (GUI) agents extend large language models from text generation to action execution in real-world digital environments. Unlike conversational systems, GUI agents perform irreversible operations such as submitting forms, granting permissions, or deleting data, making trustworthiness a core requirement. This survey identifies the execution gap as a key challenge in building trustworthy GUI agents: the misalignment between perception, reasoning, and interaction in dynamic, partially observable interfaces. We introduce a workflow-aligned taxonomy that decomposes trust into Perception Trust, Reasoning Trust, and Interaction Trust, showing how failures propagate across agent pipelines and compound through action/observation loops. We systematically review benign failure modes and adversarial attacks at each stage, together with corresponding defense mechanisms tailored to GUI settings. We further analyze evaluation practices and argue that task completion alone is insufficient for trust assessment. We highlight emerging trust-aware metrics and benchmarks that capture error cascades and the security/utility trade-off, and outline open challenges for deploying GUI agents safely and reliably.
Tags
Links
- Source: https://arxiv.org/abs/2503.23434
- Canonical: https://arxiv.org/abs/2503.23434
Trouble viewing inline? Open PDF directly â
Full Text
59,286 characters extracted from source content.
Expand or collapse full text
Towards Trustworthy GUI Agents: A Survey Yucheng Shi 1,2 , Wenhao Yu 2 , Jingyuan Huang 1 , Wenlin Yao 3 , Wenhu Chen 4 , Ninghao Liu 5 1 University of Georgia 2 Tencent AI Seattle Lab 3 Microsoft Research 4 University of Waterloo 5 Hong Kong Polytechnic University Abstract Graphical User Interface (GUI) agents extend large language models from text generation to action execution in real-world digital environ- ments. Unlike conversational systems, GUI agents perform irreversible operations such as submitting forms, granting permissions, or deleting data, making trustworthiness a core re- quirement. This survey identifies the execution gap as a key challenge in building trustworthy GUI agents: the misalignment between per- ception, reasoning, and interaction in dynamic, partially observable interfaces. We introduce a workflow-aligned taxonomy that decomposes trust into Perception Trust, Reasoning Trust, and Interaction Trust, showing how failures propagate across agent pipelines and compound through action/observation loops. We system- atically review benign failure modes and ad- versarial attacks at each stage, together with corresponding defense mechanisms tailored to GUI settings. We further analyze evaluation practices and argue that task completion alone is insufficient for trust assessment. We high- light emerging trust-aware metrics and bench- marks that capture error cascades and the se- curity/utility trade-off, and outline open chal- lenges for deploying GUI agents safely and reliably. 1 Introduction The emergence of GUI agents marks a fundamen- tal transition in how AI systems interact with the digital world. Unlike chatbots that generate text responses, GUI agents take actions, clicking but- tons, filling forms, and navigating websites that produce immediate, often irreversible, real-world consequences (Nguyen et al., 2024a; Wang et al., 2024b; Xie et al., 2024). This shift from generation to execution fundamentally changes the stakes of AI trustworthiness. The contrast is clear: when a language model hallucinates in a conversation, the user can simply ignore the response; when a GUI agent hallucinates a button that doesnât exist and clicks the wrong element, it might authorize an un- intended purchase, delete important files, or expose sensitive information (Yang et al., 2024; Levy et al., 2024). The cost of failure is no longer measured in user dissatisfaction but in tangible harm. We argue that the core challenge underlying GUI agent trustworthiness is what we term the Execu- tion Gap: the fundamental disconnect between three levels of agent operation. Perceptual Fidelity requires correctly mapping visual pixels or Docu- ment Object Model (DOM) structures to semantic understanding of interface elements. Reasoning Fidelity demands maintaining logical consistency across multi-step plans in environments that change between actions. Interaction Fidelity involves trans- lating intended actions into precise coordinates or commands that achieve the desired effect. This gap explains why techniques successful for static LLM applications often fail for GUI agents (Zheng et al., 2024; Chae et al., 2024). For instance, a model that excels at describing what it sees in an image may still click the wrong button because it cannot reliably map its understanding to actionable coordinates. A planner that generates coherent step sequences may fail when a pop-up dialog invali- dates its plan mid-execution. Existing surveys on LLM trustworthiness ad- dress privacy, bias, and hallucination (Liu et al., 2023c; Weidinger et al., 2022; Gan et al., 2024). Although these concerns apply to GUI agents, three characteristics make GUI-specific analysis essential. First, irreversibility: text generation is infinitely reversible, users simply regenerate, but GUI actions often cannot be undone, as sent emails, deleted files, and completed transactions persist (Hua et al., 2024). This asymmetry de- mands different safety architectures than those de- signed for conversational AI. Second, dynamic en- vironments: unlike static documents, GUIs change constantly through DOM updates, loading states, 1 arXiv:2503.23434v2 [cs.LG] 24 Feb 2026 pop-ups, and A/B testing, meaning the interface an agent perceives may differ from the interface it acts upon milliseconds later (Ma et al., 2024). Trust must account for environmental non-stationarity. Third, action-observation loops: GUI agents op- erate in closed loops where each action changes the environment, affecting subsequent observations, and errors compound as a wrong click leads to an unexpected screen, which leads to further misinter- pretation (Wu et al., 2025a). This survey makes three primary contributions. First, we present a workflow-aligned taxon- omy organizing trustworthiness around Perception Trust (§3), Reasoning Trust (§4), and Interaction Trust (§5), reflecting how vulnerabilities propagate through agent pipelines. Second, we provide a comprehensive analysis of defense mechanisms integrated within each trust dimension, revealing how mitigations must be stage-specific. Third, we analyze evaluation methodologies (§6) with em- phasis on the security-utility trade-off that defines practical deployment decisions. Figure 1 presents our âRisk & Mitigation Landscapeâ, a unified view mapping threats to agent modules and their real- world impacts, which serves as a roadmap for this survey. 2 Foundations: The GUI Agent Pipeline Before analyzing trustworthiness, we establish the foundations of GUI agents by examining their exe- cution pipeline, the existing execution gap at each stage, and the limitations of standard LLM safety in agentic settings. 2.1 Pipeline Architecture GUI agents typically operate through three inter- connected stages: perception, reasoning, and inter- action (Lu et al., 2023; Wang et al., 2024b). Perception converts raw interface inputs, such as screenshots, DOM structures, accessibility trees, or hybrid representations, into a semantic under- standing of the interface state (Wu et al., 2024b; Nong et al., 2024). This stage answers the question: What elements exist, and what do they represent? Existing approaches span pure vision-based per- ception using multimodal large language models (MLLMs) (Zheng et al., 2024), structured pars- ing of HTML and accessibility APIs (Deng et al., 2023), and hybrid designs that combine visual and structural cues for improved robustness (Wang et al., 2024a). Recent work on universal visual grounding further argues that fully visual percep- tion with pixel-level action execution can rival or surpass text-augmented methods (Gou et al., 2024). Reasoning operates on the perceived state and task instructions to determine the next action. This includes task decomposition, progress track- ing, and decision-making over possible action se- quences (Gu et al., 2024; Zhu et al., 2025; Koh et al., 2024b). The core question is: What should I do next to achieve the goal? Recent advances introduce world models that simulate action out- comes (Chae et al., 2024), hierarchical planning frameworks that separate high-level goals from low- level actions (Liu et al., 2025), and multi-agent sys- tems that distribute reasoning across specialized agents (Srinivas et al., 2024; Sengupta et al., 2024). Interaction executes the selected action by trans- lating abstract intentions (e.g., âclick the submit buttonâ) into concrete interface operations such as mouse clicks or touch events (Koh et al., 2024a). This stage addresses the question: How is the intended action physically performed? Reliable interaction requires accurate coordinate mapping, synchronization with dynamic UI elements, and verification that the intended effect actually oc- curs (Guan et al., 2024). 2.2 The Execution Gap at Each Stage Each stage of the pipeline introduces a distinct grounding challenge that directly affects trustwor- thiness. Perceptual Fidelity concerns the alignment be- tween raw interface signals and semantic represen- tations. Misalignment often arises from the par- tial and modality-specific interface observations. For instance, accessibility APIs expose structured yet incomplete views of the interface, while DOM parsing emphasizes logical structure but overlooks visual layout (Deng et al., 2023; Yang et al., 2024). Incorporating visual modalities introduces new fail- ure modes, as MLLMs can be manipulated by ad- versarial visual inputs that bypass textual safety alignment (Gao et al., 2024). Empirical studies further show that GUI grounding models remain highly sensitive to visual perturbations and reso- lution changes across mobile, desktop, and web environments (Zhao et al., 2025). Reasoning Fidelity requires maintaining coher- ent and adaptive plans over long action sequences. Unlike static QA tasks, GUI agents must update be- liefs after each action, handle unexpected states, and revise plans when assumptions fail. Cur- 2 1. Attacks & Threats Reasoning Attacks (Prompt Injection, Logic Exploits) Interaction Attacks (UI Redressing, Timing Attacks) Perception Attacks (Adversarial Inputs, DOM Injection) Perception Trust (Raw Inputs: Screenshots, DOM âPerceptual Fidelity: Mapping Pixels to Semantic State) 2. GUI Agent Pipeline & Trust Stages (Execution Gap) Reasoning Trust Reasoning Trust (Semantic State â Reasoning Fidelity: Planning, State Tracking â Action Plan) Interaction Trust Interaction Trust (Action Plan â Interaction Fidelity: Coordinate Mapping, Execution â Executed Action) Error Cascades Error Cascades Defense Mechanisms Robust Grounding, Multi-modal Checks Defense Mechanisms Self-check, World Models, Verification Protocols Defense Mechanisms Action Verification, Coordinate Validation Error Cascades 3. Real-world Consequences (Irreversible Impact) Data Leakage, Unintended Auth, Privacy Violation Financial Loss, System Corruption, Service Disruption Figure 1: Risk & Mitigation Landscape. This diagram maps the threat landscape of GUI agents across three dimensions: (1) specific attack vectors targeting each pipeline stage, (2) how vulnerabilities propagate through the perception-reasoning-interaction workflow, and (3) the real-world consequences of failures. Dashed arrows indicate error cascades and defense interventions. The diagram highlights that attacks on upstream modules (perception) can cascade downstream, amplifying impact. rent LLM-based agents often lack internal world models, leading to repeated irreversible actions and cascading errors in long-horizon tasks (Chae et al., 2024). More broadly, the inability to rea- son about long-term consequences fundamentally limits grounding in dynamic environments (Piatti et al., 2024). Recent analyses further reveal a mis- match between reasoning and execution: correct reasoning does not guarantee successful execution, and successful execution may conceal flawed rea- soning (Dong et al., 2025). Interaction Fidelity depends on precise action execution under variable interface conditions. Even when an agent correctly identifies a target element, mapping it to reliable pixel-level actions remains error-prone across screen resolutions, layouts, and device types (Zhao et al., 2024). These challenges are amplified in mobile environments, where agents must handle diverse screen sizes, touch interactions, and platform-specific behaviors (Yang et al., 2024; Nong et al., 2024). 2.3 Why Standard LLM Safety Falls Short Conventional LLM safety mechanisms, such as output filtering, refusal training, and alignment, are designed for static, text-based interactions (Liu et al., 2023c). They assume that outputs can be reviewed before causing harm (e.g., users can de- tect and ignore unsafe responses), and that fail- ures occur in isolated interactions. GUI agents violate these assumptions: actions execute immedi- ately, consequences may be opaque to users, and er- rors compound through closed actionâobservation loops (Kumar et al., 2024). Empirical evidence shows that refusal-trained LLMs often fail to preserve safety behaviors when deployed within agents, even when the same back- bone model behaves safely in chatbot settings (Ku- mar et al., 2024). This breakdown in safety trans- fer indicates that conversation-centric alignment may not generalize to agentic execution. More- over, the compositional nature of GUI agents in- troduces multiple interacting attack surfaces that are not captured by existing LLM safety evalua- tions (Gan et al., 2024; Wu et al., 2025a). Recent studies suggest that stronger reasoning capabili- ties can amplify catastrophic risks in autonomous agents, including deceptive behavior and unsafe autonomous action (Xu et al., 2025). 3 Perception Trust Perception trust focuses on whether agents cor- rectly interpret observed interface states. Because errors at this stage propagate downstream, percep- tual robustness is foundational to overall trustwor- thiness. We categorize perception failures into two classes: visual hallucination and adversarial attacks, as shown in Figure 2. 3.1 The Visual Hallucination Problem Visual hallucination, acting on nonexistent ele- ments or misinterpreting existing ones, is a percep- 3 Inputs (Screenshot & DOM) GUI AgentPerception (MLLM) & Failures Defenses & Outcomes Organic Hallucination Semantic Interpretation: Delete Account (Misinterpretati on of Attribute) Example: Misinterpreting element function or attribute Adversarial Attack (Visual Perturbation) Perception Module (MLLM) Adversari al Attack (Structural Injection) Semantic Interpretation: Execute Malicious Instruction (Hidden Command) Example: WIPI, AdvWeb (Hidden DOM text) GUI Agent Perception (MLLM) & Failures Defensive Mechanisms Cross-Modal Verification Inconsistency Deleted Correct Semantic State (Verified) Action: Click "Submit Order" button Incorrect Semantic State (Failure/Attack) Action: [Erroneous or Malicious Action] Semantic Interpretation: Buy Now (Targeted Hijack) Example: Imprompter, ARE (Imperceptible noise) Figure 2: Failures and Defenses. The diagram illus- trates how organic hallucinations and adversarial attacks (visual & structural) distort the agentâs semantic inter- pretation of the GUI. Defensive mechanisms like input filtering and cross-modal verification are shown as in- terventions to ensure a correct semantic state. tion failure mode in GUI agents (Bai et al., 2024; Chen et al., 2024b). Prior work identifies multiple hallucination mechanisms. Liu et al. (2023a) de- scribe object hallucination, where agents perceive UI elements absent from screenshots, a problem exacerbated in mobile settings with repetitive de- sign patterns. Jiang et al. (2024a) analyze attribute hallucination, where agents misperceive elemental properties such as color or position. More critically, Zhong et al. (2024) observe hallucination snow- balling, where early perceptual errors bias subse- quent interpretations, producing self-reinforcing failure cascades. Hallucination also interacts tightly with safety reasoning.The multimodal situational safety benchmark of Zhou et al. (2024) demonstrates that even safety-aligned MLLMs fail when visual un- derstanding is inaccurate, indicating that perceptual errors and safety violations are deeply coupled. Existing work largely treats hallucination as a training defect. We argue instead that hallucina- tion could reflect rational inference under percep- tual uncertainty. The core limitation is the ab- sence of mechanisms for uncertainty recognition and signaling. Recent uncertainty-aware training approaches (Shi et al., 2023b; Chen et al.) demon- strate that explicit uncertainty estimation can im- prove both agent reliability and trajectory evalua- tion, suggesting a promising direction for percep- tion trust. 3.2 Adversarial Perception Attacks Beyond organic failures, adversaries can deliber- ately exploit the grounding gap between human- visible interfaces and model-perceived representa- tions. Existing attacks fall into three categories. Visual perturbation attacks manipulate pixel- level inputs in ways imperceptible to humans but effective against models. Imprompter (Fu et al., 2024) and ARE (Wu et al., 2025a) show that mini- mal perturbations can reliably hijack agent behav- ior across multiple LLM backends, with success rates exceeding 60â80%. Systematic evaluations further confirm that GUI grounding models are highly sensitive to both natural noise and adversar- ial perturbations (Zhao et al., 2025). Structural injection attacks embed malicious instructions within DOM or HTML structures in- visible to users. WIPI (Wu et al., 2024a) and Ad- vWeb (Xu et al., 2024) demonstrate that indirect prompt injection via webpages can control agents in black-box settings with success rates above 90%. Fine-print injections (Chen et al., 2025a) further reveal that agents disproportionately attend to struc- turally salient but visually subtle content, rendering human oversight insufficient. Environmental and overlay attacks exploit agentsâ misinterpretation of authority and saliency cues. Adversarial pop-ups (Zhang et al., 2025) and evolving injection strategies such as EVA (Lu et al., 2025) significantly degrade task success. On mo- bile platforms, overlay attacks masquerading as system dialogs achieve attack success rates exceed- ing 90% (Yang et al., 2024; Chen et al., 2025d), while environmental injection attacks covertly ex- tract sensitive information by manipulating agent- environment interactions (Liao et al., 2024). Across modalities, these attacks exploit a shared weakness: mismatches between appearance, struc- ture, and intent representations. As a result, agents may form plausible yet incorrect interpretations of interface elements. This motivates defenses based on cross-modal consistency, rather than reliance on a single interface view. 4 3.3 Perception Defense Mechanisms Defenses against perception attacks operates across multiple stages of the perception pipeline. Existing approaches can be grouped into three categories. Input filtering aims to block malicious content before core processing. This includes classifiers for detecting prompt injection (Sharma et al., 2024), image purification methods for mitigating visual perturbations (Shi et al., 2023a), and heuristic rules for identifying suspicious DOM patterns such as hidden text or instruction-like content (Wu et al., 2024a). While effective against known attacks, static filters require continual updates and struggle against adaptive adversaries. Cross-modal verification leverages redundancy across perception modalities to detect inconsisten- cies. Discrepancies between screenshots and DOM structures, such as visually present elements absent from structural representations, can indicate manip- ulation. The ARE framework (Wu et al., 2025a) suggests that attacks typically enter through one modality but influence behavior through another, highlighting the potential of cross-modal checks. However, practical deployment remains limited by computational cost and the difficulty of formalizing consistency across heterogeneous representations. Output calibration mitigates perceptual risk at the decision stage rather than the input. CoCA (Gao et al., 2024) enhances safety awareness by condi- tioning MLLM outputs on explicit safety princi- ples, partially compensating for modality-induced degradation.Evaluation suites such as M- SafetyBench (Liu et al., 2023b) provide standard- ized assessment of manipulation resistance but do not directly prevent attacks. Open Problem. Robust, scalable cross-modal consistency checking remains largely unexplored and represents a central open challenge for percep- tion trust in GUI agents. 4 Reasoning Trust Reasoning trust focuses on whether agents make sound decisions given imperfect perceptions and evolving environments. Unlike static text gener- ation, GUI agents must sustain goal-aligned rea- soning over long action sequences, where errors accumulate and assumptions frequently break. Fig- ure 3 illustrates how these challenges intensify over extended interaction horizons. User Intent Reasoning Agent (Planner & Decision Maker) Initial GUI State Step 1: Correct Action State Update (Accurate) Step N: Negative Outcome (Failure / Harm) Step N: Goal Achieved (Success) Step 2: Correct Action Step 1: Action with Error (Compounding Uncertainty) State Update (Partial/Noisy) Step 2: Divergent Action (Plan Derailment) ...... Path A: Goal-Aligned Reasoning (Ideal) Defenses: Guardrails, World Models Defenses: Guardrails, World Models Path B: Reasoning Failures & Attacks Attack: Goal Hijacking Horizon Problem Non- Stationarity Unintended Consequences Time / Execution Steps Figure 3: Reasoning Trust and the Horizon Problem. The diagram illustrates how reasoning challenges like the horizon problem, compounding errors, and adversar- ial attacks can derail a GUI agent from its intended goal over time, contrasting with an ideal, defense-enhanced trajectory. 4.1 The Horizon Problem GUI tasks often require dozens of sequential ac- tions, creating an exponential growth in possible states as execution unfolds. This horizon problem makes maintaining coherent plans increasingly dif- ficult (Chae et al., 2024; Gu et al., 2024). Compounding Uncertainty. Even modest per- step error rates rapidly degrade task success: a 95% accurate policy succeeds only 36% of the time over 20 steps. In practice, per-step accuracy is far lower on complex interfaces (Kim et al., 2024b). TrustA- gent (Hua et al., 2024) further shows that safety awareness decays over long trajectories, with early- identified risks often ignored in later decisions. Partial Observability. Agents observe only the current interface state; critical information may re- side in background tabs, system dialogs, or hidden application states. Planning under such partial ob- servability is provably harder, yet most agents im- plicitly assume complete state information (Zhang et al., 2023). Non-Stationarity. The environment can change during execution due to system processes, network events, or human interaction. Plans generated un- 5 der static assumptions frequently fail when condi- tions shift (Ma et al., 2024). The lack of internal world models prevents agents from reasoning about long-term consequences, leading to repeated irre- versible mistakes (Chae et al., 2024). 4.2 Goal Alignment and Manipulation Beyond organic failures, reasoning trust is under- mined by attacks that exploit mismatches between user intent and agent interpretation. Goal Hijacking. Indirect instruction injection can override user goals, particularly in non-chat set- tings where refusal training fails to generalize (Ku- mar et al., 2024). Browser-based evaluations show safety-trained agents engaging in harmful behav- iors in a majority of tested scenarios. Web fraud attacks further exploit weaknesses in intent infer- ence, enabling stealthy manipulation without ex- plicit jailbreak prompts (Kong et al., 2025; Liang et al., 2025). Norm Violations. Reasoning failures also man- ifest as cultural and social norm violations. The CASA benchmark (Qiu et al., 2024) reports less than 10% norm awareness under evaluated set- tings and over 40% violation rates, indicating that agents struggle to reason about appropriate behav- ior across social contexts, an important dimension of trustworthiness. Multi-Agent Failures. As systems adopt multi- ple specialized agents, coordination becomes frag- ile. Most LLM-based agents fail to reach stable cooperation due to inability to reason about long- term group dynamics (Piatti et al., 2024). Only the strongest models achieve sustained coordination, underscoring the difficulty of distributed reasoning. 4.3 Reasoning Defense Mechanisms Defenses against reasoning failures can be grouped into three complementary strategies. Enhanced planning architectures mitigate the horizon problem through improved internal reason- ing. World-model-based approaches such as Web- Dreamer (Gu et al., 2024) simulate action outcomes before execution, while hierarchical planners sepa- rate strategic goals from tactical actions to enable re-planning (Liu et al., 2025; Nong et al., 2024). External verification systems introduce inde- pendent checks on reasoning. Guardrail agents (Xi- ang et al., 2024; Zheng et al., 2025) validate high- risk actions prior to execution, while critics such as GUI-Critic-R1 (Wanyan et al., 2025) assess poten- tial outcomes in advance. Multi-agent verification Agent's Intended Action (e.g., "Click 'Pay Now'") Interaction Gap & Grounding Challenges (Dynamic Layouts, Resolution Mismatch) Deleted Account (X) Confirm Purchase ($) Grounding Failure (Irreversible Data Loss) Privacy Risk (Screenshot Leakage of Credentials) Intended Outcome (Transaction Complete) Mitigated Outcome (Safe Termination / User Alert) Defense Mechanisms Risk-Aware Execution (Human Confirmation, Sandboxing) Access Control & Privacy Preservation (View Restriction, Data Masking) Real-World Consequences (Financial Loss, Privacy Violation) Figure 4: Interaction Trust: Risks and Defenses in Ex- ecution. The figure illustrates how coordinate ground- ing failures and privacy risks can lead to irreversible real-world consequences. Defense mechanisms like risk-aware execution and access control intervene to mitigate these threats. improves coverage (Yu et al., 2024; Sengupta et al., 2024) but incurs substantial computational cost. BlindGuard (Miao et al., 2025) extends verification to unsupervised settings without attack-specific la- bels. Training-time interventions embed safety di- rectly into reasoning. TrustAgent (Hua et al., 2024) adapts constitutional AI to agentic planning, while process reward models like GUI-Shepherd (Chen et al., 2025c) provide step-level feedback for long- horizon tasks. RapGuard (Jiang et al., 2024b) dy- namically generates context-aware safety prompts using multimodal chain-of-thought reasoning. Open Problem. Despite these advances, reliable long-horizon reasoning remains unresolved. Future progress may require architectures that decompose tasks into independently verifiable subgoals. 5 Interaction Trust Interaction trust focuses on whether agents exe- cute intended actions correctly and safely. Because this stage directly affects real systems, errors are often immediate and irreversible. We examine ir- reversibility, coordinate grounding, privacy risks, and defenses, as illustrated in Figure 4. 6 5.1 The Irreversibility Challenge GUI interactions differ fundamentally from internal reasoning: actions are executed in external systems, where their effects persist beyond the agentâs con- trol and are often difficult or impossible to undo. Three classes of problems are particularly critical. Destructive actions modify or delete data (e.g., file deletion, form submission) and may be un- recoverable. The Responsible Task Automation framework (Zhang et al., 2023) emphasizes feasi- bility and consequence prediction as prerequisites for safe execution. Financial actions commit re- sources through purchases or transfers and often require human intervention to reverse. Benchmarks show that agents readily attempt such actions with- out sufficient verification (Levy et al., 2024), ex- posing a gap between capability and caution. Au- thorization actions grant permissions via OAuth flows or access sharing, creating persistent security risks. Mobile agents are especially vulnerable to manipulation through fake system dialogs and over- lays (Yang et al., 2024). Most agents treat all ac- tions uniformly, applying identical execution logic to low- and high-stakes operations. This neglects consequence severity and represents a fundamental limitation for trustworthy interaction (Hua et al., 2024). Pre-execution critics (Wanyan et al., 2025) partially address this by evaluating action correct- ness and impact before execution. 5.2 Coordinate Grounding Failures Even with correct perception and reasoning, inter- action can fail due to imprecise action grounding. Resolution sensitivity arises when models trained on fixed resolutions misplace actions on different screen sizes. Hybrid encoders mitigate but do not eliminate this issue (Nong et al., 2024). Dynamic layouts reposition elements across win- dow sizes, zoom levels, and device orientations. Combining structural and visual cues improves robustness, yet large gaps remain relative to or- acle grounding (Wang et al., 2024a). Temporal effects such as animations and transitions can ren- der elements temporarily non-interactive, causing premature or misaligned actions. Evaluations in the GUI Testing Arena show persistent failure modes even for advanced models (Zhao et al., 2024). State-dependent controls introduce addi- tional complexity: toggle actions depend on current state. StaR (Wu et al., 2025c) demonstrates that ex- plicit state-aware reasoning improves performance on such tasks by over 30%. Explainable interac- tion frameworks such as EBC-LLMAgent (Guan et al., 2024) improve grounding by explicitly map- ping actions to UI elements, illustrating the benefits of structured interaction over end-to-end predic- tion. 5.3 Privacy Risks in Interaction Interaction introduces privacy risks that extend be- yond immediate task execution. Screenshot leak- age occurs when perception captures sensitive on- screen information such as credentials or medical data (Chen et al., 2024a). CLEAR (Chen et al., 2024a) mitigates this by exposing privacy risks and policies to users. Contextual exposure arises from action traces themselves, which can reveal private user behaviors even without explicit sensitive con- tent (Kim et al., 2024a; Ngong et al., 2025). Envi- ronmental injection attacks exploit this channel to extract personal information (Liao et al., 2024). 5.4 Interaction Defense Mechanisms Defenses must balance safety against usability un- der irreversible execution. Existing approaches fall into three categories. Risk-aware execution differentiates actions by consequence. Low-risk actions execute directly, while high-risk actions require verification or sand- boxing (OpenAI, 2025; Anthropic, 2025). ST- WebAgentBench (Levy et al., 2024) evaluates com- pliance under safety constraints, while formal ver- ification approaches such as VeriSafe Agent (Lee et al., 2025) translate instructions into verifiable specifications, achieving 94â98% accuracy. Graph- based methods like G-Safeguard (Wang et al., 2025) detect anomalous action patterns indicative of failures or attacks. Human oversight preserves user control for con- sequential actions. Confirmation prompts provide a safety valve but risk fatigue if overused (OpenAI, 2025). VeriOS (Wu et al., 2025b) improves this through proactive querying, selectively requesting human input when trustworthiness is low. Access control and privacy preservation con- strain damage even under failure. Capability min- imization limits available actions, while authenti- cated delegation provides cryptographic guarantees against goal hijacking (South et al., 2025). Privacy- preserving architectures such as PAPILLON (Siyan et al., 2024) and EcoAgent (Yi et al., 2025) reduce sensitive data exposure through selective routing and on-device verification. 7 Open Problem. Current risk classification re- mains context-insensitive: identical actions may range from benign to high-stakes depending on en- vironment. Context-aware risk assessment for GUI actions remains an open challenge. 6 Evaluation Methodologies Rigorous evaluation is essential for measuring progress in trustworthy GUI agents. This section reviews existing benchmarks and examines the se- curityâutility trade-off that governs practical de- ployment. 6.1 Trust-Aware Metrics Early benchmarks such as WebArena (Zhou et al., 2023), VisualWebArena (Koh et al., 2024a), and Mind2Web (Deng et al., 2023) primarily measure task completion. While useful for assessing ca- pability, they are insufficient for trustworthiness evaluation: policy violations are not penalized, fail- ure modes are opaque, and collateral effects are ignored. Recent benchmarks address these gaps by explicitly targeting trust-related behaviors. Ta- ble 1 in the appendix summarizes representative frameworks, organized by the pipeline stage they evaluate, along with their key metrics, innovations, and limitations. 6.2 Evaluation Dimensions Beyond existing benchmarks, we identify addi- tional dimensions necessary for comprehensive trust evaluation. Cascade metrics quantify error propagation across action sequences, capturing detection rate, recovery success, and failure depth.GUI- Shepherd (Chen et al., 2025c) enables such analysis through step-level rewards. Uncertainty calibra- tion measures whether agent confidence reflects true success likelihood. URST (Chen et al.) shows that uncertainty-aware sampling improves trajec- tory assessment. Reasoningâexecution alignment evaluates consistency between internal reasoning and executed actions. Ground-Truth Alignment (GTA) (Dong et al., 2025) distinguishes execution gaps (correct reasoning, failed action) from rea- soning gaps (successful action, flawed reasoning). Explainability supports oversight by enabling hu- mans to interpret agent decisions. XAgent (Nguyen et al., 2024b) and XMODE (Nooralahzadeh et al., 2024) demonstrate improved humanâAI collabora- tion through interpretable reasoning. 6.3 The SecurityâUtility Trade-off Trustworthy deployment requires balancing au- tomation benefits against risk. Security postures range from full autonomy (high utility, high risk) to full supervision (low risk, minimal automation), with intermediate strategies such as confirmation checkpoints and watch modes (OpenAI, 2025). Optimal trade-offs depend on context, includ- ing action reversibility, financial stakes, user ex- pertise, and regulatory constraints. Existing sys- tems reflect different choices: OpenAIâs Computer- Using Agent adopts watch modes for sensitive ac- tions (OpenAI, 2025); Anthropicâs Computer Use beta restricts social interactions (Anthropic, 2025); GuardAgent enforces per-action verification (Xi- ang et al., 2024). Static policies are often suboptimal. Adaptive autonomy, which adjusts oversight based on real- time risk, offers a more effective alternative. Ve- riOS (Wu et al., 2025b) exemplifies this approach, dynamically querying humans in untrustworthy scenarios and improving success rates by approxi- mately 20%. 7 Conclusion This survey has reframed GUI agent trustworthi- ness through the lens of the Execution Gap, the fundamental challenge of maintaining faithful map- pings between perception, reasoning, and interac- tion. By organizing analysis around the agentic workflow rather than traditional safety categories, we reveal how vulnerabilities propagate and com- pound across pipeline stages. Three central insights emerge from our analysis. First, GUI-specific challenges, irreversibility, dy- namic environments, and action-observation loops, demand approaches beyond standard LLM safety techniques. Solutions must account for the closed- loop nature of agent operation where each action changes the environment affecting subsequent ob- servations. Second, the security-utility trade-off is not merely a deployment consideration but a fun- damental research challenge. Achieving both high autonomy and high safety requires architectural innovation, not just better policies. Third, current evaluation practices are misaligned with trustwor- thiness goals. Moving beyond completion metrics to assess safety, robustness, and alignment is essen- tial for meaningful progress. 8 Limitations This survey has several limitations. First, the rapid pace of development means some recent work may be inadvertently omitted. Second, our taxonomy, while designed for clarity, may not capture all nu- ances of specific approaches. Third, the security- utility trade-off analysis relies partly on qualita- tive assessment where quantitative data is unavail- able. Fourth, our proposed future directions, while grounded in identified challenges, remain specula- tive until empirically validated. Finally, as primar- ily English-language researchers, our coverage of non-English work may be incomplete. Ethics Statement This survey discusses attack techniques and vul- nerabilities. We include such discussion because understanding threats is necessary for developing defenses. We have avoided providing implementa- tion details that would lower barriers to malicious use. All discussed attacks are from published re- search intended to improve system security. Addi- tionally, AI assistants were used only for language editing and stylistic revision, including improving clarity, conciseness, and grammar. References Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, et al. 2024. Agentharm: A benchmark for measuring harmfulness of llm agents. arXiv preprint arXiv:2410.09024. Anthropic. 2025. Agents and tools: Computer use. Ac- cessed: March 16, 2025. Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of Multimodal Large Language Models: A Survey. arXiv.org. Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sungh- wan Kim, Dongha Lee, and Jinyoung Yeo. 2024. Web agents with world models: Learning and leverag- ing environment dynamics in web navigation. arXiv preprint arXiv:2410.13232. Chaoran Chen, Zhiping Zhang, Bingcan Guo, Shang Ma, Ibrahim Khalilov, Simret A Gebreegziabher, Yanfang Ye, Ziang Xiao, Yaxing Yao, Tianshi Li, et al. 2025a. The obvious invisible threat: Llm- powered gui agentsâ vulnerability to fine-print in- jections. arXiv preprint arXiv:2504.11281. Chaoran Chen, Daodao Zhou, Yanfang Ye, Toby Li, and Yaxing Yao. 2024a. Clear: Towards contextual llm-empowered privacy policy analysis and risk gen- eration for large language model applications. arXiv preprint arXiv:2410.13387. Chiyu Chen, Xinhao Song, Yunkai Chai, Yang Yao, Haodong Zhao, Lijun Li, Jie Li, Yan Teng, Gongshen Liu, and Yingchun Wang. 2025b. Ghostei-bench: Do mobile agents resilience to environmental injection in dynamic on-device environments? arXiv preprint arXiv:2510.20333. Cong Chen, Kaixiang Ji, Hao Zhong, Muzhi Zhu, Anzhou Li, Guo Gan, Ziyuan Huang, Cheng Zou, Jiajia Liu, Jingdong Chen, et al. 2025c.Gui- shepherd: Reliable process reward and verifica- tion for long-sequence gui tasks. arXiv preprint arXiv:2509.23738. Gongwei Chen, Lirong Jie, Lexiao Zou, Weili Guan, Miao Zhang, and Liqiang Nie. Enhancing gui agent with uncertainty-aware self-trained evaluator. In The Thirty-ninth Annual Conference on Neural Informa- tion Processing Systems. Xiang Chen, Chenxi Wang, Yida Xue, Ningyu Zhang, Xiaoyan Yang, Qiang Li, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. 2024b. Unified Hallucination Detection for Multimodal Large Language Models. Annual Meeting of the Association for Computational Linguistics. Yurun Chen, Xueyu Hu, Keting Yin, Juncheng Li, and Shengyu Zhang. 2025d. Aeia-mn: Evaluating the robustness of multimodal llm-powered mobile agents against active environmental injection attacks. arXiv preprint arXiv:2502.13053. Pengzhou Cheng, Lingzhong Dong, Zeng Wu, Zon- gru Wu, Xiangru Tang, Chengwei Qin, Zhuosheng Zhang, and Gongshen Liu. 2025. Agent-scankit: Unraveling memory and reasoning of multimodal agents via sensitivity perturbations. arXiv preprint arXiv:2510.00496. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091â28114. Lingzhong Dong, Ziqi Zhou, Shuaibo Yang, Haiyue Sheng, Pengzhou Cheng, Zongru Wu, Zheng Wu, Gongshen Liu, and Zhuosheng Zhang. 2025. Say one thing, do another? diagnosing reasoning-execution gaps in vlm-powered mobile-use agents. arXiv preprint arXiv:2510.02204. Ivan Evtimov, Arman Zharmagambetov, Aaron Grattafiori, Chuan Guo, and Kamalika Chaudhuri. 2025. Wasp: Benchmarking web agent security against prompt injection attacks. arXiv preprint arXiv:2504.18575. 9 Xiaohan Fu, Shuheng Li, Zihan Wang, Yihao Liu, Ra- jesh K. Gupta, Taylor Berg-Kirkpatrick, and Ear- lence Fernandes. 2024.Imprompter: Tricking llm agents into improper tool use. arXiv preprint arXiv:2410.14923. Yuyou Gan, Yong Yang, Zhen Ma, Ping He, Rui Zeng, Yiming Wang, Qingming Li, Chunyi Zhou, Songze Li, Ting Wang, Yunjun Gao, Yingcai Wu, and Shoul- ing Ji. 2024. Navigating the risks: A survey of secu- rity, privacy, and ethics threats in llm-based agents. arXiv preprint arXiv:2411.09523. Jiahui Gao, Renjie Pi, Tianyang Han, Han Wu, Chenyang Lyu, Huayang Li, Lanqing Hong, Ling- peng Kong, Xin Jiang, and Zhenguo Li. 2024. Coca: Regaining safety-awareness of multimodal large lan- guage models with constitutional calibration. arXiv preprint arXiv:2409.11365. Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2024. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243. Yu Gu, Boyuan Zheng, Boyu Gou, Kai Zhang, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, Huan Sun, and Yu Su. 2024. Is your llm secretly a world model of the internet? model-based planning for web agents. arXiv preprint arXiv:2411.06559. Yanchu Guan, Dong Wang, Yan Wang, Haiqing Wang, Renen Sun, Chenyi Zhuang, Jinjie Gu, and Zhixuan Chu. 2024. Explainable behavior cloning: Teach- ing large language model agents through learning by demonstration. arXiv preprint arXiv:2410.22916. Wenyue Hua, Xianjun Yang, Zelong Li, Cheng Wei, and Yongfeng Zhang. 2024. Trustagent: Towards safe and trustworthy llm-based agents. Conference on Empirical Methods in Natural Language Processing. Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. 2024a. Hallucination Augmented Contrastive Learning for Multimodal Large Language Model. In 2024 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 27026â27036. IEEE. Yilei Jiang, Yingshui Tan, and Xiangyu Yue. 2024b. Rapguard: Safeguarding multimodal large language models via rationale-aware defensive prompting. arXiv preprint arXiv:2412.18826. Su Kara, Fazle Faisal, and Suman Nath. 2025. Waber: Evaluating reliability and efficiency of web agents with existing benchmarks. Hanna Kim, Minkyoo Song, Seung Ho Na, Seungwon Shin, and Kimin Lee. 2024a. When llms go online: The emerging threat of web-enabled llms. arXiv preprint arXiv:2410.14569. Jaekyeom Kim, Dong-Ki Kim, Lajanugen Logeswaran, Sungryull Sohn, and Honglak Lee. 2024b. Auto- intent:Automated intent discovery and self- exploration for large language model web agents. arXiv preprint arXiv:2410.22552. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024a.Visualwebarena: Evaluat- ing multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649. Jing Yu Koh, Stephen McAleer, Daniel Fried, and Rus- lan Salakhutdinov. 2024b. Tree search for language model agents. arXiv preprint arXiv:2407.01476. Dezhang Kong, Hujin Peng, Yilun Zhang, Lele Zhao, Zhenhua Xu, Shi Lin, Changting Lin, and Meng Han. 2025. Web fraud attacks against llm-driven multi- agent systems. arXiv preprint arXiv:2509.01211. Priyanshu Kumar, Elaine Lau, Saranya Vijayaku- mar, Tu Trinh, Scale Red Team, Elaine Chang, Vaughn Robinson, Sean Hendryx, Shuyan Zhou, Matt Fredrikson, Summer Yue, and Zifan Wang. 2024. Refusal-trained llms are easily jailbroken as browser agents. arXiv preprint arXiv:2410.13886. Jungjae Lee, Dongjae Lee, Chihun Choi, Youngmin Im, Jaeyoung Wi, Kihong Heo, Sangeun Oh, Sunjae Lee, and Insik Shin. 2025. Verisafe agent: Safeguarding mobile gui agent via logic-based action verification. arXiv preprint arXiv:2503.18492. Juyong Lee, Dongyoon Hahm, June Suk Choi, W Bradley Knox, and Kimin Lee. 2024.Mo- bilesafetybench: Evaluating safety of autonomous agents in mobile device control. arXiv preprint arXiv:2410.17520. Ido Levy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yaeli, and Segev Shlomov. 2024.St- webagentbench: A benchmark for evaluating safety and trustworthiness in web agents. arXiv preprint arXiv:2410.06703. Ruichao Liang, Le Yin, Jing Chen, Cong Wu, Xiaoyu Zhang, Huangpeng Gu, Zijian Zhang, and Yang Liu. 2025. Tipping the dominos: Topology-aware multi- hop attacks on llm-based multi-agent systems. arXiv preprint arXiv:2512.04129. Zeyi Liao, Lingbo Mo, Chejian Xu, Mintong Kang, Jiawei Zhang, Chaowei Xiao, Yuan Tian, Bo Li, and Huan Sun. 2024. Eia: Environmental injection attack on generalist web agents for privacy leakage. arXiv preprint arXiv:2409.11295. Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023a. Mitigating Hal- lucination in Large Multi-Modal Models via Robust Instruction Tuning. arXiv.org. 10 Haowei Liu, Xi Zhang, Haiyang Xu, Yuyang Wanyan, Junyang Wang, Ming Yan, Ji Zhang, Chunfeng Yuan, Changsheng Xu, Weiming Hu, et al. 2025. Pc-agent: A hierarchical multi-agent collaboration framework for complex task automation on pc. arXiv preprint arXiv:2502.14282. Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao. 2023b. Mm-safetybench: A benchmark for safety evaluation of multimodal large language models. European Conference on Com- puter Vision. Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023c. Trust- worthy llms: a survey and guideline for evaluating large language modelsâ alignment. arXiv preprint arXiv:2308.05374. Qinghua Lu, Liming Zhu, Xiwei Xu, Zhenchang Xing, Stefan Harrer, and Jon Whittle. 2023. Towards re- sponsible generative ai: A reference architecture for designing foundation model based agents. In 2024 IEEE 21st International Conference on Software Ar- chitecture Companion (ICSA-C). IEEE. Yijie Lu, Tianjie Ju, Manman Zhao, Xinbei Ma, Yuan Guo, and ZhuoSheng Zhang. 2025.Eva: Red- teaming gui agents via evolving indirect prompt in- jection. arXiv preprint arXiv:2505.14289. Xinbei Ma, Yiting Wang, Yao Yao, Tongxin Yuan, Aston Zhang, Zhuosheng Zhang, and Hai Zhao. 2024. Cau- tion for the environment: Multimodal agents are sus- ceptible to environmental distractions. arXiv preprint arXiv:2408.02544. Rui Miao, Yixin Liu, Yili Wang, Xu Shen, Yue Tan, Yiwei Dai, Shirui Pan, and Xin Wang. 2025. Blindguard: Safeguarding llm-based multi-agent systems under unknown attacks. arXiv preprint arXiv:2508.08127. Ivoline Ngong, Swanand Kadhe, Hao Wang, Keerthiram Murugesan, Justin D Weisz, Amit Dhurandhar, and Karthikeyan Natesan Ramamurthy. 2025. Protecting users from themselves: Safeguarding contextual pri- vacy in interactions with conversational agents. arXiv preprint arXiv:2502.18509. Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namy- ong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, Xintong Li, Jing Shi, Hongjie Chen, Viet Dac Lai, Zhouhang Xie, Sungchul Kim, Ruiyi Zhang, Tong Yu, Mehrab Tanjim, Nesreen K. Ahmed, Puneet Mathur, Seunghyun Yoon, Lina Yao, Branislav Kveton, Thien Huu Nguyen, Trung Bui, Tianyi Zhou, Ryan A. Rossi, and Franck Dernon- court. 2024a. GUI Agents: A Survey. arXiv preprint. ArXiv:2412.13501 [cs]. Van Bach Nguyen, Jörg Schlötterer, and Christin Seifert. 2024b. Xagent: A conversational xai agent harness- ing the power of large language models. xAI. Songqin Nong, Jiali Zhu, Rui Wu, Jiongchao Jin, Shuo Shan, Xiutian Huang, and Wenhao Xu. 2024. Mo- bileflow: A multimodal llm for mobile gui agent. arXiv preprint arXiv:2407.04346. Farhad Nooralahzadeh, Yi Zhang, Jonathan Furst, and Kurt Stockinger. 2024. Explainable multi-modal data exploration in natural language via llm agent. arXiv preprint arXiv:2412.18428. OpenAI. 2025.Computer-using agent.Accessed: March 16, 2025. Giorgio Piatti, Zhijing Jin, Max Kleiman-Weiner, Bern- hard Schölkopf, Mrinmaya Sachan, and Rada Mi- halcea. 2024. Cooperate or collapse: Emergence of sustainable cooperation in a society of llm agents. Advances in Neural Information Processing Systems, 37:111715â111759. Haoyi Qiu, A. R. Fabbri, Divyansh Agarwal, Kung- Hsiang Huang, Sarah Tan, Nanyun Peng, and Chien- Sheng Wu. 2024.Evaluating cultural and so- cial awareness of llm web agents. arXiv preprint arXiv:2410.23252. Saptarshi Sengupta, Kristal Curtis, Akshay Mallipeddi, Abhinav Mathur, Joseph Ross, and Liang Gou. 2024. Mag-v: A multi-agent framework for syn- thetic data generation and verification. arXiv preprint arXiv:2412.04494. Reshabh K Sharma, Vinayak Gupta, and Dan Grossman. 2024. Defending language models against image- based prompt attacks via user-provided specifications. 2024 IEEE Security and Privacy Workshops (SPW). Yucheng Shi, Mengnan Du, Xuansheng Wu, Zihan Guan, Jin Sun, and Ninghao Liu. 2023a. Black-box backdoor defense via zero-shot image purification. Advances in Neural Information Processing Systems, 36:57336â57366. Zhelun Shi, Zhipin Wang, Hongxing Fan, Zhen-fei Yin, Lu Sheng, Yu Qiao, and Jing Shao. 2023b. Chef: A comprehensive evaluation framework for standard- ized assessment of multimodal large language mod- els. arXiv preprint arXiv:2311.02692. Li Siyan, Vethavikashini Chithrra Raghuram, Omar Khattab, Julia Hirschberg, and Zhou Yu. 2024. Papil- lon: Privacy preservation from internet-based and local language model ensembles. arXiv preprint arXiv:2410.17127. Tobin South, Samuele Marro, Thomas Hardjono, Robert Mahari, Cedric Deslandes Whitney, Dazza Green- wood, Alan Chan, and Alex Pentland. 2025. Authen- ticated delegation and authorized ai agents. arXiv preprint arXiv:2501.09674. Sakhinana Sagar Srinivas, Geethan Sannidhi, and Venkataramana Runkana. 2024. Towards human- level understanding of complex process engineer- ing schematics: A pedagogical, introspective multi- agent framework for open-domain question answer- ing. arXiv preprint arXiv:2409.00082. 11 Shilong Wang, Guibin Zhang, Miao Yu, Guancheng Wan, Fanci Meng, Chongye Guo, Kun Wang, and Yang Wang. 2025. G-safeguard: A topology-guided security lens and treatment on llm-based multi-agent systems. arXiv preprint arXiv:2502.11127. Siyi Wang, Sinan Wang, Yujia Fan, Xiaolei Li, and Yepang Liu. 2024a.Leveraging large vision- language model for better automatic web gui testing. IEEE International Conference on Software Mainte- nance and Evolution. Yuntao Wang, Yanghe Pan, Quan Zhao, Yi Deng, Zhou Su, Linkang Du, and Tom H Luan. 2024b. Large model agents: State-of-the-art, cooperation paradigms, security and privacy, and future trends. arXiv preprint arXiv:2409.14457. Yuyang Wanyan, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Jiabo Ye, Yutong Kou, Ming Yan, Fei Huang, Xiaoshan Yang, et al. 2025. Look before you leap: A gui-critic-r1 model for pre-operative error diagnosis in gui automation. arXiv preprint arXiv:2506.04614. Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. 2022. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM conference on fairness, accountability, and transparency, pages 214â229. Chen Henry Wu, Rishi Rajesh Shah, Jing Yu Koh, Russ Salakhutdinov, Daniel Fried, and Aditi Raghunathan. 2025a. Dissecting adversarial robustness of multi- modal lm agents. In The Thirteenth International Conference on Learning Representations. Fangzhou Wu, Shutong Wu, Yulong Cao, and Chaowei Xiao. 2024a. Wipi: A new web threat for llm-driven web agents. arXiv preprint arXiv:2402.16965. Zheng Wu, Heyuan Huang, Xingyu Lou, Xiangmou Qu, Pengzhou Cheng, Zongru Wu, Weiwen Liu, Weinan Zhang, Jun Wang, Zhaoxiang Wang, et al. 2025b. Verios: Query-driven proactive human-agent-gui in- teraction for trustworthy os agents. arXiv preprint arXiv:2509.07553. Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. 2024b. Os- atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218. Zongru Wu, Rui Mao, Zhiyuan Tian, Pengzhou Cheng, Tianjie Ju, Zheng Wu, Lingzhong Dong, Haiyue Sheng, Zhuosheng Zhang, and Gongshen Liu. 2025c. See, think, act: Teaching multimodal agents to effec- tively interact with gui by identifying toggles. arXiv preprint arXiv:2509.13615. Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, Dawn Song, and Bo Li. 2024. Guardagent: Safeguard llm agents by a guard agent via knowledge-enabled reasoning. arXiv preprint arXiv:2406.09187. Junlin Xie, Zhihong Chen, Ruifei Zhang, Xiang Wan, and Guanbin Li. 2024. Large multimodal agents: A survey. arXiv preprint arXiv:2402.15116. Chejian Xu, Mintong Kang, Jiawei Zhang, Zeyi Liao, Lingbo Mo, Mengqi Yuan, Huan Sun, and Bo Li. 2024. Advweb: Controllable black-box at- tacks on vlm-powered web agents. arXiv preprint arXiv:2410.17401. Rongwu Xu, Xiaojian Li, Shuo Chen, and Wei Xu. 2025. Nuclear deployed: Analyzing catastrophic risks in decision-making of autonomous llm agents. arXiv preprint arXiv:2502.11355. Xiao Yang, Jiawei Chen, Jun Luo, Zhengwei Fang, Yinpeng Dong, Hang Su, and Jun Zhu. 2025. Mla- trust: Benchmarking trustworthiness of multimodal llm agents in gui environments. arXiv preprint arXiv:2506.01616. Yulong Yang, Xinshan Yang, Shuaidong Li, Chenhao Lin, Zhengyu Zhao, Chao Shen, and Tianwei Zhang. 2024. Security matrix for multimodal agents on mo- bile devices: A systematic and proof of concept study. arXiv preprint arXiv:2407.09295. Biao Yi, Xavier Hu, Yurun Chen, Shengyu Zhang, Hongxia Yang, Fan Wu, and Fei Wu. 2025. Ecoagent: An efficient edge-cloud collaborative multi-agent framework for mobile automation. arXiv preprint arXiv:2505.05440. Chung-En (Johnny) Yu, Brian Jalaian, and Nathaniel D. Bastian. 2024. Mitigating Large Vision-Language Model Hallucination at Post-hoc via Multi-agent Sys- tem. Proceedings of the AAAI Symposium Series, 4(1):110â113. Yanzhe Zhang, Tao Yu, and Diyi Yang. 2025. Attacking vision-language computer agents via pop-ups. arXiv preprint. Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. 2024. Agent-safetybench: Evaluating the safety of llm agents. arXiv preprint arXiv:2412.14470. Zhizheng Zhang, Xiaoyi Zhang, Wenxuan Xie, and Yan Lu. 2023. Responsible task automation: Empow- ering large language models as responsible task au- tomators. arXiv preprint arXiv:2306.01242. Haoren Zhao, Tianyi Chen, and Zhen Wang. 2025. On the robustness of gui grounding models against image attacks. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1618â1623. Kangjia Zhao, Jiahui Song, Leigang Sha, HaoZhan Shen, Zhi Chen, Tiancheng Zhao, Xiubo Liang, and Jianwei Yin. 2024. Gui testing arena: A unified benchmark for advancing autonomous gui testing agent. arXiv preprint arXiv:2412.18426. 12 Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v(ision) is a generalist web agent, if grounded. International Conference on Machine Learning. Boyuan Zheng, Zeyi Liao, Scott Salisbury, Zeyuan Liu, Michael Lin, Qinyuan Zheng, Zifan Wang, Xiang Deng, Dawn Song, Huan Sun, et al. 2025. Webguard: Building a generalizable guardrail for web agents. arXiv preprint arXiv:2507.14293. Weihong Zhong, Xiaocheng Feng, Liang Zhao, Qiming Li, Lei Huang, Yuxuan Gu, Weitao Ma, Yuan Xu, and Bing Qin. 2024. Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models. Annual Meeting of the Association for Computational Linguistics. KAI-QING Zhou, Chengzhi Liu, Xuandong Zhao, An- derson Compalas, Dawn Song, and Xin Eric Wang. 2024. Multimodal situational safety. arXiv preprint arXiv:2410.06172. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. 2023. We- barena: A realistic web environment for building au- tonomous agents. arXiv preprint arXiv:2307.13854. Zichen Zhu, Hao Tang, Yansi Li, Dingye Liu, Hongshen Xu, Kunyao Lan, Danyang Zhang, Yixuan Jiang, Hao Zhou, Chenrun Wang, Situo Zhang, Liangtai Sun, Yixiao Wang, Yuheng Sun, Lu Chen, and Kai Yu. 2025. Moba: Multifaceted memory-enhanced adap- tive planning for efficient mobile task automation. Preprint, arXiv:2410.13757. A Trustworthiness Evaluation Benchmark 13 BenchmarkTrust DimensionKey MetricsInnovationLimitation Perception Trust Evaluation ARE(Wuetal., 2025a) Adversarial robustness Attack success rate, task degrada- tion Cross-module attack flow analysis Specificattack types M- SafetyBench(Liu et al., 2023b) Visual manipula- tion Safety score across 13 scenarios Image-basedattack scenarios Syntheticattacks only Robust GUI (Zhao et al., 2025) Groundingro- bustness Accuracy under perturbationNatural/adversarial noise testing Grounding-specific Reasoning Trust Evaluation Agent- SafetyBench (Zhang et al., 2024) Multi-category safety Safety scores across 8 risk cate- gories Comprehensive risk taxonomy English-only AgentHarm(An- driushchenko et al., 2024) Harmful task han- dling Refusal rate, completion rateDualre- fusal/completion metric Narrow task scope CASA (Qiu et al., 2024) Culturalaware- ness Awareness coverage, violation rate Cross-cultural norm testing Limitedcultural coverage Agent- ScanKit(Cheng et al., 2025) Memory & rea- soning Sensitivity to perturbationsDiagnostic probingDiagnosticfocus only Interaction Trust Evaluation ST- WebAgentBench (Levy et al., 2024) Policycompli- ance CUP, Risk Ratio Safety-utilityjoint measurement Web-only MobileSafetyBench (Lee et al., 2024) Mobile safetyInjection resistance, risk manage- ment Mobile-specific sce- narios Android-only EIA (Liao et al., 2024)Privacy preserva- tion PII extraction rateEnvironmental attack testing Specific attack vec- tor GhostEI-Bench (Chen et al., 2025b) Environmental in- jection Success rate in dynamic environ- ments Executable Android emulator Mobile-focused Comprehensive Evaluation MSSBench(Zhou et al., 2024) Situational safetyContext-sensitive safety reason- ing 1,820 language-image pairs Multimodal only MLA-Trust(Yang et al., 2025) Four- dimensional Truthfulness,controllability, safety, privacy First comprehensive framework Resource intensive WASP (Evtimov et al., 2025) Prompt injectionEnd-to-end attack successRealistic attack scenar- ios Web-focused WABER (Kara et al., 2025) Reliability & effi- ciency Consistency, speed, cost Network proxy evalua- tion Benchmark- dependent ChEF (Shi et al., 2023b) Holisticassess- ment Calibration, robustness, uncer- tainty Modularevaluation recipes Not agent-specific GUITesting Arena (Zhao et al., 2024) End-to-end test- ing Task completion on real apps Real application evalu- ation Limited trust met- rics Table 1: Comprehensive comparison of trustworthiness evaluation benchmarks. CUP = Completion Under Policy. Each benchmark addresses specific trust dimensions with characteristic trade-offs between coverage and depth, suggesting that comprehensive evaluation requires benchmark combinations. 14