Paper deep dive
Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents
Seoyoung Choi, Minseok Ko, Hyunseok Lee, Kunwoong Kim, Woomin Song, Chanseok Jeon, Jinwoo Shin
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/9/2026, 5:38:34 AM
Summary
This paper investigates the impact of visual memory on GUI agents, identifying four failure modes: cognitive failure, visual state misunderstanding, hidden operation blindness, and grounding error. It finds that full-image memory reduces state-level failures but exacerbates action-level ones. To address this, the authors propose AGMem, an action-grounded visual memory framework that stores focused image crops instead of full screenshots, improving task success rates by 33.3% over full-image memory on OSWorld.
Entities (10)
Relation Signals (9)
AGMem â improves â Task Success Rate
confidence 96% ¡ AGMem improves task success rates by 33.3 % over full-image memory.
OSWorld â evaluates â AGMem
confidence 95% ¡ Experiments on OSWorld show that AGMem improves task success rates by 33.3 % over full-image memory.
GUI Agents â uses â Visual Memory
confidence 95% ¡ More recent approaches further extend this idea to visual memory by storing and retrieving screenshots from past interactions
AGMem â mitigates â Visual State Misunderstanding
confidence 94% ¡ AGMem consistently alleviates the full-image memoryâs failures in some taxonomies, (e.g., Visual State Misunderstanding reduces by 37.3%p)
AGMem â mitigates â Hidden Operation Blindness
confidence 94% ¡ Hidden Operation Blindness reduces by 26.3%
Visual Memory â increases â Hidden Operation Blindness
confidence 92% ¡ hidden operation blindness rises from 67.1% to 78.8% (+11.7%p)
Visual Memory â increases â Grounding Error
confidence 92% ¡ grounding error rises from 27.5% to 36.1% (+8.6%p)
Visual Memory â reduces â Visual State Misunderstanding
confidence 90% ¡ cognitive failure drops from 82.6% to 75.0% and visual state misunderstanding drops from 73.1% to 69.6%
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Graphical User Interface (GUI) agents are increasingly used to automate complex computer tasks across applications, websites, and operating systems. To improve their reliability, recent work has introduced experiential memory, where agents retrieve prior trajectories to guide decision-making in similar states. More recent approaches further extend this idea to visual memory by storing and retrieving screenshots from past interactions, providing agents with richer contextual information than text-only memories. However, the effect of visual memory in GUI agents remains insufficiently understood: it is unclear which failures visual memory mitigates, or which failures it exacerbates. To systematically analyze the effect of visual memory, we introduce a taxonomy of four GUI agent failures (i.e., cognitive failure, visual state misunderstanding, hidden operation blindness, and grounding error) that map to distinct stages of the perception-reasoning-action pipeline. We find that prepending full-image memory has a divergent effect on the failure distribution: it reduces state-level failures but worsens action-level ones, and increases hidden operation blindness and grounding error. Motivated by this finding, we propose Action-Grounded Visual Memory (AGMem), an action-grounded memory framework for GUI agents. The core idea of AGMem is to store image crops that capture the local GUI region closely related to a successful action or a recovery, rather than storing full screenshots. Experiments on OSWorld show that AGMem improves task success rates by 33.3 % over full-image memory. These results demonstrate that AGMem is an effective representation for visual memory in GUI agents.
Tags
Links
- Source: https://arxiv.org/abs/2606.14106v1
- Canonical: https://arxiv.org/abs/2606.14106v1
Trouble viewing inline? Open PDF directly â
Full Text
70,960 characters extracted from source content.
Expand or collapse full text
Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents Seoyoung Choi 1 Minseok Ko 2 Hyunseok Lee 3 Kunwoong Kim 3 Woomin Song 3 Chanseok Jeon 4 Jinwoo Shin 3 Abstract Graphical User Interface (GUI) agents are increas- ingly used to automate complex computer tasks across applications, websites, and operating sys- tems. To improve their reliability, recent work has introduced experiential memory, where agents re- trieve prior trajectories to guide decision-making in similar states. More recent approaches further extend this idea to visual memory by storing and retrieving screenshots from past interactions, pro- viding agents with richer contextual information than text-only memories. However, the effect of visual memory in GUI agents remains insuf- ficiently understood: it is unclear which failures visual memory mitigates, or which failures it ex- acerbates. To systematically analyze the effect of visual memory, we introduce a taxonomy of four GUI agent failures (i.e., cognitive failure, visual state misunderstanding, hidden operation blind- ness, and grounding error) that map to distinct stages of the perception-reasoning-action pipeline. We find that prepending full-image memory has a divergent effect on the failure distribution: it reduces state-level failures but worsens action- level ones, and increases hidden operation blind- ness and grounding error. Motivated by this find- ing, we propose Action-Grounded Visual Memory (AGMem), an action-grounded memory frame- work for GUI agents. The core idea of AGMem is to store image crops that capture the local GUI region closely related to a successful action or a re- covery, rather than storing full screenshots. Exper- iments on OSWorld show that AGMem improves task success rates by 33.3 % over full-image mem- ory. These results demonstrate that AGMem is an effective representation for visual memory in GUI agents. 1 Seoul National University 2 POSTECH 3 KAIST 4 Pion Corpo- ration. Correspondence to: Seoyoung Choi <chlsy07@snu.ac.kr>, Jinwoo Shin <jinwoos@kaist.ac.kr>. Proceedings of the43 rd International Conference on Machine Learning Workshop Failure Modes in Agentic AI, Seoul, South Korea. Copyright 2026 by the author(s). 1. Introduction Graphical User Interface (GUI) control is often determined by local visual states rather than by the entire screen. These local states recur across tasks and applications, providing reusable cues for deciding what action should be taken next. Recent advances in vision-language models (VLMs) have enabled GUI agents that operate directly from screenshots and natural-language instructions (Zhang et al., 2024), and prior work have scaled them by training on massive decision- making trajectories (Xu et al., 2024; Wang et al., 2026; Wu et al., 2024) Despite this progress, modern GUI agents, including frontier models such as GPT (OpenAI, 2026), Claude (Anthropic, 2026), and Gemini (Google DeepMind, 2026), still fail in non-trivial ways, and the structure of these failures should receive attention. Recent works have proposed experiential working memory to tackle this problem. These methods log past decision trajectories and, at test time, retrieve the trajectories most relevant to the current state to guide action selection (Agashe et al., 2024; Kagaya et al., 2024). More recent approaches further introduce image-based memory systems, such as Wu et al. (2025a), which provide agents with richer, domain- specific contextual information beyond text. These systems typically consume screenshots as monolithic images, with- out an explicit representation of the local regions that de- termine actions. Despite this progress, the role of visual memory in GUI agents remains insufficiently understood. Its effects have not been systematically characterized in terms of how it improves performance, which failure modes it mitigates, and where it may introduce regressions. This leads to a question: What are the core failure modes in GUI agents, and how does visual memory mitigate them? In this work, we categorize and analyze the common failure patterns of existing GUI agents. Specifically, we investigate the failure patterns in representative benchmarks, including AgentNet (Wang et al., 2026), OSWorld (Xie et al., 2024a), and WebForge (Yuan et al., 2026). On these benchmarks, we identify four recurring failure modes that together account for the majority of agent errors: (i) Visual State Misunder- standing, (i) Hidden Operation Blindness, (i) Cognitive Failure, and (iv) Grounding Error. 1 arXiv:2606.14106v1 [cs.MA] 12 Jun 2026 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents Task: Set the given image as the background of the second slide. Wrong Plan: Click slide 3 thumbnail Slide 2 is currently selected, so I'l click slide 3's thumbnail to access the next page's background settings. x (a) Cognitive Failure Task: Remove the dock on the left side of the screen. â Wrong Plan: Click slide 3 thumbnail x I'l click 'Close Tab' in the dock menu to remove the dock. x Target: Click âClose Tabâ Wrong Action: Off-Target Click (b) Grounding Error Task: Fill total earned = total hours Ă hourly rate. â Wrong Plan: Click slide 3 thumbnail x Total hour is in column C and Total Earned in column D. Iâl fill D3 with C3*24*F3 x Target: Fill Total Earned Cell Wrong Perception: Writes Total Hours Cell (c) Visual State Misunderstanding Task: Create a new folder on the bookmarks bar. â Wrong Plan: Click slide 3 thumbnail x I don't see an 'Add folder' option on the screen. I'l use Ctrl+Shift+B to create the folder directly. Ctrl + Shift + B x Wrong Action: Only toggles bookmarks bar (d) Hidden Operation Blindness Figure 1. Four failure modes of GUI agents. We observe four recurring patterns in agent failures. In (a), the agent should be working on the second page, but instead clicks the thumbnail of the third page. In (b), the agentâs target is correct, but the click is slightly off and misses the menu item. In (c), the agent misreads a cell, entering the formula into the Total Hours column instead of Total Earned. In (d), creating a folder requires an operation that is not visible on the screen, but the agent only pressesCtrl+Shift+B, a shortcut that toggles the bookmarks bar instead of creating the folder. Red and green markers denote the agentâs action and the correct one, respectively. Based on these patterns, we then analyze how introducing visual memory shifts the failure distribution. Following Wu et al. (2025a), we prepend full screenshots of relevant history entries to the agent context. For example, on OSWorld with GPT-5.4-mini, full-image memory reduces state-level fail- ures (e.g., Visual State Misunderstanding 73.1%â69.6%, but consistently worsens action-level failures, increasing Hidden Operation Blindness by 11.7%p). That is, full visual memory injects useful global context for state-level con- fusion but simultaneously crowds the visual context with task-irrelevant cues that distract from invisible operations and degrade coordinate grounding. This suggests that visual memory is valuable in principle, but which pixels are stored matters as much as whether memory is used at all. To further mitigate this problem, we introduce Action- Grounded Visual Memory (AGMem), an action-grounded information compacting framework for managing GUI agen- tic memory. The core idea of AGMem is to enable agents to access focused GUI observations through a working mem- ory and a domain-compacting retrieval system. Specifically, we introduce two focusing strategies: (i) a visual memory consisting of cropped images that catch the elements that should be focused on, and (i) memory narrowing related to the given state. To construct the memory system, we propose an automatic memory processing pipeline that can simply extract working memory trajectories from existing GUI action trajectories. At test-time, the agent retrieves relevant crop-view memories and uses them during plan- ning, allowing it to access decision-critical visual evidence while avoiding the cost of repeatedly processing full-screen images, as depicted in Figure 5. In experiments, we demonstrate AGMem consistently alle- viates the full-image memoryâs failures in some taxonomies, (e.g., Visual State Misunderstanding reduces by 37.3%p and Hidden Operation Blindness reduces by 26.3%), while improving end-to-end task accuracy by 6.8%p, on OSWorld with GPT-5.4-mini. 2 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents Table 1. The four representative failure modes that can be observed in GUI agents. Failure modeDescription Cognitive failureThe model misunderstands the task goal, current subtask, required next step, success condition, or recovery strategy (e.g., following the wrong plan step, assuming a failed action succeeded, or terminating too early). Visual state misunder- standing The model misreads the visible screen state (e.g., believing a dialog or dropdown is open when it is not, misidentifying the selected object, or incorrectly inferring that a sort order). Hidden operation blind- ness The correct action requires an operation not directly visible until another UI action reveals it (e.g., opening a menu, context menu, or overflow menu; using a keyboard shortcut; performing a non-obvious selection). Grounding errorThe model intends the correct visible target and predicts a compatible action type, but the executed coordinate or low-level action misses the target. We highlight the main contributions of this work as follows: â˘We introduce a four-class taxonomy of GUI agent failure modes and an LLM-as-Judge protocol for labeling them across online and offline benchmarks. ⢠We provide the first empirical analysis of how visual memory shifts the failure distribution, showing that full- image memory cuts state-level failures but amplifies hidden-operation and grounding failures. â˘We propose AGMem, an action-grounded visual work- ing memory framework for GUI agents with an auto- matic memory construction pipeline, which effectively mitigates the failure modes in GUI agents 2. Failure modes of GUI Agents In this section, we provide a comprehensive analysis of the trajectories of GUI agents and classify their failure modes. First, we define a taxonomy of failure modes observed in the GUI agents (Section 2.1). Then, we explain our setups (Sec- tion 2.2). Finally, we demonstrate main results of observed errors classified based on our taxonomy (Section 2.3). 2.1. Taxonomy: four failure modes We identify four recurring failure modes across benchmarks, as illustrated in Figure 1. Each mode corresponds to a dif- ferent stage of the perception-reasoning-action pipeline that a VLM-based GUI agent executes at every step: planning (cognitive failure), perception (visual state misunderstand- ing), action-space inference (hidden operation blindness), and execution (grounding error). Because these stages re- quire different kinds of evidence to fix (i.e., evidence about state for perception, about affordances for action-space in- ference, and about pixel-level alignment for execution), a memory mechanism that helps one stage need not help the other stages, which we exploit in the remainder of this sec- tion as a measurement instrument. ⢠Cognitive failure. This failure occurs when the model misunderstands the task goal, the current subtask, the required next step, the success condition, or the recovery strategy. It includes choosing the wrong action despite sufficient visible information, following the wrong plan step, assuming a failed action succeeded, terminating too early, or solving a semantically different task. â˘Visual state misunderstanding. This is when the model misreads the current visible screen. For example, the model may believe it is on the wrong page, think a dialog or dropdown is open when it is not, misidentify the selected object, or incorrectly infer that a form, filter, sort order, or progress state has already changed. â˘Hidden operation blindness. The correct action re- quires an operation that is not directly visible until an- other UI action reveals it. Examples include opening a menu, toolbar, context menu, or overflow menu; using a shortcut; or performing a non-obvious selection or drag. â˘Grounding error. This failure is observed when the model intends the correct visible target and predicts a compatible action type, but the executed coordinate or low-level action misses the target. We assign this type only when the raw reasoning indicates the same intended target as the ground truth; if the intended target is different, the failure is classified as cognitive failure or visual state misunderstanding instead. 2.2. Setup Evaluation setup. We evaluate GUI agents on three bench- marks: two online benchmarks, OSWorld (Xie et al., 2024b) and WebForge (Yuan et al., 2026), and one offline bench- mark AgentNetBench (Wang et al., 2026). Specifically, OSWorld contains 316 tasks across 10 application domains that use a1920Ă 1080viewport. WebForge is evaluated on a randomly sampled subset of 50 tasks. Each episode is lim- ited to 50 steps unless explicitly stated. Also, we evaluate AgentNetBench on pre-captured screenshot sequences and a given task instruction. We report the average task score, computed from the benchmarkâs composite action-matching metric aggregated over all tasks. Labeling setup. Based on our taxonomy in Table 1, we label failures with a Codex-based LLM-as-Judge that as- signs a label only when the evidence is directly visible in the trace. Note that the taxonomy is multi-label, so per- mode rates need not sum to100%. We label at the task level on OSWorld and WebForge, and at the action level on 3 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents Cog.VSMHOBGE 0 20 40 60 80 100 Failure rate (\%) vanilla AgentNetBench OSWorld WebForge Figure 2. Failure mode distributions. Failure mode distribution of GPT-5.4-mini across OSWorld, AgentNetBench, and WebForge. Each benchmark exhibits a distinct dominant mode: hidden oper- ation blindness on OSWorld, grounding error on WebForge, and visual state misunderstanding on AgentNetBench. AgentNetBench. The full details are given in Appendix C. GUI agent. A GUI agent is a large language model (LLM)-based autonomous agent that perceives and inter- acts with on-screen interface elements, such as screenshots, through human-like actions such as clicking and typing (Nguyen et al., 2025). We follow the standard screenshot- based GUI agent setup (Xie et al., 2024a; Cheng et al., 2024). Specifically, at each stept, an LLM-based policy receives the instructionI, a short historyh t of recent screenshot- action pairs, the current screenshoto t , and a set of retrieved memory stepsR t from a memory bankM, and emits an actiona t :a t âź Ď Âˇ | I, h t , o t , R t .A full formalization of the policy and trajectory is deferred to Appendix A. 2.3. Main results GUI agents frequently fall into four failure modes. Fig- ure 2 shows the vanilla failure distribution across the three benchmarks with GPT-5.4-mini. Each environment has a distinct dominant mode: hidden operation blindness on OS- World (67.1%), grounding error on WebForge (70.0%), and visual state misunderstanding on AgentNetBench (45.1%). Three modes-(i) cognitive failure, (i) visual state misunder- standing, and (i) grounding error-appear across all bench- marks, while hidden operation blindness is desktop-specific. Per-benchmark trends reflect environment character- istics. The dominant failure modes differ across bench- marks because each environment stresses a different part of the perception-reasoning-action pipeline. OSWorldâs high cognitive failure (82.6%) and hidden operation blind- ness (67.1%) reflect the long-horizon, menu-heavy nature of desktop applications, where essential commands are rou- tinely buried in menus, context menus, ribbons, and key- board shortcuts. WebForgeâs grounding error dominance Cog.VSMHOBGE 0 20 40 60 80 100 Failure rate (\%) Vanilla Full-image Cropped AGMem (Ours) Figure 3. Per-mode failure rates on OSWorld with GPT-5.4- mini. Full-image memory reduces state-level failures (cognitive failure, visual state misunderstanding) but worsens action-level failures (hidden operation blindness, grounding error). AGMem is the only configuration that consistently reduces all failure modes. (70.0%) reflects the dense layout of web interfaces with many small click targets, making pixel-level grounding more brittle than in larger desktop UIs; correspondingly, hid- den operation blindness is rare (8.0%) because web controls are typically exposed directly on the page. AgentNetBenchâs lower absolute rates and the relative prominence of visual state misunderstanding (45.1%) follow from its per-action evaluation, which isolates each stepâs state-recognition re- quirement and sidesteps long-horizon planning. These vari- ous aspects of the benchmarks reflect the diverse modes of GUI agent usage and suggest where each evaluation is the most useful. Full-image memory has a divergent effect. As shown in Figure 3, adding full-image memory has a divergent effect on the failure distribution. On the one hand, it reduces state- level failures by injecting global context that helps the agent re-orient to the task and the visible state: cognitive failure drops from82.6%to75.0%(â7.6%p) and visual state mis- understanding drops from73.1%to69.6%(â3.5%p). On the other hand, the same full-screen retrieved screenshots substantially expand the visual context with task-irrelevant elements that distract the model from less salient opera- tions. This shows up as an increase in two action-level failures: hidden operation blindness rises from67.1%to 78.8%(+11.7%p) and grounding error rises from27.5%to 36.1%(+8.6%p). The net effect is that full-image memory trades state-level errors for action-level errors rather than consistent improvement. Directions for improvement. This divergent effect points to a clear improvement target. To preserve the gains in state-level failures while avoiding the costs on action-level failures, visual memory should be smaller (containing only action-relevant pixels rather than full screenshots) and more selective (narrowed to memory steps aligned with the cur- 4 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents Figure 4. Conceptual illustration of action-grounded visual memory. Without memory, an agent may select an incorrect UI target. Full-image visual memory can provide useful prior experience, but the action-relevant cue may remain small or spatially ambiguous within the full screenshot, causing incorrect grounding. Action-grounded visual memory focuses the retrieved example on the region associated with the demonstrated action, making the relevant target easier to identify and localize. rent subtask). We translate these two requirements into the design of AGMem in the next section (Section 3). 3. AGMem: mitigating the side effects of visual memory In this section, we present our proposed framework Action- Grounded Visual Memory (AGMem), which addresses the two directions for improvement identified in Section 2.3. First, we formalize the problem setup and outline AGMem in Section 3.1. Then, we introduce action-grounded mem- ory entries that compact each memory entry to an action- relevant view Section 3.2. Finally, we describe action- grounded memory candidates that narrow retrieval and pro- vide a separate recovery memory for erroneous states Sec- tion 3.3. More details on AGMem are provided in Ap- pendix B, including an overview in Figure 5. 3.1. Overview To guide the agent through a given long-horizon instruction I, AGMem first decomposesIinto a list of subtasks (Gao et al., 2024; Ye et al., 2025): s = (s 1 ,s 2 ,...,s p ). Here, eachs i is a natural-language subtask derived from the instructionIalone before any action is executed. Note that the number of subtaskspis generally different from the number of stepsT, since one subtask may span multiple steps. The pair(I,s)is given to the agent throughout the tra- jectory. Before emitting each action, the agent self-reports its current subtask, so that action execution is aligned with the current subtask. We then keep the memory salient in two core components. First, an action-relevant view (Section 3.2). Second, a nar- rowed retrieval pipeline with a separate recovery memory for erroneous states (Section 3.3). 3.2. Action-grounded visual memory A desktop screenshot is often dominated by content unre- lated to the current action or state (e.g., background win- dows, system trays, and idle panels), that is not affected by the agentâs actions (Lin et al., 2025; Li et al., 2025). Stor- ing all such screenshots would unnecessarily expand the memory bank and burden the agentâs visual context with task-irrelevant cues. We therefore reduce each memory en- try to its salient form, called the action-relevant view. For each stept, we denote this view byC t , which is a tightly cropped image in which action a t has important effects. Construction from trajectories.Starting from screenshot o t at timet(Figure 7a), the agent executes actiona t , and the environment transitions to the next screenshoto t+1 . Given the consecutive screenshots(o t ,o t+1 ), we construct the action-relevant viewC t , which is a tightly cropped snapshot that isolates the GUI region in which actiona t takes effect. (Figure 7b). See Appendix B for the full algorithm. 5 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents Figure 5. Overview of AGMem. AGMem is a compact GUI trajectory memory framework based on action-relevant visual views, salient memory candidates from trajectory- and step-level retrieval, and recovery memory for erroneous states. Memory structure. We construct the memory bankM from AgentNet dataset (Wang et al., 2026),D AGNet , where we start from retaining only non-redundant and correct tra- jectories Ď . For each trajectoryĎ âD AGNet , an LLM produces a list of post-hoc subtask labelss Ď that contains information about what actually happened on the screen duringĎ, and we segmentĎalongs Ď into sub-trajectories. Consequently, the atomic unit of memory is a memory step: m t = (s j ,a t ,C t ). wheres j âs Ď is the subtask label that the step realizes,a t is the executed action, andC t is the action-relevant view. Sub-trajectory segmentation and indexing details are given in Appendix B. 3.3. Recovery-aware memory retrieval Searching the entire memory bank at every step or action is both costly and noisy, since most trajectories are irrelevant to the current task and individual steps may produce spurious visual memories. We therefore narrow the search space in two stages. We first select a small pool of trajectories at the task level, and then retrieve individual memory steps within that pool. In addition, when the agent flags an error, we redirect retrieval to a separate recovery memory. Two-stage retrieval Given the current subtask lists, a trajectory-level retriever first selects a size-kpoolM sub â Mof memory trajectories whose post-hoc subtask labels s Ď are close tosin a Sentence-Transformer (Reimers and Gurevych, 2019) embedding space, greedily covering each subtask ins. A step-level retriever then encodes each mem- ory stepâs subtask label and action-relevant view with the CLIP encoders (Radford et al., 2021), computes a fixed- weight similarity against the current state â the agentâs self- reported subtasks t and the previous stepâs viewC tâ1 â and returns the top-5 memory steps asR t . Encoder choices and more details are given in Appendix B. Recovery-aware memory A common reason behind the failures of GUI agents is error propagation, where the agent enters an erroneous state and the subsequent action pro- posals are made under a faulty premise. To address this issue, we add recovery-aware verification memory which teaches the agent how to recover. Specifically, LLM-based recovery-state detector detects the erroneous step during operation. When an error is detected, the recovery mem- ory provides examples of correct behaviors that bring the agent back from a faulty state to a valid one. The recovery memory is built from AgentNet sub-trajectories that were excluded from the main bank (redundant or incorrect) but were later corrected within the same trajectory. Specifically, each entry pairs a faulty sub-trajectory with the correct step, labeled by failure mode so retrieval can match the flagged pattern. Construction details are given in Appendix B. 4. AGMem experiments We provide empirical evidence of AGMem by analyzing how it improves the failure mode distribution that full-image memory induces (Section 4.1), and end-to-end task perfor- mance against existing baselines (Section 4.2). 4.1. Failure mode analysis We use the failure mode taxonomy of Section 2.1 to measure how AGMem is behaving on the failure distribution, on OSWorld with GPT-5.4-mini. AGMem reduces every failure modes. As shown in Ta- ble 2, AGMem consistently reduces all four failure modes against the vanilla agent. For example, visual state misun- derstanding reduces by about -40.8%p. This suggests that compact action-relevant crops are particularly effective at processing the small subset of pixels that determine state- level decisions. Notably, while hidden operation blindness and grounding error are the two failure modes that are wors- ened by full-image memory, AGMem also reduces them. This indicates the gains from memory persist even when the visual context is restricted to action-relevant regions. 6 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents Table 2. Failure Mode Analysis on OSWorld Benchmark. Comparison of overall Success Rate (Acc.) and the distribution of specific failure modes across different visual memory configurations. Method OverallFailure Mode (Root Cause %) Acc. (%)CognitiveVisual Mis.Hidden Op.Grounding Vanilla Agent18.382.673.167.127.5 + Visual Memory20.475.069.678.836.1 + Crop25.872.468.360.332.1 AGMem (Ours)27.267.132.352.522.5 * Cognitive: Cognitive Failure / Visual Mis.: Visual State Misunderstanding / Hidden Op.: Hidden Operation Blindness / Grounding: Grounding Error Table 3. AGMem improves performance by effectively leverag- ing visual memory. Accuracy (%) and Milestone Accuracy (M. Acc.) of AGMem (Ours) compared with baselines across three benchmarks, for GPT-5.4-mini. For OSWorld and WebForge, we report overall success rate; for other, step-level accuracy. Method OSWorldWebForgeAgentNet Acc.Acc.Acc.M. Acc. Vanilla agent18.32.025.824.2 + visual memory20.42.028.133.8 AGMem (Ours)27.22.028.834.6 Cropping alone is insufficient. The cropped configura- tion crops memory screenshots but performs retrieval over the full memory bank. Compared to full-image memory, cropping alone partially recovers performance on hidden operation blindness (78.8% â 60.3%) and grounding er- ror (36.1% â 32.1%), but still falls short of AGMem in every mode and especially on visual state misunderstand- ing (68.3%vs.32.3%). The remaining gap is closed by subtask-aligned retrieval narrowing, which restricts retrieval to memory steps relevant to the current subtask and prevents off-task crops from re-injecting irrelevant cues. 4.2. Main results We first present the performance of AGMem by compar- ing GUI agent performance with other baselines. Here we mainly compare with one offline benchmarks (i.e., Agent- NetBench) and two online benchmarks (i.e., OSWorld and WebForge). Furthermore, we demonstrate the token effi- ciency of AGMem by comparing average token consump- tion per sample with the visual memory baseline against other baselines. Significant performance improvement. As shown in Ta- ble 3, AGMem consistently outperforms the existing base- lines across both offline and online benchmarks, with large margins. For instance, on OSWorld, AGMem improves task accuracy by +8.9%p (from 18.3% to 27.2%) with GPT- 5.4-mini. On AgentNet Bench, AGMem further improves step accuracy from 25.8% to 28.8% with GPT-5.4-mini. These results show that AGMem is not specialized to a sin- gle benchmark or interaction setting, but generalizes across both web-based GUI tasks and online desktop-control en- vironments. This suggests that compact and task-relevant visual memory can provide useful state information for GUI agents without relying on full visual screenshots. Effectively leveraging visual memory. Notably, simply adding visual memory does not always improve GUI agent performance. For example, on OSWorld, the full-image memory baseline increases the hidden operation blindness rate from 67.1% to 78.8%, compared to the vanilla agent, indicating that storing full screenshots can introduce redun- dant or distracting context. In contrast, AGMem reduces it from 67.1% to 60.3%, outperforming both the vanilla agent and the full-image memory baseline. This demonstrates that AGMem does not merely add more visual history or information, but more effectively leverages visual memory by compacting and preserving task-relevant visual evidence. 5. Case studies Figure 4 shows a representative hidden operation blindness case in LibreOffice Impress. The task is to apply the Dis- solve transition to the first slide. Although the first slide is already selected, the transition settings are not directly visible in the main canvas and must be opened through a right-sidebar tab. The vanilla agent understands the high- level goal but guesses an incorrect route, clicking the top Slide menu and a presumed transition submenu. Full-image memory provides a related prior state, but the useful cue is still embedded in a cluttered whole-screen observation, where the relevant sidebar icon is small and competes with many unrelated UI elements. As a result, full-image memory can suggest the correct route but does not reliably ground the action to the precise affordance. In contrast, the paired crop in action-grounded memory suppresses irrelevant vi- sual context by isolating the sidebar tab or icon region. This changes the agentâs next action from a menu-based guess to clicking the transition icon in the right sidebar in the current screenshot. Thus, the memory does not provide the final transition value directly, but supplies the hidden UI route. Through action-grounding, it makes the relevant visual af- fordance salient enough to produce the correct action. 7 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents Although action-grounded memory helps the agent interpret retrieved examples, it does not by itself guarantee recovery once the agent has entered an erroneous state. In an OS- World VLC task, the agent must stream an HLS video by opening VLCâs network-stream dialog and entering a .m3u8 URL. The vanilla agent follows the correct high-level route and reaches the target dialog but fails at the final low-level interaction by misclicking the Play button and incorrectly assuming task completion. Action-grounded memory pro- vides more interpretable UI regions and example actions, but it does not correct the faulty premise that the stream was closed without playback. AGMem resolves this error propa- gation through recovery-aware verification memory. When the detector flags the trajectory as erroneous, the recovery memory supplies examples of corrective behavior from sim- ilar faulty states, allowing the verifier to replace the unstable menu-click trajectory with the task-relevant recovery ac- tion Ctrl+N, which directly reopens VLCâs network-stream dialog, and succeeds the task. This case illustrates that action-grounded memory is useful for selecting and ground- ing normal actions, but recovery-aware memory is necessary when an earlier miss creates a misleading state premise that ordinary retrieval cannot correct. 6. Related work This section provides a comprehensive review of works related to AGMem, including agentic memory, GUI agents, and compaction methods for multimodal agents. Agentic memory.Recent works have studied how agents can reuse past experience during inference. Synapse stores past computer-control trajectories as textual examples and retrieves them for similar tasks (Zheng et al., 2024). Agent Workflow Memory extracts reusable workflows from suc- cessful trajectories (Wang et al., 2025), and MemP builds procedural memory from step-level and script-level trajec- tory summaries (Fang et al., 2026). A-Mem organizes mem- ories as linked notes that can be updated over time (Xu et al., 2026a). Other works use failures as memory. For example, ExpeL (Zhao et al., 2024) extracts text insights from success and failure pairs, and ReasoningBank (Ouyang et al., 2026a) stores reasoning strategies from both successful and failed runs. These methods show that memory can improve agent decision-making. However, most of them store memory as text. For GUI tasks, text memory can miss important visual cues such as icons, layouts, selected regions, input fields, and error messages. AGMem instead stores action-relevant cropped images as memory. It also uses a separate recovery memory, so that the agent can retrieve corrective examples when it enters an erroneous state. GUI agent.GUI agents aim to control graphical interfaces from screenshots and natural language instructions. CogA- gent (Hong et al., 2024) builds a VLM-based GUI agent with high- and low-resolution visual encoders. OS-Atlas (Wu et al., 2025b) proposes a foundation action model with a unified action space, and Aguvis (Xu et al., 2025) trains a pure-vision agent for autonomous GUI interaction. Open- CUA (Wang et al., 2026) further provides open data, tools, benchmarks, and models for computer-use agents. These works mainly improve GUI agents by training stronger mod- els or collecting larger datasets. Several GUI agents also use memory during planning. Agashe et al. (2025) proposed to use narrative memories for hierarchical planning. CoMEM (WU et al., 2026) com- presses trajectories into continuous memory embeddings, and WebRAgent (Zhang et al., 2026) retrieves multimodal web trajectories for planning. These methods are close to AGMem because they also reuse previous GUI trajectories. However, they usually use text summaries, full screenshots, trajectory-level memories, or learned embeddings. Image processing. Visual focusing methods reduce ir- relevant image regions for multimodal models. Set-of- Mark (Yang et al., 2023) adds visual marks to images to help grounding, and Wu and Xie (2023) proposed to search images by zooming into useful regions. CropVLM (Car- valho et al., 2025) learns a cropping policy for frozen VLMs. For GUI agents, Song et al. (2026) proposed to reduce coordinate information during policy optimization, GUIPruner (Xu et al., 2026b) prunes visual tokens for ef- ficient high-resolution GUI agents, and FocusUI (Ouyang et al., 2026b) selects instruction-relevant visual tokens for GUI grounding. While these works mainly focus on process- ing the current observation, AGMem focuses on memories. It extracts crop-view memories from past trajectory transi- tions and retrieves only the visual evidence that is related to the current subtask and state. 7. Conclusion and discussion We took an analysis-first view of memory in GUI agents and showed that screenshot-based agents can fail in benchmark- specific ways. For example, with GPT-5.4-mini, OSWorld is dominated by cognitive failure (82.6%) and hidden op- eration blindness (67.1%), WebForge by grounding error (70.0%), and AgentNetBench by visual state misunderstand- ing (45.1%). Adding naĂŻve full-image visual memory does not move these dominant modes uniformly, while it helps where the bottleneck is recognition but amplifies template- style memory over-following where the bottleneck is action- space reasoning. AGMem targets exactly these side effects: action-relevant crops keep memory references focused on the relevant UI region, and the recovery-state detector inter- venes in residual cognitive failures, jointly yielding a9.1%p OSWorld accuracy gain over the vanilla agent. We see this 8 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents analysis-design-reanalysis loop as a useful lens for future work on memory-augmented GUI agents. Impact statement The main goal of this work is to advance the field of Machine Learning, in particular the analysis and design of memory for GUI agents. As GUI agents observe whatever appears on a userâs screen, deployments should filter sensitive UI con- tent from any persistent memory and retain human oversight to prevent automated misuse. Beyond these standard consid- erations, we do not feel any further societal consequences of our work should be particularly highlighted. References Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, et al. Large language model-brained gui agents: A survey. arXiv preprint arXiv:2411.18279, 2024. Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong.Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454, 2024. Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tian- bao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, Zhennan Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Zheng Boyuan, LI PEI- HANG, Fangyu Lei, Ruisheng Cao, Yeqiao Fu, Dongchan Shin, Martin Shin, Hu Jiarui, Yuyan Wang, Jixuan Chen, Yuxiao Ye, Danyang Zhang, Yipu Wang, Heng Wang, Diyi Yang, Victor Zhong, Y.Charles, Zhilin Yang, and Tao Yu. OpenCUA: Open foundations for computer-use agents. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URLhttps: //openreview.net/forum?id=6iRZvJiC9Q. Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218, 2024. OpenAI.Gpt-5.5systemcard,April2026. URLhttps://openai.com/index/ gpt-5-5-system-card/. Anthropic.Claudeopus4.7systemcard, April2026.URLhttps://cdn. sanity.io/files/4zrzovbb/website/ 037f06850df7fbe871e206dad004c3db5fd50340. pdf. Google DeepMind. Gemini 3.1 pro model card, Febru- ary 2026.URLhttps://deepmind.google/ models/model-cards/gemini-3-1-pro/. Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like a human. arXiv preprint arXiv:2410.08164, 2024. Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlekar, Sugiri Pranata, Akira Kinose, Koki Oguri, Felix Wick, and Yang You. Rap: Retrieval- augmented planning with contextual memory for mul- timodal llm agents, 2024. Wenyi Wu, Kun Zhou, Ruoxin Yuan, Vivian Yu, Stephen Wang, Zhiting Hu, and Biwei Huang. Auto-scaling continuous memory for gui agent.arXiv preprint arXiv:2510.09038, 2025a. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Vic- tor Zhong, and Tao Yu. Osworld: Benchmarking mul- timodal agents for open-ended tasks in real computer environments, 2024a. Peng Yuan, Yuyang Yin, Yuxuan Cai, and Zheng Wei. Web- forge: Breaking the realism-reproducibility-scalability trilemma in browser agent benchmark. arXiv preprint arXiv:2604.10988, 2026. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhou- jun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yi- heng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In The Thirty-eight Conference on Neu- ral Information Processing Systems Datasets and Bench- marks Track, 2024b. URLhttps://openreview. net/forum?id=tN61DTr4Ed. Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namy- ong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, Xintong Li, Jing Shi, Hongjie Chen, Viet Dac Lai, Zhouhang Xie, Sungchul Kim, Ruiyi Zhang, Tong Yu, Mehrab Tanjim, Nesreen K. Ahmed, Puneet Mathur, Seunghyun Yoon, Lina Yao, Branislav Kveton, Jihyung Kil, Thien Huu Nguyen, Trung Bui, Tianyi Zhou, Ryan A. Rossi, and Franck Dernoncourt. GUI agents: A survey. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Find- ings of the Association for Computational Linguistics: ACL 2025, pages 22522â22538, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 9 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl. 1158. URLhttps://aclanthology.org/2025. findings-acl.1158/. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9313â9332, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. URLhttps://aclanthology.org/2024. acl-long.505. Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, Hengxu Wang, Luowei Zhou, and Mike Zheng Shou. Assistgui: Task-oriented desktop graphical user interface automation, 2024. URLhttps: //arxiv.org/abs/2312.13108. Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, Jitong Liao, Qi Zheng, Fei Huang, Jingren Zhou, and Ming Yan. Mobile-agent-v3: Funda- mental agents for gui automation, 2025. URLhttps: //arxiv.org/abs/2508.15144. Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision-language- action model for gui visual agent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19498â19508, June 2025. Kaixin Li, Meng Ziyang, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua.Screenspot-pro: GUI grounding for profes- sional high-resolution computer use. In Workshop on Reasoning and Planning for Large Language Models, 2025. URLhttps://openreview.net/forum? id=XaKNDIAHas. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Compu- tational Linguistics, 11 2019. URLhttp://arxiv. org/abs/1908.10084. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable vi- sual models from natural language supervision. In Ma- rina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learn- ing, volume 139 of Proceedings of Machine Learn- ing Research, pages 8748â8763. PMLR, 18â24 Jul 2021. URLhttps://proceedings.mlr.press/ v139/radford21a.html. Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exemplar prompting with memory for computer control. In The Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum? id=Pc8AU1aF5e. Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Gra- ham Neubig.Agent workflow memory.In Forty- second International Conference on Machine Learning, 2025. URLhttps://openreview.net/forum? id=NTAhi2JEEE. Runnan Fang, Yuan Liang, Xiaobin Wang, Jialong Wu, Shuofei Qiao, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. Memp: Exploring agent procedural memory, 2026. URLhttps://arxiv.org/abs/ 2508.06433. Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-mem: Agentic memory for LLM agents. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026a. URLhttps: //openreview.net/forum?id=FiM0M8gcct. Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. In Michael J. Wooldridge, Jen- nifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Sympo- sium on Educational Advances in Artificial Intelligence, EAAI 2024, February 20-27, 2024, Vancouver, Canada, pages 19632â19642. AAAI Press, 2024. doi: 10.1609/ aaai.v38i17.29936. URLhttps://ojs.aaai.org/ index.php/AAAI/article/view/29936. Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long Le, Samira Daruki, Xi- angru Tang, Vishy Tirumalashetty, George Lee, Mah- san Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and Tomas Pfister. Reasoningbank: Scaling agent self- evolving with reasoning memory. In The Fourteenth International Conference on Learning Representations, 2026a. URLhttps://openreview.net/forum? id=jL7fwchScm. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual 10 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14281â14290, June 2024. Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. OS-ATLAS: Foun- dation action model for generalist GUI agents. In The Thirteenth International Conference on Learning Rep- resentations, 2025b. URLhttps://openreview. net/forum?id=n9PDaFNi8t. Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tian- bao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caim- ing Xiong. Aguvis: Unified pure vision agents for au- tonomous GUI interaction. In Forty-second International Conference on Machine Learning, 2025. URLhttps: //openreview.net/forum?id=PlihOwfx4r. Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like a human. In The Thir- teenth International Conference on Learning Represen- tations, 2025. URLhttps://openreview.net/ forum?id=lIVRgt4nLv. Wenyi WU, Kun Zhou, Ruoxin Yuan, Vivian Yu, Stephen Wang, Zhiting Hu, and Biwei Huang. Auto-scaling con- tinuous memory for GUI agent, 2026. URLhttps: //openreview.net/forum?id=cvAZKWA1HZ. Xuan Zhang, Shengbo Cai, Ziyan Jiang, Rui Meng, Zora Zhiruo Wang, Lu Guangcheng, Zhiyong Wu, Yanyi Shang, and Dehan Kong.WebRAGent: Retrieval- augmented generation for multimodal web agent plan- ning, 2026.URLhttps://openreview.net/ forum?id=L1VPZFbAcu. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023. Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms. arXiv preprint arXiv:2312.14135, 2023. Miguel Carvalho, Helder Dias, and Bruno Martins. Cropvlm: Learning to zoom for fine-grained vision- language perception. arXiv preprint arXiv:2511.19820, 2025. Yurun Song, Jiong Yin, Rongjunchen Zhang, and Ian G. Har- ris. Compress to focus: Efficient coordinate compression for policy optimization in multi-turn gui agents, 2026. URL https://arxiv.org/abs/2601.11631. Zhou Xu, Bowen Zhou, Qi Wang, Shuwen Feng, and Jingyu Xiao. Spatio-temporal token pruning for efficient high- resolution gui agents, 2026b. URLhttps://arxiv. org/abs/2602.23235. Mingyu Ouyang, Kevin Qinghong Lin, Mike Zheng Shou, and Hwee Tou Ng. Focusui: Efficient ui grounding via position-preserving visual token selection. arXiv preprint arXiv:2601.03928, 2026b. Sunil Bhutada, Nakerakanti Yashwanth, Puppala Dheeraj, and Kethavath Shekar. Opening and closing in morpho- logical image processing. World Journal of Advanced Research and Reviews, 14(3):687â695, 2022. 11 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents A. Preliminaries on memory-augmented GUI agents A graphical user interface (GUI) agent is a large language model (LLM)-based autonomous agent that perceives and interacts with on-screen interface elements, such as screenshots, through human-like actions (Nguyen et al., 2025). Given a long-horizon natural-language instructionI, the GUI agent operates overTsteps of observation-action pairs. At each stept â 0, 1,...,T â 1, the agent observes a full screenshoto t and executes an actiona t , after which the environment transitions to the next screenshot o t+1 . From the initial screenshot o 0 , we have the trajectory Ď := o 0 ,a 0 ,o 1 ,a 1 ,...,o Tâ1 ,a Tâ1 ,o T , where o T is the final screenshot at which the goal of I is achieved. The agent can be formalized by an LLM-based policyĎ. At each stept, the agent is given (i) the instructionI, (i) a historyh t = (o i ,a i ) tâ1 i=max(0,tâ3) of up to the three most recent screenshot-action pairs (Xie et al., 2024a), (i) the current screenshoto t , and (iv) a set of retrieved memory stepsR t from a memory bankM, which stores previously executed trajectories. The policy samples an action a t , and the environment then transitions to the next screenshot o t+1 : a t âź Ď Âˇ I,h t ,o t ,R t , o t+1 âź P ¡ o t ,a t , wherePdenotes the unknown environment transition induced by executinga t on the device. Because this policy operates over screenshots rather than HTML or accessibility trees, the framework is applicable to arbitrary desktop environments and applications (Xie et al., 2024a; Cheng et al., 2024). B. Algorithm details of AGMem In this section, we present our proposed framework Action-Grounded Visual Memory (AGMem) introduced in Section 3 with details. Recall that an overview of AGMem is depicted in Figure 5. To guide the agent through a given long-horizon instructionI, AGMem first decomposesIinto a list of subtasks (Gao et al., 2024; Ye et al., 2025): s = (s 1 ,s 2 ,...,s p ). Here, eachs i is a natural-language subtask derived from the instructionIalone, before any action is executed. Note that, the number of subtaskspis generally different from the number of stepsT, since one subtask may span multiple steps. The pair (I,s)is given to the agent throughout the trajectory. Before emitting each action, the agent self-reports its current subtask, so that action execution is aligned with the current subtask. VisualMemoryViewcompaction Submemory Recovery RetrieveCompaction Figure 6. Components of AGMem. AGMem compacts GUI observations into structured memories, stores them in submemory and recovery modules, and retrieves compact context for future actions. We then keep the memory salient in two core components. First (Appendix B.1), every memory step is reduced to an action-relevant view, a tight crop of the screenshot closely related to the executed action. This action-relevant view is used for every step stored in memory, so that retrieval and action prediction can focus on action-relevant views rather than full screenshots. Second (Appendix B.2), retrieval is restricted to salient candidates in two stages. A trajectory-level retriever 12 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents first selects a small pool of memory trajectories that cover the subtasks ins. Then, a step-level retriever searches for the few memory steps closest to the present state. A recovery-state detector additionally redirects retrieval to a separate recovery memory once the agent enters an erroneous state, providing examples of how to recover rather than only examples of successful execution. Figure 6 summarizes the main components of AGMem. B.1. Salient memory entries: action-relevant vision memory A desktop screenshot is often dominated by content unrelated to the current action or state (e.g., background windows, persistent toolbars, and idle panels), that are not affected by the agentâs actions (Lin et al., 2025; Li et al., 2025). Storing all such screenshots would unnecessarily expand the memory bank and provide the agentâs visual context with task-irrelevant cues. We therefore reduce each memory entry to its salient form, which we call the action-relevant view. For each stept, we denote this view by C t , which is a tightly cropped image patch that isolates the GUI region in which action a t takes effect. Construction Given the screenshoto t (Figure 7a), the agent executes actiona t , and the environment transitions to the next screenshoto t+1 . Based on the consecutive screenshots(o t ,o t+1 ), we build the action-relevant viewC t related toa t (Figure 7b). (a) Screenshot o t at time t.(b) o t+1 after a t , highlighted by the action-relevant crop C t . Figure 7. Action-relevant vision memory. Given a screenshoto t at timet,AGMem executes an actiona t ,and then crop the action- relevant vision memory C t from o t+1 . This construction proceeds in four steps. First, we compute a pixel-wise difference between the two screenshots, subtract a locally smoothed version to suppress slowly varying background, and binarize the residual under a soft and a sharp threshold. We apply standard morphological opening and closing to remove noise and connect fragmented regions into a single change mask (Bhutada et al., 2022) . Second, we take the tightest bounding box enclosing the change mask. Third, we snap the bounding box to the smallest UI container, such as a window, panel, or dialog returned by a UI parser, that fully contains it. Finally, if no qualifying container is returned by the parser, we keep the bounding box itself; if no change is detected at all, we fall back to the full screenshot. At test time at step t, the action a t has not yet been executed, so o t+1 and hence C t are not yet available. We therefore use the previous stepâs viewC tâ1 , computed from the most recent pair(o tâ1 ,o t )by the same procedure, as the visual query for memory retrieval (see Section 3.3). Fort = 0, where no previous transition exists, we use the initial full screenshoto 0 directly as the query. Memory structure We construct the memory bankMfrom AgentNet dataset (Wang et al., 2026), which we denote as D AGNet , where start from retaining only non-redundant and correct trajectories Ď . For each trajectoryĎ â D AGNet , an LLM produces a list of post-hoc subtask labelss Ď = (s 1 ,s 2 ,...,s q )containing information about what actually happened on the screen duringĎ.In contrast tos, which is derived from the instructionI alone before any action is executed,s Ď is generated post-hoc, afterĎhas been observed in full. Hence, it is also grounded in the executed trajectoryĎas well as the instructionI.Thus, its segmentation reflects what actually happened on the screen through the trajectory Ď , rather than only what the user asked for through I . We then segment Ď alongs Ď into sub-trajectories. Consequently, the atomic unit of memory is a memory step defined as: m t = (s j ,a t ,C t ). where s j âs Ď is the subtask level that the step realizes, a t is the executed action, and C t is the action-relevant view. 13 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents B.2. Recovery-aware memory retrieval Searching the entire memory bank at every step or action is both costly and noisy, since most trajectories are irrelevant to the current task and individual steps may produce spurious visual memories. We therefore narrow the search space in two stages. We first select a small pool of trajectories at the task level, and then retrieve individual memory steps within that pool. In addition, when the agent flags an error, we redirect retrieval to a separate recovery memory. Two-stage retrieval Given the current subtask lists, the trajectory-level retriever selects memory trajectories that cover the current task. For each subtasksâs, we compare it with the subtask labelss Ď of each memory trajectoryĎ âMusing a Sentence-Transformer encoder (Reimers and Gurevych, 2019). A trajectory is considered relevant tos,if any of its subtask labels is close tosin the embedding space. We greedily select a size-kpoolM sub â M, wherekis the number of memory trajectories selected at the trajectory level. This selection encourages the pool to cover all subtasks insrather than repeatedly selecting trajectories for only a single dominant subtask. After the trajectory pool is selected, the step-level retriever searches over the memory steps contained inM sub . Each memory step is represented by both its subtask label and its action-relevant view. We encode these with the CLIP encoders (Radford et al., 2021): the subtask labels are encoded by the CLIP text encoder and the action-relevant view are encoded by the CLIP image encoder. The two embeddings are combined with a fixed text-image weight of0.5for each, which is shared across all retrieval queries. The current state at steptis represented in the same way, using the current subtask self-reported by the agent and the previous stepâs viewC tâ1 as the visual query. We then retrieve the top-5 memory steps whose combined representations have the highest cosine similarity to the current state, and append them to the agentâs prompt asR t . Recovery-aware verification One of the common failure modes of GUI agents is the error propagation. Once the agent enters an erroneous state, its subsequent action proposals are made under a faulty premise. This issue is difficult to handle with the main memory bank alone, because the main memory bank only stores successful executions and therefore does not contain examples of âhow to recoverâ from mistakes. To address this issue, we propose to use an LLM-based recovery-state detector. Before each proposed action is executed, the detector inspects the recent screenshots, the executed actions, and the proposed action. It then predicts whether the agent is in an erroneous state and, if so, identifies the corresponding failure mode in Table 1. When no error is detected, the proposed action is executed as usual. When an error is detected, the proposed action is blocked, and retrieval is redirected to a separate recovery memory. This recovery memory provides examples of corrective behaviors that bring the agent back from a faulty state to a valid one. The recovery memory is constructed from AgentNet sub-trajectories that were excluded from the main memory bank as redundant or incorrect, but were later corrected within the same trajectory. Each recovery example makes a pair of a faulty sub-trajectory and the corrective step that restores a valid state. Each example is also labeled by failure mode, allowing retrieval to match the flagged failure pattern rather than rely only on global similarity. 14 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents C. Experimental details C.1. Prompts for failure mode analysis Prompt You are given an evaluation result folder from a GUI/web/action-agent benchmark. Analyze the folder and produce failure-mode statistics using the taxonomy below. The benchmark may be OSWorld, WebForge, AgentNet, or another benchmark with similar artifacts. Do not assume one fixed folder format. Inspect the folder first, infer the evaluation unit, and then apply the same root-cause definitions consistently . ## Input The user will provide: ```text RESULT_DIR=<absolute path to an evaluation result folder> ``` ## Possible Artifacts Use whatever files exist in the provided folder. Common artifacts include: - summary files - per-task or per-instance files - trajectories - raw model inputs/outputs - screenshots/images - memory/retrieval files - ground-truth action files for offline benchmarks Prefer raw model reasoning/output, executed actions, observations/screenshots, ground truth, retrieved memory, and final scores over pre-existing detector labels. ## Core Rules Do not trust detector labels, evaluator names, superficial task types, or judge explanations as root causes by themselves. Use them only as pointers. The root- cause classification must come from the actual instruction, model reasoning, raw response, executed action, observation/image, ground truth, retrieved memory, and final outcome. Use multi-label classification. One evaluation unit may have multiple root causes. Count each root cause at most once per unit. ## Unit Selection Choose the evaluation unit based on the benchmark: - Online task-completion benchmarks such as OSWorld and WebForge: use task-level units. One task is one evaluated instance. - Offline action-prediction benchmarks such as AgentNet: use action-level units. One predicted action/step is one evaluated instance. - If the benchmark is unknown, inspect the data: - If each record has a final task score or judge result, use task-level. - If each record compares a predicted action against a ground-truth action, use action-level. 15 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents ## Denominator Percentages must use the total number of evaluated units, not only failed units. Use: ```text ratio = count / total_evaluated_units * 100 ``` Otherwise: - Deduplicate repeated task IDs or action IDs when needed. - Use the number of unique evaluated units found in the folder. ## Which Units To Label Assign failure labels only to units that actually failed or partially failed. Task-level benchmarks: - Failed/partial means final score is below the success threshold, judge says incorrect, or result is false. - If scores are numeric, use`score < 0.999` as the default failed/partial threshold unless the benchmark defines another threshold. Action-level benchmarks: - Failed means the predicted action is incorrect, mismatched, unparseable, or materially different from ground truth. - If partial credit exists, treat non-perfect action scores as failed/partial unless the benchmark documentation indicates otherwise. Successful units should not receive failure labels. ## Failure Categories Classify each failed/partial unit into the following five categories. ### 1. Cognitive Failure The model makes a wrong task-level, subtask-level, semantic, or progress-state decision. Include cases where: - It misunderstands the user's actual goal. - It chooses the wrong next action or wrong subtask. - It assumes a failed action succeeded and continues. - It prematurely outputs`DONE`,`FAIL`,`WAIT`, or any equivalent terminal action. - It keeps following an invalid plan despite evidence of failure. - It makes a semantic/content reasoning mistake, such as wrong formula, wrong answer , wrong comparison, wrong extraction, wrong data transformation, wrong ordering, or wrong condition. - It predicts an action that does not correspond to the ground-truth next action because it reasoned about the task incorrectly. Examples: - Ground truth is to click Search, but the model says it should click Settings because it misunderstood the intended next step. - Task asks to calculate grades using a scale, but the model writes a formula with 16 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents reversed thresholds. - Task asks to approve a specific request, but the model opens or approves a different request. ### 2. Visual State Misunderstanding The model misreads the current visible observation or screen state. Include cases where: - It thinks a dialog, menu, page, app, sheet, slide, layer, selection, cursor focus, checkbox, search result, object, or file is in a different state from what is visible. - It says something is selected when it is not selected. - It says a dialog is gone/open when the observation shows the opposite. - It focuses on the wrong visible element because it misinterprets the screenshot/ image. - It predicts based on a nonexistent or misread visual state. Do not classify as Visual State Misunderstanding merely because the unit failed. There must be evidence in the raw response/action or image that the model's perceived visible state differs from the actual visible state. Examples: - Raw response says "the image is selected", but the screenshot shows a text placeholder selected. - The model describes a Settings page, but the observation is still a document editor. - For an offline action step, the model targets a visible element that is not the one requested by the ground truth because it misread the image. ### 3. Hidden Operation Blindness The model fails because the necessary operation is not directly visible in the current observation and must be found through a menu, submenu, dropdown, context menu, keyboard shortcut, command, file picker, save/export dialog, preferences/ settings page, API call, or other hidden operation path. Include cases where: - The correct operation requires opening a menu or submenu. - The correct operation requires a keyboard shortcut, command-line operation, or hidden app feature. - The correct operation is behind Save As, Export, Import, Preferences, Settings, Format Cells, Chart Wizard, Extension Manager, file picker, or context menu. - The model repeatedly tries visible UI controls but does not discover the hidden operation path. - The model opens the wrong hidden path or cannot navigate a hidden menu/dialog workflow. Do not classify as Hidden Operation Blindness just because the task is difficult. The failure must involve inability to find or execute a non-visible operation path. Examples: - Cannot find an export-as-PNG workflow. - Cannot locate a setting buried inside preferences. - Cannot use a required keyboard shortcut or command-line operation. ### 4. Grounding Error 17 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents The model's reasoning identifies the correct visible target, but the executed or predicted low-level action misses or fails to activate that target. Only classify as Grounding Error when the raw reasoning shows that the intended target was correct. Include cases where: - The model says it will click the correct button/link/field, but the coordinates are off. - It repeatedly clicks the correct intended target area and the observation does not change. - It drags or selects the correct object/range in intent, but the actual selected region is wrong due to coordinate/drag grounding. - Text entry targets the right field in reasoning, but focus is wrong due to click/ coordinate failure. - In action-prediction benchmarks, both prediction and ground truth intend the same element, but the predicted coordinate/bbox is outside the target. Do not classify as Grounding Error when: - The model chooses the wrong target from the start. - The model reasons about the wrong element. - The failure is due to not knowing the operation path. Examples: - Raw response says "click the close button", but repeated clicks at the coordinate do not close the dialog. - Ground truth and prediction both refer to the Search button, but the predicted coordinate lands outside the button. ### 5. Memory Over-following The model's current reasoning or action is contaminated by retrieved/example memory that does not belong to the current task context. Only classify as Memory Over-following when there is explicit or highly concrete evidence that memory content influenced the wrong action. Include cases where: - Memory/retrieval files contain unrelated task details, and the raw response/action repeats those unrelated details. - Raw response suddenly mentions a different application, file, object, target, keyword, color, number, menu, or workflow that appears in memory but not in the current task. - The model follows an example-memory action sequence even though the current observation/task calls for a different action. Do not classify as Memory Over-following when: - Memory exists but the model does not follow it. - The retrieved memory is relevant and helps solve the current task. - The model merely selects the wrong subtask from the current task's own subtask list. That is Cognitive Failure, not Memory Over-following. - There is no memory artifact in the folder. Examples: - Current task is in LibreOffice, but retrieved memory is VS Code auto-save and the model repeatedly opens "Visual Studio Code". - Current task is to click a Search button, but the raw response mentions a 18 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents remembered Settings workflow unrelated to the current observation. ## Classification Workflow For each unique evaluation unit: 1. Determine whether the unit is successful, failed, or partially failed. 2. If successful, do not assign failure labels. 3. If failed/partial, inspect all available evidence: - instruction/task metadata - ground truth - raw model input/output - reasoning text - predicted/executed action - screenshots/images/observations - memory/retrieval content - final score/judge result 4. Assign all applicable root-cause labels from the five categories. 5. Count each category at most once per unit. When deciding between categories: - Wrong target because the model reasoned about the wrong UI element: Cognitive Failure or Visual State Misunderstanding. - Correct target in reasoning but coordinate/bbox/action misses: Grounding Error. - Failure to discover a menu/shortcut/dialog/workflow: Hidden Operation Blindness. - Believing the current observation is in a state contradicted by the image: Visual State Misunderstanding. - Wrong formula, wrong extraction, wrong answer, wrong ordering, wrong task decomposition: Cognitive Failure. - Irrelevant retrieved memory leaks into current action: Memory Over-following. ## Required Output Format First give one concise sentence describing: - benchmark or inferred benchmark type - unit type - total evaluated units - success count - failed/partial count - mean final score if available Then output exactly this table format: ```markdown | Root cause | Count | Ratio | |---|---:|---:| | Cognitive Failure | N | P% | | Visual State Misunderstanding | N | P% | | Hidden Operation Blindness | N | P% | | Grounding Error | N | P% | | Memory Over-following | N | P% | ``` Percentages must be computed as: ```text count / total_evaluated_units * 100 ``` Round percentages to one decimal place. 19 Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents After the table, provide a short "Evidence Notes" section with 3 to 5 representative cases. For each representative case, include: - category - benchmark unit ID, task ID, or action ID - raw response/action file path and line number if available - screenshot/image filename if relevant - memory file/path and line number if the case is Memory Over-following - one sentence explaining why the classification is justified Do not include long copied logs. Quote only the shortest raw content needed to justify the classification. 20