Paper deep dive
Learning How to Remember: Memory Modality Selection for GUI Agents
Taeyun Roh, Suhyeong Park, Gijin Kim, Wonjune Jang, Jaewoo Kang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent advances in vision-language models (VLMs) have substantially improved the capabilities of GUI agents, while memory augmentation has emerged as a promising approach for reusing knowledge from past interactions. However, existing memory augmented GUI agents typically adopt a fixed representation strategy, providing retrieved experience as a static modality. It remains unclear whether a single memory modality is consistently optimal, or how modality preference changes across interaction steps. In this work, we conduct a systematic study of memory modality in VLM based GUI agents. Across GUI benchmarks, we find that the preferred memory representation is strongly step dependent. Motivated by the finding, we introduce \textbf{SwitchMem}, a routing framework that dynamically selects between textual and visual memory representations at each step. Across three GUI benchmarks, SwitchMem outperforms memory augmented baselines that rely on static memory representations. Our findings validate our claim that memory modality is a step dependent decision rather than a fixed design choice. Code and project resources are available at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2610.17928v1
- Canonical: https://arxiv.org/abs/2610.17928v1
Trouble viewing inline? Open PDF directly →
Full Text
62,827 characters extracted from source content.
Expand or collapse full text
Learning How to Remember: Memory Modality Selection for GUI Agents Taeyun Roh 1 Suhyeong Park 1 Gijin Kim 1 Wonjune Jang 2 Jaewoo Kang 1,3 1 Korea University 2 Myongji University 3 AIGEN Sciences nrbsld,kangj@korea.ac.kr Abstract Recent advances in vision-language models (VLMs) have substantially improved the capa- bilities of GUI agents, while memory augmen- tation has emerged as a promising approach for reusing knowledge from past interactions. However, existing memory augmented GUI agents typically adopt a fixed representation strategy, providing retrieved experience as a static modality. It remains unclear whether a single memory modality is consistently optimal, or how modality preference changes across in- teraction steps. In this work, we conduct a systematic study of memory modality in VLM based GUI agents. Across GUI benchmarks, we find that the preferred memory represen- tation is strongly step dependent. Motivated by the finding, we introduce SwitchMem, a routing framework that dynamically selects be- tween textual and visual memory representa- tions at each step. Across three GUI bench- marks, SwitchMem outperforms memory aug- mented baselines that rely on static memory representations. Our findings validate our claim that memory modality is a step dependent de- cision rather than a fixed design choice. Code and project resources are available here. 1 Introduction Recent advances in vision-language models (VLMs) have substantially improved their ability to perceive, reason over, and act upon visual en- vironments (Xie et al., 2024; Gemini Robotics Team et al., 2025; Jung et al., 2026). Such progress has enabled graphical user interface (GUI) agents, where VLMs directly interpret screenshots, rea- son about user instructions, and execute actions across desktop and mobile environments (Xu et al., 2024; Zhang et al., 2025; Rawles et al., 2025; Qin et al., 2025). Despite these advances, GUI interac- tion remains challenging: tasks often require long- horizon decision making, and errors made at one step can propagate to subsequent decisions. Same Memory Text Modality Same Memory Image Modality SUCCESS FAILURE Figure 1: The same retrieved memory can lead to differ- ent outcome for a GUI agent depending on modality. To learn from prior interactions and reduce re- peated errors, recent GUI agents augment the actor with memories constructed from past trajectories, demonstrations, interaction knowledge, and visual observations (Li et al., 2026b; Qin et al., 2026; Choi et al., 2026; Liu et al., 2026; Zeng et al., 2026). However, existing systems generally adopt prede- fined memory representations, presenting retrieved experience solely as text, visual observations, or a fixed combination of both modalities. Once cho- sen, this representation strategy typically remains unchanged across interaction steps. However, existing GUI memory systems rarely question whether a single representation modality is suitable for every interaction step. Prior work on VLMs suggests that textual and visual represen- tations provide fundamentally different forms of information and that their relative utility depends on the reasoning context. (Pezeshkpour et al., 2025; Yang et al., 2026; Long et al., 2026; Roh et al., 2026b). Text can provide a compact semantic abstraction of visual content, but converting im- ages into language may discard fine-grained infor- mation required for downstream reasoning (Yang et al., 2026). Conversely, visual representations preserve fine-grained spatial and appearance in- 1 arXiv:2610.17928v1 [cs.AI] 13 Oct 2026 AITW 48.0%52.0% LearnGUI 47.7%52.3% 050100 TextImage (a) Share among non-tied steps Figure 2: Relative frequency of Text and Image wins among interaction steps. Neither modality consistently dominates across benchmarks. formation that textual surrogates may fail to re- tain (Yang et al., 2026; Jenkins, 2026). However, when the information needed for reasoning can be captured in structured text, textual abstraction can improve performance on complex visual reason- ing tasks (Liu et al., 2023; Park et al., 2025; Xu et al., 2026). Although these findings highlight the context-dependent role of modality in VLM rea- soning, it remains unclear whether the preferred memory modality varies across interaction steps in memory enhanced GUI agents. In this work, we conduct a systematic study of memory modality in VLM based GUI agents. We show that preferred memory modality changes each step, and that the preference is revealed through models internal hidden states. Based on the obser- vations, we introduce SwitchMem, a lightweight router that selects between textual and visual mem- ories at each step. Our contributions are threefold: •We show that the preferred memory modality varies across interaction steps and that fixed modalities lead to suboptimal performance across evaluation settings. •We show that memory modality preference is reflected at the representation level within the agent. Our Directional Chain-of-Embedding (D-CoE) analysis reveals a reproducible Tex- t/Image preference signal that generalizes to unseen GUI trajectories. • Building on this representation-level signal, we propose SwitchMem, a routing framework that adaptively selects between Text and Im- age memory at each interaction step. Conse- quently, it consistently outperforms all static memory baselines across evaluation tasks. 2 Related Work Mobile GUI Agents. Mobile GUI agents exe- cute user instructions through repeated observation and interaction with mobile applications. At each step, the agent interprets the current screenshot in the context of the task instruction and interaction history, then predicts an action such as tapping, swiping, or entering text (Wang et al., 2024, 2025; Qin et al., 2025). The resulting interface state be- comes the observation for the next decision, and this process continues until the task is completed. Recent VLM based approaches integrate visual per- ception, action grounding, and sequential reasoning to support this interaction loop (Xu et al., 2024; Qin et al., 2025). Despite these advances, reliable mo- bile interaction remains challenging because each action changes the interface state, and errors in per- ception or action selection can propagate through subsequent decisions. Memory Systems for GUI Agents. Memory augmentation enables agents to extract and reuse actionable knowledge from prior interactions, sup- porting informed decisions and reducing the risk of repeated failures (Chhikara et al., 2025; Yeo et al., 2026; Xu et al., 2025b; Roh et al., 2026a). In GUI environments, existing approaches pre- serve episodic histories and retrieve relevant ex- periences as in-context examples (Agashe et al., 2025; Xu et al., 2025a; Zhang et al., 2025; Liu et al., 2026). Others transform interaction trajecto- ries into reusable textual traces, structured proce- dures, or executable knowledge that guides action generation (Li et al., 2026b; Qin et al., 2026; Xiao et al., 2026). Visual memory methods instead retain screenshots or action-relevant regions to ground de- cisions in prior interface states (Rawles et al., 2025; Choi et al., 2026; Zeng et al., 2026). While these methods differ in how memories are constructed and retrieved, they use a fixed modality after re- trieval. SwitchMem instead treats memory modal- ity as a step level decision, selecting between Text and Image based on the current interaction state. 3 Understanding Memory Modality Preference 3.1 Study Setup We compare textual and visual representations of the same retrieved demonstration at each interac- tion step. By holding the retrieved demonstration fixed, we examine how modality preference varies 2 AITWLearnGUI 35 40 45 50 55 60 65 Action score (%) 49.3 37.4 55.9 43.4 Best fixedStep oracle Figure 3: Performance gap between fixed modality and step-wise oracle modality memory selection. across steps. We use Qwen3.5-9B as the actor in all primary comparisons. At step i, let x i = (g i ,o i )(1) denote the current GUI state, whereg i is the task instruction ando i is the current screenshot. Each memory candidate is retrieved from a bank of valid demonstration steps containing a past GUI state and its successful demonstrated action. To isolate the effect of memory representation from retrieval quality, we use a shared text-based retriever for both modalities. For each statex i , the retriever selects a single demonstrationj i , which is then instantiated in either textual or visual form: m T i = g j i ,t j i ,a j i , m I i = g j i ,v j i ,a j i . (2) whereg j i is the task instruction associated with the retrieved demonstration,t j i is the textual de- scription of its GUI state,v j i is the correspond- ing screenshot, anda j i is the demonstrated action. The textual state representation is constructed from structured UI information provided by each bench- mark. At each interaction stepi, we compare the ex- pected action utility obtained under Text and Image memory. For modalityz ∈T, I, we estimate its utility usingKsampled actions from the frozen actor: ˆu z i = 1 K K X k=1 S b a z i,k ,a ∗ i ,(3) whereS b is the benchmark specific action scorer anda ∗ i is the ground-truth action. We define the step-wise memory modality preference as ˆ ∆ i = ˆu I i − ˆu T i .(4) 08162331 Transformer layer 0.5 0.6 0.7 0.8 AUROC Figure 4: layer-wise modality-preference probing with D-CoE on AITW. Using examples with at least five- of-seven agreement in sampled preference, AUROC in- creases in deeper layers, indicating a stronger modality- preference signal in later hidden representations. 3.2 Step-wise Memory Modality Preference We examine how memory modality preference varies across interaction steps using the estimated utility gap ˆ ∆ i defined in Equation 4. Neither modality consistently dominates. As shown in Figure 2, both Text and Image memory are preferred for substantial subsets of interaction steps across AITW and LearnGUI. Image memory yields a slightly higher average performance on both AITW and LearnGUI, but the weaker modal- ity on aggregate still provides higher utility on nu- merous individual execution steps. The preferred modality therefore depends on the current interac- tion state. Fixed modalities leave consistent performance on the table.The step-wise variation in modality preference implies that a single fixed representation cannot be optimal throughout an entire interaction. To quantify this effect, we define a step-wise ora- cle. At each evaluation step, we run the frozen actor separately with Text and Image memory under the same fixed evaluation seed. We score both pre- dicted actions using the benchmark specific metric, and the oracle takes the higher of the two scores. As shown in Figure 3, the step-wise oracle con- sistently outperforms the strongest fixed-modality strategy across benchmarks, revealing a persistent gap caused by static memory representation. This gap provides direct motivation for adaptive, step- level modality selection. 3 (a) Offline training Current state Instruction + screen Retrieved memory Same ID Actor + Text memory Actor + Image memory hᵀ hᴵ D-CoEshift d = hᴵ − hᵀ Preference probe Preference target t: Image vs Text (b) Inference Instruction + screen Frozen actor → s Text / Image memory eᵀ, eᴵ (same memory ID) Pairwise router: shared MLP vᶻ= [ŝ; êᶻ; ŝ ⊙êᶻ; |ŝ − êᶻ|] MLP → qᵀMLP → qᴵ r = qᴵ − qᵀ Select memory r ≥ τ: Image otherwise: Text Frozen actor Current state + one memory Action Router training ℒ= (r − t)² t (target) Figure 5: Overview of SwitchMem. During offline training, the difference between the actor’s hidden states obtained with Image and Text memory is used to construct a continuous modality preference target. A pairwise router scores both memory candidates and is trained to match their score difference to this target. At inference, the router selects one memory modality for the frozen actor to generate the next action. 3.3 Modality Preference in Hidden States Building on the finding that memory modality pref- erence varies across interaction steps, we probe the actor’s hidden representations to determine whether the optimal modality can be predicted at each step. Directional Chain-of-Embedding. Chain-of- Embedding (CoE) analyzes how different input conditions diverge throughout a VLM by com- paring their intermediate hidden representations (Long et al., 2026). We adapt this perspective to contrast the retrieved Text and Image memory con- ditions. Leth z i,ℓ denote the hidden state of the frozen actor at layerℓunder memory condition z ∈ T, I. The actor input contains the current task instruction, the current GUI screenshot, and the retrieved memory represented in modalityz. Retrieval is performed once based on a text embed- ding, and the same retrieved demonstration ID is used under both Text and Image memories. We extracth z i,ℓ from the final input token of the fully constructed prompt, immediately before action gen- eration. We retain the signed shift d i,ℓ = h I i,ℓ − h T i,ℓ ,(5) and refer to the layer-wise sequenceC i =d i,ℓ L ℓ=1 as the Directional Chain-of-Embedding (D-CoE). Modality preference reflected in hidden repre- sentations. We probe whetherd i,ℓ contains in- formation about which memory modality yields higher action utility. For each transformer layer, we reduce the shift representation with PCA and train a linear probe to predict the preferred modality. To reduce noise from stochastic action sampling, we retain only examples for which the same modal- ity is preferred in at least five of seven sampled generations. As shown in Figure 4, the modality preference is predictable from the representation shift on AITW. This suggests that modality prefer- ence becomes reflected in the actor’s hidden repre- sentations. 4 SwitchMem SwitchMem is trained in two stages. First, a prefer- ence estimator processes Text and Image memory through separate actor forward passes and maps the resulting representation shift to a preference target. Second, a pairwise router is trained to predict the preference target from the current state and the re- trieved memory candidates. The router applies a shared MLP scorer to each state–memory pair and uses the difference between the Text and Image scores as its routing score. 4.1 Preference Target Construction In this phase, the actor’s modality dependent rep- resentation shift is transformed into a continuous preference target used to train the pairwise router. 4 MethodAugmentationAITW (%)LearnGUI (%)MobileMiniWoB (%) General-purpose GPT-5.4-mini–34.00024.91251.087 Qwen3.5-9B–41.92023.48558.696 GUI specialized Aguvis-7B–33.24024.73629.348 UI-TARS-2B-SFT–23.2609.3658.696 UI-TARS-7B-SFT–29.98025.70523.913 UI-TARS-1.5-7B–36.0006.01938.043 Qwen3.5-9B Memory augmented RAG-GUIGenerated guidelines18.30024.1869.783 M3AAccessibility / SoM17.20020.89941.304 AppAgentOperation document14.78031.10425.000 TextText memory48.76035.63853.261 ImageImage memory49.30037.44157.609 BothText and Image47.08026.91960.870 EchoTrailRetrieved text step45.90027.28655.435 AGMemAction region crop45.38028.36652.174 SwitchMemAdaptive memory51.42038.13163.043 Table 1: Performance comparison on AITW, LearnGUI, and MobileMiniWoB++. AITW and LearnGUI use the official benchmark-provided step level action scores, whereas MobileMiniWoB++ reports task success rate. The memory-augmented group compares different augmentation strategies using the same frozen Qwen3.5-9B actor and evaluation protocol. We use the final-layer D-CoE representation de- fined in Equation 5: We select the final transformer block based on the layer wise probing analysis in Section 3.3. To construct the preference targets, we compare the sampled action utilities obtained with Text and Image memory. We retain the non-tied training states D pref =i| ˆu I i ̸= ˆu T i and fit a preference estimator to predict whether ˆu I i − ˆu T i > 0. To reduce sample-specific noise in the 4096-dimensional representation and obtain stable preference targets, we standardized i and reduce it to 64 dimensions using PCA. We then fit a logistic probe whose decision value defines the soft preference target: t i = w ⊤ PCA (Std(d i )) + b(6) A positive targett i favors Image memory, whereas a negative target favors Text memory. 4.2 Preference-Supervised Router Training To enable modality selection, we train a pairwise router using the continuous preference targets de- rived in Section 4.1 as supervision. Formally, given interaction stepi, lets i denote the final-token hid- den state obtained from the actor prompt contain- ing the current instruction and screenshot. For each modalityz ∈T,I, lete z i denote the embedding of the corresponding retrieved memory candidate. We projects i ande z i to a shared dimension and apply ℓ 2 normalization, yielding ˆs i and ˆe z i , respec- tively. For each modality, we construct the candi- date feature v z i = [ˆs i ; ˆe z i ; ˆs i ⊙ ˆe z i ;|ˆs i − ˆe z i |],(7) where⊙ denotes element-wise multiplication. A shared MLP scorerf θ , parameterized byθ, maps each candidate feature to a scalar score: q z i = f θ (v z i ).(8) The router then computes the relative modality- preference score as r i = q I i − q T i .(9) Because the same scorer is applied to both modali- ties,q T i andq I i are directly comparable, and their differencer i represents the relative preference for Image over Text memory. We train the router to regress the soft preference target t i using L route = 1 |D pref | X i∈D pref (r i − t i ) 2 .(10) At inference,r i is compared with a predefined thresholdτ. Image memory is selected when 5 BenchmarkBothImageSwitchesRun AITW83.836.51.722.42 LearnGUI90.881.31.794.95 MobileMiniWoB57.876.31.564.50 Table 2: Within-trajectory routing behavior of Switch- Mem. Both and Image denote the percentages of dual- modality trajectories and Image-routed steps, respec- tively. Switches and Run measure the average modality transition count and contiguous run length per trajec- tory. r i ≥ τ; otherwise, Text memory is selected. Only the selected memory is then provided to the frozen actor for action generation. 5 Experiments Benchmarks.We evaluate SwitchMem on three GUI benchmarks: MobileMiniWob++ (Rawles et al., 2025), AITW (Rawles et al., 2023), and LearnGUI (Liu et al., 2026). Together, these bench- marks cover diverse mobile GUI interaction set- tings and allow us to examine whether adaptive memory modality selection generalizes across en- vironments. Because MobileMiniWoB++ does not provide official training or validation splits, we transfer the router trained on LearnGUI while keep- ing its weights fixed. Further details on the transfer and threshold-calibration protocol are provided in Section B.1. Models and Retrieval.All primary experiments use Qwen3.5-9B (Qwen Team, 2026) as the frozen actor and Qwen3-VL-Embedding-2B (Li et al., 2026a) for top-1 retrieval and memory embedding. Text and Image memories are instantiated from the same retrieved demonstration, isolating the effect of memory representation. Baselines. WecompareSwitchMemwith general-purposeVLMs(GPT-5.4-miniand Qwen3.5-9B), GUI specialized agents, such as Aguvis (Xu et al., 2024) and UI-TARS variants (Qin et al., 2025), and memory-augmented methods. The memory-augmented group uses the same frozen Qwen3.5-9B actor and includes RAG-GUI (Xu et al., 2025a), M3A (Rawles et al., 2025), AppAgent (Zhang et al., 2025), EchoTrail (Li et al., 2026b), AGMem (Choi et al., 2026), and fixed Text, Image, and Both baselines. 5.1 Results Main results.Table 1 shows that augmenting the GUI agent with retrieved experience can improve action prediction, but its effectiveness depends on how the experience is represented. Image mem- ory tends to perform well on AITW and LearnGUI, whereas providing both modalities is more effective on MobileMiniWoB. Notably, advanced memory formulation strategies do not consistently outper- form raw context injection, suggesting that altering context representation without step-wise selectivity yields limited benefits. SwitchMem achieves the strongest performance across all three benchmarks. Rather than fixing one representation or always providing both, it selects a single memory modality at each interaction step. This allows the actor to exploit the complementary strengths of textual and visual memory across steps while avoiding the performance degradation that can arise from jointly providing both modalities. Modality preference changes within trajectories. Table 2 offers a quantitative analysis of Switch- Mem’s routing behavior across steps within a tra- jectory. Across all benchmarks, SwitchMem fre- quently utilizes both modalities within a single tra- jectory exhibiting moderate transition frequencies. Crucially, selected modalities are retained for con- secutive steps. This demonstrates that SwitchMem does not lock into a static task level modality, but exhibits state adaptive modality. 6 Analysis 6.1 Modality Utility Patterns and Adaptive Routing Alignment To uncover the mechanisms governing dynamic modality selection, we analyze interaction context from two complementary perspectives. First, we identify when Image modality is preferred over Text discussed in Section 3.2. Second, we evaluate whether SwitchMem’s learned router aligns with these utility patterns to drive overall performance gains. Figure 6 illustrates how SwitchMem’s rout- ing decisions align with these utility shifts across each dimension. 6.1.1 When Is Each Memory Modality Useful? Visually dense states favor Image memory.To quantify structural complexity, we measure ele- ment density using bounding box annotations pro- vided in each benchmark. Specifically, we define 6 Δ Utility (pt)Δ Image selection (p) 0 2 4 6 8 10 12 14 16 0 10 20 30 40 Element density Dense vs. Sparse Function vs. Spatial Post-initial vs. Initial Δ Utility (pt) Δ Image selection (p) Figure 6: Changes in Image over Text utility (left axis) and SwitchMem’s Image-selection rate (right axis). element density as the number of unique UI ele- ments divided by their fractional screen coverage, computed as the union area of the bounding boxes relative to the total screen area. Empirical analy- sis shows that element density remains positively associated with the Image-over-Text utility gap. Im- age memory yields an advantage when decisions rely on fine-grained layouts containing dense UI elements that textual abstractions fail to convey. Image memory is more useful for function- oriented actions.Following standard GUI action taxonomies (Shi et al., 2025; Lu et al., 2025), we group interaction steps into Spatial and Function actions. Spatial actions comprise coordinate-based operations such as tapping, swiping, scrolling, and dragging, whereas Function actions govern inter- face functionality, task status, or system navigation. When shifting from Spatial to Function actions, the utility advantage of Image memory grows across benchmarks, showing that functional interface com- ponents rely on visual grounding. This suggests that visual memory is particularly valuable for deci- sions requiring an understanding of interface state and functionality, beyond simply locating interac- tion targets. Image memory becomes more useful after the in- terface evolves.Compared to initial states, post- initial states exhibit a wider Image-over-Text utility gap. This finding indicates that as a trajectory pro- gresses beyond initial states, evolving UI contexts generate complex visual nuances that text-based representations fail to preserve. 6.1.2 Understanding SwitchMem’s Memory Modality Selection While Section 6.1.1 identified when visual context is useful, we now examine whether SwitchMem’s learned router dynamically aligns with these utility patterns. Furthermore, we evaluate how Switch- Mem’s adaptive routing translates into overall per- formance gains across benchmarks. SwitchMem follows modality-utility patterns. SwitchMem’s routing selections consistently align with the utility shifts identified across all three di- mensions (Figure 6). The router selects Image memory more frequently on visually dense screens, at steps whose reference actions are categorized as Function rather than Spatial, and in post-initial states. This alignment demonstrates that Switch- Mem’s router dynamically responds to systematic changes in modality utility across interaction steps. This behavior helps explain the performance im- provements in Table 1. By favoring Image in con- ditions where visual context is more useful while retaining Text in complementary states, Switch- Mem outperforms static memory strategies across all three benchmarks. 6.2 Comparison with Alternative Routing Strategies To demonstrate the advantages of SwitchMem’s routing design, we analyze its performance against alternative routing baselines. Learned routing baselines.We compare Switch- Mem with three learned alternatives. Utility and Bi- nary utilize the same router architecture as Switch- Mem, constructed upon the actor’s current-state hidden representation and retrieved-memory em- beddings. However, both are trained directly from sampled action utilities, without D-CoE-based tar- get construction. Utility regresses the utility gap us- ing MSE, whereas Binary predicts only its sign us- ing binary cross-entropy. Basic routing also learns binary preferences, but uses embeddings of the cur- rent instruction, screenshot, and retrieved memo- ries rather than the actor’s hidden states. As shown in Figure 7a, conventional routing baselines offer minimal gains over a static Image Memory pol- icy, whereas SwitchMem delivers consistent per- formance improvements. These results validate the necessity of actor-derived preference targets for guiding memory selection. Ratio-matched random routing. To determine whether SwitchMem’s gains arise from step- specific memory selection rather than the overall Text and Image usage ratio, we sweep its threshold and compare it with a random policy that indepen- 7 LearnGUI 37 37.5 38 38.5 Action score (%) 37.441 Image 37.441 Utility 37.392 Binary 37.472 Basic 38.131 SwitchMem (a) Routing baseline ablation 36 37 38 39 0255075100 Image memory usage (%) Action score (%) LearnGUI 81% Image, 38.13 SwitchMem Ratio-matched random Deployed threshold 95% paired CI (b) Modality budget frontier 1.21.62.02.42.8 20 24 28 32 36 40 Latency (s/step) LearnGUI action score (%) No memory Text Image Both M3A EchoTrail AppAgent AGMem SwitchMem (c) Latency–performance trade-off Figure 7: Empirical analysis on LearnGUI. (a) Comparison with simpler routing baselines. Utility and Binary use direct preference supervision, while Basic routing excludes actor hidden states. (b) SwitchMem versus ratio-matched random routing, where the dashed line denotes the analytic expected score and the star marks the deployed threshold. (c) Efficiency Analysis of SwitchMem. RAG-GUI is omitted in figure due to its high latency of 11.118 s/step. dently selects Image memory with probabilityp, matched to SwitchMem’s Image-selection rate. Using the step level action scores under fixed Text and Image memory, we compute the random policy’s expected performance as S rand (p) = (1− p)S Text + pS Image ,(11) whereS Text andS Image denote the mean action scores under the two fixed memory policies. Fig- ure 7b shows that SwitchMem outperforms ratio matched random routing at every evaluated inter- mediate Image selection rate. This indicates that its gains arise from assigning memory modalities to appropriate interaction steps, rather than merely using a particular Text and Image ratio. 6.3 Efficiency Analysis Figure 7c compares the latency–performance trade off on LearnGUI, with all methods measured on the same RTX PRO 6000 GPU. SwitchMem achieves the highest action score while adding only mod- est latency. SwitchMem avoids processing both memory-conditioned prompts and incurs no addi- tional autoregressive decision stage. In contrast, procedural baselines such as M3A, AppAgent, and RAG-GUI introduce additional annotation, docu- ment construction, or guideline-generation stages, resulting in higher latency despite lower perfor- mance. 6.4Generalization to a GUI specialized Actor To evaluate whether SwitchMem generalizes be- yond its base actor, we apply its preference con- struction and routing protocol to the GUI special- ized UI-TARS-7B-SFT model. As shown in Ta- ble 3, SwitchMem consistently outperforms static MethodAITWLearnGUIMobileMiniWoB++ No memory29.98025.70523.913 Text32.64038.23032.609 Image30.44040.99733.696 SwitchMem33.68041.05536.957 Table 3: Generalization to the GUI specialized UI- TARS-7B-SFT actor. SwitchMem achieves the best performance across all three benchmarks, improving over the strongest fixed memory condition. memory baselines across all three benchmarks. These results confirm that SwitchMem’s routing formulation effectively transfers across distinct actor architectures and domain-specialized back- bones. 7 Conclusion We study memory modality as a step-dependent de- cision for VLM-based GUI agents. Across bench- marks, neither Text nor Image memory consistently dominates, and their utility varies across interaction steps. We show that memory modality preference is reflected at internal state through D-CoE analysis. Based on the representation-level signal, we intro- duce SwitchMem which uses modality conditioned representation shifts to train a router that selects the suitable memory modality at each step. Switch- Mem improves over fixed modality and memory- augmentation baselines while providing only one memory modality to the actor. Together, these re- sults confirm that optimal memory selection is step- dependent, and that our SwitchMem approach can predict memory modality achieving performance gains over static alternatives. 8 Limitations Our evaluation focuses on mobile GUI benchmarks centered on Android interfaces and controlled in- teraction settings. Generalization to iOS interfaces and newer Android versions, which may introduce different visual layouts and interaction conventions, remains unverified. Our online evaluation uses emulated environ- ments rather than physical devices. Although these environments support controlled and reproducible experiments, they do not fully capture device ren- dering differences, system interruptions, network variability, or hardware constraints. Evaluation on physical devices is an important direction for future work. SwitchMem requires access to the actor’s hid- den states for both preference target construction and routing at inference. It therefore cannot be directly applied to models accessed through APIs that do not expose these representations. Extending modality routing to such models remains an open direction. References Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Wang. 2025. Agent s: An open agentic framework that uses computers like a human. In International Conference on Learning Representa- tions, volume 2025, pages 22924–22946. Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. 2025. Mem0: Building production-ready ai agents with scalable long-term memory. arXiv preprint arXiv:2504.19413. Seoyoung Choi, Minseok Ko, Hyunseok Lee, Kun- woong Kim, Woomin Song, Chanseok Jeon, and Jin- woo Shin. 2026. Naive visual memory is not enough: A failure-mode study of gui agents. arXiv preprint arXiv:2606.14106. Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonza- lez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. 2025. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:2503.20020. Ben Jenkins. 2026. Thinking in pictures: A diagnostic study of visual vs. textual chain-of-thought reasoning in vision-language models. In Proceedings of the 4th Workshop on Advances in Language and Vision Research (ALVR), pages 1–12, San Diego, California, USA. Association for Computational Linguistics. Junha Jung, Minbyul Jeong, Suhyeon Lim, Sungwook Jung, Jaehoon Yun, Taeyun Roh, Mujeen Sung, and Jaewoo Kang. 2026. Breaking failure cascades: Step- aware reinforcement learning for medical multimodal reasoning. arXiv preprint arXiv:2606.31825. Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2026a.Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of- the-art multimodal retrieval and ranking. Preprint, arXiv:2601.04720. Runze Li, Yuwen Zhai, Bo Xu, Liwu Xu, Nian Shi, Wei Zhang, Ran Lin, and Liang Wang. 2026b. Echotrail- gui: Building actionable memory for gui agents via critic-guided self-exploration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings, pages 9347– 9356. Fangyu Liu, Julian Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Man- dar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. 2023. DePlot: One-shot visual language rea- soning by plot-to-table translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10381–10399, Toronto, Canada. Associ- ation for Computational Linguistics. Guangyi Liu, Pengxiang Zhao, Liang Liu, Zhiming Chen, Yuxiang Chai, Yaozhen Liang, WenHao Wang, Siheng Chen, Zhengxi Lu, Shuai Ren, Hao Wang, Shibo He, Yong Liu, and Wenchao Meng. 2026. Lear- nAct: Few-shot mobile GUI agent with a unified demonstration benchmark. In Findings of the Asso- ciation for Computational Linguistics: ACL 2026, pages 29820–29843, San Diego, California, United States. Association for Computational Linguistics. Lin Long, Changdae Oh, Seongheon Park, and Sharon Li. 2026. Understanding language prior of lvlms by contrasting chain-of-embedding. In International Conference on Learning Representations, volume 2026, pages 94723–94754. Quanfeng Lu, Wenqi Shao, Zitao Liu, Lingxiao Du, Fan- qing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, and Ping Luo. 2025. Guiodyssey: A comprehensive dataset for cross-app gui navi- gation on mobile devices. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22404–22414. Simon Park, Abhishek Panigrahi, Yun Cheng, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. 2025. Gener- alizing from simple to hard visual reasoning: can we mitigate modality imbalance in vlms? arXiv preprint arXiv:2501.02669. Pouya Pezeshkpour, Moin Aminnaseri, and Estevam Hruschka. 2025. Mixed signals: Decoding vlms’ rea- soning and underlying bias in vision-language con- flict. In EMNLP (Findings), pages 24833–24848. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, 9 Yunxin Li, Shijue Huang, et al. 2025. Ui-tars: Pio- neering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326. Zerui Qin, Sheng Yue, Xingyuan Hua, Yongjian Fu, and Ju Ren. 2026. Executable agentic memory for GUI agent. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 101815–101840. PMLR. Qwen Team. 2026. Qwen3.5: Towards native multi- modal agents. Chris Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell- Ajala, et al. 2025. Androidworld: A dynamic bench- marking environment for autonomous agents. In In- ternational Conference on Learning Representations, volume 2025, pages 406–441. Christopher Rawles, Alice Li, Daniel Rodriguez, Ori- ana Riva, and Timothy Lillicrap. 2023.An- droidinthewild: A large-scale dataset for android device control. Advances in Neural Information Pro- cessing Systems, 36:59708–59728. Taeyun Roh, WonJune Jang, Junha Jung, and Jaewoo Kang. 2026a. CLAG: Adaptive memory organization via agent-driven clustering for small language model agents. In Findings of the Association for Computa- tional Linguistics: ACL 2026, pages 16707–16726, San Diego, California, United States. Association for Computational Linguistics. Taeyun Roh, Suhyeong Park, Dongyoung Lee, Won- june Jang, Junha Jung, Eunyeong Jo, and Jaewoo Kang. 2026b. Buzzy: Contrastive scoring to mitigate text-induced bias in multimodal multiple-choice qa. Preprint, arXiv:2603.28026. Yucheng Shi, Wenhao Yu, Zaitang Li, Yonglin Wang, Hongming Zhang, Ninghao Liu, Haitao Mi, and Dong Yu. 2025. Mobilegui-rl: Advancing mobile gui agent through reinforcement learning in online environment. arXiv preprint arXiv:2507.05720. Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024. Mobile-agent: Autonomous multi-modal mo- bile device agent with visual perception. arXiv preprint arXiv:2401.16158. Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. 2025. Mobile-agent-e: Self-evolving mo- bile assistant for complex tasks. arXiv preprint arXiv:2501.11733. Han Xiao, Guozhi Wang, Hao Wang, Shilong Liu, Yux- iang Chai, Yue Pan, Yufeng Zhou, Xiaoxin Chen, Yafei Wen, and Hongsheng Li. 2026. Ui-mem: Self- evolving experience memory for online reinforce- ment learning in mobile gui agents. arXiv preprint arXiv:2602.05832. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhou- jun Cheng, Dongchan Shin, Fangyu Lei, et al. 2024. Osworld: Benchmarking multimodal agents for open- ended tasks in real computer environments. Ad- vances in Neural Information Processing Systems, 37:52040–52094. Ran Xu, Kaixin Ma, Wenhao Yu, Hongming Zhang, Joyce C. Ho, Carl Yang, and Dong Yu. 2025a. Retrieval-augmented GUI agents with generative guidelines. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Process- ing, pages 17866–17875, Suzhou, China. Association for Computational Linguistics. Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. 2025b. A-mem: Agen- tic memory for llm agents. In Advances in Neural Information Processing Systems, volume 38, Main Conference, pages 17577–17604. Curran Associates, Inc. Yige Xu, Yongjie Wang, Zizhuo Wu, Kaisong Song, Jun Lin, and Zhiqi Shen. 2026. Do vision-language models truly perform vision reasoning? a rigor- ous study of the modality gap. arXiv preprint arXiv:2604.16256. Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tian- bao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. 2024. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454. Shijia Yang, Yunong Liu, Bohan Zhai, Ximeng Sun, Zicheng Liu, Emad Barsoum, Manling Li, and Chen- feng Xu. 2026. Captionqa: Is your caption as useful as the image itself? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 23741–23750. Woongyeong Yeo, Kangsan Kim, Jaehong Yoon, and Sung Ju Hwang. 2026. Worldmm: Dynamic multi- modal memory agent for long video reasoning. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 25599– 25609. Ziyun Zeng, Hang Hua, Bocheng Zou, Mu Cai, Rogerio Feris, and Jiebo Luo. 2026. Mementogui: Learning agentic multimodal memory control for long-horizon gui agents. arXiv preprint arXiv:2605.18652. Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2025. Appagent: Multimodal agents as smartphone users. In Proceedings of the 2025 CHI conference on human factors in computing systems, pages 1–20. 10 A Dataset and Baseline Details A.1 Dataset Details We evaluate SwitchMem on three mobile GUI benchmarks that cover complementary interaction settings: Android in the Wild (AITW), LearnGUI, and MobileMiniWoB++. Android in the Wild (AITW). AITW (Rawles et al., 2023) contains human-demonstrated Android interaction trajectories spanning diverse applica- tions and user goals. Its heterogeneous interface states and multi-step trajectories make it suitable for studying whether the preferred memory repre- sentation changes across interaction steps. We eval- uate next-action prediction using the benchmark- specific action scorer. LearnGUI. LearnGUI (Liu et al., 2026) eval- uates mobile GUI agents in demonstration aug- mented interaction settings, requiring agents to ex- ecute complex navigation tasks by conditioning on retrieved historical execution traces. Because agent performance hinges on effectively leverag- ing retrieved multimodal experience for dynamic step-wise decisions, it serves as an ideal testbed for evaluating adaptive memory routing. Memory is retrieved only from demonstration trajectories distinct from the query trajectory. Following the benchmark’s standardized protocol, we report ac- tion level execution accuracy across its diverse mo- bile interaction tasks. MobileMiniWoB++. MobileMini- WoB++ (Rawles et al., 2025) provides pro- grammatically verifiable mobile GUI tasks in an online, closed-loop execution environment. While AITW and LearnGUI evaluate agents against het- erogeneous real-world traces, MobileMiniWoB++ offers deterministic state transitions and immediate environment feedback for step-wise actions. This setup serves as a controlled testbed, allowing us to isolate and evaluate SwitchMem’s dynamic memory routing without confounding noise from real-world OS rendering anomalies or ambiguous task rewards. Following standard protocol, we report task completion success rate. Together, the three benchmarks cover human- demonstrated trajectories, demonstration-based knowledge transfer, and controlled executable in- teraction. For all benchmarks, Text and Image memory conditions use the same actor, current ob- servation, retrieved demonstration, and decoding configuration; only the representation of the re- trieved memory differs. A.2 Baseline Details General-purpose and GUI specialized baselines are evaluated from their released models or APIs. For a fair comparison, we standardize the backbone actor across all memory-augmented methods. Specifi- cally, all methods use the same frozen Qwen3.5- 9B actor, retrieval bank, and decoding protocol, differing only in how the retrieved experience is represented and provided to the actor. Fixed memory conditions.TEXT serializes the retrieved instruction, state description, and demon- strated action. IMAGE replaces the state description with its screenshot while retaining the same instruc- tion and action information. BOTH supplies both representations of the same retrieved step. RAG-GUI.RAG-GUI (Xu et al., 2025a) aug- ments the agent with external procedural knowl- edge retrieved from a corpus of GUI tutorials. Rather than appending raw retrieved documents, it filters and converts relevant content into concise, task- and progress-conditioned textual guidelines for the actor during action prediction. M3A.M3A (Rawles et al., 2025) enriches the current observation with structured interface infor- mation, including visible UI elements and recent interaction history. This structured context enables the actor to jointly reason about screen states and prior execution steps during action selection. AppAgent.APPAGENT (Zhang et al., 2025) con- structs an offline operation document by exploring target applications to detail interface elements, exe- cutable actions, and state transitions. At inference time, it retrieves documentation associated with the current screen’s UI elements and uses it to guide action generation. EchoTrail.ECHOTRAIL (Li et al., 2026b) builds reusable memory from collected GUI trajectories using self-exploration and critic-based filtering. Successful experiences are reformulated into com- pact textual traces highlighting interface states and transitions, enabling the retrieval and reuse of rele- vant interaction patterns for new tasks. AGMem.AGMEM (Choi et al., 2026) preserves action-relevant visual evidence from demonstra- tions by comparing consecutive frames to crop changed interaction regions. It provides these 11 cropped regions as visual memory, falling back to full demonstration frames when distinct regions cannot be reliably identified. To ensure fair comparison, all baselines are im- plemented as training-free, component-level mem- ory methods under the same retrieval bank, frozen actor, and decoding protocol as SWITCHMEM. Consequently, performance differences directly re- flect how each method formulates and delivers re- trieved experience to the agent. B SwitchMem Details B.1 SwitchMem Training and Evaluation Details For AITW and LearnGUI, we retrieve one memory step using the text-based retriever and represent the same retrieved ID as either Text or Image. At each training state, we sample seven actor actions per modality using temperature 0.7 and seeds 42–48, and score them with the benchmark specific action metric. Exact utility ties are excluded from pref- erence target construction. The routing threshold is selected by maximizing utility on the resulting training predictions and is frozen before test evalu- ation. The router is parameterized by a two-layer MLP (1024 → 256 → 1) with GELU activation and a0.1dropout rate, trained using AdamW with a learning rate of 0.001. AITW. We divide 20,000 steps from the official training split at the trajectory level into 10,019 preference-query steps and 9,981 memory steps, corresponding to 3,100 and 3,087 trajectories, re- spectively. The two subsets have no trajectory overlap. Each training query retrieves a demon- stration from the memory subset within the same AITW task category. For test evaluation, we use 5,000 steps from 759 held-out trajectories and al- low retrieval from the complete 20,000-step train- ing bank. LearnGUI. We use the 101 successful Online demonstration trajectories, comprising 1,500 in- teraction steps, as the source of both preference queries and candidate memories. Success is used to ensure that the memory bank contains validated experiences. We then construct preference supervi- sion by comparing the actor’s action quality under the Text and Image representations of the same re- trieved step. At evaluation, we follow LearnGUI’s offline one-shot setting and use the support demon- strations available as memory in that setting. The router and threshold remain fixed after Online train- ing, and no Offline-Test action, utility, or outcome is used for fitting or calibration. MobileMiniWoB++. MobileMiniWoB++ does not provide step level action utilities or an official training/validation split. We therefore transfer the LearnGUI router without updating its weights. The memory bank is constructed once using suite seed 1001 and contains 129 interaction steps from 50 successful trajectories. It is fixed before calibra- tion and evaluation. For threshold calibration, we use only the initial states of the 92 task templates instantiated with suite seed 43. Without observ- ing actions, rewards, or task success, we choose the cutoff whose Image-selection rate best matches the training-side routing rate on LearnGUI. The cutoff is then frozen and evaluated through full on- line rollouts on the same 92 templates instantiated with suite seed 42, yielding 63.043% task success (58/92). Suite seeds 1001, 43, and 42 share task templates but instantiate different task values, on-screen text, and target coordinates. Consequently, the memory bank, calibration set, and evaluation set contain no identical task instances. No observation from seed 42 is used to construct the memory bank, train the router, or select its threshold. Text memory is constructed without an addi- tional captioning model. For AITW, we serialize the dataset-provided UI text annotations together with the current Android activity. For LearnGUI, we use the provided element annotations, including page captions, element descriptions, and functional- ity labels. For MobileMiniWoB++, we determinis- tically serialize the Android accessibility hierarchy obtained through UIAutomator. Each textual GUI observation is paired with its task instruction and demonstrated action to form a Text memory entry. B.2 Modality-preference Probing Details For AITW and LearnGUI and each layerℓ ∈ 0, 8, 16, 23, 31, we compute the signed D-CoE shifth I i,ℓ − h T i,ℓ from the final prompt token im- mediately before action generation. Features are standardized, reduced to 64 PCA dimensions, and classified with a logistic probe. The label is re- tained only when at least five of the seven sampled generations agree on the preferred modality. The modality-preference signal becomes most pronounced in the later transformer layers. It reaches its highest AUROC at layer 31 on AITW 12 08162331 Transformer layer 0.5 0.6 0.7 0.8 AUROC Figure 8: layer-wise modality-preference probing on Learn-GUI. AUROC is computed by trajectory-grouped prediction on the 5-of-7 stable-preference subset. The dashed line denotes chance. and at layer 23 on LearnGUI. Rather than select- ing the best layer separately for each benchmark, SwitchMem consistently uses layer 31, the final transformer block. Its hidden state represents the actor’s fully integrated multimodal representation immediately before action generation. This choice provides a single benchmark-independent extrac- tion rule and avoids tuning the representation layer to each evaluation dataset, despite the higher layer- 23 probe score on LearnGUI. We do not report an analogous 5-of-7 probe for MobileMiniWoB++ because the benchmark does not provide the intermediate step level utilities re- quired to construct Text and Image preference tar- gets. We therefore evaluate MobileMiniWoB++ under a transfer setting using the router trained on LearnGUI, which precludes constructing the corre- sponding utility-based probing subset. B.3 Additional Routing Strategy Comparisons We repeat the routing comparisons on AITW using the same experimental protocol. Figure 9 shows that Utility, Binary, and Basic routing improve over the fixed Image policy, but remain below Switch- Mem. Binary is particularly competitive on AITW, whereas the advantage of SwitchMem is more pro- nounced on LearnGUI. Across both benchmarks, SwitchMem achieves the highest point estimates, supporting D-CoE-based target construction as a more consistently effective alternative to direct su- pervision. Figure 10 further shows that SwitchMem outper- forms ratio-matched random routing at every eval- AITW 48 49 50 51 52 Action score (%) 49.300 Image 50.740 Utility 51.260 Binary 50.100 Basic routing 51.420 SwitchMem Figure 9: Comparison with simpler routing baselines on AITW. Binary is closely competitive with SwitchMem. The y-axis is truncated. 0255075100 Image memory usage (%) 49 50 51 52 Action score (%) 36% Image, 51.42 AITW SwitchMem Ratio-matched random Deployed threshold 95% paired CI Figure 10: SwitchMem versus random routing on AITW. Random routing controls for aggregate Image usage but does not condition its selections on the interaction state. uated non-trivial modality budget. The same pat- tern on AITW and LearnGUI supports step-specific modality assignment, rather than the global Text– Image ratio, as a source of the adaptive-routing benefit. B.4 Efficiency Details Table 4 reports mean wall-clock latency on the same RTX PRO 6000 GPU. Requests are exe- cuted sequentially with greedy decoding, think- ing disabled, and the same 128-token output cap. The SWITCHMEM number includes its current- state hidden-state probe, the small pairwise router, and one selected-memory action call. Although SWITCHMEM introduces an additional routing 13 MethodAITWLearnGUIMobileMini No memory1.2971.2720.906 Text1.1541.3770.889 Image1.2791.6581.558 Both1.3091.6801.441 EchoTrail1.2051.3641.189 AGMem1.3721.8691.388 M3A2.1562.2513.952 AppAgent3.2982.9242.682 RAG-GUI8.81511.1182.862 SwitchMem1.4071.8481.862 Table 4: Mean inference latency in seconds per interac- tion step. Method-specific inference stages are included, while retrieval and input preprocessing are excluded. stage, its runtime overhead remains modest. Com- pared with the Image baseline, SWITCHMEM adds only 0.128 s, 0.190 s, and 0.304 s per step on AITW, LearnGUI, and MobileMiniWoB++, respec- tively, while achieving consistently better action performance. In contrast, more elaborate memory- augmented baselines such as M3A, AppAgent, and RAG-GUI introduce additional inference-time processing and incur substantially higher latency, while still underperforming SWITCHMEM. For example, RAG-GUI requires 8.815 s and 11.118 s per step on AITW and LearnGUI, compared with 1.407 s and 1.848 s for SWITCHMEM. Overall, these results show that adaptive memory routing provides consistent performance gains with only a small additional inference cost. B.5 Text and Image Memory Examples Figure 11 illustrates the text and image memory representations used by SwitchMem. Both vari- ants are constructed from the same step retrieved from a successfully completed trajectory. They contain the same past task, successful action, and natural-language action description. The only dif- ference is the representation of the past observation: text memory provides a textual description of the GUI state, whereas image memory provides the corresponding screenshot. This paired construction isolates the effect of the observation modality while preserving the demonstrated task and action. B.6 Statistical significance. We assess whether the improvement over the strongest fixed memory baseline is statistically re- liable using paired tests. For AITW and Learn- GUI, we resample complete task trajectories rather than individual steps, thereby preserving within- trajectory dependence, and construct 95% con- fidence intervals using 20,000 paired cluster- bootstrap replicates. We additionally perform a two-sided cluster-level sign-flip randomization test. For MobileMiniWoB++, where each task produces a binary success outcome, we use a paired task bootstrap and an exact McNemar test. The improvements are statistically significant on AITW (p < .001) and LearnGUI (p = .016), and remain significant after Holm correction across the three benchmarks. On MobileMiniWoB++, Switch- Mem exhibits a positive gain, but the difference is not statistically significant (p = .688), reflect- ing the limited sample size of 92 evaluation tasks. Moreover, the router is transferred directly from LearnGUI without any MobileMiniWoB++ prefer- ence supervision or target-domain training. There- fore, this result should be interpreted as evidence of cross-benchmark transfer rather than in-domain- optimized performance. 14 (a) Text-memory context Demonstration Memory Past task: Open Booking.com. Look up a one-way flight from Melbourne to Auckland. Depart on June 22. 3 adults in business class. Stopovers are allowed. Sort by cheapest price. Past GUI observation: This is a Booking.com app page where users can search for stays, flights, car rentals, and taxis. The page also shows loyalty program information and previous search results. | Messages | Notifications | Stays | Flights | Car rental | Taxi | Toronto | Staying from Sun, Sep 01 until Mon, Sep 30|1 room, 2 adults, 0 children | Search | Jaeger, you’re at Genius Level 1 in our loyalty program | 10% discounts on stays | Enjoy discounts at participating properties worldwide | 1–30 Sep, 2 adults | Saved | Bookings | Profile | Accommodation search box | Travel more, spend less | Continue your search | Miami | 10–20 Aug, 2 adults. Past action: "type":"tap", "position":[0.438194,0.121795] Past action description: On Booking.com Home Screen, click “Flights” button, to search for flights. (b) Image-memory context Demonstration Memory Past task: Open Booking.com. Look up a one-way flight from Melbourne to Auckland. Depart on June 22. 3 adults in business class. Stopovers are allowed. Sort by cheapest price. Past GUI observation: Past action: "type":"tap", "position":[0.438194,0.121795] Past action description: On Booking.com Home Screen, click “Flights” button, to search for flights. Figure 11: Paired actor prompts used by SwitchMem. Both variants contain the same retrieved step from a successfully completed trajectory and expose its demonstrated action. Text memory renders the past GUI state as a textual observation, whereas Image memory replaces only that observation with the corresponding full screenshot. The image is cropped only for visualization; the actor receives the full screenshot. The current task, current screenshot, action schema, actor, and decoding configuration are held fixed. For readability, the common system prompt and current screenshot are collapsed in the figure. DatasetBest fixed SwitchMem∆95% CI AITW49.30051.420 +2.120[1.270, 2.945] LearnGUI37.44138.131 +0.690[0.149, 1.263] MobileMiniWoB++60.87063.043 +2.174 [-3.261, 7.609] Table 5: Paired performance differences between SwitchMem and the strongest fixed memory baseline. Confidence intervals are reported in percentage points. 15 B.7 Case Studies of When Each Memory Modality Is Useful We illustrate state-dependent memory selection us- ing two states from different AITW tasks. The first is a Sparse initial state requiring a Spatial action, whereas the second is a Dense post-initial state requiring a Function action. These examples con- trast the three contextual factors examined in our modality utility analysis. Within each example, Text and Image memory represent the same retrieved demonstration, and the current instruction and screenshot are identical across memory conditions. Only the representation of the past GUI observation differs. Sparse initial state: Text memory is more use- ful. At step 0 of “Go to the official website of bestbuy,” the current screen shows a Costco sign-in page (Figure 12). Its annotated element density is 177.25, below AITW’s lower-quartile cutoff of 310.00, placing it in the Sparse group. The gold action is a tap on the browser address bar, classified as Spatial. This state is therefore Sparse, Spatial, and Initial. The retrieved demonstration has the instruction “open google chrome and search for bestbuy.com” and also depicts a Costco sign-in page (Figure 12). Its textual observation includes the browser URL and labels such as “Sign In,” “Email Address,” and “Password,” together with the demonstrated tap ac- tion. The Image condition instead provides the screenshot of that past page, while retaining the same task and action information. Under Text mem- ory, the actor correctly taps the current address bar. Under Image memory, it outputs a typing action forbestbuy.comwithout first selecting the address bar. The action scores are 1 and 0, respectively, and SwitchMem selects Text. Dense post-initial state: Image memory is more useful. At step 2 of “Search for dell alienware on walmart.com,” the current screen displays the requested search results, including a query field, fil- ter controls, product information, and pricing (Fig- ure 13). Its annotated element density is 713.27, above AITW’s upper-quartile cutoff of 567.73, placing it in the Dense group. The gold action is task_complete, categorized as Function because it signals task completion rather than a spatial in- teraction. This state is therefore Dense, Function, and Post-initial. The retrieved demonstration concerns “Search dell alienware on amazon” (Figure 13). Its tex- tual observation lists the query, search suggestions, and text-selection controls, while its screenshot depicts the query-entry state with the search text se- lected. Under Text memory, the actor unnecessarily presses Enter. Under Image memory, it correctly outputstask_complete. The action scores are 0 and 1, respectively, and SwitchMem selects Image. Together, these examples illustrate contextual variation in memory usefulness across interaction states. SwitchMem follows the higher-scoring modality in both cases, although these illustrative examples do not isolate the causal contribution of each contextual factor. 16 Current task: Go to the official website of bestbuy. (a) Current state: Step 0 Sparse / Spatial / Initial Element density: 177.25 SwitchMem: Text Score (T/I): 1/0 (b) Text memory Demonstration Memory Past task: open google chrome and search for bestbuy.com . Past GUI observation: Android activity: com.android.chrome/ com.google.android. apps.chrome.Main Visible UI text: signin.costco.com/ e0714d4-784d-46d6-a278 | COSTCO | EWHOLESALE | Sign | In | -Email Address | Password | Eorgot.Password? | Keep me signed in | Check this | box only when | private device. | on | a | New to Costco? | Create Account | Terms and Conditions | Privacy Rights. Past action (normalized): "type":"tap","position": [0.5313,0.0408] (c) Image memory Demonstration Memory Past task: open google chrome and search for bestbuy.com . Past GUI observation: Past action (normalized): "type":"tap","position": [0.5313,0.0408] Figure 12: Text-favoring AITW case. (a) A Sparse initial state requiring a Spatial action. (b,c) Text and Image representations of the same retrieved demonstration, which also shows a Costco sign-in page. Text memory yields a correct tap on the current browser address bar, whereas Image memory yields a typing action before selecting the bar. SwitchMem selects Text. T/I denotes Text/Image action-matching scores; memory coordinates are normalized and rounded. Current task: Search for dell alienware on walmart.com. (a) Current state: Step 2 Dense / Function / Post-initial Element density: 713.27 SwitchMem: Image Score (T/I): 0/1 (b) Text memory Demonstration Memory Past task: Search dell alienware on amazon. Past GUI observation: Android activity: com.android.chrome/ com.google.android. apps.chrome.Main Visible UI text: amazon.com | = | amazon | Sign|in|>|G|Define|Select|all| Copy | Share | Web search | X | dell alienware | bc | deul alienware aw3423dw | dell alienware aw3423dwf | dell alienware x14 | dell alienware monitor | dell alienware x17 | r2 | essentials under $50 | TDAMEH | holiday gifts | Great | Be | Oculus | Women’s Fashion | Pic | here | are | alienware. Past action (normalized): "type":"tap","position": [0.1878,0.2123] (c) Image memory Demonstration Memory Past task: Search dell alienware on amazon. Past GUI observation: Past action (normalized): "type":"tap","position": [0.1878,0.2123] Figure 13: Image-favoring AITW case. (a) A Dense post-initial state requiring a Function action: the requested Walmart search results are already displayed. (b,c) Text and Image representations of the same retrieved Amazon demonstration at a query-entry state. Image memory yields correct task completion, whereas Text memory yields an unnecessary Enter action. SwitchMem selects Image. T/I denotes Text/Image action-matching scores; memory coordinates are normalized and rounded. 17