Paper deep dive
TAPDreamer: Transferable Adversarial Patches for World Action Models
Xuanyu Lu, Fengqing Jiang, Kaiyuan Zheng, Yichen Feng, Yaorui Ding, Yuetai Li, Zhen Xiang, Bhaskar Ramasubramanian, Basel Alomair, Luyao Niu, Radha Poovendran
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/7/2026, 8:47:14 AM
Summary
The paper introduces TAPDreamer, a transferable adversarial attack against World Action Models (WAMs) that uses a fixed local patch optimized via a public visual encoder. The attack exploits the interaction between patch-induced changes in attention weights and value vectors to broadcast representation shifts beyond the patch footprint, causing significant degradation in task success rates across various robotic benchmarks (LIBERO, RoboTwin) and model architectures (FastWAM, DreamWAM, Motus) without requiring target-policy queries.
Entities (10)
Relation Signals (9)
TAPDreamer โ degradesperformanceof โ FastWAM
confidence 95% ยท reduces FastWAM's success rate from 97.7% to 0.0%
TAPDreamer โ degradesperformanceof โ DreamWAM
confidence 95% ยท The same patches reduce success to 1.45% and 1.00% on two DreamWAM configurations
TAPDreamer โ degradesperformanceof โ Motus
confidence 95% ยท and to 10.60% on Motus.
TAPDreamer โ evaluatedon โ LIBERO
confidence 95% ยท across 40 LIBERO tasks
TAPDreamer โ evaluatedon โ RoboTwin
confidence 95% ยท across 50 RoboTwin tasks
TAPDreamer โ targets โ World Action Models
confidence 95% ยท We propose an attack, TAPDreamer, against world action models
Attention Weights โ interactswith โ Value Vectors
confidence 90% ยท interactions between patch-induced changes in attention weights and value vectors broadcast a nearly identical representation shift
TAPDreamer โ uses โ Visual Encoder
confidence 90% ยท TAPDreamer... uses a public encoder alone to construct a fixed local perturbation
TAPDreamer โ โ
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:World models learn to predict how their environment will evolve, making them an important foundation for general-purpose robotic control. Yet world action models depend on camera inputs whose manipulation can corrupt the visual representations used across tasks and action policies. Existing attacks on these models optimize against the victim's actions or predicted futures and therefore require access to target-model outputs. In this paper, we propose an attack, TAPDreamer, against world action models that instead uses a public encoder alone to construct a fixed local perturbation that transfers across tasks and action architectures. TAPDreamer requires no target-policy queries. Our key insight is that interactions between patch-induced changes in attention weights and value vectors broadcast a nearly identical representation shift far beyond the patch footprint, and this shift remains stable across task observations. Guided by this insight, TAPDreamer uses six frames from one source task to maximize the global L1 distance between clean and patched encoder representations. In closed-loop evaluation, one frozen patch per benchmark, covering about 6.5% of the input, reduces FastWAM's success rate from 97.7% to 0.0% across 40 LIBERO tasks and from 90.86% to 0.0% across 50 RoboTwin tasks; matched random patches retain 81.5% and 79.2% success. The same patches reduce success to 1.45% and 1.00% on two DreamWAM configurations and to 10.60% on Motus. These results show that protecting downstream action generation alone is insufficient: defenses for world action models must also secure shared visual encoders against persistent local perturbations.
Tags
Links
- Source: https://arxiv.org/abs/2610.06814v2
- Canonical: https://arxiv.org/abs/2610.06814v2
Trouble viewing inline? Open PDF directly โ
Full Text
56,287 characters extracted from source content.
Expand or collapse full text
TAPDreamer: Transferable Adversarial Patches for World Action Models Xuanyu Lu 1โ Fengqing Jiang 1โ โก Kaiyuan Zheng 1โ Yichen Feng 1 Yaorui Ding 1 Yuetai Li 1 Zhen Xiang 2 Bhaskar Ramasubramanian 1,3 Basel Alomair 1,4,5 Luyao Niu 1 Radha Poovendran 1 1 University of Washington 2 University of Georgia 3 Western Washington University 4 King Abdulaziz City for Science and Technology 5 HUMAIN โ Equal Contribution โก Project Lead tapdreamer.github.io Abstract World models learn to predict how their environment will evolve, making them an important foundation for general-purpose robotic control. Yet World Action Models (WAMs) rely on camera inputs, and manipulating those inputs can corrupt the visual representations that underpin their tasks and action policies. Existing attacks on these models optimize against actions or predicted futures and require access to victim outputs or generation modules. We proposeTAPDreamer, a transferable fixed local patch whose pixel optimization uses only a public visual encoder and requires no target-policy queries. Its design addresses two requirements: a local modification should affect representations beyond its footprint, and the resulting disturbance should persist across observations. We optimize a fixed local patch on six frames from one source task by maximizing the mean L1 distance between clean and patched encoder outputs over all latent positions and channels. Mechanism analysis reveals that interactions between patch-induced changes in attention weights and value vectors broadcast a nearly identical representation shift beyond the patch footprint, while a shared perturbation response persists across task observations despite differences in clean latents. In closed-loop evaluation, one frozen patch per benchmark, covering about 6.5% of the input, reduces FastWAMโs success rate from 97.65% to 0.00% across 40 LIBERO tasks and from 90.86% to 0.00% across 50 RoboTwin tasks; matched random patches retain 81.45% and 79.20% success. The same patches reduce success to 1.45% and 1.00% on two DreamWAM configurations and to 10.60% on Motus. Behavioral analysis connects these representation shifts to grasping failures and separates rapid command recovery after patch removal from persistent physical consequences. These results show that protecting downstream action generation alone is insufficient: WAM defenses must also secure shared visual encoders against persistent local perturbations. 1. Introduction World Action Models (WAMs) are emerging as a powerful foundation for general-purpose robotic control. Rather than predicting actions from the current observation alone, a WAM learns how the visual world evolves together with the robotโs actions. This world-modeling objective captures objects, spatial relationships, and motion, and can support action generation either through explicit future imagination (Du et al., 2023; Zhou et al., 2024) or through representations learned during video prediction (Yuan et al., 2026b). By unifying visual prediction and control, WAMs promise policies that can reason about the consequences of their actions and replan as the environment changes (Zhu et al., 2025; Ye et al., 2026). This coupling, however, also creates a security risk. Current WAM pipelines implicitly trust the visual representation on which world prediction and action generation are built. This assumption is fragile. Cameras arXiv:2610.06814v2 [cs.CV] 6 Oct 2026 TAPDreamer: Transferable Adversarial Patches for World Action Models One source task Public encoder E Encoder only No target-policy queries TAPDreamer patch Drawer opening FastWAM InputOutcome Open Closed Pot placement Motus InputOutcome On stove Off stove Bowl placement DreamWAM InputOutcome On plate Off plate Clean + TAPDreamer Same fixed patch One source task Public encoder E Encoder only No target-policy queries TAPDreamer patch Drawer opening FastWAM InputOutcome Open Closed Pot placement Motus InputOutcome On stove Off stove Bowl placement DreamWAM InputOutcome On plate Off plate Clean + TAPDreamer Same fixed patch (a) Optimize once(b) Reuse the same patch Figure 1:TAPDreamerconstructs a patch using a public visual encoder and observations from one source task, then freezes it for deployment across tasks and WAMs within each benchmark. Examples compare clean and patched execution on drawer opening with FastWAM, pot placement with Motus, and bowl placement with DreamWAM. expose a direct, externally accessible input channel. Moreover, visual encoders are often reused across tasks and models, making the exposed channel transferable among tasks. A perturbation that corrupts this shared interface can therefore bypass model-specific action heads. If the channel remains exposed, a perturbation can be injected consistently. In this paper, we investigate whether an attacker can exploit this vulnerability and launch attacks against WAMs that transfer to unseen tasks and models. We proposeTAPDreamer(illustrated in Figure 1), a transferable attack that applies a patch on visual inputs to WAMs.TAPDreamermaximizes the displacement of the final latent produced by visual encoder. The search uses a few demonstration frames from a single source task. Once the patch is found, the patch pixels, area, and position are frozen and the same artifact is deployed unchanged throughout each episode and across target tasks and models. Throughout the attack,TAPDreamerdoes not require action labels, task rewards, simulator, victim-model weights, or policy outputs, making TAPDreamer simple and inexpensive to initiate. We evaluateTAPDreameron 40 LIBERO tasks and all 50 RoboTwin tasks.TAPDreamerconsistently outper- forms attack baselines. On FastWAM, a single frozen patch covering approximately 6.5% of the input reduces task success from97.65%to0.00%on LIBERO and from90.86%to0.00%on RoboTwin. Random patches retain 81.45%and79.20%success, respectively. The same frozen patches also transfer across WAM architectures without retraining or target-model queries: success falls from84.55%and86.10%with matched random patches to1.45%and1.00%on DreamWAM-uncond and DreamWAM-joint, and from82.40%to10.60%on Motus. We further trace how a fixed patch causes failure during closed-loop control. Our results show that changing a small set of pixels distorts visual features across the scene, including those used to select actions, and thus leads to reduced task success rate. 2 TAPDreamer: Transferable Adversarial Patches for World Action Models 2. World Action Model, Threat Model, and TAPDreamer 2.1. World Action Model Letํ ํก denote the robot and environment state at stepํก,ํ ํก the camera observations,ํ ํก the robotโs proprioceptive state,โthe task instruction, andโ ํก the available history. For a world modelํ, image preprocessingํบ ํ produces the visual inputํฅ ํก = ํบ ํ (ํ ํก ), and the visual encoder producesํ ํก = ํธ(ํฅ ํก ). The downstream model constructs a contextํถ ํก = ํน ํ (ํ ํก ,โ,ํ ํก ,โ ํก ),whereํdenotes the learnable parameters of the corresponding module. For a WAM that predicts future visual latents before generating actions, we denote visual predictions and generated action chunks as ห ํ ํก+1:ํก+ํพ ํฃ โผ ํ ํ (ยท| ํถ ํก ), ํ ํก:ํก+ํพ ํ โ1 โผ ํด ํ (ยท| ํถ ํก , ห ํ ํก+1:ํก+ํพ ํฃ ),(1) whereํพ ํฃ andํพ ํ are the visual prediction and action horizons. Here,ํ ํ andํด ํ denote conditional distri- butions over future latents and action chunks. During closed-loop execution, the robot executes the firstํพ ํ actions of each chunk and receives new observations: ํ ํก+ํพ ํ =โฑ (ํ ํก ,ํ ํก:ํก+ํพ ํ โ1 ), ํ ํก+ํพ ํ = ํ(ํ ํก+ํพ ํ ),(2) where1 โค ํพ ํ โค ํพ ํ ,โฑis the environment transition, andํis the observation function. The WAM then encodes the new observations and generates the next chunk. A visual perturbation can therefore affect both the current action and subsequent observations through execution-induced state changes. 2.2. Threat Model Attackerโs objective. We consider an attacker who aims to reduce the task success rate of a WAM-controlled robot by modifying a small region of its visual input. This is an untargeted, test-time attack. Attackerโs knowledge and access. The attacker has white-box access to a publicly released visual encoder, including its weights and gradients, and a small set of public demonstration images from one source task. We assume that the attacker knows the relevant visual preprocessing, including image normalization and camera-view composition. Patch optimization requires only the WAMโs encoder and a small set of demonstration frames from a single source task. The attacker does not require access to the downstream world model or action decoder or their gradients, action labels, or task rewards for pixel optimization. Attackerโs capabilities. The attacker can replace pixels within a fixed local region of the visual input before encoding. Letํbe a fixed binary mask and ํ the patch. The attacked observation is ํ ํ (ํฅ ํก ) = (1โ ํ )โ ํฅ ํก + ํ โ ํ,(3) whereโrepresents elementwise multiplication. The attacker chooses the patch content. At deployment, the patch content, position, and area remain unchanged. The same patch is reused across target tasks and models within each benchmark. The attack does not manipulate pixels outside the mask, language instructions, proprioceptive inputs, or policy parameters. The attacker cannot directly set the robotโs actions or the environment state. 3 TAPDreamer: Transferable Adversarial Patches for World Action Models Conceptual illustration 6 observations / 1 source task External + wrist Apply patch Patched observation External + wrist Public Wan VAE encoder Frozen weights Latent representation Encoder-space objective p Trainable patch Backpropagate through frozen encoder Update patch only Afteer optimization Final p* Freeze & reuse No target-policy queries during construction Task instruction Patched observation รตโ Same p* per step Target WAM Unseen task / architecture Actions aโ Environment Next observation o โ โโ Apply same p* Fixed across tasks and WAMs 1 Spatial broadcasting Patched input Local pixel change Bottleneck attention p Spatial mixing Representation change Beyond the patch footprint Local pixel change โ widespread feature shifte 2 Consistent shifts across observations Shared shift direction Obs. 1 hโ h โ Obs. 2 hโ h โ Obs. 3 hโ h โ Common component g Residual variation ฮh = h โ h = g + rแตข Forward / executionBackward / backpropagationFrozen model Conceptual illustration 6 observations / 1 source task External + wrist Apply patch Patched observation External + wrist Public Wan VAE encoder Frozen weights Latent representation Encoder-space objective p Trainable patch Backpropagate through frozen encoder Update patch only Afteer optimization Final p* Freeze & reuse No target-policy queries during construction Task instruction Patched observation รตโ Same p* per step Target WAM Unseen task / architecture Actions aโ Environment Next observation o โ โโ Apply same p* Fixed across tasks and WAMs 1 Spatial broadcasting Patched input Local pixel change Bottleneck attention p Spatial mixing Representation change Beyond the patch footprint Local pixel change โ widespread feature shifte 2 Consistent shifts across observations Shared shift direction Obs. 1 hโ h โ Obs. 2 hโ h โ Obs. 3 hโ h โ Common component g Residual variation ฮh = h โ h = g + rแตข Forward / executionBackward / backpropagationFrozen model (a) Offline patch construction(b) Fixed-patch deployment Figure 2: Overview of TAPDreamer. (a) Pixel updates maximize the latent displacement through a frozen public visual encoder. (b) The resulting patch is frozen and reused at each step across tasks and models within the same benchmark. 2.3. Attack Design and Implementation Design rationale. The attack exploits the reuse of public visual encoders across WAMs with different downstream control architec- tures. The encoder output therefore provides an attack objective independent of a particular action interface. For local pixel changes to affect broader scene representations, they must exploit propagation pathways within the encoder. We maximize the difference between the encoderโs clean and patched representations over the entire latent output, including all spatial positions and channels. The same patch is optimized over multiple source observations. Patch construction. TAPDreamersamplesํฝframesํฅ ํ ํฝ ํ=1 from different demonstrations of a single source task, caches their clean latents, and optimizes max ํโํซ local 1 ํฝํ ํ ํฝ โ๏ธ ํ=1 โํธ(ํ ํ (ํฅ ํ ))โ ํธ(ํฅ ํ )โ 1 .(4) Algorithm 1 TAPDreamer Patch Optimization Require:Frozen encoderํธ, source imagesํฅ ํ ํฝ ํ=1 , patch maskํ, pixel range[ํ,ํ], iterationsํพ, learning rate ํ Ensure: Optimized patch ํ โ 1: ํ 0 ํ โ StopGradient(ํธ(ํฅ ํ )), โํ 2: ํ 0 โผํฐ (ํ,ํ); initialize Adam state ํ 0 3: for ํ โ 0 to ํพโ 1 do 4: ํ ํ ํ (ํฅ ํ ) = (1โ ํ )โ ํฅ ํ + ํ โ ํ ํ , โํโ Eq. (3) 5: โ ํ โ 1 ํฝํ ํ ํฝ โ๏ธ ํ=1 โฆ โฆ ํธ(ํ ํ ํ (ํฅ ํ ))โ ํ 0 ํ โฆ โฆ 1 6:(ํ ํ+1 ,ํ ํ+1 )โ Adam(ํ ํ ,โโ ํ ํ โ ํ ,ํ ํ ;ํ) 7: end for 8: return ํ ํพ 4 TAPDreamer: Transferable Adversarial Patches for World Action Models Here,ํ ํ is the total number of scalar elements in the final latent, andํซ local contains patches within the fixed mask that satisfy the valid pixel range. Algorithm 1 gives the construction procedure. Figure 2 illustrates the overall pipeline. 3. Experiments 3.1. Experimental Setup Datasets and models. We conduct closed-loop evaluation on LIBERO and RoboTwin (Liu et al., 2023; Chen et al., 2025). LIBERO contains four task categories (Spatial, Object, Goal, and Long-horizon) with 10 tasks each. RoboTwin contains 50 tasks. FastWAM is the default evaluation model. DreamWAM-uncond (DW-U), DreamWAM-joint (DW- J) (Yuan et al., 2026a), and Motus (Bi et al., 2026) are used for transfer evaluation within the same benchmark. Each model configuration retains its own action generation process and preprocessing. Attack setup. We run TAPDreamer using patches optimized on source tasks; cross-model transfer occurs within each bench- mark. Patches are injected before encoding. On LIBERO, the external and wrist views form a224ร 448canvas; the main patch occupies an81ร 81region in the upper-left corner of the external view, covering 6.54% of the input. The corresponding coverage on RoboTwin is 6.59%. Patches remain visible throughout each trial in standard attack evaluations. Images used for optimization come from public LeRobot demonstrations. We filter episodes by source-task identifier, then select one frame from each of six episodes at temporal indices 5, 20, 40, 60, 15, and 35, respectively. When generating the patch,TAPDreamersetsํฝ = 6with 500 Adam gradient-ascent updates on patch pixels, using a learning rate of 0.02. Patch pixels are initialized uniformly within[โ1, 1]. After each update, they are projected back to this range, with support enforced by the fixed mask. Baselines. We compareTAPDreamerwith baselines to show its effectiveness. We evaluate the policy on unmodified observations to establish the clean baseline. For random patch, we replace the same image region with a random patch matched to the adversarial patch in position and area, to measure the effects of ordinary local occlusion and pixel replacement. Random patches are generated following a uniform distribution within admissible range of pixel values. Extra ablation of transferred VLA attack is deferred to Appendix A.2. Metrics. We evaluate attack effectiveness using closed-loop task success rate (SR), defined as SR = ํ success ํ trials ร 100%,(5) whereํ success is the number of successful trials andํ trials is the total number of trials. A trial is successful if the robot completes the task according to the benchmarkโs success criterion. Complete task sets contain both the source task and held-out tasks; cross-task transfer is evaluated on held-out tasks after freezing the patch. 5 TAPDreamer: Transferable Adversarial Patches for World Action Models Table 1: Closed-loop FastWAM success rates (percent) on complete task sets. LIBERO and RoboTwin each use one fixed patch, optimized onSpatial t6andpick diverse bottles, respectively. Small subscripts report the signed success-rate difference from Clean, in percentage points. LIBERO Overall averages the four suites. Setup LIBERO RoboTwin SpatialObjectGoalLong-hor.Overall Clean98.00 ยฑ0.00 99.60 ยฑ0.00 97.60 ยฑ0.00 95.40 ยฑ0.00 97.65 ยฑ0.00 90.86 ยฑ0.00 Random85.60 โ12.40 78.00 โ21.60 90.00 โ7.60 72.20 โ23.20 81.45 โ16.20 79.20 โ11.66 TAPDreamer 0.00 โ98.00 0.00 โ99.60 0.00 โ97.60 0.00 โ95.40 0.00 โ97.65 0.00 โ90.86 3.2. Experimental Results TAPDreamer reduces the success rate to zero on both benchmarks. We evaluate FastWAM on all four LIBERO suites with 10 tasks each (4ร 10tasks), using 50 trials per task, and on all 50 RoboTwin tasks, using 100 trials per task. On LIBERO, FastWAM achieves 97.65% success with clean inputs, whereas the frozenTAPDreamerpatch reduces performance to 0.00%. The same pattern holds on RoboTwin, where success decreases from 90.86% to 0.00%. Table 1 shows that success is 0.00% in all four LIBERO suites, covering different task configurations. TAPDreamer causes a much larger performance degradation than a matched random patch. On LIBERO, the random patch retains an 81.45% success rate, compared with 0.00% forTAPDreamer. On RoboTwin, the corresponding success rates are 79.20% and 0.00%, respectively. Thus, the performance degradation caused byTAPDreamercannot be attributed solely to obscuring the same image region; it depends on the optimized content of the TAPDreamer patch. Table 2: Frozen-patch transfer across models within each benchmark. LIBERO results use the first optimization initialization. Success rates are percentages; small subscripts report signed differences from Clean for the same model and suite, in percentage points. DW-U and DW-J denote DreamWAM-uncond and DreamWAM-joint. LIBERO Overall averages the four suites. VictimSuiteCleanRandom TAPDreamer LIBERO DW-USpatial96.20 ยฑ0.00 82.20 โ14.00 0.00 โ96.20 DW-UObject99.60 ยฑ0.00 96.00 โ3.60 0.00 โ99.60 DW-UGoal98.00 ยฑ0.00 90.00 โ8.00 5.80 โ92.20 DW-ULong-horizon96.20 ยฑ0.00 70.00 โ26.20 0.00 โ96.20 DW-UOverall97.50 ยฑ0.00 84.55 โ12.95 1.45 โ96.05 DW-JSpatial99.40 ยฑ0.00 84.20 โ15.20 0.00 โ99.40 DW-JObject100.00 ยฑ0.00 99.20 โ0.80 0.00 โ100.00 DW-JGoal98.00 ยฑ0.00 88.00 โ10.00 4.00 โ94.00 DW-JLong-horizon93.40 ยฑ0.00 73.00 โ20.40 0.00 โ93.40 DW-JOverall97.70 ยฑ0.00 86.10 โ11.60 1.00 โ96.70 RoboTwin FastWAM Overall90.86 ยฑ0.00 79.20 โ11.66 0.00 โ90.86 MotusOverall86.62 ยฑ0.00 82.40 โ4.22 10.60 โ76.02 6 TAPDreamer: Transferable Adversarial Patches for World Action Models Mechanism in the encoderAcross observations Different observations Early encoding Bottleneck attention Remaining layers Final latent Shared shift direction 1 2 3 Same optimized patch p* U Local change Residual U VAV Attention map A H Distributed change Z ฮZ = Zแต โ Z Spatial broadcasting Reference features Patch-induced change Patch footprint Obs. 1 Zโ ฮZโ Obs. 2 Zโ ฮZโ Obs. 3 Zโ ฮZโ Source templateResidual ฮH ํช = 1g โค + R H ฮZ i = ฮZ s + e i W O Z p 1 Z p 2 Z p 3 Figure 3: Mechanism overview of TAPDreamer. Alocal patch-induced change spreads through bottleneck attention and propagates to the final latent. The changes across observations share a source-template component, with observation-specific residuals. TAPDreamer transfers across downstream control architectures. Keeping the patch fixed and changing the victim model reduces average success to 1.45% on DreamWAM- uncond, 1.00% on DreamWAM-joint, and 10.60% on Motus. The corresponding matched random patches retain 84.55%, 86.10%, and 82.40% success. Table 2 reports results by benchmark, model, and suite. Transfer uses no target-model feedback for reoptimization. DreamWAM-uncond does not read explicitly generated future frames to predict actions, yet it is still strongly affected. In these models, explicit test-time future-video generation is therefore not necessary for the attack. We defer more experimental results along with ablation studies about patch variations to Appendix A. 4. Mechanism and Behavioral Analysis This section investigates how a local pixel change byTAPDreamerresults in deviations in action execution. Figure 3 summarizes impacts on latents due to TAPDreamer. 4.1. A local patch becomes a distributed latent shift Attention spreads the change introduced by TAPDreamer. We compare clean and patched activations throughout the VAE encoder. Its bottleneck contains spatial self-attention on a14ร 28grid. We call the projection of the patch region onto a feature grid the patch footprint. Table 3 reports the fraction of squared magnitude of representation change outside this footprint. For TAPDreamer, it rises from less than 1% at the attention input to about 71% at its output and 79% in the final latent. A random patch of the same size leaves changes largely within the footprint. The attention layer helps the optimized local change reach distant positions. Attention weights and values carry the change. To identify what carries this change, write the clean attention output after residual addition asํป = ํ +ํดํ ํ ํ , whereํis the residual input,ํดthe attention weights,ํthe value vectors, andํ ํ the output projection. Rows index spatial positions; for multiple heads,ํดํdenotes their concatenated outputs. Letํ ํ denote the 7 TAPDreamer: Transferable Adversarial Patches for World Action Models Table 3: Fraction of squared representation change outside the projected patch footprint: โโํ ํ โ 2 ํน /โโํโ 2 ํน , whereโํ = ํ ํ โ ํ,ํ โํ,ํป,ํdenotes attention input, attention output, and final latent, respectively, and ํ denotes positions outside the footprint at the corresponding layer. Values are medians over 48 states. Attention input Attention output Final latent Random0.0070.0080.013 TAPDreamer0.0070.7080.794 0 8 16 โฮ z โ 2 Full patchClean weightsClean values 0 75 150 L2 norm outside footprint 147.8 8.7 6.8 Random patch: 3.8 (a) Random patch (b) TAPDreamer patch(c) Attention restoration Figure 4: Spatial spread and attention restoration. (a,b) Per-position final-latent L2 changes with a shared color scale; dashed boxes mark the patch footprint. (c) Mean L2 change outside the footprint over 48 states, with 95% confidence intervals. The dotted line denotes the random patch. patched quantity and โํ = ํ ํ โ ํ, where ํ โํป,ํ,ํ,ํด. We then have โํป = โํ + (ํดโํ + โํดํ + โํดโํ )ํ ํ .(6) The terms capture changed values, changed attention weights, and their joint change, respectively. We observe that the joint change termโํดโํis the largest measured component forTAPDreamerbut is small for a random patch. This points to coordinated changes in weights and values, rather than greater attention to the patch alone. Restoration tests the attention pathway. In Figure 4(c), Full patch denotes the patched pass without intervention; Clean weights and Clean values restoreํดandํ, respectively, to their clean counterparts. Restoring the attention branch, weights, or values to their clean counterparts while retaining the patched residual branch substantially reduces the latent and action changes. Perturbations outside the patch footprint are strongly suppressed, while the remaining perturbations are concentrated near the patch. The same interventions have little effect on the random patch. Figure 4 shows the spatial changes and their magnitude outside the footprint; Table 7 in Appendix B.1 reports the retained effects after each intervention. The distributed shift is coherent. The bottleneck positions outside the footprint receive change vectors with nearly identical directions and magnitudes. Letํbe these positions andโํป ํ their matrix of output changes. Defineํ =|ํ| โ1 (โํป ํ ) โค 1as the mean change vector and ํ ํป = โํป ํ โ 1ํ โค as the position-specific residual. Then โํป ํ = 1ํ โค + ํ ํป , โโํป ํ โ ํน โฅ โ๏ธ |ํ| โํโ 2 โโํ ํป โ ํน .(7) Here1 โR |ํ| is an all-ones vector. The inequality follows from the reverse triangle inequality: when the residual is small, the common change contributes across all|ํ|positions with aggregate magnitude proportional to โ๏ธ |ํ|. These results show how a small patch produces a coherent displacement over the full representation. 8 TAPDreamer: Transferable Adversarial Patches for World Action Models Table 4: Consistency of final-latent changes outside the patch footprint. CV denotes coefficient of variation. Metric TAPDreamer Random Cross-task cosine similarity0.9710.603 Magnitude CV0.0560.071 Template cosine similarity0.9810.716 Relative L1 residual0.2060.809 L1 bound ratio0.8330.069 01050200500 Optimization step 10 2 10 3 10 4 โฮ Z โ 1 (median) (a) Latent L1 displacement 01050200500 Optimization step 0.65 0.75 0.85 0.95 Cosine similarity (b) Template agreement Init A Source (a) Init B Held-out (a) Init C Random patch Figure 5: Optimization trajectories for three initializations: (a) latent L1 displacement and (b) source-template agreement. Dotted lines denote random-patch results. 4.2. The latent shift repeats across observations Measuring transfer of the response. Section 4.1 explains how the patch reaches distant positions within one observation. Transfer across tasks also requires a similar response when the scene changes. For a fixed patchํ, letโํ(ํฅ,ํ) = ํธ(ํ ํ (ํฅ))โ ํธ(ํฅ)be its final-latent change on observationํฅ. We average the patch-induced latent changes in observations from 12 source-task states to form a fixed templateโํ src (ํ), then compare it with changes in observations from 36 held-out states. The statistics in Table 4 use latent positions outside the projected patch footprint. Cross-task cosine similarity compares full final-latent displacement fields between tasks, while template cosine similarity compares each held-out field with the source template. Cosine similarities, relative residuals, and lower-bound ratios are reported as medians. Magnitude CV is the standard deviation of change magnitudes divided by their mean, computed over all 48 states. One source template describes held-out scenes. The source template has a median cosine similarity of 0.981 with held-out changes, and the median residual โฆ โฆ โํ ํ โโํ src โฆ โฆ 1 /โโํ ํ โ 1 is 0.206 in Table 4, whereํis the target held-out state. Clean representations from the same observations have a cross-task cosine similarity of 0.371, so the shared direction is specific to the patch-induced change. The random patch produces much weaker template agreement. Figure 5 shows that cross-observation alignment emerges early, while subsequent optimization mainly increases displacement on both source and held-out observations. Appendix B.2 reports the full trajectories, including the effective rank and frozen-template L1 bound during optimization. 9 TAPDreamer: Transferable Adversarial Patches for World Action Models The shared shift retains its magnitude. TAPDreamermaximizes L1 displacement, so the size of the common response matters alongside its direction. For a held-out stateํ, write its change as the source template plus a residualํ ํ . The reverse triangle inequality gives โํ ํ = โํ src + ํ ํ , โโํ ํ โ 1 โฅ โฆ โฆ โํ src โฆ โฆ 1 โโํ ํ โ 1 .(8) Using the measured residual for each held-out state, we compute the ratio of the lower bound in Equation(8) to the observed L1 displacement. The median ratio is 0.833 (range 0.732โ0.918). Thus, even after subtracting the observation-specific residual norm, the fixed source template yields a lower bound that retains much of the measured displacement. Local features reproduce the broadcast. About 95.3% of final-latent change energy outside the footprint lies in a shared spatial component. The direction of this component has a cross-task cosine similarity of about 0.997. To probe where this consistency begins, we replace keys and values within the patch footprint with source-task templates, retain a new imageโs clean context, and recompute attention. This K/V template analysis also includes a second patch initialization. The resulting broadcast closely matches the actual patched pass in direction and magnitude. Stable local keys and values thus help reproduce the distributed shift across observations. Additional analysis. We analyze how the latent shift changes actions in Appendix B.3. 5. Related Work Transferable and localized adversarial attacks. Adversarial perturbations can alter neural-network predictions (Goodfellow et al., 2015). Cross-model transfer has been studied using ensembles (Liu et al., 2017), momentum (Dong et al., 2018), and input diversity (Xie et al., 2019). Universal perturbations reuse a single perturbation across images (Moosavi-Dezfooli et al., 2017), with data-independent constructions such as Fast Feature Fool (Mopuri et al., 2017). Feature-level objectives have been explored through feature disruption (Ganeshan et al., 2019), feature importance (Wang et al., 2021), and neuron attribution (Zhang et al., 2022). Localized attacks include LaVAN (Karmon et al., 2018) and attention-aware Patch-Fool (Fu et al., 2022); physical attacks include robust object perturbations (Eykholt et al., 2018) and camera stickers (Li et al., 2019). PhotoGuard also studies encoder-level attacks on diffusion- based image editing (Salman et al., 2023). In sequential decision-making, adversarial policies can disrupt reinforcement-learning agents through an opponentโs behavior (Gleave et al., 2020). Attack on vision-language-action policies. Vision-language-action (VLA) policies such as OpenVLA (Kim et al., 2025) andํ 0 (Black et al., 2025) map visual and language inputs to robot actions. Action-based attacks on VLA policies optimize action differences, motion directions, target actions, or action freezing (Wang et al., 2025a,b; Yan et al., 2025). Their gradient- based objectives require access through the corresponding action generation interfaces. Representation-based methods attack imageโtext alignment, feature displacement, projection features, attention relationships, or visual proprioception (Xu et al., 2025; Lu et al., 2026; Zhang et al., 2025; Fu et al., 2026; Zhang et al., 2026). Cleanโperturbed representation distances already appear in prior attacks; here, we study their effectiveness for fixed local attacks on a public generative visual encoder and the mechanism of transfer. In the WAM pathway studied here, imageโtext interaction occurs downstream of the public VAE, so the corresponding alignment 10 TAPDreamer: Transferable Adversarial Patches for World Action Models and attention objectives fall outside the default encoder-only access. We therefore define the objective directly on the encoder output and evaluate its effects on control through closed-loop experiments and mechanism interventions. Adversarial threats to world models and world action models. BadWorld (Shen et al., 2026) evaluates attacks on generated videos, without testing closed-loop robotic tasks. BadWAM (Li et al., 2026) perturbs the full visual input through online target-model queries at each replanning step. Attacking the Trusted Imagination (Chen et al., 2026) optimizes observation perturbations using end- to-end gradients through the encoder and world model, corrupting future predictions used by downstream planning or safety checks. JailWAM (Liu et al., 2026) induces unsafe actions and requires attacker control over language instructions. Unlike these WAM threats,TAPDreamerachieves closed-loop attacks across tasks and WAM architectures with a fixed local patch optimized solely through a public visual encoder, without using target-policy action outputs or future predictions. 6. Conclusion TAPDreameroptimizes a fixed local patch using only a public visual encoder and six observations from one task, yet produces strong closed-loop interference across two complete benchmarks and different downstream action architectures. Mechanism analysis reveals that interactions between attention-weight and value changes broadcast local interference to distant representations, while optimized perturbation responses remain highly consistent across task observations. Behavioral analysis connects this representation vulnerability to grasping and pose interference, and separates the immediate effects of sustained visual input from state consequences accumulated through execution. These results show that WAM robustness must cover the full path from local visual input through representation, generation, and control. Ethics statement This study aims to identify vulnerabilities of World Action Models to local visual perturbations to inform robot safety evaluation and defense research. All experiments were conducted in the LIBERO and RoboTwin simulation environments, with attacks implemented by modifying model input images. We recognize that such methods could be misused to disrupt normal robot operation and emphasize that their use should be restricted to authorized security testing and research. Our mechanism analysis suggests potential defenses that attenuate keys and values from suspicious image regions or train the encoder to preserve representations outside locally perturbed regions. These interventions target the attention-mediated spread of the patch-induced shift; their effectiveness against adaptive attacks and their impact on clean-task performance remain to be evaluated. References Hongzhe Bi, Hengkai Tan, Shenghao Xie, Zeyuan Wang, Shuhe Huang, Haitian Liu, Ruowen Zhao, Yao Feng, Chendong Xiang, Yinze Rong, Hongyan Zhao, Hanyu Liu, Zhizhong Su, Lei Ma, Hang Su, and Jun Zhu. Motus: A Unified Latent Action World Model. In CVPR, p. 35101โ35113, 2026. URLhttps: //arxiv.org/abs/2512.13030. Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky.ํ 0 : A Vision-Language-Action Flow Model for General Robot Control. In Robotics: Science and Systems (RSS), 2025. URL https://arxiv.org/abs/2410.24164. 11 TAPDreamer: Transferable Adversarial Patches for World Action Models Linghan Chen, Kaiyan Ji, and Minyu Guo. Attacking the Trusted Imagination: Oracle-Level Integrity Attacks on Imagine-then-Act World Models. arXiv preprint arXiv:2606.22966, 2026. URLhttps://arxiv.org/abs/ 2606.22966. Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Zixuan Li, Qiwei Liang, Xianliang Lin, Yiheng Ge, Zhenyu Gu, Weiliang Deng, Yubin Guo, Tian Nian, Xuanbing Xie, Qiangyu Chen, Kailun Su, Tianling Xu, Guodong Liu, Mengkang Hu, Huan-ang Gao, Kaixuan Wang, Zhixuan Liang, Yusen Qin, Xiaokang Yang, Ping Luo, and Yao Mu. RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation. arXiv preprint arXiv:2506.18088, 2025. URL https://arxiv.org/abs/2506.18088. Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting Adversarial Attacks with Momentum. In CVPR, p. 9185โ9193, 2018. URL https://arxiv.org/abs/1710.06081. Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning Universal Policies via Text-Guided Video Generation. In NeurIPS, 2023. URL https://arxiv.org/abs/2302.00111. Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust Physical-World Attacks on Deep Learning Visual Classification. In CVPR, p. 1625โ1634, 2018. URL https://arxiv.org/abs/1707.08945. Jiyuan Fu, Kaixun Jiang, Jingkai Jia, Zhaoyu Chen, Xueyao Chen, Lingyi Hong, Shuyong Gao, Chenzhi Tan, Dingkang Yang, and Wenqiang Zhang. VLA-Hijack: A Transferable Patch Attack against Vision-Language- Action Models via Visual Proprioception Hijacking. arXiv preprint arXiv:2605.28083, 2026. URLhttps: //arxiv.org/abs/2605.28083. Yonggan Fu, Shunyao Zhang, Shang Wu, Cheng Wan, and Yingyan Celine Lin. Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations? In ICLR, 2022. URLhttps://arxiv.org/ abs/2203.08392. Aditya Ganeshan, B. S. Vivek, and R. Venkatesh Babu. FDA: Feature Disruptive Attack. In ICCV, 2019. URL https://arxiv.org/abs/1909.04385. Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial Policies: Attacking Deep Reinforcement Learning. In ICLR, 2020. URL https://arxiv.org/abs/1905.10615. Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. In ICLR, 2015. URL https://arxiv.org/abs/1412.6572. Danny Karmon, Daniel Zoran, and Yoav Goldberg. LaVAN: Localized and Visible Adversarial Noise. In ICML, volume 80 of Proceedings of Machine Learning Research, p. 2507โ2515, 2018. URLhttps://arxiv.org/ abs/1801.02608. Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An Open-Source Vision-Language- Action Model. In Conference on Robot Learning (CoRL 2024), volume 270 of Proceedings of Machine Learning Research, p. 2679โ2713, 2025. URL https://arxiv.org/abs/2406.09246. Juncheng Li, Frank Schmidt, and Zico Kolter. Adversarial Camera Stickers: A Physical Camera-Based Attack on Deep Learning Systems. In ICML, volume 97 of Proceedings of Machine Learning Research, p. 3896โ3904, 2019. URL https://arxiv.org/abs/1904.00759. 12 TAPDreamer: Transferable Adversarial Patches for World Action Models Qi Li, Xingyi Yang, and Xinchao Wang. BadWAM: When World-Action Models Dream Right but Act Wrong. arXiv preprint arXiv:2607.15207, 2026. URL https://arxiv.org/abs/2607.15207. Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. arXiv preprint arXiv:2306.03310, 2023. URLhttps: //arxiv.org/abs/2306.03310. Hanqing Liu, Songping Wang, Jiahuan Long, Jiacheng Hou, Jialiang Sun, Chao Li, Yang Yang, Wei Peng, Xu Liu, Tingsong Jiang, Yao Mu, and Wen Yao. JailWAM: Jailbreaking World Action Models in Robot Control. arXiv preprint arXiv:2604.05498, 2026. URL https://arxiv.org/abs/2604.05498. Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into Transferable Adversarial Examples and Black-box Attacks. In ICLR, 2017. URL https://arxiv.org/abs/1611.02770. Hui Lu, Yi Yu, Yiming Yang, Chenyu Yi, Qixin Zhang, Bingquan Shen, Alex C. Kot, and Xudong Jiang. When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Models. In CVPR, 2026. URL https://arxiv.org/abs/2511.21192. arXiv:2511.21192. Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal Adversarial Perturbations. In CVPR, 2017. URL https://arxiv.org/abs/1610.08401. Konda Reddy Mopuri, Utsav Garg, and R. Venkatesh Babu. Fast Feature Fool: A Data Independent Approach to Universal Adversarial Perturbations. In BMVC, 2017. URL https://arxiv.org/abs/1707.05572. Hadi Salman, Alaa Khaddaj, Guillaume Leclerc, Andrew Ilyas, and Aleksander Madry. Raising the Cost of Malicious AI-Powered Image Editing. In ICML, volume 202 of Proceedings of Machine Learning Research, p. 29894โ29918, 2023. URL https://arxiv.org/abs/2302.06588. Linghui Shen, Mingyue Cui, and Xingyi Yang. BadWorld: Adversarial Attacks on World Models. arXiv preprint arXiv:2606.16519, 2026. URL https://arxiv.org/abs/2606.16519. Taowen Wang, Cheng Han, James Chenhao Liang, Wenhao Yang, Dongfang Liu, Luna Xinyu Zhang, Qifan Wang, Jiebo Luo, and Ruixiang Tang. Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Robotics. In ICCV, 2025a. URL https://arxiv.org/abs/2411.13587. arXiv:2411.13587. Xin Wang, Jie Li, Zejia Weng, Yixu Wang, Yifeng Gao, Tianyu Pang, Chao Du, Yan Teng, Yingchun Wang, Zuxuan Wu, Xingjun Ma, and Yu-Gang Jiang. FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Models. arXiv preprint arXiv:2509.19870, 2025b. URL https://arxiv.org/abs/2509.19870. Zhibo Wang, Hengchang Guo, Zhifei Zhang, Wenxin Liu, Zhan Qin, and Kui Ren. Feature Importance-aware Transferable Adversarial Attacks. In ICCV, 2021. URL https://arxiv.org/abs/2107.14185. Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan Yuille. Improving Transferability of Adversarial Examples with Input Diversity. In CVPR, 2019. URLhttps://arxiv.org/ abs/1803.06978. Haochuan Xu, Yun Sing Koh, Shuhuai Huang, Zirun Zhou, Di Wang, Jun Sakuma, and Jingfeng Zhang. Model- agnostic Adversarial Attack and Defense for Vision-Language-Action Models. arXiv preprint arXiv:2510.13237, 2025. URL https://arxiv.org/abs/2510.13237. Yuping Yan, Yuhan Xie, Yixin Zhang, Lingjuan Lyu, Handing Wang, and Yaochu Jin. When Alignment Fails: Multimodal Adversarial Attacks on Vision-Language-Action Models. arXiv preprint arXiv:2511.16203, 2025. URL https://arxiv.org/abs/2511.16203. 13 TAPDreamer: Transferable Adversarial Patches for World Action Models Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chuning Zhu, Jiannan Xiang, Ayaan Malik, Kyungmin Lee, William Liang, Nadun Ranawaka, Ji- asheng Gu, Yinzhen Xu, Guanzhi Wang, Fengyuan Hu, Avnish Narayan, Johan Bjorck, Jing Wang, Gwanghyun Kim, Dantong Niu, Ruijie Zheng, Yuqi Xie, Jimmy Wu, Qi Wang, Ryan Julian, Danfei Xu, Yilun Du, Yevgen Chebotar, Scott Reed, Jan Kautz, Yuke Zhu, Linxi Fan, and Joel Jang. World Action Models are Zero-shot Policies. arXiv preprint arXiv:2602.15922, 2026. URL https://arxiv.org/abs/2602.15922. Shanglin Yuan, Weiheng Zhao, Xin Shi, Haoyi Jiang, Xianda Guo, Liu Liu, Wenyu Liu, Wei Sui, and Xinggang Wang. DreamWAM: Beyond RGB Future Prediction for World Action Models. arXiv preprint arXiv:2608.04996, 2026a. URL https://arxiv.org/abs/2608.04996. Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-WAM: Do World Action Models Need Test-time Future Imagination? arXiv preprint arXiv:2603.16666, 2026b. URLhttps://arxiv.org/abs/2603.16666. Jianping Zhang, Weibin Wu, Jen-tse Huang, Yizhan Huang, Wenxuan Wang, Yuxin Su, and Michael R. Lyu. Improving Adversarial Transferability via Neuron Attribution-Based Attacks. In CVPR, 2022. URL https://arxiv.org/abs/2204.00008. Jinquan Zhang, Dongfu Yin, Run Yang, Yufeng Yan, Zhen Tian, and F. Richard Yu. Structure-Aware Robust Fine-Tuning: Defending Vision-Language-Action Robots Against Physical Attention Hijacking. arXiv preprint arXiv:2608.03231, 2026. URL https://arxiv.org/abs/2608.03231. Naifu Zhang, Wei Tao, Xi Xiao, Qianpu Sun, Yuxin Zheng, Wentao Mo, Peiqiang Wang, and Nan Zhang. Attention-Guided Patch-Wise Sparse Adversarial Attacks on Vision-Language-Action Models. arXiv preprint arXiv:2511.21663, 2025. URL https://arxiv.org/abs/2511.21663. Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. RoboDreamer: Learning Compositional World Models for Robot Imagination. In ICML, volume 235 of Proceedings of Machine Learning Research, p. 61885โ61896, 2024. URL https://arxiv.org/abs/2404.12377. Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets. In Robotics: Science and Systems (RSS), 2025. URL https://arxiv.org/abs/2504.02792. A. Experimental Result and Ablation Studies A.1. Patch area and position We examine how patch area affects attack effectiveness and whether the attack depends on patch position. We conduct the patch-area and patch-position ablations on FastWAM across all 40 LIBERO tasks, using 50 trials per task and keeping the remaining settings consistent with the standard experimental configuration. As shown in Figure 6, increasing patch area from 3.35% to 3.96% sharply reduces success from 65.60% to 3.85%, whereas the matched random patch changes success only from 90.80% to 89.25%. The sharp success-rate drop underTAPDreamer, compared with the small change under matched random patches, shows that the effect depends on optimized patch content rather than increased occlusion alone. We also study the effect of patch position. We consider four positions in the external camera view: upper left (0,0), upper right (0,143), lower left (143, 0), lower right (143, 143). Coordinates are(ํฆ 0 ,ํฅ 0 )in pixels. We consider two settings: reoptimized patches are freshly optimized at the given position, and moved patches are copied from the patch optimized at the upper-left corner without update. 14 TAPDreamer: Transferable Adversarial Patches for World Action Models Our results are presented in Table 5. Reoptimizing a patch at each of the four tested positions achieves 0.00% success, but moving the unchanged upper-left patch to the other three corners yields 93.50%โ95.00% success, close to the corresponding random-patch results. Thus, effective patches can be optimized at different positions, but a patch optimized at one position does not transfer directly to another. 234567 Patch area (%) 0 20 40 60 80 100 Task success rate (%) Default 0.00% CleanRandom patchTAPDreamer Figure 6: Ablation study of patch area. Random patches use the same area and placement as TAPDreamer. Table 5: Ablation study of patch position. Reoptimized patches are optimized at the given position; moved patches are copies of the unchanged upper-left patch. Success rates are percentages; subscripts report signed differences from Random at the same position, in percentage points. Setup Upper left Upper right Lower left Lower right (0, 0)(0, 143)(143, 0) (143, 143) Random81.4597.5596.2093.80 Reoptimized 0.00 โ81.45 0.00 โ97.55 0.00 โ96.20 0.00 โ93.80 Moved0.00 โ81.45 93.50 โ4.05 94.85 โ1.35 95.00 +1.20 A.2. Direct transfer of VLA patches To further broaden the attack spectrum beyond our design, we explore the potential of existing VLA attacks. Due to the incompatibility of the VLA and WAM paradigms, we run only the transfer setting, i.e., the attack targets a VLA victim and transfers the optimized patches to a WAM victim. Table 6 reports direct transfer to FastWAM of patches constructed by UADA (Wang et al., 2025a) and UPA-RFAS (Lu et al., 2026). We train UPA-RFAS and UADA patches on their respective source VLAs using the original attack settings, with the patch area matched toTAPDreamer. We then transfer these patches directly to FastWAM without further optimization. Overall success is 76.55% for UPA-RFAS and 75.85% for UADA, compared with 81.45% for the random patch: reductions of 4.90 and 5.60 percentage points, respectively.TAPDreamerreduces success to 0% across all four suites. 15 TAPDreamer: Transferable Adversarial Patches for World Action Models Table 6: Direct transfer of VLA patches to FastWAM. Values are LIBERO task success rates (percent); subscripts report signed differences from Clean in the same suite or Overall column, in percentage points. Clean, Random, and TAPDreamer use the reference results in Table 1. Setup LIBERO SpatialObjectGoalLong-hor.Overall Clean98.0099.6097.6095.4097.65 Random85.60 โ12.40 78.00 โ21.60 90.00 โ7.60 72.20 โ23.20 81.45 โ16.20 UPA-RFAS 84.20 โ13.80 76.80 โ22.80 77.40 โ20.20 67.80 โ27.60 76.55 โ21.10 UADA82.00 โ16.00 77.00 โ22.60 77.20 โ20.40 67.20 โ28.20 75.85 โ21.80 TAPDreamer 0.00 โ98.00 0.00 โ99.60 0.00 โ97.60 0.00 โ95.40 0.00 โ97.65 B. How the Latent Shift Changes Action B.1. Attention-restoration interventions We want to understand which patch-induced changes within attention contribute to perturbation propagation, particularly the roles of attention weightsํดand value vectorsํin downstream representation and action changes. To investigate this, we conduct three restoration interventions: restoring the entire attention branch, restoring only the attention weightsํด, and restoring only the value vectorsํ. Each intervention retains the patched residual branch and replaces only the specified component. We measure the remaining change relative to the clean output, normalized by the change without restoration. For latent retention, both changes are measured using the L2 norm over the entire latent, including the patch footprint; action retention uses the corresponding L2 ratio for action outputs. The random patch is evaluated with the same latent-retention metric. Because attention reconstruction uses a different numerical precision from model inference, the patched reference without restoration is computed through the same reconstruction pathway. These ratios measure retained changes, not task success rates. Table 7 shows that all three interventions leave approximately 33% of the latent change and 32% of the action change induced byTAPDreamer. In comparison, random-patch latent retention remains approximately 1.000. Thus, restoring the attention branch, its weights, or its values removes most of theTAPDreamer-induced latent and action changes, while having little effect on the random patch. Table 7: Fraction of patch-induced change retained after attention restoration. Latent and action columns reportTAPDreamer; the last column reports random-patch latent retention. Values are medians over 48 states. Restored component Latent retained Action retained Random latent retained Restore clean attention branch 0.33270.3211.000 Weights A restored to clean0.33640.3201.000 Values V restored to clean0.33270.3211.000 B.2. Optimization trajectories To examine when shared perturbation structure emerges, we track optimization across three initializations on 48 observations. Within 25 steps, template agreement reaches at least 0.964 and effective rank falls below 1.39, while source displacement is only about 12% of its final magnitude. Subsequent optimization mainly amplifies displacement on both source and held-out observations while maintaining high alignment. 16 TAPDreamer: Transferable Adversarial Patches for World Action Models Initialization AInitialization BInitialization C SourceHeld-outRandom patch 01050200500 10 2 10 3 10 4 โฮ Z โ 1 (median) (a) Latent L1 displacement 01050200500 0.65 0.75 0.85 0.95 Cosine similarity (b) Template agreement 01050200500 Optimization step 1 3 5 7 Effective rank (c) Effective rank 01050200500 Optimization step 0.0 0.3 0.6 0.9 Bound / measured (d) Frozen-template L1 bound Figure 7: Evolution of perturbation structure across observations during optimization. For three initializations, the panels show displacement on source and held-out observations, source-template agreement, the effective rank of 48 perturbation fields, and the ratio of the frozen source-template lower bound to measured L1 displacement. In (a), solid and dashed curves denote source and held-out states, respectively. Dotted lines show results with random patches. 17 TAPDreamer: Transferable Adversarial Patches for World Action Models Figure 7(a) shows that displacement grows on both source and held-out observations, indicating that the amplified perturbation extends beyond the source states. The early rise in template agreement in (b) and the effective rank approaching one in (c) indicate that a common direction emerges early and becomes dominant across observations. The high lower-bound ratio in (d) shows that the source-derived template accounts for a substantial fraction of held-out displacement even after subtracting the residual. Together, these trajectories indicate that optimization first establishes a shared direction and then mainly increases its magnitude. Metrics and legend. At each optimization checkpoint in Figure 7, we measure the final-latent changeโํ ํ on the 356 spatial cells outside the projected patch footprint. The source templateโํ src is the mean displacement field over 12 source states at that checkpoint and is held fixed when evaluating the 36 held-out states. Panel (a) reports the medianโโํ ํ โ 1 separately for the source and held-out states. Panel (b) reports the median cosine similarity โจโํ src , โํ ํ โฉ/(โโํ src โ 2 โโํ ํ โ 2 ) over held-out states, with displacement fields flattened into vectors. For panel (c), we stack the 48 flattened displacement fields as rows of a matrix. Its effective rank is exp(โ โ๏ธ ํ ํ ํ logํ ํ ), whereํ ํ = ํ 2 ํ / โ๏ธ ํ ํ 2 ํ andํ ํ are its singular values (with0 log 0 = 0). A value near one indicates that the displacement fields largely share one direction. Panel (d) reports the median ratio (โโํ src โ 1 โโโํ ํ โโํ src โ 1 )/โโํ ํ โ 1 over held-out states: the frozen-template lower bound in Equation(8) divided by the measured L1 displacement. Colors identify the three initializations in all panels. Only panel (a) uses solid lines with circles for source states and dashed lines with squares for held-out states; panels (b)โ(d) show one summary curve per initialization. Gray dotted horizontal lines show the corresponding random-patch results, rather than an additional initialization. B.3. Behavioral effects The preceding analyses show how patch-induced perturbations propagate through attention and how optimiza- tion establishes a shared direction before amplifying its magnitude. We now ask how these representation changes affect action commands and, in turn, lead to task failure. Visible execution failures. The same representation shift can affect different actions as a task unfolds. In Figure 8, the patch drives the end effector below its clean trajectory and prevents task completion. We use same-state action comparisons, internal restoration, and patch removal to connect the shift to commands and physical consequences. Immediate action changes. Holding the physical state fixed, we compare actions from clean and patched observations before their tra- jectories diverge. In FastWAM, the patch suppresses lifting, biases pose commands downward, and shifts gripper commands toward opening. A fit in which each patched six-dimensional pose command mixes its clean counterpart with a shared target explains 78.2% of the variation across 1,500 paired commands. These are state-level measurements, not independent episodes. The same bias can prevent a lift, lose an object after grasping, or move the end effector away from its intended position, depending on the task stage. These action patterns emerge from the encoder-level objective, which specifies no target action. Action pathways across models. The disturbed visual representation also reaches actions through different downstream designs. In Motus, restoring the branch that represents visual world dynamics recovers about 93%โ99% of the action change, 18 TAPDreamer: Transferable Adversarial Patches for World Action Models Clean t=30t=120t=210t=290t=380 TAPDreamer t=30t=120t=210t=290t=380 (a) Put the alphabet soup and tomato sauce in the basket โ0.1 0.0 โ0.2 0.2 0.5 0.7 x (m) y (m) z height (m) (b) End-effector trajectory Figure 8: Example of closed-loop behavior under TAPDreamer. Left: observation sequences under clean and perturbed conditions. Right: corresponding three-dimensional end-effector position trajectories. Green dashed and red solid paths denote Clean and TAPDreamer, respectively. The triangle marks the starting position and stars mark the endpoints; ํก counts environment execution steps. Positions are in meters in the world coordinate frame. whereas restoring the visual-language understanding branch recovers about 10%โ27%. In DreamWAM-joint, restoring predicted future representations also substantially recovers actions. These interventions link encoder interference to control while showing that its route depends on the modelโs action pathway. Continued exposure and recovery. Gripper commands shift toward closing at the first clean replan after removal, indicating that the immediate command bias depends on continued visual interference. Whether the task recovers then depends on the physical state: some pose deviations can be corrected, while dropped or misplaced objects and missed manipulation opportunities may persist. In the duration sweep, some tasks finish after removal following 20 corrupted replans, whereas continuous exposure maintains strong failure. Repeated exposure lets the fixed patch keep injecting the same latent change as execution accumulates physical consequences. 19