Paper deep dive

Structure-Aware Robust Fine-Tuning: Defending Vision-Language-Action Robots Against Physical Attention Hijacking

Jinquan Zhang, Dongfu Yin, Run Yang, Yufeng Yan, Zhen Tian, F. Richard Yu

Year: 2026Venue: arXiv preprintArea: cs.ROType: PreprintEmbeddings: 47

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Vision-Language-Action (VLA) policies promise general robotic manipulation, but their robustness against physical-world attacks remains fragile. In particular, we show that physically realizable adversarial patches can reliably induce failures by triggering a mechanism we call policy-critical action-to-vision attention hijacking, where action-conditioned attention is diverted from task-relevant regions to a localized patch. To demonstrate the threat, we propose Attention-Guided Semantic Disruption (AGSD), an Expectation-over-Transformation (EOT) optimized printable patch that jointly (i) concentrates action-to-vision attention on the patch and (ii) disrupts vision-language semantic alignment, yielding strong cross-task and cross-architecture transfer. To mitigate such attacks, we introduce Structure-Aware Robust Fine-Tuning (SARF), a zero-inference-overhead defense that fine-tunes only the visual encoder using feature anchoring, policy-critical attention correction, and language-guided geometric consistency restricted to semantically relevant regions. On LIBERO, SARF reduces OpenVLA's failure rate under AGSD from 100% to 14.2%-56.8% (28.6% average) across suites while preserving clean performance, and on a real PiPER manipulator it improves average success under AGSD from 23.0% to 65.0%. These results highlight mechanism-level robustness as a practical path to securing VLA robots against physical attention hijacking.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

45,716 characters extracted from source content.

Expand or collapse full text

Structure-Aware Robust Fine-Tuning: Defending Vision-Language-Action Robots Against Physical Attention Hijacking Jinquan Zhang1,2, Dongfu Yin1,∗, Run Yang1,2, Yufeng Yan1,2, Zhen Tian1, F. Richard Yu3 This work was supported in part by the Research Task Assignment Project from Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) under Grant No. GML-26420004, and in part by the Shenzhen Science and Technology Program under Grant No. KJZD20240903104400001.1Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, China.2Shenzhen University, Shenzhen, China.3Carleton University, Ottawa, Canada.∗Corresponding author: Dongfu Yin, yindongfu@gml.ac.cn. Abstract Vision-Language-Action (VLA) policies promise general robotic manipulation, but their robustness against physical-world attacks remains fragile. In particular, we show that physically realizable adversarial patches can reliably induce failures by triggering a mechanism we call policy-critical action-to-vision attention hijacking, where action-conditioned attention is diverted from task-relevant regions to a localized patch. To demonstrate the threat, we propose Attention-Guided Semantic Disruption (AGSD), an Expectation-over-Transformation (EOT) optimized printable patch that jointly (i) concentrates action-to-vision attention on the patch and (i) disrupts vision–language semantic alignment, yielding strong cross-task and cross-architecture transfer. To mitigate such attacks, we introduce Structure-Aware Robust Fine-Tuning (SARF), a zero-inference-overhead defense that fine-tunes only the visual encoder using feature anchoring, policy-critical attention correction, and language-guided geometric consistency restricted to semantically relevant regions. On LIBERO, SARF reduces OpenVLA’s failure rate under AGSD from 100% to 14.2–56.8% (28.6% avg.) across suites while preserving clean performance, and on a real PiPER manipulator it improves average success under AGSD from 23.0% to 65.0%. These results highlight mechanism-level robustness as a practical path to securing VLA robots against physical attention hijacking. I INTRODUCTION Figure 1: Motivation and overview of SARF against physical attention hijacking. (a) PiPER tabletop manipulation conditioned on a natural-language instruction. (b) The original VLA succeeds on clean inputs but fails under a printable AGSD patch; SARF restores success under the same attack. (c) SARF fine-tunes only the visual encoder in a teacher–student framework using feature anchoring (ℒfeatL_feat), policy-critical action-token attention distillation (ℒpcadL_pcad), and language-guided geometric consistency (ℒgeoL_geo), with zero inference overhead. Vision–Language–Action (VLA) policies map images and natural-language instructions directly to low-level actions and have enabled increasingly general robotic manipulation [29, 9, 20, 3]. Because VLAs close the perception–action loop end-to-end, localized visual failures can immediately translate into erroneous physical behaviors, making robustness a first-order safety concern. A practical and low-cost threat is the physically realizable adversarial patch: a printable localized pattern that remains effective under viewpoint and illumination changes when optimized with Expectation-over-Transformation (EOT) [4, 2]. Recent studies report that patch attacks can substantially degrade VLA manipulation in both simulation and real settings [7, 14, 22]. Existing defenses (e.g., the defense accompanying EDPA) largely rely on adversarial fine-tuning to align global representations between clean and patched observations [24]. However, manipulation often depends on sparse task-critical evidence (e.g., end-effector and target object), suggesting that robustness may hinge on stabilizing where the policy attends, not only what it represents. We identify a failure mode that is particularly damaging for VLA control: policy-critical action-to-vision attention hijacking. In modern VLAs, action outputs are driven by cross-attention from a small set of action-query tokens to visual tokens. A patch can become a strong attention attractor, capturing attention mass from these action queries and suppressing attention on task-relevant regions, thereby derailing long-horizon trajectories and motivating defenses that stabilize action-conditioned attention pathways (Fig. 1). Guided by this observation, we propose (i) a stress-test attack to expose the mechanism and (i) a targeted defense. On the attack side, we introduce Attention-Guided Semantic Disruption (AGSD), an EOT-optimized printable patch that jointly concentrates action-to-vision attention on the patch and disrupts vision–language semantic alignment, improving transfer across tasks and architectures. To mitigate this vulnerability, we propose Structure-Aware Robust Fine-Tuning (SARF), a zero-inference-overhead teacher–student robust fine-tuning framework that updates only the visual encoder while freezing the multimodal backbone and action head. SARF combines feature anchoring, policy-critical action-token attention distillation to counteract hijacking, and language-guided geometric consistency restricted to semantically relevant regions. Our contributions are threefold: • We present AGSD, an EOT-optimized printable patch that targets policy-critical action-to-vision attention, jointly inducing attention hijacking and semantic misalignment to achieve strong cross-task and cross-architecture transfer. • We propose SARF, which achieves robust fine-tuning with zero inference overhead by combining feature anchoring, policy-critical attention correction, and language-guided geometric consistency restricted to semantically relevant regions. • We evaluate our methods on LIBERO [13] and a PiPER tabletop setup, demonstrating the broad vulnerability of multiple VLA architectures (e.g., π0 _0 and OpenVLA variants) to AGSD, and showing that SARF consistently enhances the robustness of OpenVLA across diverse physical patch scenarios while preserving clean performance. Figure 2: Physical attention hijacking and our attack–defense pipeline. (a) A VLA policy encodes an RGB observation x and instruction L and produces low-level actions via cross-attention. (b) AGSD optimizes a printable patch δ under EOT to hijack action-to-vision attention and disrupt vision–language alignment. (c) SARF performs teacher–student robust fine-tuning that updates only the student visual encoder EvSE_v^S (freezing π and the action head) using feature anchoring ℒfeatL_feat, policy-critical attention distillation ℒpcadL_pcad, and language-guided geometric consistency ℒgeoL_geo . (d) At deployment, SARF restores successful execution under the same patch with zero inference overhead. I Related Work I-A VLA / Robot Foundation Policies for Manipulation Vision–Language–Action (VLA) policies enable robots to execute open-vocabulary manipulation by conditioning low-level control on images and natural-language instructions. RT-2 demonstrates that scaling vision–language knowledge can benefit robotic control [29], while OpenVLA provides an open-source foundation VLA trained on large-scale robot demonstrations [9]. Octo further advances open generalist robot policies across diverse robots and sensors [20], and π0 _0 explores flow-based action generation for general robot control [3]. Complementary lines investigate alternative action modeling and cross-modal fusion for robotic perception [8, 6, 5]. However, current foundation policies primarily prioritize scaling and zero-shot generalization, leaving their vulnerabilities to physical adversarial threats largely underexplored. I-B Physical Patch Attacks and Robustness Evaluation for VLA Systems Printable adversarial patches, optimized via Expectation-over-Transformation (EOT) [4, 2], pose a realistic physical threat whose perception errors are often amplified by closed-loop robotic execution. Recent evaluation efforts highlight that VLA robustness under physical variations and adversarial conditions remains underexplored and can degrade dramatically under patch attacks [7, 14]. Beyond general robustness evaluation, VLA-specific adversarial studies propose patch objectives tailored to robotic action spaces, including UADA/UPA-style attacks that directly induce action-level discrepancies [22]. EDPA constructs model-agnostic patches by increasing representation discrepancy and image–text misalignment, and proposes a corresponding defense via adversarial fine-tuning [24]. More recent works emphasize transferability and universality across models and sim-to-real conditions (e.g., UPA-RFAS) [15], propose attention-guided sparse attacks for efficiency and stealthiness (e.g., ADVLA) [28], and provide broader benchmarking suites for adversarial and backdoor threats across the VLA lifecycle (e.g., AttackVLA) [11]. Despite these advances, existing physical attacks mostly target global feature disruption or text-to-vision misalignment, overlooking the specific attention pathways that drive VLA action generation. To expose this blind spot, our AGSD explicitly exploits policy-critical action-to-vision attention combined with semantic disruption, achieving strong transferability. I-C Defenses and Robust Fine-Tuning Against Patch Attacks Classical defenses against adversarial perturbations include adversarial training [16, 27], data augmentation and domain randomization [21], and input-time purification using generative models [17, 18, 19]. For localized patch threats, PatchGuard provides provable robustness by combining small receptive fields with robust masking [23]. However, purification-style defenses often introduce inference overhead and must be carefully evaluated against adaptive attacks to avoid a false sense of security [1]. In the VLA setting, EDPA’s defense improves robustness by adversarially fine-tuning the visual encoder to better align clean and patched representations [24], but it does not explicitly stabilize the attention mechanism that determines where the policy looks when producing actions. Meanwhile, robust/continual adaptation methods aim to preserve pretrained capabilities during fine-tuning, e.g., via regularization or distillation [10, 12], attention transfer [26], or parameter merging to improve robustness without extensive retraining [25]. Crucially, prior defenses in VLA settings either introduce inference overhead or focus solely on aligning global representations, failing to stabilize the fundamental attention mechanisms that determine action outputs. To overcome this, our SARF framework achieves zero-inference-overhead robustness by directly correcting policy-critical attention and enforcing localized geometric consistency. I Methodology We formalize the physical threat model, introduce AGSD, and then present SARF for improving VLA robustness against physical attention hijacking. I-A Attention-Guided Semantic Disruption (AGSD) We model the physical attack as a printable, localized adversarial patch optimization problem. Given the original RGB observation ∈ℝH×W×3x ^H× W× 3, the patched adversarial input ′x is defined as ′=(−)⊙+⊙,x =(1-m) +m δ, (1) where δ denotes the adversarial patch, ∈0,1H×Wm∈\0,1\^H× W is a binary spatial mask indicating the patch location, and ⊙ is the element-wise product. The attacker aims to find an optimal patch ∗ δ^* that remains effective under common physical imaging variations. We adopt the Expectation over Transformation (EOT) framework: ∗=argmint∼[ℒAGSD(t(′))], δ^*= *argmin_ δ\;E_t [L_AGSD (t(x ) ) ], (2) where t(⋅)t(·) is sampled from a transformation distribution T (e.g., random rotation, translation, and perspective tilting), and the expectation encourages robustness in real-world deployment. To effectively attack a VLA model, we define the attack total loss as a weighted combination of three terms: ℒAGSD=λattnℒattn−λdispℒdisp−λmisalignℒmisalign,L_AGSD= _attnL_attn- _dispL_disp- _misalignL_misalign, (3) where λattn _attn, λdisp _disp, and λmisalign _misalign are balancing coefficients. Note that ℒdispL_disp and ℒmisalignL_misalign are subtracted in Eq. (3), meaning that minimizing ℒAGSDL_AGSD effectively maximizes these two losses, thus disrupting feature consistency and image–text alignment. We define each component as follows. I-A1 Attention Guidance Loss (ℒattnL_attn) This loss directly manipulates the cross-modal attention mechanism to “hijack” the model’s perceptual focus. It forces the policy-critical action queries to over-attend to the patch region, thereby diverting action-to-vision cross-attention away from task-relevant evidence: ℒattn=−1|act||Kpatch|∑q∈act∑k∈KpatchA¯q,k,L_attn=- 1|Q_act||K_patch| _q _act _k∈ K_patch A_q,k, (4) where actQ_act denotes the set of policy action-query tokens that condition the action head, KpatchK_patch is the set of visual key tokens inside the patch region (determined by m), and A¯q,k A_q,k is the mean cross-attention weight aggregated from the last three cross-attention layers. The leading negative sign maximizes the attention weight from the action queries to the adversarial patch. I-A2 Feature Dispersion Loss (ℒdispL_disp) To destabilize semantic features, we adopt an InfoNCE-style objective: ℒdisp=−log⁡exp⁡(sim(adv,clean)/τnce)∑j=1Bexp⁡(sim(adv,clean(j))/τnce),L_disp=- (sim(z_adv,z_clean)/ _nce) _j=1^B (sim(z_adv,z_clean^(j))/ _nce), (5) where advz_adv and cleanz_clean denote the embeddings of the adversarial and clean images, B is the batch size, τnce _nce is the temperature, and sim(⋅,⋅)sim(·,·) is cosine similarity. Since ℒdispL_disp is subtracted in Eq. (3), the optimizer effectively maximizes it, which reduces the similarity between advz_adv and cleanz_clean and induces strong feature-space perturbations. I-A3 Image–Text Misalignment Loss (ℒmisalignL_misalign) This loss aims to maximize image–text misalignment in the joint embedding space: ℒmisalign=1B∑i=1B‖sim(adv(i),text(i))−sim(clean(i),text(i))‖1,L_misalign= 1B _i=1^B \|sim(z_adv^(i),z_text^(i))-sim(z_clean^(i),z_text^(i)) \|_1, (6) where text(i)z_text^(i) is the embedding of the i-th text instruction. By maximizing this L1L_1 distance (implicitly via Eq. (3)), the attacker forces the image–text matching of the adversarial image to deviate significantly from the correct alignment of the clean image. I-B Proposed Defense: Structure-Aware Robust Fine-Tuning (SARF) To defend against the above patch attack, we propose SARF. SARF fine-tunes only the visual encoder of the VLA model and uses a Teacher model pretrained on clean data to guide the recovery of a perturbed Student model. The overall SARF objective consists of feature anchoring, attention rectification, and geometric consistency: ℒSARF=λfeatℒfeat+λpcadℒpcad+λgeoℒgeo.L_SARF= _featL_feat+ _pcadL_pcad+ _geoL_geo. (7) I-B1 Feature Anchor Loss (ℒfeatL_feat) To prevent catastrophic forgetting during fine-tuning, we enforce the Student’s high-level semantic features to remain directionally consistent with the Teacher: ℒfeat=1−1N∑i=1NS(i)⋅T(i)‖S(i)‖‖T(i)‖,L_feat=1- 1N _i=1^N z_S^(i)·z_T^(i)\|z_S^(i)\|\|z_T^(i)\|, (8) where N is the number of visual patch tokens, and S(i)z_S^(i) and T(i)z_T^(i) are the i-th patch-level feature vectors from the Student and Teacher, respectively. This loss anchors the feature space by minimizing the cosine distance (1−cos⁡θ)(1- θ). I-B2 Policy-Critical Attention Distillation (ℒpcadL_pcad) Motivated by the VLA policy structure, we distill attention only for Action Tokens that determine action outputs. We align Student and Teacher attention distributions using a symmetric Jensen–Shannon Divergence (JSD) objective: ℒpcad=1H|act|∑h=1H∑q∈actDJS(PT(h)∥PS(h)),L_pcad= 1H _act _h=1^H _q _actD_JS\! (P_T^(h) P_S^(h) ), (9) where H is the number of heads and actQ_act indexes Action tokens. For each h and q∈actq _act, we form attention distributions over visual tokens P(h)(k∣q)=Softmax(Aq,k(h)/τattn)P^(h)(k q)=Softmax(A^(h)_q,k/ _attn) and minimize DJS(PT(h)(⋅∣q)∥PS(h)(⋅∣q))D_JS(P_T^(h)(· q) P_S^(h)(· q)) to align policy-critical action-to-vision attention. I-B3 Language-Guided Geometric Consistency (ℒgeoL_geo) To preserve the geometry of task-relevant objects while suppressing background noise, we introduce a language-guided geometric mask. We first compute the importance of each patch i using Teacher text-to-vision attention: mi=maxq∈txt⁡Aq,iTm_i= _q _txtA_q,i^T, and construct a sharpened pairwise mask ij=(mi⋅mj)2M_ij=(m_i· m_j)^2. Since this mask is derived from the clean Teacher stream rather than the perturbed Student stream, it provides a stable reference for task-relevant regions. Based on this mask, the geometric loss is defined as ℒgeo=∑i,jij⋅(GijS−GijT)2∑i,jij+ϵ,L_geo= _i,jM_ij· (G_ij^S-G_ij^T )^2 _i,jM_ij+ε, (10) where the Gram element GijG_ij captures pairwise feature relations (texture and structure): Gij=(i)⋅(j)‖(i)‖‖(j)‖.G_ij= z^(i)·z^(j)\|z^(i)\|\|z^(j)\|. (11) This loss constrains only the regions activated by M, enabling structure denoising on task-critical objects (e.g., the end-effector and manipulated objects) while automatically down-weighting background perturbations. IV Experiments We first demonstrate that physical patch attacks are feasible and can severely degrade Vision–Language–Action (VLA) robots. We then analyze the underlying failure mechanism via attention-level diagnostics, propose SARF as a targeted solution, and validate robustness both in simulation and on a real PiPER system. Unless otherwise specified, results are reported on four LIBERO suites (Spatial, Object, Goal, Long). IV-A Experimental Setup IV-A1 Benchmarks We evaluate on four LIBERO simulation task suites (reporting Failure Rate, FR) and a real-world PiPER tabletop setup (reporting Success Rate, SR). For PiPER, we conduct 100 independent trials per condition with randomized initial object poses, camera viewpoints, and distances to assess physical robustness. IV-A2 Models We evaluate representative VLA policies, including OpenVLA and its fine-tuned variant OpenVLA-oft, as well as a stronger foundation-policy baseline π0 _0 (Pi0). All models are evaluated using their default action interfaces and inference settings to ensure fair comparison. IV-A3 Attacks We consider localized, printable adversarial patches and compare: (i) Clean (no patch), (i) Random (non-optimized printable patch), (i) UADA and UPA from [22], (iv) EDPA from [24], and (v) AGSD (ours). Unless otherwise stated, all methods use the same patch size and placement distribution. IV-A4 Defenses We report three settings on OpenVLA: Original (no defense), AF (Defense Method in the EDPA), and SARF (ours). SARF fine-tunes only the visual encoder and distills policy-critical action-to-vision attention from a clean Teacher model, with language-guided geometric consistency restricted to semantically relevant regions. We instantiate SARF on OpenVLA because its action-conditioned visual attention is accessible; extension to diffusion- or flow-style decoders requires identifying analogous policy-critical visual pathways. IV-A5 Metrics For LIBERO, we report Failure Rate (FR, %). For the real robot, we report Success Rate (SR, %). When evaluating attack effectiveness (e.g., AGSD), a higher FR indicates a stronger attack (worse robustness). When evaluating defense effectiveness (e.g., SARF), a lower FR indicates better robustness while preserving performance on clean inputs. IV-A6 Implementation details and EOT settings To ensure stable optimization by balancing inherent gradient magnitudes and prioritizing critical objectives, we set the AGSD coefficients to λattn=0.8 _attn=0.8, λdisp=0.2 _disp=0.2, and λmisalign=0.5 _misalign=0.5. These values are selected such that the attention-guidance loss acts as the primary driver of optimization, while the remaining terms provide auxiliary semantic disruption. For the SARF defense, we set λfeat=0.5 _feat=0.5, λpcad=1.0 _pcad=1.0, and λgeo=0.3 _geo=0.3. This configuration reflects our core design principle: prioritizing the distillation of policy-critical attention (ℒpcadL_pcad) to counteract hijacking, while leveraging ℒfeatL_feat and ℒgeoL_geo as structure-aware constraints to preserve the pretrained visual backbone’s integrity. In LIBERO, the adversarial patch occupies 5% of the image area. On the real PiPER robot, we evaluate printed physical patches with sizes of 20×20 cm, 15×15 cm, 8×8 cm, and 5×5 cm. To ensure physical realizability, all attacks are optimized and evaluated under an Expectation-over-Transformation (EOT) framework that incorporates random in-plane rotation (θ∼(−30∘,30∘)θ (-30 ,30 )), translation (up to 10% of the image dimensions), scaling (s∼(0.9,1.1)s (0.9,1.1)), random patch placement (uniformly sampled within the image bounds), and mild perspective/illumination jitter. SARF uses the same data budget as AF and updates only the visual encoder; the deployed architecture and inference latency remain unchanged. Figure 3: Printable AGSD patch examples. Representative AGSD patches optimized under the same EOT pipeline. Columns correspond to the four LIBERO suites (Spatial, Object, Goal, Long); rows show patches optimized for OpenVLA and OpenVLA-oft using primary and wrist camera observations. These patches are used in our physical patch-attack evaluations. IV-B Physical Attention Hijacking IV-B1 Quantitative results of attack effectiveness As shown in Table I, even non-optimized Random patches already increase failure rates compared to Clean inputs, indicating that localized physical perturbations can impair manipulation. However, optimized patches are substantially more damaging; AGSD drives OpenVLA to 100% failure across all suites, improving over Random by 24.4–64.2 points. Crucially, AGSD transfers beyond the attacked model: on OpenVLA-oft, failure rises from 10.0–32.0% (Random) to 93.6–100% (AGSD), and on the stronger π0 _0 baseline from 5.2–48.6% (Random) to 48.8–80.2% (AGSD). Relative to EDPA, AGSD is comparable on OpenVLA while remaining markedly stronger on OpenVLA-oft and π0 _0, supporting cross-architecture transfer. Consistently, Table I (Original column) confirms that, under the same evaluation protocol used for defense studies, the undefended OpenVLA exhibits near-certain failure under AGSD across all suites. TABLE I: Attack effectiveness on LIBERO (Failure Rate, %). Failure rates (mean± ) of OpenVLA, OpenVLA-oft, and _0 under Clean, Random, and optimized patch attacks (EDPA, AGSD) across four suites. Higher FR indicates stronger attacks; best (highest) results among optimized attacks are in bold. Suite Method Failure Rate (FR↑ ) OpenVLA OpenVLA-oft _0 Spatial Clean 14.2 ± 0.5 2.4 ± 0.4 3.4 ± 0.3 Random 35.8 ± 1.3 10.0 ± 1.1 6.0 ± 0.8 EDPA[24] 100 ± 0.0 39.7 ± 0.9 29.8 ± 1.6 AGSD (Ours) 100 ± 0.0 97.2 ± 0.6 48.8 ± 2.1 Object Clean 11.6 ± 0.4 2.6 ± 0.3 2.0 ± 0.2 Random 44.6 ± 1.2 20.4 ± 1.5 5.2 ± 0.5 EDPA[24] 100 ± 0.0 52.3 ± 0.8 39.5 ± 1.7 AGSD (Ours) 100 ± 0.0 93.6 ± 1.2 50.4 ± 2.4 Goal Clean 20.8 ± 1.5 3.0 ± 0.6 10.4 ± 1.1 Random 42.0 ± 1.2 16.2 ± 1.3 16.8 ± 1.4 EDPA[24] 100 ± 0.0 80.8 ± 0.4 44.3 ± 2.0 AGSD (Ours) 100 ± 0.0 100 ± 0.0 70.8 ± 1.8 Long Clean 46.2 ± 2.0 4.8 ± 0.7 42.0 ± 1.6 Random 75.6 ± 2.4 32.0 ± 2.1 48.6 ± 1.9 EDPA[24] 100 ± 0.0 86.4 ± 1.9 70.7 ± 1.6 AGSD (Ours) 100 ± 0.0 100 ± 0.0 80.2 ± 1.5 Figure 4: AGSD objective ablation. Failure rates (FR, %) of OpenVLA on four LIBERO suites under four printable patch variants: Random, Attn-only, Disp+Misalign, and Full AGSD. Higher FR indicates a stronger attack; error bars denote variability across repeated evaluations. IV-B2 AGSD produces printable and deployable patches To illustrate physical plausibility, Fig. 3 visualizes representative printable adversarial patches optimized by AGSD against OpenVLA and OpenVLA-oft under the same EOT transformation pipeline. Across suites and viewpoints (primary and wrist cameras), the optimized patterns are concrete and reproducible, supporting real-world deployment rather than relying on imperceptible digital noise. IV-B3 Mechanistic insight via objective ablation To understand why AGSD is particularly effective, we conduct an objective-level ablation in Fig. 4. Optimizing only the attention-guidance term (Attn-only) already increases FR, indicating that explicitly manipulating policy-critical action-to-vision attention can drive failures. Optimizing semantic disruption without explicit attention guidance (Disp + Misalign) also degrades performance by destabilizing visual features and vision–language alignment. Crucially, the Full AGSD objective that couples attention guidance with semantic disruption achieves the strongest attacks across all LIBERO suites, suggesting that these components are complementary and together yield more transferable physical patches. IV-B4 Qualitative attention hijacking under AGSD While Table I and Fig. 4 establish the strength and contributing factors of AGSD, we further provide qualitative attention evidence in Fig. 5. Under AGSD, the Original policy’s action-to-vision attention repeatedly collapses onto the patch region across rollout timesteps, diverting focus away from task-relevant evidence. This motivates defenses that explicitly stabilize policy-critical attention pathways. IV-C SARF for Stabilizing Policy-Critical Attention IV-C1 Goal and evaluation protocol We evaluate SARF as a drop-in defense against physically realizable patch attacks: it introduces zero inference overhead (no test-time modules and unchanged policy interface) and updates only the visual encoder during training. TABLE I: OpenVLA robustness under physical patch attacks (FAILURE RATE,%). Failure rates (mean± ) on four LIBERO suites for Original, AF(Defense Method in the EDPA), and SARF under Clean, Random, and optimized attacks (UADA/UPA/EDPA/AGSD); lower is better and best is in bold. SARF–AGSD uses an adaptive (re-optimized) AGSD patch against the SARF-tuned model. Suite Method Failure Rate (FR↓ ) Original AF[24] SARF (Ours) Spatial Clean 14.2 ± 0.5 17.9 ± 0.8 14.4 ± 0.6 Random 35.8 ± 1.3 19.4 ± 1.1 15.0 ± 0.7 UADA[22] 98.8 ± 0.2 65.4 ± 2.5 16.2 ± 0.9 UPA[22] 99.0 ± 0.1 46.6 ± 2.1 15.8 ± 0.8 EDPA[24] 100 ± 0.0 39.4 ± 1.8 16.5 ± 1.0 AGSD (Ours) 100 ± 0.0 90.2 ± 1.5 17.0 ± 1.1 Object Clean 11.6 ± 0.4 17.3 ± 0.9 11.8 ± 0.5 Random 44.6 ± 1.2 16.0 ± 1.0 12.4 ± 0.6 UADA[22] 92.0 ± 1.5 58.8 ± 2.3 13.2 ± 0.8 UPA[22] 94.2 ± 1.1 43.9 ± 2.0 12.8 ± 0.7 EDPA[24] 100 ± 0.0 58.6 ± 2.4 13.6 ± 0.9 AGSD (Ours) 100 ± 0.0 99.8 ± 0.2 14.2 ± 1.0 Goal Clean 20.8 ± 1.5 22.8 ± 1.2 21.0 ± 1.4 Random 42.0 ± 1.2 23.0 ± 1.5 22.2 ± 1.6 UADA[22] 98.6 ± 0.4 91.6 ± 1.8 24.6 ± 1.7 UPA[22] 96.4 ± 0.8 68.3 ± 2.2 23.8 ± 1.5 EDPA[24] 100 ± 0.0 73.9 ± 2.5 25.4 ± 1.8 AGSD (Ours) 100 ± 0.0 97.2 ± 0.6 26.5 ± 2.0 Long Clean 46.2 ± 2.0 49.0 ± 2.1 46.6 ± 1.9 Random 75.6 ± 2.4 50.2 ± 2.5 48.8 ± 2.2 UADA[22] 99.8 ± 0.2 97.4 ± 0.8 53.2 ± 2.6 UPA[22] 99.8 ± 0.1 86.7 ± 1.5 51.5 ± 2.4 EDPA[24] 100 ± 0.0 91.2 ± 1.2 54.0 ± 2.7 AGSD (Ours) 100 ± 0.0 99.0 ± 0.5 56.8 ± 2.9 Figure 5: AGSD hijacks policy-critical attention, while SARF restores task-relevant focus. Action-to-vision cross-attention heatmaps overlaid on the RGB observation at four rollout timesteps (t=0,T/3, 2T/3,Tt\!=\!0,\,T/3,\,2T/3,\,T). Rows show Clean/Random baselines (Original), AGSD-attacked Original, and AGSD-attacked SARF. AGSD causes attention to collapse onto the patch region across time, whereas SARF suppresses patch fixation and re-centers attention on task-relevant evidence. IV-C2 Main robustness gains under strong transferable patches Table I reports OpenVLA failure rates (FR; lower is better) under diverse physical patch attacks. Under the strongest stress-test attack, AGSD, the undefended policy fails almost always (FR=100%=100\%) on all four suites. In contrast, SARF reduces FR to 17.0% / 14.2% / 26.5% / 56.8% on Spatial/Object/Goal/Long, corresponding to absolute drops of 83.0 / 85.8 / 73.5 / 43.2 points, respectively. Averaged across suites, SARF decreases FR from 100.0% to 28.6% under AGSD (a 71.4-point reduction). The higher failure rate on the Long suite suggests that minor residual attention jitter can compound over long-horizon trajectories, though the improvement over the undefended baseline remains substantial. IV-C3 Consistent improvements across attack baselines Beyond AGSD, SARF yields consistent robustness gains across prior patch baselines. Averaged over suites, FR decreases from 97.3% (UADA) to 26.8%, from 97.4% (UPA) to 26.0%, and from 100.0% (EDPA) to 27.4%. These results indicate that SARF improves robustness broadly under diverse optimization objectives and transfer settings. IV-C4 Comparison to adversarial fine-tuning (AF) Compared with AF (the EDPA defense), SARF achieves markedly lower FR under strong attacks. For example, under AGSD, AF remains highly vulnerable with an average FR of 96.6%, whereas SARF reduces it to 28.6% (an additional 67.9-point reduction over AF). Similar margins hold for EDPA (65.8% to 27.4%) and UADA (78.3% to 26.8%) on average. This massive performance gap highlights that for end-to-end visuomotor policies, stabilizing where the model looks is far more effective than broadly aligning what the model represents. IV-C5 Robustness under an adaptive attacker Importantly, the AGSD results for SARF are obtained using an adaptive setting where the patch is re-optimized against the frozen SARF-tuned model using the same AGSD objective and EOT settings. Even under this stronger threat model, SARF maintains substantially lower FR than both the undefended policy and AF. IV-C6 Clean performance is preserved Robustness gains do not come at the cost of clean-task degradation: on clean inputs, FR changes only marginally from 14.2/11.6/20.8/46.2% (Original) to 14.4/11.8/21.0/46.6% (SARF) across suites (average 23.2% to 23.5%). IV-C7 Mechanism-level evidence: attention is re-centered under attack Fig. 5 provides an attention-level diagnosis consistent with the quantitative improvements. Under AGSD, the original policy exhibits persistent patch fixation where action-to-vision cross-attention collapses onto the patch region across rollout timesteps. After SARF, attention is re-centered on semantically relevant regions under the same physical patch, supporting the mechanism-targeted explanation that stabilizing policy-critical attention helps prevent long-horizon failures. IV-D Real-Robot Validation on PiPER IV-D1 Protocol and metric We evaluate three real-world manipulation tasks on a PiPER tabletop platform (Table I) and report Success Rate (SR, %). For each task, we run 100 physical trials with randomized initial object poses and randomized trial order. We report clean performance for the undefended policy (Original), and evaluate robustness under a printed AGSD patch by comparing Original, AF [24], and SARF (ours). To reflect realistic sensing variations, we vary the camera viewpoint and distance across trials. Figure 6: Real-robot qualitative comparison under physical patch attacks (PiPER tabletop). Four keyframes (Start/Grasp/Approach/Outcome) under Clean and Random baselines (Original), and under an AGSD patch with the Original and SARF policies. The AGSD patch causes failure for the original policy, while SARF restores successful execution under the same printed patch (zero inference overhead). TABLE I: Real-robot results on PiPER under a printed AGSD patch (Success Rate, %). Success rates are averaged over 100 physical trials per condition. We compare the undefended policy (Original), the strongest simulation baseline (AF), and our SARF. SARF decisively restores manipulation capabilities under the same physical patch. Task Clean Under AGSD Attack Original Original AF [24] SARF (Ours) pick & place 79.0 32.0 51.0 74.0 open drawer 71.0 23.0 42.0 63.0 stack 66.0 14.0 29.0 58.0 Average 72.0 23.0 40.7 65.0 IV-D2 SARF recovers real-world performance under a printed AGSD patch As shown in Table I, the printed AGSD patch substantially degrades real-world performance, reducing the average success rate of the undefended policy (Original) from 72.0% (clean) to 23.0% under attack. The simulation baseline AF [24] partially improves robustness (40.7% on average), whereas SARF restores manipulation much more effectively under the same printed patch, achieving 65.0% average success (a 42.0-point gain over the attacked Original). The gains are consistent across tasks: for pick & place, SARF reaches 74.0% under attack, close to the clean performance of 79.0%; for the precision-heavy stack task, SARF improves success from 14.0% to 58.0% (a 44.0-point gain), although an 8.0-point gap to the clean setting remains. These results suggest that SARF mitigates physical attention hijacking in the real world, although residual errors can still arise from remaining attention leakage to the patch under challenging sensing variations. IV-E Ablation Study TABLE IV: SARF ablation on LIBERO-Spatial (Failure Rate, %). Failure rates of OpenVLA under Clean, Random Patch, and AGSD Attack. Lower FR indicates better performance. Method Failure Rate (FR%↓ ) Clean Random Patch AGSD Attack only ℒpcadL_pcad 34.7 ± 2.0 41.1 ± 2.4 54.4 ± 2.5 only ℒgeoL_geo 37.9 ± 1.5 59.8 ± 1.9 81.6 ± 1.2 w/o ℒfeatL_feat 31.6 ± 2.5 34.9 ± 2.8 37.2 ± 3.0 w/o ℒpcadL_pcad 14.2 ± 1.1 44.6 ± 2.2 87.5 ± 1.4 w/o ℒgeoL_geo 14.0 ± 1.3 21.5 ± 1.6 35.8 ± 2.1 Full SARF 14.4 ± 0.6 15.0 ± 0.7 17.0 ± 1.1 To validate the individual contribution of each SARF component towards defending against physical attacks and preserving original capabilities, we conduct an ablation study on LIBERO-Spatial (Table IV). While Full SARF achieves the strongest robustness under Random Patch and AGSD Attack while maintaining near-original clean performance, removing ℒpcadL_pcad collapses robustness under AGSD (FR spikes to 87.5%). Furthermore, dropping ℒfeatL_feat severely harms clean performance (31.6% FR), and removing ℒgeoL_geo yields a moderate robustness drop (35.8% FR). These results demonstrate that all three objectives are synergistic and indispensable: policy-critical attention correction drives the core defense, while feature anchoring and geometric consistency ensure the stability of the original visuomotor policy. V CONCLUSIONS We characterized physically realizable patch threats to Vision–Language–Action (VLA) manipulation and pinpointed a mechanism-level failure—policy-critical action-to-vision attention hijacking—that causally links localized visual perturbations to compounding control errors. Building on this diagnosis, we introduced AGSD as a stress-test physical attack that combines attention capture with vision–language semantic disruption, and we proposed SARF, a zero-inference-overhead defense that strengthens robustness by updating only the visual encoder while preserving the deployed policy interface. Across LIBERO suites and on a real PiPER platform, SARF consistently improves robustness under AGSD while maintaining clean-task performance, and attention visualizations corroborate that it re-centers action-conditioned attention onto task-relevant evidence during execution. The ablation results further show that policy-critical attention distillation is the dominant factor for attack robustness, while feature anchoring and geometric consistency help preserve clean-task stability. Overall, our findings suggest that mechanism-targeted robustness—stabilizing the attention pathways that directly drive action generation—offers a practical and deployable route to securing VLA robots against physical attacks without adding test-time complexity. In future work, we will extend SARF to natural clutter and distractors, further reduce residual long-horizon attention jitter, and validate the defense across broader VLA backbones, embodiments, and third-party adaptive attacks. References [1] A. Athalye, N. Carlini, and D. Wagner (2018) Obfuscated gradients give a false sense of security: circumventing defenses to adversarial examples. In International conference on machine learning, p. 274–283. Cited by: §I-C. [2] A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok (2018) Synthesizing robust adversarial examples. In International conference on machine learning, p. 284–293. Cited by: §I, §I-B. [3] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. (2024) π0 _0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164. Cited by: §I, §I-A. [4] T. B. Brown, D. Mané, A. Roy, M. Abadi, and J. Gilmer (2017) Adversarial patch. arXiv preprint arXiv:1712.09665. Cited by: §I, §I-B. [5] H. Cai, D. Yin, F. R. Yu, and S. Xiong (2025) DSTR: dual scenes transformer for cross-modal fusion in 3d object detection. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 3064–3073. Cited by: §I-A. [6] H. Cai, D. Yin, F. Yu, and S. Xiong (2025) MFT: modal fusion transformer for cross-modal fusion in 3d object detection. In ICASSP 2025–2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §I-A. [7] H. Cheng, E. Xiao, Y. Wang, C. Yu, M. Sun, Q. Zhang, J. Cao, Y. Guo, N. Liu, K. Xu, et al. (2024) Manipulation facing threats: evaluating physical vulnerabilities in end-to-end vision language action models. arXiv preprint arXiv:2409.13174. Cited by: §I, §I-B. [8] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song (2025) Diffusion policy: visuomotor policy learning via action diffusion. The International Journal of Robotics Research 44 (10-11), p. 1684–1704. Cited by: §I-A. [9] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. (2024) Openvla: an open-source vision-language-action model. arXiv preprint arXiv:2406.09246. Cited by: §I, §I-A. [10] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. (2017) Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114 (13), p. 3521–3526. Cited by: §I-C. [11] J. Li, Y. Zhao, X. Zheng, Z. Xu, Y. Li, X. Ma, and Y. Jiang (2025) AttackVLA: benchmarking adversarial and backdoor attacks on vision-language-action models. arXiv preprint arXiv:2511.12149. Cited by: §I-B. [12] Z. Li and D. Hoiem (2018) Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence 40 (12), p. 2935–2947. Cited by: §I-C. [13] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone (2023) Libero: benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems 36, p. 44776–44791. Cited by: 3rd item. [14] H. Liu, J. Long, J. Wu, J. Hou, H. Tang, T. Jiang, W. Zhou, and W. Yao (2025) Eva-vla: evaluating vision-language-action models’ robustness under real-world physical variations. arXiv preprint arXiv:2509.18953. Cited by: §I, §I-B. [15] H. Lu, Y. Yu, Y. Yang, C. Yi, Q. Zhang, B. Shen, A. C. Kot, and X. Jiang (2025) When robots obey the patch: universal transferable patch attacks on vision-language-action models. arXiv preprint arXiv:2511.21192. Cited by: §I-B. [16] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu (2017) Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083. Cited by: §I-C. [17] W. Nie, B. Guo, Y. Huang, C. Xiao, A. Vahdat, and A. Anandkumar (2022) Diffusion models for adversarial purification. arXiv preprint arXiv:2205.07460. Cited by: §I-C. [18] P. Samangouei, M. Kabkab, and R. Chellappa (2018) Defense-gan: protecting classifiers against adversarial attacks using generative models. arXiv preprint arXiv:1805.06605. Cited by: §I-C. [19] Y. Song, T. Kim, S. Nowozin, S. Ermon, and N. Kushman (2017) Pixeldefend: leveraging generative models to understand and defend against adversarial examples. arXiv preprint arXiv:1710.10766. Cited by: §I-C. [20] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. (2024) Octo: an open-source generalist robot policy. arXiv preprint arXiv:2405.12213. Cited by: §I, §I-A. [21] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel (2017) Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), p. 23–30. Cited by: §I-C. [22] T. Wang, C. Han, J. Liang, W. Yang, D. Liu, L. X. Zhang, Q. Wang, J. Luo, and R. Tang (2025) Exploring the adversarial vulnerabilities of vision-language-action models in robotics. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 6948–6958. Cited by: §I, §I-B, §IV-A3, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I. [23] C. Xiang, A. N. Bhagoji, V. Sehwag, and P. Mittal (2021) \patchguard\: A provably robust defense against adversarial patches via small receptive fields and masking. In 30th USENIX Security Symposium (USENIX Security 21), p. 2237–2254. Cited by: §I-C. [24] H. Xu, Y. S. Koh, S. Huang, Z. Zhou, D. Wang, J. Sakuma, and J. Zhang (2025) Model-agnostic adversarial attack and defense for vision-language-action models. arXiv preprint arXiv:2510.13237. Cited by: §I, §I-B, §I-C, §IV-A3, §IV-D1, §IV-D2, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I. [25] Y. Yadav, Z. Zhou, A. Wagenmaker, K. Pertsch, and S. Levine (2025) Robust finetuning of vision-language-action robot policies via parameter merging. arXiv preprint arXiv:2512.08333. Cited by: §I-C. [26] S. Zagoruyko and N. Komodakis (2016) Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928. Cited by: §I-C. [27] H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan (2019) Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, p. 7472–7482. Cited by: §I-C. [28] N. Zhang, W. Tao, X. Xiao, Q. Sun, Y. Zheng, W. Mo, P. Wang, and N. Zhang (2025) Attention-guided patch-wise sparse adversarial attacks on vision-language-action models. arXiv preprint arXiv:2511.21663. Cited by: §I-B. [29] B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. (2023) Rt-2: vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, p. 2165–2183. Cited by: §I, §I-A.

Instant research discovery