Paper deep dive
Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration
Ninghao Zhang, Bin Zhu, Shijie Zhou, Jingjing Chen
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/13/2026, 12:15:18 AM
Summary
The paper introduces ICBench, a diagnostic benchmark for evaluating linguistic grounding in Vision-Language-Action (VLA) models by injecting contradictory instructions. It identifies 'linguistic blindness' in models like Pi0, Pi0.5, and OpenVLA-OFT, where visual priors override semantic instructions. To mitigate this, the authors propose Instruction-Guided Attention Recalibration (IGAR), a train-free, inference-time mechanism that rebalances attention to restore linguistic influence without requiring model retraining.
Entities (6)
Relation Signals (3)
ICBench → evaluates → VLA Models
confidence 95% · ICBench, a controlled benchmark that injects out-of-distribution (OOD) semantic contradictions into task instructions
IGAR → improves → VLA Models
confidence 95% · IGAR... restores linguistic grounding in Vision-Language-Action (VLA) models
Pi0 → exhibits → Linguistic Blindness
confidence 90% · Evaluations on three representative VLA architectures, including Pi0... show that these models frequently succeed at tasks despite logically impossible instructions
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision-Language-Action (VLA) models enable robots to perform manipulation tasks directly from natural language instructions and are increasingly viewed as a foundation for generalist robotic policies. However, their reliability under Out-of-Distribution (OOD) instructions remains underexplored. In this paper, we reveal a critical failure mode in which VLA policies continue executing visually plausible actions even when the language instruction contradicts the scene. We refer to this phenomenon as linguistic blindness, where VLA policies prioritize visual priors over instruction semantics during action generation. To systematically analyze this issue, we introduce ICBench, a diagnostic benchmark constructed from the LIBERO dataset that probes language-action coupling by injecting controlled OOD instruction contradictions while keeping the visual environment unchanged. Evaluations on three representative VLA architectures, including Pi0, Pi0.5 and OpenVLA OFT, show that these models frequently succeed at tasks despite logically impossible instructions, revealing a strong visual bias in action generation. To mitigate this issue, we propose Instruction-Guided Attention Recalibration (IGAR), a train-free inference-time mechanism that rebalances attention distributions to restore the influence of language instructions. IGAR operates without retraining or architectural modification and can be directly applied to existing VLA models. Experiments across 30 LIBERO tasks demonstrate that IGAR substantially reduces erroneous execution under OOD contradictory instructions while preserving baseline task performance. We additionally validate the approach on a real Franka robotic arm, where IGAR effectively prevents manipulation triggered by inconsistent instructions.
Tags
Links
- Source: https://arxiv.org/abs/2603.06001v1
- Canonical: https://arxiv.org/abs/2603.06001v1
Trouble viewing inline? Open PDF directly →
Full Text
41,591 characters extracted from source content.
Expand or collapse full text
Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration Ninghao Zhang 1* , Bin Zhu 2*† , Shijie Zhou 3 , and Jingjing Chen 3 1 Tsinghua University 2 Singapore Management University 3 Fudan University Abstract. Vision-Language-Action (VLA) models enable robots to per- form manipulation tasks directly from natural language instructions and are increasingly viewed as a foundation for generalist robotic policies. However, their reliability under Out-Of-Distribution (OOD) instructions remains underexplored. In this paper, we reveal a critical failure mode in which VLA policies continue executing visually plausible actions even when the language instruction contradicts the scene. We refer to this phe- nomenon as linguistic blindness, where VLA policies prioritize visual priors over instruction semantics during action generation. To systemati- cally analyze this issue, we introduce ICBench, a diagnostic benchmark constructed from the LIBERO dataset that probes language–action cou- pling by injecting controlled OOD instruction contradictions while keep- ing the visual environment unchanged. Evaluations on three represen- tative VLA architectures, including π 0 , π 0.5 , and OpenVLA-OFT, show that these models frequently succeed at tasks despite logically impossi- ble instructions, revealing a strong visual bias in action generation. To mitigate this issue, we propose Instruction-Guided Attention Re- calibration (IGAR), a train-free inference-time mechanism that rebal- ances attention distributions to restore the influence of language instruc- tions. IGAR operates without retraining or architectural modification and can be directly applied to existing VLA models. Experiments across 30 LIBERO tasks demonstrate that IGAR substantially reduces erro- neous execution under OOD contradictory instructions while preserving baseline task performance. We additionally validate the approach on a real Franka robotic arm, where IGAR effectively prevents manipulation triggered by inconsistent instructions. Keywords: Vision-Language-Action Models· Robotic Manipulation· Attention Recalibration 1 Introduction Vision-Language-Action (VLA) models are emerging as a promising paradigm for building generalist robotic policies. By integrating large-scale vision-language models [1,7,22,25] with action generation modules, these systems enable robots 1 * Equal contribution. 2† Corresponding author and project lead. Email: binzhu@smu.edu.sg arXiv:2603.06001v1 [cs.RO] 6 Mar 2026 2N. Zhang et al. Fig. 1: Linguistic blindness in Vision-Language-Action (VLA) models. Under normal instructions (left), the robot completes the task correctly. Under contradictory instructions (right), a structured form of OOD linguistic input, the robot often follows the same visually plausible trajectory while ignoring the instruction. to execute complex manipulation tasks directly from natural language instruc- tions across diverse environments [3,4,14,15,20,23,30,31]. Despite these advances, the reliability of VLA models in real-world deploy- ment remains a critical concern. In safety-critical environments, robots must strictly obey linguistic constraints provided by users. However, we observe that modern VLA models frequently execute visually plausible actions even when the instruction is semantically inconsistent with the scene. As illustrated in Fig. 1, a robot can successfully execute tasks such as “pick up the black bowl” or “put the wine bottle on the cabinet” once a VLA model is well trained. However, when the instruction is modified to contradict the scene, for example, “pick up the white bowl” when no white bowl is present, or “put the wine bottle under the cab- inet” when this is physically impossible, the robot often continues to execute the original visually plausible trajectory while ignoring the instruction. These con- tradictory instructions represent a structured form of out-of-distribution (OOD) linguistic input, revealing a fundamental failure mode that we term linguistic blindness: the tendency of VLA policies to prioritize visual priors over instruc- tion semantics during action generation. This vulnerability poses serious risks for real-world robotics. Unlike conversational AI systems, errors in robotic con- trol directly translate into physical actions that may damage objects, violate safety constraints, or cause hazardous behavior [11,24,28]. Ensuring that robots remain sensitive to language instructions is therefore essential for trustworthy embodied intelligence [29]. However, diagnosing linguistic grounding failures in VLA models is challeng- ing. Existing evaluations primarily measure task success under valid instruc- Restoring Linguistic Grounding in VLAs3 tions [13, 16, 18], which cannot distinguish whether successful execution arises from genuine language grounding or from purely visual heuristics. To address this limitation, we introduce ICBench, a controlled diagnostic benchmark built upon the LIBERO dataset [18]. ICBench isolates the coupling between language and action by injecting structured instruction contradictions while keeping the visual scene unchanged. Specifically, we modify task instructions by altering ob- ject attributes or spatial relations of the target location, as shown in Fig. 1. These modifications create semantically inconsistent instructions that function as con- trolled OOD linguistic perturbations. Under such contradictions, a language- grounded policy should detect the inconsistency and fail to execute the task, whereas a linguistically insensitive policy may still succeed by relying on visual priors. Using ICBench, we conduct a systematic analysis of three representa- tive VLA architectures: π 0 [4], π 0.5 [3], and OpenVLA-OFT [14]. Our evaluation reveals a striking phenomenon: across multiple task suites, VLA models often maintain high success rates even when instructions are logically impossible or inconsistent. This demonstrates that action generation is frequently dominated by visual cues, with language playing only a limited role in guiding behavior. To address this issue, we propose Instruction-Guided Attention Recalibration (IGAR), a train-free intervention that restores linguistic grounding by rebal- ancing attention distributions during the forward pass. IGAR identifies atten- tion sink tokens through hidden-state spike analysis, selects grounding-critical cross-modal heads, and redistributes attention mass toward under-weighted in- struction tokens. Importantly, IGAR requires no gradient updates, no additional training data, and no architectural modification, making it a lightweight plug- and-play module for deployed robotic VLA policies. Extensive experiments on 30 tasks from the LIBERO benchmark demonstrate the effectiveness of IGAR. Un- der OOD contradictory instructions, IGAR significantly reduces erroneous task execution and substantially increases the Linguistic Grounding Score (LGS), in- dicating stronger reliance on instruction semantics. At the same time, IGAR preserves VLA baseline task performance under normal instructions. We further validate our approach on a real Franka robotic arm, where IGAR successfully interrupts the manipulation tasks when contradictory instructions are applied. 2 Related Work 2.1 Vision-Language-Action Models The integration of Vision Language Models (VLMs) with robotic control has ac- celerated the transition from task-specific policies to generalist Vision-Language- Action (VLA) models. Early milestones like RT-1 [5] and RT-2 [31] formulated motor control as sequence-to-sequence generation, a paradigm rapidly scaled by massive robotic datasets [20]. Subsequent models further improve action generation through stronger vision-language backbones and advanced control paradigms. For example, OpenVLA [15] leverage large pretrained VLMs, while architectures like Octo [9], π 0 [4], OpenVLA-OFT [14], RDT [19], GR00T [2] π 0.5 [3] refine continuous control through diffusion and flow-matching paradigms. 4N. Zhang et al. Despite these advances, the internal mechanisms fusing visual and textual modal- ities during action generation remain poorly understood, particularly regarding how language instructions influence control decisions. 2.2 Linguistic Grounding and Modality Bias Multimodal foundation models are known to exhibit strong modality biases that can undermine reliable language grounding. Multimodal foundation models are vulnerable to modality collapse and visual hallucination. Evaluations [17] show Vision-Language Models (VLMs) frequently hallucinate absent objects due to strong textual priors. Conversely, VLMs often exhibit bag-of-words behavior or disregard order and composition information in text [27]. Existing works have also identified attention imbalance in LLMs and VLMs, where certain visual to- kens attract disproportionately high attention regardless of their relevance to the instruction [12,26]. This vulnerability becomes critical in VLA architectures: un- like conversational agents, ignoring linguistically grounded constraints in robotic control can lead to catastrophic physical failures [11]. Recognizing this risk, initial efforts have sought to address linguistic vul- nerabilities in embodied control. To mitigate text-following biases, data-centric strategies like CAST [10] and CounterfactualVLA [21] expand training corpora with synthesized counterfactual scenarios, particularly within navigation and au- tonomous driving. SayCan [6] grounds language in robotic affordances by com- bining large language models with value functions of low-level skills for real-world action selection. In addition, the community has also started probing policy resilience, such as LIBERO-PLUS [8], which evaluates robustness against per- turbations such as camera viewpoints and instructions. However, most existing works focus on task success under valid instructions and do not explicitly disen- tangle whether success arises from genuine language grounding or from visually driven heuristics. In contrast, this paper introduces a controlled diagnostic set- ting that isolates language–action coupling by injecting structured contradictory instructions while keeping the visual scene unchanged. 3 ICBench: A Controlled Instruction Contradiction Benchmark To rigorously evaluate linguistic grounding in Vision-Language-Action (VLA) models, we introduce ICBench, a controlled benchmark that injects out-of-distribution (OOD) semantic contradictions into task instructions while keeping the visual scene and environment dynamics unchanged. The objective of ICBench is di- agnostic rather than adversarial: it probes whether language meaningfully in- fluences action generation. If a model genuinely integrates linguistic constraints into motor planning, contradictory instructions should prevent successful exe- cution. Conversely, if the policy predominantly relies on visual priors, it may continue executing visually valid behaviors despite semantic inconsistencies. Restoring Linguistic Grounding in VLAs5 3.1 Instruction Contradiction Construction Let a standard task be defined by instruction ℓ and environment observation o t . ICBench constructs a modified instruction ̃ ℓ such that: (1) The environment observation o t remains unchanged. (2) ̃ ℓ is semantically incompatible with the scene configuration. By holding the environment fixed, ICBench enables con- trolled measurement of language–action coupling without confounding percep- tion or control quality. Under this setting, the contradictory instruction ̃ ℓ can be viewed as an OOD linguistic perturbation relative to the original task instruc- tion. A language-grounded policy should fail or abstain from execution, while a linguistically insensitive policy may still succeed by exploiting visual priors. Importantly, within ICBench, high task success under contradictory instructions indicates weak linguistic grounding. Therefore, task success rate (SR) serves as a proxy for instruction adherence rather than manipulation capability. 3.2 Contradiction Taxonomy and Design Principles Contradiction Taxonomy. ICBench introduces four structured contradiction types. Each perturbation modifies only a minimal subset of instruction tokens while preserving grammatical plausibility and fluency. These modifications intro- duce controlled OOD linguistic inputs without altering the visual environment. V1: Operand Attribute Substitution. The manipulated object’s descriptive attribute (e.g., colour) is replaced with a non-existent alternative. For instance, “pick up the black bowl”→ “pick up the white bowl”. This tests grounding of object-level attribute semantics. V2: Target Attribute Augmentation. A contradictory attribute is inserted into the target location description. For example, “place it on the plate”→ “place it on the black plate.” This evaluates grounding of relational destination constraints. V3: Dual Attribute Perturbation. Both operand and target attributes are contradicted simultaneously. For example, “put black bowl on the plate”→ “put white bowl on black plate”. This increases semantic inconsistency while main- taining syntactic validity. V4: Spatial Relation Substitution. The spatial preposition is replaced with a contradictory alternative. For instance, “put the block on the table”→ “put the block under the table”. Unlike attribute substitutions, spatial contradictions directly alter trajectory planning, thereby testing deeper relational grounding. Design Principles. ICBench is designed with three key principles: (1) Mini- mal Surface Perturbation. Only a small number of tokens are modified to ensure the instruction remains natural and close to the training distribution. (2) De- terministic Unsatisfiability. Each contradictory instruction is guaranteed to be incompatible with the fixed scene configuration, eliminating ambiguous evalu- ation cases. (3) Different Semantic Complexity. Perturbations range from local attribute mismatches (V1–V3) to spatial inconsistencies (V4), enabling analysis of shallow versus deep linguistic grounding. 6N. Zhang et al. Fig. 2: Overview of the IGAR framework. IGAR is a train-free and plug-and- play intervention that restores linguistic grounding in VLA models via three stages: (1) detecting attention sink tokens through hidden-state spike analysis, (2) selecting grounding heads that exhibit cross-modal imbalance, and (3) redistributing attention from sink tokens to instruction tokens. 4 IGAR: Instruction-Guided Attention Recalibration As illustrated in Fig. 2, we propose Instruction-Guided Attention Recalibration (IGAR), a train-free inference-time mechanism that restores linguistic ground- ing in Vision-Language-Action (VLA) models by correcting vision-dominant at- tention sink [12, 26] imbalance during action prediction. We observe that, in modern VLA architectures, action-query tokens disproportionately attend to visually salient tokens, forming vision-dominant attention sinks that suppress instruction tokens even when the instruction semantically constrains the action. We attribute linguistic blindness to a structural cross-modal competition im- balance induced by attention sink dynamics. IGAR addresses this imbalance by detecting attention sinks through hidden-state spike analysis, selectively target- ing cross-modal fusion heads, and recalibrating attention mass toward instruc- tion tokens during inference. Our method operates entirely within the forward pass, requires no gradient updates or retraining, does not modify model param- eters, and can be seamlessly integrated into transformer-based VLA backbones without architectural changes, making it suitable as a plug-and-play module for deployed robotic policies. 4.1 Problem Formulation Instruction Contradiction Setting. A VLA model f θ maps observation o t and a language instruction ℓ to action a t = f θ (o t ,ℓ). Under ICBench, we evaluate the model using a semantically contradictory instruction ̃ ℓ, while keeping the observation unchanged. By construction, ̃ ℓ is incompatible with the visual scene Restoring Linguistic Grounding in VLAs7 and cannot be satisfied through valid task execution. This setup isolates the role of language instruction grounding in action generation. A policy that genuinely integrates instruction semantics should fail or abstain under contradiction. A policy that relies primarily on visual priors may still succeed despite the semantic inconsistency. Linguistic Grounding Score. Let SR(f θ ,ℓ) denote the task success rate of policy f θ when executing under standard instruction ℓ. We define the Linguistic Grounding Score (LGS) as: LGS( ̃ ℓ) = SR(f θ ,ℓ)− SR(f θ , ̃ ℓ),(1) where SR(f θ , ̃ ℓ) is the success rate respect to the standard instruction when exe- cuting the task using contradictory instruction ̃ ℓ. As a result, a perfect grounded model should fail under contradiction, yielding LGS( ̃ ℓ) = SR(f θ ,ℓ). A linguisti- cally insensitive model that ignores instruction semantics will succeed regardless of contradiction, yielding LGS≈0. Instruction Visual Attention Ratio. To quantify instruction influence during action decoding, let V and T denote the visual and text token sets. Let ̄ A be head-averaged attention matrix defined as follows: ̄ A = 1 H H X h=1 A h ,(2) where H denotes the number of attention heads in the final transformer layer of VLA. For the action-predicting query position s, Instruction Visual Attention Ratio (IVAR) is formalized as follows: IVAR = P j∈T ̄ A s,j P j∈V∪T ̄ A s,j .(3) Low IVAR under contradiction signals attention sink imbalance suppressing linguistic tokens. IGAR aims to restore instruction influence while preserving baseline execution performance. 4.2 Instruction-Guided Attention Recalibration IGAR operates within the model’s forward pass by selectively recalibrating at- tention distributions in the transformer layers. Rather than globally modifying attention weights, IGAR intervenes only in structurally imbalanced heads iden- tified through attention sink analysis. The procedure consists of three stages: (1) attention sink token detection, (2) grounding head selection, and (3) attention redistribution. Attention Sink Token Detection. Let H ∈R N×D denote the hidden states from an intermediate transformer layer, where N is the sequence length and D the hidden dimension. 8N. Zhang et al. We first compute the RMS norm per token: r i = v u u t 1 D D X d=1 H 2 i,d , i = 1,...,N.(4) However, large token norms alone do not necessarily indicate structural domi- nance. To detect localized extreme activations, which is characteristic of atten- tion sinks, we introduce a spike ratio per feature dimension: φ(d) = max i |H i,d | mean i |H i,d | + ε ,(5) where ε is a small constant for numerical stability. We select the top-k dimensions (k=5) from the set D spike = d : φ(d) > γ with γ=3.0. This spike criterion ensures that only dimensions with localized extreme activations (few tokens with high values, not all tokens uniformly high) are selected, avoiding false-positive sink identification. A token i is then classified as a sink token if: S = i : max d∈D spike |H i,d | > τ , τ = 20.(6) We partition sinks into visual sinks S V =S∩V and text sinks S T =S∩T . Grounding Head Selection. Not all attention heads are equally susceptible to sink distortion. For each head h in layer l and each query position q beyond the image region, we evaluate two conditions: c 1 (h,q) : P j∈S V A h q,j P j∈V A h q,j + ε ≤ ρ,(7) c 2 (h,q) : X j∈V A h q,j ≥ α.(8) A head-query pair (h,q) is selected for reallocation if both conditions hold, with ρ=0.4 and α=0.01. Condition c 1 ensures the head is not already dominated by visual sinks (structural attention rather than semantic), while c 2 ensures the head allocates meaningful attention to visual tokens (filtering out text-only heads). When no visual sinks are detected (i.e., S V =∅), c 1 is trivially satisfied and selection proceeds on c 2 alone. Attention Redistribution. For each selected head-query pair (h,q), IGAR redistributes attention from sink tokens to non-sink tokens by first scaling down the attention of sink tokens by a factor p = 0.6. This operation frees a total budget Ω h q = (1− p) X j∈S T A h q,j (9) Restoring Linguistic Grounding in VLAs9 which is then reallocated to non-sink tokens proportionally to their original attention weights: A ′ h q,j = A h q,j + Ω h q · A h q,j P j ′ ∈T ns A h q,j ′ + ε , ∀j ∈T ns (10) where T ns =T T denotes the set of non-sink text tokens. 5 Experiments We evaluate the proposed Instruction-Guided Attention Recalibration (IGAR) under the controlled OOD instruction contradiction setting introduced by ICBench to answer three key questions: – Q1: Do current Vision-Language-Action (VLA) models exhibit linguistic blindness under contradictory instructions? – Q2: Can IGAR restore language grounding during action generation? – Q3: Does IGAR preserve performance on valid in-distribution tasks? 5.1 Experimental Setup VLAs and Benchmark. We evaluate three representative VLA architectures (π 0 [4], π 0.5 [3], and OpenVLA-OFT [14]) across 30 simulated robot manipula- tion tasks from the LIBERO benchmark [18], covering Spatial, Object, and Goal suites. For each task variant, we perform 50 independent rollouts. These environ- ments cover diverse robotic manipulation settings, including spatial reasoning, object manipulation, and goal-conditioned tasks. Our ICBench is constructed by introducing controlled contradictory instructions into the LIBERO tasks. Evaluation Metrics. We report two primary evaluation metrics. Task Suc- cess Rate (SR) measures the percentage of successful task completions. Under standard instructions, a high SR indicates correct task execution and is therefore desirable. However, under the ICBench contradiction setting, a high SR becomes undesirable, as it implies that the policy completes the task despite contradictory instructions, suggesting that the model relies primarily on visual priors rather than instruction semantics. To quantify the influence of language instructions on action generation, we adopt the Linguistic Grounding Score (LGS) defined in Sec. 4.1. LGS measures the performance gap between executions under valid in- structions and contradictory instructions, with higher values indicating stronger reliance on linguistic instructions. Implementation Details. IGAR is applied uniformly at inference time with- out any task-specific tuning. For attention-based decoders (π 0.5 and OpenVLA- OFT), interventions span the initial 16 layers using unified constraints for visual- sink portion boundaries (ρ=0.4) and text-sink decay (p=0.6), alongside fixed detection thresholds (τ =20, γ=3.0). For π 0 ’s continuous flow-matching diffu- sion, we deploy a discrete contrastive formulation that restricts divergence trig- gers (τ detect =0.15) and interpolation boundaries (τ max =0.50). This standardized configuration isolates the inherent robustness of IGAR across fundamentally dis- parate generation paradigms. 10N. Zhang et al. Table 1: Linguistic Blindness Diagnosis on ICBench. Success Rate (SR, %) and Linguistic Grounding Score (LGS) under normal and contradictory instructions. High SR and low LGS under contradiction (V1-V4) indicate that the policy ignores instruction semantics and follows visual priors. Highlighted cells denote SR≥ 90%, revealing severe linguistic blindness. π 0 π 0.5 OpenVLA-OFT Suite InstructionSR LGSSR LGSSRLGS Spatial Normal96.8 –97.4 –97.6– V1 (operand attribute)90.4 6.496.2 1.297.8-0.2 V2 (target attribute) 96.2 0.697.8 -0.496.41.2 V3 (dual attribute)89.8 7.096.4 1.096.21.4 V4 (spatial relation) 92.4 4.497.6 -0.292.45.2 Object Normal98.8 –98.4 –98.4– V1 (operand attribute)93.8 5.096.2 2.297.80.6 V2 (target attribute)90.2 8.690.4 8.096.22.2 V3 (dual attribute)98.2 0.696.4 2.099.8-1.4 V4 (spatial relation)91.6 7.288.2 10.299.6-1.2 Goal Normal95.8 –97.6 –98.0– V1 (operand attribute)90.2 5.693.8 3.897.80.2 V2 (target attribute)84.4 11.490.4 7.298.2-0.2 V3 (dual attribute)88.2 7.694.2 3.498.4-0.4 V4 (spatial relation)76.4 19.493.6 4.090.27.8 5.2 Diagnosing Linguistic Blindness Table 1 presents the VLA baseline evaluation under ICBench, comparing perfor- mance under normal instructions and contradictory instructions. Across all three VLAs, we observe a consistent pattern: modern VLA models frequently ignore contradictory language inputs and continue executing visually plausible actions, revealing a systemic form of linguistic blindness. First, models often maintain extremely high success rates even when the instruction is semantically invalid. Across the Spatial and Object suites, success rates frequently exceed 90% under contradictory instructions, particularly for π 0.5 and OpenVLA-OFT. This indi- cates that the policy continues to complete the task by relying on visual cues rather than instruction semantics. As a consequence, the resulting LGS remain very low, suggesting that language contributes minimally to action generation in these scenarios. Second, the severity of linguistic blindness varies across architec- tures. π 0 shows relatively higher LGS values compared to π 0.5 and OpenVLA- OFT, indicating slightly stronger language grounding. In contrast, π 0.5 often exhibits low LGS values, implying that contradictory instructions have almost no influence on the executed actions. These results demonstrate that despite their multimodal design, current VLA models often rely on vision-dominant ex- ecution strategies. Language instructions exert limited influence, though they Restoring Linguistic Grounding in VLAs11 directly constrain the physical feasibility of the action, highlighting a critical vulnerability for deploying VLA systems in real-world settings. 5.3 IGAR Restores Linguistic Grounding Fig. 3: Attention visualization of OpenVLA-OFT with and without IGAR. We visualize cross-modal attention maps under normal and contradictory instructions. The baseline policy attends primarily to salient regions regardless of instruction se- mantics, while IGAR redistributes attention toward instruction-relevant objects and spatial regions, mitigating visual attention sinks and improving linguistic grounding. Table 2 reports the performance of VLA models under contradictory instruc- tions after applying IGAR. Compared with the baseline results in Table 1, IGAR consistently reduces erroneous task execution while substantially increasing the LGS, indicating that action generation becomes more sensitive to instruction semantics. First, IGAR effectively suppresses visually driven execution under contradictory instructions. Across all suites, SR drops significantly compared to the baseline setting, particularly for the π 0 and OpenVLA-OFT. For example, in the Goal suite, SR decreases to as low as 36.4% under spatial contradictions (V4), while LGS increases dramatically to 59.4. This indicates that the policy refrains from executing actions that violate instruction semantics, demonstrating restored language grounding. Second, the improvement varies across model archi- tectures and suites. The largest gains are observed for π 0 , where LGS consistently exceeds 40 and reaches up to 59.4 in goal suites. OpenVLA-OFT also shows sub- stantial improvements, achieving LGS values above 30 in several goal tasks. In 12N. Zhang et al. contrast, the π 0.5 model exhibits more limited gains, showing strong reliability on visual cues. Overall, the results demonstrate that IGAR effectively restores the influence of language instructions during action generation. By redistribut- ing attention toward instruction tokens, the method mitigates vision-dominant biases and enables VLA policies to better detect and respond to semantically inconsistent instructions. Table 2: IGAR mitigation under ICBench. We evaluate VLA models under con- tradictory instructions after applying IGAR. SR measures task completion despite contradiction (lower is better), while LGS measures the influence of language instruc- tions (higher is better). Green cells indicates LGS≥ 10. π 0 π 0.5 OpenVLA-OFT Suite. ContradictionSR LGSSR LGSSRLGS Spatial V176.420.495.8 1.686.411.2 V2 84.212.693.6 3.886.211.4 V380.416.494.8 2.690.27.4 V476.220.699.6 -2.288.49.2 Object V188.210.694.2 4.288.410.0 V286.412.490.4 8.084.214.2 V3 93.6 5.287.610.888.210.2 V490.4 8.488.410.082.416.0 Goal V146.449.490.2 7.466.431.6 V240.255.682.814.870.227.8 V346.249.692.4 5.264.233.8 V436.459.496.2 1.458.439.6 5.4 Baseline Preservation Under IGAR Table 3 evaluates whether IGAR affects performance under normal (non-contradictory) instructions. We compare the VLA baseline success rates with results obtained when IGAR is applied to the same tasks. Overall, IGAR preserves baseline per- formance with only marginal impact. Across all suites, the π 0 model shows neg- ligible performance changes, with an average decrease of only 0.4%. Similarly, OpenVLA-OFT maintains almost identical performance, with an average change of +0.5%. These results indicate that the attention recalibration introduced by IGAR does not interfere with correct instruction following when the language input is consistent with the visual scene. 5.5 Ablation Study: Hyperparameter Sensitivity We systematically analyze the influence of key hyperparameters governing IGAR’s intervention constraints using OpenVLA-OFT on the libero_goal benchmark. As detailed in Figure 4, we evaluate the LGS to measure the effectiveness of Restoring Linguistic Grounding in VLAs13 Table 3: Baseline preservation under IGAR. Success Rate (SR) comparison un- der normal (non-contradictory) instructions for baseline and baseline with our IGAR. "OFT" refers to "OpenVLA-OFT". Suitesπ 0 π 0 +IGARπ 0.5 π 0.5 +IGAROFT OFT+IGAR Spatial96.8 96.4 (−0.4)97.498.2 (+0.8)97.6 97.6 (+0.0) Object 98.8 98.2 (−0.6)98.4 94.4 (−4.0)98.499.8 (+1.4) Goal95.8 95.6 (−0.2)97.6 96.4 (−1.2)98.098.2 (+0.2) Average97.1 96.7 (−0.4)97.8 96.3 (−1.5)98.0 98.5 (+0.5) Fig. 4: Hyperparameter Sensitivity. Impact of the text-sink decay factor (p), head selection bound (ρ), and number of intervened layers (L) in terms of linguistic ground- ing performance (LGS). Results are reported using the OpenVLA-OFT architecture on the libero_goal benchmark suite. The dashed lines indicate the selected values. language-action grounding. Applying an overly aggressive text-sink decay factor (p < 0.3) or an insufficient one (p > 0.8) disrupts the optimal reinforcement of instructional tokens, directly reducing the overall LGS. Similarly, excessively loose visual-sink bounds (ρ > 0.6) fail to adequately intercept structural visual heads responsible for attention sinks, while overly strict boundaries interfere with normal processing. Furthermore, intervening across too many transformer layers (L > 24) tends to corrupt deep representations that are highly specialized for continuous action regression, whereas intervening on too few layers fails to provide a strong enough corrective signal. Our default configurations (p = 0.6, ρ = 0.4, L = 16) strategically target the mid-level layers where cross-modal semantic fusion primarily occurs, fully maximizing the LGS and effectively mit- igating linguistic blindness. 5.6 Real-World Evaluation We further validate IGAR on a real robotic platform to examine whether the lin- guistic grounding improvements observed in simulation transfer to physical ma- nipulation. As shown in Fig. 5, experiments are conducted on a Franka Research 3 robotic arm equipped with two Intel RealSense D435 cameras, one mounted on the wrist and another providing a third-person viewpoint. We evaluate the π 0 policy on a cube placement task: “place the blue cube into the open drawer”. Under normal instructions, both the π 0 policy and π 0 +IGAR reliably complete the task, confirming that IGAR preserves the original policy behavior when the 14N. Zhang et al. Fig. 5: Real-world experiments. We test the π 0 policy with and without IGAR on the task "placing the blue cube into the open drawer". Under normal instructions (left), both policies successfully place the blue cube into the open drawer. Under contradictory instructions (right), the π 0 policy still executes a visually plausible trajectory and produces a fake success. In contrast, IGAR restores linguistic grounding and prevents incorrect task execution, resulting in a deserved failure. instruction is consistent with the scene. However, when the instruction becomes contradictory (e.g., requesting a non-existent object attribute or an impossible spatial relation), the baseline policy frequently continues executing the visually plausible trajectory and completes the manipulation despite the semantic in- consistency. This behavior results in fake success, where the robot performs the correct physical action but violates the instruction semantics. In contrast, IGAR restores the influence of language during action generation. Given contradictory instructions, the recalibrated policy refrains from completing the task and in- stead produces safe behaviors such as hovering or empty grasp attempts, effec- tively interrupting the manipulation. These outcomes represent deserved failures, indicating that the policy correctly detects the instruction inconsistency. 6 Conclusion In this paper, we have revealed a critical reliability issue in Vision-Language- Action (VLA) models that we term linguistic blindness, where policies prioritize visual priors over instruction semantics during action generation. To system- atically diagnose this failure mode, we introduce ICBench, a controlled bench- mark that evaluates language–action grounding under structured contradictory instructions. We further propose Instruction-Guided Attention Recalibration (IGAR), a train-free inference-time intervention that restores linguistic ground- ing by redistributing attention from sink tokens to instruction tokens. Experi- ments on three representative VLA architectures demonstrate that modern mod- els often execute visually plausible actions even when instructions are logically inconsistent with the scene, while IGAR effectively mitigates such failures with- out degrading baseline task performance. Restoring Linguistic Grounding in VLAs15 References 1. Beyer, L., Steiner, A., Pinto, A.S., Kolesnikov, A., Wang, X., Salz, D., Neumann, M., Alabdulmohsin, I., Tschannen, M., Bugliarello, E., et al.: Paligemma: A ver- satile 3b vlm for transfer. arXiv preprint arXiv:2407.07726 (2024) 2. Bjorck, J., Castañeda, F., Cherniadev, N., Da, X., Ding, R., Fan, L., Fang, Y., Fox, D., Hu, F., Huang, S., et al.: Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734 (2025) 3. Black, K., Brown, N., Darpinian, J., Dhabalia, K., Driess, D., Esmail, A., Equi, M.R., Finn, C., Fusai, N., Galliker, M.Y., Ghosh, D., Groom, L., Hausman, K., brian ichter, Jakubczak, S., Jones, T., Ke, L., LeBlanc, D., Levine, S., Li-Bell, A., Mothukuri, M., Nair, S., Pertsch, K., Ren, A.Z., Shi, L.X., Smith, L., Springen- berg, J.T., Stachowicz, K., Tanner, J., Vuong, Q., Walke, H., Walling, A., Wang, H., Yu, L., Zhilinsky, U.: Pi0.5: a vision-language-action model with open-world generalization. In: 9th Annual Conference on Robot Learning (2025) 4. Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al.: pi0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164 (2024) 5. Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakr- ishnan, K., Hausman, K., Herzog, A., Hsu, J., et al.: Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 (2022) 6. Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., et al.: Do as i can, not as i say: Grounding language in robotic affordances. In: Conference on robot learning. p. 287–318. PMLR (2023) 7. Driess, D., Xia, F., Sajjadi, M.S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al.: Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 (2023) 8. Fei, S., Wang, S., Shi, J., Dai, Z., Cai, J., Qian, P., Ji, L., He, X., Zhang, S., Fei, Z., et al.: Libero-plus: In-depth robustness analysis of vision-language-action models. arXiv preprint arXiv:2510.13626 (2025) 9. Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., et al.: Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213 (2024) 10. Glossop, C., Chen, W., Bhorkar, A., Shah, D., Levine, S.: Cast: Counterfactual la- bels improve instruction following in vision-language-action models. arXiv preprint arXiv:2508.13446 (2025) 11. Guiochet, J., Machin, M., Waeselynck, H.: Safety-critical advanced robots: A sur- vey. Robotics and Autonomous Systems 94, 43–52 (2017) 12. Kang, S., Kim, J., Kim, J., Hwang, S.J.: See what you are told: Visual attention sink in large multimodal models. In: The Thirteenth International Conference on Learning Representations (2025) 13. Khazatsky, A., Pertsch, K., Nair, S., Balakrishna, A., Dasari, S., Karamcheti, S., Nasiriany, S., Srirama, M.K., Chen, L.Y., Ellis, K., et al.: Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945 (2024) 14. Kim, M.J., Finn, C., Liang, P.: Fine-tuning vision-language-action models: Opti- mizing speed and success. arXiv preprint arXiv:2502.19645 (2025) 15. Kim, M.J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E.P., Sanketi, P.R., Vuong, Q., et al.: Openvla: An open-source vision-language-action model. In: 8th Annual Conference on Robot Learning (2024) 16N. Zhang et al. 16. Li, C., Zhang, R., Wong, J., Gokmen, C., Srivastava, S., Martín-Martín, R., Wang, C., Levine, G., Lingelbach, M., Sun, J., et al.: Behavior-1k: A benchmark for em- bodied ai with 1,000 everyday activities and realistic simulation. In: Conference on Robot Learning. p. 80–93. PMLR (2023) 17. Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.R.: Evaluating object hal- lucination in large vision-language models. In: Proceedings of the 2023 conference on empirical methods in natural language processing. p. 292–305 (2023) 18. Liu, B., Zhu, Y., Gao, C., Feng, Y., Liu, Q., Zhu, Y., Stone, P.: Libero: Benchmark- ing knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems 36, 44776–44791 (2023) 19. Liu, S., Wu, L., Li, B., Tan, H., Chen, H., Wang, Z., Xu, K., Su, H., Zhu, J.: Rdt- 1b: a diffusion foundation model for bimanual manipulation. In: The Thirteenth International Conference on Learning Representations (2025) 20. O’Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., et al.: Open x-embodiment: Robotic learn- ing datasets and rt-x models. In: 2024 IEEE International Conference on Robotics and Automation (ICRA). p. 6892–6903. IEEE (2024) 21. Peng, Z., Ding, W., You, Y., Chen, Y., Luo, W., Tian, T., Cao, Y., Sharma, A., Xu, D., Ivanovic, B., et al.: Counterfactual vla: Self-reflective vision-language-action model with adaptive reasoning. arXiv preprint arXiv:2512.24426 (2025) 22. Team, G., Anil, R., Borgeaud, S., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., Millican, K., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023) 23. Team, G.R., Abeyruwan, S., Ainslie, J., Alayrac, J.B., Arenas, M.G., Armstrong, T., Balakrishna, A., Baruch, R., Bauza, M., Blokzijl, M., et al.: Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:2503.20020 (2025) 24. Team, G.R., Choromanski, K., Devin, C., Du, Y., Dwibedi, D., Gao, R., Jindal, A., Kipf, T., Kirmani, S., Leal, I., et al.: Evaluating gemini robotics policies in a veo world simulator. arXiv preprint arXiv:2512.10675 (2025) 25. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bash- lykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023) 26. Xiao, G., Tian, Y., Chen, B., Han, S., Lewis, M.: Efficient streaming language models with attention sinks. In: The Twelfth International Conference on Learning Representations (2024) 27. Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., Zou, J.: When and why vision-language models behave like bags-of-words, and what to do about it? In: The Eleventh International Conference on Learning Representations (2023) 28. Zacharaki, A., Kostavelis, I., Gasteratos, A., Dokas, I.: Safety bounds in human robot interaction: A survey. Safety science 127, 104667 (2020) 29. Zhang, B., Zhang, Y., Ji, J., Lei, Y., Dai, J., Chen, Y., Yang, Y.: Safevla: Towards safety alignment of vision-language-action model via constrained learning. arXiv preprint arXiv:2503.03480 (2025) 30. Zhang, S., Xu, Z., Liu, P., Yu, X., Li, Y., Gao, Q., Fei, Z., Yin, Z., Wu, Z., Jiang, Y.G., et al.: Vlabench: A large-scale benchmark for language-conditioned robotics manipulation with long-horizon reasoning tasks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. p. 11142–11152 (2025) 31. Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al.: Rt-2: Vision-language-action models transfer web knowledge to robotic control. In: Conference on Robot Learning. p. 2165–2183 (2023)