Paper deep dive
What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies
Vivek Chavan, Pengtao Xie, Yahuan Shi, Oliver Heimann, Kevin Haninger, Jörg Krüger
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/8/2026, 2:21:18 AM
Summary
This paper investigates conditional visual grounding failures in visuomotor imitation policies, where policies fail to select correct visual targets (objects or receptacles) in the presence of visually similar distractors, despite retaining underlying manipulation skills. Using Action Chunking with Transformers (ACT) and a Vision-Language-Action (VLA) model, the authors diagnose that failures are specific to manipulation phases (picking vs. placement) and visual similarity types (color vs. shape). They propose interventions including distractor augmentation, phase-dependent attention regularization, and appearance-based visual prompting, which significantly improve robustness in simulation and on a physical UR3e robot.
Entities (9)
Relation Signals (7)
UR3e → usedfor → hardware evaluation
confidence 95% · These interventions substantially improve robustness in simulation and on a physical UR3e.
ACT → usedin → conditional visual grounding diagnosis
confidence 95% · Using Action Chunking with Transformers (ACT), we systematically introduce distractor objects and receptacles... and localize failures
Visual Distractors → causesfailurein → ACT
confidence 92% · visuomotor imitation policies can achieve high performance... yet fail when visually similar objects or receptacles are introduced.
Phase-Dependent Attention Regularization → improvesrobustnessof → ACT
confidence 90% · ACT-Modified reaches 94.5% and 88.5% on Tasks 1 and 2... These interventions substantially improve robustness
Distractor Augmentation → improvesrobustnessof → ACT
confidence 90% · Augmentation alone reaches 100.0% on Task 1 and 64.0% on Task 2... showing that substantial robustness can be recovered
Visual Prompting → improvesrobustnessof → ACT
confidence 90% · ACT-Modified... appearance-based visual prompting... substantially improve robustness
$\\pi_{0.5}$ → exhibits → conditional visual grounding failures
confidence 88% · We further examine the same failure pattern in a pretrained vision-language-action policy... where the observed state of a medical instrument determines the correct destination.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Visuomotor imitation policies can achieve high performance under in-distribution visual conditions yet fail when visually similar objects or receptacles are introduced. We study this behavior as a problem of conditional visual grounding: the visual target required for successful control changes with the manipulation phase and, in more complex tasks, with the observed task state. Using Action Chunking with Transformers (ACT), we systematically introduce distractor objects and receptacles with controlled color and shape similarity and localize failures to picking and placement. We find that distractor sensitivity is specific to both the type of visual similarity and the manipulation stage. Guided by this diagnosis, we evaluate distractor augmentation, phase-dependent attention regularization, and appearance-based visual prompting as complementary interventions for improving target selection while preserving spatial information required for control. These interventions substantially improve robustness in simulation and on a physical UR3e. We further examine the same failure pattern in a pretrained vision-language-action policy on a state-conditioned instrument-handling task, where the observed state of a medical instrument determines the correct destination. Together, the results show that visual distractors can cause incorrect object or destination selection even when the underlying manipulation skill remains intact, and that explicitly improving target selection can substantially recover performance across distinct visuomotor policy-learning regimes.
Tags
Links
- Source: https://arxiv.org/abs/2609.05376v1
- Canonical: https://arxiv.org/abs/2609.05376v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
20,556 characters extracted from source content.
Expand or collapse full text
What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies Vivek Chavan 1,2,∗ Pengtao Xie 2,∗,† Yahuan Shi 2,† Oliver Heimann 1 Kevin Haninger 1,‡ J ̈ org Kr ̈ uger 1,2,‡ 1 Fraunhofer Institute for Production Systems and Design Technology IPK 2 Technische Universit ̈ at Berlin ∗ Equal contribution. † Work conducted as part of student projects at Fraunhofer IPK. ‡ Supervision Abstract—Visuomotor imitation policies can achieve high per- formance in curated environments yet fail when visually sim- ilar objects compete with task-relevant entities. We study this behavior as a problem of conditional visual grounding: which visual entity matters depends on the current manipulation phase and, in more complex tasks, on the inferred task state. Using Action Chunking with Transformers (ACT), we systematically vary object and receptacle competitors and localize failures to picking and placement. The resulting errors are cue- and phase-specific and are accompanied by corresponding changes in learned visual representations. Guided by this diagnosis, we evaluate distractor augmentation, phase-dependent attention reg- ularization, and appearance-based visual prompting as comple- mentary interventions for strengthening task-relevant grounding while preserving spatial information required for control. These interventions substantially improve robustness in simulation and on a physical UR3e. We further examine the same diagnosis- intervention principle in a pretrained vision-language-action policy on a state-conditioned instrument-handling task, where the correct destination depends on the observed instrument state. Together, the results support conditional visual grounding as a useful framework for diagnosing and improving robustness across distinct visuomotor policy-learning regimes. 8 Page paper! Index Terms—Robot Imitation Learning, Visual Grounding, Visual Distractors, ACT, Vision-Language-Action Models. I. INTRODUCTION Vision-based imitation policies are commonly trained and evaluated under curated visual conditions, while deployment scenes can contain visually similar objects, receptacles, and irrelevant clutter [1], [2]. Adding such competitors changes neither the manipulation objective nor the underlying robot dynamics, yet can substantially reduce policy success [2], [3]. Aggregate success alone, however, cannot distinguish between a policy that has lost the ability to execute the manipulation and one that retains the motor behavior but applies it to the wrong entity. We study this distinction as conditional visual grounding. Given candidate referents R t , an instruction ℓ, and execu- tion context c t , correct action requires selecting a context- dependent referent r ⋆ t ∈ R t . Crucially, the relevant referent is not necessarily static. For Action Chunking with Trans- formers (ACT) [4], we examine phase-conditioned relevance: the manipulated object is critical during target acquisition and grasping, whereas the receptacle becomes critical during placement. In our vision-language-action (VLA) case study, relevance is additionally state-conditioned: the observed state of a medical instrument determines which receptacle is the correct destination. We ask whether visual failures can be localized to the cue, referent, and execution stage at which grounding becomes incorrect, and whether this diagnosis suggests effective in- terventions. Controlled ACT experiments first expose distinct object- and receptacle-selection bottlenecks. We then evaluate lightweight object-centric interventions and validate their ef- fect in simulation and on a physical UR3e. Finally, we examine whether the same diagnosis-intervention principle remains useful in a pretrained VLA policy on a state-conditioned routing task. I. RELATED WORK Visuomotor imitation policies are vulnerable to spurious observation-action correlations [5], and recent work has sys- tematically quantified generalization failures under changes in object appearance, scene configuration, and visual dis- tractors [1]. In particular, Causal-ACT shows that ACT can exploit irrelevant distractor correlations [3], while ImitDiff and DRAIL improve robustness through semantic guidance and task-relevant region-aware representations, respectively [2], [6]. Complementary approaches provide explicit spatial or visual guidance through grounding masks and visual prompts [7], [8]. Visual robustness is likewise unresolved in pretrained vision- language-action policies: BYOVLA demonstrates sensitivity to task-irrelevant visual content and mitigates it through run- time observation interventions [9], while recent work explicitly supervises task-relevant factors in VLA action representa- tions [10]. Our focus is complementary; we jointly diagnose which visual cues induce incorrect grounding, which task referent is affected, and when the resulting error becomes behaviorally consequential, and use this diagnosis to moti- vate interventions across distinct visuomotor policy-learning regimes. arXiv:2609.05376v1 [cs.RO] 4 Sep 2026 I. METHOD a) Controlled diagnosis with ACT.: We study two sim- ulated pick-and-place tasks designed to separate target and destination grounding. In Task 1, a randomly placed object is moved to a fixed receptacle; in Task 2, both object and re- ceptacle positions are randomized, requiring visual localization during both picking and placement. ACT is trained from 100 clean scripted demonstrations per task. At evaluation, held-out object and receptacle competitors match the target in color, shape, or neither, with one to three competitors per class. The full mixed condition contains one competitor from each object and receptacle class. In addition to end-to-end success, we measure P(pick), P(lift | pick), and P(place | pick, lift), allowing failures to be localized within the manipulation sequence. Simulation results aggregate three training seeds, four evaluation seeds, and 50 closed-loop rollouts per seed. b) Diagnosis-driven intervention.: The diagnosis moti- vates three complementary mechanisms targeting incorrect grounding. Image-space copy-paste augmentation inserts syn- thetic competitors into otherwise clean demonstrations, re- ducing the reliability of incidental appearance correlations. Phase-dependent attention regularization supervises one de- coder cross-attention head toward the currently relevant object or receptacle and away from synthetic distractors. Appearance- based visual prompting provides positionless crops of the target object and receptacle, with a jointly trained phase predictor selecting the appropriate prompt during execution. Figure 1 summarizes the resulting ACT-Modified architecture. Let A denote the supervised head’s normalized attention, L φ the phase-prediction loss, M φ the phase-relevant mask, and M d the distractor mask. Training uses L =L ACT + λ φ L φ + λ + (1−⟨A,M φ ⟩) + λ − ⟨A,M d ⟩. (1) The visual crop specifies target appearance but contains no ex- plicit full-frame coordinates. Simulation crops and masks use privileged segmentation, while hardware prompts are obtained from an initial manual annotation followed by SAM 2 segmen- tation [11]. Because prompt-bearing policies receive additional target information, augmentation-only and augmentation-plus- attention variants are evaluated separately to isolate robustness gains that do not depend on explicit target specification. c) Pretrained-VLA case study.: To examine whether the same grounding perspective remains useful beyond task- specific ACT policies, we fine-tune π 0.5 [12] on 231 tele- operated episodes of a seven-step instrument-handling pro- cedure observed through fixed-base and wrist cameras. Five subgoals are evaluated, including two spatially ambiguous routing subgoals in which the correct destination depends on the observed state of the medical instrument. We compare a prompt-trained checkpoint with and without its expected RGB cue and a regularized checkpoint evaluated on clean RGB input. Five trials per subgoal and condition yield 75 trials. For the routing subgoals, the prompt variants receive the generic destination instruction highlighted, whereas the regularized variant receives explicit left/right destination text; the conditions are therefore not directly comparable. TABLE I END-TO-END SUCCESS UNDER FULL MIXED DISTRACTORS. SIMULATION CONTAINS 600 ROLLOUTS PER CELL; HARDWARE CONTAINS 20 TRIALS PER CELL. T1 USES A FIXED RECEPTACLE AND T2 A RANDOMIZED RECEPTACLE. Simulation (%)UR3e (%) MethodT1T2T1T2 Standard ACT39.514.00.00.0 Augmentation only100.064.0-- ACT-Modified94.588.565.060.0 IV. EXPERIMENTS AND RESULTS a) ACT failures are cue-specific and stage-localized.: Standard ACT reaches 98.5% and 99.7% success in clean Tasks 1 and 2, confirming that both manipulation routines are learned. Under three color-matched object competitors in Task 2, P(pick) falls to 39.2%, while P(lift | pick) and P(place | pick, lift) remain 93.8% and 95.5%. In contrast, two shape-matched receptacle competitors leave P(pick) at 97.5% but reduce conditional placement to 33.9%. Within the evaluated assets, object selection is therefore most sensitive to color, whereas receptacle selection is most sensitive to shape. The effect increases with competitor count and compounds when both bottlenecks are active: under full mixed distractors, end-to-end success falls to 39.5% on Task 1 and 14.0% on Task 2. The preservation of downstream conditional success after correct selection indicates that much of the degradation arises from incorrect grounding rather than loss of the learned manipulation routine. b) Object-centric interventions recover robustness.: Ta- ble I summarizes the central behavioral results under full mixed distractors. Augmentation alone reaches 100.0% on Task 1 and 64.0% on Task 2 without explicit target prompts, showing that substantial robustness can be recovered by disrupting spurious appearance correlations during training. Its remaining Task 2 deficit is concentrated in randomized- receptacle localization. ACT-Modified reaches 94.5% and 88.5% on Tasks 1 and 2, respectively. On the physical UR3e, standard and modified ACT remain comparable in clean scenes: 17/20 versus 17/20 successes on Task 1 and 16/20 versus 15/20 on Task 2. Under mixed distractors, standard ACT fails all tested trials, whereas ACT- Modified succeeds in 13/20 and 12/20 trials. Given the sample size, these experiments establish that the behavioral ordering observed in simulation persists on hardware rather than re- producing the complete mechanism analysis in the physical setting. c) Robust representations require invariance without loss of task geometry.: To examine what changes inside the policy, we compare standard ACT, augmentation-only ACT, and ACT- Modified at the vision encoder, transformer memory, and decoder state. Matched clean and full-distractor activations are evaluated using cosine shift, pick/place centroid separation, and Task 2 left/right container structure. Transformer Encoder Transformer Encoder Transformer Decoder Phase Predictor CLS ... Learned Query ... Learned Query ... Predicted Action Chunk ... GT Action Chunk Cross-Attention Phase-Gated Selection Vision Encoder Vision Encoder Vision Encoder Cross-Attention ... Robot Proprioception Prompt 1 Full View Prompt 2 Only for Training Training & Inference Addition with Positional Embedding Copy-Paste Augmentation Attention Regularization Fig. 1. ACT-Modified architecture. The full camera observation and phase-selected visual prompt are encoded into the visual memory consumed by the ACT transformer decoder. A phase predictor selects the object or receptacle prompt according to the current manipulation phase, while robot proprioception (qpos) provides state conditioning. The decoder combines visual memory, learned queries, cross-attention, and positional embeddings to predict an action sequence. The action-sequence encoder and CLS latent path form the CVAE posterior during training only; during inference, future actions are unavailable and the latent is fixed to the prior mean z = 0. TABLE I TASK 2 DECODER-STATE DIAGNOSTICS. SHIFT IS THE PLACE-PHASE CLEAN-TO-FULL COSINE DISTANCE. PHASE RETENTION MEASURES FULL-TO-CLEAN PICK/PLACE CENTROID SEPARATION. GEOMETRY IS THE FULL-DISTRACTOR CONTAINER-POSITION SILHOUETTE. MetricStandard DataAug Modified Place shift ↓0.19930.00110.0125 Phase retained ↑13.1%94.8%108.5% Container silhouette ↑0.00950.08890.2037 Conditional placement ↑33.3%66.3%100.0% Standard ACT’s decoder state shifts substantially under distractors during placement and retains only 13.1% of its clean pick/place separation. Both object-centric variants strongly suppress this shift and preserve phase structure. The augmentation-only model provides a critical counterex- ample to distractor invariance as a sufficient explanation of robustness: it is more invariant than ACT-Modified, yet retains weaker container-position geometry and reaches only 66.3% conditional placement compared with 100.0% for ACT-Modified. Across the three policies, destination-geometry preservation follows placement performance more closely than clean-to-distractor invariance alone. Robust behavior is there- fore associated with suppressing nuisance-induced variation while preserving the spatial structure required by the currently relevant referent. The representation analyses remain correlational, and at- tention maps are likewise treated as diagnostic rather than causal explanations. We therefore complement them with a direct prompt intervention. In a separate cup-selection probe, replacing the target prompt with a distractor prompt increases distractor selection from 0-1% to 35-91%. Because the visual crop contains target appearance but no explicit full-frame coordinates, this intervention shows that prompt identity can actively redirect closed-loop target selection rather than merely correlate with successful behavior. d) The VLA case study provides preliminary cross-regime evidence.: The prompt-trained π 0.5 checkpoint succeeds in 20/25 trials when evaluated without its expected RGB cue and in 25/25 when the cue is restored. The entire deficit is concentrated in the ambiguous routing subgoals: routing success increases from 5/10 to 10/10, while the remaining instrument-handling subgoals are unaffected. The regularized clean-input checkpoint also reaches 25/25 overall and 10/10 on routing. Thus, the observed deficit again localizes to selection of a context-dependent destination rather than to execution of the remaining manipulation sequence. Because the experiment is small and the guided conditions are not equally informed, it provides convergent behavioral evidence for the conditional- grounding perspective rather than evidence that ACT and π 0.5 share the same internal shortcut. V. DISCUSSION AND OUTLOOK The ACT and VLA studies instantiate conditional visual grounding differently. In ACT, manipulation phase determines whether the object or receptacle is relevant; in the VLA task, the observed instrument state determines the relevant (a) Standard: clean/full shift(b) DataAug: clean/full shift(c) Modified: clean/full shift (d) Standard: container geometry(e) DataAug: container geometry(f) Modified: container geometry Fig. 2. Robust placement requires both distractor invariance and retained target geometry. The top row compares clean and full-distractor Task 2 place-phase decoder states. The bottom row colors the corresponding representations by the target container’s left/right position, with clean and full conditions shown in a shared within-model projection. Augmentation-only ACT exhibits the strongest clean/full invariance, whereas ACT-Modified preserves substantially clearer container-position structure. The t-SNE projections are illustrative; quantitative comparisons in Table I are computed in the original feature space. destination. Across both settings, competent motor behavior can coexist with incorrect selection of a context-dependent ref- erent. The contribution is therefore not that both architectures fail identically, but that analyzing what must be grounded when provides a common framework for localizing visual failures and designing targeted interventions. The ACT representation analysis further shows that robustness requires not only sup- pressing distractor-induced variation but also preserving the task-relevant geometry required for control. Several limitations bound these conclusions. The observed ACT color-shape hierarchy is specific to the evaluated assets and requires validation with fully counterbalanced visual fac- tors. Prompt-bearing ACT variants receive additional target information, although augmentation-only results demonstrate substantial robustness gains without this advantage. Attention and representation analyses remain correlational, the hardware evaluation contains only 20 trials per cell, and the exploratory VLA study contains only five trials per subgoal with non- equivalent guidance conditions. The VLA task additionally uses a visible contamination surrogate and fixed routing rule rather than general contamination understanding. REFERENCES [1] A. Xie, L. Lee, T. Xiao, and C. Finn, “Decomposing the generalization gap in imitation learning for visual robotic manipulation,” in 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024, p. 3153–3160. [2] Y. Dong, H. Ge, Y. Zeng, J. Zhang, B. Tian, H. Zhu, Y. Jia, R. Wang, Z. Xue, G. Zhou, L. Ma, and G. Tian, “ImitDiff: Transferring foundation- model priors for distraction robust visuomotor policy,” arXiv preprint arXiv:2502.09649, 2025. [3] Y. Chen, Y. Zhang, G. D’urso, N. Lawrance, and B. Tidd, “Improving generalization ability of robotic imitation learning by resolving causal confusion in observations,” arXiv preprint arXiv:2507.22380, 2025. [4] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning fine-grained bi- manual manipulation with low-cost hardware,” 2023, arXiv:2304.13705. [5] P. de Haan, D. Jayaraman, and S. Levine, “Causal confusion in imitation learning,” in Advances in Neural Information Processing Systems, vol. 32, 2019. [Online]. Available: https://arxiv.org/abs/1905.11979 [6] S. Hattori, H. Sasaki, T. Hachimine, Y. Mizutani, and T. Matsubara, “Task-relevant and irrelevant region-aware augmentation for generaliz- able vision-based imitation learning in agricultural manipulation,” arXiv preprint arXiv:2603.04845, 2026. [7] H. Huang, X. Chen, Y. Chen, H. Li, X. Han, Z. Wang, T. Wang, J. Pang, and Z. Zhao, “RoboGround: Robotic manipulation with grounded vision-language priors,” arXiv preprint arXiv:2504.21530, 2025. [Online]. Available: https://arxiv.org/abs/2504.21530 [8] M. A. Muttaqien, T. Motoda, R. Hanai, and Y. Domae, “Visual prompting for robotic manipulation with annotation-guided pick-and- place using ACT,” arXiv preprint arXiv:2508.08748, 2025. [Online]. Available: https://arxiv.org/abs/2508.08748 [9] A. J. Hancock, A. Z. Ren, and A. Majumdar, “Run-time observation interventions make vision-language-action models more visually robust,” arXiv preprint arXiv:2410.01971, 2024. [Online]. Available: https://arxiv.org/abs/2410.01971 [10] X. Jia, B. Yang, Z. Ge, X. Nie, Y. Zhou, C. Fan, Y. Li, Y. Chai, C. Jing, Z. Liang, Q. Bu, H. Cao, C. Wu, Q. Li, Z. Yang, C. Zhang, H. Li, Z. Wu, J. Yan, and Y.-G. Jiang, “Guidedvla: Specifying task-relevant factors via plug-and-play action attention specialization,” 2026. [Online]. Available: https://arxiv.org/abs/2605.12369 [11] N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ̈ adle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C.-Y. Wu, R. Girshick, P. Doll ́ ar, and C. Feichtenhofer, “Sam 2: Segment anything in images and videos,” 2024. [Online]. Available: https://arxiv.org/abs/2408.00714 [12] K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky, “π 0.5 : A vision-language-action model with open-world generalization,” in Proceedings of the 9th Conference on Robot Learning, ser. Proceedings of Machine Learning Research, vol. 305.PMLR, 2025, p. 17–40. [Online]. Available: https://proceedings.mlr.press/v305/black25a.html