Paper deep dive
Sparse Feature Policy Unlearning Mitigates State Hallucination in Vision-Language-Action Models
Jiho Lee, Jeongeun Park, Heayoun Choi, Taekyung Kim, Eunwoo Kim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/11/2026, 2:35:16 AM
Summary
This paper introduces SOUL (Sparse feature pOlicy UnLearning), a method to mitigate 'state hallucination' in Vision-Language-Action (VLA) models. State hallucination is defined as a failure mode where the robot acts as if an unrealized state (e.g., grasping an object) has been achieved. The authors use Sparse Autoencoders (SAEs) to identify specific sparse features associated with these hallucinations versus successful behaviors. SOUL selectively unlearns the policy knowledge linked to hallucination features while retaining knowledge for successful manipulation, resulting in reduced hallucination rates and improved task success without compromising existing capabilities.
Entities (11)
Relation Signals (9)
SOUL → mitigates → State Hallucination
confidence 95% · Sparse Feature Policy Unlearning Mitigates State Hallucination in Vision-Language-Action Models
SOUL → evaluatedon → Franka Robot
confidence 92% · Experiments across VLA architectures... and a real-world Franka robot
SOUL → evaluatedon → LIBERO-plus
confidence 92% · Experiments across VLA architectures... on LIBERO-Plus
SOUL → evaluatedon → RoboCasa
confidence 92% · Experiments across VLA architectures... on RoboCasa
SOUL → uses → Sparse Autoencoder
confidence 92% · mechanistic interpretation via sparse autoencoders reveals that hallucination-associated sparse features are activated
SOUL → targets → Forget Set
confidence 90% · sparse features identified from hallucination failures... serve as explicit forgetting... targets... form the forget set
SOUL → targets → Retain Set
confidence 90% · sparse features identified from... successful behaviors serve as explicit... retention targets... form the retain set
OpenVLA → validatedwith → SOUL
confidence 90% · We validate our approach with OpenVLA [2] and π0.5
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision-Language-Action (VLA) models have shown strong generalization in robotic manipulation by leveraging rich representations from pretrained vision-language models. However, their deployment in real-world environments remains limited by recurring unreliable behaviors. In this work, we study state hallucination, a recurring failure pattern in which a VLA continues acting as if an unrealized robot-object state had been achieved. Our analyses find that state hallucination coincides with weakened attention to task-relevant visual regions, and a mechanistic interpretation via sparse autoencoders reveals that hallucination-associated sparse features are activated when these failures occur. Based on this analysis, we propose SOUL (Sparse feature pOlicy UnLearning), which selectively unlearns policy knowledge associated with state hallucination behaviors, where sparse features identified from hallucination failures and successful behaviors serve as explicit forgetting and retention targets, respectively. Experiments across VLA architectures in simulated and real-world environments show that our method substantially reduces hallucinated failures and improves task success without substantially compromising the existing manipulation capabilities. These results suggest that interpretable feature analysis provides a practical basis for selectively modifying undesirable knowledge in robot policies.
Tags
Links
- Source: https://arxiv.org/abs/2610.09496v1
- Canonical: https://arxiv.org/abs/2610.09496v1
Trouble viewing inline? Open PDF directly →
Full Text
45,273 characters extracted from source content.
Expand or collapse full text
Sparse Feature Policy Unlearning Mitigates State Hallucination in Vision-Language-Action Models Jiho Lee Affiliation: Jiho Lee, Heayoun Choi, and Eunwoo Kim are with the School of Computer Science Engineering, Chung-Ang University, Seoul, 06974, Republic of Korea (e-mail: j2hoooo, heayounchoi, eunwoo@cau.ac.kr). Jeongeun Park Affiliation: Jeongeun Park is with the School of Interactive Computing, Georgia Institute of Technology, Atlanta, GA, 30308, USA (e-mail: jpark3456@gatech.edu) Heayoun Choi Affiliation: Jiho Lee, Heayoun Choi, and Eunwoo Kim are with the School of Computer Science Engineering, Chung-Ang University, Seoul, 06974, Republic of Korea (e-mail: j2hoooo, heayounchoi, eunwoo@cau.ac.kr). Taekyung Kim Affiliation: Taekyung Kim is with the NAVER AI Lab, Seongnam, 13561, Republic of Korea (e-mail: taekyung.k@navercorp.com). *Corresponding authors. Eunwoo Kim Affiliation: Jiho Lee, Heayoun Choi, and Eunwoo Kim are with the School of Computer Science Engineering, Chung-Ang University, Seoul, 06974, Republic of Korea (e-mail: j2hoooo, heayounchoi, eunwoo@cau.ac.kr). Abstract Vision-Language-Action (VLA) models have shown strong generalization in robotic manipulation by leveraging rich representations from pretrained vision-language models. However, their deployment in real-world environments remains limited by recurring unreliable behaviors. In this work, we study state hallucination, a recurring failure pattern in which a VLA continues acting as if an unrealized robot-object state had been achieved. Our analyses find that state hallucination coincides with weakened attention to task-relevant visual regions, and a mechanistic interpretation via sparse autoencoders reveals that hallucination-associated sparse features are activated when these failures occur. Based on this analysis, we propose SOUL (Sparse feature pOlicy UnLearning), which selectively unlearns policy knowledge associated with state hallucination behaviors, where sparse features identified from hallucination failures and successful behaviors serve as explicit forgetting and retention targets, respectively. Experiments across VLA architectures in simulated and real-world environments show that our method substantially reduces hallucinated failures and improves task success without substantially compromising the existing manipulation capabilities. These results suggest that interpretable feature analysis provides a practical basis for selectively modifying undesirable knowledge in robot policies. I INTRODUCTION Vision-language-action models (VLAs) have emerged as a promising approach for bringing foundation models into robot manipulation [1, 2, 3]. By integrating visual and language understanding with action generation, VLAs can translate high-level instructions into continuous robot control. Large-scale pretraining on heterogeneous robot and vision-language data has further enabled generalization across tasks, objects, environments, and robot embodiments [4, 5, 6]. Despite this progress, VLAs still exhibit unreliable behaviors, including unstable or premature actions, long-horizon drift, failures to recover from execution errors, and infeasible actions [7, 8, 9, 10, 11, 12, 13]. This is particularly concerning in physical robot execution, where erroneous actions directly affect the robot and its environment. Within this broader set of failures, we focus on a recurring failure pattern in which a VLA continues a manipulation sequence based on a robot-object state that has not actually been achieved. We refer to this behavioral pattern as state hallucination, because the policy acts as if an unrealized physical state were true. For example, a VLA may close its gripper before reaching an object and subsequently execute transport and place motions as though the object has been successfully grasped, as shown in Fig. 11(a). Such a mismatch can persist across subsequent actions, allowing a local execution error to propagate over multiple stages. This recurring and persistent behavior motivates us to investigate whether policy knowledge associated with state hallucination can be selectively removed while preserving successful manipulation capabilities. (a) Representative example of state hallucination (b) Overview of sparse feature policy unlearning for state hallucination (c) Performance after unlearning Fig. 1: Sparse feature policy unlearning mitigates hallucinated beliefs in VLAs. (a) Examples of grasp, transport, and place state hallucinations. (b) State hallucination occurs when a VLA mistakes its hallucinated belief for the actual robot-object state and generates actions based on this false belief. During this mismatch, hallucination features become strongly activated, and the proposed unlearning suppresses their activation while reinforcing success features. (c) Our sparse feature policy unlearning reduces hallucination while improving clean (hallucination-free) task success. We first examine how the policy’s internal representations differ between hallucinated and successful behaviors. Such hallucinated rollouts exhibit weakened attention to task-relevant regions, such as the gripper and target object, together with more spatially dispersed attention. Through a mechanistic analysis with sparse autoencoders, we also observe that sparse features exhibit distinct activation patterns during hallucinated and normal executions. To characterize these sparse feature activation patterns, we define a feature association score that contrasts activations across execution outcomes, such as failures with hallucination, failures without hallucination, and successful executions. Based on this score, we categorize sparse features according to whether they are more strongly associated with hallucinated failures or successful executions. Building on these observations, we propose Sparse feature pOlicy UnLearning (SOUL), which selectively unlearns policy knowledge associated with state hallucination, where sparse features identified from hallucinated failures and successful behaviors serve as explicit forgetting and retention targets, respectively. This feature-level policy unlearning objective suppresses activations of the forgetting targets while enhancing activations of the retention targets, thereby mitigating recurring hallucination behavior while minimizing interference with useful manipulation capabilities. We validate our approach with OpenVLA [2] and π0.5 _0.5 [3] on LIBERO-Plus [9], RoboCasa [14], and a real-world Franka robot, achieving an average 21.7%p reduction in hallucination rate and a 22.1%p improvement in task success while preserving 89% of behaviors that were successful before unlearning. These results demonstrate that interpretable feature analysis can reveal internal representations associated with recurring policy failures, providing a practical basis for selectively unlearning such behaviors in embodied foundation models. Our contributions are as follows: • We characterize state hallucination as a recurring failure pattern in which a VLA continues acting as though an unrealized robot-object state has been achieved. • We analyze state hallucination through visual attention and sparse features, revealing weakened attention to task-relevant regions and distinct feature patterns associated with state hallucination. • We introduce a sparse feature policy unlearning method that selectively unlearns recurring state hallucination behavior by suppressing hallucination features while enhancing features that support successful manipulation. I Related Work I-A Vision-Language-Action Models Vision-language-action models have become a major direction for generalist robot manipulation. Prior work has improved generalization through larger pretrained vision-language representations, heterogeneous robotic data, and stronger action generation architectures [1, 2, 3, 4, 5, 6, 15, 16, 17, 18]. Deployment reliability has also been improved through intermediate reasoning [19, 20], failure detection and recovery [12, 21], and test-time sampling or verification [22, 23]. These approaches primarily focus on improving policy outputs during deployment, rather than directly modifying policy knowledge associated with recurring failures. Recent concurrent mechanistic interpretability studies have begun to analyze and steer VLA behavior at inference time by intervening on representations, such as feed-forward activation directions [24] and sparse features [25]. In contrast, we formulate the selective modification of undesirable policy behavior as a policy unlearning problem, where knowledge associated with recurring hallucinated behavior is removed while representations supporting successful execution are preserved. Encoding this modification in the pretrained policy parameters enables persistent behavior correction without repeated intervention at inference time. I-B Machine Unlearning Machine unlearning aims to make a trained model forget knowledge associated with selected data without retraining the model from scratch [26, 27]. Prior work has studied unlearning mainly in classification tasks, where the influence of selected training data is removed from trained models [28, 29], and in language tasks, where factual knowledge or undesired responses are suppressed [30, 31]. A central challenge is balancing the removal of targeted knowledge with the preservation of useful model capabilities. This trade-off is particularly consequential for robot policies, where modifying one behavior can inadvertently disrupt multiple tightly coupled skills. Motivated by this, we introduce feature-level selective policy unlearning by explicitly identifying features for forgetting and retention to suppress undesirable behavior while preserving useful manipulation capabilities. I-C Sparse Autoencoders Sparse autoencoders (SAEs) decompose dense model activations into sparse latent representations, enabling individual features to be associated with more localized and interpretable patterns in model behavior [32, 33, 34]. They are trained to reconstruct the original activations under a sparsity constraint, such that only a small subset of latent features is active for each input. A commonly used variant is the Top-k SAE, which imposes this sparsity constraint explicitly by retaining only the k largest latent activations. In VLAs, SAE features have recently been used to interpret and repeatedly steer policy at test time [25]. In contrast, the proposed method identifies recurring internal patterns associated with unreliable behaviors using sparse features and translates these patterns into forgetting targets to guide targeted policy unlearning. Fig. 2: State hallucination is associated with distinct attention patterns. Representative clean success (CS, hallucination-free success) and hallucination failure (HF, failure with hallucination) rollouts under the same task show distinct attention patterns. HF exhibits weaker attention to task-relevant regions (e.g., gripper and target object) and increased attention to the background. Clean success Hallucination failure Gripper & target object Background & target object Gripper & Background I Analysis of State Hallucination in VLAs I-A What Is State Hallucination? We define state hallucination as a behavioral execution pattern in which a VLA continues to generate actions based on a robot-object state that has not actually been achieved. State hallucination can occur at different manipulation stages: • A grasp hallucination occurs when the gripper closes in free space without contacting the target object. Note that grasp failures involving object contact, such as inaccurate localization or slippage, are not considered hallucinations. • A transport hallucination occurs when, after either a grasp failure or a grasp hallucination, the gripper remains closed and continues moving from the grasp location toward the destination while the object remains ungrasped. • A place hallucination occurs when the gripper opens at the destination without holding the target object. We label these events by tracking ground-truth robot-object states in simulation and observable interaction states in the real world. Based on these criteria, we categorize each rollout as clean success (CS), hallucination failure (HF), or non-hallucination failure (NF). Here, HF denotes failed executions that exhibit at least one state hallucination event, whereas CS and NF denote successful and failed executions without state hallucination, respectively. I-B Attention Analysis of Hallucinated and Normal Executions To examine how the policy behaves internally during state hallucination, we compare internal attention patterns between CS and HF executions. At each rollout step, we aggregate the attention from action tokens to visual tokens and project the resulting attention scores onto the input image, revealing which visual regions the policy attends to while generating actions. For visualization, we show representative time steps immediately preceding the corresponding manipulation event, such as grasping. As shown in Fig. 2, CS executions exhibit attention concentrated on task-relevant regions, particularly the target object and gripper. In contrast, HF executions show weaker attention to these regions and increased attention to background regions. These differences suggest that state hallucination is accompanied by a distinct shift in the policy’s visual attention. This motivates us to look beyond attention and examine whether hallucinated executions are also reflected in the internal representations of the policy. CS rollout HF rollout Success event Hallucination event CS PickPlaceFridgeDrawerToShelf HF Statistics HF PickPlaceDrawerToCounter PickPlaceCounterToSink HF (a) Hallucination feature CS Pick up the black bowl on the cookie box and place it on the plate. HF Statistics (b) Success feature Fig. 3: Sparse features associated with state hallucination can be identified. For each feature, we show clean success (CS) and hallucination failure (HF) rollouts with their temporal activations, ze,t,r,jz_e,t,r,j, together with the distribution of rollout-level maximum activations, me,r,jm_e,r,j. (a) A hallucination feature becomes strongly activated around hallucination events, while remaining weak in CS. (b) A success feature is strongly activated during successful subtask execution or task completion in CS, while remaining weak in HF. The boxplots show consistent activation differences between CS and HF across rollouts. I-C Sparse Feature Characterization of State Hallucination To investigate whether state hallucination is reflected in distinguishable internal representations of VLAs, we analyze policy activations using sparse autoencoders (SAEs) [32]. SAEs decompose dense representations into sparse features, allowing us to characterize how individual feature activations vary across different behaviors, such as clean success, hallucination failure, and non-hallucination failure. Through this characterization, we find that sparse features are selectively associated with state hallucination and successful behaviors. SAE training. To extract sparse features from policy activations, we collect visual-token activations from the residual streams of transformer layers throughout each rollout together with their execution outcome labels. The visual token activations allow sparse features to be examined with respect to semantic scene regions. For an input sample x, let xp∈ℝdx_p ^d denote the activation corresponding to visual token p. We flatten the collected activations across rollout steps and visual tokens, treating each xpx_p as an individual SAE training sample. The encoder maps xpx_p to a latent representation and retains only the k largest positive activations, yielding a sparse feature vector zp(x)z_p(x), whose j-th element zp,j(x)z_p,j(x) denotes the activation of SAE feature j at visual token p. The decoder reconstructs the original policy activation xpx_p from zp(x)z_p(x). After training, we freeze the SAE and use its encoder to extract sparse feature activations for subsequent analysis. Sparse feature categorization. Motivated by the region-level attention differences observed in Section I-B, we characterize how sparse features are activated across semantic scene regions. We obtain semantic regions from the masks provided by the simulation benchmarks and manually annotate the corresponding regions for real-world observations. For an input sample x, let ℛr(x)R_r(x) denote the set of visual tokens belonging to region r. We define the region-conditioned feature activation as: zr,j(x)=1|ℛr(x)|∑p∈ℛr(x)zp,j(x).z_r,j(x)= 1|R_r(x)| _p _r(x)z_p,j(x). (1) This quantity measures how strongly feature j is expressed within semantic region r. For rollout e at time step t, we denote the corresponding region-conditioned feature activation by ze,t,r,jz_e,t,r,j. Since individual execution events, such as grasp, may occur over limited portions of a rollout, we summarize the temporal activation of each region-feature pair by its maximum over time, me,r,j=maxtze,t,r,j.m_e,r,j= _tz_e,t,r,j. We then average these rollout-level activations within each execution outcome. Let ℰcE_c denote the set of rollouts belonging to outcome c∈CS,NF,HFc∈\CS,NF,HF\. The outcome-level activation is computed as: m¯c,r,j=1|ℰc|∑e∈ℰcme,r,j. m_c,r,j= 1|E_c| _e _cm_e,r,j. (2) To examine how individual sparse features are associated with different execution outcomes, we define the following feature association scores: sr,jhall s^hall_r,j =m¯HF,r,j−12(m¯CS,r,j+m¯NF,r,j), = m_HF,r,j- 12 ( m_CS,r,j+ m_NF,r,j ), (3) sr,jsucc s^succ_r,j =m¯CS,r,j−12(m¯HF,r,j+m¯NF,r,j). = m_CS,r,j- 12 ( m_HF,r,j+ m_NF,r,j ). Here, larger sr,jhalls^hall_r,j and sr,jsuccs^succ_r,j indicate that feature j at region r is more strongly activated in hallucination failures and clean successes, respectively, relative to the other outcomes. Based on these scores, we categorize sparse features according to their stronger association with hallucination failures or clean successes, and refer to them as hallucination features and success features, respectively. Sparse features are selectively associated with state hallucination. Fig. 3 visualizes representative hallucination and success features across HF and CS rollouts, showing both their temporal activations and the distributions of their rollout-level activations. For each rollout, the temporal activation ze,t,r,jz_e,t,r,j shows how the sparse feature activation changes over time throughout the rollout. The hallucination feature remains weak during CS rollouts, but becomes strongly activated around state hallucination events in HF rollouts. Its activation is concentrated around hallucinated execution periods rather than remaining uniformly high throughout the rollout, indicating that the feature captures a temporally localized pattern associated with state hallucination. Across HF rollouts spanning different tasks and visual scenes, similar activation patterns repeatedly appear around hallucination events, revealing a consistent feature pattern associated with the behavior. Conversely, the success feature becomes strongly activated during successful subtask execution or task completion in CS rollouts, while remaining weaker in HF. Beyond these trajectory-level representative rollouts, boxplots on the right summarize the distributions of the rollout-level maximum activation me,r,jm_e,r,j across CS and HF rollouts. Consistent with the temporal activation patterns, the hallucination feature shows higher activation in HF, whereas the success feature shows higher activation in CS. These differences remain consistent across tasks and visual conditions. Together, these results show that state hallucination and successful execution are associated with distinguishable sparse feature patterns within the VLA representation. More visualizations are provided in Section V and the supplementary materials. Fig. 4: Overall framework. The proposed method provides a framework for improving robot policies by identifying and selectively modifying internal representations associated with undesirable behaviors. It first extracts visual-region-specific sparse features from policy activations and analyzes their activation patterns across execution outcomes, including clean success (CS), hallucination failure (HF), and non-hallucination failure (NF), characterized by task performance and hallucination occurrence. Based on these associations, region-feature pairs are selected as forget or retain targets. The policy is then updated to suppress hallucination-associated features while preserving reliable behavior. In this way, the policy is improved. IV Sparse Feature Policy Unlearning (SOUL) Building on the sparse feature analysis, we formulate the state hallucination mitigation as a selective policy unlearning problem. Specifically, internal representations associated with state hallucination are suppressed while those supporting successful execution are enhanced. To this end, we target sparse features associated with hallucination failure and clean success as explicit forgetting and retention, respectively. The overall framework is illustrated in Fig. 4. Feature selection for forgetting and retention. We construct the forget and retain sets using the feature association scores defined in Section I-C. For forgetting, we prioritize features strongly associated with hallucination failures, distinguishable from non-hallucination failures, and weakly associated with clean success. The first hallucination association is captured by sr,jhalls^hall_r,j, while the other two criteria are combined as sr,jagg=sr,jsep−sr,jsuccs^agg_r,j=s^sep_r,j-s^succ_r,j, where sr,jsep=m¯HF,r,j−m¯NF,r,js^sep_r,j= m_HF,r,j- m_NF,r,j measures separation from non-hallucination failures and subtracting sr,jsuccs^succ_r,j penalizes association with successful execution. We then define the final forget score as sr,jℱ=sr,jhall+sr,jaggs^F_r,j=s^hall_r,j+s^agg_r,j and select the top-ranked features to form the forget set ℱF. For retention, we select features with high sr,jsuccs^succ_r,j as the retain set ℛR, favoring features strongly associated with clean success. The resulting sets ℱF and ℛR define the sparse feature targets to be suppressed and enhanced, respectively, during policy unlearning. Feature-guided policy unlearning. Given the forget set ℱF and retain set ℛR, we update the VLA policy using a feature-guided unlearning objective. During unlearning, the SAE encoder remains frozen and is used only to compute sparse feature activations, while the VLA policy parameters θ are optimized. For hallucination failure samples, we define the forgetting objective to penalize activations of the selected forget features as: ℒforget=x∈HF[1|ℱ|∑(r,j)∈ℱ(zr,j(x,θ))2].L_forget=E_x _HF [ 1|F| _(r,j) (z_r,j(x;θ) )^2 ]. (4) For clean success samples, we define the retention objective to encourage activation of the selected retain features as: ℒretain=−x∈CS[1|ℛ|∑(r,j)∈ℛzr,j(x,θ)].L_retain=-E_x _CS [ 1|R| _(r,j) z_r,j(x;θ) ]. (5) In summary, the forgetting objective suppresses feature activations associated with state hallucination, while the retention objective reinforces features associated with successful execution, helping preserve useful policy representations during policy unlearning. The final objective jointly balances forgetting and retention as ℒunlearn=ℒforget+λrℒretain,L_unlearn=L_forget+ _rL_retain, where λr _r controls the strength of the retention objective. TABLE I: Performance comparison of hallucination and task success on benchmarks. All values are reported as percentages (%). ‘Hall.’ denotes hallucination. ↓ and ↑ indicate that lower and higher values are better, respectively. Method Hallucination ↓ Hallucination failure ↓ Overall success ↑ Clean success ↑ Grasp Hall. ↓ Transport Hall. ↓ Place Hall. ↓ LIBERO-Plus with OpenVLA Original Policy [2] 70.0 60.0 26.0 16.0 70.0 38.0 16.0 Gradient Ascent [31] 84.0 84.0 0.0 0.0 84.0 62.0 0.0 SOUL (Ours) 44.0 38.0 58.0 52.0 44.0 20.0 6.0 RoboCasa with π0.5 _0.5 Original Policy [3] 40.0 37.1 41.9 39.0 39.0 7.6 1.9 Gradient Ascent [31] 60.0 59.0 4.8 3.8 60.0 0.0 0.0 SOUL (Ours) 22.9 21.0 54.3 52.4 22.9 3.8 0.0 Before unlearning (Original policy) After unlearning (SOUL, proposed method) Retain Forget (a) LIBERO-Plus with OpenVLA Retain Forget (b) RoboCasa with π0.5 _0.5 Activation: Fig. 5: Visualizations of rollout behavior and feature activations on benchmarks. We compare the original policy and the proposed method, showing the temporal activations of selected forget and retain features during execution. V Experiments V-A Setup Evaluation protocol. We conduct experiments with the autoregressive OpenVLA [2] and flow-matching-based π0.5 _0.5 [3] in LIBERO-Plus [9], RoboCasa [14], and a real-world setup with a 7-DoF Franka Research 3 robotic arm. LIBERO-Plus provides realistic visual perturbations at multiple severity levels, while RoboCasa provides diverse kitchen environments and everyday manipulation tasks. To provide a controlled setting for evaluating state hallucination while minimizing confounding effects from task and motion planning, we focus on pick-and-place tasks, a fundamental manipulation primitive underlying many complex tasks, with a structured grasp-transport-place sequence.11 1 For LIBERO-Plus, we use the base tasks put the bowl on the stove, pick up the black bowl on the cookie box and place it on the plate, and pick up the tomato sauce and place it in the basket from the goal, spatial, and object suites, respectively, together with their corresponding lighting and background evaluation variants. For RoboCasa, we use PickPlaceCounterToStandMixer, PickPlaceDrawerToCounter, PickPlaceCounterToSink, PickPlaceCabinetToCounter, PickPlaceCounterToCabinet, and PickPlaceFridgeDrawerToShelf, together with their corresponding evaluation configurations. To our knowledge, no prior work has established an unlearning baseline for robot policies. For a direct comparison from an unlearning perspective, we therefore adapt gradient ascent, a commonly used optimization-based unlearning method [30, 31], as a baseline for VLA policies by maximizing the action prediction loss on hallucination failure samples. SAE-based feature analysis and unlearning. We use Top-k SAEs [33, 35] to decompose visual-token activations into sparse latent features, whose fixed number of active features provides a consistent basis for comparing activation patterns across executions. The SAE objective combines the standard reconstruction loss with an auxiliary reconstruction loss using the top-kauxk_aux inactive features to provide additional learning signals for rarely activated features. We set the SAE latent dimension to 40964096, the number of active features to k=100k=100 and kaux=512k_aux=512. From the sparse feature analysis, we select the top five region-feature pairs associated with hallucination and success as forgetting and retention targets, respectively. For policy unlearning, we initialize the model from the publicly released pretrained policy and optimize LoRA adapters [36] with our unlearning objective while keeping the original model weights frozen. We uniformly sample up to 2020 time steps per rollout to reduce bias toward longer trajectories. For feature analysis, we use layers 0, 8, 16, 24, and 31 of the OpenVLA language backbone and layers 0, 5, 11, and 17 of the PaliGemma backbone in π0.5 _0.5. Metrics. To evaluate task performance and state hallucination, we report task-level success metrics and rollout-level hallucination metrics. Overall success and clean success measure the fraction of rollouts that successfully complete the task and the fraction that complete the task without state hallucination, respectively. Hallucination and hallucination failure measure the fraction of rollouts containing at least one state hallucination event and the fraction that both contain state hallucination and result in task failure, respectively. We further report type-specific hallucination rates for grasp, transport, and place, defined as the fraction of rollouts containing at least one hallucination of each type. V-B Main Results Quantitative results. To quantitatively evaluate whether our proposed sparse feature policy unlearning (SOUL) improves policy reliability by reducing state hallucination without compromising task success, we compare three policies: the original pretrained policy, gradient-ascent unlearning, and SOUL. As shown in TABLE I, the original policies exhibit average hallucination and hallucination failure rates of 55.0%55.0\% and 48.6%48.6\% across evaluation settings, respectively. Gradient-ascent unlearning does not consistently mitigate hallucination and substantially degrades task success, suggesting that directly forgetting hallucination samples can interfere with useful manipulation capabilities. In contrast, SOUL reduces the average hallucination rate from 55.0%55.0\% to 33.5%33.5\%, while increasing clean success from 27.5%27.5\% to 52.2%52.2\%. Hallucination rates of the grasp, transport, and place stages are also consistently reduced, showing that the improvement extends across manipulation stages rather than being limited to one stage. These results show that SOUL improves policy reliability by selectively mitigating recurring state hallucination behaviors while preserving useful manipulation capabilities. Before unlearning (Original policy) After unlearning (SOUL, proposed method) Retain Forget (a) ‘Pick fries and put it on the plate’ with OpenVLA Retain Forget (b) ‘Throw the coffee in the trash bin’ with π0.5 _0.5 Activation: Fig. 6: Real-world rollouts and selected sparse feature activations before and after applying the proposed unlearning, SOUL. We compare the original policy and SOUL across representative real-world rollouts spanning both models and tasks. Qualitative results. To qualitatively examine how SOUL modifies sparse feature activations, Fig. 5 compares rollout behaviors and the temporal activations of selected forget and retain features before and after applying unlearning across the evaluation settings. Before unlearning, the original policies exhibit state hallucination, with strong forget feature activation emerging around hallucination events while retain feature activation remains weak. After unlearning, the policies successfully complete the tasks without state hallucination, accompanied by suppressed forget feature activation and increased retain feature activation during the corresponding manipulation stages. This pattern is consistently observed for both OpenVLA on LIBERO-Plus and π0.5 _0.5 on RoboCasa. These results show that SOUL shifts the selected sparse feature activations in the intended directions, suppressing the activation of hallucination-associated features while increasing that of features associated with clean successful execution. Real-world evaluation. We further evaluate SOUL on a Franka Research 3 robot using OpenVLA and π0.5 _0.5, conducting 50 rollouts across both models under diverse visual conditions. Starting from the publicly released pretrained models, we first fine-tune each model for its target task using LoRA and then apply SOUL to the task-specific policies. Quantitatively, SOUL increases clean success by an average of 20%p while reducing hallucination failure by 22%p across both settings. As shown in Fig. 6, the original policies exhibit state hallucination, accompanied by strong activation of forget features and weak activation of retain features. After applying SOUL, policies complete the tasks without hallucination, with suppressed forget feature activations and increased retain feature activations. These results show that SOUL remains effective in physical robot execution under diverse real-world visual conditions across both VLA architectures. (a) Preservation ↑ (b) Correction ↑ Fig. 7: Behavior preservation and correction after applying SOUL. (a) Preservation measures the fraction of originally successful task executions and non-hallucinated events that remain successful and non-hallucinated, respectively, after SOUL. (b) Correction measures the fraction of originally failed task executions and hallucinated events that become successful and non-hallucinated, respectively, after SOUL. Higher values indicate better preservation or correction. V-C Analyses Behavior preservation and correction. To further examine the behavior-level changes underlying the reductions in hallucination and improvements in task success reported in TABLE I, we analyze the unlearning trade-off between preserving desirable behaviors and correcting unreliable behaviors. Fig. 7 reports preservation and correction at both the task and event levels. SOUL achieves an average preservation rate of 89% across originally successful task executions and non-hallucinated events. Meanwhile, it converts 40% of originally failed task executions into successful ones and corrects 78% of hallucinated events across the grasp, transport, and place stages. These results demonstrate a favorable unlearning trade-off, substantially correcting undesirable hallucination behaviors while largely preserving behaviors that were already successful or hallucination-free. TABLE I: Comparison of feature selection at different spatial granularities. ‘Global’ selects features from activations pooled across all visual tokens, ‘Tokenized’ from individual visual-token activations, and ‘Region-guided’ from activations aggregated within each semantic region. Method Hall. ↓ Hall. failure ↓ Overall success ↑ Clean success ↑ Global 45.8 41.4 42.7 38.3 Tokenized 41.5 35.5 47.3 41.3 Region-guided 33.5 29.5 56.2 52.2 Effect of region guidance. To evaluate the effect of spatial grounding on sparse feature selection for policy unlearning, we compare global, tokenized, and region-guided feature selection using OpenVLA and π0.5 _0.5 on LIBERO-Plus and RoboCasa, respectively, with results averaged in TABLE I. Global feature selection improves over the original policy, but is less effective than tokenized and region-guided selection, likely because pooling across visual tokens removes spatial information needed to isolate localized hallucination-associated patterns. Tokenized selection performs better by preserving token-level localization, while region-guided selection achieves the lowest hallucination rates and the highest success rates. These results show that preserving spatial structure during feature selection improves policy unlearning, with region guidance achieving the best balance between hallucination reduction and task success. Contributions of forget score and retain objective. To analyze the roles of the forget score sr,jℱs^F_r,j and the retain objective ℒretainL_retain, we conduct ablations using OpenVLA and π0.5 _0.5 on LIBERO-Plus and RoboCasa, respectively, with results averaged in TABLE I. The first row corresponds to the proposed method. Without the retain objective (second row), hallucination reduction is maintained, but clean success decreases, showing its importance in preserving successful behavior. Removing the aggregation score from the forget score (third row), both hallucination reduction and task success degrade, showing that incorporating the separation and success criteria enables more selective forgetting target selection. Together, combining score aggregation with the retain objective achieves the best balance, supporting their complementary roles in selective policy unlearning. TABLE I: Ablation study of the forget score formulation and retain objective. The aggregated score sr,jagg=sr,jsep−sr,jsuccs^agg_r,j=s^sep_r,j-s^succ_r,j complements sr,jhalls^hall_r,j for forget feature selection. sr,jhalls^hall_r,j sr,jaggs^agg_r,j ℒretainL_retain Hall. ↓ Hall. failure ↓ Overall success ↑ Clean success ↑ ✓ ✓ ✓ 33.5 29.5 56.2 52.2 ✓ ✓ 36.8 29.9 55.2 48.3 ✓ ✓ 39.9 36.9 50.7 47.7 VI Conclusion We have presented a study of selective policy unlearning in VLAs, targeting state hallucination as a recurring undesirable execution behavior. Our analysis showed that state hallucination is accompanied by weakened attention to task-relevant visual regions and distinct region-conditioned sparse feature patterns associated with hallucinated executions. Based on these findings, we proposed SOUL, which selectively unlearns policy knowledge associated with state hallucination by suppressing hallucination features while preserving success features. Experiments across VLA architectures in simulated and real-world environments show that SOUL reduces state hallucination and improves task success while largely preserving existing manipulation capabilities. These results show that interpretable sparse feature analysis can move beyond diagnosing VLA failures to provide actionable targets for policy unlearning, enabling selective modification of recurring undesirable behaviors in embodied robot policies. References [1] B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, et al. (2023) Rt-2: vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, p. 2165–2183. Cited by: §I, §I-A. [2] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, et al. (2025) OpenVLA: an open-source vision-language-action model. In Conference on Robot Learning, p. 2679–2713. Cited by: §I, §I, §I-A, TABLE I, §V-A. [3] K. Black et al. (2025) π0.5 _0.5: a vision-language-action model with open-world generalization. In Proceedings of the 9th Conference on Robot Learning, Vol. 305, p. 17–40. Cited by: §I, §I, §I-A, TABLE I, §V-A. [4] Open X-Embodiment Collaboration et al. (2024) Open x-embodiment: robotic learning datasets and RT-X models. In 2024 IEEE International Conference on Robotics and Automation, p. 6892–6903. External Links: Document Cited by: §I, §I-A. [5] NVIDIA et al. (2025) GR00T N1: an open foundation model for generalist humanoid robots. In ArXiv Preprint, External Links: 2503.14734 Cited by: §I, §I-A. [6] Gemini Robotics Team et al. (2025) Gemini robotics: bringing ai into the physical world. arXiv preprint arXiv:2503.20020. Cited by: §I, §I-A. [7] Q. Gu et al. (2026) Safe: multitask failure detection for vision-language-action models. Advances in Neural Information Processing Systems 38, p. 40041–40076. Cited by: §I. [8] B. Zhang et al. (2026) Safevla: towards safety alignment of vision-language-action model via constrained learning. Advances in Neural Information Processing Systems 38, p. 153335–153373. Cited by: §I. [9] S. Fei, S. Wang, J. Shi, Z. Dai, J. Cai, P. Qian, L. Ji, X. He, S. Zhang, Z. Fei, et al. (2025) Libero-plus: in-depth robustness analysis of vision-language-action models. arXiv preprint arXiv:2510.13626. Cited by: §I, §I, §V-A. [10] X. Zhou, Y. Xu, G. Tie, et al. (2025) Libero-pro: towards robust and fair evaluation of vision-language-action models beyond memorization. arXiv preprint arXiv:2510.03827. Cited by: §I. [11] Y. Zhang, Y. Qi, and X. Zheng (2025) Experiences from benchmarking vision-language-action models for robotic manipulation. arXiv preprint arXiv:2511.11298. Cited by: §I. [12] Z. Lin, J. Duan, H. Fang, et al. (2025) Failsafe: reasoning and recovery from failures in vision-language-action models. arXiv preprint arXiv:2510.01642. Cited by: §I, §I-A. [13] H. Soh and E. Lim (2026) Action hallucination in generative visual-language-action models. arXiv preprint arXiv:2602.06339. Cited by: §I. [14] X. Zhou et al. (2024) RoboCasa: large-scale simulation of household tasks for generalist robots. In Robotics: Science and Systems X, External Links: Document Cited by: §I, §V-A. [15] D. Driess et al. (2023) PaLM-e: an embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning, p. 8469–8488. Cited by: §I-A. [16] D. Ghosh et al. (2024) Octo: an open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, External Links: Document Cited by: §I-A. [17] K. Black et al. (2025) π0 _0: a vision-language-action flow model for general robot control. In Proceedings of Robotics: Science and Systems, External Links: Document Cited by: §I-A. [18] M. Shukor et al. (2025) Smolvla: a vision-language-action model for affordable and efficient robotics. arXiv preprint arXiv:2506.01844. Cited by: §I-A. [19] Q. Zhao et al. (2025) Cot-vla: visual chain-of-thought reasoning for vision-language-action models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 1702–1713. Cited by: §I-A. [20] F. Lin, R. Nai, Y. Hu, J. You, et al. (2026) OneTwoVLA: a unified vision-language-action model with adaptive reasoning. In The Fourteenth International Conference on Learning Representations, Cited by: §I-A. [21] C. Liu, W. Tan, L. Zhu, F. Li, J. Li, G. Yang, and H. T. Shen (2026) Self-correcting vla: online action refinement via sparse world imagination. arXiv preprint arXiv:2602.21633. Cited by: §I-A. [22] J. Kwok et al. (2025) RoboMonkey: scaling test-time sampling and verification for vision-language-action models. In Conference on Robot Learning, p. 3200–3217. Cited by: §I-A. [23] S. Jang, D. Kim, C. Kim, Y. Kim, and J. Shin (2026) Verifier-free test-time sampling for vision-language-action models. In The Fourteenth International Conference on Learning Representations, Cited by: §I-A. [24] B. Häon, K. C. Stocking, I. Chuang, and C. Tomlin (2025) Mechanistic interpretability for steering vision-language-action models. In Proceedings of the 9th Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 305, p. 2743–2762. Cited by: §I-A. [25] M. A. Khan, N. Boskov, F. M. Anwar, and M. A. Khan (2025) Controlling vision–language–action policies through sparse latent directions. In Mechanistic Interpretability Workshop at NeurIPS 2025, Cited by: §I-A, §I-C. [26] Y. Cao and J. Yang (2015) Towards making systems forget with machine unlearning. In IEEE symposium on security and privacy, Cited by: §I-B. [27] T. T. Nguyen, T. T. Huynh, Z. Ren, P. L. Nguyen, A. W. Liew, H. Yin, et al. (2025) A survey of machine unlearning. ACM Transactions on Intelligent Systems and Technology 16 (5), p. 1–46. Cited by: §I-B. [28] A. Golatkar, A. Achille, and S. Soatto (2020) Eternal sunshine of the spotless net: selective forgetting in deep networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 9301–9309. Cited by: §I-B. [29] A. Thudi et al. (2022) Unrolling sgd: understanding factors influencing machine unlearning. In IEEE European symposium on security and privacy, Cited by: §I-B. [30] Y. Yao and X. Xu (2024) Large language model unlearning. Advances in Neural Information Processing Systems, p. 105425–105475. Cited by: §I-B, §V-A. [31] P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, and J. Z. Kolter (2024) Tofu: a task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121. Cited by: §I-B, TABLE I, TABLE I, §V-A. [32] R. Huben et al. (2024) Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, Cited by: §I-C, §I-C. [33] L. Gao et al. (2025) Scaling and evaluating sparse autoencoders. In International Conference on Learning Representations, Vol. 2025, p. 26721–26754. Cited by: §I-C, §V-A. [34] B. Bussmann, P. Leask, and N. Nanda (2024) Batchtopk sparse autoencoders. arXiv preprint arXiv:2412.06410. Cited by: §I-C. [35] A. Makhzani and B. Frey (2013) K-sparse autoencoders. arXiv preprint arXiv:1312.5663. Cited by: §V-A. [36] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, Cited by: §V-A.