Paper deep dive

SAVE: Sparse Autoencoder-Driven Visual Information Enhancement for Mitigating Object Hallucination

Sangha Park, Seungryong Yoo, Jisoo Mok, Sungroh Yoon

Year: 2024Venue: arXiv preprintArea: Multimodal SafetyType: EmpiricalEmbeddings: 60

Models: LLaVA-1.6, LLaVA-NeXT, Qwen2-VL

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/11/2026, 1:04:02 AM

Summary

SAVE (Sparse Autoencoder-Driven Visual Information Enhancement) is a training-free framework that mitigates object hallucination in Multimodal Large Language Models (MLLMs) by steering internal model activations along identified 'visual understanding' latent features discovered via Sparse Autoencoders (SAEs).

Entities (5)

Object Hallucination · problem · 100%SAVE · framework · 100%Sparse Autoencoder · technique · 100%LLaVA-1.6 · model · 95%Qwen2-VL · model · 95%

Relation Signals (3)

SAVE → utilizes → Sparse Autoencoder

confidence 100% · we leverage SAEs to capture visual information processing of MLLMs

SAVE → mitigates → Object Hallucination

confidence 95% · SAVE (Sparse Autoencoder-Driven Visual Information Enhancement), a framework that mitigates hallucination

Sparse Autoencoder → identifies → Visual Understanding Features

confidence 90% · SAE features most indicative of the model's visual information processing, referred to as visual understanding features

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Although Multimodal Large Language Models (MLLMs) have advanced substantially, they remain vulnerable to object hallucination caused by language priors and visual information loss. To address this, we propose SAVE (Sparse Autoencoder-Driven Visual Information Enhancement), a framework that mitigates hallucination by steering the model along Sparse Autoencoder (SAE) latent features. A binary object-presence question-answering probe identifies the SAE features most indicative of the model's visual information processing, referred to as visual understanding features. Steering the model along these identified features reinforces grounded visual understanding and effectively reduces hallucination. With its simple design, SAVE outperforms state-of-the-art training-free methods on standard benchmarks, achieving a 10\%p improvement in CHAIR\_S and consistent gains on POPE and MMHal-Bench. Extensive evaluations across multiple models and layers confirm the robustness and generalizability of our approach. Further analysis reveals that steering along visual understanding features suppresses the generation of uncertain object tokens and increases attention to image tokens, mitigating hallucination. Code is released at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

59,434 characters extracted from source content.

Expand or collapse full text

SAVE: Sparse Autoencoder-Driven Visual Information Enhancement for Mitigating Object Hallucination Sangha Park 1 Seungryong Yoo 1 Jisoo Mok 2† Sungroh Yoon 1,3† 1 Department of Electrical and Computer Engineering, Seoul National University 2 Daegu Gyeongbuk Institute of Science and Technology 3 IPAI, AIIS, ASRI, INMC, and ISRC, Seoul National University Abstract Although Multimodal Large Language Models (MLLMs) have advanced substantially, they remain vulnerable to ob- ject hallucination caused by language priors and visual in- formation loss. To address this, we propose SAVE (Sparse Autoencoder-Driven Visual Information Enhancement), a framework that mitigates hallucination by steering the model along Sparse Autoencoder (SAE) latent features. A binary object-presence question-answering probe identifies the SAE features most indicative of the model’s visual in- formation processing, referred to as visual understanding features. Steering the model along these identified features reinforces grounded visual understanding and effectively reduces hallucination. With its simple design, SAVE out- performs state-of-the-art training-free methods on standard benchmarks, achieving a 10%p improvement in CHAIR S and consistent gains on POPE and MMHal-Bench. Exten- sive evaluations across multiple models and layers confirm the robustness and generalizability of our approach. Fur- ther analysis reveals that steering along visual understand- ing features suppresses the generation of uncertain object tokens and increases attention to image tokens, mitigating hallucination. Code is released at https://github. com/wiarae/SAVE. 1. Introduction Multimodal Large Language Models (MLLMs) have achieved strong performance across various multimodal tasks such as image captioning, visual question answering, and multimodal reasoning by aligning visual inputs with pretrained language models [6, 21]. However, these mod- els are prone to object hallucination—the generation of text describing non-existent objects or attributes not grounded in the visual input [1, 16]. Unlike hallucination in uni- †Corresponding authors “Is there ice cream in the image?” SAE Decoder Encoder ... ... YesNo ... ... Layer SAVE Latent Steering Vanilla SAVE (Ours) Steered to correct answer GT: No Layer Figure 1. Illustration of SAVE. While vanilla model produces a hallucinated response, our approach—using SAE latent steer- ing—successfully generates a correct, grounded response. modal LLMs, which mainly arises from linguistic priors, hallucination in MLLMs additionally involves failures in visual grounding, stemming from misaligned attention or suppressed visual cues [11, 24]. To mitigate object hallucination, prior studies have fo- cused on reducing language priors and alleviating visual in- formation loss through strategies such as contrastive decod- ing [16, 34] and latent steering [19, 25]. More recently, studies have observed that the internal behavior of MLLMs differs between responses containing hallucinated objects and those correctly describing ground-truth objects [13, 14], suggesting that hallucination arises from a structured shift in internal dynamics rather than random failure. These insights motivate approaches aimed at strengthening the model’s visual information processing, which first calls for a clear understanding of the model’s visual encoding capa- bilities. Sparse autoencoders (SAEs), widely used in mechanis- tic interpretability [27], decompose model activations into sparse and often monosemantic features, where each latent 1 arXiv:2512.07730v2 [cs.CV] 11 Dec 2025 dimension corresponds to a semantically meaningful direc- tion [2, 35]. Originally proposed for LLMs [? ], SAEs have recently been extended to multimodal models [30, 41]. Building on these efforts, we leverage SAEs to capture vi- sual information processing of MLLMs and enable targeted steering that reinforces grounded visual understanding. Surprisingly, we demonstrate that SAEs, when guided by a well-designed probe, provide an effective approach for capturing a model’s visual information processing. We for- mulate binary object-presence question-answering as such a probe, enabling the computation of a separation score that measures differences in SAE activations between correct and hallucinated responses. SAE features with the high- est separation scores are identified as visual understand- ing features, serving as the primary latent directions of the model’s visual information processing, while features with high separation in the opposite direction are consid- ered hallucinated features. Our analysis further reveals that these SAE features are semantically disentangled: visual understanding features predominantly activate on correct responses, whereas hallucinated features are more frequent in hallucinated ones. This selective activation faithfully re- flects the model’s visual information processing and reveals its underlying failure modes. Building on these insights, we propose SAVE (Sparse Autoencoder-Driven Visual Information Enhancement), which mitigates object hallucination by steering along vi- sual understanding features. Reinforcing these features ef- fectively reduces hallucination, whereas steering along hal- lucinated features amplifies it. Although derived from an object-level probe, the identified features reliably mitigate hallucination through latent steering, revealing the model’s susceptibility to visual information loss and language pri- ors—the key drivers of object hallucination. We evaluate SAVE on POPE [18], CHAIR [31], and MMHal-Bench [33]. With its simple design, SAVE con- sistently outperforms multiple state-of-the-art training-free approaches.In particular, compared to their respec- tive base models, CHAIR S improves by about 10%p on LLaVA-1.6 [22], 5%p on LLaVA-NeXT [23], and 20%p on Qwen2-VL [38]. Moreover, experiments show that apply- ing steering at different layer positions effectively mitigates hallucination, demonstrating our approach is layer-wise applicable.By validating on both LLaVA models and Qwen2-VL [38], we demonstrate that our method is ro- bust and generalizable across MLLMs. Finally, our analysis shows that steering along visual understanding features mit- igates hallucination by lowering the probability of uncertain object tokens and increasing attention to image tokens. Our contributions are as follows: 1. We propose SAVE, a novel framework that leverages SAE features to interpret a model’s visual informa- tion processing through binary object-presence probing, thereby enabling effective hallucination mitigation. 2. We demonstrate that steering along discovered vi- sual understanding features significantly reduces hal- lucination and consistently outperforms state-of-the-art training-free methods. 3. Through extensive experiments across multiple MLLMs (LLaVA-1.6, LLaVA-NeXT, Qwen2-VL) and layer-wise steering, we demonstrate the robustness and generaliz- ability of our SAE-based approach. 2. Related work 2.1. Object Hallucination in MLLMs Object hallucination—where models generate content that is factually incorrect, irrelevant, or inconsistent with vi- sual inputs [1]—is a persistent challenge in MLLMs. Re- cent studies address this issue through three main ap- proaches.Decoding-based methods (e.g., VCD [16], SID [12], DeCo [37], ED [4]) enforce token-level grounding by adjusting the logits during generation. Attention-based approaches (e.g., PAI [24], VAR [15], AD-H [39]) refine attention maps to improve image-text alignment. Steering- based methods (e.g., VTI [25], VISTA [19]) manipulate in- ternal activations to suppress hallucinated content. These efforts suggest hallucination is shaped by controllable inter- nal dynamics. We build on this view by introducing SAEs for representation-level analysis and control of hallucina- tion in multimodal models. 2.2. Mechanistic Interpretability with SAE Mechanistic interpretability [27] aims to uncover how neu- ral networks produce their outputs by analyzing internal ac- tivations and identifying human-interpretable features as- sociated with individual neurons [8, 28]. However, ana- lyzing individual neurons is challenging due to their pol- ysemantic nature—a phenomenon caused by superposition, where models encode more features than available dimen- sions. [8]. To address the polysemanticity of neurons, SAEs decompose activations into a sparse set of latent direc- tions that form a high-dimensional feature dictionary [2]. SAEs have been used to discover interpretable and steer- able features of LLMs, including those related to bias and safety [5, 35]. Building on this, several studies have applied SAEs to vision-language models such as CLIP, enabling visual con- cept extraction and image generation steering [7]. More re- cently, SAEs have been extended to MLLMs, where they are used to analyze visual concepts [20], track the emer- gence of multimodal alignment [26], and identify semanti- cally meaningful features across modalities [41]. This pa- per extends these efforts by utilizing SAEs for hallucination control in MLLMs, leveraging their interpretability to ana- lyze and steer internal representations. 2 Is there in the image? Visual Understanding Feature:Hallucinated feature: ... ... Correct Set Hallucinated Set Yes ... ...... Layer 푙 Layer 푙 + 1 GT (zebra) or Hallucinated (mountain) Object (a) Object presence QA Answer SAE Ye s -> Yes No -> No (b) Collect SAE activation (c) Identify features Ye s No Ye s -> No No -> Yes Choose max activated feature: Figure 2. Overall framework of SAVE. (a) We obtain SAE ac- tivations using object-presence yes/no questions, then (b) split them by response correctness. (c) Separation scores are computed to identify features associated with visual understanding (correct cases) and hallucination (incorrect cases). 3. Preliminaries 3.1. Sparse Autoencoder SAEs are a powerful tool for discovering interpretable structure in model activations. Inspired by classical dictio- nary learning [29], they learn a sparse representation of the input by constraining the intermediate activations such that only a small number of latent units are active at a time. To enforce this sparsity, the SAE architecture proposed by OpenAI [10] introduces a two-layer autoencoder with a TopK activation function, which retains only the top-k acti- vations and zeroes out the rest. Given a model representa- tion x, the SAE defines the encoding and decoding process as: a(x) = TopK (ReLU(W enc (x− b pre ) + b enc )),(1) SAE(x) = W dec · a(x) + b dec .(2) Here, W enc and b enc denote the encoder’s weight matrix and bias, while W dec and b dec are the corresponding decoder parameters. A learned normalization offset b pre is applied to the input x prior to encoding. We denote the input as x ∈R T×d n , where T is the number of tokens and d n is the hidden dimension of the model. The encoder produces a sparse activation a(x)∈R T×d M , where the latent dimen- sion d M is much larger than the input dimension d n (i.e., d M ≫ d n ). This allows the input to be represented over a wide dictionary of features. We refer to the sparse out- put a(x) as the latent activation, and to each row of W dec as a latent direction, which serves as a basis vector in the learned dictionary. Each dimension of the d M -dimensional latent space is indexed by j ∈ 1,...,d M , and we refer to each index j as a distinct latent feature, representing a specific decoder direction. For a given input x, the activa- tion value a t,j (x) denotes the strength of feature j at token position t. SAE is trained to minimize the following loss function: L(x) =∥x− SAE(x)∥ 2 2 | z L reconstruction +λ∥a(x)∥ 0 |z L sparsity .(3) where the first term encourages accurate reconstruction of the input, and the second imposes an ℓ 0 -based sparsity penalty on the activation vector a(x). The hyperparameter λ controls the balance between these objectives. This train- ing setup reconstructs the input as a sparse linear combi- nation of latent directions, which tend to represent human- interpretable features. The sparsity constraint promotes monosemanticity, where each latent unit captures a distinct, coherent concept [2, 5, 10, 35]. 3.2. Multimodal SAE Recent work extends sparse autoencoders (SAEs) from text- only LLMs to large multimodal models, showing that SAEs can disentangle cross-modal, open-semantic features and even steer model behavior (e.g., feature-level interventions and transfer across model scales) [20, 41]. Building on this motivation, the multimodal-SAE [30] is integrated into in- termediate transformer layers of LLaVA-1.6-Mistral-7B to decompose residual stream activations into sparse, semanti- cally meaningful features. Following prior work on scaling SAEs [10, 35], a ReLU-based encoder–decoder architecture with an 8× expansion ratio is employed and trained to mini- mize a combination of reconstruction loss and sparsity reg- ularization. The SAE learns a dictionary of 32k latent fea- tures, with sparse activations that promote disentangled visual-language representations. It is trained on roughly 1.2M image–caption pairs from ShareGPT4V [3].To capture cross-layer visual grounding behavior, SAEs are trained at layers 8, 12, 16, 20, and 24. Additional details on density scheduling, token counts, and optimization are provided in Section A. 4. Visual Understanding Feature Object hallucination can be mitigated by enhancing visual information of the model [16, 34]. Prior studies have shown 3 Visual Understanding Activation | AUROC 0.93 Density 0 100 0 10 20 30 40 75 50 25 0.000.050.100.150.200.25 correct response hallucinated response Hallucinated Activation | AUROC 0.82 0.00.30.20.1 0.4 0.5 Activation Density (a) CHAIR S Scores (↓) 40 30 20 Score 31.2 31.0 21.4 38.0 CHAIR I Scores (↓) 7.9 7.7 5.4 11.3 10 8 6 12 Base RandomVisual Understanding Hallucinated (b) Score Visual Understanding Feature Hallucinated Feature (c) Figure 3. Analysis of identified features. (a) AUROC of visual-understanding and hallucinated features, computed by activation analysis on correct versus hallucinated responses. (b) CHAIR evaluation of steering behavior toward random, visual-understanding, and hallucinated features. (c) Visualization of each feature using its top-activated images and the most responsive patches. In the image, a person is holding a hot dog in their right hand. The person is seated at a table, which is adorned with a can of soda and a bottle of beer. LLaVA-1.6 Visual Understanding Feature Hallucination No Hallucination Hallucination, Speculative Hallucinated Feature In the image, a hand is holding a hot dog, which is a golden bun with a hot dog inside. The table is surrounded by people, suggesting that this might be a social gathering or a meal shared among friends or family. In the image, a person is holding a hot dog in their hand. The hot dog is intricately designed with various toppings, including what appears to be onions and mustard. The hot dog is placed on a paper design, which is brown in color. Figure 4. Qualitative examples of model steering. Steering to- ward visual understanding features mitigates object hallucination, whereas steering toward hallucinated features yields responses that remain hallucinated (highlighted in red) and include speculative wording (highlighted in blue). that the model’s internal representations are distinct when producing ground-truth versus hallucinated tokens [13, 14]. Building on these insights, we leverage the disentangled na- ture of SAEs to identify features most relevant to visual grounding and steer the model along these directions. This targeted steering enhances the model’s visual information, guiding it toward faithful object descriptions and reducing hallucinations. 4.1. Collect SAE Activations SAE features are characterized by how frequently they are activated under specific concepts.To identify vi- sual understanding features, we introduce a probe for the model’s visual information processing.Following prior work [37], we formulate the probe as a binary object-presence question-answering task, in which the model determines whether a queried object is present in the image (see Figure 2). To avoid biasing the extracted SAE features toward a particular response (e.g., consistently favoring “yes” or “no”), we construct 10,000 balanced queries—5,000 for objects present (GT) and 5,000 for ob- jects absent (hallucinated)—using LURE [42], a benchmark of hallucinated objects generated by GPT-3.5. Each activa- tion a(x) (as defined in Equation (1)) is assigned toX correct if the response is correct, or X hallu otherwise. Experimen- tal results verifying that there is no bias toward any specific response are provided in Section 5.4. 4.2. Identify Visual Understanding Feature While SAE features are typically associated with a concept based on frequent activation under that concept, a more ro- bust identification of visual understanding features can be achieved by explicitly contrasting activations between cor- rect and hallucinated responses [9]. For each feature j, the activation frequency in each set is computed as f correct j = 1 N correct N correct X i=1 1 a j (x correct i ) > 0 ,(4) f hallu j = 1 N hallu N hallu X i=1 1 a j (x hallu i ) > 0 (5) where a j (x) denotes the activation of feature j for input x. A separation score is then defined as s j = f correct j − f hallu j (6) which captures the degree to which a feature is preferen- tially activated for visually grounded predictions. The fea- ture with the highest s j is selected as the visual understand- ing feature, i.e., arg max j s j . Conversely, features strongly associated with hallucina- tion can be identified by reversing the separation score, s j = f hallu j − f correct j .(7) 4 CHAIR POPE MMHal-Bench RandomPopularAdversarial CHAIR S ↓CHAIR I ↓F1↑Acc↑F1↑Acc↑F1↑Acc↑Score↑HalRate↓ LLaVA- 1.6-7B Base 31.27.992.1692.1989.6689.7386.3285.902.880.41 VCD38.610.190.0189.9387.2787.2383.7282.962.650.48 DeCO38.411.792.3291.0588.7988.1683.5581.532.640.47 Devils36.09.890.1990.4889.4990.0386.6086.862.250.54 VTI29.45.891.7891.7588.9189.4884.4684.972.540.47 VISTA26.811.391.9391.9989.7189.8086.2885.862.930.41 SAVE (Ours)21.45.492.7192.6189.7489.7786.5286.373.120.36 LLaVA- NeXT-8B Base34.29.092.7492.6589.8889.7385.6084.672.830.45 VCD35.49.790.5390.4488.1688.0084.9884.202.900.45 DeCO35.611.092.7592.6189.7289.5085.3884.302.750.46 Devils21.47.090.6790.8288.6688.9385.1784.932.220.55 VTI32.85.491.7891.7589.2289.2083.9982.992.540.57 VISTA33.48.392.3792.2689.6889.5385.5184.603.020.42 SAVE (Ours)28.07.092.8592.7889.9489.8385.6884.803.210.29 Qwen2- VL-7B Base40.07.290.5690.8988.1788.6386.4386.703.310.32 SAVE (Ours)20.25.992.7292.6889.0988.9385.9085.173.700.23 Table 1. Results on CHAIR, POPE, and MMHal-Bench benchmarks, comparing SAVE with state-of-the-art training-free methods. Best and second-best results are highlighted in bold and underlined, respectively. Because most baselines are built on LLaVA, for Qwen2-VL we compare only against the vanilla model. Metric descriptions are provided in Section 5.2. Activation Analysis The resulting features exhibit clear discriminative behavior in our experiments: visual under- standing features are highly activated for correct responses (AUROC=0.93), while hallucination features are selectively activated for hallucinated responses (AUROC=0.82), as il- lustrated in Figure 3(a). 4.3. Steering Model Behavior Adjusting the model’s internal representations along these identified features strengthens visual grounding and miti- gates object hallucinations. Formally, an SAE reconstructs a hidden representation as x≈ X j a j (x)W dec [j, :] + b dec ,(8) which is equivalently expressed as Equation (2), where a j (x) is the activation of feature j and W dec [j, :] is its de- coder direction. Modulating a single feature j by a scalar α (the steering strength) shifts the representation along its decoder direction, producing x steered = x + αW dec [j, :],(9) which corresponds to activation steering in the latent space [36]. Model Behavior Analysis Steering along the visual un- derstanding feature effectively reduces object hallucination, whereas steering along the hallucination feature amplifies it. As shown in Figure 3(b), CHAIR S drops from 31.2 to 21.4 when steering toward the visual understanding feature, while steering toward the hallucination feature increases it to 38.0, with a random baseline obtained by averaging five randomly selected features in between. Qualitative exam- ples in Figure 4 further confirm this trend. 4.4. Feature Visualization To analyze the semantics of each SAE feature, we visualize the top-activated images and their most responsive patches. Following [41], all images are processed through the model to obtain hidden representations X ∈R |D|×n img ×d , where |D| is the number of image samples (indexed by k), n img is the number of image tokens per image (indexed by j), and d is the hidden feature dimension. These are then projected into the SAE latent space as Z ∈R |D|×n img ×d s , where d s is the number of SAE latent features (indexed by i). We first compute the mean activation of each feature across image tokens: ̄ Z[k,i] = 1 n img n img X j=1 Z[k,j,i], ̄ Z ∈R |D|×d s , and select the top-3 most influential images per feature as Top3Images(i) = TopK 3 ̄ Z[:,i] . Within each selected image k, the most responsive patch 5 The image appears to be a painting of a keel-billed toucan perched on a branch. The image you've provided is not of a vicuna. What are the colors of the parachutes in the sky? The parachutes in the sky are orange and blue. The parachutes in the sky are green, blue, and orange. Adversarial Attribute LLaVA-NeXT SAVE (Ours) LLaVA-NeXT What is a vicuna standing in the sand looking at? SAVE (Ours) Figure 5. Qualitative examples in MMHal-Bench. In the adversar- ial category, SAVE correctly identifies the absence of misleading objects, while in the attribute category, it preserves all relevant at- tribute information. for feature i is determined as p ∗ k,i = arg max j∈[1,n img ] Z[k,j,i]. These images and patches are then used for qualitative vi- sualization of each SAE feature. Qualitatively, visual understanding features tend to fo- cus on object regions, whereas hallucinated features often attend to background or context patches. Figure 3(c) illus- trates this contrast by highlighting the top-activated patches for each feature. 5. Experiments In this section, we evaluate the effectiveness of SAVE, which steers the model toward visual-understanding fea- tures, by comparing it against state-of-the-art training-free approaches on both binary yes/no tasks and open-ended captioning to demonstrate its effectiveness in mitigating hallucination.We conduct comparative experiments on LLaVA-1.6, LLaVA-NeXT, and Qwen2-VL. 5.1. Experimental setting Model architectures We evaluate on LLaVA-1.6- Mistral-7B [22] and LLaVA-NeXT-LLaMA3-8B [23], re-implementing all baselines on these two models for fair comparison. We additionally include Qwen2-VL-7B [38], trained following the LLaVA-NeXT procedure. Further de- tails on training the SAE for LLaVA-NeXT and Qwen2-VL are provided in Section A. Baselines Because SAVE operates by manipulating a model’s internal representations, we compare it against LLaVA-1.6-7B CHAIR S CHAIR S 24.0 21.6 26.6 29.0 21.4 Base=31.2 Qwen2-VL-7B 20.2 23.0 27.8 23.6 20.8 Base=40.0 Figure 6. CHAIR S results across five steering layers (8, 12, 16, 20, 24) for LLaVA-1.6 and Qwen2-VL-7B. The dotted line indicates the performance of the base model. post-hoc hallucination-mitigation methods.We adopt VCD [16], which mitigates hallucination through parallel contrastive learning across time steps. For more recent training-free strategies, we compare with DeCO [37] and Devils [14], which apply dynamic preceding-layer selec- tion and head-guided attention intervention, respectively. We also evaluate latent steering baselines: VTI [25], which leverages averaged intervention vectors, and VISTA [19], which derives image-specific vectors from contrastive dif- ferences with or without visual tokens. These serve as inference-time baselines most comparable to our method. 5.2. Benchmark and metric We evaluate on three standard hallucination benchmarks. Further evidence of our method’s effectiveness in vi- sual understanding, evaluated on M-Vet [40] and A-OKVQA [32], is provided in Section C. CHAIR The CHAIR metric [31] evaluates object hallu- cination in image captioning by comparing generated cap- tions with ground truth object labels. We follow Huang et al. [11] and report both sentence-level (CHAIR S ) and instance-level (CHAIR I ) scores using 500 MSCOCO 2014 validation images. CHAIR S = |captions with hallucinated objects| |all captions| is defined as the proportion of captions containing halluci- nated objects, and CHAIR I = |hallucinated objects| |all objects mentioned| measures the proportion of hallucinated object mentions. POPE The POPE benchmark [18] evaluates object hallu- cination on 500 MSCOCO images using six binary object- presence questions per image across three splits (random, popular, adversarial). It probes whether a model asserts the presence of objects not grounded in the image, providing a focused measure of hallucination robustness. Performance is reported using F1 score and accuracy. 6 Steering DirectionTop-kSAVE MetricHallucinationBothTop 3Top 5Visual & Top 1 CHAIR S ↓29.8 (+8.4)27.4 (+6.0)30.2 (+8.8)30.6 (+9.2)21.4 CHAIR I ↓6.5 (+1.1)6.9 (+1.5) 7.2 (+1.8)7.5 (+2.1)5.4 Table 2. Ablation study of SAVE along two axes: steering direction and top-k feature selection. All experiments are conducted on the CHAIR benchmark using LLaVA-1.6 with steering applied at layer 24 and α = 15. The red delta values indicate how much worse each setting performs compared to SAVE (Visual, Top 1). MMHal-Bench MMHal-Bench benchmark [33] evalu- ates MLLMs across 96 image-question pairs spanning eight categories, including attributes, counting, and adversarial objects. It emphasizes complex reasoning and visual un- derstanding, with model responses evaluated by GPT-4. 5.3. Results Main Results On CHAIR [31], SAVE reduces CHAIR S by 32% on LLaVA-1.6 (31.2→21.4), consistently out- performing all baselines, and by 18% on LLaVA-NeXT (34.2→28.0), achieving competitive performance, sec- ond only to Devils [14], and by 49% on Qwen2-VL (40.0→20.2). On POPE [18], SAVE achieves the highest average F1 and accuracy across all evaluation settings (ran- dom, popular, adversarial). Notably, it outperforms all base- lines in the random split across all models and remains com- petitive in the popular and adversarial splits, indicating ro- bust generalization across diverse model architectures. On MMHal-Bench [33], SAVE achieves the highest average score and lowest hallucination rate on both LLaVA models and Qwen2-VL, again outperforming all baseline methods, which represent the strongest inference-time competitors. Beyond the quantitative results, Figure 5 illustrates exam- ple responses where hallucination is effectively mitigated. Layer-wise Results Since an SAE can only operate meaningfully on the layer where it was trained, effective layer-wise steering requires a separately trained SAE for each target layer. We also train a Multimodal-SAE for Qwen2-VL-7B-Instruct [38] on the same five layers (8, 12, 16, 20, 24), to evaluate the generalizability of our approach across different model architectures, following exactly the same procedure used for LLaVA-NeXT [41]. As shown in Figure 6, steering at early layers (8, 12) or late layers (24) generally outperforms mid-layer steer- ing (16, 20), indicating that intermediate representations are less responsive to latent manipulation.This trend is consistent across both LLaVA-1.6 and Qwen2-VL-7B- Instruct, with layers 12 and 24 in LLaVA-1.6 and lay- ers 8 and 24 in Qwen2-VL-7B-Instruct yielding the low- est CHAIR S .Moreover, the effective steering strength varies by layer: early layers respond best to small magni- tudes (α = 3), mid-layers benefit from moderate strengths (α ∈ 3, 5), and deeper layers require stronger interven- tion (α ∈ 5, 10, 15) to achieve optimal performance. These results highlight that semantically meaningful, steer- able features emerge primarily in early and late layers, and the required steering intensity naturally increases toward deeper layers. Additional results for layer-wise SAE steer- ing are provided in Section B. Ablation study SAVE performs steering using the top-1 visual-understanding SAE feature. To justify this choice, we first compare alternative steering directions.The hallucination-related feature is applied in the opposite direction to counteract hallucinated activations, but—as shown in the Steering Direction columns of Table 2—it yields worse CHAIR scores than the visual-understanding direction. This indicates that the visual-understanding fea- ture provides the primary corrective signal, whereas the hal- lucination feature contributes less targeted information. We also vary the number of steering features and observe that steering with additional directions (top-3, top-5) further degrades performance, as shown in the Top-k columns of Table 2. This suggests that the top-1 visual-understanding feature alone captures the essential information for hallu- cination mitigation, while adding more features introduces noise. 5.4. Analysis In this section, we analyze the effects of steering the model along the visual-understanding latent direction. We first test whether such steering introduces any inherent response bias (e.g., toward answering yes or no). We then study how this steering improves performance through two complementary lenses: (1) the evolution of token-level predictions across layers and (2) attention shifts between query and image to- kens. Unless otherwise noted, all steering is applied at layer 25 of LLaVA-NeXT. Response Shift Analysis As discussed in 4.1, we ana- lyzed the transitions where model predictions changed from incorrect to correct after steering along the identified visual understanding features. On the POPE dataset, the number of yes→no transitions is 37, while the number of no→yes transitions is 41—an approximately balanced pattern. This 7 Describe this image in detail. .... In the distance, a boat can be seen floating on the water, adding a touch of life to this peaceful scene... boat hill mountain cliff line 0.00 0.30 0.20 0.10 Vanilla - Last Layer Vanilla SAVE (Ours) SAVE (Ours)- Last Layer mountain hill small lush green 0.00 0.30 0.20 0.10 Layer Probability of “boat” Figure 7. Comparison of vanilla and steered model outputs for a hallucinated token (”boat”). The left panel shows the user query, image, and a hallucinated caption generated by the vanilla model. The right panel presents layer-wise probabilities for ”boat” and the top-5 tokens at the final layer. SAVE suppresses the probability of the hallucinated token and instead produces visually grounded alternatives, demonstrating reduced hallucination. Image Query SAVE (Ours) Original Average Attention Figure 8. Attention scores for the generated token, averaged across all layers. Steering increases attention to the image and decreases attention to the text. symmetry, averaged over the three POPE types (random, popular, and adversarial), indicates that our steering does not favor a particular response type and therefore does not introduce answer bias. Layer-wise token probability To analyze the evolution of hallucinated token probabilities across layers, we com- pare the vanilla LLaVA-NeXT model with SAVE. We first generate image captions with the vanilla model on the CHAIR benchmark and identify hallucinated words using the ground-truth annotations. Following DeCO [37], we then evaluate both models while conditioning on a fixed prefix extracted from the vanilla model’s caption (e.g., ‘In the distance, a’ in Figure 7) to probe layer-wise predic- tions. As shown in Figure 7, the vanilla model sharply in- creases the probability of the hallucinated token “boat” at the penultimate layer, indicating that hallucination emerges late in decoding. In contrast, SAVE—steered along visual- understanding features—shows no such spike and instead consistently favors the grounded token “mountain.” This suggests that latent steering with sparse autoencoders sup- presses late-stage hallucinatory drift and promotes align- ment with visually grounded concepts. Attention-score We follow Liu et al. [25] to analyze the attention distribution from a single generated token to both query and image tokens, measured on the POPE bench- mark. Specifically, we compute average attention scores over the final 8 layers (layers 25–32), where steering is ap- plied. As shown in Figure 8, steering toward the correct la- tent decreases attention to query tokens while increasing at- tention to image tokens. This shift indicates that SAVE mit- igates text-biased generation [43] and encourages the model to rely more on visual evidence, resulting in more grounded predictions. 6. Conclusion We present SAVE, the first simple and effective frame- work that leverages SAEs to mitigate object hallucina- tion in MLLMs. SAVE identifies latent features that cap- ture the model’s visual understanding and steers the model along these directions, effectively reducing hallucinated responses. Extensive experiments across diverse MLLM architectures demonstrate that SAVE consistently outper- forms state-of-the-art training-free approaches, validating both the effectiveness and generalizability of our framework for hallucination mitigation. 8 Acknowledgements This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Ko- reagovernment(MSIT)[No.2022R1A3B1077720; No.2022R1A5A7083908], Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [No.RS-2025-02263754; No.RS-2022-I220959; No.RS- 2021-I211343, Artificial Intelligence Graduate School Program (Seoul National University)], the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2025. This research was also conducted as part of the Sovereign AI Foundation Model Project (Data Track), organized by MSIT and supported by the National Information Society Agency (NIA) of Korea [No.2025-AI Data-wi43]. References [1] Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930, 2024. 1, 2 [2] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yi- fan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah.Towards Monosemanticity: Decomposing Language Models With Dictionary Learning, 2023. 2, 3 [3] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pages 370–387. Springer, 2024. 3 [4] Yeongjae Cho, Keonwoo Kim, Taebaek Hwang, and Sung- zoon Cho. Do you keep an eye on what i ask? mitigat- ing multimodal hallucination via attention-guided ensemble decoding. In The Thirteenth International Conference on Learning Representations, 2025. 2 [5] Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023. 2, 3 [6] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Thirty- seventh Conference on Neural Information Processing Sys- tems, 2023. 1 [7] Gytis Daujotas. Case study: Interpreting, Manipulating, and Controlling CLIP with sparse autoencoders, 2024. 2 [8] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield- Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition, 2022. 2 [9] Javier Ferrando, Oscar Balcells Obeso, Senthooran Raja- manoharan, and Neel Nanda. Do i know this entity? knowl- edge awareness and hallucinations in language models. In The Thirteenth International Conference on Learning Rep- resentations, 2025. 4 [10] Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Rep- resentations, 2025. 3, 1 [11] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu.Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13418–13427, 2024. 1, 6 [12] Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao.Self-introspective de- coding: Alleviating hallucinations for large vision-language models.In The Thirteenth International Conference on Learning Representations, 2025. 2 [13] Nicholas Jiang, Anish Kachinthaya, Suzanne Petryk, and Yossi Gandelsman. Interpreting and editing vision-language representations to mitigate hallucinations.In The Thir- teenth International Conference on Learning Representa- tions, 2025. 1, 4 [14] Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large vision- language models: Interpreting, detecting and mitigating ob- ject hallucinations via attention lens. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 25004–25014, 2025. 1, 4, 6, 7 [15] Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models. In The Thirteenth International Confer- ence on Learning Representations, 2025. 2 [16] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 13872–13882, 2024. 1, 2, 3, 6 [17] Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger llms supercharge multimodal capa- bilities in the wild, 2024. 1 [18] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292–305, Singapore, 2023. Association for Computational Linguistics. 2, 6, 7 [19] Zhuowei Li, Haizhou Shi, Yunhe Gao, Di Liu, Zhenting Wang, Yuxiao Chen, Ting Liu, Long Zhao, Hao Wang, and 9 Dimitris N. Metaxas. The hidden life of tokens: Reducing hallucination of large vision-language models via visual in- formation steering, 2025. 1, 2, 6 [20] Hyesu Lim, Jinho Choi, Jaegul Choo, and Steffen Schnei- der. Sparse autoencoders reveal selective remapping of vi- sual concepts during adaptation. In The Thirteenth Interna- tional Conference on Learning Representations, 2025. 2, 3 [21] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 1 [22] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306, 2024. 2, 6 [23] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 2, 6, 1 [24] Shi Liu, Kecheng Zheng, and Wei Chen. Paying more at- tention to image: A training-free method for alleviating hal- lucination in lvlms. In European Conference on Computer Vision, pages 125–140. Springer, 2024. 1, 2 [25] Sheng Liu, Haotian Ye, and James Zou. Reducing hallucina- tions in large vision-language models via latent space steer- ing. In The Thirteenth International Conference on Learning Representations, 2025. 1, 2, 6, 8 [26] Hantao Lou, Changye Li, Jiaming Ji, and Yaodong Yang. Sae-v: Interpreting multimodal models for enhanced align- ment, 2025. 2 [27] Chris Olah. Mechanistic interpretability, variables, and the importance of interpretable bases, 2022. 1, 2 [28] ChrisOlah,NickCammarata,LudwigSchubert, GabrielGoh,MichaelPetrov,andShanCarter. Zoom in:An introduction to circuits.Distill, 2020. https://distill.pub/2020/circuits/zoom-in. 2 [29] Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):3311–3325, 1997. 3 [30] Achyuta Rajaram, Sarah Schwettmann, Jacob Andreas, and Arthur Conmy. Line of sight: On linear representations in vllms. arXiv preprint arXiv:2506.04706, 2025. 2, 3 [31] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image cap- tioning. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 4035– 4045, Brussels, Belgium, 2018. Association for Computa- tional Linguistics. 2, 6, 7 [32] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi.A-okvqa: A benchmark for visual question answering using world knowl- edge. In European conference on computer vision, pages 146–162. Springer, 2022. 6, 3 [33] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell.Aligning large multimodal models with factu- ally augmented RLHF. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13088–13110, Bangkok, Thailand, 2024. Association for Computational Linguistics. 2, 7 [34] Wei Suo, Lijun Zhang, Mengyang Sun, Lin Yuanbo Wu, Peng Wang, and Yanning Zhang. Octopus: Alleviating hal- lucination via dynamic contrastive decoding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 29904–29914, 2025. 1, 3 [35] Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lind- sey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunning- ham, Nicholas L Turner, Callum McDougall, Monte Mac- Diarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Ex- tracting interpretable features from claude 3 sonnet. Trans- former Circuits Thread, 2024. 2, 3, 1 [36] Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without opti- mization. arXiv e-prints, pages arXiv–2308, 2023. 5 [37] Chenxi Wang, Xiang Chen, Ningyu Zhang, Bozhong Tian, Haoming Xu, Shumin Deng, and Huajun Chen. MLLM can see? dynamic correction decoding for hallucination mitiga- tion. In The Thirteenth International Conference on Learn- ing Representations, 2025. 2, 4, 6, 8 [38] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2, 6, 7 [39] Tianyun Yang, Ziniu Li, Juan Cao, and Chang Xu. Un- derstanding and mitigating hallucination in large vision- language models via modular attribution and intervention. In The Thirteenth International Conference on Learning Repre- sentations, 2025. 2 [40] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 6, 3 [41] Kaichen Zhang, Yifei Shen, Bo Li, and Ziwei Liu. Large multi-modal models can interpret features in large multi- modal models, 2024. 2, 3, 5, 7, 1 [42] Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. In The Twelfth International Con- ference on Learning Representations, 2024. 4, 1 [43] Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating hallucinations in large vision- language models via image-biased decoding. arXiv preprint arXiv:2402.18476, 2024. 8 10 SAVE: Sparse Autoencoder-Driven Visual Information Enhancement for Mitigating Object Hallucination Supplementary Material A. Experimental Details LURE Following LURE [42],we generate hallu- cinated objects for our visual information processing probe using GPT-3.5, which predicts objects likely to co-occur with the given image and prompt. Specifically, we prompt GPT-3.5 with: "List three other objects that you think are most likely to appear with the objects in the scene described below." Model Architectures • LLaVA-1.6 For more details than those provided in 3.2, the SAEs are trained with a density factor of λ = 5, which is linearly increased from 0 during the first 5% of training steps to encourage sparsity. Training uses a total of 1.5B tokens with batches of 4096, shuffled to balance text and image tokens. The learning rate is 5e-5, decayed to zero over the final 20% of training, and optimization is per- formed using Adam. • LLaVA-NeXT The Multimodal-SAE proposed by Zhang et al. [41] integrates a SAE into the 25th transformer layer of LLaVA-NeXT-LLaMA3-8B [17], where the hid- den representation at that layer serves as the SAE input x. The SAE is trained on the LLaVA-NeXT supervised fine-tuning dataset [23], which consists of approximately 779,000 samples, using the AnyRes strategy for process- ing images of varying resolutions. mage and text inputs are preprocessed in the same way as during supervised fine-tuning. The SAE is configured with 2 17 latent features and em- ploys top-k sparsity, following Gao et al. [10] for the sparsity mechanism. We set k = 256 to match the ac- tivation patterns observed in Templeton et al. [35], pro- moting disentangled and semantically meaningful repre- sentations. Unless otherwise noted, all experiments are conducted using these SAE settings. • Qwen2-VL Qwen2-VL-7B is trained following the exact same process as LLaVA-NeXT. Configurations Table 1 reports results under the follow- ing configurations. For LLaVA-1.6, we evaluate POPE us- ing steering at layer 24 with a strength (α) of 10, CHAIR at layer 24 with a strength of 15, and MMHal-Bench at layer 20 with a strength of 5. For LLaVA-NeXT, all benchmarks are evaluated with steering at layer 24, using a strength of 10 for POPE and MMHal-Bench and 15 for CHAIR. We set VISTABaseVTISAVE (Ours) 3.17 3.50 3.50 1.42 4.33 3.08 3.50 3.27 3.67 1.17 2.75 4.25 2.25 2.50 2.84 3.42 3.42 2.92 Counting Comparison Adversarial Attribute Average Other Holistic Environment Relation Figure S1.Task-wise MMHal-Bench performance on LLaVA-NeXT for different steering methods. maxnewtokens to 2 for POPE, 256 for MMHal-Bench, and 512 for CHAIR. Steering We apply steering—i.e., modifying the encoded representation—only to the input query tokens, not to the tokens generated by the model. The steering mechanism described in Section 4.3 applies to LLaVA-1.6. Since LLaVA-NeXT is trained differently, it requires a distinct steering strategy, which we also adopt for Qwen because it follows the same training process. In our experiments, this strategy is applied to both models, as formulated in the equation below, where T denotes tokens. z(x) = ReLU(W enc (x− b pre ) + b enc )(10) ˆz(x)[T,j] = α(steering)(11) a(x) = TopK(ˆz(x))(12) x steered = W dec · a(x) + b dec (13) Software and HardwareAll experiments were con- ducted using PyTorch and an NVIDIA A40 GPU. B. Detailed Results MMHal-Bench MMHal-Bench reports scores across di- verse tasks. As a detailed breakdown of Table 1, Figure S1 visualizes the task-wise performance on LLaVA-NeXT. 1 89.38 89.42 89.65 89.24 89.27 89.32 89.58 89.22 Visual Understanding POPE HallucinatedBaseRandom Figure S2.POPE evaluation of steering toward random, visual-understanding, and hallucinated features. 2.83 2.86 3.21 2.23 0.45 0.43 0.29 0.55 Visual Understanding MMHal-Bench Hallucination rate Score BaseRandomHallucinated Figure S3. MMHal-Bench evaluation of steering toward random, visual-understanding, and hallucinated features. Compared with the base model and various steering meth- ods, SAVE consistently outperforms all baselines across ev- ery task. Steering Model Behavior Additional results for the LLaVA-1.6 steering experiments presented in 4.3 are shown in Figure S2 and Figure S3. We report F1 and Accu- racy for POPE, and GPT-4 Score with hallucination rate for MMHal-Bench.Across both benchmarks, steering along visual-understanding features mitigates hallucination, whereas steering along hallucinated features amplifies it. For this experiment, we follow the same configurations as in Table 1. Layer-wise Steering Strength Ablation Table S1 and Table S2 provide the detailed results corresponding to Fig- ure 6, for LLaVA-1.6 and Qwen2-VL, respectively. In Fig- ure 6, we report the best performance per layer, selecting the steer strength that achieves the highest score. The optimal LayerSteer CHAIRPOPE CHAIR S CHAIR I F1Acc 8 1.5 23.86.188.2687.99 3– 12 321.66.089.2788.90 5– 16 326.66.889.1989.19 5 –89.1588.78 20 531.67.989.3189.21 1033.08.189.2689.17 1529.010.389.0688.97 24 532.28.489.4189.21 1029.47.289.6589.58 15 21.45.489.5589.46 Table S1. Performance of LLaVA-1.6 on CHAIR and POPE across layers and steering strengths. LayerSteer CHAIRPOPE CHAIR S CHAIR I F1Acc 8 1.517.85.189.2488.93 320.25.989.2288.89 12 323.06.588.4888.92 524.26.988.5488.99 16 327.86.788.9989.22 526.86.589.0189.24 20 523.86.488.7589.03 1023.66.388.7889.07 1524.46.888.7989.08 24 5 20.88.085.1686.61 10 21.08.085.2886.71 1521.48.285.2886.71 Table S2. Performance of Qwen2-VL on CHAIR and POPE across layers and steering strengths. steer strength varies across layers: lower strengths (1.5 or 3) are more effective in early layers, while higher strengths (10 or 15) perform better in later layers. Through extensive ab- lations, we observe that overly strong steering at any layer can corrupt the model’s output—e.g., generating repeated blanks or meaningless responses such as “If you have any questions about the image, please provide more informa- tion” (as observed in LLaVA-1.6, layer 8, steer strength 3). In the tables, a “–” indicates such corrupted outputs. Al- though the severity varies by layer, these findings highlight the importance of carefully selecting an appropriate steer strength to avoid response degradation. 2 MethodInference Time (s)Generated TokensTotal GFLOPsFLOPs / Token Vanilla7.99821538,166.48177.52 VTI12.55324838,662.83155.90 VISTA8.57319237,822.56196.99 Devils 8.3906037,163.19619.39 DeCo10.85022067,947.49308.85 SAVE (Ours)8.86317440,281.54231.50 Table S3. Inference efficiency comparison across methods. ModelMethodRecOCRKnowGenSpatMathTotal LLaVA-1.6 Base 45.236.533.636.035.526.542.2 SAVE (Ours)44.342.030.532.142.726.543.7 LLaVA-NeXT Base 40.941.025.427.542.419.241.8 SAVE (Ours)43.143.130.232.543.122.742.2 Table S4. Per-task MMVet evaluation results on LLaVA-1.6 and LLaVA-NeXT. MethodLLaVA-1.6LLaVA-NeXT Base66.8965.58 SAVE (Ours)70.0466.81 Table S5.A-OKVQA multiple-choice accuracy (%) on LLaVA-1.6 and LLaVA-NeXT. C. Additional Results Inference Efficiency We compare SAVE with sev- eral recent training-free methods—all reproduced on the LLaVA-NeXT backbone for consistency—using FLOPs and latency-based metrics. FLOPs are measured over the token generation process, which dominates overall infer- ence computation. As shown in Table S3, SAVE achieves a favorable trade-off between efficiency and effectiveness: it generates a similar number of tokens while maintaining lower total FLOPs and FLOPs per token than strong base- lines such as DeCo and Devils. Moreover, it runs faster than both DeCo and VTI, demonstrating efficiency in terms of both computation and latency. VQA & General MLLM benchmark • M-VET M-Vet [40] assesses visual understanding across six tasks—Recognition, OCR, Knowledge, Lan- guage Generation, Spatial Awareness, and Math. Ta- ble S4 shows that SAVE outperforms the base models on both LLaVA-1.6 and LLaVA-NeXT in terms of total score, using layer 20 with a steering strength of 3 for LLaVA-1.6, and layer 24 with a steering strength of 10 for LLaVA-NeXT. • A-OKVQA A-OKVQA [32] is a crowdsourced bench- mark of about 25K diverse questions that require com- monsense reasoning about visual scenes beyond simple knowledge-base queries. As shown in Table S5, SAVE achieves higher multiple-choice accuracy than the base models, indicating that steering along visual understand- ing features not only mitigates hallucination but also en- hances general visual understanding. Results are reported using LLaVA-1.6 with layer 20 and steering strength 5, and LLaVA-NeXT with layer 24 and steering strength 10. Steer Strength We conduct an ablation study on steering strength (Figure S4). For LLaVA-NeXT at layer 24, we evaluate steer strengths of 3, 5, 10, 15, 20. The best results are obtained with a strength of 10 on POPE and MMHal- Bench, and 15 on CHAIR. Extending the scope of SAE feature identification be- yond object presence We conduct experiments on four visual reasoning tasks—existence (which corresponds to the object-presence signal used in the object-presence-only setting), count, position, and color—to evaluate the scala- bility of SAE-based feature extraction. In this setup, we use AMBER, a dataset organized by task type (see Figure S5), to obtain SAE activations, while evaluation is performed on MME to ensure generalizability. As shown in Table S6, LURE features naturally improve performance on the exis- tence task, as they explicitly encode existence-related cues. This also explains why the existence scores of LURE and AMBER settings are identical. Interestingly, the LURE steering also improves performance on the color task, sug- gesting that visual cues associated with object existence im- plicitly contribute to color understanding as well. 3 Steer Strength CHAIR S CHAIR I CHAIR 31.8 32.2 32.0 29.2 31.4 8.2 8.1 8.5 7.2 7.5 Score Hallucination Rate Steer Strength MMHal-Bench 3.10 3.00 3.22 2.85 3.01 0.40 0.43 0.40 0.43 0.43 Accuracy F1 89.42 89.47 89.50 89.44 89.43 89.00 89.07 89.10 89.06 88.98 Steer Strength POPE Figure S4. Steer strength ablation results for LLaVA-NeXT on POPE, CHAIR, and MMHal-Bench. TaskVanillaLUREAmber Existence195200 (+5)200 (+5) Color155160 (+5)160 (+5) Count115115 (+0)120 (+5) Position93.393.3 (+0) 98.3 (+5) Table S6.Amber-derived SAE features outperform object- presence-only features particularly on count and relation (posi- tion) tasks. In contrast, SAE features extracted from the AMBER dataset—reflecting a richer variety of task-specific sig- nals—lead to further gains in count and position. This sug- gests that when SAE directions encode broader aspects of the model’s visual understanding, steering along those di- rections yields more comprehensive and balanced improve- ments across tasks. Overall, these findings demonstrate that our SAE-based latent steering approach is not limited to ob- ject presence, but can be effectively extended to support a wide range of visual reasoning objectives. Qualitative Results for CHAIR Figure S6 shows qual- itative CHAIR results. By steering along the identified visual-understanding features to enhance visual informa- tion, SAVE produces more visually grounded answers. Statistical Test We conducted statistical significance testingonhallucinationratesbetweenLLaVA-1.6 Base (CHAIR S =31.2,CHAIR I =7.9) and LLaVA-1.6 SAVE (Ours) (CHAIR S =21.4, CHAIR I =5.4). Since the sample-level hallucination scores deviated from normality (Shapiro-Wilk p < 0.001), we employed the Wilcoxon signed-rank test, a non-parametric paired test assessing whether the median differences between models are sys- tematically biased. At the instance level (CHAIR I ), SAVE achieved a modest yet statistically significant improvement (W = 4998.0, p = 0.0188, significant at p < 0.05). At the sentence level (CHAIR S ), the reduction was highly significant (W = 2600.0, p = 0.00002, significant at p < 0.001), demonstrating that SAVE consistently pro- “Is there a tissue in this image?” “Is the cream pink in this image??” “Are there two plates in this image?” “Is there direct contact between the strawberry and plate?” Existence Color Count Relation [N] [Y] [N] [Y] Figure S5. Examples of diverse question types from AMBER. For each task, both ‘yes’ and ‘no’ answer cases are included. duces fewer hallucinated captions than the baseline, with the most pronounced effect at the sentence level. 4 LLaVA-NeXT:The room itself is well-lit, with multiple screens and a clock visible in the background. The presence of these screens and the clock suggests that this is a professional setting, possibly a conference hall or seminar room. Ours:The setting seems to be a conference room or seminar hall, as there are multiple people seated in the background. The room is well- lit, with several screens and monitors placed around the room, likely for presentations or other multimedia content. LLaVA-Next: The person is seated in a chair, their body angled slightly to the left, creating a sense of depth in the image. Their right arm is casually resting on the armrest of the chair, while their left arm is draped over the back of the chair, adding a relaxed vibe to their otherwise formal appearance. Ours: The background is blurred, but it appears to be an indoor setting with a wall that has some text on it, although the text is not clearly legible. The lighting in the image is soft and seems to be coming from the left side, casting a gentle shadow on the person's right side. LLaVA-NeXT: The bathroom itself is bathed in a soft, warm light that accentuates the beige tiles and the white sink. The sink, located on the right side of the image, is adorned with a few toiletries -a bottle of soap and a toothbrush holder. Ours: The bathroom has a beige color scheme with white tiles on the walls and a window with frosted glass to the left. On the window sill, there are two bottles, one of which appears to be a bottle of hand soap. Figure S6. Qualitative comparison between vanilla LLaVA-NeXT and our method on the CHAIR benchmark. Our steered model generates visually grounded captions, while vanilla LLaVA-NeXT exhibits object hallucination. Hallucinated words are highlighted in red, and correct responses are shown in green. 5

Instant research discovery