Paper deep dive

Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination

Zichuan Wang, Songlin Yang, Bo Peng, Zhenchen Tang, Yang Li, Beibei Dong, Jing Dong

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 50

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large Vision-Language Models (LVLMs) often suffer from object hallucination, generating objects that are absent from the image. Prior work largely attributes this to insufficient visual attention. However, we find that both real and hallucinated objects receive equally strong visual attention in the model's mid-to-late layers, suggesting that the key issue may not be how much the model attends, but what it attends to and why. To this end, we decode the visual features of high-attention regions using Logit Lens, and observe that regions corresponding to real objects can be correctly decoded to the target object tokens, whereas those for hallucinated objects cannot. Building on this, we identify two hallucination mechanisms: (i) visual uncertainty, triggered by semantically similar or confusable regions; masking these regions eliminates the hallucination. (ii) contextual prior, triggered by strong co-occurrence priors; even when the initially attended region is masked, the hallucination persists and attention drifts to other regions. Based on these findings, we propose a simple yet effective training-free Detect-Mitigate framework comprising a Logit-Lens Consistency Check to detect hallucination and targeted remedies: High-Attention Regions Masking (HARM) for visual uncertainty hallucination, and Visual Evidence Enhanced Decoding (VEED) for contextual prior hallucination. Our approach achieves state-of-the-art results on multiple hallucination benchmarks. Code will be available.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

49,031 characters extracted from source content.

Expand or collapse full text

Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination Zichuan Wang 1,2 Songlin Yang 3 Bo Peng 2 * Zhenchen Tang 1,2 Yang Li 2 Beibei Dong 2 Jing Dong 2 1 School of Artificial Intelligence, University of Chinese Academy of Sciences 2 New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Science 3 Hong Kong University of Science and Technology wangzichuan2024,tangzhenchen2024,liyang2022,dongbeibei2022@ia.ac.cn syangds@connect.ust.hk,bo.peng,jdong@nlpr.ia.ac.cn Abstract Large Vision-Language Models (LVLMs) often suffer from object hallucination, generating objects that are absent from the image. Prior work largely attributes this to insuf- ficient visual attention. However, we find that both real and hallucinated objects receive equally strong visual attention in model’s mid-to-late layers, suggesting that the key issue may not be how much the model attends, but what it attends to and why. To this end, we decode the visual features of high-attention regions using Logit Lens, and observe that regions corresponding to real objects can be correctly de- coded to the target object tokens, whereas those for halluci- nated objects cannot. Building on this, we identify two hal- lucination mechanisms: (i) visual uncertainty, triggered by semantically similar or confusable regions, masking these regions eliminates the hallucination. (i) contextual prior, triggered by strong co-occurrence priors, even when the initially attended region is masked, the hallucination per- sists and attention drifts to other regions. Based on these findings, we propose a simple yet effective training-free De- tect–Mitigate framework comprising a Logit-Lens Consis- tency Check to detect hallucination and targeted remedies: High-Attention Regions Masking (HARM) for visual uncer- tainty hallucination, and Visual Evidence Enhanced Decod- ing (VEED) for contextual prior hallucination. Our ap- proach achieves state-of-the-art results on multiple hallu- cination benchmarks. Code will be available. 1 1. Introduction Large Vision-Language Models (LVLMs) have made sig- nificant strides in recent years, demonstrating impressive capabilities on cross-modal tasks [3, 8, 10, 19, 20, 25]. * Corresponding author. 1 https://github.com/wzczc/SADT The image features a table with chairs placed outside, surrounded by a beautiful garden setting. ....... The table is adorned with a vase, a bowl, and a cup, adding to the inviting atmosphere. LLaVA Describe this image. User Real token ‘chair’Hallucinated token ‘bowl’ (a) Attention Heatmap Mean Attention Magnitude (± Std) on COCO Subset (b) Figure 1. Different from prior work that attributes object halluci- nation to insufficient visual attention, we find that real and halluci- nated objects often have the same attention magnitudes, as shown qualitatively in (a) and quantitatively in (b), reflecting “Same At- tention, Different Truths”. However, they still suffer from object hallucination [4, 22, 27], i.e., generating objects that are not present in the image, undermining their reliability and practical use. A prevailing view attributes hallucination to insuffi- cient visual attention, either because textual priors dom- inate the multimodal interaction and suppress visual fo- cus [5, 16, 22, 36, 40], or because improper processing of visual information leads to inadequate attention to salient regions [2, 15, 17]. Accordingly, many methods attempt to amplify or redistribute attention [11, 15, 17, 28, 43], or in- ject extra visual guidance during training/inference [7, 18, 34, 45]. However, our experiments reveal that this expla- arXiv:2608.07302v1 [cs.CV] 7 Aug 2026 nation is incomplete: the key issue may not be how much the model attends to the image, but what it attends to and why. To support this claim, we first examine how much the model attends. By quantifying the attention strength asso- ciated with object tokens, we find that visual attention con- sistently peaks in the mid-to-late layers, which we define as the “image attention stage”. Both real and hallucinated ob- ject tokens exhibit similar attention magnitudes and focus on specific regions in this stage, as shown in Fig. 1. Thus, the occurrence of hallucination cannot be explained simply by insufficient visual attention. Next, to understand what the model attends to, we an- alyze the semantics behind these high-attention regions us- ing the Logit Lens [31], which uses the model’s textual de- coding head to translate visual representations into text to- kens. We observe that regions corresponding to real object tokens can be accurately decoded into their target tokens, whereas those corresponding to hallucinated object tokens fail to produce semantically consistent outputs. This reveals a semantic misalignment between visual semantics and out- put tokens within high-attention regions, showing that hal- lucination is typically accompanied by inconsistent seman- tic decoding, rather than by the lack of visual attention. To further probe why the model attends to these regions when hallucination occurs, we mask these regions and ex- amine how the model’s output changes. This intervention reveals two distinct hallucination mechanisms. (1) Visual Uncertainty Hallucination: triggered by confusing or se- mantically related regions (e.g., a fuzzy ceramic-like area mistaken for a “bowl”). Masking these high-attention re- gions directly eliminates the hallucination. This shows that the model cannot extract precise semantics from highly un- certain visual evidence, leading to decoding inconsistency. (2) Contextual Prior Hallucination: dominated by co- occurrence priors (e.g., generating “microwave” in a de- scription of a kitchen even though it is absent). In this case, masking the high-attention regions does not remove the hal- lucination, and attention shifts to other regions. Although these regions can be correctly decoded into their target to- kens, they are not selected in the final output due to the sup- pression by strong priors. Here, the attention mechanism behaves more like a “routine”: the model has learned that it must attend to some region before generating an object and thus anchors attention merely to satisfy this requirement. Based on these findings, we propose a simple yet ef- fective training-free Detect–Mitigate framework to pre- cisely locate and reduce hallucinations. First, we propose the Logit-Lens Consistency Check to detect hallucinated to- kens by comparing whether the semantic decoding of high- attention regions in the mid-to-late layers is consistent with the token being generated. Then, we propose corresponding mitigation methods for each category. For the visual un- certainty cases, we perform simple High-Attention Regions Masking (HARM) to remove unreliable visual evidence. For the contextual prior cases, we propose Visual Evidence Enhanced Decoding (VEED). Here, the high-attention re- gions can still be decoded to the correct tokens via Logit Lens, but are suppressed during generation due to the strong priors. We therefore inject the true visual evidence from these regions into the generation process, enhancing the contribution of image-grounded semantics and suppressing contextual priors. Finally, we apply our strategy to multi- ple LVLMs on several hallucination benchmarks, achieving SOTA results, demonstrating its effectiveness and general- ity. In summary, our contributions are as follows: • We reveal the “same attention, different truth” phe- nomenon, showing the key of object hallucination is the semantic inconsistency between high-attention visual fea- tures and the text output, not the attention strength. Based on this, we categorize object hallucination into two core mechanisms: visual uncertainty and contextual prior. • We propose a simple yet effective training-free “Detect- Mitigate” framework, which includes: (1) the Logit-Lens Consistency Check for detection and (2) two mitigation strategies targeting each mechanism: High-Attention Re- gions Masking and Visual Evidence Enhanced Decoding. • Our method achieves SOTA performance on multiple object hallucination benchmarks across several LVLMs, proving its effectiveness and generalizability. 2. Related Work 2.1. Large Vision-Language Models Recent years have witnessed remarkable progress in Large Vision-Language Models (LVLMs), which integrate lan- guage models with visual encoders to perform unified mul- timodal capacity. Early work such as CLIP [32] established large-scale vision–language pretraining via contrastive im- age–text alignment. Subsequent models, including BLIP- 2 [19] and Flamingo [1], further bridged modality gaps via learnable adapters or cross-attention mechanisms. More re- cently, instruction-tuned LVLMs such as LLaVA [25, 26], Shikra [6], mPLUG-Owl2 [42], Qwen-VL [3] and In- ternVL [8] have demonstrated impressive open-ended gen- eration and reasoning capabilities. Despite these advances, LVLMs still frequently hallucinate objects, i.e., describe ob- jects that are absent from the image, which undermines their reliability in practical applications. 2.2. Object Hallucination in LVLMs Object hallucination is a critical challenge for LVLMs. Ex- isting mitigation methods fall into two categories. The first is training-stage methods, such as introducing additional datasets [7, 24], incorporating new training objectives [13, 29], and reinforcement learning methods [41, 44, 46]. How- ever, they are costly in terms of time and resources. The sec- ond is inference-stage methods, including contrastive de- coding [18, 21], guided decoding [9, 45], and visual en- hancement [2, 43]. Recently, a growing body of work has begun to investigate the internal mechanisms of hallucina- tion at inference time, with many studies attributing it to in- sufficient visual attention. [18, 28, 40, 43] find that textual priors often dominate multimodal interaction. In parallel, [2, 15, 17, 36] show that improper processing of visual in- formation also results in weak visual attention. Building on these, [15, 17, 43] propose reallocating or amplifying at- tention to strengthen the visual pathway, while [18, 28, 37] leverage visually guided to counteract language priors. Our work follows the attention-based analysis but takes a step further. We argue that the “insufficient visual attention” ex- planation is incomplete, as our findings show real and hal- lucinated objects can receive comparable attention magni- tudes, while the semantic consistency between the attended regions and the textual output may be the key. 3. Object Hallucination Analysis In this section, we analyze the mechanisms underlying ob- ject hallucination in LVLMs. We begin with the necessary preliminaries, then compare the attention patterns of hallu- cinated and non-hallucinated objects and use the Logit Lens to identify their semantic differences. Finally, we conduct intervention experiments that categorize object hallucina- tion into two distinct types. 3.1. Preliminary LVLM Generation Process. LVLMs typically consists of a vision encoder, a cross-modal projector and a LLM. Given an image I and a textual prefix Q <t = q 1 ,...,q t−1 , the vision encoder maps I to image tokens V =v 1 ,...,v N , which the projector M transforms into ̃v i = M (v i ). These tokens are then concatenated with system tokens S = s 1 ,...,s M and the text prefix to form the input sequence X = [s 1 ,...,s M , ̃v 1 ,..., ̃v N , q 1 ,...,q t−1 ], which is fed into a multi-layer transformer decoder with causal masking. At generation step t, the output head W U maps the hidden state h (L) t of the last layer L to the next-token distribution: p(x t | I,X <t ) = softmax(W U h (L) t ).(1) Attention Mechanism.In LVLMs, attention trans- fers information between tokens. The autoregressive trans- former uses causal attention: at step t, the query can only attend to tokens at positions j ≤ t, while positions j > t are masked. Let q t be the query, and k j be the keys. The mask m tj is defined as m tj = 0 when j ≤ t and−∞ otherwise. The attention weight between tokens are then computed as: α tj = softmax q ⊤ t k j √ d k + m tj .(2) 0.0 0.2 0.4 0.6 0.8 Attention Ratio (a) 148121620242832 Layer 0.0 0.2 0.4 0.6 0.8 Attention Ratio (b) SystemImageTextImage Attention Stage Figure 2. Attention Ratio across layers. (a) Object attention; (b) Non-object attention. Non-object tokens exhibit minimal attention to the image, while for object tokens we partition the attention into four stages according to their variations through layers, with the mid-to-late layers (20-27) being the Image-Attention Stage. Logit Lens. Logit Lens is an interpretability method that decodes the hidden states of intermediate layers in LLM into vocabulary space, revealing how information evolves across layers. Given a hidden state h (l) t , the output head W U produces a probability distribution over the vocabulary: p (l) lens (·| h (l) t ) = softmax W U h (l) t .(3) In this work, we apply it to the hidden states of image tokens to interpret how the model processes visual information at different layers. 3.2. Same Attention: Attention Magnitude Is Not to Blame To analyze whether attention magnitude play a role in gen- erating hallucinated object tokens, we adopt LLaVA-1.5- 7B [26] as an example, randomly sampling 500 images from the COCO 2014 validation set [23] and use the prompt “Describe this image.” for generation. For each generated token, we collect its attention distribution data across all the layers. We first compare the attention distributions of gen- erated object tokens versus non-object tokens, as shown in Fig. 2. It is observable that, apart from the initial 1-2 layers, non-object tokens exhibit minimal attention to the image. In contrast, object tokens exhibit a four-stage pattern: • Initialization Stage: In the first 1–2 layers, attention is high on both image and text, as the model rapidly aggre- gates multimodal context. Figure 3. Both the real object token (a) ‘chair’ and the hallucinated object token (b) ‘bowl’ exhibit attention that converges on specific regions during the image-attention stage (layers 20–27). • Text-Attention Stage: In the early-to-mid layers (3-19), text attention dominates while image attention remains relatively low. • Image-Attention Stage: In the mid-to-late layers (20- 27), attention to the image increases and becomes dom- inant, while text attention drops. The model focuses on extracting evidence related to the candidate objects. • Language-Organization Stage: In the final layers (28-32), image attention decreases again while text attention rises, as the model is shifting to organize and generate a gram- matically and logically coherent output. We then ask whether real and hallucinated object to- kens differ in how much they attend to the image. We employ CHAIR [33] to identify real and hallucinated ob- ject tokens and quantify their average image attention dur- ing the Image-Attention Stage, as shown in Fig. 1 (b). The results show no systematic difference between the two cat- egories. To provide further qualitative evidence, we visu- alize the evolution of image attention across layers when generating object tokens. As shown in Fig. 3, both real and hallucinated object tokens focus their attention on specific regions during the Image-Attention Stage, with comparable magnitudes. These observations indicate that object hallu- cination is not caused by insufficient visual attention. 3.3. Different Truths: Semantic Inconsistency Revealed by Logit-Lens The previous section showed that hallucination is not re- lated to how much the model attends to the image. In the (a) Real Object Tokens (b) Hallucinated Object Tokens Figure 4. Via the Logit-Lens, semantics decoded from the high- attention regions in the Image-Attention Stage align with the (a) real object tokens, but diverge from (b) the hallucinated object tokens. following, we show its relation to what the model attends to. Specifically, we use Logit-Lens [31] to “read out” the semantics of the attended visual regions. During the Image- Attention Stage, we project the hidden states of the top-k attended image tokens into the vocabulary space, thereby decoding what the model semantically “recognizes”. The results are shown in Fig. 4. We find: • For Real Objects (Semantic Consistency): When the model generates a real object token (e.g., ‘suitcase’), the highly-attended visual regions (i.e., the actual suitcase re- gions) can also be decoded to consistent tokens like ‘suit’ or ‘lug’ by Logit-Lens. This indicates that the real token attention is faithfully grounded in the visual evidence. • For Hallucinated Objects (Semantic Inconsistency): When the model generates a hallucinated object (e.g., ‘cell phone’), a clear semantic inconsistency emerges. The highly-attended visual regions (e.g., a ‘blurry patch on the ground’) do not decode to the hallucinated target token by Logit-Lens. This reveals a critical disconnect: the semantics “read” from the attended region are incon- sistent with the final output. This finding uncovers a key phenomenon of object hal- lucination: hallucinated object tokens exhibit a semantic inconsistency, where the visual evidence extracted by the attention mechanism does not support the generated to- ken. This insight directly motivates our subsequent detec- tion method, and empirical evidence for this phenomenon can be seen in the detection experiments in Sec. 5.1. 3.4. Two Types: Visual Uncertainty and Contextual Prior The previous section showed that hallucinations can ap- pear around the regions with strong attention during the Image-Attention Stage. In this section, we further inves- tigate why the model attends to these regions by devising a High-Attention Regions Masking experiment. Let the attention weight at layer l on the j-th image token during the t-th generation step (targeting object token o t ) be α (l) tj . LetS IA be the set of layers in the Image-Attention Stage. We first obtain the average attention that o t assigns to each image token j inS IA : ̄α tj = 1 |S IA | X l∈S IA α (l) tj .(4) We then select the top-k high-attention image tokens Ω t = TopK( ̄α t ,k), mask corresponding patches in the image and regenerate the response to observe the model’s behavior. As illustrated in Fig. 5, this experiment reveals two types of object hallucinations with distinct underlying causes: • Visual Uncertainty Hallucination: In these cases, the hallucinated token is directly related with high-attention regions. When we mask these regions, the hallucinated token disappears, indicating that the hallucination is visu- ally rooted. Empirically, such regions are often ambigu- ous, blurry, or uncertain areas (e.g., a round, ceramic-like area leading to the generation of “bowl”; or a dark, blurry rectangular region resulting in the output “cell phone”). These observations suggest that the model may attempt to forcibly extract certain semantics from highly uncertain visual evidence, which is insufficient to support the gener- ated object, leading to semantic inconsistency. Once this uncertain evidence source is removed, the model loses its visual anchor for generating hallucination. • Contextual Prior Hallucination: In contrast, we also observe hallucination where masking the high-attention regions does not remove the hallucinated token. Instead, hallucination persists and attention shifts to other regions. Logit Lens analysis further shows that these attended re- gions can be decoded into the correct objects actually present in the image, yet the model still outputs the hal- lucinated token. This pattern is consistent with the gener- ation being dominated by strong contextual priors [28] (e.g., outputting “microwave” in a kitchen description even when no microwave is present). In such cases, the attention mechanism is more like a procedural “routine”: its internal rules require it to “attend somewhere” before generating an object. So it arbitrarily “grounds” its atten- tion on some regions to satisfy this procedural require- ment, while the actual generation decision is mainly con- trolled by the contextual prior. We further perform a statistical analysis on the halluci- nation instances over 500 images. For LLaVA-1.5-7b, the ratio of “Visual Uncertainty” to “Contextual Prior” hallu- cination is approximately 2:1. The findings in this section expose two different types of object hallucination, demon- strating that a single, one-size-fits-all mitigation strategy is likely insufficient, thereby laying a solid foundation for our subsequent design of a targeted mitigating method. Question: Describe this image. Response: The image features a table with chairs placed outside, surrounded by a beautiful garden setting. There are two chairs positioned on either side of the table, and a third chair is located closer to the table. The table is adorned with a vase, a bowl, and a cup, adding to the inviting atmosphere. Original image Attention map for HOT Masked image Question: Describe this image. Response: The image features a wooden table with a vase of flowers on top, situated in a garden setting. The table is surrounded by three chairs, two on the left side and the other on the right side. The chairs are positioned in such a way that they face the table, creating a comfortable seating arrangement. Question: Describe this image. Response: The image depicts a large, well-equipped kitchen with a stainless steel refrigerator freezer positioned on the left side of the room. The kitchen features a microwave placed in the middle of the space. A table is situated in the right corner of the room, accompanied by a chair. Question: Describe this image. Response: The image features a kitchen with a large stainless steel refrigerator freezer on the left side of the room. The refrigerator is next to a picture on the wall. The kitchen is equipped with a microwave and a table with a chair placed in front of it. Attention map for HOT （ a ) V i s u a l U n c e r t a i n t y （ b ) C o n t e x t u a l P r i o r Figure 5. Two types of hallucination. (a) Disappears after masking high-attention regions, driven by visual uncertainty. (b) Persists after masking with attention shifting to other regions, driven by contextual prior. HOT here means Hallucinated Object Token. 4. Detect-Mitigate Framework For Object Hal- lucination Based on the analysis in Sec. 3, we propose a simple yet ef- fective Detect–Mitigate framework, as shown in Fig. 6. We first introduce a Logit-Lens Consistency Check (LLCC) to detect potential hallucinated tokens in real time (motivated by Sec. 3.3). We then determine which type (from Sec. 3.4) each detected hallucination belongs to and apply a targeted mitigation strategy. Details are as follows. 4.1. Hallucination Detection via Logit-Lens Consis- tency Check Our detection module aims to identify object hallucination in real-time within the generated sequence. This process is based on the finding from Sec. 3.3: real objects are semanti- cally consistent with their visual evidence, whereas halluci- nated objects are not. The detection process is divided into three stages: (1) Object Token Identification, (2) Semantic Consistency Check and (3) Hallucination Classification. 4.1.1. Object Token Identification To precisely locate object tokens while avoiding unnec- essary computation on non-object tokens, we design an identification step. First, inspired by the observation from Sec. 3.2 that object tokens exhibit significantly higher im- age attention than non-object tokens, we calculate the aver- age image attention A img (o t ) for each generated token o t within the Image-Attention StageS IA : A img (o t ) = 1 |S IA | X l∈S IA   N img X j=1 α (l) tj   ,(5) The image showcases a modern kitchen with a black and white color scheme. The kitchen features a center island with a sink, a refrigerator, and a microwave. There are two dining tables in the room, one located near the center island and the other closer to the refrigerator. Real microwave refrigerator sink Object Token Identification via S IA Hallucination Logit Lens Logit-Lens Consistency Check The image depicts a modern kitchen with a black and white color scheme. The kitchen features a sink and a microwave. There are two chairs positioned in the kitchen, one on the left side and another on the right side. Type 1: refrigerator Type 2: sink High-Attention Regions Masking ...... The kitchen features a countertop and a microwave. There are two chairs in the kitchen, both located near the countertop. Visual Evidence Enhanced Decoding Final Output Figure 6. The overall structure of our Detect-Mitigate framework. It consists of three components. First, Logit-Lens Consistency Check (LLCC) detects hallucinated object tokens in the response. Then, High-Attention Regions Masking (HARM) categorizes them into two different types of hallucination while generating an intermediate output. If type 2 hallucination is detected, the output is further processed by Visual Evidence Enhanced Decoding (VEED) to produce the final result. where α (l) tj is the attention weight of o t on the j-th image token at layer l, and N img is the total number of image to- kens. We treat o t with A img (o t ) > τ attn as object tokens, where τ attn is a threshold hyper-parameter. 4.1.2. Semantic Consistency Check For each object token, we first collect its attention to all im- age tokens withinS IA and select the Top-k attended tokens: Ω t = TopK α (l) tj , k ,l∈S IA .(6) We apply Logit Lens to each hidden state h(ω i t ),ω i t ∈ Ω t and take the highest-probability decoded token: v i t = arg max v∈V softmax W U h(ω i t ) v ,i = 1,...,k. (7) whereV denotes the vocabulary. Then we compare whether o t is semantically consistent with each token v i t . Since v i t might be a synonym of o t (e.g., o t =car, v i t =vehicle), we adopt a standard similar to [33, 38], using WordNet [30] or another semantic similarity function Sim(·,·): Label(o t ) = ( 0 (Real),if Sim(o t ,v i t ) > τ sim , 1 (Hallucination), otherwise. (8) where τ sim is the semantic similarity threshold. 4.1.3. Hallucination Classification When a token o t is detected as “hallucination”, we further classify its type. Following the same procedure as Sec. 3.4, we mask the high-attention regions withinS IA and observe whether the hallucination “disappears”. Let the newly gen- erated output sequence be O new , then: Type(o t ) = ( 1 (Visual Uncertainty) if o t /∈ O new , 2 (Contextual Prior)if o t ∈ O new . (9) 4.2. Targeted Mitigation Methods for Hallucination A single strategy cannot effectively handle hallucinations arising from different mechanisms. We therefore propose two simple yet effective methods tailored to each class. 4.2.1. Visual Uncertainty: High-Attention Region Mask- ing As analyzed in Sec. 3.4, this type of hallucination occurs when the model anchors its attention on a high-uncertainty visual region, and the generation heavily depends on this er- roneous evidence. Therefore, we employ a simple masking strategy, identical to that in Sec. 3.4. Let all the Top-k high-attention image tokens of each hal- lucinated object token be Ω and the original image be I . Its binary mask M satisfies: M = ( 1, if pixel/patch p∈ Ω 0, otherwise .(10) Let μ be a simple replacement value, e.g., the mean image color or a zero vector (black). The masked image I mask is: I mask = (1−M)⊙ I +M⊙ μ.(11) We then regenerate with the same prompt using I mask . By removing the high-uncertainty evidence, this strategy effec- tively reduces hallucination of this type. 4.2.2. Contextual Prior: Visual Evidence Enhanced De- coding The second type is insensitive to masking. Its root cause is that strong contextual priors dominate the output, while the real visual evidence is ignored. To address this, we propose Visual Evidence Enhanced Decoding (VEED). During the decoding stage of the language model, we directly inject vi- sual semantics from high-attention regions to increase their weight in the final logits, thereby exploiting correct visual evidence and suppressing the influence of the priors. Let h(ω max t ) denote the hidden state of the most attended visual region at step t. We extract its visual logits using Logit Lens (computed in Sec. 4.1): z vis t = W U h(ω max t ). Let z mask t = logits θ (y t | y <t ,I mask ) be the model’s decod- ing logits at step t when the high-attention region is masked. We fuse them to form the enhanced distribution for type 2: p t = softmax (1− α)· z vis t + α· z mask t (12) = softmax (1− α)· W U h(ω max t ) + α· logits θ (y t | y <t ,I mask ) Here, α controls the strength of visual-evidence injec- tion: a lower value gives higher weight to visual semantics from the high-attention region in the final decision. 5. Experiments In this section, we empirically validate the effectiveness of our proposed Detect–Mitigate framework. We first assess the hallucination detection method in Sec. 4.1, and then the mitigation method in Sec. 4.2. 5.1. Hallucination Detection 5.1.1. Experimental Settings Following [12, 15], we randomly sample 500 images from COCO2014 dataset, generate descriptions with LLaVA-1.5- 7B, and use CHAIR [33] to label hallucinated and real ob- jects. We compare our approach against three recent meth- ods: Uncertainty Score [47], which identifies hallucinations via uncertainty estimates; InterConf [14], which aggre- gates decoded probabilities from all layers; and SVAR [15], which exploits attention weights from intermediate layers 5–18. We set τ attn to 0.15 and k to 3. Following [38], we set τ sim to 0.8 to match semantically similar words. Addi- tional parameter experiments are in the appendix. 5.1.2. Results and Analysis As shown in Tab. 1, LLCC achieves the best Precision, Re- call, and F1. By explicitly localizing object-token atten- tion and performing a semantic consistency check on the attended regions, it directly tests whether the generation is supported by visual evidence. In contrast, baselines rely on decoding probabilities or cross-layer aggregates, weakening their causal link to the image. Uncertainty Score measures uncertainty based on probability, which lacks a theoretical link between probability and uncertainty and is undermined by LLM’s overconfidence [35]. InterConf merely assesses whether the object exists in the image, lacking explicit lo- calization, so language priors or misleading visual cues can make it wrong, and its reliance on all layers further degrades accuracy. SVAR focuses on attention “quantity” (Summed Table 1. Hallucination Detection Results. Best results are in bold, and second-best are underlined . MethodPrecisionRecallF1 Score Uncertainty Score0.59650.64150.6182 InterConf0.67170.69070.6811 SVAR 0.65000.72220.6842 LLCC(Ours)0.78700.79550.7932 Ratio) over “quality”. Our analysis (Sec. 3.2 / Sec. 3.3) shows that attention aligns with image evidence primarily in the later Image-Attention Stage, while it aggregates signals from non-image-dominant layers, which introduces noise and dilutes the signal. Our findings (Sec. 3.4) also sug- gest quantity maybe not a very reliable metric. While for our method, it uniquely assesses the “reasonableness of the generation” by first locking onto the visual source for ob- ject tokens and then checking if that source provides consis- tent semantic support. Notably, it can also identify different causal types of hallucinations, which prior methods cannot. 5.2. Hallucination Mitigation 5.2.1. Experimental Settings Benchmark and metrics. We evaluate on two standard benchmarks: CHAIR [33] and AMBER [38]. CHAIR measures whether objects mentioned in the generated text appear in the image and reports sentence-level and instance- level metrics (CHAIR S and CHAIR I , lower is better): CHAIR I = |hallucinated objects| |all objects mentioned| , CHAIR S = |sentences with hallucinated object| |all sentences| Following common settings [12, 15, 28], we randomly sam- ple 500 images from the COCO2014 validation set for CHAIR evaluation. AMBER is a multi-dimensional bench- mark comprising 1004 images with detailed object annota- tions. We report four metrics under its generative setting: CHAIR (rate of hallucinated objects), Cover (object cover- age), Hal (fraction of responses containing any hallucina- tion), and Cog (tendency to produce cognitively tempting target objects), where higher is better for Cover and lower is better for the others. Baselines. We compare against three common decoding strategies and five representative hallucination mitigation methods. Greedy Search selects the most probable next to- ken at each step. Beam Search maintains multiple hypothe- ses and outputs the highest-probability sequence. Nucleus Sampling samples from tokens whose cumulative probabil- ity exceeds a threshold p. VCD [18] contrasts outputs gen- erated from the original and a perturbed image to suppress language priors. OPERA [12] introduces dynamic penalties on over-confident decoding steps. DeCo [37] selects an in- termediate “anchor” layer to correct final-layer logits. Dev- ils [15] exploits mid-layer visual signals by aligning multi- Table 2. CHAIR hallucination evaluation results on multiple LVLMs. Best results are in bold, and second-best are underlined. DeCo and Devils do not report analyses on Qwen2-VL so their experimental settings are unspecified. We mark them as “–”. Method LLaVA-1.5-7BLLaVA-1.5-13BShikra-7BQwen2-VL-7B CHAIR S ↓ CHAIR I ↓CHAIR S ↓ CHAIR I ↓CHAIR S ↓ CHAIR I ↓CHAIR S ↓ CHAIR I ↓ Greedy49.820.447.819.858.422.231.412.7 Beam 50.020.148.218.959.723.130.811.9 Nucleus59.025.054.221.758.022.033.314.0 VCD 56.322.950.318.952.419.832.113.2 OPERA42.918.742.116.438.116.728.010.3 DeCo37.316.538.315.740.216.9-- Devils32.113.735.414.132.813.1-- PAI29.813.233.213.537.915.024.78.6 Ours26.810.031.312.431.412.724.08.3 Table 3. AMBER hallucination evaluation results. Best results are in bold, and second-best are underlined. Method LLaVA-1.5-7BLLaVA-1.5-13BShikra-7B CHAIR↓ Cover↑ Hal↓ Cog↓CHAIR↓ Cover↑ Hal↓ Cog↓CHAIR↓ Cover↑ Hal↓ Cog↓ Greedy6.951.032.03.36.852.031.73.510.652.047.05.2 Beam7.849.735.54.18.748.738.83.810.051.344.14.8 Nucleus7.951.135.54.38.451.538.13.911.152.249.45.7 VCD6.452.133.22.97.653.235.33.410.353.745.25.0 OPERA6.847.328.32.76.949.031.53.38.251.639.83.9 DeCo6.646.427.62.86.248.527.72.97.548.237.94.0 Devils3.550.219.91.35.651.130.63.46.351.432.12.8 PAI4.250.718.41.64.951.926.23.05.950.932.73.4 Ours2.851.214.71.24.052.024.12.55.352.330.32.0 head attentions. And PAI [28] amplifies the attention of image tokens and contrasts its output with no-image logits. Implementation details. We implement our method with greedy decoding as a representative case and test on three VLMs: LLaVA-1.5 [26], Shikra [6] and Qwen2-VL [39]. For Beam search, we set n beam = 3. For Nucleus sampling, we set top-p = 0.9 and temperature τ = 0.7. Following the original papers, we pair VCD with nucleus sampling, ap- ply OPERA and DeCo on top of beam search, and combine Devils and PAI with greedy decoding. All experiments are conducted on a single NVIDIA A100 40GB GPU. 5.2.2. Results and Analysis As shown in Tab. 2 and Tab. 3, our method consistently reduces hallucination while maintaining (even slightly im- proving) object coverage.On CHAIR, we achieve the largest drop among the four models. On AMBER, we obtain the lowest hallucination rates across models while keeping strong coverage, which is slightly higher than the original 51.0 and second only to VCD’s 52.1 on LLaVA- 1.5-7B. Notably, most baselines decrease on Cover rate, in- dicating that they sacrifice valid content to mitigate hallu- cination, whereas our approach achieves the lowest hallu- cination rates without eroding the original expression. We attribute the advantage to our two-stage framework: (i) ex- plicit object-token localization and cause-aware hallucina- tion detection, and (i) targeted mitigation: masking only a minimal set of image tokens and applying decoding en- hancement only for the second hallucination type, thereby limiting impact on original content.Although absolute scores vary with model scale and architecture, the rela- tive ranking remains stable, demonstrating robustness of our method. Additional ablations are provided in the appendix. 6. Conclusion In this work, we revisit object hallucination in LVLMs via the lens of attention and Logit Lens analysis.By analyzing object token generation process, we reveal a “same attention, different truths” phenomenon: both real and hallucinated objects exhibit similar attention patterns in the image-attention stage, yet only the former are semantically decodable from the attended regions via Logits-Lens.Building on this, we pro- pose a training-free Detect–Mitigate framework that first detects hallucinated object tokens via Logit-Lens Consistency Check and then applies cause-aware High Attention Regions Masking for visual uncertainty hal- lucination and Visual Evidence Enhanced Decoding for contextual prior hallucination.Extensive experiments demonstrate the effectiveness and generality of our method. References [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022. 2 [2] Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Ping Chen, Xiaoqin Zhang, and Shi- jian Lu. Mitigating object hallucinations in large vision- language models with assembly of global and local attention. In Proceedings of the Computer Vision and Pattern Recogni- tion Conference, pages 29915–29926, 2025. 1, 3 [3] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 1, 2 [4] Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930, 2024. 1 [5] Junzhe Chen, Tianshu Zhang, Shiyu Huang, Yuwei Niu, Lin- feng Zhang, Lijie Wen, and Xuming Hu. Ict: Image-object cross-level trusted intervention for mitigating object halluci- nation in large vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 4209–4221, 2025. 1 [6] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao.Shikra: Unleashing multi- modal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023. 2, 8 [7] Zhiyang Chen, Yousong Zhu, Yufei Zhan, Zhaowen Li, Chaoyang Zhao, Jinqiao Wang, and Ming Tang. Mitigating hallucination in visual language models with visual supervi- sion. arXiv preprint arXiv:2311.16479, 2023. 1, 2 [8] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185–24198, 2024. 1, 2 [9] Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Halc: Object hallucination re- duction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425, 2024. 3 [10] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. Advances in neural information processing systems, 36:49250–49267, 2023. 1 [11] Xuan Gong, Tianshi Ming, Xinpeng Wang, and Zhihua Wei. Damro: Dive into the attention mechanism of lvlm to re- duce object hallucination. arXiv preprint arXiv:2410.04514, 2024. 1 [12] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu.Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13418–13427, 2024. 7 [13] Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learn- ing for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27036–27046, 2024. 2 [14] Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. Interpreting and editing vision-language rep- resentations to mitigate hallucinations.arXiv preprint arXiv:2410.02762, 2024. 7 [15] Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large vision- language models: Interpreting, detecting and mitigating ob- ject hallucinations via attention lens. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 25004–25014, 2025. 1, 3, 7 [16] Omri Kaduri, Shai Bagon, and Tali Dekel. What’s in the im- age? a deep-dive into the vision of vision language models. In Proceedings of the Computer Vision and Pattern Recogni- tion Conference, pages 14549–14558, 2025. 1 [17] Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models. arXiv preprint arXiv:2503.03321, 2025. 1, 3 [18] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 13872–13882, 2024. 1, 3, 7 [19] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning, pages 19730– 19742. PMLR, 2023. 1, 2 [20] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195– 22206, 2024. 1 [21] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Ja- son Eisner, Tatsunori B Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text genera- tion as optimization. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12286–12312, 2023. 3 [22] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.Evaluating object hallucina- tion in large vision-language models.arXiv preprint arXiv:2305.10355, 2023. 1 [23] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ́ ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. 3 [24] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Ya- coob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning.arXiv preprint arXiv:2306.14565, 2023. 2 [25] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. 1, 2 [26] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306, 2024. 2, 3, 8 [27] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiu- tian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253, 2024. 1 [28] Shi Liu, Kecheng Zheng, and Wei Chen. Paying more at- tention to image: A training-free method for alleviating hal- lucination in lvlms. In European Conference on Computer Vision, pages 125–140. Springer, 2024. 1, 3, 5, 7, 8 [29] Xinyu Lyu, Beitao Chen, Lianli Gao, Hengtao Shen, and Jingkuan Song. Alleviating hallucinations in large vision- language models through hallucination-induced optimiza- tion. Advances in Neural Information Processing Systems, 37:122811–122832, 2024. 2 [30] George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41, 1995. 6 [31] nostalgebraist.Interpreting gpt:The logit lens. https : / / w . lesswrong . com / posts / AcKRB8wDpdaN6v6ru / interpreting - gpt - the-logit-lens, 2020. 2, 4 [32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748–8763. PmLR, 2021. 2 [33] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image cap- tioning. arXiv preprint arXiv:1809.02156, 2018. 4, 6, 7 [34] Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan ̈ O Arık, and Tomas Pfister. Mitigating ob- ject hallucination in mllms via data-augmented phrase-level alignment. arXiv preprint arXiv:2405.18654, 2024. 1 [35] Zailong Tian, Zhuoheng Han, Yanzhe Chen, Haozhe Xu, Xi Yang, Richeng Xuan, Houfeng Wang, and Lizi Liao. Over- confidence in llm-as-a-judge: Diagnosis and confidence- driven solution. arXiv preprint arXiv:2508.06225, 2025. 7 [36] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024. 1, 3 [37] Chenxi Wang, Xiang Chen, Ningyu Zhang, Bozhong Tian, Haoming Xu, Shumin Deng, and Huajun Chen. Mllm can see? dynamic correction decoding for hallucination mitiga- tion. arXiv preprint arXiv:2410.11779, 2024. 3, 7 [38] Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, et al. Amber: An llm-free multi-dimensional bench- mark for mllms hallucination evaluation.arXiv preprint arXiv:2311.07397, 2023. 6, 7 [39] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 8 [40] Huyu Wu, Meng Tang, Xinhan Zheng, and Haiyun Jiang.When language overrules: Revealing text domi- nance in multimodal large language models. arXiv preprint arXiv:2508.10552, 2025. 1, 3 [41] Zhihe Yang, Xufang Luo, Dongqi Han, Yunjian Xu, and Dongsheng Li. Mitigating hallucinations in large vision- language models via dpo: On-policy data hold the key. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 10610–10620, 2025. 2 [42] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug- owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 13040–13051, 2024. 2 [43] Hao Yin, Guangzong Si, and Zilei Wang. Clearsight: Vi- sual signal enhancement for object hallucination mitigation in multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14625–14634, 2025. 1, 3 [44] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional hu- man feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13807– 13816, 2024. 2 [45] Linxi Zhao, Yihe Deng, Weitong Zhang, and Quanquan Gu. Mitigating object hallucination in large vision-language models via classifier-free guidance. arXiv e-prints, pages arXiv–2402, 2024. 1, 3 [46] Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Ji- aqi Wang, and Conghui He. Beyond hallucinations: Enhanc- ing lvlms through hallucination-aware direct preference op- timization. arXiv preprint arXiv:2311.16839, 2023. 2 [47] Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754, 2023. 7

Instant research discovery