Paper deep dive

Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens

Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, Xu Yang

Year: 2024Venue: arXiv preprintArea: Multimodal SafetyType: EmpiricalEmbeddings: 82

Models: LLaVA-1.5-13B, LLaVA-1.5-7B, MiniGPT-4-7B, Shikra-7B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/12/2026, 7:23:06 PM

Summary

The paper investigates object hallucinations in Large Vision-Language Models (LVLMs) by analyzing visual information processing in middle layers. It identifies two distinct stages—'visual information enrichment' and 'semantic refinement'—and proposes the Visual Attention Ratio (VAR) metric to detect hallucinations. The authors demonstrate that hallucinated tokens result from inconsistent attention across heads and propose an inference-time method to mitigate hallucinations by adjusting visual attention without additional training.

Entities (5)

LVLM · model-architecture · 100%Object Hallucination · phenomenon · 99%LLaVA-1.5 · model · 98%Logit Lens · interpretability-technique · 95%Visual Attention Ratio · metric · 95%

Relation Signals (3)

LVLM → exhibits → Object Hallucination

confidence 95% · LVLMs often exhibit undesirable object hallucinations

Middle Layers → processes → Visual Information

confidence 95% · the middle layers are crucial... propagate visual data to object tokens

Visual Attention Ratio → detects → Object Hallucination

confidence 92% · utilize the metric in this stage to detect hallucinated object tokens

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Hallucinations in Large Vision-Language Models (LVLMs) significantly undermine their reliability, motivating researchers to explore the causes of hallucination. However, most studies primarily focus on the language aspect rather than the visual. In this paper, we address how LVLMs process visual information and whether this process causes hallucination. Firstly, we use the attention lens to identify the stages at which LVLMs handle visual data, discovering that the middle layers are crucial. Moreover, we find that these layers can be further divided into two stages: ''visual information enrichment'' and ''semantic refinement'' which respectively propagate visual data to object tokens and interpret it through text. By analyzing attention patterns during the visual information enrichment stage, we find that real tokens consistently receive higher attention weights than hallucinated ones, serving as a strong indicator of hallucination. Further examination of multi-head attention maps reveals that hallucination tokens often result from heads interacting with inconsistent objects. Based on these insights, we propose a simple inference-time method that adjusts visual attention by integrating information across various heads. Extensive experiments demonstrate that this approach effectively mitigates hallucinations in mainstream LVLMs without additional training costs. Code is available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

81,551 characters extracted from source content.

Expand or collapse full text

Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens Zhangqi Jiang1 Junkai Chen2,3 Beier Zhu4 Tingjin Luo1 Yankun Shen2,3 Xu Yang2,3†footnotemark: 1National University of Defense Technology 2Southeast University 3Key Laboratory of New Generation Artificial Intelligence Technology & Its Interdisciplinary Applications (Southeast University), Ministry of Education 4Nanyang Technological University sxdxjzq, junkai.chen.0917@gmail.com beier.zhu@ntu.edu.sg tingjinluo@hotmail.com 220242353, xuyang_palm@seu.edu.cn Corresponding author. Abstract Hallucinations in Large Vision-Language Models (LVLMs) significantly undermine their reliability, motivating researchers to explore the causes of hallucination. However, most studies primarily focus on the language aspect rather than the visual. In this paper, we address how LVLMs process visual information and whether this process causes hallucination. Firstly, we use the attention lens to identify the stages at which LVLMs handle visual data, discovering that the middle layers are crucial. Moreover, we find that these layers can be further divided into two stages: “visual information enrichment” and “semantic refinement” which respectively propagate visual data to object tokens and interpret it through text. By analyzing attention patterns during the visual information enrichment stage, we find that real tokens consistently receive higher attention weights than hallucinated ones, serving as a strong indicator of hallucination. Further examination of multi-head attention maps reveals that hallucination tokens often result from heads interacting with inconsistent objects. Based on these insights, we propose a simple inference-time method that adjusts visual attention by integrating information across various heads. Extensive experiments demonstrate that this approach effectively mitigates hallucinations in mainstream LVLMs without additional training costs.111Code: https://github.com/ZhangqiJiang07/middle_layers_indicating_hallucinations. 1 Introduction Figure 1: Illustration of our findings: (i) visual information is primarily processed in the middle layers where (a) the model extracts the visual information and then (b) interprets the semantics embedded in image tokens; (i) for real tokens like “cat”, the attention weights over image tokens are generally higher than hallucinated ones like “TV” in (a); (i) the model may combine the visual features extracted from multiple objects to produce hallucinations. Building on the foundations of Large Language Models (LLMs), Large Vision-Language Models (LVLMs) [27, 49, 4, 54, 44, 32, 51] have emerged as powerful tools for understanding visual inputs and generating textual responses aligned with user intent. Their multi-modal nature has enabled a large variety of vision-language tasks, e.g., image captioning, visual question answering, and visual dialog. Despite generating coherent textual responses, these models often exhibit undesirable object hallucinations – producing objects that do not exist in the visual input. Although various approaches, such as visual instruction fine-tuning [26, 20, 14, 46], the integration of external expert models [45, 5, 50, 43], and contrastive decoding strategies [24, 9, 42, 28], have been proposed to address the object hallucination issue in LVLMs, the underlying mechanisms driving these hallucinations remain poorly understood. A few studies have preliminarily explored the causes of object hallucination, identifying the language bias as the primary factor, with examples including the “anchor pattern” [19] and “text inertia” [28]. In essence, these studies found that LVLMs tend to prioritize internal textual knowledge over external visual information as more tokens are generated. However, these studies have overlooked the devils in the visual parts – while intuitively, improper processing of visual information within the LVLMs could contribute more to the emergence of object hallucinations. In this paper, we delve into how LVLMs process visual information from image tokens and how this affects the generation of object hallucinations. Building on the three-stage mechanism identified in LLMs for retrieving factual knowledge [12], we examine whether LVLMs exhibit similar distinct stages during inference and, if so, which stage is critical for processing visual information. We propose a simple score called Visual Attention Ratio (VAR) to examine the distribution of visual attention across layers, revealing that the middle layers play a crucial role in processing visual information. To study how the model processes visual information in these middle layers, we leverage the logit lens [29] method to decode the hidden states of image tokens into LVLMs’ vocabulary, identifying two crucial stages as illustrated in Fig. 1: a visual information enrichment stage propagates visual information to the object token, and a semantic refinement stage interprets the encoded visual information through text. Secondly, focusing on these middle layers, we observe that the hallucinated tokens often exhibit (1) less active attention patterns, and (2) inconsistent attentions across different heads during visual information enrichment stage, compared to real ones. To quantify these observations, we first introduce a metric based on VAR, and confirm that real object tokens tend to assign higher attention weights over image tokens during visual information enrichment than hallucinated ones, illustrated in Fig. 1. Relying on our finding, we utilize the metric in this stage to detect hallucinated object tokens, achieving AUROC and mAP up to 74% and 88% on LLaVA-1.5-7B [27]. Second, given the various attention heads used in the Multi-Heads Self-Attention (MHSA) mechanism [40] for information aggregation, we explore the head behavior by visualizing the attention heatmaps over the input image. As illustrated in Fig. 1, we find that the heads interact with inconsistent objects during visual information enrichment when generating hallucinations. Our empirical findings provide valuable insights into the way object hallucinations are generated, intuitively described as follows (Fig. 1): In visual information enrichment, the model’s ambiguous interactions with multiple objects result in limited and mixed visual information being propagated to the object token. Subsequently, in semantic refinement, this imprecise visual information leads the model to mistakenly associate semantic information from different objects, ultimately causing hallucinations. Inspired by our findings, we propose a simple method to adjust the visual information process in the middle layers by integrating attention information from various heads during inference. Extensive experiments on three mainstream LVLMs show significant hallucination mitigation compared to the original models, with average reductions in CHAIRI and CHAIRS up to 6.3 and 24.1 points, respectively, while preserving detail in descriptions. 2 Related Work Large Vision-Language Models. The recent integration of advanced open-source LLMs, like Llama [38, 39] and Vicuna [6], has greatly expanded the capabilities of LVLMs, enabling them to perform more complex vision-language tasks. Typically, the architecture of recent LVLMs incorporates three key components: a vision branch, such as CLIP [33] and EVA [8], to encode image inputs, a modality connector to transform image features into image embeddings aligning with text modality, and a pretrained language model to process image and prompt embeddings to generate responses. LLaVA-1.5 [27] employs an MLP to map the output of the vision branch into 576 image embeddings. Similarly, Shikra [4] utilizes one linear layer to align image features. While MiniGPT-4 [54] adopts a learnable querying transformer to establish vision-language connections, with just 32 image tokens as LVLM input. Despite their impressive capabilities, the above LVLMs suffer from severe object hallucinations. In this paper, we focus on interpreting, detecting, and mitigating their hallucinations. Object Hallucination in LVLMs. Object hallucination [52, 48] is a prevalent and critical issue in current LVLMs where the model erroneously generates descriptions of non-existent objects in images, posing significant risks in high-stakes fields such as medical imaging [18, 23], sequential recommendation systems [15, 35], and autonomous driving [7]. Previous studies to mitigate this issue have primarily focused on visual instruction fine-tuning [26, 20, 14, 46, 53], incorporating external expert models [45, 5, 50, 43], and improving decoding strategies [19, 24, 28, 42, 9]. Despite numerous efforts, our understanding of the underlying mechanism driving these errors remains limited. Recent studies like OPERA [19] have pointed specific “anchor patterns” in text tokens that always align with the onset of hallucinated contents, and PAI [28] has identified “text inertia” as another contributing factor, as LVLMs reproduce identical hallucinated descriptions when solely input part of historical generated text. Contrary to these perspectives that focus on language bias, we investigate the causes of object hallucinations from the view of visual information processing through the lens of attention. Interpretability in Foundation Models. In the field of NLP, numerous studies have analyzed LLMs to explore the internal model knowledge and understand model behavior in specific settings from the perspectives of attention maps [3, 47], neural activation patterns [17], and intermediate hidden states [36, 13]. The logit lens [29], which transforms the hidden states at each layer into the vocabulary space using the model’s own linear projector before softmax, offers insightful observations into the next token prediction process [11, 16] in LLMs. Our work adapts the logit lens method to analyze hidden states of image tokens, enabling us to interpret the LVLMs’ process of understanding the visual information via textual vocabulary. Few studies explore the internal mechanisms of LVLMs [1, 2, 10], notable efforts include [30], which extends an unimodal causal tracing tool for studying neural mechanisms in BLIP’s image-conditioned text generation, and [31], which introduces a dictionary learning-based framework to extract multi-modal concepts for interpreting intermediate representations. Compared to prior works, our study leverages the internal signals from attention weights to uncover the generation of hallucinated objects. 3 Understanding Object Hallucinations In this section, we start by introducing the generation process of LVLMs and the analytical tools, including visual attention ratio and logit lens. We then conduct case studies focusing on three key aspects: the processing of visual information, the attention patterns of object tokens, and the behavior of heads during object token generation. 3.1 Preliminary Notation. The input to LVLMs is initialized with a sequence of d-dimensional image tokens 1,…,nsubscript1…subscript\v_1,...,v_n\ v1 , … , vitalic_n and instruction text (prompt) tokens 1,…,msubscript1…subscript\t_1,...,t_m\ t1 , … , titalic_m . Typically structured as a Transformer decoder [40], the LVLM outputs responses in an autoregressive manner: at time step k, the model processes the initial input tokens 1,…,n,1,…,msubscript1…subscriptsubscript1…subscript\v_1,...,v_n,t_1,...,t_m\ v1 , … , vitalic_n , t1 , … , titalic_m , followed by the sequence of k−11k-1k - 1 previously generated tokens 1,…,k−1subscript1…subscript1\y_1,...,y_k-1\ y1 , … , yitalic_k - 1 , to predict the next token ksubscripty_kyitalic_k. Let L denote the number of Transformer layers, each with H heads. At layer ℓ∈[L]ℓdelimited-[] ∈[L]ℓ ∈ [ L ], we denote the attention weights in head h∈[H]ℎdelimited-[]h∈[H]h ∈ [ H ] as k(ℓ,h)∈ℝak×aksuperscriptsubscriptℓℎsuperscriptℝsubscriptsubscriptA_k^( ,h)∈R^a_k× a_kAitalic_k( ℓ , h ) ∈ blackboard_Raitalic_k × aitalic_k, where ak=n+m+k−1subscript1a_k=n+m+k-1aitalic_k = n + m + k - 1. Additionally, we define iℓsubscriptsuperscriptℓv _ivroman_ℓitalic_i and k−1ℓsubscriptsuperscriptℓ1y _k-1yroman_ℓitalic_k - 1 as the hidden states for the image token isubscriptv_ivitalic_i and generated token k−1subscript1y_k-1yitalic_k - 1 at layer ℓ ℓ, respectively. The real and hallucinated object tokens are noted as subscripty_ rysansserif_r and subscripty_ hysansserif_h, respectively. Visual Attention Ratio. For the k-th token ksubscripty_kyitalic_k, we define the Visual Attention Ratio (VAR) in each head hℎh at layer ℓ ℓ: VAR(ℓ,h)⁢(k)≜∑i=1nk(ℓ,h)⁢(ak,i),≜superscriptVARℓℎsubscriptsuperscriptsubscript1superscriptsubscriptℓℎsubscript VAR^( ,h)(y_k) _i=1^nA_k^% ( ,h)(a_k,i),VAR( ℓ , h ) ( yitalic_k ) ≜ ∑i = 1n Aitalic_k( ℓ , h ) ( aitalic_k , i ) , (1) where k(ℓ,h)⁢(ak,i)superscriptsubscriptℓℎsubscriptA_k^( ,h)(a_k,i)Aitalic_k( ℓ , h ) ( aitalic_k , i ) represents the attention weights of the newly generated token ksubscripty_kyitalic_k assigned to the image token isubscriptv_ivitalic_i. VAR quantifies the extent of the token ksubscripty_kyitalic_k’s interaction with visual information: a higher VAR score indicating a greater contribution from image tokens during ksubscripty_kyitalic_k’s generation. Logit Lens. We utilize this technique to investigate how the model interprets the visual hidden state iℓsuperscriptsubscriptℓv_i vitalic_iroman_ℓ during inference via text. We denote the linear projector before softmax as ∈ℝ||×dsubscriptsuperscriptℝW_V∈R^|V|× dWcaligraphic_V ∈ blackboard_R| V | × d, which is applied to k−1Lsubscriptsuperscript1y^L_k-1yitalic_Litalic_k - 1 to predict the probability of next token over the vocabulary VV. Then, the logit lens transforms the hidden state iℓsuperscriptsubscriptℓv_i vitalic_iroman_ℓ of the image tokens to the prediction probability distribution over the vocabulary by the linear projector subscriptW_VWcaligraphic_V: ⁢(|iℓ)=softmax⁢(⋅iℓ)∈ℝ||,conditionalsuperscriptsubscriptℓsoftmax⋅subscriptsuperscriptsubscriptℓsuperscriptℝ (V|v_i )=softmax(% W_V·v_i ) ^|V% |,p ( V | vitalic_iroman_ℓ ) = softmax ( Wcaligraphic_V ⋅ vitalic_iroman_ℓ ) ∈ blackboard_R| V | , (2) where j⁢(|iℓ)subscriptconditionalsuperscriptsubscriptℓp_j(V|v_i )pitalic_j ( V | vitalic_iroman_ℓ ) corresponds to the j-th text token in the vocabulary. To explain the processed image token, the text token with the highest probability is considered as the model’s interpretation of the hidden state iℓsuperscriptsubscriptℓv_i vitalic_iroman_ℓ at layer ℓ ℓ. Figure 2: (a) Distribution of visual attention ratio for real object tokens across heads and layers in LLaVA-1.5-7B, sorted row-wise by attention ratios. Note that the high attention in the 00-th layer (bottom row, green rectangle) is not consistent across all LVLMs, as Shikra and MiniGPT-4 models fail to exhibit this pattern, see Sec. C.1. (b) The logit contribution of attention sublayers to real token prediction. We find the middle layers continuously assign higher attention weights to image tokens and exhibit two different contribution patterns to the correct prediction. Figure 3: Case study of image hidden state interpretation in LLaVA-1.5-7B via logit lens. The heatmap illustrates the retrieved texts of the image hidden states across layers in the three distinct regions, differentiated by color. Our findings reveal that the model’s semantic comprehension of image tokens emerges in the later middle layers (19-26) while remaining largely absent in earlier layers. The real and hallucinated object tokens are presented in blue and red in the description, respectively. 3.2 Experimental Setup for Case Study We conduct case studies with a randomly selected subset of 2,000 images from COCO 2014 validation set [25]. These images represent diverse scenes and activities across 80 common object categories, each paired with multiple object annotations. The 7B version of LLaVA-1.5 serves as the main focus for the subsequent analysis. In Sec. C.1, we further repeat our case studies on other LVLMs, including Shikra and MiniGPT-4, and observe consistent findings. To investigate the internal patterns in generating real and hallucinated object tokens, we use the model with greedy search to generate captions for the selected images, prompted by “Please help me describe the image in detail.”. Real and hallucinated object tokens are then identified using ground truth annotations as a reference. For multi-token objects, only the first token is considered. As a result, we obtained 1,842 hallucinated tokens and 4,397 real tokens. 3.3 Finding 1: Middle Layers Matter for Visual Information Interaction To assess the contribution of visual information to token generations, we compute the VAR score (Eq. 1) for real object tokens subscripty_ rysansserif_r across all layers ℓ∈[L]ℓdelimited-[] ∈[L]ℓ ∈ [ L ] and heads h∈[H]ℎdelimited-[]h∈[H]h ∈ [ H ]. We calculate the mean of VAR(ℓ,h)⁢()superscriptVARℓℎsubscript VAR^( ,h)(y_ r)VAR( ℓ , h ) ( ysansserif_r ) for all real object tokens in our case dataset, with the results depicted in Fig. 2 (a). The visualization shows that the middle layers, i.e., 5-26 layers, continuously exhibit high attention weights to the image tokens, indicating that visual information interaction primarily occurs in the middle layers. We further analyze how the model processes visual information by examining signals from the MHSA sublayers. Using the logit lens, we quantify the contribution of the MHSA sublayers to real token prediction as ⁢(|ℓ)subscriptconditionalsubscriptsuperscriptℓp_ o(V|a _ r)psansserif_o ( V | aroman_ℓsansserif_r ), where ℓsubscriptsuperscriptℓa _ raroman_ℓsansserif_r is the output of the MHSA sublayer at layer ℓ ℓ, and osansserif_o denotes the index of the real object token subscripty_ rysansserif_r in the vocabulary. Fig. 2 (b) reveals two patterns in the contributions from the MHSA sublayers: lower in 5-18 layers and higher in 19-26 layers, reflecting varied use of visual information. To better understand these two patterns, we use the logit lens to map the hidden states of image token iℓsuperscriptsubscriptℓv_i vitalic_iroman_ℓ to text. Specifically, for each image token isubscriptv_ivitalic_i at layer ℓ ℓ, we retrieve the text token in the vocabulary with the highest probability, i.e., argmax1≤j≤||⁢j⁢(|iℓ)subscriptargmax1subscriptconditionalsuperscriptsubscriptℓ argmax_1≤ j≤|V|\p_j(V| % v_i )\argmax1 ≤ j ≤ | V | pitalic_j ( V | vitalic_iroman_ℓ ) . A case is shown in Fig. 3, revealing two stages in the middle layers with different visual information usage: • Stage 1: Visual Information Enrichment. The first stage, in layers 5–18, shows that retrieved text tokens are less related to the corresponding image patches, suggesting the model is unable to interpret visual information. This misalignment might explain the low prediction contributions observed in Fig. 2 (b). Meanwhile, these layers continue to show high VAR scores, indicating that the object tokens are accumulating visual information through self-attention. In Secs. 3.4 and 3.5, we show that such attention serves as a key indicator to identify hallucinations. • Stage 2: Semantic Refinement. The second stage, in layers 19–26, shows that the retrieved text tokens are semantically consistent with the image patches, suggesting the model is able to interpret and utilize the visual information encoded in the image tokens. Given the high VAR during this stage, we term it as semantic refinement where the model actively interacts with the semantic information of image tokens to reason object token prediction. 3.4 Finding 2: Inactive Visual Attention in Middle Layers Implies Hallucination Figure 4: Case study of the visual attention ratio distribution over heads and layers in LLaVA-1.5-7B. Left: real objects; Right: hallucinated objects. We find that the hallucinated object tokens exhibit inactive attention patterns during visual information enrichment compared to the real ones. Given that visual information is primarily processed in the middle layers, a natural question arises: does the model display unusual attention patterns in these layers when generating hallucinated object tokens? Fig. 4 presents the VAR distribution for real object tokens (‘bus’ and ‘car’) and hallucinated ones (‘people’ and ‘fire hydrant’) from the case of Fig. 3. We observe that the hallucinated object tokens demonstrate an inactive attention pattern during the visual information enrichment stage (5-18 layers) compared to the real ones. We conjecture that such pattern weakens the interaction between the object token and image tokens in this stage, limiting the propagation of visual information and potentially leading to hallucinations. To quantify this observation, we introduce a metric, Summed Visual Attention Ratio (SVAR), calculated by averaging VAR scores over all heads and summing across selected layers [ℓs,ℓe]subscriptℓsubscriptℓ[ _s, _e][ ℓitalic_s , ℓitalic_e ]. Specifically, for a real object token subscripty_ rysansserif_r in [ℓ5,ℓ18]subscriptℓ5subscriptℓ18[ _5, _18][ ℓ5 , ℓ18 ], we compute: SVAR5⁢-⁢18⁢()≜1H⁢∑ℓ=518∑h=1HVAR(ℓ,h)⁢().≜subscriptSVAR5-18subscript1superscriptsubscriptℓ518superscriptsubscriptℎ1superscriptVARℓℎsubscript SVAR_5 -18(y_ r) 1H% _ =5^18 _h=1^H VAR^( ,h)(y_ r% ).SVAR5 - 18 ( ysansserif_r ) ≜ divide start_ARG 1 end_ARG start_ARG H end_ARG ∑ℓ = 518 ∑h = 1H VAR( ℓ , h ) ( ysansserif_r ) . (3) We conduct a statistical experiment on the SVAR5⁢-⁢18subscriptSVAR5-18 SVAR_5 -18SVAR5 - 18 metric for both real and hallucinated tokens. The results, depicted in Fig. 5 (a), and the associated statistical test detailed in Sec. C.2, indicate a salient difference: the model assigns significantly higher attention weights to image tokens when generating real object tokens during visual information enrichment compared to hallucinated ones. Fig. 5 (b) shows a similar trend in LLaVA-1.5-13B. Figure 5: SVAR5⁢-⁢18subscriptSVAR5-18 SVAR_5 -18SVAR5 - 18 score distributions across object token types for the 7B (a) and 13B (b) versions of LLaVA-1.5. 3.4.1 Application: Object Hallucination Detection Building upon our Finding 2, we apply the SVAR5⁢-⁢18subscriptSVAR5-18 SVAR_5 -18SVAR5 - 18 metric to detect object hallucinations. We assess the efficacy of SVAR5⁢-⁢18subscriptSVAR5-18 SVAR_5 -18SVAR5 - 18 in reflecting object hallucination using our case dataset, where a positive sample is a real object token and a negative sample is a hallucinated one. For comparison, we use the internal confidence metric recently proposed in [21] as a baseline. The internal confidence uses the maximum probability of the object token within all image hidden states iℓ:i∈[n],ℓ∈[L]conditional-setsuperscriptsubscriptℓformulae-sequencedelimited-[]ℓdelimited-[]\v_i :i∈[n], ∈[L]\ vitalic_iroman_ℓ : i ∈ [ n ] , ℓ ∈ [ L ] , mapped by the logit lens for detection. The qualitative results are presented in Fig. 6 and the results for LLaVA-1.5-13B are shown in Sec. C.1. Compared to internal confidence, using the simple SVAR5⁢-⁢18subscriptSVAR5-18 SVAR_5 -18SVAR5 - 18 metric improves the AUROC by 8.82% and the AP by 3.53%, verifying the utility of our metric. This further suggests that the middle layers provide crucial indicator information for object hallucination. Figure 6: Object hallucination detection curves for LLaVA-1.5-7B. We show the ROC and Precision-Recall curves of SVAR5⁢-⁢18subscriptSVAR5-18 SVAR_5 -18SVAR5 - 18 metric for object hallucination detection on the case study dataset. Figure 7: Visualization of attention maps over image for ‘bus’ (real) and ‘people’ (hallucinated) object tokens in heads (ℓ,h)ℓℎ( ,h)( ℓ , h ). We find that the heads tend to interact with multiple objects when generating hallucinated object tokens. More examples can be found in Sec. C.4. Beyond the hand-crafted SVAR5⁢-⁢18subscriptSVAR5-18 SVAR_5 -18SVAR5 - 18 measure, we explore the potential of middle layers to indicate hallucinations by training a simple model to derive metrics directly from raw visual attention states. Experimental setting. We train a two-layer MLP by inputting the concatenated VAR scores across all heads in the layer range of interest [ℓs,ℓe]subscriptℓsubscriptℓ[ _s, _e][ ℓitalic_s , ℓitalic_e ] to classify the real and hallucinated object tokens. Specifically, the input vector ⁢()subscriptx(y_ r)x ( ysansserif_r ) for the real token subscripty_ rysansserif_r can be written as: ⁢()=[VAR(ℓs,1),VAR(ℓs,2),…,VAR(ℓe,H)]⊤.subscriptsuperscriptsuperscriptVARsubscriptℓ1superscriptVARsubscriptℓ2…superscriptVARsubscriptℓtopx(y_ r)=[ VAR^( _s,1), VAR^% ( _s,2),…, VAR^( _e,H)] .x ( ysansserif_r ) = [ VAR( ℓitalic_s , 1 ) , VAR( ℓitalic_s , 2 ) , … , VAR( ℓitalic_e , H ) ]⊤ . (4) We split the case dataset into training and test sets in the ratio of 8:2, and train the MLP with cross-entropy loss. The model performance is evaluated using accuracy, AUROC, and recall metrics. Training details are in Sec. B.2. Results. We experiment with four layer ranges in LLaVA-1.5-7B: [0,4]04[0,4][ 0 , 4 ], [5,18]518[5,18][ 5 , 18 ], [19,26]1926[19,26][ 19 , 26 ], and [27,31]2731[27,31][ 27 , 31 ], as shown in Tab. 1. We can see using the layers of visual information enrichment (5-18) achieves the best performance among other ranges, validating the effectiveness of our Finding 2. We observe that other layers also encode hallucination-related information which can be learned to detect hallucinations. However, compared to the explicit pattern in 5-18 layers, the implicit patterns contained in other ranges require computation costs for feature learning. 3.5 Finding 3: Attention Heads Interact with Multiple Objects Incurring Object Hallucination Considering the MHSA mechanism incorporates various attention heads to aggregate information, we analyze heatmaps over the image for both real and hallucinated object tokens (‘bus’ vs ‘people’) to investigate the head behavior during visual information enrichment. A case is shown in Fig. 7 and more examples can be found in Sec. C.4. We can see that each head focuses on distinct local details, yet the attention distribution for the real token consistently aligns with the spatial extent of the corresponding object. In contrast, for the hallucinated token, the heads interact with inconsistent objects in the image during this stage. We speculate that such unstable head behavior encodes mixed information from multiple objects into the object token, potentially leading to hallucinations. Layers 0-4 5-18 19-26 27-31 0-31 Accuracy 82.61 84.46 81.49 77.72 84.46 AUROC 89.57 90.19 87.34 84.30 90.28 Recall 66.22 72.34 68.88 63.56 71.28 Table 1: Object hallucination detection results (%) of different layer ranges of interest on LLaVA-1.5-7B. 4 Object Hallucination Mitigation Method LLaVA-1.5-7B LLaVA-1.5-13B Shikra-7B MiniGPT-4-7B Avg. CS↓subscriptabsentC_S _S ↓ CI↓subscriptabsentC_I _I ↓ F1↑ ↑ CS↓subscriptabsentC_S _S ↓ CI↓subscriptabsentC_I _I ↓ F1↑ ↑ CS↓subscriptabsentC_S _S ↓ CI↓subscriptabsentC_I _I ↓ F1↑ ↑ CS↓subscriptabsentC_S _S ↓ CI↓subscriptabsentC_I _I ↓ F1↑ ↑ CS↓subscriptabsentC_S _S ↓ CI↓subscriptabsentC_I _I ↓ Decoding Strategy Greedy 53.0 15.6 76.7 49.8 14.6 78.2 57.6 15.7 75.3 31.8 12.0 71.1 48.1 14.5 Beam 55.6 15.4 77.5 50.4 13.8 79.0 59.0 16.3 74.6 29.2 9.9 71.2 48.6 13.9 OPERA 45.6 13.1 79.1 42.6 13.2 77.8 41.4 13.7 73.5 25.4 9.6 71.2 38.8 12.4 Contrastive Decoding VCD† 58.6 18.2 72.8 53.6 15.3 75.8 56.4 15.5 75.2 41.4 14.1 68.2 52.5 15.8 PAI† 24.2 7.1 75.2 33.0 9.2 77.8 38.6 10.1 76.2 23.2 8.2 71.4 29.8 8.7 Ours† 25.0 6.7 76.1 25.8 8.8 77.3 23.8 9.4 72.7 21.4 8.0 70.8 24.0 8.2 Δ Δ% ↑ ↑3.3% ↓ ↓5.6% ↓ ↓21.8% ↓ ↓4.3% ↓ ↓38.3% ↓ ↓6.9% ↓ ↓7.8% ↓ ↓2.4% ↓ ↓19.5% ↓ ↓5.7% Table 2: CHAIR hallucination evaluation and F1 results on three LVLMs with max new token set to 512. † † denotes using the greedy decoding strategy. Our method outperforms other baselines and also preserves the richness of the descriptions. Δ Δ% denotes the relative performance improvement with respect to the second-best method. 4.1 Heads Guided Attention Intervention Inspired by the insights from Sec. 3 that improper processing of visual information during inference can lead to object hallucinations, we aim to correct this process to alleviate object hallucination. Specifically, we adjust the attention weights assigned to image tokens, i.e., k(ℓ,h)⁢(ak,i)i=1nsuperscriptsubscriptsuperscriptsubscriptℓℎsubscript1\A_k^( ,h)(a_k,i)\_i=1^n Aitalic_k( ℓ , h ) ( aitalic_k , i ) i = 1n, in each head at the middle layers during inference for intervention. To modify the attention weights in time step k, we extract the attention score matrix k(ℓ,h)superscriptsubscriptℓℎS_k^( ,h)Sitalic_k( ℓ , h ) before softmax operation: k(ℓ,h)=(ℓ,h⁢ℓ,h⊤dk)k,superscriptsubscriptℓℎsubscriptsubscriptℓℎsuperscriptsubscriptℓℎtopsubscriptS_k^( ,h)= ( Q_ ,hK_ ,h^% d_k )_k,Sitalic_k( ℓ , h ) = ( divide start_ARG Qroman_ℓ , h Kroman_ℓ , h⊤ end_ARG start_ARG square-root start_ARG ditalic_k end_ARG end_ARG )k , (5) where ℓ,hsubscriptℓℎQ_ ,hQroman_ℓ , h and ℓ,h∈ℝak×dksubscriptℓℎsuperscriptℝsubscriptsubscriptK_ ,h∈R^a_k× d_kKroman_ℓ , h ∈ blackboard_Raitalic_k × ditalic_k represent the query and key matrices of dimension dksubscriptd_kditalic_k, respectively. Guided by the attention pattern difference in Finding 2, we amplify the original attention scores in the layers of visual information enrichment by adding positive values to enhance visual information interaction. Furthermore, leveraging the head behavior observed in Finding 3, we compute these positive values by averaging the absolute attention scores across all heads, where only the consistent regions interacted by different heads receive large enhancement. By integrating the attention information from various heads, we can find a more faithful and object-related direction for attention shift to reduce hallucinations. Formally, for each image token i∈[n]delimited-[]i∈[n]i ∈ [ n ] across all heads h∈[H]ℎdelimited-[]h∈[H]h ∈ [ H ] in layer ℓ∈[ℓs,ℓe]ℓsubscriptℓsubscriptℓ ∈[ _s, _e]ℓ ∈ [ ℓitalic_s , ℓitalic_e ], we adjust the visual attention scores by: k(ℓ,h)⁢(ak,i)=k(ℓ,h)⁢(ak,i)+α⁢1H⁢∑h=1H|k(ℓ,h)⁢(ak,i)|,superscriptsubscriptℓℎsubscriptsuperscriptsubscriptℓℎsubscript1superscriptsubscriptℎ1superscriptsubscriptℓℎsubscriptS_k^( ,h)(a_k,i)=S_k^( ,h)(a_k,i)+α% 1H _h=1^H|S_k^( ,h)(a_k,i)|,Sitalic_k( ℓ , h ) ( aitalic_k , i ) = Sitalic_k( ℓ , h ) ( aitalic_k , i ) + α divide start_ARG 1 end_ARG start_ARG H end_ARG ∑h = 1H | Sitalic_k( ℓ , h ) ( aitalic_k , i ) | , (6) where the [ℓs,ℓe]subscriptℓsubscriptℓ[ _s, _e][ ℓitalic_s , ℓitalic_e ] denotes the range of visual information enrichment, and the parameter α is defined as a balance factor to control the intervention strength. 4.2 Experimental Setup and Results Models. We conduct experiments on three representative LVLMs to evaluate the effectiveness and generalization of our method, including the LLaVA-1.5 [27], Shikra [4], and MiniGPT-4 [54]. In our experiments, we use the 7B and 13B versions of LLaVA-1.5 to study the scale effect, while other LVLMs are 7B models. Baselines. We adopt two commonly used decoding strategies and three mainstream hallucination mitigation approaches as the baseline methods, including greedy decoding, beam search decoding [37], OPERA [19], VCD [24], and PAI [28]. Greedy decoding selects the token with the highest probability over the vocabulary as the predicted next token. Beam search decoding maintains multiple beams and selects the top tokens with the highest cumulative probabilities at the end of generation. Improved on beam search, OPERA introduces an overtrust penalty term on the attention weights during inference with a rollback strategy to mitigate hallucinations. Different from the above three decoding approaches, VCD and PAI can be categorized as contrastive decoding methods. Specifically, VCD subtracts the output logits of the distorted visual input from the original output logits to reduce the statistical priors. In addition to using contrastive decoding, PAI further manipulates the attention matrix to overcome language bias. In our experiments, we use the default parameters of these baselines and unify Nbeam=5subscriptbeam5N_ beam=5Nbeam = 5 for the beam search decoding and OPERA. Benchmark and Metrics. Following [19], we conduct experiments on 500 random images from the COCO 2014 validation set. To evaluate the degree of object hallucinations in the image captioning task, we adopt the CHAIR [34] criteria which computes the proportion of all objects mentioned in the caption that are not present in the ground-truth annotations. CHAIR provides two main metrics, including CHAIRI (CIsubscriptC_ICitalic_I) and CHAIRS (CSsubscriptC_SCitalic_S) that assess instance-level and sentence-level hallucinations, respectively: CI=|hallucinated objects||all mentioned objects|,CS=|captions w/ hallucinated objects||all captions|,formulae-sequencesubscripthallucinated objectsall mentioned objectssubscriptcaptions w/ hallucinated objectsall captionsC_I= | \hallucinated objects\|| % \all mentioned objects\|,C_S= | \% captions w/ hallucinated objects\|| \all captions\|,Citalic_I = divide start_ARG | hallucinated objects | end_ARG start_ARG | all mentioned objects | end_ARG , Citalic_S = divide start_ARG | captions w/ hallucinated objects | end_ARG start_ARG | all captions | end_ARG , where lower values indicate fewer hallucinations. Following [28], we also adopt the F1 score to assess the richness and accuracy of the generated descriptions. Implementation details. Our method contains two parameters: the visual information enrichment range [ℓs,ℓe]subscriptℓsubscriptℓ[ _s, _e][ ℓitalic_s , ℓitalic_e ] and the balance factor α in Eq. 6. Utilizing the VAR feature and logit lens described in Sec. 3.1, we can easily establish suitable ranges for any other LVLMs. In our experiment, we set the ranges as [5,18]518[5,18][ 5 , 18 ] for both versions of LLaVA-1.5, [3,13]313[3,13][ 3 , 13 ] for Shikra, and [3,14]314[3,14][ 3 , 14 ] for MiniGPT-4. We set α to 0.5 for LLaVA-1.5, MiniGPT-4, and 0.55 for Shikra. We implement our method with greedy decoding as a baseline. Results. Tab. 2 presents the experimental results for the selected LVLMs with max new tokens of 512. From the Tab. 2, our approach outperforms the three decoding strategies in reducing hallucinations and achieves an average reduction of 19.5% in CSsubscriptC_SCitalic_S and 5.7% in CIsubscriptC_ICitalic_I, compared to the second-best baseline. Compared to VCD, our method not only achieves superior hallucination mitigation performance but also better preserves the richness of the descriptions. Compared to PAI which requires the cost of once additional forward process for contrastive decoding, our approach achieves comparable or superior performance by only adjusting attention weights during inference, validating the efficacy of our method. Moreover, the consistent reduction of hallucinations across four different LVLMs confirms the generalizability of our findings. To evaluate the impact of generated token lengths, we conduct experiments on LLaVA-1.5-7B by varying the max new token in 64,128,25664128256\64,128,256\ 64 , 128 , 256 . The results, reported in Tab. 3, show that our method consistently reduces hallucinations across different token lengths and achieves the best performance on average, demonstrating its robustness. We provide qualitative results in Sec. C.7. Method max new token 64 max new token 128 max new token 256 Avg. CS↓subscriptabsentC_S _S ↓ CI↓subscriptabsentC_I _I ↓ F1↑ ↑ CS↓subscriptabsentC_S _S ↓ CI↓subscriptabsentC_I _I ↓ F1↑ ↑ CS↓subscriptabsentC_S _S ↓ CI↓subscriptabsentC_I _I ↓ F1↑ ↑ CS↓subscriptabsentC_S _S ↓ CI↓subscriptabsentC_I _I ↓ Decoding Strategy Greedy 23.8 8.0 74.7 52.4 15.5 76.7 53.0 15.6 76.7 43.1 13.0 Beam 17.6 6.0 75.1 54.4 14.8 77.6 55.6 15.4 77.5 42.5 12.1 OPERA 19.0 6.3 74.4 44.2 12.9 78.8 45.6 13.1 79.1 36.3 10.8 Contrastive Decoding VCD† 26.0 9.3 73.3 56.8 16.9 74.5 58.6 18.2 72.8 47.1 14.8 PAI† 19.8 6.2 74.0 24.2 7.3 75.2 24.2 7.2 75.2 22.7 6.9 Ours† 18.2 5.4 73.9 24.8 7.1 76.0 25.0 7.1 76.1 22.7 6.5 Table 3: CHAIR hallucination evaluation and F1 results on LLaVA-1.5-7B with varying max new token in 64,128,25664128256\64,128,256\ 64 , 128 , 256 . † † denotes using the greedy decoding strategy. 5 Ablations and Discussions Layers 0-4 5-18 19-26 27-31 Greedy CS↓subscriptabsentC_S _S ↓ 31.0 25.0 53.4 53.4 53.0 CI↓subscriptabsentC_I _I ↓ 10.4 6.7 14.3 15.7 15.6 F1↑ ↑ 77.1 76.1 77.5 77.2 76.7 Table 4: CHAIR hallucination evaluation and F1 score among four layer ranges on LLaVA-1.5-7B with max new token set to 512. Layers of attention intervention. Motivated by the empirical observation of implicit hallucination patterns across layers beyond the 5-18 range in Sec. 3.4.1, we investigate whether interventions in these layers can also mitigate object hallucinations. We apply our approach (Eq. 6) to the layer ranges [0,4]04[0,4][ 0 , 4 ], [19,26]1926[19,26][ 19 , 26 ], and [27,31]2731[27,31][ 27 , 31 ] in LLaVA-1.5-7B, and the results are reported in Tab. 4. We observe that the 5-18 layer range largely outperforms the other ranges while using some ranges (19-26 and 27-31) even enhanced hallucinations instead. The result suggests that explicit patterns in visual information enrichment are more effective than implicit patterns, facilitating more effective hallucination mitigation through simple attention intervention. Different attention intervention strategies. To validate the efficacy of our method in adjusting attention weights to reduce object hallucinations, we compare it with the inference intervention component proposed in PAI [28]. The inference intervention component roughly excites attention weights by computing a weighted sum of the attention scores and their absolute values across individual heads. We apply this approach to the same layer ranges in ours on LLaVA-1.5-7B, Shikra, and MiniGPT-4. Tab. 5 suggests that our attention intervention method surpasses that of PAI, as our method integrates attention information from various heads to shift attention toward a more truthful direction. Figure 8: Parameter sensitivity of α with CHAIR metrics. Method LLaVA-1.5-7B Shikra-7B MiniGPT-4-7B CS↓subscriptabsentC_S _S ↓ CI↓subscriptabsentC_I _I ↓ CS↓subscriptabsentC_S _S ↓ CI↓subscriptabsentC_I _I ↓ CS↓subscriptabsentC_S _S ↓ CI↓subscriptabsentC_I _I ↓ PAI⋆ 26.6 8.1 45.8 13.6 30.2 9.5 Ours 25.0 6.7 23.8 9.4 21.4 8.0 Table 5: CHAIR hallucination evaluation and F1 results on three LVLMs. ⋆ ⋆ denotes only using the inference intervention component in the same layer range as ours. Greedy decoding is adopted. Balance factor α sensitivity. To examine the influence of α on intervention efficacy, we vary its value in the range of 0.3,0.4,0.5,0.6,0.70.30.40.50.60.7\0.3,0.4,0.5,0.6,0.7\ 0.3 , 0.4 , 0.5 , 0.6 , 0.7 and the results of CHAIR metrics and F1 score for the 7B version of both LLaVA-1.5 and MiniGPT-4 are shown in Fig. 8. We find that a lower α limits hallucination mitigation effectiveness, whereas a higher α compromises the richness of descriptions. These results indicate that an appropriate α value (e.g., 0.5 for both selected LVLMs) balances the reduction of hallucinations with the maintenance of detailed visual descriptions. 6 Conclusion and Insights We introduce a new perspective, visual information processing, to investigate the underlying mechanisms driving object hallucination. Analyzing through the lens of attention, we identify three key findings and discover a simple yet effective method for reducing hallucinations at inference time. Our study deepens the understanding of how hallucinations are produced in LVLMs. We conclude with three insights for future direction: 1) Based on LVLMs’ interpretation in semantic refinement stage, the image tokens may be further categorized into sub-groups, such as object tokens, color tokens, and textual tokens, enabling us to explore how the LVLMs extract different types of visual information. 2) While we focus on visual information processing, there may exist more granular, stage-wise mechanisms that could explain model behavior more precisely. 3) Instead of external interventions on LVLM outputs, such as contrastive decoding, we suggest that targeted modifications to specific internal states, like attention weights in middle layers, offer a more effective method for calibration. Our insights may aid the community in developing more trustworthy and reliable LVLMs against hallucinations. Acknowledgment This work is supported by the National Science Foundation of China (62376281, 62206048), the Natural Science Foundation of Jiangsu Province (BK20220819), and the Fundamental Research Funds for the Central Universities (2242024k30035). References Cao et al. [2020] Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. Behind the scene: Revealing the secrets of pre-trained vision-and-language models. In Computer Vision – ECCV 2020, pages 565–580, 2020. Cao et al. [2024] Min Cao, Yang Bai, Ziyin Zeng, Mang Ye, and Min Zhang. An empirical study of clip for text-based person search. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 465–473, 2024. Chefer et al. [2021] Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 397–406, 2021. Chen et al. [2023] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. Chen et al. [2024] Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Halc: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425, 2024. Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2023. Cui et al. [2024] Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 958–979, 2024. Fang et al. [2023] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19358–19369, 2023. Favero et al. [2024] Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14303–14312, 2024. Gandelsman et al. [2024] Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting CLIP’s image representation via text-based decomposition. In International Conference on Learning Representations, 2024. Geva et al. [2022] Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 30–45, 2022. Geva et al. [2023] Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 12216–12235, 2023. Ghandeharioun et al. [2024] Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models. In International Conference on Machine Learning, pages 15466–15490. PMLR, 2024. Gunjal et al. [2024] Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 18135–18143, 2024. Guo et al. [2024] Wei Guo, Hao Wang, Luankang Zhang, Jin Yao Chin, Zhongzhou Liu, Kai Cheng, Qiushi Pan, Yi Quan Lee, Wanqi Xue, Tingjia Shen, et al. Scaling new frontiers: Insights into large recommendation models. arXiv preprint arXiv:2412.00714, 2024. Halawi et al. [2024] Danny Halawi, Jean-Stanislas Denain, and Jacob Steinhardt. Overthinking the truth: Understanding how language models process false demonstrations. In International Conference on Learning Representations, 2024. He et al. [2024] Jinwen He, Yujia Gong, Zijin Lin, Yue Zhao, Kai Chen, et al. Llm factoscope: Uncovering llms’ factual discernment through measuring inner states. In Findings of the Association for Computational Linguistics ACL 2024, pages 10218–10230, 2024. He et al. [2023] Yuting He, Guanyu Yang, Rongjun Ge, Yang Chen, Jean-Louis Coatrieux, Boyu Wang, and Shuo Li. Geometric visual similarity learning in 3d medical image self-supervised pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9538–9547, 2023. Huang et al. [2024] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13418–13427, 2024. Jiang et al. [2024a] Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27036–27046, 2024a. Jiang et al. [2024b] Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. Interpreting and editing vision-language representations to mitigate hallucinations. arXiv preprint arXiv:2410.02762, 2024b. Kingma [2014] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. Kong et al. [2024] Youyong Kong, Xiaotong Zhang, Wenhan Wang, Yue Zhou, Yueying Li, and Yonggui Yuan. Multi-scale spatial-temporal attention networks for functional connectome classification. IEEE Transactions on Medical Imaging, 2024. Leng et al. [2024] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13882, 2024. Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014. Liu et al. [2024a] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In International Conference on Learning Representations, 2024a. Liu et al. [2024b] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024b. Liu et al. [2024c] Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. arXiv preprint arXiv:2407.21771, 2024c. nostalgebraist [2020] nostalgebraist. Interpreting gpt: The logit lens. https://w.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens, 2020. Palit et al. [2023] Vedant Palit, Rohan Pandey, Aryaman Arora, and Paul Pu Liang. Towards vision-language mechanistic interpretability: A causal tracing tool for blip. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2856–2861, 2023. Parekh et al. [2024] Jayneel Parekh, Pegah Khayatan, Mustafa Shukor, Alasdair Newson, and Matthieu Cord. A concept-based explainability framework for large multimodal models. arXiv preprint arXiv:2406.08074, 2024. Peng et al. [2025] Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536, 2025. Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021. Rohrbach et al. [2018] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 4035–4045, 2018. Shen et al. [2024] Tingjia Shen, Hao Wang, Chuhan Wu, Jin Yao Chin, Wei Guo, Yong Liu, Huifeng Guo, Defu Lian, Ruiming Tang, and Enhong Chen. Optimizing sequential recommendation models with scaling laws and approximate entropy. arXiv preprint arXiv:2412.00430, 2024. Su et al. [2024] Weihang Su, Changyue Wang, Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, and Yiqun Liu. Unsupervised real-time hallucination detection based on the internal states of large language models. arXiv preprint arXiv:2403.06448, 2024. Sutskever et al. [2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, page 3104–3112, 2014. Touvron et al. [2023a] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. Touvron et al. [2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b. Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017. Wang et al. [2023] Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An llm-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311.07397, 2023. Wang et al. [2024] Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. arXiv preprint arXiv:2403.18715, 2024. Wu et al. [2024] Junfei Wu, Qiang Liu, Ding Wang, Jinghao Zhang, Shu Wu, Liang Wang, and Tieniu Tan. Logical closed loop: Uncovering object hallucinations in large vision-language models. arXiv preprint arXiv:2402.11622, 2024. Yin et al. [2023] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023. Yin et al. [2024] Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67(12):1–13, 2024. Yu et al. [2024] Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12944–12953, 2024. Yuksekgonul et al. [2024] Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, and Besmira Nushi. Attention satisfies: A constraint-satisfaction lens on factual errors of language models. In International Conference on Learning Representations, 2024. Zhai et al. [2024] Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, Chunyuan Li, and Manling Li. Halle-control: Controlling object hallucination in large multimodal models. arXiv preprint arXiv:2310.01779, 2024. Zhang et al. [2024] Yi-Kai Zhang, Shiyin Lu, Yang Li, Yanqing Ma, Qingguo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, and Han-Jia Ye. Wings: Learning multimodal llms without text-only forgetting. In Advances in Neural Information Processing Systems, pages 31828–31853, 2024. Zhao et al. [2024a] Linxi Zhao, Yihe Deng, Weitong Zhang, and Quanquan Gu. Mitigating object hallucination in large vision-language models via classifier-free guidance. arXiv preprint arXiv:2402.08680, 2024a. Zhao et al. [2024b] Zhen Zhao, Jingqun Tang, Binghong Wu, Chunhui Lin, Shu Wei, Hao Liu, Xin Tan, Zhizhong Zhang, Can Huang, and Yuan Xie. Harmonizing visual text comprehension and generation. In Advances in Neural Information Processing Systems, pages 97499–97522, 2024b. Zhou et al. [2024] Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. In International Conference on Learning Representations, 2024. Zhu et al. [2023] Beier Zhu, Yulei Niu, Saeil Lee, Minhoe Hur, and Hanwang Zhang. Debiased fine-tuning for vision-language models by prompt regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3834–3842, 2023. Zhu et al. [2024] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In International Conference on Learning Representations, 2024. Supplementary Material Appendix A Limitations Despite the simplicity and effectiveness of our hallucination detection and mitigation, there are several limitations: • First, the SVAR metric, used to detect hallucinated object tokens, is limited by the inherent attention behavior of LVLMs. When LVLM consistently exhibits extremely high visual attention ratios at nearly all layers, such as the case of Shikra illustrated in Fig. 13 (a), this may weaken the effectiveness of the SVAR metric. • Second, although the use of the VAR score and logit lens approach can intuitively distinguish two stages of visual information processing, identifying the specific range of these stages remains somewhat subjective. However, leveraging learnable strategies, such as training a set of learnable weights for layers based on the signals from VAR distribution and prediction contributions, could potentially achieve automatic localization of these stages, and we leave this for future work. Model No. of Real No. of Hallucinated LLaVA-1.5-7B 4,397 1,842 LLaVA-1.5-13B 4,488 1,700 Shikra-7B 4,263 1,794 MiniGPT-4-7B 2,999 981 Table 6: Statistical information of case datasets. Appendix B Experiment Details B.1 Datasets for Case Study Tab. 6 reports the statistical information of the synthetic datasets used in our case studies. Additionally, Fig. 9 illustrates the positional distributions of real and hallucinated object tokens for the four selected LVLMs. Figure 9: Real and hallucinated object token distributions by their position in description (%). Figure 10: Illustration of detecting hallucinated object tokens by training an MLP classifier on the concatenated VAR scores. Hyperparameters LLaVA-1.5-7B Optimizer Adam [22] (β1subscript1 _1β1, β2subscript2 _2β2) (0.9, 0.999) Hidden size 64, 128, 256, 512 Learning rate 1e-2, 1e-3, 1e-4 No. of epochs 200 Table 7: Training hyperparameters of the two-layer MLP for hallucination detection on LLaVA-1.5-7B. B.2 MLP Training Details Fig. 10 shows the training pipeline of the object hallucination detector. Tab. 7 details the hyperparameters used to train the two-layer MLP, designed for detecting hallucinated object tokens as described in Sec. 3.4.1. The Adam optimizer is employed to train the classifier with the number of epochs set to 200. For each layer range, we utilize a gride search strategy to find the optimal hidden layer size and learning rate within the ranges of 64, 128, 256, 512 and 1e-2, 1e-3, 1e-4, respectively. Appendix C Additional Results C.1 Case Study Results In this subsection, we conduct additional experiments on LLaVA-1.5-13B, Shikra-7B, and MiniGPT-4-7B to examine whether other models also share similar characteristics with LLaVA-1.5-7B. LLaVA-1.5-13B. Fig. 11 (a) and (b) show the VAR score distribution and the prediction contributions from the MHSA sublayers, respectively. We find the same two patterns in the middle layers analogous to those found in LLaVA-1.5-7B as described in Sec. 3.3, suggesting the model scale generalization of our findings. Fig. 15 presents qualitative comparisons of hallucination detection between the SVAR5⁢-⁢18subscriptSVAR5-18SVAR_5-18SVAR5 - 18 metric and the internal confidence method, demonstrating the superiority of our metric. Figure 11: (a) Distribution of visual attention ratio for real object tokens across heads and layers in LLaVA-1.5-13B, sorted row-wise by attention ratios. (b) The logit contribution of attention sublayers to real token prediction. MiniGPT-4-7B. Fig. 12 (a) and (b) depict the VAR score distribution and the prediction contributions from the MHSA sublayers, respectively. Similar to LLaVA-1.5-7B, the two patterns in the middle layers where the model exhibits continuous higher visual attention can be observed. Notably, we can see that MiniGPT-4-7B does not exhibit the same high attention as LLaVA-1.5 at the 0-th layer. In our experiments, layers 3-14 are selected as the range of the visual information enrichment stage. Fig. 14 (b) reports the SVAR3⁢-⁢14subscriptSVAR3-14SVAR_3-14SVAR3 - 14 value distribution across the two token types, demonstrating a similar trend to LLaVA-1.5-7B as described in Sec. 3.4. These results suggest the model generalization of our findings. The qualitative results of hallucination detection are presented in Fig. 16. Figure 12: (a) Distribution of visual attention ratio for real object tokens across heads and layers in MiniGPT-4-7B, sorted row-wise by attention ratios. (b) The logit contribution of attention sublayers to real token prediction. Shikra-7B. As shown in Fig. 13 (a), Shikra continuously exhibits extremely high VAR scores across layers. To clearly analyze the VAR distribution, a seventh-order polynomial is used to fit the summed VAR values over all heads of each layer (depicted by a red curve). Compared to other layers, we can see that the middle layers exhibit relatively higher VAR scores, aligning with our observation from LLaVA-1.5-7B. Combined with the prediction contributions from MHSA sublayers in Fig. 13 (b), we can also identify two distinct patterns in the middle layers. Like MiniGPT-4-7B, Shikra-7B exhibits low visual attention at the 0-th layer. In our experiments, layers 3-13 are selected as the range of the visual information enrichment stage. Fig. 14 (a) presents the SVAR3⁢-⁢13subscriptSVAR3-13SVAR_3-13SVAR3 - 13 value distribution across the two token types, demonstrating a similar trend to other LVLMs. The comparison results of hallucination detection, displayed in Fig. 17, show that our simple SVAR3⁢-⁢13subscriptSVAR3-13SVAR_3-13SVAR3 - 13 metric performs comparably to the more complex baseline that projects the hidden states of all image tokens at all layers into the vocabulary space. Compared to other LVLMs, the decreased performance of the SVAR metric on Shikra-7B may be attributed to the extremely high VAR scores across nearly all layers, potentially reducing the sensitivity of our metric to attention pattern differences. Figure 13: (a) Distribution of visual attention ratio for real object tokens across heads and layers in Shikra-7B, sorted row-wise by attention ratios. Note that the red curve represents a seventh-order polynomial fit to the values of attention ratios summed over heads in each layer. (b) The logit contribution of attention sublayers to real token prediction. Figure 14: SVAR3⁢-⁢13subscriptSVAR3-13SVAR_3-13SVAR3 - 13 and SVAR3⁢-⁢14subscriptSVAR3-14SVAR_3-14SVAR3 - 14 score distributions across object token types for Shikra-7B (a) and MiniGPT-4-7B (b), respectively. C.2 Results of Statistical Tests To assess the statistical significance of the SVAR score being higher for real object tokens than for hallucinated ones during visual information enrichment, we conduct a one-tailed t-test for each LVLM. We present the results in Tab. 8 for LLaVA-1.5-7B, Tab. 9 for LLaVA-1.5-13B, Tab. 10 for Shikra-7B, and Tab. 11 for MiniGPT-4-7B. Across all models, the results consistently indicate that significantly higher attention weights are assigned to image tokens when generating real object tokens, compared to hallucinated ones. Figure 15: Object hallucinations detection curves for LLaVA-1.5-13B. Figure 16: Object hallucinations detection curves for MiniGPT-4-7B. Figure 17: Object hallucinations detection curves for Shikra-7B. C.3 Numerical Results of α Sensitivity Tab. 12 presents the sensitivity results of the balance factor α, used in our attention intervention method (Eq. 6), on LLaVA-1.5-7B, LLaVA-1.5-13B, and MiniGPT-4-7B. In addition to modulating the trade-off between hallucination mitigation and description richness as discussed in Sec. 5, we find that LLaVA-1.5-7B and LLaVA-1.5-13B are more sensitive to changes in α compared to MiniGPT-4-7B. A possible reason for this increased sensitivity may be that LLaVA-1.5 uses substantially more image tokens than MiniGPT-4 (576 versus 32), potentially magnifying the impact of the parameter α. LLaVA-1.5-7B Real Hallucinated SVAR5⁢-⁢18subscriptSVAR5-18 SVAR_5 -18SVAR5 - 18 score 1.70 1.25 t-statistic 32.44 p-value 1.03E-213 df 6,237 Table 8: Results of one-tailed t-tests for LLaVA-1.5-7B. The null hypothesis is the mean SVAR5⁢-⁢18subscriptSVAR5-18 SVAR_5 -18SVAR5 - 18 score of real object tokens is less than or equal to the mean SVAR5⁢-⁢18subscriptSVAR5-18 SVAR_5-18SVAR5 - 18 score of hallucinated ones. LLaVA-1.5-13B Real Hallucinated SVAR5⁢-⁢18subscriptSVAR5-18 SVAR_5 -18SVAR5 - 18 score 1.50 1.06 t-statistic 32.24 p-value 3.25E-211 df 6,186 Table 9: Results of one-tailed t-tests for LLaVA-1.5-13B. The null hypothesis is the mean SVAR5⁢-⁢18subscriptSVAR5-18 SVAR_5-18SVAR5 - 18 score of real object tokens is less than or equal to the mean SVAR5⁢-⁢18subscriptSVAR5-18 SVAR_5 -18SVAR5 - 18 score of hallucinated ones. Shikra-7B Real Hallucinated SVAR3⁢-⁢13subscriptSVAR3-13 SVAR_3-13SVAR3 - 13 score 5.86 5.43 t-statistic 22.56 p-value 1.50E-108 df 6,055 Table 10: Results of one-tailed t-tests for Shikra-7B. The null hypothesis is the mean SVAR3⁢-⁢13subscriptSVAR3-13 SVAR_3-13SVAR3 - 13 score of real object tokens is less than or equal to the mean SVAR3⁢-⁢13subscriptSVAR3-13 SVAR_3 -13SVAR3 - 13 score of hallucinated ones. MiniGPT-4-7B Real Hallucinated SVAR3⁢-⁢14subscriptSVAR3-14 SVAR_3-14SVAR3 - 14 score 2.25 1.67 t-statistic 22.06 p-value 4.21E-102 df 3,978 Table 11: Results of one-tailed t-tests for MiniGPT-4-7B. The null hypothesis is the mean SVAR3⁢-⁢14subscriptSVAR3-14 SVAR_3-14SVAR3 - 14 score of real object tokens is less than or equal to the mean SVAR3⁢-⁢14subscriptSVAR3-14 SVAR_3 -14SVAR3 - 14 score of hallucinated ones. α LLaVA-1.5-7B LLaVA-1.5-13B MiniGPT-4-7B CS↓subscriptabsentC_S _S ↓ CI↓subscriptabsentC_I _I ↓ F1↑ ↑ CS↓subscriptabsentC_S _S ↓ CI↓subscriptabsentC_I _I ↓ F1↑ ↑ CS↓subscriptabsentC_S _S ↓ CI↓subscriptabsentC_I _I ↓ F1↑ ↑ Greedy 53.0 15.6 76.7 49.8 14.6 78.2 31.8 12.0 71.1 0.3 41.8 12.2 77.9 44.4 12.5 78.1 28.0 10.0 70.5 0.4 41.6 11.5 78.1 44.6 13.2 77.5 27.2 10.8 71.2 0.5 25.0 6.7 76.1 25.8 8.8 77.3 22.4 8.6 70.8 0.6 1.0 0.9 49.3 6.4 3.3 57.7 20.6 8.6 69.5 0.7 1.6 2.0 36.8 2.4 26.3 40.0 14.6 5.9 67.4 Table 12: Numerical results of balance factor α sensitivity. LLaVA-1.5-7B Greedy Beam OPERA VCD† PAI† Ours CHAIR↓ ↓ 7.7 9.1 7.3 8.6 4.9 4.3 Hal↓ ↓ 35.4 39.8 31.5 39.5 24.4 20.2 Cog↓ ↓ 4.3 4.8 2.9 4.5 1.6 1.2 Table 13: AMBER results on LLaVA-1.5-7B with max new token set to 512. † † denotes using the greedy decoding strategy. C.4 Attention Heads Behavior Visualization We exhibit more visualization examples of LLaVA-1.5-7B in Figs. 18 and 19 to validate that the heads interact with inconsistent objects in the image during visual information enrichment when generating hallucinated object tokens. C.5 Comparison Results on AMBER Benchmark We further evaluate our approach on AMBER [41] benchmark, which contains 1,004 images for the generative task. The results presented in Tab. 13 demonstrate its superior performance. C.6 Some Intuition Behind Layer Division The layers of LLaVA-1.5-7B are divided into four ranges according to the patterns identified from the results of VAR score (Fig. 2 (a)) and logit lens (Fig. 2 (b) and Fig. 3). We find Range 1 (layer 0-4): low-level image processing, VAR attention pattern differs in different models; Range 2 (layer 5-18): visual information enrichment, accumulates the visual information exhibiting high VAR scores and low logit contribution; Range 3 (layer 19-26): semantic refinement, interacts semantic information of image tokens with high VAR scores and reasons object token prediction with high logit contribution; Range 4 (layer 27-31): grammar concern, guarantees the coherence and correctness of response with low VAR scores, in which the model tends to interpret image tokens as punctuation marks or conjunctions (Fig. 3). These patterns can be generalized to divide other LVLMs similarly. It is worth noting that the first and last divisions are not fixed at 5 layers but vary with models. C.7 Qualitative Results of Hallucination Mitigation We provide sample results from our hallucination mitigation method as described in Sec. 4.1, which corrects the attention distribution over image tokens, in Fig. 20, Fig. 21 and Fig. 22 for LLaVA-1.5-7B, MiniGPT-4-7B, and Shikra-7B, respectively. Figure 18: Attention heads behavior in LLaVA-1.5-7B. Figure 19: Attention heads behavior in LLaVA-1.5-7B. Figure 20: Qualitative results of hallucination mitigation on LLaVA-1.5-7B. The real and hallucinated object words are marked in blue and red, respectively. Our prompt is “Please help me describe the image in detail.”. Figure 21: Qualitative results of hallucination mitigation on MiniGPT-4-7B. The real and hallucinated object words are marked in blue and red, respectively. Our prompt is “Please help me describe the image in detail.”. Figure 22: Qualitative results of hallucination mitigation on Shikra-7B. The real and hallucinated object words are marked in blue and red, respectively. Our prompt is “Please help me describe the image in detail.”.

Instant research discovery