Paper deep dive
Focus Matters: Phase-Aware Suppression for Hallucination in Vision-Language Models
Sohyeon Kim, Sang Yeon Yoon, Kyeongbo Kong
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 4/10/2026, 2:14:34 AM
Summary
The paper introduces a training-free, inference-time intervention for Large Vision-Language Models (LVLMs) to mitigate object hallucinations. By analyzing attention dynamics in vision encoders, the authors identify a three-phase structure: diffusion, focus, and rediffusion. They demonstrate that suppressing low-attention tokens specifically during the 'focus' phase—using a Determinantal Point Process (DPP) to maintain diversity—effectively reduces hallucinations without the high computational cost of iterative adversarial optimization.
Entities (4)
Relation Signals (3)
vision encoder → exhibits → Three-phase attention structure
confidence 98% · identify a consistent three-phase structure of visual information processing: diffusion, focus, and rediffusion
Focus-Aware Suppression → mitigates → Object Hallucination
confidence 95% · the proposed approach consistently reduces hallucination metrics
Determinantal Point Process → optimizes → Token Selection
confidence 92% · employs a Determinantal Point Process (DPP) to preserve diverse visual cues while filtering redundant tokens
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Vision-Language Models (LVLMs) have achieved impressive progress in multimodal reasoning, yet they remain prone to object hallucinations, generating descriptions of objects that are not present in the input image. Recent approaches attempt to mitigate hallucinations by suppressing unreliable visual signals in the vision encoder, but many rely on iterative optimization for each input, resulting in substantial inference latency. In this work, we investigate the internal attention dynamics of vision encoders in LVLMs and identify a consistent three-phase structure of visual information processing: diffusion, focus, and rediffusion. Our analysis reveals that hallucination behavior is particularly sensitive to tokens receiving low attention during the focus phase. Motivated by this observation, we propose a lightweight inference-time intervention that selectively suppresses such tokens during the focus phase. The method operates in a training-free manner using statistics from a single forward pass and employs a Determinantal Point Process (DPP) to preserve diverse visual cues while filtering redundant tokens. Extensive experiments across multiple LVLM backbones and decoding strategies demonstrate that the proposed approach consistently reduces hallucination metrics while maintaining competitive caption quality. Moreover, compared to adversarial uncertainty estimation methods, our approach achieves comparable hallucination mitigation with negligible additional inference latency.
Tags
Links
- Source: https://arxiv.org/abs/2604.03556v1
- Canonical: https://arxiv.org/abs/2604.03556v1
Trouble viewing inline? Open PDF directly →
Full Text
155,803 characters extracted from source content.
Expand or collapse full text
Focus Matters: Phase-Aware Suppression for Hallucination in Vision-Language Models Sohyeon Kim 1 , Sang Yeon Yoon 2 , and Kyeongbo Kong 1† 1 Pusan National University, shkim0503, kbkong@pusan.ac.kr 2 Pukyong National University, palsc@pukyong.ac.kr Project Page: https://cvsp-lab.github.io/FocusMatters/ Abstract. Large Vision-Language Models (LVLMs) have achieved im- pressive progress in multimodal reasoning, yet they remain prone to ob- ject hallucinations, generating descriptions of objects that are not present in the input image. Recent approaches attempt to mitigate hallucinations by suppressing unreliable visual signals in the vision encoder, but many rely on iterative optimization for each input, resulting in substantial inference latency. In this work, we investigate the internal attention dy- namics of vision encoders in LVLMs and identify a consistent three-phase structure of visual information processing: diffusion, focus, and rediffu- sion. Our analysis reveals that hallucination behavior is particularly sen- sitive to tokens receiving low attention during the focus phase. Motivated by this observation, we propose a lightweight inference-time interven- tion that selectively suppresses such tokens during the focus phase. The method operates in a training-free manner using statistics from a sin- gle forward pass and employs a Determinantal Point Process (DPP) to preserve diverse visual cues while filtering redundant tokens. Extensive experiments across multiple LVLM backbones and decoding strategies demonstrate that the proposed approach consistently reduces hallucina- tion metrics while maintaining competitive caption quality. Moreover, compared to adversarial uncertainty estimation methods, our approach achieves comparable hallucination mitigation with negligible additional inference latency. 1 Introduction Qwen-2.5-VL Vision encoder time Total time 3030 Intern-VL-2.5 Vision encoder time 4040 Total time Time(s) 20 10 Time(s) 20 10 Time(s) 30 20 10 Time(s) 30 20 10 Orign. 0 AUE Ours Orign. 0 AUE Ours Orign. 0 AUE Ours Orign. 0 AUE Ours Qwen-2.5-VL Intern-VL-2.5 Orign.AUEOursOrign.AUEOurs CHAIRS↓ CHAIRI↓ 31.4 7.6 27.8 7.4 27.8 7.3 28.6 7.1 28.4 7.0 27.0 6.8 Fig. 1: Runtime comparison and halluci- nation mitigation performance (CHAIR) across recent LVLMs. Large Vision-Language Models (LVLMs) have recently demonstrated impressive progress in multimodal reasoning and image-groundded language generation. Despite these advances, they remain prone to object hallucination [12], gen- erating descriptions of objects that are not present in the input image. Such hallucinations undermine the reliability of LVLMs and limit their deployment in applications that require trustworthy visual grounding. † Corresponding author. arXiv:2604.03556v1 [cs.CV] 4 Apr 2026 2Kim et al. Existing research on hallucination mitigation can be broadly categorized into decoder-stage and vision encoder-stage interventions. Decoder-based meth- ods [5, 8–10, 15] primarily address hallucinations during text generation by ad- justing token probabilities or reweighting attention within the language model. However, hallucinations that manifest during language generation can also be in- fluenced by the quality of upstream visual representations, making interventions at the vision encoder an orthogonal and complementary direction. Recently, Ad- versarial Uncertainty Estimation (AUE) [20] estimates the uncertainty of visual tokens using iterative adversarial optimization with Projected Gradient Descent (PGD) [17], and suppresses the influence of unreliable tokens during visual en- coding. As illustrated in Fig. 1, although AUE effectively reduces hallucinations, its iterative adversarial optimization requires repeated per-image updates, re- sulting in substantially higher runtime and limiting its practicality in large-scale or real-time inference scenarios. In this work, we revisit hallucination mitigation from the perspective of at- tention dynamics within the vision encoder. Through layer-wise analysis of at- tention distributions across multiple LVLM backbones, we observe that visual information processing consistently follows a hierarchical structure consisting of three phases: diffusion, focus, and rediffusion. Attention is broadly distributed during the diffusion phase, becomes highly concentrated on a small subset of tokens during the focus phase, and spreads again during the rediffusion phase as representations propagate to deeper layers. To understand how these phases relate to hallucination behavior, we conduct controlled experiments that modulate token influence across different phases. Specifically, we partition tokens based on attention statistics and scale the rep- resentations of the lower-attention group. Interestingly, hallucination metrics exhibit strong sensitivity to these manipulations during the focus phase, while similar interventions in the diffusion and rediffusion phases have significantly weaker effects. To further analyze how these interventions affect the interaction between the vision encoder and the language model, we employ the Visual Atten- tion Ratio (VAR) [9], which measures how strongly generated tokens attend to visual inputs during decoding. Our analysis shows that suppressing low-attention tokens during the focus phase increases the VAR during generation, suggesting that reducing noisy visual signals encourages the language model to rely more strongly on visual evidence rather than language priors. Motivated by these observations, we propose a simple hierarchical masking strategy that selectively suppresses low-attention tokens during the focus phase. The method operates in a training-free manner using statistics obtained from a single forward pass, eliminating the need for iterative optimization. As shown in Fig. 1, the proposed approach achieves hallucination mitigation performance comparable to AUE while introducing negligible additional runtime. Extensive experiments demonstrate that the proposed approach consistently reduces hallu- cinations across multiple LVLM backbones under the CHAIR [19] metric, while maintaining strong caption fidelity. Our main contributions are summarized as follows: Focus Matters3 – A low-overhead, training-free inference-time intervention. We pro- pose a hallucination mitigation strategy that selectively suppresses visual to- kens within the vision encoder without additional adversarial optimization, achieving performance improvements at a computational cost comparable to standard inference. – Analysis of layer-wise attention dynamics in vision encoders. Through layer-wise attention analysis across multiple LVLM backbones, we identify a consistent three-phase structure of visual information processing consisting of diffusion, focus, and rediffusion. – Correlation between phase-specific token suppression and halluci- nation. Controlled experiments modulating token influence across phases reveal that intervention during the focus phase is closely associated with re- duced hallucinations. VAR analysis further shows that this change correlates with the language model’s visual reference ratio and language bias. – Comprehensive validation through quantitative and qualitative eval- uations. We validate the proposed approach across standard hallucination benchmarks, including CHAIR and POPE [11], and complement these re- sults with sentence-level analysis based on ground-truth captions. 2 Related Works 2.1 Large Vision-Language Models. Driven by the strong language understanding and generation capabilities of large language models (LLMs) [4,22,23], Large Vision-Language Models (LVLMs) [1– 3, 13, 14] have achieved remarkable performance across a wide range of multi- modal tasks. A typical LVLM consists of three core components: (1) a vision encoder, (2) a modality connector, and (3) an LLM. The vision encoder [6,18,27] transforms an input image into visual features. The modality connector aligns the encoded visual features with the textual embedding space. Finally, the LLM performs reasoning and text generation based on the aligned visual and textual embeddings. Built upon this architecture, various LVLMs have demonstrated impressive performance; however, they persistently suffer from hallucination, generating content that is inconsistent with the input image. 2.2 Mitigating hallucinations in LVLMs. Approaches to mitigating hallucination in LVLMs can be broadly categorized into training-based and training-free methods. Recent training-based approaches typically employ a Supervised Fine-Tuned (SFT) LVLM as the reference policy and apply preference optimization using hallucination-specific preference data [7, 21,25,26,28]. However, such methods incur substantial costs for data collection and additional training compared to standard models. To overcome these limitations, training-free methods that directly intervene during inference without parameter updates have been actively explored. These 4Kim et al. methods can be further divided into three categories based on their point of intervention. The first category adjusts the output distribution at the decoding stage of the LLM [5,8,10]. While these methods achieve meaningful improvements through decoding-level manipulation alone, they are fundamentally limited in that they cannot correct erroneously extracted visual information itself. The second category directly modulates the attention mechanism within the LLM after visual tokens have been passed to it [9,15]. However, since these meth- ods attempt post-hoc correction after the vision encoder has already forwarded visual features containing noise and distortions to the LLM, they are structurally limited in their ability to eliminate the root causes of hallucination. The third category identifies the source of hallucination in the uncertainty within the vision encoder and intervenes directly at this level. Adversarial Un- certainty Estimation (AUE) [20] injects adversarial perturbations based on Pro- jected Gradient Descent (PGD) [17] to detect visual tokens with high epistemic uncertainty and masks them at intermediate layers of the vision encoder. While this method carries significant merit in proactively controlling hallucination at the visual token level, it requires iterative adversarial optimization for each input image, incurring computational overhead that renders it impractical for real-time inference. To address the limitations of existing methods, we propose a training-free approach based on the observation that attention dynamics in vision encoders follow a hierarchical structure of diffusion, focus, and rediffusion. Using attention statistics from a single forward pass, the method identifies uncertain tokens and suppresses them during the focus phase, mitigating hallucination without additional computational overhead. 3 Layer-wise Attention Dynamics in Vision Encoders In this section, we analyze how interactions among visual tokens evolve across layers of the vision encoder. First, we introduce a quantitative metric that mea- sures the concentration of attention distributions. Using this metric, we then examine the layer-wise evolution of attention patterns across several LVLM backbones and identify a consistent three-phase structure of visual information processing. 3.1 Attention Concentration Metric To quantify how attention distributions change across layers, we analyze two statistics derived from the attention maps of the vision encoder: the attention entropy and the maximum attention score. At a given layer ℓ and attention head h, we define the attention distribution assigned by the class token (CLS token) to the N spatial patch tokens as A (ℓ,h) =A (ℓ,h) cls,i N i=1 .(1) Focus Matters5 LLaV (a (b ShikrQwen-2.5-VDeepseek-VL 112112112231122 Max score / EntropMax score / EntropMax score / EntropMax score / Entrop LayeLayeLayeLaye Layer Layer Layer Layer Layer 1Layer 1Layer 1Layer 1Layer 2Layer 2 Max score / Entrop 0.0 0.0 0.0 Diffusion PhasFocus PhasRediffusion Phas 0.0 112 Layer Inde Fig. 2: Layer-wise attention dynamics across various LVLM backbones. (a) The progression of the maximum attention score to entropy ratio (R (l) ) across vision encoder layers. (b) Visualization of attention maps, demonstrating a consistent three- phase visual processing structure: diffusion, focus, and rediffusion. For architectures without an explicit class token, such as Qwen-2.5-VL [1], we instead compute the average attention distribution across all visual query tokens: A (ℓ,h) = 1 N N X j=1 A (ℓ,h) j,i N i=1 . (2) The attention entropy is defined as H (ℓ,h) =− N X i=1 A (ℓ,h) i logA (ℓ,h) i ,(3) which measures how broadly attention mass is distributed across tokens. Complementarily, we compute the maximum attention score M (ℓ,h) = max i A (ℓ,h) i ,(4) which captures the dominance of the most attended token. To summarize the relative concentration of attention at each layer, we define the following ratio R (ℓ) = E h [M (ℓ,h) ] E h [H (ℓ,h) ] . (5) This ratio provides a simple measure of attention concentration. While en- tropy captures how broadly attention is distributed across tokens, the maximum attention score reflects the dominance of the most attended token. By combining 6Kim et al. these two statistics, R (ℓ) increases when attention becomes concentrated on a small subset of tokens and decreases when attention is broadly distributed. 3.2 Phase Structure of Layer-wise Attention Dynamics Using the above metric, we analyze the layer-wise evolution of R (ℓ) across several LVLM vision encoders, including LLaVA [13], Shikra [2], Qwen-2.5-VL [1], and DeepSeek-VL2-small [24]. As shown in Fig. 2(a), although the absolute magnitude of R (ℓ) varies across models, the overall trend with respect to network depth remains remarkably consistent. Specifically, the metric remains relatively low in the early layers, increases sharply in the intermediate layers, and decreases again in the later layers. Based on these observations, we partition the vision encoder into three distinct phases exhibiting similar attention characteristics: the diffusion phase, the focus phase, and the rediffusion phase, as illustrated in Fig. 2(b). Phase 1: Diffusion. In the diffusion phase, R (ℓ) remains relatively low, indi- cating that attention is broadly distributed across many tokens. Attention maps show that the locations receiving higher attention vary across layers, suggest- ing that the encoder references a wide range of tokens without concentrating strongly on specific regions. Phase 2: Focus. In the focus phase, R (ℓ) increases sharply and reaches its peak. Attention becomes highly concentrated on a small subset of tokens, while most other tokens receive negligible attention. Phase 3: Rediffusion. In the rediffusion phase, the previously concentrated attention distribution becomes more diffuse again and R (ℓ) gradually decreases. Importantly, this three-phase structure appears consistently across different LVLM backbones despite differences in architecture and scale. Building upon this observation, the next section investigates how modulating tokens within each phase influences hallucination behavior. 4 Correlation Analysis Between Phase-Specific Token Modulation and Hallucination In this section, we examine how the three visual processing phases identified in Sec. 3 relate to the model’s hallucination behavior. We first analyze how hallucination metrics change when tokens within each phase are modulated. We then investigate how such interventions affect the language model’s utilization of visual information using the Visual Attention Ratio (VAR) [9]. 4.1 Hallucination Analysis under Phase-Specific Token Modulation To analyze the influence of tokens identified in each phase on the model output, we systematically modulated the influence of low-attention tokens. Specifically, tokens were partitioned based on attention statistics computed from the lay- ers immediately preceding the intervention. We then varied the influence of the Focus Matters7 !" # " % 8 6416/.257,25+-*6546,0+, 3 /.)6( ,'0)65 ?B % E E E E E E ?Q ?Q ?Q ?Q ?Q ?Q !" # Z % !" # Z % !" # Z % !" # Z % !" # Z % !"# Z % E eB cbeB cbgf gf gf y 5416/.257,25+-*6546, 0+,3 /.)6( ,'0)65 eB cb f " %% f " %% f " % % B % B " B % B " B % B " Fig. 3: Impact of masking strategies across different processing phases on hallucination metrics. Masking visual tokens during the focus phase (Mask 2) ef- fectively reduces hallucinations (indicated by lower CHAIR S (CS) and CHAIR I (CI) scores) while preserving object recognition capabilities (F1 score), highlighting the fo- cus phase as the effective intervention point. lower-attention group by scaling their attention logits, which allowed us to con- tinuously adjust their contribution to the attention distribution. Two extreme cases were additionally evaluated: (i) masking, which suppresses the contribution of these tokens, and (i) inverse masking, which retains only these tokens while suppressing the remaining ones. To examine whether token selection strategies affect this behavior, we com- pared two approaches: a simple attention rank-based Top-k selection and a De- terminantal Point Process (DPP) [16] selection, which jointly accounts for token importance and token diversity. Hallucination changes were evaluated using the CHAIR [19] benchmark together with additional analysis based on ground-truth captions. Effect of Phase-Specific Token Modulation We applied token modulation independently to the diffusion, focus, and rediffusion phases and analyzed the resulting changes in the CHAIR metric (sentence-level CHAIR S and instance- level CHAIR I ), which measure the frequency of generating non-existent objects during caption generation (Fig. 3). In the diffusion and rediffusion phases, modifying the influence of low-attention tokens produced only minor variations in the CHAIR metrics. Neither suppress- ing these tokens nor amplifying their influence resulted in consistent improve- ments or degradations compared to the baseline (no masking). In contrast, interventions applied during the focus phase produced a markedly different behavior. Suppressing low-attention tokens led to a consistent reduc- tion in both CHAIR S and CHAIR I across both Top-k and DPP token selection strategies. Conversely, amplifying the influence of these tokens through inverse masking increased hallucination metrics relative to the baseline. These results in- dicate that hallucination behavior is particularly sensitive to tokens that receive relatively low attention during the focus phase. 8Kim et al. GT captio Phase 1 maskin Phase 2 maskin Phase 3 maskin No maskin An orange-and-white cat is lying inside a white toilet bowl. The cat’s head is raised and facing toward the camera. The cat is positioned on its back with its belly visible. The cat has green eyes and upright ears. The toilet seat is up, forming a white oval rim around the cat A cat is sitting in a sink. The cat is orange and white. The cat has a pink nose. The cat is looking at the camera. The sink is white. A cat is sitting in a white bowl. The cat is orange and white. The cat is looking at the camera. The bowl is on a table. The bowl is filled with water A cat is sitting in a toilet bowl. The cat is orange and white. The cat is looking at the camera. The toilet is white. The toilet is open A cat is sitting in a toilet bowl. The cat is orange and white. The cat has a pink nose. The cat is looking at the camera. The toilet bowl is white. GT captioGT captio Phase 1 maskin Phase 1 maskin Phase 2 maskin Phase 2 maskin Phase 3 maskin Phase 3 maskin No maskinNo maskin Two pugs are sitting on a green slatted bench. Leashes are attached to the dogs and lie across the bench. The pug on the left is facing forward with its front paws apart. The pug on the right is also facing forward. A low wall covered in small square mosaic tiles is behind the bench Two dogs are sitting on a bench. The dogs are on a leash. The dogs are facing each other. The dogs are of different sizes. The bench is green Two dogs are sitting on a bench. The dogs are on a leash. The dogs are facing each other. The dogs are in front of a building. The dogs are sitting next to each other Two dogs are sitting on a bench. The dogs are on a leash. The dogs are facing the same direction. The dogs are of the same breed. The dogs are sitting next to each other Two dogs are sitting on a bench. The dogs are on a leash. The dogs are facing each other. The dogs are sitting next to each other. The bench is green. A man is seated at a table indoors. He is wearing a dark long-sleeve top. He is smiling while looking toward an open laptop. His hands are positioned on the laptop keyboard as if typing. A dark circular sticker with a tree graphic is attached to the laptop lid A man is sitting at a table with a laptop. He is wearing a dark long-sleeve top. The man is smiling while using the laptop. The man is using a mouse with the laptop. The table is white in color A person is sitting at a table with a laptop. The person is wearing a black shirt. The person is smiling. The person is using the laptop. The table is white A man is sitting in front of a laptop. The person is wearing a black shirt. The man is smiling. The man is using the laptop. The laptop has a sticker on it A man is sitting at a table with a laptop. The person is wearing a black shirt. The man is smiling while using the laptop. The table is made of wood. The laptop has a sticker on the back Fig. 4: Qualitative examples of hallucination behavior across masking phases. GT captions are shown for reference, and hallucinated statements are high- lighted in red. While no masking or masking in the diffusion (Phase 1) and rediffusion (Phase 3) phases produces inconsistent results, masking in the focus phase (Phase 2) consistently reduces hallucinations and yields captions more consistent with the image content. Comparison of Token Selection Strategies We next compare the impact of different token selection strategies. Top-k selection removes tokens purely based on attention ranking and yields noticeable reductions in CHAIR metrics. However, this approach also leads to a decrease in the F1 score, suggesting that purely rank-based removal may discard potentially useful visual cues. To address this issue, we employ a DPP for token selection. DPP considers both token importance and similarity among tokens, enabling the selection of a more diverse subset of visual features. Applying DPP-based masking within the focus phase achieves hallucination reduction comparable to Top-k while main- taining higher F1 scores. This indicates that incorporating diversity in token selection provides a better balance between hallucination mitigation and visual information preservation. Qualitative Analysis using Ground-Truth Captions While CHAIR pro- vides a quantitative measure of hallucination frequency, it does not capture how hallucinations manifest at the sentence level. To complement this evaluation, we conducted a qualitative analysis using Ground-Truth (GT) captions. For this analysis, we constructed hallucination-free GT captions for COCO images by generating initial descriptions using GPT-5.2 and refining them through human verification. Each model was then configured to generate multiple de- Focus Matters9 # 0 ; G C U MOLK J RI J K J U MOLK J RI J K J Y Y Y Y _ _Va Va_ Va Va VaY VaY Va Va Va Va Va Va Va Va Va Va Va Va Va Va Va Y__ oI J w Ry R MO syy J RRRI J C C U MOU MO oI J G C VaVZZ VaVZg VaVV VaY_Z VaVZ Fig. 5: Visual Attention Ratio analysis under different masking conditions. Left: Distribution of image-level mean VAR across masking settings. Masking in the focus phase(Phase 2) yields a significantly higher VAR compared to the baseline (No masking) (p < 0.001). Right: Layer-head VAR heatmaps showing increased visual attention in intermediate layers when DPP masking is applied. scriptive sentences per image, and the outputs were compared against the GT captions. Without masking, the baseline model frequently generated hallucinated state- ments describing non-existent objects or relationships. Masking in the diffusion or rediffusion phases produced inconsistent outcomes, often preserving existing hallucinations or introducing new ones. In contrast, masking applied to the fo- cus phase consistently replaced hallucinated statements with descriptions that aligned more closely with the visual content. These observations suggest that token modulation during the focus phase contributes to more reliable visual grounding at the sentence level. 4.2 Language Bias and Visual Attention Ratio (VAR) Although Sec. 4.1 shows that masking during the focus phase correlates with reduced hallucinations, it remains unclear how this intervention affects the in- teraction between the vision encoder and the language model. To investigate this relationship, we analyze the VAR, which measures the extent to which generated tokens attend to visual inputs during decoding. Definition of VAR VAR is defined as the total attention weight assigned to visual tokens by a generated token y k at language model layer l and head h: VAR (l,h) (y k ) = N X i=1 A (l,h) k (a k,i ),(6) where A (l,h) k (a k,i ) denotes the attention score assigned by the generated token to visual token a k,i . A higher VAR indicates stronger reliance on visual information during text generation, whereas a lower VAR suggests that the language model relies more heavily on language priors. 10Kim et al. VAR Dynamics under Phase-Specific Masking To evaluate how masking influences the language model’s use of visual context, we compared the distribu- tion of image-level mean VAR values under different phase conditions (Fig. 5). The results show that masking applied during the focus phase produces a statistically significant increase in the mean VAR compared to the baseline. In contrast, interventions applied during the diffusion or rediffusion phases result in only marginal changes. Layer-wise VAR heatmaps further reveal increased visual attention in in- termediate layers of the language model when focus phase masking is applied. These observations suggest that suppressing low-attention tokens during the fo- cus phase encourages the language model to reference visual tokens more strongly during decoding. Together with the hallucination results in Sec. 4.1, these find- ings indicate that phase-specific token modulation in the vision encoder can influence both hallucination behavior and the downstream utilization of visual information. 5 Experiments Based on the analysis of attention dynamics presented in Sec. 3 and the phase- specific masking study in Sec. 4, we evaluate the effectiveness and generality of the proposed focus-phase token masking strategy across a variety of LVLM ar- chitectures. Our experiments aim to answer three key questions: (1) whether the proposed phase-aware masking consistently reduces hallucinations across differ- ent models, (2) whether the method is compatible with existing hallucination mitigation techniques, and (3) whether it introduces negligible computational overhead compared with prior approaches such as AUE. 5.1 Experimental Setup Models and Baselines We evaluate the proposed method on LVLMs with di- verse architectures and scales. As primary baselines, we employ LLaVA-1.5-7B and LLaVA-1.5-13B [13], both of which adopt the CLIP-L/336px vision en- coder [18]. We further include Shikra-7B [2], which also utilizes the CLIP-L backbone but differs in the visual-language alignment mechanism. To examine the scalability of the proposed approach to recent LVLM architectures, we addi- tionally evaluate performance on Qwen-2.5-VL [1] and Intern-VL-2.5 [3]. These models are evaluated under standard greedy decoding to measure the direct impact of the proposed vision encoder intervention. To demonstrate the compatibility of our approach with existing hallucina- tion mitigation techniques, we combine our method with several representative decoding-time approaches, including OPERA [8], VCD [10], PAI [15], and Dev- ils [9]. These methods address hallucination primarily at the language model stage, allowing us to evaluate whether the proposed vision encoder intervention provides complementary improvements. Finally, we include Adversarial Uncer- tainty Estimation (AUE) [20] as a direct baseline for comparison. Unlike our Focus Matters11 approach, AUE identifies unreliable tokens using iterative PGD-based adversar- ial perturbations [17]. This comparison allows us to evaluate whether similar hallucination mitigation can be achieved without the additional optimization overhead required by AUE. DPP-Based Token Selection and Mask Configuration The masking in- tervention is applied to the focus phase identified in Sec. 3. To select tokens for suppression, we construct a DPP kernel that accounts for both token importance and token redundancy. Token importance is estimated using the mean attention score computed from the diffusion-to-focus transition layers (layers 7–11), which immediately precede the focus phase. To model token similarity and redundancy, we com- pute cosine similarity between visual token embeddings extracted from layer 11. These two components jointly define the DPP kernel, enabling the selection of a diverse subset of tokens that captures important visual information while reduc- ing redundancy. During inference, only the selected tokens are retained while the remaining tokens are suppressed through hard masking. The masking operation is applied to the focus phase layers (layers 12–18), where attention concentration is highest according to the analysis in Sec. 3. The masking ratio is configured per model to reflect differences in token characteristics and architecture: 60% for LLaVA models, 35% for Shikra-7B, 65% for Qwen2.5-VL, and 40% for Intern-VL-2.5 Benchmarks To evaluate hallucination behavior, we employ two widely used benchmarks: CHAIR [19] and POPE [11]. CHAIR measures hallucinations at both the sentence level (C S := CHAIR S and instance level (C I := CHAIR I )) using captions generated for 500 randomly sampled images from the COCO dataset. These metrics quantify the frequency with which models generate descriptions containing objects that are not present in the input image: CHAIR S = |sentences containing hallucinated objects| |all sentences| , CHAIR I = |hallucinated objects| |all mentioned objects| . (7) POPE evaluates hallucinations using binary queries that ask whether spe- cific objects are present in the image. The benchmark contains three evaluation splits (Random, Popular, and Adversarial), totaling 9,000 prompts, and reports classification accuracy for object existence queries. 5.2 Experimental results Quantitative Results. Table 1 summarizes the results on CHAIR and POPE across multiple LVLM backbones and decoding strategies, comparing the original models (Orig.), the iterative AUE baseline, and our focus-phase masking method. 12Kim et al. GreedyOPERAVCDPAIDevils ModelBench Metric Orig. AUE Ours (∆) Orig. AUE Ours (∆) Orig. AUE Ours (∆) Orig. AUE Ours (∆) Orig. AUE Ours (∆) LLaVA-1.5-7b CHAIR CHAIR S ↓ 45.0 30.228.8(-1.4) 44.2 29.831.6(+1.8) 46.2 34.235.6(+1.4) 32.4 19.417.4(-2.0) 23.2 13.012.8(-0.2) CHAIR I ↓ 13.3 10.410.2(-0.2) 13.5 11.211.2(0.0) 14.9 11.812.5(+0.7) 9.6 7.66.5(-1.1) 7.9 7.26.3(-0.9) F1↑74.2 71.872.0(+0.2) 74.4 72.371.4(-0.9) 71.3 70.570.4(-0.1) 74.6 67.768.0(+0.3) 72.0 63.565.1(+1.6) POPE ran.↑87.4 85.6 86.4(+0.8) 86.3 84.585.2(+0.7) 83.7 78.980.6(+1.7) 87.4 85.886.2(+0.4) 87.7 84.585.4(+0.9) pop.↑84.3 83.083.1(+0.1) 83.3 81.981.5(-0.4) 80.9 77.177.5(+0.4) 84.5 83.283.0(-0.2) 85.3 83.083.2(+0.2) adv.↑79.3 78.478.5(+0.1) 79.3 77.977.5(-0.4) 77.3 73.374.7(+1.4) 79.7 78.878.5(-0.3) 79.9 78.778.9(+0.2) LLaVA-1.5-13b CHAIR CHAIR S ↓ 41.0 29.829.0(-0.8) 37.8 27.826.4(-1.4) 48.8 32.235.4(+3.2) 33.8 21.024.0(+3.0) 25.0 17.018.8(+1.8) CHAIR I ↓ 11.8 9.8 10.2(+0.4) 11.5 10.79.6(-1.1) 13.5 11.311.7(+0.4) 10.7 7.587.8(+0.22) 7.2 7.46.9(-0.5) F1↑75.3 71.472.5(+1.1) 75.0 71.573.1(+1.6) 73.1 71.070.7(-0.3) 75.2 70.870.8(0.0) 73.9 66.369.0(+2.7) POPE ran.↑86.6 82.483.5(+1.1) 86.5 78.180.4(+2.3) 81.8 75.377.7(+2.4) 87.2 82.983.8(+0.9) 87.9 82.085.5(+3.5) pop.↑83.9 82.482.3(-0.1) 83.4 79.779.8(+0.1) 80.0 76.776.9(+0.2) 84.1 82.882.5(-0.3) 85.8 81.083.3(+2.3) adv.↑79.5 76.577.3(+0.8) 79.4 75.176.0(+0.9) 77.2 73.174.0(+0.9) 79.8 76.877.6(+0.8) 81.5 76.178.3(+2.2) Shikra-7b CHAIR CHAIR S ↓ 52.4 47.244.6(-2.6) 35.2 28.831.6(+2.8) 54.6 48.848.2(-0.6) 34.0 22.623.2(+0.6) 27.8 20.821.8(+1.0) CHAIR I ↓ 16.1 14.514.1(-0.4) 12.8 10.712.6(+1.9) 16.8 16.215.8(-0.4) 9.9 7.67.2(-0.4) 10.7 10.510.3(-0.2) F1↑71.5 72.270.7(-1.5) 70.0 68.866.5(-2.3) 70.9 70.768.2(-2.5) 72.1 69.767.5(-2.2) 70.0 66.767.0(+0.3) POPE ran.↑81.4 81.0 81.2(+0.2) 82.5 81.281.8(+0.6) 79.5 78.079.5(+1.5) 81.2 79.178.6(-0.5) 80.5 78.578.4(-0.1) pop.↑80.7 79.879.5(-0.3) 80.5 78.679.2(+0.6) 78.2 77.575.7(-1.8) 80.7 78.077.3(-0.7) 77.9 75.175.7(+0.6) adv.↑77.1 76.176.4(+0.3) 77.4 75.876.5(+0.7) 76.2 74.674.1(-0.5) 77.0 75.074.5(-0.5) 75.5 73.374.4(+1.1) Table 1: Results on CHAIR and POPE benchmark. Comparison of Origin (baseline), AUE, and Ours across decoding strategies. Numbers in parentheses in the ours columns denote the AUE-relative delta ∆ = ours− AUE. For CHAIR S /CHAIR I (lower is better), negative ∆ indicates improvement and is highlighted in blue. For F1 and POPE (higher is better), positive ∆ indicates improvement and is highlighted in blue. LLaVA-1.5-7B Vision encoder time Total time 4 6 Time(s) 3 2 Time(s) 4 2 1 Orign. 0 AUE Ours Orign. 0 AUE Ours Shikra-7B Vision encoder time Total time 4 6 Time(s) 3 2 Time(s) 4 2 1 Orign. 0 AUE Ours Orign. 0 AUE Ours Qwen-2.5-VLIntern-VL-2.5 Vision encoder time Vision encoder time Total timeTotal time 4040 3030 30 Time(s) 20 10 30 Time(s) 20 10 Time(s) 20 10 Time(s) 20 10 Orign. 0 AUE Ours Orign. 0 AUE Ours Orign. 0 AUE Ours Orign. 0 AUE Ours Fig. 6: Per-sample latency (seconds). Vision encoder time vs total time for Orig./AUE/Ours. Qwen-2.5-VLIntern-VL-2.5 Bench MetricOrig. AUE OursOrig. AUE Ours CHAIR CHAIR S ↓31.4 27.827.828.6 28.427.0 CHAIR I ↓7.6 7.47.37.1 7.06.8 F1↑75.5 75.874.776.3 76.676.1 POPE ran.↑81.3 79.680.394.27 94.1794.13 pop.↑80.8 79.279.988.68 88.5388.65 adv.↑80.5 78.779.689.60 85.7385.43 Table 2: Greedy decoding results (CHAIR/POPE). Overall, the proposed method achieves consistent reductions in CHAIR metrics across models, while maintaining compet- itive F1 and POPE performance. Under greedy decoding, our method substantially reduces hallucinations on LLaVA-1.5-7B (CHAIR S : 45.0 → 28.8; CHAIR I : 13.3 → 10.2) and shows similar trends on other backbones. Moreover, when combined with decoder-stage mitigation methods such as PAI and Devils, we observe additional reductions in CHAIR in several settings, suggesting that a vision-encoder intervention can be complementary to language-side approaches rather than redundant. We also report results on recent high-resolution LVLMs (Qwen-2.5-VL and Intern-VL-2.5) in Table 2, where our method improves or matches hallucination metrics relative to both Orig. and AUE under greedy decoding. Focus Matters13 S hikra+AU LLaVA+AU S hikr LLaV Please describe the image in detail Please describe the image in detail S hikra+OurLLaVA+Our The image features a young child, likely a toddler, holding a black umbrella with a red handle. The child is standing on a sidewalk, with the umbrella providing protection from the rain. In the background, there is a truck parked further away from the child. The scene also includes several potted plants, with one plant placed close to the child and two others situated further away. The potted plants add a touch of greenery to the environment The image features a man wearing a red jacket and a black hat, talking on his cell phone while standing in a crowded city street. He appears to be engaged in a conversation, possibly discussing his whereabouts or sharing information with someone. There are several other people in the scene, some of them carrying backpacks and handbags. The backpacks are visible in various positions, with one near the center of the image, another towards the left side, and the third one closer to the right side. The handbags are also scattered throughout the scene, with one near the center, another on the left side, and the third one on the right side. The busy street is filled with people, creating a lively atmosphere in the city The image features a young child, likely a toddler, holding a black umbrella with a red handle. The child is standing on a sidewalk, with the umbrella shielding them from the rain. In the background, there is a truck parked, which might be the one that has brought the rain. Additionally, there are two potted plants in the scene, one located near the left edge of the image and the other towards the right side. These plants add some greenery to the environment, creating a pleasant atmosphere for the little girl under the umbrella The image features a man wearing a red jacket and a black hat, talking on his cell phone. He is surrounded by a crowd of people, some of whom are also wearing hats. The scene appears to be a busy street with many individuals going about their day. There are several traffic lights visible in the background, indicating that the location is likely an urban area with a well-regulated traffic system. The man with the cell phone is standing out from the crowd, as he is the only one actively engaged in a conversation The image is a blurry photograph of a young child standing on a sidewalk, holding a black umbrella. The child appears to be looking at something or someone, possibly a camera capturing the scene. The umbrella is open, providing a clear view of the child's face. In the background, there is a truck parked on the street, which might suggest that the child is in a public area or near a parking lot. The overall scene gives a sense of curiosity and playfulness, as the child is likely enjoying their time outdoors The image features a man wearing a red jacket and a black hat, talking on his cell phone. He is surrounded by a crowd of people, some of whom are also wearing red jackets. The man appears to be the main focus of the scene, as he is the only one holding a cell phone. The other people in the image are engaged in various activities, such as walking or standing around. The scene is bustling with activity, and the man's red jacket stands out against the crowd Fig. 7: Qualitative examples on the CHAIR dataset.Our method reduces hal- lucinated object mentions compared to vanilla LVLMs and AUE. To make the comparison with AUE explicit, we report the AUE-relative delta ∆ = Ours−AUE in parentheses in Table 1. Negative ∆ indicates improvement for CHAIR S /CHAIR I , while positive ∆ indicates improvement for F1 and POPE. Across most settings, our method achieves hallucination mitigation comparable to AUE, and in several cases further improves CHAIR while preserving caption fidelity. Inference Efficiency. While our method and AUE often yield similar hal- lucination mitigation performance, they differ fundamentally in computational cost. AUE relies on iterative PGD-based optimization to identify uncertain to- kens, requiring multiple forward/backward passes per image. In contrast, our method constructs the masking decision using statistics from a single forward pass, avoiding iterative gradient computations. Figure 6 compares per-sample latency by reporting the vision encoder time (V isionencoder time ) and total inference time (Total time ) for Orig./AUE/Ours. To highlight practical overhead, we compare the latency increase of each method relative to the original baseline. Across all evaluated backbones, AUE introduces a large overhead due to iterative PGD optimization, and this effect becomes particularly severe for high-resolution models. For example, on Qwen-2.5-VL, AUE increases V isionencoder time from 0.074s to 31.24s (a +31.17s overhead) and Total time from 2.60s to 33.46s (a +30.86s overhead). In contrast, our method increases V isionencoder time to 0.286s (a +0.212s overhead) and Total time to 2.81s (a +0.21s overhead), remaining close to the original baseline. Similar trends are observed on LLaVA-1.5-7B, Shikra-7B, and Intern-VL-2.5, indicating that our approach achieves hallucination mitigation with negligible additional latency overhead compared to AUE. Qualitative Results Figure. 7 presents qualitative examples on the CHAIR dataset. Vanilla LVLMs under greedy decoding often hallucinate objects that are absent from the input image. AUE alleviates some hallucinations, but spurious object mentions still appear in certain cases. In contrast, our method tends to reduce such hallucinated mentions and produces descriptions that align more closely with the visual content. 14Kim et al. 5.3 Spatial Analysis of Mask Patterns OURSAUE #Masked token num: 231#Masked token num: 261 OURSAUE #Masked token num: 347#Masked token num: 321 Fig. 8: Comparison of masking patterns between AUE and our DPP-based method. In this section, we compare the spatial char- acteristics of DPP-based masking and PGD- based AUE masking. The proposed method selects tokens by jointly considering impor- tance and diversity while fixing the num- ber of retained tokens, whereas AUE removes tokens using a threshold on an uncertainty map, which can lead to varying retained to- ken counts. For fair comparison, we visualize the masks under similar retained-token bud- gets. Both methods show similar selection ten- dencies around object boundaries or high- contrast regions, indicating that they capture salient visual cues. However, AUE produces more continuous and dense removal patterns, while DPP masking selects tokens in a more spatially dispersed manner, sup- pressing redundancy while preserving diverse visual patterns. 6 Conclusion In this paper, we revisited the object hallucination problem in LVLMs from the perspective of attention dynamics in vision encoders. Through layer-wise analysis, we identified a consistent three-phase structure of visual information processing—diffusion, focus, and rediffusion—that appears across diverse LVLM architectures. Our controlled experiments revealed that hallucination behavior is particularly sensitive to tokens receiving low attention during the focus phase. Suppressing the influence of these tokens consistently reduces hallucination met- rics, while amplifying them leads to the opposite effect. This observation provides a new perspective on how internal attention dynamics within vision encoders in- fluence hallucination in multimodal generation. Motivated by this insight, we propose a lightweight inference-time method that selectively suppresses low- attention tokens during the focus phase using DPP-based selection, relying only on statistics from a single forward pass without iterative optimization. Extensive experiments across multiple LVLM backbones and decoding strategies demon- strated that the proposed approach achieves hallucination mitigation compa- rable to existing methods while introducing negligible computational overhead. These results suggest that phase-aware token modulation in vision encoders of- fers a practical direction for improving the reliability of multimodal generation systems. Focus Matters15 References 1. Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 (2025) 2. Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., Zhao, R.: Shikra: Unleash- ing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195 (2023) 3. Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al.: Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 (2024) 4. Chiang, W.L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., et al.: Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) 2(3), 6 (2023) 5. Chuang, Y.S., Xie, Y., Luo, H., Kim, Y., Glass, J.R., He, P.: Dola: Decoding by contrasting layers improves factuality in large language models. In: The Twelfth International Conference on Learning Representations (2024) 6. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. The Ninth International Conference on Learning Representations (2021) 7. Fu, Y., Xie, R., Sun, X., Kang, Z., Li, X.: Mitigating hallucination in multimodal large language model via hallucination-targeted direct preference optimization. In: Findings of the Association for Computational Linguistics: ACL 2025. p. 16563– 16577 (2025) 8. Huang, Q., Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., Yu, N.: Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 13418–13427 (2024) 9. Jiang, Z., Chen, J., Zhu, B., Luo, T., Shen, Y., Yang, X.: Devils in middle lay- ers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 25004–25014 (2025) 10. Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., Bing, L.: Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 13872–13882 (2024) 11. Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, X., Wen, J.R.: Evaluating object hallu- cination in large vision-language models. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. p. 292–305. Association for Computational Linguistics (2023) 12. Liu, H., Xue, W., Chen, Y., Chen, D., Zhao, X., Wang, K., Hou, L., Li, R., Peng, W.: A survey on hallucination in large vision-language models (2024) 13. Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning (2023) 14. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: NeurIPS (2023) 16Kim et al. 15. Liu, S., Zheng, K., Chen, W.: Paying more attention to image: A training-free method for alleviating hallucination in lvlms. In: European Conference on Com- puter Vision. p. 125–140. Springer (2024) 16. Macchi, O.: The coincidence approach to stochastic point processes. Advances in Applied Probability 7(1), 83–122 (1975) 17. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. The Sixth International Conference on Learning Representations (2017) 18. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transfer- able visual models from natural language supervision. In: Proceedings of the 38th International Conference on Machine Learning. p. 8748–8763. PMLR (2021) 19. Rohrbach, A., Hendricks, L.A., Burns, K., Darrell, T., Saenko, K.: Object halluci- nation in image captioning. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. p. 4035–4045 (2018) 20. Seo, H., Kang, D.U., Cho, H., Lee, J., Chun, S.Y.: On epistemic uncertainty of visual tokens for object hallucinations in large vision-language models. Advances in neural information processing systems (2025) 21. Sun, Z., Shen, S., Cao, S., Liu, H., Li, C., Shen, Y., Gan, C., Gui, L., Wang, Y.X., Yang, Y., et al.: Aligning large multimodal models with factually augmented rlhf. In: Findings of the Association for Computational Linguistics: ACL 2024. p. 13088–13110 (2024) 22. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023) 23. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bash- lykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023) 24. Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., Xie, Z., Wu, Y., Hu, K., Wang, J., Sun, Y., Li, Y., Piao, Y., Guan, K., Liu, A., Xie, X., You, Y., Dong, K., Yu, X., Zhang, H., Zhao, L., Wang, Y., Ruan, C.: Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding (2024) 25. Xie, Y., Li, G., Xu, X., Kan, M.Y.: V-dpo: Mitigating hallucination in large vision language models via vision-guided direct preference optimization. In: Findings of the Association for Computational Linguistics: EMNLP 2024. p. 13258–13273 (2024) 26. Yang, Z., Luo, X., Han, D., Xu, Y., Li, D.: Mitigating hallucinations in large vision-language models via dpo: On-policy data hold the key. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 10610– 10620 (2025) 27. Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language im- age pre-training. In: Proceedings of the IEEE/CVF international conference on computer vision. p. 11975–11986 (2023) 28. Zhao, Z., Wang, B., Ouyang, L., Dong, X., Wang, J., He, C.: Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839 (2023) Focus Matters17 Organization of the Supplementary This supplementary material provides additional technical details, extended anal- yses, and qualitative results that complement the main manuscript. In particu- lar, it includes further explanations of the proposed phase-aware token suppres- sion framework, additional empirical evidence supporting the observations in the main paper, and implementation details related to token selection and efficiency. The document is organized as follows: – Appendix A describes the formulation and empirical procedure used to determine the phase boundaries within vision encoders. – Appendix B provides additional analysis of continuous token modula- tion, showing how hallucination behavior changes when the influence of low- attention tokens is adjusted across phases. – Appendix C extends the Visual Attention Ratio (VAR) analysis to multiple LLM backbones, examining the consistency of the observed trends across architectures. – Appendix D presents the formulation of the Determinantal Point Process (DPP) kernel and the efficient greedy MAP inference procedure used in our method. – Appendix E describes the Ground-Truth (GT) caption-based evaluation protocol used for sentence-level qualitative analysis. – Appendix F provides extended qualitative examples on the CHAIR and POPE datasets, together with visual comparisons of spatial masking pat- terns. – Appendix G reports quantitative results on additional hallucination bench- mark. A Phase Boundary Determination In Sec. 3 of the main manuscript, we empirically observed a consistent three- phase visual processing structure (Diffusion, Focus, and Rediffusion) across mul- tiple LVLM vision encoders. To clarify how the focus phase boundaries used in our experiments are identified, we describe the procedure for determining the interval F = [l start ,l end ] based on the dynamics of the attention concentration ratio. The concentration ratio at layer l is defined as R (l) = M (l) H (l) , (8) where M (l) denotes the maximum attention score and H (l) denotes the attention entropy at layer l. This ratio increases when attention becomes concentrated on a small subset of tokens and decreases when attention is more evenly distributed. 18Kim et al. A.1 Onset of the Focus Phase (l start ) During the initial Diffusion Phase, attention is broadly distributed and the layer- wise change of the concentration metric remains relatively small. To identify the transition into the Focus Phase, we examine the discrete forward difference ∆R (l) = R (l) − R (l−1) .(9) The start layer l start is determined as the first layer where the increase in concentration exceeds the baseline variation observed in the early layers: l start = min n l| ∆R (l) > μ base + λ· σ base o ,(10) where μ base and σ base denote the mean and standard deviation of ∆R (l) com- puted from the early diffusion layers (e.g., the first 25% of the network depth), and λ is a confidence multiplier. This criterion identifies the layer where attention concentration begins to increase beyond the relatively stable baseline behavior of the diffusion stage. A.2 Offset of the Focus Phase (l end ) Empirically, the Focus Phase corresponds to the main region where the concen- tration ratio remains high after the initial transition. Across the evaluated LVLM architectures (with depths ranging from 24 to 32 layers), this stage typically oc- cupies a contiguous block covering roughly 30% ∼ 40% of the total network depth L. Rather than enforcing a fixed length, we determine the window length K relative to the shape of the concentration curve following l start . Models exhibiting a sharp concentration peak tend to require a shorter window (approximately 30% of L), whereas models with broader concentration plateaus require slightly longer windows (closer to 40% of L) to capture the main high-concentration region. The end layer is therefore determined as l end = l start + K− 1.(11) This procedure captures the primary region of concentrated attention before the concentration metric begins to decrease again during the Rediffusion Phase. A.3 Phase Boundaries Across Evaluated Models Applying the above procedure provides a consistent rationale for the layer in- tervals used in the main experiments without requiring model-specific hyperpa- rameter tuning as visualized for each model in Fig. 9. LLaVA-1.5 & Shikra (L = 24). The gradient ∆R (l) remains close to zero in early layers and exhibits a clear increase around layer 11. Setting l start = 11 and using a window length of K = 7 (≈ 29% of L) captures the main concentration region before the metric declines, yielding Layers 11–17. Focus Matters19 11111111 Layer IndeLayer IndeLayer IndeLayer Inde 222223 LLaVShikrQwen-2.5-V InternVL-2. Max score / EntropMax score / EntropMax score / EntropMax score / Entrop Fig. 9: Layer-wise attention dynamics across evaluated LVLM backbones. The solid lines indicate the ratio of maximum attention score to entropy (R (l) ). The shaded regions represent the identified Diffusion (yellow), Focus (red), and Rediffusion (purple) phases based on our gradient-based boundary formulation. Qwen-2.5-VL (L = 32). The concentration gradient exceeds the early base- line around layer 17 and continues to increase. Using K = 10 (≈ 31% of L) captures the dominant concentration peaks around layers 24 and 26 before the metric decreases near layer 27, resulting in Layers 17–26. InternVL-2.5 (L = 24). The first consistent increase in ∆R (l) occurs around layer 9. Using the proportional window K = 7 (≈ 29% of L) isolates the pri- mary concentration region, yielding Layers 9–15. Although a secondary spike occasionally appears in the final layers (e.g., in InternVL-2.5), this peak typi- cally corresponds to late-stage representation aggregation rather than the pri- mary attention concentration stage. Therefore, our formulation focuses on the first dominant concentration peak, which consistently marks the transition into the focus phase. B In-Depth Analysis of Token Modulation in the Focus Phase To further examine the observations reported in Section 4.1 regarding the sen- sitivity of hallucination metrics to token influence within the focus phase, we conducted continuous token modulation experiments. Instead of applying bi- nary masks alone, we systematically modulated the influence of low-attention tokens by directly shifting their pre-softmax attention logits. As illustrated in Fig. 10, increasing the logit values (i.e., boosting the in- fluence) of these low-attention tokens tends to amplify hallucination behavior. When their influence is strengthened, the models increasingly generate descrip- tions containing objects that are not present in the image (highlighted in red), and in some cases the generated captions deviate from the actual visual content. Conversely, subtracting from these logits (i.e., suppressing their influence) pro- gressively reduces the occurrence of hallucinated entities, producing descriptions that are more consistent with the visual evidence. Fig. 11 further presents additional qualitative examples demonstrating the same trend across multiple scenes. Across diverse images, strengthening the in- fluence of low-attention tokens consistently introduces hallucinated objects or re- lationships, whereas suppressing their influence leads to more visually grounded 20Kim et al. P has e 2 P has e 2 P has e 2 Or igin . Or igin . Or igin . De cre a si n g i n fl ue n ce of l ow-a tte n ti on toke n In cre a si n g i n fl ue n ce of l ow-a tte n ti on toke n Pl e a se de scr i be the i m a ge i n de ta i l Pl e a se de scr i be the i m a ge i n de ta i l Pl e a se de scr i be the i m a ge i n de ta i l The image feat ures a cit ys cape w ith a large ind us t rial b uild ing in the b ackgro und , s urro und ed by a fo res t o f s kys crapers . The ind us t rial b uild ing is a pro minent feat ure in the s cene, towering over the o ther b uild ings . The cit ys cape is b us tling w ith act ivit y, as there are s everal cars and t rucks s cat tered thro ugho ut the area. In ad d it io n to the vehicles , there are t wo peo ple vis ib le in the s cene, o ne near the center and ano ther tow ard s the right s id e. The pres ence o f thes e ind ivid uals ad d s a s ens e o f life to the o therw is e ind us t rial s et t ing The image d epicts a co zy living ro o m w ith a large co uch and a chair placed in the center o f the ro o m. The co uch is po s it io ned agains t the w all, w hile the chair is s it uated in the mid d le o f the ro o m. A co ffee t ab le is lo cated in fro nt o f the co uch, and a d ining t ab le can b e s een nearby. There are s everal po t ted plants in the ro o m, w ith o ne placed near the co uch, ano ther near the d ining t ab le, and a third o ne clo s er to the chair. A vas e is als o pres ent in the ro o m, ad d ing to the d eco rat io n. In ad d it io n to the furnit ure, there are t wo cups placed o n the co ffee t ab le, and a b ow l can b e s een o n the d ining t ab le. A clo ck is mo unted o n the w all, and a b o o k is res t ing o n the co uch The image feat ures a vas e filled w ith flowers placed o n a t ab le. The vas e is po s it io ned in the center o f the s cene, and the flowers are arranged in a vis ually appealing manner. The t ab le appears to b e a d ining t ab le, and the vas e is the main fo cus o f the s cene. In the b ackgro und , there is a chair lo cated o n the left s id e o f the image. Ad d it io nally, there are t wo cups placed o n the t ab le, o ne near the left ed ge and the o ther clo s er to the center. The overall s et t ing s ugges ts a co zy and invit ing at mo s phere The image feat ures a cit ys cape w ith a large ind us t rial b uild ing in the b ackgro und , s urro und ed by a fo res t o f t rees . The b uild ing is a pro minent feat ure in the s cene, towering over the s urro und ing area. The cit ys cape als o includ es a clo ck tower, ad d ing to the urb an at mo s phere. In the fo regro und , there is a pers o n s t and ing near the center o f the image, po s s ib ly ad miring the view o r go ing ab o ut their d aily act ivit ies . The co mb inat io n o f the ind us t rial b uild ing, the clo ck tower, and the t rees creates a unique and interes t ing b lend o f urb an and nat ural elements The image d epicts a co zy living ro o m w ith a large co uch and a chair placed in the center o f the ro o m. The co uch is po s it io ned agains t the w all, w hile the chair is s it uated in the mid d le o f the ro o m. A co f fee t ab le is lo cated in fro nt o f the co uch, and a d ining t ab le can b e s een nearby. There are s everal po t ted plants in the ro o m, ad d ing a to uch o f greenery to the s pace. One plant is placed o n the lef t s id e o f the ro o m, ano ther is lo cated near the center, and the third o ne is s it uated o n the right s id e. A vas e is als o pres ent in the ro o m, placed o n the left s id e o f the co uch. In ad d it io n to the f urnit ure, there are t wo b o o ks in the ro o m, o ne o n the left s id e and the o ther o n the right s id e o f the co uch. A clo ck is mo unted o n the w all, and a remo te co nt ro l can b e s een o n the right s id e o f the co uch The image feat ures a vas e filled w ith flowers placed o n a t ab le. The vas e is po s it io ned in the center o f the s cene, and the flowers are arranged in a vis ually appealing manner. The t ab le appears to b e a s mall ro und t ab le, and the vas e is the main fo cus o f the s cene. In the b ackgro und , there is a chair lo cated o n the lef t s id e o f the image. Ad d it io nally, there are t wo cups placed o n the t ab le, o ne near the left ed ge and the o ther clo s er to the center. A po t ted plant can als o b e s een in the b ackgro und , ad d ing a to uch o f greenery to the s cene The image feat ures a large ind us t rial co mplex w ith s everal t all b uild ings , includ ing a clo ck tower, s it uated in a cit y. The b uild ings are s urro und ed by t rees , creat ing a pleas ant co nt ras t b et ween the urb an enviro nment and the nat ural land s cape. In the fo regro und , there is a clo ck mo unted o n o ne o f the b uild ings , w hich is vis ib le f ro m a d is t ance. The clo ck is po s it io ned near the center o f the s cene, making it a pro minent feat ure. The overall at mo s phere o f the image is that o f a b us tling cit y w ith a to uch o f nat ure, as the t rees provid e a s erene b ackd ro p to the ind us t rial co mplex The image d epicts a co zy living ro o m w ith a large co uch and a chair placed in the center o f the ro o m. The co uch is po s it io ned agains t the w all, w hile the chair is s it uated in the mid d le o f the ro o m. A co f fee t ab le is lo cated in f ro nt o f the co uch, and a d ining t ab le can b e s een nearby. There are s everal po t ted plants s cat tered thro ugho ut the ro o m, ad d ing a to uch o f greenery to the s pace. A vas e is als o pres ent , placed o n the d ining t ab le. A cat is res t ing o n the co uch, making the ro o m feel even mo re invit ing and co mfo rt ab le The image feat ures a vas e filled w ith aut umn leaves , placed o n a t ab le. The vas e is po s it io ned in the center o f the s cene, and the leaves ins id e it are arranged in a vis ually appealing manner. The t ab le appears to b e a d ining t ab le, and the vas e is the main fo cus o f the s cene. In the b ackgro und , there are t wo cups placed o n the t ab le, o ne clo s er to the lef t s id e and the o ther near the right s id e. Ad d it io nally, there is a b ow l lo cated near the lef t ed ge o f the t ab le The image feat ures a large ind us t rial co mplex w ith s everal t all b uild ings , includ ing a clo ck tower, s it uated in a cit y. The b uild ings are s urro und ed by t rees , creat ing a pleas ant co nt ras t b et ween the urb an enviro nment and the nat ural land s cape.There are mult iple clo cks vis ib le o n the b uild ings , ind icat ing the impo rt ance o f t imekeeping in this area. The clo cks are lo cated at vario us heights and po s it io ns , ad d ing to the overall architect ural interes t o f the co mplex . The s cene is capt ured in a s lightly b lurry manner, giving it a s lightly art is t ic appearance The image d epicts a co zy living ro o m w ith a large co uch and a chair placed in the center o f the ro o m. The co uch is po s it io ned agains t the w all, w hile the chair is s it uated in the mid d le o f the ro o m. A co f fee t ab le is lo cated in f ro nt o f the co uch, and a d ining t ab le can b e s een nearby. There are s everal po t ted plants s cat tered thro ugho ut the ro o m, ad d ing a to uch o f greenery to the s pace. A vas e is als o pres ent , placed o n the co ffee t ab le. A cat is res t ing o n the co uch, making the ro o m feel even mo re invit ing and co mfo rt ab le The image feat ures a vas e filled w ith a variet y o f aut umn leaves , creat ing a b eaut if ul and co lo rful d is play. The vas e is placed o n a t ab le, w hich is po s it io ned in fro nt o f a mirro r. The leaves in the vas e are arranged in a w ay that s howcas es their nat ural b eaut y, making the arrangement vis ually appealing. The co mb inat io n o f the vas e, leaves , and mirro r creates a co zy and invit ing at mo s phere in the ro o m The image feat ures a cit ys cape w ith a large ind us t rial b uild ing in the b ackgro und . The b uild ing is s urro und ed by s everal t all, s kinny towers , w hich are likely part o f the ind us t rial co mplex. The s ky ab ove the s cene is b lue, creat ing a pleas ant at mo s phere. In the fo regro und , there are t rees that ad d a to uch o f greenery to the urb an land s cape. The co mb inat io n o f the ind us t rial b uild ing, towers , and t rees creates a vis ually interes t ing s cene that capt ures the es s ence o f a b us tling cit y The image feat ures a co zy living ro o m w ith a co uch and a chair placed in the center o f the ro o m. The co uch is po s it io ned o n the left s id e o f the ro o m, w hile the chair is s it uated o n the right s id e. A co ffee t ab le is lo cated in the mid d le o f the ro o m, s urro und ed by the co uch and chair. In ad d it io n to the s eat ing, there are s everal d eco rat ive elements in the ro o m. A po t ted plant is placed near the center o f the ro o m, and a vas e can b e s een o n the right s id e. A cat is als o pres ent in the s cene, s it t ing o n the co uch, ad d ing a to uch o f w armth and co mfo rt to the living s pace The image feat ures a vas e filled w ith a variet y o f aut umn leaves , creat ing a b eaut iful and co lo rful d is play. The vas e is placed o n a t ab le, and the leaves are arranged in a w ay that s howcas es their nat ural b eaut y. The leaves are o f d ifferent s izes and co lo rs , ad d ing a s ens e o f d epth and variet y to the arrangement . The overall s cene is a d elight ful repres ent at io n o f the b eaut y o f aut umn leaves De cre a si n g i n fl ue n ce of l ow-a tte n ti on toke n In cre a si n g i n fl ue n ce of l ow-a tte n ti on toke n De cre a si n g i n fl ue n ce of l ow-a tte n ti on toke n In cre a si n g i n fl ue n ce of l ow-a tte n ti on toke n Fig. 10: Qualitative examples of continuous token modulation in the focus phase. Increasing the influence of low-attention tokens introduces hallucinated objects in the generated descriptions (highlighted in red), while suppressing their influence reduces hallucination and produces captions more consistent with the visual content. descriptions. This consistent qualitative pattern supports the observation that hallucination behavior is closely related to the influence of low-attention tokens during the focus phase. Taken together, these results provide empirical evidence that tokens receiving relatively low attention in the focus phase can introduce noisy signals into the generation process when their influence is amplified. The observed monotonic response further indicates that hallucination dynamics are closely tied to token influence within this phase. C Extended Visual Attention Ratio (VAR) Analysis In Sec. 4.2, we showed that suppressing low-attention tokens during the focus phase increases the Visual Attention Ratio (VAR) [9], indicating that the lan- guage model places greater emphasis on visual inputs during text generation. To further examine whether this behavior is consistent across different language Focus Matters21 P has e 2 P has e 2 P has e 2 Or igin . Or igin . Or igin . De cre a si n g i n fl ue n ce of l ow-a tte n ti on toke n De cre a si n g i n fl ue n ce of l ow-a tte n ti on toke n De cre a si n g i n fl ue n ce of l ow-a tte n ti on toke n In cre a si n g i n fl ue n ce of l ow-a tte n ti on toke n In cre a si n g i n fl ue n ce of l ow-a tte n ti on toke n In cre a si n g i n fl ue n ce of l ow-a tte n ti on toke n Pl e a se de scr i be the i m a ge i n de ta i l Pl e a se de scr i be the i m a ge i n de ta i l Pl e a se de scr i be the i m a ge i n de ta i l The image feat ures a b o at flo at ing o n a b o d y o f w ater, w ith a pers o n s t and ing o n the b ack o f the b o at . The b o at is quite large and has a cab in o n the b ack, provid ing a co mfo rt ab le s pace fo r the pers o n to s t and . The b o at is als o equipped w ith a cano py, o ffering s had e and pro tect io n fro m the elements . In ad d it io n to the pers o n o n the b o at , there are t wo d o gs o n b o ard , o ne lo cated near the mid d le o f the b o at and the o ther clo s er to the fro nt . The d o gs s eem to b e enjoying their t ime o n the b o at , acco mpanying their ow ner o n this o ut ing The image feat ures a man wearing a s uit and t ie, s t and ing in a hallw ay. He appears to b e lo o king d ow n, po s s ib ly at his cell pho ne. The man is the main fo cus o f the s cene, w ith his s uit and t ie clearly vis ib le. In the b ackgro und , there are s everal b o o ks s cat tered aro und , po s s ib ly ind icat ing a s t ud y o r wo rk area. Ad d it io nally, there are t wo o ther peo ple in the s cene, o ne o n the left s id e and ano ther o n the right s id e, b ut they are no t the main s ub ject o f the image The image feat ures a large clo ck tower w ith a ro und clo ck face, towering over a cit y. The clo ck face is illuminated , making it s t and o ut agains t the s ky. The tower is s it uated in fro nt o f a b uild ing, and the clo ck is po s it io ned at the to p o f the tower. In the b ackgro und , there are s everal peo ple s cat tered thro ugho ut the s cene, po s s ib ly enjoying the view o r go ing ab o ut their d aily act ivit ies . The overall at mo s phere o f the image is that o f a b us tling cit y w ith a pro minent land mark The image feat ures a b o at f lo at ing o n a b o d y o f w ater, w ith a d o g s it t ing o n the b ack o f the b o at . The d o g appears to b e enjoying the rid e and is lo o king b ack at the camera. The b o at is quite large, o ccupying a s ignif icant po rt io n o f the s cene.There are t wo peo ple vis ib le in the image, o ne near the mid d le o f the b o at and ano ther clo s er to the right s id e. The b o at is equipped w ith a co uch, provid ing a co mfo rt ab le s eat ing area fo r the pas s engers . The overall at mo s phere o f the s cene is relaxed and leis urely, w ith the d o g and peo ple enjoying their t ime o n the w ater The image feat ures a man wearing a s uit and t ie, s t and ing in f ro nt o f a mirro r. He is t aking a s elf ie w ith his cell pho ne, capt uring his reflect io n in the mirro r. The man appears to b e fo cus ed o n the t as k at hand , and the ref lect io n o f the cell pho ne can b e s een in the mirro r as well. In the b ackgro und , there are s everal b o o ks s cat tered aro und , po s s ib ly ind icat ing a s t ud y o r wo rks pace. The b o o ks are o f vario us s izes and are placed at d ifferent heights , creat ing a s ens e o f d epth in the s cene The image feat ures a large clo ck tower w ith a t rain o n to p o f it , towering over a cit y. The clo ck is pro minently d is played o n the tower, making it a fo cal po int o f the s cene. The t rain is s it uated o n the to p o f the tower, ad d ing an interes t ing architect ural element to the s t ruct ure. In the b ackgro und , there are s everal peo ple s cat tered thro ugho ut the s cene, po s s ib ly enjoying the view o r go ing ab o ut their d aily act ivit ies . The co mb inat io n o f the clo ck tower, t rain, and peo ple creates a lively and b us tling at mo s phere in the cit y The image feat ures a b o at f lo at ing o n a b o d y o f w ater, w ith a d o g s it t ing in the f ro nt o f the b o at . The b o at is a s mall, w hite, and s ilver-co lo red ves s el, and it appears to b e a s peed b o at . The d o g is s it t ing o n the fro nt o f the b o at , enjoying the rid e.There are t wo peo ple vis ib le in the s cene, o ne near the mid d le o f the b o at and ano ther clo s er to the right s id e. The b o at is equipped w ith a co uch, w hich is lo cated in the mid d le o f the b o at , provid ing a co mfo rt ab le s eat ing area fo r the pas s engers The image feat ures a man wearing a s uit and t ie, s t and ing in fro nt o f a mirro r. He is ho ld ing a camera in his hand , po s s ib ly t aking a s elf ie o r capt uring a mo ment . The man appears to b e well-d res s ed and fo cus ed o n his t as k. In the b ackgro und , there are s everal b o o ks s cat tered aro und , po s s ib ly ind icat ing a s t ud y o r wo rks pace. The b o o ks are o f vario us s izes and are placed at d if ferent heights , creat ing a s ens e o f d epth in the s cene The image feat ures a large, o ld - fas hio ned t rain engine w ith a numb er 71 o n its s id e, s it t ing o n to p o f a hill. The t rain is s urro und ed by a lus h green fo res t , creat ing a pict ures que s cene. The t rain appears to b e a pro minent and eye-catching element in the image. In the b ackgro und , there are a few peo ple s cat tered aro und the area, po s s ib ly enjoying the view o r t aking a s t ro l thro ugh the fo res t . The co mb inat io n o f the t rain, the fo res t , and the peo ple creates a s erene and capt ivat ing at mo s phere The image feat ures a b o at f lo at ing o n a b o d y o f w ater, po s s ib ly a lake o r a river. The b o at is a s mall, w hite, and s ilver- co lo red ves s el w ith a b lack s t ripe running alo ng its s id e. The b o at appears to b e a mo to rb o at , and it is carrying a d o g o n b o ard . The d o g is s it t ing o n the b ack o f the b o at , enjoying the rid e.The b o at is po s it io ned in the mid d le o f the s cene, w ith the d o g b eing the main fo cus o f the image. The w ater appears calm, provid ing a s erene enviro nment fo r the b o at and its canine co mpanio n The image feat ures a man wearing a s uit and t ie, s t and ing in fro nt o f a mirro r. He is ho ld ing a camera in his hand , po s s ib ly t aking a pict ure o f hims elf o r capt uring a mo ment . The man appears to b e fo cus ed o n the camera, po s s ib ly ad jus t ing the s et t ings o r preparing to t ake a s ho t . The ref lect io n o f the man and the camera can b e s een in the mirro r, creat ing a s ens e o f d epth in the s cene The image feat ures a large, o ld - fas hio ned t rain engine w ith a numb er 71 o n its s id e, s it t ing o n a hills id e. The t rain is ad o rned w ith lights , giving it a fes t ive appearance. The t rain is po s it io ned in the center o f the s cene, w ith a b uild ing vis ib le in the b ackgro und .There are s everal peo ple s cat tered aro und the area, po s s ib ly ad miring the t rain o r enjoying the view. So me o f them are clo s er to the t rain, w hile o thers are f urther aw ay, creat ing a s ens e o f d epth in the s cene The image feat ures a b o at flo at ing o n a b o d y o f w ater, po s s ib ly a lake o r a river. The b o at is large and has a s ilver co lo r. On the b o at , there is a d o g s it t ing in the fro nt , enjoying the rid e. The d o g is po s it io ned near the center o f the b o at , lo o king o ut over the w ater. The b o at appears to b e a mo to rb o at , and the d o g s eems to b e the o nly o ccupant at the mo ment The image feat ures a man wearing a s uit and t ie, s t and ing in fro nt o f a mirro r. He is ho ld ing a camera in his hand , po s s ib ly t aking a pict ure o f hims elf o r capt uring a mo ment . The man appears to b e the main fo cus o f the s cene, w ith the camera and the mirro r reflect ing his image The image feat ures a large, o ld -fas hio ned t rain engine s it t ing o n a hills id e, s urro und ed by a fo res t . The t rain is pro minently d is played in the fo regro und , w ith its impres s ive s ize and d es ign. The s ky in the b ackgro und is a b eaut iful s had e o f b lue, ad d ing to the overall at mo s phere o f the s cene Fig. 11: Additional qualitative examples demonstrating the effect of token influence modulation across diverse scenes. Strengthening the influence of low- attention tokens tends to introduce hallucinated objects or relationships, whereas sup- pressing their influence produces descriptions that better align with the visual evidence. decoder configurations, we extend the VAR analysis to two additional LVLMs: LLaVA-1.5-13B and Shikra-7B. These models represent two distinct decoder characteristics. First, LLaVA-1.5-13B employs a larger 13B parameter language model, allowing us to examine whether the observed effect remains consistent under increased language model capacity. Second, Shikra-7B is optimized for ref- erential dialogue and spatial grounding, where the language model is trained to interpret spatial coordinates through natural language expressions. This provides a complementary setting to evaluate whether the intervention remains effective when the decoder is explicitly trained for spatial reasoning. As illustrated in Fig. 12 and Fig. 13, both models exhibit a consistent trend. Applying DPP masking during the focus phase (Phase 2) produces a noticeable increase in the image-level mean VAR compared to the no-masking baseline. In contrast, applying the same intervention during the diffusion phase (Phase 1) or the rediffusion phase (Phase 3) results in only minor changes in VAR. The heatmaps further illustrate how this shift occurs within the language model. When focus-phase masking is applied, intermediate layers allocate higher atten- 22Kim et al. &% - &% 8 &% D &% QOQO _% YTS^RSTS_% YTS^RSTS ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` RS RS ys^u^% n luuS^^^RS%O _% _% ` `` ` `gd ` `g ` ` `gd Fig. 12: VAR analysis on LLaVA-1.5-13B. Left: Distribution of image-level mean VAR across masking strategies. Focus-phase masking (Phase 2) produces a significantly higher VAR than the baseline (no masking), indicating increased reliance on visual tokens during generation. Right: Layer-head VAR heatmaps comparing DPP masking and no masking. Applying focus-phase masking increases visual attention across inter- mediate decoder layers. $ , $ 7 $ C $ PNPN ^$VTXS ]Q S ^$VTXS ]Q S b b h _j_j _j_j _j_j _j_j _j_j bbbbhh tQ tQ ]]$VTz x ]]]Q $N ^$VT^$VT _j__ _j _j _j _j _j _j _j`l_ _j`` _j eh _j` h Fig. 13: VAR analysis on Shikra-7B. Left: Distribution of image-level mean VAR under different masking phases. Focus-phase masking consistently yields higher VAR compared to the baseline. Right: Layer-head VAR heatmaps illustrating the increase in visual attention when DPP masking is applied during the focus phase. tion weights to visual tokens across multiple heads, indicating that the decoder relies more strongly on visual evidence during generation. Taken together, these results suggest that suppressing low-attention tokens during the focus phase systematically increases the model’s reliance on visual inputs across different decoder architectures and model scales. D Detailed Formulation and Efficiency of Phase-Aware DPP Masking In Section 5.1 of the main manuscript, we introduced a training-free token sup- pression strategy based on a Determinantal Point Process (DPP) [16] applied during the focus phase. This appendix provides the detailed formulation and implementation used to construct the DPP mask and apply it within the vision Focus Matters23 encoder. The procedure involves three functional components of the encoder layers: the source layers, the feature layer, and the target layers. D.1 Extraction of Token Importance and Similarity To determine which visual tokens should be retained, we estimate both the importance of individual tokens and the similarity among them. These quantities are extracted from reference layers immediately preceding the focus phase. Source Layers (Token Importance). We first estimate the structural impor- tance of each token using attention statistics from the source layers. Specifically, we extract the self-attention matrices from the multi-head attention modules of these layers and average them across heads and layers to obtain ̄ A∈ R N×N , (12) where N denotes the sequence length. The importance score of token i is then computed as the total attention it receives: q i = N X j=1 ̄ A j,i . (13) Feature Layer (Token Similarity). To measure semantic redundancy be- tween tokens, we extract the visual feature embeddings F∈ R N×C (14) from a designated feature layer preceding the focus phase. Each token feature vector is ℓ 2 -normalized ̃ f i = f i ∥f i ∥ 2 ,(15) and the cosine similarity matrix is computed as S i,j = ̃ f ⊤ i ̃ f j , (16) yielding S∈ R N×N .(17) D.2 DPP Kernel Construction Using both token importance and token similarity, we construct a positive semi- definite L-ensemble kernel L∈ R N×N ,(18) whose elements are defined as L i,j = q i · S i,j · q j .(19) Under this formulation, the diagonal entries L i,i = q 2 i (20) reflect the absolute importance of individual tokens, while the off-diagonal terms penalize the joint selection of semantically redundant tokens. 24Kim et al. OriginAUE [20]Ours ModelEnc. Time Total Enc. Time Total Enc. Time Total LLaVA-1.50.0083.207 4.588 (+4.580) 7.307 (+4.100) 0.198 (+0.190) 3.182 (-0.025) Shikra0.0063.267 4.586 (+4.580) 7.677 (+4.410) 0.188 (+0.182) 3.267 (+0.000) Qwen-2.5-VL0.0742.598 31.244 (+31.170) 33.456 (+30.858) 0.286 (+0.212) 2.813 (+0.215) InternVL2.50.0472.299 38.768 (+38.721) 41.291 (+38.992) 0.259 (+0.212) 2.722 (+0.423) Table 3: Inference time per sample (seconds). Values in parentheses indicate the change relative to the origin baseline. Our DPP-based masking introduces only a small overhead, while AUE significantly increases latency. D.3 Fast Greedy MAP Inference and Efficiency The goal is to select a subset of K tokens that maximizes the DPP posterior probability (MAP inference). Since exact MAP inference for a DPP is NP-hard, we adopt an efficient greedy approximation algorithm. Starting from an empty set, the algorithm iteratively selects the token that yields the largest marginal gain in the determinant of the corresponding sub-kernel matrix. This process is accelerated using Cholesky factorization updates, which allow efficient incremen- tal updates without recomputing determinants from scratch. Importantly, this selection process is executed using statistics obtained from a single forward pass of the vision encoder. Therefore, the mask generation does not require iterative gradient computations. The resulting inference overhead is summarized in Ta- ble 3. Compared with the baseline models, our method introduces only a small increase in vision encoder processing time, while maintaining nearly identical to- tal inference latency. In contrast, the adversarial uncertainty estimation method (AUE) requires iterative gradient-based optimization and therefore introduces substantially higher latency across all evaluated architectures. D.4 Mask Generation and Target Layer Application After selecting the subset of K tokens, we construct an additive attention mask M over the token sequence. To preserve global image semantics, the class token ([CLS]) is always retained. Selected tokens and the [CLS] token are assigned a mask value of 0, while unselected tokens receive −∞. The mask M is applied to the self-attention modules of the target layers corresponding to the focus phase. During the forward pass, the mask is added to the pre-softmax attention logits, forcing the attention weights of the masked tokens to become zero after the softmax operation. This phase-aware masking suppresses low-attention tokens during the focus phase while allowing the remaining tokens to dominate the attention distribution used for downstream generation. Focus Matters25 GT ca pt i o P has e 1 mas k in P has e 2 mas k in P has e 3 mas k in No mas k in A s milin g child is s ittin g at a k itche n co u n te r. The child is we ar in g a ye llow z ip- u p ho o die . So me do n u ts in the fo re gro u n d have co lo rf u l icin g an d s pr in k le s . Seve ral plate s o f do n u ts are ar ran ge d o n the co u n te r in the fo re gro u n d. A win dow with a wo o de n f rame is vis ible above a s in k in the backgro u n d A gir l is s ittin g at a table with a var ie ty o f do n u ts in f ro n t o f he r. The gir l is we ar in g a ye llow jacke t. The table is f ille d with dif fe re n t type s o f do n u ts , s o me o f which are f ro s te d. The re is a cu p o n the table , po s s ibly co n tain in g a beve rage to acco mpany the do n u ts . The s ce n e take s place in do o r s , like ly in a ho me o r a co z y cafe A gir l is s ittin g at a table with a var ie ty o f do n u ts in f ro n t o f he r. The gir l is we ar in g a ye llow s we ats hirt. The table is cove re d with a var ie ty o f do n u ts . The gir l is s milin g an d appe ar s to be e n joyin g the do n u ts . The table is lo cate d in a k itche n are a. The gir l appe ar s to be the main fo cu s o f the image A gir l is s ittin g at a table with a var ie ty o f do n u ts in f ro n t o f he r. The table is f ille d with dif fe re n t type s o f do n u ts , in clu din g cho co late an d s pr in k le d o n e s . The re is a bowl o n the table , po s s ibly co n tain in g mo re do n u ts o r o the r tre ats . The s ce n e take s place in a k itche n . The gir l is ho ldin g a do n u t in he r han d A little gir l is s ittin g in f ro n t o f a table with a var ie ty o f do n u ts . The gir l is we ar in g a ye llow s hirt. The table is cove re d with a var ie ty o f do n u ts . The gir l is s milin g. The gir l is po s in g fo r a pictu re GT ca pt i oGT ca pt i o P has e 1 mas k in P has e 1 mas k in P has e 2 mas k in P has e 2 mas k in P has e 3 mas k in P has e 3 mas k in No mas k in No mas k in A re d s o fa is po s itio n e d o n the r ight s ide o f the image . An o pe n lapto p s its o n to p o f a s mall blu e table n e ar the ce n te r- le f t. A cable r u n s f ro m the lapto p down to the f lo o r. A win dow with cu rtain s is o n the back wall, le ttin g in daylight. A white table lamp s its o n the win dows ill to the r ight A lapto p is s ittin g o n a wo o de n table . A win dow is vis ible in the backgro u n d. A pe r s o n is s ittin g o n a co u ch in the ro o m. A re mo te co n tro l is place d o n the table . A bo o k is lo cate d o n the table n ext to the lapto p A lapto p co mpu te r is s ittin g o n a table . The lapto p is o pe n . The table is in f ro n t o f a win dow. The win dow has a cu rtain . The co u ch is in f ro n t o f the win dow A lapto p co mpu te r is s ittin g o n a table . A re d co u ch is in the backgro u n d. A win dow is vis ible in the backgro u n d. A re mo te co n tro l is o n the table . A bo o k is o n the table A lapto p co mpu te r is s ittin g o n a table . A re d co u ch is in the backgro u n d. A win dow is vis ible in the backgro u n d. A re mo te co n tro l is o n the table . A bo o k is o n the table A f lu f f y gray- brown tabby cat is s ittin g o n to p o f a s tain le s s s te e l re f r ige rato r. Wo o de n k itche n cabin e ts are mo u n te d above an d be hin d the re f r ige rato r. A bu lle tin bo ard with pin n e d n o te s is han gin g o n the wall to the le f t o f the re f r ige rato r. A f rame d pictu re is han gin g above the bu lle tin bo ard o n the le f t wall. Seve ral pape r s an d magn e ts are attache d to the r ight s ide o f the re f r ige rato r. A cat is s ittin g o n to p o f a re f r ige rato r. The re f r ige rato r is me tallic an d s ilve r in co lo r. The cat is lo o k in g down . The re f r ige rato r is lo cate d in a k itche n . The re f r ige rato r is o pe n A cat is s ittin g o n to p o f a re f r ige rato r. The re f r ige rato r is me tallic an d s ilve r in co lo r. The cat is lo o k in g down . The re f r ige rato r is lo cate d in a k itche n . The re f r ige rato r is o pe n A cat is s ittin g o n to p o f a re f r ige rato r. The re f r ige rato r is me tallic an d s ilve r in co lo r. The cat is lo o k in g down . The re f r ige rato r is lo cate d in a k itche n . The re f r ige rato r is o pe n A cat is s ittin g o n to p o f a re f r ige rato r. The re f r ige rato r is me tallic an d s ilve r in co lo r. The cat is lo o k in g at the came ra. The re f r ige rato r is lo cate d in a k itche n . The cat is o n the re f r ige rato r do o r Fig. 14: Sentence-level qualitative analysis using Ground-Truth (GT) cap- tions. Focus-phase masking (Phase 2) consistently mitigates hallucinated statements compared to No masking or interventions in Phase 1 and 3. E Qualitative Analysis Methodology using Ground-Truth Captions In Section 4.1 of the main text, we presented a qualitative analysis to observe how hallucinations manifest at the sentence level under different phase-masking con- ditions. This section details the procedure for constructing and utilizing Ground- Truth (GT) captions for this analysis, the results of which are visualized in Fig. 14 and Fig. 15. To construct reference descriptions for comparison, we first generated an ini- tial set of eight descriptive sentences for randomly sampled images from the COCO dataset using a large language model. These descriptions were subse- quently reviewed and corrected by human annotators to remove inaccurate ob- ject mentions or relationships, resulting in GT captions that are consistent with the visual content of the images. Subsequently, we examined the text generated by the LLaVA-1.5-7B model across four distinct configurations: the baseline (No masking), Phase 1 masking (Diffusion), Phase 2 masking (Focus), and Phase 3 masking (Rediffusion). For each configuration, the model was prompted to generate eight descriptive sen- 26Kim et al. GT ca pt i o P has e 1 mas k in P has e 2 mas k in P has e 3 mas k in No mas k in A plu s h te ddy be ar is vis ible n e ar the ce n te r o f the image . The te ddy be ar is be ige with a dar ke r n o s e an d eye s . The te ddy be ar is partially hidde n amo n g gre e n fe r n s an d ve ge tatio n . A lo n g, an gle d pie ce o f wo o d le an s u pward in f ro n t o f the te ddy be ar. The re are de n s e gre e n bu s he s an d tre e s fo r min g the backgro u n d. A te ddy be ar is s ittin g o n a pile o f s tick s . The te ddy be ar is brown in co lo r. The te ddy be ar is we ar in g a brown s hirt. The bran ch is s u r ro u n de d by le ave s . The image is in black an d white A te ddy be ar is s ittin g o n a pile o f s tick s . The te ddy be ar is brown in co lo r. The te ddy be ar is we ar in g a brown s hirt. The bran ch is s u r ro u n de d by le ave s . The image is in black an d white A brown te ddy be ar is s ittin g o n a pile o f bran che s . The te ddy be ar is we ar in g a brown s hirt. The be ar is s u r ro u n de d by gre e n le ave s . The be ar is s ittin g o n a lo g. The be ar is in a fo re s t s e ttin g A cat is s ittin g in a to ile t bowl. The cat is o ran ge an d white . The cat is lo o k in g at the came ra. The to ile t is white . The to ile t is o pe n GT ca pt i oGT ca pt i o P has e 1 mas k in P has e 1 mas k in P has e 2 mas k in P has e 2 mas k in P has e 3 mas k in P has e 3 mas k in No mas k in No mas k in Two lo n g bo ats are f lo atin g o n a large bo dy o f wate r. The n e are r bo at co n tain s many pe o ple s e ate d clo s e ly to ge the r. So me o f the paddle s in the n e are r bo at are re d, while o the r s are dar ke r co lo r s . A s e co n d bo at with a gro u p o f pe o ple is vis ible farthe r away n e ar the to p- ce n te r o f the image . The re s t o f the s ce n e is o pe n wate r with s mall r ipple s an d n o s ho re lin e vis ible A gro u p o f pe o ple is r idin g in a lo n g bo at. The bo at is f ille d with pe o ple s ittin g clo s e to ge the r. The bo at is re d an d white . The bo at is o n the wate r. The pe o ple in the bo at are e n joyin g the ir time to ge the r A gro u p o f pe o ple is r idin g in a lo n g bo at. The bo at is f ille d with pe o ple s ittin g clo s e to ge the r. The bo at is re d an d white . The bo at is o n the wate r. The pe o ple in the bo at are e n joyin g the ir time to ge the r A gro u p o f pe o ple is r idin g in a bo at o n the wate r. The bo at is f ille d with pe o ple , an d they are all we ar in g life jacke ts . The bo at is white an d re d in co lo r. The pe o ple in the bo at are e n joyin g the ir time to ge the r. The pe o ple in the bo at are s ittin g clo s e to e ach o the r A gro u p o f pe o ple are r idin g in a bo at. The bo at is f ille d with pe o ple . The bo at is o n the wate r. The pe o ple are e n joyin g the ir time o n the bo at. The bo at is a can o e A man is s e ate d at a table in do o r s . He is we ar in g a dar k lo n g- s le eve to p. He is s milin g while lo o k in g toward an o pe n lapto p. His han ds are po s itio n e d o n the lapto p keybo ard as if typin g. A dar k circu lar s ticke r with a tre e graphic is attache d to the lapto p lid A gro u p o f cows is walk in g down a ro ad. The re are s eve ral pe o ple s tan din g o n the s ide o f the ro ad. The pe o ple are watchin g the cows as they walk by. The cows are brown in co lo r. The pe o ple are s tan din g n e ar the cows A gro u p o f cows is walk in g down a ro ad. The cows are brown an d white . The cows are walk in g in a lin e . The re are two pe o ple in the image , o n e n e ar the cows an d an o the r f u rthe r away. The pe o ple are walk in g be hin d the cows A gro u p o f cows is walk in g down a ro ad. The cows are brown an d white . The cows are walk in g in f ro n t o f a ho u s e . The cows are walk in g in a lin e . The cows are walk in g o n a hill. The pe o ple are walk in g be hin d the cows A gro u p o f cows is walk in g down a ro ad. The cows are walk in g in f ro n t o f a car. The re are s eve ral pe o ple s tan din g o n the s ide o f the ro ad. The pe o ple are watchin g the cows as they walk by. The image is in black an d white Fig. 15: Additional qualitative examples using Ground-Truth (GT) captions. The comparisons highlight that applying the intervention strictly to the focus phase is crucial for ensuring factual alignment without degrading the overall sentence structure. tences per image. We then conducted a sentence-by-sentence comparison against the GT captions to qualitatively observe how phase-specific interventions influ- ence the introduction of hallucinated statements. Please note that while the models generated eight sentences to ensure com- prehensive visual coverage, Fig. 14 and Fig. 15 display a representative subset of approximately five sentences per setting. This subset is shown to facilitate clear side-by-side visual comparison of the generated descriptions across different in- tervention phases while maintaining visual clarity. F Extended Qualitative Results and Spatial Mask Comparisons F.1 Extended Qualitative Results on CHAIR and POPE Datasets This subsection provides additional qualitative examples from the CHAIR and POPE benchmarks to further illustrate the impact of the proposed phase-aware token suppression. The evaluation encompasses the five distinct LVLM archi- tectures analyzed in the main text: LLaVA-1.5-7B, LLaVA-1.5-13B, Shikra-7B, Qwen-2.5-VL, and InternVL-2.5. Focus Matters27 The provided examples compare the text generation outputs of the original baseline models, the AUE method, and our proposed approach.Specifically, Figs. 16 – 20 present extended text generation examples evaluated under the CHAIR benchmark. Furthermore, Figs. 21 – 25 display qualitative comparisons based on the POPE benchmark, which evaluates models using targeted object-existence queries (e.g., "Is there a [object] in the image?"). Consistent with the quantitative results reported in the main manuscript, the unmasked baseline models frequently output descriptions containing objects ab- sent from the input image. Applying our phase-aware DPP masking suppresses these hallucinated mentions and yields descriptions more tightly grounded to the actual visual content. The qualitative improvements are consistently observ- able across the descriptive generation tasks (CHAIR) and the targeted object- existence queries (POPE) for all five evaluated architectures. F.2 Spatial Mask Comparisons: Ours vs. AUE Expanding on the spatial analysis presented in Section 5.3 of the main text, Fig. 26 provides further visual comparisons between the masking patterns gen- erated by our DPP-based approach and the PGD-based AUE method. To ensure a fair and consistent comparison, the visualizations are constructed under similar masked-token budgets for both methods. The extended examples confirm the distinct spatial characteristics of the two selection strategies. The AUE method, which removes tokens based on an absolute uncertainty thresh- old, tends to produce continuous and dense removal patterns. In contrast, the proposed DPP method jointly models token importance and semantic similarity. Consequently, it produces a more spatially dispersed masking pattern, effectively filtering out redundant visual features while preserving a diverse set of salient visual cues required for accurate multimodal reasoning. G Evaluation on Additional Hallucination Benchmarks To further validate the generalization of our proposed method beyond the CHAIR and POPE benchmarks, we additionally evaluated our approach on the AMBER dataset. AMBER offers a comprehensive evaluation suite comprising both gener- ative and discriminative tasks, evaluating three types of hallucination: existence, attribute, and relation. For the generative task, AMBER employs four metrics: CHAIR↓ measures the proportion of hallucinated objects, Cover↑ assesses the coverage of ground- truth objects, Hal↓ captures the fraction of responses containing hallucinations, and Cog↓ quantifies cognitively plausible hallucinations. For the discriminative task, AMBER assesses whether the model correctly rejects hallucinatory prompts by answering "Yes" or "No" to targeted queries regarding object existence, attributes, and relations. Performance is measured using standard binary classification metrics—Accuracy↑, Precision↑, Recall↑, 28Kim et al. LLaVA-1.5-7BShikra-7B Bench MetricOrig.OursOrig.Ours Gen. CHAIR ↓8.26.212.211.5 Cover ↑48.546.248.947.5 Hal ↓34.924.749.744.8 Cog ↓ 3.52.45.64.6 Disc. Acc. ↑70.071.173.573.3 Precision ↑90.089.585.785.1 Recall ↑ 61.763.972.874.2 F1 ↑73.274.678.779.3 Table 4: Evaluation results on the AMBER dataset. The best result in each row is highlighted in bold. and F1-score↑—where Precision and Recall treat hallucinatory questions with a ground-truth answer of "No" as the positive class. Table 4 reports the evaluation results on the AMBER benchmark for the LLaVA-1.5-7B and Shikra-7B architectures. Across both models, the proposed phase-aware DPP masking consistently reduces hallucination-related generation metrics, such as Hal↓ and Cog↓. At the same time, the method maintains or slightly improves the discriminative metrics, including Acc↑, Recall↑, and F1↑. These results indicate that suppressing low-attention tokens during the focus phase can mitigate hallucination while preserving the model’s ability to cor- rectly identify object existence. Overall, the consistent improvements observed on AMBER suggest that the proposed approach generalizes beyond caption- based evaluations and remains effective under the benchmark’s object-existence evaluation setting. Focus Matters29 LLaVA-1.5-7B+AUE LLaVA-1.5-7B+AUE LLaVA-1.5-7B+AUE LLaVA-1.5-7B+AUE LLaVA-1.5-7B+AUE LLaVA-1.5-7B+Our LLaVA-1.5-7B+Our LLaVA-1.5-7B+Our LLaVA-1.5-7B+Our LLaVA-1.5-7B+Our LLaVA-1.5-7 LLaVA-1.5-7 LLaVA-1.5-7 LLaVA-1.5-7 LLaVA-1.5-7 Please describe the image in detail Please describe the image in detail Please describe the image in detail Please describe the image in detail Please describe the image in detail The image features a red double-decker bus driving down a city street. The bus is quite large and occupies a significant portion of the scene. There are several people walking along the sidewalk, with some of them carrying backpacks. In addition to the bus, there are two cars visible in the scene, one located behind the bus and another further down the street. The presence of pedestrians and vehicles indicates that this is a busy urban area The image features a boat floating on a body of water, with a dog sitting in the front of the boat. The boat is a small, white, and silver-colored vessel, and it appears to be a speed boat. The dog is sitting on the front of the boat, enjoying the ride.There are two people visible in the scene, one near the middle of the boat and another closer to the right side. The boat is equipped with a couch, which is located in the middle of the boat, providing a comfortable seating area for the passengers The image features a large blue and yellow train traveling down the tracks. The train is positioned in the middle of the scene, with its front end visible. The train appears to be a passenger train, as it is designed for transporting people.There are several people in the image, with some standing near the train and others further away. They seem to be observing the train as it passes by. The scene captures the essence of a busy train station, with people waiting for their ride or simply enjoying the view The image depicts a busy subway station with a large crowd of people waiting for the train. Among the crowd, a woman wearing a white headscarf is standing near the edge of the platform, holding a shopping bag. She appears to be waiting for the train to arrive.There are several other people in the scene, some of them carrying handbags and backpacks. The subway station is filled with people of various heights and positions, creating a lively atmosphere. The train is visible in the background, indicating that it is approaching the station The image features a bench sitting in a flooded area, surrounded by water. The bench is positioned near a pier, and it appears to be partially submerged in the water. The scene also includes a boat in the background, further emphasizing the flooded environment.There are several people in the scene, with one person standing close to the bench and others scattered around the area. Some of them are closer to the boat, while others are near the water's edge. The presence of people suggests that they might be observing the flooded area or possibly dealing with the aftermath of the flood The image features a red double-decker bus driving down a street, with a large building in the background. The bus is the main focus of the scene, and it appears to be a tour bus, possibly in London. There are several people visible in the image, with one person standing near the bus and others scattered around the scene. The street appears to be busy, with the bus driving down the road and people going about their day The image features a boat floating on a body of water, possibly a lake or a river. The boat is white and has a dog sitting on the front of it, enjoying the ride. The dog is positioned towards the right side of the boat, and it appears to be looking back at the camera. The boat is quite large, covering a significant portion of the water's surface The image features a yellow and blue train car traveling down the tracks. The train is large and occupies a significant portion of the image. The train is moving along the tracks, and it appears to be a passenger train. The train is situated in a rural area, giving it a sense of being in the middle of nowhere. The train is also described as a commuter train, which suggests that it is designed for daily transportation of passengers The image depicts a busy subway station with a large number of people waiting for the train. Among the crowd, a woman wearing a headscarf is standing near the edge of the platform, holding a shopping bag. She appears to be waiting for the train to arrive.There are several other people in the scene, some of them carrying handbags and backpacks. The subway station is filled with commuters, creating a bustling atmosphere. The train itself is visible in the background, indicating that it is approaching the station The image features a bench situated near a body of water, possibly a lake or a river. The bench is positioned close to the water's edge, providing a relaxing spot for people to sit and enjoy the view. The water appears to be calm and serene, with no visible signs of activity or disturbances. The bench is the main focal point of the scene, and it appears to be the only object in the image The image features a large red double-decker bus driving down a street. The bus is prominently displayed in the foreground, occupying a significant portion of the scene. There are several people visible in the background, walking or standing along the street. Some of them are closer to the bus, while others are further away. The scene captures the bustling atmosphere of a busy city street The image features a boat floating on a body of water, possibly a lake or a river. The boat is large and has a silver color. On the boat, there is a dog sitting in the front, enjoying the ride. The dog is positioned near the center of the boat, looking out over the water. The boat appears to be a motorboat, and the dog seems to be a companion on this journey The image features a yellow and blue train traveling down the tracks. The train is large and occupies a significant portion of the image. The train is moving swiftly, and its colorful design makes it stand out against the background. The train appears to be a passenger train, possibly a commuter train, as it is designed for public transportation The image depicts a busy subway station with a large number of people waiting for their train. Among the crowd, a woman wearing a headscarf is standing near the edge of the platform, possibly waiting for her train. She is holding a handbag in her hand.There are several other people in the scene, some of them carrying handbags as well. The subway station is filled with people, creating a bustling atmosphere. The image captures the essence of public transportation and the daily lives of people in a city The image features a bench situated near a body of water, possibly a lake or a river. The bench is positioned on the edge of the water, providing a perfect spot for relaxation and enjoying the view. The bench is made of metal and appears to be weathered, adding a sense of history and charm to the scene.In the background, there are several boats visible, some of which are floating on the water. The presence of these boats adds to the overall atmosphere of the scene, making it a picturesque and serene location Fig. 16: Extended qualitative results on the CHAIR benchmark using LLaVA-1.5-7B. 30Kim et al. LLaVA-1.5-13B+AUE LLaVA-1.5-13B+AUE LLaVA-1.5-13B+AUE LLaVA-1.5-13B+AUE LLaVA-1.5-13B+AUE LLaVA-1.5-13B+Our LLaVA-1.5-13B+Our LLaVA-1.5-13B+Our LLaVA-1.5-13B+Our LLaVA-1.5-13B+Our LLaVA-1.5-13 LLaVA-1.5-13 LLaVA-1.5-13 LLaVA-1.5-13 LLaVA-1.5-13 Please describe the image in detail Please describe the image in detail Please describe the image in detail Please describe the image in detail Please describe the image in detail The image features a motorcycle parked on a street, with a sidecar attached to it. The motorcycle is an old-fashioned model, and the sidecar is made of wood. The motorcycle is positioned on the left side of the scene, while the sidecar is located on the right side.There are two people in the scene, one standing near the motorcycle and the other standing further back. The motorcycle and sidecar are the main focus of the image, showcasing their vintage design and unique features The image features a man sitting on a bench surrounded by a large flock of pigeons. The man appears to be feeding the birds, as they gather around him in various positions. Some pigeons are perched on the bench, while others are standing or flying nearby.In the background, there are a few cars parked, and a handbag can be seen placed close to the bench. The scene captures a moment of interaction between the man and the pigeons, creating a peaceful and lively atmosphere The image features a young child standing on a sidewalk, holding a black and white umbrella to protect themselves from the rain. The child is positioned in the center of the scene, with the umbrella covering a significant portion of the image.In the background, there is a car parked on the street, and a truck can be seen further down the road. Additionally, there are two chairs placed near the edge of the scene, possibly belonging to a nearby outdoor seating area The image features a cityscape with a large industrial building in the foreground, surrounded by trees. The building has two tall smokestacks, which are prominent in the scene. In the background, there are several other buildings, including a clock tower, which adds to the urban atmosphere.The cityscape is bustling with activity, as there are multiple cars and a truck visible in the scene. The cars are scattered throughout the area, with some closer to the foreground and others further in the background. The truck is located near the center of the image, adding to the busy city vibe The image features a man sitting on a couch in a living room, holding a Nintendo Wii remote control in his hand. He appears to be playing a video game, possibly on a Wii console. The couch is positioned in the center of the room, with a dining table located nearby.There are several chairs in the room, with one close to the dining table and another one further away. A few books can be seen scattered around the room, possibly indicating that the man enjoys reading or has been reading before playing the video game The image features a motorcycle with a black frame and a silver engine. The motorcycle is parked on a street, and the front wheel is visible in the foreground. The motorcycle is old and has a number 60 on the front. The scene is set against a black background, which highlights the motorcycle's details The image features a man sitting on a park bench surrounded by a large flock of pigeons. The man appears to be feeding the birds, as they gather around him in various positions. Some birds are perched on the bench, while others are standing or flying nearby. The scene captures the interaction between the man and the pigeons, creating a peaceful and lively atmosphere in the park The image features a young child standing on a sidewalk, holding a black umbrella to protect themselves from the rain. The child is positioned in the center of the scene, with the umbrella covering a significant portion of their body.In the background, there is a truck parked on the street, and a bench can be seen further down the sidewalk. The scene captures a typical rainy day, with the child trying to stay dry under their umbrella The image features a large industrial building with two tall smokestacks in the background. The building is situated in front of a clock tower, which is visible in the middle of the scene. The clock tower appears to be a prominent landmark in the area. The industrial building is surrounded by trees, adding a touch of nature to the scene. The overall atmosphere of the image suggests a blend of urban and natural elements The image features a man lying on a couch, holding a Wii remote in his hand. He appears to be playing a video game, possibly on a Nintendo Wii console. The couch is positioned in the center of the room, with a chair located to the left side of the scene.There are several books scattered around the room, with some placed on the floor and others on a nearby surface. A dining table can be seen in the background, along with a couple of chairs placed around it. The room seems to be a comfortable and cozy space for the man to relax and enjoy his gaming session The image features a motorcycle parked on a street, with a close-up view of the front wheel and the engine. The motorcycle is an old- fashioned model, possibly a vintage or antique, and it appears to be in good condition. The motorcycle is parked next to a bush, adding a touch of greenery to the scene. The overall atmosphere of the image suggests a sense of nostalgia and appreciation for the classic design of the motorcycle The image features a man sitting on a bench surrounded by a large flock of pigeons. The man appears to be feeding the birds, as they are all gathered around him. There are at least 13 pigeons in the scene, with some perched on the bench and others flying nearby. The man is seated in the middle of the scene, with the birds surrounding him on all sides. The scene captures a peaceful moment between the man and the birds, as they share a meal together The image features a young child standing on a sidewalk, holding a black and white umbrella to protect themselves from the rain. The child is positioned in the center of the scene, with the umbrella covering a significant portion of their body.In the background, there is a truck parked on the street, partially visible on the right side of the image. The scene captures the child's innocence and the simple joy of playing in the rain The image features a cityscape with a large building in the foreground, surrounded by several other buildings. The main building has a clock tower, which stands out among the other structures. In the background, there are two tall smokestacks, likely part of a factory or industrial complex.The scene also includes a few trees scattered throughout the area, adding some greenery to the urban landscape. The overall atmosphere of the image is that of a bustling city with a mix of industrial and architectural elements The image features a man sitting on a couch, holding a Wii remote in his hand. He appears to be playing a video game, possibly on a Nintendo Wii console. The man is wearing a black shirt and is positioned in the center of the scene.In the background, there is a dining table with chairs placed around it. The table is located on the left side of the image, and the chairs are arranged in a semi-circle formation. The scene suggests a casual and relaxed atmosphere, with the man enjoying his gaming session Fig. 17: Extended qualitative results on the CHAIR benchmark using LLaVA-1.5-13B. Focus Matters31 Please describe the image in detail. Shikra-7B The image features a tennis court with two tennis players actively engaged in a match. One player is on the ground, possibly taking a swing at a low ball, while the other player is standing next to a tennis racket, possibly preparing for their next move. There are several people in the background, likely watching the game or waiting for their turn to play. Some of them are sitting on a bench, while others are standing around the court. Additionally, there is a backpack placed on the ground near the edge of the court, which might belong to one of the players or spectators. Shikra-7B+AUE The image captures a tennis match in progress, with several players and spectators present. A man is leaning over, holding a tennis racket, and preparing to hit the ball. There are multiple people in the stands, watching the match with keen interest.In addition to the players and spectators, there is a tennis ball visible on the court. A backpack can also be seen near the edge of the court, possibly belonging to one of the players. The scene is lively, showcasing the excitement of the tennis match and the dedication of the players and spectators. Shikra-7B+Ours The image features a tennis court where a game is taking place. There are multiple people present, including a player holding a tennis racket, likely preparing to hit the sports ball in the air. Another person is seen holding a tennis racket as well, possibly waiting for their turn to play. In addition to the players, there are several spectators watching the game from various positions around the court. A bench is located near the playing area, providing a place for people to sit and enjoy the match. The scene is lively, with everyone engaged in the tennis game or waiting for their turn to play. Please describe the image in detail. Shikra-7B The image depicts a picturesque scene of a herd of sheep grazing in a lush green field. There are at least 13 sheep spread across the field, with some closer to the foreground and others further back towards the edges of the scene. The landscape features a beautiful mountain in the background, adding to the serene atmosphere of the setting.In addition to the sheep, there are a few vehicles visible in the image, including cars and trucks, located near the field, possibly for transportation or farming purposes. Shikra-7B+AUE The image features a picturesque scene of a group of white sheep grazing on a grassy hillside. There are at least nine sheep spread across the field, with some closer to the front and others further back. The environment appears to be a blend of a mountain and a green pasture, providing a serene and natural setting for the sheep. Shikra-7B+Ours The image features a picturesque scene of a herd of sheep grazing in a lush green field. There are at least 13 sheep spread across the field, with some closer to the foreground and others further back towards the edge of the scene. The landscape is characterized by a gentle slope, and the sheep appear to be enjoying the fresh grass.In the background, there are mountains visible, emphasizing the natural beauty of the landscape. The combination of the vibrant green field and the serene mountainous backdrop creates a tranquil and idyllic atmosphere. Please describe the image in detail. Shikra-7B The image features a beach scene with a small metal boat prominently positioned at the edge of the water. The boat appears to be sitting on the shore, partially surrounded by a body of water.There are several other boats scattered throughout the scene, both close to and further away from the main boat. Some of these boats are situated on the left side of the image, while others can be found on the right side.In addition to the boats, there are numerous people in the background enjoying the beach atmosphere. They are dispersed in various groups, likely engaging in different activities or conversations.A bench is also visible in the middle of the scene, providing a place to relax and take in the view. Shikra-7B+AUE The image features a sandy beach with a group of boats near the water. There are several small boats, including a metal boat, beached on the shore. The metal boat is positioned closer to the center of the scene, while other boats are scattered around the beach.In addition to the boats, there are a few people enjoying their time on the beach. One person can be seen closer to the left side of the image, another is located further back, and two more people appear on the right side of the scene. The beach and the boats provide a picturesque and relaxing atmosphere. Shikra-7B+Ours The image features a beach scene with a small boat sitting on the sand near the water. The boat appears to be surrounded by a body of water, possibly an ocean, and is positioned close to the edge of the shore. There are several people scattered throughout the beach area, enjoying the sun and the water. Some of them are closer to the boat, while others are further away, exploring the beach or engaging in different activities. The overall atmosphere of the scene is relaxed and leisurely, with people enjoying their time by the water. Please describe the image in detail. Shikra-7B The image features a large body of water with a small airplane flying over the water, possibly approaching a landing. The plane appears to be a passenger jet, and it is located towards the right side of the scene. There are also several people in the vicinity of the airplane, possibly enjoying the view or waiting for their flight. Some of these people are located near the center of the image, while others can be found towards the right side. Shikra-7B+AUE The image features a large body of water with a small island nearby. There are several people on the island and a plane flying in the sky, seemingly approaching the island. The plane appears to be a large passenger jet, and it is positioned towards the right side of the image. The water surrounding the island and the plane gives the impression of a serene and picturesque setting. Shikra-7B+Ours The image features a large body of water with a small island nearby. There is a plane flying in the sky, appearing as a blurry image. The island is located in the middle of the water, surrounded by the lake. The scene creates a sense of serenity and calmness with the presence of the water and the airplane soaring through the sky. Please describe the image in detail. Shikra-7B The image features a man kiteboarding on a large body of water, with a kite attached to his wrists and surfboard in tow. The man is skillfully balancing on the surfboard while being propelled by the kite. There are several other people in the scene, likely enjoying the water activities as well. Some of them can be seen in the background, both near the shore and further out in the water. There is also a boat visible in the scene, situated near the right side of the image. Shikra-7B+AUE The image features a man kiteboarding over a large body of water, with a blue sky in the background. The man is skillfully balancing on a board while being propelled by a colorful kite that is flying high above the ocean. There are several other people in the scene, likely enjoying the water activities as well. Some of them are kiteboarding or windsurfing, while others are simply watching the activities from a distance. The image captures a lively and exciting atmosphere with people having a great time on the water. Shikra-7B+Ours The image features a man kiteboarding on top of a body of water, with a surfboard attached to his feet. He is skillfully balancing on the surfboard while being pulled by a large kite that is flying in the sky above him. There are several other people in the scene, likely enjoying the water activities as well. Some of them can be seen on the water's surface, either close to the man or further away. The kite is visible in the sky, with its position and size varying across the scene. Fig. 18: Extended qualitative results on the CHAIR benchmark using Shikra-7B. 32Kim et al. Please describe the image in detail. Qwen-2.5-VL The image features a man sitting on a wooden bench in a park, surrounded by a large flock of pigeons. The birds are scattered around the bench, with some sitting on the ground and others flying nearby. The man appears to be enjoying the company of the birds, as he is smiling and looking at them.In the background, there are several cars parked, and a truck can be seen further away. The scene captures a peaceful moment in the park where the man and the birds coexist. Qwen-2.5-VL+AUE The image features a man sitting on a bench surrounded by a large flock of pigeons. The man appears to be feeding the birds, as they gather around him, some even sitting on his lap. The birds are scattered all around the bench, with some close to the man and others further away.In the background, there are a few cars parked, and a truck can be seen in the distance. The scene captures a moment of interaction between the man and the pigeons, creating a peaceful and serene atmosphere. Qwen-2.5-VL+Ours The image features a man sitting on a wooden bench surrounded by a large flock of pigeons. The man appears to be enjoying the company of the birds, as they gather around him. There are at least 13 pigeons in the scene, with some sitting on the bench and others standing nearby.In addition to the pigeons, there are two birds flying in the sky, one near the top left corner and the other in the top right corner. The scene captures a peaceful moment between the man and the birds, creating a sense of harmony and connection with nature. Please describe the image in detail. Qwen-2.5-VL The image features a young boy wearing a grey shirt and blue jean shorts, standing on a skateboard in a driveway. He appears to be enjoying his time skateboarding, possibly practicing his moves. In the background, there are several potted plants placed around the area, adding a touch of greenery to the scene. A clock can also be seen hanging on the wall, possibly indicating the time for the boy as he skateboards. Qwen-2.5-VL+AUE The image features a young boy wearing a grey shirt and blue jeans, standing on a sidewalk and playing with a skateboard. He appears to be enjoying his time outdoors, possibly in a park or a similar setting. The boy is positioned in the center of the scene, with the skateboard placed on the ground in front of him.In the background, there are two potted plants, one on the left side and the other on the right side of the image. Additionally, there are two vases, one located near the center of the scene and the other towards the right side. The scene also includes a clock, which is placed on the left side of the image. Qwen-2.5-VL+Ours The image features a young boy wearing a gray shirt and blue jeans, standing on a sidewalk and playing with a skateboard. He appears to be in the middle of a trick, possibly performing a kickflip. The boy is the main focus of the scene, and his skateboard is positioned under his feet.In the background, there are several potted plants placed along the sidewalk, adding a touch of greenery to the scene. The plants are scattered at various distances from the boy, with some closer to the foreground and others further in the background. Please describe the image in detail. Qwen-2.5-VL The image features a baseball sitting on top of a red and orange striped object, which appears to be a baseball glove. The baseball is positioned in the center of the glove, surrounded by the vibrant colors of the striped material. The glove is placed on a surface, possibly a table or a counter, and the baseball seems to be the main focus of the scene. Qwen-2.5-VL+AUE The image features a baseball glove with a baseball inside, placed on a surface. The baseball glove is positioned in the center of the scene, and the baseball is nestled within it. The glove appears to be a catcher's mitt, and the baseball is in the process of being caught. The scene captures the essence of a baseball game, with the baseball player ready to make a catch. Qwen-2.5-VL+Ours The image features a baseball sitting on top of a baseball glove, which is placed on a surface. The baseball glove is made of leather and is positioned in the center of the scene. The baseball itself is white and appears to be in good condition. The glove and baseball are the main focus of the image, creating a sense of anticipation for a game or practice session. Please describe the image in detail. Qwen-2.5-VL The image features a man standing on a sandy beach, holding a frisbee in his hand. He appears to be preparing to throw the frisbee, possibly engaging in a game of beach frisbee with someone else. The man is wearing a white shirt, and his position suggests that he is focused on the task at hand.In the background, there is a boat visible on the water, adding to the beach atmosphere. The scene captures a moment of leisure and outdoor activity, with the man enjoying the beach and the frisbee game. Qwen-2.5-VL+AUE The image captures a man playing with a frisbee on a sandy beach. He is in the middle of a throw, with the frisbee flying through the air. The man is wearing a white shirt and appears to be enjoying the activity.In the background, there is a body of water, possibly the ocean, which adds to the beach atmosphere. The scene is set against a blue sky, creating a pleasant and relaxing environment for the man and his frisbee game. Qwen-2.5-VL+Ours The image features a man standing on a beach, wearing a white shirt and black shorts. He is in the process of throwing a frisbee, with his arm raised and the frisbee in the air. The man appears to be enjoying his time outdoors, possibly playing a game of frisbee with friends or family.The beach setting is characterized by the presence of a body of water, which can be seen in the background. The man is the main focus of the scene, with the frisbee and the beach serving as the backdrop. Please describe the image in detail. Qwen-2.5-VL This image captures a snowy urban street scene, likely following a heavy snowfall. The ground and parked cars on both sides of the street are covered with a thick layer of snow, indicating recent snowfall. The street is lined with bare trees, their branches laden with snow, and multi-story residential buildings with brick facades and white window frames.In the foreground, there are two individuals engaged in cross-country skiing. One person, wearing green ski pants, a teal jacket, and a red hat, is carrying a child dressed in winter clothing, including a light-colored hat and a backpack. The child appears to be learning or practicing skiing, as they are holding ski poles and standing on skis. Another individual, dressed in dark clothing, is also skiing nearby, appearing to assist or guide the child.The atmosphere suggests a cold winter day, with people enjoying outdoor activities despite the challenging weather conditions. The scene conveys a sense of community and resilience against the elements. Qwen-2.5-VL+AUE This image captures a snowy urban street scene, likely after a heavy snowfall. The ground and parked cars on both sides of the street are covered with a thick layer of snow. There are several people in the image, engaged in winter activities.In the foreground, two individuals are skiing on the snow-covered road. One person is wearing green ski pants, a teal jacket, and a red hat, carrying a child dressed in white and brown. The child appears to be learning how to ski, as they are being supported by the adult. Both are using ski poles for balance. Further down the street, another individual is seen skiing alone, wearing a blue outfit with a white hat. They also have ski poles and appear to be enjoying the activity.The background shows a row of residential buildings, some with brick facades and others with light-colored siding. The trees lining the street are bare, indicating it's winter, and their branches are laden with snow. A few other people can be seen further down the street, possibly walking or engaging in similar winter activities. The overall atmosphere is one of a quiet, wintry day in an urban neighborhood. Qwen-2.5-VL+Ours This image captures a snowy urban street scene, likely following a heavy snowfall. The ground and the roofs of parked cars are covered with a thick layer of snow, indicating recent snowfall. The street is lined with multi- story brick buildings on both sides, typical of an urban residential area.In the foreground, there are three individuals engaged in cross- country skiing. One person appears to be an adult wearing green ski pants and a red hat, carrying a child dressed in winter clothing. Another adult, dressed in dark clothing, is also skiing, accompanied by a child in blue ski gear. The skiers are using poles for balance and propulsion, and their tracks are visible in the snow.The street itself is partially cleared, with some areas still covered in snow, suggesting that the snowfall has recently occurred or that the area is not heavily trafficked. In the background, other individuals can be seen walking along the street, possibly clearing snow or heading out. The trees lining the street are bare, with snow clinging to their branches, adding to the wintry atmosphere. The overall mood of the image is calm and quiet, with a sense of community activity amidst the snow. Fig. 19: Extended qualitative results on the CHAIR benchmark using Qwen- 2.5-VL. Focus Matters33 )+/ '$,& # ,"/! , ' /,+ )+/ '$,& # ,"/! , ' /,+ )+/ '$,& # ,"/! , ' /,+ )+/ '$,& # ,"/! , ' /,+ )+/ '$,& # ,"/! , ' /,+ CQMNE=PKMNFMAETODNPNTPDIPBNE@F>>HNDTM@MNGQMHMN PNKH>IAN>SNAM>ABMNEDNKPOQMHMFNE@NPNBE<E@KNH>>=:N R@NOQMNS>HMKH>I@F7NPN=P@NEDNDOP@FE@KNP@FN PAAMPHDNO>N5MNABP4E@KNPN<EFM>NKP=M7NQ>BFE@KNPN GQEOMNKP=MNT>@OH>BBMH:N3MNEDNGMPHE@KNPNABPEFN 5IOO>@LIANDQEHO7N5BIMN2MP@D7NP@FNGQEOMND>T1D:N 3EDNMJAHMDDE>@NEDNS>TIDMF7NP@FNQMNDMM=DNO>N5MN T>@TM@OHPOE@KN>@NOQMNKP=M9 0MQE@FNQE=7NOQMHMNPHMNOQHMMN>OQMHNE@FE<EFIPBDN DMPOMFN>@NPN5MEKMNT>ITQ:NC>NOQMNBMSO7NPN=P@NEDN DEOOE@KNGEOQNQEDNBMKDNTH>DDMF7NGMPHE@KNPNGQEOMNOL DQEHONGEOQNPNKHPAQETNFMDEK@7N1QP1ENDQ>HOD7NP@FN GQEOMND@MP1MHD:N3MNQPDNPN@MIOHPBNMJAHMDDE>@N P@FNEDNB>>1E@KNO>GPHFDNOQMNDOP@FE@KN=P@:N;MJON O>NQE=7NPNG>=P@NEDNDMPOMF7NGMPHE@KNPN5BPT1NOP@1N O>ANP@FNFPH1NAP@OD:N6QMNEDNQ>BFE@KNPNHMFNAEBB>GN P@FNPAAMPHDNO>N5MNGPOTQE@KNOQMNKP=M:N8@NOQMN SPHNHEKQO7NP@>OQMHNAMHD>@NEDNAPHOEPBB4N<EDE5BM7N GMPHE@KNPNBEKQOLT>B>HMFNO>ANP@FNFPH1NAP@OD7N GEOQNOQMEHNBMKDNTH>DDMF9 CQMNH>>=NQPDNPNT>?4NPO=>DAQMHM7NGEOQNPNBPHKMN GE@F>GNT><MHMFN54N<MHOETPBN5BE@FDNE@NOQMN 5PT1KH>I@F:NCQMHMNEDNPNA>OOMFNABP@ON@MPHNOQMN GE@F>G7NP@FNPND=PBBNYXWVUNGEOQNPNSMGNEOM=D7N E@TBIFE@KNPNGPOMHN5>OOBMNP@FNPNT>IABMN>SNHMFN TIAD7NEDN<EDE5BMNE@NOQMNB>GMHNBMSONT>H@MH:NCQMN ><MHPBBNDMOOE@KNDIKKMDODNPNHMBPJMFNP@FND>TEPBN M@<EH>@=M@O7NBE1MB4NPNKPOQMHE@KN>SNSHEM@FDN>HN SP=EB4NM@2>4E@KNPN<EFM>NKP=MNDMDDE>@ CQMNE=PKMNDQ>GDNPN5BPT1NP@FNGQEOMNTPON DOP@FE@KN>@NPN5MF7NHMPTQE@KNE@O>NPNGQEOMNTIAN GEOQNT>B>HSIBNFMDEK@D:NCQMNTIANEDNABPTMFN>@NPN @EKQODOP@FN@MJONO>NPNBP=ANGEOQNPN5MEKMN BP=ADQPFM:NCQMN@EKQODOP@FNPBD>NQ>BFDNPN5>JN>SN OEDDIMD7NPN5>OOBMN>SN\><MNB>OE>@7NP@FNPND=PBBN ~yx|yzw:NCQMN5MFNQPDNPNOMPBN5MFDAHMPFNP@FNPN GQEOMNAEBB>G:NCQMNH>>=NQPDNG>>FM@NAP@MBE@KN>@N OQMNGPBB7NP@FNOQMN><MHPBBNBEKQOE@KNEDNGPH=7N THMPOE@KNPNT>?4NPO=>DAQMHM CQMNE=PKMNFMAETODNPNOM@@EDN=POTQNOP1E@KNABPTMN >@NPNTBP4NT>IHO:NƒNSM=PBMNOM@@EDNABP4MHNEDNE@NOQMN S>HMKH>I@F7N5M@FE@KN><MHNO>NAET1NIANPNOM@@EDN 5PBB:N6QMNEDNGMPHE@KNPNBEKQON5BIMNO>ANP@FNGQEOMN DQ>HOD:NCQMNT>IHONEDNDIHH>I@FMFN54NPNKHMM@N SM@TM7NP@FNOQMHMNPHMNDM<MHPBNAM>ABMNDOP@FE@KN P@FNDEOOE@KN>@NOQMN5BMPTQMHDNE@NOQMN5PT1KH>I@F:N CQMN5BMPTQMHDNPHMNAPHON>SNPNBPHKMHNDOHITOIHMNOQPON PAAMPHDNO>N5MN ¡£§ ¢¡§œ¡ ¡ž¦§ ¢¥Ÿ•¡¤¦ ¢ž§œ›7NPDN E@FETPOMFN54NOQMNHPEBE@KDNP@FNOQMNAHMDM@TMN>SN GPOMHNE@NOQMN5PT1KH>I@F:NCQMNDMOOE@KNEDN >IOF>>HD7NP@FNOQMHMNPHMNOHMMDN<EDE5BMNE@NOQMN FEDOP@TM:NCQMND14NEDN><MHTPDO7NDIKKMDOE@KNPN TB>IF4NFP4 CQMNE=PKMNDQ>GDNOG>NGQEOMN5>GBDNSEBBMFNGEOQNPN THMP=4ND>IA:NCQMND>IANT>@OPE@DNAEMTMDN>SN 5H>T>BENP@FNDAE@PTQ7NKE<E@KNEONPN<E5HP@ONKHMM@N T>B>H:NCQMN5>GBDNPHMNABPTMFN>@NPNG>>FM@N DIHSPTM7NP@FNOQMHMNEDNPNG>>FM@NDA>>@NHMDOE@KN >@NPND=PBBNG>>FM@NABPOMN5MQE@FNOQMN5>GBD:NC>N OQMNHEKQO7NOQMHMNEDNPNDA>>@NP@F-ª- ̈©¬«NHMDOE@KN>@N PN5MEKMNTB>OQN@PA1E@:NCQMN><MHPBBNDMOOE@KN DIKKMDODNPNDE=ABM7NHIDOETN=MPB7NA>DDE5B4N Q>=M=PFM:NCQMNBEKQOE@KNEDND>SO7NQEKQBEKQOE@KN OQMNOMJOIHMDNP@FNT>B>HDN>SNOQMNS>>F CQMNE=PKMNSMPOIHMDNPNABIDQNOMFF4N5MPHNDEOOE@KN >@NPN5MEKMNDIHSPTM:NCQMN5MPHNEDNFHMDDMFNE@NPN T>?47NKHMM@NP@FNGQEOMNH>5MNGEOQNPNGQEOMNSIHN OHE=:NCQMNH>5MNQPDNPNHMFNFHPGDOHE@KNOEMFNE@N²° ̄±®°PONOQMNSH>@O:NCQMNOMFF4N5MPHNQPDNPNBEKQON 5H>G@NSIH7NGEOQNFPH1MHN5H>G@NPTTM@ODN>@NEODN MPHD7NAPGD7NP@FNOQMNOEAN>SNEODN@>DM:NRODNM4MDNPHMN D=PBBNP@FN5BPT17NP@FNEONQPDNPNSHEM@FB47NE@<EOE@KN MJAHMDDE>@:NCQMN><MHPBBNPAAMPHP@TMNEDNGPH=N P@FNE@<EOE@K7NDIKKMDOE@KNPNT>=S>HOE@KNP@FN SMDOE<MNPO=>DAQMHM CQMNE=PKMNFMAETODNPNTPDIPBNE@F>>HNDTM@MN GQMHMNPNKH>IAN>SNAM>ABMNEDNKPOQMHMFNE@NPNBE<E@KN H>>=:NR@NOQMNS>HMKH>I@F7NPN=P@NEDNDOP@FE@KNP@FN PAAMPHDNO>N5MNABP4E@KNPN<EFM>NKP=M7NQ>BFE@KNPN GQEOMNKP=MNT>@OH>BBMH:N3MNEDNGMPHE@KNPNABPEFN 5IOO>@LIANDQEHO7N5BIMN2MP@D7NP@FNGQEOMND>T1D:N 3EDNMJAHMDDE>@NEDNS>TIDMF7NP@FNQMNDMM=DNO>N5MN T>@TM@OHPOE@KN>@NOQMNKP=M9 0MQE@FNQE=7NOQMHMNPHMNOQHMMN>OQMHNE@FE<EFIPBDN DMPOMFN>@NPN5MEKMNT>ITQ:NC>NOQMNBMSO7NPN=P@NEDN DEOOE@KNGEOQNQEDNBMKDNTH>DDMF7NGMPHE@KNPNGQEOMNOL DQEHONGEOQNPNKHPAQETNFMDEK@7N1QP1ENDQ>HOD7NP@FN GQEOMND@MP1MHD:N3MNQPDNPNHMBPJMFNA>DOIHMNP@FN EDNB>>1E@KNO>GPHFDNOQMNDOP@FE@KN=P@:N;MJONO>N QE=7NPNG>=P@NEDNDMPOMF7NGMPHE@KNPN5BPT1NOP@1N O>ANP@FNFPH1NAP@OD:N6QMNEDNQ>BFE@KNPNHMFNAEBB>GN P@FNPAAMPHDNO>N5MNGPOTQE@KNOQMNKP=M:N8@NOQMN SPHNHEKQO7NP@>OQMHNAMHD>@NEDNAPHOEPBB4N<EDE5BM7N GMPHE@KNPNBEKQOLT>B>HMFNO>ANP@FNFPH1NAP@OD9 CQMNH>>=NQPDNPNT>?4NPO=>DAQMHM7NGEOQNPNBPHKMN GE@F>GNT><MHMFN54N<MHOETPBN5BE@FDNE@NOQMN 5PT1KH>I@F:NCQMHMNEDNPNA>OOMFNABP@ON@MPHNOQMN GE@F>G7NP@FNPND=PBB ̧·¶ ́³μNGEOQNPNSMGNEOM=D7N E@TBIFE@KNPNGPOMHN5>OOBMNP@FND>=MNHMFNTIAD7NEDN <EDE5BMNE@NOQMNB>GMHNBMSONT>H@MH:NCQMN><MHPBBN DMOOE@KNDIKKMDODNPNTPDIPB7ND>TEPBNKPOQMHE@K7N A>DDE5B4NPNKP=MN@EKQONP=>@KNSHEM@FD CQMNE=PKMNDQ>GDNPN5BPT1NP@FNGQEOMNTPON DOP@FE@KN>@NPN5MF7NHMPTQE@KNF>G@NO>NPN @EKQODOP@F:NCQMNTPONEDNE@OMHPTOE@KNGEOQNPNGQEOMN TIANOQPONQPDNT>B>HSIBNFMDEK@DN>@NEO:NCQMN @EKQODOP@FNEDNTBIOOMHMFNGEOQN<PHE>IDNEOM=D7N E@TBIFE@KNPN5>JN>SNOEDDIMD7NPNTP@FBM7NPN5>OOBMN>SN B>OE>@7NP@FNPNSMGN>OQMHND=PBBN>52MTOD:NCQMHMNEDNPN BP=ANGEOQNPN5MEKMNDQPFMN>@NOQMN@EKQODOP@F7N AH><EFE@KNPNGPH=NBEKQO:NCQMNGPBBN5MQE@FNOQMN @EKQODOP@FNEDNAP@MBMFNGEOQNG>>F7NP@FNOQMN5MFN QPDNPNOMPBLT>B>HMFN5MFDAHMPFNGEOQNPNGQEOMN AEBB>G:NCQMN><MHPBBNDTM@MNDIKKMDODNPNT>?47N F>=MDOETNDMOOE@K CQMNE=PKMNFMAETODNPNOM@@EDN=POTQNOP1E@KNABPTMN >@NPNTBP4NT>IHO:NƒNSM=PBMNOM@@EDNABP4MHNEDNE@NOQMN S>HMKH>I@F7N5M@FE@KN><MHNO>NAET1NIANPNOM@@EDN 5PBB:N6QMNEDNGMPHE@KNPNBEKQON5BIMNO>ANP@FNGQEOMN DQ>HOD:NCQMNT>IHONEDNDIHH>I@FMFN54NPNKHMM@N SM@TM7NP@FN5M4>@FNOQMNSM@TM7NOQMHMNEDNPNBPHKMN 5IEBFE@KNGEOQN=IBOEABMNBM<MBD:NCQMN5IEBFE@KNQPDNPN KHMM@NP@FNGQEOMNDOHEAMFNPG@E@KNP@FNDM<MHPBN GE@F>GD:NCQMHMNPHMNAM>ABMNDOP@FE@KNP@FNDEOOE@KN >@NOQMNIAAMHNBM<MBN>SNOQMN5IEBFE@K7NGPOTQE@KNOQMN =POTQ:NCQMNDMOOE@KNPAAMPHDNO>N5MN>IOF>>HD7N GEOQNOHMMDN<EDE5BMNE@NOQMN5PT1KH>I@F:NCQMND14NEDN ><MHTPDO7NDIKKMDOE@KNPNTB>IF4NFP4 CQMNE=PKMNDQ>GDNOG>NGQEOMN5>GBDNSEBBMFNGEOQNPN THMP=4ND>IANT>@OPE@E@KN5H>T>BENP@FNDAE@PTQ:N CQMN5>GBDNPHMNABPTMFN>@NPNG>>FM@NDIHSPTM7NP@FN OQMHMNEDNPNG>>FM@NTIOOE@KN5>PHFNAPHOEPBB4N<EDE5BMN 5MQE@FNOQM=:NC>NOQMNHEKQON>SNOQMN5>GBD7NOQMHMNEDN PNDA>>@NHMDOE@KN>@NPN5MEKMNTB>OQN@PA1E@:NCQMN ><MHPBBNDMOOE@KNDIKKMDODNPNDE=ABM7NQ>=ML T>>1MFN=MPB7NA>DDE5B4NPNQMPBOQ4NP@FN@IOHEOE>IDN >AOE>@ CQMNE=PKMNSMPOIHMDNPNABIDQNOMFF4N5MPHNDEOOE@KN >@NPN5MEKMNDIHSPTM:NCQMN5MPHNEDNFHMDDMFNE@NPN T>?47NKHMM@NP@FNGQEOMNH>5MNGEOQNPNGQEOMNSIHN OHE=:NCQMNH>5MNQPDNPNHMFNFHPGDOHE@KNOEMFNE@N½» º¼¹»PONOQMNSH>@O:NCQMNOMFF4N5MPHNQPDNPNBEKQON 5H>G@NSIHNP@FNEDNDEOOE@KNGEOQNEODNBMKDNDOHMOTQMFN >IONE@NSH>@O:NCQMN5PT1KH>I@FNEDNDBEKQOB4N5BIHHMF7N M=AQPDE?E@KNOQMN5MPHNPDNOQMN=PE@NDI52MTO:NCQMN ><MHPBBNDTM@MNKE<MDNPNGPH=NP@FNE@<EOE@KNSMMB7N DIKKMDOE@KNPNT>=S>HOP5BMNP@FNHMBPJMFNDMOOE@K CQMNE=PKMNFMAETODNPNKH>IAN>SNAM>ABMNE@NPNBE<E@KN H>>=NDMOOE@K:NƒN=P@NDOP@FDNE@NOQMNTM@OMH7N Q>BFE@KNPNGQEOMNÃEENHM=>OM7NDIKKMDOE@KNQMNEDN ABP4E@KNPN<EFM>NKP=M:N3MNEDNGMPHE@KNPNABPEFN 5IOO>@LIANDQEHO7N5BIMN2MP@D7NP@FNGQEOMND>T1D:N 3EDNMJAHMDDE>@NEDNS>TIDMF7NP@FNQMNPAAMPHDNO>N 5MNT>@TM@OHPOE@KN>@NOQMNKP=M9 C>NQEDNBMSO7NPN=P@NEDNDMPOMFN>@NPN5MEKMNT>ITQ7N GMPHE@KNPNGQEOMNOLDQEHONGEOQNPNKHPAQETNFMDEK@7N 1QP1ENDQ>HOD7NP@FNGQEOMND@MP1MHD:N3MNEDNB>>1E@KN O>GPHFDNOQMNDOP@FE@KN=P@7NA>DDE5B4NGPOTQE@KN OQMNKP=M9 8@NOQMNHEKQONDEFMN>SNOQMNE=PKM7NPNG>=P@NEDN DMPOMFN>@NOQMNDP=MNT>ITQ7NQ>BFE@KNPNHMFNAEBB>G:N 6QMNEDNGMPHE@KNPN5BPT1NO>ANP@FN5BIMN2MP@D:N3MHN MJAHMDDE>@NEDN@MIOHPB7NP@FNDQMNDMM=DNO>N5MN >5DMH<E@KNOQMNDTM@M9 R@NOQMN5PT1KH>I@F7NOQMHMNEDNPNBPHKMNGE@F>GNGEOQN <MHOETPBN5BE@FD7NP@FNPNA>OOMFNABP@ONEDN<EDE5BMN @MPHNOQMNGE@F>G:NCQMNH>>=NEDNGMBBLBEO7NP@FN OQMHMNEDNPNAPE@OE@KN>@NOQMNGPBBNO>NOQMNHEKQO:NCQMN ><MHPBBNPO=>DAQMHMNEDNTPDIPBNP@FNHMBPJMF CQMNE=PKMNDQ>GDNPN5BPT1NP@FNGQEOMNTPON DOP@FE@KN>@NPN5MF7NBMP@E@KN><MHNPN@EKQODOP@F:N CQMNTPONEDNFHE@1E@KNSH>=NPNGQEOMNTIANGEOQN T>B>HSIBNFMDEK@DN>@NEO:NCQMN@EKQODOP@FNEDN TBIOOMHMFNGEOQN<PHE>IDNEOM=D7NE@TBIFE@KNPN5>JN >SNOEDDIMD7NPN5>OOBMN>SN\><MNB>OE>@7NPNBP=ANGEOQN PN5MEKMNDQPFM7NP@FNPNSMGN>OQMHND=PBBN>52MTOD:N CQMN5MFNQPDNPNOMPBLT>B>HMFN5MFDAHMPF7NP@FN OQMHMNEDNPNGQEOMNAEBB>GN5MQE@FNOQMNTPO:NCQMNGPBBN 5MQE@FNOQMN@EKQODOP@FNP@FN5MFNEDNAP@MBMFNGEOQN BEKQOLT>B>HMFNG>>F:NCQMN><MHPBBNDTM@MNEDNT>?4N P@FNF>=MDOET CQMNE=PKMNFMAETODNPNOM@@EDN=POTQNOP1E@KN ABPTMN>@NPNTBP4NT>IHO:NƒNSM=PBMNOM@@EDNABP4MHNEDN E@NOQMNS>HMKH>I@F7N5M@FE@KN><MHNO>NAET1NIANPN OM@@EDN5PBB:N6QMNEDNGMPHE@KNPNBEKQON5BIMNO>ANP@FN GQEOMNDQ>HOD:NCQMNT>IHONEDNDIHH>I@FMFN54NPN KHMM@NSM@TM7NP@FNOQMHMNPHMNDM<MHPBNDAMTOPO>HDN P@FNABP4MHDN<EDE5BMNE@NOQMN5PT1KH>I@F9 0MQE@FNOQMNT>IHO7NOQMHMNEDNPNBPHKMN5IEBFE@KNGEOQN =IBOEABMNBM<MBD7NSMPOIHE@KNPNKHMM@NP@FNGQEOMN DOHEAMFNPG@E@K:NCQMN5IEBFE@KNQPDNDM<MHPBN GE@F>GDNP@FNPN5PBT>@4NPHMPNGQMHMN=>HMN AM>ABMNPHMNKPOQMHMF:NCQMNDMOOE@KNPAAMPHDNO>N 5MN>IOF>>HD7NGEOQNOHMMDN<EDE5BMNE@NOQMN 5PT1KH>I@F7NDIKKMDOE@KNPNAPH1N>HNHMTHMPOE>@PBN PHMP:NCQMND14NEDN><MHTPDO7NE@FETPOE@KNPNTB>IF4N FP4 CQMNE=PKMNDQ>GDNOG>NGQEOMN5>GBDNSEBBMFNGEOQNPN THMP=4ND>IANT>@OPE@E@KN5H>T>BENP@FNDAE@PTQ:N CQMND>IANPAAMPHDNO>N5MNPNBEKQO7NTHMP=4N5H>OQ7N A>DDE5B4NPN5H>T>BENP@FNDAE@PTQND>IA:NCQMN 5H>T>BENSB>HMODNPHMN5HEKQONKHMM@7NP@FNOQMN DAE@PTQNBMP<MDNPHMN<EDE5BM7NPFFE@KNPNO>ITQN>SN KHMM@NO>NOQMNTHMP=4N5PDM:NCQMN5>GBDNPHMN ABPTMFN>@NPNG>>FM@NDIHSPTM7NP@FNOQMHMNEDNPN G>>FM@NDA>>@NHMDOE@KN>@NPNTB>OQN@PA1E@N 5MDEFMNOQMN5>GBD:NCQMN><MHPBBNDMOOE@KN DIKKMDODNPNDE=ABM7NQMPBOQ4N=MPB7NBE1MB4NDMH<MFN E@NPNHIDOETN>HNQ>=M4NM@<EH>@=M@O CQMNE=PKMNSMPOIHMDNPNABIDQNOMFF4N5MPHNDEOOE@KN >@NPN5MEKMNDIHSPTM:NCQMN5MPHNEDNFHMDDMFNE@NPN KHMM@NP@FNGQEOMNH>5MNGEOQNPNHMFNDOHE@KNOEMFN PH>I@FNEODN@MT1:NCQMNH>5MNQPDNPNSBISS4NGQEOMN T>BBPHNP@FNTISSD7NPFFE@KNO>NOQMNT>?4N PAAMPHP@TM:NCQMNOMFF4N5MPHNQPDNPNBEKQON5H>G@N SIHNP@FNPNSHEM@FB4NMJAHMDDE>@NGEOQN5BPT1NM4MDN P@FNPND=PBBN5BPT1N@>DM:NCQMN><MHPBBNDTM@MN KE<MDNPNGPH=NP@FNE@<EOE@KNSMMB7NHM=E@EDTM@ON>SN PNSMDOE<MN>HNQ>BEFP4NDMOOE@K Fig. 20: Extended qualitative results on the CHAIR benchmark using InternVL-2.5. 34Kim et al. LLaVA-1.5-7B+AUE LLaVA-1.5-7B+AUE LLaVA-1.5-7B+AUE LLaVA-1.5-7B+AUE LLaVA-1.5-7B+AUE LLaVA-1.5-7B+AUE LLaVA-1.5-7B+Our LLaVA-1.5-7B+Our LLaVA-1.5-7B+Our LLaVA-1.5-7B+Our LLaVA-1.5-7B+Our LLaVA-1.5-7B+Our LLaVA-1.5-7 LLaVA-1.5-7 LLaVA-1.5-7 LLaVA-1.5-7 LLaVA-1.5-7 LLaVA-1.5-7 Is there a tie in the image Is there a tie in the image Is there a bicycle in the image Is there a toaster in the image Is there a scissors in the image Is there a couch in the image yes, there is a tie in the image, which is placed on the coffee table yes, there is a tie in the image, and it is being worn by a man no, there is no bicycle in the image. the image features a man standing in front of a store with a display of soccer jerseys yes, there is a toaster in the image, and the small white dog is laying on top of it yes, there is a pair of scissors in the image, located near the cat no, there is no couch in the image. the image shows a person lying on the floor no, there is no tie in the image. the image features a couch, a television, and a coffee table yes, there is a tie in the image, and it is being worn by a man no, there is no bicycle in the image. the image features a store with a variety of clothing items displayed outside no, there is no toaster in the image. the image features a white dog laying on a table no, there are no scissors in the image. the image features a cat lying on a bed with blankets yes, there is a couch in the image, and a man is sitting on it no, there is no tie in the image. the image features a couch, a table, and a television no, there is no tie in the image. the image features a woman standing next to a table with a cake on it yes, there is a bicycle in the image, and it is parked next to a store no, there is no toaster in the image. the image features a white dog laying on a table no, there are no scissors in the image. the image features a cat laying on a bed yes, there is a couch in the image, and a man is standing in front of it Fig. 21: Extended qualitative results on the POPE benchmark using LLaVA- 1.5-7B. Focus Matters35 LLaVA-1.5-13B+AUE LLaVA-1.5-13B+AUE LLaVA-1.5-13B+AUE LLaVA-1.5-13B+AUE LLaVA-1.5-13B+AUE LLaVA-1.5-13B+AUE LLaVA-1.5-13B+Our LLaVA-1.5-13B+Our LLaVA-1.5-13B+Our LLaVA-1.5-13B+Our LLaVA-1.5-13B+Our LLaVA-1.5-13B+Our LLaVA-1.5-13 LLaVA-1.5-13 LLaVA-1.5-13 LLaVA-1.5-13 LLaVA-1.5-3 LLaVA-1.5-13 Is there a cake in the image Is there a dining table in the image Is there a wine glass in the image Is there a kite in the image Is there a tie in the image Is there a refrigerator in the image yes, there is a cake in the image. it is placed on the kitchen counter no, there is no dining table in the image. the image shows a living room with a couch, a coffee table, and a television yes, there is a wine glass in the image, and it is placed on a table next to a pizza yes, there is a kite in the image, which is hanging on the wall yes, there is a tie in the image. it is located near the train tracks and the train yes, there is a refrigerator in the image, located on the beach no, there is no cake in the image. the image features a kitchen with a sink, a window, and a dishwasher yes, there is a dining table in the image yes, there is a wine glass in the image, and it is placed on top of a pizza no, there is no kite in the image. the image features a living room with a fireplace, a couch, and a table yes, there is a tie in the image. it is located near the train tracks and is part of the train station no, there is no refrigerator in the image no, there is no cake in the image. the image features a kitchen with a stove and a window yes, there is a dining table in the image, located in the living room no, there is no wine glass in the image. the image features a close-up of a pizza on a pan no, there is no kite in the image. the image features a living room with a fireplace, a couch, and a table no, there is no tie in the image. the image features a train and a sign no, there is no refrigerator in the image. the image features a beach scene with a blue umbrella, a chair, and people enjoying the beach Fig. 22: Extended qualitative results on the POPE benchmark using LLaVA- 1.5-13B. 36Kim et al. Shikra-7B+AUE Shikra-7B+AUE Shikra-7B+AUE Shikra-7B+AUE Shikra-7B+AUE Shikra-7B+AUE Shikra-7B+Our Shikra-7B+Our Shikra-7B+Our Shikra-7B+Our Shikra-7B+Our Shikra-7B+Our Shikra-7 Shikra-7 Shikra-7 Shikra-7 Shikra-7 Shikra-7 Is there a dining table in the image Is there a dining table in the image Is there a pizza in the image Is there a spoon in the image Is there a car in the image Is there a bicycle in the image no, there is no dining table in the image. the man is holding a pizza in a kitchen setting no, there is no dining table in the image. the people are standing around a food stand yes, there is a pizza in the image, and people are waiting at a bus stop to get on a bus yes, there is a spoon in the image, and it is covered with food yes, there is a car in the image, and it is driving behind the herd of sheep that are crossing the street no, there is no bicycle in the image. the main subject is a brightly colored bus or double- decker bus with an attached store no, there is no dining table in the image. the man is holding a pizza in his hands and placing it in an oven yes, there is a dining table in the image, with a woman and two other people standing near it no, there is no pizza in the image. the image features a city bus and a large building, with people walking on the sidewalk yes, there is a spoon in the image the answer is no yes, there is a bicycle in the image. the bicycle is green and red, and it is parked next to a street lamp yes, there is a dining table in the image, and a man is putting something on it yes, there is a dining table in the image, and a group of people is standing around it no, there is no pizza in the image. the image features a white bus and people standing on the sidewalk near a city street no, there is no spoon in the image. the image features a dish containing food, but there is no spoon present no, there is no car in the image. the photo features a herd of sheep running along the road yes, there is a bicycle in the image, along with a street scene and a double-decker bus Fig. 23: Extended qualitative results on the POPE benchmark using Shikra- 7B. Focus Matters37 Qwen-2.5-VL+AUE Qwen-2.5-VL+AUE Qwen-2.5-VL+AUE Qwen-2.5-VL+AUE Qwen-2.5-VL+AUE Qwen-2.5-VL+AUE Qwen-2.5-VL+Our Qwen-2.5-VL+Our Qwen-2.5-VL+Our Qwen-2.5-VL+Our Qwen-2.5-VL+Our Qwen-2.5-VL+Our Qwen-2.5-V Qwen-2.5-V Qwen-2.5-V Qwen-2.5-V Qwen-2.5-V Qwen-2.5-V Is there a backpack in the image Is there a bottle in the image Is there a spoon in the image Is there a potted plant in the image Is there a skis in the image Is there a chair in the image no, there is no backpack in the image. the image shows a black cat standing next to a bottle of wine labeled "notro." there is also a clear glass bowl on the right side of the image, but no backpack is visible yes, there is a bottle visible on the bathroom counter in the background of the image. it appears to be a small, white bottle, possibly containing lotion or another personal care product no, there is no spoon in the image. the image shows a slice of cake on a plate with whipped cream and orange segments on top, along with a spoon placed next to the plate no, there is no spoon in the image. the image shows a slice of cake on a plate with whipped cream and orange segments on top, along with a spoon placed next to the plate no, there are no skis visible in the image. the person in the air appears to be performing a snowboard trick, as indicated by the snowboard under their feet yes, there is a chair visible in the image. it appears to be a wooden chair with a woven seat and backrest, positioned behind the group of people no, there is no backpack in the image. the image shows a black cat standing next to a bottle of wine labeled "notro." there are no visible backpacks or similar items in the background or foreground no, there is no bottle visible in the image. the image shows a child standing on a stool in a bathroom, wearing adult-sized boots and holding a toothbrush. there are no bottles present in the scene yes, there is a spoon in the image. it is placed on the right side of the plate with the cake yes, there is a potted plant in the image. it is located under an umbrella on a wooden deck. the pot appears to be red and contains some greenery yes, there is a pair of skis visible in the image. they can be seen in the background, being used by a person who appears to be performing a jump or trick on the snow yes, there is a chair in the image. it appears to be a wooden chair with a cushioned seat and backrest, which the woman and child on the right are sitting on yes, there appears to be a black backpack in the background of the image, partially visible behind the cat and the wine bottle no, there is no bottle visible in the image. the image shows a young child standing on a toilet, wearing adult-sized boots and a diaper, and holding what appears to be a toothbrush. there are no bottles present in the background or elsewhere in the image yes, there is a spoon in the image. it is placed on the right side of the plate with the cake slice yes, there is a potted plant in the image. it is located on a wooden deck and appears to be a small potted plant with some greenery inside it. the pot is red, and there is an umbrella above it, which might be providing shade for the plant yes, there is a pair of skis visible in the image. they can be seen in the air, indicating that someone is performing a ski jump or trick. the skier appears to be mid-air, with their skis extended above them no, there is no chair visible in the image. the people appear to be standing or sitting on a surface that is not clearly identifiable as a chair Fig. 24: Extended qualitative results on the POPE benchmark using Qwen- 2.5-VL. 38Kim et al. InternVL-2.5+AUE InternVL-2.5+AUE InternVL-2.5+AUE InternVL-2.5+AUE InternVL-2.5+AUE InternVL-2.5+AUE InternVL-2.5+Our InternVL-2.5+Our InternVL-2.5+Our InternVL-2.5+Our InternVL-2.5+Our InternVL-2.5+Our InternVL-2. InternVL-2. InternVL-2. InternVL-2. InternVL-2. InternVL-2. Is there a bottle in the image Is there a dining table in the image Is there a baseball bat in the image Is there a sink in the image Is there a cup in the image Is there a person in the image yes, there is a bottle in the image yes, there is a dining table in the image. it is located in the background, behind the people who are standing in line yes, there is a baseball bat in the image yes, there is a sink in the image. the man is standing in front of a stainless steel sink yes, there is a cup in the image. it is located on the beach, near the surfboard and the umbrella yes, there is a person in the image no, there is no bottle in the image no, there is no dining table in the image. the image shows a group of people standing in line at a food truck, with no visible dining tables no, there is no baseball bat in the image. the image shows a person playing catch with a ball, but there is no baseball bat visible yes, there is a sink in the image. the man is standing in front of a stainless steel sink no, there is no cup visible in the image. the main objects present are beach chairs, umbrellas, surfboards, and bags no, there is no person in the image no, there is no bottle in the image no, there is no dining table in the image. the image shows a group of people standing in line at a food truck, with no visible dining tables no, there is no baseball bat in the image. the image shows a person playing catch with a baseball there is no visible sink in the image. the main focus is on the man, the blender, and the pot. so, the answer is no no, there is no cup in the image. the image features a beach scene with surfboards, a towel, and an umbrella no, there is no person in the image Fig. 25: Extended qualitative results on the POPE benchmark using InternVL-2.5. Focus Matters39 OUR # Masked token: 23 # Masked token: 23 # Masked token: 23 # Masked token: 34 # Masked token: 23 # Masked token: 23 # Masked token: 23 # Masked token: 23 # Masked token: 23 # Masked token: 23 # Masked token: 23 # Masked token: 34 # Masked token: 26 # Masked token: 27 # Masked token: 23 # Masked token: 31 # Masked token: 25 # Masked token: 24 # Masked token: 29 # Masked token: 27 # Masked token: 23 # Masked token: 24 # Masked token: 25 # Masked token: 30 OUR OUR OUR OUR OUR OUR OUR OUR OUR OUR OUR AUE AUE AUE AUE AUE AUE AUE AUE AUE AUE AUE AUE Fig. 26: Spatial mask comparisons between our DPP-based method and the AUE method under similar masked-token budgets. Our method produces a more spatially dispersed pattern, effectively preserving diverse visual cues while filtering redundancy.