Paper deep dive
Beyond Dominant Patches: Spatial Credit Redistribution For Grounded Vision-Language Models
Niamul Hassan Samin, Md Arifur Rahman, Abdullah Ibne Hanif Arean, Juena Ahmed Noshin, Md Ashikur Rahman
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 10:40:11 AM
Summary
The paper identifies 'spatial credit collapse' as a root cause of object hallucination in Vision-Language Models (VLMs), where early transformer layers concentrate attention on sparse visual patches, suppressing contextual evidence. To address this, the authors propose Spatial Credit Redistribution (SCR), a training-free, inference-time method that redistributes hidden-state activations from high-attention source patches to their spatial neighbors. SCR reduces hallucination rates significantly across seven model configurations (Chameleon, LLaVA, Qwen, InternVL2) on benchmarks like POPE and CHAIR while preserving generation quality and incurring negligible latency overhead.
Entities (17)
Relation Signals (16)
Spatial Credit Collapse → causes → Object Hallucination
confidence 95% · We identify a contributing cause of this behavior, which we term spatial credit collapse
Spatial Credit Redistribution → evaluatedon → AMBER
confidence 95% · We evaluate SCR across seven model configurations ... on five benchmarks: ... and AMBER.
Spatial Credit Redistribution → evaluatedon → POPE
confidence 95% · We evaluate SCR across seven model configurations ... on five benchmarks: POPE
Spatial Credit Redistribution → evaluatedon → CHAIR
confidence 95% · We evaluate SCR across seven model configurations ... on five benchmarks: ... CHAIR
Spatial Credit Redistribution → evaluatedon → MME
confidence 95% · We evaluate SCR across seven model configurations ... on five benchmarks: ... MME
Spatial Credit Redistribution → evaluatedon → HallusionBench
confidence 95% · We evaluate SCR across seven model configurations ... on five benchmarks: ... HallusionBench
Spatial Credit Redistribution → mitigates → Object Hallucination
confidence 95% · SCR reduces POPE-Adversarial hallucination by 4.6-6.0 percentage points
Spatial Credit Collapse → correlateswith → Attention Entropy
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision-Language Models (VLMs) often hallucinate objects that are not present in the input image. We identify a contributing cause of this behavior, which we term spatial credit collapse: in early transformer layers, hidden-state activation concentrates on a small number of visual patches, suppressing surrounding contextual evidence and increasing reliance on language priors. Across seven models we observe a strong correlation between visual attention entropy and hallucination rate (r = -0.65, p < 0.001), suggesting that reduced spatial credit diversity contributes to hallucination. To address this issue we propose Spatial Credit Redistribution (SCR), a training-free inference-time method. SCR uses a lightweight two-pass procedure. A diagnostic pass identifies the top-K high-attention source patches and their spatial neighbors. A redistribution pass then scales each source by 1/lambda (~0.91) and injects a (lambda - 1) weighted copy of its hidden state into neighboring patches, restoring suppressed visual context without modifying model weights. Because the diagnostic pass is performed once per image and reused across the output sequence, the added latency is negligible (<0.5 ms per token for 100-token responses). We evaluate SCR across seven model configurations from four VLM families (Chameleon, LLaVA-1.5, Qwen-VL/Qwen2-VL, and InternVL2) on five benchmarks: POPE, CHAIR, MME, HallusionBench, and AMBER. SCR reduces POPE-Adversarial hallucination by 4.6-6.0 percentage points and CHAIR-s by 41-51 percent while preserving caption quality (CIDEr drop <=0.8). Compared with prior inference-time methods including OPERA, VCD, OA-VCD, DoLa, VLI, SID, and CRoPS, SCR achieves a better trade-off between hallucination reduction, generation quality, and latency.
Tags
Links
- Source: https://arxiv.org/abs/2602.22469v2
- Canonical: https://arxiv.org/abs/2602.22469v2
Trouble viewing inline? Open PDF directly →
Full Text
39,080 characters extracted from source content.
Expand or collapse full text
Beyond Dominant Patches: Spatial Credit Redistribution For Grounded Vision-Language Models Niamul Hassan Samin 1 , Md Arifur Rahman 1 , Abdullah Ibne Hanif Arean 1,3 , Juena Ahmed Noshin 2 , and Md Ashikur Rahman 1 1 The KOW Company 2 American International University Bangladesh (AIUB) 3 University of Dhaka Abstract. Vision-Language Models (VLMs) frequently hallucinate ob- jects absent from the input image. We identify spatial credit collapse as a root cause: in early transformer layers, hidden-state activation con- centrates on sparse visual patches, suppressing contextual evidence and inflating reliance on language priors. An entropy-hallucination correla- tion (r=−0.65, p<0.001) across seven models empirically motivates this diagnosis. To address this, we propose Spatial Credit Redistribution (SCR), a training-free, inference-time method with a two-pass design. A diagnostic pass (run once per image) identifies the top-K high-attention “source” patches and maps their 8-connected spatial neighbors. A redistribution pass then scales each source by 1/λ≈0.91 and injects a (λ−1)=0.10- weighted copy of its hidden state into each neighbor, amplifying the aggregate ℓ 2 norm by≈51% on average across models (range 48-56%) to restore suppressed visual context-without modifying any model weights. Since the diagnostic pass is amortized over the full output sequence, per-token overhead is negligible (< 0.5 ms for 100-token responses). We evaluate seven configurations spanning four model families-Chameleon (7B/30B), LLaVA-1.5 (7B/13B), Qwen-VL/Qwen2-VL (7B), InternVL2- 7B-on five benchmarks (POPE, CHAIR, MME, HallusionBench, AM- BER). SCR reduces POPE-Adversarial hallucination rate by 4.6-6.0 p and CHAIR-s by 41-51% relative, while preserving CIDEr within 0.8 p. On the joint (hallucination rate, generation quality, latency) Pareto fron- tier, SCR dominates OPERA, VCD, OA-VCD, DoLa, VLI, SID, and CRoPS-running ≈3-6× faster than OPERA/VCD at typical response lengths. On two stronger recent models (LLaVA-1.5-13B, InternVL2- 7B), our CRoPS † reproduction achieves marginally lower HR (0.2-0.4 p) at the cost of 3-4 p CIDEr degradation versus vanilla, compared to SCR’s ≤0.8 p. A Uniform-Smooth ablation confirms that attention- guided source selection is essential: random selection yields only 2.6- 3.4 p vs. SCR’s 4.6-6.0 p. Keywords: Object Hallucination· Spatial Credit Redistribution· Vision- Language Models· Attention Entropy· Training-Free Inference arXiv:2602.22469v2 [cs.CV] 4 Mar 2026 2N. H. Samin et al. 1 Introduction Vision-Language Models (VLMs) [17, 22, 23, 33] have achieved strong perfor- mance across vision and multimodal tasks by aligning visual encoders [6, 27] with Large Language Models (LLMs). Most of them fall victim to object hal- lucination. This occurs when a VLM generates descriptions of objects that are not present in the input image [8, 13, 18, 21]. Existing hallucination mitigation methods often rely on expensive retraining models, such as RLHF [26,31] or in- struction tuning [20], or trade off generation fluency through aggressive decoding constraints [14,16]. These methods primarily treat hallucinations as a language modeling problem and do not directly address the underlying visual grounding problem. In this work, we present evidence that hallucination is associated with vi- sual evidence becoming overly concentrated in sparse regions of the representa- tion. This concentration reduces the diversity of contextual information avail- able to the model and increases reliance on language priors (statistical patterns learned from text training data). Formally, we define spatial credit at image- token position i as c i ∝ ∥∂ logP(y|I,Q)/∂h i ∥·∥h i ∥ 2 , and spatial credit col- lapse as the condition in which the normalized distribution (c i / P j c j ) becomes highly peaked-its Shannon entropy H credit < H min -so that contextual evidence from non-dominant patches is effectively suppressed. We propose Spatial Credit Redistribution (SCR) as a corrective, training-free intervention; the entropy- hallucination correlation (r =−0.65, p < 0.001) and the Uniform-Smooth abla- tion together provide strong empirical grounding for the mechanism. Our main contributions are: (1) an empirically grounded set of design princi- ples linking spatial credit entropy to hallucination; (2) Spatial Credit Redistribu- tion (SCR)-a training-free, two-pass intervention with a principled consistency guarantee between the diagnostic and redistribution passes; and (3) a compre- hensive evaluation on five benchmarks (POPE, CHAIR, MME, HallusionBench, AMBER) across four VLM families (Chameleon, LLaVA, Qwen, InternVL2) at scales of 7B, 13B, and 30B. 2 Related Work Vision-Language Models and Hallucination. Most VLMs follow connector- based (BLIP-2 [17], LLaVA [22,23], InternVL2 [3]) or early-fusion (Chameleon [33]) architectures, yet all remain prone to object hallucination [8,18]. Hallucination Mitigation through Training. HIO [25] reduces halluci- nations but requires full retraining. RLHF [4, 31], instruction tuning [20], and DPO [28,38] all improve hallucination rates through training, but the substantial computational cost limits their scalability to larger models. Inference-Time Decoding Approaches. VCD [16] applies visual con- trastive decoding (+153 ms); OPERA [14] applies an attention penalty via beam search (+267 ms); OA-VCD [12] refines VCD with adversarial vision-encoder per- turbations (+72 ms); DoLa [5] contrasts early- and late-layer logits for factuality Beyond Dominant Patches: SCR for Grounded Vision-Language Models3 (a) Two-pass SCR pipeline - Pass 1: (1) extract attention, (2) se- lect top-K sources, (3) map 8-connected neighbors; Pass 2: (4) re- distribute credit in early layers (model-dependent; see Section 4), (5) generate corrected output. (b) 8-connected neighbor- hood. Source patch (orange) redistributes activation to 8 neighbors (teal): h n ← h n + (λ−1)h s ; source is scaled by 1/λ. Fig. 1: SCR methodology overview. (a) Full two-pass inference pipeline. (b) Spatial credit redistribution via 8-connected neighborhood structure. (+38 ms), though without specifically targeting visual grounding. All decoding- based methods improve hallucination at non-trivial latency cost, versus the +43- 46 ms (small models) to +54-56 ms (large models) incurred by SCR. Training-Free Representation Interventions. VLI [24] applies bi-causal attention steering, CRoPS [1] extends contrastive decoding to vision- and text- deficit modes (both concurrent preprints), and SID [15] proposes self-introspective decoding. Mechanistic Analysis. Attention manipulation [36] has been studied for efficiency and interpretability; feature attribution [32], universal attention head analysis [10], and knowledge erasure [9] are related, yet none directly address the spatial credit collapse that causes hallucinations. Hallucination Evaluation. CHAIR [29] measures hallucinated object rates in free-form COCO captions (Type I: open-ended) via sentence-level (CHAIR- s) and instance-level (CHAIR-i) metrics. POPE [18] probes object existence as yes/no queries (Type I: discriminative) across random, popular, and adversarial splits. MME [8], HallusionBench [11], and AMBER [34] assess broader general- ization across perception, visual illusion, and attribute/relational hallucination. 3 Methodology Figure 1a illustrates the two-pass pipeline: Pass 1 captures attention maps, se- lects top-K source patches, and maps 8-connected neighbors; Pass 2 redistributes 4N. H. Samin et al. credit through early layers (0-15 for Chameleon-7B/Qwen/InternVL2, 0-20 for Chameleon-30B, 0-11 for LLaVA). 3.1 Patch Selection and Neighbor Mapping Attention Map Computation. All four architectures use unified causal self- attention over an interleaved sequence of image and text tokens; none employs a separate cross-attention module. For Chameleon, image tokens are discrete VQ- VAE codebook entries; for LLaVA, Qwen, and InternVL2, they are projected patch embeddings. In all cases, we locate the N v image token positions in the input sequence, where N v = H img ×W img /P 2 for patch size P, and extract self- attention weights from text query positions to those visual positions. Weights are averaged across all heads and layers 8-16 to obtain the mean spatial attention map A∈R N v . For Qwen2-VL, which supports variable-resolution inputs, N v is computed dynamically from the image dimensions. Top-K Selection and Neighbor Mapping. Ablations (Appendix E) iden- tify K = 32 as the optimal number of source patches; border rows and columns are excluded to prevent attention-sink artifacts. To avoid redistribution inter- ference, sources are processed in descending attention order: a patch cannot act as both source and neighbor, and conflicts among overlapping source-neighbor pairs are resolved by priority, with higher-attention sources claiming neighbors first. 3.2 Intervention Mechanism Injection point relative to LayerNorm. All four architectures (Chameleon, LLaVA-1.5, Qwen/Qwen2-VL, InternVL2) use Pre-LayerNorm (Pre-RMSNorm) transformer blocks [37]. Each block l processes the residual stream x via: x ← x + Attn LN 1 (x) ,x ← x + FFN LN 2 (x) . SCR hooks are placed on the residual stream immediately after the second residual addition of block l (i.e., before block l+1’s LN 1 ). Because the residual stream is never renormalized at this position, the ℓ 2 -norm amplification persists across layers via additive skip connections, explaining the 48-56% aggregate norm increase at the final intervention layer (Appendix C, Table C2). The intervention redistributes activation from source patch h i s to its 8- connected spatial neighbors (see Figure 1b), while the source itself is scaled down using a fixed factor λ = 1.10. h i n ← h i n + (λ− 1)· h i s , h i s ← 1 λ · h i s (1) The aggregate ℓ 2 norm increases by ≈51% on average (range 48-56%), am- plifying suppressed spatial context while retaining 91% of each source activation (1/λ≈0.91); later layers are left unchanged to avoid interfering with language generation. Beyond Dominant Patches: SCR for Grounded Vision-Language Models5 Two-pass consistency. Pass 1 source selection uses the model’s fixed weight matrices to compute attention; SCR modifies only residual-stream magnitudes, not the weights, so dominant-patch structure is preserved across passes. Empir- ically, the Jaccard overlap between Pass 1 source sets and a hypothetical post- redistribution diagnostic run is 0.87±0.04 across 500 POPE images and all seven models-the residual 13% churn introduces < 0.2 p HR sensitivity (Appendix C, Table C3). 3.3 Empirical Motivation and Design Principles Empirical Finding (EF1)-Norm-Credit Proxy. We define visual credit at location i as c i ∝ ∂ logP(y|I,Q) ∂h i · ∥h i ∥ 2 . In early transformer layers, ∥h i ∥ 2 alone achieves Pearson r=0.72 correlation with c i , making it a tractable proxy (Appendix C for full per-model validation). Credit Concentration Entropy. We quantify spatial credit concentration via: H credit =− n X i=1 c i P j c j log c i P j c j .(2) Design Principle 1-Hallucination-Entropy Relationship. Low credit entropy (H credit < H min ) is empirically associated with elevated hallucination; we observe r =−0.65 (p < 0.001) across all models (Section 5, Tab. 6). Design Principle 2-8-Connected Neighborhoods. Natural images ex- hibit spatial autocorrelation decaying with patch distance [7,30]. An 8-connected neighborhood captures diagonal correlations at distance √ 2 (ρ( √ 2)≈0.62; Ap- pendix A) that a 4-connected scheme ignores, without over-spreading to low- correlation patches. Ablation results in Figure 4a confirm this as the optimal connectivity. Design Principle 3-Peak-Preserving Expansion. For λ ∈ [1.05, 1.15], SCR approximates the following entropy-maximization objective (Appendix A): max H ′ λ 1 H credit (H ′ )− λ 2 ∥H ′ − H∥ 2 F s.t. max i ∥h ′ i ∥≥ τ max j ∥h j ∥, with τ = 1/λ ≈ 0.91, ensuring the dominant patch retains sufficient activation to remain discriminative. 4 Experimental Setup Selected Models. Chameleon-7B/30B [33] (layers 0-15 / 0-20); as no published POPE results exist for Chameleon, we evaluate it ourselves using the official model weights; LLaVA-1.5-7B/13B [22] (layers 0-11 for both; validated by the layer-selection ablation in Appendix E, which shows diminishing returns beyond layer 11 regardless of depth); Qwen-VL/Qwen2-VL-7B [2,35] (layers 0-15); and InternVL2-7B [3] (layers 0-15). All layer ranges were selected on a held-out validation set of 500 COCO train2014 images, fully disjoint from the POPE 6N. H. Samin et al. and CHAIR evaluation sets; no evaluation data was used in hyperparameter selection. Computational Setup. All experiments were run on a workstation with an AMD Ryzen Threadripper PRO CPU and a single NVIDIA A100 GPU (fp16 precision). Benchmarks. POPE [18] in 3 splits (1,000 samples each); COCO [19] (3,000- image val2014 subset) for CHAIR [29] and CIDEr evaluation; MME [8] (percep- tion subtask); HallusionBench [11]; and AMBER [34] (object, attribute, and relation hallucination). Baselines. OPERA [14] (+267 ms), VCD [16] (+153 ms), OA-VCD [12] (+72 ms), DoLa [5] (+38 ms), SID [15], VLI [24], CRoPS [1]. OPERA, VCD, OA-VCD, DoLa, and SID use their official released code with default hyper- parameters. As official implementations of VLI † and CRoPS † are not publicly available at the time of submission, we implement both following the descriptions and hyperparameters in their respective arXiv papers; reimplementation bias is bounded by reproducing VCD and OA-VCD within 0.3 p of their published results under identical settings. All baselines were run on the same hardware un- der identical decoding settings (greedy, temperature 1.0) to ensure fair latency and quality comparisons. Uniform-Smooth is a controlled ablation that ap- plies the same operation as SCR (λ=1.10, K=32) but selects sources uniformly at random, isolating the contribution of attention-guided selection. Metrics. Hallucination Rate HR = FP/(FP+TN) (false positive rate for ab- sent objects [18]), Accuracy, CIDEr, CHAIR-s, CHAIR-i, and Attention Entropy. 5 Results 5.1 Multi-Model and Benchmark Evaluation Tabs. 1 and 2 report POPE-Adversarial and CIDEr results for all seven model configurations. Figure 2 summarizes the hallucination-rate trends visually. SCR reduces hallucination rates (HR) while maintaining generation quality across all tested configurations and model families. Observations. SCR achieves the best or second-best HR across all seven configurations, with CIDEr preserved within 0.8 p of vanilla in every case. Two exceptions emerge on the stronger recent models: on LLaVA-1.5-13B, our CRoPS † reproduction achieves HR = 12.6% vs. SCR’s 12.8% (0.2 p gap), and on InternVL2-7B, it achieves HR = 10.8% vs. SCR’s 11.2% (0.4 p gap). In both cases CRoPS † trades generation quality to reach that lower HR-CIDEr degrades by 3.4-4.0 p relative to vanilla (LLaVA-13B:−4.0 p; InternVL2:−4.1 p), ver- sus SCR’s ≤0.8 p. On the joint (HR, CIDEr) Pareto frontier, SCR therefore dominates all baselines across all seven configurations, including our CRoPS † reproduction. All gains are significant (p<0.001, paired t-test, 1,000 POPE sam- ples). The Uniform-Smooth ablation confirms attention guidance is essential: random selection yields only 2.6-3.4 p vs. SCR’s 4.6-6.0 p (≈1.7× gap). Beyond Dominant Patches: SCR for Grounded Vision-Language Models7 Table 1: POPE-Adversarial HR (%↓), Accuracy (%↑), CIDEr (↑) for 7B models. Mean±std over 3 runs.Green = best HR;Blue = second best. ∆HR relative to Vanilla. Uniform-Smooth: same SCR operation with random (not attention-guided) source selection. † Concurrent preprint, author-reimplemented (no official code; see §4). ModelMethodHR↓Acc↑ CIDEr↑ ∆HR↓ Cham-7B Vanilla19.5±.8 80.2±.7 102.3±1.4 - OPERA16.8±1.0 82.8±.7 100.3±1.3 −2.7 DoLa17.2±1.0 82.5±.8 101.8±1.4 −2.3 VCD15.1±1.3 84.6±1.0 97.4±2.0 −4.4 OA-VCD14.7±1.1 85.0±.9 99.2±1.5 −4.8 VLI † 15.8±.9 83.8±.8 100.6±1.1 −3.7 CRoPS † 14.5±1.1 85.2±.9 98.2±1.5 −5.0 SID15.3±.9 84.4±.7 99.5±1.2 −4.2 Uniform-Smooth 16.3±1.0 83.5±.8 99.8±1.4 −3.2 SCR 13.5±.786.2±.5 101.7±.9 −6.0 LLaVA-1.5-7B Vanilla18.8±.7 80.9±.6 108.5±1.3 - OPERA16.2±.9 83.4±.7 106.6±1.1 −2.6 DoLa16.8±1.0 82.9±.8 107.6±1.3 −2.0 VCD15.0±1.2 84.6±.9 103.8±1.7 −3.8 OA-VCD14.5±1.0 85.2±.8 105.3±1.3 −4.3 VLI † 15.2±.8 84.5±.7 106.2±1.1 −3.6 CRoPS † 14.4±1.1 85.3±.8 104.5±1.4 −4.4 SID15.2±.8 84.5±.6 105.8±1.0 −3.6 Uniform-Smooth 16.0±.9 83.7±.8 105.6±1.3 −2.8 SCR13.1±.786.5±.5 107.8±.8 −5.7 Qwen-VL Vanilla18.5±.6 81.2±.6 115.2±1.0 - OPERA15.6±.9 84.0±.6 113.4±1.0 −2.9 DoLa16.0±.8 83.7±.7 114.5±1.0 −2.5 VCD14.4±1.0 85.2±.7 111.2±1.4 −4.1 OA-VCD13.8±.9 85.9±.7 112.5±1.1 −4.7 VLI † 14.8±.7 84.9±.5 113.2±.9 −3.7 CRoPS † 13.9±.9 85.8±.6 111.8±1.2 −4.6 SID14.1±.7 85.6±.6 112.5±.8 −4.4 Uniform-Smooth 15.1±.8 84.6±.7 111.7±1.0 −3.4 SCR 12.9±.586.8±.4 114.6±.7 −5.6 Qwen2-VL-7B Vanilla17.0±.5 82.7±.5 118.5±.9- OPERA14.6±.8 85.1±.5 116.7±.9 −2.4 DoLa15.2±.7 84.5±.6 117.9±.9 −1.8 VCD13.1±.9 86.5±.7 114.6±1.3 −3.9 OA-VCD12.9±.8 86.8±.6 115.4±1.0 −4.1 VLI † 13.7±.6 86.0±.5 116.3±.8 −3.3 CRoPS † 12.6±.8 87.1±.6 115.1±1.1 −4.4 SID13.5±.6 86.2±.5 116.0±.7 −3.5 Uniform-Smooth 14.2±.7 85.6±.6 115.1±.9 −2.8 SCR 11.4±.688.2±.4 117.9±.6 −5.6 InternVL2-7B Vanilla15.8±.5 83.8±.4 121.4±.8- OPERA13.5±.7 86.0±.5 119.4±.8 −2.3 DoLa13.8±.7 85.7±.5 120.8±.8 −2.0 VCD12.4±.9 87.2±.7 117.2±1.1 −3.4 OA-VCD12.0±.8 87.6±.6 118.5±.9 −3.8 VLI † 12.9±.6 86.7±.5 119.4±.7 −2.9 CRoPS † 10.8±.788.0±.5 117.3±1.1 −5.0 SID12.2±.6 87.4±.5 118.5±.7 −3.6 Uniform-Smooth 13.2±.7 86.5±.5 117.9±.9 −2.6 SCR11.2±.5 88.5±.3 120.7±.5 −4.6 8N. H. Samin et al. Table 2: POPE-Adversarial HR (%↓), Accuracy (%↑), CIDEr (↑) for large-scale mod- els. Mean±std over 3 runs. Color coding as Tab. 1. On LLaVA-13B, our CRoPS † reproduction reaches the lowest HR but degrades CIDEr by 4.0 p vs. SCR’s 0.7 p; SCR Pareto-dominates on all three axes per our reproduction. ModelMethodHR↓ Acc↑ CIDEr↑ ∆HR↓ Cham-30B Vanilla18.2±.7 81.5±.6 108.2±1.2- OPERA15.4±1.0 84.2±.6 106.8±1.1 −2.8 DoLa16.0±.9 83.6±.7 107.6±1.1 −2.2 VCD14.1±1.1 85.6±.8 103.8±1.7 −4.1 OA-VCD13.7±1.0 86.1±.7 105.2±1.1 −4.5 VLI † 14.9±.8 84.8±.6 106.2±1.0 −3.3 CRoPS † 13.6±.9 86.1±.7 104.4±1.3 −4.6 SID13.9±.8 85.8±.6 105.1±.9 −4.3 Uniform-Smooth 15.4±.9 84.3±.7 105.3±1.1 −2.8 SCR 12.6±.687.1±.4 107.4±.8 −5.6 LLaVA-1.5-13B Vanilla17.5±.6 82.2±.5 114.8±1.1- OPERA14.9±.9 84.8±.6 112.8±1.0 −2.6 DoLa15.6±.8 84.1±.7 114.2±1.0 −1.9 VCD13.8±1.0 85.8±.8 110.6±1.5 −3.7 OA-VCD13.2±.9 86.5±.7 112.2±.9 −4.3 VLI † 14.3±.7 85.4±.6 112.4±.9 −3.2 CRoPS † 12.6±.987.0±.6 110.8±1.2 −4.9 SID13.6±.7 86.1±.5 111.9±.8 −3.9 Uniform-Smooth 14.9±.8 84.8±.6 112.5±1.0 −2.6 SCR12.8±.6 87.4±.4 114.1±.7 −4.7 5.2 POPE Split Analysis 5.3 Latency SCR incurs a one-time overhead from the diagnostic forward pass: +43-46 ms for smaller models (Cham-7B, LLaVA-7B, Qwen variants) and +54-56 ms for larger models (Cham-30B, LLaVA-13B). This is well below OPERA [14] (+267 ms), VCD [16] (+153 ms), OA-VCD [12] (+72 ms), and DoLa [5] (+38 ms). SCR does not increase memory usage, since all modifications are applied through in-place hooks on the residual stream. All latency numbers are end-to-end wall-clock time on a single NVIDIA A100 in fp16. Amortization. Unlike VCD/OPERA which apply overhead at every decod- ing step (2N passes for N tokens), SCR’s diagnostic pass runs once per image and is reused across the full output sequence. Per-token cost for a 100-token response is < 0.5 ms-negligible compared to decoding-based overheads. For very short responses (≤5 tokens, e.g., VQA), the fixed diagnostic-pass cost (+43- 46 ms or +54-56 ms depending on model size) is not amortized; in such settings OPERA’s beam-search penalty remains comparable, while DoLa [5] (+38 ms) offers a lower fixed cost at reduced HR gain. Beyond Dominant Patches: SCR for Grounded Vision-Language Models9 Chameleon 7B Chameleon 30B LLaVA-1.5 7B LLaVA-1.5 13B Qwen-VLQwen2-VL 7B InternVL2 7B Model Architecture 0 5 10 15 20 25 Hallucination Rate (%) -6.0 -5.6 -5.7 -4.7 -5.6 -5.6 -4.6 Mean: -5.4p Vanilla SCR (Ours) Fig. 2: POPE-Adversarial HR for all seven VLM configurations. SCR reduces HR by ≈4.6-6.0 p across all model families. Table 3: POPE split analysis (HR%) across model architectures (Vanilla→SCR). Ad- versarial split matches Tabs. 1 and 2; consistent gains across all splits and architectures. ModelRandom Popular Adversarial Chameleon-7B 12.5→8.5 15.8→10.9 19.5→13.5 Chameleon-30B 11.2→7.8 14.5→9.4 18.2→12.6 LLaVA-1.5-7B 11.8→8.6 15.0→10.4 18.8→13.1 LLaVA-1.5-13B 10.5→7.4 13.8→10.0 17.5→12.8 Qwen-VL11.5→8.2 14.8→10.1 18.5→12.9 Qwen2-VL-7B 10.2→6.8 13.5→9.2 17.0→11.4 InternVL2-7B 9.5→6.8 12.5→8.5 15.8→11.2 10N. H. Samin et al. For latency-critical deployments, an optional One-Pass SCR variant (a lightweight 2-layer MLP predictor;≈120K parameters) reduces overhead to +6- 11 ms at a cost of ≈1.3 p HR increase; full details in Appendix F. 5.4 Caption-Level Hallucination: CHAIR Results Tab. 4 shows CHAIR metrics for all seven models (full eight-baseline breakdown in Appendix B). 5.5 Extended Benchmark Evaluation: MME, HallusionBench, AMBER Tab. 5 reports results on MME Perception [8], HallusionBench [11], and AM- BER [34]. Discussion. Gains are strongest for object hallucination (AMBER-O:−4.8 p avg.), moderate for attributes (−2.0 p), and minimal for relations (−0.9 p), consistent with SCR’s spatial-grounding scope. HallusionBench gains are mod- est (+2.8 p) as many errors involve compositional reasoning; per-subtask break- downs are in Appendix B. 5.6 Empirical Prediction Validation We validate predictions P1-P4 (P1-P2 from Design Principle 1, P3 from Design Principle 2, P4 from Design Principle 3; see Section 3) across all seven models; EF1 is in Appendix C. P1: Entropy-Hallucination Correlation. Across samples and models, lower spatial credit entropy is associated with higher hallucination rates. Tab. 6 shows negative correlations across architectures. Samples with H<4.0 show elevated HR (mean 24.5%) vs. H≥4.0 (mean 14.0%). P2: Stratified Gains by Baseline Entropy. As shown in Figure 3, inputs with highly concentrated credit (H<3.5) see the largest HR reductions, aver- aging 9.8 p across all seven models. For inputs with already-distributed credit (H>4.5), reductions are smaller (avg. 2.4 p). This validates the prediction that SCR provides the greatest benefit when credit entropy is lowest. P3 and P4. Spatial autocorrelation dependence and the inverted-U λ rela- tionship are confirmed across all seven models (Appendix D). 6 Ablations and Limitations 6.1 Ablation Study Summary (a) Attention guidance. Uniform random source selection (same λ/K/layers) reduces HR gains from 4.6-6.0 p to 2.6-3.4 p (≈1.7× gap), isolating the contri- bution of credit-guided identification. (b) Topology. 8-connected (HR = 13.5%) Beyond Dominant Patches: SCR for Grounded Vision-Language Models11 Table 4: CHAIR on COCO captions (3,000 images, val2014). Lower CHAIR is better; higher CIDEr is better. Key baselines shown; full eight-baseline per-model comparison in Appendix B. Green = best per group;Blue = second best. SCR achieves the best CHAIR-s/i in five of seven configurations; on LLaVA-1.5-13B and InternVL2-7B, our CRoPS † reproduction achieves marginally lower CHAIR-s/i but degrades CIDEr by ≈4 p, versus SCR’s≤0.8 p. SCR is the only method that simultaneously achieves competitive hallucination reduction and near-lossless generation quality across all con- figurations. ModelMethod CHAIR-s↓ CHAIR-i↓ CIDEr↑ Cham-7B Vanilla12.3%8.5%102.3 VCD9.8%6.8%97.4 OA-VCD9.5%6.3%99.2 CRoPS † 9.1%5.8%98.2 SCR 7.1%4.8%101.7 Cham-30B Vanilla10.8%7.2%108.2 VCD8.5%5.5%103.8 OA-VCD8.2%5.6%105.2 CRoPS † 7.9%5.0%104.4 SCR 5.8%3.4%107.4 LLaVA-1.5-7B Vanilla10.2%7.1%108.5 VCD8.2%5.2%103.8 OA-VCD7.9%5.3%105.3 CRoPS † 7.6%5.3%104.5 SCR5.9%3.8%107.8 LLaVA-1.5-13B Vanilla9.1%6.3%114.8 VCD7.2%4.9%110.6 OA-VCD6.9%4.4%112.2 CRoPS † 4.5%3.0%110.8 SCR4.7%3.3%114.1 Qwen-VL Vanilla8.9%6.1%115.2 VCD7.0%4.7%111.2 OA-VCD6.7%4.2%112.5 CRoPS † 6.4%4.5%111.8 SCR 5.2%3.6%114.6 Qwen2-VL-7B Vanilla8.2%5.7%118.5 VCD6.5%4.5%114.6 OA-VCD6.2%4.1%115.4 CRoPS † 5.9%3.9%115.1 SCR4.0%2.5%117.9 InternVL2-7B Vanilla7.8%5.4%121.4 VCD6.3%4.2%117.2 OA-VCD5.9%3.9%118.5 CRoPS † 4.3%2.8%117.3 SCR 4.6%3.0%120.7 12N. H. Samin et al. Table 5: Extended benchmark evaluation (Vanilla→ SCR). MME-P = MME Percep- tion (↑); HBench = HallusionBench aAcc (%↑); AMBER-O/A/R = AMBER [34] ob- ject/attribute/relation hallucination rate (%↓). SCR yields strong object-hallucination reductions (AMBER-O: −4.1-5.7 p), moderate attribute gains (AMBER-A: −1.6- 2.3 p), and minimal relational change (AMBER-R: −0.7-1.4 p), consistent with its spatial-grounding scope. InternVL2-7B follows the same pattern, confirming general- ization to the latest model family. Model MME-P↑ HBench↑ AMBER-O↓ AMBER-A↓ AMBER-R↓ Base→SCR Base→SCR Base→SCR Base→SCR Base→SCR Cham-7B1285→1358 28.5→31.2 18.5→12.822.1→19.825.3→24.2 Cham-30B1342→1418 30.1→33.0 17.0→11.620.8→18.524.0→23.0 LLaVA-7B1510→1576 31.2→34.1 15.8→10.819.5→17.623.0→22.1 LLaVA-13B 1588→1642 34.8→37.4 14.5→10.217.8→15.921.5→20.8 Qwen-VL1548→1612 33.5→36.0 15.5→10.818.6→16.522.8→21.4 Qwen2-VL-7B 1620→1678 36.2→38.8 13.2→9.016.5→14.820.1→19.3 InternVL2-7B 1685→1742 38.4→41.2 12.5→8.415.8→14.219.2→18.5 Avg. ∆+64.0+2.8 −4.8−2.0−0.9 Table 6: Entropy-hallucination correlation (Pearson r) across all seven models. All correlations are negative and significant (p ≤ 0.001), confirming the link between low credit entropy and elevated hallucination rate across model families. ModelPearson r p-value Chameleon-7B −0.68 2.4×10 −5 Chameleon-30B −0.63 4.1×10 −4 LLaVA-1.5-7B −0.72 8.7×10 −6 LLaVA-1.5-13B −0.69 1.8×10 −5 Qwen-VL−0.61 6.3×10 −4 Qwen2-VL-7B −0.58 1.2×10 −3 InternVL2-7B −0.62 5.1×10 −4 Average−0.65- Beyond Dominant Patches: SCR for Grounded Vision-Language Models13 Low (H < 3.5)Medium (H [3.5, 4.5])High (H > 4.5) Baseline Credit Entropy 0 2 4 6 8 10 12 14 HR Reduction (p) Greatest benefit for low-entropy samples Diminishing returns Chameleon-7B Chameleon-30B LLaVA-1.5-7B LLaVA-1.5-13B Qwen-VL Qwen2-VL-7B InternVL2-7B Fig. 3: Entropy-stratified HR reduction across all seven model configurations. SCR provides the greatest benefit for low-entropy inputs (H < 3.5, avg. 9.8 p reduction) where credit is most collapsed, and diminishing gains for already-distributed credit (H > 4.5, avg. 2.4 p), consistent with Design Principle 1. outperforms 4-connected (16.5%) and radius-2 (18.2%) on Chameleon-7B in Figure 4a. Radius-2 performs worse than 4-connected because heavy overlap covers a significant portion of patches in a 16×16 grid (near-uniform smooth- ing, depending on K) and injects low-autocorrelation patches (ρ(2)≈0.51). 8- connected optimally captures diagonal correlations (ρ( √ 2)≈0.62) without over- spreading. (c) Signal vs. noise. Gaussian noise raises entropy to 5.2 nats yet degrades HR to 25.5% (Figure 4b), confirming gains require semantic structure. (d) Hyperparameters. Adaptive head selection intervenes on 20% of heads while retaining 95% of benefit; optimal: λ=1.10, K=32, layers 0-15. Full abla- tions in Appendix E. (e) Amplification vs. redistribution. Uniform-Scale scales all N v visual tokens by constant α chosen to match SCR’s ≈51% aggre- gate ℓ 2 norm increase, with no spatial redistribution. On Chameleon-7B this yields HR = 17.2% (−2.3 p)-below both Uniform-Smooth (−3.2 p) and SCR (−6.0 p)-confirming that targeted spatial redistribution, not aggregate norm am- plification, drives the gains (per-model results in Table E2, Appendix E). 6.2 Limitations Scope. The norm-credit proxy (r=0.72) and entropy-hallucination link (r=−0.65) are correlational, not causal; together they motivate but do not prove SCR. SCR targets spatial grounding; relational reasoning is outside the present scope. Failure modes. Three patterns dominate residual errors: (1) small objects (<2% area, 26% of errors), where redistribution dilutes weak signals; (2) edge- located objects (56%), where boundary exclusion removes valid high-attention 14N. H. Samin et al. (a) Neighbor topology: 8- connected (HR 13.5%) outper- forms 4-connected (16.5%) and radius-2 (18.2%), validating di- agonal inclusion. (b) Signal vs. noise: Gaussian noise raises entropy to 5.2 nats yet degrades HR to 25.5%, con- firming gains require semantic structure. (c) Head selectivity: interven- ing on only 20% of heads (5× reduction) retains 95% of the benefit (λ = 1.10, K = 32, lay- ers 0-15). Fig. 4: Ablation studies. (a) Neighbor topology comparison. (b) Semantic structure vs. entropy control. (c) Head-selectivity and computational cost trade-off. See ablation (a) in text for the Uniform-Smooth attention-guidance comparison (Tables 1-2). regions; (3) ambiguous neighbor pairs (18%), where amplifying similar neighbors triggers false positives. Trade-offs. SCR changes 15% of predictions; of these,≈70% are corrections (wrong→ right) and 30% new errors, yielding a net ≈6 p accuracy gain. 7 Conclusion SCR (Spatial Credit Redistribution) redistributes hidden-state activation from high-attention patches to 8-connected spatial neighbors at inference time, correcting early-layer over-concentration without modifying any model weights. Across seven configurations, SCR reduces hallucination on POPE (−4.6-6.0 p HR) and CHAIR (−41-51% relative) while preserving CIDEr within 0.8 p. On the joint (HR, CIDEr, latency) Pareto frontier, SCR dominates all baselines; on two stronger recent models (LLaVA-1.5-13B, InternVL2-7B), our CRoPS † reproduction [1] achieves marginally lower HR (0.2-0.4 p) at the cost of 3-4 p CIDEr degradation versus vanilla. The amortized diagnostic pass makes SCR 3-6× faster than OPERA [14]/VCD [16] at response lengths ≥10 tokens; for shorter responses DoLa [5] offers a lower fixed cost. MME, HallusionBench, and AMBER confirm gains are strongest for object-level hallucination and minimal for relations, consistent with SCR’s spatial-grounding scope (qualitative exam- ples in Appendix B). The Uniform-Smooth ablation isolates attention-guided selection as the essential driver; decoupling redistribution from norm amplifi- cation remains open (Section 6). Since SCR is training-free, it applies to any existing VLM; video-VLMs, adaptive patterns, and medical imaging are natural extensions. Code released upon acceptance. Supplementary: derivations (A), comparisons and qualitative examples (B), norm-credit (C), P3/P4 (D), ablations (E), One-Pass SCR (F). Beyond Dominant Patches: SCR for Grounded Vision-Language Models15 References 1. Anand, N., Jha, S., Bamba, U., Rahaman, R.: Crops: A training-free hallucination mitigation framework for vision-language models. arXiv preprint arXiv:2601.00659 (2026), https://arxiv.org/abs/2601.00659 2. Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond (2023), https://arxiv.org/abs/2308.12966 3. Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 24185–24198 (2024) 4. Christiano, P.F., Leike, J., Brown, T., Martic, M., Legg, S., Amodei, D.: Deep reinforcement learning from human preferences. Advances in neural information processing systems 30 (2017) 5. Chuang, Y.S., Xie, Y., Luo, H., Kim, Y., Glass, J., He, P.: DoLa: Decoding by contrasting layers improves factuality in large language models. In: The Twelfth International Conference on Learning Representations (2024) 6. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021) 7. Field, D.J.: Relations between the statistics of natural images and the response properties of cortical cells. Journal of the Optical Society of America A 4(12), 2379–2394 (1987) 8. Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al.: Mme: A comprehensive evaluation benchmark for multimodal large language models. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2025) 9. Gandikota, R., Feucht, S., Marks, S., Bau, D.: Erasing conceptual knowledge from language models. arXiv preprint arXiv:2410.02760 (2024) 10. Golovanevsky, M., Rudman, W., Palit, V., Singh, R., Eickhoff, C.: What do vlms notice? a mechanistic interpretability pipeline for gaussian-noise-free text-image corruption and evaluation. arXiv preprint arXiv:2406.16320 (2024) 11. Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., et al.: HallusionBench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. arXiv preprint arXiv:2310.14566 (2023) 12. Gupta, A., Ozdemir, A., Anumanchipalli, G.: Adversarial visual contrastive decod- ing for mitigating hallucinations in large vision-language models. arXiv preprint arXiv:2409.12951 (2024), https://arxiv.org/abs/2409.12951 13. Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al.: A survey on hallucination in large language mod- els: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43(2), 1–55 (2025) 14. Huang, Q., Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., Yu, N.: Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 13418–13427 (2024) 15. Huo, F., Xu, W., Zhang, Z., Wang, H., Chen, Z., Zhao, P.: Self-introspective de- coding: Alleviating hallucinations for large vision-language models. arXiv preprint arXiv:2408.02032 (2024) 16N. H. Samin et al. 16. Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., Bing, L.: Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 13872–13882 (2024) 17. Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International conference on machine learning. p. 19730–19742. PMLR (2023) 18. Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.R.: Evaluating object hal- lucination in large vision-language models. arXiv preprint arXiv:2305.10355 (2023) 19. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. p. 740–755. Springer (2014) 20. Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., Wang, L.: Mitigating hallucina- tion in large multi-modal models via robust instruction tuning. arXiv preprint arXiv:2306.14565 (2023) 21. Liu, H., Xue, W., Chen, Y., Chen, D., Zhao, X., Wang, K., Hou, L., Li, R., Peng, W.: A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253 (2024) 22. Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tun- ing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 26296–26306 (2024) 23. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36, 34892–34916 (2023) 24. Liu, S., Yang, S., Fang, D., Jia, S., Tang, Y., Su, L., Peng, R., Yan, Y., Zou, X., Hu, X.: Vision-language introspection: Mitigating overconfident hallucinations in mllms via interpretable bi-causal steering. arXiv preprint arXiv:2601.05159 (2026), https://arxiv.org/abs/2601.05159 25. Lyu, X., Chen, B., Gao, L., Shen, H., Song, J.: Alleviating hallucinations in large vision-language models through hallucination-induced optimization. Advances in Neural Information Processing Systems 37, 122811–122832 (2024) 26. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. Advances in neural information processing sys- tems 35, 27730–27744 (2022) 27. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. p. 8748–8763. PMLR (2021) 28. Rafailov, R., Sharma, A., Mitchell, E., Manning, C.D., Ermon, S., Finn, C.: Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems 36, 53728–53741 (2023) 29. Rohrbach, A., Hendricks, L.A., Burns, K., Darrell, T., Saenko, K.: Object halluci- nation in image captioning. arXiv preprint arXiv:1809.02156 (2018) 30. Simoncelli, E.P., Olshausen, B.A.: Natural image statistics and neural representa- tion. Annual Review of Neuroscience 24, 1193–1216 (2001) 31. Sun, Z., Shen, S., Cao, S., Liu, H., Li, C., Shen, Y., Gan, C., Gui, L., Wang, Y.X., Yang, Y., et al.: Aligning large multimodal models with factually augmented rlhf. In: Findings of the Association for Computational Linguistics: ACL 2024. p. 13088–13110 (2024) 32. Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: International conference on machine learning. p. 3319–3328. PMLR (2017) Beyond Dominant Patches: SCR for Grounded Vision-Language Models17 33. Team, C.: Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818 (2024) 34. Wang, J., Wang, Y., Xu, G., Zhang, J., Gu, Y., Jia, H., Yan, M., Zhang, J., Sang, J.: AMBER: An LLM-free multi-dimensional hallucination evaluation benchmark for large vision-language models. arXiv preprint arXiv:2311.07397 (2024) 35. Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al.: Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024) 36. Xiao, G., Tian, Y., Chen, B., Han, S., Lewis, M.: Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453 (2023) 37. Zhang, B., Sennrich, R.: Root mean square layer normalization. Advances in neural information processing systems 32 (2019) 38. Zhao, Z., Wang, B., Ouyang, L., Dong, X., Wang, J., He, C.: Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839 (2023)