Paper deep dive
Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilization
Aditi Sarker, Rafi Ibn Sultan, Hui Zhu, Dongxiao Zhu, Prashant Khanduri
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/15/2026, 7:36:37 AM
Summary
The paper introduces INFUSE, a framework for mitigating hallucinations in Large Vision-Language Models (LVLMs) through implicit feature stabilization. Unlike explicit methods that intervene at inference time, INFUSE builds perturbation-invariance into model weights during fine-tuning. It employs a two-stage process: Stage 1 stabilizes visual and textual representations using perturbation-averaged anchors and contrastive objectives, while Stage 2 aligns these stabilized representations across modalities. Theoretical proofs show that anchor variance shrinks with more views, bounding hallucination behavior under Lipschitz decoders. Experiments on LLaVA-1.5, LLaVA-1.6, and Qwen3-VL-8B-Instruct demonstrate significant reductions in hallucination metrics (AMBER CHAIR, ObjHal, MMHal, HallusionBench, POPE) without inference-time overhead.
Entities (18)
Relation Signals (13)
INFUSE → evaluatedon → Qwen3-VL-8B-Instruct
confidence 95% · On LLaVA-1.5, LLaVA-1.6, and Qwen3-VL-8B-Instruct, INFUSE reduces AMBER CHAIR...
INFUSE → evaluatedon → LLaVA-1.5
confidence 95% · On LLaVA-1.5, LLaVA-1.6, and Qwen3-VL-8B-Instruct, INFUSE reduces AMBER CHAIR...
INFUSE → evaluatedon → LLaVA-1.6
confidence 95% · On LLaVA-1.5, LLaVA-1.6, and Qwen3-VL-8B-Instruct, INFUSE reduces AMBER CHAIR...
INFUSE → mitigates → Hallucination
confidence 95% · Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilization
INFUSE → uses → Implicit Feature Stabilization
confidence 95% · We propose implicit stabilization instead: perturbation-invariance is built into the model weights during fine-tuning
INFUSE → improves → MMHal
confidence 92% · improves ObjHal, MMHal, HallusionBench, and POPE
INFUSE → improves → POPE
confidence 92% · improves ObjHal, MMHal, HallusionBench, and POPE
INFUSE → improves → ObjHal
confidence 92% · improves ObjHal, MMHal, HallusionBench, and POPE
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Vision-Language Models (LVLMs) are prone to hallucinations: they fluently describe objects, attributes, and scenes that are not in the image. We connect part of this failure to a measurable property of their representations, feature instability, where mild semantics-preserving perturbations of the input cause large changes in the learned embeddings; hallucination rates rise together with this variability. Existing stability-motivated remedies are explicit, in the sense that they intervene at inference time through latent steering or constrained decoding, and pay for it on every query. We propose implicit stabilization instead: perturbation-invariance is built into the model weights during fine-tuning, and nothing extra runs at deployment. Our framework, INFUSE, first stabilizes visual and textual representations around perturbation-averaged and ground-truth anchors, then aligns the stabilized representations across modalities with bidirectional contrastive objectives. We prove that the anchor's root-mean-square deviation from the perturbation-mean representation shrinks at rate $1/\sqrt{K}$ in the number of views, and that under a Lipschitz decoder, this bounds how much any perturbation can change the model's hallucination behavior. On LLaVA-1.5, LLaVA-1.6, and Qwen3-VL-8B-Instruct, INFUSE reduces AMBER CHAIR by 46-63% relative to each base model, improves ObjHal, MMHal, HallusionBench, and POPE, and preserves VQA-v2 and TextVQA, all with no inference-time overhead.
Tags
Links
- Source: https://arxiv.org/abs/2608.29924v1
- Canonical: https://arxiv.org/abs/2608.29924v1
Trouble viewing inline? Open PDF directly →
Full Text
71,441 characters extracted from source content.
Expand or collapse full text
Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilization Aditi Sarker† Rafi Ibn Sultan† Hui Zhu† Dongxiao Zhu and Prashant Khanduri†Department of Computer Science, Wayne State University, MI, USA∗Institute for AI and Data Science (AIDaS), Wayne State University, MI, USAEmail: aditi1, rafis, hui, dzhu, khanduri.prashant@wayne.edu Abstract Large Vision-Language Models (LVLMs) are prone to hallucinations: they fluently describe objects, attributes, and scenes that are not in the image. We connect part of this failure to a measurable property of their representations, feature instability, where mild semantics-preserving perturbations of the input cause large changes in the learned embeddings; hallucination rates rise together with this variability. Existing stability-motivated remedies are explicit, in the sense that they intervene at inference time through latent steering or constrained decoding, and pay for it on every query. We propose implicit stabilization instead: perturbation-invariance is built into the model weights during fine-tuning, and nothing extra runs at deployment. Our framework, INFUSE, first stabilizes visual and textual representations around perturbation-averaged and ground-truth anchors, then aligns the stabilized representations across modalities with bidirectional contrastive objectives. We prove that the anchor’s root-mean-square deviation from the perturbation-mean representation shrinks at rate 1/K1/ K in the number of views, and that under a Lipschitz decoder, this bounds how much any perturbation can change the model’s hallucination behavior. On LLaVA-1.5, LLaVA-1.6, and Qwen3-VL-8B-Instruct, INFUSE reduces AMBER CHAIR by 46-63% relative to each base model, improves ObjHal, MMHal, HallusionBench, and POPE, and preserves VQA-v2 and TextVQA, all with no inference-time overhead. 1 Introduction Large Vision-Language Models (LVLMs) couple pre-trained vision encoders with large language models and now perform well across captioning, visual question answering (VQA), and instruction following [1, 2, 3], which has pushed them into settings where reliability matters [4, 5]. They remain prone to hallucinations: fluent, plausible responses unsupported by the visual evidence (Figure 1). The failure mode is well documented on dedicated benchmarks such as POPE, AMBER, MMHal-Bench, and HallusionBench [6, 7, 8, 9], and surveyed at length [10, bai2024hallucination, 11, 12]. Figure 1: Object-existence hallucination. The LVLM incorrectly describes a non-existent chair in an image containing only a sofa. Feature instability and hallucination. Our starting point is an empirical observation about LVLM representations. Semantically equivalent observations of a scene should produce consistent embeddings. In practice, they do not: existing LVLMs produce substantially different visual features under controlled input perturbations such as additive noise, masking, or partial region removal. Figure 2 quantifies this on LLaVA-1.6. Embedding variance grows monotonically with perturbation strength (a), and hallucination rates grow under the same perturbations (b). Correlation alone does not establish causation. What we add is interventional evidence: when we directly train the representations to be perturbation-invariant (Stage 1 [S1], previewed in the same figure and ablated in Table 6), both embedding variance and hallucination rates drop substantially, before any cross-modal objective is applied. Taken together, the two observations support treating representation stability as an actionable training target, complementing the well-studied role of cross-modal misalignment [13, 14, 15]. Implicit vs. explicit stabilization. Stability-motivated remedies exist, but current ones are explicit: they impose stability from outside the trained model at inference time. Latent-space steering (VTI) [16] shifts visual activations along pre-computed directions at every query. Contrastive and constrained decoding methods (VCD, OPERA) [17, 18] correct instability’s downstream symptoms during generation. These interventions leave the underlying representation space unchanged and pay a cost on every query. We take the opposite route: build perturbation-invariance into the weights during fine-tuning, so the deployed model is stable without any intervention during inference. Training-time hallucination methods to date, such as supervised fine-tuning, RLHF, preference optimization, and contrastive alignment [19, 20, 13, 14, 15, 21], optimize output behavior or cross-modal correspondence; none optimizes intra-modal representation stability directly during training. Conversely, the methods that do target stability act only at inference time. Our approach. We propose INFUSE (Implicit Feature Stabilization and Alignment), a two-stage training framework. Stage 1 ([S1]) stabilizes representations within each modality: visual embeddings are pulled toward perturbation-averaged anchors whose estimation error contracts at rate 1/K1/ K in the number of views (Proposition 3.1), and generated-text embeddings are pulled toward human-corrected truthful-response anchors and repelled from hallucinated captions. Stage 2 ([S2]) then aligns the stabilized representations across modalities with bidirectional contrastive objectives plus a standard generation loss. The ordering is deliberate. Alignment acts on representations that are already low-variance, so positive pairs are matched on semantics rather than on appearance noise. Our contributions are: 1. We articulate the distinction between implicit (training-time, weight-level) and explicit (inference-time, intervention-level) feature stabilization for LVLMs, present correlational and interventional evidence that instability is an actionable factor in hallucination, and give a formal justification: a variance-contraction guarantee for perturbation-averaged anchors and a Lipschitz argument connecting embedding stability to stability of the output (Proposition 3.1). 2. We instantiate implicit stabilization in INFUSE, with fully specified anchor constructions (Algorithm 1), an asymmetric stop-gradient design related to but distinct from BYOL/SimSiam-style self-distillation [22, 23], and a stabilize-then-align schedule. 3. Across LLaVA-1.5-7B, LLaVA-1.6-7B, and Qwen3-VL-8B-Instruct, INFUSE reduces AMBER CHAIR by 46-63% relative to the corresponding base models, improves ObjHal, MMHal, HallusionBench, and POPE, and preserves or improves VQA-v2 and TextVQA, at lower training cost than the strongest training-time baseline and with nothing extra running at inference. (a) Embedding variance. (b) Hallucination rate. Figure 2: Sensitivity to image perturbations (LLaVA-1.6). (a) The baseline model’s embedding variance grows with perturbation strength; training with our [S1] objective yields lower variance. (b) Hallucination rates evaluated on a fixed smaller AMBER subset increase under the same perturbations, while [S1] consistently reduces them. Baseline curves motivate the approach; [S1] curves preview the effect of intervention, quantified in Table 6. 2 Related Work Hallucination in LVLMs. Modern LVLMs [1, 24, 2, 25, 26, 3] remain vulnerable to hallucination despite strong benchmark performance; surveys [11, 12, 27, 28] catalogue types and causes. Mitigation approaches [29, 30, 31, 32, 33, 34, 35, 36] divide into inference-time and training-time families. Explicit (inference-time) stabilization and correction. Inference-time methods mitigate hallucinations without updating model parameters through decoding refinement, response verification, attention intervention, or post-hoc correction [37, 18, 35, 38, 17, 31, 32, 36]. Closest to our motivation is VTI [16], which attributes hallucinations to unstable visual features and stabilizes them during inference. In contrast, INFUSE learns stable representations during training, eliminating the need for per-query intervention. On the same LLaVA-1.5 backbone, INFUSE outperforms VTI on adversarial POPE (Table 4). More generally, inference-time methods improve generation without explicitly strengthening the underlying representations and often incur additional inference cost [39, 16]. Implicit (training-time) mitigation. Training-based methods reduce hallucinations by improving supervision, preference alignment, or cross-modal learning. Representative approaches include contrastive learning and object-level supervision [13], RLHF-based methods such as HALVA [15], RLHF-V [19], and Fact-RLHF [8], preference optimization methods including DPO [20, 40], CHiP [14], Antidote [41], and recent preference learning approaches [34], as well as textual embedding refinement [30]. While these methods improve supervision or image–text alignment, they do not explicitly learn perturbation-invariant visual and textual representations, leaving feature instability largely unaddressed. In contrast, INFUSE first stabilizes visual and textual features before cross-modal alignment. Our method constructs perturbation-averaged visual anchors with lagged-parameter repulsion and ground-truth textual anchors with hallucinated-caption negatives, drawing inspiration from asymmetric self-supervised learning methods such as BYOL [22] and SimSiam [23]. Figure 3: Overview of INFUSE. [S1] Intra-modal stabilization establishes perturbation-invariant anchors within each modality: perturbed views of each image are encoded, projected, and averaged into a low-variance visual anchor toward which the current embedding is pulled (and pushed from its lagged embedding); the ground-truth caption embedding anchors the textual branch, with hallucinated captions as negatives. [S2] Inter-modal alignment aligns the stabilized representations bidirectionally (V→ and T→ ), pulling paired image-text embeddings together and pushing unpaired combinations apart. 3 The Proposed INFUSE Framework As illustrated in Figure 3, INFUSE first learns stable representations within each modality ([S1], intra-modal stabilization) and then aligns the stabilized representations across modalities ([S2], cross-modal alignment), together with a standard generation objective that preserves generated language quality. Problem setup. Let (vi,ti)i=1N\(v_i,t_i)\_i=1^N be image-caption pairs. An LVLM comprises a frozen vision encoder VθV_θ, a multimodal projector FαF_α, and a language model LβL_β; following standard practice, only (α,β)(α,β) are trained. Given viv_i, the model generates t^i=Lβ(Fα(Vθ(vi))) t_i=L_β(F_α(V_θ(v_i))). For a text sequence t, we write Lβ(t)∈ℝdL_β(t) ^d for the final-token hidden state of the decoder, used as a sentence-level embedding following [16, 13, 14]. 3.1 [S1]: Intra-Modal Stabilization Vision-to-vision (V→ ) stabilization. A single image is one observation of a scene; it cannot tell us whether the representation of that scene is stable. Controlled perturbations provide multiple views of the same input while altering portions of its visual evidence. A well-grounded model should embed them consistently, and large dispersion among views indicates reliance on cues that the perturbations disturb. We therefore construct a low-variance anchor by view averaging and train the representation toward it. Definition 3.1 (Perturbation-averaged visual anchor). For image viv_i, draw K i.i.d. controlled perturbations Ck(vi)k=1K\C_k(v_i)\_k=1^K (random masking in our implementation), embed each view as zikv≔Fα(Vθ(Ck(vi)))z_ik^v F_α(V_θ(C_k(v_i))), and define the anchor z¯iv≔1K∑k=1Kzikv z_i^v 1K _k=1^Kz_ik^v, treated as a constant (stop-gradient) in the loss. Definition 3.2 (Lagged reference embedding). At optimization step s, the lagged embedding is the current image’s embedding under the parameters of the previous step, ziv≔sg[Fα(s−1)(Vθ(vi))]z_i^v \! [F_α^(s-1)(V_θ(v_i)) ], where sg[⋅]sg[·] denotes stop-gradient. It represents the pre-update, potentially perturbation-sensitive state of the representation. The current (learnable) embedding is ziv+≔Fα(Vθ(vi))z_i^v^+ F_α(V_θ(v_i)), with gradients through the projector. The V→ objective is ℒV→V≔−1N∑i=1Nlogesim(ziv+,z¯iv)/τvvesim(ziv+,z¯iv)/τvv+esim(ziv+,ziv)/τvv,L_V→ V\! \!- 1N _i=1^N\! e^sim(z_i^v^+, z_i^v)/ _ve^sim(z_i^v^+, z_i^v)/ _v+e^sim(z_i^v^+,z_i^v)/ _v, (1) where sim(⋅,⋅)sim(·,·) is cosine similarity and τvv _v a temperature. Why repel from the lagged embedding? The repulsion term is an anti-collapse and escape mechanism, not a semantic judgment about the previous state. With the anchor held constant, the attraction term alone admits degenerate optima in which the projector barely moves. Repelling from the lagged embedding breaks this symmetry: each update must move measurably away from the pre-update state and toward the low-variance anchor. The construction belongs to the same family as the predictor and stop-gradient asymmetries that prevent collapse in BYOL [22] and SimSiam [23], with an explicit repulsive term in place of their architectural asymmetry. Nor can the repulsion discard grounded semantics wholesale, since the attraction target z¯iv z_i^v is built from views of the same image: the optimization moves within the image’s semantic neighborhood, away only from its unstable directions. Text-to-text (T→ ) stabilization. For each viv_i we use the human-corrected truthful response tit_i and a hallucinated (rejected) caption t~i t_i, with embeddings zit≔Lβ(ti)z_i^t L_β(t_i) and z~it≔Lβ(t~i) z_i^t L_β( t_i). Let zit+≔Lβ(t^i)z_i^t^+ L_β( t_i) denote the embedding of the model’s generated response t^i t_i; gradients flow through the decoder’s hidden states on the generated token sequence (token identities are held fixed), so the objective shapes the representation of what the model actually generates. The generated embedding is the anchor of the contrastive comparison. It is pulled toward the ground-truth embedding and repelled from hallucinated-caption embeddings across the batch, ℒT→T≔−1N∑i=1Nlogesim(zit+,zit)/τttesim(zit+,zit)/τtt+∑j=1Nesim(zit+,z~jt)/τtt.L_T→ T\! \!- 1N\! _i=1^N\! e^sim(z_i^t^+,\,z_i^t)/ _te^sim(z_i^t^+\!,z_i^t)/ _t+ _j=1^Ne^sim(z_i^t^+\!, z_j^t)/ _t. (2) Here, zitz_i^t acts as a fixed truthful anchor pulling the generated embedding toward grounded semantics, while hallucinated embeddings push it away from hallucination-prone regions. Algorithm 1 INFUSE Stage 1 (intra-modal stabilization) 1: pairs (vi,ti,t~i)\(v_i,t_i, t_i)\, frozen VθV_θ, projector FαF_α, decoder LβL_β, views K, temperatures τvv,τtt _v, _t 2: for each optimization step s=1,2,…s=1,2,… do 3: for each i in the batch do 4: draw views Ck(vi)k=1K\C_k(v_i)\_k=1^K; z¯iv←1K∑kFα(s)(Vθ(Ck(vi))) z_i^v← 1K _kF_α^(s)(V_θ(C_k(v_i))) ⊳ stop-grad 5: ziv←sg[Fα(s−1)(Vθ(vi))]z_i^v [F_α^(s-1)(V_θ(v_i))]; ziv+←Fα(s)(Vθ(vi))z_i^v^+← F_α^(s)(V_θ(v_i)) 6: zit←sg[Lβ(ti)]z_i^t [L_β(t_i)]; z~it←sg[Lβ(t~i)] z_i^t [L_β( t_i)]; zit+←Lβ(t^i)z_i^t^+← L_β( t_i) 7: end for 8: update α with ∇αℒV→V _αL_V→ V (Equation 1); update β with ∇βℒT→T _βL_T→ T (Equation 2) 9: end for Overall, [S1] optimizes minαℒV→V+minβℒT→T _αL_V→ V+ _βL_T→ T, yielding stable embeddings in both modalities that serve as reliable references for [S2]. 3.2 Theoretical Justification Two questions deserve an answer: why is a perturbation-averaged anchor the right training target, and how does embedding stability transfer to the model’s outputs? Proposition 3.1 (Anchor contraction and output stability). Fix an image v and let zk=Fα(Vθ(Ck(v)))z_k=F_α(V_θ(C_k(v))), k=1,…,Kk=1,…,K, be embeddings of i.i.d. semantics-preserving views with common mean μv=[z1] _v=E[z_1] (the perturbation-mean representation of v) and Cov[z1]⪯σ2IdCov[z_1] σ^2I_d. Then the anchor z¯=1K∑kzk z= 1K _kz_k satisfies ‖z¯−μv‖22≤dσ2K.E\,\| z- _v\|_2^2\;≤\; d\,σ^2K. Suppose further the decoder’s conditional output distribution is L-Lipschitz in the visual embedding under total variation, i.e., TV(pβ(⋅∣z),pβ(⋅∣z′))≤L∥z−z′∥2TV\! (p_β(· z),\,p_β(· z ) )≤ L\|z-z \|_2. Then for any set ℋH of hallucinatory responses, |Prpβ(⋅∣z)[ℋ]−Prpβ(⋅∣μv)[ℋ]|≤L∥z−μv∥2. |\, _p_β(· z)[H]- _p_β(· _v)[H]\, |\;≤\;L\,\|z- _v\|_2. Proof. The first bound is the variance of an i.i.d. mean: ‖z¯−μv‖22=1K2∑ktr(Cov[zk])≤dσ2KE\| z- _v\|_2^2= 1K^2 _ktr(Cov[z_k])≤ d\,σ^2K. For the second, |Prpβ(⋅∣z)[ℋ]−Prpβ(⋅∣μv)[ℋ]|≤TV(pβ(⋅∣z),pβ(⋅∣μv))≤L∥z−μv∥2| _p_β(· z)[H]- _p_β(· _v)[H]| (p_β(· z),p_β(· _v))≤ L\|z- _v\|_2, by the definition of total variation and the Lipschitz assumption. ∎ The first inequality says that the anchor’s root-mean-square deviation from the perturbation-mean representation μv _v contracts at rate 1/K1/ K. Averaging views is therefore a statistically principled way to obtain a lower-variance estimate of the perturbation-mean representation as K increases. Empirically, performance generally improves with larger K, with diminishing returns at higher values, as reported in the supplementary material. The second inequality says that once embeddings concentrate near μv _v, an input perturbation can change the probability of any hallucinatory output by at most L times the embedding displacement it causes. The proposition should be read for what it establishes: a bound on how much perturbations can shift the model’s hallucination behavior, not a guarantee that hallucination rates fall. That they do fall under stabilization is an empirical result, shown by the ablations in Table 6. As for the Lipschitz assumption, it is a mild smoothness condition on the decoder’s conditional distribution over the continuous projected embedding; softmax decoders with bounded logit gradients satisfy it on any bounded embedding domain. 3.3 [S2]: Cross-Modal Alignment Stable intra-modal representations are necessary but not sufficient. Reliable generation also requires accurate correspondence between modalities, since LVLMs connect pre-trained encoders and decoders independently through a light projector [25, 2, 3]. That architecture creates a representation gap; INFUSE closes it through training rather than architectural modification, and does so after stabilization, so that alignment matches semantics rather than appearance noise. For each pair (vi,ti)(v_i,t_i), we take uiv=Lβ(Fα(Vθ(vi)))u_i^v=L_β(F_α(V_θ(v_i))) and uit=Lβ(ti)u_i^t=L_β(t_i) (final decoder hidden states) and optimize a bidirectional InfoNCE objective in the style of CLIP [1, 13, 17]: ℒV→T≔−1N∑i=1Nlogesim(uiv,uit)/τvt∑j=1Nesim(uiv,ujt)/τvt,L_V→ T - 1N _i=1^N e^sim(u_i^v,u_i^t)/ _vt _j=1^Ne^sim(u_i^v,u_j^t)/ _vt, (3) ℒT→V≔−1N∑i=1Nlogesim(uit,uiv)/τtv∑j=1Nesim(uit,ujv)/τtv.L_T→ V - 1N _i=1^N e^sim(u_i^t,u_i^v)/ _tv _j=1^Ne^sim(u_i^t,u_j^v)/ _tv. (4) [S2] jointly optimizes (α,β)(α,β) with minα,β[ℒV→T+ℒT→V+ℒG] _α,β\,[L_V→ T+L_T→ V+L_G], where ℒGL_G is the standard supervised generation loss preserving fluency and downstream ability. Distinct roles of Equation (2) and Equations (3)-(4). Though contrastive in form, the two stages act on different objects with different negatives. Equation (2) is intra-textual: it shapes the embedding of the model’s own generated text against truthful and hallucinated caption directions, with no visual term. Equations (3)-(4) are cross-modal, matching image embeddings to caption embeddings with unpaired-sample negatives. Figure 4 shows that the V→ and T→ stabilization branches are complementary, with their combination in [S1] yielding lower hallucination rates than either branch alone. Table 6 further shows that adding [S2] after [S1] provides additional improvements, supporting the distinct and complementary roles of intra-modal stabilization and cross-modal alignment. Implementation. We use K=100K=100 masked views and temperature τ=0.07τ=0.07 for all contrastive objectives, and train [S1] and [S2] for 5 epochs each. Sensitivity analyses over τ and K are reported in Table 8 and Table 9, respectively, while Figure 2 examines the effect of varying perturbation strength. 4 Experimental Evaluation We evaluate INFUSE on LLaVA-1.5-7B [26], LLaVA-1.6-7B [3], and Qwen3-VL-8B-Instruct [42], covering CLIP+Vicuna [1, 43] and Qwen-family [44, 42] architectures. Training data. Following prior preference-alignment work, we draw training data from RLHF-V [19]. Each selected item contains an image and a prompt with a preferred (chosen) response and a rejected response. We use the preferred response as the grounded textual target tit_i and the rejected response as the negative response t~i t_i. INFUSE is trained on a 2K-sample subset. Baselines and comparison protocol. We separate two kinds of comparison and label them explicitly in Table 1. Controlled comparisons are methods we re-ran under our exact setting (same backbone, training data, and evaluation protocol): the base models and LLaVA-1.6-7B + DPO [20], CHiP [14] (LLaVA-1.6). Literature-reported results (marked ⋄) are numbers taken from the original publications: RLHF-V [19] (Muffin [45] architecture with BEiT-3 [46] and Vicuna-13B), HALVA [15] (LLaVA-1.5), which use their own training data and protocols. They contextualize our results but are not like-for-like, and we confine our strongest claims to the controlled block. We additionally compare against inference-time methods (Regular decoding, VCD [17], OPERA [18], VTI [16]) on POPE. 4.1 Results Hallucination benchmarks. Table 1 reports results on ObjHal, MMHal, HallusionBench, and AMBER (generative). Across all three backbones, INFUSE consistently reduces hallucinations while maintaining competitive performance across complementary metrics. Compared with the corresponding base models, it improves all 11 metrics on both LLaVA-1.6 and Qwen3-VL, and all but one metric on LLaVA-1.5, where HallusionBench fA decreases slightly (24.9→ 23.1). Under controlled comparisons, INFUSE outperforms DPO on 10 of the 11 metrics, with DPO retaining only a small advantage on HallusionBench fA (26.7 vs. 25.7). Compared with CHiP, evaluated under the same backbone, training data, and evaluation protocol, INFUSE achieves better performance on 9 of the 11 metrics, with CHiP outperforming INFUSE only on HallusionBench aA and AMBER Cog. The largest gains are observed on AMBER, where INFUSE reduces CHAIR from 7.8 to 4.2 (46%) on LLaVA-1.5, from 8.3 to 3.1 (63%) on LLaVA-1.6, and from 5.9 to 2.9 (51%) on Qwen3-VL. Similar improvements are observed on ObjHal, where both response and mention rates are consistently reduced, and on MMHal, where INFUSE achieves the highest overall scores across all three backbones. Overall, these results demonstrate that the proposed training strategy generalizes effectively across diverse LVLM architectures. Fine-grained MMHal analysis. Table 2 reports the GPT-4-based fine-grained evaluation on MMHal-Bench. Compared with existing methods, INFUSE consistently improves overall performance while substantially reducing hallucinations. Specifically, INFUSE with LLaVA-1.6 achieves the highest overall GPT-4 score (3.87), whereas INFUSE with Qwen3-VL-8B-Instruct attains the lowest hallucination rate (15.12%) and achieves the best performance in seven of the eight fine-grained evaluation categories. These consistent improvements across diverse reasoning types demonstrate that INFUSE enhances visual grounding and mitigates hallucinations in a broad range of multimodal reasoning scenarios. Fine-grained discriminative analysis. Table 3 reports AMBER’s discriminative evaluation, including attribute-level (F1A) and relation-level (F1R) scores that probe finer-grained grounding than object existence. A natural worry about our design is that final-token anchors sacrifice fine-grained detail. The discriminative results say otherwise: for the backbones with corresponding base rows (LLaVA-1.6 and Qwen3-VL), INFUSE improves both F1A and F1R. In addition, on LLaVA-1.6, INFUSE improves F1R relative to the base model, whereas CHiP reduces it (62.0 vs. 69.6 for the base model). Token- and region-level extensions may capture still finer structure, which we discuss in the Limitations. Comparison to explicit (inference-time) stabilization. Table 4 compares INFUSE with representative inference-time mitigation methods on the POPE adversarial split. On the shared LLaVA-1.5 backbone, INFUSE outperforms VTI, improving accuracy from 82.57% to 83.56% and F1 from 82.11 to 83.21, while also achieving higher precision and recall. With stronger backbones, INFUSE reaches 88.25% accuracy with LLaVA-1.6 and 90.40% with Qwen3-VL-8B, with the latter achieving the best overall F1 of 90.15. General vision-language understanding. Table 5 evaluates whether INFUSE preserves general multimodal capabilities on VQAv2 [47] and TextVQA [48]. Across all three backbones, INFUSE matches or improves the corresponding base models, achieving the best performance on LLaVA-1.6 and Qwen3-VL-8B. These results show that hallucination mitigation does not come at the expense of general vision-language understanding. Table 1: Hallucination evaluation on four benchmarks. ⋄ = literature-reported (original paper’s backbone, data, and protocol; see text). Best and second-best are highlighted in green and orange. Model ObjHal† MMHal‡ HallusionBench§ AMBER¶ R.↓ M.↓ Overall↑ R.↓ qA↑ fA↑ aA↑ CHAIR↓ Cover↑ Hal↓ Cog↓ Base models (re-run) LLaVA-1.5-7B [26] 46.3 22.6 2.4 52.1 10.6 24.9 46.9 7.8 51.0 36.4 4.2 LLaVA-1.6-7B [3] 14.1 7.4 2.8 42.7 15.8 20.8 51.6 8.3 61.0 48.6 4.2 Qwen3-VL-8B-Instruct [42] 11.1 6.3 2.8 38.5 18.6 26.4 68.0 5.9 62.3 26.5 3.1 Literature-reported⋄ RLHF-V⋄ [19] 12.2 7.5 2.5 51.0 – – – 6.3 46.1 25.1 2.1 HALVA⋄ [15] – – – – 13.9 20.1 49.1 – – – – Controlled comparison (re-run, same data and protocol) LLaVA-1.6-7B + DPO 14.0 7.1 2.3 46.2 19.4 26.7 53.2 6.3 57.1 41.4 3.6 LLaVA-1.6-7B + CHiP [14] 5.8 3.9 3.1 40.2 21.3 22.0 56.8 3.9 55.4 25.7 1.8 INFUSE (ours) INFUSE + LLaVA-1.5-7B 5.1 3.7 3.78 18.75 39.1 23.1 49.7 4.2 61.1 28.2 2.1 INFUSE + LLaVA-1.6-7B 4.87 3.4 3.87 16.02 41.0 25.7 56.3 3.1 68.1 24.3 2.9 INFUSE + Qwen3-VL-8B-Instruct 4.13 3.2 3.82 15.12 48.1 32.0 72.1 2.9 67.4 13.0 1.0 † ObjHal-COCO2017 [49, 50]. ‡ MMHal [8]. § HallusionBench [9]. ¶ AMBER [7]. Table 2: Fine-grained GPT-4 evaluation results on MMHal-Bench (MMHal) [8]. Best and second-best results are highlighted in green and orange, respectively. ↑ indicates higher is better, while ↓ indicates lower is better. Model MMHal-Bench Overall↑ Hallu.↓ Attr.↑ Adv.↑ Comp.↑ Count.↑ Rel.↑ Env.↑ Hol.↑ Other↑ Base models and existing methods LLaVA [3] 2.78 42.71 3.75 3.50 3.50 1.50 1.92 4.08 1.75 2.83 LLaVA + DPO 2.3 46.2 4.17 2.92 3.00 2.67 2.67 4.25 1.17 1.00 LLaVA + CHiP [14] 3.1 40.2 4.17 3.33 2.67 2.67 2.25 4.08 1.67 1.92 VTI [16] 2.90 51.00 3.75 2.42 3.50 2.42 2.83 3.50 2.42 2.33 INFUSE (ours) INFUSE + LLaVA-1.5 3.78 18.75 3.67 3.58 3.50 2.83 2.58 4.07 1.71 2.83 INFUSE + LLaVA-1.6 3.87 16.02 3.71 3.61 3.52 2.92 2.57 4.12 3.42 3.58 INFUSE + Qwen3-VL-8B-Instruct 3.82 15.12 3.76 3.73 3.57 3.50 2.87 4.46 3.50 3.72 Model F1↑ F1E↑ F1A↑ F1R↑ LLaVA-1.6 (7B) 87.0 95.1 81.5 69.6 Qwen3-VL-8B 85.4 94.2 82.5 70.8 LLaVA-1.6 + CHiP 86.9 98.3 80.3 62.0 INFUSE + LLaVA-1.5 87.1 95.5 81.6 70.1 INFUSE + LLaVA-1.6 88.6 98.5 82.1 71.0 INFUSE + Qwen3-VL-8B 91.0 97.8 84.0 78.0 Table 3: Discriminative results on AMBER. Green and orange denote the best and second-best results. Method Acc. Prec. Rec. F1 LLaVA-1.5 Regular⋄ 76.03 76.11 76.80 76.45 LLaVA-1.5 + OPERA⋄ 80.88 82.16 79.76 80.94 LLaVA-1.5 + VCD⋄ 77.31 73.43 86.47 79.42 LLaVA-1.5 + VTI⋄ 82.57 84.33 80.01 82.11 INFUSE + LLaVA-1.5 83.56 85.00 81.50 83.21 INFUSE + LLaVA-1.6 88.25 92.69 83.06 87.61 INFUSE + Qwen3-VL-8B 90.40 92.55 87.88 90.15 Table 4: Comparison with inference-time mitigation methods on POPE. Green and orange denote the best and second-best results. Model VQA-v2 TextVQA LLaVA-1.5 (Base) 76.2 56.4 LLaVA-1.6 (Base) 80.2 60.9 Qwen3-VL-8B (Base) 78.4 61.4 INFUSE + LLaVA-1.5 76.8 57.0 INFUSE + LLaVA-1.6 81.0 61.5 INFUSE + Qwen3-VL-8B 78.8 62.1 Table 5: General vision-language understanding. Green and orange denote the best and second-best results. Ablation study. We evaluate the contribution of each component in INFUSE. Figure 4 shows that both V→ and T→ stabilization independently reduce hallucination rates on the AMBER discriminative benchmark and MMHal, while their combination ([S1]) achieves the lowest hallucination rates. Table 6 further demonstrates that these gains translate to improved generative performance on AMBER, with [S1] consistently outperforming the corresponding baselines and [S2] providing additional improvements. Training cost. Table 7 reports aggregate GPU-hours, computed as the number of GPUs multiplied by wall-clock training time. For this cost comparison, CHiP is evaluated using its official 5K+ training recipe, whereas INFUSE uses our 2K training set. CHiP requires 12.5 GPU-hours, compared with 6.2 GPU-hours for INFUSE. These values therefore compare the computational cost of the respective training recipes and should not be confused with the controlled comparison in Table 1, where CHiP is re-run using our matched training-data setting. Figure 4: [S1] Ablation on hallucination rate (%). V→ and T→ each reduce hallucinations, while their combination achieves the lowest rates on AMBER and MMHal for LLaVA-1.6-7B and Qwen3-VL-8B-Instruct. Table 6: Ablation study. [S1]: ℒV→V+ℒT→TL_V→ V+L_T→ T; [S2]: ℒV→T+ℒT→V+ℒGL_V→ T+L_T→ V+L_G. green = best. ↑ higher is better, ↓ lower is better. Model CHAIR↓ Cover↑ Hal↓ Cog↓ LLaVA-1.5-7B (base) 7.8 51.0 36.4 4.2 LLaVA-1.6-7B (base) 8.3 61.0 48.6 4.2 Qwen3-VL-8B-Instruct (base) 5.9 62.3 26.5 3.1 LLaVA-1.6-7B + DPO 6.3 57.1 41.4 3.6 LLaVA-1.6-7B + CHiP 3.9 55.4 25.7 1.8 INFUSE (LLaVA-1.5-7B backbone) INFUSE [S1]: ℒV→V+ℒT→TL_V→ V+L_T→ T 7.3 52.3 32.5 3.7 INFUSE [S1] + [S2]: ℒV→T+ℒT→V+ℒGL_V→ T+L_T→ V+L_G 4.2 61.1 28.2 2.1 INFUSE (LLaVA-1.6-7B backbone) INFUSE [S1]: ℒV→V+ℒT→TL_V→ V+L_T→ T 6.8 60.2 35.4 3.6 INFUSE [S1] + [S2]: ℒV→T+ℒT→V+ℒGL_V→ T+L_T→ V+L_G 3.1 68.1 24.3 2.9 INFUSE ( Qwen3-VL-8B-Instruct backbone) INFUSE [S1]: ℒV→V+ℒT→TL_V→ V+L_T→ T 4.8 62.7 20.1 1.4 INFUSE [S1] + [S2]: ℒV→T+ℒT→V+ℒGL_V→ T+L_T→ V+L_G 2.9 67.4 13.0 1.0 ¶AMBER [7]. Table 7: Training cost comparison (LLaVA-1.6 backbone). CHiP cost is reported using its official training recipe, whereas INFUSE uses our 2K training setting. This cost comparison is therefore not the controlled 2K-data comparison reported in Table 1. Method Training Data Epochs GPU-hours Base SFT 5K 5 12 CHiP(official recipe) 5K+ 3 12.5 INFUSE 2K 5+5 6.2 Temperature sensitivity analysis. We evaluate the sensitivity of the contrastive temperature parameter τ. As shown in Table 8, INFUSE remains stable across a broad range of values (0.01≤τ≤0.200.01≤τ≤ 0.20), consistently reducing hallucinations relative to the baseline. The best performance is achieved at τ=0.07τ=0.07, which we use throughout the paper. Table 8: Sensitivity analysis of temperature parameter τ. Temperature parameter, τ CHAIR↓ Hal.↓ 0.01 4.1 27.6 0.05 3.5 25.8 0.07 (chosen) 3.1 24.3 0.10 3.3 25.1 0.20 3.7 26.4 Qualitative example. Figure 5 shows a representative AMBER example. The baseline and CHiP outputs assert unsupported content (“grass being too tall,” “living room,” “park”); INFUSE sticks to the visible scene. Figure 5: Qualitative comparison on AMBER. LLaVA and CHiP produce hallucinated content: “grass being too tall”, “living room”, and “park”; INFUSE (green) describes visually supported content. 5 Limitations INFUSE operates on sentence-level (final-token) embeddings. The discriminative results suggest that attribute- and relation-level grounding improves anyway, but token- and region-level stabilization may capture finer detail and is a natural extension. Our textual negatives come from a single curated pool (RLHF-V rejected captions). The transfer we observe to AMBER, MMHal, HallusionBench, and POPE, none of which shares RLHF-V’s distribution, suggests the learned stability is not tied to that pool’s specific hallucination patterns, but broader negative sources deserve study. Our evaluation covers single-image, single-turn settings at the 7-8B scale; multi-image and multi-turn interactions, and larger models, remain open. Finally, Proposition 3.1 controls perturbation-induced variation of the output distribution. Why stabilization also lowers absolute hallucination rates, which we establish empirically, is a question worth considering as a theory of its own. 6 Conclusion We introduced the distinction between explicit (inference-time) and implicit (training-time) feature stabilization for LVLM hallucination mitigation, and instantiated the implicit route in INFUSE: perturbation-averaged and ground-truth anchors stabilize each modality, and bidirectional contrastive alignment then grounds the stabilized representations across modalities. The approach is theoretically motivated, with anchor contraction at rate 1/K1/ K and Lipschitz control of output variation; cheap to train, at 6.2 GPU-hours (3.1 hours on two H100s) on 2K samples; and free at inference. Across three backbones, it reduces AMBER CHAIR by 46-63% and AMBER hallucination rate by 23-51% relative to the corresponding base models, while preserving VQA-v2 and TextVQA performance. We take this as evidence that implicit feature stabilization is a practical training principle for building more reliable LVLMs. References [1] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §1, §2, §3.3, §4. [2] J. Li, D. Li, S. Savarese, and S. Hoi (2023) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, p. 19730–19742. Cited by: §1, §2, §3.3. [3] H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee (2024) LLaVA-next: improved reasoning, ocr, and world knowledge. External Links: Link Cited by: Appendix B, §1, §2, §3.3, Table 1, Table 2, §4. [4] G. Lee, L. Shi, E. Latif, Y. Gao, A. Bewersdorff, M. Nyaaba, S. Guo, Z. Liu, G. Mai, T. Liu, et al. (2025) Multimodality of ai for education: toward artificial general intelligence. IEEE Transactions on Learning Technologies 18, p. 666–683. Cited by: §1. [5] T. Tu, S. Azizi, D. Driess, M. Schaekermann, M. Amin, P. Chang, A. Carroll, C. Lau, R. Tanno, I. Ktena, et al. (2024) Towards generalist biomedical ai. Nejm Ai 1 (3), p. AIoa2300138. Cited by: §1. [6] Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. Wen (2023) Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, p. 292–305. Cited by: Table 10, Appendix B, §1. [7] J. Wang, Y. Wang, G. Xu, J. Zhang, Y. Gu, H. Jia, J. Wang, H. Xu, M. Yan, J. Zhang, et al. (2023) Amber: an llm-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311.07397. Cited by: Table 10, Appendix B, §1, Table 1, Table 6. [8] Z. Sun, S. Shen, S. Cao, H. Liu, C. Li, Y. Shen, C. Gan, L. Gui, Y. Wang, Y. Yang, et al. (2024) Aligning large multimodal models with factually augmented rlhf. In Findings of the Association for Computational Linguistics: ACL 2024, p. 13088–13110. Cited by: Table 10, Appendix B, §1, §2, Table 1, Table 2, Table 2. [9] T. Guan, F. Liu, X. Wu, R. Xian, Z. Li, X. Liu, X. Wang, L. Chen, F. Huang, Y. Yacoob, et al. (2024) Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 14375–14385. Cited by: Table 10, Appendix B, §1, Table 1. [10] H. Liu, W. Xue, Y. Chen, D. Chen, X. Zhao, K. Wang, L. Hou, R. Li, and W. Peng (2024) A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253. Cited by: §1. [11] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, et al. (2025) A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43 (2), p. 1–55. Cited by: §1, §2. [12] P. Sahoo, P. Meharia, A. Ghosh, S. Saha, V. Jain, and A. Chadha (2024) A comprehensive survey of hallucination in large language, image, video and audio foundation models. Findings of the Association for Computational Linguistics: EMNLP 2024, p. 11709–11724. Cited by: §1, §2. [13] C. Jiang, H. Xu, M. Dong, J. Chen, W. Ye, M. Yan, Q. Ye, J. Zhang, F. Huang, and S. Zhang (2024) Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 27036–27046. Cited by: §1, §1, §2, §3.3, §3. [14] J. Fu, S. Huangfu, H. Fei, X. Shen, B. Hooi, X. Qiu, and S. Ng (2025) CHiP: cross-modal hierarchical direct preference optimization for multimodal llms. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, p. 90675–90695. External Links: Link Cited by: §1, §1, §2, §3, Table 1, Table 2, §4. [15] P. Sarkar, S. Ebrahimi, A. Etemad, A. Beirami, S. Arik, and T. Pfister (2025) Mitigating object hallucination in mllms via data-augmented phrase-level alignment. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, p. 73796–73846. External Links: Link Cited by: §1, §1, §2, Table 1, §4. [16] S. Liu, H. Ye, and J. Zou (2025) Reducing hallucinations in large vision-language models via latent space steering. In The Thirteenth International Conference on Learning Representations, Cited by: §1, §2, §3, Table 2, §4. [17] S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing (2024) Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 13872–13882. Cited by: §1, §2, §3.3, §4. [18] Q. Huang, X. Dong, P. Zhang, B. Wang, C. He, J. Wang, D. Lin, W. Zhang, and N. Yu (2024) Opera: alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13418–13427. Cited by: §1, §2, §4. [19] T. Yu, Y. Yao, H. Zhang, T. He, Y. Han, G. Cui, J. Hu, Z. Liu, H. Zheng, M. Sun, et al. (2024) Rlhf-v: towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13807–13816. Cited by: Appendix A, §1, §2, Table 1, §4, §4. [20] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. Cited by: §1, §2, §4. [21] Z. Yan, Z. Li, Y. He, C. Wang, K. Li, X. Li, X. Zeng, Z. Wang, Y. Wang, Y. Qiao, et al. (2025) Task preference optimization: improving multimodal large language models with vision task alignment. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 29880–29892. Cited by: §1. [22] J. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko (2020) Bootstrap your own latent: a new approach to self-supervised learning. Advances in Neural Information Processing Systems 33, p. 21271–21284. Cited by: item 2, §2, §3.1. [23] X. Chen and K. He (2021) Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 15750–15758. Cited by: item 2, §2, §3.1. [24] J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. (2022) Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35, p. 23716–23736. Cited by: §2. [25] H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023) Visual instruction tuning. Advances in neural information processing systems 36, p. 34892–34916. Cited by: §2, §3.3. [26] H. Liu, C. Li, Y. Li, and Y. J. Lee (2024) Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 26296–26306. Cited by: Appendix B, §2, Table 1, §4. [27] S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen (2024) A survey on multimodal large language models. National Science Review 11 (12), p. nwae403. Cited by: §2. [28] Z. Chen, Y. Min, J. Zhang, B. Yan, J. Wang, X. Wang, and S. Shan (2026) A survey of multimodal hallucination evaluation and detection. International Journal of Computer Vision 134 (3), p. 131. Cited by: §2. [29] X. Zhu, J. Fang, S. Wang, B. Zhu, Z. Wang, Y. Yang, and X. He (2026) Mitigating hallucinations in large vision-language models without performance degradation. arXiv preprint arXiv:2604.20366. Cited by: §2. [30] A. Agrawal, K. Gouthaman, R. Aralikatti, G. Jagatap, J. Yuan, S. Baskar, V. Kamarshi, A. Fanelli, and F. Huang (2026) Towards mitigating hallucinations in large vision-language models by refining textual embeddings. In Findings of the Association for Computational Linguistics: ACL 2026, p. 42032–42048. Cited by: §2, §2. [31] L. Yu, Z. Chen, P. Kuang, Z. Feng, F. Zhou, L. Wang, and G. Dobbie (2026) Causally-grounded dual-path attention intervention for object hallucination mitigation in lvlms. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 36021–36029. Cited by: §2, §2. [32] M. Fazli, B. Wei, and Z. Zhu (2026) Inject to heal: alleviating hallucination in lvlms via context embedding injection. In Findings of the Association for Computational Linguistics: ACL 2026, p. 41177–41193. Cited by: §2, §2. [33] C. Yu, Y. Xu, Y. Chen, and W. Zhang (2026) Optimizing lvlms with on-policy data for effective hallucination mitigation. In 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 4733–4743. Cited by: §2. [34] B. Lim, J. Yun, J. Kwon, K. Kim, and Y. Kim (2026) Aligning with your own voice: self-corrected preference learning for hallucination mitigation in lvlms. In Findings of the Association for Computational Linguistics: ACL 2026, p. 35829–35849. Cited by: §2, §2. [35] J. Wu, Y. Shen, H. Sun, and M. Cao (2025) Mitigating hallucinations in large vision-language models via dual contrastive decoding. In Proceedings of the 7th ACM International Conference on Multimedia in Asia, p. 1–8. Cited by: §2, §2. [36] J. Ji, Q. Liu, W. Yang, and Z. He (2026) CausalLens: sensitivity-guided multi-head causal intervention for hallucination mitigation in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 4199–4209. Cited by: §2, §2. [37] Y. Park, D. Lee, J. Choe, and B. Chang (2025) Convis: contrastive decoding with hallucination visualization for mitigating hallucinations in multimodal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 6434–6442. Cited by: §2. [38] C. Wang, J. Yang, and Y. Zhou (2025) MINT: mitigating hallucinations in large vision-language models via token reduction. arXiv preprint arXiv:2502.00717. Cited by: §2. [39] W. Zhou, S. Bai, D. P. Mandic, Q. Zhao, and B. Chen (2024) Revisiting the adversarial robustness of vision language models: a multimodal perspective. arXiv preprint arXiv:2404.19287. Cited by: §2. [40] Z. Yang, X. Luo, D. Han, Y. Xu, and D. Li (2025) Mitigating hallucinations in large vision-language models via dpo: on-policy data hold the key. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 10610–10620. Cited by: §2. [41] Y. Wu, L. Zhang, H. Yao, J. Du, K. Yan, S. Ding, Y. Wu, and X. Li (2025) Antidote: a unified framework for mitigating lvlm hallucinations in counterfactual presupposition and object perception. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 14646–14656. Cited by: §2. [42] S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y. Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y. Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang, Q. Wang, Y. Wang, T. Xie, Y. Xu, H. Xu, J. Xu, Z. Yang, M. Yang, J. Yang, A. Yang, B. Yu, F. Zhang, H. Zhang, X. Zhang, B. Zheng, H. Zhong, J. Zhou, F. Zhou, J. Zhou, Y. Zhu, and K. Zhu (2025) Qwen3-vl technical report. CoRR abs/2511.21631. External Links: Link Cited by: Table 1, §4. [43] L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. (2023) Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36, p. 46595–46623. Cited by: §4. [44] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. (2023) Qwen technical report. Cited by: §4. [45] T. Yu, J. Hu, Y. Yao, H. Zhang, Y. Zhao, C. Wang, S. Wang, Y. Pan, J. Xue, D. Li, et al. (2023) Reformulating vision-language foundation models and datasets towards universal multimodal assistants. arXiv preprint arXiv:2310.00653. Cited by: §4. [46] W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, et al. (2023) Image as a foreign language: beit pretraining for vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 19175–19186. Cited by: §4. [47] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh (2017) Making the v in vqa matter: elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 6904–6913. Cited by: §4.1. [48] A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach (2019) Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §4.1. [49] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft coco: common objects in context. In European conference on computer vision, p. 740–755. Cited by: Table 1. [50] A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Saenko (2018) Object hallucination in image captioning. arXiv preprint arXiv:1809.02156. Cited by: Table 10, Appendix B, Table 1. [51] I. Loshchilov and F. Hutter (2017) Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: Appendix A, Appendix B. [52] E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: Appendix A, Appendix B. Appendix This appendix is organized as follows. Appendix A provides pseudocode and implementation-level details of the two training stages of INFUSE. Appendix B describes training and data configurations, including architecture, optimization hyperparameters, and evaluation settings. Appendix C evaluates robustness of the trained model under visual perturbations. Appendix D reports additional quantitative results, including response-length statistics and fine-grained ObjHal metrics. Appendix E presents representative hallucination cases for qualitative comparison against LLaVA, DPO, and CHiP. Appendix A Algorithms for INFUSE Our method follows a two-stage training framework. [S1] performs intra-modal stabilization, constructing perturbation-invariant anchors for both modalities; [S2] performs inter-modal alignment on the stabilized representations. We describe both stages at the level of the implementation, expanding on Algorithm 1 of the main paper. [S1] Intra-modal stabilization. We train on a subset N of the RLHF-V dataset [19] with ||=N=2000|N|=N=2000 samples. Each sample is a triplet (vi,ti,t~i)(v_i,t_i, t_i): input image, ground-truth caption, and hallucinated (rejected) caption. Masked-view visual anchors. At each optimization step, for each image viv_i in the batch we generate K=100K=100 masked views Ck(vi)k=1K\C_k(v_i)\_k=1^K with patch-mask ratio ρ=0.97ρ=0.97, embed each view through the frozen CLIP vision encoder VθV_θ and the current projector, zikv≔Fα(Vθ(Ck(vi)))z_ik^v F_α\! (V_θ(C_k(v_i)) ), and form the anchor z¯iv≔1K∑k=1Kzikv, z_i^v 1K _k=1^Kz_ik^v, (5) treated as a constant (stop-gradient) in the loss, as in Definition 3.1 of the main paper. Averaging suppresses high-variance, instability-driven directions while preserving consistent semantic structure (Proposition 3.1). The clean-image embedding ziv+≔Fα(Vθ(vi))z_i^v^+ F_α\! (V_θ(v_i) ), which receives gradients, is pulled toward z¯iv z_i^v and repelled from the lagged embedding zivz_i^v computed under the previous step’s parameters with stop-gradient (Definition 3.2 of the main paper), yielding the visual stability loss ℒV→VL_V→ V of Equation (1). Since the vision encoder is frozen, the K view embeddings require forward passes only. Textual anchors. The ground-truth caption is encoded under the current decoder parameters with stop-gradient to form the truthful anchor zit≔sg[Lβ(ti)]z_i^t [L_β(t_i) ], and the hallucinated caption gives the negative direction z~it≔sg[Lβ(t~i)] z_i^t [L_β( t_i) ]. The model’s response t^i t_i is decoded from viv_i, and zit+≔Lβ(t^i)z_i^t^+ L_β( t_i) denotes its embedding; gradients flow through the decoder’s hidden states on the generated token sequence (token identities held fixed). The textual stability loss ℒT→TL_T→ T of Equation (2) pulls zit+z_i^t^+ toward the truthful anchor and pushes it away from hallucinated-caption embeddings z~jt z_j^t across the batch, discouraging drift of the generated representation into hallucination-prone regions of the text space. [S1] objective. The overall objective combines both terms, ℒ[S1]≔ℒV→V+ℒT→TL_[S1] _V→ V+L_T→ T, minimized with AdamW [51] over the projector and decoder parameters (α,β)(α,β); since ℒV→VL_V→ V depends only on α and ℒT→TL_T→ T only on β, this is equivalent to the per-modality updates of Algorithm 1 in the main paper. The stage yields stabilized components Fα(1)F^(1)_α and Lβ(1)L^(1)_β; detailed steps are in Algorithm 2. Algorithm 2 [S1] Intra-modal stabilization (implementation) 1: ≔(vi,ti,t~i)i=1NN \(v_i,t_i, t_i)\_i=1^N, epochs T1T_1, initial Fα,LβF_α,L_β, step size η 2: Fα(1),Lβ(1)F^(1)_α,L^(1)_β 3: Split N into mini-batches ss=1S\S_s\_s=1^S of size NmN_m 4: for e=1e=1 to T1T_1 do ⊳ epoch loop 5: for each mini-batch sS_s do ⊳ one optimization step 6: for each i∈si _s do 7: draw views Ck(vi)k=1K\C_k(v_i)\_k=1^K; z¯iv←1K∑kFα(Vθ(Ck(vi))) z_i^v← 1K _kF_α\! (V_θ(C_k(v_i)) ) ⊳ stop-grad 8: ziv←sg[Fαprev(Vθ(vi))]z_i^v [F_α^prev (V_θ(v_i) ) ]; ziv+←Fα(Vθ(vi))z_i^v^+← F_α (V_θ(v_i) ) 9: t^i←DECODE(Lβ,vi) t_i (L_β;v_i); zit+←Lβ(t^i)z_i^t^+← L_β( t_i) 10: zit←sg[Lβ(ti)]z_i^t [L_β(t_i) ]; z~it←sg[Lβ(t~i)] z_i^t [L_β( t_i) ] 11: end for 12: Compute ℒV→VL_V→ V on sS_s via Equation (1); ℒT→TL_T→ T via Equation (2) 13: ℒ[S1]←ℒV→V+ℒT→TL_[S1] _V→ V+L_T→ T 14: αprev←α^prev←α; (α,β)←AdamW((α,β),∇(α,β)ℒ[S1],η)(α,β) ((α,β), _(α,β)L_[S1],η ) 15: end for 16: end for 17: return Fα(1)←FαF^(1)_α← F_α, Lβ(1)←LβL^(1)_β← L_β Embedding stability under perturbations. [S1] explicitly reduces representational variance induced by input perturbations. As shown in Figure 2(a) of the main paper, the baseline exhibits progressively larger embedding variance as perturbation strength increases, whereas representations learned after [S1] maintain substantially lower variance at all strengths. The density distribution in Figure 6 corroborates this: [S1] shifts the entire variance distribution toward lower values, so the improvement holds across perturbed samples rather than only on average. The stabilization objective achieves this by pulling the current representation toward the perturbation-averaged anchor while repelling the lagged embedding, reducing perturbation-induced drift without feature collapse. Figure 6: Embedding-variance distribution before and after intra-modal stabilization. [S1] shifts the entire distribution toward lower variance, indicating consistently more stable, perturbation-invariant visual representations across samples. [S2] Inter-modal alignment. Starting from the stabilized projector and decoder of [S1], this stage enforces cross-modal grounding using the same ||=N=2000|N|=N=2000 RLHF-V samples, now as image-caption pairs (vi,ti)(v_i,t_i). For each image, the visual-conditioned decoder embedding uivu^v_i is obtained by passing viv_i through the frozen VθV_θ, the projector FαF_α, and the decoder LβL_β, taking the end-of-sequence (EOS) token embedding; the EOS embedding of the caption gives uitu^t_i. The bidirectional InfoNCE objectives of Equations (3)-(4) pull each paired (uiv,uit)(u^v_i,u^t_i) together and push unpaired combinations apart, and the supervised generation loss ℒGL_G preserves linguistic quality. The final objective, ℒ[S2]=ℒV→T+ℒT→V+ℒGL_[S2]=L_V→ T+L_T→ V+L_G, is minimized with AdamW over (α,β)(α,β), yielding Fα(2)F^(2)_α and Lβ(2)L^(2)_β. Detailed steps are in Algorithm 3. Algorithm 3 [S2] Inter-modal alignment (implementation) 1: Input: =(vi,ti)i=1NN=\(v_i,t_i)\_i=1^N, epochs T2T_2, step size η, Fα(1)F^(1)_α, Lβ(1)L^(1)_β 2: Output: Fα(2)F^(2)_α, Lβ(2)L^(2)_β 3: Initialize: Fα←Fα(1)F_α← F^(1)_α; Lβ←Lβ(1)L_β← L^(1)_β 4: Split N into mini-batches ss=1S\S_s\_s=1^S of size NmN_m 5: for e=1e=1 to T2T_2 do 6: for s=1s=1 to S do 7: uiv←Lβ(Fα(Vθ(vi)))u^v_i← L_β (F_α (V_θ(v_i) ) ) ∀i∈s∀ i _s ⊳ vision EOS 8: uit←Lβ(ti)u^t_i← L_β(t_i) ∀i∈s∀ i _s ⊳ text EOS 9: Compute ℒV→TL_V→ T on sS_s via Equation (3) 10: Compute ℒT→VL_T→ V on sS_s via Equation (4) 11: Compute the generation loss ℒGL_G on sS_s 12: ℒ[S2]←ℒV→T+ℒT→V+ℒGL_[S2] _V→ T+L_T→ V+L_G 13: (α,β)←AdamW((α,β),∇(α,β)ℒ[S2],η)(α,β) ((α,β), _(α,β)L_[S2],η ) 14: end for 15: end for 16: return Fα(2)←FαF^(2)_α← F_α, Lβ(2)←LβL^(2)_β← L_β Effect of the complete two-stage framework. Figure 7 compares the base models with the final INFUSE model after both stages. Across LLaVA-1.6-7B and Qwen3-VL-8B-Instruct, the complete framework consistently reduces hallucination on AMBER and MMHal: for LLaVA-1.6-7B, from approximately 49%49\% to 24%24\% on AMBER and from 43%43\% to 16%16\% on MMHal; for Qwen3-VL-8B-Instruct, from approximately 27%27\% to 13%13\% and 39%39\% to 15%15\%, respectively. The stable intra-modal representations learned in [S1] thus provide a reliable foundation for the bidirectional alignment of [S2]. Figure 7: Hallucination reduction after complete two-stage training. Baseline models vs. the complete INFUSE framework ([S1]+[S2]) on AMBER and MMHal. Lower is better. A note on training efficiency. INFUSE is computationally lightweight for three reasons. First, it preserves the original architecture and keeps the vision encoder frozen, fine-tuning only the projector, LoRA [52] adapters on the last four decoder layers, and the associated parameters; this substantially lowers memory and optimization cost relative to full fine-tuning. Second, stability anchors are built from forward passes through the frozen encoder followed by simple averaging, with no auxiliary networks or learnable modules. Third, all stabilization and alignment objectives apply only during training; the inference pipeline is identical to the base model, with no added latency, memory, or deployment complexity. Appendix B Training and Data Details Training configuration for [S1]. We train INFUSE on 2× NVIDIA H100 GPUs with bf16 mixed precision and fully synchronized gradient updates. For LLaVA-1.5-7B [26] and LLaVA-1.6-7B [3], LoRA [52] adapters (rank 16, scaling 32) are attached only to the last four transformer layers of the Vicuna LLM; only these LoRA parameters and the vision–language projector are updated, while the CLIP ViT-L/336 vision encoder remains frozen. We optimize with AdamW [51] at a peak learning rate of 2×10−52× 10^-5 under cosine decay with no warm-up, batch size Nm=16N_m=16, for T1=5T_1=5 epochs on the N=2000N=2000-sample RLHF-V subset. Each image is expanded into K=100K=100 masked views at a 97%97\% patch-mask ratio, and only the V→ and T→ objectives are active. Experiment tracking uses Weights & Biases. Training configuration for [S2]. [S2] adds the cross-modal objectives (V→ , T→ ) and the instruction-following generation loss. The trainable set is unchanged (last four decoder layers via LoRA and the projector); the vision encoder stays frozen to avoid re-introducing encoder-level instability during cross-modal alignment. We use AdamW with a base learning rate of 2×10−52× 10^-5 and a cosine schedule, batch size Nm=16N_m=16, for T2=5T_2=5 epochs. We use RLHF-V image-truthful-response pairs, where paired samples form the positives and other samples in the mini-batch provide unpaired negatives for the bidirectional contrastive objectives. Qwen3-VL training configuration. For Qwen3-VL-8B-Instruct, we keep the vision encoder frozen and train the multimodal projector together with LoRA adapters on the last four language-model decoder layers. We use LoRA rank r=16r=16, scaling factor α=32α=32, and dropout 0.050.05. Both stages are trained using AdamW with a learning rate of 2×10−52× 10^-5, weight decay 0.010.01, BF16 precision, and an effective batch size of 16. Stage [S1] is trained for 5 epochs, and Stage [S2] is initialized from the Stage [S1] checkpoint and trained for another 5 epochs. We use K=100K=100 masked views and a contrastive temperature of τ=0.07τ=0.07 throughout training. Sensitivity to the number of perturbation views. We evaluate the sensitivity of INFUSE to the number of perturbation views K used to construct the perturbation-averaged visual anchor. Specifically, we vary K∈10,25,50,100K∈\10,25,50,100\ on LLaVA-1.6 while keeping all other training and evaluation settings fixed. As shown in Table 9, increasing K from 10 to 100 consistently improves hallucination mitigation, reducing CHAIR from 4.2 to 3.1 and the hallucination rate from 29.5% to 24.3%. Increasing K further to 200 provides no additional improvement, with CHAIR and Hal. slightly increasing to 3.2 and 24.5%, respectively. These results indicate diminishing returns at larger values of K and motivate our choice of K=100K=100 throughout the main experiments. Table 9: Sensitivity to the number of perturbation views K on LLaVA-1.6. Lower values are better for both metrics. K CHAIR↓ Hal.↓ 10 4.2 29.5 25 3.7 27.0 50 3.3 25.3 100 3.1 24.3 Evaluation settings. We evaluate on five hallucination benchmarks: AMBER [7], Object HalBench [50], MMHal-Bench [8], POPE [6], and HallusionBench [9], summarized in Table 10. Together they cover fine-grained object grounding, attribute faithfulness, multimodal reasoning consistency, and object-existence verification. All inference uses deterministic decoding (temperature =0=0) for reproducibility. Metrics. Object HalBench reports response-level and mention-level hallucination rates. MMHal-Bench reports GPT-4-assessed overall correctness and response-level hallucination rate. HallusionBench reports question-pair accuracy (qA), figure accuracy (fA), and overall accuracy (aA). AMBER (generative) reports CHAIR, object coverage (Cover), response-level hallucination rate (Hal), and cognition-driven hallucination rate (Cog); its discriminative track reports F1, F1E, F1A, and F1R. POPE reports accuracy, precision, recall, and F1 over Yes/No object-existence questions on MSCOCO images. Table 10: Evaluation datasets used for hallucination analysis and multimodal grounding. Dataset Brief description Object HalBench [50] Detects object hallucination in image captioning and open-ended descriptions. MMHal-Bench [8] Multimodal hallucination dataset based on VQA, spanning multiple question types and object categories. HallusionBench [9] Robustness to visual illusions and prior-driven hallucinations across 346 images and 1,129 questions. AMBER [7] Evaluator-independent benchmark for generative hallucination and visual-textual consistency. POPE [6] Binary object-existence benchmark with Yes/No questions over MSCOCO images. Appendix C Robustness Evaluation To assess how well the INFUSE-trained model maintains performance under visual corruption, we evaluate several perturbation strategies applied to the input image at test time, with the model fixed. All rows use INFUSE + LLaVA-1.5. Perturbation details. Masking (0.3 / 0.6) removes 30%30\% or 60%60\% of spatial patches at random, simulating partial occlusion at two severities. Gaussian noise adds pixel-level perturbations ϵ∼(0,σ2I)ε (0,σ^2I). Cropping selects a random spatial window and resizes it to the original resolution, preserving global semantics while altering local structure. Rotation applies a small random angular shift. Together, these span mild geometric variation through heavier distortion. Results. Table 11 summarizes results on HallusionBench (aA, qA) and AMBER (Cover, Hal). Clean images give the reference performance; corrupted inputs generally reduce accuracy and coverage. Rotation and cropping preserve performance best, while masking and Gaussian noise degrade semantic consistency more severely, with visible drops in qA and increased hallucination on AMBER. Perturbations that maintain global structure are therefore better tolerated, while heavy occlusion or random noise remains the harder regime. Table 11: Robustness of the INFUSE + LLaVA-1.5 model under test-time perturbations. aA and qA on HallusionBench; Cover and Hal on AMBER. HallusionBench AMBER Perturbation aA↑ qA↑ Cover↑ Hal↓ Clean images (reference) 49.7 39.1 61.1 28.2 Masking (0.3) 45.3 23.7 49.2 30.2 Masking (0.6) 42.2 22.7 45.2 35.1 Gaussian 42.6 22.9 50.0 39.0 Cropping 45.0 36.1 47.0 31.2 Rotation 48.0 37.5 48.0 30.1 Appendix D Additional Quantitative Results Response length across benchmarks. Response length is a useful behavioral signal: overly long outputs often accompany hallucination-prone generation (invented objects, fabricated attributes, scene embellishment), while overly short outputs can indicate lost informativeness. As shown in Table 12, INFUSE maintains output lengths comparable to DPO and CHiP and to the base model, indicating that its hallucination reductions do not come from truncating responses. Table 12: Average output length (tokens). Baseline rows use the LLaVA-1.6 backbone. Model AMBER MMHal ObjHal LLaVA-1.6 131 44 164 LLaVA-1.6 + DPO 127 42 160 LLaVA-1.6 + CHiP 124 43 151 INFUSE + LLaVA-1.5 127 45 162 INFUSE + LLaVA-1.6 125 45 164 INFUSE + Qwen3-VL-8B-Instruct 128 44 162 Fine-grained hallucination and grounding on ObjHal. Table 13 decomposes ObjHal into response-level hallucination (Resp. Hall), object-level hallucination (Obj. Hall), response and object correctness, and object recall. INFUSE + Qwen3-VL attains the lowest response-level hallucination (4.13) and object-level hallucination (3.2), while CHiP achieves the highest response correctness (95.08) and object correctness (96.79). On LLaVA-1.5, response hallucination decreases from 46.3 to 5.1 and object hallucination from 22.6 to 3.7. On LLaVA-1.6, the corresponding reductions are from 14.1 to 4.87 and from 7.4 to 3.4, respectively. These results show that INFUSE substantially reduces both response- and object-level hallucination while maintaining strong grounding performance. Table 13: Fine-grained results on Object HalBench (ObjHal). Bold = best, underline = second best. ↑ higher is better, ↓ lower is better. Baseline row uses the LLaVA-1.6 backbone. Model Resp. Hall↓ Obj. Hall↓ Resp. Corr.↑ Obj. Corr.↑ Obj. Recall↑ LLaVA-1.6 14.1 7.4 85.92 92.63 55.03 LLaVA-1.6 + DPO 14.0 7.1 88.97 93.39 52.83 LLaVA-1.6 + CHiP 5.8 3.9 95.08 96.79 48.95 INFUSE + LLaVA-1.5 5.1 3.7 95.01 94.40 49.01 INFUSE + LLaVA-1.6 4.87 3.4 94.86 95.40 51.08 INFUSE + Qwen3-VL-8B-Instruct 4.13 3.2 94.83 96.34 52.75 Appendix E Representative Hallucination Examples Figures 8-12 present five representative cases from MMHal-Bench and HallusionBench, covering object counting, geometric reasoning, table interpretation, chart comprehension, and map-based spatial comparison. Across these scenarios, the base models frequently hallucinate quantities, rankings, trends, or spatial relations unsupported by the visual evidence, while CHiP and especially INFUSE produce more conservative, visually grounded responses. Figure 8: MMHal-Bench: object-counting hallucination. The base LLaVA and Qwen models hallucinate an extra object, claiming six oysters where five are visible. CHiP corrects the count, and INFUSE consistently identifies five oysters. Figure 9: HallusionBench: visual-geometric reasoning. For a right triangle with legs 3 and 4, the hypotenuse is x=5x=5. Asked whether x exceeds 5, the base models incorrectly answer Yes; CHiP answers correctly, and INFUSE consistently answers No. Figure 10: HallusionBench (VS/table): table interpretation. Given the 2008 Olympics medal table with China leading in golds, the base models incorrectly answer No when asked whether the shown ranking is correct; CHiP and INFUSE answer Yes, grounding in the visible entries. Figure 11: HallusionBench (VS/chart): chart comprehension. Switzerland has the longest bar in the import-value chart. Asked whether France leads, the base models incorrectly answer Yes, hallucinating a ranking that contradicts the bar lengths; CHiP and INFUSE answer No. Figure 12: HallusionBench (VS/map): geographic-size reasoning. Texas is visibly larger than California, yet the base LLaVA model answers No when asked; CHiP and INFUSE answer Yes, matching the map.