Paper deep dive
ReWEIGH the Evidence: Calibrating Token-Level Ordinal Visual Evidence to Mitigate Hallucinations in Large Vision-Language Models
Jihae Jeong, Junha Choi, Hwanjo Yu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/21/2026, 4:49:35 AM
Summary
The paper introduces ReWEIGH, a training-free decoding intervention for Large Vision-Language Models (LVLMs) that mitigates hallucinations by calibrating token-level ordinal visual evidence. ReWEIGH aggregates visual-token readouts using Dense Mean Reciprocal Rank (DMRR) across visual positions and compares candidate tokens against a token-specific reference estimated from unlabeled images. This method reduces hallucinated object mentions by up to 21.3% with minimal latency overhead (1.33% per token) across various model architectures and sizes.
Entities (7)
Relation Signals (6)
ReWEIGH → mitigates → Hallucination
confidence 95% · ReWEIGH reduces hallucinated object mentions by up to 21.3% while largely preserving or improving descriptive and general performance.
LVLMs → suffersfrom → Hallucination
confidence 95% · Large vision-language models (LVLMs) often hallucinate, generating content that the input image does not support.
ReWEIGH → evaluatedon → LLaVA-1.5-7B
confidence 92% · On four 7B backbones, ReWEIGH reduces hallucinated object mentions... The reductions extend to 11 models from six architecture families... LLaVA-1.5-7B
ReWEIGH → evaluatedon → Qwen2.5-VL-7B
confidence 92% · On four 7B backbones... Qwen2.5-VL-7B
ReWEIGH → uses → Dense Mean Reciprocal Rank
confidence 90% · We pool visual evidence through reciprocal ranks, formalized in Section 4.2 as dense mean reciprocal rank (DMRR).
ReWEIGH → reduces → CHAIR
confidence 88% · Across four 7B backbones, ReWEIGH reduces CHAIRI (21) by up to 21.3%
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large vision-language models (LVLMs) often hallucinate, generating content that the input image does not support. Preventing such content during decoding calls for a candidate-specific measure of how strongly the image supports the token under consideration. The model's visual-token states offer a natural source of this evidence because projecting each state through the output head reveals which vocabulary items that position favors. These position-wise readouts cannot be pooled directly because their probability magnitudes are not comparable across visual positions. Vocabulary ranks provide a scale-invariant basis for pooling, but tokens still differ systematically in their typical rank-based evidence. We propose ReWEIGH, a training-free decoding intervention that aggregates these ranks across visual positions and compares each candidate with a token-specific reference estimated from unlabeled images. At inference, ReWEIGH caches the image evidence during prefill and applies a bounded penalty only to candidates that fall below their reference. On four 7B backbones, ReWEIGH reduces hallucinated object mentions by up to 21.3% while largely preserving or improving descriptive and general performance. With evidence cached, the average added latency is 1.33% per token, and the reductions extend across six architecture families to 32B parameters.
Tags
Links
- Source: https://arxiv.org/abs/2608.19075v1
- Canonical: https://arxiv.org/abs/2608.19075v1
Trouble viewing inline? Open PDF directly →
Full Text
119,395 characters extracted from source content.
Expand or collapse full text
ReWEIGH the Evidence: Calibrating Token-Level Ordinal Visual Evidence to Mitigate Hallucinations in Large Vision-Language ModelsBoth authors contributed equally to this work.Corresponding author. Jihae Jeong Junha Choi11footnotemark: 1 Hwanjo Yu Affiliation: Pohang University of Science and Technology (POSTECH) Affiliation: wisdomjeong, 2002junha, hwanjoyu@postech.ac.kr Abstract Large vision-language models (LVLMs) often hallucinate, generating content that the input image does not support. Preventing such content during decoding calls for a candidate-specific measure of how strongly the image supports the token under consideration. The model’s visual-token states offer a natural source of this evidence because projecting each state through the output head reveals which vocabulary items that position favors. These position-wise readouts cannot be pooled directly because their probability magnitudes are not comparable across visual positions. Vocabulary ranks provide a scale-invariant basis for pooling, but tokens still differ systematically in their typical rank-based evidence. We propose ReWEIGH, a training-free decoding intervention that aggregates these ranks across visual positions and compares each candidate with a token-specific reference estimated from unlabeled images. At inference, ReWEIGH caches the image evidence during prefill and applies a bounded penalty only to candidates that fall below their reference. On four 7B backbones, ReWEIGH reduces hallucinated object mentions by up to 21.3%21.3\% while largely preserving or improving descriptive and general performance. With evidence cached, the average added latency is 1.33%1.33\% per token, and the reductions extend across six architecture families to 32B parameters. 1 Introduction Large vision-language models (LVLMs) have emerged as general-purpose systems for visual understanding and reasoning (18; 5). However, they often hallucinate content that conflicts with the input image or lacks visual support (14; 25). These errors can arise when strong linguistic priors outweigh evidence from the image (13; 6). Because the resulting text remains fluent and plausible, the text alone often provides little indication of the error. This failure mode motivates interventions that act during decoding, before the model emits content without visual support. Existing decoding interventions face a trade-off between computational cost and grounding specificity. Contrastive methods compute corrective logits under altered visual or textual conditioning, which commonly requires additional forward or decoding passes (13; 6; 26; 12). Lighter methods use attention to control reliance on visual tokens (8; 19) or use uncertainty to decide when to reinject visual features (28). These methods assume that attention or uncertainty indicates insufficient grounding. Attention reveals where the model routes information, while uncertainty reflects confidence in the output distribution. Neither signal directly measures how strongly the image supports a particular candidate token. Output confidence also misses many errors, as hallucinated object mentions persist even in its top quartile (Table 15, Appendix B.6). Existing methods therefore offer no lightweight, candidate-specific measure of whether the model’s visual states provide unusually weak support for the token under consideration. The LVLM’s internal visual states provide a natural source of token-specific evidence without auxiliary model passes. Applying the output head to each visual-token state yields a vocabulary readout that reveals which words each visual position favors (20; 11). Prior lens-guided decoding uses the resulting probability magnitudes (3), but their values depend on the sharpness of each position’s output distribution. Pooling these probabilities can therefore give disproportionate weight to sharper visual positions. Vocabulary ranks avoid this scale dependence and provide a stable ordinal basis for aggregation. However, ranks do not support a single criterion for all tokens. Some tokens routinely rank near the top across visual positions, while others have lower typical evidence. Visual support must therefore be assessed against the candidate token’s own evidence baseline. Based on this analysis, we propose ReWEIGH (Reference-calibrated Weighting via Ordinal Evidence for Image-Grounded Hallucination Mitigation), a training-free decoding intervention. During offline calibration on unlabeled images, ReWEIGH aggregates the visual readouts using dense mean reciprocal rank (DMRR) and estimates each candidate token’s typical evidence level. It registers a token for intervention only when this reference estimate is stable. At inference, ReWEIGH computes and caches the image’s DMRR evidence during prefill. When a registered candidate falls below its reference, the method applies a bounded logit penalty. Across four 7B backbones, ReWEIGH reduces CHAIRI (21) by up to 21.3%21.3\% while largely preserving or improving descriptive and general multimodal performance. The reductions extend to 11 models from six architecture families, with model sizes ranging from 7B to 32B. Each model requires a one-time offline calibration on unlabeled images. With image-level evidence cached, the average added latency is 1.33%1.33\% per token. Our contributions are as follows: • We show that probability pooling remains sensitive to position-specific distribution shapes after affine scale equalization. Rank aggregation is invariant to strictly increasing transformations at each position and provides a stable basis for pooling visual readouts. • We identify a strong token-dependent component in ordinal visual evidence. We then introduce candidate-conditioned, token-specific calibration that assesses each token against its typical evidence baseline. • We develop a bounded, training-free decoding intervention that reduces hallucination across diverse LVLM backbones and architectures while largely preserving descriptive and general multimodal utility with minimal inference overhead. 2 Related Work Training-free decoding methods mitigate LVLM hallucinations without updating the backbone. Contrastive methods derive reference logits from altered visual inputs or views (13; 6; 2), altered instructions or self-generated descriptions (26; 12), or selectively retained vision tokens (10). These approaches typically require an auxiliary decoding trajectory and make the corrective signal depend on how the counterfactual condition is constructed. Other methods control attention or visual-token reliance (8; 19; 7) or use uncertainty-triggered reinjection of visual features (28). Although these signals can strengthen visual conditioning, they do not directly provide a token-specific reference for determining whether a candidate has unusually weak support from the image. Our approach belongs most directly to internal-representation interventions. Vocabulary projection maps hidden states into language space through the output head (20) and, in LVLMs, reveals signals that help distinguish visually grounded objects from hallucinated ones (11). ReVisiT selects one vision token at each decoding step and uses its constrained probability projection to refine the output (3). Consequently, evidence distributed across the remaining visual positions is not explicitly aggregated. DeCo instead blends earlier-layer predictions from the current generation position (24), rather than directly summarizing visual-token states. Activation Steering Decoding learns hallucination directions from labeled hidden states and contrasts predictions from opposite steering passes (22), but does not measure candidate-specific support from the image. In contrast, we aggregate scale-invariant ordinal readouts over all visual positions once during prefill, cache the resulting image-level evidence throughout generation, and interpret each candidate relative to its token-specific reference estimated from unlabeled calibration data. 3 Two Obstacles to Decoding with Internal Visual Evidence Figure 1: Across-position spread of the rank-10 probability before and after scale equalization. Error bars are retrospective image-bootstrap 95% intervals, and equalized bars are point estimates. Figure 2: Decomposition of DMRR variability on LLaVA-1.5-7B. Each bar sums to 100%. Visual Evidence from Internal States. LVLMs frequently hallucinate unsupported content in open-ended descriptions (14; 25), but their internal visual states can provide a proxy for the visual evidence associated with individual words. Projecting a visual-position hidden state through the language model’s output head yields a vocabulary-space readout of the words associated with that position (20). Such readouts distinguish grounded objects from hallucinated ones (11), and grounded alternatives remain highly ranked even at decoding steps that produce hallucinations (3). Formally, let P denote the set of visual-token positions and j(ℓ)h_j^( ) the hidden state at position j∈Pj∈ P and language-model layer ℓ . Following the logit-lens view, we apply the model’s final output normalization NormNorm and vocabulary projection headW_head to define the visual-position readout j(ℓ)=head(Norm(j(ℓ))).z_j^( )=W_head\! (Norm\! (h_j^( ) ) ). (1) Each component of j(ℓ)z_j^( ) is the vocabulary score that visual position j assigns to the corresponding vocabulary item. Turning these internal readouts into a decoding intervention, however, requires two steps. First, we must aggregate evidence from many visual positions without distortion (Section 3.1). Second, we must interpret the aggregated value relative to each candidate token (Section 3.2). These two problems determine ReWEIGH’s evidence measurement and per-candidate calibration, respectively. 3.1 Scale-Invariant Aggregation Across Visual Positions Probability magnitudes are not directly comparable across visual positions. A direct approach applies softmax to Eq. (1) and averages a token’s probability across positions. Prior lens-guided decoding likewise represents visual evidence using these probability magnitudes (3; 11). However, the average inherits the sharpness of each position-wise distribution. At a given position, a small probability can indicate weak support for the candidate. It can also result from a distribution concentrated on another token. To measure this dependence, we use 500 development images and record, for each image, the probability of the token at vocabulary rank 10 at every visual position. The dispersion persists from rank 1 through rank 1,000 (Appendix B.2). Figure 1 reports the image-median ratio between the 90th and 10th percentiles of this probability across positions. The ratio is 13.8×13.8× for LLaVA-1.5-7B (16) and 7.8×7.8× for Qwen2.5-VL-7B (1). Thus, the probability at the same ordinal position can differ by up to an order of magnitude across visual positions. Equalizing each position’s logit scale to the image median reduces these ratios only to 11.0×11.0× and 4.2×4.2×, respectively, leaving substantial dispersion in place. Therefore, an affine correction alone is insufficient. This dependence propagates to the intervention itself. We test sensitivity to position-specific scale while preserving vocabulary order. We retain each position’s readout direction but reassign the observed positive scale factors among visual positions. We then reapply the same calibrated edit rule. This rescaling does not change the vocabulary ordering within any position. Any resulting change in the edit is therefore due to scale alone. Section 4.4 defines suppression strength as the per-candidate edit magnitude in [0,1][0,1]. Under the probability readout, its mean absolute change is 0.0450.045 in LLaVA-1.5-7B and 0.0460.046 in Qwen2.5-VL-7B. As shown in Table 9 of Appendix B.2, the rank-derived suppression strengths remain exactly unchanged. Ranks, by contrast, are preserved under any strictly increasing transformation at each position. Our narrow claim is that ordinal information provides a transformation-invariant basis for aggregating evidence across visual positions. We pool visual evidence through reciprocal ranks, formalized in Section 4.2 as dense mean reciprocal rank (DMRR). Among order-only statistics, DMRR is a simple parameter-free choice requiring no truncation depth or fitted weights. It emphasizes positions that rank the token near the top while compressing differences among deep ranks. Figure 3: Out-of-sample calibration error of token-specific, global, and shuffled references. Errors are in b0b_0 units with every token counted once. Shading spans the full shuffle range, and the tick marks its median. 3.2 Token-Specific Interpretation of Aggregated Evidence Replacing magnitude with rank makes visual positions comparable, but it does not make vocabulary items interchangeable. Some tokens routinely appear near the head of visual-position rankings. Others have lower typical levels. Figure 2 decomposes DMRR variability over calibration observations of LLaVA-1.5-7B. Each observation is an image-level DMRR value from a decoding step where the token is a plausible next-token candidate. In the image-balanced decomposition, every pair of a token and an image contributes equally. Differences between tokens explain 66.4%66.4\% of the variability, rather than differences between images for the same token. A single global reference therefore conflates evidence that is unusually low for one token with evidence that is entirely typical for another. This structure suggests using each token’s calibration median as its reference. To assess whether this token-level structure generalizes beyond the calibration data, we fit token-specific medians on 500 unlabeled MS COCO images (15) and evaluate them on 4,969 disjoint images. For each token, the calibration error is the gap between its held-out median evidence and the stored reference. Relative to a single global reference, the token-specific medians reduce this error by 92%92\% in LLaVA-1.5-7B and 91%91\% in Qwen2.5-VL-7B. They also outperform all 1,000 random reassignments of references to tokens (Figure 3, Appendix B.3). The gain comes from the correspondence between tokens and their references, not from the distribution of reference values alone. References estimated from few or highly variable observations can misstate typical evidence and risk suppressing well-grounded candidates. ReWEIGH therefore registers a token only when uncertainty in its estimated reference has little effect on the resulting edit, abstaining otherwise (Section 4.3). Together, these diagnostics motivate ordinal aggregation, a token-specific reference, and a stability safeguard for uncertain references. Section 4 turns these ingredients into a bounded decoding intervention. 4 ReWEIGH: Token-Calibrated Ordinal Suppression 4.1 Overview Figure 4: Overview of ReWEIGH. Measure reads visual-token representations as vocabulary rankings. Register constructs a reliable token-specific reference table from unlabeled calibration images. Intervene applies a bounded negative shift to candidates with an evidence deficit. Measure is shared by offline calibration and online inference, whereas registration runs only once. Figure 4 summarizes ReWEIGH’s Measure–Register–Intervene workflow. Offline, Measure aggregates visual-token readouts and Register builds a reliability-filtered token-reference table from unlabeled images. At inference, Measure computes image-level evidence once during prefill, and Intervene uses the cached evidence and frozen table throughout decoding. The resulting training-free method requires no additional model forward pass or external verifier. Algorithm 1 provides the complete procedure. 4.2 Measure: Dense Ordinal Visual Evidence The Measure module quantifies how strongly the image-token states represent token v. Let V denote the language model’s vocabulary. Recall from Eq. (1) that j(ℓ)∈ℝ||z_j^( ) ^|V| is the LM-head readout of visual position j∈Pj∈ P at layer ℓ . Let rankj(v)rank_j(v) denote the rank of zj(ℓ)(v)z_j^( )(v) among the vocabulary scores in descending order, such that the largest score has rank 1. We define the image-level evidence for token v as its dense mean reciprocal rank (DMRR) across visual positions: DMRRI(v)=1|P|∑j∈P1rankj(v).DMRR_I(v)= 1|P| _j∈ P 1rank_j(v). (2) Equation (2) depends only on within-position vocabulary rankings and thus does not require score magnitudes to be comparable across visual positions. Reciprocal rank gives greater weight to positions where v ranks near the top while limiting the contribution of low-ranked positions. The quantity DMRRI(v)DMRR_I(v) depends on the image and readout layer but is independent of the autoregressive step. We therefore compute it once during prefill and cache it for the entire response. CHAIR AMBER Cost Method CHAIRS↓_S CHAIRI↓_I F1↑ CHAIR↓ Cover↑ Hal.↓ Cog.↓ Acc.↑ F1↑ Score↑ Lat.↓ Mem.↓ LLaVA-1.5-7B 52.60 15.61 80.66 3.69 50.47 19.02 3.82 71.96 74.81 85.56 1.00× 1.00× VCD 59.00 17.97 78.50 6.25 51.81 32.07 4.49 66.81 70.37 82.06 2.01× 1.06× OPERA 51.20 14.19 80.96 3.70 48.79 17.03 2.70 74.94 78.09 87.20 7.26× 1.51× DoLa 53.00 15.87 79.28 4.52 51.03 21.12 3.84 72.17 75.09 85.28 1.10× 1.00× PAI 42.40 13.51 75.55 4.68 43.11 19.32 1.97 38.29 18.93 57.12 2.14× 1.02× ReVisiT 51.40 15.44 80.57 4.00 55.15 24.90 2.72 73.09 77.80 86.90 1.02× 1.03× ReWEIGH 44.80 12.67 80.85 2.98 50.21 16.43 3.13 71.96 74.81 85.91 1.02× 1.00× Qwen2.5-VL-7B 31.60 9.58 70.80 4.52 55.80 20.32 1.02 77.85 87.89 91.68 1.00× 1.00× VCD 33.80 9.65 70.27 5.43 63.30 26.99 1.77 81.77 86.66 90.61 2.03× 1.01× OPERA 20.20 7.01 63.79 3.51 63.33 20.92 1.42 83.70 88.17 92.33 8.03× 1.04× DoLa 23.60 16.76 60.32 4.47 51.71 18.92 1.56 82.55 87.15 91.33 1.25× 1.00× PAI 26.20 9.61 66.98 4.25 49.47 18.03 1.10 81.34 86.26 91.00 2.04× 1.00× ReVisiT 24.00 6.75 66.99 4.27 62.30 24.90 1.42 83.28 87.70 91.71 1.05× 1.25× ReWEIGH 25.40 7.54 71.83 3.80 54.65 18.82 0.93 78.19 87.97 92.08 1.01× 1.00× InstructBLIP-7B 49.60 14.03 79.01 4.64 53.74 24.40 3.80 75.91 81.40 88.38 1.00× 1.00× VCD 56.20 17.67 76.48 7.25 53.24 36.16 4.59 70.23 75.94 84.34 1.98× 1.02× OPERA 50.80 14.77 79.02 4.81 52.08 23.80 3.53 75.90 81.32 88.25 11.52× 1.18× DoLa 67.00 20.26 76.84 7.14 55.26 35.86 7.09 75.76 81.06 86.96 1.21× 1.00× PAI 61.00 16.70 77.79 4.63 52.03 22.81 4.04 75.30 80.03 87.70 2.19× 1.01× ReVisiT 43.00 15.02 74.97 4.22 50.99 22.91 2.13 54.36 50.95 73.37 1.01× 1.02× ReWEIGH 46.00 12.50 79.80 4.05 53.28 21.51 3.11 75.96 81.44 88.69 1.01× 1.00× Table 1: Main results on CHAIR and AMBER with decoding cost. The AMBER columns include generative and discriminative metrics, followed by the benchmark’s combined score. Lat. and Mem. are per-token latency and peak allocated memory relative to each backbone’s greedy decoding under the cached-evidence protocol. Green and red shade improvements and degradations of more than 5% against each backbone’s greedy baseline, darker shades mark more than 20%, and unshaded cells differ by less than 5%. Bold and underlined values indicate the best and second-best result, respectively, within each backbone block. Arrows indicate the preferred direction. 4.3 Register: Uncertainty-Aware Token Calibration The Register module asks whether a token-specific reference is reliable enough to drive an intervention. We run the base model without logit intervention on N unlabeled calibration images that are disjoint from the evaluation data. For calibration image IiI_i with TiT_i decoding steps, let i,tC_i,t be the candidate set obtained by top-p selection under the base-model distribution, subject to minimum and maximum sizes. We define the candidate-conditioned calibration multiset as Dv=⨄1≤i≤N, 1≤t≤Tiv∈i,tDMRRIi(v),D_v= _ subarrayc1≤ i≤ N,\;1≤ t≤ T_i\\ v _i,t subarray \DMRR_I_i(v) \, (3) where ⨄ denotes multiset union and therefore preserves repeated candidate occurrences. Thus, DvD_v records the visual evidence for v only at steps where the model considers it a plausible next-token candidate. We summarize the observations using a token-specific reference and a pooled normalization scale: b(v) b(v) =median(Dv), =median(D_v), (4) b0 b_0 =median(⨄vDv), =median\! ( _vD_v ), The token reference b(v)b(v) estimates the typical evidence level of token v as a decoder candidate under the calibration distribution (Section 3.2), whereas b0b_0 provides a shared normalization scale across tokens. Tokens occur at different frequencies during calibration, so the reliability of b(v)b(v) also varies. To account for this uncertainty, we construct a nominal 95% order-statistic range over the candidate-occurrence multiset DvD_v, denoted by [blo(v),bhi(v)][b_lo(v),b_hi(v)]. We use it as a conservative stability diagnostic rather than a coverage-guaranteed confidence interval (Appendix C.3). We then define the normalized edit induced by a reference b and observation x as e(b,x)=clip(b−xb0,0,1).e(b,x)=clip\! ( b-xb_0,0,1 ). (5) We quantify estimation uncertainty by its effect on the bounded edit: Δe(v)=1|Dv|∑x∈Dv|e(bhi(v),x)−e(blo(v),x)|. e(v)= 1|D_v| _x∈ D_v |e\! (b_hi(v),x )-e\! (b_lo(v),x ) |. (6) We register token v only if its range exists and Δe(v)<0.5 e(v)<0.5. Otherwise, the method abstains from editing it. The frozen table T stores the global scale b0b_0 and maps each registered vocabulary token v∈ℛv to its token-specific reference b(v)b(v). Appendix C.3 provides the range construction and Appendix A the candidate-set configuration. 4.4 Intervene: Bounded Evidence-Deficit Suppression The Intervene module determines how strongly to suppress a candidate that lacks visual support. For a registered token, we reuse Eq. (5) to compute an image-specific suppression strength sI(v) s_I(v) =e(b(v),DMRRI(v)). =e\! (b(v),DMRR_I(v) ). (7) At decoding step t, let t∈ℝ||z_t ^|V| denote the decoder logits before intervention, zt(v)z_t(v) their component for token v∈v , and tC_t the candidate set constructed exactly as during calibration. Given maximum penalty β≥0β≥ 0, the edited decoder logit is zt′(v)=zt(v)−βsI(v),v∈t∩ℛ,zt(v),otherwise.z _t(v)= casesz_t(v)-β s_I(v),&v _t ,\\[3.0pt] z_t(v),&otherwise. cases (8) Backbone Metric Base VCD OPERA DoLa PAI ReVisiT ReWEIGH LLaVA-1.5-7B MMHal Score↑ 2.76 2.16 2.57 2.65 2.28 2.33 2.78 MMHal Hall. Rate (%)↓ 51.04 65.62 53.12 53.12 43.75 58.33 48.96 M-Vet Acc.↑ 35.41 32.84 36.74 36.06 18.26 33.58 37.20 Qwen2.5-VL-7B MMHal Score↑ 4.28 4.11 4.50 3.69 3.27 3.78 4.42 MMHal Hall. Rate (%)↓ 15.62 17.71 17.71 20.83 26.04 19.79 15.62 M-Vet Acc.↑ 56.65 60.87 70.96 46.97 44.86 56.38 59.54 InstructBLIP-7B MMHal Score↑ 2.34 2.19 2.34 2.33 2.36 2.00 2.54 MMHal Hall. Rate (%)↓ 55.21 60.42 55.21 54.17 54.17 60.42 48.96 M-Vet Acc.↑ 29.17 30.78 35.23 31.61 31.19 20.09 29.63 LLaVA-NeXT-7B MMHal Score↑ 3.84 3.46 3.71 3.96 3.55 3.47 4.02 MMHal Hall. Rate (%)↓ 31.25 41.67 37.50 32.29 38.54 37.50 29.17 M-Vet Acc.↑ 48.21 46.19 52.02 52.43 49.50 49.31 50.05 Table 2: MMHal-Bench and M-Vet results. Hall. Rate is the percentage of examples with score below 3, and M-Vet accuracy is on a 0–100 scale. The lower clipping bound makes the update suppression-only. Evidence at or above the token reference yields no edit, whereas lower evidence can reduce a candidate’s logit by at most β. Candidates without registered references and tokens outside tC_t remain unchanged. At each decoding step, we recompute the candidate set from the pre-edit distribution. In contrast, we compute sIs_I once per image and cache it throughout decoding. Overall, online inference combines the frozen calibration table T with a single DMRR computation at prefill, after which each decoding step requires only candidate selection and bounded logit updates. Appendix E.1 traces this arithmetic at a recorded decoding step, where full, proportional, and zero suppression occur within a single candidate set. 5 Experiments 5.1 Experimental Setup Models. We evaluate four 7B LVLMs with different visual interfaces and language backbones, namely LLaVA-1.5-7B (16), Qwen2.5-VL-7B (1), InstructBLIP-7B (5), and LLaVA-NeXT-7B (17). Appendix A lists the exact checkpoint identifiers. Configuration CHAIR AMBER CS↓C_S\, CI↓C_I\, F1 ↑ CHAIR ↓ Cover ↑ Score ↑ [3pt][3pt] Full method 44.8 12.67 80.85 2.98 50.21 85.91 Global reference 58.4 17.45 80.31 4.10 52.78 85.36 Shuffled references 56.0 14.74 81.54 4.11 51.63 85.36 Mismatched evidence 50.2 14.45 79.73 3.70 49.35 85.56 No abstention 44.4 12.64 80.90 2.97 50.37 86.62 Bidirectional update 50.4 16.07 79.74 3.72 51.22 85.54 Unbounded update 39.2 29.71 69.09 3.44 48.26 85.69 Table 3: Component ablations on LLaVA-1.5-7B. CSC_S and CIC_I denote CHAIRS and CHAIRI, and arrows indicate the preferred direction. Arm definitions are in Table 19. Benchmarks. CHAIR (21) evaluates object hallucination in MS COCO captions. AMBER (25) provides generative and discriminative evaluations. MMHal-Bench (23) evaluates open-ended multimodal responses. We use M-Vet (27) to measure general multimodal capability. Because a model can reduce hallucination simply by saying less, we read hallucination metrics together with recall-oriented measures and general capability. Appendix A.3 gives the splits, prompts, metrics, and evaluator contracts. Implementation. We calibrate on 500 MS COCO training images and define the candidate set as a top-p probability prefix with p=0.9p=0.9, restricted to between 2 and 50 tokens. We select the readout layer and β once per backbone on 500 images sampled from MS COCO train2014 with seed 42. We then freeze the selected setting across benchmarks. CHAIR results come from a separate 500-image subset of MS COCO val2014, so no evaluation image enters operating-point selection. We compare with VCD (13), OPERA (8), DoLa (4), PAI (19), and ReVisiT (3). Appendix A provides further details. Figure 5: Calibration-size sweep on LLaVA-1.5-7B. Dashed lines mark base decoding, and the open circle marks the deployed 500-image table. 5.2 Main Results Visual Grounding and Hallucination Mitigation. Table 1 and Appendix D summarize fine-grained visual grounding and hallucination mitigation on CHAIR and AMBER. Across the four models, ReWEIGH reduces CHAIRI by 10.3%10.3\%–21.3%21.3\% while largely preserving or improving F1, so it does not simply suppress object mentions. The improvement generalizes to 11 models spanning six architecture families and model sizes from 7B to 32B, and ReWEIGH reduces both CHAIRS and CHAIRI for every model (Appendix D.4). On AMBER generation, ReWEIGH lowers CHAIR and hallucination rate and improves the AMBER score for all four backbones. OPERA reduces CHAIRS more strongly on Qwen, and PAI reaches a lower CHAIRS on LLaVA-1.5, but both give up more F1 or discriminative performance. ReWEIGH balances hallucination reduction and retained content more consistently. Under the shading rule of Table 1, ReWEIGH is the only method that never degrades a baseline metric by more than 5%5\% on any backbone. Its largest regression is a 2.1%2.1\% relative decrease in AMBER coverage. Open-ended Reliability and General Multimodal Utility. Table 2 shows that ReWEIGH improves MMHal-Bench quality and M-Vet accuracy across all four backbones, while reducing or maintaining the MMHal-Bench hallucination rate. 5.3 Component and Counterfactual Analysis Impact of the Ordinal Readout. Under a matched mean edit budget, replacing DMRR with probability pooling in Table 13 increases CHAIRS from 44.844.8 to 50.050.0. This comparison supports the ordinal readout as part of the complete intervention. Impact of Token-Specific Calibration. Table 3 shows that global or shuffled references degrade performance. Because shuffling preserves the marginal reference distribution, the degradation isolates the correspondence between tokens and their references. Impact of Calibration Size. Figure 5 refits the calibration table on nested COCO subsets from 10 to 1,000 images at the frozen operating point. Ten images already recover half of the CHAIRS reduction, and 100 images match the default CHAIRS while keeping F1 above base decoding. We fixed the 500-image calibration table before these runs, so the reported results do not sit at this sweep’s optimum (Appendix D.3). Impact of Image-Specific Evidence. The mismatched-evidence arm in Table 3 rotates the cached evidence vector across images while holding every other component fixed. Appendix D extends this counterfactual with further substitution variants (Table 21). Every substitution recovers less than half of the full method’s CHAIRS reduction and pushes F1 below base decoding, so the intervention draws on evidence about the current image, not only on which tokens are registered. Impact of Conservative Suppression. Table 3 shows that bidirectional updates degrade CHAIRS, CHAIRI, and F1. Unbounded updates lower CHAIRS but induce repetition, which sharply increases CHAIRI and reduces F1. In contrast, removing abstention has little effect on the aggregate metrics and affects only about 3.5%3.5\% of candidate positions. We therefore treat registration as a conservative reference-stability rule rather than a measured source of aggregate gains. 5.4 Inference Efficiency The Lat. and Mem. columns of Table 1 report per-token decoding latency and peak allocated memory relative to greedy decoding, measured separately on three backbones over 100 prompts each and excluding offline calibration. Across the three backbones, ReWEIGH adds 1.33%1.33\% latency after the evidence has been cached. Including online DMRR computation during prefill yields 2.40%2.40\% end-to-end latency overhead. Peak allocated memory increases by 0.31%0.31\% under the cached-evidence protocol. Appendix C.5 details both latency paths and the evidence-cache footprint. 6 Conclusion We introduced ReWEIGH, a training-free decoder that aggregates vocabulary ranks across visual positions and compares each candidate with a reliably estimated token-specific reference. A bounded, one-sided penalty converts evidence deficits into logit edits, and prefill-time computation lets generation reuse the evidence. Across four 7B backbones, ReWEIGH reduces hallucination while preserving or improving descriptive and general multimodal utility, and the reductions extend to 11 models across six architecture families from 7B to 32B. These results support token-calibrated ordinal readouts as a practical interface between internal visual states and generation. Limitations ReWEIGH requires access to visual-token hidden states, the output normalization, and the vocabulary head, so it cannot operate through closed APIs. Our current implementation uses a separate calibration table, readout layer, and intervention strength for each backbone. We evaluate 11 models from six architecture families, but we calibrate and tune each model separately. Our experiments therefore do not establish whether these components transfer across backbones without additional calibration or development data. Future work could derive references that transfer across models and choose operating points automatically. The method derives its corrective signal entirely from the model’s visual representations and does not consult external sources. It can lower the logits of candidates that the image does not support, but it cannot add facts that neither the image nor the backbone contains. It therefore offers no direct way to correct errors that require current or specialized knowledge. Our open-ended and general multimodal benchmarks do not test this type of factual error. Future work could combine ordinal visual evidence with retrieval or external verification. We conduct all calibration, development, and evaluation in English. We construct token-specific references over each model’s vocabulary, so differences in multilingual tokenization and alignment may change candidate frequencies and visual-evidence baselines. Our results do not show whether the method behaves similarly in other languages, especially when languages differ in morphology or tokenization. We could repeat calibration in another language, but we do not know whether a reference table can transfer across languages or requires separate calibration. Future work should test the method in multilingual settings. Ethical Considerations Reducing visually unsupported statements can improve the reliability of LVLM outputs, but the method does not make a model safe for high-stakes use. A bounded intervention can still leave hallucinations unchanged or replace them with different errors. Users should independently verify claims in medical, legal, scientific, and public-safety settings. The method requires access to internal model states and an offline calibration corpus. Deployments should respect the licenses, privacy constraints, and provenance requirements of both model checkpoints and calibration images. Calibration data can also encode geographic, cultural, and frequency biases. Token-specific references may reproduce these biases by treating common visual concepts as better calibrated than rare ones. We report abstention for unstable references and avoid presenting lower benchmark scores as evidence of universal factual reliability. References Bai et al. (2025) S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin Qwen2.5-VL technical report. External Links: 2502.13923, Document, Link Cited by: §C.1, §3.1, §5.1. Chen et al. (2024) Z. Chen, Z. Zhao, H. Luo, H. Yao, B. Li, and J. Zhou HALC: object hallucination reduction via adaptive focal-contrast decoding. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, Vienna, Austria, p. 7824–7846. External Links: Link Cited by: §2. Cho and Kim (2026) B. Cho and J. Kim Revisit what you see: revealing visual semantics in vision tokens to guide LVLM decoding. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), San Diego, California, United States, p. 5794–5824. External Links: Document, Link Cited by: §1, §2, §3, §3.1, §5.1. Chuang et al. (2024) Y. Chuang, Y. Xie, H. Luo, Y. Kim, J. R. Glass, and P. He DoLa: decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §5.1. Dai et al. (2023) W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. C. Hoi InstructBLIP: towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 49250–49267. External Links: Document, Link Cited by: §C.1, §1, §5.1. Favero et al. (2024) A. Favero, L. Zancato, M. Trager, S. Choudhary, P. Perera, A. Achille, A. Swaminathan, and S. Soatto Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, p. 14303–14312. External Links: Document, Link Cited by: §1, §1, §2. Gong et al. (2024) X. Gong, T. Ming, X. Wang, and Z. Wei DAMRO: dive into the attention mechanism of LVLM to reduce object hallucination. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA, p. 7696–7712. External Links: Document, Link Cited by: §2. Huang et al. (2024) Q. Huang, X. Dong, P. Zhang, B. Wang, C. He, J. Wang, D. Lin, W. Zhang, and N. Yu OPERA: alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, p. 13418–13427. External Links: Document, Link Cited by: §1, §2, §5.1. Hudson and Manning (2019) D. A. Hudson and C. D. Manning GQA: a new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, p. 6700–6709. External Links: Document, Link Cited by: §A.6, §D.3. Huo et al. (2025) F. Huo, W. Xu, Z. Zhang, H. Wang, Z. Chen, and P. Zhao Self-introspective decoding: alleviating hallucinations for large vision-language models. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2. Jiang et al. (2025) N. Jiang, A. Kachinthaya, S. Petryk, and Y. Gandelsman Interpreting and editing vision-language representations to mitigate hallucinations. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §1, §2, §3, §3.1. Kim et al. (2024) J. Kim, H. J. Kim, Y. J. Kim, and Y. M. Ro CODE: contrasting self-generated description to combat hallucination in large multi-modal models. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, p. 133571–133599. External Links: Document, Link Cited by: §1, §2. Leng et al. (2024) S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, p. 13872–13882. External Links: Document, Link Cited by: §1, §1, §2, §5.1. Li et al. (2023) Y. Li, Y. Du, K. Zhou, J. Wang, X. Zhao, and J. Wen Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, p. 292–305. External Links: Document, Link Cited by: §1, §3. Lin et al. (2014) T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick Microsoft COCO: common objects in context. In Computer Vision – ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars (Eds.), Lecture Notes in Computer Science, Vol. 8693, Zurich, Switzerland, p. 740–755. External Links: Document, Link Cited by: §A.2, §A.2, §A.6, §3.2. Liu et al. (2024a) H. Liu, C. Li, Y. Li, and Y. J. Lee Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, p. 26296–26306. External Links: Document, Link Cited by: §C.1, §3.1, §5.1. Liu et al. (2024b) H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee LLaVA-NeXT: improved reasoning, OCR, and world knowledge. Note: Blog post, LLaVA project blogPublished 30 January 2024. Accessed 2026-08-04 External Links: Link Cited by: §C.1, §5.1. Liu et al. (2023) H. Liu, C. Li, Q. Wu, and Y. J. Lee Visual instruction tuning. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 34892–34916. External Links: Document, Link Cited by: §C.1, §1. Liu et al. (2024c) S. Liu, K. Zheng, and W. Chen Paying more attention to image: a training-free method for alleviating hallucination in LVLMs. In Computer Vision – ECCV 2024, A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol (Eds.), Lecture Notes in Computer Science, Vol. 15141, Milan, Italy, p. 125–140. External Links: Document, Link Cited by: §1, §2, §5.1. nostalgebraist (2020) nostalgebraist Interpreting GPT: the logit lens. Note: Blog post, LessWrongPublished 31 August 2020; cross-posted to the AI Alignment Forum. Accessed 2026-08-04 External Links: Link Cited by: §1, §2, §3. Rohrbach et al. (2018) A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Saenko Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, p. 4035–4045. External Links: Document, Link Cited by: §A.3.1, §1, §5.1. Su et al. (2025) J. Su, J. Chen, H. Li, Y. Chen, L. Qing, and Z. Zhang Activation steering decoding: mitigating hallucination in large vision-language models through bidirectional hidden state intervention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, p. 12964–12974. External Links: Document, Link Cited by: §2. Sun et al. (2024) Z. Sun, S. Shen, S. Cao, H. Liu, C. Li, Y. Shen, C. Gan, L. Gui, Y. Wang, Y. Yang, K. Keutzer, and T. Darrell Aligning large multimodal models with factually augmented RLHF. In Findings of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand, p. 13088–13110. External Links: Document, Link Cited by: §A.3.3, §A.6, §5.1. Wang et al. (2025) C. Wang, X. Chen, N. Zhang, B. Tian, H. Xu, S. Deng, and H. Chen MLLM can see? dynamic correction decoding for hallucination mitigation. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2. Wang et al. (2023) J. Wang, Y. Wang, G. Xu, J. Zhang, Y. Gu, H. Jia, J. Wang, H. Xu, M. Yan, J. Zhang, and J. Sang AMBER: an LLM-free multi-dimensional benchmark for MLLMs hallucination evaluation. External Links: 2311.07397, Document, Link Cited by: §A.3.2, §A.6, §1, §3, §5.1. Wang et al. (2024) X. Wang, J. Pan, L. Ding, and C. Biemann Mitigating hallucinations in large vision-language models with instruction contrastive decoding. In Findings of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand, p. 15840–15853. External Links: Document, Link Cited by: §1, §2. Yu et al. (2024) W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang M-Vet: evaluating large multimodal models for integrated capabilities. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, Vienna, Austria, p. 57730–57754. External Links: Link Cited by: §A.3.4, §A.6, §5.1. Zou et al. (2025) X. Zou, Y. Wang, Y. Yan, Y. Lyu, K. Zheng, S. Huang, J. Chen, P. Jiang, J. Liu, C. Tang, and X. Hu Look twice before you answer: memory-space visual retracing for hallucination mitigation in multimodal large language models. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, Vancouver, Canada, p. 80873–80899. External Links: Link Cited by: §1, §2. Appendix A Experimental and Evaluation Details A.1 Model Checkpoints Table 4 lists the exact model checkpoints used in our experiments together with their language backbones. We use the abbreviated model names in the first column consistently throughout the paper. Model Checkpoint ID Language backbone InstructBLIP-7B Salesforce/LAVIS blip2_vicuna_instruct/vicuna7b Vicuna-7B LLaVA-1.5-7B liuhaotian/llava-v1.5-7b Vicuna-7B-v1.5 LLaVA-NeXT-7B liuhaotian/llava-v1.6-mistral-7b Mistral-7B-Instruct-v0.2 Qwen2.5-VL-7B Qwen/Qwen2.5-VL-7B-Instruct Qwen2.5-7B InternVL2.5-8B OpenGVLab/InternVL2_5-8B internlm/internlm2_5-7b-chat Gemma-3-12B google/gemma-3-12b-it Gemma-3-12B InstructBLIP-13B Salesforce/LAVIS blip2_vicuna_instruct/vicuna13b lmsys/vicuna-13b-v1.1 LLaVA-1.5-13B liuhaotian/llava-v1.5-13b Vicuna-13B-v1.5 InternVL2.5-26B OpenGVLab/InternVL2_5-26B internlm/internlm2_5-20b-chat Gemma-3-27B google/gemma-3-27b-it Gemma-3-27B Qwen2.5-VL-32B Qwen/Qwen2.5-VL-32B-Instruct Qwen2.5-32B Table 4: Model checkpoints and their language backbones. A.2 Calibration Data and Operating-Point Selection For each backbone, we sample 500 images from MS COCO train2014 (15) with seed 4242. The calibration images are disjoint from both the seed-42 tuning subset used for operating-point selection and the evaluation manifests. We run the unmodified model greedily and, at each step, collect the DMRR value of every token in the shortest probability prefix whose mass reaches 0.9. We enforce a minimum of two candidates and cap the prefix at 50 tokens. We condition token observations on candidacy and do not sweep the vocabulary at every decoding step. Backbone ℓ β b0b_0 Registered LLaVA-1.5 29 1.1 0.00324809 2,580 LLaVA-NeXT 30 1.2 0.00274012 3,734 InstructBLIP 29 0.5 0.00184383 2,708 Qwen2.5-VL 27 1.1 0.000653155 7,028 Table 5: Frozen backbone-specific operating points and reference-stability table statistics. For operating-point selection, we sample 500 images from MS COCO train2014 with seed 42. The initial grid pairs the three decoder layers with β∈0.5,0.7,0.9,1.1,1.3β∈\0.5,0.7,0.9,1.1,1.3\, which gives 15 configurations. We then identify the most promising pair of adjacent strengths and evaluate their midpoint at all three layers. This adds three configurations and gives 18 in total per backbone. We inspect CHAIRS, CHAIRI, recall, and F1 together and select the operating point qualitatively rather than optimizing a single aggregate score. We then freeze the selected layer and β for all remaining benchmarks. Table 5 lists the frozen operating point (ℓ , β, b0b_0) and the number of registered tokens for each backbone. We report CHAIR results on 500 images sampled from MS COCO val2014. MS COCO defines train2014 and val2014 as distinct image splits and explicitly limits near-duplicates across splits (15). No evaluation image is therefore used for operating-point selection, and we use no benchmark-specific strength. The calibration-size sweep in Figure 5 uses a single nested sequence built from the frozen seed-4242 panel. We sort the panel by COCO image ID and take the first n images as the subsets up to n=500n=500. For larger sizes, we continue the same random stream, sample 500 additional images without replacement from the remaining pool, sort them by image ID, and append them to the frozen panel. The 700- and 1,000-image sets are prefixes of this extended sequence, so every smaller subset is contained in every larger one. The 500-image arm reproduces the deployed configuration exactly. Refitting from this prefix recovered the deployed b0b_0, all 2,580 registered token references, and their observation counts, and the arm reuses the frozen evaluation output. A.3 Evaluation Protocols Table 6 summarizes the evaluation splits and reported metrics. Unless otherwise stated, we follow the setup in each benchmark’s original paper and released official evaluator as closely as possible. We select CHAIR hyperparameters on the seed-42 COCO train2014 tuning subset and report results on the separate COCO val2014 subset listed in the table. We hold prompts, decoding limits, and evaluator contracts fixed across the base model, ReWEIGH, and the compared methods. Benchmark Split Records Max. Reported metrics CHAIR MS COCO val500 500 512 CSC_S, CIC_I, object recall, F1, length AMBER–Generative official generative 1,004 128 CHAIR, Cover, Hal., Cog. AMBER–Discriminative official discriminative 14,216 8 accuracy, precision, recall, F1, unparsed, AMBER score MMHal-Bench full set 96 256 quality score, hallucination rate M-Vet original test 218 1,024 total score Table 6: Evaluation datasets, record counts, maximum numbers of newly generated tokens (Max.), and reported metrics. A.3.1 CHAIR We evaluate CHAIR (21) on a fixed 500-image subset of MS COCO val2014, disjoint from the tuning and calibration images. The image identifiers are fixed by the evaluation manifest sampled with seed 42. We query every image with the exact prompt “Please describe this image in detail.” and cap generation at 512 new tokens. We use the official CHAIR object vocabulary and synonym mapping. Following the benchmark protocol, the reference object set for each image is the union of objects in the MS COCO instance annotations and the reference captions. CHAIRS is the fraction of captions that contain at least one hallucinated object mention, whereas CHAIRI is the fraction of generated object mentions that are hallucinated. Object recall is corpus-micro recall, computed by summing the numbers of recovered and reference object categories across images. We additionally derive object precision as 1−CHAIRI1-CHAIR_I and report its harmonic mean with object recall as F1. Length is the mean number of whitespace-delimited words in the generated captions. Every method uses the same fixed manifest, prompt, synonym mapping, and scorer. As described above, we select the readout layer ℓ and β on the COCO train2014 tuning subset before scoring the 500 COCO val2014 captions. A.3.2 AMBER AMBER (25) contains a 1,004-example generative split and a 14,216-question discriminative split. The latter comprises 4,924 existence, 7,628 attribute, and 1,664 relation questions. For the generative split, we use the exact prompt “Describe this image.” and generate at most 128 new tokens. The official noun-based scorer extracts and lemmatizes nouns with NLTK and matches them using the released association and safe-word resources together with the en_core_web_lg similarity model at threshold 0.8. We report its CHAIR, object coverage (Cover), caption-level hallucination rate (Hal.), and cognitive hallucination (Cog.) metrics. These quantities use AMBER’s released vocabulary and should not be numerically equated with MS COCO CHAIRI. For the discriminative split, we preserve each released yes/no question verbatim, add no answer-format suffix, and cap generation at 8 new tokens. To avoid silently mapping a free-form response to a binary class, we normalize Unicode and case, remove leading whitespace and punctuation, and accept a response only if its first token is yes or no. We mark all other responses as unparsed and count them as incorrect for accuracy. An unparsed example whose ground-truth label is no also counts as a false negative for recall. We treat no, which denotes the hallucinatory case, as the positive class and report accuracy, precision, recall, F1, and the number and percentage of unparsed responses. F1 uses the scorer’s 10−410^-4 denominator smoothing. The reported combined score is AMBERScore=12[(100−CHAIR)+F1],AMBER\ Score= 12 [(100-CHAIR)+F1 ], (9) with both constituent metrics expressed on a 0–100 scale. A.3.3 MMHal-Bench We use all 96 MMHal-Bench examples (23), comprising 12 examples from each of eight question types: attribute, adversarial, comparison, counting, relation, environment, holistic, and other. We pass the released question verbatim without an additional prompt suffix and allow at most 256 new tokens. We retain the released five-example evaluator prompt and rubric without modification. This text-only evaluator receives the released image-content object list, question, human reference answer, and model response. It does not receive the image itself. Its rubric assigns an integer score from 0 to 6 according to informativeness and the presence of hallucination. Scores 0–2 denote a response with hallucination, and scores 3–6 denote a response without hallucination. We report the mean score and the percentage of examples assigned a score below 3. All configurations are judged with the frozen gpt-4.1-mini-2025-04-14 snapshot at temperature 0. The parser extracts a “Rating: s” entry for s∈0,…,6s∈\0,…,6\, and an output that does not yield a unique valid rating receives score 0. Thus, parse failures receive identical treatment for every method and stay in the denominator. A.3.4 M-Vet We evaluate the original M-Vet test set (27), which contains 218 questions over 200 images. Each released question is used verbatim without an answer-format suffix, and generation is capped at 1,024 new tokens. The official text-only grading contract retains the released six-example prompt without modification. It compares the question, released reference answer, and model prediction, and returns one of 0.0,0.1,…,1.0\0.0,0.1,…,1.0\. The prompt explicitly preserves the benchmark’s <AND> semantics, which require all referenced elements, and <OR> semantics, which accept any listed alternative. We use gpt-4.1-mini-2025-04-14 at temperature 0. An evaluator output that does not yield a valid score receives score 0. The total M-Vet score is 100 times the mean item grade. The evaluator also defines capability breakdowns over recognition (150), OCR (96), knowledge (84), generation (80), spatial reasoning (75), and mathematics (26). These capability tags overlap, so their sample counts do not form a partition of the 218 questions. The same judge snapshot, prompt contract, and one-grade policy are used for every M-Vet row in Table 2. A.4 Generation Settings The base model and ReWEIGH use batch size 1, eager attention, one beam, and no sampling. The generation seed is 42. The checkpoint-native repetition penalty is 1.05 for Qwen2.5-VL and 1.0 for the other backbones. Benchmark-specific decoding limits are fixed in the evaluation manifests, and the method and base model use the same limit and prompt. For all baseline methods, we use the authors’ official implementations and the default configurations specified in the corresponding papers and repositories. Each baseline therefore keeps the decoding regime of its original paper. OPERA searches with five beams, which also accounts for the decoding cost we report for it in Table 1, and VCD samples instead of decoding greedily. A.5 Hardware and Software Each generation job uses an NVIDIA A100-SXM4-80GB GPU, with approximately 250 total GPU-hours used across all experiments. LLaVA-1.5 uses Python 3.10.20, PyTorch 2.1.2+cu121, and Transformers 4.40.0. LLaVA-NeXT and Qwen2.5-VL use Python 3.10.20, PyTorch 2.5.1+cu124, and Transformers 4.52.1. The original LAVIS InstructBLIP environment uses Python 3.9.16, PyTorch 2.0.1+cu117, and Transformers 4.33.2. LLaVA and InstructBLIP run in FP16, and Qwen2.5-VL runs in BF16. For every run, we record the model revision, the evaluation-manifest and prompt hashes, the calibration-table hash, the decoding configuration, and the runtime environment. A.6 Artifact Licenses and Intended Use We use all datasets, model checkpoints, and evaluation tools for non-commercial research, consistent with their intended use and stated terms. MS COCO annotations are released under the Creative Commons Attribution 4.0 license, and the underlying images are governed by the Flickr Terms of Use (15). AMBER (25) and MMHal-Bench (23) are released under the Apache-2.0 license, and GQA (9) under C BY 4.0. The M-Vet evaluation code is Apache-2.0 and its data are distributed under C BY-NC 4.0 (27). Among the checkpoints in Table 4, Qwen2.5-VL-7B and LLaVA-NeXT-7B (Mistral-7B-Instruct-v0.2 backbone) are Apache-2.0, LLaVA-1.5-7B is subject to the Llama 2 Community License through its Vicuna-v1.5 backbone, and InstructBLIP-7B follows the non-commercial research terms of its LLaMA-based Vicuna backbone. The AMBER scorer depends on NLTK (Apache-2.0) and the spaCy en_core_web_lg model (MIT), and the LLM judge is accessed through the OpenAI API under its terms of use. Qualitative figures reproduce individual benchmark images solely for scholarly analysis. Appendix B Additional Diagnostic Analyses B.1 Scope of the Diagnostic Analyses The analyses in Section 3 answer two design questions. The first asks which property of a visual-position vocabulary readout can be pooled across positions without inheriting position-specific cardinal scale. The second asks which reference makes the pooled evidence interpretable for a particular token. These analyses motivate an intervention design. We do not claim that rank is a universally superior hallucination detector or that token-specific centering improves standalone classification. We conduct the diagnostic analyses on LLaVA-1.5-7B and Qwen2.5-VL-7B at their selected readout layers. The scale-dispersion and token-identity-share measurements additionally cover all four evaluated backbones. The two models have different visual-token counts and different absolute DMRR scales. We therefore report within-backbone effects and do not compare absolute evidence values across models. B.2 Rank-Preserving Scale Analysis Let j(ℓ)z_j^( ) be the vocabulary readout of visual position j. Multiplying this vector by a positive scalar preserves its complete vocabulary ordering but changes the probability distribution obtained by softmax. The controlled test retains the observed readout directions and reassigns positive scale factors among visual positions. It is an invariance stress test rather than a model of a naturally occurring perturbation. Its role is to isolate sensitivity to position-wise cardinal scale while every within-position vocabulary ordering stays fixed. Consequently, every token retains its rank within each position, while a probability aggregate can change because the positions now contribute distributions with different sharpness. The probability arm replaces DMRR with the logarithm of each token’s position-averaged softmax probability, floored at 10−810^-8, and mirrors the calibration of Section 4.3: the token reference is the candidate-conditioned median of this readout, and registration applies the same Δe(v)<0.5 e(v)<0.5 rule. One substitution is forced by the readout’s range. The pooled median of the log-probability readout is negative (−8.6-8.6 for LLaVA-1.5-7B and −10.2-10.2 for Qwen2.5-VL-7B), so reusing it as the normalization scale b0b_0 in Eq. (5) would flip the sign of the normalized deficit, clipping every evidence deficit to zero while directing suppression toward evidence surpluses. The probability arm therefore normalizes by a positive scale solved on the calibration images so that its mean deployed edit matches the DMRR arm’s under the same maximum penalty (0.9660.966 for LLaVA-1.5-7B and 0.9050.905 for Qwen2.5-VL-7B), and the registration rule is evaluated under the same positive scale. With this substitution, the two arms share the bounded evidence-deficit form of Eqs. (5) and (7) under a matched mean edit budget and differ in the underlying readout and its normalization scale. The maximum rank change under this manipulation is zero. With the original calibration table fixed, the probability-derived suppression strengths shift by a mean absolute 0.04450.0445 for LLaVA and 0.04620.0462 for Qwen, whereas the rank-derived suppression strengths remain exactly unchanged. Equalizing the scale closes only 6.94%6.94\% and 8.76%8.76\% of the full probability-to-rank gap in suppression strengths. The equalization comparison uses the raw position-averaged probability rather than its logarithm, whose pooled calibration median is positive and therefore serves directly as the normalization scale, with the penalty refit so this arm meets the same mean edit budget as the DMRR arm. Scale equalization also leaves most of the fixed-rank dispersion in place. Setting every position’s logit scale to the image median reduces the rank-10 p90/p10 spread from 13.82×13.82× to 11.05×11.05× on LLaVA and from 7.79×7.79× to 4.25×4.25× on Qwen. Between 67%67\% and 95%95\% of the log-probability spread survives across the measured depths 1–1,000 for both backbones (Table 8). Position-wise distribution shape therefore differs beyond what a per-position rescaling can correct. The result identifies cardinal scale as one nuisance in probability pooling, not as a complete explanation of the end-to-end difference between the two interventions. Three tables document the position-scale diagnostics. Table 7 extends the dispersion measurement to all four evaluated backbones. It fixes a probability threshold and reports the across-position spread of its rank depth, complementing Figure 1, which fixes a vocabulary rank and reports the spread of its probability. At every selected readout layer, the rank depth of a fixed probability threshold varies across positions by at least a factor of four on 5,000 development images, and the 200-image measurement closely matches. Table 8 reports the depth-wise spreads on 500 development images before and after scale equalization. Its rank-10 rows underlie Figure 1, and the original rank-10 spreads carry retrospective image-bootstrap 95% intervals of [2.568, 2.723][2.568,\,2.723] and [2.032, 2.068][2.032,\,2.068] in log units, or [13.0×, 15.2×][13.0×,\,15.2×] and [7.6×, 7.9×][7.6×,\,7.9×] as ratios. Table 9 reports the rescaling stress test and the scale-equalization closure with their bootstrap intervals. Threshold τ LLaVA-1.5-7B InstructBLIP-7B LLaVA-NeXT-7B Qwen2.5-VL-7B 10−310^-3 8.6 (8.5) 4.1 (3.9) 9.2 (9.2) 5.8 (5.7) 10−410^-4 62.5 (59.4) 7.1 (6.7) 87.7 (84.0) 7.4 (7.4) 10−510^-5 93.8 (89.4) 4.6 (4.6) 477.3 (459.4) 11.5 (10.9) Table 7: Across-position dispersion of the rank depth of a fixed probability threshold. Each cell is the image-median p90/p10 ratio of Nj(τ)N_j(τ), the number of vocabulary items with probability at least τ at visual position j. Values in parentheses use the 200-image measurement. LLaVA-1.5-7B Qwen2.5-VL-7B Rank depth k Original Equalized Surv. Original Equalized Surv. 1 4.64 3.86 83% 4.11 3.53 86% 2 3.59 2.91 81% 3.29 2.70 82% 5 2.55 2.15 84% 2.47 1.87 76% 10 2.63 2.40 91% 2.05 1.45 70% 20 — — — 1.79 1.20 67% 50 3.43 3.26 95% 1.58 1.18 75% 100 3.87 3.64 94% 1.53 1.28 84% 500 4.63 4.14 90% — — — 1,000 4.84 4.22 87% 1.83 1.57 86% Table 8: Across-position dispersion of the rank-k log probability before and after scale equalization. Each cell is the image-median p90–p10 spread in log units. Surv. is the share of the original spread that survives equalization. Depths absent from a backbone’s stored grid are marked —. LLaVA-1.5-7B Qwen2.5-VL-7B Signal Quantity Value 95% CI Value 95% CI Rank Suppression-strength change 0†0 — 0†0 — Probability Suppression-strength mean |Δ|| | 0.0445 [0.0430, 0.0460][0.0430,\,0.0460] 0.0462 [0.0447, 0.0478][0.0447,\,0.0478] Candidate-score Spearman 0.964 — 0.961 — Edited-set Jaccard 0.805 — 0.857 — Strongest edited candidate changed 18.6% — 17.8% — One-step winner changed 1.7% — 2.5% — Equalization Probability-to-rank gap closed 6.94% [6.52, 7.36][6.52,\,7.36] 8.76% [6.50, 11.11][6.50,\,11.11] Table 9: Order-preserving rescaling stress test at the selected readout layers. The probability rows report how the probability-derived edit field reacts to the rescaling, and the winner is the candidate-local one-step argmax after the edit. The equalization row reports the closed share of the probability-to-rank gap. Intervals are image-bootstrap 95%, retrospective for the rescaling rows and paired for the equalization row. † marks an exact zero. The shaded row is the rank signal used by ReWEIGH. LLaVA-1.5-7B Qwen2.5-VL-7B Support Readout policy CHAIRS↓_S CHAIRI↓_I Recall↑ F1↑ CHAIRS↓_S CHAIRI↓_I Recall↑ F1↑ Matched Ordinal readout (DMRR) 45.8 12.78 75.2 80.77 26.0 8.64 58.0 70.99 Standardized (z-score) 52.8 14.87 76.9 80.83 24.4 6.45 55.6 69.77 Native Base decoding 52.6 16.07 77.3 80.48 28.8 8.26 56.6 69.97 Ordinal readout (DMRR) 45.8 12.77 75.2 80.77 25.8 8.59 57.8 70.82 Standardized (z-score) 52.4 14.90 76.9 80.81 24.0 6.38 55.0 69.29 Mean probability 49.2 13.30 77.4 81.81 25.2 8.94 57.6 70.58 Standardized probability 49.4 13.80 77.0 81.34 26.6 7.17 57.3 70.86 Table 10: End-to-end control between the ordinal and standardized readouts. Matched rows evaluate both policies on identical candidate support. Native rows let each policy form its own candidate sets, which is why the two blocks differ slightly. The native probability rows are not the matched-budget rank–probability comparison of Table 13. Shaded rows are the ordinal readout used by ReWEIGH. Arrows indicate the preferred direction. Table 10 reports the standardized-readout control referenced in Section 3.1. The control replaces the ordinal readout with a per-position standardized (z-scored) readout of the same lens under matched calibration and suppression. The comparison runs in a self-contained harness in which each backbone regenerates its own base, with Qwen2.5-VL-7B at repetition penalty 1.0, so its values are not numerically comparable with the main results. On LLaVA-1.5-7B, the ordinal readout lowers CHAIRS from 52.852.8 to 45.845.8 under matched support, while recall decreases from 76.976.9 to 75.275.2. On Qwen2.5-VL-7B, the standardized readout lowers CHAIRS from 26.026.0 to 24.424.4, while the ordinal readout raises recall from 55.655.6 to 58.058.0. These mixed point estimates do not establish cross-backbone superiority, but they show that per-position standardization does not reproduce the ordinal intervention’s behavior. B.3 Token-Conditioned Reference Analysis After replacing probability magnitude with rank, the remaining evidence still depends on token identity. In the image-balanced LLaVA decomposition reported in the main paper, 66.36%66.36\% of DMRR variability is associated with differences between tokens and 33.64%33.64\% with differences between images for the same token. The decomposition is descriptive. Its role is to test whether a single global reference is well matched to the evidence being interpreted. Each observation is the image-level value DMRRI(v)DMRR_I(v) attached to one candidate occurrence of token v. Repeated occurrences within an image share one value because we compute DMRRI(v)DMRR_I(v) once per image. The decomposition splits the total sum of squares of these observations into a between-token component, the variability of per-token means, and a between-image, within-token remainder, and reports each component’s share of the total. The occurrence-weighted variant counts every candidate occurrence once, whereas the image-balanced variant counts every token–image pair once, subject to the rare-token filter stated in Figure 2. We evaluate this question out of sample using 500 calibration images and 4,969 disjoint images. For each token observed in the held-out set, we form the median of its held-out DMRR observations and score a stored reference b by the normalized gap between the two in b0b_0 units. The token-unweighted aggregate takes the median of this gap with every token counted once, and the candidate-mass-weighted aggregate weights each token by its held-out candidate mass. The correct token-specific median reduces error relative to a global reference by more than 90%90\% in both analyzed backbones under token-unweighted and candidate-mass-weighted aggregation. Randomly shuffling the mapping between tokens and reference values preserves the marginal reference distribution but performs worse than the correct mapping in all 1,000 permutations, which corresponds to the smallest attainable one-sided permutation p-value under this null (p=1/1001p=1/1001). This control separates the value distribution of the calibration table from the token-to-reference correspondence. (a) Weighting sensitivity of the decomposition (LLaVA-1.5-7B) Analysis unit Between tokens Between images, within token Token–image balanced, min. 1 image/token 66.40% 33.60% Token–image balanced, min. 2 images/token 66.33% 33.67% Token–image balanced, min. 5 images/token (primary) 66.36% 33.64% Token–image balanced, min. 10 images/token 66.59% 33.41% Occurrence weighted 74.62% 25.38% (b) Token-identity share (ICC) on 5,000 development images Backbone ICC (simple) ICC (ANOVA) LLaVA-1.5-7B 0.470 0.655 InstructBLIP-7B 0.440 0.455 LLaVA-NeXT-7B 0.441 0.456 Qwen2.5-VL-7B 0.525 0.757 Table 11: Token-identity component of DMRR variability. Panel (a) varies the rare-token filter and the weighting unit of the decomposition in Figure 2. Panel (b) reports the intraclass correlation of token identity on image-unique observations at the selected readout layers. Out-of-sample error (b0b_0 units) Backbone Aggregation Global Shuffled Token-specific Reduction LLaVA-1.5-7B token-unweighted median 0.849 1.010 0.067 92.1% candidate-mass-weighted mean 4.211 5.195 0.341 91.9% Qwen2.5-VL-7B token-unweighted median 0.831 0.510 0.077 90.7% candidate-mass-weighted mean 7.506 8.501 0.612 91.9% Table 12: Out-of-sample calibration error of global, shuffled, and token-specific references. The shuffled column is the median of 1,000 shuffles, and Reduction is the error decrease of the correct reference relative to the global reference. The shaded column is the reference used by ReWEIGH. Two tables document the token-reference analysis. Table 11 varies the rare-token filter and the weighting unit of the decomposition, and it reports the token-identity share for all four evaluated backbones on 5,000 development images. Table 12 lists the out-of-sample calibration errors behind Figure 3, including the shuffled-reference null for both analyzed backbones. B.4 Detection Quality Does Not Predict Mitigation This section tests whether standalone detection quality identifies the signal that best drives an autoregressive intervention. We choose the ordinal readout in Section 3.1 for comparability across visual positions and for calibratability, not for detection quality. We compare rank- and probability-derived signals at the emitted-object detection level, in a raw score-to-edit negative control, and in the complete calibrated intervention. Signal CHAIRS↓_S CHAIRI↓_I Removed↑ Introduced↓ Net↑ Rank 44.8 12.67 329 198 131 Probability 50.0 14.03 283 227 56 Table 13: Matched-budget end-to-end comparison of the rank and probability signals on LLaVA-1.5-7B. Removed and Introduced count hallucinated-object occurrences relative to the same base captions, and Net is their difference. The shaded row is the rank signal used by ReWEIGH. Arrows indicate the preferred direction. At the detection level, the emitted-object hallucination AUROC reaches 0.6790.679 with the rank readout and 0.7180.718 with the probability readout on LLaVA-1.5-7B, and 0.6190.619 and 0.6340.634 on Qwen2.5-VL-7B. This metric orders grounded and hallucinated object tokens that the base trajectory emitted. It does not specify the full candidate-vector edit or the autoregressive trajectory after an intervention. The raw score-to-edit control asks whether the uncalibrated signals already differ as local intervention targets. At every step, each arm suppresses exactly one candidate by the same amount, without b(v)b(v), registration, normalization, or clipping. The contrast measures local switch selectivity within a fixed four-token precursor window. Raw rank minus probability is −0.49-0.49 p (95%95\% image-bootstrap interval [−1.19,+0.22][-1.19,\,+0.22]) on LLaVA-1.5-7B and −0.93-0.93 p ([−1.72,−0.16][-1.72,\,-0.16]) on Qwen2.5-VL-7B, so the bare rank signal provides no local advantage. Rank alone therefore does not explain the final method’s behavior. Table 13 gives the corresponding end-to-end comparison on LLaVA-1.5-7B under a matched mean edit budget. The calibrated rank intervention lowers CHAIRS from 50.050.0 to 44.844.8 relative to the calibrated probability intervention. It also removes more hallucinated-object occurrences from the shared base captions and introduces fewer new ones. Figure 6 illustrates the same distinction on one image. The probability intervention generates an unsupported fork, whereas the rank intervention avoids it while preserving the grounded pizza and dining table. The example is illustrative. The aggregate comparison in Table 13, rather than this selected image, supports the conclusion. LLaVA-1.5-7B Base The image features a delicious pepperoni pizza with a generous amount of pepperoni toppings, placed on a black tray. The pizza is cut into slices, making it easy to serve and enjoy. The pizza is served on a dining table, and a fork is placed nearby, ready for use. The table also has a bowl, possibly containing additional toppings or condiments. The scene is set in a cozy dining area, inviting people to gather around and enjoy the pizza together. Matched probability The image features a delicious pepperoni pizza sitting on top of a black pan, ready to be served. The pizza is cut into slices and has a generous amount of pepperoni toppings. The pan is placed on a dining table, and a fork is resting nearby, indicating that it is ready to be enjoyed. The pizza appears to be freshly baked and is served on a pizza pan, making it an appetizing sight for anyone who loves pepperoni pizza. ++ReWEIGH (rank) The image features a delicious pepperoni pizza sitting in a black pan on a dining table. The pizza is cut into slices, making it easy to serve and enjoy. The pan is placed on a dining table, which occupies most of the background of the image. The pizza appears to be freshly baked and ready to be eaten. Figure 6: Matched-budget rank and probability interventions on one CHAIR image. The visible utensil is a pizza server rather than a fork. Red marks scorer-flagged object mentions, and blue marks scorer-matched objects. These results do not isolate a single causal mediator or show that calibration alone creates the gap. They establish the conclusion relevant to our design. The ordering of the two signals on standalone detection does not match their ordering on end-to-end mitigation, where the token-calibrated rank intervention performs better under the matched budget. Detection quality alone therefore does not identify the signal of an effective autoregressive intervention. B.5 From Detection to Allocation Table 14 examines where the calibrated rank and probability edit fields place their edits on 4,969 held-out images, complementing the raw score-to-edit control of Appendix B.4. The unit of analysis is the four-step precursor window before each CHAIR object mention. A window counts as switched when the applied edit changes the one-step argmax at any of its steps, and switch selectivity is the switch rate before hallucinated mentions minus the switch rate before grounded mentions. A deterministic tokenizer-piece taxonomy, rather than contextual part-of-speech tagging, classifies precursor tokens into structural words, lexical words, punctuation and boundary pieces, and special fragments. An allocation-exclusive event is a window that switches under exactly one field’s candidate assignment, read at that field’s first switched step. The Effect rows exchange only the candidate assignment between the two calibrated fields while the edit-magnitude multiset stays fixed, using either field as the magnitude donor. On LLaVA-1.5-7B, the rank-derived assignment yields higher hallucination-linked precursor switch selectivity under both donors, while on Qwen2.5-VL-7B the contrasts are directionally consistent but their intervals include zero. The Role rows decompose the rank-donor effect by precursor token role, and the advantage concentrates on structural tokens. The Share rows report the fraction of allocation-exclusive first edit events that fall on structural tokens, which is likewise larger under the rank field. These are descriptive allocation statistics. They locate the difference between the two calibrated fields without establishing that structural-token edits are individually necessary. LLaVA-1.5-7B Qwen2.5-VL-7B Group Quantity Value 95% CI Value 95% CI Effect Rank-magnitude donor +2.32+2.32 p [+1.09,+3.53][+1.09,\,+3.53] +0.55+0.55 p [−1.00,+2.12][-1.00,\,+2.12] Probability-magnitude donor +2.06+2.06 p [+0.95,+3.14][+0.95,\,+3.14] +1.37+1.37 p [−0.15,+2.91][-0.15,\,+2.91] Role Structural tokens +2.36+2.36 p [+1.28,+3.48][+1.28,\,+3.48] +1.03+1.03 p [−0.36,+2.46][-0.36,\,+2.46] Lexical content tokens +0.33+0.33 p [−0.39,+1.04][-0.39,\,+1.04] +0.51+0.51 p [−0.65,+1.72][-0.65,\,+1.72] Punctuation tokens −0.74-0.74 p [−1.56,+0.09][-1.56,\,+0.09] +0.72+0.72 p [−1.19,+2.62][-1.19,\,+2.62] Share Rank-allocation-only events 62.6% — 55.1% — Probability-allocation-only events 50.0% — 40.6% — Table 14: Candidate-allocation analysis of the calibrated rank and probability edit fields. Effect and Role values are rank-minus-probability selectivity differences in percentage points with image-bootstrap 95% intervals. Share values are fractions of allocation-exclusive events that fall on structural tokens. Role and Share use the rank-magnitude donor. The definitions are in the accompanying text. B.6 Output Confidence and the Evidence Deficit Table 15 asks whether output confidence alone identifies the hallucinated mentions that the evidence signal targets. Among object-mention onsets in the top quartile of output confidence, 13.9%13.9\% are hallucinated on LLaVA-1.5-7B and 7.3%7.3\% on Qwen2.5-VL-7B, and the hallucinated onsets carry a positive median evidence deficit while grounded onsets in the same quartile carry none. The separation is clear on LLaVA-1.5-7B. On Qwen2.5-VL-7B the direction agrees but the interval reaches zero. High output confidence therefore does not subsume the calibrated evidence signal. Quantity LLaVA-1.5-7B Qwen2.5-VL-7B Hallucinated onsets in the top-confidence quartile 13.9% [10.3, 17.7][10.3,\,17.7] 7.3% [4.4, 10.7][4.4,\,10.7] (count) 52 / 373 20 / 274 Median evidence deficit at those onsets 1.18 0.10 Median evidence deficit, grounded onsets in the same quartile 0.00 0.00 Median difference [95%[95\% CI]] 1.18 [0.47, 5.56][0.47,\,5.56] 0.10 [0.00, 7.04][0.00,\,7.04] Table 15: Object-mention onsets in the top quartile of output confidence. Brackets are image-bootstrap 95% intervals, and the evidence deficit is max(b(v)−DMRRI(v),0)/b0 (b(v)-DMRR_I(v),0)/b_0 before clipping. Appendix C Method and Implementation Details C.1 Architecture-Specific Visual Positions The set P always refers to positions that carry visual information inside the language-model sequence. For LLaVA-1.5 and LLaVA-NeXT, these are the projected vision-encoder patch embeddings inserted in place of the image placeholder (18; 16; 17). For InstructBLIP, P contains the projected Q-Former query embeddings supplied to the Vicuna language model (5). For Qwen2.5-VL, P contains the merged visual embeddings inserted into the multimodal sequence (1). We exclude text positions and image-boundary control tokens. At the selected language-model layer ℓ , we collect j(ℓ)h_j^( ) for every j∈Pj∈ P. The readout applies the final output normalization and the language model’s vocabulary head, exactly as in Eq. (1). We do not train a translator or probe for intermediate layers. We define all ranks over the native vocabulary of the corresponding language backbone. Algorithm 1 Calibration and inference with ReWEIGH. 1: Calibration stage 2: Unlabeled calibration set calD_cal, frozen model M, readout layer ℓ 3: Frozen token table T 4: Initialize an empty evidence multiset DvD_v for each token v 5: for each image–prompt pair (I,x)∈cal(I,x) _cal do 6: Compute DMRRI(v)DMRR_I(v) for all tokens using Eq. (2) 7: Decode with the frozen model M and collect candidate sets tC_t 8: Add DMRRI(v)DMRR_I(v) to DvD_v whenever v∈tv _t 9: end for 10: Compute the pooled scale b0b_0 and token references b(v)b(v) by Eq. (4) 11: for each token v with Dv≠∅D_v≠ do 12: Estimate the reference interval [blo(v),bhi(v)][b_lo(v),b_hi(v)] 13: Compute e(b,x)e(b,x) and Δe(v) e(v) using Eqs. (5) and (6) 14: if the interval exists and Δe(v)<0.5 e(v)<0.5 then 15: Register (v,b(v))(v,b(v)) in T 16: end if 17: end for 18: Store b0b_0 in T 19: 20: Inference stage 21: Image I, prompt x, frozen model M, readout layer ℓ , penalty β, token table T 22: Generated sequence y 23: During prefill, compute DMRRI(v)DMRR_I(v) once 24: Cache sI(v)s_I(v) for every registered token using Eq. (7) 25: while the response is not complete do 26: Select candidates tC_t from the pre-edit distribution 27: Apply zt′(v)=zt(v)−βsI(v)z _t(v)=z_t(v)-β s_I(v) to v∈t∩ℛv _t 28: Decode the next token from the edited logits t′z _t 29: end while 30: return y C.2 Candidate Construction and Reference Storage We construct the candidate set from the unedited next-token distribution by sorting tokens by probability and taking tC_t as the shortest prefix whose cumulative mass reaches 0.90.9. We expand the prefix to two tokens when the threshold would produce a singleton and truncate it at 50 tokens when the distribution is flat. The same rule applies during calibration and inference. Whenever v∈tv _t during calibration, the current image-level value DMRRI(v)DMRR_I(v) is appended to DvD_v. The stored table contains b(v)b(v) only for registered tokens, together with the shared normalization unit b0b_0 and the registration mask. Inference needs neither the calibration images nor their hidden states. We never substitute the pooled value b0b_0 for a missing token reference. C.3 Reference-Stability Registration Let n=|Dv|n=|D_v| and write the calibration observations for token v as Dv=x1,…,xnD_v=\x_1,…,x_n\, with order statistics x(1)≤⋯≤x(n)x_(1)≤·s≤ x_(n). We select the largest integer k satisfying the nominal order-statistic rule 1−2Pr[Y≤k−1]≥0.95,Y∼Bin(n,12),1-2 \! [Y≤ k-1 ]≥ 0.95, Y (n, 12), (10) and define the lower and upper endpoints as [blo(v),bhi(v)]=[x(k),x(n+1−k)]. [b_lo(v),b_hi(v) ]= [x_(k),x_(n+1-k) ]. (11) The resulting endpoints define a nominal 95% order-statistic reference range. Calibration observations can repeat within images and decoding trajectories, so we do not interpret this range as a confidence interval with independent sampling or guaranteed 95% coverage. We use it only to test whether moving the stored reference between the two order-statistic endpoints materially changes the downstream edit. If no valid k exists, the token is not registered. The implied minimum sample size is six, so we do not introduce a separate minimum-count parameter. We compute the binomial tail exactly for n≤200n≤ 200 and use a continuity-corrected normal approximation for larger collections. We evaluate the range endpoints in the normalized edit space. For a candidate reference b and an observed evidence value x, define e(b,x)=clip(b−xb0,0,1).e(b,x)=clip\! ( b-xb_0,0,1 ). (12) We measure the edit variation induced by the reference range as Δe(v)=1|Dv|∑x∈Dv|e(bhi(v),x)−e(blo(v),x)|. e(v)= 1|D_v| _x∈ D_v |e\! (b_hi(v),x )-e\! (b_lo(v),x ) |. (13) We register a token only when Δe(v)<0.5 e(v)<0.5. This engineering criterion measures reference stability in the quantity that reaches the decoder rather than attaching a coverage claim to the raw median. Table 24 contrasts this rule with minimum-count registration at the frozen operating point (Appendix D.3). C.4 Complete Procedure Algorithm 1 summarizes the complete calibration and inference stages. Here calD_cal is the unlabeled calibration image–prompt set, M is the frozen LVLM, and T is the frozen token table. Both stages construct candidate sets with the rule of Appendix C.2, and the reference interval [blo(v),bhi(v)][b_lo(v),b_hi(v)] follows the order-statistic construction of Appendix C.3. At inference, the algorithm computes DMRRIDMRR_I once during prefill, caches the suppression strength sI(v)s_I(v) for registered tokens, and edits only candidates with registered references at each decoding step. C.5 Computational Cost Group Quantity Greedy ReWEIGH Overhead Latency Cached evidence (mean ratio) 1.000000×1.000000× 1.013323×1.013323× +1.33%+1.33\% Online DMRR computation (mean ratio) 1.000000×1.000000× 1.023974×1.023974× +2.40%+2.40\% Memory Peak allocated memory (MiB) 14,353 14,397 +0.31%+0.31\% Footprint Disk, 500 images (MiB) — 183.11 — Per image (KiB) — 375.0 — Table 16: Inference-cost accounting. Latency ratios average measurements from three backbones, with 100 prompts per backbone. Memory and cache-footprint measurements use LLaVA-1.5-7B. Overhead is relative to greedy decoding, and the shaded column is ReWEIGH. For |P||P| visual positions and a vocabulary of size |||V|, exact dense ranking requires sorting the vocabulary readout at each position during prefill. A direct implementation costs O(|P|||log||)O(|P||V| |V|) for this one-time operation. The cached evidence requires O(||)O(|V|) memory, and the registered reference table requires O(|ℛ|)O(|R|) memory. Once the candidate set has been formed, the additional per-step editing work is O(|t|)O(|C_t|). The method introduces no additional backbone forward pass and does not repeat the visual-position projection as the response grows. Table 16 quantifies these costs. The latency rows average the backbone-specific ratios measured over 100 prompts for each of the three backbones in Table 1. With the evidence already cached, the mean latency ratio is ×1.013323× 1.013323, corresponding to 1.33%1.33\% overhead over greedy decoding. When DMRR is computed online during prefill, the mean end-to-end latency ratio is ×1.023974× 1.023974, corresponding to 2.40%2.40\% overhead. The Footprint rows report the stored evidence cache, which occupies 375 KiB per image across the three cached readout layers. Deployment reads only the selected layer’s slice of this cache. Appendix D Additional Quantitative Results D.1 Full CHAIR and AMBER Results Tables 17 and 18 provide the complete per-benchmark results, including the metrics omitted from the main-paper summary table and the LLaVA-NeXT-7B backbone. Backbone Metric Base VCD OPERA DoLa PAI ReVisiT ReWEIGH LLaVA-1.5-7B CHAIRs↓ 52.60 59.00 51.20 53.00 42.40 51.40 44.80 CHAIRi↓ 15.61 17.97 14.19 15.87 13.51 15.44 12.67 Recall↑ 77.25 75.26 76.63 74.95 67.06 76.94 75.26 F1↑ 80.66 78.50 80.96 79.28 75.55 80.57 80.85 Qwen2.5-VL-7B CHAIRs↓ 31.60 33.80 20.20 23.60 26.20 24.00 25.40 CHAIRi↓ 9.58 9.65 7.01 16.76 9.61 6.75 7.54 Recall↑ 58.17 57.49 48.54 47.30 53.20 52.27 58.73 F1↑ 70.80 70.27 63.79 60.32 66.98 66.99 71.83 InstructBLIP-7B CHAIRs↓ 49.60 56.20 50.80 67.00 61.00 43.00 46.00 CHAIRi↓ 14.03 17.67 14.77 20.26 16.70 15.02 12.50 Recall↑ 73.09 71.41 73.65 74.15 72.96 67.06 73.34 F1↑ 79.01 76.48 79.02 76.84 77.79 74.97 79.80 LLaVA-NeXT-7B CHAIRs↓ 34.00 42.40 31.40 35.80 40.60 33.20 32.60 CHAIRi↓ 8.67 10.87 7.98 9.54 10.62 8.75 7.78 Recall↑ 64.82 65.94 63.46 67.25 66.94 65.94 63.70 F1↑ 75.83 75.80 75.11 77.14 76.55 76.56 75.36 Table 17: Full CHAIR results. Bold and underlined values are the best and second-best result within each backbone block, and the shaded column is ReWEIGH. Arrows indicate the preferred direction. Gen Disc Score Method CHAIR↓ Cover↑ Hal.↓ Cog.↓ Acc.↑ Prec.↑ Rec.↑ F1↑ Unp.%↓ AMBER↑ LLaVA-1.5-7B 3.69 50.47 19.02 3.82 71.96 92.65 62.73 74.81 0.08 85.56 VCD 6.25 51.81 32.07 4.49 66.81 86.22 59.45 70.37 0.67 82.06 OPERA 3.70 48.79 17.03 2.70 74.94 92.89 67.36 78.09 0.01 87.20 DoLa 4.52 51.03 21.12 3.84 72.17 92.35 63.27 75.09 0.08 85.28 PAI 4.68 43.11 19.32 1.97 38.29 73.46 10.86 18.93 16.66 57.12 ReVisiT 4.00 55.15 24.90 2.72 73.09 85.75 71.25 77.80 1.75 86.90 ReWEIGH 2.98 50.21 16.43 3.13 71.96 92.65 62.73 74.81 0.08 85.91 Qwen2.5-VL-7B 4.52 55.80 20.32 1.02 77.85 84.41 91.68 87.89 9.06 91.68 VCD 5.43 63.30 26.99 1.77 81.77 84.18 89.29 86.66 14.78 90.61 OPERA 3.51 63.33 20.92 1.42 83.70 85.01 91.57 88.17 7.94 92.33 DoLa 4.47 51.71 18.92 1.56 82.55 85.17 89.21 87.15 22.00 91.33 PAI 4.25 49.47 18.03 1.10 81.34 84.32 88.29 86.26 6.89 91.00 ReVisiT 4.27 62.30 24.90 1.42 83.28 85.62 89.88 87.70 17.07 91.71 ReWEIGH 3.80 54.65 18.82 0.93 78.19 84.28 92.01 87.97 8.53 92.08 InstructBLIP-7B 4.64 53.74 24.40 3.80 75.91 83.49 79.43 81.40 0.08 88.38 VCD 7.25 53.24 36.16 4.59 70.23 81.84 70.83 75.94 1.90 84.34 OPERA 4.81 52.08 23.80 3.53 75.90 83.66 79.10 81.32 0.04 88.25 DoLa 7.14 55.26 35.86 7.09 75.76 84.10 78.23 81.06 0.07 86.96 PAI 4.63 52.03 22.81 4.04 75.30 86.29 74.62 80.03 0.06 87.70 ReVisiT 4.22 50.99 22.91 2.13 54.36 88.66 35.75 50.95 2.47 73.37 ReWEIGH 4.05 53.28 21.51 3.11 75.96 83.50 79.48 81.44 0.02 88.69 LLaVA-NeXT-7B 5.45 56.64 32.47 2.64 64.62 99.02 56.70 72.10 19.93 83.33 VCD 7.55 56.21 41.33 3.11 69.46 95.28 56.75 71.13 17.33 81.79 OPERA 5.45 54.94 31.27 2.48 74.93 97.91 63.55 77.07 15.39 85.81 DoLa 6.19 58.24 33.27 3.03 72.21 98.68 58.88 73.76 20.05 83.79 PAI 5.66 56.55 33.37 2.60 67.57 98.73 51.77 67.92 25.89 81.13 ReVisiT 4.81 59.53 31.47 2.58 63.03 94.61 46.92 62.73 56.25 78.96 ReWEIGH 4.62 56.03 26.49 2.17 65.07 98.97 57.03 72.36 19.22 83.87 Table 18: Full AMBER results. Gen and Disc denote the generative and discriminative evaluations, respectively. Bold and underlined values are the best and second-best result within each backbone block, and shaded rows are ReWEIGH. Arrows indicate the preferred direction. D.2 Ablation Definitions and Diagnostics Figure 7: Layer-specific sensitivity to the intervention strength β on the CHAIR validation set. Circles mark the selected operating point (ℓ=29 =29, β=1.1β=1.1). All ablations use LLaVA-1.5-7B with readout layer L=29L=29, β=1.1β=1.1, the same frozen reference-stability table, greedy decoding, and seed 42. We evaluate 500 CHAIR validation images and all 1,004 examples in the AMBER generative split. Each arm modifies exactly one component of the full method (Table 19). Arm Modification Global reference Replace the token-specific reference b(v)b(v) with the single pooled scale b0b_0. Shuffled references Randomly permute b(v)b(v) across token identities while preserving the multiset of reference values. Mismatched evidence Rotate the image-conditioned evidence DMRRI(v)DMRR_I(v) across examples with a fixed derangement, so no example retains its own evidence. No abstention Apply the fallback b0b_0 to candidates absent from the registered set ℛR instead of leaving them unchanged. Bidirectional update Remove the suppression-only rectifier (the lower clipping bound), permitting evidence to either decrease or increase a candidate logit. Unbounded update Remove the upper clipping bound on the intervention magnitude. Table 19: Definitions of the component ablations. Table 20 reports the CHAIR diagnostics omitted from the main paper. The mismatched-evidence arm fires more often than the full method (87.8% versus 80.4%) but performs worse, confirming that the identity of the visual evidence matters. The unbounded arm produces 765 bound-escape events and a maximum repetition count of 122. Its apparent CHAIRS gain is therefore inseparable from severe degeneration and coverage loss. Configuration CHAIRS↓_S CHAIRI↓_I Recall↑ F1↑ Length Fire (%) Escapes Full method 44.8 12.67 75.3 80.85 90.3 80.4 0 Global reference 58.4 17.45 78.2 80.31 90.3 67.0 0 Shuffled references 56.0 14.74 78.1 81.54 92.9 72.6 0 Mismatched evidence 50.2 14.45 74.6 79.73 90.2 87.8 0 No abstention 44.4 12.64 75.3 80.90 90.2 82.7 0 Bidirectional update 50.4 16.07 75.9 79.74 92.0 98.7 0 Unbounded update 39.2 29.71 67.9 69.09 106.3 79.3 765 Table 20: Detailed CHAIR ablation results. Fire is the fraction of candidate positions at which the intervention is active, and Escapes counts updates that would exceed the full method’s clipping bound. The shaded row is the full method. Table 21 expands this counterfactual into a full image-specificity study at the same frozen operating point. It replaces the cached evidence vector while holding every other component fixed. The fixed-derangement row is the mismatched-evidence arm above. A cyclic-shift swap and an image-blind control (DMRRI≡0DMRR_I≡ 0) behave the same way, each recovering less than half of the full method’s CHAIRS reduction and pushing F1 below base decoding. Evidence supplied CHAIRS↓_S CHAIRI↓_I Recall↑ F1↑ None (base decoding) 52.6 15.61 77.3 80.66 Own image (ReWEIGH) 44.8 12.67 75.3 80.85 Wrong image, fixed derangement 50.2 14.45 74.6 79.73 Wrong image, cyclic shift 50.4 15.01 75.0 79.66 Image-blind (DMRRI≡0DMRR_I≡ 0) 49.8 13.98 74.1 79.61 Table 21: Image-specificity study on LLaVA-1.5-7B. Each row replaces the cached evidence vector with the stated substitute. The fixed-derangement row is the mismatched-evidence arm of Table 3. Arrows indicate the preferred direction, and the shaded row is the unmodified method. On AMBER generation, the well-behaved arms show the same overall degradation pattern as on CHAIR (Table 22). Abstention is a conservative safeguard rather than a performance component. It withholds the edit for tokens whose reference estimate is unstable, so one unreliable reference does not distort otherwise sound generation. The arm that removes it touches only about 3.5% of candidate positions and shifts CHAIRS by 0.40.4 points and CHAIR F1 by 0.050.05 points. On the discriminative split, removing abstention raises F1 from 74.81 to 76.20 while moving the predicted yes/no counts from 7,822/6,383 to 7,420/6,769. The prediction shift favors “no”, the majority label, and lowers precision as recall rises. We therefore read this F1 movement as majority-class drift rather than improved visual grounding, and such marginal movement does not outweigh the safeguard. Gen Disc Score Configuration CHAIR↓ Cover↑ Hal.↓ Cog.↓ Acc.↑ Prec.↑ Rec.↑ F1↑ Unp.%↓ AMBER↑ Full method 2.98 50.21 16.43 3.13 71.96 92.65 62.73 74.81 0.08 85.91 Global reference 4.10 52.78 21.91 3.51 71.98 92.65 62.75 74.82 0.05 85.36 Shuffled references 4.11 51.63 21.51 4.14 71.98 92.65 62.75 74.82 0.05 85.36 Mismatched evidence 3.70 49.35 18.03 3.15 71.96 92.65 62.73 74.81 0.08 85.56 No abstention 2.97 50.37 16.14 3.01 72.82 91.17 65.46 76.20 0.19 86.62 Bidirectional update 3.72 51.22 19.02 2.94 71.95 92.65 62.72 74.80 0.08 85.54 Unbounded update 3.44 48.26 16.14 2.62 71.96 92.65 62.73 74.81 0.08 85.69 Table 22: Detailed AMBER ablation results. Gen and Disc denote the generative and discriminative evaluations, respectively. The shaded row is the full method. Arrows indicate the preferred direction. Table 23 lists the full layer × β grid behind Figure 7, covering layers 28–30 and β from 0 to 1.5. We selected the frozen operating point (ℓ=29 =29, β=1.1β=1.1) on the seed-42 COCO train2014 tuning subset before these validation runs. It is not the per-cell optimum on this split. ℓ=28 =28 ℓ=29 =29 ℓ=30 =30 β CS↓C_S CI↓C_I F1↑ CS↓C_S CI↓C_I F1↑ CS↓C_S CI↓C_I F1↑ 0.0 52.6 15.61 80.66 52.6 15.61 80.66 52.6 15.61 80.66 0.1 53.2 15.37 80.81 53.0 15.11 80.99 52.8 15.18 81.20 0.2 52.0 15.26 80.72 52.2 14.76 80.95 51.6 15.14 81.02 0.3 49.6 15.10 80.55 48.6 14.23 81.08 49.8 14.27 81.27 0.4 48.8 14.51 80.12 47.6 13.95 80.72 49.2 14.32 80.87 0.5 47.4 13.49 80.60 48.6 14.00 80.73 49.6 14.26 80.48 0.6 48.2 13.34 80.74 47.2 13.28 80.73 50.2 13.72 80.61 0.7 48.0 12.99 81.14 46.8 12.96 81.15 47.6 13.55 80.65 0.8 47.0 12.73 81.25 46.2 13.07 81.03 48.8 13.40 80.57 0.9 46.8 13.27 81.09 46.6 13.23 80.89 47.8 13.29 80.94 1.0 47.8 13.34 81.13 46.2 13.14 80.79 48.2 13.42 80.63 1.1 47.8 13.07 80.89 44.8 12.67 80.85 47.4 13.93 80.37 1.2 48.2 13.65 80.46 46.8 13.22 80.51 46.4 13.70 80.30 1.3 46.6 13.84 80.31 47.8 13.21 80.47 48.8 14.14 79.79 1.4 47.8 13.68 80.73 47.8 13.67 79.95 49.0 14.11 79.80 1.5 43.8 12.27 81.02 47.0 13.35 80.09 47.6 13.54 79.76 Table 23: Full layer × β grid on the CHAIR validation split (LLaVA-1.5-7B). CSC_S and CIC_I denote CHAIRS and CHAIRI, the β=0β=0 rows equal base decoding, and the shaded cell is the frozen operating point. D.3 Design-Space and Data Robustness Table 24 contrasts the reference-stability rule of Section 4.3 with minimum-count registration (n(v)≥5n(v)≥ 5 or n(v)≥10n(v)≥ 10) under an otherwise frozen configuration. Every rule recovers most of the CHAIR improvement over base decoding, and the three rules stay within 1.21.2 CHAIRS points of one another on every backbone. The reference-stability rule also avoids the AMBER F1 drop that the n(v)≥5n(v)≥ 5 rule shows on InstructBLIP-7B. A fixed occurrence cutoff, however, is an external constant. The counts it gates on grow with the calibration budget, and the evidence scale behind each reference varies across backbones with different visual-token counts, so the same threshold acts differently as either factor changes. We use the reference-stability rule because it gates each token on the normalized edit that reaches the decoder, which adapts to both factors without a separate minimum-count parameter (Appendix C.3). Backbone Registration rule |ℛ||R| CHAIRS↓_S CHAIRI↓_I Recall↑ F1↑ AMBER-disc F1↑ LLaVA-1.5-7B none (base) – 52.6 15.61 77.3 80.66 74.81 count n(v)≥5n(v)≥ 5 3,252 44.6 12.69 75.4 80.91 74.81 count n(v)≥10n(v)≥ 10 2,476 44.8 12.70 75.4 80.91 74.81 reference stability (Δe<0.5 e<0.5) 2,580 44.8 12.67 75.3 80.85 74.81 InstructBLIP-7B none (base) – 49.6 14.03 73.1 79.01 81.40 count n(v)≥5n(v)≥ 5 3,524 46.2 12.50 73.3 79.80 78.51 count n(v)≥10n(v)≥ 10 2,714 46.2 12.54 73.3 79.74 81.44 reference stability (Δe<0.5 e<0.5) 2,708 46.0 12.50 73.3 79.80 81.44 LLaVA-NeXT-7B none (base) – 34.0 8.67 64.8 75.83 72.10 count n(v)≥5n(v)≥ 5 4,791 32.2 7.66 64.3 75.78 72.36 count n(v)≥10n(v)≥ 10 3,572 32.2 7.72 64.1 75.68 72.36 reference stability (Δe<0.5 e<0.5) 3,734 32.6 7.78 63.7 75.36 72.36 Qwen2.5-VL-7B none (base) – 31.6 9.58 58.2 70.80 87.89 count n(v)≥5n(v)≥ 5 9,286 26.2 7.64 58.4 71.57 87.97 count n(v)≥10n(v)≥ 10 6,458 26.6 7.87 58.4 71.50 87.97 reference stability (Δe<0.5 e<0.5) 7,028 25.4 7.54 58.7 71.83 87.97 Table 24: Design-space contrast of registration rules. |ℛ||R| is the number of registered tokens. Arrows indicate the preferred direction, and the shaded rows are the reference-stability rule used by ReWEIGH. Table 25 lists the calibration-size sweep behind Figure 5, including recall. The nested subsets share the frozen operating point, and the 500-image row reuses the deployed table and evaluation output (Appendix A). Calibration images CHAIRS↓_S CHAIRI↓_I Recall↑ F1↑ None (base decoding) 52.6 15.61 77.3 80.66 10 48.6 14.29 74.64 79.80 25 48.0 14.04 75.14 80.19 50 48.2 13.99 76.01 80.70 100 44.8 13.29 75.76 80.87 200 45.2 12.82 75.14 80.71 500 44.8 12.67 75.26 80.85 700 44.4 12.62 75.82 81.19 1,000 43.8 12.35 76.07 81.45 Table 25: Calibration-size sweep on LLaVA-1.5-7B. Arrows indicate the preferred direction, and the shaded row is the deployed 500-image configuration. Table 26 refits the calibration table on different unlabeled corpora at the frozen operating point. GQA calibration (9) matches the MS COCO default on CHAIRS, and AMBER and pooled calibration remain well below base decoding. The method therefore does not require calibrating on the evaluation distribution, although corpus shift can move the operating point. Calibration corpus Images |ℛ||R| CHAIRS↓_S CHAIRI↓_I Recall↑ F1↑ None (base decoding) – – 52.6 15.61 77.3 80.66 MS COCO (default) 500 2,580 44.8 12.67 75.3 80.85 GQA 500 2,429 44.6 13.15 75.4 80.71 AMBER 1,004 2,982 48.0 12.97 75.6 80.93 Pooled (COCO+GQA+AMBER) 2,004 3,900 46.6 13.22 75.9 81.00 Table 26: Calibration-corpus transfer on LLaVA-1.5-7B. |ℛ||R| is the number of registered tokens. Arrows indicate the preferred direction, and the shaded row is the default configuration. D.4 Scaling across Model Sizes and Architectures We further evaluate ReWEIGH across 11 models spanning six architecture families and model sizes from 7B to 32B. Results use the 500 examples in CHAIR val and reference-stability registration with the fixed b0b_0 scale. CHAIRS, CHAIRI, recall, and F1 are reported on a 0–100 scale. The scaling study assigns operating points by family. A 13B or 32B model reuses the β of its 7B backbone in Table 5 and places its readout layer at the same depth from the final decoder layer. We tune the two Gemma-3 models separately using a restricted search. For each model, we fix the readout to the third-to-last decoder layer and compare only β∈0.5,1.0β∈\0.5,1.0\ on the seed-42 COCO train2014 tuning subset. We select the setting by jointly reviewing CHAIRS, CHAIRI, recall, and F1. This yields (ℓ,β)=(45,1.0)( ,β)=(45,1.0) for Gemma-3-12B and (ℓ,β)=(59,0.5)( ,β)=(59,0.5) for Gemma-3-27B. For InternVL2.5-8B and InternVL2.5-26B, we use the same operating-point search described in Appendix A. On the seed-42 COCO train2014 tuning subset, we pair the final three decoder layers with β∈0.5,0.7,0.9,1.1,1.3β∈\0.5,0.7,0.9,1.1,1.3\ and then evaluate one additional midpoint at all three layers. We jointly review CHAIRS, CHAIRI, recall, and F1 over the resulting 18 configurations. This procedure yields (ℓ,β)=(31,1.1)( ,β)=(31,1.1) for the 8B model and (ℓ,β)=(45,1.1)( ,β)=(45,1.1) for the 26B model. We freeze these settings before evaluating the models on the separate 500-image COCO val2014 subset reported in Table 27. As shown in Table 27, the selected ReWEIGH settings reduce both CHAIRS and CHAIRI for every model. This trend holds across all tested sizes and architectures. F1 improves for six of the 11 models. For the remaining models, F1 stays within 0.70.7 points of base decoding, and each decrease occurs where recall also falls. Model Method CHAIRS↓_S CHAIRI↓_I Recall↑ F1↑ InstructBLIP-7B Greedy 49.60 14.03 73.09 79.01 ReWEIGH 46.00 12.50 73.30 79.80 LLaVA-1.5-7B Greedy 52.60 15.61 77.25 80.66 ReWEIGH 44.80 12.67 75.30 80.85 LLaVA-NeXT-7B Greedy 34.00 8.67 64.82 75.83 ReWEIGH 32.60 7.78 63.70 75.36 Qwen2.5-VL-7B Greedy 31.60 9.58 58.17 70.80 ReWEIGH 25.40 7.54 58.70 71.83 InternVL2.5-8B Greedy 32.40 8.21 69.36 79.01 ReWEIGH 30.00 7.51 68.18 78.50 Gemma-3-12B Greedy 49.80 8.95 70.60 79.53 ReWEIGH 46.80 8.53 70.04 79.33 InstructBLIP-13B Greedy 58.00 17.23 70.79 76.31 ReWEIGH 54.60 15.95 70.04 76.41 LLaVA-1.5-13B Greedy 51.80 14.76 77.50 81.19 ReWEIGH 45.00 11.70 75.95 81.66 InternVL2.5-26B Greedy 31.20 8.16 69.98 79.43 ReWEIGH 28.60 7.19 69.48 79.47 Gemma-3-27B Greedy 53.00 9.51 72.03 80.21 ReWEIGH 50.60 9.17 70.79 79.57 Qwen2.5-VL-32B Greedy 59.40 10.93 75.33 81.62 ReWEIGH 54.60 10.15 74.58 81.51 Table 27: CHAIR results across model sizes and architecture families. Shaded cells mark ReWEIGH. Appendix E Qualitative Results and Error Analysis This appendix complements the quantitative results with recorded case studies. It unfolds the intervention arithmetic at a single decoding step, presents additional side-by-side examples from CHAIR and AMBER, and illustrates failure modes that appear when individual design components are removed. Throughout, red marks object mentions that the benchmark scorer counts as hallucinated, and blue marks objects that the scorer matches to the reference annotations. The rank–probability output comparison appears separately in Appendix B.4. E.1 Mechanism Case Study Table 28 and Figure 8 unfold the intervention arithmetic at a single recorded decoding step of LLaVA-1.5-7B. At this step the base model’s argmax is the object token cars, which is absent from the image. Its evidence readout falls far below the token-specific reference (DMRRI(v)≪b(v)DMRR_I(v) b(v)), the deficit saturates the normalization (sI(v)=1s_I(v)=1), and the token is displaced by the full bound β. The evidence-supported candidate traffic sits above its own reference, receives no edit, and becomes the post-edit argmax, so the caption continues with the traffic lights that the image does contain. The same table shows the two other regimes at work within a single candidate set: registered tokens whose evidence clears their reference are left untouched, and a partially supported candidate receives a proportional intermediate penalty. Figure 8(b) shows a complementary case for LLaVA-NeXT-7B in which the suppressed hallucinated object is directly replaced by a grounded one. All values come from recorded traces of the deployed configuration, and no quantity is recomputed post hoc. Evidence (×10−3× 10^-3) Candidate zt(v)z_t(v) DMRRI(v)DMRR_I(v) b(v)b(v) Deficit sI(v)s_I(v) βsI(v)β\,s_I(v) zt′(v)z _t(v) cars 17.000 0.758 8.321 7.563 1.000 1.100 15.900 people 16.359 2.225 6.128 3.903 1.000 1.100 15.259 traffic 16.344 40.513 6.984 0.000 0.000 0.000 16.344 p 15.891 7.390 23.583 16.192 1.000 1.100 14.791 pal 15.703 72.867 124.762 51.895 1.000 1.100 14.603 stop 15.063 10.986 5.882 0.000 0.000 0.000 15.063 ped 15.039 4.393 4.743 0.350 0.108 0.119 14.920 clock 14.953 0.760 7.693 6.933 1.000 1.100 13.853 Table 28: Per-candidate arithmetic of ReWEIGH at a single recorded decoding step. The step uses the deployed LLaVA-1.5-7B configuration of Table 5. The table lists the eight candidates with the highest pre-edit logits among the K=22K=22 candidates, including the pre-edit argmax (bold) and the emitted token (highlighted). The columns reproduce the arithmetic of Eqs. (5), (7), and (8) from the recorded trace. (a) LLaVA-1.5-7B (b) LLaVA-NeXT-7B Figure 8: Mechanism case studies read from recorded traces. Each panel plots decoder logits before (filled) and after (open) the edit at one recorded decoding step, and each arrow spans the bounded suppression βsI(v)β s_I(v). The annotation under each candidate compares its evidence DMRRI(v)DMRR_I(v) with its reference b(v)b(v) in units of 10−310^-3. Panel (a) shows LLaVA-1.5-7B at step 71 with four candidates from Table 28 that span full, proportional, and zero suppression. Panel (b) shows LLaVA-NeXT-7B (β=1.2β=1.2, layer-30 readout) at step 60. Red marks the hallucinated pre-edit argmax, and blue marks the grounded token that becomes the post-edit argmax. E.2 Additional CHAIR Examples Figure 9 provides additional examples from CHAIR. We compare the response from the base model with the response obtained using ReWEIGH under the same image and prompt. These examples cover different backbones and common hallucination patterns, including contextual scene completion, repeated object generation, and removal of peripheral errors while preserving recognized objects. (a) InstructBLIP-7B Base The image features a living room with a fireplace, where a red duffel bag is placed on the floor next to a chair. There are several other bags and suitcases scattered around the room, including a black bag and a brown bag. The room also contains a television, which is placed on a table in the center of the room. A potted plant can be seen on the left side of the room, and a book is placed on the right side of the room. The overall scene suggests that the room is being prepared for a trip or a move. ++ReWEIGH The image features a living room with a fireplace, where several bags and suitcases are placed on the floor next to the fireplace. There are three bags visible, with one of them being a red one, and the other two being black. The bags are placed close to each other, with one bag located closer to the fireplace and the other two bags further away. There is also a box placed on the floor near the fireplace, which is likely to contain additional items. The room appears to be well-organized, with the bags and box neatly arranged. (b) LLaVA-1.5-7B Base The image captures a lively scene of people enjoying water sports on a large body of water. There are two people in the water, one of them is parasailing, while the other is windsurfing. The parasailer is being pulled by a boat, and the windsurfer is skillfully navigating the water. In the sky above, a kite can be seen flying, adding to the excitement of the scene. The water is filled with various boats, including a couple of smaller boats near the shore and a larger boat further out in the water. The presence of multiple boats and people participating in water sports creates a vibrant and active atmosphere. ++ReWEIGH The image captures a beautiful scene of a person kiteboarding in the ocean, surrounded by a few other kiteboarders. The person in the center of the scene is skillfully riding a kiteboard, while others are also enjoying the water nearby. In the sky, there are two kites visible, one towards the left side and the other towards the right. The kites are flying high above the water, adding to the excitement of the kiteboarding activity. The overall atmosphere of the scene is lively and full of adventure. (c) InstructBLIP-7B Base The image features a black and white cat sitting on top of a black suitcase placed on a bed. The cat is lying on the suitcase, which is open and partially visible. The cat’s body is stretched out, and its head is resting on the top of the suitcase. The suitcase is placed on the bed, which is covered with a blanket. There are several other items on the bed, including a book, a cup, and a bottle. The cat seems to be enjoying its time on the suitcase, possibly taking a nap or simply relaxing. ++ReWEIGH The image features a black and white cat sitting on top of a black suitcase placed on a bed. The cat is lying on the suitcase, which is open and partially visible on the bed. The cat is positioned in the center of the suitcase, with its head and body stretched out on top of it. The suitcase is placed on a bed, which is covered with a blanket and has a pillow on the left side. The cat seems to be enjoying its resting spot on the suitcase, possibly waiting for its owner to return. Figure 9: Qualitative CHAIR examples of three common hallucination patterns. Panels (a), (b), and (c) show contextual scene completion, repeated object generation, and peripheral errors, respectively. Red marks CHAIR-hallucinated objects, and blue marks ground-truth objects. Across these examples, ReWEIGH selectively suppresses the flagged object mentions while retaining or introducing descriptions of matched objects. This behavior applies to both isolated object mentions and relations built around those objects. E.3 AMBER Examples Figure 10 shows a generative example from AMBER, whose per-image annotations provide both a ground-truth object list and a hallucination vocabulary. The pattern mirrors the CHAIR examples. The base model imports a scene-typical beach setting that the annotations flag, while ReWEIGH avoids the flagged term. The candidate example was flagged mechanically from the annotation vocabulary and then manually verified against the image before inclusion. (a) LLaVA-1.5-7B Query: Describe this image. Base The image captures a lively beach volleyball game in progress. Two men are playing against each other, both reaching for the same volleyball that is in the air. They are both wearing sunglasses, and their athletic attire suggests they are fully engaged in the game. The volleyball is positioned in the middle of the scene, with both players trying to make a successful hit. The net is visible in the background, and the beach setting adds to the excitement of the game. ++ReWEIGH The image captures a dynamic moment during a volleyball game. Two men are playing against each other, both reaching for the same volleyball that is in the air. They are both wearing black shirts and are positioned close to the net. The volleyball is located in the center of the scene, with both players trying to grab it. The net is visible in the background, and the two men are the main focus of the image as they compete for the ball. Figure 10: Generative example from AMBER. Red marks terms from the annotated hallucination vocabulary, and blue marks annotated ground-truth objects. E.4 Failure Modes of Ablated Components Figure 11 illustrates, on actual generations, the two characteristic failure modes behind the component ablation in Table 3. Removing the bound (Unbounded update) turns suppression into unbounded logit displacement. Once the evidence-poor continuation is displaced far enough, the decoder falls into a two-token loop and repeats the same objects until the length limit, with a maximum single-object repetition of 122. This collapse inflates caption length and drives the mention-level hallucination rate far above the base model. Replacing the token-specific reference b(v)b(v) with a single global reference (Global reference) misallocates suppression across candidates. On the example shown, it derails an otherwise clean continuation into five object classes that neither the base model nor the full method produces (book, bottle, dining table, potted plant, and vase). The full method keeps the base model’s two flagged objects (chair and remote) and adds none. Both examples are the top-ranked cases under a preregistered mechanical ordering rather than hand-picked selections. The ordering ranks degeneration cases by their maximum single-object repetition and misallocation cases by the number of newly introduced object classes, breaking ties by image ID. The corresponding aggregate diagnostics appear in the ablation diagnostics tables. (a) LLaVA-1.5-7B Unbounded update → repetition collapse −-Bounding (unbounded update) The image depicts a busy city scene with multiple buses, cars, a bicycle, motorcycle, truck, motorcycle, taxi, […] bus, car, bus, car, bus, car, bus, car, bus, car, bus, car, bus, car, bus, car, bus, car, bus, car, bus, car, bus, car, bus, car, bus, car, bus, car ++ReWEIGH (full) The image depicts a busy city street filled with various vehicles, including a pink bus, several cars, and a truck. The pink bus is driving down the street, surrounded by other vehicles. There are multiple people visible in the scene, some of them standing or walking near the vehicles, while others are inside the bus or other vehicles. The street is quite congested, with traffic in both directions, including cars and buses. The scene captures the bustling atmosphere of a city, with people going about their daily routines and commuting using different modes of transportation. (b) LLaVA-1.5-7B Global reference → misdirected suppression −-Token reference (global reference) The image shows a man in a black shirt standing in a living room, unboxing a large flat-screen TV. He is carefully removing the plastic covering from the TV, which is placed on a wooden table. The room is furnished with a couch and a chair, and there is a remote control on the couch. In the background, there is a dining table with a bottle on it. The living room also features a potted plant, a vase, and a book, adding to the cozy atmosphere of the space. ++ReWEIGH (full) The image shows a man standing in a living room, carefully wrapping a large flat-screen TV in plastic. The TV is placed on a wooden stand, and the man is using plastic to protect it from dust and potential damage during transportation. The living room is furnished with a couch and a chair, both located towards the left side of the room. There is also a remote control placed on the couch, likely for operating the TV or other electronic devices. The overall atmosphere suggests that the man is taking care to ensure the TV’s safe and secure transportation to its new location. Figure 11: Failure modes of ablated components on LLaVA-1.5-7B. Arm definitions are in Table 19. In (a), the arm caption is abridged, and the omitted middle, marked […], repeats only the two tokens “bus” and “car”. Red in (a) marks the repeated tokens in the shown tail. In (b), red marks CHAIR-hallucinated objects, and blue marks ground-truth objects. Appendix F Use of AI Assistants In preparing this manuscript, we used AI-based writing assistants in a limited capacity to suggest alternative phrasings, correct grammatical errors, and improve readability. The authors reviewed and approved all such suggestions. AI assistants were not used to formulate the research ideas, design the method or experiments, implement the system, or generate or verify citations, factual claims, or dataset statistics. Separately from writing assistance, the official protocols of MMHal-Bench and M-Vet specify an LLM-based automatic evaluator. Following these protocols, we score both benchmarks with the frozen gpt-4.1-mini-2025-04-14 judge under identical settings for every compared method (Appendix A.3). This judge is a component of the evaluation pipeline defined by the benchmarks and is distinct from the writing assistance disclosed above. All other reported quantities are computed by deterministic scorers that involve no generative model. All technical content, implementation details, analyses, and reported findings were produced and verified by the authors, who take full responsibility for the manuscript.