Paper deep dive
LookBack: Where and How to Score LVLM Responses via Visual Reference Usage
Beomsik Cho, Jinhyeong Kim, Dongseok Lee, Jaehyung Kim
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Vision-Language Models (LVLMs) integrate visual perception with language generation, enabling responses that span image understanding and complex reasoning. However, LVLMs do not just inherit the text-level hallucinations; they also hallucinate against the image, producing fluent responses ungrounded in what they see. This makes LVLM response scoring inherently harder, and our diagnostics show that existing confidence-based metrics adopted from LLMs are insufficient for LVLMs. Specifically, removing the input image barely changes confidence-based selection, suggesting that output-space confidence primarily captures textual plausibility rather than agreement with the image. To address this gap, we propose LookBack, a training-free LVLM response scoring method that augments token likelihood with visual lookback score, a lightweight measure of how strongly each response token refers to image tokens. Across four benchmarks and three models, LookBack consistently improves Best-of-$N$ selection over existing baselines with negligible additional overhead.
Tags
Links
- Source: https://arxiv.org/abs/2608.11847v1
- Canonical: https://arxiv.org/abs/2608.11847v1
Trouble viewing inline? Open PDF directly →
Full Text
60,771 characters extracted from source content.
Expand or collapse full text
LookBack: Where and How to Score LVLM Responses via Visual Reference Usage Beomsik Cho Jinhyeong KimDongseok LeeJaehyung KimYonsei Universitybscho333, mjmps0726, jaehyungk@yonsei.ac.kr Abstract Large Vision-Language Models (LVLMs) integrate visual perception with language generation, enabling responses that span image understanding and complex reasoning. However, LVLMs do not just inherit the text-level hallucinations; they also hallucinate against the image, producing fluent responses ungrounded in what they see. This makes LVLM response scoring inherently harder, and our diagnostics show that existing confidence-based metrics adopted from LLMs are insufficient for LVLMs. Specifically, removing the input image barely changes confidence-based selection, suggesting that output-space confidence primarily captures textual plausibility rather than agreement with the image. To address this gap, we propose LookBack, a training-free LVLM response scoring method that augments token likelihood with visual lookback score, a lightweight measure of how strongly each response token refers to image tokens. Across four benchmarks and three models, LookBack consistently improves Best-of-N selection over existing baselines with negligible additional overhead. 11 1 Code: https://github.com/bscho333/LookBack. **footnotetext: Equal contribution. 1 Introduction With the recent success of Large Language Models (LLMs) (32; 1; 31), Large Vision-Language Models (LVLMs) have integrated visual understanding with text generation, demonstrating strong capability from image perception to complex reasoning (7; 24; 42; 2; 43). Despite these advances, LVLMs inherit a fundamental challenge of autoregressive generation: they can produce responses that are fluent and plausible yet factually incorrect. In multimodal settings, this challenge is further complicated by visual hallucination, where a response asserts objects, attributes, or relations that are not supported by the image (29; 21; 20; 10). Consequently, reliably identifying which responses are faithful to the image becomes an important problem for LVLMs. A common setting for improving reliability is Best-of-N selection, originating from the LLM literature: multiple candidate responses are sampled, and the best one is selected (37; 11; 30). The core of Best-of-N is the selection criterion that scores each candidate, and a prominent way is to train reward models or verifiers to assess response quality (6; 22). This idea has recently been extended to LVLMs through multimodal reward models that evaluate the visual correctness of candidates (36; 3). While effective, these methods require additional models, task-specific supervision, or preference annotations, which limits their applicability. Figure 1: Overview of LookBack: combining output-space confidence with visual reference usage. Given an image and a question, an LVLM generates multiple candidate responses and our goal is to select the proper one. Output-space confidence can assign high scores to fluent but visually unsupported tokens, while visual reference usage highlights tokens that directly look back to the image but does not by itself ensure response correctness. LookBack (Ours) combines these two signals at the token level by calibrating token confidence with visual reference usage, and aggregates token scores at the response level with visual relevance weights. A natural auxiliary-free direction is the model’s own output-space confidence, i.e., how plausible a response is under its output distribution. In the LLM literature, such confidence signals have proven a useful selection criterion that requires no external verifier (38; 17). This makes the model’s own output distribution an appealing candidate for scoring LVLM responses, as it requires no auxiliary model. However, LVLM response scoring differs from its text-only counterpart in one crucial respect: response quality depends on an additional source of evidence, the image. Through a diagnostic analysis, we find that confidence-based selection remains nearly as effective even when the confidence is computed without conditioning on the image. This reveals a gap between image-conditioning and image-sensitivity: the confidence score reflects a response’s textual plausibility rather than its agreement with the image. Since visual hallucination arise from a mismatch between the response and the image, such an image-insensitive signal cannot reliably detect them. To close this gap, we incorporate an attention-based measure of visual lookback score that captures how strongly each generation step attends back to vision tokens. This score serves as a lightweight proxy for visual reference usage. Specifically, we propose LookBack, a training-free LVLM response scoring method that calibrates output-space confidence using visual lookback scores. At the token level, LookBack combines each token’s likelihood with its visual lookback score to form a lookback calibrated token score. At the response level, it aggregates these token scores under a visual relevance distribution, giving greater weight to tokens with stronger visual reference usage. This distinction is crucial because LVLM responses contain both visually diagnostic content words and generic fluency words: lookback calibrated token scoring captures whether each word is visually supported, while response-level aggregation determines how much visually relevant words contribute to the LookBack. As a result, LookBack favors responses whose high-confidence tokens are strongly tied to the visual input, without requiring external verifiers, additional training, or extra inference passes. We evaluate LookBack in Best-of-N selection across four benchmarks that require visual understanding and three representative LVLMs: LLaVA-1.5-7B (24), Qwen2.5-VL-7B (2), and InternVL3-8B (43). Across these settings, LookBack achieves the highest model-wise average performance for all three LVLMs and generally improves over both linguistic- and vision-side baselines. 2 Related Works Output-space response scoring in LLMs. Stochastic decoding in LLMs yields multiple candidate responses that may follow different inference paths (16; 28), making Best-of-N selection a natural strategy to improve the performance and reliability (11; 30). Existing methods typically score candidates from the generated output alone, using answer consistency (37), external reward models or verifiers (6; 33; 22; 35), or training-free model-intrinsic signals (38; 17; 18; 12). However, the output-space score is less reliable in LVLMs, since fluent and confident responses can still contain visual claims unsupported by the image. LookBack retains the lightweight, model-intrinsic nature of such scores while incorporating visual reference usage. Response scoring in LVLMs. Unlike LLM response scoring, LVLM response scoring must evaluate both linguistic plausibility and visual faithfulness simultaneously. Prior work has addressed this challenge using external multimodal evaluators or reward models, such as VLM-as-a-judge models, multimodal critics, and process reward models (19; 39; 36; 3). Other approaches estimate image–response alignment through cross-modal matching or calibrated visual constraints (13; 41), or analyze LVLM reliability through uncertainty, hallucination detection, and visual-grounding signals (20; 27). While these works emphasize the need for visual evidence, they often introduce external models, additional training, or extra scoring passes. In contrast, LookBack scores responses using token likelihood and visual lookback score which can be freely obtained during generation, providing a training-free and efficient alternative for LVLM response selection. 3 Motivation To identify a reliable scoring function for LVLM response, we begin with output-space confidence, as it has shown consistent effectiveness in LLM scoring across inference-time selection (17) and training-time reward (40). Our investigation asks whether this signal remains reliable when response quality depends not only on linguistic plausibility, but also on visual input. We instantiate this investigation in a Best-of-N (BoN) response selection setting, where multiple candidate responses are generated for the same input and then ranked by a scoring function. Our analysis covers both discriminative and generative LVLM responses on MS-COCO (23) dataset: VQAv2 (9) for visual question answering and CHAIR (29) for object hallucination evaluation. For each benchmark, we randomly sample 1,000 instances and generate N=25N=25 candidate responses with LLaVA-1.5-7B (24) and Qwen2.5-VL-7B (2). As a representative output-space confidence scorer, we adopt Self-Certainty (SC) (17), a state-of-the-art BoN scoring method based on the token-averaged KL divergence between the model’s predicted distribution and a uniform distribution. 3.1 Is Output-Space Confidence Sensitive to Visual Input? We first examine whether SC is affected by the visual input by comparing its score distributions under image-present and image-absent conditions. For each generated response, we compute SC under two conditions: with the input image (SC w/ image) and without the input image (SC w/o image). Figure 2 shows that across all model–benchmark combinations, the SC distributions under the two conditions are nearly identical in shape and mean, suggesting that SC scores are largely unaffected by the removal of the image at the population level. Figure 2: SC score distributions remain similar with and without the image. SC score distributions under image-present and image-removed conditions highly overlap, with closely aligned means across setups. To further assess this weak image sensitivity at the selection level, we measure the top-1 agreement ratio: the fraction of instances for which SC w/ image and SC w/o image select the same top-ranked candidate from N=25N=25, directly reflecting whether the input image influences Best-of-N selection under SC. The solid bars in Figure 3 report this agreement. If the image were a critical determinant of response quality, we would expect the two scorers to select different top-1 candidates. However, we observe agreement ratios of 0.36–0.64, far above the random baseline of 1/N=0.041/N=0.04, indicating that removing the image barely changes which response SC selects. These results reveal a gap between image-conditioning and image-sensitivity: although LVLMs are conditioned on the image during generation, their output-space confidence can remain largely unchanged even without the image. Thus, output-space confidence alone cannot reliably indicate whether a response is grounded in the image. Figure 3: Top-1 agreement with image-conditioned SC: image-removal consistency and visual-lookback complementarity. We report the fraction of instances in which two scorers select the same top-ranked response from N=25N=25 candidates. 3.2 Visual Lookback Complements Token-Level Confidence The weak image-sensitivity of output-space confidence raises a natural question: what model-internal signal can reflect whether each generation step consults the visual input? Unlike text-only LLM, LVLMs receive the image as explicit vision tokens in the input context. Although visual information may be propagated implicitly through image-conditioned text representations and previous output states, the original vision tokens remain available in the causal context throughout decoding. Namely, each response-token prediction can therefore directly look back to the visual input. We quantify this behavior by the fraction of attention directed from each output token to the vision tokens. We call this token-level quantity the visual lookback score AtA_t, which serves as a lightweight proxy for visual reference usage; its formal definition is given in Section 4.1. Figure 4: Token-level POS analysis of visual lookback score and Self-Certainty. We group generated words into a visual set and a textual set, and report the mean z-score of each signal within each group. Visual lookback scores are higher for the visual set, whereas Self-Certainty is higher for the textual set. To examine whether this signal indeed captures a token-level visual reference, we analyze how it varies across different types of generated words. If visual lookback score captures how much a generation step refers back to visual evidence, it should be higher for tokens that express visually referential content than for tokens that primarily serve grammatical or discourse functions. To this end, we conduct a token-level Part-of-Speech (POS) analysis on the same generated responses from Sec. 3.1. Generated words are tagged with spaCy and grouped into two sets following prior works (5; 8; 26): a visual set, containing words more likely to express visually referential content—nouns, proper nouns, adjectives, and numerals—and a textual set, containing words that primarily serve grammatical or discourse roles, such as auxiliaries, determiners, pronouns, and conjunctions. Over the same word groups, we also analyze SC to test whether output-space confidence favors visually referential words despite its weak image sensitivity at the response-selection level. For words split into multiple subtokens, we use the mean subtoken score as the word-level score, and independently z-score normalize each score dimension over all analyzed output tokens before word-level aggregation. As shown in Figure 4, across all model-benchmark combinations, words in the visual set exhibit above-average visual lookback score but below-average SC, whereas words in the textual set show the opposite pattern. This opposing pattern suggests that visual lookback score captures a token-level tendency complementary to SC: SC is higher for linguistically predictable words, whereas visual lookback score is higher for words that are more visually referential. A fine-grained per-POS breakdown in Figure 8 further shows that this trend is not driven by a single POS category, but broadly appears across content-oriented categories. This token-level complementarity also appears at the response level. The hatched bars in Figure 3 show that SC and visual lookback score rarely select the same top-ranked response, with the lowest agreement ratio reaching 0.01, below the random baseline of 0.040.04. This indicates that the two signals induce substantially different response rankings. Figure 1 illustrates this behavior qualitatively. SC assigns high scores to fluent, linguistically predictable tokens including function words and assertion markers, whereas visual lookback score highlights object names, attributes, actions, and spatial relations that require direct consultation of the image. This divergence is particularly pronounced when the model generates confident but visually unsupported assertions: confidence rewards linguistic fluency regardless of visual grounding, whereas visual lookback score reflects whether the prediction step directly attended to the visual input. Overall, these observations suggest that output-space confidence and visual lookback score capture complementary aspects of LVLM generation. Output-space confidence is useful for identifying linguistically likely responses, but it may remain high even when a response is driven primarily by language priors rather than visual evidence. Visual lookback score captures whether generation steps attend back to the explicit vision tokens, but high visual attention alone does not guarantee correctness. Therefore, a visually grounded response score should combine token-level confidence with token-level visual lookback score, while placing greater emphasis on response positions where visual evidence is most relevant. 4 Method Based on the observations in Section 3, our goal is to define a response score that converts output-space confidence into vision-aware confidence using only model-internal signals, without extra inference passes. Specifically, we design the score according to following objectives: O1. Preserve language confidence. The score should retain token-level model confidence, so that implausible or low-probability responses are not favored. O2. Incorporate visual reference usage. Tokens that are likely under the output distribution should be further assessed by how strongly they interact with the visual reference. O3. Aggregate according to visual relevance. Since not all output tokens are equally informative for visual grounding, visually relevant token positions should contribute more. Preliminary. Let pθp_θ denote an LVLM parameterized by θ. Given a text query x and vision tokens v encoded from an input image ℐI, the model generates a response =(y1,…,yT)y=(y_1,…,y_T) autoregressively: pθ(∣,)=∏t=1Tpθ(yt∣,,y<t),p_θ(y ,v)= _t=1^Tp_θ(y_t ,v,y_<t), (1) and we denote the per-token probability as pt=pθ(yt∣,,y<t)p_t=p_θ(y_t ,v,y_<t). Under stochastic decoding, we sample N candidate responses (i)i=1N\y^(i)\_i=1^N for the same input (,)(x,v), and select the best candidate under a training-free score S: ∗=argmaxiS((i)∣,).y^*= _i\,S(y^(i) ,v). (2) 4.1 Visual Lookback Score For each generated token yty_t, we estimate its visual lookback score: how strongly the prediction step attends back to the original vision tokens. Let CtC_t denote the full causal context at the prediction step for yty_t, comprising text query tokens, vision tokens, and previous output tokens. Let v⊂CtP_v⊂ C_t denote the positions corresponding to vision tokens v. Let at,k(ℓ,h)a_t,k^( ,h) be the attention weight from the query position predicting yty_t to context position k at layer ℓ and head h. We define the token-level visual lookback score AtA_t as: At=1LH∑ℓ=1L∑h=1H∑k∈vat,k(ℓ,h)∑k∈Ctat,k(ℓ,h),A_t= 1LH _ =1^L _h=1^H _k _va_t,k^( ,h) _k∈ C_ta_t,k^( ,h), (3) where L and H denote the number of layers and attention heads, respectively. Since CtC_t is the full causal context, the denominator equals one under softmax normalization; we retain the ratio form to make explicit that AtA_t measures the fraction of attention directed to vision tokens. As AtA_t is obtained from the attention weights produced during the generation forward pass, it does not require additional inference or external model. A larger AtA_t indicates greater visual reference usage, meaning that the prediction step allocates a larger fraction of its attention to the explicit vision tokens. 4.2 Lookback-Calibrated Token Score Since ptp_t and AtA_t capture complementary aspects of token generation, we define a token-level score that preserves output-space confidence while accounting for visual lookback score. We define the lookback-calibrated token score utu_t as ut=log(pt)+αlog(At),u_t= (p_t)+α (A_t), (4) equivalently written as ut=log(pt⋅Atα)u_t= (p_t· A_t^α). The hyperparameter α controls the strength of this lookback calibration. When α=0α=0, utu_t reduces to pure output-space confidence logpt p_t (O1). When α>0α>0, utu_t is larger for tokens that are both highly probable and strongly attend to vision tokens, and smaller for tokens whose confidence is weakly supported by the visual reference (O1, O2). 4.3 Weighting Tokens with Visual Relevance To obtain a response-level score, we aggregate token-level scores utt=1T\u_t\_t=1^T under a weight distribution q over output token positions: S(∣,):=t∼q[ut]=∑t=1Tq(t)ut.S(y ,v):=E_t q [u_t ]= _t=1^Tq(t)\,u_t. (5) Uniform averaging corresponds to q(t)=1Tq(t)= 1T, but it assumes every token is equally informative, which is often not the case. In visually grounded responses, tokens such as object names, attributes, counts, and relations are generally more diagnostic than function words or generic phrases. The aggregation distribution should therefore concentrate on positions that interact more strongly with the visual reference, motivating q derived from AtA_t. We define qλq_λ as a visual relevance distribution over output tokens, derived as the solution to an entropy-regularized relevance maximization: qλ=argmaxq∈ΔT[λt∼q[log(At)]+H(q)],q_λ= _q∈ _T [λ\,E_t q [ (A_t) ]+H(q) ], (6) where ΔT _T is the probability simplex over T output positions, and H(q)=−∑tq(t)log(q(t))H(q)=- _tq(t) (q(t) ) is the entropy of q. The first term encourages q to concentrate on positions with high visual lookback score, while the entropy term keeps the distribution smooth and prevents hard selection. The hyperparameter λ controls how sharply q concentrates on high-AtA_t positions. Proposition 1. The solution to the entropy-regularized relevance maximization is qλ(t)=Atλ∑j=1TAjλ.q_λ(t)= A_t^λ _j=1^TA_j^λ. (7) Proof. See Appendix A.2. ∎ When λ=0λ=0, qλq_λ is uniform; as λ increases, qλq_λ assigns more mass to positions with larger visual lookback score (O3). Final score. Instantiating S with qλq_λ gives the final response score LookBack: S(∣,) S(y ,v) =t∼qλ[log(pt)+αlog(At)] =E_t q_λ [ (p_t )+α (A_t ) ] (8) =∑t=1TAtλ(log(pt)+αlog(At))∑j=1TAjλ. = _t=1^T A_t^λ ( (p_t )+α (A_t ) ) _j=1^TA_j^λ. LookBack measures the expected lookback calibrated token score under a visual relevance distribution. Intuitively, it evaluates model confidence in a response while emphasizing tokens that interact more strongly with the explicit visual reference. 5 Experiments 5.1 Setups Benchmarks and metrics. We evaluate on four image-grounded benchmarks suited to Best-of-N response selection: VQAv2 (9), CHAIR (29), AMBER (34), and HallusionBench (10). We chose this suite so that it (1) requires visual evidence beyond language plausibility, (2) provides sufficient candidate diversity and selection headroom, and (3) covers complementary response formats, including short-answer QA, open-ended generation, and discriminative visual-grounding evaluation. For a consistent main comparison, we report a single higher-is-better primary metric per benchmark: accuracy for VQAv2, F1 for CHAIR and AMBER, and GPT-evaluated correctness for HallusionBench. See Appendix B.1 for details. Table 1: Main results. Best-of-N selection performance for N=5N=5 and N=25N=25 across four benchmarks and three LVLMs. Average denotes the average score over all benchmark–N combinations within each model. Higher scores (↑ ) indicate better performance. The best results in each setup are bolded, and the second-best are underlined. Model Method VQAv2 CHAIR HallusionBench AMBER Average N=5N=5 N=25N=25 N=5N=5 N=25N=25 N=5N=5 N=25N=25 N=5N=5 N=25N=25 LLaVA-1.5-7B Random 62.57 60.40 70.20 71.52 44.84 43.82 63.90 64.08 60.16 SC 65.93 65.70 72.76 73.53 47.21 46.19 64.10 63.45 62.36 USC 63.78 61.62 71.07 70.87 43.09 43.20 64.61 63.96 60.28 CLIPScore 64.07 62.27 71.89 71.93 43.15 43.65 66.08 65.25 61.03 VAUQ 66.03 66.53 72.48 73.15 45.01 45.35 64.28 64.22 62.13 Ours 66.63 67.60 74.03 74.43 47.38 48.22 65.33 65.74 63.67 Qwen2.5-VL-7B Random 63.40 62.23 74.18 73.22 57.19 57.36 74.19 73.84 66.95 SC 66.33 65.47 74.41 71.97 57.36 58.04 74.43 74.30 67.78 USC 66.88 66.73 74.63 73.96 56.80 56.91 74.33 74.64 68.11 CLIPScore 64.07 62.17 74.47 74.80 57.36 58.21 74.17 74.35 67.45 VAUQ 66.67 66.83 75.15 73.24 57.02 57.19 74.77 74.50 68.17 Ours 68.23 67.47 75.42 75.28 60.07 61.93 74.98 75.92 69.91 InternVL3-8B Random 62.17 61.33 77.88 77.78 56.35 57.19 79.82 79.45 69.00 SC 66.70 66.67 78.35 78.24 56.35 56.85 81.66 81.67 70.81 USC 70.33 69.39 78.34 78.28 58.15 58.38 80.04 79.72 71.58 CLIPScore 63.47 63.47 78.32 78.27 59.39 59.39 79.68 79.98 70.25 VAUQ 68.23 69.77 78.49 78.28 57.19 57.19 81.47 81.65 71.53 Ours 70.33 72.57 79.37 78.22 57.02 56.85 81.77 82.07 72.27 Models and baselines. For the experiments, we consider three representative LVLMs: LLaVA-1.5-7B (25), Qwen2.5-VL-7B (2), and InternVL3-8B (43). Also, to evaluate the effectiveness of LookBack, we consider two categories of baselines: language side and vision side. (1) Language side: Self-Certainty (SC) (17) measures model’s token distributional confidence leveraging KL divergence with uniform distribution. Universal Self-Consistency (USC) (4) select most consistent response by prompting all candidates responses to model. (2) Vision side: CLIPScore (13) leverages a pretrained vision-language encoder to measure the cosine similarity between the text embedding of a generated response and the corresponding image embedding. VAUQ (27) estimates model uncertainty by masking a fixed proportion of visual attention weights and computing the entropy of the resulting output distribution. To evaluate the gain of each method, we additionally consider the simplest baseline, random selection (Random). Implementation details. For all models, to ensure sufficient diversity among candidates, we generate N=25N=25 candidate responses per input using nucleus sampling (15) with temperature=1.2 and top-p=0.9. LookBack has two hyperparameters, α and λ, which control the strength of token-level visual lookback calibration and response-level visual relevance weighting, respectively. We set these hyperparameters on a per-model basis. Specifically, we use (α,λ)=(7.0,1.5)(α,λ)=(7.0,1.5) for LLaVA-1.5-7B, (0.5,1.25)(0.5,1.25) for Qwen2.5-VL-7B, and (0.25,1.25)(0.25,1.25) for InternVL3-8B. See Appendix B.2 for details. 5.2 Main Results Table 1 reports Best-of-N selection performance for N=5N=5 and N=25N=25 across four benchmarks and three LVLMs. Across all setups, LookBack improves the average score from 65.37% to 68.62%, achieving a 4.97% relative gain over random selection. While SC achieves competitive performance as an output-space baseline, LookBack improves over SC in most settings and achieves higher average scores. This suggests that visually grounded response selection benefits from measuring not only how confident a response is, but whether that confidence is supported by visual reference usage. USC is also competitive in InternVL3-8B, but its gains are less stable across models and candidate budgets. Since USC relies on the LVLM itself to select candidate responses, its effectiveness may depend on the model’s own selection ability, not just on the quality of the sampled candidates. Compared with VAUQ which estimates reliability from a perturbed visual condition, LookBack uses visual evidence in a more candidate-specific way; LookBack directly measures whether high-confidence tokens look back to the visual input.. This makes LookBack well suited to Best-of-N selection, where the score must distinguish candidates whose confident claims are visually supported from those that are merely plausible. Figure 5: Best-of-N scaling. We report Best-of-N selection performance as the candidate pool size increases from N=1N=1 to N=25N=25 on HallusionBench with LLaVA-1.5-7B (top) and Qwen2.5-VL-7B (bottom). 5.3 Scaling Results We further investigate how each scoring method benefits from varied candidate budget. Figure 5 shows Best-of-N accuracy as N increases from 1 to 25 on HallusionBench for LLaVA-1.5-7B and Qwen2.5-VL-7B. LookBack maintains a consistent advantage over all baselines across the full range of N. While SC improves gradually with N, LookBack maintains a clear margin as the candidate pool grows, especially on Qwen2.5-VL-7B. CLIPScore and VAUQ do not consistently benefit from larger candidate pools. This suggests that scaling Best-of-N requires a reliable grounding-aware score, rather than simply increasing the number of candidates. Overall, LookBack shows stable performance gains as N increases. Figure 6: Ablation study. We ablate token-level calibration strength α and the response-level visual weighting strength λ on AMBER with LLaVA-1.5-7B. Top: performance when varying α with fixed λ. Bottom: performance when varying λ with fixed α. Dashed lines show the corresponding variants without the other component. 5.4 Additional Analyses Ablation study. Figure 6 ablates the two hyperparameters of LookBack on AMBER with LLaVA-1.5-7B: α for token-level visual lookback calibration and λ for response-level visual relevance weighting. In the top panel, increasing α yields modest gains when λ=0λ=0 because aggregation remains uniform. With λ=1.5λ=1.5, however, the same increase leads to stronger performance, showing that token-level calibration is more effective when visually relevant tokens are weighted more heavily. In the bottom panel, increasing λ improves performance when α=0α=0, but with α=7α=7, performance peaks at a moderate λ and declines as the aggregation becomes too sharp. These results show that token-level lookback calibration and visual relevance-based aggregation are complementary. Figure 7: Scoring overhead. We compare the average scoring time per response across three LVLMs on CHAIR. Error bars indicate standard deviation. Scoring overhead. To evaluate the computational efficiency of LookBack, we measure post-generation scoring overhead in milliseconds per response on CHAIR across three LVLMs. As shown in Figure 7, LookBack adds only modest overhead and remains substantially cheaper than VAUQ and USC, which require perturbation-based uncertainty estimation or additional candidate-comparison passes. This shows that LookBack provides a favorable efficiency–grounding trade-off: unlike SC, it incorporates an explicit visual-grounding signal, yet unlike heavier visual scorers, it obtains this signal directly from the LVLM’s internal likelihood and attention statistics without external models or expensive post-hoc verification. See Appendix B.3 for measurement setup and hardware configuration. 6 Conclusion We presented LookBack, a training-free response scoring method for LVLMs that calibrates output-space confidence with visual lookback. Our motivation analysis shows that confidence-based scoring can remain largely insensitive to the image, even when computed under image-conditioned generation, and therefore tends to capture textual plausibility rather than visual grounding. To address this gap, LookBack combines token likelihood with a token-level visual lookback score and aggregates these calibrated token scores according to a visual relevance distribution. Across multiple LVLMs and visual understanding benchmarks, LookBack consistently improves Best-of-N selection over both linguistic and vision side baselines, while requiring no auxiliary verifier, training, or extra inference passes. Limitations and Future Work LookBack requires access to internal attention weights, which limits its applicability to black-box LVLMs. In addition, visual lookback is only a proxy for visual reference usage, not a guarantee of factual correctness: a model can strongly attend to image tokens while still generating an incorrect or unsupported response. The reliability of this proxy may also depend on model-specific attention behavior, as attention distributions and their calibration can vary across LVLM architectures. In addition, our current evaluation focuses on image-grounded Best-of-N selection with relatively concise responses. Whether LookBack generalizes to long-form multimodal reasoning, multi-image or video inputs, and more complex grounding scenarios remains an important direction for future work. Interestingly, our formulation suggests a broader perspective on source-aware scoring for generative models, beyond LVLM response selection. The same principle may apply whenever a response should be grounded in a designated part of the input context, such as retrieved documents in RAG, tool outputs, or instruction tokens for prompt following. In such settings, a reliable scorer should assess not only whether a response is likely under the model’s output distribution, but also whether high-confidence generation steps refer back to the intended source. We believe this perspective opens a path toward lightweight, model-internal scoring methods for source-grounded generation beyond the visual domain. Broader Impact and Ethical Implications LookBack is intended to improve the reliability of LVLM response selection by favoring candidates whose high-confidence tokens are more strongly tied to the visual input. This direction may be beneficial in applications where visually unsupported responses can mislead users, such as image-based assistance, educational tools, accessibility interfaces, and content analysis systems. Because LookBack requires neither additional training nor an auxiliary evaluator, it may also provide a lightweight mechanism for improving response selection in resource-constrained settings. At the same time, LookBack should not be interpreted as a guarantee of correctness or safety. Visual lookback is only a proxy for reference usage: a response can strongly attend to image tokens while still being incorrect, biased, or harmful. These risks are especially consequential in high-stakes applications, including medical, legal, surveillance, and accessibility-critical settings. We therefore recommend using LookBack only as a response-selection aid, together with task-specific validation and human oversight. References Achiam et al. (2023) J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §1. Bai et al. (2025) S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: §1, §1, §3, §5.1. Chen et al. (2025) X. Chen, B. Liu, X. Wang, Y. Wang, and C. Lu Vrprm: process reward modeling via visual reasoning. arXiv preprint arXiv:2508.03556. Cited by: §1, §2. Chen et al. (2023) X. Chen, R. Aksitov, U. Alon, J. Ren, K. Xiao, P. Yin, S. Prakash, C. Sutton, X. Wang, and D. Zhou Universal self-consistency for large language model generation. arXiv preprint arXiv:2311.17311. Cited by: §5.1. Chen et al. (2024) Z. Chen, Z. Zhao, H. Luo, H. Yao, B. Li, and J. Zhou Halc: object hallucination reduction via adaptive focal-contrast decoding. In Proceedings of the International Conference on Machine Learning (ICML), Cited by: §3.2. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §1, §2. Dai et al. (2023) W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi InstructBLIP: towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1. Dong et al. (2025) X. Dong, S. Dong, J. Wang, J. Huang, L. Zhou, Z. Sun, L. Jing, J. Lan, X. Zhu, and B. Zheng INTER: mitigating hallucination in large vision-language models by interaction guidance sampling. In International Conference on Computer Vision (ICCV), Cited by: §3.2. Goyal et al. (2017) Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh Making the v in vqa matter: elevating the role of image understanding in visual question answering. In Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §B.1, §3, §5.1. Guan et al. (2024) T. Guan, F. Liu, X. Wu, R. Xian, Z. Li, X. Liu, X. Wang, L. Chen, F. Huang, Y. Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §B.1, §1, §5.1. Gui et al. (2024) L. Gui, C. Gârbacea, and V. Veitch Bonbon alignment for large language models and the sweetness of best-of-n sampling. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2. Gwak et al. (2025) M. Gwak, G. Son, and J. Kim Revisiting the uniform information density hypothesis in llm reasoning traces. arXiv preprint arXiv:2510.06953. Cited by: §2. Hessel et al. (2021) J. Hessel, A. Holtzman, M. Forbes, R. Le Bras, and Y. Choi Clipscore: a reference-free evaluation metric for image captioning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §2, §5.1. Hinton (2002) G. E. Hinton Training products of experts by minimizing contrastive divergence. Neural computation 14 (8), p. 1771–1800. Cited by: §A.1. Holtzman et al. (2019) A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. Cited by: §5.1. Kadavath et al. (2022) S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. Cited by: §2. Kang et al. (2025) Z. Kang, X. Zhao, and D. Song Scalable best-of-n selection for large language models via self-certainty. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2, §3, §3, §5.1. Lee et al. (2025) D. Lee, J. Hong, D. Kim, and J. Kim Training-free llm verification via recycling few-shot examples. arXiv preprint arXiv:2506.17251. Cited by: §2. Lee et al. (2024) S. Lee, S. Kim, S. Park, G. Kim, and M. Seo Prometheus-vision: vision-language model as a judge for fine-grained evaluation. In Findings of the Association for Computational Linguistics: ACL 2024, Cited by: §2. Leng et al. (2024) S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §2. Li et al. (2023) Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. Wen Evaluating object hallucination in large vision-language models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §1. Lightman et al. (2024) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. Lin et al. (2014) T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick Microsoft coco: common objects in context. In European Conference on Computer Vision (ECCV), Cited by: §3. Liu et al. (2024) H. Liu, C. Li, Y. Li, and Y. J. Lee Improved baselines with visual instruction tuning. In Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §1, §3. Liu et al. (2023) H. Liu, C. Li, Q. Wu, and Y. J. Lee Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §5.1. Min et al. (2025) K. Min, M. Kim, K. Lee, D. Lee, and K. Jung Mitigating hallucinations in large vision-language models via summary-guided decoding. In Findings of the Association for Computational Linguistics: NAACL 2025, Cited by: §3.2. Park et al. (2026) S. Park, C. Oh, H. K. Choi, X. Du, and S. Li VAUQ: vision-aware uncertainty quantification for lvlm self-evaluation. arXiv preprint arXiv:2602.21054. Cited by: §2, §5.1. Qiu and Miikkulainen (2024) X. Qiu and R. Miikkulainen Semantic density: uncertainty quantification for large language models through confidence measurement in semantic space. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. Rohrbach et al. (2018) A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Saenko Object hallucination in image captioning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §B.1, §1, §3, §5.1. Snell et al. (2025) C. Snell, J. Lee, K. Xu, and A. Kumar Scaling llm test-time compute optimally can be more effective than scaling model parameters. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. Team et al. (2023) G. Team, R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. Cited by: §1. Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §1. Uesato et al. (2022) J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275. Cited by: §2. Wang et al. (2023a) J. Wang, Y. Wang, G. Xu, J. Zhang, Y. Gu, H. Jia, J. Wang, H. Xu, M. Yan, J. Zhang, et al. Amber: an llm-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311.07397. Cited by: §B.1, §5.1. Wang et al. (2024) P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui Math-shepherd: verify and reinforce llms step-by-step without human annotations. In Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §2. Wang et al. (2025) W. Wang, Z. Gao, L. Chen, Z. Chen, J. Zhu, X. Zhao, Y. Liu, Y. Cao, S. Ye, X. Zhu, et al. Visualprm: an effective process reward model for multimodal reasoning. arXiv preprint arXiv:2503.10291. Cited by: §1, §2. Wang et al. (2023b) X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. Wang and Zhou (2024) X. Wang and D. Zhou Chain-of-thought reasoning without prompting. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2. Xiong et al. (2025) T. Xiong, X. Wang, D. Guo, Q. Ye, H. Fan, Q. Gu, H. Huang, and C. Li Llava-critic: learning to evaluate multimodal models. In Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2. Zhao et al. (2026) X. Zhao, Z. Kang, A. Feng, S. Levine, and D. Song Learning to reason without external rewards. In International Conference on Learning Representations (ICLR), Cited by: §3. Zhou et al. (2024) Y. Zhou, Z. Fan, D. Cheng, S. Yang, Z. Chen, C. Cui, X. Wang, Y. Li, L. Zhang, and H. Yao Calibrated self-rewarding vision language models. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. Zhu et al. (2024) D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny Minigpt-4: enhancing vision-language understanding with advanced large language models. In International Conference on Learning Representations (ICLR), Cited by: §1. Zhu et al. (2025) J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y. Duan, W. Su, J. Shao, et al. Internvl3: exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479. Cited by: §1, §1, §5.1. Appendix A Additional Theoretical Analysis A.1 Log-Linear Product-of-Experts Interpretation The grounded token score can be written as ut=log(pt)+αlog(At)=log(pt⋅Atα).u_t= (p_t)+α (A_t)= (p_t· A_t^α ). (9) This resembles a product-of-experts-style score 14, where output-space confidence ptp_t and visual lookback score AtA_t serve as complementary factors that are combined multiplicatively into an unnormalized token-level score. The hyperparameter α controls the relative strength of the visual factor. Thus, tokens with low visual lookback score receive a lower combined score even when their output probability is high. A.2 Proof of Proposition 1 We prove that qλq_λ defined by qλ=argmaxq∈ΔT[λτ∼q[log(Aτ)]+H(q)]q_λ= _q∈ _T [λ\,E_τ q [ (A_τ) ]+H(q) ] (10) has the closed-form solution qλ(t)=Atλ∑j=1TAjλ.q_λ(t)= A_t^λ _j=1^TA_j^λ. (11) Here, ΔT=q∈ℝ≥0T:∑t=1Tq(t)=1 _T= \q _≥ 0^T: _t=1^Tq(t)=1 \ (12) is the set of distributions over output positions. Proof. Expanding the objective using the definition of H(q)H(q), the problem becomes max∑t=1Tq∈ΔTq(t)[λlog(At)−log(q(t))]. _q∈ _T _t=1^Tq(t) [λ (A_t)- (q(t)) ]. (13) Assuming At>0A_t>0 for all t, this is a strictly concave maximization over the probability simplex ΔT _T, so a unique solution exists. Introducing a Lagrange multiplier η for the constraint ∑tq(t)=1 _tq(t)=1, the Lagrangian is ℒ=∑t=1Tq(t)λlog(At) = _t=1^Tq(t)λ (A_t) −∑t=1Tq(t)log(q(t)) - _t=1^Tq(t) (q(t)) (14) +η(∑t=1Tq(t)−1). +η\! ( _t=1^Tq(t)-1 ). Setting ∂ℒ/∂q(t)=0 /∂ q(t)=0 gives λlog(At)−log(q(t))−1+η=0,λ (A_t)- (q(t))-1+η=0, (15) which implies log(q(t))=λlog(At)+(η−1). (q(t))=λ (A_t)+(η-1). (16) Therefore, q(t)∝Atλ.q(t) A_t^λ. (17) Applying the normalization constraint ∑tq(t)=1 _tq(t)=1 yields qλ(t)=Atλ∑j=1TAjλ.q_λ(t)= A_t^λ _j=1^TA_j^λ. (18) ∎ Interpretation. The exponent λ controls the sharpness of qλq_λ. When λ=0λ=0, qλq_λ is the uniform distribution over output positions, recovering uniform averaging. As λ→∞λ→∞, qλq_λ concentrates on the position with the largest AtA_t, approaching hard selection of the most visually attended token. For finite λ>0λ>0, qλq_λ acts as a soft selector that prefers visually attended positions while maintaining nonzero weight on all positions. A.3 Geometric Mean Interpretation We show that the proposed score S(∣,)S(y ,v) can be interpreted as the log of a visual-relevance-weighted geometric mean. This provides a justification for using a normalized weighted average rather than an unnormalized sum over output tokens. Corollary A.0.1. Let qλq_λ be as defined in Proposition 1. Then exp(S(∣,))=∏t=1T(pt⋅Atα)qλ(t). \! (S(y ,v) )= _t=1^T (p_t· A_t^α )^q_λ(t). (19) That is, exp(S) (S) is the visual-relevance-weighted geometric mean of the token-level product scores pt⋅Atαp_t· A_t^α. Proof. By definition, S(∣,)=∑t=1Tqλ(t)[log(pt)+αlog(At)].S(y ,v)= _t=1^Tq_λ(t) [ (p_t)+α (A_t) ]. (20) Since log(pt)+αlog(At)=log(pt⋅Atα), (p_t)+α (A_t)= (p_t· A_t^α ), (21) we have S(∣,)=∑t=1Tqλ(t)log(pt⋅Atα).S(y ,v)= _t=1^Tq_λ(t) (p_t· A_t^α ). (22) Exponentiating both sides gives exp(S(∣,))=∏t=1T(pt⋅Atα)qλ(t). \! (S(y ,v) )= _t=1^T (p_t· A_t^α )^q_λ(t). (23) ∎ Implication. A standard log-likelihood sum, ∑tlog(pt) _t (p_t), scales with response length T, making scores difficult to compare across candidates of different lengths. In contrast, the proposed score is equivalent to maximizing the visual-relevance-weighted geometric mean ∏t=1T(pt⋅Atα)qλ(t), _t=1^T (p_t· A_t^α )^q_λ(t), (24) where the exponents satisfy ∑tqλ(t)=1 _tq_λ(t)=1. Thus, the score compares candidates using a normalized weighted average of token-level product scores rather than an unnormalized sum. Unlike uniform length normalization with q(t)=1/Tq(t)=1/T, the weighting qλq_λ adapts the normalization to visual relevance. Figure 8: Visual lookback and SC emphasize different POS groups. We report mean z-scored Visual lookback and Self-Certainty (SC) scores for each POS category. Across models and benchmarks, visual POS categories tend to receive higher Visual lookback scores, whereas textual categories tend to receive higher SC scores. Appendix B Additional Experimental Details Table 2: Oracle performance across benchmarks and LVLMs. We report Oracle Best-of-N selection performance for N=5N=5 and N=25N=25. Average denotes the average score across the four benchmarks for each candidate budget. Higher scores (↑ ) indicate better performance. Benchmark LLaVA-1.5-7B Qwen2.5-VL-7B InternVL3-8B N=5N=5 N=25N=25 N=5N=5 N=25N=25 N=5N=5 N=25N=25 VQAv2 84.07 92.50 82.83 91.17 88.60 94.80 CHAIR 84.17 90.55 86.11 91.47 86.73 91.72 HallusionBench 80.71 94.42 83.42 92.89 80.54 91.71 AMBER 76.96 83.83 84.97 89.91 88.81 92.76 Average 81.48 90.33 84.33 91.36 86.17 92.75 B.1 Benchmark Details We select benchmarks that are informative for Best-of-N response selection. In particular, the candidate pool should contain meaningful variation in response quality, so that a response scorer has non-trivial room to improve over random selection. We therefore consider both candidate diversity and oracle–average gaps when constructing the evaluation suite. Table 2 reports oracle Best-of-N performance, where the oracle selects the highest-scoring response among the sampled candidates for each input. The large gap between oracle and average performance indicates that the sampled candidate pools contain substantial selection headroom. For the main comparison, we report a single higher-is-better metric for each benchmark. This avoids mixing heterogeneous native metrics in the main table while still allowing a consistent comparison of response scorers. We provide benchmark-specific details below. VQAv2. VQAv2 (9) is a short-answer visual question answering benchmark where each question is paired with multiple human annotations. We use VQAv2 to evaluate whether a scorer can select answers that are grounded in the visual content of the image. For each generated response, we parse the predicted answer and evaluate it against the ground-truth annotations. The primary metric is accuracy: Acc(ans)=min# human annotators that provided ans3,1.Acc(ans)= \ \# human annotators that provided ans3,1 \. Dataset-level accuracy is computed by averaging the per-question scores. CHAIR. CHAIR (29) evaluates object hallucination in open-ended image descriptions. Following the CHAIR setup, generated captions are parsed into object mentions and matched to MS-COCO object categories using the benchmark’s synonym mapping. For each image, let GiG_i denote the set of ground-truth objects and PiP_i denote the set of predicted object mentions extracted from the generated response. A predicted object is counted as hallucinated if it does not appear in GiG_i. Although CHAIR conventionally reports CHAIRS and CHAIRI, we report F1 as the primary metric to obtain a single higher-is-better score for Best-of-N selection: Preci=|Pi∩Gi||Pi|,Reci=|Pi∩Gi||Gi|,Prec_i= |P_i∩ G_i||P_i|, _i= |P_i∩ G_i||G_i|, F1i=2⋅Preci⋅ReciPreci+Reci,F1=1||∑i∈F1i.F1_i= 2·Prec_i·Rec_iPrec_i+Rec_i, 1= 1|D| _i F1_i. When the denominator is zero, the corresponding precision, recall, or F1 term is set to zero. AMBER. AMBER (34) is a multidimensional hallucination benchmark for LVLMs. In the generative setting, AMBER evaluates whether generated responses contain visual instances that are supported by the image. Unlike CHAIR, which focuses on object hallucination, AMBER covers a broader set of visual error types, including object existence, attributes, and relations. For each sample, we compare generated visual instances against the ground-truth visual instance set. Instances absent from the ground-truth set are treated as hallucinated. For consistency with CHAIR and the main table, we report F1 as the primary higher-is-better metric: F1i=2⋅Preci⋅ReciPreci+Reci,F1=1||∑i∈F1i.F1_i= 2·Prec_i·Rec_iPrec_i+Rec_i, 1= 1|D| _i F1_i. This provides a unified measure that rewards both avoiding hallucinated visual claims and covering ground-truth visual content. HallusionBench. HallusionBench (10) evaluates visually grounded reasoning under cases where language priors and visual evidence can conflict. We use it as a discriminative visual-grounding benchmark. For each candidate response, we use a GPT-based evaluator to judge whether the response is visually grounded and consistent with the reference answer. The evaluator returns a correctness label for each response, and we treat uncertain or invalid judgments as incorrect. The primary metric is GPT-evaluated correctness: Correctness=1||∑i∈Judge(Ii,qi,yi)=Correct.Correctness= 1|D| _i 1\Judge(I_i,q_i,y_i)= Correct\. This single accuracy-style metric allows HallusionBench to be compared with the other benchmarks under the same Best-of-N selection framework. B.2 Baseline Implementation Details For USC, when the method failed to produce a valid selection, we fell back to the first sampled response. To reduce the order bias in USC methods, we report the average accuracy over three independent runs (m=3m=3). For VAUQ, we follow the layer choices specified in the original paper: 10–25 for LLaVA-1.5-7B, 12–26 for Qwen2.5-VL-7B, and 10–25 for InternVL3-8B. We use fixed masking ratios of K=0.6K=0.6, 0.50.5, and 0.40.4 for the three models, respectively, and set α=1α=1 throughout all experiments. We implement VAUQ following its attention-knockout-based visual masking procedure as mentioned in paper. Table 3: Hardware and software configuration for scoring overhead measurement. LLaVA-1.5-7B Qwen2.5-VL-7B / InternVL3-8B GPU NVIDIA RTX A6000 NVIDIA H200 CPU AMD EPYC 9354 Intel Xeon Platinum 8568Y+ CUDA 11.7 12.4 cuDNN 8.5.0 9.1.0 Python 3.10.15 3.9.21 PyTorch 2.0.1+cu117 2.6.0+cu124 Transformers 4.31.0 4.57.6 Random seed 42 42 B.3 Scoring Overhead Measurement We measure post-generation scoring overhead in milliseconds per response. Candidate generation is excluded from timing: all scorers are given the same generated candidate responses, and we measure only the additional wall-clock time required to assign scores after generation. Measurements are conducted on CHAIR for LLaVA-1.5-7B, Qwen2.5-VL-7B, and InternVL3-8B. For SC and LookBack, scoring uses model-internal quantities from the LVLM scoring pass, including token probabilities and attention weights. For CLIPScore, we include the external CLIP encoder forward pass. For VAUQ, we include visual-attention perturbation and uncertainty computation. For USC, which scores all candidates from the same input in one teacher-forcing forward pass, we divide the total candidate-set scoring time by the number of candidates. We report mean milliseconds per response with standard deviation across evaluated samples. Due to resource availability, LLaVA-1.5-7B is measured on an NVIDIA RTX A6000, while Qwen2.5-VL-7B and InternVL3-8B are measured on an NVIDIA H200. Therefore, absolute latency values should be interpreted as scorer-overhead comparisons within each model, rather than as direct speed comparisons across LVLMs. Appendix C Detailed Quantitative Results C.1 PoS In Section 3.2, we group generated words into a visual-reference-prone content set and a textual-function set to compare visual lookback score with output-space confidence. Here, we provide a fine-grained breakdown by individual POS tags. We use the same generated responses and scoring setup as in Section 3.2. Each word is assigned a POS tag using spaCy. For words split into multiple subtokens, we average the corresponding subtoken scores to obtain a word-level score. Visual lookback score and Self-Certainty are independently standardized over all analyzed output words within each model–benchmark setup, and we report the mean z-score for each POS category. Figure 8 shows that content-oriented POS categories, such as nouns, proper nouns, adjectives, and numerals, generally exhibit higher visual lookback score than textual-function categories. In contrast, textual-function categories often show relatively higher certainty, reflecting their linguistic predictability. This fine-grained analysis supports the coarse POS grouping used in the main text. At the same time, some rare or context-dependent categories exhibit noisier behavior, which motivates our use of grouped POS sets rather than drawing conclusions from individual POS tags. Appendix D Additional Qualitative Results In Figures 9 and 10, we additionally present the qualitative examples to compare SC, Visual lookback score, and our final score. Figure 9 visualizes token-level scores for two representative cases in which LookBack selects the correct top-ranked response, while both SC and Visual Lookback Score select incorrect responses. These examples illustrate why neither confidence nor visual lookback alone is sufficient. SC often assigns large scores to a small number of highly confident tokens, but these tokens need not correspond to visually grounded evidence for the question. In contrast, Visual lookback Score broadly highlights tokens that refer to visible entities in the image, but it can also emphasize irrelevant or distracting visual content. For example, in the counting case, Visual lookback Score attends to image-referential words such as cars, house, and truck, but fails to distinguish the question-relevant evidence needed to answer the number of parked cars. LookBack instead emphasizes tokens that are both visually grounded and relevant to the question, such as the single car in the image, leading to the correct selection. A similar pattern appears in Figure 10: Visual lookback score highlights visible entities such as the child, toothbrush, and person, while LookBack assigns high scores to tokens supporting the question-relevant relation between the two people. These qualitative results support our design choice of combining token-level confidence with visual lookback and aggregating the resulting scores according to visual relevance. Figure 9: Token-level comparison of response scorers. SC favors a fluent but incorrect response, and Visual Lookback Score highlights visually referential tokens but remains insufficient on its own. LookBack combines confidence and visual lookback to select the visually grounded response. Darker highlights indicate higher token-level scores. Figure 10: Token-level behavior on a visually complex example. Visual lookback Score broadly highlights image-referential tokens, whereas SC places large scores on a few confident tokens. LookBack instead emphasizes question-relevant visual evidence, selecting the correct response. Darker highlights indicate higher token-level scores. Appendix E Usage of AI assistants In preparing this work, we utilized AI-based writing assistants in a limited way to suggest alternative phrasings, correct grammatical errors, and improve the readability. All research ideas, experimental designs, implementation details, and reported results were determined, implemented, and verified by the authors, and all model outputs and quantitative results in the paper were generated and checked through our own code and experiments. AI assistants did not generate any factual content reported in the paper (e.g., experimental results, dataset statistics, or citations), ensuring the originality, scientific contributions, technical content, methodology, and experimental findings are entirely attributable to the authors.