Paper deep dive

GHOST-Q: Towards Studying Grounding Hallucinations Overlooked Under Same-score TradeOffs in Quantized VLMS

Saim Rehman, Muhammad Shafique

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 23

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 9/26/2026, 2:41:14 AM

Summary

The paper introduces GHOST-Q, a framework for evaluating vision-language models (VLMs) under post-training quantization (FP16, INT8, NF4). It demonstrates that preserving aggregate accuracy scores does not guarantee the preservation of visual grounding reliability, as quantization can redistribute specific grounding errors. Additionally, the study reveals that memory reduction via quantization does not necessarily translate to lower inference latency on A100 GPUs and that open-ended generation is subject to significant budget censoring that varies by model architecture and precision.

Entities (14)

Idefics3-8B · model · 99%InternVL3-8B · model · 99%Qwen3-VL-8B · model · 99%GHOST-Q · evaluation-framework · 98%A100 · hardware · 95%FP16 · precision-format · 95%INT8 · precision-format · 95%NF4 · precision-format · 95%AMBER · benchmark · 92%HallusionBench · benchmark · 92%MMStar · benchmark · 92%POPE · benchmark · 92%Visual Grounding Hallucination · phenomenon · 90%Generation Budget Censoring · phenomenon · 88%

Relation Signals (10)

GHOST-Q → evaluates → Qwen3-VL-8B

confidence 95% · We present GHOST-Q, a cross-precision controlled evaluation of three 8B VLM families... We evaluate this intersection using Qwen3-VL-8B

GHOST-Q → evaluates → InternVL3-8B

confidence 95% · We present GHOST-Q, a cross-precision controlled evaluation of three 8B VLM families... We evaluate this intersection using... InternVL3-8B

GHOST-Q → evaluates → Idefics3-8B

confidence 95% · We present GHOST-Q, a cross-precision controlled evaluation of three 8B VLM families... We evaluate this intersection using... and Idefics3-8B

Idefics3-8B → testedwith → NF4

confidence 95% · We evaluate this intersection using... and Idefics3-8B... in FP16, bitsandbytes LLM.int8(), and NF4

Qwen3-VL-8B → testedwith → NF4

confidence 95% · We evaluate this intersection using Qwen3-VL-8B... in FP16, bitsandbytes LLM.int8(), and NF4

InternVL3-8B → testedwith → NF4

confidence 95% · We evaluate this intersection using... InternVL3-8B... in FP16, bitsandbytes LLM.int8(), and NF4

NF4 → reduces → Model Footprint

confidence 93% · NF4 by 64.3–65.7%... NF4 dominates INT8 in both footprint and median latency

Quantization → doesnotguarantee →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Post-training quantization of vision--language models (VLMs) is typically assessed through aggregate task accuracy and memory savings, but preserving a headline score does not guarantee preservation of visual grounding behavior. We present GHOST-Q, a cross-precision controlled evaluation of three 8B VLM families under FP16, INT8, and NF4 across utility and hallucination-sensitive benchmarks. Rather than comparing only aggregate accuracy, we pair FP16 and quantized predictions item by-item to quantify how compression redistributes grounding successes and failures. Five of six quantized variants preserve MMStar accuracy within $\pm2$ percentage points, yet 10 of 36 paired effects remain significant after false-discovery-rate correction, nine on hallucination-sensitive conditions. Same-device A100 profiling further demonstrates that substantial memory reduction does not necessarily mean lower inference latency. Finally, an open-ended AMBER audit reveals strong generation budget censoring whose severity varies by architecture and precision. These results show that quantized VLMs should be evaluated jointly for aggregate utility, grounding reliability, generation behavior, and realized deployment efficiency.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

19,632 characters extracted from source content.

Expand or collapse full text

GHOST-Q: Towards Studying Grounding Hallucinations Overlooked Under Same-score TradeOffs in Quantized VLMS Saim Rehman Muhammad Shafique Abstract Post-training quantization of vision–language models (VLMs) is typically assessed through aggregate task accuracy and memory savings, but preserving a headline score does not guarantee preservation of visual grounding behavior. We present GHOST-Q, a cross-precision controlled evaluation of three 8B VLM families under FP16, INT8, and NF4 across utility and hallucination-sensitive benchmarks. Rather than comparing only aggregate accuracy, we pair FP16 and quantized predictions item-by-item to quantify how compression redistributes grounding successes and failures. Five of six quantized variants preserve MMStar accuracy within ±2± 2 percentage points, yet 10 of 36 paired effects remain significant after false-discovery-rate correction, nine on hallucination-sensitive conditions. Same-device A100 profiling further demonstrates that substantial memory reduction does not necessarily mean lower inference latency. Finally, an open-ended AMBER audit reveals strong generation-budget censoring whose severity varies by architecture and precision. These results show that quantized VLMs should be evaluated jointly for aggregate utility, grounding reliability, generation behavior, and realized deployment efficiency. Index Terms: Vision-Language Models, Quantization, Hallucination, Multimodal Reliability †address: eBRAIN Lab, Division of Engineering, New York University Abu Dhabi (NYUAD), Abu Dhabi, UAE sr7849, muhammad.shafique@nyu.edu 1 Introduction Large vision-language models (VLMs) increasingly serve as general multimodal interfaces, but their memory footprint motivates aggressive compression. Quantization methods such as LLM.int8(), GPTQ, SmoothQuant, AWQ, and NF4-based QLoRA reduce transformer memory while often retaining conventional task performance [1, 2, 3, 4, 5]. In parallel, a distinct literature shows that strong VLMs can hallucinate objects, attributes, relations, or answers that are weakly grounded in visual evidence [6, 7, 8, 9]. This motivates the following question: if quantization preserves multimodal utility, does it also preserve the model’s pattern of grounding successes and failures? Moreover, nominally lower precision need not yield lower latency unless the software stack and kernels realize that efficiency. We evaluate this intersection using Qwen3-VL-8B [10], InternVL3-8B [11], and Idefics3-8B [12], each in FP16, INT8, and NF4. Motivating case study. Recent VLM quantization methods such as MBQ and VLMQ improve low-bit deployment by explicitly accounting for modality imbalance, token importance, and quantization sensitivity [13, 14, 15]. However, their primary validation still centers on retained multimodal benchmark performance and compression efficiency, while hallucination benchmarks such as POPE and AMBER evaluate grounding failures without studying how compression redistributes them [6, 7]. This leaves an important gap: a quantized model may preserve its headline utility score while changing which individual examples it grounds correctly. In our motivating case, Idefics3-NF4 changes MMStar accuracy by only +0.40+0.40 p, yet POPE accuracy falls by 1.241.24 p, with 257 FP16-correct predictions becoming wrong versus 145 FP16 errors being corrected (BH-adjusted q=3.0×10−7q=3.0× 10^-7). This demonstrates why aggregate accuracy alone is insufficient to certify grounding reliability after quantization. Our main contributions are as follows: • We introduce a controlled cross precision evaluation protocol that compares FP16, INT8, and NF4 VLMs under matched examples, preprocessing, prompts, and decoding conditions across three architectures and multiple utility and grounding-sensitive benchmarks. • We move beyond aggregate accuracy by introducing an item-level paired reliability analysis that distinguishes harmful and beneficial precision-induced prediction flips using bootstrap confidence intervals, exact McNemar tests, FDR correction, and matched-pair effect sizes. • We jointly characterize behavioral reliability and realized deployment efficiency on the same A100 platform, showing when memory compression is decoupled from actual inference latency. • We study generation-budget sensitivity in open-ended VLM evaluation, showing that fixed decoding budgets can introduce architecture- and precision-dependent censoring and therefore confound naive hallucination comparisons. 2 Background Figure 1: Overview of the Ghost-Q evaluation pipeline and main analyses. Quantization. To begin, LLM.int8() introduced mixed-precision 8-bit matrix multiplication for large transformers [1]; GPTQ uses approximate second-order information for one-shot low-bit weight quantization [2]; SmoothQuant targets W8A8 PTQ [3]; and AWQ protects activation-salient weight channels [4]. QLoRA introduced NormalFloat-4 (NF4) and double quantization [5]. For multimodal models, Q-VLM targets VLM PTQ through cross-layer dependency modeling [16]; MBQ accounts for different vision/language token sensitivities [13]; VLMQ incorporates token importance into a Hessian-based objective [14]; and Das et al. systematically study bit width, quantization method, and component sensitivity across multimodal tasks [15]. Grounding and hallucination. POPE operationalizes object hallucination through polling-style questions [6]. AMBER provides LLM-free discriminative evaluation of existence, attribute, and relation hallucination [7]. HallusionBench probes image-context reasoning under entangled language hallucination and visual illusions [8]. Surveys document the broader VLM hallucination landscape [9]. Our focus, rather, is complementary: we test whether compression itself changes grounding-sensitive decisions even when aggregate utility is nearly preserved. 3 Methodology 3.1 Controlled Cross-Precision Evaluation Ghost-Q compares each quantized VLM against its own FP16 reference under matched inputs and evaluation conditions. For architecture m, precision p, dataset d, and example i, candidate responses are scored deterministically from model logits, y^im,p=arg⁡maxc∈dsm,p(c∣xi). y_i^m,p= _c _ds_m,p(c x_i). We compute accuracy Am,p,dA_m,p,d and define the paired precision effect as Δm,p,d=Am,p,d−Am,FP16,d. _m,p,d=A_m,p,d-A_m,FP16,d. All FP16–quantized comparisons use the same examples, preprocessing, and candidate sets, isolating precision as the experimental factor. For MMStar, |Δ|≤2| |≤ 2 p is used only as a practical reporting tolerance and not as a statistical equivalence test. 3.2 Paired Grounding-Shift Analysis Aggregate accuracy can remain stable even when different examples change correctness after quantization. We therefore compare FP16 and quantized predictions item-by-item and count harmful C→ and beneficial W→ transitions. We form 10,000 paired bootstrap resamples for confidence intervals and apply the exact McNemar test [17] to discordant pairs. Benjamini–Hochberg correction controls false discovery across all 36 FP16–quantized comparisons [18]. Effect direction is summarized by the matched-pair odds ratio OR=nW→C/nC→W,OR=n_W→ C/n_C→ W, where OR >1>1 indicates more beneficial than harmful flips and OR <1<1 the reverse. 3.3 Open-Ended Generation Audit Forced-choice evaluation isolates precision-induced decision changes from free-form decoding. To examine whether the decoding itself is also precision-sensitive, we additionally evaluate open-ended AMBER generation under identical prompts, greedy decoding, and a shared token budget. We record whether each response naturally terminates or reaches the generation ceiling and use a higher-budget FP16 stopping-length probe to distinguish natural stopping behavior from budget-induced truncation. 4 Experimental Setup 4.1 Models and Quantization We evaluate Qwen3-VL-8B [10], InternVL3-8B [11], and Idefics3-8B [12] in FP16, bitsandbytes LLM.int8(), and NF4. NF4 uses FP16 compute and double quantization [5], while INT8 uses the mixed-precision LLM.int8() path [1]. Neither quantized loader requires a calibration dataset. Module audits show that 92.35–93.38% of logical linear weights are quantized; detected vision and language linear layers are converted while the final LM head remains FP16. 4.2 Benchmarks MMStar provides 1,500 multimodal utility questions [19]. Grounding-sensitive evaluation uses POPE (9,000 examples), AMBER existence (4,924), attribute (7,628), and relation (1,664) items [6, 7], and 951 HallusionBench image examples [8]. Each model–precision configuration therefore produces 25,667 forced-choice predictions, giving 231,003 predictions overall. For open-ended evaluation, we use the 1,004-image AMBER generative subset with the official “Describe this image.” prompt, greedy decoding, and a common 256-token ceiling. Moreover, Qwen3-VL uses the same fixed 1-megapixel longest-edge image budget across all three precisions. 4.3 Hardware Profiling All nine configurations are profiled sequentially on the same NVIDIA A100-SXM4-80GB GPU. After 10 warm-up examples, we perform three measured repeats over the same 200 POPE inputs, yielding 600 measured inferences per configuration. We report runtime model footprint, peak allocated memory, median/p95 end-to-end latency, throughput, and repeat-wise variability. Power telemetry is treated only as run-level characterization rather than per-inference energy. 5 Results 5.1 Aggregate Utility is Mostly Preserved Table 1 summarizes utility and grounding-sensitive accuracy. Five of six quantized variants remain within the ±2p± 2\,p MMStar tolerance; InternVL3-NF4 is the only exception, at −2.07p-2.07\,p relative to FP16. Table 1: Accuracy (%) across utility and grounding-sensitive tasks. “Macro” is the unweighted mean of POPE, three AMBER subsets, and HallusionBench. Model Prec. M* POPE A-E A-A A-R Hall. Macro Qwen3 FP16 63.53 88.68 92.73 86.88 85.22 72.77 85.25 INT8 64.07 88.51 93.70 86.72 85.58 73.92 85.69 NF4 62.80 87.89 94.01 86.43 85.40 71.71 85.09 InternVL3 FP16 66.47 90.87 91.96 87.02 83.35 65.93 83.83 INT8 65.67 91.00 92.18 86.93 83.35 64.98 83.69 NF4 64.40 90.92 92.73 86.63 84.92 64.56 83.95 Idefics3 FP16 47.33 87.43 89.40 77.31 87.02 54.15 79.06 INT8 46.93 86.98 89.32 76.68 87.38 54.47 78.96 NF4 47.73 86.19 90.62 77.81 85.64 54.05 78.86 5.2 Preserved Averages Hide Paired Grounding Shifts The grounding macro changes by at most 0.43p0.43\,p from FP16, yet paired analysis reveals substantial item-level redistribution (Fig. 2). Ten of 36 comparisons remain significant after FDR correction, nine on hallucination-sensitive conditions, and the effects are bidirectional. Table 2 reports all significant effects; matched ORs range from 13.0 for Qwen3 INT8 on AMBER existence to 0.52 for Qwen3 NF4 on POPE. Table 2: FDR-significant FP16–quantized paired effects. Δ is quantized minus FP16 accuracy (p). C→ /W→ denote harmful/beneficial flips. q is the Benjamini–Hochberg-adjusted McNemar p-value. Matched OR >1>1 indicates more beneficial than harmful flips; OR <1<1 indicates the reverse. Model Task Prec. Δ C→ W→ q OR [95% CI] Qwen3 A-exist INT8 +0.97 4 52 <10−8<10^-8 13.00 [4.78, 49.50] Qwen3 A-exist NF4 +1.28 14 77 <10−8<10^-8 5.50 [3.09, 10.53] Idefics3 POPE NF4 -1.24 257 145 3.0×10−73.0× 10^-7 0.56 [0.46, 0.69] Qwen3 POPE NF4 -0.79 149 78 2.6×10−52.6× 10^-5 0.52 [0.39, 0.69] Idefics3 A-exist NF4 +1.22 67 127 1.4×10−41.4× 10^-4 1.90 [1.40, 2.59] InternVL3 A-exist NF4 +0.77 27 65 5.6×10−45.6× 10^-4 2.41 [1.52, 3.92] Idefics3 POPE INT8 -0.46 87 46 2.5×10−32.5× 10^-3 0.53 [0.36, 0.76] InternVL3 A-rel. NF4 +1.56 20 46 8.4×10−38.4× 10^-3 2.30 [1.33, 4.10] Idefics3 A-attr. INT8 -0.63 148 100 1.1×10−21.1× 10^-2 0.68 [0.52, 0.88] InternVL3 MMStar NF4 -2.07 74 43 1.9×10−21.9× 10^-2 0.58 [0.39, 0.86] 5.3 Memory Compression Does Not Imply A100 Speedup Table 3 shows that INT8 reduces model footprint by 42.9–43.8% and NF4 by 64.3–65.7%. However, neither improves median end-to-end latency over FP16 on the tested A100/Transformers-bitsandbytes stack: INT8 incurs 43.5–280.4% higher latency and NF4 +2.2–69.6%. Repeat-wise median-latency CV remains below 0.62%, and NF4 dominates INT8 in both footprint and median latency for all three architectures. These results characterize the tested stack rather than quantization algorithms in general. Figure 2: Accuracy change relative to each architecture’s FP16 baseline. Stars mark comparisons significant after Benjamini–Hochberg correction over all 36 paired tests (q<0.05q<0.05). Table 3: Same-device A100 profile. E2E values are median/p95 latency over 600 measured inferences per configuration. Model Prec. Foot. Peak E2E50 E2E95 Thr. (GiB) (GiB) (ms) (ms) ex/s Qwen3 FP16 16.33 16.53 76.6 86.9 13.00 INT8 9.33 9.63 291.3 309.0 3.40 NF4 5.83 6.18 129.8 134.7 7.70 InternVL3 FP16 14.80 16.01 354.5 363.8 3.68 INT8 8.41 9.63 566.0 578.7 2.12 NF4 5.22 6.55 373.9 381.1 3.43 Idefics3 FP16 15.76 16.96 416.3 537.2 2.34 INT8 8.86 10.09 597.5 733.3 1.64 NF4 5.41 6.73 425.5 539.2 2.30 Figure 3: Measured footprint–latency trade-off on the same A100 GPU. Quantization consistently moves left (smaller footprint) but not down (faster inference); NF4 dominates INT8 in this two-dimensional efficiency plane for all three architectures. 5.4 Open-ended Generation Reveals Budget Sensitivity Table 4: AMBER open-ended generation under a common 256-token budget. Entries are responses that hit the token ceiling out of 1,004. Model FP16 INT8 NF4 Qwen3-VL 794 (79.1%) 768 (76.5%) 734 (73.1%) InternVL3 920 (91.6%) 913 (90.9%) 844 (84.1%) Idefics3 1001 (99.7%) 1001 (99.7%) 1002 (99.8%) All nine configurations complete the 1,004-image subset, yielding 9,036 valid responses, but 7,977 (88.28%) reach the common 256-token ceiling. Censoring varies across both architectures and precision settings (Table 4), showing that quantization can alter open-ended generation-length behavior under otherwise identical decoding conditions. A separate 1,024-token FP16 diagnostic confirms that 256 tokens is below natural stopping length: none of 90 generations reaches the 1,024-token ceiling, with maximum lengths of 400, 635, and 769 tokens for Qwen3-VL, Idefics3, and InternVL3, respectively. We therefore treat the 256-token experiment as a controlled budget-sensitivity audit rather than an uncensored generative-hallucination estimate. 6 Discussion and Conclusion Across three 8B VLM families, GHOST-Q shows that preserved aggregate utility does not guarantee preserved grounding behavior or realized deployment efficiency: quantization can redistribute item-level errors, and substantial memory savings need not reduce latency on the tested stack. Open-ended evaluation further reveals precision-dependent generation-budget censoring. Although limited to three model families, bitsandbytes weight quantization, and batch-one A100 inference, these results motivate evaluating quantized VLMs jointly in terms of utility, paired grounding reliability, generation conditions, and realized deployment behavior. Acknowledgment This work was supported in part by the NYUAD Center for CyberSecurity (CCS), funded by Tamkeen under the NYUAD Research Institute grant G1104. This research was carried out on the High Performance Computing resources at New York University Abu Dhabi. Generative AI Use Disclosure During the preparation of this work, the authors used Generative AI tools (specifically ChatGPT and Grammarly) for language editing, text refinement, and visual refinement of the methodology figure. The authors reviewed and edited all generated or refined content as needed and take full responsibility for the publication’s content. References [1] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer (2022) LLM.int8(): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339. Cited by: §1, §2, §4.1. [2] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh (2022) GPTQ: accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. Cited by: §1, §2. [3] G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han (2023) SmoothQuant: accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, Cited by: §1, §2. [4] J. Lin et al. (2023) AWQ: activation-aware weight quantization for LLM compression and acceleration. arXiv preprint arXiv:2306.00978. Cited by: §1, §2. [5] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer (2023) QLoRA: efficient finetuning of quantized LLMs. arXiv preprint arXiv:2305.14314. Cited by: §1, §2, §4.1. [6] Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. Wen (2023) Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Cited by: §1, §1, §2, §4.2. [7] J. Wang et al. (2023) AMBER: an LLM-free multi-dimensional benchmark for MLLMs hallucination evaluation. arXiv preprint arXiv:2311.07397. Cited by: §1, §1, §2, §4.2. [8] T. Guan et al. (2024) HallusionBench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §1, §2, §4.2. [9] H. Liu et al. (2024) A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253. Cited by: §1, §2. [10] S. Bai et al. (2025) Qwen3-VL technical report. arXiv preprint arXiv:2511.21631. Cited by: §1, §4.1. [11] J. Zhu et al. (2025) InternVL3: exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479. Cited by: §1, §4.1. [12] H. Laurençon, A. Marafioti, V. Sanh, and L. Tronchon (2024) Building and better understanding vision-language models: insights and future directions. arXiv preprint arXiv:2408.12637. Cited by: §1, §4.1. [13] S. Li et al. (2025) MBQ: modality-balanced quantization for large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 4167–4177. Cited by: §1, §2. [14] Y. Xue, Y. Huang, J. Shao, and J. Zhang (2025) VLMQ: efficient post-training quantization for large vision-language models via hessian augmentation. arXiv preprint arXiv:2508.03351. Cited by: §1, §2. [15] G. Das, V. La, E. Lau, A. Shrivastava, and M. Gwilliam (2026) Towards understanding best practices for quantization of vision-language models. arXiv preprint arXiv:2601.15287. Cited by: §1, §2. [16] C. Wang, Z. Wang, X. Xu, Y. Tang, J. Zhou, and J. Lu (2024) Q-VLM: post-training quantization for large vision-language models. arXiv preprint arXiv:2410.08119. Cited by: §2. [17] Q. McNemar (1947) Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12 (2), p. 153–157. Cited by: §3.2. [18] Y. Benjamini and Y. Hochberg (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B 57 (1), p. 289–300. Cited by: §3.2. [19] L. Chen et al. (2024) Are we on the right way for evaluating large vision-language models?. In Advances in Neural Information Processing Systems, Cited by: §4.2.

Instant research discovery