Paper deep dive

Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption

Mehmet Kaan Erol

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 62

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/31/2026, 1:29:33 AM

Summary

This study investigates the 'edge reliability gap' in compressed vision-language models (VLMs) by comparing a 7B-parameter quantized model (Qwen2.5-VL-7B) with a 500M-parameter FP16 model (SmolVLM2-500M). Using a diagnostic framework of Object Blindness, Semantic Drift, and Prior Bias, the research reveals that compact models exhibit qualitatively distinct failure signatures, particularly in negation robustness and dataset-dependent miscalibration, despite similar aggregate performance on coarse metrics.

Entities (5)

COCO Captions · dataset · 100%Qwen2.5-VL-7B · vlm · 100%SmolVLM2-500M · vlm · 100%VQAv2 · dataset · 100%GPT-4o · model · 95%

Relation Signals (3)

SmolVLM2-500M → demonstratesnegationcollapse → 12.5pp

confidence 95% · the compact model exhibits a qualitatively distinct failure signature: a 12.5pp larger negation collapse

Qwen2.5-VL-7B → exhibitsfailuremode → Semantic Drift

confidence 90% · Semantic Drift (B) as the dominant failure mode on VQAv2 and on COCO for Qwen

SmolVLM2-500M → exhibitsfailuremode → Object Blindness

confidence 90% · mixed Object Blindness / Semantic Drift profile for SmolVLM2 on COCO

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:The rapid compression of large vision-language models (VLMs) for edge deployment raises an underexplored question: do compact models fail differently, not merely more often? This study compares a 7-billion-parameter quantised VLM (Qwen2.5-VL-7B, 4-bit NF4) against a 500-million-parameter FP16 model (SmolVLM2-500M) across 4,000 samples from VQAv2 and COCO Captions. A three-category error taxonomy (Object Blindness, Semantic Drift, Prior Bias) is applied as a diagnostic framework. A text-only GPT-4o judge reveals Semantic Drift (B) as the dominant failure mode on VQAv2 and on COCO for Qwen, with a mixed Object Blindness / Semantic Drift profile for SmolVLM2 on COCO; Prior Bias (C) is present on VQAv2 but absent on COCO for both models. Confidence calibration is measured via Expected Calibration Error (ECE) using geometric mean token probability, compositional reasoning is probed with structured negation probes across four templates, and a blur robustness experiment completes the evaluation. For this model pair, the compact model exhibits a qualitatively distinct failure signature: a 12.5pp larger negation collapse (-33.2pp vs. -20.8pp, Wald 95% CI [8.2, 16.8]pp, p < 10^-8), driven almost entirely by COCO while the VQAv2 gap is not statistically significant (4.5pp, p=0.19). The most discriminating template is false_yn: SMOLVLM2-500M responds "Yes" (incorrectly claiming a depicted object is absent) on 100% of COCO trials vs. 14% for Q WEN 2.5-VL-7B. Asymmetric dataset-dependent miscalibration and a blur experiment with two controlled ablations complete the analysis. The fully reproducible pipeline is released for systematic safety auditing of compressed VLMs prior to edge deployment.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

61,893 characters extracted from source content.

Expand or collapse full text

Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption Mehmet Kaan Erol Marmara University, Institute of Pure and Applied Sciences kaanerol@marun.edu.tr https://github.com/knrl/quantifying-vlms Abstract The rapid compression of large vision-language models (VLMs) for edge deployment raises an underexplored question: do compact models fail differently, not merely more often? This study compares a 7-billion-parameter quantised VLM (Qwen2.5-VL-7B, 4-bit NF4) against a 500-million-parameter FP16 model (SmolVLM2-500M) across 4,000 samples from VQAv2 and COCO Captions on identical hardware (single NVIDIA RTX 5090). A three-category error taxonomy (Object Blindness, Semantic Drift, Prior Bias) is applied as a diagnostic framework; a text-only GPT-4o judge reveals Semantic Drift (B) as the dominant failure mode on VQAv2 and on COCO for Qwen, with a mixed Object Blindness / Semantic Drift profile for SmolVLM2 on COCO; Prior Bias (C) is present on VQAv2 but absent on COCO for both models. Confidence calibration is measured via Expected Calibration Error (ECE) using geometric mean token probability (§3.6); compositional reasoning is probed with structured negation probes across four templates; and a blur robustness experiment completes the evaluation. For this model pair, the compact model exhibits a qualitatively distinct failure signature: a 12.5pp12.5\,p larger negation collapse (−33.2pp-33.2\,p vs. −20.8pp-20.8\,p; Wald 95% CI [8.2,16.8]pp[8.2,16.8]\,p, p<10−8p<10^-8), driven almost entirely by COCO (per-dataset gap 20.5pp20.5\,p, p<10−16p<10^-16) while the VQAv2 gap is not statistically significant (4.5pp4.5\,p, p=0.19p=0.19). The most discriminating template is false_yn: SmolVLM2-500M responds “Yes” (incorrectly claiming a depicted object is absent) on 100% of COCO trials vs. 14% for Qwen2.5-VL-7B. Asymmetric dataset-dependent miscalibration and a blur experiment with two controlled ablations complete the analysis. The fully reproducible pipeline is released for systematic safety auditing of compressed VLMs prior to edge deployment. 1. Introduction The proliferation of multimodal AI at the network edge—in smartphones, autonomous vehicles, robotics, and IoT devices—has created enormous pressure to compress large vision-language models [29, 10, 57]. Techniques including quantisation [13, 16, 34], pruning [15, 42], and architecture miniaturisation [29, 1] have produced models that run comfortably within 1–5 GiB VRAM budgets. The standard evaluation narrative is straightforward: smaller or more quantised models achieve lower aggregate accuracy. However, aggregate accuracy conceals the structure of failure. Consider two models with similar accuracy on a VQA benchmark. One consistently misses rare object categories (a perceptual failure); the other hallucinates plausible but incorrect attributes with high confidence (a linguistic failure). These models carry fundamentally different deployment risks. An autonomous vehicle assistant that says “I don’t see a stop sign” (uncertainty-aware) is preferable to one that confidently asserts “there is no stop sign” when one is clearly present (overconfident hallucination) [20, 31]. For safety-critical edge applications, the type of error matters as much as its rate. Despite extensive work on VLM compression [27, 13, 16, 34, 47] and hallucination benchmarking [31, 41, 20, 35], no prior study has jointly measured error taxonomy, calibration quality, and negation robustness across size regimes for compressed VLMs on identical hardware. This gap is instantiated for one representative compact-vs.-large VLM pair; the primary contribution is the evaluation methodology and diagnostic framework: C1. A three-category error taxonomy—Object Blindness, Semantic Drift, Prior Bias—as a reusable diagnostic framework for VLM failures, with two auxiliary categories (Spatial/Relational Error D, Other E) for residual cases (§3.3). A heuristic classifier provides a zero-cost reproducibility baseline; a text-only GPT-4o judge (n=400n=400 API calls, no image access) provides the authoritative labels in Table 5 (prompt in Appendix D). A vision-capable re-judge remains an open extension. C2. The edge reliability gap for this model pair: SmolVLM2-500M achieves 47.8% VQAv2 / 92.2% COCO accuracy vs. Qwen2.5-VL-7B at 55.6% / 91.0%; ECE of 0.329 vs. 0.265 (average); negation drop of −33.2pp-33.2\,p vs. −20.8pp-20.8\,p. Because the two models differ simultaneously in scale, precision, and architecture, these differences characterise the specific pairing rather than isolating any single factor (§5.2). A blur experiment (n=100n=100) finds equal degradation for both models (Δ=3pp =3\,p each, ρ=1.0ρ=1.0, McNemar p=0.68p=0.68, not significant); controlled ablations show NF4 quantisation adds 3pp3\,p blur sensitivity and scale within the Qwen architecture confers robustness (ρarch=0.75 _arch=0.75 for SmolVLM2 vs. Qwen-3B FP16). C3. A fully reproducible evaluation pipeline encompassing all experiments, dataset streaming, and figure generation, enabling systematic safety auditing of VLMs prior to edge deployment. 2. Related Work 2.1 VLM Compression for Edge Deployment Model compression for vision-language tasks has been approached through three complementary strategies. Weight quantisation methods including GPTQ [16], bitsandbytes NF4 [13], AWQ [34], and SmoothQuant [47] dramatically reduce memory requirements but are primarily evaluated on language benchmarks, leaving vision-language grounding quality under deployment conditions under-studied. Structured pruning approaches such as DepGraph [15], SlimVLM [12], and LLaVA-PruMerge [42] reduce model footprint but reveal that visual projection layers are disproportionately sensitive. Architecture miniaturisation via MobileVLM [10], LLaVA-Phi [57], Phi-3.5-Vision [1], and the SmolVLM series [29] pushes the parameter frontier below 1B, demonstrating non-trivial VQA performance at significant accuracy cost on fine-grained tasks. Recent systematic benchmarks reveal that compression effects extend beyond accuracy. LVLM-Compress-Bench [27] shows that weight and KV-cache quantisation produce capability-specific trade-offs across recognition, spatial reasoning, and hallucination. Bouguerra et al. [7] demonstrate that CLIP quantisation always impacts reliability—altering calibration, OOD detection, and shift robustness—with effects depending on pre-training data and quantisation recipe. Bhatnagar et al. [6] show that multimodal tokens exhibit higher variance than text tokens, making some layers highly sensitive to sub-4-bit quantisation. Crucially, across all these lines of work, the structural characteristics of remaining errors are not systematically reported. 2.2 Hallucination in Vision-Language Models Hallucination benchmarks have established important categories of VLM failure. POPE [31] measures object existence hallucination via binary polling. CHAIR [41] quantifies hallucinated objects in captions. HallusionBench [20] constructs adversarial visual illusions to disentangle language-prior hallucination from visual perception errors. NegatedHallu [35] and HallE-Control [51] study hallucination at the small-model end, finding that lightweight models exhibit qualitatively different hallucination profiles: more frequent Prior Bias and fewer Semantic Drift errors. Hallucination mitigation has been approached through RLHF [44], retrospective revision [55], visual contrastive decoding [30], and attention penalty methods [23]. Importantly, all of these interventions are developed and evaluated for models with ≥ 7B parameters. Their effectiveness at the 0.5B scale—where language prior dominance is likely stronger—remains unknown. Comprehensive surveys [24, 26] frame hallucination as a lifecycle-wide failure mode and emphasise structured uncertainty estimation as a diagnostic backbone. 2.3 Robustness Under Visual Corruption ImageNet-C [22] established synthetic corruptions as a standard robustness benchmark. For VLMs, MMRobustBench [32] introduces 14 visual corruptions across seven benchmarks, finding that robustness and accuracy are weakly correlated. Tu et al. [45] evaluate 84 CLIP models under ten corruption types, finding that encoder architecture strongly affects blur sensitivity and that fine-tuning can reduce beneficial shape biases. Bao et al. [5] show that common corruptions induce “embedding variance collapse” in CLIP encoders—intra-class and inter-class variances shrink with corruption severity. Ramos et al. [39] demonstrate that processing and acquisition shifts are systematically encoded in visual features and can dominate semantic predictions. Geirhos et al. [17] show that texture bias in vision models makes them particularly sensitive to blur, which suppresses high-frequency detail. For VLMs with dynamic resolution systems like Qwen2.5-VL, multi-scale tiling may partially compensate for blur degradation, whereas fixed-resolution encoders like SigLIP-400M may lose high-frequency information more readily. This architectural difference—among several confounded factors including model scale and precision—motivates our blur robustness hypothesis (§4.4). 2.4 Confidence Calibration A classifier is calibrated if its predicted confidence matches its empirical accuracy [21]. Expected Calibration Error (ECE) [37] is the canonical scalar summary. Modern deep networks are systematically overconfident [21], and RLHF fine-tuning consistently degrades calibration [48]. For VLMs, calibration is substantially less studied. Xuan et al. [49] show systematic miscalibration across VLM types and develop Visual Confidence-Aware Prompting. Zhao et al. [53] find that VLMs output near-100% verbalized confidence on hallucinated objects. Groot & Valdenegro-Toro [19] report persistent overconfidence in GPT-4V and Gemini Pro Vision. The interaction between quantisation and calibration is largely unexplored; our study provides the first ECE comparison between a 4-bit quantised 7B model and an FP16 0.5B model. 2.5 Negation and Compositional Reasoning Negation remains challenging for neural language models. Ettinger [14] demonstrates that BERT-family models fail negation tests via shallow lexical matching. NegatedHallu [35] finds that LLaVA-1.5-7B incorrectly agrees with false negative visual statements in over 40% of cases. NegVQA [52] constructs naturally negated VQA questions, finding a consistent negation performance gap that is larger for smaller models. Mechanistic accounts [50, 40] suggest that attention-based models may struggle to propagate negation scope effectively—the token “not” modifies semantics locally, but attention patterns can dilute this signal. Empirically, this effect appears to be associated with smaller LLM backbones [52], though whether the underlying mechanism is insufficient attention capacity, inadequate training data coverage of negated constructions, or some other factor remains an open question. 3. Evaluation Framework 3.1 Models Under Study Two models spanning a representative size-vs-efficiency frontier are selected, both deployable on a single consumer GPU (Table 1). Table 1: Models under study. Both are loaded simultaneously on a single NVIDIA RTX 5090 (32 GiB VRAM). Combined footprint ≈ 5.5–6 GiB. Property Qwen2.5-VL-7B SmolVLM2-500M Provider Alibaba / Qwen HuggingFace Parameters ∼ 7.6B ∼ 0.5B Precision NF4 4-bit FP16 VRAM ∼ 4.5 GiB ∼ 1.0 GiB Vision encoder Qwen2.5-VL SigLIP-400M LLM backbone Qwen2.5-7B SmolLM2-135M Qwen2.5-VL-7B is a 7B-parameter model from Alibaba [4], quantised to 4-bit NF4 via bitsandbytes with double quantisation and float16 compute dtype. Its native dynamic resolution system tiles high-resolution images into variable-length visual token sequences, providing multi-scale representation. SmolVLM2-500M is a 500M-parameter model from HuggingFace [29], running in native FP16. It uses SigLIP-400M as its vision encoder and SmolLM2-135M as its language backbone. Design note (three-variable confound). This comparison simultaneously varies scale (7B vs. 0.5B), numerical precision (NF4 vs. FP16), and architecture (Qwen2.5-VL vs. SmolVLM2). All reported differences are therefore associative rather than causally attributed to any single factor. Two controlled ablations (§5.2) partially disentangle these factors by adding Qwen2.5-VL-7B in native FP16 (isolates precision) and Qwen2.5-VL-3B in FP16 (isolates scale within architecture), forming a partial 2×22×2 factorial over precision and scale within the Qwen2.5-VL architecture family. 3.2 Datasets Two established benchmarks are used (Table 2), streamed via HuggingFace datasets with streaming=True. Corrupted or zero-size images are skipped and logged. Table 2: Evaluation datasets. Both are streamed; results are continuously flushed with batch-level checkpointing. Dataset Split Samples Task VQAv2 [3] validation 2,000 Open VQA COCO Captions val 2,000 Captioning Correctness metric. For VQAv2, a prediction is correct if the normalised ground-truth string is a substring of the normalised prediction or vice versa (soft match), approximating the official 10-annotator VQA metric. For COCO Captions, correctness requires keyword overlap: the prediction must contain at least one content word (length ≥ 3) from the ground-truth caption. This is a deliberately coarse proxy whose weakness has concrete downstream consequences. Because matching a single common noun suffices, captions with severe semantic errors may still register as “correct”, inflating both models’ COCO accuracy and compressing the accuracy gap. The near-parity between SmolVLM2-500M (92.2%) and Qwen2.5-VL-7B (91.0%) on COCO is therefore not interpretable as evidence of equivalent captioning quality—standard captioning metrics (CIDEr, METEOR, BLEU-4) would likely reveal a larger quality gap. Including even one standard metric (CIDEr is trivially available via pycocoevalcap) would ground the COCO comparison; this is left for future work and is a clear limitation. Implications for negation experiments. The COCO “both-correct” set—on which the negation probes are computed—is defined using this coarse keyword-overlap criterion. If any rows in the both-correct set are false positives (semantically incorrect captions that happen to share a content word with the reference), the negation probes for those rows operate on an unreliable baseline. Because the keyword-overlap criterion is permissive rather than strict, false positives are the primary risk: the both-correct set may include rows where one or both models already produced poor captions. The 90.0% both-correct rate for COCO is consistent with this concern (a stricter metric would likely yield a lower rate). The negation gap itself—which measures relative change from baseline—is less affected than the absolute accuracy levels, but readers should bear the coarse metric in mind when interpreting COCO negation results. 3.3 Error Taxonomy Three primary error categories target structurally distinct failure modes, plus two auxiliary categories: • Object Blindness (A): Model fails to identify a salient object that is visually prominent. The answer references an irrelevant object or claims nothing is visible. • Semantic Drift (B): Model identifies the correct object category but produces a semantically inconsistent description—wrong colour, count, action, or spatial relation. • Prior Bias (C): Model generates a plausible response consistent with statistical priors about the scene, independent of actual image content. • Spatial/Relational Error (D): Model produces an incorrect spatial relation, object layout, or count. This category captures failures that involve quantitative or positional reasoning rather than object identity or attribute errors. Category D failures are excluded from the three-way A/B/C percentage distribution but are reported as raw counts (Table 5). • Other (E): Failure does not clearly fit A–D. Excluded from the A/B/C distribution; raw counts reported. Categories A–C are mutually exclusive; each failure receives the label of its most proximal cause—the failure mode that, if corrected alone, would make the prediction correct. Categories D and E capture residual cases and are excluded from the A/B/C percentages in Table 5. The heuristic applies D before B: any question containing spatial or counting keywords is assigned D regardless of token overlap, so D counts may include cases that would otherwise qualify as B. The GPT-4o judge prompt (Appendix D) includes all five categories; the main text focuses on A–C as the primary failure modes of interest. Appendix B Example 3 illustrates a Category D case. 3.4 Heuristic Taxonomy Classifier Taxonomy labels are assigned by a rule-based heuristic classifier, which applies a deterministic decision tree to the question, ground-truth, and prediction strings: Spatial/Relational (D) if the question contains spatial or count keywords; Object Blindness (A) if negation words appear or no ground-truth tokens overlap the prediction; Semantic Drift (B) if partial ground-truth token overlap is present; Prior Bias (C) if the prediction is verbally fluent but shares no tokens with the ground truth; Other (E) otherwise. The classifier has no image access. A text-only GPT-4o-as-judge [54, 9] was subsequently executed as the primary taxonomy judge (March 2026; n=400n=400 API calls, text only, no image access); results appear in Table 5 and §4.2. The GPT-4o prompt is preserved in Appendix D. The heuristic remains a zero-cost reproducibility fallback; note that both judges operate on text only (question, ground-truth, prediction) without image access—a vision-capable re-judge remains an open extension. Implementation note. SmolVLM2 outputs retain the chat-template prefix “Assistant:” after decoding. A clean_prediction() helper strips this prefix before taxonomy classification to prevent spurious token overlap from corrupting the A/B/C decision logic. The correctness metrics are unaffected because their normalisation already removes such tokens. A failure-profile concordance score is computed using Cohen’s κ formula [11] between taxonomy labels for each model’s failures. Because the two failure sets consist of different images, this is not standard inter-rater κ; it measures whether the taxonomy categories are used with similar relative frequency across models—a descriptive comparison, not a reliability estimate. True inter-rater κ (requiring shared cases) is left as future work. 3.5 Gaussian Blur Perturbation Visual robustness is measured by re-running both models on Gaussian-blurred versions of a 100-sample subset where both models were originally correct (the baseline-correct set). Blur is applied (kernel 5×55×5, σ=2.0σ=2.0), corresponding approximately to severity level 2 on the ImageNet-C scale [22]. This severity was chosen to produce non-trivial but not catastrophic degradation—the inflection point where encoder resolution differences are most diagnostic. The relative robustness ratio ρ quantifies differential blur sensitivity: ρ=ΔSmolΔQwen,ρ= _Smol _Qwen\,, (1) where Δmodel=Accclean−Accblurred _model=Acc_clean-Acc_blurred. ρ>1ρ>1 indicates that SmolVLM2-500M is proportionally more sensitive to blur. 3.6 Confidence Calibration Per-token log-probabilities are aggregated into a sequence-level confidence score via geometric mean token probability: conf(s)=exp⁡(1|s|∑t=1|s|log⁡p(st∣s<t,x)),conf(s)= \! ( 1|s| _t=1^|s| p(s_t s_<t,x) )\,, (2) where s is the generated token sequence and x is the multimodal input. Predictions are binned into M=10M=10 equal-width confidence bins over [0,1][0,1]. Expected Calibration Error [37] is: ECE=∑m=1M|Bm|N|acc(Bm)−conf¯(Bm)|,ECE= _m=1^M |B_m|N |acc(B_m)- conf(B_m) |\,, (3) where |Bm||B_m| is the bin population, acc(Bm)acc(B_m) is fraction correct in bin m, and conf¯(Bm) conf(B_m) is the arithmetic mean of per-sequence geometric-mean confidences within bin m. This two-level aggregation—geometric mean within sequences, arithmetic mean across sequences in a bin—is consistent with standard LLM calibration practice [25]. Proxy validity caveats. Three limitations apply. First, for yes/no VQA questions, sequence-level log-probabilities conflate answer confidence with generation fluency; a first-token proxy would be cleaner. Second, variable-length COCO captions accumulate low-probability tokens, artificially lowering the geometric mean and likely contributing to SmolVLM2-500M’s COCO underconfidence (ECE =0.431=0.431). Third, Kadavath et al. [25] elicit explicit “P(True)” confidence rather than extracting it from generation log-probabilities; the approaches are methodologically distinct. Despite these caveats, token log-probability geometric mean is the standard proxy for auto-regressive models, and the most anomalous finding—Qwen2.5-VL-7B’s constant ≈ 0.999 confidence on VQAv2—is robust to any reasonable proxy choice. 3.7 Negation Stress Tests Four negation operator templates are applied to up to 100 both-correct rows per dataset (Table 3), yielding up to 800 judgements per model per dataset. Baseline accuracy is 100% by construction, making the negation drop a clean measure of compositional sensitivity. Table 3: Negation probe templates and correctness criteria. For false_yn, the placeholder answer is the model’s original correct answer—an object that is depicted. The question therefore asks whether something present is absent; the semantically correct reply is “no”. A model that says “yes” is hallucinating (incorrectly claiming the depicted object is absent). Template Correctness criterion is_not Does not contain original answer absent Does not contain original answer false_yn Contains “no” in the first 20 normalised characters (rejects false premise)111Implemented as "no" in resp[:20] in negation_probes.py: the response is lowercased and stripped, then the first 20 characters are checked for the substring “no”. This window is narrow enough to capture the answer token while avoiding false positives from explanatory text later in the response. counter Contains a non-trivial word ≠ original A model that collapses “no dog” into “dog” at the embedding level will answer as if the negation were absent. The false_yn template is the most discriminating: it asserts a false premise about a depicted object being absent, and the correct response is “No”. A model responding “Yes” is hallucinating. 4. Experiments and Results All experiments run on a single NVIDIA RTX 5090 (32 GiB VRAM) with deterministic seeding (SEED=42), torch.backends.cudnn.deterministic=True, and benchmark=False. The pipeline consists of six sequential phases: sanity check, batch inference, blur robustness, ECE calibration, negation probes, and LLM-as-judge taxonomy validation. 4.1 Baseline Accuracy Table 4 presents clean-image accuracy on 2,000 samples per dataset. Qwen2.5-VL-7B leads SmolVLM2-500M by +7.8pp+7.8\,p on VQAv2 but trails by −1.2pp-1.2\,p on COCO Captions. The COCO near-parity should be interpreted with caution: the single-keyword overlap metric used for COCO correctness (§3) is deliberately coarse, and near-parity under this metric is not evidence of equivalent captioning quality—standard metrics (CIDEr, METEOR, BLEU-4) would likely reveal a larger quality gap. VQAv2 demands fine-grained object recognition and attribute reasoning where the larger model’s richer representations are associated with higher accuracy. Table 4: Clean-image accuracy (n=2,000n=2,000 per dataset). Model VQAv2 COCO Combined SmolVLM2-500M (FP16) 47.8% 92.2% 70.0% Qwen2.5-VL-7B (NF4) 55.6% 91.0% 73.3% Δ (Qwen−-Smol) +7.8 p −-1.2 p +3.3 p 4.2 Error Taxonomy Distribution A text-only GPT-4o semantic judge assigned taxonomy labels to n=100n=100 failures per model × dataset (400400 labels total). The judge receives the question, ground-truth answer, and model prediction as text and returns a structured JSON label (category A–E plus free-text reason); no image is sent to the judge. A rule-based heuristic is also available as a zero-cost reproducibility fallback. As with all model comparisons in this study, the three-variable confound applies (§5.2). VQAv2 taxonomy. Table 5 shows the distribution for VQAv2 failures. Semantic Drift (B) dominates for both models (SmolVLM2: 69.4%, Qwen NF4: 51.9% of A–C labels), consistent with both models identifying the correct object category but producing wrong attributes (colour, count, or relation). Qwen NF4 shows more Prior Bias (C) (20.4% vs. 4.2%), suggesting the larger model generates more plausible-but-wrong predictions anchored to scene priors rather than specific image content. Spatial/Relational Errors (D) are 19 (SmolVLM2) and 24 (Qwen), reflecting VQAv2’s spatial and counting questions; these are excluded from the A–C percentage. Both models always produce text in failure cases (0% empty-output rate). COCO Captions taxonomy. The GPT-4o judge breaks the 100% Object Blindness artefact that the keyword-mismatch heuristic produced for COCO. Semantic Drift (B) is dominant for Qwen NF4 (82.8% of A–C labels), and the two failure categories are roughly split for SmolVLM2 (Object Blindness A 46.6%, Semantic Drift B 53.4%). Prior Bias (C) is 0% for both models on COCO, consistent with captions being open-ended descriptions where statistical scene priors are harder to confuse with specific errors. Spatial Error (D) is rare (SmolVLM2: 8, Qwen: 3), as expected for free-form captioning. The markedly different COCO failure profiles between the two models (Qwen: Semantic Drift dominant; SmolVLM2: split A/B) suggest that the larger model’s stronger language backbone produces caption text that semantically misrepresents scene content, while the compact model more often omits key objects entirely. Table 5: Error taxonomy distribution from GPT-4o text-only judge (% of A–C labelled failures; n=100n=100 per model × dataset, seed 42). D and E reported as raw counts; both are excluded from the A–C percentage denominator. Dataset Error Type SmolVLM2-500M Qwen2.5-VL-7B VQAv2 Object Blindness (A) 26.4% 27.8% Semantic Drift (B) 69.4% 51.9% Prior Bias (C) 4.2% 20.4% Spatial Error (D) (19) (24) Other (E) (9) (22) COCO Object Blindness (A) 46.6% 17.2% Semantic Drift (B) 53.4% 82.8% Prior Bias (C) 0.0% 0.0% Spatial Error (D) (8) (3) Other (E) (4) (4) Figure 1: GPT-4o text-only error-taxonomy distribution across VQAv2 and COCO Captions (n=200n=200 failures per dataset; SmolVLM2 and Qwen NF4 averaged). VQAv2: Semantic Drift (B) dominates (39.0% combined), followed by Spatial Error (D, 21.5%); Object Blindness (A) is 17.0% and Prior Bias (C) is 7.0%. COCO: Semantic Drift (B) dominates (62.0% combined), with Object Blindness (A) at 28.5%; Prior Bias (C) is 0% for both models. Failure-profile concordance. The failure-profile concordance score (Cohen’s κ formula applied across different failure sets; see §3.4) is κ=0.017κ=0.017 on VQAv2 (slight), consistent with the two models exhibiting broadly similar failure distributions on VQAv2 but with Qwen2.5-VL-7B showing more Prior Bias (C) (20.4%) than SmolVLM2-500M (4.2%). On COCO Captions, κ=−0.038κ=-0.038 (less than chance), reflecting the divergent COCO failure profiles: Qwen2.5-VL-7B concentrates Semantic Drift (B, 82.8%) while SmolVLM2-500M is split between Object Blindness and Semantic Drift (46.6% A / 53.4% B). A proper inter-rater consistency estimate (test-retest double-labelling of a shared subset) is left as future work. Figure 2: Qualitative failure grid: three representative errors from llm_judge_labels.json. Columns show the input image, ground truth, Qwen2.5-VL-7B output (green), and SmolVLM2-500M output (red). Rows 1–2: Object Blindness on VQAv2 and COCO. Row 3: Spatial Error (VQAv2). 4.3 Confidence Calibration Table 6 presents ECE results across both models and datasets. The three-variable confound (§5.2) applies: the ECE differences below are a joint function of scale, precision, and architecture. In particular, Qwen’s confidence saturation (p≈0.999p≈ 0.999 for all predictions) may be partly an artefact of aggregating token-level log-probabilities over short VQA answers in a closed-answer setting—the softmax probability of the dominant token approaches 1.0 when the model’s instruction-tuned distribution is sharply peaked, independent of genuine calibration quality. The finding is nonetheless operationally meaningful: regardless of its cause, a constant confidence function provides zero discriminative signal for deployment gating. The calibration landscape is asymmetric and dataset-dependent—a finding with important implications for confidence-based deployment gating. Table 6: Expected Calibration Error (M=10M=10 bins, n=2,000n=2,000 per dataset). Lower is better; a perfectly calibrated model has ECE =0=0. Model VQAv2 COCO Average SmolVLM2-500M (FP16) 0.228 0.431 0.329 Qwen2.5-VL-7B (NF4) 0.443 0.087 0.265 VQAv2 calibration. Qwen2.5-VL-7B exhibits the degenerate extreme of miscalibration on VQAv2: all 2,000 predictions fall in the top confidence bin (≥ 0.9; mean confidence 0.999) while achieving only 55.6% accuracy, yielding ECE =0.443=0.443. This is not routine overconfidence—it is a constant confidence function. Any threshold set below 0.999 passes every prediction unconditionally (including all erroneous ones); any threshold at or above 0.999 rejects everything. Qwen2.5-VL-7B’s token-level confidence carries zero discriminative signal for deployment gating on this task. SmolVLM2-500M distributes predictions across multiple bins (bins 3–9) with moderate overconfidence (ECE =0.228=0.228), meaning that while its confidences are imperfect, they carry some discriminative signal—a notable contrast with Qwen2.5-VL-7B’s saturated output. COCO calibration. The pattern reverses. Qwen2.5-VL-7B places all 2,000 predictions in the top bin again, but with 91.1% accuracy the miscalibration is much smaller (ECE =0.087=0.087). SmolVLM2-500M clusters predictions predominantly in bins 3–6 (confidence 0.37–0.63) while achieving 92.2% accuracy, resulting in severe underconfidence miscalibration (ECE =0.431=0.431). Implications. Qwen2.5-VL-7B’s saturated confidence on VQAv2 renders any confidence-based gate equivalent to no gate—every prediction is admitted at maximum certainty, including the 44.4% that are wrong. Conversely, SmolVLM2-500M’s COCO underconfidence would suppress many correct answers. No single threshold works across tasks for either model; task-adaptive calibration [21] is essential. Figure 3 visualises this pattern. Figure 3: Reliability diagrams for both models on VQAv2 (left) and COCO Captions (right). The dashed diagonal represents perfect calibration. Qwen2.5-VL-7B (blue) on VQAv2 collapses its entire 2,000-prediction distribution to a single point at confidence ≈ 0.999, accuracy ≈ 0.556—annotated and isolated far to the right of the diagonal. This is the defining visual of confidence-function degeneration: the model issues maximum confidence unconditionally, regardless of correctness. SmolVLM2-500M (red, dashed) distributes predictions across multiple bins: below the diagonal on VQAv2 (overconfident) and above it on COCO (underconfident). 4.4 Robustness Under Gaussian Blur Table 7 presents accuracy under Gaussian blur (σ=2.0σ=2.0) on the n=100n=100 baseline-correct subset.222Both models answered correctly on all 100 images under clean conditions, giving a 100% clean baseline by construction. 95% bootstrap CIs (10,000 resamples) and McNemar’s test. The both-correct subset from which these 100 rows are drawn constitutes 38.2% of VQAv2 and 90.0% of COCO Captions; see §4.5 for full subset statistics. Table 7: Gaussian blur robustness (kernel 5×55×5, σ=2.0σ=2.0, n=100n=100 both-correct). Drop shown with 95% bootstrap CI (10,000 resamples, percentile method). Original accuracy is 100% by construction. McNemar p tests the paired null of no differential degradation. Model Clean Blurred Drop 95% CI SmolVLM2-500M (FP16) 100.0% 97.0% 3.0 p [0.0, 7.0] p Qwen2.5-VL-7B (NF4) 100.0% 97.0% 3.0 p [0.0, 7.0] p Sensitivity ratio ρ 1.00 [0.00, 5.00] McNemar p (paired, continuity-corrected) 0.683 The bootstrap CIs span zero for both models, and McNemar’s test (p=0.683p=0.683, 3 discordant pairs each) detects no significant differential degradation at n=100n=100. Two controlled ablations (Table 8) examine whether quantisation or scale modulate this equal sensitivity. Table 8: Four-model blur ablation (kernel 5×55×5, σ=2.0σ=2.0, n=100n=100). Original accuracy for SmolVLM2 and Qwen-7B NF4 is 100% by construction; Qwen-7B FP16 baseline is 97% and Qwen-3B baseline is 81% (these models were not part of the selection criterion). Model Prec. Clean Blurred Drop Ret.† SmolVLM2-500M (0.5B) FP16 100.0% 97.0% −-3.0 p 0.970 Qwen2.5-VL-3B‡ FP16 81.0% 77.0% −-4.0 p 0.951 Qwen2.5-VL-7B FP16 97.0% 97.0% 0.0 p 1.000 Qwen2.5-VL-7B NF4 100.0% 97.0% −-3.0 p 0.970 ρorig _orig (SmolVLM2 / Qwen-7B NF4) 1.00 ρprecision _precision (SmolVLM2 / Qwen-7B FP16) ∞ (degenerate: 0 ppdenom.) ρscale _scale (Qwen-3B / Qwen-7B, both FP16) ∞ (degenerate: 0 ppdenom.) ρarch _arch (SmolVLM2 / Qwen-3B, both FP16) 0.75 †Retention = blurred/clean (own baseline). ‡Qwen-3B baseline is 81%: the 100-image pool was drawn from rows correct for SmolVLM2 & Qwen-7B NF4, not Qwen-3B. Three findings emerge from the n=100n=100 ablation. First, NF4 quantisation adds blur sensitivity: the NF4 variant drops 3.0pp3.0\,p whereas the FP16 variant shows 0pp0\,p drop (97%→ 97%), so the precision ratio is degenerate (ρprecision=∞ _precision=∞). Second, scale within the Qwen architecture confers robustness: Qwen-7B FP16 (0pp0\,p drop) is more robust than Qwen-3B FP16 (4pp4\,p drop), also with a degenerate ratio (ρscale=∞ _scale=∞). Third, at comparable precision and scale, SmolVLM2 (3pp3\,p) is marginally less sensitive than Qwen-3B (4pp4\,p), giving ρarch=0.75 _arch=0.75—counter to the SigLIP encoder bottleneck hypothesis. The degenerate ρ values for precision and scale indicate that Qwen-7B FP16 is fully robust to this blur level on this pool; the 4pp4\,p Qwen-3B drop should be interpreted cautiously given its lower baseline accuracy (81%) on the SmolVLM2-selected pool. Figure 4 shows the clean vs. blurred accuracy for both models at σ=2σ=2. Interpreting degenerate ρ values. Two entries show ρ=∞ρ=∞ because Qwen-7B FP16 shows 0pp0\,p degradation (97%→ 97%), making the denominator zero. The key signal is directional: SmolVLM2 and Qwen-NF4 both degrade by 3pp3\,p, Qwen-3B by 4pp4\,p, while Qwen-7B FP16 is fully robust at this blur level. Both-correct subset caveat. Sampling only from rows where both models were originally correct ensures a clean 100% baseline but may over-represent visually simple images, limiting headroom for degradation. A companion script (stratified_robustness.py) extends the experiment to all four correctness strata; results are left as future work. Figure 4: Accuracy under clean (σ=0σ=0) and Gaussian-blurred (σ=2σ=2) conditions for both models (n=100n=100 both-correct images). Both conditions are directly measured; no extrapolation is performed. Both models drop 3.03.0 p, giving ρ=1.00ρ=1.00 (95% CI [0.00,5.00][0.00,5.00]; McNemar p=0.683p=0.683; differential degradation not significant). 4.5 Negation Stress Tests Table 9 presents per-dataset negation accuracy. The negation gap is strongly dataset-dependent: on COCO, SmolVLM2-500M degrades 20.5pp20.5\,p more than Qwen2.5-VL-7B (95% CI [15.8,25.2][15.8,25.2], p<10−16p<10^-16), while on VQAv2 the gap is only 4.5pp4.5\,p and not statistically significant (p=0.19p=0.19). The aggregate gap—12.5pp12.5\,p (95% CI [8.2,16.8]pp[8.2,16.8]\,p, z=5.69z=5.69, p<10−8p<10^-8)—is therefore driven almost entirely by the COCO templates. Instruction-following asymmetry caveat. The false_yn template (“Is it true that [answer] is NOT shown in this image?”) uses the model’s own correct answer and requires rejection of a false negation. This is a well-designed logical inversion probe, but it also tests instruction-following capability: Qwen2.5-VL-7B’s 7B instruction-tuned backbone may find it inherently easier to parse and comply with the complex prompt structure, independent of negation reasoning per se. The 0% result for SmolVLM2-500M is therefore partly confounded with the model’s general instruction-following capacity. A control template testing other forms of logical inversion—not involving the model’s own prior answer—would help isolate negation reasoning specifically; this is left as future work. Table 9: Aggregate negation accuracy (n=100n=100 both-correct rows per dataset × 4 templates = 800 judgements per model). Baseline accuracy is 100% by construction. Wilson 95% CIs in brackets. Model Baseline Negation 95% CI Drop SmolVLM2-500M (FP16) 100.0% 66.8% [63.4, 69.9] −-33.2 p Qwen2.5-VL-7B (NF4) 100.0% 79.2% [76.3, 81.9] −-20.8 p Gap (Qwen −- Smol) 12.5 p Wald 95% CI [8.2, 16.8] p z=5.69z=5.69, p<10−8p<10^-8 Per-template breakdown. Table 10 reveals highly template-specific behaviour. The most discriminating template is false_yn: SmolVLM2-500M achieves 2% success on VQAv2 (Wilson 95% CI [0.6, 7.0]) and 0% on COCO (CI [0.0, 3.7])—responding “Yes” (the hallucination response) on 98–100% of trials. Qwen2.5-VL-7B scores 49% (CI [39.4, 58.7]) and 86% (CI [77.9, 91.5]) respectively. The counter template shows a similar asymmetry on VQAv2 (39% vs. 100%). Conversely, both models handle is_not and absent well on COCO (≥ 97%), suggesting that open-ended negation is easier than logical inversion under a false premise. Per-dataset breakdown. The aggregate gap masks a dramatic asymmetry: COCO accounts for nearly all of it (20.5pp20.5\,p, p<10−16p<10^-16), driven by the false_yn collapse, while VQAv2 is not significant (4.5pp4.5\,p, p=0.19p=0.19). The strongest negation claim rests entirely on COCO. Table 10: Per-template negation success rates (%) across VQAv2 and COCO. n=100n=100 per cell. Bold indicates the most discriminating comparisons. VQAv2 COCO Template SmolVLM2-500M Qwen2.5-VL-7B SmolVLM2-500M Qwen2.5-VL-7B is_not 94 47 100 97 absent 99 56 100 99 false_yn 2 49 0 86 counter 39 100 100 100 Average 58.5 63.0 75.0 95.5 Interpretation. The false_yn collapse suggests SmolVLM2-500M processes the negated prompt as a positive object-identification question, responding “Yes” as if “NOT” were absent—achieving 0% on COCO. This failure mode is invisible to standard accuracy metrics. Interestingly, SmolVLM2-500M outperforms Qwen2.5-VL-7B on is_not and absent for VQAv2 (94/99% vs. 47/56%), possibly because Qwen2.5-VL-7B’s larger language model anchors more strongly to the original answer—though tokenisation and instruction-tuning differences could also contribute. This cross-template divergence shows that negation comprises distinct compositional skills, not a monolithic capability. Figure 5 shows the full breakdown. Both-correct subset caveat. The 12.5pp12.5\,p negation gap is measured only on both-correct images (38.2% of VQAv2, 90.0% of COCO). This set is enriched for visually unambiguous cases; the negation gap on harder images is uncharacterised. For COCO, 90% coverage makes the subset nearly representative. The false_yn collapse is therefore a pure compositional failure; the total deficit on arbitrary images remains uncharacterised. Under the most pessimistic assumption (zero additional collapse outside the both-correct set), selection-weighted lower bounds are ≈1.7pp≈1.7\,p for VQAv2 and ≈18.5pp≈18.5\,p for COCO. The VQAv2 bound is notably small, confirming selection bias is a material concern there. However, harder images likely exhibit greater negation collapse, so true population-level gaps may exceed the measured values. The false_yn finding (0% for SmolVLM2-500M on COCO) is unaffected by selection bias, as it operates only on correctly-identified objects. Figure 5: Negation probe success rates per template across VQAv2 (left) and COCO Captions (right). Qwen2.5-VL-7B (blue) and SmolVLM2-500M (red) are shown side-by-side. The false_yn template is the most discriminating: SmolVLM2-500M collapses to 2% on VQAv2 and 0% on COCO, while Qwen2.5-VL-7B scores 49% and 86%. 5. Discussion 5.1 Structural Characterisation of the Reliability Gap For this specific model pair, the compact model fails differently, not just more often. The four evaluation axes—taxonomy, calibration, blur robustness, and negation—reveal qualitatively distinct failure profiles. Because the two models differ simultaneously in scale, architecture, and numerical precision (§5.2), the interpretations below are candidate explanations consistent with the observed patterns, not causally established conclusions. (1) No differential blur sensitivity; quantisation and scale effects directional. No significant blur gap is detected (Table 7). The controlled ablations (Table 8) reveal directional signals: NF4 quantisation adds sensitivity, scale confers robustness, and the SigLIP encoder bottleneck hypothesis is not supported (ρarch=0.75 _arch=0.75). These remain directional at n=100n=100. (2) Language head dominance (hypothesised). The false_yn collapse (0–2% success) shows SmolVLM2-500M affirming false premises about present objects being absent on 98–100% of trials—consistent with its 135M-parameter language backbone overriding visual reasoning. This does not constitute mechanistic proof: the collapse could also arise from instruction-tuning differences, tokenisation artefacts, or negation-specific training data coverage. The template also tests instruction-following capacity; Qwen2.5-VL-7B’s larger backbone may parse the complex prompt more easily, independent of negation ability (§4.5). (3) Calibration as a deployment signal (operational, not diagnostic). The asymmetric calibration pattern (Table 6, Figure 3) means confidence scores carry different informativeness depending on task and model. Regardless of whether Qwen2.5-VL-7B’s constant p≈0.999p≈ 0.999 on VQAv2 reflects genuine calibration collapse or a tokenisation artefact, the practical consequence is the same: zero discriminative value for deployment gating. No single threshold policy works across tasks for either model. The ECE differences are a joint function of scale and precision (NF4 vs. FP16) and cannot be attributed to either factor alone. 5.2 The Quantisation Confound A critical limitation is that Qwen2.5-VL-7B runs in 4-bit NF4 while SmolVLM2-500M runs in FP16. The observed reliability gap is therefore a joint function of: (i) model scale (7B vs. 0.5B), (i) numerical precision (NF4 vs. FP16), and (i) architecture (Qwen2.5-VL vs. SmolVLM2). Recent work on quantisation-specific reliability effects [7, 6, 43] suggests that 4-bit quantisation can either improve or worsen calibration depending on whether the base model is under- or over-confident, making the ECE comparison a joint effect rather than a pure scale effect. FP16 precision ablation (completed, n=100n=100). Qwen2.5-VL-7B in native FP16 was re-run on the same 100 both-correct blur rows as §4.4 using (hardware: RTX 5090, 32 GiB VRAM; footprint ≈ 16 GiB). Qwen-7B FP16 shows 0pp0\,p drop (97%→ 97%), while Qwen-7B NF4 drops 3pp3\,p (100%→ 97%). Note that Qwen-7B FP16 starts at 97% original accuracy on this pool (not 100%), because the pool was selected for SmolVLM2 / Qwen-NF4 correctness. The 3pp3\,p precision effect (NF4 vs. FP16) directionally confirms that quantisation hurts blur robustness; ρprecision _precision is degenerate (∞) because the FP16 denominator is 0pp0\,p. Scale ablation within Qwen architecture (completed, n=100n=100). Qwen2.5-VL-3B in FP16 was run on the same 100 rows using (VRAM: ≈ 7 GiB). Qwen-3B drops 4.0pp4.0\,p (81%→ 77%), while SmolVLM2-500M drops 3.0pp3.0\,p, giving ρarch=3.0/4.0=0.75 _arch=3.0/4.0=0.75 (Table 8). SmolVLM2 is marginally less blur-sensitive than Qwen-3B at comparable precision, counter to the SigLIP encoder bottleneck hypothesis. Scale within the Qwen architecture shows a clear benefit: Qwen-7B FP16 (0pp0\,p) is substantially more robust than Qwen-3B FP16 (4pp4\,p), consistent with the larger LLM backbone reconstructing semantics from degraded visual tokens, though the ρscale _scale ratio is degenerate (∞). The Qwen-3B baseline on this pool is only 81% (because the pool was selected for correctness on SmolVLM2 and Qwen-7B, not Qwen-3B), so the 4pp4\,p drop estimate carries a low-baseline caveat. 5.3 Error Taxonomy Interpretation Semantic Drift (B) dominates VQAv2 failures for both models (Table 5), consistent with correct object identification but wrong attributes. Qwen2.5-VL-7B shows notably more Prior Bias (C) on VQAv2 (20.4% vs. 4.2%), suggesting its stronger language backbone generates more plausible-but-wrong predictions anchored to scene priors. The failure-profile concordance (κ=0.017κ=0.017 on VQAv2, −0.038-0.038 on COCO) confirms divergent failure structures, particularly on COCO where Qwen2.5-VL-7B concentrates on Semantic Drift while SmolVLM2-500M splits between Object Blindness and Semantic Drift. The GPT-4o judge operates text-only and cannot verify object presence in images; a vision-capable judge would strengthen COCO categorisation. 5.4 Implications for Edge Deployment These findings have direct implications for edge deployment: Safety-critical applications. The false_yn collapse in SmolVLM2-500M is particularly concerning for safety-critical applications. The template asks “Is it true that [depicted object] is NOT shown?”—the correct answer is “No”. SmolVLM2-500M answers “Yes” on 98–100% of trials, effectively agreeing that present objects are absent. In an operational context, a model that confirms “Is it true that no obstacles are present?” with “Yes” when obstacles exist represents a dangerous false safety assertion, not merely a negation reasoning lapse. Models deployed in autonomous driving, medical imaging, or security contexts must be stress-tested with such logical-inversion probes before deployment. Confidence-based gating. Confidence thresholds for routing uncertain queries collapse for Qwen2.5-VL-7B on VQAv2: constant ≈ 0.999 confidence admits every prediction, including the 44.4% that are wrong. For SmolVLM2-500M on COCO, underconfidence over-routes correct answers to fallbacks. Task-adaptive calibration [21] is essential; off-the-shelf confidence gates without per-task recalibration risk silent failures. Multi-modal robustness testing. A minimum deployment-readiness suite should include: (i) blur/noise perturbation at ImageNet-C severity levels 1–3; (i) negation probes across at least two templates; and (i) ECE measurement with reliability diagrams. The released pipeline provides a reference implementation. 6. Conclusion The central finding is qualitative: for this model pair, the compact model fails differently, not just more often. The error taxonomy reveals distinct failure profiles (§4.2); negation probes expose a striking compositional collapse in SmolVLM2-500M, driven by COCO (§4.5); calibration analysis uncovers asymmetric, dataset-dependent miscalibration that renders confidence-based gating ineffective (§4.3); and blur ablations provide directional evidence on quantisation and scale effects (§4.4). As a single model pair (N=1N=1), generalisation is the most important open question. Aggregate accuracy is insufficient for edge deployment readiness. Compressed VLMs should undergo structured reliability auditing—calibration analysis, perturbation stress-testing, and compositional reasoning probes—before safety-relevant deployment. The taxonomy and pipeline introduced here provide a reusable framework; all code and artefacts are released for full reproducibility. References [1] M. Abdin et al., “Phi-3 technical report: A highly capable language model locally on your phone,” arXiv:2404.14219, 2024. [2] A. Agrawal, D. Batra, D. Parikh, and A. Kembhavi, “Don’t just assume; look and answer: Overcoming priors for visual question answering,” in CVPR, 2018. [3] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering,” in CVPR, 2017. [4] Shuai Bai et al., “Qwen2.5-VL technical report,” arXiv:2502.13923, 2025. [5] W. Bao, R. Deng, and J. He, “Mint: A simple test-time adaptation of vision-language models against common corruptions,” arXiv:2510.22127, 2025. [6] S. Bhatnagar, A. Xu, K. Tan, and N. Ahuja, “LUQ: Layerwise ultra-low bit quantization for multimodal large language models,” arXiv:2509.23729, 2025. [7] A. Bouguerra, D. Montoya, A. Gomez-Villa, F. Arnez, and C. Mraidha, “Can less precise be more reliable? A systematic evaluation of quantization’s impact on CLIP beyond accuracy,” arXiv:2509.21173, 2025. [8] Z. Chen et al., “InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,” in CVPR, 2024. [9] C.-H. Chiang and H.-y. Lee, “Can large language models be an alternative to human evaluations?” in ACL, 2023. [10] X. Chu et al., “MobileVLM: A fast, strong and open vision language assistant for mobile devices,” arXiv:2312.16886, 2023. [11] J. Cohen, “A coefficient of agreement for nominal scales,” Educ. Psych. Meas., vol. 20, no. 1, p. 37–46, 1960. [12] A. Dai et al., “SlimVLM: Structured pruning for compact vision-language models,” arXiv, 2024. [13] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Efficient finetuning of quantized LLMs,” in NeurIPS, 2023. [14] A. Ettinger, “What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models,” TACL, vol. 8, p. 34–48, 2020. [15] G. Fang et al., “DepGraph: Towards any structural pruning,” in CVPR, 2023. [16] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate post-training quantization for generative pre-trained transformers,” in ICLR, 2023. [17] R. Geirhos et al., “ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness,” in ICLR, 2019. [18] J. Geng, F. Cai, Y. Wang, H. Koeppl, P. Nakov, and I. Gurevych, “A survey of confidence estimation and calibration in large language models,” in NAACL, p. 6577–6595, 2024. [19] T. Groot and M. Valdenegro-Toro, “Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models,” arXiv:2405.02917, 2024. [20] T. Guan et al., “HallusionBench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models,” in CVPR, 2024. [21] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in ICML, 2017. [22] D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” in ICLR, 2019. [23] Q. Huang et al., “OPERA: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation,” in CVPR, 2024. [24] L. Huang et al., “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” ACM Trans. Inf. Syst., vol. 43, p. 1–55, 2023. [25] S. Kadavath et al., “Language models (mostly) know what they know,” arXiv:2207.05221, 2022. [26] J. Kou et al., “Beyond isolated fixes: A comprehensive survey on hallucination mitigation with a three-dimensional taxonomy and integrative framework,” in IEEE UV, 2024. [27] S. Kundu et al., “LVLM-Compress-Bench: Benchmarking the broader impact of large vision-language model compression,” arXiv:2503.04982, 2025. [28] J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, no. 1, p. 159–174, 1977. [29] H. Laurençon, A. Marafioti, V. Sanh, and L. Tronchon, “SmolVLM: Redefining small and efficient multimodal models,” HuggingFace Technical Report, 2024. [30] S. Leng et al., “Mitigating object hallucinations in large vision-language models through visual contrastive decoding,” in CVPR, 2024. [31] Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. R. Wen, “Evaluating object hallucination in large vision-language models,” in EMNLP, 2023. [32] T. Li et al., “MMRobustBench: Benchmarking the robustness of large vision-language models,” arXiv, 2024. [33] X. Li, Y. Fang, M. Liu, Z. Ling, Z. Tu, and H. Su, “Distilling large vision-language model with out-of-distribution generalizability,” in ICCV, 2023. [34] J. Lin, J. Tang, H. Tang, S. Yang, X. Dang, and S. Han, “AWQ: Activation-aware weight quantization for LLM compression and acceleration,” in MLSys, 2024. [35] H. Lovenia, W. Dai, S. Cahyawijaya, Z. Ji, and P. Fung, “Negative object presence evaluation (NOPE) to measure object hallucination in vision-language models,” arXiv:2310.05338, 2023. [36] D. Misra, A. Goyal, B. Runwal, and P. Chen, “Uncovering the hidden cost of model compression,” in CVPRW, 2024. [37] M. P. Naeini, G. F. Cooper, and M. Hauskrecht, “Obtaining well calibrated probabilities using Bayesian binning into quantiles,” in AAAI, 2015. [38] A. Radford et al., “Learning transferable visual models from natural language supervision,” in ICML, 2021. [39] R. Ramos, V. Stojnić, G. Kordopatis-Zilos, Y. Nakashima, G. Tolias, and N. García, “Processing and acquisition traces in visual encoders: What does CLIP know about your camera?” arXiv:2508.10637, 2025. [40] A. Ravichander et al., “CONDAQA: A contrastive reading comprehension dataset for reasoning about negation,” in EMNLP, 2022. [41] A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Saenko, “Object hallucination in image captioning,” in EMNLP, 2018. [42] Y. Shang et al., “LLaVA-PruMerge: Adaptive token reduction for efficient large multimodal models,” arXiv:2403.15388, 2024. [43] E. Slyman, A. Kanneganti, S. Hong, and S. Lee, “You never know: Quantization induces inconsistent biases in vision-language foundation models,” arXiv:2410.20265, 2024. [44] Z. Sun et al., “Aligning large multimodal models with factually augmented RLHF,” arXiv:2309.14525, 2023. [45] W. Tu, W. Deng, and T. Gedeon, “Toward a holistic evaluation of robustness in CLIP models,” IEEE TPAMI, vol. 47, p. 8280–8296, 2024. [46] M. Usama, S. Asim, S. Ali, S. Wasim, and U. Mansoor, “Analysing the robustness of vision-language-models to common corruptions,” arXiv:2504.13690, 2025. [47] G. Xiao et al., “SmoothQuant: Accurate and efficient post-training quantization for large language models,” in ICML, 2023. [48] M. Xiong et al., “Can LLMs express their uncertainty? An empirical evaluation of confidence elicitation in LLMs,” in ICLR, 2024. [49] W. Xuan, Q. Zeng, H. Qi, J. Wang, and N. Yokoya, “Seeing is believing, but how much? A comprehensive analysis of verbalized calibration in vision-language models,” arXiv:2505.20236, 2025. [50] J. Yu et al., “Negation as a modality binding failure: VLMs cannot negate across vision and language,” arXiv, 2023. [51] B. Zhai et al., “HallE-Control: Controlling object hallucination in large multimodal models,” arXiv:2310.01779, 2023. [52] Y. Zhao et al., “NegVQA: Probing negation understanding in vision-language models,” arXiv, 2023. [53] Y. Zhao et al., “Object-level verbalized confidence calibration in vision-language models via semantic perturbation,” arXiv:2504.14848, 2025. [54] L. Zheng et al., “Judging LLM-as-a-judge with MT-Bench and Chatbot Arena,” in NeurIPS, 2023. [55] Y. Zhou et al., “Analyzing and mitigating object hallucination in large vision-language models,” arXiv:2310.00754, 2023. [56] A. Zhou, J. Wang, Y.-X. Wang, and H. Wang, “Distilling out-of-distribution robustness from vision-language foundation models,” arXiv:2311.01441, 2023. [57] W. Zhu et al., “LLaVA-Phi: Efficient multi-modal assistant with small language model,” arXiv, 2024. Supplementary Material Appendix A Detailed Calibration Bin Statistics Tables 11–14 report the full M=10M=10 bin statistics for all model–dataset combinations. The contrast between SmolVLM2-500M’s distributed confidence on VQAv2 (predictions spread across bins 2–9) and Qwen2.5-VL-7B’s extreme concentration (all 2,000 predictions in bin 9) is the most salient feature. Table 11: SmolVLM2-500M calibration bins on VQAv2 (n=2,000n=2,000, ECE =0.228=0.228). Bin range Count Avg. Conf. Accuracy [0.2,0.3)[0.2,0.3) 1 0.268 0.0% [0.3,0.4)[0.3,0.4) 23 0.373 39.1% [0.4,0.5)[0.4,0.5) 148 0.459 46.0% [0.5,0.6)[0.5,0.6) 305 0.554 42.3% [0.6,0.7)[0.6,0.7) 448 0.655 45.3% [0.7,0.8)[0.7,0.8) 529 0.751 45.8% [0.8,0.9)[0.8,0.9) 402 0.843 49.8% [0.9,1.0)[0.9,1.0) 144 0.933 72.2% Table 12: SmolVLM2-500M calibration bins on COCO (n=2,000n=2,000, ECE =0.431=0.431). Bin range Count Avg. Conf. Accuracy [0.2,0.3)[0.2,0.3) 18 0.286 77.8% [0.3,0.4)[0.3,0.4) 223 0.370 90.1% [0.4,0.5)[0.4,0.5) 867 0.456 91.4% [0.5,0.6)[0.5,0.6) 713 0.543 94.3% [0.6,0.7)[0.6,0.7) 170 0.633 93.5% [0.7,0.8)[0.7,0.8) 8 0.730 87.5% [0.8,0.9)[0.8,0.9) 1 0.818 0.0% Table 13: Qwen2.5-VL-7B calibration bins on VQAv2 (n=2,000n=2,000, ECE =0.443=0.443). All predictions fall in a single bin, indicating extreme overconfidence. Bin range Count Avg. Conf. Accuracy [0.9,1.0)[0.9,1.0) 2,000 0.999 55.6% All other bins 0 — — Table 14: Qwen2.5-VL-7B calibration bins on COCO (n=2,000n=2,000, ECE =0.087=0.087). All predictions fall in a single bin; high task accuracy keeps ECE low. Bin range Count Avg. Conf. Accuracy [0.9,1.0)[0.9,1.0) 2,000 0.998 91.1% All other bins 0 — — Appendix B Qualitative Failure Examples Example 1 — Category B (Semantic Drift), VQAv2. [Manually verified; heuristic assigns A due to yes/no GT structure.] Q: “Are the boats seaworthy?” GT: “yes.” SmolVLM2-500M: “The boats are not seaworthy, as they are not designed for such conditions.” Qwen2.5-VL-7B: “It is not possible to definitively determine from the image alone whether the boat in the foreground is seaworthy…” By visual inspection, both models identify the boats (correct object class) but produce wrong attribute judgements about seaworthiness—a Semantic Drift failure. The heuristic incorrectly assigns Category A because the GT token “yes” does not appear in predictions (zero text overlap). This is a concrete illustration of the heuristic’s failure mode on yes/no VQAv2 questions: the A-vs-B distinction collapses to whether the GT string happens to appear in the output, not whether the model identified the correct object. Example 2 — Category A (Object Blindness), COCO. Prompt: “Generate a short caption for this image.” GT: “Two women waiting at a bench next to a street.” SmolVLM2-500M: “A woman is sitting on a bench and talking to another woman.” Qwen2.5-VL-7B: “Two individuals wait at a bus stop in an urban setting with modern buildings and palm trees in the background.” Heuristic classifier: Both models miss or under-specify primary scene elements. Example 3 — Category D (Spatial Error), VQAv2. Q: “How many wheels do you see?” GT: “6.” SmolVLM2-500M: “There are two wheels visible in the image.” Qwen2.5-VL-7B: “In the image, there are two wheels visible on the motorcycle in the foreground.” Heuristic classifier: Both models undercount visible wheels — a spatial/quantitative reasoning error. Appendix C Negation Probe Template Prompts Table 15: Full negation probe template prompts. answer is replaced with the model’s original correct answer; question with the original question text. Template Prompt text is_not “What object or thing is NOT depicted in this image?” absent “Name something absent from this image.” false_yn “Is it true that ‘answer’ is NOT shown in this image? Answer yes or no.” counter “The answer to the question ‘question’ is NOT ‘answer’. What is the correct answer?” Appendix D GPT-4o Judge Prompt System: You are an expert evaluator of vision-language model outputs. Classify the following model error into exactly one category: [A: Object Blindness | B: Semantic Drift | C: Prior Bias | D: Spatial/Relational Error | E: Other]. Definitions: A - Model fails to identify a salient visible object. B - Correct object category, wrong attribute (colour, count, action). C - Plausible output consistent with scene priors, wrong for this image. D - Incorrect spatial relation or layout description. E - Does not clearly fit A-D. User: Dataset: dataset Question/Prompt: question Ground Truth: ground_truth Model Output: prediction Respond with JSON: "category":"A|B|C|D|E", "confidence": 0-1, "reasoning": "..." Appendix E Reproducibility Details All experiments share a deterministic seed (SEED=42) applied to Python random, os.environ["PYTHONHASHSEED"], torch.manual_seed, and torch.cuda.manual_seed_all. CuDNN is run in deterministic mode with benchmarking disabled. The 100-sample robustness subset is selected with random.sample(both_correct, 100) after this seed, ensuring identical subsets across runs given the same inference CSV. The complete pipeline can be executed via a single command: bash run_pipeline.sh All phases auto-commit their outputs to version control. Hardware: NVIDIA RTX 5090 (32 GiB VRAM), single-GPU. Software: PyTorch 2.x, transformers 4.x, bitsandbytes, datasets (streaming mode).

Instant research discovery