Paper deep dive
GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinations
Boyuan Chen, Minghao Shao, Siddharth Garg, Ramesh Karri, Muhammad Shafique
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/22/2026, 6:19:42 AM
Summary
GroundCount is a framework that mitigates counting hallucinations in Vision-Language Models (VLMs) by augmenting them with explicit spatial grounding from CNN-based object detection models (ODMs) like YOLO. The approach uses three strategies: prompt-based augmentation, feature-level fusion, and a hybrid of both, achieving up to 81.3% accuracy and reducing inference time by up to 23% in stronger models.
Entities (5)
Relation Signals (3)
GroundCount â augments â Vision-Language Models
confidence 100% ¡ We propose GroundCount, a framework that augments VLMs with explicit spatial grounding from ODMs
GroundCount â mitigates â Counting Hallucinations
confidence 95% ¡ GroundCount, a framework that augments VLMs with explicit spatial grounding from ODMs to mitigate counting hallucinations.
YOLOv13x â providesgroundingfor â GroundCount
confidence 95% ¡ For object detection, we utilize YOLOv13x as our primary ODM
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision Language Models (VLMs) exhibit persistent hallucinations in counting tasks, with accuracy substantially lower than other visual reasoning tasks (excluding sentiment). This phenomenon persists even in state-of-the-art reasoning-capable VLMs. Conversely, CNN-based object detection models (ODMs) such as YOLO excel at spatial localization and instance counting with minimal computational overhead. We propose GroundCount, a framework that augments VLMs with explicit spatial grounding from ODMs to mitigate counting hallucinations. In the best case, our prompt-based augmentation strategy achieves 81.3% counting accuracy on the best-performing model (Ovis2.5-2B) - a 6.6pp improvement - while reducing inference time by 22% through elimination of hallucination-driven reasoning loops for stronger models. We conduct comprehensive ablation studies demonstrating that positional encoding is a critical component, being beneficial for stronger models but detrimental for weaker ones. Confidence scores, by contrast, introduce noise for most architectures and their removal improves performance in four of five evaluated models. We further evaluate feature-level fusion architectures, finding that explicit symbolic grounding via structured prompts outperforms implicit feature fusion despite sophisticated cross-attention mechanisms. Our approach yields consistent improvements across four of five evaluated VLM architectures (6.2--7.5pp), with one architecture exhibiting degraded performance due to incompatibility between its iterative reflection mechanisms and structured prompts. These results suggest that counting failures stem from fundamental spatial-semantic integration limitations rather than architecture-specific deficiencies, while highlighting the importance of architectural compatibility in augmentation strategies.
Tags
Links
- Source: https://arxiv.org/abs/2603.10978v1
- Canonical: https://arxiv.org/abs/2603.10978v1
Trouble viewing inline? Open PDF directly â
Full Text
50,418 characters extracted from source content.
Expand or collapse full text
GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinations Boyuan Chen 1,2 , Minghao Shao 1,2 , Siddharth Garg 1 , Ramesh Karri 1 , Muhammad Shafique 2 , 1 Tandon School of Engineering, New York University, NY, USA 2 eBRAIN Lab, Division of Engineering, New York University Abu Dhabi, UAE boyuan.chen, minghao.shao, sg175, rkarri, muhammad.shafique@nyu.edu AbstractâVision Language Models (VLMs) exhibit persistent hallucinations in counting tasks, with accuracy substantially lower than other visual reasoning tasks (excluding sentiment). This phenomenon persists even in state-of-the-art reasoning- capable VLMs. Conversely, CNN-based object detection models (ODMs) such as YOLO excel at spatial localization and instance counting with minimal computational overhead. We propose GroundCount, a framework that augments VLMs with explicit spatial grounding from ODMs to mitigate counting hallucina- tions. In the best case, our prompt-based augmentation strategy achieves 81.3% counting accuracy on the best-performing model (Ovis2.5-2B) - a 6.6p improvement - while reducing inference time by 22% through elimination of hallucination-driven reason- ing loops for stronger models. We conduct comprehensive abla- tion studies demonstrating that positional encoding is a critical component, being beneficial for stronger models but detrimental for weaker ones. Confidence scores, by contrast, introduce noise for most architectures and their removal improves performance in four of five evaluated models. We further evaluate feature-level fusion architectures, finding that explicit symbolic grounding via structured prompts outperforms implicit feature fusion despite sophisticated cross-attention mechanisms. Our approach yields consistent improvements across four of five evaluated VLM architectures (6.2â7.5p), with one architecture exhibiting de- graded performance due to incompatibility between its iterative reflection mechanisms and structured prompts. These results suggest that counting failures stem from fundamental spatial- semantic integration limitations rather than architecture-specific deficiencies, while highlighting the importance of architectural compatibility in augmentation strategies. I. INTRODUCTION Vision-Language Models (VLMs) have demonstrated re- markable progress across a wide range of multimodal tasks [1], benefiting from large-scale training and fine-tuning on massive image-text corpora. This joint learning paradigm significantly enhances both contextual reasoning and instruction-following abilities [2]. Despite such advances, VLMs exhibit halluci- nations by generating object attributes, entities, or spatial relations inconsistent with the visual input [3], [4]. A growing body of work attributes this issue to an imbal- anced cross-modal attention mechanism: the language decoder often over-attends to textual priors while under-utilizing visual tokens [5]â[7]. Consequently, several methods have been pro- posed to mitigate hallucination, either through decoding-level adjustments [8]â[10] or training-level regularization [11], [12]. Motivated by layer-level analysis showing that hallucination primarily occurs when later layers deviate from correct logits, another direction proposes steering strategies [13]â[15] that preserve the correct momentum from earlier layers. These approaches have yielded measurable improvements on popular VLMs, including LLaVA-v1.5 [16], LLaVA-v1.6 [2], Instruct- BLIP [17], MiniGPT-2 [18], and mPLUG-Owl2 [19]. Recent work jointly trains vision encoders and language decoders on diverse multimodal datasets [20], and incorpo- rates reinforcement learning to enhance reasoning capabilities [21]â[23]. These efforts have led to significantly stronger multimodal representations and improved factual alignment. However, even state-of-the-art (sota) VLMs continue to exhibit systematic hallucination in counting tasks. As we demonstrate in Section I and Table I, counting consistently remains the lowest-accuracy task across all evaluated VLMs (excluding sentiment, which is relatively subjective), with accuracies ranging from 64.0% to 74.7%. This is substantially lower than other visual reasoning tasks such as object recognition (70.3% to 89.6%) and attribute identification (83.2% to 86.5%). Re- cent research also supports this finding [24], [25]. The persistence of counting hallucinations in reasoning- capable VLMs presents a unique challenge. Unlike earlier VLMs where attention steering and decoding adjustments proved effective, modern reasoning VLMs already incorporate reflection mechanisms [20], [22], [23] that direct attention to visual modalities. Furthermore, the iterative reasoning process makes it difficult to apply layer-level vector steering, as there is no single âcorrectâ token to steer toward during multi-step reasoning. This suggests that existing hallucination mitigation strategies, while effective for simpler VLMs, are insufficient for addressing systematic failures in compositional tasks like counting. On the other hand, counting the number of occurrences of an object is nearly trivial for object detection models (ODMs), such as YOLO [26] and DETR [27]. These models achieve very high accuracy in object detection while requiring minimal inference time compared to auto-regressive VLMs. Moreover, ODMs provide structured outputs including bounding boxes, confidence scores, and class labels. This information explicitly addresses the spatial and compositional reasoning needed for arXiv:2603.10978v1 [cs.CV] 11 Mar 2026 accurate counting. This observation suggests a complemen- tary approach: rather than attempting to fix VLM attention mechanisms or reasoning processes, we can augment VLMs with explicit grounding information from specialized object detection models. Our contributions are as follows: ⢠We provide a comprehensive analysis of sota VLMs in counting tasks, demonstrating that counting remains sys- tematically the lowest-accuracy task across all evaluated models despite advances in reasoning capabilities. ⢠We introduce GroundCount, an object-detection-driven augmentation pipeline that increases VLM counting accu- racy by 6.2 to 7.5p across four of five evaluated models (up to 81.3% on the PhD benchmark) with negligible memory overhead and reduced inference time for stronger models (Ovis2.5-2B: -22%, Molmo2-4B: -23%). ⢠We conduct three groups of ablation studies analyzing the impact of different ODM outputs (confidence scores, positional encoding, detection thresholds) on VLM per- formance, providing insights into why grounding infor- mation reduces hallucination. ⢠We evaluate a fusion architecture that combines VLMs with ODMs at the feature level through fine-tuning, ana- lyzing the trade-offs between prompt-based augmentation and architectural integration. I. RELATED WORKS VLM Hallucination in Counting Tasks: Vision-Language Models exhibit persistent hallucinations in counting tasks de- spite advances in reasoning capabilities. One major hypothesis attributes this to vision transformers (ViTs) having difficulties noticing fine-grained image details [28], [29], as ViTs process images through global patch attention rather than the local receptive fields that facilitate spatial instance discrimination [28], [30]. Recent studies [24], [31] systematically demonstrate that counting remains a fundamental challenge for VLMs, with accuracy substantially lower than other visual reasoning tasks. The autoregressive nature of VLM decoding may also contribute to counting errors, as the model must maintain accurate object tallies across many generation steps without explicit grounding in visual features. VLM Hallucination Mitigation Methods: Existing ap- proaches to mitigate VLM hallucinations operate at multiple levels. Decoding-level methods adjust the generation process through contrastive decoding [12], layer contrasting [7], or attention-guided decoding [9]. Training-level approaches in- corporate regularization [11] or specialized fine-tuning ob- jectives. More recent steering strategies [14], [15] preserve correct activations from earlier layers to prevent hallucina- tion drift. However, these methods primarily target general hallucinations and show limited effectiveness on systematic compositional failures like counting, particularly in reasoning- capable VLMs that already employ reflection mechanisms. Combining CNN with Transformers: The complementary strengths of CNNs and transformers have motivated hybrid architectures. DETR [27] pioneered end-to-end object detec- tion by combining CNN backbones with transformer decoders, demonstrating how structured CNN features can enhance transformer reasoning. More recent work explores various fusion strategies for integrating convolutional inductive biases with transformer flexibility [32], [33]. Our fusion architecture extends this paradigm to multimodal settings by grounding VLM patch tokens with CNN-based object detection features. Fusion Training and Cross-Modal Integration: Training multimodal fusion architectures requires careful alignment of features from different modalities and architectural paradigms [34], [35]. Cross-attention mechanisms have proven effective for selectively integrating information across modalities [27], [36], [37]. Feature-wise Linear Modulation (FiLM) [38] pro- vides adaptive conditioning mechanisms that allow one modal- ity to modulate anotherâs representations. Recent work on vision-language pre-training emphasizes joint optimization of vision encoders and language decoders [20], though these typically focus on transformer-only architectures. Our fusion approach specifically targets integrating CNN-based object detection with transformer-based VLMs, requiring novel archi- tectural components to bridge the representational gap between local CNN features and global ViT patch embeddings. Object Detection for Visual Reasoning: Object detection models provide structured, localized visual information that can ground higher-level reasoning. YOLO [39] and its vari- ants achieve real-time detection with high accuracy through efficient CNN architectures. These models excel at spatial localization and instance counting, which are precisely the capabilities where VLMs struggle. Prior work has explored using detected objects as input to VQA systems [40], [41], but typically through learned feature fusion during training rather than explicit textual grounding [42], [43]. Our prompt- TABLE I: Correctness rates (%) of different VLMs on the PhD benchmark using greedy decoding, with explicit thinking mode enabled when available (i.e., the output begins with a prior reasoning segment enclosed by special tokens such as "<think>...</think>"). The PhD benchmark consists of questions from five task types listed in the Task column on the left. For each subset (excluding sentiment), the task with the lowest correctness rate is highlighted with a light-red background. Results across all four evaluation settings are shown; baseline corresponds to standard VQA without adversarial context. VLMMolmo2-4BOvis2.5-2BR-4BQwen3-VL-2BInternVL3.5-1B Task base seciccccsbase seciccccsbase seciccccsbase seciccccsbase seciccccs object 89.6 79.2 65.9 89.288.5 85.1 84.0 84.687.9 85.3 82.1 90.487.7 84.3 83.183.170.3 68.3 67.9 82.0 attribute 85.244.715.1 93.986.5 76.8 75.3 89.286.5 75.5 64.5 92.786.5 77.8 74.2 88.183.2 75.2 73.680.9 positional 80.8 66.4 47.3 89.180.9 73.0 65.3 88.680.6 71.5 54.5 94.178.3 71.3 63.6 86.870.2 61.3 49.2 85.9 counting 67.4 62.9 33.187.974.767.652.079.873.769.444.287.969.463.553.383.164.054.041.3 85.5 sentiment 67.3 52.9 32.0 93.668.4 59.6 59.2 94.968.1 57.9 50.0 96.268.1 50.9 50.3 91.066.2 57.7 59.7 83.3 based approach (Plan A) uniquely leverages ODM outputs as interpretable, structured prompts that augment VLMs without architectural modification, while Plans B and C extend this with feature-level integration for deeper cross-modal ground- ing. I. ANALYSIS - HALLUCINATION IN COUNTING TASKS We first validate that state-of-the-art VLMs systematically exhibit counting hallucinations through controlled experi- ments. We employ consistent benchmark and model selections across all evaluations in this work. A. VQA Benchmark We employ the PhD benchmark [44] to evaluate vision- language model capabilities. The benchmark comprises 33,688 visual question-answer (VQA) pairs across 16,844 images with binary (yes/no) ground-truth labels. Questions are categorized into five task types: attribute, counting, object, positional, and sentiment. To assess robustness against misleading informa- tion, the benchmark provides three challenge variants: (1) sec (specious context) with plausible but potentially misleading descriptions, (2) icc (incorrect context) where VLMs receive explicitly incorrect textual descriptions, and (3) ccs (contra- dictory common sense) featuring 1,506 VQA pairs on 753 AI-generated images that violate common-sense expectations. Compared to earlier benchmarks such as POPE [45] and CHAIR [46], PhD provides larger sample sizes and more granular task categorization. B. Selection of VLM and ODM We evaluate five state-of-the-art open-source VLMs with parameters not exceeding 4B: R-4B [21], Ovis2.5-2B [23], Qwen3-VL-2B-Thinking (abbreviated as Qwen3-VL-2B for the rest of the paper) [47], InternVL3.5-1B [22], and Molmo2- 4B [48]. This selection spans diverse architectural designs, with several incorporating advanced reasoning mechanisms such as reinforcement learning (R-4B) and iterative reflection (InternVL3.5). All models are loaded in float32 precision. We allocate generous computational budgets of 1,024 tokens for both output generation and thinking steps, using greedy decoding to ensure reproducibility. For object detection, we utilize YOLOv13x [26] as our primary ODM, selected for its state-of-the-art accuracy and minimal inference latency. C. Results and Discussion Table I presents the comprehensive evaluation results. Our findings clearly demonstrate: counting consistently remains the lowest-accuracy task in standard evaluation, with attribute identification often showing steeper degradation under adversarial contexts in higher-capacity models (other than sentiment, which is relatively subjective). In the base evaluation setting, counting accuracy ranges from 64.0% to 74.7% (mean: 69.8%), representing a sub- stantial gap compared to other visual reasoning tasks: object recognition achieves 70.3%â89.6% (mean: 84.8%), attribute identification reaches 83.2%â86.5% (mean: 85.6%), and po- sitional reasoning attains 70.2%â80.9% (mean: 78.2%). This Counting Task Input Imageí í LanguagePromptí Q User:Howmanypeopleare thereinthisimage? VLM Vision Encoder Image Tokens CLS Patch 1 Patch 2 ... Patch m CNN-Based ODM Let ODM detect the image first. Process with positional info and append in the prompt. Gr F U S E *FUSION NETWORK* Language Decoder Input ... new Patch 1 new Patch 2 CLS new Patch m Te x t 1 Te x t n ... Output To ke n1 Output To ke n2 ... Language Responseí q Language Transformerí í EncoderHead Vector Extraction ... Extracts local and positional information for a set of objects. Pooled to a long vector with the first dimension of 1. Different VLM Vector by Patch ViT tokens query CNN features as key/value pairs. FiLM applies learned parameters from CNN to modulate the ViT features. Branch A: MLP FiLM Layer Branch B: Gated Linear + GELU Cross Gating Linear + GELU DepthwiseConv1d Linear + GELU Cross-Attention Weighted Combination Information Bottleneck Adaptive Residual Layer Norm Compresses features to force using both modalities Dual-branch fusion in complex non-linear feature transformations (Branch A) and adaptive modality selection (Branch B), minimizing the risks of fusion strategy failing. Weights are learned at combination. Residual layer for both ViT and CNN vectors to avoid gradient vanishing. Each new patch token contains both local features and filtered global features. GroundCount B â Fusion w/ Finetuning Fusing ODMâs latent vector at the last layer with each image patch token from VLM. Then, use a fusion network to map the fused vector back to the original dimensions. The fusion network is light-weight and trainable. Must train fusion network; optionally train VLM, ODM, Language Transformer. Global CNN Vector VLM Language Transformer Available Models GroundCount A â Prompt Augmentation GroundCount C â Prompt + Fusion Augmentation Combining A & B. The training set also incorporates ODM info. CNN-Based ODM person1 middle-left; person2 ... User:Howmanypeopleare thereinthisimage? # ODM detections: person1 middle left; ... VLM Language Transformer VLM Vision Encoder Pass the augmented prompt to VLM language transformer. Fig. 1: Structural overview of three strategies in our proposed fusion framework - A, B and C. In GroundCount A, we run inference with ODM on the image, and then include its output in the VLM prompt. In GroundCount B, we fuse the VLM and ODM on the visual patch latent vector using a light-weight network. To ensure correct information delivery, we finetune the network with our original counting task mutation from COCO. The fusion block is required to be trained; Other modules - VLM, ODM, and language transformer - are optionally frozen. GroundCount C incorporates both plans by including both prompt-level information and architectural-level integration. The training data also includes ODM detections in the textual input. represents an average accuracy deficit of 15.0p and 15.8p compared to object recognition and attribute identification, respectively. Performance degradation under misleading contexts further reveals the fragility of counting abilities. Under sec and icc conditions, counting accuracy drops by 4.3â10.0p and 16.1â 34.3p, respectively, the steepest degradation among all task categories. This suggests that VLM counting mechanisms are particularly susceptible to distraction from textual priors, consistent with hypotheses about imbalanced cross-modal at- tention [5]. Interestingly, the ccs subset reveals a reversal: VLMs achieve substantially higher counting accuracy (79.8%â87.9%) on AI-generated images. We hypothesize this occurs due to cleaner spatial layouts and reduced perceptual ambiguity in synthetic images, though this improved performance does not reflect real-world applicability. Model-Specific Patterns. Examining individual models re- veals that counting failures are not simply correlated with parameter count or training paradigm. Molmo2-4B achieves only 67.4% baseline accuracy despite having the largest pa- rameter count among evaluated models (tied with R-4B at 4B), while the smaller Ovis2.5-2B reaches 74.7%. However, Molmo2 demonstrates the second smallest accuracy drop under misleading contexts (4.5p for sec, second to 4.3p for sec of R-4B), suggesting robust attention mechanisms. InternVL3.5-1B exhibits the steepest degradation under sec (10.0p) in the counting task, indicating that iterative reflection mechanisms require sufficient capacity to filter spurious textual information. These findings have critical implications. Despite incorpo- rating advanced reasoning capabilities, modern VLMs con- tinue to exhibit systematic counting failures. This suggests the problem lies not in reasoning depth but in fundamental spatial- semantic integration. Moreover, models explicitly addressing cross-modal attention balance still achieve only 64.0% to 74.7% counting accuracy, indicating that attention rebalancing strategies have limited impact. Layer-level steering strate- gies [14] prove ineffective for iterative reasoning processes where no single âcorrectâ token exists to steer toward during multi-step counting. These observations motivate our approach: rather than re- fining attention mechanisms within existing VLM paradigms, we augment VLMs with explicit grounding from specialized object detection models that excel precisely where VLMs fail. IV. METHODOLOGY - AUGMENTING VLMS WITH OBJECT DETECTION MODEL We propose GroundCount, a framework that augments VLMs with explicit spatial grounding from object detection models to mitigate counting hallucinations. Our approach operates on the observation that CNN-based ODMs excel at spatial localization and instance counting, precisely where VLMs exhibit systematic failures. We present three implemen- tation strategies with different computational trade-offs. A. GroundCount A: Prompt-Based ODM Augmentation Positional encoding based on each detectionâs center upper-leftupper-centerupper-right middle-leftmiddle-centermiddle-right lower-leftlower-centerlower-right ### Result from Object Detection Model: skateboard 1 lower-left: 0.80 person 1 lower-left: 0.92 person 2 upper-center: 0.93 person 3 upper-center: 0.68 person 4 upper-center: 0.81 person 5 upper-right: 0.93 ODM Detections with bounding boxes and confidence values *Object in sequence of the following priorities: 1. left to right 2. lower to upper Fig. 2: Our pipeline of converting ODM outputs to descriptive text. The image is #000000000077.jpg from COCO-train2017, showing 5 young people skateboarding. The bounding boxes (bbox) come from YOLOv13xâs detection: yellow ones are person objects; orange ones are skateboard objects. The location of each object is determined by the center of their corresponding bbox. Two skateboard objects were not included due to low confidence. The most straightforward approach augments VLM prompts with structured textual descriptions of ODM detections. As illustrated in Figure 2, we first pass the input image through YOLOv13x to obtain bounding boxes, class labels, and confi- dence scores, which are then converted into natural language prompts. Spatial Encoding: We discretize image space into a 3Ă3 grid (upper/middle/lower Ă left/center/right) and assign each detection to a grid cell based on its bounding box center coordinates. This coarse spatial encoding preserves relative positioning information while remaining interpretable for lan- guage models. Object Sequencing: Detections are ordered by (1) horizon- tal position (left-to-right), then (2) vertical position (lower-to- upper). This consistent ordering enables VLMs to maintain spatial coherence when processing multiple instances of the same object class. Prompt Construction: For each detection, we generate a string in the format: "[class] [index] [position]: [confidence]", where the index distinguishes multiple instances of the same class. The complete ODM output is appended to the original user prompt as structured context (see Figure 2). This approach requires no architectural modifications or training, enabling plug-and-play augmentation across differ- ent VLM families. The computational overhead is minimal: YOLOv13x inference (âź0.1s) is negligible compared to VLM auto-regressive decoding (7-40s depending on model architec- ture and reasoning complexity), and sometimes reduces total inference time by preventing hallucination-induced reasoning loops, particularly for stronger models (Figure 3). B. GroundCount B: Feature-Level Fusion Architecture While prompt augmentation proves effective, it relies on the VLMâs ability to correctly interpret textual descriptions of spatial information. To enable direct feature-level grounding, we propose a fusion architecture that integrates CNN features from the ODM with ViT patch tokens from the VLMâs vision encoder. Architecture Overview: As shown in Figure 1, our fusion network operates between the VLMâs vision encoder and lan- guage decoder. For each ViT patch embedding p i âR d vit , we extract the corresponding spatial region from the ODMâs final convolutional layer, yielding a local CNN feature c i âR d cnn . We additionally extract a global CNN feature vector gâR d cnn via adaptive pooling. Dual-Branch Fusion: The fusion network employs two parallel branches to integrate multimodal features: Branch A (Feature Transformation): Applies Feature-wise Linear Modulation (FiLM) [38] to enable CNN features to adaptively modulate ViT representations: h A i = FiLM(p i , c i ) =Îł i â p i +β i whereÎł i ,β i = MLP(c i ) are learned affine parameters. Branch B (Selective Attention): Uses cross-attention to allow ViT patches to selectively query relevant CNN features: h B i = CrossAttn(p i , [c i , g]) The branches are combined via learned gating: h i = Îą i h A i + (1â Îą i )h B i , where Îą i = Ď(MLP([p i , c i ])). This dual-branch design provides robustness. Branch A enforces strong CNN influence through multiplicative modulation, while Branch B enables adaptive selection. This minimizes risk if one fusion strategy proves suboptimal. Information Bottleneck: Following the dual-branch fusion, we apply dimensionality reduction to enforce multimodal in- formation integration. This bottleneck prevents the model from trivially bypassing fusion by relying solely on one modality. Training Data Preparation: We construct training data from the COCO train2017 dataset [49] using ground-truth object annotations. For each image, we apply the same spatial encoding and sequencing rules used for ODM detections in Plan A. Specifically, each ground-truth bounding box is assigned to a 3Ă3 grid cell based on its center coordinates, and objects are ordered left-to-right, then lower-to-upper. The model is trained to generate structured spa- tial descriptions in the format: "[class] [index] in [position]" for each object instance. For example, given an image with multiple birds, the target output would be: âbird 1 in upper-left; bird 2 in middle-center; bird 3 in lower-rightâ. This format mirrors the ODM detection output structure, enabling the fusion model to learn consistent spatial grounding patterns that align with both ground-truth annotations and runtime ODM predictions. Training Strategies: We evaluate four training configura- tions (Table I): ⢠B.1: Train fusion network only (frozen VLM, frozen ODM) ⢠B.2: Train fusion network + fine-tune VLM and ODM ⢠B.3: Train fusion network + fine-tune language decoder only ⢠B.4: Train fusion network + fine-tune VLM, ODM, and language decoder C. GroundCount C: Combined Prompt and Fusion Our final approach combines both strategies: structural fusion (Plan B.4) with prompt augmentation (Plan A). This hy- brid design provides complementary benefits. Feature-level fu- sion enables implicit spatial grounding, while textual prompts offer explicit, interpretable object counts. The combined ap- proach aims to leverage both implicit feature integration and explicit symbolic reasoning. D. Implementation Details All experiments use YOLOv13x as the object detection model with default confidence threshold 0.5. For Plan A, we filter detections below this threshold before prompt con- struction. VLMs are evaluated with greedy decoding, 1024- token output budget, and float32 precision. Fusion network training uses AdamW optimizer (β 1 = 0.9,β 2 = 0.999), learning rate 2Ă 10 â5 with cosine annealing, batch size 1, and training with a maximum training budget of 40k steps; best checkpoints selected per Table I. All experiments are conducted on NVIDIA A100 GPUs with 80GB memory. V. RESULTS & ANALYSIS A. Main Results: Comparing GroundCount Strategies Table I presents a comprehensive comparison of our pro- posed GroundCount strategies on the PhD benchmarkâs count- ing subset, evaluated on Ovis2.5-2B. Prompt Augmentation (Plan A) achieves the highest counting accuracy at 81.3%, representing a substantial 6.6p improvement over the 74.7% baseline. Remarkably, Plan A also reduces average inference time from 10.0s to 7.8s (a 22% Baseline - VLM inference Q: Are there 2 bowls in the image? Augmented Prompt: ... ### Detection from Object Detection Model: ... bowl 1 lower-left: 0.95 bowl 2 middle-left: 0.82 ... COCO_val2014_000000189241.jpg, Qwen3 Label: Yes (There are 2 bowls on the left corner. One is partially covered by the other one closer to us.) VLM Output from Augmented Prompt: So, let's check the image for bowls. First, look at the objects. There's a white bowl on the left, labeled ""bowl 1 lower-left: 0.95"". Then another bowl, ""bowl 2 middle-left: 0.82"". So that's two bowls. The question is ""Are there two bowls in the image?"" So we need to count. Let's confirm: the white bowl and the other bowl. Yes, two bowls. So the answer should be YES. </think> see bowl 1 lower-left (white bowl) and bowl 2 middle-left (another bowl). There are two bowls in the image. This matches the question asking for two bowls. So the answer is YES. final answer is: YES VLM Output from Raw Question Prompt: So, let's look at the image. First, identify the bowls. There's a white bowl on the left, and another white bowl? Wait, the image shows a wooden table. Let's check: there's a white bowl with a spoon, and another white bowl? Wait, no, the description says ""two bowls"". Let's count. The image has a white bowl (the one with the spoon) and maybe another? Wait, looking at the image: there's a white bowl on the left, and is there another? Wait, the left side has a white bowl, and maybe another? Wait, no, let's check again. The image shows a wooden table with various items. There's a white bowl (the one with the spoon), and is there another bowl? Wait, the problem is asking for two bowls. Let's count. The white bowl is one. Is there another bowl? [...] Endlessly re-checks the image, without arriving to a final deterministic conclusion. Distracted by unrelated objects. Realizes potential missing object from textual prompt, yet fails to find from image. ODM Augmentation Refers to ODM detections with position and confidence value for each object. Double-checks the objects by looking at the image. ODM Augmentation improves accuracy while also saving inference time by reducing tokens wasted in hallucination. Fig. 3: Illustration with real evaluation example for ODM prompt augmentation in GroundCount A. The image is #000000189241.jpg from COCO-val2014, which is used for a counting question in our selected benchmark, PhD. The tested VLM is Qwen3-VL-2B-Thinking. The question asks for a correctness judgment on the number of bowls in the image. There are indeed 2 bowls in the image, with one bowl partially covered by another. The baseline VLM, whose output is showcased in the left frame, fails to find the second bowl and re-thinks iteratively, exhibiting behaviors of hallucination. On the other hand, with information from the object detection model (ODM) appended in the prompt, the VLM successfully finds the second bowl with a double-check. speedup). This counterintuitive result stems from VLMs gen- erating fewer tokens when provided with explicit grounding information (Figure 3). The negligible computational overhead of YOLOv13x inference (âź0.1s) is more than offset by reduced VLM generation, making Plan A both more accurate and faster. This efficiency gain is particularly notable in complex reasoning scenarios where VLMs typically engage in extended chain-of-thought generation. Fusion Architectures (Plan B) demonstrate varied out- comes depending on training configuration. Training only the fusion network (B.1) yields minimal improvement (75.2%), suggesting that frozen VLM and ODM representations re- main misaligned. Conversely, jointly fine-tuning the fusion network, VLM, and ODM (B.2) degrades performance to 72.7%, indicating potential catastrophic forgetting. The most effective fusion strategy (B.4) fine-tunes the fusion network, VLM, ODM, and language decoder together, achieving 78.0% accuracy with 7.7s inference time. While this represents a meaningful 3.3p gain over baseline, it falls short of Plan Aâs prompt-based approach. Combined Strategy (Plan C) integrates prompt augmen- tation with fusion architecture (B.4), yielding 78.2% accuracy and 4.8s inference (the fastest among approaches that exceed baseline accuracy). The reduced inference time stems from the fine-tuned language decoder generating more concise re- sponses. However, the accuracy remains below Plan A alone, suggesting that the fusion network may introduce noise that partially counteracts the benefits of explicit textual grounding. These findings reveal a critical insight: for counting tasks, explicit symbolic grounding via prompts outperforms implicit feature-level fusion. This aligns with recent work emphasizing the importance of interpretable intermediate representations in multimodal reasoning [37]. The fusion architectureâs underper- formance likely stems from the fundamental representational gap between CNN spatial features and ViT global patch embeddings. B. Ablation Study: Decomposing ODM Information Figure 4 presents ablation results across five VLM fami- lies, systematically removing different components of ODM information. Confidence Scores: Removing confidence values (NoCon- fidence) produces mixed results. While Ovis2.5-2B shows a modest 0.7p accuracy drop, four models (Molmo2-4B, R- 4B, Qwen3-VL-2B, InternVL3.5-1B) actually improve by 0.1 to 4.8p without confidence scores, suggesting that confidence values may introduce noise for certain VLM architectures. Positional Encoding: The impact of spatial position in- formation reveals a critical architectural divide. Remov- ing positional encoding (NoPosition) causes degradation in the two strongest baseline models: Molmo2-4B (1.8p loss) and Ovis2.5-2B (0.6p loss). However, the three weaker modelsâR-4B, Qwen3-VL-2B, and InternVL3.5- 1Bâparadoxically improve by 0.7 to 4.4p without positional information, suggesting these models struggle to correctly interpret spatial encodings and perform better with position- agnostic object lists. Baseline Full ODM Info No- Confidence No- Position Low- thresholdPointing Molmo2-4B Ovis2.5-2B R-4B Qwen3-VL-2B InternVL3.5-1B ODM-only (ref) 67.4 (7.9s) 74.9 (6.1s) 75.0 (5.8s) 73.1 (4.3s) 58.0 (12.0s) 71.1 (4.4s) 74.7 (10.0s) 81.3 (7.8s) 80.6 (10.4s) 80.7 (15.3s) 66.3 (27.5s) NA 73.7 (19.2s) 79.9 (23.1s) 80.4 (21.1s) 80.6 (22.1s) 68.3 (66.8s) NA 69.4 (39.6s) 75.6 (47.2s) 76.0 (43.9s) 77.7 (41.0s) 60.6 (69.7s) NA 64.0 (44.5s) 62.5 (82.4s) 67.3 (56.6s) 66.9 (76.6s) 61.2 (99.5s) NA 72.8 (0.1s) 60 65 70 75 80 Accuracy (%) Fig. 4: Results of GroundCount A across all model families and including ablation studies. Each block contains the accuracy and average inference time for that group of experiment on the PhD counting subset. The bottom row marks the result of running the object detection model only. The right-most column records the special pointing mode for Molmo2 model only. Detection Threshold: Lowering the confidence threshold from 0.5 to 0.3 (LowThreshold) uniformly harms performance across all models, with accuracy decreasing by 1.3 to 16.9p relative to the Full ODM augmentation, and inference time increasing by 4.1 to 55.0s relative to the no-augmentation baseline. The degradation is particularly severe for stronger models (Molmo2-4B: 16.9p, Ovis2.5-2B: 15.0p). The dra- matic slowdown occurs because VLMs must process sub- stantially more false-positive detections. This establishes that detection precision substantially outweighs recall for counting augmentation. Model-Specific Patterns: Molmo2-4B uniquely supports a pointing mode where bounding boxes are overlaid on the input image. Interestingly, this visual grounding (71.1%) underper- forms textual ODM augmentation (74.9%), suggesting that Molmoâs architecture processes structured textual descriptions more effectively than visual annotations. C. ODM-Only Baseline Running YOLOv13x alone achieves 72.8% accuracy in 0.1s. This establishes an important reference point: the ODM al- ready captures most counting information, yet VLMs struggle to extract it from raw images. When properly grounded with ODM outputs, the strongest VLMs (Plan A: Ovis2.5-2B at 81.3%) exceed ODM-only performance by 8.5p, demonstrat- ing that VLMs contribute valuable contextual reasoning when supplied with explicit spatial priors. D. Cross-Model Consistency and Architectural Variations The benefits of ODM augmentation show significant vari- ation across VLM architectures. Four of five evaluated mod- els exhibit substantial accuracy improvements with Plan A: Molmo2-4B (7.5p), Ovis2.5-2B (6.6p), R-4B (6.2p), and Qwen3-VL-2B (6.2p). However, InternVL3.5-1B presents a notable exception, showing degraded performance (from 64.0% baseline to 62.5% with full ODM augmentation). This reveals that InternVL3.5âs architecture appears incompatible with explicit textual ground- ing, potentially due to its iterative reflection mechanisms being disrupted by structured prompts. The modelâs improvements in NoPosition (+4.4p) and NoConfidence (+4.8p) ablations suggest it performs better with minimal explicit guidance. This divergence suggests that while counting hallucina- tions generally stem from fundamental limitations in spatial- semantic integration, the effectiveness of explicit grounding depends critically on architectural compatibility. E. Implications and Future Directions Our results establish that counting hallucinations in VLMs reflect deeper challenges in spatial-semantic integration. The success of prompt-based augmentation over sophisticated fu- sion architectures suggests that current VLMs are better equipped to process structured symbolic representations than to learn implicit cross-modal alignments. However, the ar- chitectural variationâparticularly InternVL3.5âs negative re- sponseâindicates that augmentation strategies must be tai- lored to specific VLM designs. Several limitations warrant future investigation. First, our fusion experiments used only 40k training steps on binary classification tasks. More extensive pre-training may unlock greater potential. Second, we focused on CNN-based detec- tors (YOLO). Exploring transformer-based ODMs like DETR or Grounding DINO could reduce the representational gap. Third, understanding why certain architectures reject explicit grounding could inform better augmentation strategies. Finally, efficient caching for repeated queries would enhance practical applicability. VI. CONCLUSION We present GroundCount, a framework that mitigates count- ing hallucinations in VLMs through explicit grounding from object detection models. Our evaluation across five state-of- the-art VLMs shows counting remains the lowest-accuracy task (64.0 to 74.7%, excluding sentiment). Our prompt-based augmentation (Plan A) achieves 81.3% counting accuracyâa TABLE I: Performance comparison of different GroundCount schemes - prompt augmentation (Plan A), architectural fusion with training (Plan B), and both (Plan C). Plan A gains the highest accuracy; Plan B.4 and Plan C have the lowest inference time among schemes that exceed baseline accuracy, due to training on language transformer. Training SchemeAcc(%) Time(s) Best Steps Baseline74.710.0NA Plan A - ODM Prompt Augmentation81.37.8NA Plan B.1 - Fusion only75.217.810k Plan B.2 - Fusion + VLM, ODM72.714.65k Plan B.3 - Fusion + LangTrans71.44.35k Plan B.4 - Fusion + VLM, ODM 78.07.75k + LangTrans Plan C - Plan A + Plan B.478.24.85k 6.6p improvement over baselineâwhile reducing inference time by 22%. Ablation studies reveal positional encoding is beneficial for stronger models (0.6 to 1.8p), while detec- tion precision substantially outweighs recall (1.3 to 16.9p degradation at lower thresholds). Critically, we find that explicit symbolic grounding via structured prompts outper- forms feature-level fusion on Ovis2.5-2B, while prompt-based augmentation achieves consistent gains across four of five architectures. This indicates counting failures stem from fun- damental spatial-semantic integration limitations rather than architecture-specific deficiencies. The gap between ODM-only (72.8%) and augmented VLM performance (81.3%) demon- strates that VLMs contribute valuable contextual reasoning when properly grounded. IMPACT STATEMENT This work improves VLM reliability in counting tasks, enhancing trustworthiness for accessibility tools, inventory systems, and educational technologies. We acknowledge that improved counting accuracy could amplify privacy concerns if applied to surveillance without appropriate safeguards. We encourage responsible deployment with proper consent and adherence to privacy regulations. Reducing hallucinations con- tributes to developing more reliable multimodal AI systems. ACKNOWLEDGMENTS This work has been supported in parts by the NYUAD Cen- ter for Cyber Security (CCS), funded by Tamkeen under the NYUAD Research Institute Award G1104. Experiments are performed with NYUAD Jubail High Performance Computing (HPC). REFERENCES [1] M. Shao et al., âSurvey of different large language model architectures: Trends, benchmarks, and challenges,â IEEE Access, 2024. [2] H. Liu et al., âLlava-next: Improved reasoning, ocr, and world knowl- edge,â January 2024. [3] L. Zhao et al., âMitigating object hallucination in large vision-language models via image-grounded guidance,â in Forty-second International Conference on Machine Learning, 2025. [4] X. Zou et al., âLook twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language mod- els,â The Forty-second International Conference on Machine Learning (ICML), 2025. [5] S. Leng et al., âMitigating object hallucinations in large vision-language models through visual contrastive decoding,â in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024, p. 13 872â13 882. [6] X. Wang et al., âMitigating hallucinations in large vision-language mod- els with instruction contrastive decoding,â in Findings of the Association for Computational Linguistics ACL 2024, 2024, p. 15 840â15 853. [7] Y.-S. Chuang et al., âDola: Decoding by contrasting layers improves factuality in large language models,â in The Twelfth International Conference on Learning Representations, 2024. [8] X. Xu et al., âMitigating hallucinations in multi-modal large language models via image token attention-guided decoding,â in Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the As- sociation for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Albuquerque, New Mexico: Association for Computational Linguistics, Apr. 2025, p. 1571â1590. [9] S. Yin et al., âClearsight: Visual signal enhancement for object halluci- nation mitigation in multimodal llms,â in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, p. 16 520â16 530. [10] Z. Jiang et al., âDevils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens,â in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2025. [11] Y. Wu et al., âAntidote: A unified framework for mitigating lvlm hallucinations in counterfactual presupposition and object perception,â in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2025, p. 14 646â14 656. [12] Z. Yang et al., âMitigating hallucinations in large vision-language models via dpo: On-policy data hold the key,â in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, p. 10 610â10 620. [13] Z. Li et al., âThe hidden life of tokens: Reducing hallucination of large vision-language models via visual information steering,â in Forty-second International Conference on Machine Learning, 2025. [14] K. Wang et al., âDAMO: Decoding by accumulating activations mo- mentum for mitigating hallucinations in vision-language models,â in The Thirteenth International Conference on Learning Representations, 2025. [15] C. Wang et al., âMLLM can see? dynamic correction decoding for hallucination mitigation,â in The Thirteenth International Conference on Learning Representations, 2025. [16] H. Liu et al., âVisual instruction tuning,â in NeurIPS, 2023. [17] W. Dai et al., âInstructblip: Towards general-purpose vision-language models with instruction tuning,â 2023. [18] J. Chen et al., âMinigpt-v2: large language model as a unified interface for vision-language multi-task learning,â 2023. [19] Q. Ye et al., âmplug-owl2: Revolutionizing multi-modal large language model with modality collaboration,â in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2024. [20] J. Zhu et al., âInternvl3: Exploring advanced training and test-time recipes for open-source multimodal models,â 2025. [Online]. Available: https://arxiv.org/abs/2504.10479 [21] Q. Yang et al., âR-4b: Incentivizing general-purpose auto-thinking capability in mllms via bi-mode annealing and reinforce learning,â 2025. [Online]. Available: https://arxiv.org/abs/2508.21113 [22] W. Wang et al., âInternvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency,â 2025. [Online]. Available: https://arxiv.org/abs/2508.18265 [23] S. Lu et al., âOvis2.5 technical report,â 2025. [Online]. Available: https://arxiv.org/abs/2508.11737 [24] A. Vo et al., âVision language models are biased: Counting legs of an animal is surprisingly hard,â in 2nd AI for Math Workshop @ ICML 2025, 2025. [25] X. Guo et al., âYour vision-language model canât even count to 20: Exposing the failures of vlms in compositional counting,â 2025. [26] M. Lei et al., âYolov13: Real-time object detection with hypergraph- enhanced adaptive visual perception,â 2025. [27] N. Carion et al., âEnd-to-end object detection with transformers,â in Computer Vision â ECCV 2020: 16th European Conference, Glasgow, UK, August 23â28, 2020, Proceedings, Part I.Berlin, Heidelberg: Springer-Verlag, 2020, p. 213â229. [28] W. Park et al., âSECOND: Mitigating perceptual hallucination in vision- language models via selective and contrastive decoding,â in Forty-second International Conference on Machine Learning, 2025. [29] M. Raghu et al., âDo vision transformers see like convolutional neural networks?â in Proceedings of the 35th International Conference on Neural Information Processing Systems, ser. NIPS â21.Red Hook, NY, USA: Curran Associates Inc., 2021. [30] M. Naseer et al., âIntriguing properties of vision transformers,â in Proceedings of the 35th International Conference on Neural Information Processing Systems, ser. NIPS â21.Red Hook, NY, USA: Curran Associates Inc., 2021. [31] S. Sengupta et al., âCan vision-language models count? a synthetic benchmark and analysis of attention-based interventions,â 2025. [32] Z. Liu et al., âA convnet for the 2020s,â in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, p. 11 976â11 986. [33] Z. Dai et al., âCoatnet: marrying convolution and attention for all data sizes,â in Proceedings of the 35th International Conference on Neural Information Processing Systems, ser. NIPS â21. Red Hook, NY, USA: Curran Associates Inc., 2021. [34] J. Lu et al., ViLBERT: pretraining task-agnostic visiolinguistic represen- tations for vision-and-language tasks.Red Hook, NY, USA: Curran Associates Inc., 2019. [35] H.Tanetal.,âLXMERT:Learningcross-modalityencoder representations from transformers,â in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).Hong Kong, China: Association for Computational Linguistics, Nov. 2019, p. 5100â5111. [Online]. Available: https://aclanthology.org/D19-1514/ [36] J.-B. Alayrac et al., âFlamingo: a visual language model for few-shot learning,â in Proceedings of the 36th International Conference on Neural Information Processing Systems, ser. NIPS â22. Curran Associates Inc., 2022. [37] J. Li et al., âBlip-2: bootstrapping language-image pre-training with frozen image encoders and large language models,â in Proceedings of the 40th International Conference on Machine Learning, ser. ICMLâ23. JMLR.org, 2023. [38] E. Perez et al., âFilm: visual reasoning with a general conditioning layer,â in Proceedings of the Thirty-Second AAAI Conference on Ar- tificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, ser. AAAIâ18/IAAIâ18/EAAIâ18. AAAI Press, 2018. [39] J. Redmon et al., âYou only look once: Unified, real-time object detection,â in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016, p. 779â788. [40] D. A. Hudson et al., âGqa: A new dataset for real-world visual reasoning and compositional question answering,â in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, p. 6700â6709. [41] L. H. Li et al., âWhat does BERT with vision look at?â in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.Online: Association for Computational Linguistics, Jul. 2020, p. 5265â5275. [42] P. Anderson et al., âBottom-up and top-down attention for image captioning and visual question answering,â in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, p. 6077â6086. [43] Y.-C. Chen et al., âUniter: Universal image-text representation learning,â in Proceedings of the European Conference on Computer Vision, 2020, p. 104â120. [44] J. Liu et al., âPhd: A chatgpt-prompted visual hallucination evaluation dataset,â in CVPR, 2025. [45] Y. Li et al., âEvaluating object hallucination in large vision-language models,â in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.Singapore: Association for Compu- tational Linguistics, Dec. 2023, p. 292â305. [46] A. Rohrbach et al., âObject hallucination in image captioning,â in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Oct.- Nov. 2018, p. 4035â4045. [47] S. Bai et al., âQwen3-vl technical report,â 2025. [48] C. Clark et al., âMolmo2: Open weights and data for vision-language models with video understanding and grounding,â 2026. [49] T.-Y. Lin et al., âMicrosoft coco: Common objects in context,â in Computer VisionâECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, 2014, p. 740â755.