Paper deep dive
NumerosityVLM: A Cognitively Inspired Benchmark for Interpreting Numerosity Representations in Vision-Language Models
Yiming Fu, Fangjun Li, Xiujin Liu, Ruidong Ma, Hang Yu, Zhichen Lu, Kanwei He, Alessandro Di Nuovo, Angelo Cangelosi, Zhegong Shangguan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/22/2026, 2:15:20 AM
Summary
The paper introduces NumerosityVLM, a cognitively inspired diagnostic benchmark consisting of 10,800 synthetic images designed to evaluate numerosity perception in Vision-Language Models (VLMs). The benchmark orthogonally manipulates object size, spatial arrangement, and numerosity while progressively ablating texture, shape, and color to disentangle visual factors. Evaluating seven open-source VLMs reveals that model architecture is the primary driver of performance variance, with linearly separable numerosity signals emerging in early vision encoder layers, while performance differences across models are largely attributed to the language model component.
Entities (13)
Relation Signals (11)
InternVL3-8B → achievesaccuracy → 77.21%
confidence 95% · InternVL3-8B... Accuracy 77.21%
Qwen3VL-8B-Instruct → achievesaccuracy → 83.97%
confidence 95% · Qwen3VL-8B-Instruct... Accuracy 83.97%
NumerosityVLM → containscondition → Space-Incongruent
confidence 95% · Space-Incongruent. ISA is fixed... while objects are constrained within a convex hull of approximately fixed perimeter.
NumerosityVLM → containscondition → Texture-Ablated
confidence 95% · Texture-Ablated. Textured objects are replaced with simplified shape templates
NumerosityVLM → containscondition → Shape-Ablated
confidence 95% · Shape-Ablated. Objects are further replaced by solid-colored dots
NumerosityVLM → containscondition → Color-Ablated
confidence 95% · Color-Ablated. Colored dots are converted to grayscale
NumerosityVLM → containscondition → Size-Incongruent
confidence 95% · The benchmark orthogonally manipulates object size... Size-Incongruent. TSA is fixed... while ISA varies inversely with N
NumerosityVLM → evaluates → Vision-Language Models
confidence 95% · We introduce a cognitively inspired diagnostic benchmark, NumerosityVLM... Evaluating seven VLMs in a zero-shot setting
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision-language models (VLMs) achieve strong performance on high-level multimodal tasks, yet numerosity perception, a cognitive ability that emerges in human infants before language acquisition, remains poorly understood in current models, as existing counting benchmarks entangle numerosity with correlated visual factors. We introduce a cognitively inspired diagnostic benchmark, NumerosityVLM, comprising 10,800 synthetic images across six controlled conditions. The benchmark orthogonally manipulates object size, spatial arrangement, and numerosity, while progressively ablating texture, shape, and color. Evaluating seven VLMs in a zero-shot setting, multi-factor analysis reveals that model architecture explains the largest proportion of performance variance (partial $\omega^{2}=0.325$), far exceeding visual conditions. Layer-wise probing further shows that linearly separable numerosity signals consistently emerge at early stages of the vision encoder, while performance differences across evaluated models are primarily associated with the language model component. Code and data are publicly available at this https URL, and this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.15425v1
- Canonical: https://arxiv.org/abs/2608.15425v1
Trouble viewing inline? Open PDF directly →
Full Text
35,628 characters extracted from source content.
Expand or collapse full text
NumerosityVLM: A Cognitively Inspired Benchmark for Interpreting Numerosity Representations in Vision–Language Models Yiming Fu 1 , Fangjun Li 1 , Xiujin Liu 2 , Ruidong Ma 5 , Hang Yu 3 , Zhichen Lu 4 , Kanwei He 1 , Alessandro Di Nuovo 5 , Angelo Cangelosi 1 , and Zhegong Shangguan 1⋆ 1 Cognitive Robotics Lab, Department of Computer Science, The University of Manchester, Manchester M13 9PL, UK 2 University of Michigan, MI 48109, USA 3 Tufts University, MA 02155, USA 4 ENSTA, Institut Polytechnique de Paris, Palaiseau 91120, France 5 Sheffield Hallam University, Sheffield S1 1WB, UK Abstract. Vision-language models (VLMs) achieve strong performance on high-level multimodal tasks, yet numerosity perception, a cognitive ability that emerges in human infants before language acquisition, re- mains poorly understood in current models, as existing counting bench- marks entangle numerosity with correlated visual factors. We introduce a cognitively inspired diagnostic benchmark, NumerosityVLM, com- prising 10,800 synthetic images across six controlled conditions. The benchmark orthogonally manipulates object size, spatial arrangement, and numerosity, while progressively ablating texture, shape, and color. Evaluating seven VLMs in a zero-shot setting, multi-factor analysis re- veals that model architecture explains the largest proportion of perfor- mance variance (partial ω 2 = 0.325), far exceeding visual conditions. Layer-wise probing further shows that linearly separable numerosity sig- nals consistently emerge at early stages of the vision encoder, while performance differences across evaluated models are primarily associ- ated with the language model component. Code and data are publicly available at https://github.com/fuy3/NumerosityVLM-Benchmark, and https://huggingface.co/datasets/fuy3/NumerosityVLM. Keywords: Vision–Language Models· Numerosity Perception· Human- Inspired Vision· Cognitive Benchmark· Mechanistic Interpretability 1 Introduction Vision–language models (VLMs) have achieved remarkable performance on mul- timodal tasks such as visual question answering [20], scene understanding [21], and chart interpretation [23], yet whether they also possess fundamental percep- tual abilities remains an open question. Recent studies suggest a gap between ⋆ Corresponding author. Email: zhegong.shangguan@manchester.ac.uk arXiv:2608.15425v1 [cs.CV] 15 Aug 2026 2Y. Fu et al. high-level reasoning and low-level visual perception in VLMs [14], raising con- cerns that strong benchmark performance may not reflect genuine numerical understanding. Numerosity perception, which emerges in human infants before language acquisition [18], provides a natural testbed for evaluating such abili- ties. Existing counting benchmarks, however, suffer from two major limitations: general-purpose benchmarks [5,7] cover only limited numerosity ranges, whereas real-world datasets [24] entangle numerosity with visual factors such as object size, density, and appearance. While cognitively inspired evaluations have been explored [22], they still lack systematic control over visual variables known to influence numerosity judgments. Consequently, current benchmarks cannot re- liably attribute counting failures to deficiencies in visual perception, language decoding, or reliance on shortcut cues. To address these limitations, we introduce NumerosityVLM, a controlled diagnostic benchmark with six experimental conditions and evaluate seven repre- sentative open-source VLMs from four architectural families, enabling systematic investigation of numerosity perception through controlled behavioral evaluation and mechanistic analysis. Our contributions are summarized as follows: (i) We introduce a cognitively inspired benchmark that orthogonally controls object size, space, and numerosity, together with progressive cue ablation, enabling systematic diagnosis of numerosity perception in VLMs. (i) We demonstrate through multi-factor attribution analysis that model ar- chitecture accounts for the largest proportion of counting performance variance, substantially exceeding visual conditions and object categories. (i) We reveal via layer-wise probing that numerosity signals consistently emerge in early vision layers, while the performance gap strongly aligns with the language model’s mapping from latent representations to textual outputs. 2 Related Work Human numerosity perception. Human numerosity perception relies on two com- plementary mechanisms: within the subitizing range (N ≤ 4), enumeration is rapid and nearly error-free [15], whereas larger numerosities are processed by the Approximate Number System (ANS), whose estimation variability follows Weber’s law and increases proportionally with magnitude [10, 17]. Numerosity judgments are also systematically influenced by non-numerical visual cues such as object size, density, and convex hull extent [11]. Counting evaluation in vision–language models. Object counting has become a common benchmark for evaluating VLM perception. Existing datasets are broadly categorized into general-purpose benchmarks [5, 7, 19], which mainly focus on small numerosities, and counting-specific datasets [4, 9, 16], which ex- tend to larger and denser scenes but rely on complex real-world images, making systematic error attribution difficult. Recent studies have moved beyond perfor- mance evaluation toward diagnostic analysis. CountCLIP [12] and subsequent NumerosityVLM3 [Text Output] [Image Input] Patch Embedding Vision Encoder (ViT) Vision-Language Alignment Module 0 Vision EncoderLanguage ModelOutput ACC 0.5 0.3 0.7 0.9 0.95 0.85 0.81 0.74 0.75 InternVL3-8B InternVL3-1B 0.08 Conclusion 2: Early vision encoder layers show separa- ble numerosity signals, while cross-model performance variations appear to be associated with differences in the language model‛s representation-to-text alignment. Conclusion 1: Cross-model performance differences are primarily associated with architectural design and multimodal integration strategies Large Language Model 0.19 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.27 0.70 0.17 0.75 0.80 0.86 0.21 0.40 0.72 0.35 0.79 0.79 0.86 0.20 0.28 0.71 0.15 0.73 0.81 0.86 0.19 0.23 0.70 0.12 0.75 0.76 0.67 0.21 0.22 0.66 0.09 0.80 0.75 0.90 0.22 0.23 0.78 0.08 0.81 0.77 0.87 set 1 CLIP-ViT-B/32 LLaVA-1.5-7B LLaVA-OneVi- sion-7B InternVL3-1B InternVL3-8B Qwen3-VL-4B Qwen3-VL-8B set 2set 3 Model X Condition Interaction Accuracy set 4set 5set 6 APPLE BUTTERFLY PEOPLE Object Category Condition Variation Baseline condition: preserves the natural covariance between number and total area SET 1 Size-incongruent condition: Object size increases while fixing other conditions SET 2 Space-incongruent condition: Spatial density increases while fixing other conditions SET 3 Text-ablated condition: Images are regenerated SET 4 Shape-ablated condition: Objects are replaced by solid-coloured circles SET 5 Color-ablated condition: Colored dots are converted to grayscale SET 6 The 3D orthogonal space defined by the Numer-osity, Size, and spacing Dataset Setup Analysis Orthogonal control VLM Structure Numerosity Setting 012345678910 LEVEL 1 Small Numbers Large Numbers LEVEL 2LEVEL 3LEVEL 4LEVEL 12 ...... ...... ...... 100 Spacing Coverage Closeness Sparsity Field Area Convex Hull Total Perimeter Total Surface Area Item Surface Area/Perimeter Size Numerosity ect... Fig. 1: Overview of the proposed NumerosityVLM benchmark framework. Left: The diagnostic benchmark comprises three object categories, six controlled conditions, and 12 numerosity levels. Right: Layer-wise probing is conducted across the full VLM pipeline to localize the source of numerosity perception failures. work [2,19] attributed counting failures to dataset bias, visual clutter, and atten- tion limitations, while Testolin et al. [22] introduced a cognitively inspired eval- uation framework. Numerosity representations have also been shown to emerge in convolutional networks [6] and large language models [1]. However, existing studies still lack both systematic factor-isolated evaluation for diagnosing the visual determinants of numerosity perception and mechanistic analysis of how numerical representations are formed throughout the VLM pipeline. 3 The Diagnostic Counting Benchmark As illustrated in Figure 1 and inspired by studies of human numerosity percep- tion, we introduce NumerosityVLM, a controlled synthetic benchmark designed for the mechanistic evaluation of vision–language models (VLMs). 3.1 Design Principles All images are synthesized on a 1024 × 1024 pixel, uniformly white canvas, with non-overlapping objects constrained to maintain a minimum pairwise inter- object distance greater than 5 pixels and to lie entirely within the image bound- aries. The dataset is generated automatically using Python scripts, and for each image we record metadata comprising individual surface area (ISA), total surface area (TSA), convex hull area, mean spatial sparsity, and object coordinates. 4Y. Fu et al. Fig. 2: Representative dataset stimuli under the six controlled conditions. Columns show the designed controlled conditions. Rows illustrate numerosities of 1, 11, and 100 using examples from the apple, butterfly, and human-figure categories. 3.2 Controlled Conditions The benchmark comprises two complementary groups of conditions: (i) three orthogonal control conditions that independently manipulate object size and spatial arrangement, and numerosity. (i) three visual-cue ablation conditions that progressively remove texture, shape, and color information. Examples of samples from all six conditions are shown in Figure 2. Orthogonal Control Conditions Let N denote the number of objects, ISA the individual surface area, and TSA = N · ISA the total surface area. Baseline. ISA is fixed at 50× 50 pixels 2 , so TSA increases monotonically with N. Object positions are randomly sampled. This setting preserves the natural covariance between numerosity and total area. Size-Incongruent. TSA is fixed at 2.5× 10 5 pixels 2 , while ISA varies inversely with N, subject to ISA > 28× 28 pixels 2 to avoid overly small objects. This isolates numerosity from total occupied area. Space-Incongruent. ISA is fixed at 50× 50 pixels 2 while objects are constrained within a convex hull of approximately fixed perimeter. As N increases, spatial density increases correspondingly. This dissociates numerosity from convex hull area and introduces controlled crowding. NumerosityVLM5 Visual Cue Ablation Conditions All three ablation conditions reuse the coordinates and ISA from the Baseline condition, ensuring that positions and sizes are identical. Each condition removes one visual cue in a progressive chain. Texture-Ablated. Textured objects are replaced with simplified shape templates rendered using the dominant color of each category. This examines whether predictions depend on texture cues. Shape-Ablated. Objects are further replaced by solid-colored dots with the same ISA and color as in the Texture-Ablated condition. This removes semantic shape information while preserving spatial configuration and color statistics. Color-Ablated. Colored dots are converted to grayscale while preserving position, size, and luminance. This isolates the influence of color on numerosity perception. 3.3 Numerical Range and Dataset Statistics To cover both the subitizing and approximate number system (ANS) regimes, we include all integers from 1 to 4 and logarithmically spaced values from 5 to 100 with a ratio of approximately 1.45: 1, 2, 3, 4, 5, 7, 11, 18, 29, 46, 74, 100.(1) Across all six conditions, the benchmark includes three object categories: apples, butterflies, and human figures. Each category contains five subtype variations that differ in visual appearance, such as color and shape, as illustrated in the Object Category section of Figure 1. For each image, 1–5 subtypes are randomly sampled according to the target numerosity, providing within-category visual diversity. For each condition, object category, and numerosity level, we generate 50 random instances, yielding a total of 6× 3× 12× 50 = 10,800 images. 4 Experimental Setup We evaluate seven representative open-source VLMs on the proposed benchmark to identify the main factors affecting numerosity perception and to assess po- tential biases or learning shortcuts. Both open-ended and closed-ended prompt strategies are compared. All experiments are conducted on a single NVIDIA L4 GPU (24 GB) using official pre-trained weights in BF16 precision. 4.1 Model Selection We select seven open-source models spanning four VLM architectures, as de- tailed in Table 1. CLIP [13] aligns visual and text embeddings via contrastive learning. LLaVA [8] projects visual features into the LLM token space through a lightweight MLP. InternVL3 [25] adopts multi-modal joint pretraining with tighter vision-language fusion. Qwen3VL [3] introduces DeepStack, injecting multi-scale visual features into corresponding LLM layers. 6Y. Fu et al. Table 1: Overview of the seven evaluated open-source vision-language models, includ- ing the backbone architectures of their vision and language components. Model VersionYear Input Size Vision Model Language Model CLIPCLIP-ViT-B/322021 224×224 ViT-B/32Transformer LLaVA LLaVA-1.5-7B-hf2023 336×336 ViT-L/14Vicuna-7B LLaVA-OneVision- Qwen2-7B-ov-hf 2024 384×384 SigLIPQwen2-7B InternVL InternVL3-1B-hf2025 448×448 InternViTQwen2.5-0.5B InternVL3-8B-hf2025 448×448 InternViTQwen2.5-7B QwenVL Qwen3VL-4B-Instruct 2025 Dynamic Qwen3-ViT Qwen3-4B Qwen3VL-8B-Instruct 2025 Dynamic Qwen3-ViT Qwen3-8B 4.2 Statistical Attribution Analysis Prompting Strategy Both open-ended and closed-ended prompting strategies were employed during benchmark evaluation. The prompts corresponding to each strategy are specified as follows: – Open-ended: How many objects are in this image? – Closed-ended: How many objects are in this image? Choose one answer from the following options: 1, 2, 3, 4, 5, 7, 11, 18, 29, 46, 74, 100 To ensure consistent evaluation, identical prompt templates are applied across all compatible models. Since CLIP does not support text generation, it is eval- uated only with closed-ended prompts, while open-ended prompting is applied to the other selected generative VLMs. Numeric responses were batch-extracted from model outputs using heuristic rules and regular expressions. Evaluation Matrix To further evaluate the numerosity perception of vision- language models, we define five complementary metrics to quantify overall per- formance and error magnitude. Accuracy measures exact-match correctness be- tween predicted and ground-truth counts. Mean Absolute Error (MAE) quanti- fies the average absolute deviation: MAE = 1 n n X i=1 |ˆy i − y i |,(2) where n is the total number of samples. To account for scale effects across dif- ferent numerosities, we compute Normalized Absolute Error (NAE): NAE = 1 n n X i=1 |ˆy i − y i | y i ,(3) NumerosityVLM7 which normalizes the absolute deviation by the ground-truth value. This nor- malization is motivated by Weber’s law, reflecting the ratio-dependent nature of human numerosity perception and allowing larger deviations for higher counts without disproportionately affecting performance. In addition, we introduce MAE error and NAE error , which are averaged only over mispredicted samples to characterize the magnitude of errors: MAE error = 1 m m X i=1 |ˆy i − y i |, NAE error = 1 m m X i=1 |ˆy i − y i | y i , (4) where m denotes the number of mispredicted samples. Multi-factor Contribution Analysis To quantify the contributions of dif- ferent factors to counting performance, we perform a multi-factor ANOVA on image-level Accuracy with three categorical factors, Model, Condition, and Sub- set, together with their pairwise interactions. Effect sizes are quantified using partial ω 2 , providing a statistical attribution of performance differences. 4.3 Systematic Bias Analysis Visual Bias Analysis To evaluate whether controlled visual factors induce systematic performance biases, we conducted a series of hierarchical controlled comparisons following the dataset’s factor-isolation strategy. Analyses were per- formed separately within two model groups stratified by overall performance. Specifically, we compared: (1) Baseline vs. Size-Incongruent and Baseline vs. Space-Incongruent to assess robustness under size and spatial controls; (2) Base- line vs. Texture-Ablated to isolate the effect of removing semantic texture cues; (3) Texture-Ablated vs. Shape-Ablated to evaluate the contribution of structured shape information relative to minimal dot stimuli; and (4) Shape-Ablated vs. Color-Ablated to measure the influence of color cues. Statistical significance was assessed using a generalized estimating equation (GEE) framework. Numerosity Bias Analysis To examine magnitude-dependent counting be- havior, we further computed residuals as residual = ˆy − y and analyzed their distribution across numerosity levels. Grouped mean residuals reveal systematic over- or under-counting and potential scale-dependent bias. Layer-wise Numerosity Probing To better understand the internal mech- anisms underlying numerosity perception in vision-language models and to an- alyze potential sources of performance differences, we conduct a layer-wise nu- merosity probing study using the proposed strictly controlled synthetic dataset. Let M denote a VLM under evaluation. For each layer i, we denote its hidden representation as h i =x i,0 ,x i,1 ,...,x i,N ,(5) 8Y. Fu et al. where each x i,j ∈R d is a token embedding in the latent space of layer i. We extract representations throughout the full multimodal pipeline, includ- ing (i) the visual embedding layer, (i) the outputs of the Attention and MLP submodules in each Vision Encoder block, (i) each layer of the vision-language alignment module, and (iv) each decoder layer of the language model. Directly training a probe on the full token set h i is undesirable due to its high dimen- sionality. We therefore adopt two aggregation strategies. For the global token representation (e.g., a CLS token), we use h global i = x i,0 ,(6) while for the patch representations, we compute the arithmetic mean over the remaining tokens: h mean i = 1 N N X j=1 x i,j .(7) These complementary strategies enable comparison between a dedicated global summarization token and a spatially aggregated latent representation. For each aggregated feature, we train a linear Support Vector Machine (SVM) with a fixed regularization parameter (C = 1) using data from the Shape-Ablated condition, while keeping the backbone model frozen. To verify that our obser- vations are not sensitive to probe hyperparameters, we additionally perform a grid search over C ∈ 0.01, 0.1, 1, 10, 100 for one representative model. Layer- wise probing accuracy is then compared with the model’s final textual counting performance to assess whether numerosity information is linearly decodable at intermediate stages and how such decodability relates to end-task performance. Language-backbone Numerical Geometry Motivated by evidence that LLMs may encode approximately logarithmically compressed numerical rep- resentations [1], we examined whether differences in language-backbone num- ber geometry align with differences in VLM counting performance. For each n∈1,..., 100, we presented the prompt “The number isn.” to each language backbone and extracted its hidden representation from a common intermediate transformer layer ( L = 12). We applied Principal Component Analysis (PCA) to the 100 representations and denoted the projection of each number onto the first principal component as s n . We quantified numerical ordering using the Spearman correlation betweennands n , and compared logarithmic,s n = a log(n) +b, and linear,s n = an + b, fits usingR 2 . These geometry measures were descriptively compared with the corresponding VLMs’ end-task exact-match accuracy. 5 Results 5.1 Statistical Attribution Analysis Overall Performance NumerosityVLM9 Table 2: Overall performance comparison across different VLMs. ModelAccuracy (%) ↑ MAE error ↓ NAE error ↓ InternVL3-1B15.5621.402.24 CLIP-ViT-B/3220.3924.062.97 LLaVA-1.5-7B27.2432.231.95 LLaVA-OneVision-Qwen2-7B71.2818.140.35 InternVL3-8B77.2118.470.35 Qwen3VL-4B-Instruct77.7324.970.36 Qwen3VL-8B-Instruct83.9720.850.31 Table 3: Performance comparison in details across different dataset conditions. Group Dataset Condition Accuracy (Mean± Std) MAE (Mean± Std) NAE (Mean± Std) Tier-1 Baseline0.7780± 0.058621.5381± 1.4629 0.3316± 0.0340 Tier-1 Size-Incongruent0.7909± 0.052322.7540± 2.5135 0.3730± 0.0436 Tier-1 Space-Incongruent0.7772± 0.061923.2677± 4.0634 0.3493± 0.0422 Tier-1 Texture-Ablated0.7205± 0.034419.1333± 4.7798 0.3641± 0.0184 Tier-1 Shape-Ablated0.7771± 0.085318.6371± 4.1403 0.3171± 0.0247 Tier-1 Color-Ablated0.8092± 0.040818.2291± 2.9610 0.3501± 0.0203 Tier-2 Baseline0.2019± 0.051827.2530± 8.9281 2.9104± 1.0483 Tier-2 Size-Incongruent0.3220± 0.080424.1742± 5.5354 1.2958± 0.2227 Tier-2 Space-Incongruent0.2087± 0.054125.9040± 5.1623 2.9413± 1.4558 Tier-2 Texture-Ablated0.1817± 0.046427.8706± 4.6050 2.6056± 0.3610 Tier-2 Shape-Ablated0.1718± 0.056828.3641± 4.4524 2.9368± 0.6242 Tier-2 Color-Ablated0.1783± 0.066921.8029± 2.2649 1.6320± 0.9012 Closed-Ended Setting. As shown in Table 2, the evaluated models cluster into two performance tiers. Tier-1 mainly consists of recent architectures with larger parameter scales, achieving a mean accuracy of 77.55% ± 5.19 and a mean NAE error of 0.34 ± 0.02. In contrast, Tier-2 models obtain a substantially lower mean accuracy of 21.06% ± 5.87 and a higher mean NAE error of 2.39 ± 0.53, slightly above the 8.34% random baseline. Figure 3 (c) further reveals distinct error patterns between the two tiers: Tier-1 models exhibit more symmetric error distributions around ground-truth counts, whereas Tier-2 models fail to achieve fine-grained numerical discrimination and tend to produce a limited set of repeated outputs regardless of the true count. Detailed per-model results are provided in Appendix Table 1. To assess the benchmark’s generalization beyond open-source models, we further evaluate the proprietary frontier Gemini 3 under the same protocol, with behavioral results reported in Appendix Figure 1. Open-Ended Setting. Quantitative metrics are omitted for the open-ended set- ting due to context-length limitations that caused some responses to be truncated before valid numerical answers could be extracted. Representative outputs and error distributions are shown in Appendix Table 2 and Appendix Figure 2. 10Y. Fu et al. Fig. 3: Comprehensive evaluation of numerosity understanding in VLMs. (a–b) Ac- curacy across conditions and mean residuals indicating underestimation at higher nu- merosity. (c–d) Output-layer and linear-probe confusion matrices for selected models. 5.2 Systematic Bias Analysis Visual Bias Analysis Orthogonal control conditions. Under the Size-Incongruent condition (total area fixed; numerosity inversely correlated with individual surface area), models ex- hibited a statistically significant improvement over Baseline (p < 0.01 for Tier 1, p < 0.001 for Tier 2) across both performance tiers, with Tier 2 models increas- ing their overall accuracy by 12.01%. In contrast, under the Space-Incongruent condition (convex hull perimeter fixed), performance remained comparable to Baseline across both tiers: Tier 1 models achieved 77.72% mean accuracy versus 77.80% at Baseline, while Tier 2 models scored 20.87% compared to 20.19%, with no statistically significant differences observed. Visual cue ablation conditions. Under the Texture-Ablated condition, where tex- tured object images were replaced with abstract, shape-matched colored silhou- ettes, models showed a significant decline relative to Baseline (p < 0.001) across both tiers. Tier 1 models improved when individual visual cues were selectively removed, where accuracy increased from 72.05%± 3.44% at the Texture-Ablated condition to 77.71% ± 8.53% under shape ablation and further to 80.92% ± 4.08% with color ablation (p < 0.001). Tier 2 models were largely unaffected, with Texture-Ablated accuracy at 18.17%, 17.18% under shape ablation, and 17.83% under color ablation, as shown in Figure 3 (a), Table 3, and Table 4. NumerosityVLM11 Table 4: Pairwise statistical comparisons across experimental conditions. Reported p- values are shown before and after multiple-comparison correction. Significance levels: ∗ p < 0.05, ∗ p < 0.01, ∗ p < 0.001; − indicates no significant difference. Group ComparisonRaw p-value Corrected p-value Sig. Tier-1 Baseline vs Size-incongruent1.96× 10 −3 2.75× 10 −3∗ Tier-1 Baseline vs Space-incongruent 8.31× 10 −1 8.49× 10 −1 − Tier-1 Baseline vs Texture-ablated1.63× 10 −29 5.72× 10 −29∗ Tier-1 Texture-ablated vs Shape-ablated 3.87× 10 −40 2.71× 10 −39∗ Tier-1 Shape-ablated vs Color-ablated 1.79× 10 −14 4.17× 10 −14∗ Tier-2 Baseline vs Size-incongruent2.53× 10 −52 1.77× 10 −51∗ Tier-2 Baseline vs Space-incongruent 6.79× 10 −1 6.79× 10 −1 − Tier-2 Baseline vs Texture-ablated5.81× 10 −12 1.36× 10 −11∗ Tier-2 Texture-ablated vs Shape-ablated 3.68× 10 −2 5.15× 10 −2 − Tier-2 Shape-ablated vs Color-ablated 1.29× 10 −1 1.51× 10 −1 − Contribution Attribution A three-way ANOVA with effect size estimation indicates that Model explains a substantial proportion of the variance (par- tial ω 2 = 0.325), whereas Condition (partial ω 2 = 0.005) and Subset (partial ω 2 = 0.001) contribute negligibly. Although several interaction terms reach sta- tistical significance, their effect sizes remain small, suggesting that architectural differences are the primary driver of performance variation. 5.3 Numerosity Bias Analysis Tier-1 models perform strongly within the subitizing range (1–4), achieving 94.94% average accuracy with a mean NAE of 0.02, approaching the near error- free level characteristic of human subitizing, as in Figure 3 (b). In the lower ANS range (5–20), models maintain 85.56% average accuracy with a mean NAE of 0.04, and residual curves remain centered around zero. As numerosity increases from 20 to 100, average accuracy decreases to 52.15% while the mean NAE rises to around 0.17. With further increases in numerosity, models exhibit a systematic underestimation trend: all evaluated models show consistent negative residuals over 50, with stronger models tending to exhibit a later onset of this bias. 5.4 Layer-wise Analysis of Numerosity Representations Figure 4 presents linear probing results for representative models from the two tiers. Overall, linearly separable numerosity signals emerge in early visual layers, while cross-tier differences are primarily reflected in the language modeling and decoding stages. Similar patterns are observed across all evaluated models and under SVM hyperparameter optimization, as shown in Appendix Figures 3–7. Vision encoder. Numerosity information is detectable at early patch-level representations, while global features initially remain uninformative but gradu- ally align with patch-level signals as depth increases. By the end of the vision 12Y. Fu et al. Fig. 4: Layer-wise probing results. Numerosity signals emerge early in the vision encoder, while performance differences primarily arise during language modeling. encoder, both tiers encode robust numerosity representations, suggesting that visual encoding is not the main source of performance variation. Language model. After vision-language alignment, numerosity information becomes less stable in global representations but continues to strengthen with depth in the language model. A clear divergence emerges between tiers in later layers: Tier-1 models retain more structured numerosity information throughout decoding, whereas Tier-2 models degrade rapidly toward the output stage. Per- formance differences are therefore mainly associated with language-level trans- formation rather than visual feature extraction. 5.5 Language-backbone Number Geometry As shown in Figure 5 and Table 5, Tier-1 language backbones exhibited more ordered and compressed numerical geometry, with mean Spearman ρ = 0.92± 0.02 and logarithmic-fit R 2 = 0.95± 0.01, compared with 0.73± 0.03 for linear fits. In contrast, Tier-2 backbones showed weaker ordering (ρ = 0.64±0.16) and more variable scaling, with logarithmic and linear R 2 values of 0.43± 0.36 and 0.34±0.12, respectively. CLIP was the only model better characterized by a linear than a logarithmic fit (R 2 = 0.37 vs. 0.17). Overall, language-backbone number geometry followed a broadly similar trend to VLM counting performance, despite imperfect correspondence across individual models. NumerosityVLM13 Fig. 5: PC1 projections across evaluated models. Tier-1 models (a–d) show clear mono- tonic organization, whereas Tier-2 models (e–g) exhibit weaker or inconsistent scaling. Table 5: Geometric and scaling analysis of internal number representations for 1–100. An asterisk (*) denotes models from the Instruct series. VLMLLMSpearman ρ↑ PC1 Var. Log R 2 ↑ Linear R 2 Log Slope LLaVA-OV-7BQwen2-7B*0.93660.27140.95390.76276.5417 InternVL3-8BQwen2.5-7B*0.92340.34890.94880.73329.2618 Qwen3VL-8BQwen3-8B0.92110.30090.94730.70138.2357 Qwen3VL-4BQwen3-4B0.89510.38670.93240.72744.5485 InternVL3-1BQwen2.5-0.5B*0.81000.42900.84390.44741.2121 LLaVA-1.5-7B Vicuna-7B-V1.50.49010.94160.28500.21043.6450 CLIP-ViT-B/32 Transformer0.61160.33280.1661 0.37150.5987 6 Conclusion and Limitations We introduce a cognitively inspired diagnostic benchmark, NumerosityVLM, with 10,800 strictly controlled synthetic images for evaluating numerosity per- ception in VLMs. Multi-factor analysis shows that model architecture explains the largest proportion of counting performance variance, while layer-wise prob- ing reveals that numerosity representations emerge early in vision encoders and that cross-model differences primarily arise during language decoding, suggest- ing that the observed performance differences are more strongly associated with later representation-to-output processing than with the initial availability of vi- sual numerosity information. The benchmark relies on synthetic stimuli with strict factor isolation, which may limit generalization to real-world scenes, and the probing results provide correlational rather than causal evidence. Future work will extend the framework to more complex real-world settings and ex- plore causal intervention methods. 14Y. Fu et al. Acknowledgements This work was supported by ERC eTALK Project (Grant No. EP/Y029534/1) and by Innovate UK (Grant No. 10089807) through the Horizon Europe project PRIMI (Grant Agreement No. 101120727). The authors gratefully acknowledge the support of Research IT and the use of the Research VM Platform at The University of Manchester. References 1. AlquBoj, H., AlQuabeh, H., Bojkovic, V., Hiraoka, T., El-Shangiti, A.O., Nwadike, M., Inui, K.: Number representations in llms: A computational parallel to human perception. arXiv preprint arXiv:2502.16147 (2025) 2. Amini-Naieni, N., Han, T., Zisserman, A.: Countgd: Multi-modal open-world counting. Advances in Neural Information Processing Systems 37, 48810–48837 (2024) 3. Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025) 4. Dai, S., Liu, J., Cheung, N.M.: Referring expression counting. In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 16985– 16995. IEEE (2024) 5. Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al.: Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394 (2023) 6. Kim, G., Jang, J., Baek, S., Song, M., Paik, S.B.: Visual number sense in untrained deep neural networks. Science advances 7(1), eabd6127 (2021) 7. Li, B., Ge, Y., Ge, Y., Wang, G., Wang, R., Zhang, R., Shan, Y.: Seed- bench: Benchmarking multimodal large language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 13299– 13308 (2024) 8. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: NeurIPS (2023) 9. Nguyen, T., Pham, C., Nguyen, K., Hoai, M.: Few-shot object counting and detec- tion. In: European Conference on Computer Vision. p. 348–365. Springer (2022) 10. Nieder, A.: The neuronal code for number. Nature Reviews Neuroscience 17(6), 366–382 (2016) 11. Nieder, A.: The calculating brain. Physiological reviews 105(1), 267–314 (2025) 12. Paiss, R., Ephrat, A., Tov, O., Zada, S., Mosseri, I., Irani, M., Dekel, T.: Teaching clip to count to ten. In: Proceedings of the IEEE/CVF international conference on computer vision. p. 3170–3180 (2023) 13. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. p. 8748–8763. PMLR (2021) 14. Rahmanzadehgervi, P., Bolton, L., Taesiri, M.R., Nguyen, A.T.: Vision language models are blind. In: Proceedings of the Asian Conference on Computer Vision. p. 18–34 (2024) 15. Railo, H., Koivisto, M., Revonsuo, A., Hannula, M.M.: The role of attention in subitizing. Cognition 107(1), 82–104 (2008) NumerosityVLM15 16. Ranjan, V., Sharma, U., Nguyen, T., Hoai, M.: Learning to count everything. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 3393–3402. IEEE (2021) 17. Rinaldi, L., Marelli, M.: The use of number words in natural language obeys weber’s law. Journal of Experimental Psychology: General 149(7), 1215 (2020) 18. Sarnecka, B.W., Carey, S.: How counting represents number: What children must learn and when they learn it. Cognition 108(3), 662–674 (2008) 19. Sengupta, S., Moradinasab, N., Liu, J., Brown, D.E.: Can vision-language models count? a synthetic benchmark and analysis of attention-based interventions. arXiv preprint arXiv:2511.17722 (2025) 20. Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., Beißwenger, J., Luo, P., Geiger, A., Li, H.: Drivelm: Driving with graph visual question answering. In: European conference on computer vision. p. 256–274. Springer (2024) 21. Siméoni, O., Vo, H.V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., Massa, F., Haziza, D., Wehrstedt, L., Wang, J., Darcet, T., Moutakanni, T., Sentana, L., Roberts, C., Vedaldi, A., Tolan, J., Brandt, J., Couprie, C., Mairal, J., Jégou, H., Labatut, P., Bojanowski, P.: DINOv3 (2025), https://arxiv.org/abs/2508.10104 22. Testolin, A., Hou, K., Zorzi, M.: Visual enumeration remains challenging for mul- timodal generative ai. PloS one 20(9), e0331566 (2025) 23. Xia, R., Ye, H., Yan, X., Liu, Q., Zhou, H., Chen, Z., Shi, B., Yan, J., Zhang, B.: Chartx & chartvlm: A versatile benchmark and foundation model for complicated chart reasoning. IEEE Transactions on Image Processing (2025) 24. You, Z., Yang, K., Luo, W., Lu, X., Cui, L., Le, X.: Few-shot object counting with similarity-aware feature enhancement. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. p. 6315–6324 (2023) 25. Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., et al.: Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 (2025)