Paper deep dive
Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs
Xiaofu Chen, Stella Frank, Yova Kementchedjhieva
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/14/2026, 6:07:49 AM
Summary
This paper investigates whether vision encoders and Vision-Language Models (VLMs) encode 'canonical color' (conceptual knowledge of an object's typical color) as semantic information, distinct from visible pixel color. Using a dataset of 708 objects with canonical color labels, the authors probe frozen vision encoders (CLIP, SigLIP, DINOv2, ViT-MAE, Swin-V2) using grayscale images to remove chromatic cues. They find that canonical color remains linearly decodable from grayscale inputs, indicating a conceptual link to object identity. Furthermore, VLM post-training significantly alters where this information is accessible within the model architecture.
Entities (11)
Relation Signals (6)
CLIP â istypeof â Vision Encoder
confidence 99% · CLIP (Radford et al., 2021) ... are language-supervised models
DinoV2 â istypeof â Vision Encoder
confidence 99% · DINOv2 (Oquab et al., 2024) ... are self-supervised vision models
Canonical Color â isdecodablefrom â Vision Encoders
confidence 95% · canonical color remains decodable from grayscale images... indicating a conceptual link.
Canonical Color â istiedto â Object Identity
confidence 93% · canonical color... is tied to predicted object identity, indicating a conceptual link.
VLM Post-training â affects â Color Decodability
confidence 92% · VLM post-training can have a surprisingly large effect on color decodability in the vision encoder.
Grayscale Images â removes â Chromatic Cues
confidence 90% · color can be directly removed from the image input, by gray-scaling the image
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Visual encoders construct a representation of the image input for Vision-Language models. How much conceptual, as opposed to immediately visible, information does this representation contain? We use canonical color as a controlled test case to ask whether vision encoders make canonical-color information linearly accessible, even when color is removed from the input image. We construct a dataset of objects with canonical colors, and probe vision encoders for both color and object identity using color and grayscale images. We find that canonical color remains decodable from grayscale images, and is tied to predicted object identity, indicating a conceptual link. Extending this analysis to full VLMs, we find that VLM post-training can have a surprisingly large effect on color decodability in the vision encoder. Overall, canonical color provides a usefully controllable lens for tracing object-level conceptual semantic information in vision encoders and VLMs.
Tags
Links
- Source: https://arxiv.org/abs/2609.09124v1
- Canonical: https://arxiv.org/abs/2609.09124v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
49,549 characters extracted from source content.
Expand or collapse full text
Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs Xiaofu Chen 1 Stella Frank 2 Yova Kementchedjhieva 1 1 MBZUAI 2 Technical University of Denmark xiaofu.chen, yova.kementchedjhieva@mbzuai.ac.ae stefra@dtu.dk Abstract Visual encoders construct a representation of the image input for Vision-Language models. How much conceptual, as opposed to imme- diately visible, information does this represen- tation contain? We use canonical color as a controlled test case to ask whether vision en- coders make canonical-color information lin- early accessible, even when color is removed from the input image. We construct a dataset of objects with canonical colors, and probe vi- sion encoders for both color and object iden- tity using color and grayscale images. We find that canonical color remains decodable from grayscale images, and is tied to predicted ob- ject identity, indicating a conceptual link. Ex- tending this analysis to full VLMs, we find that VLM post-training can have a surprisingly large effect on color decodability in the vision encoder. Overall, canonical color provides a usefully controllable lens for tracing object- level conceptual semantic information in vision encoders and VLMs. 1 Introduction Humans associate object concepts with typical at- tributes, including canonical color. Such attributes are part of conceptual representations, and color can be especially diagnostic for some object cate- gories (McRae et al., 2005; Tanaka and Presnell, 1999). Human studies also show that object knowl- edge can affect perceived color (Hansen et al., 2006; Witzel et al., 2011). Thus, knowing that bananas are yellow is different from seeing yellow pixels in a particular image. For text-only models, acquiring such typical knowledge can be difficult because text often states unusual rather than obvi- ous properties, a problem known as reporting bias (Gordon and Van Durme, 2013). Multimodal se- mantic work has therefore used visual grounding to complement linguistic evidence (Bruni et al., 2014; Kiela and Bottou, 2014; Silberer and Lapata, 2014). Vision encoders have direct access to object colors during training, but it remains unclear whether they encode canonical color as object-level conceptual knowledge or simply expose chromatic cues from the input during inference. Recent work has begun to address this question with representation-level probing. Linear probes are widely used to study what information is ac- cessible in intermediate representations, although their results depend on the probe and control set- ting (Alain and Bengio, 2017; Hewitt and Liang, 2019; Belinkov, 2022). Onea ̧t Ì a et al. (2025) show that frozen vision encoders encode many semantic attributes, including color. However, their stimuli are full-color RGB images, so color prediction may reflect visible pixels rather than canonical color knowledge. In VLMs, final answers can also be shaped by language priors, dataset bias, hallucina- tion, or weak objectâattribute binding (Goyal et al., 2017; Rohrbach et al., 2018; Li et al., 2023; Yuk- sekgonul et al., 2023). Golovanevsky et al. (2025) study such conflicts with counterfactual images, e.g., a blue strawberry, but full-model behavior does not reveal where canonical color information is represented within the visionâlanguage pipeline. In this work, we use canonical color as a con- trolled probe of object-level semantic information in vision encoders and VLMs. Color is a useful case because, unlike many other visual attributes, it can be directly removed from the image input, by gray-scaling the image, while leaving the object structure intact. Because canonical color is defined at the object-category level, its relation to object identity is part of the question we study, rather than a confound. We ask whether canonical color remains linearly accessible through object-level representations after visible color is removed, and whether object recognition alone accounts for its decodability. In other words, we probe the encoder: How is a black-and-white banana like black-and- white daffodils and lemons? Furthermore, if visual encoders represent canon- arXiv:2609.09124v1 [cs.CV] 8 Sep 2026 ical colors as part of their object concept represen- tations, how is this information passed on to the language backbone in VLM architectures? Does VLM training preserve the conceptual organiza- tion in the visual encoder, or is there a conceptual reorganization towards the language model? To answer these questions, we construct a dataset of object classes annotated with basic canonical color labels and train linear probes on frozen rep- resentations. RGB inputs provide an upper-bound setting with visible color cues, while our Grayscale setting removes chromatic information and reduces simple luminance cues through grayscale conver- sion and histogram equalization. We validate this control with Visual-CounterFact (Golovanevsky et al., 2025) by comparing actual-color prediction from recolored images with canonical-color predic- tion from both original and counterfactual images. We then extend the probing logic to VLMs by com- paring matched pre- and post-VLM vision towers and decoder-side visual-token representations. Our results show that canonical color remains linearly decodable even after direct color cues are removed. Counterfactual validation indicates that this residual signal is better aligned with canoni- cal object color than with actual surface color. We also find that canonical color is related to object- class information, but the two signals are only par- tially aligned. Finally, VLM post-training changes where object and color information are most lin- early accessible: in some models, this information becomes less accessible in the standalone vision tower but reappears after the visual interface or inside decoder-side visual-token states. 2 Related Work Semantic feature norms describe object concepts through typical properties, including perceptual attributes such as color (McRae et al., 2005; De- vereux et al., 2014; Bannert and Bartels, 2013). Recent work has used probing to ask whether pretrained representations encode such attributes. Onea ̧t Ì a et al. (2025) probe image encoders, multi- modally trained image encoders, and language-only models for perceptual, functional, and encyclope- dic attributes. Our work follows this representation- level view, but focuses on canonical color as a controlled case study. Color is a special attribute because it is directly visible in RGB images: a successful color probe may rely on pixel-level chromatic cues rather than object-level canonical color.We therefore introduce input-side controls to remove chromatic information and further reduce the influence of luminance cues. Linear probes are useful for tracing what in- formation is accessible in frozen representations (Alain and Bengio, 2017), but probe accuracy alone does not show that a model uses that information in its final predictions. Prior work therefore em- phasizes careful probe design, control tasks, and cautious interpretation (Hewitt and Liang, 2019; Belinkov, 2022). We follow this view by treating probing as a measure of linear accessibility, and by validating our color-controlled setting with coun- terfactual images. Work on VLMs further shows that model predic- tions can be shaped by language priors, dataset bias, hallucination, and weak objectâattribute binding (Goyal et al., 2017; Rohrbach et al., 2018; Li et al., 2023; Zhao et al., 2022; Yuksekgonul et al., 2023). More directly related to color, Tang et al. (2023) identify Concept Association Bias, where VLMs fill in strongly associated missing concepts across modalities. Liang et al. (2025) show that color per- ception, reasoning, and robustness remain challeng- ing for modern VLMs. Golovanevsky et al. (2025) further study conflicts between visual evidence and memorized priors using Visual CounterFact, where objects are recolored to non-canonical colors. Un- like these behavioral studies, we localize where object and canonical color information are linearly decodable across vision encoders, VLM-trained vision towers, visual interfaces, and decoder-side visual-token representations. 3 Data and Probing Setup This section describes our controlled probing setup. We first construct a dataset of objectâcolor pairs, where each object class is associated with a canoni- cal color label. We then apply image-side controls to separate visible pixel color from canonical object color. Finally, we pass the images through frozen vision encoders, extract layer-wise representations, and train linear probes to predict canonical color and object identity from these representations. 3.1 Canonical Color Dataset We construct a dataset of 708 object classes, each associated with one of 10 canonical-color labels, with a target of five images per class. Our dataset extends the objectâcolor pairs com- piled by Golovanevsky et al. (2025) with additional pairs sourced from Wikidata. We retain entries whose structured has-color property contains a sin- gle value, which provides the canonical-color label. To reduce the label-space complexity, we map these values to ten basic colors: black, blue, brown, gray, green, pink, purple, red, white, and yellow. The resulting objectâcolor pairs are manually screened for validity. For each class, we target five images by com- bining those linked from Wikidata with additional Google Search results, retaining up to five images that clearly depict the object and are consistent with its canonical color. Because PokĂ©mon were over-represented, we downsample them to 50 classes while prioritizing low-frequency colors (gray, brown, and red). The resulting color distribution is shown in Figure 7; white is the most frequent label with 99 classes and gray the least frequent with 48. 3.2 Input-side Controls for Color Probing A probe that predicts the canonical color of an object from a vision encoderâs representation may rely on color information that is directly present in the input image. RGB images contain real pixel- level color, so high probing accuracy in this setting does not by itself show that the encoder stores color as concept-level knowledge. We therefore treat RGB as an upper-bound setting and remove this direct color channel in the controlled settings. To remove pixel color, we first convert each image to grayscale by using a single luminance channel and replicating it across three channels. We then apply per-image histogram equalization to reduce brightness-based cues that may still cor- relate with color. We refer to this processed in- put as grayscale throughout the paper. Thus, grayscale denotes grayscale conversion followed by per-image histogram equalization. 3.3 Probing Tasks and Evaluation We probe the internal representations of frozen vi- sion encoders to ask two questions. First, do these representations contain information that can predict an objectâs canonical color? Second, how does this color information relate to object-class informa- tion? To answer these questions, we use two linear probing tasks. The color probe predicts one of the 10 color classes in our dataset, while the object- class probe predicts the object identity among 708 object classes. We compare the layer-wise trajectories of the two probes to test whether canonical color decod- ability is associated with category-level object se- mantics. Because the two tasks have different label spaces, 10 color classes versus 708 object classes, their absolute accuracies are not directly compara- ble. We therefore focus on how their accuracies change across layers and whether the two trends become aligned. Canonical color labels are defined over object categories rather than over individual image in- stances. For example, the label âyellowâ for an image containing a banana reflects the canonical color associated with the object category banana. Thus, a color probe that predicts âyellowâ for such an image may rely on representations that support category-level identification of the object as a ba- nana, or on other image-level cues that correlate with canonical color. A color signal that appears when object-class recognition is still weak may re- flect lower-level visual regularities or coarse object information. A color signal that strengthens to- gether with object-class recognition is more likely to be linked to category-level object semantics. 4 Controlled Probing of Canonical Color in Vision Encoders 4.1 Experimental Setup Vision Encoders. We evaluate five vision en- coders that cover different supervision signals and architectural designs. CLIP (Radford et al., 2021) and SigLIP (Zhai et al., 2023) are language- supervised models trained with imageâtext align- ment objectives, so their visual representations are shaped by natural language supervision. DI- NOv2 (Oquab et al., 2024) and ViT-MAE (He et al., 2022) are self-supervised vision models, but they use different objectives: DINOv2 learns through self-distillation, whereas ViT-MAE learns by recon- structing masked image patches. We also include Swin-V2 (Liu et al., 2022), a hierarchical vision transformer, to test whether the observed patterns hold beyond standard ViT-style backbones. All models are used at the ViT-Base scale, or at the cor- responding hierarchical scale for Swin-V2. Model repositories are listed in Appendix A. Features.For each input image, we cache hidden states from all transformer blocks and use mean- pooled patch features for all main probing results. For Swin-V2, adjacent blocks within each stage are averaged to form 12 layer-indexed features, match- ing the ViT-Base models. Probing. For each vision encoder, input setting, layer, and representation type, we train an inde- pendent linear probe on the cached image-level features. Each probe is an L2-regularized multino- mial logistic regression classifier trained for up to 1000 optimization iterations. The vision encoders remain frozen throughout probing. For color probing, we use 5-fold object-class- level cross-validation to avoid leakage across im- ages of the same object class. We split the 708 object classes into five disjoint folds. In each fold, the probe is trained on all images from the training object classes and evaluated on the held-out object classes. At test time, we average the predicted prob- ability vectors across all images of each test object class and take the argmax, producing one color pre- diction per object class. We report accuracy across the five folds. For object-class recognition, we use a per-class image hold-out setting, since holding out entire object classes would remove the target labels from training. For each object class with at least two images, we hold out one image for testing and use the remaining images for training. Object classes with only one image are used for training only. This task tests whether different images of the same object class are grouped in the representation space. 4.2 Validating the Color-controlled Setting Before using grayscale as our main controlled set- ting, we first validate what this preprocessing re- moves and what it preserves (Figure 1). All experi- ments use patch-averaged representations from the same five frozen encoders used in the main experi- ments, with results averaged across encoders. First, color probing accuracy drops sharply from RGB to grayscale, confirming that RGB probing is strongly driven by visible pixel color. Second, object-class recognition also drops under grayscale, but remains clearly decodable and continues to improve with depth. Thus, grayscale removes direct color cues while still preserving enough object-level structure for semantic probing. These two panels do not tell us whether the re- maining color signal reflects the actual surface color in the image or the canonical color of the object. To separate these possibilities, we repeat the probing analysis on Visual-CounterFact, which contains paired original and recolored images of 36912 Layer 0 25 50 75 Accuracy (%) ColorObject Class 36912 Layer 20 40 60 Accuracy (%) Counterfactual Color RGBGrayscaleMajority Baseline Actual On Counterfactual Canonical On Counterfactual Canonical On Original Majority Baseline Figure 1: Validation of the grayscale control. Left: color probing drops after visible color cues are removed. Middle: object-class recognition remains decodable. Right: Visual-CounterFact separates actual surface color from canonical object color. Dashed lines show majority baselines; shaded bands show±1standard deviation across encoders. the same object (Golovanevsky et al., 2025). We ap- ply the same grayscale preprocessing to all images. The actual-color probe on counterfactual images stays close to its majority baseline, indicating that the counterfactual surface color is mostly removed. In contrast, the two canonical-color probes, one on counterfactual images and one on original images, are both well above their majority baseline and show similar layer-wise trends. The actual-color probe and the canonical-color probes use differ- ent label spaces. We therefore use the actual-color probe only as a check that the counterfactual sur- face color is not recoverable after grayscale pre- processing. The main validation comes from com- paring the canonical-color probes on the original and the counterfactual images, which use the same canonical-color label space. Their similar layer- wise behavior shows that canonical color remains decodable from both counterfactual and original images after visible color cues are removed. Overall, these controls address two types of residual cues. First, grayscale conversion removes chromatic cues, histogram equalization reduces simple brightness cues, and Visual-CounterFact tests whether the counterfactual surface color can still be decoded after preprocessing. The actual- color probe remains near the majority baseline, while the canonical-color probes on the original and counterfactual images remain above baseline. Second, grayscale preserves object-correlated struc- 24681012 Layer 25 30 35 40 45 Accuracy (%) CLIP SigLIP DINOv2 ViT-MAE Swin-V2 Figure 2: Layer-wise color probing accuracy under grayscale inputs using patch-averaged representations. The figure compares five vision encoders: CLIP, SigLIP, DINOv2, ViT-MAE, and Swin-V2. ture, such as texture, material, and luminance lay- out. We do not treat this structure as a confound: our claim is not that canonical color can be de- coded independently of object identity, but that it can be linearly decoded through object-level rep- resentations, as examined in Section 4.4. In short, the control separates canonical color from surface color, not canonical color from object identity. 4.3 Canonical Color Decodability Across Encoders We next ask whether canonical color remains lin- early decodable across different vision encoders after visible color cues have been removed. Fig- ure 2 shows that all five encoders remain above chance and the 13.98% majority-class baseline un- der grayscale inputs. Most models improve from early to middle layers, suggesting that canonical- color-predictive information becomes more acces- sible in later representations. Detailed best-layer results, confidence intervals, and significance tests are reported in Table 4 in Appendix B. Across encoders, differences in peak accuracy are modest. DINOv2 reaches the highest overall peak, while SigLIP and Swin-V2 also perform strongly in the middle layers. CLIP improves across layers but does not clearly outperform the self-supervised encoders, suggesting that the signal does not depend exclusively on imageâtext super- vision. Several models, especially ViT-MAE and to a lesser extent SigLIP and DINOv2, show some degradation in their final layers, which may reflect increasing specialization for their pretraining ob- jectives. Because this cross-encoder comparison holds model scale approximately fixed, we additionally conduct a controlled within-family comparison of ModelParams RGB Gray Gray + HE DINOv2-S22M71.146.241.8 DINOv2-B86M77.048.643.8 DINOv2-L304M79.451.046.0 DINOv2-g1.1B80.651.848.0 Table 1: Best-layer canonical-color accuracy (%) across different DINOv2 model scales. All results are obtained using the same object-class-level five-fold protocol and mean-pooled patch features as in the main experiments. HE denotes histogram equalization. 36912 30 40 CLIP Spearman = 0.895 36912 SigLIP Spearman = 0.644 0 50 36912 30 40 DINOv2 Spearman = 0.881 36912 ViT-MAE Spearman = 0.755 0 50 36912 30 40 Swin-V2 Spearman = 0.601 0 50 Color Accuracy (%) Class Accuracy (%) Layer Class RecognitionColor Figure 3: Layer-wise object-class recognition and color probing under grayscale inputs. Solid and dashed lines show object-class and color probing accuracy, respec- tively. Each subplot reports the Spearman correlation between the two layer-wise curves. DINOv2-S/B/L/g using the same probing protocol. Canonical-color accuracy increases monotoni- cally with scale in all three input settings (Spear- manâsÏ = 1.00): from 46.2% to 51.8% for Gray and from 41.8% to 48.0% for Gray+HE. The RGBâ Gray gap remains broadly stable across sizes, while entity CLS accuracy follows the same monotonic trend. Thus, canonical-color decodability under grayscale inputs is not specific to a single Base- scale operating point and becomes stronger with model scale. Together, the cross-encoder and scale compar- isons establish that canonical color remains linearly accessible after visible color cues are removed. They do not, however, determine whether this ac- cessibility is mediated by object-level recognition; the next section examines this relationship directly. 4.4 Relation to Object-Class Recognition We analyze how color probe and object recognition probe results relate. Both probes are trained on the same patch-averaged representations at each layer, under the grayscale setting. Figure 3 shows that 3691236912369123691236912 0 20 40 60 80 Consistency (%) CLIPSigLIPDINOv2ViT-MAESwin-V2 Layer Figure 4:Predicted-object color accuracy under grayscale inputs. Bars report the percentage of test entities for which the color probe predicts the canonical color of the object predicted by the object probe. Error bars show±1 standard deviation across five folds. object-class recognition improves with depth for all models, although ViT-MAE remains lower in absolute accuracy. Color probing also tends to im- prove from early to middle layers, but its layer-wise trajectory does not always match object-class recog- nition. For example, some models show high color accuracy before object recognition reaches its peak, while others show a late-layer drop in color prob- ing despite continued gains in object recognition. This suggests that color-predictive information is associated with object-level representations, but is not fully determined by object-class recognition accuracy alone. We measure Spearman correlation as a measure of layer-wise association between the two accuracy curves (shown in Figure 3), to establish whether layers with higher object-class recognition accu- racy also tend to have higher color probing accu- racy. All models show a strong positive correlation, with CLIP and DINOv2 having the strongest layer- wise alignment. This suggests that color-predictive information and object-class information become more accessible in similar parts of the encoder, al- though the two signals are not perfectly aligned. Figure 4 measures color accuracy under the ob- ject class assumed by the object probe. For each test entity, letËobe the object predicted by the object probe,Ëcbe the color predicted by the color probe, andg(Ëo)be the canonical color of the predicted object. We count the prediction as correct when Ëc = g(Ëo). This metric does not requireËoto match the ground-truth object; it asks whether the color prediction is correct relative to the modelâs own object prediction. The score increases in later layers for most en- coders, reaching around 55â65% for CLIP, SigLIP, DINOv2, and Swin-V2, with ViT-MAE lower. This provides stronger evidence that the color probe of- ten tracks object-level canonical color information, rather than behaving independently of object iden- tity. At the same time, the scores are not near ceiling, which suggests that the encoders do not expose canonical color knowledge equally well for all object classes. 5 VLM Post-training and the Location of Semantic Decodability We next ask whether VLM post-training changes where canonical color and object information are linearly decodable. We do not evaluate final VLM answers, since they can be confounded by decoder- side world knowledge. Instead, we probe frozen representations across matched pre-/post-VLM vi- sion towers and decoder-side visual-token states. 5.1 Model-dependent changes in the vision tower We first test whether post-training changes linear accessibility in the standalone vision tower. We compare three matched pre-/post-VLM pairs: Ope- nAI CLIP ViT-L/14@336px versus the CLIP vi- sion tower in Molmo (Deitke et al., 2025), Ope- nAI CLIP ViT-L/14@224 versus the CLIP vision tower in mPLUG-Owl (Ye et al., 2023), and SigLIP- So400m@224 versus the SigLIP vision tower in PaliGemma (Beyer et al., 2024). Within each pair, the pre- and post-VLM encoders are evaluated at the input resolution used by the VLM; across pairs, we compare the direction of change rather than ab- solute accuracy. Model repositories are provided in Appendix A. All models are evaluated under the same grayscale setting. For each model and layer, we ex- tract patch-averaged features and train linear probes on frozen representations. Canonical-color prob- ing uses object-class-level cross-validation, while object recognition uses the same image-level entity- recognition protocol as before. The red- and blue-circle curves in Figure 5 show that post-training has model-dependent effects on vision-tower decodability. In the two CLIP-based pairs, Molmo and mPLUG-Owl, the post-VLM vi- sion towers yield lower linear-probe accuracy than the corresponding original CLIP encoders. The reduction is moderate for canonical color but sub- stantially larger for object recognition. Thus, in 41224D4D12D28 0 10 20 30 40 50 Color Accuracy (%) CLIP Molmo 41224D4D12D32 0 10 20 30 40 50 CLIP mPLUG-Owl 41227D4D12D18 0 10 20 30 40 50 SigLIP PaliGemma 41224D4D12D28 Layer Index (ViT | D=Decoder) 0 20 40 60 Object Accuracy (%) 41224D4D12D32 Layer Index (ViT | D=Decoder) 0 20 40 60 41227D4D12D18 Layer Index (ViT | D=Decoder) 0 20 40 60 Pre-VLM ViT (CLIP / SigLIP)Post-VLM ViTDecoder · Visual-AvgBlank Baseline Figure 5: Probing semantic decodability across vision towers and decoder visual-token states under grayscale inputs. Columns show matched pre-/post-VLM pairs: CLIPâMolmo, CLIPâmPLUG-Owl, and SigLIPâPaliGemma. Top row: canonical-color probing. Bottom row: object recognition. Red circles show the pre-VLM vision encoder. Blue circles show the post-VLM vision tower. Blue squares show the decoder visual-avg readout. Dotted gray lines show blank-image baselines; shaded bands show±1 standard deviation across folds. these two models, post-training makes object-level information less linearly accessible from the stan- dalone vision tower. The SigLIPâPaliGemma pair behaves differ- ently. Under the same grayscale probing setup, the post-VLM PaliGemma vision tower closely fol- lows the original SigLIP encoder for both canon- ical color and object recognition. Reduced stan- dalone vision-tower decodability is therefore not a universal consequence of VLM post-training: the two CLIP-based VLMs show clear reductions, whereas PaliGemma largely preserves the original SigLIP pattern. Detailed numerical comparisons and paired significance tests are reported in Table 5 in Appendix B. 5.2Visual-interface and decoder-side probing To test the reformatting hypothesis, we ask whether information weakly decodable from the post-VLM vision tower becomes linearly accessible again downstream. Keeping each VLM frozen, we run the same grayscale images through the model, cache hidden states after the visual interface and at each decoder layer, and apply the same linear probes as before. At each decoder layer, we aver- age the hidden states at the visual-token positions passed to the language model, yielding the visual- token average (visual-avg), the closest decoder- side analogue of the patch-averaged vision features. Decoder layersd 1 ,...,d N are plotted after the vi- sion layers only to indicate where information be- comes linearly accessible in the full stack. A blank- image baseline tests whether the recovered signal depends on visual input. The blue-square curves in Figure 5 show that ob- ject and canonical-color information can become linearly accessible again in decoder-side represen- tations. In Molmo, canonical-color decodability returns close to the original CLIP level from the first decoder layer, while object recognition in- creases toward the original CLIP peak in later lay- ers. mPLUG-Owl shows the same pattern: despite weak standalone-tower decodability, its decoder- side representations recover much of the reduction for both tasks. PaliGemma behaves somewhat dif- ferently: its post-VLM vision tower already closely tracks the original SigLIP encoder, so its decoder- side readout remains broadly comparable rather than showing a substantial recovery. To locate where the recovery enters the stack, we additionally probe the mPLUG-Owl visual in- terface for object recognition. Figure 6 shows that the post-VLM ViT readout remains low, whereas accuracy is already substan- tially higher at the Q-Former-style visual abstractor output, before the visual tokens enter the LLM de- coder. The first decoder layer changes this value 48121624 (L1)D1 (L6)D8D16D24D32 0 10 20 30 40 50 60 70 Object Accuracy (%) CLIP ViTQ-FormerLLaMA Decoder Pre-VLM ViT (Original OpenAI CLIP) Post-VLM Path (mPLUG-Owl ViT Q-Former Decoder) Q-Former Layer (6 Abstractor Layers) Figure 6: Object-recognition probing along the mPLUG- Owl visual path under grayscale inputs. Red shows the original CLIP encoder; blue shows the post-VLM path through the visual abstractor and LLaMA decoder. The main increase occurs at the visual interface. only slightly, while later layers further refine it, sug- gesting that the main increase occurs at the visual interface rather than in the language decoder alone. This pattern reflects a change in linear accessi- bility, not proof that the post-VLM ViT contains no object information; a nonlinear probe might recover additional information. We focus on information directly accessible to a simple linear readout to re- main consistent with the earlier experiments. The blank-image baseline remains near the majority- class baseline, suggesting that the recovered signal depends on visual input. Overall, these results are consistent with re- formatting rather than removal. In Molmo and mPLUG-Owl, object and canonical-color informa- tion becomes less linearly accessible in the stan- dalone tower but reappears after the learned vi- sual interface and in decoder-side representations. PaliGemma, whose vision tower changes less sub- stantially, largely preserves the original decodabil- ity pattern. 5.3 Prompt-level behavior under color-controlled inputs As a final behavioral check, we query the VLMs with color- and object-related prompts. This exper- iment is not intended to localize color information; instead, it illustrates how final answers can mix vis- ible evidence, object recognition, canonical-color priors, and output formatting. We evaluate the same 708 object classes after converting all images to grayscale. We use three prompts: Visual Color, âWhat color is the ob- ject in this image? Answer with just one word,â for which the expected answer is gray; Canonical Color, âWhat is the typical real-world color of this object? Answer with just one word,â which targets typical rather than visible color; and Object, âWhat object is shown in this image? Answer with one or MetricMolmo mPLUG PaliGemma Visual color23.414.917.3 Canonical color44.525.64.1 Object category53.057.161.3 Dec. color probe42.740.643.2 Dec. object probe59.757.569.9 Table 2: Prompt-level results on grayscale images. The first three rows report VLM answer accuracy; the last two report peak decoder visual-avg probe accuracy. two words,â which tests object identification. For the two color prompts, we obtain class-level predictions by majority vote over images of the same class. Because exact matching to Wikidata names is too strict for fine-grained, scientific, and proper names, we evaluate object recognition using manually verified acceptable labels derived from the entity name and Wikidata description. VLM outputs are not used to construct these label sets. Table 2 shows that final answers are not a clean measure of visual color perception. All models perform poorly on Visual Color despite gray be- ing the expected answer, suggesting effects from object-level color priors or response preferences. Canonical Color improves performance substan- tially for Molmo and also for mPLUG-Owl, while PaliGemma performs poorly. Category-level object recognition is higher but remains imperfect under grayscale inputs. Thus, final answers conflate vi- sual evidence, object recognition, priors, and out- put normalization, whereas internal visual-token probes provide a cleaner view of where object and canonical-color information are linearly decodable. 6 Conclusion Using canonical color as a controlled probe, we find that it remains linearly decodable from grayscale images even though RGB probing is strongly influenced by visible color. Counterfactual and object probes show that this signal aligns more closely with canonical than surface color and is re- lated to, but not fully determined by, object identity. VLM post-training can reduce tower-level decod- ability while the information reappears after the visual interface or within decoder-side states, a pat- tern consistent with representational reformatting rather than removal and illustrating how canoni- cal color information traces object-level semantics across the visionâlanguage stack. Limitations Our study focuses on canonical color as a con- trolled case study. Color is useful because visible chromatic cues can be removed with grayscale con- version and histogram equalization, but this also means that our empirical conclusions should not be directly generalized to all visual attributes. Other attributes, such as material, texture, function, or affordance, may require different controls and eval- uation protocols. Our analysis is also based on linear probing. This allows us to compare representations across models and layers in a simple and consistent way, but it only measures information that is linearly accessible from the chosen readout. A nonlinear probe or a downstream task-specific head might recover additional information. Therefore, our re- sults should be interpreted as evidence about linear decodability, rather than as a complete measure of all information contained in the representations. Acknowledgments Stella Frank was funded by NNF project 0094281. References Guillaume Alain and Yoshua Bengio. 2017. Under- standing intermediate layers using linear classifier probes. In International Conference on Learning Representations Workshop Track. Michael M Bannert and Andreas Bartels. 2013. Decod- ing the yellow of a gray banana. Current Biology, 23(22):2268â2272. Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguis- tics, 48(1):207â219. Lucas Beyer, Andreas Steiner, AndrĂ© Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, and 1 others. 2024. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726. Elia Bruni, Nam Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. Journal of Arti- ficial Intelligence Research, 49:1â47. Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, and 1 others. 2025. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 91â104. Barry J. Devereux, Lorraine K. Tyler, Jeroen Geertzen, and Billi Randall. 2014. The centre for speech, lan- guage and the brain (CSLB) concept property norms. Behavior Research Methods, 46(4):1119â1127. Michal Golovanevsky, William Rudman, Michael A Lepori, Amir Bar, Ritambhara Singh, and Carsten Eickhoff. 2025. Pixels versus priors: Controlling knowledge priors in vision-language models through visual counterfacts. In Proceedings of the 2025 Con- ference on Empirical Methods in Natural Language Processing, pages 24848â24863. Jonathan Gordon and Benjamin Van Durme. 2013. Re- porting bias and knowledge acquisition. In Proceed- ings of the 2013 workshop on Automated knowledge base construction, pages 25â30. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Thorsten Hansen, Maria Olkkonen, Sebastian Walter, and Karl R. Gegenfurtner. 2006. Memory modulates color appearance. Nature Neuroscience, 9(11):1367â 1368. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Pi- otr DollĂĄr, and Ross Girshick. 2022. Masked autoen- coders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000â16009. John Hewitt and Percy Liang. 2019. Designing and in- terpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733â2743, Hong Kong, China. Association for Computational Linguistics. Douwe Kiela and LĂ©on Bottou. 2014. Learning image embeddings using convolutional neural networks for improved multi-modal semantics. In Proceedings of the 2014 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 36â45, Doha, Qatar. Association for Computational Linguis- tics. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucina- tion in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292â305, Sin- gapore. Association for Computational Linguistics. Yijun Liang, Ming Li, Chenrui Fan, Ziyue Li, Dang Nguyen, Kwesi Cobbina, Shweta Bhardwaj, Jiuhai Chen, Fuxiao Liu, and Tianyi Zhou. 2025. Color- bench: Can vlms see and understand the colorful world? a comprehensive benchmark for color per- ception, reasoning, and robustness. arXiv preprint arXiv:2504.10514. Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. 2022. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 12009â 12019. Ken McRae, George S. Cree, Mark S. Seidenberg, and Chris McNorgan. 2005. Semantic feature production norms for a large set of living and nonliving things. Behavior Research Methods, 37(4):547â559. Dan Onea ̧t Ì a, Desmond Elliott, and Stella Frank. 2025. Seeing what tastes good: Revisiting multimodal dis- tributional semantics in the billion parameter era. In Findings of the Association for Computational Lin- guistics: ACL 2025, pages 24174â24191. Maxime Oquab, TimothĂ©e Darcet, ThĂ©o Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fer- nandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Woj- ciech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, and 7 others. 2024. DINOv2: Learning robust vi- sual features without supervision. In Transactions on Machine Learning Research. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748â8763. PMLR. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallu- cination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. Associa- tion for Computational Linguistics. Carina Silberer and Mirella Lapata. 2014. Learning grounded meaning representations with autoencoders. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 721â732, Baltimore, Maryland. Association for Computational Linguistics. James W. Tanaka and Lynn M. Presnell. 1999. Color diagnosticity in object recognition. Perception & Psychophysics, 61(6):1140â1153. Yingtian Tang, Yutaro Yamada, Yoyo Zhang, and Ilker Yildirim. 2023. When are lemons purple? the con- cept association bias of vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14333â14348, Singapore. Association for Compu- tational Linguistics. Christoph Witzel, Hanna Valkova, Thorsten Hansen, and Karl R. Gegenfurtner. 2011. Object knowledge modulates colour appearance. i-Perception, 2(1):13â 49. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, and 1 others. 2023. mplug-owl: Modularization empowers large lan- guage models with multimodality. arXiv preprint arXiv:2304.14178. Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2023. When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975â11986. Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. 2022. VL-CheckList: Evaluating pre-trained vision- language models with objects, attributes and relations. arXiv preprint arXiv:2207.00221. A Additional Dataset and Model Details This appendix provides additional details on dataset construction, label sources, evaluation splits, and the model repositories used in our experiments. A.1 Dataset Construction and Label Source The dataset contains 708 object concepts. Canonical-color labels are obtained from the Wikidata color property (P462). We retain only entities associated with a single color value and map the source values to ten basic categories: black, blue, brown, gray, green, pink, purple, red, white, and yellow. This reduced label space makes the probing task tractable and allows direct comparison across object classes.The canonical-color labels are therefore derived from structured Wikidata entries rather than assigned through free-form human annotation. For each object concept, we collect images linked from its Wikidata entry and supplement them with Google Search results, with a target of five images per concept. Manual inspection is used only for image quality control and does not assign or revise the canonical-color labels. We remove images that do not clearly depict the target object or are inconsistent with its canonical color. Because filtering is performed after image collection, some concepts contain fewer than five images. A.2 Class Distribution and Sampling We retain the resulting imbalance across canonical- color labels rather than forcing all categories to have the same size. During collection, PokĂ©mon formed a large subset of the candidate concepts. We therefore cap this subset at 50 concepts while prioritizing the less frequent gray, brown, and red categories. The final dataset contains 69 black, 69 blue, 53 brown, 48 gray, 73 green, 81 pink, 65 purple, 61 red, 99 white, and 90 yellow concepts. White is the most frequent label and gray is the least frequent. A.3 Evaluation Splits and Variable Image Counts For color probing, we use object-class-level five- fold cross-validation. All images of the same object concept are assigned to the same fold. In each run, the probe is trained on four folds and evaluated on object concepts from the held-out fold. The test classes are therefore unseen during training, so this setting evaluates canonical-color generalization to WhiteYellowPinkGreenBlackBluePurpleRedBrownGray 0 20 40 60 80 100 Number of entities 99 90 81 73 6969 65 61 53 48 majority baseline = 13.98% Figure 7: Distribution of the 708 object concepts across the ten canonical-color labels. new object concepts rather than to new images of known concepts. At test time, we average the predicted color- probability vectors over all available images of each object concept and take the class with the highest average probability. This produces one color prediction per object concept regardless of its number of images. We report results against the approximately 14% majority-class baseline. The cross-validation procedure defines the held-out ob- ject concepts but does not rebalance the color-label distribution. For object recognition, we instead use a within- class image hold-out because holding out an entire object class would remove its target label from training. For each concept with at least two images, we hold out one image for testing and use the re- maining images for training. Concepts represented by a single image are used for training only. A.4 Model Repositories B Numerical Results and Statistical Tests This section provides numerical summaries of the layer-wise comparisons in the main text. All results use histogram-equalized grayscale inputs and five folds. For the encoder comparison, we report best- layer accuracy with 95% confidence intervals, one- samplet-tests against the 13.98% majority base- line, and pairedt-tests for the RGB-to-grayscale difference. For the matched VLM pairs, we report best-layer pre-/post-training differences and paired t-tests over the five folds. Vision encoders. Table 4 reports the best-layer canonical-color results for the five vision encoders. All grayscale accuracies are significantly above the majority baseline, while all RGB-to-grayscale reductions are also significant. RoleModelHugging Face repository Vision encoderCLIP ViT-B/32 https://huggingface.co/openai/clip-vit-base-patch32 Vision encoderSigLIP Base https://huggingface.co/google/siglip-base-patch16-224 Vision encoderDINOv2 Base https://huggingface.co/facebook/dinov2-base Vision encoderViT-MAE Base https://huggingface.co/facebook/vit-mae-base Vision encoderSwin-V2 Base https://huggingface.co/microsoft/swinv2-base-patch4-window12-192-22k Pre-VLM vision towerCLIP ViT-L/14@336px https://huggingface.co/openai/clip-vit-large-patch14-336 Pre-VLM vision towerCLIP ViT-L/14@224px https://huggingface.co/openai/clip-vit-large-patch14 Pre-VLM vision towerSigLIP-So400m@224px https://huggingface.co/google/siglip-so400m-patch14-224 VLMMolmo-7B-D https://huggingface.co/allenai/Molmo-7B-D-0924 VLMmPLUG-Owl https://huggingface.co/MAGAer13/mplug-owl-llama-7b VLMPaliGemma 3B Mix 224 https://huggingface.co/google/paligemma-3b-mix-224 Table 3: Hugging Face model repositories used in our experiments. EncoderGray Acc.95% CIvs. 13.98% RGBâGray Drop CLIP41.5%[38.4, 44.6] p < .00134.9 p (p < .001) DINOv243.8%[37.0, 50.7] p < .00132.8 p (p < .001) SigLIP42.4%[37.7, 47.2] p < .00139.5 p (p < .001) ViT-MAE40.3%[35.3, 45.4] p < .00135.3 p (p < .001) Swin-V240.2%[35.8, 44.6] p < .00137.1 p (p < .001) Table 4: Best-layer canonical-color probing under histogram-equalized grayscale inputs. Confidence inter- vals are computed over five folds. Significance relative to the 13.98% majority baseline is assessed using one- samplet-tests; RGB-to-grayscale differences use paired t-tests. Here, p denotes percentage points. VLM PairTaskPre-VLM Post-VLMChange Paired Test CLIPâMolmoColor43.9%36.8% â7.1 p < .05 CLIPâMolmoObject65.0%26.2% â38.8 p p < .001 CLIPâmPLUG-Owl Color43.0%30.8% â12.3 p < .05 CLIPâmPLUG-Owl Object64.4%7.0% â57.4 p p < .001 SigLIPâPaliGemma Color43.3%42.9% â0.4 ppn.s. SigLIPâPaliGemma Object73.3%71.5% â1.9 p p < .001 Table 5: Best-layer grayscale probing accuracy for matched standalone vision towers before and after VLM post-training. Significance is assessed using paired t-tests over five folds. Statistical significance should be interpreted together with effect magnitude: the PaliGemma object difference is consistent across folds but only 1.9 percentage points. Pre-/post-VLM vision towers. Table 5 com- pares the standalone vision towers before and after VLM post-training. The two CLIP-based pairs show substantial reductions, particularly for object recognition, whereas the SigLIPâPaliGemma pair changes little in absolute magnitude.