Paper deep dive
Know-Thy-Not: Training-Free Targeted Negation in Dual-Encoder Vision-Language Models
DoYoung Kim, Changkyu Choi, Adín Ramírez Rivera, YunSeok Choi, Jee-Hyong Lee
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Dual-encoder vision-language models systematically fail at negation, scoring ''a cat without a dog'' nearly identically to ''a cat with a dog.'' We show that this failure is inherent to the single-embedding framework. Under contrastive training, single-embedding methods can at best ignore the negated concept (dilution) but never penalize its presence (rejection). We propose Know-Thy-Not (KTN), a training-free method that introduces an image-side rejection dimension to penalize the negated concept directly in the visual representation. We also introduce COCO-Confusion, the first benchmark requiring genuine rejection to succeed. Across 9 backbones, KTN improves COCO-Confusion mAP by up to +25.6 over vanilla VLMs and +9.6 over the strongest existing method.
Tags
Links
- Source: https://arxiv.org/abs/2610.17440v1
- Canonical: https://arxiv.org/abs/2610.17440v1
Trouble viewing inline? Open PDF directly →
Full Text
57,714 characters extracted from source content.
Expand or collapse full text
Know-Thy-Not: Training-Free Targeted Negation in Dual-Encoder Vision-Language Models DoYoung Kim Affiliation: Sungkyunkwan University, South Korea Changkyu Choi Affiliation: University of Oslo, Norway Adín Ramírez Rivera Affiliation: University of Oslo, Norway YunSeok Choi Jee-Hyong Lee Abstract Dual-encoder vision-language models systematically fail at negation, scoring “a cat without a dog” nearly identically to “a cat with a dog.” We show that this failure is inherent to the single-embedding framework. Under contrastive training, single-embedding methods can at best ignore the negated concept (dilution) but never penalize its presence (rejection). We propose Know-Thy-Not (KTN), a training-free method that introduces an image-side rejection dimension to penalize the negated concept directly in the visual representation. We also introduce COCO-Confusion, the first benchmark requiring genuine rejection to succeed. Across 9 backbones, KTN improves COCO-Confusion mAP by up to +25.6 over vanilla VLMs and +9.6 over the strongest existing method. 22footnotetext: Corresponding authors. 1 Introduction Dual-encoder vision-language models (VLMs) [29, 19] have become foundational backbones for multimodal systems, gaining structural flexibility by aligning images and text in a shared embedding space. This flexibility has made VLMs the default starting point for nearly every modern vision-language application, ranging from zero-shot image classification [3, 30] and text-image retrieval [8, 11] to image generation without task-specific finetuning [21, 20, 22]. However, a growing body of work has revealed that these models struggle with compositional reasoning: they fail at attribute binding [25] and spatial understanding [9], with dedicated benchmarks [15, 7, 17] consistently exposing near-chance performance on structured linguistic tests. Asking a VLM to retrieve “a car without a traffic light” returns images dominated by traffic lights (Figure 1, top), as the model treats the query as if the negation were absent. This failure is consistent and severe: On VOC2007, CLIP scores 82% on affirmative queries but only 3% on their negated counterparts [2]. Negation is not a peripheral linguistic phenomenon. It is essential to expressing what users do not want, and its absence undermines VLMs as reliable retrieval and grounding systems. Recent work has attempted to close this gap from two directions. Retraining-based methods such as ConCLIP [24] and NegationCLIP [18] augment training data with negation captions and retrain the model, but show modest improvements despite substantial cost. Alternatively, the training-free method [1] extracts the negated concept from the query and suppresses it by projecting out its direction from the text embedding, achieving notable gains without retraining. Both, however, still represent the negated query within a single embedding vector, which we show is structurally insufficient for negation. We introduce a formal analysis which reveals that the optimal text coefficient (6) on the negated concept is driven to zero by the non-negative nature of natural visual data. This implies that approaches operating solely on the text embedding are inherently bounded. The conventional embedding correction approach, which cancels the direction of the negated concept from the text embedding, is one such case. While this limits the model’s tendency to reward the negated concept, it never verifies whether the negated concept is actually present in the image. We argue that this is dilution, not rejection: VLMs need not merely ignore the negated concept (dilution) but actively penalize its presence (rejection). This stems from a representational asymmetry: visual concepts manifest as presence or absence, never as anti-presence, leaving no dimension on which a single text vector can push back against the negated concept. We attribute this dilution to a structural limitation of contrastive training: under the in-kind alignment component of contrastive training [29, 19], the model learns to treat the negated concepts as irrelevant noise to be ignored, rather than as signals to be actively rejected. Prior data-centric retraining attempts yield only marginal gains in practice. To escape this limitation, we propose Know-Thy-Not (KTN), a training-free method that scores a query along two independent dimensions. A text-side dimension affirms c+c^+ via a corrected text embedding, while an image-side dimension detects c−c^- in the image and penalizes its presence. We further introduce COCO-Confusion, a benchmark whose galleries contain images with the negated concept, making it the first to measure rejection rather than dilution. Existing benchmarks place the negated concept in unfiltered galleries where it rarely co-occurs with the affirmed concept, so methods that merely ignore the negation already score well. COCO-Confusion enforces co-occurrence in every gallery, making genuine rejection (not dilution) necessary to succeed. Across 9 backbones spanning CLIP, EVA-02, SigLIP, and SigLIP2, KTN consistently improves negation handling, with the largest gains on COCO-Confusion (+9.6 mAP over Embedding Correction [1]), where prior methods plateau. Query: “a car is present but no traffic light” (COCO-Confusion, SigLIP2 SO400M) Vanilla × × × × × Embedding correction [1] × × × ✓ ✓ Proposed method ✓ ✓ × ✓ ✓ Figure 1: Top-5 retrieval on our COCO-Confusion benchmark (leftmost is rank 1). A vanilla VLM ignores the negation, retrieving primarily confusing images that contain a traffic light. The embedding correction approach [1] removes the negated direction from the text embedding; while this filters out some confusing images (ranks 4–5), it still places traffic-light scenes at the top. In contrast, our proposed Know-Thy-Not (KTN) introduces an explicit image-grounding step, successfully retrieving correct images. Green borders indicate images containing only the affirmed concept (2), while red borders indicate confusing images containing the additional negated concept (3). 2 Preliminaries Vision-language retrieval. We consider a dual-encoder vision-language model [29, 19, 26], consisting of a text encoder f:→ℝdf_T ^d and an image encoder fℐ:ℐ→ℝdf_I ^d, both producing ℓ2 _2-normalized embeddings in a shared d-dimensional space. Given a text query q∈q and a gallery ijj=1M⊂ℐ\i_j\_j=1^M , image retrieval ranks M candidate images by their inner product to the query s(q,i)=f(q)⊤fℐ(i).s(q,i)=f_T(q) f_I(i). (1) Negation queries. We define a negation query as a compositional semantic constraint q=(c+,c−)q=(c^+,c^-), typically formulated as “c+``c^+ without c−”c^-′ or equivalent exclusionary construction. Here, c+c^+ denotes the affirmed concept and c−c^- denotes the negated concept. To evaluate the faithfulness of the retrieval system, we define a binary indicator function ϕ(i,c)∈0,1φ(i,c)∈\0,1\ that denotes the presence of concept c in image i. For a given q=(c+,c−)q=(c^+,c^-), we partition the gallery ℐI into three mutually exclusive sets: ∙ Correct set ℐ+~I^+: Images satisfying the affirmed concept while strictly excluding the negated concept ℐ+=i∈ℐ|ϕ(i,c+)=1,ϕ(i,c−)=0.I^+= \i \; |\;φ(i,c^+)=1,\;φ(i,c^-)=0 \. (2) ∙ Confusing set ℐ±~I^±: Images containing the affirmed concept but failing the negation constraint ℐ±=i∈ℐ|ϕ(i,c+)=1,ϕ(i,c−)=1.I^±= \i \; |\;φ(i,c^+)=1,\;φ(i,c^-)=1 \. (3) ∙ Irrelevant set ℐ−~I^-: Images that fail to satisfy the primary affirmed concept ℐ−=i∈ℐ|ϕ(i,c+)=0.I^-= \i \; |\;φ(i,c^+)=0 \. (4) In Figure 1, the affirmed concept c+c^+ is “car” and the negated concept c−c^- is “traffic light”. Faithful negation retrieval ranks ℐ+I^+ above ℐ±I^±, where ℐ±I^± is the most challenging distractor. 3 Why single text embeddings cannot reject negation Existing approaches only dilute the negated concept rather than reject it. We formalize this limitation by proving that contrastive training provides no gradient toward the negative coefficient that rejection requires. We confirm this limitation empirically on COCO-Confusion. 3.1 Existing approaches only dilute, not reject Data-centric strategies typically augment training sets with negated captions and retrain the model to enhance its grasp of negation. However, these approaches often yield marginal improvements. Despite the substantial computational overhead of retraining, the model’s ability to distinguish between a caption and its negation remains far from satisfactory [2, 18]. To address this bottleneck, Aggarwal et al. [1] introduced a training-free embedding correction method that isolates the negated concept c−c^- from the query q and projects out its direction from the original query embedding ~∝f(q)−λ⋅projectc−f(q), t\; \;f_T(q)-λ· project_c^-f_T(q), (5) where λ is a scalar hyperparameter. While this approach yields notable empirical gains, we observe that the resulting embedding ~∈ℝd t ^d remains agnostic to the visual grounding of c−c^-. The subtraction operation modifies the text embedding in isolation, failing to verify the absence of c−c^- against its actual features grounded in the visual manifold. Consequently, such embedding correction merely dilutes the semantic influence of c−c^- without penalizing its material presence in the image (see Figure 2). We posit that faithful negation handling requires an explicit mechanism of rejection—a capability that is structurally precluded by the conventional single embedding format. This observation forms the crux of our argument: the conventional single-embedding VLMs lack the expressive bandwidth to encode exclusionary logic within the same representation used for semantic matching. Query: “a photo of a cat without a dog”VanillaEmbedding correction [1]Proposed method Ignore: Activates on both Dog negation ignored Dilute: Activates on cat Dog feature suppressed Reject: Activates on cat Dog explicitly penalized Figure 2: Given the query “A photo of a cat without a dog”, a vanilla VLM ignores the negation and activates on both concepts (left). Embedding correction approach [1] removes the dog direction from the text embedding but never verifies the visual features, which amounts to passively diluting the concept (center). Faithful negation requires explicitly locating the dog in the image and penalizing its presence (right), which forms the crux of our method Know-Thy-Not (KTN). 3.2 A structural barrier in single-embedding negation handling Under standard ℓ2 _2-normalized contrastive training, the contrastive objective provides no learning signal toward the negative coefficient that rejection requires. Moreover, query-side post-hoc correction is also bounded by dilution. We formalize both limits below. Setup. For each concept c, let c∗∈ℝd e_c^* ^d denote a unit-norm proxy vector, ideally representing the concept c in the space shared by both encoders. We define αc(i):=fℐ(i)⊤c∗(image activation),βc(q):=f(q)⊤c∗(text coefficient). _c(i):=f_I(i) e_c^* (image activation), _c(q):=f_T(q) e_c^* (text coefficient). (6) For analytical tractability we adopt a linear concept model: fℐ(i)=∑cαc(i)c∗f_I(i)= _c _c(i) e_c^* on an orthonormal concept basis. We focus on negation queries q=q= “c+c^+ without c−c^-”. Assumption 1 (Concept Cone, idealized). In the idealized model, concept activations are non-negative: αc(i)≥0 _c(i)≥ 0, with αc(i)>0 _c(i)>0 iff c∈ic∈ i. This captures the intuition that semantic similarity measures presence, not anti-presence. Lemma 1 (Rejection criterion). For q=q= “c+c^+ without c−c^-”, consider images i+∈ℐ+i^+ ^+ and i±∈ℐ±i^± ^± that differ only along the c−c^- axis. Under the linear concept model, s(i+,q)−s(i±,q)=−αc−(i±)βc−(q).s(i^+,q)-s(i^±,q)=- _c^-(i^±)\, _c^-(q). (7) Faithful retrieval demands s(i+,q)>s(i±,q)s(i^+,q)>s(i^±,q), and since αc−(i±)>0 _c^-(i^±)>0, this is equivalent to βc−(q)<0. _c^-(q)<0. (8) Theorem 1 (In-kind alignment carries no gradient toward βc−(q)<0 _c^-(q)<0). Consider the positive-alignment objective (q)=f(q)⊤(q),(q):=i∈ℐq[fℐ(i)],A(q)=f_T(q) μ(q), μ(q):=E_i _q[f_I(i)], (9) the positive-pair component of standard contrastive losses. For q=q= “c+c^+ without c−c^-”, every i∈ℐqi _q satisfies c−∉ic^-∉ i, so μc−(q)=0 _c^-(q)=0, and decomposing (q)=∑kβk(q)μk(q)A(q)= _k _k(q)\, _k(q) gives ∂(q)∂βc−(q)≡ 0. \; (q)∂ _c^-(q)\;≡\;0.\; (10) The objective is invariant in βc−(q) _c^-(q): positive-alignment training provides no gradient that could drive βc−(q) _c^-(q) below zero. Corollary 1 (Optimal text coefficient under unit-norm constraint). Under the unit-norm constraint ‖f(q)‖=1\|f_T(q)\|=1 standard for ℓ2 _2-normalized encoders, the maximizer of (q)=β(q)⊤(q)A(q)=β(q) μ(q) is, by Cauchy–Schwarz, β∗(q)=(q)/‖(q)‖β^*(q)= μ(q)/\| μ(q)\|. Since μc−(q)=0 _c^-(q)=0 for negation queries, βc−∗(q)=0β^*_c^-(q)=0. Combined with Lemma 1, this implies s(i+,q)=s(i±,q)s(i^+,q)=s(i^±,q) at the optimum: even an optimally-trained encoder under in-kind alignment cannot rank correct above confusing. (Details in Appendix A.) Beyond in-kind alignment. Hard-negative supervision lies outside the positive-alignment regime but, under token frequency asymmetry, collapses to a generic negation offset rather than concept-specific rejection (Appendix A). Post-hoc embedding correction still only reaches β~c−(q)=0 β_c^-(q)=0 and fails to overcome the Lemma 1 barrier. β~c−(q)=0 β_c^-(q)=0 is the ceiling of all existing approaches, training-based or post-hoc, yet remains insufficient for rejection. 3.3 The limit of dilution-based approaches We demonstrate these limitations with an example query on COCO-Confusion (Figure 3). Raw SigLIP2 Embedding Correction Figure 3: Dilution cannot separate correct from confusing images. Top-5 similarity scores for “a car is present but no traffic light” on COCO-Confusion, SigLIP2 SO400M. Raw SigLIP2 (left): all top-5 are confusing (red) with uniform scores (≈0.15≈ 0.15). Embedding Correction (right): scores drop overall (≈0.12≈ 0.12), but red (confusing) and green (correct) bars remain interleaved, failing to separate correct from confusing. Vanilla VLM. Vanilla VLM does not process the negation word “no,” so the query is effectively treated as “a car is present and traffic light.” As a result, images containing both car and traffic light receive the highest similarity score—all top-5 results are confusing images (red bar) with nearly uniform scores (≈ 0.15≈\,0.15). Embedding correction. Embedding Correction dilutes the traffic light concept by removing its direction from the text embedding. As a result, the overall scores of confusing images decrease (≈ 0.12≈\,0.12). However, since Embedding Correction ignores traffic light rather than rejecting it, images containing both car and traffic light still receive high similarity scores. The score bars remain nearly uniform across confusing (red) and correct (green) images, failing to separate the two groups. This confirms the dilution ceiling: single-embedding scoring cannot reject c−c^-. Negation needs multi-dimensional scoring. One dimension to affirm c+c^+, another to penalize the image-side activation of c−c^-. KTN realizes this decomposition. 4 Proposed method: Know-Thy-Not Section 3 established that single-embedding scoring cannot achieve rejection: contrastive training provides no gradient toward βc−(q)<0 _c^-(q)<0 (Theorem 1), and post-hoc correction only reaches β~c−(q)=0 β_c^-(q)=0, still short of the rejection condition required by Lemma 1. All single-embedding approaches share this limitation. We propose Know-Thy-Not (KTN), a training-free method that extends scoring from one dimension to two: (i) a text-side affirmation dimension that handles c+c^+ via a corrected text embedding ~ t, and (i) an image-side rejection dimension that handles c−c^- by detecting and penalizing its visual presence in i. The full scoring is s(q,i)=~⊤fℐ(i)⏟Dim 1: Affirm c+−wn⋅pc−(i)⏟Dim 2: Reject c−,s(q,i)\;=\; t f_I(i)_Dim 1: Affirm c^+\;-\;\;w_n· p_c^-(i)_Dim 2: Reject c^-, (11) pc−(i)p_c^-(i) is the relative grounding score defined in Eq. (13), and wnw_n is a scaling parameter. Dimension 1 operates on the text side and is constrained by Theorem 1—by itself, it can only achieve dilution. Dimension 2 operates on the image side using a separate computation that does not pass through f(⋅)f_T(·), hence is not bound by Theorem 1’s unit-norm budget. The two dimensions share no parameter budget, allowing pc−(i)p_c^-(i) to provide the rejection signal that Lemma 1 requires and Corollary 1 forbids in any single-dimension formulation. 4.1 Text-side affirmation Given a natural-language query q such as “a cat without a dog”, we first extract the negated concept c−c^- and the affirmed concepts +C^+ via spaCy’s dependency parser [6]. Negation cues—no, not, without, absent, missing, free of—are identified by dependency relation (neg) or lemma matching, and the head noun of the negation span yields c−c^-; remaining noun phrases form +C^+. We then compute the corrected text embedding by projecting out the c−c^- direction: ~=normalize(f(q)−projectc−f(q)) t\;=\; normalize\! (f_T(q)- project_c^-f_T(q) ) (12) where projc−(⋅)proj_c^-(·) denotes orthogonal projection onto the c−c^- direction, making ~ t orthogonal to c− e_c^-. This achieves βc−(~)=0 _c^-( t)=0, matching the constrained optimum of Corollary 1 — yet by Lemma 1 still insufficient for rejection. 4.2 Image-side rejection The second scoring dimension produces an image-conditioned penalty that grows when c−c^- is grounded in image i and vanishes when it is not. A naive thresholding on fℐ(i)⊤c−f_I(i) e_c^- fails because raw similarity scales vary substantially across images: some images yield globally high similarity to all concepts, producing false positives. We then compute a relative detection score pc−(i)p_c^-(i) using a per-image reference τ(i)τ(i) pc−(i)=σ(κ(fℐ(i)⊤c−τ(i)))−12σ(κ(fℐ(i)⊤c+−τ(i))),p_c^-(i)\;=\;σ\! (κ\, (f_I(i) e_c^--τ(i) ) )- 12σ\! (κ\, (f_I(i) e_c^+-τ(i) ) ), (13) τ(i)=1|+|∑c∈+fℐ(i)⊤c.τ(i)\;=\; 1|C^+| _c ^+f_I(i) e_c. (14) where σ(⋅)σ(·) is the sigmoid acting as a soft thresholding operator — penalizing c−c^- only when αc−(i) _c^-(i) exceeds τ(i)τ(i) — and κ controls the sharpness of the transition. Real VLMs deviate from the ideal encoder of Assumption 1: αc−(i) _c^-(i) remains non-zero even when c−c^- is absent. We therefore anchor the threshold to the image’s own affirmed-concept similarity, τ(i)=1|+|∑c∈+αc(i)τ(i)= 1|C^+| _c ^+ _c(i), providing a query-relevant, image-specific baseline. When c−c^-’s similarity exceeds this baseline, the first term drives pc−(i)→1p_c^-(i)→ 1, and otherwise it falls to 00. The second term further suppresses false positives that arise when c+c^+ and c−c^- are semantically correlated (e.g., cat and dog). In such cases αc−(i) _c^-(i) may exceed τ(i)τ(i) even when c−c^- is absent, and discounting by the c+c^+-side activation prevents correct images from spurious penalty. Crucially, pc−(i)p_c^-(i) is computed entirely from fℐ(i)f_I(i) and concept directions—it does not pass through the text-query embedding f(q)f_T(q), which is what allows the rejection dimension to penalize c−c^- regardless of the alignment-induced ceiling on βc−∗ _c^-^*. 4.2.1 Why KTN avoids the constraint Corollary 1 bounds the contribution of any single text-side coordinate: under the unit-norm constraint, βc−∗=0 _c^-^*=0 exactly. Dimension 1 respects this bound—~⊤fℐ(i) t f_I(i) is a single-vector dot product and inherits the βc−∗=0 _c^-^*=0 ceiling. Dimension 2 sidesteps the bound by introducing a separate scoring axis computed outside the text-side unit-sphere budget. The total score is no longer a single dot product; it is a two-dimensional composition where each dimension operates in its own budget. The matched-pair test of Lemma 1—which requires a negative ranking signal—is satisfied by Dimension 2. The asymmetry of natural visual data—presence learns; absence does not—is resolved by decomposing the problem into two dimensions: Dimension 1 handles affirmation, where the data measure naturally provides positive signal (μc+>0 _c^+>0); Dimension 2 handles rejection, where the data measure provides no signal (μc−=0 _c^-=0) but direct image-side detection circumvents this gap entirely. 5 Experiments 5.1 Introducing COCO-Confusion: A benchmark for negation rejection Existing negation retrieval benchmarks such as NegBench evaluate retrieval over a large, unfiltered gallery (e.g., 5,000 COCO [12] images), where the negated concept rarely co-occurs with the affirmed concept in the same image. In such settings, even simple embedding correction methods perform well because there are few confusing images to distinguish from correct ones. This means existing benchmarks cannot distinguish between ignoring and rejecting the negated concept. To expose this critical gap, we construct COCO-Confusion, a retrieval benchmark specifically designed to test whether a method can reject confusing images that contain the negated concept. Construction. We build COCO-Confusion from COCO val2017 using bounding box annotations. For each concept pair (c+,c−)(c^+,c^-), we partition the annotated images into two groups: correct images containing c+c^+ but not c−c^-, and confusing images containing both c+c^+ and c−c^-. We consider two settings based on query complexity: COCO-Confusion-2C uses one affirmed and one negated concept (e.g., “A car is present but no traffic light”; 950 pairs), and COCO-Confusion-3C uses two affirmed concepts and one negated concept (e.g., “An apple and banana are present but no bowl”; 1,740 pairs). Each group is capped at 50 images per pair to ensure balanced galleries. We verify all images using OWLv2 [16], an open-vocabulary object detector, to filter annotation noise where the negated concept is heavily occluded, truncated, or visually ambiguous in the bounding-box annotations, removing pairs where the detector cannot confirm concept presence. (details in Appendix D) The final benchmark contains 2,690 verified pairs across 4,708 unique images. Evaluation metric. For each pair, the gallery consists of its correct and confusing images. We report mean Average Precision (mAP), which measures how well a method ranks correct images above confusing ones. High mAP indicates successful rejection of confusing images beyond dilution. Why COCO-Confusion matters. On existing retrieval benchmarks, Embedding Correction method achieves near-optimal performance because confusing images are rare in the gallery. On COCO-Confusion, every gallery is designed to contain confusing images, making it the first benchmark that explicitly measures negation rejection rather than dilution. 5.2 Experimental setup Backbones. We evaluate on 9 pretrained VLMs spanning three model families and diverse scales: CLIP (ViT-B/32, L/14, H/14 [4]; LAION-2B [23]), EVA-02 [5] (ViT-B/16, L/14; Merged-2B), and SigLIP/SigLIP2 (ViT-B/16, B/32, L/16, SO400M; WebLI with sigmoid loss). All weights are public and frozen. Evaluation tasks. We evaluate on three complementary tasks. (1) MCQ: Given an image and four caption options, select the correct one. We report accuracy on COCO MCQ and VOC2007 MCQ from NegBench. This measures negation comprehension. (2) Standard negation retrieval: Given a negated query, retrieve matching images from a large gallery. We report Recall@5 on COCO and MSRvtt [27] negated retrieval from NegBench. This measures negation handling on natural galleries where confusing images are rare. (3) COCO-Confusion retrieval: Given a negated query, retrieve correct images from a gallery containing both correct and confusing images. We report mAP on COCO-Confusion-2C and COCO-Confusion-3C setting. This measures negation rejection. Qualitative retrieval examples are provided in Appendix. Baselines. We compare against Raw VLM, which uses standard cosine similarity with no negation processing, and Embedding Correction, a training-free method that removes the negated concept direction from the text embedding. Since Embedding Correction reports results only for CLIP ViT-B/32 and does not release code, we use their reported numbers for this backbone. For CLIP ViT-B/32, we additionally compare against NegCLIP and NegationCLIP. Table 1: Negation performance on CLIP ViT-B/32 [19]. We compare against training-based negation methods [28, 18] and training-free embedding correction method (EC) [1] across MCQ, Retrieval, and our COCO-Confusion benchmark. (*) indicates models fine-tuned on C12M-NegFull [2]. Best in bold; second-best underlined. MCQ Retrieval COCO VOC2007 MSRvtt COCO MSRvtt Conf-2C Conf-3C Model R@5 R-Neg@5 R@5 R-Neg@5 mAP mAP CLIP [19] 24.4 27.2 30.1 65.3 57.9 55.7 48.8 63.2 60.5 NegCLIP [28] 28.5 30.4 28.4 69.5 65.2 57.0 51.6 64.4 61.4 CLIP* (NegBench) [2] 54.4 54.9 45.3 54.9 52.3 48.8 43.1 68.8 65.0 NegationCLIP [18] 35.9 44.2 37.4 62.8 62.0 52.5 49.7 72.9 68.4 EC [1] 70.0 75.5 51.6 65.2 58.3 55.7 51.0 73.2 70.8 KTN (Ours) 79.5 92.1 66.9 65.2 62.0 55.7 51.9 82.8 75.2 Table 2: Negation performance across multiple VLM backbones [29, 19, 26, 5]. Each backbone shows three rows: Vanilla VLM, embedding correction (EC) [1], and our KTN. Bold indicates best-per-metric across the three methods. KTN consistently outperforms both vanilla and embedding correction on MCQ comprehension and Confusion retrieval, while preserving standard retrieval R@5. MCQ Retrieval COCO VOC2007 MSRvtt COCO MSRvtt Conf-2C Conf-3C Model Backbone R@5 R-Neg@5 R@5 R-Neg@5 mAP mAP Vanilla CLIP [19] ViT-L/14 25.4 25.3 30.9 70.8 64.1 60.4 55.0 62.1 59.4 EC [1] 48.4 45.3 41.9 70.8 66.6 60.4 55.9 73.8 67.4 KTN (Ours) 79.6 92.4 69.1 70.8 67.8 60.5 56.5 82.4 74.8 Vanilla CLIP [19] ViT-H/14 30.2 31.9 29.9 73.2 66.8 62.6 55.0 62.0 59.3 EC [1] 53.9 49.9 43.3 73.2 68.5 62.6 57.8 72.2 66.1 KTN (Ours) 79.4 92.2 68.1 73.2 69.9 62.6 59.2 81.6 74.0 Vanilla EVA-02 [5] ViT-B/16 30.2 27.8 29.2 66.6 61.2 58.7 51.1 62.3 58.7 EC [1] 44.3 40.0 43.5 66.6 63.5 58.7 53.8 76.1 68.5 KTN (Ours) 77.1 91.2 63.8 66.6 64.4 58.7 53.5 84.4 76.2 Vanilla EVA-02 [5] ViT-L/14 31.5 31.9 30.4 70.9 65.5 62.8 56.9 61.1 57.9 EC [1] 38.3 25.4 39.7 70.9 67.7 62.8 58.2 77.4 68.1 KTN (Ours) 79.7 92.4 65.9 70.9 68.5 62.8 58.4 84.6 76.2 Vanilla SigLIP [29] ViT-B/16 28.8 31.1 32.2 72.3 65.0 56.1 49.7 62.3 60.2 EC [1] 56.3 77.1 51.6 72.3 68.6 56.1 50.5 79.2 72.2 KTN (Ours) 71.1 82.9 59.2 72.3 69.6 56.1 51.5 85.5 78.1 Vanilla SigLIP2 [26] ViT-B/32 27.1 27.3 28.2 73.2 64.8 56.9 51.3 61.8 60.2 EC [1] 63.3 78.2 51.2 73.2 68.4 56.9 49.2 81.4 74.4 KTN (Ours) 78.5 93.4 61.4 73.2 69.5 56.9 50.5 86.7 79.2 Vanilla SigLIP2 [26] ViT-L/16 25.2 24.1 28.8 77.1 71.0 64.1 56.8 61.1 59.1 EC [1] 57.7 71.8 53.9 77.1 74.0 64.1 58.0 78.6 71.4 KTN (Ours) 80.5 94.6 64.6 77.1 75.3 64.1 58.4 87.0 79.6 Vanilla SigLIP2 [26] So400m 23.9 23.8 28.9 78.0 71.5 64.9 58.0 61.5 59.8 EC [1] 55.1 71.1 54.0 78.0 74.6 64.9 59.5 78.6 69.8 KTN (Ours) 81.0 94.6 65.7 77.9 75.9 64.9 59.6 87.1 79.3 5.3 Main results MCQ. Our method substantially improves negation comprehension across all MCQ benchmarks. On CLIP ViT-B/32 (Table 1), we achieve 79.5% on COCO, 92.1% on VOC2007, and 66.9% on MSRvtt, improving over Embedding Correction by +9.5, +16.6, and +15.3 points respectively. Both training-based baselines fall far behind: NegCLIP and NegationCLIP show only marginal gains over the raw baseline despite requiring retraining. These improvements generalize consistently across all backbones (Table 2): MCQ accuracy improves by +42 to +57 points on COCO, +51 to +71 points on VOC2007, and +31 to +37 points on MSRvtt relative to the raw baseline. The gains are consistent regardless of model family (CLIP, EVA-02, SigLIP, SigLIP2) or scale (ViT-B to SO400M). Retrieval. On standard negation retrieval, where confusing images are rare in the gallery, our method preserves R@5 and consistently improves R-Neg@5. On CLIP ViT-B/32, R@5 remains at 65.2 while R-Neg@5 improves from 58.3 to 62.0 (+3.7). Across all backbones, R-Neg@5 gains range from +1.5 to +4.7 on COCO and +1.5 to +4.2 on MSRvtt, with no degradation in R@5. COCO-Confusion. The largest gains appear on COCO-Confusion, the benchmark designed to test negation rejection rather than dilution. On CLIP ViT-B/32, our method achieves 82.8% mAP on Conf-2C and 75.2% on Conf-3C, improving over Embedding Correction by +9.6 and +4.4 points. Across all backbones, the pattern is striking: raw baselines cluster around 61–65% mAP regardless of model scale, confirming that larger models do not resolve negation failure. Embedding Correction provides only modest improvement. Our method consistently reaches 81–87% on Conf-2C and 74–79% on Conf-3C, with the best result of 87.1% mAP on SigLIP2 SO400M. This confirms that the image-side rejection dimension successfully separates correct from confusing images, precisely the gap that single-embedding scoring cannot close. 5.4 Ablation study Table 3: Ablation study (SigLIP2 SO400M). Dimension 1 alone (Proj) achieves dilution. Dimension 2 provides rejection. Method COCO MCQ R-Neg@5 Conf-2C SigLIP2 23.9 71.5 61.5 + Proj (Dim 1) 55.1 74.6 78.6 + Ours (Dim 1+2) 81.0 75.9 87.1 Table 3 isolates the contribution of each scoring dimension on SigLIP2 SO400M. Dimension 1 alone (Proj) already yields a large gain over Raw on MCQ (+31.2p) and COCO-Confusion (+17.1p), confirming that dilution is effective when the task requires distinguishing affirmative from negated captions. However, adding Dimension 2 KTN produces a further +25.9p on MCQ and +8.5p on COCO-Confusion, gains that are inaccessible to any single-embedding method by Theorem 1. On standard negation retrieval (R-Neg@5), the gap is small (+1.3p), consistent with the observation that confusing images are rare in unfiltered galleries and dilution alone suffices. The ablation confirms that the dominant source of improvement is the image-side rejection dimension, precisely the signal that the dilution ceiling forbids. 6 Related work Negation failure in VLMs. A growing body of work documents that VLMs systematically fail on negation. The ARO benchmark [28] showed that CLIP behaves as a bag-of-words and largely ignores word order and negation. NegBench [2] evaluated multiple VLMs across 18 negation tasks and found near-chance performance, with CLIP scoring 82% on affirmative queries but only 3% on negated ones for VOC2007. NegationCLIP [18] traced this failure to a training data shortfall and reported that only 0.7% of LAION-400M captions contain any negation word. DCSM [10] provides a theoretical account and argues that no CLIP-like joint embedding space can support both basic content representation and negation simultaneously. Approaches for improving negations awareness. Data-centric methods augment training data with negation captions and retrain the model. ConCLIP [24] constructed 228K negation triples and fine-tuned with negated-caption distractors. NegationCLIP [18] generated visually grounded negation captions through LLM and MLLM pipelines. However, these approaches require retraining and show limited generalization. On the training-free side, the embedding correction method [1] showed that CLIP text embeddings follow compositional arithmetic and proposed a rule-based projection that extracts negated concepts and removes their semantic direction from the embedding. This is effective at reducing negation bias but limited to dilution rather than rejection. We go beyond this limitation by adding a separate dimension that actively penalizes the negated concept’s visual presence. 7 Conclusion Representing negation within a single embedding vector is fundamentally limited. Even with perfect training data and infinite capacity, the non-negative nature of visual concept activations drives the text coefficient on negated concepts to zero, making dilution the theoretical ceiling of single-embedding scoring. We proposed Know-Thy-Not (KTN), a training-free multi-dimensional scoring method that breaks this ceiling by separating affirmation and rejection into independent dimensions. Combined with COCO-Confusion, a benchmark designed to measure rejection beyond dilution, experiments across nine backbones confirm consistent improvements in MCQ, negation retrieval, and confusion rejection. Beyond performance gains, COCO-Confusion makes the distinction between dilution and rejection measurable for the first time, revealing a gap that prior benchmarks could not expose. Limitations KTN’s image-side rejection relies on cosine similarity, which can yield high scores for c−c^- even when it is absent from the image. We mitigate this with adaptive thresholding using a per-image reference τ(i)τ(i) (Eq. 14), but it does not fully eliminate false positives. Our method targets dual-encoder VLMs (CLIP, SigLIP) that score via a single dot product. Extending to generative VLMs like LLaVA [14, 13] is a natural next step. Their scoring is conditioned on autoregressive decoding rather than embedding similarity, so the text coefficient βc− _c^- analyzed in our impossibility result does not directly apply. Still, the broader principle that rejection requires a scoring signal beyond a single embedding axis likely transfers, and we expect LLaVA-style architectures to also benefit from a multi-dimensional formulation that rejects the negated concept rather than merely diluting it. References [1] B. Aggarwal, A. More, M. Soni, and S. D. Bhat (2026) Seeing what’s not there: negation understanding needs more than training. In International conference on learning representations (ICLR), Cited by: Appendix C, Figure 1, Figure 1, §1, §1, Figure 2, Figure 2, Figure 2, §3.1, Table 1, Table 1, Table 1, Table 2, Table 2, Table 2, Table 2, Table 2, Table 2, Table 2, Table 2, Table 2, Table 2, §6. [2] K. Alhamoud, S. Alshammari, Y. Tian, G. Li, P. H. Torr, Y. Kim, and M. Ghassemi (2025) Vision-language models do not understand negation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 29612–29622. Cited by: §A.2, §1, §3.1, Table 1, Table 1, Table 1, §6. [3] M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev (2023) Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), p. 2818–2829. Cited by: §1. [4] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. (2021) An image is worth 16x16 words: transformers for image recognition at scale. International conference on learning representations (ICLR). Cited by: §5.2. [5] Y. Fang, Q. Sun, X. Wang, T. Huang, X. Wang, and Y. Cao (2024) Eva-02: a visual representation for neon genesis. Image and Vision Computing 149, p. 105171. Cited by: §5.2, Table 2, Table 2, Table 2, Table 2. [6] M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd (2020) spaCy: Industrial-strength Natural Language Processing in Python. External Links: Document Cited by: §4.1. [7] C. Hsieh, J. Zhang, Z. Ma, A. Kembhavi, and R. Krishna (2023) Sugarcrepe: fixing hackable benchmarks for vision-language compositionality. Advances in neural information processing systems (NeurIPS) 36, p. 31096–31116. Cited by: §1. [8] C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. Le, Y. Sung, Z. Li, and T. Duerig (2021) Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning (ICML), p. 4904–4916. Cited by: §1. [9] A. Kamath, J. Hessel, and K. Chang (2023) What’s “up” with vision-language models? investigating their struggle with spatial reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 9161–9175. Cited by: §1. [10] R. Kang, Y. Song, G. Gkioxari, and P. Perona (2025) Is CLIP ideal? no. can we fix it? yes!. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 22436–22446. Cited by: §6. [11] J. Li, D. Li, C. Xiong, and S. Hoi (2022) Blip: bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning (ICML), p. 12888–12900. Cited by: §1. [12] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft coco: common objects in context. In European conference on computer vision (ECCV), p. 740–755. Cited by: §5.1. [13] H. Liu, C. Li, Y. Li, and Y. J. Lee (2024) Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), p. 26296–26306. Cited by: Limitations. [14] H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023) Visual instruction tuning. Advances in neural information processing systems (NeurIPS) 36, p. 34892–34916. Cited by: Limitations. [15] Z. Ma, J. Hong, M. O. Gul, M. Gandhi, I. Gao, and R. Krishna (2023) Crepe: can vision-language foundation models reason compositionally?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 10910–10921. Cited by: §1. [16] M. Minderer, A. Gritsenko, and N. Houlsby (2023) Scaling open-vocabulary object detection. Advances in Neural Information Processing Systems (NeurIPS) 36, p. 72983–73007. Cited by: §5.1. [17] L. Parcalabescu, M. Cafagna, L. Muradjan, A. Frank, I. Calixto, and A. Gatt (2022) VALSE: a task-independent benchmark for vision and language models centered on linguistic phenomena. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), Vol. 1, p. 8253–8280. Cited by: §1. [18] J. Park, J. Lee, J. Song, S. Yu, D. Jung, and S. Yoon (2025) Know" no" better: a data-driven approach for enhancing negation awareness in clip. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 2825–2835. Cited by: §1, §3.1, Table 1, Table 1, Table 1, §6, §6. [19] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning (ICML), p. 8748–8763. Cited by: §1, §1, §2, Table 1, Table 1, Table 1, Table 2, Table 2, Table 2, Table 2. [20] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen (2022) Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1 (2), p. 3. Cited by: §1. [21] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022) High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), p. 10684–10695. Cited by: §1. [22] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. (2022) Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems (NeurIPS) 35, p. 36479–36494. Cited by: §1. [23] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. (2022) Laion-5b: an open large-scale dataset for training next generation image-text models. Advances in neural information processing systems (NeurIPS) 35, p. 25278–25294. Cited by: §5.2. [24] J. Singh, I. Shrivastava, M. Vatsa, R. Singh, and A. Bharati (2024) Learn" no" to say" yes" better: improving vision-language models via negations. arXiv preprint arXiv:2403.20312. Cited by: §1, §6. [25] T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross (2022) Winoground: probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 5238–5248. Cited by: §1. [26] M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafa, et al. (2025) Siglip 2: multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786. Cited by: §2, Table 2, Table 2, Table 2, Table 2, Table 2. [27] J. Xu, T. Mei, T. Yao, and Y. Rui (2016) Msr-vtt: a large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), p. 5288–5296. Cited by: §5.2. [28] M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou (2023) When and why vision-language models behave like bags-of-words, and what to do about it?. International conference on learning representations (ICLR). Cited by: Table 1, Table 1, Table 1, §6. [29] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer (2023) Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision (ICCV), p. 11975–11986. Cited by: §1, §1, §2, Table 2, Table 2, Table 2. [30] X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer (2022) Lit: zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), p. 18123–18133. Cited by: §1. Appendix A Theoretical details A.1 Per-image budget compression For each image i, let (i):=c:αc(i)>0P(i):=\c: _c(i)>0\ and (i):=c:αc(i)=0N(i):=\c: _c(i)=0\, with K(i):=|(i)|≪||K(i):=|P(i)| |C|. That is, (i)P(i) is the set of concepts visually present in image i, (i)N(i) is the set of concepts absent from it, and K(i)K(i) is the number of present concepts, which is typically much smaller than the total concept vocabulary |||C|. Recall that for each concept c, we define the text coefficient βc(q):=f(q)⊤ec _c(q):=f_T(q) e_c, which measures how much the text embedding of query q activates the direction of concept c. Suppose the text query assigns uniform affirmed-concept margin γ+>0 _+>0 across present concepts and uniform negated-concept margin γ−>0 _->0 across absent concepts: βc+(q)≥γ+∀c∈(i),βc−(q)≤−γ−∀c∈(i). _c^+(q)≥ _+\;\;∀ c (i), _c^-(q)≤- _-\;\;∀ c (i). (15) The unit-norm constraint ∑cβc(q)2≤1 _c _c(q)^2≤ 1 then implies K(i)γ+2+(||−K(i))γ−2≤1,K(i) _+^2+(|C|-K(i)) _-^2≤ 1, (16) and consequently |γ−|≤1−K(i)γ+2||−K(i)=(1||).| _-|≤ 1-K(i) _+^2|C|-K(i)=O\! ( 1 |C| ). (17) Equation (17) complements Corollary 1: the conclusion βc−∗(q)=0β^*_c^-(q)=0 arises both from optimization (the unit-norm maximizer of in-kind alignment) and from representation geometry (no unit vector can assign uniform negated concept margin across the dense set of absent concepts). Per-concept negated concept margin is compressed at rate (1/||)O(1/ |C|) regardless of the training objective. A.2 Hard-negative supervision and frequency collapse We approximate the text encoder as token-additive: f(q)≈∑t∈tokens(q)Wtf_T(q)≈ _t (q)W_t. For negation queries q=q= “c+c^+ without c−c^-” sampled uniformly over (c+,c−)∈2(c^+,c^-) ^2, the per-step participation rates are • “without” cue token: rate 11 (appears in every negation example); • concept token c: rate 1/||1/|C| in the negated position, dominantly trained by affirmation contexts elsewhere. The learned embedding is therefore biased toward f(“c+ without c−”)≈c+⏟affirm+generic⏟query-indep. offset+(1/||)⏟concept-specific.f_T(``c^+ without c^-′)\;≈\; e_c^+_affirm+ n_generic_query-indep. offset+ O(1/|C|)_concept-specific. (18) Generic offset is not concept-specific rejection. Although generic n_generic may yield βc−(q)<0 _c^-(q)<0, it fails to distinguish the negated concept c−c^- from any unmentioned-but-allowed concept r. Faithful retrieval requires βc−(q)<βr(q) _c^-(q)< _r(q) for all such r, but a query-independent offset penalizes c−c^- and r identically. This is empirically confirmed by NegBench’s PCA visualization of ConCLIP, where ‘not(c)’ embeddings collapse onte a single point regardless of c. [2] Consequently, the conventional dual-encoder VLMs exhibit global conservatism, diluting the negated concepts rather than specifically rejecting them. Appendix B Implementation details B.1 Hyperparameter settings KTN uses one hyperparameters, all fixed across all backbones and task families. We do not tune any hyperparameter per backbone or per dataset. Table 4: KTN hyperparameters. All values fixed across backbones and tasks. N denotes the candidate-pool size for the task (N=4N=4 for MCQ, N=||N=|G| for retrieval). Symbol Value Description κ 50 Sigmoid sharpness in pc−(i)p_c^-(i) wnw_n 1/N1/ N Rejection weight, scaled by candidate-pool size B.2 Concept embedding Each concept c in the vocabulary C is encoded as the L2-normalized average of four prompt templates: c=normalize(14∑j=14normalize(f(Tj(c)))),e_c\;=\; normalize\! ( 14 _j=1^4normalize\! (f_T(T_j(c)) ) ), (19) with templates T1(c)=T_1(c)=“a photo of a c”, T2(c)=T_2(c)=“a picture of a c”, T3(c)=T_3(c)=“an image showing a c”, and T4(c)=T_4(c)=“a c in a photo”. B.3 Negation parsing We use spaCy’s en_core_web_sm for dependency parsing. A token is marked as a negation cue if its dependency arc is neg or its lemma is in the cue set: ⬇ NEG_CUES = "no", "not", "without", "nothing", "nowhere", "none", "never", "absent", "missing", "free of" The head noun of the dependency subtree rooted at the cue is taken as the negated concept c−c^-. Remaining noun phrases attached to the sentence root populate the affirmed concept set +C^+. The negation-stripped text estripe_strip used in t~ t’s multi-view averaging is constructed by removing all tokens in the negation subtree and re-joining with normalized whitespace and trailing punctuation. Example. For the query “A car is present but no traffic light”: • Negation cue: no ⇒ subtree covers “no traffic light” • c−=c^-= “traffic light” • +=“car”C^+=\`` car′\ • estripe_strip encodes “A car is present” B.4 Image and video preprocessing We follow the standard CLIP preprocessing pipeline (resize, center-crop, normalize) at the backbone-specific input resolution (224 for B/32, 336 for L/14, 384 for SigLIP2 SO400M, etc.). For each image x, we encode both the original and its horizontal mirror, and use the L2-normalized mean of the two feature vectors: fℐ(x)=normalize(12(fℐenc(x)+fℐenc(flip(x)))).f_I(x)\;=\; normalize\! ( 12 (f_I^enc(x)+f_I^enc( flip(x)) ) ). (20) For MSR-VTT video retrieval, we extract four evenly-spaced frames per video (at quartile time positions) using decord.VideoReader. Each frame is encoded, and we use the L2-normalized mean of the four normalized frame features: fℐ(v)=normalize(14∑k=14normalize(fℐenc(vk))).f_I(v)\;=\; normalize\! ( 14 _k=1^4 normalize~\! (f_I^enc(v_k) ) ). (21) Appendix C Embedding correction reimplementation Aggarwal et al. [1] report results only for CLIP ViT-B/32 and do not release code. We re-implement their method following the procedure described in their paper: the negated concept direction is projected out from the query embedding with λ=1.9λ=1.9, anchored to a neutral text embedding as described in their work. Table 2 compares Embedding Correction and our method across all backbones. Appendix D COCO-Confusion: Construction details This appendix details the construction of COCO-Confusion, our benchmark designed to evaluate whether a VLM can reject a negated concept rather than merely dilute it. While Section 5 introduces the benchmark, here we provide the full pipeline, statistics, and qualitative examples. D.1 Motivation: Why a new benchmark is neeeded Existing negation retrieval benchmarks (e.g., NegBench) evaluate retrieval over a large unfiltered gallery (e.g., 5,000 COCO images). In such pools, the negated concept rarely co-occurs with the affirmed concept in the same image. As a result, even simple embedding correction-based methods perform competitively because there are few confusing images to distinguish from correct ones. This makes existing benchmarks inadequate for measuring rejection: without confusing distractors, dilution suffices. COCO-Confusion is constructed precisely to expose this gap. For every query, the gallery is composed of images that all contain the affirmed concept, partitioned into: • Correct images: contain c+c^+ but not c−c^- • Confusing images: contain c+c^+ and c−c^- A method that merely dilutes c−c^- scores both groups similarly. Only a method that genuinely rejects c−c^- separates them. D.2 Concept pair selection We use the 80 object categories of COCO from the val2017 split, which contains 5,000 images with bounding-box annotations. For each image we extract its set of present categories from the official annotations. Two-object pairs. For every ordered pair (c+,c−)(c^+,c^-) with c+≠c−c^+≠ c^-, we partition images that contain c+c^+ into: ℐ+(c+,c−) ^+(c^+,c^-) =i:c+∈i∧c−∉i, =\\,i:c^+∈ i\, \,c^-∉ i\,\, ℐ±(c+,c−) ^±(c^+,c^-) =i:c+∈i∧c−∈i. =\\,i:c^+∈ i\, \,c^-∈ i\,\. We retain a pair only if |ℐ+|≥Mmin|I^+|≥ M_ and |ℐ±|≥Mmin|I^±|≥ M_ with Mmin=10M_ =10. From each subset we sample up to Mmax=50M_ =50 images uniformly at random (seed 42). The query template is: “A c+c^+ is present but no c−c^-.” Three-object pairs. For every triple of distinct categories (a,b,c)(a,b,c) with at least MminM_ images containing all three, we generate three query partitions: ((a,b),c)((a,b),c), ((a,c),b)((a,c),b), and ((b,c),a)((b,c),a), each treating two as affirmed and one as negated. Selection criteria are identical to the 2-object case applied to the partitioned subsets. The query template is: “A c1+c^+_1 and c2+c^+_2 are present but no c−c^-.” Coverage. The selection procedure covers 76 of 80 COCO categories in 2-object pairs and 62 of 80 in 3-object pairs (Table 5). Categories absent from the benchmark are those with insufficient co-occurrence frequency to satisfy Mmin=10M_ =10 in matched-pair construction (e.g., toaster, hair drier). Table 5: Concept coverage in COCO-Confusion. 2-object 3-object Total pairs (post-verification) 950 1,740 Unique affirmed concepts 75 62 Unique negated concepts 76 62 COCO categories used 76/80 62/80 D.3 OWLv2 verification COCO bounding-box annotations are accurate but may have boundary cases (heavy occlusion, partial visibility). To ensure clean labels, we re-verify every image–concept pair using OWLv2, an open-vocabulary object detector. We use the public checkpoint google/owlv2-base-patch16-ensemble with text queries of the form “a photo of a c.” For each image i and concept set q=c+∪c−C_q=\c^+\∪\c^-\, OWLv2 produces a maximum detection confidence scorec(i)∈[0,1]score_c(i)∈[0,1] per concept. We retain images according to the following rules at threshold τOWL=0.1 _OWL=0.1: • Correct image kept iff maxc∈+scorec(i)≥τOWL _c ^+score_c(i)≥ _OWL and scorec−(i)<τOWLscore_c^-(i)< _OWL. • Confusing image kept iff maxc∈+scorec(i)≥τOWL _c ^+score_c(i)≥ _OWL and scorec−(i)≥τOWLscore_c^-(i)≥ _OWL. • Pair kept iff ≥5≥ 5 verified images remain in both subsets. The low threshold τOWL=0.1 _OWL=0.1 favors recall, reducing the risk of falsely classifying a partially-visible c−c^- object as “absent.” Retention rates. Pair-level retention is high: 99.9% (2-object) and 97.1% (3-object), consistent with COCO annotations being already accurate. Filtering primarily refines the per-pair image lists rather than discarding pairs. Table 6: COCO-Confusion final statistics. 2-object 3-object Total pairs 950 1,740 D.4 Evaluation protocol For each query, the retrieval gallery is the union of its verified correct and confusing images. We compute mean Average Precision (mAP) over the gallery, with correct images as ground-truth positives. The benchmark-level metric is the macro-average of per-pair mAP across the 950 (Conf-2C) or 1,740 (Conf-3C) pairs. D.5 Qualitative examples Figure 4 shows a 2-object pair (cat / dog). Figure 5 shows a 3-object pair (apple + banana / bowl). In each figure, the top row shows verified correct images (green border, contain the affirmed concept(s) but not the negated concept) and the bottom row shows verified confusing images (red border, contain both the affirmed and negated concepts). Correct: cat present, no dog Confusing: cat present, dog also present Figure 4: 2-object COCO-Confusion example. Query: “a cat is present but no dog.” Top row: correct images, containing a cat but no dog. Bottom row: confusing images, containing both a cat and a dog. All ten images contain a cat—only the negated concept differs across rows. A method that merely dilutes the “dog” direction in the text embedding scores all ten images similarly. Only a method that genuinely rejects c−c^- separates correct from confusing. Correct: apple + banana present, no bowl Confusing: apple + banana present, bowl also present Figure 5: 3-object COCO-Confusion example. Query: “an apple and banana are present but no bowl.” The query requires affirming both apple and banana while rejecting bowl. All shown images contain an apple and a banana. Only the bottom row also contains a bowl. D.6 What makes COCO-Confusion different Compared to NegBench standard retrieval (1 query against a 5,000-image gallery dominated by irrelevant images): 1. Curated galleries with high confusing density. Each pair gallery contains 50–100 images, of which 20–50% contain the negated concept. Standard retrieval has <1%<1\% confusing images per query. 2. OWLv2 cross-validation. Every image label is double-verified by an external open-vocabulary detector, eliminating bounding-box edge cases. Together these properties make COCO-Confusion a stress test for rejection, while standard retrieval benchmarks remain useful for measuring dilution-level negation handling.