Paper deep dive
Looking Beyond the Window: Global-Local Aligned CLIP for Training-free Open-Vocabulary Semantic Segmentation
ByeongCheol Lee, Hyun Seok Seong, Sangeek Hyun, Gilhan Park, WonJun Moon, Jae-Pil Heo
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/26/2026, 1:43:16 AM
Summary
GLA-CLIP (Global-Local Aligned CLIP) is a training-free open-vocabulary semantic segmentation framework that addresses the semantic discrepancy and grid artifacts caused by independent sliding-window inference. It introduces a Key-Value Token Extension to incorporate global context, a Proxy Anchor mechanism to mitigate local window bias, and a dynamic normalization scheme to adapt attention strength based on object scale.
Entities (5)
Relation Signals (3)
GLA-CLIP → extends → CLIP
confidence 98% · GLA-CLIP can be equipped on existing methods and broad their receptive field.
GLA-CLIP → improves → Open-Vocabulary Semantic Segmentation
confidence 95% · Extensive experiments validate the effectiveness of GLA-CLIP in enhancing training-free open-vocabulary semantic segmentation performance.
GLA-CLIP → uses → DINO
confidence 90% · The input image is first divided into overlapping windows and processed using frozen backbones: a Vision Foundation Model (VFM, e.g., DINO) and CLIP.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:A sliding-window inference strategy is commonly adopted in recent training-free open-vocabulary semantic segmentation methods to overcome limitation of the CLIP in processing high-resolution images. However, this approach introduces a new challenge: each window is processed independently, leading to semantic discrepancy across windows. To address this issue, we propose Global-Local Aligned CLIP~(GLA-CLIP), a framework that facilitates comprehensive information exchange across windows. Rather than limiting attention to tokens within individual windows, GLA-CLIP extends key-value tokens to incorporate contextual cues from all windows. Nevertheless, we observe a window bias: outer-window tokens are less likely to be attended, since query features are produced through interactions within the inner window patches, thereby lacking semantic grounding beyond their local context. To mitigate this, we introduce a proxy anchor, constructed by aggregating tokens highly similar to the given query from all windows, which provides a unified semantic reference for measuring similarity across both inner- and outer-window patches. Furthermore, we propose a dynamic normalization scheme that adjusts attention strength according to object scale by dynamically scaling and thresholding the attention map to cope with small-object scenarios. Moreover, GLA-CLIP can be equipped on existing methods and broad their receptive field. Extensive experiments validate the effectiveness of GLA-CLIP in enhancing training-free open-vocabulary semantic segmentation performance. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2603.23030v1
- Canonical: https://arxiv.org/abs/2603.23030v1
Trouble viewing inline? Open PDF directly →
Full Text
70,093 characters extracted from source content.
Expand or collapse full text
Looking Beyond the Window: Global-Local Aligned CLIP for Training-free Open-Vocabulary Semantic Segmentation ByeongCheol Lee, Hyun Seok Seong, Sangeek Hyun, Gilhan Park, WonJun Moon, Jae-Pil Heo Sungkyunkwan University bc7817, gustjrdl95, hsi1032, a01152a, wjun0830, jaepilheo@skku.edu Corresponding author. Abstract A sliding-window inference strategy is commonly adopted in recent training-free open-vocabulary semantic segmentation methods to overcome limitation of the CLIP in processing high-resolution images. However, this approach introduces a new challenge: each window is processed independently, leading to semantic discrepancy across windows. To address this issue, we propose Global-Local Aligned CLIP (GLA-CLIP), a framework that facilitates comprehensive information exchange across windows. Rather than limiting attention to tokens within individual windows, GLA-CLIP extends key-value tokens to incorporate contextual cues from all windows. Nevertheless, we observe a window bias: outer-window tokens are less likely to be attended, since query features are produced through interactions within the inner window patches, thereby lacking semantic grounding beyond their local context. To mitigate this, we introduce a proxy anchor, constructed by aggregating tokens highly similar to the given query from all windows, which provides a unified semantic reference for measuring similarity across both inner- and outer-window patches. Furthermore, we propose a dynamic normalization scheme that adjusts attention strength according to object scale by dynamically scaling and thresholding the attention map to cope with small-object scenarios. Moreover, GLA-CLIP can be equipped on existing methods and broad their receptive field. Extensive experiments validate the effectiveness of GLA-CLIP in enhancing training-free open-vocabulary semantic segmentation performance. Codes are available at github.com/2btlFe/GLA-CLIP Figure 1: Comparison of segmentation consistency near window boundaries. (a) We evaluate segmentation inconsistency using the Boundary Error Rate (BER). BER is defined as the proportion of pixels near adjacent window boundaries where predicted labels differ despite identical ground-truth. ProxyCLIP yields high BER due to its lack of cross-window interaction, whereas BER is significantly reduced in ours by incorporating global context into attention process. More details in Appendix A. (b) ProxyCLIP exhibits grid artifacts (marked with white circles), caused by the limited receptive field within individual windows. In contrast, ours mitigates these artifacts by leveraging contextual information beyond local windows. 1 Introduction Open-vocabulary semantic segmentation (OVSS) aims to assign pixel-level semantic labels from an open and potentially unbounded vocabulary, enabling models to generalize beyond a fixed set of training categories. Recent advances in vision-language models, such as CLIP [34], have made it possible to approach this task without additional training, by leveraging the image-text aligned embedding space. This has led to a growing interest in Training-Free OVSS [53, 3, 42, 23, 18, 24, 38, 28, 50], where segmentation is performed only with minimal modifications to the CLIP architecture or processing pipeline. However, a fundamental limitation remains: CLIP is pretrained on low-resolution inputs (e.g., 224×224), which hinders its applicability to high-resolution segmentation tasks. To address this, recent works adopt a sliding-window strategy [53, 42, 23, 18, 24, 22], where high-resolution images are divided into overlapping crops and processed independently. By extending precise text-aligned embeddings to high-resolution settings without compromising spatial details, this simple approach has led to a significant performance boost. However, we argue that this strategy still has a critical limitation: each window lacks access to the broader scene context, which leads to inconsistent predictions across windows. One clear example of this limitation is illustrated in Fig. 1, where independently processed windows produce inconsistent predictions near window boundaries and introduce grid-like artifacts. Specifically, adjacent windows often assign different semantic labels to neighboring pixels along shared boundaries, even when those pixels belong to the same ground-truth class, resulting in visible discontinuities and structured artifacts. These issues highlight the absence of global context integration and motivate the need for a mechanism that enforces semantic consistency across windows. To address this issue, we propose Global-Local Aligned CLIP (GLA-CLIP), a novel training-free framework that explicitly aligns local and global semantics. We begin by introducing a Key-Value Extension mechanism that allows each query to attend to an unbounded set of key-value tokens across the entire image. This design enables broader context integration, thereby mitigating the semantic discrepancy across independently processed windows. However, we observe that even with globally extended key-value tokens, the attention remains locally biased: query tokens disproportionately attend to patches within the same window while underweighting semantically similar tokens outside the window. This is attributed to the fact that query embeddings are constructed solely from inner-window features, limiting their global awareness. To alleviate this issue, we derive proxy tokens to serve as globally representative queries by aggregating the patches highly similar to the given query across the entire image. These proxies help reduce local bias and encourage semantically aligned attention across windows. Finally, we introduce a dynamic attention normalization strategy that adjusts attention magnitudes based on object scale, estimated from each query’s similarity distribution. This adaptation ensures that small objects are not overwhelmed by irrelevant global tokens, while large objects benefit from richer contextual clues. Our results across multiple benchmarks demonstrate the effectiveness of our proposed approaches in mitigating the semantic discrepancy between different sliding windows. Our contributions are summarized as follows: • To our knowledge, this is the first work to identify and address the challenge stemming from the sliding window: locally bounded semantics in each window incur inconsistent prediction between windows. • We extend the set of key-value tokens to cover the entire image, allowing each query to reason over global context regardless of its local window. • We introduce globally representative proxy queries by aggregating high confidence tokens, enabling semantically consistent attention across windows. • We propose a dynamic attention normalization mechanism that adapts the influence of global tokens based on the estimated object scale, improving robustness across varying object sizes and domains. • Our framework can expand the receptive field of any baseline model without additional training. Figure 2: Overview of our proposed framework. The input image is first divided into overlapping windows and processed using frozen backbones: a Vision Foundation Model (VFM, e.g., DINO) and CLIP. We introduce a Key-Value Token Extension, where VFM features from the current window vfmF_vfm serve as query tokens, while key tokens are gathered from all windows to provide global context. The corresponding value tokens V are extracted from the final transformer layer of CLIP. Cross-attention is then applied, followed by a projection layer to generate the final visual features visualF_visual. To stabilize attention across windows, each query token is replaced with a semantically representative proxy anchor. Finally, a dynamic normalization scheme adjusts attention strength based on object size, approximated by the number of positive samples associated with each proxy anchor. 2 Related Work 2.1 Open-Vocabulary Semantic Segmentation The emergence of pre-trained vision-language models [34, 19] enables aligning text and image features for open-vocabulary semantic segmentation [29, 35]. Early methods follow a two-stage pipeline, where class-agnostic masks are first generated and then matched with text embeddings using models like CLIP [14, 15, 49, 27, 30]. While these methods are effective, they depend on external mask generators that are separate from the vision-language model. This separation can reduce visual-semantic consistency. To address this, single-stage methods [54, 51, 48] have been introduced. These methods directly predict segmentation masks from CLIP features and often use techniques such as attention bias [48] or mask pooling [51]. Other approaches include ODISE [47], which uses frozen diffusion features with a Mask2Former decoder [9], and CAT-Seg [11], which improves alignment by pixel-text similarity and text encoder fine-tuning. More recently, EBSeg [37] enhances generalization by adjusting image-text similarity distribution during training. Meanwhile, training-free approaches have gained attention for utilizing CLIP without requiring further training. 2.2 Training-Free OVSS Training-free open-vocabulary semantic segmentation makes predictions without any additional training or fine-tuning. Instead of introducing learnable parameters, these approaches enhance the segmentation ability of CLIP by modifying its image encoder [26, 43] and leveraging vision foundation models [21, 24]. MaskCLIP [53] enhances spatial sensitivity in CLIP by discarding the query and key embeddings in the final self-attention block, using only the value embeddings. Subsequent methods have modified the attention mechanism by replacing query-key attention with value-value or self-self attention (e.g., query-query, key-key) [26, 3, 42, 38]. In addition, some approaches remove the final feed-forward layer and residual connections, as these components have been found to introduce noise in the resulting segmentation masks [18, 23]. ProxyCLIP [24] introduces a proxy attention mechanism that enhances spatial and semantic alignment in CLIP. However, most methods rely on sliding-window inference due to low training resolution, which makes it difficult for the model to understand the entire image and often leads to inconsistent results across different windows. To address these challenges, our method enhances global context and consistency by introducing key-value token extension and proxy-similarity. In doing so, it effectively mitigates attention imbalance and overcomes the inherent limitations of sliding-window inference. 3 Method 3.1 Preliminary 3.1.1 Open-Vocabulary Semantic Segmentation (OVSS) OVSS leverages the visual-language alignment of CLIP [34]. Given an image X, the CLIP visual encoder converts it as visual tokens visual∈ℝN×DF_visual ^N× D, where N is the number of tokens and D is dimension size. Meanwhile, the text encoder processes a set of text prompts to project them to a text tokens textF_text. The cosine similarity between visual and text tokens determines logits, eventually categorizing each visual token to the text corresponding to the highest logits. Recently, ProxyCLIP [24] has shown notable advances in OVSS by leveraging Vision Foundation Model (VFM) features. We adopt ProxyCLIP as our baseline, which replaces the attention map in the final transformer block of CLIP with a self-similarity map computed from features of a VFM. Specifically, the self-similarity map S is defined as: =vfm⋅vfm⊤∈ℝN×N, =F_vfm·F_vfm ^N× N, (1) where (⋅)(·) indicate matrix multiplication and vfmF_vfm denotes the visual features extracted from a VFM (e.g., DINO [6]), which serves as the query and key embeddings in the attention mechanism. To better integrate the self-similarity map into CLIP for open-vocabulary segmentation, they modulate and threshold the similarity scores to integrate semantically positive tokens. This refinement suppresses irrelevant token interactions and enables the similarity map to function as an effective attention map for segmentation tasks: =γ(−βN2∑i,j[]ij), =γ (S- βN^2 _i,j [S ]_ij ), (2) ℳij _ij =0,Aij≥0−∞,Aij<0, = cases0,&A_ij≥ 0\\ -∞,&A_ij<0, cases (3) =SoftMax(+ℳ), =SoftMax(A+M), (4) where β and γ are fixed shift and scale hyperparameters. The refined attention map Attn is applied to the value tokens ∈ℝN×DV ^N× D from the final transformer block of CLIP. The final output is obtained via a projected matrix multiplication, following prior findings that skip residual and feed-forward layers [23]: visual=Proj(⋅), _visual=Proj(Attn·V), (5) where Proj denotes the projection layer from the last transformer block of CLIP. Figure 3: Visualization of attention maps for an anchor query token. Proxy-based attention enhances focus on semantically relevant regions across both inner- and outer-window areas. The subsequent dynamic normalization further suppresses irrelevant responses, especially from noisy tokens, yielding sharper and more semantically consistent attention distributions. 3.1.2 Sliding-window Since CLIP is pre-trained with a fixed input resolution of 224×224224× 224 pixels, it cannot directly process high-resolution inputs. Conventionally, this limitation is addressed by applying an overlapped sliding-window strategy, where the input image is divided into L sub-windows of CLIP-compatible size. We follow this approach and perform OVSS independently on each sub-window. For overlapping regions, the logits are averaged across windows to produce the final per-patch predictions. 3.2 Key-Value Extension for Contextual Cue While the sliding-window inference strategy is effective for adapting CLIP to high-resolution inputs by preserving its pre-trained resolution, it inherently restricts the model’s receptive field to local window regions. This constrained field of view prevents the model from accessing holistic scene-level context, thereby hindering its ability to aggregate semantically relevant cues across spatially distant regions. The limitation becomes especially pronounced when semantically coherent regions (such as large objects or spatially continuous stuff categories) are partitioned across multiple windows. Such contextual discontinuity often leads to inconsistent or erroneous semantic predictions, as shown in Fig. 1. To enhance contextual awareness in attention computation, we propose a key-value token extension, which expands the attention scope of each window by allowing it to reference visual information from the entire image, not just its local region. Specifically, we collect the visual features extracted from the VFM to serve as global key embeddings, and simultaneously retrieve the corresponding value tokens from the final transformer layer of CLIP. By aggregating these tokens across all L windows, we construct a unified set of global key globalK_global and value tokens globalV_global as follows: global _global =[vfm(1);vfm(2);…;vfm(L)]∈ℝ(LN)×D, =[F_vfm^(1);F_vfm^(2);…;F_vfm^(L)] ^(LN)× D, (6) global _global =[(1);(2);…;(L)]∈ℝ(LN)×D. =[V^(1);V^(2);…;V^(L)] ^(LN)× D. (7) With these global tokens, we perform cross-window attention by computing similarity between the local query tokens from the current window and the aggregated global key tokens, followed by weighted aggregation over the corresponding value tokens: ext _ext =⋅global⊤∈ℝN×(LN), =Q·K_global ^N×(LN), (8) visual _visual =Proj(ext⋅global)∈ℝN×D. =Proj(A_ext·V_global) ^N× D. (9) Here, ∈ℝN×DQ ^N× D denotes the set of query tokens extracted from the vfmF_vfm of the current sliding window. These serve as the local anchors in the attention operation and initiate semantic integration with globally gathered keys and values. 3.3 Proxy Anchor for Stable Attention After applying the key-value token extension, we observe a persistent locality bias: attention is predominantly concentrated on tokens within the same window (inner-window), while downweighting semantically similar tokens from other windows (outer-window), as shown in Fig. 3(a). Notably, even when outer-window tokens correspond to the same spatial location in the original image (i.e., overlapping regions), attention remains biased toward inner-window tokens. This behavior limits the model’s capacity to aggregate consistent semantics across windows. To alleviate this bias, we introduce a proxy-based attention mechanism that replaces each query token with a semantically representative proxy aggregated from tokens with high similarity across all windows. This proxy serves as a stable anchor that reflects the underlying semantics of the region, allowing attention to be allocated based on semantic coherence rather than window-local bias. Consequently, our method promotes more globally consistent attention patterns and enhances cross-window integration. Specifically, for a given query token, we follow [36] to repeatedly compute the mean vector of high-similarity neighbors, gradually converging on a proxy that best represents the shared semantics around the anchor. We define the initial proxy-query i(0)Q_i^(0), which stems from i-th token of local query tokens Q and construct the index set of high-confident tokens i(0)P_i^(0), which indicates a higher cosine similarity compared to a pre-defined threshold ρ as below: i(0) _i^(0) =j|i(0)⋅global,j>ρ,j∈, = \j\, |\,Q_i^(0)·K_global,j>ρ,\ j \, (10) where J denotes the index set comprising all patch features of all windows. Also, the t-th proxy is defined as follows: i(t) _i^(t) =1|i(t−1)|∑j∈i(t−1)global,j. = 1 |P_i^(t-1) | _j _i^(t-1)K_global,j. (11) By repeating T steps of Eq. (10) and (11) for generating refined proxy, we obtain i(T)Q_i^(T). This operation is computed for all query tokens. Using the proxy proxy=⋃i(T)Q_proxy= _iQ_i^(T), proxy anchor-based attention map for the current window is represented as follows: proxy=proxy⋅global⊤∈ℝN×(BN). _proxy=Q_proxy·K_global ^N×(BN). (12) The proxy is located at the center of the high-similarity embedding samples, aggregated from both inner- and outer-window regions. As a result, the proxy inherently balances the contributions of tokens across windows, allowing attention to be allocated more uniformly to semantically relevant regions, as shown in Fig. 3(b). 3.4 Dynamic Normalization across Object Scales Our key-value extension allows each query to attend to tokens from global windows. While this enables broader contextual integration, it also increases the risk of attending to irrelevant or semantically mismatched tokens (i.e., negatives). This issue is strongly influenced by object scale: when a query corresponds to a small object or fine-grained region, the number of relevant positive tokens tends to be low. In such cases, attention is more easily dominated by irrelevant tokens, leading to inaccurate or noisy predictions. To address this, we propose a dynamic normalization strategy that modulates and thresholds the similarity map in a scale-aware manner. The key idea is that when a query is associated with only a few positive tokens, as is often the case with small objects, the attention becomes more prone to being dominated by irrelevant or negative tokens, leading to degraded prediction quality. Our method adaptively tightens the normalization in such cases to suppress the effect of these irrelevant signals. Conversely, when a query corresponds to a large object and is surrounded by many positive tokens, the normalization becomes more permissive to enhance their relative influence in the attention computation. Additionally, the influence of irrelevant tokens increases with the number of outer-window tokens, amplifying noise in the attention computation. To handle this more effectively, our normalization strategy adjusts its scaling behavior based on the number of windows, suppressing the impact of such noise under larger global contexts. Specifically, we introduce two adaptive variables, u and w, as adaptive counterparts to the fixed hyperparameters in ProxyCLIP [24] to modulate normalization accordingly. The attention score for the i-th query anchor is computed as: i _i =i(proxy−NL∑j=1,…,NL[proxy]ij), =w_i (S_proxy- uNL _j=1,...,NL [S_proxy ]_ij ), (13) where u is a shifting variable dependent on the number of windows L, controlling the overall influence of extended tokens. u is defined as: =1+λ1log(1+L), =1+ _1 (1+L), (14) where is a fixed coefficient. As L increases, u grows, resulting in more conservative normalization to suppress noisy token influence. Next, we define the scaling variable iw_i based on the number of high confidence tokens |i||P_i| associated with each query: i=1+λ2|i|. _i=1+ _2|P_i|. (15) A smaller |i||P_i|, often observed in small objects, leads to stronger amplification of the most relevant tokens while suppress the irrelevant tokens, encouraging scale-aware attention allocation. Note that, λ1 _1 and λ2 _2 are shared across all datasets. Finally, replacing A in Eq. (9) with Attn in Eq. (13) yields the final visual tokens for prediction. The masking and softmax operations are then applied, as formulated in Eq. (4). A key distinction of our method is a per-query normalization strategy, compared to the prior approach [24] that applies identical normalization across all windows and tokens. By adapting attention modulation at the level of each anchor token, our formulation enables fine-grained control over the influence of global context. This scale-aware behavior is particularly effective for balancing attention across varying object sizes. Furthermore, Dynamic Normalization copes with object scale variations, which differ among datasets, and thus removes the necessity of dataset-specific hyperparameters unlike recent methods [50, 22, 8]. 4 Experiment Model Train External Model *DS Hyperparameter With Background Without Background Avg V21 PC60 C-Obj V20 PC59 C-Stf City ADE [gray]0.93CLIP-DINOiser Setting [gray]0.93448 Image Resize, 448 Crop size, 224 Stride, Without rename trick† GroupViT [46] ✓ - ✘ 50.4 18.7 27.5 79.7 23.4 15.3 11.1 9.2 29.4 ReCo [40] ✓ - ✘ 25.1 19.9 15.7 57.7 22.3 14.8 21.6 11.2 23.5 CLIP-DIY [44] ✘ - ✘ 59.9 19.7 31.0 79.7 19.8 13.3 11.6 9.9 30.6 TCL [7] ✓ - ✘ 55.0 30.4 31.6 83.2 33.9 22.4 24.0 17.1 37.2 CLIP-DINOiser [45] ✓ DINO ✘ 62.2 32.4 35.0 80.2 35.9 24.6 31.7 20.0 40.3 ProxyCLIP + GLA‡ ✘ DINO ✘ 63.4 35.8 37.3 80.4 39.4 26.3 33.5 19.1 41.9 [gray]0.93ClearCLIP Setting [gray]0.93448 Image Resize, 448 Crop size, 224 Stride, Without rename trick ClearCLIP [23] ✘ - ✘ 51.8 32.6 33.0 80.9 35.9 23.9 30.0 16.7 38.1 ClearCLIP + GLA ✘ - ✘ 55.6 33.2 34.6 79.7 36.5 24.6 32.7 17.4 39.3 [gray]0.93ProxyCLIP Setting [gray]0.93448 Image Resize, 336 Crop size, 112 Stride, With rename trick on Background Classes ProxyCLIP [24] ✘ DINO ✘ 61.3 35.3 37.5 80.3 39.1 26.5 38.1 20.2 42.3 ProxyCLIP + GLA ✘ DINO ✘ 63.2 35.8 37.7 81.5 39.7 26.8 38.5 20.3 42.9 [gray]0.93SCLIP Setting [gray]0.93336 Image Resize, 224 Crop size, 112 Stride, With rename trick SCLIP [42] ✘ - ✘ 59.1 30.4 30.5 80.4 34.2 22.4 32.2 16.1 38.2 SCLIP + GLA ✘ - ✘ 59.8 32.0 32.8 80.5 35.1 24.2 35.1 18.0 39.8 LaVG [20] ✘ - ✘ 62.1 31.6 34.2 82.5 34.7 23.2 26.2 15.8 38.8 NACLIP [18] ✘ - ✘ 58.9 32.2 33.2 79.7 35.2 23.3 35.5 17.4 39.4 ResCLIP [50] ✘ - ✓ 61.1 33.5 35.0 86.0 36.8 24.7 35.9 18.0 41.4 FreeCP [8] ✘ - ✓ 65.8 35.3 37.2 84.3 38.0 24.9 33.3 18.4 42.2 DIH-CLIP [16] ✘ - ✘ 64.2 36.0 37.4 84.9 39.7 24.5 40.2 19.6 43.3 FLOSS [2] ✘ - ✘ - - - 80.2 35.9 23.6 37.0 18.4 - CASS [22] ✘ DINO ✓ 65.8 36.7 37.8 87.8 40.2 26.7 39.4 20.4 44.4 ProxyCLIP + GLA (Best§) ✘ DINO ✓ 66.7 36.3 37.7 84.7 40.2 27.2 41.2 20.5 44.3 ProxyCLIP + GLA ✘ DINO ✘ 66.3 36.1 37.7 84.2 39.9 26.9 40.8 20.0 44.0 Table 1: Open-vocabulary semantic segmentation results on 8 datasets using CLIP ViT-B/16. *DS Hyperparameter: dataset-specific hyperparameters (not used in Ours). †Rename trick: multiple textual prompts per class. ‡GLA: Adaptation for ClearCLIP [23], SCLIP [42], and ProxyCLIP [24]. § Best: optionally uses dataset-specific hyperparameters (u, w), by manual tuning. 4.1 Experimental Settings Datasets and Evaluation Metrics We evaluate the proposed model on eight semantic segmentation benchmarks. We note that five datasets (Pascal VOC21 [17], Pascal Context60 [32], COCO-Object [4], ADE20K [52], and Cityscapes [12]) include an explicit background class, while other datasets (Pascal VOC20 [17], Pascal Context59 [32], and COCO-Stuff 164K [4]) only contain object classes. In the same order, they comprise 21, 60, 81, 150, 19, 20, 59, and 171 semantic classes, respectively. Following standard practice, we report performance using mean Intersection‑over‑Union (mIoU). Implementation Details. Our settings follow existing training-free open-vocabulary semantic segmentation models [45, 23, 24, 42], including text prompt policy, background threshold, image resizing, sliding window crop size, and stride (details in the Appendix B). We use frozen CLIP (ViT-B/16) to extract features and DINO (ViT-B/8) as VFM, similar to ProxyCLIP [24]. Meanwhile, the query and key tokens from CLIP can also be utilized to form attention maps, replacing VFM features. ClearCLIP [23] and SCLIP [42] adopt this strategy, and our framework can similarly support it. However, in this case, a separate query smoothing step and hyperaprater tuning is necessary, as detailed in the Appendix C. All experiments are conducted on a single NVIDIA Titan RTX GPU. Proxy-anchor hyperparameters are set to ρ=0.6ρ=0.6 with 2 proxy generation steps, and Dynamic Normalization uses λ1=0.3 _1=0.3 and λ2=30 _2=30. 4.2 Comparison with State-of-the-Art Methods Quantitative Results. As shown in Tab. 1, we compare our method with existing open-vocabulary semantic segmentation (OVSS) approaches and report its performance under each setting for a fair evaluation. Our method achieves an average mIoU gain of 1.6% under the CLIP-DINOiser setting, demonstrating clear advantages over training-based models. When integrated with ClearCLIP [23], ProxyCLIP [24], and SCLIP [42], it further yields improvements of 1.2%, 0.6%, and 1.6%, respectively, highlighting its adaptability on any baseline model. In contrast to recent training-free OVSS methods [10, 50, 8, 22], which depend on dataset-specific hyperparameters to enhance performance at the cost of generality, our approach attains an average mIoU of 44.0%, outperforming existing models without tuning. When dataset-specific hyperparameters are optionally applied, our model achieves even higher scores, comparable to all baseline methods. These results indicate that our approach is both effective and generalizable, making it readily applicable across diverse datasets. Figure 4: Qualitative results among ProxyCLIP [24], CASS [22], Ours on Pascal VOC21 [17], COCOstuff [4], and Cityscapes [12]. Qualitative Results. Fig. 4 shows qualitative comparisons with recent state-of-the-art methods on Pascal VOC21 [17], COCO-Stuff164k [4], and Cityscapes [12]. Our method generates cleaner segmentation maps with fewer misclassified pixels and effectively suppresses the grid-like artifacts commonly seen in ProxyCLIP [24] and CASS [22]. By leveraging contextual cues from global windows, our model produces more consistent predictions and preserves clear object boundaries. For example, in the rightmost scene in Cityscapes, ProxyCLIP misclassifies the road center as a sidewalk, whereas ours accurately distinguishes the road and sidewalk regions without noisy or inconsistent outputs. 5 Further Analysis KVE Proxy Norm mIoU Fixed Best Dyna (a) 30.8 (b) ✓ ✓ 43.1 (c) ✓ ✓ 43.0 (d) ✓ ✓ ✓ ✓ 44.3 (e) ✓ ✓ ✓ ✓ 44.0 Table 2: Ablation study of each component in our method. KVE: Key-Value Extension, Proxy: Proxy Anchor-based Attention, Norm: Normalization (Best, Dynamic). Dataset-specific tuning is applied for ‘Best’ normalization configuration, with detailed results in Tab. 1. 5.1 Ablation Study To validate the effectiveness of each component, we conduct an ablation study based on five experimental configurations, as summarized in Tab. 2. We begin with a baseline (a) that computes attention maps using features from DINO, restricted to inner-window tokens only, without any form of normalization. Comparing (a) and (b), we observe a clear performance improvement when applying key-value extension along with dynamic normalization. This demonstrates the importance of incorporating global context and stabilizing the attention computation. Next, comparing (b) and (e), the introduction of proxy-based attention in (e) leads to an additional performance gain. This improvement stems from the proxy’s ability to produce more semantically grounded and stable attention distributions, especially in outer-window regions. We also compare (c) and (e) to isolate the contribution of the key-value extension. The results show that incorporating key-value tokens from outer windows yields a performance improvement, confirming that these extended tokens provide valuable contextual cues for disambiguating local predictions. Finally, experiment (d) reports results using dataset-specific hyperparameters, which were searched to maximize performance for each dataset. When comparing (d) and our method (e), we find that the performance gap is minimal, indicating that our adaptive normalization approach effectively generalizes across datasets without the need for manual tuning. 5.2 Effect of Attention Stabilization We visualize the effect of each method on attention maps in Fig. 3. In Fig. 3(a), the attention maps after applying Key-Value Extension show that attention initially concentrates on a few tokens within the inner-window region, indicating that the model struggles to distribute attention across object boundaries and capture object-level semantics beyond individual windows. Over time, the strong attention weights originally confined to inner-window positives gradually spread to outer regions, leading to a holistic, object-level attention allocation. In Fig. 3(b), after applying Proxy-Attention, attention extends across the entire object regardless of window boundaries, showing that proxy queries effectively stabilize attention. Subsequently, in Fig. 3(c), introducing Dynamic Normalization suppresses irrelevant tokens and focuses attention on meaningful object regions, highlighting its role in refining attention distribution. 5.3 Object Scale via high confidence Tokens We conduct experiments to validate the assumption that high confidence tokens are correlated with object scale in Dynamic Normalization. During proxy construction, we use the number of high confidence tokens, |i||P_i|, as an estimate of object scale and incorporate it into the attention scaling factor, as defined in Eq. 15. In Fig. 5, we analyze whether the number of high confidence tokens aligns with the ground-truth object scale on the Cityscapes [12], which contains objects of diverse sizes. After key-value extension, we measure both the number of patch tokens (#Pos) and high confidence tokens (#High confidence) for each class per image, and report the class-wise average attention scaling factor w, which is inversely proportional to #High confidence. We observe consistent trends across all three metrics: large objects (e.g., Road and Terrain) exhibit a greater number of high confidence tokens and correspondingly lower attention scaling factors, whereas small objects (e.g., Person and Rider) contain fewer tokens and thus receive higher attention scaling. Figure 5: Class-wise object scale and 1/w in Cityscapes Figure 6: Hyperparameter sensitivity experiment. 5.4 Hyperparameter Sensitivity We leverage four hyperparameters, ρ, T, λ1 _1, and λ2 _2, and conduct a sensitivity analysis to examine their effect (Fig. 6). Firstly, ρ controls the selection of high confidence tokens in the cosine similarity scale. If ρ is too low, negatives may be fused with positives, causing semantic collapse in proxy anchors and degrading performance, while too high ρ prevents proxies from aggregating sufficient positives, also reducing performance. The repetition step T in proxy anchor construction slightly affects performance, as a higher T can lead to semantic collapse. In Dynamic Normalization, λ1 _1 determines the attention masking threshold. A low λ1 _1 lowers the threshold, causing fewer tokens to be masked and more dispersed attention, whereas a higher λ1 _1 increases masking and helps to focus attention effectively. In contrast, λ2 _2 has minimal impact on performance, revealing its robustness. Overall, all hyperparameters exhibit a broad range to consistently outperform the baseline model. CLIP Model V21 PC59 C-Stf Avg. ViT-B/32 ProxyCLIP 57.3 35.2 23.6 38.7 CASS 58.2 36.5 24.4 39.7 !15Ours gray!1561.8 gray!1537.1 gray!1525.2 gray!1541.4 ViT-L/14 ProxyCLIP 60.7 38.2 26.2 41.7 CASS 62.1 39.1 26.3 42.5 !15Ours gray!1563.5 gray!1539.1 gray!1526.7 gray!1543.1 Table 3: Segmentation results of different CLIP backbones. VFM Model V21 PC59 C-Stf Avg. DINOv2 [33] ProxyCLIP 59.6 37.3 25.3 40.7 !15+GLA gray!1560.8 gray!1537.8 gray!1525.7 gray!1541.4 DINOv3 [41] ProxyCLIP 62.8 38.8 26.1 42.6 !15+GLA gray!1563.6 gray!1538.9 gray!1526.3 gray!1542.9 Table 4: Segmentation results of different VFM backbones. 5.5 Generalization Across Backbones To assess the generalizability of our framework across varying CLIP backbone capacities, we evaluate performance using ViT-B/32 and ViT-L/14. As shown in Tab. 3, ProxyCLIP and CASS exhibits a substantial performance drop of 4.4%, 4.5% in average mIoU when switching to the lower capacity ViT-B/32. In contrast, our method incurs only a 3.0% decrease, demonstrating improved robustness to representational capacity degradation. This indicates that our model more effectively leverages the available features across diverse CLIP architectures, and is less susceptible to variations in visual encoder strength. Furthermore, in Fig. 4, we conduct backbone exchanges in VFM to demonstrate the adaptability of our module. Specifically, we replace the backbone from DINO ViT-B/16 to DINOv2-reg ViT-L/14 [33, 13] and DINOv3 ViT-B/16 [41] to verify the generality of our method. In each case, our approach yields performance improvements of +0.7% and +0.3%, respectively. Additionally, our method is compatible with complementary techniques such as multi-layer feature fusion and text-prompt modifications, as discussed in the Appendix H, I. Owing to its generality, our framework can be readily integrated with new backbones and future improvements. 6 Conclusion & Limitation In this paper, we propose GLA-CLIP, a training-free method for OVSS that resolves inconsistent predictions across sliding windows. Using Key-Value Extension, each query attends to globally aggregated tokens, mitigating semantic discrepancies. We further address window bias and small object neglect by Proxy Anchor and Dynamic Normalization. While the Key–Value Extension utilizes whole tokens from other windows, computational cost may increase. More research is needed to efficiently leverage high-quality tokens across different windows. While our method can be adopted on any backbone features, certain hyperparameters may need to be adjusted to achieve optimal performance. We hope future research will address both memory efficiency and hyperparameter generalization. Acknowledgements This work was supported in part by MSIT/IITP (No. RS-2022-I220680, RS-2020-I201821, RS-2019-I190421, RS-2024-00459618, RS-2024-00360227, RS-2024-00437633, RS-2024-00437102, RS-2025-25442569), MSIT/NRF (No. RS-2024-00357729), and KNPA/KIPoT (No. RS-2025-25393280). References [1] S. Bai, Y. Liu, Y. Han, H. Zhang, and Y. Tang (2024) Self-calibrated clip for training-free open-vocabulary segmentation. arXiv preprint arXiv:2411.15869. Cited by: Appendix H. [2] Y. Benigmim, M. Fahes, T. Vu, A. Bursuc, and R. de Charette (2025) FLOSS: free lunch in open-vocabulary semantic segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision. Cited by: Table 1. [3] W. Bousselham, F. Petersen, V. Ferrari, and H. Kuehne (2024) Grounding everything: emerging localization properties in vision-language transformers. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 3828–3837. Cited by: §1, §2.2. [4] H. Caesar, J. Uijlings, and V. Ferrari (2018) Coco-stuff: thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 1209–1218. Cited by: §B.1, §B.2, §B.3, §B.4, Figure 4, Figure 4, §4.1, §4.2. [5] W. Cai, K. Jin, J. Hou, C. Guo, L. Wu, and W. Yang (2025) Vdd: varied drone dataset for semantic segmentation. Journal of Visual Communication and Image Representation 109, p. 104429. Cited by: §J.1. [6] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin (2021) Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, p. 9650–9660. Cited by: §3.1.1. [7] J. Cha, J. Mun, and B. Roh (2023) Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 11165–11174. Cited by: Table 1. [8] Q. Chen, L. Yang, Y. Chen, N. Zhao, J. Lai, J. Shao, and X. Xie (2025) Training-free class purification for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 23124–23134. Cited by: §3.4, §4.2, Table 1. [9] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar (2022) Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 1290–1299. Cited by: §2.1. [10] Z. Chi, Y. Wu, L. Gu, H. Liu, Z. Wang, Y. Zhang, Y. Wang, and K. Plataniotis (2025) Plug-in feedback self-adaptive attention in clip for training-free open-vocabulary segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 22815–22825. Cited by: §4.2. [11] S. Cho, H. Shin, S. Hong, A. Arnab, P. H. Seo, and S. Kim (2024) Cat-seg: cost aggregation for open-vocabulary semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 4113–4123. Cited by: §2.1. [12] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele (2016) The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 3213–3223. Cited by: §J.2, Figure 13, Figure 13, §B.3, Appendix F, Figure 4, Figure 4, §4.1, §4.2, §5.3. [13] T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski (2023) Vision transformers need registers. arXiv preprint arXiv:2309.16588. Cited by: Appendix C, §5.5. [14] J. Ding, N. Xue, G. Xia, and D. Dai (2022) Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 11583–11592. Cited by: §2.1. [15] Z. Ding, J. Wang, and Z. Tu (2022) Open-vocabulary universal image segmentation with maskclip. arXiv preprint arXiv:2208.08984. Cited by: §2.1. [16] S. Duan, X. Yang, and N. Wang (2025) DIH-clip: unleashing the diversity of multi-head self-attention for training-free open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 22794–22803. Cited by: Table 1. [17] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman (2011) The pascal visual object classes challenge 2012 (voc2012) results (2012). Cited by: §J.2, Appendix L, §B.1, §B.2, §B.3, §B.4, Figure 4, Figure 4, §4.1, §4.2. [18] S. Hajimiri, I. B. Ayed, and J. Dolz (2025) Pay attention to your neighbours: training-free open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 5061–5071. Cited by: §1, §1, §2.2, Table 1. [19] C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. Le, Y. Sung, Z. Li, and T. Duerig (2021) Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, p. 4904–4916. Cited by: §2.1. [20] D. Kang and M. Cho (2024) In defense of lazy visual grounding for open-vocabulary semantic segmentation. In European Conference on Computer Vision, p. 143–164. Cited by: Table 1. [21] L. Karazija, I. Laina, A. Vedaldi, and C. Rupprecht (2023) Diffusion models for zero-shot open-vocabulary segmentation. arXiv e-prints, p. arXiv–2306. Cited by: §2.2. [22] C. Kim, D. Ju, W. Han, M. Yang, and S. J. Hwang (2025) Distilling spectral graph for object-context aware open-vocabulary semantic segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 15033–15042. Cited by: §A.2, Appendix K, Figure 11, Figure 11, Figure 12, Figure 12, Figure 13, Figure 13, Appendix I, §1, §3.4, Figure 4, Figure 4, §4.2, §4.2, Table 1. [23] M. Lan, C. Chen, Y. Ke, X. Wang, L. Feng, and W. Zhang (2024) Clearclip: decomposing clip representations for dense vision-language inference. In European Conference on Computer Vision, p. 143–160. Cited by: §B.1, §B.2, Appendix C, §1, §1, §2.2, §3.1.1, §4.1, §4.2, Table 1, Table 1. [24] M. Lan, C. Chen, Y. Ke, X. Wang, L. Feng, and W. Zhang (2024) Proxyclip: proxy attention improves clip for open-vocabulary segmentation. In European Conference on Computer Vision, p. 70–88. Cited by: §A.2, Appendix K, Figure 11, Figure 11, Figure 12, Figure 12, Figure 13, Figure 13, Appendix L, §B.1, §B.3, Appendix E, §1, §1, §2.2, §3.1.1, §3.4, §3.4, Figure 4, Figure 4, §4.1, §4.2, §4.2, Table 1, Table 1. [25] K. Li, R. Liu, X. Cao, X. Bai, F. Zhou, D. Meng, and Z. Wang (2025) Segearth-ov: towards training-free open-vocabulary segmentation for remote sensing images. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 10545–10556. Cited by: §J.1. [26] Y. Li, H. Wang, Y. Duan, and X. Li (2023) Clip surgery for better explainability with enhancement in open-vocabulary tasks. arXiv e-prints, p. arXiv–2304. Cited by: §2.2. [27] F. Liang, B. Wu, X. Dai, K. Li, Y. Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu (2023) Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 7061–7070. Cited by: §2.1. [28] Y. Liu, G. Wang, J. Zhang, Q. Liu, and D. Huang (2025) Unveiling the knowledge of clip for training-free open-vocabulary semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 5649–5657. Cited by: §1. [29] T. Lüddecke and A. Ecker (2022) Image segmentation using text and image prompts. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 7086–7096. Cited by: §2.1. [30] H. Luo, J. Bao, Y. Wu, X. He, and T. Li (2023) Segclip: patch aggregation with learnable centers for open-vocabulary semantic segmentation. In International conference on machine learning, p. 23033–23044. Cited by: §2.1. [31] Y. Lyu, G. Vosselman, G. Xia, A. Yilmaz, and M. Y. Yang (2020) UAVid: a semantic segmentation dataset for uav imagery. ISPRS journal of photogrammetry and remote sensing 165, p. 108–119. Cited by: §J.1. [32] R. Mottaghi, X. Chen, X. Liu, N. Cho, S. Lee, S. Fidler, R. Urtasun, and A. Yuille (2014) The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 891–898. Cited by: Appendix K, Figure 12, Figure 12, §B.1, §B.2, §B.3, §B.4, Appendix F, §4.1. [33] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. (2023) Dinov2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. Cited by: §5.5, Table 4. [34] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §1, §2.1, §3.1.1. [35] Y. Rao, W. Zhao, G. Chen, Y. Tang, Z. Zhu, G. Huang, J. Zhou, and J. Lu (2022) Denseclip: language-guided dense prediction with context-aware prompting. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 18082–18091. Cited by: §2.1. [36] H. S. Seong, W. Moon, S. Lee, and J. Heo (2024) Progressive proxy anchor propagation for unsupervised semantic segmentation. In European Conference on Computer Vision, p. 472–490. Cited by: §3.3. [37] X. Shan, D. Wu, G. Zhu, Y. Shao, N. Sang, and C. Gao (2024) Open-vocabulary semantic segmentation with image embedding balancing. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 28412–28421. Cited by: §2.1. [38] T. Shao, Z. Tian, H. Zhao, and J. Su (2024) Explore the potential of clip for training-free open vocabulary semantic segmentation. In European Conference on Computer Vision, p. 139–156. Cited by: §1, §2.2. [39] Y. Shi, M. Dong, and C. Xu (2024) Harnessing vision foundation models for high-performance, training-free open vocabulary segmentation. arXiv preprint arXiv:2411.09219. Cited by: Appendix G. [40] G. Shin, W. Xie, and S. Albanie (2022) Reco: retrieve and co-segment for zero-shot transfer. Advances in neural information processing systems 35, p. 33754–33767. Cited by: Table 1. [41] O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, et al. (2025) Dinov3. arXiv preprint arXiv:2508.10104. Cited by: §5.5, Table 4. [42] F. Wang, J. Mei, and A. Yuille (2024) Sclip: rethinking self-attention for dense vision-language inference. In European Conference on Computer Vision, p. 315–332. Cited by: Appendix K, Appendix L, §B.1, §B.4, §1, §1, §2.2, §4.1, §4.2, Table 1, Table 1. [43] F. Wang, J. Mei, and A. Yuille (2024) Sclip: rethinking self-attention for dense vision-language inference. In European Conference on Computer Vision, p. 315–332. Cited by: §2.2. [44] M. Wysoczańska, M. Ramamonjisoa, T. Trzciński, and O. Siméoni (2024) Clip-diy: clip dense inference yields open-vocabulary semantic segmentation for-free. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 1403–1413. Cited by: Table 1. [45] M. Wysoczańska, O. Siméoni, M. Ramamonjisoa, A. Bursuc, T. Trzciński, and P. Pérez (2024) CLIP-dinoiser: teaching clip a few dino tricks for open-vocabulary semantic segmentation. In European Conference on Computer Vision, p. 320–337. Cited by: §J.2, §B.1, §4.1, Table 1. [46] J. Xu, S. De Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang (2022) Groupvit: semantic segmentation emerges from text supervision. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 18134–18144. Cited by: §B.1, §B.2, Table 1. [47] J. Xu, S. Liu, A. Vahdat, W. Byeon, X. Wang, and S. De Mello (2023) Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 2955–2966. Cited by: §2.1. [48] M. Xu, Z. Zhang, F. Wei, H. Hu, and X. Bai (2023) SAN: side adapter network for open-vocabulary semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (12), p. 15546–15561. Cited by: §2.1. [49] M. Xu, Z. Zhang, F. Wei, Y. Lin, Y. Cao, H. Hu, and X. Bai (2022) A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In European Conference on Computer Vision, p. 736–753. Cited by: §2.1. [50] Y. Yang, J. Deng, W. Li, and L. Duan (2025) ResCLIP: residual attention for training-free dense vision-language inference. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 29968–29978. Cited by: §1, §3.4, §4.2, Table 1. [51] Q. Yu, J. He, X. Deng, X. Shen, and L. Chen (2023) Convolutions die hard: open-vocabulary segmentation with single frozen convolutional clip. Advances in neural information processing systems 36, p. 32215–32234. Cited by: §2.1. [52] B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Torralba (2019) Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision 127 (3), p. 302–321. Cited by: §J.2, Figure 11, Figure 11, §B.3, Appendix D, §4.1. [53] C. Zhou, C. C. Loy, and B. Dai (2022) Extract free dense labels from clip. In European Conference on Computer Vision, p. 696–712. Cited by: §B.1, §B.2, §1, §1, §2.2. [54] Z. Zhou, Y. Lei, B. Zhang, L. Liu, and Y. Liu (2023) Zegclip: towards adapting clip for zero-shot semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 11175–11185. Cited by: §2.1. Supplementary Material Appendix A Boundary Error Rate (BER) A.1 Definition of BER In Fig. 1, to measure prediction inconsistencies caused by the sliding-window mechanism, we introduce the Boundary Error Rate (BER). BER measures how often adjacent pixel pairs across window boundaries, represented as p,q\p,q\ in the equation, share the same ground-truth label but receive different predictions. BER=∑(p,q)∈ℬ[(yp=yq)∧(y^p≠y^q)]∑(p,q)∈ℬ[yp=yq]×100BER= Σ _(p,q) 1[(y_p=y_q) ( y_p≠ y_q)]Σ _(p,q) 1[y_p=y_q]× 100 (16) Notation: • p,qp,q: A pair of adjacent pixels located across a sliding window boundary. • ℬ=(p,q)B=\(p,q)\: The set of all adjacent pixel pairs across window boundaries. • yp,yqy_p,y_q: Ground-truth labels of pixels p and q. • y^p,y^q y_p, y_q: Predicted labels of pixels p and q. • [⋅]1[·]: Indicator function. The denominator counts the number of adjacent pixel pairs across window boundaries that share the same ground-truth label. The numerator counts how many of those are predicted as different classes. BER reflects the proportion of inconsistencies. A.2 Complementarity of BER BER captures sliding-window grid artifacts overlooked by mIoU. In Fig. 7, ours shows better sample mIoU than ProxyCLIP [24] & CASS [22] and also produces natural masks with fewer grid artifacts, which BER correctly reflects. At dataset-level, gains in both mIoU & BER shows accurate pixel classification and visually coherent segmentation. Figure 7: Qualitative Results with Sample-wise mIoU and BER. Appendix B Desciption of each setting B.1 CLIP-DINOiser Setting We follow CLIP-DINOiser [45] setting in Tab. 1 to conduct fair comparison. We resize input images with a short side of 448 and perform sliding-window inference with a 448 × 448 window and 224 stride for all dataset. We do not perform rename trick on each classnames and use only the standard ImageNet prompts following [46, 53]. Additionally, naive CLIP-DINOiser setting does not offer the background threshold unlike ClearCLIP [23, 42, 24], we follow the background threshold of ClearCLIP to identify background in Pascal VOC21 [17] , Pascal Context60 [32] , COCO-Object [4]. B.2 ClearCLIP Setting We follow ClearCLIP [23] setting in Tab. 1 to conduct fair comparison. We resize input images with a short side of 448 and perform sliding-window inference with a 448 × 448 window and 224 stride for all dataset. We do not perform rename trick on each classnames and use only the standard ImageNet prompts following [46, 53]. The background threshold for Pascal VOC21 [17], Pascal Context60 [32], COCO-Object [4] is assigned to 0.5, 0.15, 0.4, for each. B.3 ProxyCLIP Setting In ProxyCLIP [24] setting in Tab 1, We resize the images to accommodate varying dataset specification: a shorter side of 336 pixels for PASCAL [17, 32] and COCO [4] datasets and 448 pixels for Cityscapes [12] and ADE20K [52] datasets. We adopt a sliding-window strategy with a 336 × 336 window and 112 × 112 stride. For the background class, rather than directly using the text prompt “background”, we employ a renaming strategy in which multiple class names associated with background semantics are grouped and used as substitutes. This follows the official ProxyCLIP implementation. The background threshold for Pascal VOC21 [17], Pascal Context60 [32], COCO-Object [4] is assigned to 0.2, 0.15, 0.25, for each. B.4 SCLIP Setting In SCLIP [42] setting in Tab 1, We resize input images with a short side of 336 and perform sliding-window inference with a 224 × 224 window and 112 stride. Only for the Cityscapes, we resize the short side of 560. We use rename trick on both background class and several other classes, following official SCLIP code. The background threshold for Pascal VOC21 [17], Pascal Context60 [32], COCO-Object [4] is assigned to 0.1, 0.1, 0.1, for each. Figure 8: Effect of Smoothing on ClearCLIP attention map Appendix C Query Smoothing for Proxy-Anchor Construction in CLIP features As we mentioned in the Implementation Details (Sec. 4.1), our method incorporates a query smoothing step during proxy-anchor construction to ensure stable attention when generating attention maps from CLIP internal features, as in ClearCLIP [23]. This query smoothing is necessary because CLIP feature maps contain noisy high-norm patches and these high-norm patches lost its semantic [13]. By this reason high-norm patches do not have to be included in attention process. However, when the query token itself corresponds to a high-norm patch, the model tends to propagate this noise by repeatedly attending to other high-norm patches across windows, ultimately leading to incorrect semantic decisions. To mitigate this issue, we replace each raw query token with a smoothed token obtained by averaging it with neighbor patches. Specifically, we average each token with its eight spatial neighbors within the inner-window and with the patches at the same spatial position in adjacent overlapping windows: i(0)=i+∑j∈N(i)j+∑k∈O(i)k1+|N(i)|+|O(i)|. _i^(0)= Q_i+ _j∈ N(i)Q_j+ _k∈ O(i)Q_k1+|N(i)|+|O(i)|. (17) Here, N(i)N(i) denotes the local neighbors within the innner-window, and O(i)O(i) represents the overlapping patches from adjacent outer-windows. This averaging suppresses high-norm activations of query token before the progressive positive mining step, allowing the model to aggregate meaningful positives across windows without reinforcing noisy high-norm responses. Qualitatively, Fig. 8 illustrates this process. In Fig. 8(a-b), the cropped images and its pca visualization identify the location of the high-norm patches. When adapting Key-Value Extension, high-norm patches in background regions (e.g., sky) activate attention both within inner and outer windows (Fig. 8(c)), and even with proxy-anchor stabilization, these noisy patches can dominate the attention map (Fig. 8(d)). By replacing query tokens with their smoothed versions (Eq. 17), attention is redirected toward semantically meaningful tokens (Fig. 8(e)), demonstrating that our method effectively suppresses noisy high-norm patches and constructs stable attention maps. Appendix D Semantic Collapse in Proxy Anchors We examine the potential risk of semantic collapse during proxy-anchor construction. In this process, a small number of negative tokens may be mistakenly selected as high-confident tokens, potentially leading to semantic collapse of the proxy anchor. To assess the robustness of our high-confident token filtering, we perform a binary classification using GT labels. As shown in Tab. 5, the lowest precision is 93.9% on ADE20K [52], while all other datasets achieve even higher precision. These results indicate that semantic collapse rarely occurs, with at most 6% of negative tokens being incorrectly incorporated into the proxy anchor. Metric V21 PC59 C-Stf City ADE Avg. Precision (%) 98.7 96.3 94.2 95.6 93.9 96.2 Table 5: High confidence token classification precision. Appendix E Analysis for Attention Masking and Scaling As discussed in Sec. 3.4, the Key–Value Extension can lead to the neglect of small objects due to the additional negative tokens from outer windows. Quantitatively, the total number of tokens increases by a factor of the number of windows L, substantially altering the statistics of the similarity map. For small objects, most of the newly introduced tokens correspond to background (negative) regions. This circumstance is clearly shown in Fig. 9(a-b), where background tokens (red) remain largely unmasked in Fig. 9(b) compared with Fig. 9(a), illustrating that low values of β=1.2β=1.2 and γ=3.0γ=3.0 fail to suppress negative tokens from the outer windows. While increasing fixed u and w (which correspond to β and γ in ProxyCLIP [24]) can quantitatively suppress this additional negative tokens, setting them too high can over-mask the attention. This over-masking removes parts of the object itself, restricting attention to local sub-regions and ultimately degrading performance (see Fig. 9c-d). Conversely, moderately stronger settings such as (=1.4,=5.0)(u=1.4,w=5.0) achieve a balance: they suppress the negative tokens introduced by Key–Value Extension while preserving full object coverage, as evidenced qualitatively in Fig. 9. These results directly support the normalization tightening strategy discussed in Sec. 3.4 of the main paper. Specifically, the “best” results reported in Tab. 1 and Tab. 2 are obtained by tightening the normalization parameters in this manner, compensating for the substantial increase of negative tokens introduced by Key–Value Extension. The observations from this results highlight a fundamental limitation of fixed masking and scaling: no single set of static hyperparameters can robustly handle both small and large objects across varying window configurations. This motivates the introduction of Dynamic Normalization, which adaptively adjusts masking and scaling to maintain consistent attention behavior regardless of object scale, window count, or dataset characteristics. Lastly, Dynamic Normalization removes the need for dataset-specific hyperparameters, providing a single formulation that generalizes well across datasets and window configurations. Figure 9: Effect of Key-Value Extension and u. Appendix F High confidence tokens High confidence token count is not a universally accurate proxy for object type; our dynamic normalization serves as an effective heuristic. Beyond the class-wise analysis in Section 5.3, we group instances based on ground-truth (GT) object size and analyze the corresponding number of high confidence tokens in Pascal Context60 [32] and Cityscapes [12] (in Fig. 10). Specifically, for each image, we compute the number of high confidence tokens associated with each GT instance, assign it to a size bin according to its GT region size, and report the interquartile range (25th–75th percentile) and median within each bin. This analysis shows that the number of high confidence tokens provides a reliable proxy for object size, demonstrating its effectiveness in capturing scale information. Figure 10: Object scale (GT vs high Confidence). Appendix G Adapting to SAM-based models We conduct experiments to evaluate the adaptability of our method to SAM-based OVSS model [39]. Our method can be integrated into SAM-based models, which utilize components such as SAM masks and SAM refinement. While SAM-based models generally exhibit lower inference speed, they achieve superior performance. We aim to demonstrate that our method is compatible with SAM-based models and has the potential to achieve state-of-the-art performance in this setting. As shown in Tab. 6, our method attains 46.1% mIoU, representing a +0.3% improvement over the baseline, Trident, which is SAM-based model. This result confirms that our method can be effectively adapted to SAM-based models, and that our contribution is independent of existing approaches. Model V21 PC 60 C- Obj V20 PC 59 C- Stf City ADE Avg Trident 67.1 38.6 41.1 84.5 42.2 28.3 42.9 21.9 45.8 gray!15Trident + GLA gray!1567.4 gray!1538.9 gray!1541.4 gray!1584.8 gray!1542.4 gray!1528.3 gray!1543.6 gray!1521.7 gray!1546.1 Table 6: Trident adaptation with GLA. Appendix H Adapting to Multi-layer Feature Fusion As we mentioned on Sec 5.5, we conduct experiments to evaluate the compatibility of our method with multi-layer feature fusion in the training-free OVSS task, which has been widely adopted recently. We also examine whether our model demonstrates performance improvements when combined with multi-layer feature fusion, which can validate that our method independently enhances performance and can be considered a standalone technique. In Tab. 7, we compare our method with and without multi-layer feature fusion. Following the approach of SC-CLIP [1], which leverages multi-layer feature fusion, we observe that our method combined with multi-layer feature fusion achieves an average mIoU of 44.9%, representing a +0.9% improvement over our baseline. This result demonstrates that our method independently improves baseline performance and can be effectively adapted to multi-layer feature fusion. Model V21 PC 60 C- Obj V20 PC 59 C- Stf City ADE Avg Ours 66.3 36.1 37.7 84.2 39.9 26.9 40.8 20.0 44.0 gray!15 Ours + Fusion gray!1567.9 gray!1537.0 gray!1538.1 gray!1584.1 gray!1540.9 gray!1527.5 gray!1543.2 gray!1520.3 gray!1544.9 Table 7: Effect of multi-layer feature fusion. Appendix I Adapting to CASS for SOTA Performance As mentioned in Sec. 5.5, we evaluate the compatibility of our method with CASS [22] to demonstrate that text prompt embedding modifications can be effectively integrated. We adapt our method to CASS by combining GLA-CLIP with two key components of CASS: (1) Object-Guided Text Embedding Adjustment, which refines text embeddings by fusing them with the mean of visual tokens, and (2) Object Perspective Patch-Text Similarity, which processes the entire image instead of individual windows and incorporates the full-image logits into the final prediction. Using the dataset-specific hyperparameters provided in the official CASS code, our approach achieves state-of-the-art performance, with an average mIoU of 44.7% in Tab. 8. One of our contribution is that ours can be combined with text embedding modification and logit fusion. Model V21 PC 60 C- Obj V20 PC 59 C- Stf City ADE Avg CASS 65.8 36.7 37.8 87.8 40.2 26.7 39.4 20.4 44.4 ProxyCLIP + GLA 66.3 36.1 37.7 84.2 40.8 26.9 39.9 20.0 44.0 gray!15 CASS + GLA gray!1567.2 gray!1536.8 gray!1536.2 gray!1588.5 gray!1539.7 gray!1527.2 gray!1540.6 gray!1521.0 gray!1544.7 Table 8: CASS adaptation with GLA Appendix J Domain Shift: Remote Sensing Datset J.1 Evaluation on remote sensing dataset To demonstrate the generality of our GLA module beyond the eight web datasets in Tab. 1, we evaluate it on three remote sensing datasets following SegEarth-OV [25]. Since SegEarth-OV also adopts a sliding-window OVSS pipeline, our module can be directly integrated. We conduct experiments on Vaihingen111https://w.isprs.org/education/benchmarks/UrbanSemLab, UAVid [31], and VDD [5]. As shown in Tab. 9, GLA improves the average mIoU from 38.8% to 39.6% (+0.8%), demonstrating strong cross-domain generalization. We expect our module to benefit any sliding-window-based setting. Model Vaih. UAVid VDD Avg ClearCLIP 27.3 36.2 39.3 34.3 ProxyCLIP 30.6 41.4 44.3 38.8 gray!15 ProxyCLIP + GLA gray!1531.9 (+1.3) gray!1541.9 (+0.5) gray!1545.0 (+0.7) gray!1539.6 (+0.8) Table 9: Performance comparison across remote sensing datasets. J.2 Comparison with training-based model Training-based OVSS models are vulnerable to domain shift due to reliance on fine-tuning. In domain-shift experiments (Tab. 10), ours and CLIP-DINOiser [45] (which is the representative Training-based model) show small gap on 8 web-image datasets (e.g. pascal voc [17], ade20k [52], cityscapes [12], etc), it becomes larger on 3 remote sensing datasets, indicating limited robustness and motivating training-free OVSS that preserves zero-shot capability of CLIP. Model Web (8) Vaih. UAVid VDD Avg (3) gray!15 Ours gray!1541.9 gray!1531.9 gray!1541.9 gray!1545.0 gray!1539.6 CLIP-DINOiser 40.3 18.6 33.3 37.7 29.9 Table 10: Domain-shift evaluation on remote sensing datasets. Appendix K Computation cost. We analyze the computational cost of our model to assess its practical adaptability. Due to the Key–Value Extension, our method must forward batched images to reference tokens from outer-window; forwarding each window independently is not feasible. In addition, the Key–Value Extension increases the number of tokens involved in the attention computation. While ProxyCLIP [24] requires a complexity of O(LN2D)O(LN^2D), our approach incurs O(L2N2D)O(L^2N^2D) complexity on attention operation. Despite this theoretical increase, the actual latency and memory overhead remain modest. We measure the number of sliding windows, latency, and memory usage on the Pascal Context59 [32] following the SCLIP [42] setting (Tab 11). We conduct the experiments on a NVIDIA RTX A6000 GPU. By varying the sliding-window stride, increasing the number of windows leads to higher memory usage in ours. Though even with more windows, ours remains more efficient than CASS [22], using less memory while achieving competitive mIoU. Context59 (336px×497px) #Win Lat. Mem. mIoU CASS 1 5799 ms 7.6 GB 40.2 ProxyCLIP (Baseline) 8 429 ms 1.8 GB 38.8 Ours (Stride=224) 6 415 ms 1.7 GB 39.5 Ours (Stride=112) 8 431 ms 1.9 GB 39.9 Ours (Stride=98) 12 441 ms 2.4 GB 39.9 Table 11: Latency & memory usage comparison on PC59 Appendix L Setting ablation. We tune model hyperparameters for each setting while applying the same configuration consistently to both the baseline and our method. As shown in Tab. 12, our method consistently outperforms the ProxyCLIP [24] on Pascal VOC21 [17] across various configurations. The default setting is defined as crop size: 224, stride: 112, image size: 336, background threshold: 0.1, rename trick: enabled (following SCLIP [42] setting). We vary each parameter individually to demonstrate the robustness of our method. VOC21 Default Crop size Stride Img resize Bg. th. Rename 224→224→ 112→112→ 336→336→ 0.1→0.1→ O→ 280 224 448 0.2 X ProxyCLIP 63.3 64.4 59.7 61.8 61.0 60.4 gray!15 Ours gray!1566.3 gray!1566.5 gray!1563.6 gray!1565.3 gray!1564.6 gray!1563.3 Table 12: Setting ablation on VOC21 (mIoU). Figure 11: Qualitative results on ADE20K [52] with 150 categories. We compare GLA-CLIP with ProxyCLIP [24], CASS [22]. Figure 12: Qualitative results on Pascal Context60 [32] with 60 categories.. We compare GLA-CLIP with ProxyCLIP [24], CASS [22]. Figure 13: Qualitative results on Cityscapes [12] with 19 categories. We compare GLA-CLIP with ProxyCLIP [24], CASS [22].