Paper deep dive
Domain Recentering and Confidence-Weighted Prior Calibration for Vision-Language Models
Youngeun Seol, Jimin Shin, Heeseo Yoon, Uiwon Hwang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/25/2026, 5:19:11 AM
Summary
The paper proposes Domain Recentering with Confidence Calibration (DRC), a training-free method to adapt Vision-Language Models like CLIP under distribution shifts. DRC uses a Gaussian Mixture Model to compute soft, image-specific bias vectors from unlabeled target data, subtracting this bias from visual embeddings. It further corrects residual class preferences via confidence-weighted prior calibration on logits. DRC outperforms existing methods on cross-domain and out-of-distribution benchmarks.
Entities (9)
Relation Signals (5)
DRC → evaluatedon → ImageNet
confidence 95% · with gains over CLIP also holding under ImageNet distribution shifts.
DRC → improves → CLIP
confidence 95% · DRC achieves the highest average accuracy on cross-domain datasets, exceeding zero-shot CLIP by 4.13 and 5.07 points
CLIP → suffersfrom → distribution_shift
confidence 95% · under distribution shift, visual embeddings drift from fixed text embeddings.
DRC → uses → Gaussian Mixture Model
confidence 95% · DRC models the structure of unlabeled CLIP visual features using a Gaussian mixture model.
DRC → outperforms → UMFC
confidence 90% · Among compared methods, DRC achieves the highest average accuracy... UMFC [5] corrects visual encoder bias by subtracting from each feature the mean of its hard-assigned cluster.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision-language models such as CLIP achieve strong zero-shot classification, yet under distribution shift, visual embeddings drift from fixed text embeddings. Training-free calibration avoids the per-sample optimization of prompt learning, but prior feature calibration gives each image the full bias of one hard cluster. We propose Domain Recentering with Confidence Calibration (DRC), a training-free method adapting CLIP from a set of unlabeled target images. DRC fits a Gaussian mixture once and subtracts from each embedding a posterior-weighted average of component means. It then removes residual class preference with a log-prior correction, estimating the prior from confidence-weighted predictions. Among compared methods, DRC achieves the highest average accuracy on cross-domain datasets, exceeding zero-shot CLIP by 4.13 and 5.07 points with ViT-B/16 and ResNet-50, with gains over CLIP also holding under ImageNet distribution shifts.
Tags
Links
- Source: https://arxiv.org/abs/2609.29358v1
- Canonical: https://arxiv.org/abs/2609.29358v1
Trouble viewing inline? Open PDF directly →
Full Text
21,451 characters extracted from source content.
Expand or collapse full text
Domain Recentering and Confidence-Weighted Prior Calibration for Vision-Language Models Youngeun Seol⋆ Jimin Shin⋆ Heeseo Yoon⋆ †thanks: * Equal contribution, listed in alphabetical order Uiwon Hwang† †thanks: †Corresponding author: Uiwon Hwang (uiwon.hwang@ewha.ac.kr) Abstract Vision-language models such as CLIP achieve strong zero-shot classification, yet under distribution shift, visual embeddings drift from fixed text embeddings. Training-free calibration avoids the per-sample optimization of prompt learning, but prior feature calibration gives each image the full bias of one hard cluster. We propose Domain Recentering with Confidence Calibration (DRC), a training-free method adapting CLIP from a set of unlabeled target images. DRC fits a Gaussian mixture once and subtracts from each embedding a posterior-weighted average of component means. It then removes residual class preference with a log-prior correction, estimating the prior from confidence-weighted predictions. Among compared methods, DRC achieves the highest average accuracy on cross-domain datasets, exceeding zero-shot CLIP by 4.13 and 5.07 points with ViT-B/16 and ResNet-50, with gains over CLIP also holding under ImageNet distribution shifts. Index Terms: Vision-language models, domain recentering, confidence-weighted calibration †address: Department of Computer Science and Engineering, Ewha Womans University 1 Introduction Vision-language models have emerged as powerful foundation models for visual recognition. CLIP [7] learns a multi-modal embedding space for images and text from natural language supervision at scale, enabling zero-shot classification by aligning an image’s embedding with the text embeddings of class descriptions. However, when target images differ from the pretraining distribution, their visual embeddings can become misaligned with the fixed text embeddings, degrading classification accuracy. Recent works have adapted CLIP under distribution shift along two broad directions. One line performs prompt learning with few-shot labeled data [12, 11] or test-time entropy minimization [9, 2], requiring labels or per-image backpropagation. Another line calibrates features without training, using unlabeled target data. UMFC [5] corrects visual encoder bias by subtracting from each feature the mean of its hard-assigned cluster. However, a feature near a cluster boundary then receives one cluster’s full mean, so a small change in the feature can flip its correction. Correcting the image features alone also leaves the text classifier unchanged, so any preference it holds for certain classes persists. We propose Domain Recentering with Confidence Calibration (DRC), which models the structure of unlabeled CLIP visual features using a Gaussian mixture model. Rather than assigning each image to a single cluster, DRC uses mixture-component posterior probabilities to construct a soft, image-specific bias vector as a weighted combination of component means, so the correction changes continuously as the feature moves between components. This vector is subtracted from the image feature to suppress domain-related variation while preserving class-discriminative content. DRC further corrects residual class preferences through confidence-weighted prior calibration, without any backpropagation or parameter updates. Among the compared methods, DRC achieves the highest average accuracy on ten cross-domain datasets with both ResNet-50 and ViT-B/16, and it also improves the average accuracy of zero-shot CLIP over four ImageNet distribution shifts. 2 Related Work Prompt learning adapts CLIP [7] to downstream data by tuning its textual prompts. CoOp [12] and CoCoOp [11] learn these prompts from labeled few-shot data. Test-time adaptation instead uses unlabeled test samples to improve robustness under distribution shifts. TPT [9] optimizes prompts for each test sample, and DiffTPT [2] extends TPT with diffusion-based augmentation. Both repeat backpropagation for every test sample, which makes inference costly and motivates training-free adaptation that leaves the pretrained model unchanged. Training-free methods instead correct the systematic biases that zero-shot vision-language models exhibit in representations and predictions [5, 6]. UMFC [5] subtracts each hard-assigned cluster’s mean, giving all images within a cluster the same correction. Label-free logit adjustment [13] offsets label bias in predictions. Our method instead assigns each image its own bias by weighting Gaussian mixture component means with the image’s soft assignment. It then estimates the class prior from the recentered predictions with confidence weighting, so that the prior targets the bias left after feature correction. 3 Method 3.1 Background CLIP. CLIP [7] maps images and text into a shared embedding space via a visual encoder EvE_v and a text encoder EtE_t. For zero-shot classification, each class name ycy_c is inserted into prompt templates and encoded by EtE_t, and the averaged embedding serves as the classifier weight ct_c. Given an image xix_i, its visual feature i=Ev(xi)v_i=E_v(x_i) is compared to each ct_c via scaled cosine similarity ℓic=exp(τ)(i∥i∥2)⊤c∥c∥2, _ic= (τ) ( v_i _i _2 ) t_c _c _2, (1) where exp(τ) (τ) is the learned logit scale of CLIP, clipped at 100100. CLIP predicts the class with the highest logit ℓic _ic, and under domain shift, iv_i can become misaligned with the fixed ct_c. Problem Setting. We adapt a frozen CLIP model to a target domain whose images may differ from the pretraining distribution. An unlabeled adaptation set u=xii=1ND_u=\x_i\_i=1^N of target-domain images is given. DRC estimates its statistics once from uD_u without any target labels and then classifies each test image with these fixed statistics. The visual encoder EvE_v, the text encoder EtE_t, and the classifier weights ct_c remain unchanged throughout. Figure 1 illustrates the DRC pipeline, which recenters each image feature with a Gaussian mixture fitted on uD_u and then calibrates the logits with a confidence-weighted class prior. 3.2 Domain Recentering DRC models the features of uD_u with a Gaussian mixture and estimates the bias of each image from all mixture components, weighting each component by how strongly the image belongs to it. Figure 1: Overview of DRC. DRC recenters CLIP image features with a GMM-based soft bias correction and calibrates the resulting logits with a confidence-weighted class prior. Each image, whether in uD_u or at test time, is represented by averaging its ℓ2 _2-normalized feature with that of its horizontal flip, followed by re-normalization i=norm(norm(Ev(xi))+norm(Ev(Flip(xi)))).f_i=norm (norm(E_v(x_i))+norm(E_v(Flip(x_i))) ). (2) With the mean =1N∑i=1Ni μ= 1N _i=1^Nf_i over uD_u, the centered features are projected via PCA onto min(16,D,N) (16,D,N) dimensions, where D is the CLIP embedding dimension, so that the mixture is fitted on the dominant directions of variation i=(i−).z_i=P(f_i- μ). (3) A diagonal-covariance GMM with K components, p()=∑k=1Kπk(,k,k),p(z)= _k=1^K _kN(z; ν_k, _k), (4) is fitted to i\z_i\, where πk _k, k ν_k, and k _k denote the mixture weight, mean, and diagonal covariance of component k. Unlike hard clustering such as K-means, the GMM assigns each feature softly to the K components through its posterior qi,k=p(k∣i)q_i,k=p(k _i). Each component mean is computed in the original embedding space from the features whose most probable component is k, ℐk=i∣k=argmaxjqi,jI_k=\i k= _jq_i,j\, so that it summarizes the features belonging mainly to that component k=1|ℐk|∑i∈ℐki.m_k= 1|I_k| _i _kf_i. (5) For an input feature f, DRC estimates its bias by weighting these means with the posterior ()=∑k=1Kqk()k,qk()=p(k∣(−)).b(f)= _k=1^Kq_k(f)\,m_k, q_k(f)=p (k (f- μ) ). (6) Since qk()q_k(f) varies continuously with f, the bias changes gradually as a feature moves between components, whereas a hard assignment would switch it abruptly at component boundaries. The recentered feature is obtained by subtracting this bias and re-normalizing ^=−β()‖−β()‖2, f= f-β\,b(f) \|f-β\,b(f) \|_2, (7) where β controls the strength of recentering. 3.3 Confidence-Weighted Prior Calibration Recentering acts only on the image features and leaves the classifier weights ct_c unchanged, so the class preference of the zero-shot classifier [13, 6] can persist in the recentered predictions. DRC estimates this preference from predictions on uD_u and removes it with a log-prior adjustment, weighting each prediction by its confidence. For each xi∈ux_i _u, let ℓ^ic _ic be the logit from the recentered feature. The class probability is then pi(c)=exp(ℓ^ic)∑c′exp(ℓ^ic′).p_i(c)= ( _ic) _c ( _ic ). (8) Since an uncertain prediction provides weak evidence of class preference, each prediction is weighted by one minus its normalized entropy Hi=−∑cpi(c)logpi(c),wi=1−HilogC,H_i=- _cp_i(c) p_i(c), w_i=1- H_i C, (9) where C is the number of classes, so that near-uniform predictions receive weights close to zero. The confidence-weighted class prior is πcConf=∑i=1Nwipi(c)∑i=1Nwi. _c^Conf= _i=1^Nw_i\,p_i(c) _i=1^Nw_i. (10) The computed distribution summarizes the class preference reflected in reliable predictions on the adaptation set. A large πcConf _c^Conf indicates that class c receives a relatively large amount of prediction mass, while a small value denotes that the class is less frequently preferred by the model. Confidence weighting reduces the influence of uncertain predictions that could otherwise distort this estimate. The log-prior correction and its zero-centered form are rc=−log(πcConf+ϵ),r~c=rc−1C∑c′=1Crc′,r_c=- ( _c^Conf+ε), r_c=r_c- 1C _c =1^Cr_c , (11) where ϵε is a small constant for numerical stability. A test image with recentered feature f is classified with the calibrated logits ℓc′=ℓ^c+r~c,ℓ^c=exp(τ)^⊤c∥c∥2. _c= _c+ r_c, _c= (τ)\, f t_c _c _2. (12) This adjustment lowers the logits of classes that receive excessive prediction mass on uD_u and raises those of less preferred classes. 4 Experiments 4.1 Experimental Setup Datasets. We evaluate DRC on the cross-domain benchmark and the out-of-distribution (OOD) benchmark. For cross-domain evaluation, we use the same ten datasets as TPT [9], which are Aircraft, Caltech101, Cars, DTD, EuroSAT, Flower102, Food101, Pets, SUN397, and UCF101. The OOD benchmark measures robustness to distribution shifts with four variants of ImageNet [1], ImageNet-A [4], ImageNet-V2 [8], ImageNet-R [3] and ImageNet-Sketch [10]. Method Aircraft Caltech101 Cars DTD EuroSAT Flower102 Food101 Pets SUN397 UCF101 Average CLIP-ResNet-50 16.11 87.26 55.89 40.37 25.79 62.77 74.82 82.97 60.85 59.48 56.63 CoOp 15.12 86.53 55.32 37.29 26.20 61.55 75.59 87.00 58.15 59.05 56.18 CoCoOp 14.61 87.38 56.22 38.53 28.73 65.57 76.20 88.39 59.61 57.10 57.23 TPT 17.58 87.02 58.46 40.84 28.33 62.69 74.88 84.49 61.46 60.82 57.66 DiffTPT 17.60 86.89 60.71 40.72 41.04 63.53 79.21 83.40 62.72 62.67 59.85 UMFC 17.76 85.48 56.25 42.85 37.63 65.57 77.47 86.10 60.49 63.39 59.30 DRC (Ours) 18.81 87.99 59.66 44.33 44.84 66.30 78.83 88.01 62.46 65.72 61.70 CLIP-ViT-B/16 23.22 93.55 66.11 45.04 50.42 66.99 82.86 86.92 65.63 65.16 64.59 CoOp 18.47 93.70 64.51 41.92 46.39 68.71 85.30 89.14 64.15 66.55 63.88 CoCoOp 22.29 93.79 64.90 45.45 39.23 70.85 83.97 90.46 66.89 68.44 64.63 TPT 24.78 94.16 66.87 47.75 42.44 68.98 84.67 87.79 65.50 68.04 65.10 DiffTPT 25.60 92.49 67.01 47.00 43.13 70.10 87.23 88.22 65.74 68.22 65.47 UMFC 25.47 91.44 66.35 45.57 50.98 70.60 85.99 88.91 65.73 68.91 66.00 DRC (Ours) 27.39 92.17 68.82 47.46 61.73 71.78 86.80 91.11 67.52 72.43 68.72 Table 1: Results on the Cross-Domain Benchmark with CLIP-ResNet-50 and CLIP-ViT-B/16. CoOp and CoCoOp serve as supervised transfer baselines trained on ImageNet with 16 labeled samples per class. CLIP, CoOp, CoCoOp, and TPT results are taken from the original TPT paper [9], and DiffTPT results are taken from the original DiffTPT paper [2]. Method ImageNet -A -V2 -R -S Average OOD Average CLIP-ResNet-50 59.81 23.24 52.91 60.72 35.48 46.43 43.09 CoOp 63.33 23.06 55.40 56.60 34.67 46.61 42.43 CoCoOp 62.81 23.32 55.72 57.74 34.48 46.81 42.82 TPT 60.74 26.67 54.70 59.11 35.09 47.26 43.89 DiffTPT 60.80 31.06 55.80 58.80 37.10 48.71 45.69 UMFC 59.70 23.47 52.85 60.94 35.82 46.56 43.27 DRC (Ours) 62.12 23.77 55.44 61.39 39.94 48.53 45.14 CLIP-ViT-B/16 68.34 49.89 61.88 77.65 48.24 61.20 59.42 CoOp 71.51 49.71 64.20 75.21 47.99 61.72 59.28 CoCoOp 71.02 50.63 64.07 76.18 48.75 62.13 59.91 TPT 68.98 54.77 63.45 77.06 47.94 62.44 60.81 DiffTPT 70.30 55.68 65.10 75.00 46.80 62.28 60.52 UMFC 68.21 50.33 61.99 77.98 48.77 61.46 59.77 DRC (Ours) 70.17 49.71 63.83 78.14 52.09 62.79 60.94 Table 2: Results on the OOD Benchmark with CLIP-ResNet-50 and CLIP-ViT-B/16. CoOp and CoCoOp serve as supervised transfer baselines trained on ImageNet with 16 labeled samples per class. CLIP, CoOp, CoCoOp, and TPT results are taken from the original TPT paper [9], and DiffTPT results are taken from the original DiffTPT paper [2]. Implementation details. Text embeddings obtained from dataset-specific prompt templates are averaged into a single vector for each class, and image features are extracted from each image and its horizontal flip. DRC uses PCA and a diagonal-covariance GMM (ninit=5n_init=5, seed 42). The adaptation set uD_u is the validation split of each cross-domain dataset and the unlabeled evaluation images for ImageNet and its variants. K and β are selected by accuracy on the validation split of each cross-domain dataset, and on the ImageNet validation set for the OOD benchmark, with the ImageNet values reused for all four variants. 4.2 Comparison with State-of-the-Art Cross-Domain Benchmark. Table 1 evaluates DRC across ten datasets with disjoint class spaces. DRC achieves the highest average accuracy with both CLIP-ViT-B/16 and CLIP-ResNet-50, reaching 68.72% and 61.70%, respectively. These results improve over zero-shot CLIP by 4.13 and 5.07 points, and DRC ranks first on 7 and 6 of the 10 datasets, respectively. On EuroSAT, DRC improves over the best competing method by 10.75 and 3.80 points, and on UCF101, the gains are 3.52 and 2.33 points, respectively. OOD Benchmark. Table 2 further evaluates DRC on the OOD benchmark. With CLIP-ViT-B/16, DRC achieves the highest Average and OOD Average of 62.79% and 60.94%. With CLIP-ResNet-50, DRC achieves 48.53% and 45.14%, improving over zero-shot CLIP by 2.10 and 2.05 points, respectively. The gains are particularly pronounced under strong appearance shifts. DRC achieves the highest accuracy on ImageNet-R/S for both backbones (61.39%/39.94% for CLIP-ResNet-50, 78.14%/52.09% for CLIP-ViT-B/16). Prior calibration Recentering without with None 64.59 68.35 Hard 66.89 68.57 Soft 67.01 68.72 Table 3: Ablation on the cross-domain benchmark with CLIP-ViT-B/16, reporting the average accuracy over the ten datasets. The bottom-right entry is DRC. 4.3 Ablation Study We perform ablation studies on the cross-domain benchmark with CLIP-ViT-B/16 to examine the effectiveness of domain recentering and confidence-weighted prior calibration. As shown in Table 3, both components improve over the CLIP baseline. Soft recentering outperforms hard recentering both without and with prior calibration. Confidence-weighted prior calibration provides the largest gain, increasing the average accuracy to 68.35%. Combining both components yields the best result, with DRC achieving an average accuracy of 68.72%. 5 Conclusion In this paper, we proposed DRC, a training-free method that improves the robustness of CLIP under distribution shift without any backpropagation or parameter updates. DRC fits a Gaussian mixture to unlabeled target features and subtracts from each feature a soft, image-specific bias, so that the correction varies continuously across mixture components. It then corrects class preferences remaining in the recentered predictions with a confidence-weighted class prior. DRC achieves the highest average accuracy among the compared methods on the cross-domain benchmark and improves the OOD Average of zero-shot CLIP with both backbones. 6 Acknowledgements This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS-2025-00561169), Global – Learning & Academic research institution for Master’s · PhD students, and Postdocs (G-LAMP) Program of the National Research Foundation of Korea (NRF) grant funded by the Ministry of Education (No. RS-2025-25442252), and Institute of Information & communications Technology Planning & Evaluation (IITP) under the Leading Generative AI Human Resources Development (IITP-2027-RS-2026-25546026) grant funded by the Korea government (MSIT). References [1] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) Imagenet: a large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, p. 248–255. Cited by: §4.1. [2] C. Feng, K. Yu, Y. Liu, S. Khan, and W. Zuo (2023) Diverse data augmentation with diffusions for effective test-time prompt tuning. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), p. 2704–2714. Cited by: §1, §2, Table 1, Table 2. [3] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, et al. (2021) The many faces of robustness: a critical analysis of out-of-distribution generalization. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), p. 8320–8329. Cited by: §4.1. [4] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song (2021) Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 15262–15271. Cited by: §4.1. [5] J. Liang, R. Hou, M. Hu, H. Chang, S. Shan, and X. Chen (2024) UMFC: unsupervised multi-domain feature calibration for vision-language models. Advances in Neural Information Processing Systems 37, p. 114072–114093. Cited by: §1, §2. [6] S. Parashar, Z. Lin, T. Liu, X. Dong, Y. Li, D. Ramanan, J. Caverlee, and S. Kong (2024) The neglected tails in vision-language models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 12988–12997. Cited by: §2, §3.3. [7] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §1, §2, §3.1. [8] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar (2019) Do imagenet classifiers generalize to imagenet?. In International conference on machine learning, p. 5389–5400. Cited by: §4.1. [9] M. Shu, W. Nie, D. Huang, Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao (2022) Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems 35, p. 14274–14289. Cited by: §1, §2, §4.1, Table 1, Table 2. [10] H. Wang, S. Ge, Z. Lipton, and E. P. Xing (2019) Learning robust global representations by penalizing local predictive power. Advances in neural information processing systems 32. Cited by: §4.1. [11] K. Zhou, J. Yang, C. C. Loy, and Z. Liu (2022) Conditional prompt learning for vision-language models. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), p. 16795–16804. Cited by: §1, §2. [12] K. Zhou, J. Yang, C. C. Loy, and Z. Liu (2022) Learning to prompt for vision-language models. International journal of computer vision 130 (9), p. 2337–2348. Cited by: §1, §2. [13] X. Zhu, B. Zhu, Y. Tan, S. Wang, Y. Hao, and H. Zhang (2024) Enhancing zero-shot vision models by label-free prompt distribution learning and bias correcting. Advances in Neural Information Processing Systems 37, p. 2001–2025. Cited by: §2, §3.3.