Paper deep dive
Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignment
Yaze Zhao, Yixiong Zou, Yuhua Li, Ruixuan Li
Intelligence
Status: succeeded | Model: anthropic/claude-sonnet-4.6 | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/24/2026, 1:52:49 AM
Summary
This paper addresses the local misalignment problem in CLIP-based Cross-Domain Few-Shot Learning (CDFSL), where domain gaps and scarce training data exacerbate CLIP's inability to capture fine-grained local visual patterns in target domains. The authors propose CC-CDFSL, a cycle consistency-based regularization framework with two cycle paths (T-I-T and I-T-I) and a Semantic Anchor mechanism to improve local vision-language alignment, interpretability, and classification performance across various benchmarks, backbones, and fine-tuning methods.
Entities (34)
Relation Signals (27)
CC-CDFSL → addresses → Local Misalignment Problem
confidence 99% · we propose the CC-CDFSL method with cycle consistency...to address this problem
Yaze Zhao → affiliatedwith → Huazhong University of Science and Technology
confidence 99% · Yaze Zhao, Yixiong Zou, Yuhua Li, Ruixuan Li School of Computer Science and Technology, Huazhong University of Science and Technology
Ruixuan Li → affiliatedwith → Huazhong University of Science and Technology
confidence 99% · Yaze Zhao, Yixiong Zou, Yuhua Li, Ruixuan Li School of Computer Science and Technology, Huazhong University of Science and Technology
Yuhua Li → affiliatedwith → Huazhong University of Science and Technology
confidence 99% · Yaze Zhao, Yixiong Zou, Yuhua Li, Ruixuan Li School of Computer Science and Technology, Huazhong University of Science and Technology
Yixiong Zou → affiliatedwith → Huazhong University of Science and Technology
confidence 99% · Yaze Zhao, Yixiong Zou, Yuhua Li, Ruixuan Li School of Computer Science and Technology, Huazhong University of Science and Technology
CC-CDFSL → appliedto → Cross-Domain Few-Shot Learning
confidence 99% · CC-CDFSL method...for CLIP-based CDFSL scenarios
CC-CDFSL → uses → Cycle Consistency
confidence 99% · we propose a regularization framework based on cycle consistency, named CC-CDFSL
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Cross-Domain Few-Shot Learning (CDFSL) adapts models trained with large-scale general data (source domain) to downstream target domains with only scarce training data, where the research on vision-language models (e.g., CLIP) is still in the early stages. Typical downstream domains, such as medical diagnosis, require fine-grained visual cues for interpretable recognition, but we find that current fine-tuned CLIP models can hardly focus on these cues, albeit they can roughly focus on important regions in source domains. Although current works have demonstrated CLIP's shortcomings in capturing local subtle patterns, in this paper, we find that the domain gap and scarce training data further exacerbate such shortcomings, much more than that of holistic patterns, which we call the local misalignment problem in CLIP-based CDFSL. To address this problem, due to the lack of supervision in aligning local visual features and text semantics, we turn to self-supervision information. Inspired by the translation task, we propose the CC-CDFSL method with cycle consistency, which translates local visual features into text features and then translates them back into visual features (and vice versa), and constrains the original features close to the translated back features. To reduce the noise imported by richer information in the visual modality, we further propose a Semantic Anchor mechanism, which first augments visual features to provide a larger corpus for the text-to-image mapping, and then shrinks the image features to filter out irrelevant image-to-text mapping. Extensive experiments on various benchmarks, backbones, and fine-tuning methods show we can (1) effectively improve the local vision-language alignment, (2) enhance the interpretability of learned patterns and model decisions by visualizing patches, and (3) achieve state-of-the-art performance.
Tags
Links
- Source: https://arxiv.org/abs/2603.17655v2
- Canonical: https://arxiv.org/abs/2603.17655v2
Trouble viewing inline? Open PDF directly →
Full Text
66,711 characters extracted from source content.
Expand or collapse full text
Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignment Yaze Zhao, Yixiong Zou, Yuhua Li, Ruixuan Li School of Computer Science and Technology, Huazhong University of Science and Technology zyaz, yixiongz, idcliyuhua, rxli@hust.edu.cn Corresponding author. Abstract Cross-Domain Few-Shot Learning (CDFSL) adapts models trained with large-scale general data (source domain) to downstream target domains with only scarce training data, where the research on vision-language models (e.g., CLIP) is still in the early stages. Typical downstream domains, such as medical diagnosis, require fine-grained visual cues for interpretable recognition, but we find that current fine-tuned CLIP models can hardly focus on these cues, albeit they can roughly focus on important regions in source domains. Although current works have demonstrated CLIP’s shortcomings in capturing local subtle patterns, in this paper, we find that the domain gap and scarce training data further exacerbate such shortcomings, much more than that of holistic patterns, which we call the local misalignment problem in CLIP-based CDFSL. To address this problem, due to the lack of supervision in aligning local visual features and text semantics, we turn to self-supervision information. Inspired by the translation task, we propose the C-CDFSL method with cycle consistency, which translates local visual features into text features and then translates them back into visual features (and vice versa), and constrains the original features close to the translated back features. To reduce the noise imported by richer information in the visual modality, we further propose a Semantic Anchor mechanism, which first augments visual features to provide a larger corpus for the text-to-image mapping, and then shrinks the image features to filter out irrelevant image-to-text mapping. Extensive experiments on various benchmarks, backbones, and fine-tuning methods show we can (1) effectively improve the local vision-language alignment, (2) enhance the interpretability of learned patterns and model decisions by visualizing patches, and (3) achieve state-of-the-art performance. Code is available at C-CDFSL. 1 Introduction Figure 1: Fine-grained visual cues (marked as red boxes) are crucial in specialized fields like interpretable medical diagnosis (a). However, in target domains, the fine-tuned CLIP cannot focus on these subtle patterns (b), but in source domains (c), CLIP can still roughly capture all important regions for recognition. Therefore, we hypothesize that the domain gap and scarce training data exacerbate CLIP’s shortcomings in capturing subtle patterns, much more than that of holistic patterns, which we aim to address. Deep learning has achieved breakthrough progress in computer vision [4, 7, 46, 44, 19]. However, these achievements are mainly in general domains where large-scale training data is available. In real-world applications, particularly in downstream domains like the medical diagnosis of rare diseases, acquiring sufficient labeled samples is not only costly but also often constrained by privacy and security concerns. To address this practical challenge, Cross-Domain Few-Shot Learning (CDFSL) [12, 54] has emerged. Its objective is to adapt models from a data-rich source domain to downstream target domains with few labeled data. Recently, vision-language foundation models like CLIP [29], pretrained on vast image-text pairs, have provided a robust foundation [40, 36] for CDFSL due to their generalizability. However, the research is still in its early stages [48]. Figure 2: (a) To validate our hypothesis, we measure the alignment score between global / local features and text features, and we find the alignments of both global and local features are harmed under the CDFSL task, but the local features show a larger decline in the alignment score, verifying our hypothesis. (b) Our proposed method effectively improves the local alignment in target domains. Typical downstream domains, such as medical diagnosis, always rely on fine-grained visual cues for interpretable recognition. For example, as illustrated in Fig. 1a, different categories of lung diseases in X-ray images may manifest as subtle abnormalities in texture or density within the lung area [35], as shown by red boxes in Fig. 1a. However, in Fig. 1b, we find current CLIP models fine-tuned on target domains always fail to focus on these subtle visual patterns, where the heatmap can only coarsely outline the contour of the body (e.g., the lung border or road edge), but cannot focus on these red boxes. Similar phenomena are also observed in other target-domain images. However, it may not be an issue in source-domain images (Fig. 1c), since the model can roughly highlight all important areas of the image. Since some works have found CLIP’s shortcomings in capturing subtle local patterns, we hypothesize that the domain gap and scarce training data exacerbate CLIP’s shortcomings in capturing subtle local patterns more than holistic patterns. To quantitatively validate this issue, we further measure the alignment scores [8] between CLIP’s global features (CLS token), or local patch features, and their text semantics (Fig. 2a). The results show that the domain gap harms the alignment between visual and text features, where the local features are more vulnerable, verifying our hypothesis. Therefore, in this paper, we aim to address the misalignment between local visual features and text features exacerbated by CDFSL (Fig. 2b). To achieve this goal, following the pretraining of CLIP, during the target-domain fine-tuning, we need to align each local visual feature (patch feature) with its corresponding text descriptions. However, firstly, image patches lack supervision that provides direct labels of texts, making it difficult to directly associate patch features with text semantics. Secondly, patches may contain noise or irrelevant regions (e.g., background) due to richer information in the visual modality, and there is insufficient supervision to remove the noise in few-shot scenarios. To solve these problems, we turn to self-supervision information for the local feature alignments. Inspired by translation [25, 22], we propose a regularization framework based on cycle consistency, named C-CDFSL, to explicitly constrain the bidirectional mapping between local visual features and text features, just like translating local visual features into text features and then translating them back into visual features (and vice versa). Specifically, we design two cycle paths: Text-to-Image-to-Text (T-I-T) and Image-to-Text-to-Image (I-T-I). To reduce the noise or irrelevant regions in the visual modality, we further propose a Semantic Anchor (SA) mechanism, which first augments visual features to provide a larger corpus for the text-to-image mapping, and then shrinks the effective visual features to filter out irrelevant image-to-text mappings. Additionally, these cycle paths also enhance interpretability by enabling visualization of patch-level semantic similarities, providing insights into learned patterns and model decisions for interpretable CDFSL. In summary, our contributions are listed as follows: • To the best of our knowledge, we are the first to find and address the local-feature-alignment problem in the CLIP-based CDFSL scenarios. • To address this problem, we propose the C-CDFSL method to provide self-supervision between local visual features and text features, by translating local visual features into text features and then translating back into visual features (and vice versa), and constraining the original features close to the translated back features. • To reduce the noise caused by the richer information in the visual modality, we further propose a Semantic Anchor mechanism, which first augments the visual features and then shrinks them to improve the fine-grained mapping between modalities. • Extensive experiments on various datasets, backbones, and fine-tuning methods demonstrate that C-CDFSL consistently improves performance and outperforms state-of-the-arts with better patch-level interpretability. 2 Related Work Cross-Domain Few-Shot Learning aims to enable a model trained on a data-rich source domain to rapidly adapt to a target domain. The main challenge is the significant distribution difference between the source and target domains [53]. Existing research primarily revolves around three perspectives: metric-based learning methods [39] achieve efficient classification by constructing similarity metrics in a feature space; meta-learning methods [33, 17] optimize rapid adaptation by simulating few-shot tasks; and transfer learning methods [28, 24] leverage pre-trained models from the source domain, adapting to the target domain through fine-tuning or feature alignment. Now vision-language models (VLMs) provide new opportunities for Source-Free Cross-Domain Few-Shot Learning [38, 47] (SF-CDFSL). Parameter-Efficient Fine-Tuning (PEFT) in VLMs. Recent research on the efficient fine-tuning of vision-language large models focuses on three primary approaches. Prompt-based methods [50, 49, 20] freeze encoder parameters, introducing trainable vectors into text or visual inputs to guide task-specific outputs. Adapter-based methods [10, 43] insert lightweight bottleneck modules into specific layers of pre-trained models, utilizing residual connections to integrate features for task adaptation. Low-Rank Adaptation (LoRA) methods [15, 41] employ low-rank matrix decomposition for weight updates, significantly reducing parameter counts while preserving inference efficiency. Although effective for in-domain fine-tuning [45], these methods only align image-level CLS tokens with text, leaving local patches without explicit semantic supervision. Consequently, fine-grained cues are prone to being overlooked when transferring across domains (Figs. 1 and 2). Local Alignment in VLMs. Some approaches explore local alignment during the pretraining phase of VLMs, such as transforming image embeddings into region representations given spatial hints [3], introducing region-text contrastive learning, or hard fine-grained negative sample learning [36]. Additionally, recent advancements in downstream fine-tuning have increasingly focused on local alignment strategies to enhance fine-grained cross-modal understanding. Notable approaches include employing region-based or local feature matching with LLM-generated text descriptions to capture detailed semantics [5, 2] or refine the similarity score [21]. However, these methods often rely on explicit region annotations or additional auxiliary losses requiring extensive labeled data, which is impractical in CDFSL scenarios where annotated data is scarce. In contrast, local alignment in CDFSL remains unexplored, and we find that the domain gap and limited data worsen local visual feature misalignment more than global features. So we leverage self-supervised cycle consistency to achieve robust patch-level alignment, which also enhances model interpretability. 3 Method 3.1 Preliminaries Cross-Domain Few-Shot Learning (CDFSL) requires a model trained on a source domain (e.g., miniImageNet) to adapt to target domains (e.g., EuroSAT) using a few labeled samples (K-way N-shot setting), and evaluates its classification performance on the target domain. In K-way N-shot classification tasks, the support set S contains K classes with N labeled samples each (N is usually small) , denoted as =(xi,yi)i=1K×NS=\(x_i,y_i)\_i=1^K× N, where the label yi∈1,…,Ky_i∈\1,…,K\. The query set =(xjq,yjq)Q=\(x_j^q,y_j^q)\ assesses the model’s performance, and ∩=∅S = . We mainly focus on the fine-tuning on the target domain with only ||=N⋅K|S|=N· K samples. Visual-Language Models exemplified by CLIP [29] have significantly advanced vision-related tasks. CLIP uses a dual-encoder architecture with separate image and text encoders. The image encoder, which can be a ResNet [13] or ViT [7], extracts features from an image I into a feature vector vI=fimage(I)∈ℝdv_I=f_image(I) ^d, where d is the feature dimension. In the Vision Transformer, the input image I∈ℝH×W×3I ^H× W× 3 is split into M=⌊Hp⌋×⌊Wp⌋M= Hp × Wp patches of size p×p×3p× p× 3. These patches are mapped to patch embeddings to form the initial image embedding sequence E0∈ℝM×dvE_0 ^M× d_v. After concatenating a class token c0c_0 and adding position embeddings, the sequence is processed by L Transformer layers. The final class token cLc_L is projected to obtain vI=Pv(cL)v_I=P_v(c_L). The text encoder processes the input text T′T by converting it into a sequence of discrete tokens. Each token is embedded into a vector to form the initial text embedding T0′∈ℝN×dtT _0 ^N× d_t. Special beginning (b0b_0) and end (e0e_0) tokens are added, and position embeddings are included. The sequence is then processed by L Transformer layers. The final text feature vector vT=ftext(T′)=Pt(eL)v_T=f_text(T )=P_t(e_L) is obtained by projecting the output vector of the EOS token eLe_L. For downstream image classification tasks with C categories and their corresponding text descriptions D1,D2,…,DCD_1,D_2,…,D_C, the image I is classified into the category with the highest similarity. Under the standard CLIP fine-tuning framework, global feature G∈ℝdG ^d generated by the CLS token is typically used for classification tasks. The original loss function is expressed as: ℒCE=−logp(y=c|x)=−logexp(sim(G,Tc)/τ)∑i=1Cexp(sim(G,Ti)/τ)L_CE=- p(y=c|x)=- (sim(G,T_c)/τ) _i=1^C (sim(G,T_i)/τ) (1) Here, τ is the temperature scaling parameter, and TcT_c is the text feature corresponding to the true class label c. Figure 3: Overview of our framework, consisting of three key components: (a) the Text-to-Image-to-Text (T-I-T) cycle-consistency module, (b) the Semantic Anchor (SA) module, and (c) the Image-to-Text-to-Image (I-T-I) cycle-consistency module. The process begins with the SA module augmenting raw images to create a larger corpus, followed by extracting local image features from these images and transforming them via MLP to align with the text feature space. The T-I-T cycle then uses text features to select semantically relevant patches and maps them back to reconstruct text features, enhancing local feature alignment. Subsequently, the SA module shrinks the feature set to select class-relevant anchor patches, which are used in the I-T-I cycle to map these anchor visual features through text features to augmented image features. Our model improves local alignment and interpretability in cross-domain few-shot learning. 3.2 C-CDFSL In the ViT architecture used in the image encoder, the local features of image patches contain rich visual information such as edges, shapes, and colors [42], making them important for interpretable diagnosis in specialized domains like medical imaging. As discussed earlier, CLIP pre-training relies solely on the CLS token to represent the whole image and leaves local patches unsupervised, leading to criticism of CLIP’s limited ability to capture fine-grained visual details [36]. We go one step further and find that downstream cross-domain tasks demand stronger local discrimination. For instance, Pneumothorax appears as a thin dark band along the lung border, while Nodules manifest as faint circular shadows in chest X-rays (Fig. 1a). Fig. 1b shows that target-domain fine-tuning harms CLIP’s local features more severely, so we hypothesize that the domain gap and scarce training data exacerbate CLIP’s local feature misalignment more than global feature misalignment. To validate this, we quantitatively measure the alignment scores for global and local patch features on the support set: Ag _g =1||∑i=1||icls⋅yi‖icls‖‖yi‖, = 1|S| _i=1^|S| v^cls_i·t_y_i\|v^cls_i\|\|t_y_i\|, Al _l =1||M∑i=1||∑j=1Mi,j⋅yi‖i,j‖‖yi‖ = 1|S|M _i=1^|S| _j=1^M v_i,j·t_y_i\|v_i,j\|\|t_y_i\| -8.5359pt (2) where M is the number of patches per image, and yi∈ℝdt_y_i ^d is the normalized text feature for the true class label yiy_i. Fig. 2a shows that in cross-domain tasks, the decrease in AlA_l is significantly greater than that of AgA_g, confirming that domain discrepancies amplify CLIP’s inability to capture fine-grained discriminative information. Drawing inspiration from this limitation, this study introduces a novel self-supervision strategy based on cycle consistency. By explicitly constraining the bidirectional mapping relationship between local features and text semantics, we effectively improve the local alignment between visual and textual modalities, and enhance interpretability. The overall framework of our method is shown in Figure 3. Semantic Anchor Module: Augmentation Phase. To reduce noise and irrelevant regions in the visual modality, we introduce the Semantic Anchor (SA) mechanism, operating in an augment-then-shrink manner. In the augmentation phase, we generate A augmented views for each image and include the original image, flattening all patch features into a single augmented feature matrix: aug∈ℝ((A+1)⋅M)×dX_aug ^((A+1)· M)× d (3) This enlarges the candidate corpus for the first hop of the T-I-T cycle. Next, we concatenate these features across the entire support set to form the local feature matrix: ′=Concat(aug(i)∣i∈)∈ℝH×d,H=||(A+1)ML =Concat (\X_aug^(i) i \ ) ^H× d, H=|S|(A+1)M (4) Local Features Transformation. Since CLIP pre-training uses global features to align visual and textual modalities, directly using local image features may result in a significant modality gap. Thus, we first transform local image features ′∈ℝH×dL ^H× d generated from all patches across the support set to reduce the distribution discrepancy with the textual feature space via a two-layer MLP: =MLP(′)=ReLU(′⋅W1)⋅W2L=MLP(L )=ReLU(L · W_1)· W_2 (5) Text-to-Image-to-Text Cycle Consistency. In the proposed T-I-T cycle, we first compute the cosine similarity matrix txt∈ℝC×HD^txt ^C× H between text features ∈ℝC×dT ^C× d (C is the number of categories) and local image features: j,itxt=j⋅i‖j‖‖i‖,[b]txt∈ℝC×M,b∈[1,2,…,||(A+1)]D^txt_j,i= T_j·L_i\|T_j\|\|L_i\|, ^txt_[b] ^C× M, b∈[1,2,...,|S|(A+1)] (6) Table 1: Accuracies (%) of target domain datasets of 5-way 1-shot and 5-shot tasks. Refer to the Appendix for the extended table. Task Method Mark backbone ISIC ChestX EuroSAT CropDisease Avg 5-way 1-shot StepSPT [37] TPAMI-25 ViT/CLIP 32.97 22.84 70.01 84.84 52.68 Tip-Adapter [43] ECCV-22 ViT/CLIP 32.68 22.24 75.44 77.15 51.87 AMU-Tuning [31] CVPR-24 ViT/CLIP 32.29 21.56 72.24 80.20 51.57 LP++ [18] CVPR-24 ViT/CLIP 33.63 21.72 73.05 81.84 52.56 LDC [23] CVPR-25 ViT/CLIP 33.72 22.32 74.39 84.07 53.62 CoOp [50] IJCV-22 ViT/CLIP 29.47 20.95 68.16 79.27 49.46 CoOp + OURS - ViT/CLIP 31.96 21.37 71.17 81.80 51.58 CLIP-Adapter [10] IJCV-24 ViT/CLIP 30.52 21.32 67.87 82.05 50.44 CLIP-Adapter + OURS - ViT/CLIP 32.47 22.14 72.89 82.01 52.37 Maple [20] CVPR-23 ViT/CLIP 33.56 20.89 75.05 82.67 53.04 Maple + OURS - ViT/CLIP 34.00 21.41 78.92 85.21 54.89 CLIP-LoRA [41] CVPR-24 RN50/CLIP 32.01 21.76 57.79 65.24 44.20 CLIP-LoRA + OURS - RN50/CLIP 35.21 22.75 59.23 72.85 47.51 CLIP-LoRA [41] CVPR-24 ViT/CLIP 35.23 21.73 81.49 85.11 55.89 CLIP-LoRA + OURS - ViT/CLIP 38.13 22.21 86.07 88.91 58.83 Δ - - +2.90 +0.48 +4.58 +3.80 +2.94 5-way 5-shot StepSPT [37] TPAMI-25 ViT/CLIP 52.12 26.36 89.40 96.01 65.97 Tip-Adapter [43] ECCV-22 ViT/CLIP 46.96 24.07 87.24 94.19 63.12 AMU-Tuning [31] CVPR-24 ViT/CLIP 44.60 23.34 88.47 94.26 62.66 LP++ [18] CVPR-24 ViT/CLIP 48.49 23.89 87.48 94.47 63.58 LDC [23] CVPR-25 ViT/CLIP 49.70 25.89 90.82 96.71 65.78 CoOp [50] IJCV-22 ViT/CLIP 42.56 22.22 85.53 93.48 60.95 CoOp + OURS - ViT/CLIP 42.75 22.76 86.43 93.82 61.44 CLIP-Adapter [10] IJCV-24 ViT/CLIP 44.09 23.53 84.48 93.34 61.36 CLIP-Adapter + OURS - ViT/CLIP 46.67 23.87 86.99 93.89 62.86 Maple [20] CVPR-23 ViT/CLIP 46.72 22.29 89.29 93.51 62.95 Maple + OURS - ViT/CLIP 48.77 23.05 91.72 95.16 64.67 CLIP-LoRA [41] zhengzaipao CVPR-24 RN50/CLIP 46.53 23.00 77.26 87.42 58.55 CLIP-LoRA + OURS - RN50/CLIP 49.11 23.55 79.23 90.99 60.72 CLIP-LoRA [41] CVPR-24 ViT/CLIP 50.68 24.44 92.63 96.20 65.99 CLIP-LoRA + OURS - ViT/CLIP 54.72 25.47 94.35 97.08 67.90 Δ - - +4.04 +1.03 +1.72 +0.88 +1.91 All features are processed to be L2-normalized for brevity in the rest, so cosine similarity is expressed as the dot product. Next, for each text feature jT_j, we select the most relevant local image feature to construct the similar image feature matrix ∗∈ℝC×dL^* ^C× d: j∗=argmaxij,itxt∈ℝdL^*_j=L_arg _iD^txt_j,i ^d (7) Then, we compute the reverse similarity matrix txtE^txt: txt=∗⋅⊤∈ℝC×CE^txt=L^*·T ^C× C (8) where j,ktxtE^txt_j,k represents the cosine similarity between the image patch feature j∗L^*_j (selected as the most relevant to the j-th text feature jT_j) and the k-th text feature kT_k. The cycle consistency loss is constructed by maximizing the cosine similarity between each text feature jT_j and the reconstructed text feature jrec∈ℝdT^rec_j ^d: ℒcyc_txt=1−1C∑j=1Csim(j,jrec)L_cyc\_txt=1- 1C _j=1^Csim(T_j,\,T^rec_j) (9) where jrec=argmaxkj,ktxtT^rec_j=T_\,arg _kE^txt_j,k is the text feature corresponding to the j-th text feature after reverse mapping. This process encourages the model to learn more discriminative and semantically meaningful local representations (Fig. 9). Semantic Anchor Module: Shrinking Phase. Before the I-T-I cycle, we filter out noisy or irrelevant patches to reduce semantic drift by selecting the most class-relevant anchor features. For the b-th image and the j-th class, we pick the top-k indices of the most similar patches: ℐj(b)=top−ki([b],j,itxt)I^(b)_j=top-k_i (D^txt_[b],j,i ) (10) We merge and deduplicate to obtain the anchor index set: ℐanchor=unique(⋃b=1||(A+1)⋃j=1Cℐj(b)),V=|ℐanchor|I_anchor=unique ( _b=1^|S|(A+1) _j=1^CI^(b)_j ), V= |I_anchor | (11) The anchor visual features are extracted as: anchor=ℐanchor∈ℝV×dX_anchor=L_I_anchor ^V× d (12) These anchor features represent the local features most relevant to class semantics, suppressing low-information regions (Fig. 7), and are used exclusively in the I-T-I cycle. Image-to-Text-to-Image Cycle Consistency. Mirroring the above T-I-T consistency process but in another direction, we use text features as an intermediary to achieve cyclic mapping from original to augmented image features. Specifically, for each anchor n∈anchorx_n _anchor, we leverage text features to bridge original and its augmented image features: 1. Identify the most similar text feature: tn=argmaxj(n⋅j)∈ℝdt_n=T_argmax_j(x_n·T_j) ^d (13) 2. Using the selected text feature tnt_n as an intermediary, retrieve the most similar patch only in the augmented feature space of nx_n (i.e., augX_aug ) to complete the cyclic mapping: m∗=argmaxm(tn⋅aug,m),^n=aug,m∗∈ℝdm^*=argmax_m(t_n·X_aug,m), x_n=X_aug,m^* ^d (14) We compute the cosine distance between anchors and their retrieved counterparts to construct the image-to-text-to-image cycle consistency loss: ℒcyc_img=1−1V∑n=1Vsim(n,^n)L_cyc\_img=1- 1V _n=1^Vsim(x_n, x_n) (15) This approach ensures reliable localization of local image features in the augmented space, enhancing the model’s robustness to input variations (Tab. 3). Consistency Loss. Finally, the total loss combines the original cross-entropy loss with both the text-to-text and image-to-image cycle consistency losses: ℒtotal=ℒCE+λ1ℒcyc_txt+λ2ℒcyc_imgL_total=L_CE+ _1L_cyc\_txt+ _2L_cyc\_img (16) where λ1 _1 and λ2 _2 are balancing hyperparameters. During inference, we assign the label to each query sample by computing the highest cosine similarity between its visual feature and the text features of all classes. 4 Experiments 4.1 Implementation Details According to the benchmark provided by [12], our model is directly fine-tuned on four target domain datasets: CropDiseases [27] (plant diseases), EuroSAT [14] (satellite imagery), ISIC2018 [6] (skin lesions), and ChestX [35] (chest X-rays). Following [37], all experiments in the main paper are conducted based on the CLIP model with the ViT-Base/16 backbone. Performance evaluations with other backbone variants are detailed in Supplementary Table 5. For all baseline methods, we adhere to the experimental settings reported in their respective original publications. For a fair comparison, all models are fine-tuned for 100 epochs on a single NVIDIA RTX 4090 GPU. Evaluation is conducted 100 times under the 1-shot setting and 400 times under the 5-shot setting, with the mean classification accuracy reported. For the hyper-parameters in Eq. 16, we set λ1 _1 and λ2 _2 based on empirical observation on a validation set via grid search; see the Appendix for details. k in the Semantic Anchor module is fixed to 10 across all experiments. 4.2 Comparison with State-of-the-Art Methods We evaluate our C-CDFSL framework in comparison with the most competitive state-of-the-art (SOTA) methods. These competing approaches cover representative technical paradigms in PEFT of vision-language models, including prompt learning (CoOp [50]) in the text branch, adapter-based methods (Tip-Adapter [43], CLIP-Adapter [10]), multi-modal prompt learning (Maple [20]), and low-rank adaptation (CLIP-LoRA [41]) —as well as recent advanced tuning strategies (AMU-Tuning [31], LP++ [18], LDC [23], StepSPT [37]). As presented in Tab. 1, our C-CDFSL consistently enhances the performance of all baseline methods. Figure 4: Base-to-new generalization on 11 datasets. To investigate whether the proposed C-CDFSL can be re-purposed beyond cross-domain few-shot learning, we further evaluate it on the widely used base-to-new generalization setting [49, 11, 20] across 11 commonly adopted datasets. As illustrated in Figure 4, we achieve substantial gains on challenging fine-grained datasets, such as EuroSAT (+3.6% in New), StanfordCars (+1.0% in New), and DTD (+2.7% in New). These results indicate that cycle-consistent patch-level alignment not only mitigates domain shift in few-shot settings but also enhances generalization to unseen classes by learning semantically richer local representations. 4.3 Ablation Study The effects of each component. According to Table 2, both cycles boost performance over the baseline, but the T-I-T cycle outperforms the I-T-I cycle. We identify two primary reasons for this: First, using the StanfordCars dataset as an example, where category names are vehicle names, image patches (e.g., tires or headlights) struggle to accurately match corresponding text features in few-shot scenarios due to the limited number of categories, resulting in text features that fail to effectively guide visual feature learning. Second, the noise or redundant information in image local features is difficult to completely eliminate, causing I-T-I reconstructed image features to deviate from their original semantics, whereas T-I-T focuses on the most semantically relevant patches, reducing interference from irrelevant patches and enhancing the robustness of local alignment. Retrieval strategies in I-T-I cycle. Tab. 3 presents the impact of different retrieval strategies on model performance. Cross-view retrieval, by using text features as an intermediary to match semantic anchor features only in the augmented image feature space, significantly enhances the model’s robustness to input transformations (e.g., rotation, flipping), adapting to distribution differences in cross-domain few-shot learning. In contrast, intra-image retrieval and all-images retrieval are limited to the original image feature space and overly large selection range separately, lacking the diversity of augmented views and being susceptible to noise, resulting in less accurate semantic alignment. Table 2: Ablation study of the 5-way 5-shot task. T-I-T I-T-I SA ISIC ChestX EuroSAT Crop. Ave. 50.68 24.44 92.63 96.20 65.98 ✓ 51.13 25.15 93.79 96.37 66.61 ✓ ✓ 54.30 25.35 94.33 96.95 67.73 ✓ ✓ 53.81 25.14 93.83 97.01 67.45 ✓ ✓ ✓ 54.72 25.47 94.35 97.08 67.90 Table 3: Impact of different retrieval strategies. Retrieval ISIC ChestX EuroSAT Crop. Ave. Cross-view 53.77 25.13 93.75 96.84 67.37 Intra-image 53.59 25.01 93.63 96.77 67.25 All-images 53.66 25.02 93.46 96.64 67.19 Hybrid coefficient λ1 _1 and λ2 _2 balance the contributions of T-I-T and I-T-I cycle losses, respectively. We conducted a grid search over [0.0,7.0][0.0,7.0], finding that λ1=3.0 _1=3.0 and λ2=2.0 _2=2.0 yield the best accuracy on ISIC2018 (see Fig. 5). As shown in the line graph, increasing λ1 _1 enhances the T-I-T path’s contribution, significantly improving performance. However, when λ1>3.0 _1>3.0, performance plateaus, indicating that excessive focus on local alignment may neglect global features. The effect of λ2 _2 is similar but more pronounced, as the I-T-I path is more susceptible to noise. Figure 5: Ablation study on hybrid coefficient in Eq. 16. Anchor Patch Number k in Eq. 10. In Fig. 6, we investigate the influence of the anchor-patch budget k (as defined in Eq. 10, controlling the number of patches that survive the Semantic Anchor shrinking phase and enter the I-T-I cycle for each image-class pair) on 5-way 5-shot accuracy. Results demonstrate that a small k(≤3)k(≤ 3) under-utilizes informative cues, reducing the average accuracy from 54.67% to 54.28%, whereas an excessively large k(≥13)k(≥ 13) re-introduces noisy regions and pushes accuracy back down to 54.21%. Setting k=10k=10 strikes an optimal balance, preserving fine-grained semantics while suppressing distractors and yielding the peak 54.72% on the ISIC2018 dataset. Therefore, we adopt k=10k=10 throughout all main experiments across the four datasets. Figure 6: Ablation study on hyperparameter k in Eq. 10. 4.4 Meaningful Patch Selection in SA As shown in Fig. 7, the Semantic Anchor module only retains the top-k most reliable patch-text matches, filtering out ambiguous mappings and reducing semantic drift. Figure 7: Semantic Anchor (SA) selection across four domains. Red boxes indicate the top-k=10k\!=\!10 patch proposals retained by the SA shrinking phase (Eq. 11). 4.5 Improved Local Alignment Fig. 8 illustrates the heatmaps of the baseline model and our C-CDFSL across four datasets using Grad-CAM [30]. The comparison reveals that our method reduces interference from irrelevant features, highlighting image regions most relevant to the text. This demonstrates that leveraging textual semantic guidance significantly enhances the model’s capacity to concentrate on discriminative features during CDFSL. Additional results can be found in Supp. Figs. 16 and 17. Quantitative analysis in Fig. 2b also confirms that our method improves local alignment scores. Figure 8: Our method (the third row) outperforms the baseline (the second row) in highlighting key details, offering a clearer view of important features in data-scarce scenarios. 4.6 Interpretability T-I-T cycle pathway. To validate the interpretability of our approach, we show the T-I-T cycle consistency pathway, as illustrated in Figure 9. Taking the ChestX-ray image in pathway-1 as an example, the initial text label is “Infiltration”. Through the T-I-T pathway, the model identifies a local region of interest (highlighted by the red box) and maps this patch back to the text space, resulting in the reconstructed text “Mass”. Even though the reconstructed text does not exactly match the initial label, this inconsistency itself provides valuable interpretability. On one hand, it demonstrates that the model is able to focus on disease-relevant regions and detect potential abnormal structures within the image. On the other hand, in medical imaging, different pathological types such as “Infiltration” and “Mass” may share similar visual characteristics on chest X-rays, making such semantic deviations in patch-level reconstruction understandable. This phenomenon indicates that the model not only localizes abnormal areas but also captures the fine-grained semantic relationships between different diseases, providing a valuable reference for clinicians in their further analysis and diagnosis. Figure 9: The T-I-T pathway indicates that the model can focus on disease-relevant regions and capture fine-grained semantic relationships between different diseases, providing valuable interpretability for medical analysis even when the reconstructed text does not fully match the initial label. I-T-I cycle pathway. Figure 10 depicts the I-T-I cycle consistency pathway (Due to space constraints, we present more results in the Supplementary Material). It shows that the model can extract and retain essential semantics in the text domain, and that these semantics can, in turn, be used to re-focus on meaningful regions within the image. Activation Map and Semantic Anchors. Our model’s activation maps (Fig. 8) precisely highlight discriminative fine-grained cues. Semantic Anchor’s shrink phase provides interpretability by visualizing the model’s focus (Fig. 7). Figure 10: The I-T-I cycle pathway demonstrates how the model extracts and retains essential semantics in the text domain, which are then used to re-focus on meaningful regions within the image’s augmented feature space. Table 4: Prototype-classification accuracy of the vision encoder. ISIC ChestX EuroSAT CropDiseases CLIP-LoRA 49.83 23.79 93.0 95.85 + Ours 50.68 24.51 93.64 96.24 4.7 Beneficial to Discriminability Notably, the above inconsistency is not a model failure but rather evidence that our method successfully learns transferable, fine-grained visual concepts shared across semantically related categories. By identifying these common concepts, the model is compelled to learn more nuanced representations that capture the subtle distinctions between classes, thereby improving feature discriminability. To verify discriminability, we conducted prototype classification using only the vision encoder. Tab. 4 shows consistent improvements after applying our cycle-consistent regularizer. The enhanced feature discriminability is also visually confirmed by the t-SNE plots in Supplementary Figure 15. 5 Conclusion We proposed C-CDFSL, a self-supervised regularization framework that enhances local feature alignment in CLIP-based CDFSL. By introducing the T-I-T, I-T-I cycle, and complementary Semantic Anchor module, our method effectively captures fine-grained semantics, improving robustness and interpretability. Extensive experiments on various benchmark datasets demonstrate superior performance. Acknowledgments This work is supported by the National Natural Science Foundation of China under grants 62206102; the National Key Research and Development Program of China under grant 2024YFC3307900; the National Natural Science Foundation of China under grants 62436003, 62376103 and 62302184; Major Science and Technology Project of Hubei Province under grant 2025BAB011 and 2024BAA008; Hubei Science and Technology Talent Service Project under grant 2024DJC078; and Ant Group through CCF-Ant Research Fund. The computation is completed in the HPC Platform of Huazhong University of Science and Technology. References [1] D. Bolya, P. Huang, P. Sun, J. H. Cho, A. Madotto, C. Wei, T. Ma, J. Zhi, J. Rajasegaran, H. Rasheed, J. Wang, M. Monteiro, H. Xu, S. Dong, N. Ravi, D. Li, P. Dollár, and C. Feichtenhofer (2025) Perception encoder: the best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504.13181. Cited by: §13. [2] G. Chen, W. Yao, X. Song, X. Li, Y. Rao, and K. Zhang (2023) PLOT: prompt learning with optimal transport for vision-language models. In ICLR 2023, Cited by: §2. [3] H. Chen, Z. Lai, H. Zhang, X. Wang, M. Eichner, K. You, M. Cao, B. Zhang, Y. Yang, and Z. Gan (2024) Contrastive localized language-image pre-training. CoRR abs/2410.02746. Cited by: §2. [4] X. Chen, Y. Wang, Y. Du, S. Hassoun, and L. Liu (2023) On separate normalization in self-supervised transformers. In NeurIPS 2023, Cited by: §1. [5] H. Choi, Y. K. Jang, and C. Eom (2025) GOAL: global-local object alignment learning. In CVPR 2025, p. 4070–4079. Cited by: §2. [6] N. Codella, V. Rotemberg, P. Tschandl, M. E. Celebi, S. Dusza, D. Gutman, B. Helba, A. Kalloo, K. Liopyris, M. Marchetti, et al. (2019) Skin lesion analysis toward melanoma detection 2018: a challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03368. Cited by: §4.1, §6. [7] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021) An image is worth 16x16 words: transformers for image recognition at scale. In ICLR 2021, Cited by: §1, §3.1. [8] S. Eslami and G. de Melo (2024) Mitigate the gap: investigating approaches for improving cross-modal alignment in CLIP. CoRR abs/2406.17639. External Links: Link, Document, 2406.17639 Cited by: §1. [9] Y. Fu, Y. Xie, Y. Fu, and Y. Jiang (2023) StyleAdv: meta style adversarial training for cross-domain few-shot learning. In CVPR 2023, p. 24575–24584. Cited by: §11. [10] P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, and Y. Qiao (2024) CLIP-adapter: better vision-language models with feature adapters. Int. J. Comput. Vis. 132 (2), p. 581–595. Cited by: §2, Table 1, Table 1, §4.2. [11] Y. Guo and X. Gu (2025) MMRL: multi-modal representation learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025, p. 25015–25025. Cited by: §4.2. [12] Y. Guo, N. C. Codella, L. Karlinsky, J. V. Codella, J. R. Smith, K. Saenko, T. Rosing, and R. Feris (2020) A broader study of cross-domain few-shot learning. In Computer vision–ECCV 2020: 16th European conference, glasgow, UK, August 23–28, 2020, proceedings, part XXVII 16, p. 124–141. Cited by: §1, §4.1, §6, §6. [13] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In CVPR 2016,, p. 770–778. Cited by: §3.1. [14] P. Helber, B. Bischke, A. Dengel, and D. Borth (2019) Eurosat: a novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, p. 2217–2226. Cited by: §4.1, §6. [15] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In ICLR 2022, Cited by: §2. [16] S. X. Hu, D. Li, J. Stühmer, M. Kim, and T. M. Hospedales (2022) Pushing the limits of simple pipelines for few-shot learning: external data and fine-tuning make a difference. In CVPR 2022, p. 9058–9067. Cited by: §11. [17] Y. Hu and A. J. Ma (2022) Adversarial feature augmentation for cross-domain few-shot classification. In Computer Vision - ECCV 2022 - 17th European Conference, S. Avidan, G. J. Brostow, M. Cissé, G. M. Farinella, and T. Hassner (Eds.), Lecture Notes in Computer Science, Vol. 13680, p. 20–37. Cited by: §2. [18] Y. Huang, F. Shakeri, J. Dolz, M. Boudiaf, H. Bahig, and I. B. Ayed (2024) LP++: a surprisingly strong linear probe for few-shot clip. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: Table 1, Table 1, §4.2. [19] Y. Jiang, Y. Zou, Y. Li, and R. Li (2025) Revisiting pool-based prompt learning for few-shot class-incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 1303–1313. Cited by: §1. [20] M. U. Khattak, H. A. Rasheed, M. Maaz, S. H. Khan, and F. S. Khan (2023) MaPLe: multi-modal prompt learning. In CVPR 2023, p. 19113–19122. Cited by: §2, Table 1, Table 1, §4.2, §4.2, §9. [21] J. Li, H. Li, S. M. Erfani, L. Feng, J. Bailey, and F. Liu (2024) Visual-text cross alignment: refining the similarity score in vision-language models. In ICML 2024, Cited by: §2. [22] M. Li, H. Huang, L. Ma, W. Liu, T. Zhang, and Y. Jiang (2018) Unsupervised image-to-image translation with stacked cycle-consistent adversarial networks. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss (Eds.), Lecture Notes in Computer Science, Vol. 11213, p. 186–201. Cited by: §1. [23] S. Li, F. Liu, Z. Hao, X. Wang, L. Li, X. Liu, P. Chen, and W. Ma (2025-06) Logits deconfusion with clip for few-shot learning. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), p. 25411–25421. Cited by: Table 1, Table 1, §4.2. [24] H. Liang, Q. Zhang, P. Dai, and J. Lu (2021) Boosting the generalization capability in cross-domain few-shot learning via noise-enhanced supervised autoencoder. In ICCV 2021, p. 9404–9414. Cited by: §2. [25] M. Liu, T. M. Breuel, and J. Kautz (2017) Unsupervised image-to-image translation networks. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett (Eds.), p. 700–708. Cited by: §1. [26] R. Ma, Y. Zou, Y. Li, and R. Li (2025) Reconstruction target matters in masked image modeling for cross-domain few-shot learning. In AAAI-25, T. Walsh, J. Shah, and Z. Kolter (Eds.), p. 19305–19313. Cited by: §11. [27] S. P. Mohanty, D. P. Hughes, and M. Salathé (2016) Using deep learning for image-based plant disease detection. Frontiers in plant science, p. 215232. Cited by: §4.1, §6. [28] C. P. Phoo and B. Hariharan (2021) Self-training for few-shot transfer across extreme task differences. In ICLR 2021, Cited by: §2. [29] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021) Learning transferable visual models from natural language supervision. In ICML 2021, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, p. 8748–8763. Cited by: §1, §3.1. [30] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra (2017) Grad-cam: visual explanations from deep networks via gradient-based localization. In IEEE International Conference on Computer Vision, ICCV 2017, Cited by: §10.1, §4.5. [31] Y. Tang, Z. Lin, Q. Wang, P. Zhu, and Q. Hu (2024) AMU-tuning: effective logit bias for clip-based few-shot learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, p. 23323–23333. Cited by: Table 1, Table 1, §4.2. [32] M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafa, O. Hénaff, J. Harmsen, A. Steiner, and X. Zhai (2025) SigLIP 2: multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786. Cited by: §13. [33] H. Tseng, H. Lee, J. Huang, and M. Yang (2020) Cross-domain few-shot classification via learned feature-wise transformation. In ICLR 2020, Cited by: §2, §6. [34] R. Walsh, I. I. Osman, and M. S. Shehata (2023) Masked embedding modeling with rapid domain adjustment for few-shot image classification. IEEE Trans. Image Process., p. 4907–4920. Cited by: §11. [35] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers (2017) Chestx-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 2097–2106. Cited by: §1, §4.1, §6. [36] C. Xie, B. Wang, F. Kong, J. Li, D. Liang, G. Zhang, D. Leng, and Y. Yin (2025) FG-CLIP: fine-grained visual and textual alignment. CoRR abs/2505.05071. Cited by: §1, §2, §3.2. [37] H. Xu, L. Liu, T. Liu, S. Zhi, S. Sun, and M. Cheng (2025) Step-wise distribution-aligned style prompt tuning for source-free cross-domain few-shot learning. IEEE Transactions on Pattern Analysis and Machine Intelligence (), p. 1–16. External Links: Document Cited by: §11, §13, Table 1, Table 1, §4.1, §4.2. [38] H. Xu, L. Liu, S. Zhi, S. Fu, Z. Su, M. Cheng, and Y. Liu (2024) Enhancing information maximization with distance-aware contrastive learning for source-free cross-domain few-shot learning. IEEE Trans. Image Process., p. 2058–2073. Cited by: §11, §2. [39] M. Xue, Q. Huang, H. Zhang, J. Hu, J. Song, M. Song, and C. Jin (2024) ProtoPFormer: concentrating on prototypical parts in vision transformers for interpretable image recognition. In IJCAI 2024, p. 1516–1524. Cited by: §2. [40] L. Yuan, D. Chen, Y. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li, C. Liu, M. Liu, Z. Liu, Y. Lu, Y. Shi, L. Wang, J. Wang, B. Xiao, Z. Xiao, J. Yang, M. Zeng, L. Zhou, and P. Zhang (2021) Florence: A new foundation model for computer vision. CoRR abs/2111.11432. Cited by: §1. [41] M. Zanella and I. Ben Ayed (2024) Low-rank few-shot adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, p. 1593–1603. Cited by: §2, Table 1, Table 1, Table 1, Table 1, §4.2, §8. [42] F. Zeng, Z. Cheng, F. Zhu, H. Wei, and X. Zhang (2025) Local-prompt: extensible local prompts for few-shot out-of-distribution detection. In The Thirteenth International Conference on Learning Representations, Cited by: §3.2. [43] R. Zhang, W. Zhang, R. Fang, P. Gao, K. Li, J. Dai, Y. Qiao, and H. Li (2022) Tip-adapter: training-free adaption of CLIP for few-shot classification. In Computer Vision - ECCV 2022 - 17th European Conference, p. 493–510. Cited by: §2, Table 1, Table 1, §4.2. [44] Z. Zhang, G. Chen, Y. Zou, Z. Huang, Y. Li, and R. Li (2024) Micm: rethinking unsupervised pretraining for enhanced few-shot learning. In Proceedings of the 32nd ACM International Conference on Multimedia, p. 7686–7695. Cited by: §1. [45] Z. Zhang, G. Chen, Y. Zou, Z. Huang, and Y. Li (2025) Decoupling template bias in clip: harnessing empty prompts for enhanced few-shot learning. arXiv preprint arXiv:2512.08606. Cited by: §2. [46] Z. Zhang, G. Chen, Y. Zou, Y. Li, and R. Li (2024) Learning unknowns from unknowns: diversified negative prototypes generator for few-shot open-set recognition. In Proceedings of the 32nd ACM International Conference on Multimedia, p. 6053–6062. Cited by: §1. [47] Z. Zhang, G. Chen, Y. Zou, Y. Li, and R. Li (2026) Reclaiming lost text layers for source-free cross-domain few-shot learning. arXiv preprint arXiv:2603.05235. Cited by: §2. [48] Z. Zhang, Y. Zou, Y. Li, R. Li, and G. Chen (2026) Mind the discriminability trap in source-free cross-domain few-shot learning. arXiv preprint arXiv:2603.13341. Cited by: §1. [49] K. Zhou, J. Yang, C. C. Loy, and Z. Liu (2022) Conditional prompt learning for vision-language models. In CVPR 2022, p. 16795–16804. Cited by: §2, §4.2. [50] K. Zhou, J. Yang, C. C. Loy, and Z. Liu (2022) Learning to prompt for vision-language models. Int. J. Comput. Vis.. Cited by: §2, Table 1, Table 1, §4.2. [51] L. Zhuo, Z. Wang, Y. Fu, and T. Qian (2024) Prompt as free lunch: enhancing diversity in source-free cross-domain few-shot learning through semantic-guided prompting. CoRR abs/2412.00767. Cited by: §11, §13. [52] Y. Zou, Y. Liu, Y. Hu, Y. Li, and R. Li (2024) Flatten long-range loss landscapes for cross-domain few-shot learning. In CVPR 2024, p. 23575–23584. Cited by: §11. [53] Y. Zou, R. Ma, Y. Li, and R. Li (2024) Attention temperature matters in vit-based cross-domain few-shot learning. In NeurIPS 2024, Cited by: §11, §2. [54] Y. Zou, S. Yi, Y. Li, and R. Li (2024) A closer look at the CLS token for cross-domain few-shot learning. In NeurIPS 2024, Cited by: §1, §11. Supplementary Material 6 Detailed Dataset Description Our experimental setup follows the BSCD-FSL [12] benchmark, addressing the challenge of significant distributional shifts across four distinct target domain datasets. Detailed information on these datasets is provided below: CropDiseases [27] is a dataset including 54,306 images of 14 crop species (Apple, Blueberry, Cherry, Corn, Grape, Orange, Peach, Bell Pepper, Potato, Raspberry, Soybean, Squash, Strawberry, and Tomato) with 26 diseases (or healthy). The samples of this dataset are listed in Fig. 11. CropDiseases images are natural images, but are very specialized (specific to the agriculture industry), so the domain gap here is larger than in the previous cross-domain setting [33]. Figure 11: Samples from CropDiseases. EuroSAT [14] is a dataset for land use and land cover classification. EuroSAT based on Sentinel-2 satellite imagery, covers 13 spectral bands and consists of 10 categories including Industrial Buildings, Residential Buildings, Annual Crop, Permanent Crop, River, Sea & Lake, Herbaceous Vegetation, Highway, Pasture and Forest, with a total of 27,000 annotated and geographically referenced images. Compared to CropDiseases, EuroSAT images are less similar to miniImagenet (source domain dataset) as they have lost perspective distortion, but are still color images of natural scenes. The samples of this dataset are listed in Fig. 12. Figure 12: Samples from EuroSAT. The ISIC2018 [6] dataset was published by the International Skin Imaging Collaboration (ISIC) as a large-scale dataset of dermoscopy images containing 10,015 images of seven skin injury types (melanoma, melanocytic nevus, basal cell carcinoma, actinic keratosis, benign keratosis, dermatofibroma, or a vascular lesion). ISIC2018 images are even less similar to miniImagenet as they have lost perspective distortion and no longer represent natural scenes. The samples of this dataset are listed in Fig. 13. Figure 13: Samples from ISIC2018. ChestX-ray14 is the largest lung X-ray database to date, which contains more than 100,000 pre-X-ray views for 14 lung diseases. Categories 1 to 14 correspond to 14 lung diseases, and category 15 indicates no disease. [35] studied the images of eight diseases in this database and constructed the ChestX-ray8 dataset, which comprises 108,948 frontal view X-ray images of 32,717 unique patients with the text-mined eight disease image labels (where each image can have multi-labels) from the associated radiological reports using natural language processing. In this work, we use ChestX-ray8 for cross-domain testing, consistent with [12]. ChestX is the most dissimilar to miniImagenet across the four target domains as its images have lost perspective distortion, do not represent natural scenes, and have lost 2 color channels. The samples of this dataset are listed in Fig. 14. Figure 14: Samples from ChestX. 7 Applying Our Method to other ViT Variants Table 5 presents a comparative analysis of our method against the CLIP-LoRA baseline across other ViT architectures, specifically ViT-B/32, and ViT-L/14, under the 1-shot setting. As observed, our method consistently demonstrates significant performance improvements over the baseline across all tested ViT variants and target domains. Table 5: Comparison with ViT-based methods in 1-shot. Backbone Method ChestX ISIC EuroSAT Crop. Ave. ViT-B/32 Baseline 21.23 36.32 79.79 81.13 54.62 + Ours 21.36 38.26 82.88 83.98 56.62 ViT-L/14 Baseline 22.90 38.45 85.83 89.83 59.25 + Ours 23.19 38.72 90.11 90.77 60.70 8 Experimental Data of Figure 2 Figure 2 in the main paper quantifies the hypothesis that the domain gap and scarce training data in CDFSL hurt patch-level local alignment more severely than global alignment. Table 6 reports these alignment scores for both global (CLS-token) and local (patch-level) representations of the baseline, CLIP-LoRA [41]. As observed, all cross-domain datasets exhibit lower local alignment scores compared to miniImageNet, with the exception of EuroSAT, whose images are dominated by foreground regions, confirming the baseline’s inability to capture fine-grained cues under data scarcity. To further validate the efficacy of the proposed C-CDFSL framework, we compare the local alignment scores before and after applying our method. Table 7 summarizes the results. Consistently, our method significantly improves these scores, demonstrating that the cycle-consistency and semantic anchor mechanisms successfully enhance patch-level semantics alignment under domain shift without additional annotations. Table 6: Alignment Scores of Global and Local Features. Dataset Global Feature Local Features Far Domain miniImageNet 0.7056 0.3821 ✗ ChestX 0.6768 0.2391 ✓ ISIC2018 0.7261 0.2360 ✓ EuroSAT 0.6768 0.3845 ✓ CropDiseases 0.7012 0.3811 ✓ Table 7: Comparison of Local Feature Alignment Scores. Method ChestX ISIC EuroSAT Crop. Ave. Baseline 0.2391 0.2360 0.3845 0.3811 0.3102 + Ours 0.3293 0.2661 0.3796 0.3877 0.3407 Table 8: Base-to-new generalization on 11 datasets. Ours (C-CDFSL) consistently improves new-class accuracy and harmonic mean (HM) over MaPLe, validating the cross-task generality of cycle-consistent patch-level alignment. (a) Average over 11 datasets. Base New HM MaPLe 82.19 74.55 78.18 + Ours 82.26 75.75 78.87 (b) ImageNet. Base New HM MaPLe 75.57 70.88 73.15 + Ours 75.59 71.10 73.28 (c) Caltech101. Base New HM MaPLe 98.02 94.43 96.19 + Ours 97.91 95.49 96.68 (d) OxfordPets. Base New HM MaPLe 95.64 97.80 96.71 + Ours 95.82 97.99 96.89 (e) StanfordCars. Base New HM MaPLe 72.56 73.80 73.17 + Ours 72.66 74.84 73.73 (f) Flowers102. Base New HM MaPLe 96.11 72.44 82.61 + Ours 96.07 73.90 83.19 (g) Food101. Base New HM MaPLe 90.75 91.84 91.29 + Ours 90.83 92.00 91.41 (h) FGVCAircraft. Base New HM MaPLe 38.34 34.59 36.37 + Ours 38.00 36.01 36.98 (i) SUN397. Base New HM MaPLe 80.92 78.21 79.54 + Ours 81.04 79.08 80.05 (j) DTD. Base New HM MaPLe 79.86 59.54 68.28 + Ours 79.90 62.28 70.00 (k) EuroSAT. Base New HM MaPLe 92.66 68.39 78.70 + Ours 93.57 71.98 81.37 (l) UCF101. Base New HM MaPLe 83.66 78.04 80.75 + Ours 83.51 79.07 81.23 9 Generalization to Base-to-New Setting The base-to-new generalization task refers to a scenario where a dataset is evenly divided by class into non-overlapping base classes and new classes. After a model is trained on a few-shot dataset from the base classes, its generalization ability is tested on the new classes. The harmonic mean (HM) of the classification accuracies on both is used to evaluate the overall performance: HM=21Base+1New=2⋅Base⋅NewBase+NewHM= 2 1Base+ 1New= 2·Base·NewBase+New (17) As shown in Table 14(l), C-CDFSL consistently outperforms MaPLe [20] across all datasets in terms of new-class accuracy and harmonic mean, with marginal improvements on base classes. 9.1 Better Class Separation As illustrated in the Figure 15, features extracted by CLIP-LoRA tend to form less compact and more overlapping clusters, with different classes not well separated in the embedding space. In contrast, our method produces more distinct and compact clusters for each class, resulting in clearer class boundaries and reduced intra-class variance. This indicates that our approach achieves better class separation compared to the baseline. For ChestX, t-SNE embeddings are omitted: the grayscale chest X-rays exhibit minuscule lesion regions that yield highly overlapping clusters, rendering any inter-method distinctions imperceptible. CropDiseases EuroSAT ISIC Figure 15: t-SNE visualization of feature distributions on different datasets. The first row shows results from CLIP-LoRA. The second row shows results from our proposed method. Different colors denote different classes. 10 Visualization 10.1 Improved Focus on Relevant Semantics Figure 16 and Figure 17 respectively present the attention maps and Grad-CAM [30] heatmaps produced by the baseline model and our proposed method across four datasets. This demonstrates that our method enhances the model’s ability to focus on critical features in cross-domain few-shot learning tasks and simultaneously improves local alignment between local visual patches and textual semantics. ISIC EuroSAT ChestX CropDiseases Figure 16: Row 2 and Row 3 display the attention maps of the baseline and C-CDFSL, respectively, showing that the latter localizes critical regions more precisely. CropDiseases EuroSAT ISIC ChestX Figure 17: The heatmap for CLIP-LoRA (the second row) and our C-CDFSL (the third row) in four target domains. Figure 18: Illustration of the Text-to-Image-patch-to-Text (T-I-T) cycle consistency pathway. Each specific class text is first used to locate the most semantically relevant local patch within the image. The selected image patch is then mapped back to the text space in an attempt to reconstruct the original class label. And the patches in the figure are resized to a resolution of 64×64 for better presentation. Figure 19: Additional visualizations of the Image-to-Text-to-Image (I-T-I) cycle consistency pathway. Each pathway shows a selected patch within the initial image, the most similar text derived from this patch, and a semantically re-focused patch within its corresponding image in the augmented image space. 11 Comparison with SOTA CDFSL Methods We also evaluate our C-CDFSL framework in comparison with the most competitive state-of-the-art (SOTA) CDFSL methods, covering a range of settings such as different backbones, the use of source datasets, and whether fine-tuning is performed on the target domain (FT). MEM-FS [34], StyleAdv [9], FLoR [52], DAMIM [26], AttnTemp [53], CD-CLS [54], PMF [16], IM-DCL [38], StepSPT [37], and SeGD-VPT [51] are introduced as competitors. Tabs. 9 and 10 present that our method achieves new SOTA average accuracies of 67.90% and 58.85% for the 5-way 5-shot and 5-way 1-shot classification tasks, respectively. Notably, IM-DCL [38] excels on ChestX, which we attribute to its ResNet-based backbone being better suited for capturing local features prevalent in medical images. Table 9: Comparison with state-of-the-art CDFSL works by the 5-way 5-shot classification. Method Backbone Mark Source Target CropDiseases EuroSAT ISIC ChestX Ave. MEM-FS ViT/DINO TIP-23 93.74 86.49 47.38 26.67 63.57 StyleAdv ViT/DINO CVPR-23 ✓ - 94.85 88.57 47.73 26.97 64.53 FLoR ViT/DINO CVPR-24 ✓ - 95.28 90.41 49.52 27.28 65.48 DAMIM ViT/DINO AAAI-25 ✓ - 95.52 89.50 50.76 27.28 65.77 AttnTemp ViT/DINO NeurIPS-24 ✓ - 95.53 90.13 53.09 27.72 66.62 CD-CLS ViT/DINO NeurIPS-24 ✓ ✓ 96.27 91.53 54.69 27.66 67.54 PMF ViT/DINO CVPR-22 ✓ ✓ 92.96 85.98 50.12 27.27 64.08 StyleAdv-FT ViT/DINO CVPR-23 ✓ ✓ 95.99 90.12 51.23 26.97 66.08 FLoR-FT ViT/DINO CVPR-24 ✓ ✓ 96.47 90.75 53.06 27.02 66.83 DAMIM-FT ViT/DINO AAAI-25 ✓ ✓ 96.34 91.18 54.86 27.82 67.78 IM-DCL RN10 TIP-24 - ✓ 95.73 89.47 52.74 28.93 66.72 StepSPT ViT/CLIP TPAMI-25 - ✓ 96.01 89.40 52.12 26.36 65.97 SeGD-VPT ViT/CLIP M-24 - ✓ 96.93 93.81 53.10 23.20 66.76 CLIP-LoRA ViT/CLIP CVPR-24 - ✓ 96.20 92.63 50.68 24.44 65.99 CLIP-LoRA + Ours ViT/CLIP Ours - ✓ 97.08 94.35 54.72 25.47 67.90 Δ - - - - +0.88 +1.72 +4.04 +1.03 +1.91 Table 10: Comparison with state-of-the-art CDFSL works by the 5-way 1-shot classification. Method Backbone Mark Source Target CropDiseases EuroSAT ISIC ChestX Ave. MEM-FS ViT/DINO TIP-23 81.11 68.11 32.97 22.76 51.24 StyleAdv ViT/DINO CVPR-23 ✓ - 81.22 72.15 33.05 22.92 52.34 FLoR ViT/DINO CVPR-24 ✓ - 81.81 72.39 34.20 22.78 52.80 DAMIM ViT/DINO AAAI-25 ✓ - 82.34 72.87 34.66 22.97 53.21 AttnTemp ViT/DINO NeurIPS-24 ✓ - 84.02 74.35 34.92 23.19 54.12 CD-CLS ViT/DINO NeurIPS-24 ✓ ✓ 84.53 74.97 35.56 23.39 54.62 PMF ViT/DINO CVPR-22 ✓ ✓ 80.79 70.74 30.36 21.73 50.91 StyleAdv-FT ViT/DINO CVPR-23 ✓ ✓ 84.11 74.93 33.99 22.92 53.99 FLoR-FT ViT/DINO CVPR-24 ✓ ✓ 83.55 73.09 35.49 23.26 53.85 DAMIM-FT ViT/DINO AAAI-25 ✓ ✓ 83.90 73.61 36.35 23.38 54.31 IM-DCL RN10 TIP-24 - ✓ 84.37 77.14 38.13 23.98 55.91 StepSPT ViT/CLIP TPAMI-25 - ✓ 84.84 70.01 32.97 22.84 52.68 CLIP-LoRA ViT/CLIP CVPR-24 - ✓ 85.11 81.49 35.23 21.73 55.89 CLIP-LoRA +Ours ViT/CLIP Ours - ✓ 88.91 86.07 38.13 22.21 58.83 Δ - - - - +3.80 +4.58 +2.90 +0.48 +2.94 12 Interpretability T-I-T cycle pathway Figure 18 presents additional visualizations of the Text-to-Image-to-Text (T-I-T) cycle consistency pathway. The pathways demonstrate the model’s ability to establish fine-grained semantic connections between textual descriptions and visual regions. Even when the reconstructed text does not exactly match the original text (e.g., “River” vs. “Highway or Road” of the EuroSAT for Pathway-2), the inconsistency provides valuable insights into the model’s understanding and reasoning at a local level. For instance, both “River” and “Highway or Road” categories represent linear structures (elongated shapes) in satellite imagery. Note that while the Semantic Anchor (SA) Module’s augmentation phase expands the corpus for the first hop of the T-I-T cycle, for display convenience, only the original image is used for retrieval in these visualizations. I-T-I cycle pathway As explicitly mentioned in the main paper, Figure 19 provides more detailed visualizations of the Image-to-Text-to-Image (I-T-I) cycle consistency pathway: the model (1) extracts semantics from an anchor patch (red box), (2) maps it to the most similar text label, and (3) retrieves a semantically matching patch only in the augmented view (red box). It highlights the model’s ability to maintain semantic consistency across different visual transformations and through the textual modality. 13 Generalization beyond CLIP We focus on the Source-Free Cross-Domain Few-Shot Learning (SF-CDFSL) benchmark established by StepSPT [37] and SeGD-VPT [51], where only a pre-trained model and scarce target-domain shots are available; CLIP is widely adopted as the default backbone in this setting for fair comparison. To verify our findings and method are generalizable to other baselines, we take SigLIP2 [32] and PE-Core [1] as two vision-language baselines with strengthened fine-grained representations. Specifically, SigLIP2 integrates Location-aware Captioners (LocCa) and vision-only self-supervised learning (including SILC and TIPS) to enhance dense prediction and localization capabilities. PE-Core aligns and tunes intermediate-layer features to capture fine-grained spatial representations. As presented in Tab. 11, our method achieves consistent performance gains across diverse backbones. Table 11: Accuracy on different backbones in 1-shot. Method Backbone ISIC ChestX EuroSAT CropDiseases Average CLIP-LoRA RN50/CLIP 32.01 21.76 57.79 65.24 44.20 + OURS RN50/CLIP 35.21 22.75 59.23 72.85 47.51 SigLIP2-LoRA ViT/SigLip2 26.48 20.53 63.05 81.84 47.98 + OURS ViT/SigLip2 29.53 22.00 68.12 83.39 50.76 PE-Core-LoRA ViT/PE-Core 38.05 22.45 82.16 89.01 57.92 + OURS ViT/PE-Core 40.72 22.67 83.92 90.48 59.45 Moreover, we also reproduce the alignment experiments (Fig. 2 in the paper) in Tab. 12. We can see that although these two methods strengthen the fine-grained representations, the phenomenon of degraded local alignment still exists under extreme domain shifts. Quantitatively, Tab. 12 shows that our method consistently improves local alignment scores by +26.3% for SigLIP2 and +18.6% for PE-Core, despite different ways to pretrain the VLM. Table 12: Feature alignments of other baselines. Method Feature type ISIC ChestX EuroSAT CropDiseases Average SigLIP2-LoRA global 0.1511 0.1380 0.1226 0.1691 0.14520 SigLIP2-LoRA local 0.0330 0.0310 0.0126 0.0383 0.02872 + OURS local 0.0421 0.0363 0.0181 0.0486 0.03627 PE-Core-LoRA global 0.2795 0.2493 0.2459 0.2810 0.26392 PE-Core-LoRA local 0.0191 0.0096 0.0264 0.0088 0.01597 + OURS local 0.0238 0.0111 0.0286 0.0123 0.01895 In summary, these experiments verify that our findings and designs are generalizable to other baselines. 14 Hyperparameters in Eq. 16 Table 13 lists the cycle-consistency weights λ1 _1 (T-I-T) and λ2 _2 (I-T- I) for each target dataset, determined by grid search on the validation split. All experiments use k=10k=10 for selecting anchor patches. Table 13: Hyperparameters λ1 _1 and λ2 _2 for each dataset. ChestX ISIC2018 EuroSAT CropDiseases λ1 _1 3 3 1.5 1 λ2 _2 0.5 2 0.2 1.5