Paper deep dive
CARE: Constrained Attention Refinement for Fine-Grained Visual Classification via Teacher-Student Distillation
Ruibo Wen, Hang Shao, Yiming Lei
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/9/2026, 8:23:08 AM
Summary
The paper introduces CARE, a framework for interpretable fine-grained visual classification (FGVC) using teacher-student distillation. It employs a frozen DINOv2 backbone, a class-specific attention student for prediction and explanation, and a training-only auxiliary query teacher that fuses multi-level intermediate representations. The student is regularized with diversity and sparsity losses to improve trait localization. CARE achieves state-of-the-art results on CUB, Oxford-IIIT Pet, Stanford Dogs, and Stanford Cars datasets.
Entities (11)
Relation Signals (7)
CARE → achievesaccuracyon → CUB
confidence 95% · reaching 78.5% Top-1 accuracy on CUB
CARE → usesbackbone → DINOv2
confidence 95% · CARE is built on a frozen DINOv2 Vision Transformer
CARE → employstechnique → Teacher-Student Distillation
confidence 92% · CARE... via teacher-student distillation
CARE → employstechnique → Logit-Standardized Knowledge Distillation
confidence 90% · transfers logit-standardized class-discriminative knowledge to the student through logit-standardized knowledge distillation
CARE → outperforms → Prompt-CAM
confidence 90% · Compared with Prompt-CAM, CARE increases Top-1 accuracy from 74.1% to 78.5% on CUB
CARE → usesloss → Diversity Loss
confidence 88% · We regularize these maps with diversity and sparsity losses
CARE → usesloss → Sparsity Loss
confidence 88% · The sparsity loss is the normalized entropy of the selected heads
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Fine-grained visual classification requires models to recognize subtle local traits while exposing the visual evidence behind their predictions. Class-specific attention pathways provide a natural basis for interpretable recognition, but their constrained prediction structure limits discriminative capacity and underuses intermediate representations from strong pretrained backbones. To address this problem, we propose CARE, a constrained attention refinement framework for interpretable fine-grained recognition via teacher-student distillation. CARE keeps the final prediction and explanation within a class-specific attention student, while introducing a training-only auxiliary query teacher that reads selected intermediate DINOv2 layers with learnable queries. The teacher fuses multi-level representations and transfers logit-standardized class-discriminative knowledge to the student. To further refine the explanation pathway, we design diversity and sparsity terms to regularize student attention heads, reducing redundancy and encouraging compact trait localization. Experiments on CUB, Oxford-IIIT Pet, Stanford Dogs, and Stanford Cars show that CARE achieves strong classification performance under an interpretable frozen-backbone setting, reaching 78.5% Top-1 accuracy on CUB. Faithfulness analysis with insertion and deletion metrics further indicates that the top-ranked attention regions retain class-relevant evidence for explanation.
Tags
Links
- Source: https://arxiv.org/abs/2610.11153v1
- Canonical: https://arxiv.org/abs/2610.11153v1
Trouble viewing inline? Open PDF directly →
Full Text
41,965 characters extracted from source content.
Expand or collapse full text
CARE: Constrained Attention Refinement for Fine-Grained Visual Classification via Teacher-Student Distillation Ruibo Wen Affiliation: College of Computer Science and Technology, Qingdao University, Qingdao, China E-mail leiyiming@qdu.edu.cn Hang Shao Affiliation: College of Computer Science and Technology, Qingdao University, Qingdao, China E-mail leiyiming@qdu.edu.cn Yiming Lei🖂 Affiliation: College of Computer Science and Technology, Qingdao University, Qingdao, China E-mail leiyiming@qdu.edu.cn Abstract Fine-grained visual classification requires models to recognize subtle local traits while exposing the visual evidence behind their predictions. Class-specific attention pathways provide a natural basis for interpretable recognition, but their constrained prediction structure limits discriminative capacity and underuses intermediate representations from strong pretrained backbones. To address this problem, we propose CARE, a constrained attention refinement framework for interpretable fine-grained recognition via teacher-student distillation. CARE keeps the final prediction and explanation within a class-specific attention student, while introducing a training-only auxiliary query teacher that reads selected intermediate DINOv2 layers with learnable queries. The teacher fuses multi-level representations and transfers logit-standardized class-discriminative knowledge to the student. To further refine the explanation pathway, we design diversity and sparsity terms to regularize student attention heads, reducing redundancy and encouraging compact trait localization. Experiments on CUB, Oxford-IIIT Pet, Stanford Dogs, and Stanford Cars show that CARE achieves strong classification performance under an interpretable frozen-backbone setting, reaching 78.5% Top-1 accuracy on CUB. Faithfulness analysis with insertion and deletion metrics further indicates that the top-ranked attention regions retain class-relevant evidence for explanation. The source code is publicly available at https://github.com/panpan0814/CARE. Keywords: Fine-grained recognition Interpretable vision transformers Visual prompt tuning Knowledge distillation †footnotetext: 🖂 Corresponding author. 1 Introduction Fine-grained visual classification (FGVC) aims to separate categories that differ in subtle local cues, including species, breeds, and object subtypes. In biodiversity monitoring, industrial inspection, and medical image analysis, a small visual detail may determine the correct category. Models used in these settings therefore need to expose the visual evidence behind their predictions, rather than rely on background bias or holistic shortcuts [13, 6]. Vision Transformers (ViTs) are strong backbones for FGVC because their token interactions capture both local parts and global context [5, 14, 6]. Self-supervised models such as DINO and DINOv2 further provide transferable frozen representations for downstream recognition tasks [2, 16]. Post-hoc attribution methods can explain decisions after a predictive model has been trained, but they do not require the classifier to use trait-level evidence when making predictions [20, 1, 3]. Intrinsically interpretable models connect prediction and explanation more directly, although they often introduce additional structures that are not always suitable for lightweight adaptation of frozen ViT backbones [18, 15]. Visual Prompt Tuning (VPT) adapts frozen ViTs with learnable visual prompt tokens and provides a parameter-efficient alternative to full fine-tuning [8]. Its focus, however, is mainly on recognition performance, and it does not provide class-specific visual explanations. Prompt-CAM extends prompt-based adaptation to intrinsic interpretability by using class-specific prompt outputs for prediction, with prompt-to-patch attention maps serving as class-level explanations [4]. This design is efficient and compatible with frozen ViTs, but prediction is still limited to final-layer prompt representations and a shared scoring head. As a result, it may not fully use intermediate Transformer representations that contain complementary local, part-level, and semantic evidence. We propose CARE, a constrained attention refinement framework for interpretable fine-grained recognition. CARE keeps a class-specific prompt-attention student as the final prediction and explanation pathway, and improves it through training supervision. Diversity and sparsity losses regularize the student attention heads to reduce redundancy and encourage compact trait localization. A training-only auxiliary query teacher reads selected intermediate DINOv2 layers and transfers multi-level discriminative knowledge to the student through logit-standardized knowledge distillation (KD) [7, 21]. During inference, the teacher branch is removed, so prediction and explanation are produced by the student alone. Our contributions are summarized as follows: • We formulate constrained attention refinement for interpretable FGVC, aiming to improve discriminative capacity while keeping prediction and explanation within a class-specific attention pathway. • We introduce a training-only auxiliary query teacher that reads multi-level DINOv2 representations and distills logit-standardized class-discriminative knowledge to the student. • We apply diversity and sparsity regularization to student attention heads, reducing redundancy and encouraging compact localization of fine-grained traits. • We evaluate CARE on CUB, Oxford-IIIT Pet, Stanford Dogs, and Stanford Cars using classification accuracy, insertion/deletion faithfulness, component ablations, and teacher-layer analysis. 2 Related Work Fine-grained visual classification. FGVC distinguishes visually similar categories using subtle local traits. Early methods relied on part localization, bilinear interactions, or region modeling [28, 13]. Transformers reduce the need for hand-crafted part modeling by representing images as patch tokens [5, 22, 14], and TransFG adapts ViTs to FGVC through discriminative token selection [6]. Furthermore, self-supervised foundation models such as DINO and DINOv2 provide strong frozen representations for downstream recognition [2, 16]. Our work studies how to exploit multi-level DINOv2 representations while preserving class-specific visual evidence for interpretable fine-grained recognition. Prompt learning and parameter-efficient adaptation. Prompt learning adapts pretrained models with a small number of trainable parameters. VPT introduces visual prompts into frozen ViTs [8], while CoOp and MaPLe adapt vision-language models with learned prompts [30, 10]. Recent work on PEFT further improves prompt-based adaptation [19]. These methods mainly target recognition performance rather than intrinsic explanation. Prompt-CAM extends prompt-based adaptation by using class-specific prompts for both prediction and explanation [4]. In contrast, CARE focuses on refining the class-specific attention pathway through attention regularization and training-only teacher-student distillation. Visual attribution and interpretability. Post-hoc methods such as Grad-CAM, Attention Rollout, and Transformer attribution explain trained classifiers after the decision function has been learned [20, 1, 3]. They are useful diagnostic tools, but they do not require the classifier itself to rely on interpretable visual evidence. Intrinsic methods instead build explanation structures into the model. MCTformer uses class tokens for localization [26], ProtoTree and ProtoPFormer use prototype reasoning [15, 27], and INTR uses class-specific queries in an encoder–decoder architecture [18]. Class-specific prompt-attention models offer a simpler and more parameter-efficient form of intrinsic explanation, but their constrained prediction pathway can limit discriminative capacity. CARE addresses this limitation by preserving the attention-based explanation pathway while strengthening it with multi-level teacher supervision. Knowledge distillation and intermediate representations. Knowledge distillation transfers softened outputs or intermediate features from a teacher to a student [7]. Decoupled distillation and logit standardization further refine the transferred signal [29, 21]. In ViTs, intermediate layers encode complementary cues, ranging from local patterns to semantic abstractions. Visual Query Tuning shows that learnable queries can access such representations efficiently [24]. Inspired by this idea, the Auxiliary Query Teacher in CARE reads selected DINOv2 layers, produces teacher logits, and transfers multi-level discriminative knowledge to the Constrained Attention Student only during training. 3 Methodology Figure 1: Overall architecture of CARE. The framework consists of (a) a Constrained Attention Student branch for prediction and explanation, and (b) a training-only Auxiliary Query Teacher branch for multi-layer distillation. 3.1 Overview Given an input image x and label y∈1,…,Cy∈\1,…,C\, our goal is to strengthen the discriminative capacity of a class-specific attention pathway while preserving it as the final prediction and explanation source. Fig. 1 shows the proposed CARE framework, which is built on a frozen DINOv2 Vision Transformer [16]. During training, CARE contains two branches: (a) a Constrained Attention Student for prediction and explanation, and (b) a training-only Auxiliary Query Teacher for multi-layer distillation. The Constrained Attention Student follows a class-specific prompt-attention design for prediction and explanation. Class-specific prompt outputs are scored by a shared vector to produce logits, and the corresponding prompt-to-patch multi-head attention maps are used as class-specific explanations. We regularize these maps with diversity and sparsity losses to reduce redundancy across heads and encourage more compact localization. This keeps the explanation source tied to the same class-specific pathway used for prediction. The Auxiliary Query Teacher is introduced to exploit hierarchical evidence that may not be fully represented by the final student prompt outputs. It reads features from selected intermediate layers, where cues such as local structure, part boundaries, texture, and higher-level semantics are useful for fine-grained recognition. The frozen backbone extracts both intermediate and final-layer features. The student produces logits and attention maps through the final class-specific attention pathway. In parallel, the teacher uses learnable query tokens to read selected intermediate layers, fuses the query-conditioned readouts, and predicts teacher logits. These logits supervise the student through logit-standardized KD [7, 21]. At inference, the teacher branch is removed, and the student alone provides both prediction and explanation. 3.2 Constrained Attention Student Branch The student branch follows the class-specific prompt-attention principle of Prompt-CAM [4]. Instead of using a generic CLS-token classifier, class-specific prompt tokens are used to compute category scores from their corresponding prompt outputs. The same prompts also query image patches through attention, so the prompt-to-patch attention maps can be used as class-specific visual evidence. Let FθF_θ denote the frozen DINOv2 ViT with N Transformer layers. The input image x is mapped to patch tokens 0∈ℝM×DE_0 ^M× D, where M is the number of image patches and D is the hidden dimension. We introduce class-specific prompt tokens =[1,…,C]⊤∈ℝC×DP=[p^1,…,p^C] ^C× D, where c∈ℝDp^c ^D is the learnable prompt token for class c. After the final Transformer layer, the class prompt outputs are Ns=[Ns,1,…,Ns,C]⊤∈ℝC×DZ_N^s=[z_N^s,1,…,z_N^s,C] ^C× D. A shared scoring vector s∈ℝDw_s ^D and bias bs∈ℝb_s map each class prompt to a scalar logit: scs=s⊤Ns,c+bs,c=1,…,C.s_c^s=w_s z_N^s,c+b_s, c=1,…,C. (1) The student logit vector and classification loss are s=[s1s,…,sCs]⊤∈ℝC,ℒclss=−logexp(sys)∑c=1Cexp(scs).s^s=[s_1^s,…,s_C^s] ^C, _cls^s=- (s_y^s) _c=1^C (s_c^s). (2) For class c and attention head r, let c,r∈ℝM α^c,r ^M denote the final-layer prompt-to-patch attention map over image patches. With R attention heads, the attention-map set for class c is c=c,rr=1RA^c=\ α^c,r\_r=1^R. The Constrained Attention Student keeps the class-specific prediction pathway and adds ℒdivL_div and ℒspL_sp to regularize the student attention maps. 3.3 Auxiliary Query Teacher Branch The Auxiliary Query Teacher reads features from selected layers of the frozen backbone. Let the frozen backbone contain N Transformer layers indexed as ℓ0,ℓ1,…,ℓN−1 _0, _1,…, _N-1. The selected teacher-layer set is denoted as =ℓi,ℓj,ℓkS=\ _i, _j, _k\, where 0≤i<j<k≤N−10≤ i<j<k≤ N-1. Unless otherwise stated, we use the zero-based DINOv2 layer indices 4,8,11\4,8,11\. For layer ℓ∈ , the context tokens are ℓ=[ℓ1,…,ℓMℓ]⊤∈ℝMℓ×DH_ =[h_ ^1,…,h_ ^M_ ] ^M_ × D, where MℓM_ denotes the number of context tokens available to the teacher branch. These tokens contain the intermediate non-prompt representations produced by the frozen backbone. We introduce NqN_q learnable query tokens ℓ∈ℝNq×DU_ ^N_q× D for each selected layer, with Nq=8N_q=8 in the main configuration. The query, key, and value matrices are ℓ _ =ℓℓQ, =U_ W_ ^Q, ℓ _ =ℓℓK, =H_ W_ ^K, ℓ _ =ℓℓV, =H_ W_ ^V, (3) where ℓQ,ℓK,ℓV∈ℝD×DW_ ^Q,W_ ^K,W_ ^V ^D× D. The layer-wise cross-attention matrix is ℓ=softmaxctx(ℓℓ⊤D)∈ℝNq×Mℓ,A_ =softmax_ctx ( Q_ K_ D ) ^N_q× M_ , (4) where the softmax is applied over the context-token dimension. Equivalently, Aℓ,q,m=exp(⟨ℓq,ℓm⟩/D)∑m′=1Mℓexp(⟨ℓq,ℓm′⟩/D).A_ ,q,m= ( _ ^q,K_ ^m / D) _m =1^M_ ( _ ^q,K_ ^m / D). (5) The query-conditioned readout is ℓ=ℓℓ∈ℝNq×D.R_ =A_ V_ ^N_q× D. (6) In implementation, this readout is realized as a residual multi-head cross-attention module followed by a lightweight feed-forward network, and the readout dimension is kept the same as the backbone embedding dimension. We average the query readouts and fuse the selected layers: ρ(ℓ)=1Nq∑q=1Nqℓq,=ϕ(Concat[ρ(ℓi),ρ(ℓj),ρ(ℓk)]),ρ(R_ )= 1N_q _q=1^N_qR_ ^q, =φ (Concat[ρ(R_ _i),ρ(R_ _j),ρ(R_ _k)] ), (7) where ρ(ℓ)∈ℝDρ(R_ ) ^D, ϕ:ℝ3D→ℝDφ:R^3D ^D is a lightweight fusion MLP, and ∈ℝDg ^D is the fused teacher representation. The teacher logits and auxiliary loss are t=t+t,ℒaux=−logexp(syt)∑c=1Cexp(sct).s^t=W_tg+b_t, _aux=- (s_y^t) _c=1^C (s_c^t). (8) Here, t∈ℝC×DW_t ^C× D, t∈ℝCb_t ^C, and t∈ℝCs^t ^C. The teacher branch is used only for training-time supervision and is not used as the explanation source. 3.4 Logit-Standardized Knowledge Distillation Teacher and student logits may have different scales, so we standardize them before distillation [21]. For ∈ℝCs ^C, ^=−μ()σ()+ϵ,μ()=1C∑j=1Csj,σ()=(1C∑j=1C(sj−μ())2)1/2. s= s-μ(s)1σ(s)+ε, μ(s)= 1C _j=1^Cs_j, σ(s)= ( 1C _j=1^C(s_j-μ(s))^2 )^1/2. (9) Here, ∈ℝC1 ^C is an all-one vector and ϵε is a small constant for numerical stability. The standardized teacher and student distributions are t=softmax(sg(^t)T),s=softmax(^sT),q^t=softmax ( sg( s^t)T ), ^s=softmax ( s^sT ), (10) where T>0T>0 is the distillation temperature, t,s∈ℝCq^t,q^s ^C are probability distributions, and sg(⋅)sg(·) denotes stop-gradient. Thus, the KD term transfers knowledge from the teacher to the student without updating the teacher through the distillation loss. The KL divergence and KD loss are defined as: KL(t∥s)=∑c=1Cqctlog(qct+ϵqcs+ϵ),ℒKD=T2KL(t∥s).KL(q^t\,\|\,q^s)= _c=1^Cq_c^t ( q_c^t+εq_c^s+ε ), _KD=T^2KL(q^t\,\|\,q^s). (11) This loss transfers the teacher’s relative class distribution to the Constrained Attention Student. 3.5 Attention Diversity and Sparsity Regularization For the ground-truth class y, the normalized attention map of head r is ~y,r=y,r∑m=1Mαmy,r+ϵ∈ℝM, α^y,r= α^y,r _m=1^M _m^y,r+ε ^M, (12) where αmy,r _m^y,r denotes the attention value on patch m and ϵε is a small constant. The diversity loss is the average pairwise cosine similarity between different heads: ℒdiv=1R(R−1)∑r,r′=1r≠r′R(~y,r)⊤~y,r′‖~y,r‖2‖~y,r′‖2+ϵ.L_div= 1R(R-1) _ subarraycr,r =1\\ r≠ r subarray^R ( α^y,r) α^y,r \| α^y,r\|_2\| α^y,r \|_2+ε. (13) For sparsity, we first select the top-k attention heads of the ground-truth class and then regularize their entropy. The head-selection score is defined by the maximum prompt-to-patch attention response: ηry=maxm∈1,…,Mαmy,r. _r^y= _m∈\1,…,M\ _m^y,r. (14) The selected head index set is ℐky=TopKr∈1,…,R(sg(ηry),k),I_k^y=TopK_r∈\1,…,R\ (sg( _r^y),k ), (15) where sg(⋅)sg(·) denotes stop-gradient. The discrete top-k operation determines which heads are regularized, while gradients are back-propagated through the selected attention maps. The sparsity loss is the normalized entropy of the selected heads: ℒsp=1|ℐky|∑r∈ℐkyℋ(~y,r)logM,L_sp= 1|I_k^y| _r _k^y H ( α^y,r ) M, (16) where ℋ(~y,r)=−∑m=1Mα~my,rlog(α~my,r+ϵ).H ( α^y,r )=- _m=1^M α_m^y,r ( α_m^y,r+ε ). (17) Both regularizers act directly on the Constrained Attention Student attention maps. 3.6 Overall Objective and Training The total objective is ℒtotal=ℒclss+λauxℒaux+λKDℒKD+λdivℒdiv+λspℒsp. _total=\;L_cls^s+ _auxL_aux+ _KDL_KD+ _divL_div+ _spL_sp. (18) Here, λaux _aux, λKD _KD, λdiv _div, and λsp _sp are scalar loss weights. The attention regularizers do not introduce additional trainable parameters; they only impose constraints on student attention maps. The trainable parameters include the class-specific prompt tokens, the shared student scoring head, and the Auxiliary Query Teacher branch, while the DINOv2 backbone remains frozen. During inference, the final prediction and explanation are produced by the Constrained Attention Student branch, and the Auxiliary Query Teacher is removed. 4 Experiments 4.1 Datasets and Implementation Details Datasets We evaluate CARE on CUB-200-2011 [23], Stanford Dogs [11], Oxford-IIIT Pet [17], and Stanford Cars [12]. CUB is used for ablation studies, teacher-layer selection, and hyperparameter analysis. The remaining datasets test whether the observed behavior transfers across different fine-grained domains. Experimental settings All prompt-based models use a frozen DINOv2 ViT-B/14 backbone [16] with an input resolution of 224×224224× 224. Unless otherwise stated, models are trained for 60 epochs with SGD, momentum 0.9, weight decay 0.001, batch size 64, learning rate 0.005, and 10 warm-up epochs. For CARE, we use zero-based teacher-layer indices 4,8,11\4,8,11\, Nq=8N_q=8, λaux=0.5 _aux=0.5, λKD=1.0 _KD=1.0, T=2.0T=2.0, λdiv=0.05 _div=0.05, λsp=0.01 _sp=0.01, and k=4k=4 for sparsity regularization. Evaluation Metrics Classification performance is measured by Top-1 accuracy. For explanation faithfulness, we report insertion and deletion scores. Class-specific prompt-attention methods are evaluated using their intrinsic attention maps, while Grad-CAM, Layer-CAM, and Attention Rollout are evaluated on the DINOv2 Linear Probing classifier [20, 9, 1]. Baselines We compare CARE with both recognition-oriented and interpretable baselines. DINOv2 Linear Probing is included as a recognition-oriented reference with a frozen ViT-B/14 backbone and a linear head. For interpretable fine-grained recognition, we compare with INTR [18], ProtoTree [15], TesNet [25], and the class-specific prompt-attention baseline Prompt-CAM [4]. Unless otherwise stated, Linear Probing, Prompt-CAM, and CARE are evaluated under the same frozen DINOv2 setting, while the remaining interpretable baseline results follow the corresponding published reports. Prompt-CAM is the closest baseline, while CARE further introduces constrained attention refinement with training-only teacher-student distillation. 4.2 Main Results The classification results show that CARE consistently improves the class-specific prompt-attention baseline across all four fine-grained benchmarks, as reported in Table 1. Compared with Prompt-CAM, CARE increases Top-1 accuracy from 74.1% to 78.5% on CUB, from 81.3% to 81.9% on Stanford Dogs, from 92.7% to 93.6% on Oxford-IIIT Pet, and from 84.3% to 87.1% on Stanford Cars. These gains suggest that the auxiliary query teacher and logit-standardized distillation improve the discriminative ability of the class-specific attention pathway while preserving its intrinsic explanation mechanism. Table 1: Top-1 classification accuracy (%) on four fine-grained recognition benchmarks. Gray values denote recognition-oriented Linear Probing results provided for reference. Bold indicates the best result among the compared interpretable methods. Dataset Linear Probing INTR ProtoTree TesNet Prompt-CAM CARE CUB 89.6 71.8 68.5 77.7 74.1 78.5 Dog 87.3 72.5 68.4 76.5 81.3 81.9 Pet 96.2 90.4 – – 92.7 93.6 Cars 88.2 86.8 70.0 84.6 84.3 87.1 4.3 Faithfulness Analysis Explanation faithfulness is evaluated with insertion and deletion scores under the two settings defined above: post-hoc explanations on the DINOv2 Linear Probing classifier and intrinsic attention maps from class-specific prompt-attention methods. All methods follow the same perturbation protocol. Saliency maps are upsampled to the image resolution, pixels are ranked by descending saliency, and each score is computed as the area under the target-class confidence curve over 50 perturbation steps. The quantitative results are reported in Table 2. Under the intrinsic prompt-attention setting, CARE improves insertion over Prompt-CAM on CUB, Oxford-IIIT Pet, and Stanford Dogs. This suggests that the top-ranked attention regions selected by CARE contain more prediction-relevant evidence during the insertion process. On CUB, the regularization-only variant also improves insertion, indicating that diversity and sparsity constraints can improve the quality of selected-head evidence. Deletion results are more mixed: CARE is slightly worse than Prompt-CAM on CUB, but performs better on Oxford-IIIT Pet and Stanford Dogs. The corresponding insertion and deletion curves are shown in Fig. 2. Table 2: Insertion/deletion faithfulness evaluation under post-hoc and class-specific prompt-attention explanation settings. Prompt-CAM + Reg. denotes the student-only CARE variant with ℒdiv+ℒspL_div+L_sp, without the Auxiliary Query Teacher or KD. (a) Post-hoc methods on CUB Method Ins.↑ Del.↓ Grad-CAM 0.61 0.17 Layer-CAM 0.77 0.28 Att. Rollout 0.73 0.28 (b) Class-specific prompt-attention methods Method CUB Pet Dog Ins.↑ Del.↓ Ins.↑ Del.↓ Ins.↑ Del.↓ Prompt-CAM 0.78 0.13 0.79 0.40 0.68 0.31 Prompt-CAM + Reg. 0.85 0.15 – – – – CARE 0.86 0.16 0.86 0.39 0.83 0.29 (a) CUB (b) Oxford-IIIT Pet (c) Stanford Dogs Figure 2: Insertion and deletion curve comparisons on CUB, Oxford-IIIT Pet, and Stanford Dogs. The CUB curves include Prompt-CAM, Prompt-CAM with diversity and sparsity attention regularization, and CARE. The Oxford-IIIT Pet and Stanford Dogs curves include Prompt-CAM and CARE. Table 3: Component ablation study of CARE on the CUB dataset. The teacher branch includes ℒauxL_aux by default. CARE w/o Teacher/KD removes both the Auxiliary Query Teacher and KD, leaving only attention regularization on the student. Setting Teacher KD Reg. Top-1 Prompt-CAM – – – 74.1 CARE w/o Reg. ✓ ✓ – 78.3 CARE w/o Teacher/KD – – ✓ 73.4 CARE ✓ ✓ ✓ 78.5 4.4 Ablation Study Component ablation. The component ablation on CUB examines how each part of CARE contributes to the final performance, including the Auxiliary Query Teacher, logit-standardized KD, and attention regularization, as reported in Table 3. The teacher-distillation variant improves the Prompt-CAM baseline from 74.1% to 78.3%, indicating that multi-level teacher supervision is the main source of the accuracy gain. In contrast, applying ℒdivL_div and ℒspL_sp without teacher supervision reduces accuracy to 73.4%, suggesting that attention regularization alone can over-constrain the class-specific attention pathway. When teacher-guided distillation and attention regularization are combined, accuracy reaches 78.5%, outperforming both the Prompt-CAM baseline and the teacher-only variant. This result suggests that multi-level teacher supervision and attention regularization play complementary roles in improving the Constrained Attention Student. KD-weight sensitivity. Although the framework contains several loss weights, we focus the sensitivity analysis on λKD _KD because it directly controls the strength of knowledge transfer from the Auxiliary Query Teacher to the Constrained Attention Student. The accuracy curve in Fig. 3(a) shows that the best result is obtained at λKD=1.0 _KD=1.0, reaching 78.55% Top-1 accuracy on CUB. The faithfulness curves in Fig. 3(b) show that insertion and deletion scores vary only slightly across the tested KD weights. We therefore use λKD=1.0 _KD=1.0 as the default setting in the main experiments. (a) Classification accuracy (b) Faithfulness scores Figure 3: Sensitivity analysis of the KD loss weight λKD _KD on CUB. (a) reports Top-1 classification accuracy. (b) reports insertion and deletion faithfulness scores. All other hyperparameters are kept identical to the default CARE configuration. Prompt-CAM CARE Input Top-1 Top-2 Top-3 Top-4 Input Top-1 Top-2 Top-3 Top-4 CUB Pet Dog Car Figure 4: Visualization of class-specific attention maps from the top-4 attention heads on CUB, Oxford-IIIT Pet, Stanford Dogs, and Stanford Cars. Table 4: Query-token analysis on the CUB dataset. Setting NqN_q Top-1 Train. Params Total Params Prompt-CAM – 74.1 1.84M 87.56M CARE 4 77.1 25.05M 110.77M CARE 8 78.5 25.06M 110.78M CARE 16 76.7 25.07M 110.79M Table 5: Layer combination study on the CUB dataset. Teacher indices Top-1 4, 8, 11 78.5 5, 8, 11 76.9 4, 7, 11 76.7 4, 9, 11 76.6 3, 7, 10 77.1 4.5 Query Token Analysis The number of learnable query tokens controls the readout capacity of the Auxiliary Query Teacher for multi-level feature aggregation. We analyze this factor on CUB by fixing the selected teacher-layer indices to 4,8,11\4,8,11\ and changing only NqN_q, with Prompt-CAM included as the baseline without teacher query tokens. The results are reported in Table 5. With Nq=4N_q=4, CARE reaches 77.1% Top-1 accuracy, improving over Prompt-CAM but remaining below the default setting. This suggests that four query tokens may provide insufficient capacity to summarize intermediate representations. Increasing NqN_q to 8 raises accuracy to 78.5%, indicating that moderate query-token capacity helps the teacher extract multi-level evidence. Increasing NqN_q to 16 lowers accuracy to 76.7%, suggesting that excessive query tokens may introduce redundant or unstable readouts for distillation. The parameter counts change only marginally across different values of NqN_q, so the accuracy differences mainly reflect teacher readout capacity rather than parameter scaling. We therefore use Nq=8N_q=8 as the default setting. 4.6 Selecting Complementary Teacher Layers The selected teacher-layer indices control the depth distribution of the multi-level supervision provided by the Auxiliary Query Teacher. As reported in Table 5, the zero-based index combination 4,8,11\4,8,11\ achieves the best accuracy, while changing the early, middle, or late layer reduces performance. This suggests that the teacher branch benefits from complementary representations across different Transformer depths. This pattern is consistent with the hierarchy of ViT features: earlier layers preserve local appearance cues, middle layers provide part-level structure, and deeper layers encode semantic information. The 4,8,11\4,8,11\ combination therefore provides a more balanced teacher signal than alternatives at similar depths. 4.7 Qualitative Discussion Qualitative visualizations are used to examine whether the refined class-specific attention maps highlight meaningful fine-grained regions across CUB, Oxford-IIIT Pet, Stanford Dogs, and Stanford Cars, as shown in Fig. 4. Prompt-CAM and CARE follow the same visualization protocol for selecting the displayed attention heads. Specifically, we adopt the iterative head-pruning procedure used by the Prompt-CAM baseline: at each step, one candidate head is replaced by a uniform attention pattern, the model is re-evaluated, and the head whose removal causes the smallest decrease in target-class confidence is pruned. The four remaining heads are visualized as trait-level attention maps. This visualization-time selection is separate from the training-time sparsity regularization in Sec. 3.5. The selected heads generally correspond to semantically meaningful fine-grained regions, such as the head, beak, neck, ears, or other object-specific local parts. Since the heads are selected according to their contribution to target-class confidence rather than human visual preference, some maps may still contain contextual or diffuse responses. Compared with Prompt-CAM, CARE produces more compact and less redundant attention patterns in many examples. This qualitative behavior is consistent with the insertion/deletion results and supports the role of attention regularization in refining class-specific localization. 5 Limitations and Discussion CARE has two main limitations. First, the Auxiliary Query Teacher introduces additional training-time parameters and computational overhead, mainly from the learnable query tokens, multi-layer query readout, feature fusion, and teacher classifier. However, the teacher branch is discarded during inference, so the additional cost does not affect the deployed student model. Second, our experiments are conducted primarily on natural fine-grained datasets. Applying CARE to domains such as medical or industrial imagery requires further evaluation under domain-specific variations, robustness requirements, and reliability constraints. 6 Conclusion We presented CARE, a constrained attention refinement framework for faithful fine-grained visual classification. CARE uses a Constrained Attention Student and a training-only Auxiliary Query Teacher to transfer multi-level discriminative knowledge while preserving class-specific attention as the prediction and explanation source. Experiments on four fine-grained benchmarks show consistent accuracy gains over the class-specific prompt-attention baseline and competitive faithfulness results, indicating that teacher-student distillation and attention regularization can strengthen the interpretable attention pathway without replacing its intrinsic localization mechanism. Acknowledgements This work was supported in part by the National Natural Science Foundation of China (No. 62306075) and the Natural Science Foundation of Shandong Province (No. ZR2026QC0725). References [1] S. Abnar and W. Zuidema (2020) Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 4190–4197. Cited by: §1, §2, §4.1. [2] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin (2021) Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 9650–9660. Cited by: §1, §2. [3] H. Chefer, S. Gur, and L. Wolf (2021) Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 782–791. Cited by: §1, §2. [4] A. Chowdhury, D. Paul, Z. Mai, J. Gu, Z. Zhang, K. S. Mehrab, E. G. Campolongo, D. Rubenstein, C. V. Stewart, A. Karpatne, T. Berger-Wolf, Y. Su, and W. Chao (2025) Prompt-CAM: making vision transformers interpretable for fine-grained analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 4375–4385. Cited by: §1, §2, §3.2, §4.1. [5] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021) An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. [6] J. He, J. Chen, S. Liu, A. Kortylewski, C. Yang, Y. Bai, and C. Wang (2022) TransFG: a transformer architecture for fine-grained recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, p. 852–860. Cited by: §1, §1, §2. [7] G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, Cited by: §1, §2, §3.1. [8] M. Jia, L. Tang, B. Chen, C. Cardie, S. Belongie, B. Hariharan, and S. Lim (2022) Visual prompt tuning. In Proceedings of the European Conference on Computer Vision (ECCV), p. 709–727. Cited by: §1, §2. [9] P. Jiang, C. Zhang, Q. Hou, M. Cheng, and Y. Wei (2021) LayerCAM: exploring hierarchical class activation maps for localization. IEEE Transactions on Image Processing 30, p. 5875–5888. Cited by: §4.1. [10] M. U. Khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan (2023) MaPLe: multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 19113–19122. Cited by: §2. [11] A. Khosla, N. Jayadevaprakash, B. Yao, and L. Fei-Fei (2011) Novel dataset for fine-grained image categorization. In Proceedings of the CVPR Workshop on Fine-Grained Visual Categorization, Cited by: §4.1. [12] J. Krause, M. Stark, J. Deng, and L. Fei-Fei (2013) 3D object representations for fine-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops, p. 554–561. Cited by: §4.1. [13] T. Lin, A. RoyChowdhury, and S. Maji (2015) Bilinear CNN models for fine-grained visual recognition. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), p. 1449–1457. Cited by: §1, §2. [14] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo (2021) Swin transformer: hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 10012–10022. Cited by: §1, §2. [15] M. Nauta, R. van Bree, and C. Seifert (2021) Neural prototype trees for interpretable fine-grained image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 14933–14943. Cited by: §1, §2, §4.1. [16] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P. Huang, S. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jégou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski (2024) DINOv2: learning robust visual features without supervision. Transactions on Machine Learning Research. Cited by: §1, §2, §3.1, §4.1. [17] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar (2012) Cats and dogs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 3498–3505. Cited by: §4.1. [18] D. Paul, A. Chowdhury, X. Xiong, F. Chang, D. Carlyn, S. Stevens, K. L. Provost, A. Karpatne, B. Carstens, D. Rubenstein, C. Stewart, T. Berger-Wolf, Y. Su, and W. Chao (2024) A simple interpretable transformer for fine-grained image classification and analysis. In International Conference on Learning Representations (ICLR), Cited by: §1, §2, §4.1. [19] L. Ren, C. Chen, L. Wang, and K. A. Hua (2025) DA-VPT: semantic-guided visual prompt tuning for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2. [20] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra (2017) Grad-CAM: visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), p. 618–626. Cited by: §1, §2, §4.1. [21] S. Sun, W. Ren, J. Li, R. Wang, and X. Cao (2024) Logit standardization in knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 15731–15740. Cited by: §1, §2, §3.1, §3.4. [22] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou (2021) Training data-efficient image transformers and distillation through attention. In Proceedings of the International Conference on Machine Learning (ICML), p. 10347–10357. Cited by: §2. [23] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie (2011) The caltech-ucsd birds-200-2011 dataset. Technical report Technical Report CNS-TR-2011-001, California Institute of Technology. Cited by: §4.1. [24] J. Wang, Y. Huang, S. Wu, C. Chen, and M. Tan (2023) Visual query tuning: towards effective usage of intermediate representations for parameter and memory efficient transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 7725–7735. Cited by: §2. [25] J. Wang, H. Liu, X. Wang, and L. Jing (2021) Interpretable image recognition by constructing transparent embedding space. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Cited by: §4.1. [26] L. Xu, W. Ouyang, M. Bennamoun, F. Boussaid, and D. Xu (2022) Multi-class token transformer for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 4310–4319. Cited by: §2. [27] M. Xue, Q. Huang, H. Zhang, L. Z. Meng, J. Song, and M. Song (2024) ProtoPFormer: concentrating on prototypical parts in vision transformers for interpretable image recognition. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), p. 1517–1525. Cited by: §2. [28] N. Zhang, J. Donahue, R. Girshick, and T. Darrell (2014) Part-based R-CNNs for fine-grained category detection. In Proceedings of the European Conference on Computer Vision (ECCV), p. 834–849. Cited by: §2. [29] B. Zhao, Q. Cui, R. Song, Y. Qiu, and J. Liang (2022) Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 11953–11962. Cited by: §2. [30] K. Zhou, J. Yang, C. C. Loy, and Z. Liu (2022) Learning to prompt for vision-language models. International Journal of Computer Vision 130, p. 2337–2348. Cited by: §2.