Paper deep dive
FD$^2$: A Dedicated Framework for Fine-Grained Dataset Distillation
Hongxu Ma, Guang Li, Shijie Wang, Dongzhan Zhou, Baoli Sun, Takahiro Ogawa, Miki Haseyama, Zhihui Wang
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/27/2026, 1:32:57 AM
Summary
FD^2 is a framework for fine-grained dataset distillation that enhances decoupled distillation methods by incorporating counterfactual attention learning (CAL) and two specific constraints: a fine-grained characteristic constraint for intra-class compactness and inter-class separability, and a similarity constraint to promote diversity among distilled samples.
Entities (5)
Relation Signals (3)
FD^2 → utilizes → Counterfactual Attention Learning
confidence 98% · FD^2 leverages Counterfactual Attention Learning (CAL) to extract attention maps
FD^2 → augments → Decoupled Dataset Distillation
confidence 95% · FD^2, a dedicated framework for Fine-grained Dataset Distillation that augments decoupled DD with fine-grained supervision.
FD^2 → improves → SRe^2L++
confidence 92% · FD^2 consistently improves representative decoupled distillation methods.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Dataset distillation (DD) compresses a large training set into a small synthetic set, reducing storage and training cost, and has shown strong results on general benchmarks. Decoupled DD further improves efficiency by splitting the pipeline into pretraining, sample distillation, and soft-label generation. However, existing decoupled methods largely rely on coarse class-label supervision and optimize samples within each class in a nearly identical manner. On fine-grained datasets, this often yields distilled samples that (i) retain large intra-class variation with subtle inter-class differences and (ii) become overly similar within the same class, limiting localized discriminative cues and hurting recognition. To solve the above-mentioned problems, we propose FD$^{2}$, a dedicated framework for Fine-grained Dataset Distillation. FD$^{2}$ localizes discriminative regions and constructs fine-grained representations for distillation. During pretraining, counterfactual attention learning aggregates discriminative representations to update class prototypes. During distillation, a fine-grained characteristic constraint aligns each sample with its class prototype while repelling others, and a similarity constraint diversifies attention across same-class samples. Experiments on multiple fine-grained and general datasets show that FD$^{2}$ integrates seamlessly with decoupled DD and improves performance in most settings, indicating strong transferability.
Tags
Links
- Source: https://arxiv.org/abs/2603.25144v1
- Canonical: https://arxiv.org/abs/2603.25144v1
Trouble viewing inline? Open PDF directly →
Full Text
66,001 characters extracted from source content.
Expand or collapse full text
FD 2 : A Dedicated Framework for Fine-Grained Dataset Distillation Hongxu Ma 1 , Guang Li 2⋆ , Shijie Wang 3 , Dongzhan Zhou 4 , Baoli Sun 5 , Takahiro Ogawa 2 , Miki Haseyama 2 , and Zhihui Wang 5∗ 1 Zhejiang University, China 2 Hokkaido University, Japan 3 The University of Queensland, Australia 4 Shanghai AI Laboratory, China 5 Dalian University of Technology, China Abstract. Dataset distillation (D) compresses a large training set into a small synthetic set, reducing storage and training cost, and has shown strong results on general benchmarks. Decoupled D further improves efficiency by splitting the pipeline into pretraining, sample distillation, and soft-label generation. However, existing decoupled methods largely rely on coarse class-label supervision and optimize samples within each class in a nearly identical manner. On fine-grained datasets, this often yields distilled samples that (i) retain large intra-class variation with subtle inter-class differences and (i) become overly similar within the same class, limiting localized discriminative cues and hurting recognition. To solve the above-mentioned problems, we propose FD 2 , a dedicated framework for Fine-grained Dataset Distillation. FD 2 localizes discrimi- native regions and constructs fine-grained representations for distillation. During pretraining, counterfactual attention learning aggregates discrim- inative representations to update class prototypes. During distillation, a fine-grained characteristic constraint aligns each sample with its class prototype while repelling others, and a similarity constraint diversifies at- tention across same-class samples. Experiments on multiple fine-grained and general datasets show that FD 2 integrates seamlessly with decou- pled D and improves performance in most settings, indicating strong transferability. Code will be released. Keywords: Dataset Distillation· Fine-grained· Decoupled Distillation 1 Introduction Recently, the rapid growth of large-scale datasets has driven substantial progress in artificial intelligence, but it also brings high computational cost and long train- ing time [8,9,15]. Dataset distillation (D) addresses this challenge by learning a ⋆ Correspondence to: guang@lmd.ist.hokudai.ac.jp and zhihuiwang@dlut.edu.cn arXiv:2603.25144v1 [cs.CV] 26 Mar 2026 2H. Ma and G. Li et al. FloridaIvoryWhiteBrownSageAnnaNelsonAmericanGreenChuck jaygullbreastedcreeperthrasherhummingbirdsharpgoldfinchvioletearwill nuthatchtailedwidow sparrow 0.1 0.2 0.3 0.4 0.5 0.6 Intra - class dispersion Fine-grained characteristics OriginSRe 2 L++ 0.0 0.2 0.4 0.6 Inter - class distance (a) One group id IPC = 1id IPC = 2id IPC = 3id IPC = 4id IPC = 5 SRe 2 L++ Ours (b) Fig. 1: (a) On CUB-200-2011, 10 classes are randomly sampled to compare the origi- nal training set and the SRe 2 L++ distilled set, where intra-class dispersion (bars) and inter-class distance (line) are reported. (b) Attention heatmaps of Black-footed Alba- tross in CUB-200-2011, comparing the attended regions of distilled samples produced by SRe 2 L++ and FD 2 . compact synthetic dataset that preserves the training utility of the original data, such that a model trained on the distilled set achieves performance close to train- ing on the full dataset while using far fewer resources [6,20,21,23,48,58]. Existing D methods can be broadly grouped into matching-based approaches, e.g., gra- dient matching [57, 58], distribution matching [28, 45, 60], trajectory matching [1,7,22], decoupled methods [4,5,55], and generative methods [13,24,26,42,59]. Most of these techniques are evaluated on general-purpose benchmarks, including ImageNette and ImageWoof [8], where they have shown promising performance. Among these, decoupled methods [4, 5, 55] split dataset distillation into three stages: model pretraining, sample distillation, and soft-label generation, which significantly improves efficiency while preserving competitive accuracy on gen- eral benchmarks. However, when directly applied to fine-grained datasets (e.g., CUB-200-2011), existing decoupled methods suffer from two limitations: (i) the distilled set preserves unfavorable fine-grained characteristics, and (i) distilled samples within the same class become overly similar. Fine-Grained Dataset Distillation3 For (i), Fig. 1a visualizes 10 randomly selected CUB-200-2011 classes, where the original data exhibit large intra-class variation and subtle inter-class differ- ences, which we term the fine-grained characteristic. This characteristic degrades the model’s ability to extract discriminative cues, thereby limiting fine-grained recognition performance. A representative decoupled method, SRe 2 L++ [5], largely inherits this unfavorable characteristic in the distilled set, and in some classes, the characteristic is even worse than that of the full dataset, making it difficult for the student model to learn discriminative representations. We at- tribute this to the fact that decoupled D relies primarily on coarse class-label supervision during distillation: it enforces class-level semantic consistency but does not explicitly encourage intra-class compactness or inter-class separability at the fine-grained level. For (i), the attention heatmaps in Fig. 1b show that SRe 2 L++ attends to highly consistent regions across same-class distilled sam- ples, which reduces coverage of discriminative parts and weakens the diversity of discriminative cues available for the student model. This behavior is induced by sample-wise iterative synthesis under nearly identical optimization across it- erations, which drives the same-class samples to converge to similar solutions. Together, these issues limit the quality of fine-grained distilled data and conse- quently degrade the student model’s performance on fine-grained recognition. To address these issues, we propose FD 2 , a dedicated framework for Fine- grained Dataset Distillation that augments decoupled D with fine-grained su- pervision. FD 2 leverages Counterfactual Attention Learning (CAL) to extract attention maps that localize discriminative regions and to construct class proto- types summarizing these representations. These attention-based signals provide fine-grained supervision for the subsequent distillation process. During pretrain- ing, CAL is used to obtain attention maps and update class prototypes that capture discriminative cues. During distillation, the attention maps are used to localize discriminative regions and are fused with backbone features to form fine-grained representations. Based on these representations, we introduce two complementary constraints. A fine-grained characteristic constraint pulls each distilled sample toward its target-class prototype while pushing it away from other-class prototypes, improving intra-class compactness and inter-class sepa- rability. A similarity constraint encourages diversity among same-class distilled samples by separating their attention distributions, enabling coverage of different discriminative regions. Both constraints are incorporated into existing distilla- tion objectives as plug-in terms, introducing minimal overhead while preserving the original decoupled distillation pipeline. Experiments on multiple fine-grained and general datasets demonstrate that FD 2 consistently improves representative decoupled distillation methods. The main contributions of this paper are summarized as follows: – We propose FD 2 , a dedicated framework for fine-grained dataset distillation that augments decoupled D with fine-grained supervision. The proposed method improves the structure of the distilled dataset by simultaneously enhancing inter-class separability and promoting within-class diversity. 4H. Ma and G. Li et al. – We introduce a fine-grained characteristic constraint to improve intra-class compactness and inter-class separability, and a similarity constraint to di- versify attention across same-class distilled samples, thereby providing richer localized discriminative cues. – Extensive experiments on multiple fine-grained and general datasets demon- strate that FD 2 can be integrated into representative decoupled methods without modification of their workflow and achieves consistent improvements in most settings, with particularly strong gains on fine-grained datasets. 2 Related Work Dataset Distillation. Dataset distillation (D) aims to synthesize a compact set of training samples from a large dataset, substantially reducing storage and training cost while retaining performance close to training on the full data. Existing D methods can be broadly categorized into gradient matching [19, 30, 57, 58, 62], distribution matching [27, 29, 36, 45, 52, 60], trajectory matching [1,2,7,14], decoupled distillation [4,39,40,51,54,55], and generative distillation [13,25,41,50,53,59,61]. Decoupled Dataset Distillation. SRe 2 L [55] pioneers decoupled distillation by separating model pretraining from sample distillation. SRe 2 L++ [5] improves robustness via stronger augmentation and batch-specific soft labels. FADRM [4] enriches pixel-space information with multi-scale residual connections (with FADRM+ extending to multi-model distillation). G-VBSM [39] enhances gener- alization across multiple models, while CDA [54] stabilizes optimization through a curriculum strategy. LPLD [51] revisits the necessity of large-scale soft labels and proposes lighter-weight alternatives. Despite their effectiveness on general- purpose benchmarks, these methods are not designed with fine-grained datasets in mind and do not explicitly address fine-grained separability or within-class diversity in the distilled set. Fine-grained Image Recognition. Fine-grained image recognition (FGIR) often relies on localized discriminative regions, motivating methods that mine and ag- gregate part-level cues. Cross-X [31] exploits cross-layer interaction with cross- category constraints. PMG [10] learns multi-granularity cues via progressive training. DP-Net [46] injects learnable positional cues and dynamically aligns them with visual content. CSC-Net [47] improves localization by enforcing class- specific semantic coherency. CAL [35] introduces counterfactual attention learn- ing to enhance attention quality and guide models toward key local cues. 3 Approach 3.1 Preliminaries Dataset distillation (D) aims to synthesize a compact training set that pre- serves the training utility of a large labeled dataset. Let T = (x i ,y i ) N i=1 de- note the original training set. The objective is to distill a substantially smaller Fine-Grained Dataset Distillation5 1. Model pretraining Backbone CAL + Soft labels Full dataset Distilled images 2. Distillation (A group of N S images of the same class) Prototypes Backbone CAL + Current distilled image Previous distilled images (total: 푖 -1) ℒ other ℒ cls + Fine-grained characteristic↓ Representation Previous attention maps (total: 푖 -1) Current attention map λℒ F Attention maps (Current & Previous) Similarity↓ (1−λ)ℒ S + id IPC = 1 ... No Similarity↓ id IPC = N S ... Backbone CAL + 3. Soft label generation Representation Current class prototype Fine-grained characteristic↓ Similarity↓ id IPC = 푖 Other class prototypes Fig. 2: Overview of FD 2 . 1. We pretrain a Backbone+CAL teacher and maintain class prototypes online. 2. We distill images in the same-class groups of size N S , adding a fine- grained characteristic constraint (prototype alignment or separation) and a similarity constraint (diverse attention), together with class-supervision from both branches. 3. We generate soft labels using the backbone branch. set D =( ̃x j , ̃y j ) M j=1 with M ≪ N, such that models trained on D achieve per- formance comparable to those trained on T . This objective can be expressed by minimizing the discrepancy between the losses obtained under the two training regimes: sup (x,y)∼T |L(f θ T (x),y)−L(f θ D (x),y)|≤ δ,(1) whereL(·,·) denotes the task loss (e.g., cross-entropy), and θ T and θ D represent the parameters obtained by training on T and D, respectively. Accordingly, dataset distillation can be formulated as the following optimization problem: arg min D,|D| sup (x,y)∼T |L(f θ T (x),y)−L(f θ D (x),y)|. (2) Under a given distillation budget, the distilled setD maintains an equal number of images for each class. 3.2 Overview of FD 2 The overall framework of FD 2 is illustrated in Fig. 2. FD 2 follows the decoupled distillation paradigm and can be integrated into existing decoupled methods as an additive module. We first pretrain a Backbone+CAL teacher by jointly op- timizing the backbone and CAL classifiers while maintaining fine-grained class prototypes online. During distillation, distilled images are optimized in a sample- wise manner and organized into the same-class groups of size N S . For each cur- rent distilled sample, its feature representation and attention map are obtained 6H. Ma and G. Li et al. from the Backbone+CAL teacher, and the original distillation objective is aug- mented with two constraints. The fine-grained characteristic constraint pulls the sample toward the prototype of the target class and pushes it away from prototypes of other classes, improving intra-class compactness and inter-class separability. The similarity constraint encourages within-class diversity by in- creasing the discrepancy between the attention map of the current sample and those of previously generated samples of the same class, thereby promoting cov- erage of different discriminative regions. Finally, during the soft-label generation stage, soft labels are produced by the backbone branch. Since downstream eval- uation trains a standard backbone student, this design helps reduce bias caused by architectural mismatch. 3.3 Counterfactual Attention Learning (CAL) To address the limitations of previous methods on fine-grained datasets, we in- troduce Counterfactual Attention Learning (CAL) [35], a simple and effective approach that provides fine-grained class prototypes and precise attention maps. CAL leverages counterfactual intervention to construct a factual–counterfactual contrastive signal, enabling more reliable attention localization and discrimina- tive representation learning under weak supervision. Given an input image x with label y, a backbone feature extractor f (·) pro- duces the final-stage feature map F, and an attention predictor g(·) generates M attention maps A = A m M m=1 . CAL then applies an attention-weighted ag- gregation operator Φ(·) (e.g., bilinear attention pooling) to obtain the factual representation z: F = f (x), A = g(F ), z = Φ(F,A).(3) To explicitly quantify the contribution of attention, CAL performs a counter- factual intervention by replacing A with an alternative (or perturbed) attention set ̄ A while keeping F fixed. This operation produces a counterfactual represen- tation ˆz, and the difference between factual and counterfactual predictions forms an effect prediction: ˆz = Φ(F, ̄ A), p raw = Wz, p eff = p raw − W ˆz,(4) where W denotes the linear classifier of CAL, p raw represents the factual logit, and p eff measures the gain induced by the factual attention relative to the coun- terfactual attention. Intuitively, if attention focuses on truly discriminative re- gions, replacing it with ̄ A weakens class evidence, which increases the discrim- inability of p eff and is therefore encouraged during training. Meanwhile, CAL maintains a feature center for each class as a class prototype. Let c y denote the prototype of class y. The prototype is updated online using a momentum rule, and a center regularizer encourages the factual representation z to approach the corresponding class prototype: c y ← (1− μ)c y + μ Norm(z), L center = z− Norm(c y ) 2 2 , (5) Fine-Grained Dataset Distillation7 where Norm(·) denotes feature normalization and μ∈ (0, 1) denotes the momen- tum coefficient. Subsequently, the pretraining objective of CAL is defined as: L CAL =L ce (p raw ,y) +L ce (p eff ,y) + ηL center ,(6) where L ce (·,·) denotes the cross-entropy loss and η controls the strength of the center regularization. This objective enables CAL to identify discriminative re- gions more reliably and to learn discriminative representations during training, which improves classification accuracy. During the pretraining, distillation, and soft-label generation stages, the teacher adopts the Backbone+CAL architecture. In the post-evaluation stage, following the standard protocol of prior methods, a plain Backbone student model is used. Notably, during pretraining, the backbone classifier and the CAL classifier share the same backbone features and are jointly optimized. This de- sign is motivated by the use of a backbone student model in the post-evaluation stage. If soft labels generated by the CAL branch are used directly, instability may be introduced into the probability outputs of the student model. Formally, the backbone branch predicts: p b = W b · GAP(F ),(7) where W b denotes the backbone classifier and GAP(·) denotes global average pooling. The overall pretraining loss is defined as a weighted combination: L pre = (1− α)L ce (p b ,y) + αL CAL ,(8) where α ∈ [0, 1] controls the relative contribution of the two branches. Joint optimization ensures that the backbone teacher model preserves the ability to generate stable soft labels for student training, while still benefiting from the discriminative representations and prototype aggregation provided by CAL. 3.4 Constraints Suppressing unfavorable fine-grained characteristics and reducing similarity among distilled samples are two key strategies for fine-grained dataset distillation. There- fore, two specialized constraints are introduced. Fine-grained Characteristic Constraint. Suppressing unfavorable fine-grained characteristics in distilled samples reduces the difficulty for the student model to learn discriminative representations. A straightforward strategy aligns the representation of a sample with the prototype of its class while increasing the distance to the prototypes of other classes: L F ( ̃x y,i ) = β ℓ 2 (z y,i ,c y ) + (1− β) 1− E k̸=y ℓ 2 (z y,i ,c k ) , β ∈ [0, 1],(9) where ̃x y,i denotes the i-th distilled image of class y in a group, β is a weighting hyperparameter, ℓ 2 (u,v) = ∥u− v∥ 2 /(∥u∥ 2 +∥v∥ 2 + ε) is a symmetrically nor- malized Euclidean metric, z y,i denotes the representation of the sample, and c y 8H. Ma and G. Li et al. and c k denote the prototypes of the current class and other classes, respectively. According to Eq. (9), the intra-class compactness and inter-class separability of the distilled images are enhanced. Similarity Constraint. To suppress similarity among distilled samples and en- courage diverse discriminative regions, the distance between the attention map of the current sample and the attention maps of previously distilled samples is maximized in the attention space: L S ( ̃x y,i ) = 1− E j<i [ℓ 2 A y,i ,A y,j ) ,1 < i≤ N S , (10) and L S is omitted when i = 1. Here, A y,i denotes the attention map of the current sample, A y,j denotes the attention maps of previous samples of the same class, and N S denotes the number of distilled images in the group that satisfy this constraint. Since pretraining adopts a dual-classifier architecture, distillation also super- vises class signals with both classifiers to leverage their discriminative capability: L cls ( ̃x y,i ) = (1− α)L ce p y,i b ,y + αL ce p y,i cal ,y , α∈ [0, 1]. (11) Here, α is a weighting hyperparameter, and p y,i b and p y,i cal denote the logits pro- duced by the backbone and CAL branches, respectively. Let L other denote the original loss of the underlying distillation method. The final objective for opti- mizing ̃x y,i is L total =L other +L cls + λL F + (1− λ)L S ,(12) where λ controls the relative contribution of the two proposed constraints. The training process is summarized in Algorithm 1. The proposed constraints are jointly optimized with the original objective, which suppresses undesirable fine- grained characteristics and similarity without degrading the effectiveness of the underlying method. A theoretical analysis of these constraints is provided in the Appendix. 4 Experiments 4.1 Experimental Settings Datasets. Evaluation of FD 2 is conducted on three fine-grained datasets, CUB- 200-2011 [44], FGVC-Aircraft [33], and Stanford Cars [18], together with two general datasets, ImageNette and ImageWoof, which are subsets of ImageNet- 1K [8]. All input images are resized to 224× 224. Comparative Methods. Comparison of FD 2 is performed with three state-of-the- art dataset distillation baselines under the same experimental environment on a single H200 GPU. Since FD 2 is designed for the decoupled distillation paradigm, SRe 2 L++ [5] is selected as the primary baseline. This method extends SRe 2 L [55] Fine-Grained Dataset Distillation9 Algorithm 1 FD 2 Require: Class index y, number of classes K, group size N S Require: Pretrained teacher φ(·) (Backbone+CAL), prototypes c k K k=1 Require: Weights α,β,λ, original loss L other 1: for i = 1 to N S do 2: Initialize distilled image ̃x y,i 3: if i > 1 then 4:for j = 1 to i− 1 do 5:(·,·,·,A y,j )← φ( ̃x y,j ) ▷Attention maps of previous same-class samples 6:end for 7: end if 8: for each step do 9:(p y,i b , p y,i cal , z y,i , A y,i )← φ( ̃x y,i ) ▷Logits, representation and attention map 10:Compute L F by Eq. (9) 11:if i > 1 then 12:Compute L S by Eq. (10) using A y,j i−1 j=1 and A y,i 13:else 14:L S ← 0 15:end if 16:Compute L cls by Eq. (11) 17: L total ←L other +L cls + λL F + (1− λ)L S 18:Update ̃x y,i by gradient descent on L total 19: end for 20: end for through stronger data augmentation and batch-specific soft labels. To evalu- ate plug-and-play transferability, FADRM+ [4] is also included. FADRM+ is an ensemble-based decoupled method that improves both efficiency and per- formance through multi-scale residual connections. In addition, RDED [43] is adopted as a representative approach that constructs distilled sets by cropping real images, which can be interpreted as exploiting localized regions during the distillation process. In cross-architecture generalization experiments and ablation studies, CUB-200- 2011 is used as the default dataset unless otherwise specified. Additional com- parisons and efficiency analysis with other advanced methods on fine-grained datasets are provided in the Appendix. 4.2 Main Results Fine-grained Datasets. As shown in Tab. 1, FD 2 improves the performance of decoupled methods in most settings when compared with SRe 2 L++ and FADRM+. This result indicates that the mitigation of unfavorable fine-grained characteristics and excessive similarity among samples can improve the quality of distilled data and facilitate the learning of discriminative representations by the student model. Notably, the improvements are typically more pronounced when IPC= 1, with particularly clear gains on the Stanford Cars dataset. For example, with ResNet18, SRe 2 L++ FD 2 exceeds SRe 2 L++ by 12.1%, and FADRM+ FD 2 10H. Ma and G. Li et al. Table 1: Top-1 accuracy is reported in the post-evaluation stage to compare FD 2 with three state-of-the-art methods. For fine-grained datasets, where the number of samples per class is limited, the value of IPC is set to 1, 3, and 5. Following the standard protocol of each method, post-evaluation is conducted for 300 epochs for RDED, which constructs distilled sets by cropping real images, and for 400 epochs for the decoupled baselines SRe 2 L++ and FADRM+. Evaluation of FD 2 is performed by integrating it into SRe 2 L++ and FADRM+ (denoted by the subscript “ FD 2 ”) while keeping the same post-evaluation settings as the corresponding baseline methods. DatasetStudent IPC RDEDSRe 2 L++SRe 2 L++ FD 2 FADRM+FADRM+ FD 2 CUB-200-2011 ResNet18 1 38.3 53.456.4 (↑ 3.0)54.855.0 (↑ 0.2) 3 52.660.064.9 (↑ 4.9)64.064.6 (↑ 0.6) 5 63.9 63.567.0 (↑ 3.5)66.467.5 (↑ 1.1) ResNet50 1 33.461.170.1 (↑ 9.0)66.266.5 (↑ 0.3) 3 49.065.173.7 (↑ 8.6)70.470.6 (↑ 0.2) 5 58.668.175.5 (↑ 7.4)71.572.0 (↑ 0.5) FGVC-Aircraft ResNet18 1 22.152.658.2 (↑ 5.6)55.060.5 (↑ 5.5) 3 36.466.676.1 (↑ 9.5)72.975.1 (↑ 2.2) 5 38.668.380.0 (↑ 11.7)74.077.6 (↑ 3.6) ResNet50 1 12.8 59.367.8 (↑ 8.5)70.873.8 (↑ 3.0) 3 34.8 68.076.7 (↑ 8.7)76.178.9 (↑ 2.8) 5 45.272.279.0 (↑ 6.8)78.681.0 (↑ 2.4) Stanford Cars ResNet18 1 33.052.464.5 (↑ 12.1)60.374.1 (↑ 13.8) 3 69.468.275.2 (↑ 7.0)75.084.8 (↑ 9.8) 5 76.170.981.4 (↑ 10.5)77.786.6 (↑ 8.9) ResNet50 1 12.865.680.7 (↑ 15.1)73.985.3 (↑ 11.4) 3 70.2 76.686.5 (↑ 9.9)78.987.8 (↑ 8.9) 5 75.978.088.3 (↑ 10.3)82.289.0 (↑ 6.8) exceeds FADRM+ by 13.8%. This observation suggests that FD 2 can still pro- vide effective discriminative cues even when only a single distilled sample is available for each class. In contrast, RDED exhibits the lowest overall perfor- mance, which indicates that coarse cropping of real images does not reliably capture key discriminative regions. General datasets. To verify the applicability of FD 2 beyond fine-grained datasets, we further evaluate it on ImageNette and ImageWoof. Since SRe 2 L++ does not report results on ImageWoof, we use RDED and FADRM+ as baselines. As shown in Tab. 2, on ImageWoof (which can be viewed as a 10-class fine-grained subset of ImageNet), FADRM+ FD 2 achieves clear improvements in most settings, supporting the effectiveness of FD 2 . On ImageNette, the effect of FD 2 depends on the student architecture: with ResNet50, FADRM+ FD 2 yields clear improve- ments across all IPC settings, while with the smaller ResNet18, the gains are more limited and become noticeable mainly at IPC= 50. This suggests that the additional fine-grained cues introduced by FD 2 are more effectively utilized by higher-capacity students. RDED consistently performs the worst across datasets and IPC settings, indicating that a cropping-based manner provides limited use- ful information on general datasets. Overall, FD 2 improves performance in most settings, with reduced gains mainly observed for small-capacity students on sim- pler datasets. Fine-Grained Dataset Distillation11 Table 2: Top-1 accuracy in the post-evaluation stage is reported to compare FD 2 with two baselines. On general datasets, IPC is set to 1, 10, and 50. Following the standard protocols, post-evaluation is conducted for 300 epochs for both RDED and FADRM+. FD 2 is evaluated by integrating it into FADRM+ (denoted by the subscript “ FD 2 ”) under the same post-evaluation setting. Dataset IPC ResNet18ResNet50 RDEDFADRM+FADRM+ FD 2 RDEDFADRM+FADRM+ FD 2 ImageNette 1 35.839.238.6 (↓ 0.6) 27.031.939.0 (↑ 7.1) 10 61.469.069.5 (↑ 0.5) 55.068.171.4 (↑ 3.3) 50 80.484.686.7 (↑ 2.1) 81.885.492.3 (↑ 6.9) ImageWoof 1 20.8 22.822.1 (↓ 0.7) 17.819.930.3 (↑ 10.4) 10 38.557.360.7 (↑ 3.4) 35.254.164.6 (↑ 10.5) 50 68.5 72.680.0 (↑ 7.4) 67.071.780.2 (↑ 8.5) SRe 2 L++(samples) Ours(samples) SRe 2 L++(center) Ours(center) t-SNE (a) 0255075125150175200 100 Class index 0.15 0.10 0.20 Nearest - neighbor center distance Per-class Nearest-neighbor Center Distance SRe 2 L++ Ours (b) Fig. 3: (a) t-SNE feature distribution. (b) Nearest-neighbor center distance of each class for distilled images on CUB-200-2011. 4.3 Analysis Fine-grained Characteristic. As shown in Fig. 3a, we visualize distilled samples on CUB-200-2011 using t-SNE [32] in the feature space, where 10 classes are randomly sampled with 5 distilled images per class. Compared with SRe 2 L++, same-class samples distilled with FD 2 exhibit higher intra-class compactness. Meanwhile, Fig. 3b shows larger nearest-neighbor center distances between classes under FD 2 , indicating improved inter-class separability. These results demon- strate that the proposed fine-grained characteristic constraint reduces intra-class variance while enhancing inter-class separability. Similarity. As illustrated in Fig. 1b, attention heatmaps of the black-footed alba- tross class in CUB-200-2011 show that SRe 2 L++ produces highly homogeneous distilled samples within the same class, where the attended regions are nearly identical across samples. When the similarity constraint is applied within a group of id IPC =1-4, FD 2 encourages more diverse attention distributions, enriching the discriminative regions in the distilled samples. Although RDED introduces some local information through cropping, this coarse strategy does not reliably cap- ture key discriminative cues. As a result, the performance of RDED is generally lower than that of FD 2 , as shown in Tab. 1 and Tab. 2. 12H. Ma and G. Li et al. Table 3: Top-1 accuracy on CUB-200-2011 for cross-architecture generalization at IPC=3. StudentSRe 2 L++SRe 2 L++ FD 2 FADRM+FADRM+ FD 2 ShuffleNetV2 [56]38.547.6 (↑ 9.1)46.446.7 (↑ 0.3) MobileNetV2 [37]57.162.9 (↑ 5.8)64.965.8 (↑ 0.9) DenseNet121 [17]62.065.8 (↑ 3.8)68.668.1 (↓ 0.5) ResNet18 [15] 60.064.9 (↑ 4.9)64.064.6 (↑ 0.6) ResNet50 [15]64.267.7 (↑ 3.5)70.470.6 (↑ 0.2) Transferability. FD 2 functions as an add-on module that can be integrated into different decoupled distillation methods without modifying the original train- ing pipeline. As shown in Tab. 1, integrating FD 2 into SRe 2 L++ yields clear accuracy improvements across datasets and IPC settings. When integrated into FADRM+, improvements are also observed on most datasets, although the gains on CUB-200-2011 remain relatively limited, while larger improvements appear on Stanford Cars. These results indicate that FD 2 consistently enhances the performance of different decoupled methods and demonstrates strong transfer- ability. 4.4 Cross-Architecture Generalization Cross-architecture generalization is an important criterion for evaluating the quality of distilled datasets, as it reflects transferability across different student architectures and practical applicability. To examine this property, we compare SRe 2 L++, FADRM+, and their FD 2 -augmented variants. As shown in Tab. 3, integrating FD 2 improves performance for most architectures and methods. On DenseNet121 [17], FADRM+ performs slightly better than FADRM+ FD 2 , while SRe 2 L++ FD 2 consistently outperforms SRe 2 L++. These results indicate that FD 2 provides strong cross-architecture generalization and stable performance across diverse evaluation settings. 4.5 Ablation Study To reduce computational cost and runtime, FD 2 is integrated into SRe 2 L++, and all ablation studies are conducted using ResNet18 + CAL. 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 CAL ratio 훼 66 68 70 72 74 76 Backbone Top - 1 (%) 54.8 0.3 71.6 Number Attention maps =8 Number Attention maps =16 Number Attention maps =32 66 68 70 72 74 76 CAL Top - 1 (%) 76.1 CUB-200-2011 Fig. 4: Top-1 accuracy under different settings during pretraining. Table 4: Accuracy at IPC= 3 on CUB- 200-2011 and FGVC-Aircraft for distilled datasets obtained with different CAL ratios during distillation. α CUB-200-2011 FGVC-Aircraft 0.162.875.1 0.363.375.9 0.563.474.8 0.762.175.6 0.962.475.2 Fine-Grained Dataset Distillation13 Impact of the Cal Ratio α in the Model Pretraining and Distillation. Since the backbone classifier and the CAL classifier are jointly optimized, we first perform an ablation study in the pretraining stage to determine an appropriate CAL ratio, as shown in Fig. 4. With the number of attention maps fixed to 8, different CAL ratios are evaluated. When the CAL ratio exceeds 0.5, the accuracy of both the backbone branch and the CAL branch decreases noticeably. Therefore, in experiments with 16 and 32 attention maps, the CAL ratio is restricted to 0.1–0.5. Considering all configurations, the pretrained model with 8 attention maps and a CAL ratio of 0.5 is selected for the distillation stage. During distillation, the effect of the CAL ratio on post-evaluation perfor- mance is further examined with IPC= 3. As shown in Tab. 4, on CUB-200-2011, the highest Top-1 accuracy (63.4%) is obtained at CAL ratio = 0.5. On FGVC- Aircraft, the best accuracy (75.9%) is achieved at CAL ratio = 0.3, which is also optimal in the pretraining stage. These results suggest that using the same CAL ratio in both stages helps maintain consistent feature distributions and attention strengths, thereby reducing stage mismatch. Therefore, adopting the same CAL ratio across stages provides a stable configuration. Table 5: Impact of the hy- perparameter β in the fine- grained characteristic con- straint. Method β Acc SRe 2 L++ FD 2 0.00 63.5 0.25 63.7 0.50 64.8 0.75 64.7 1.00 64.1 Table 6: Impact of the group size N S in the simi- larity constraint. Method N S Acc SRe 2 L++ FD 2 2 64.1 3 64.9 4 66.1 5 64.6 Table 7: Ablation study of different constraint combi- nations. Method L F L S Acc SRe 2 L++ FD 2 – – 63.4 ✓ – 64.8 –✓ 64.6 ✓ 64.9 Impact of the β in the Fine-grained Characteristic Constraint. To study the rela- tive weighting between alignment with same-class prototypes and repulsion from other-class prototypes, we enable the fine-grained characteristic constraint alone and ablate β with IPC= 3. As shown in Tab. 5, the best post-evaluation accuracy is achieved at β = 0.5 (64.8%). This result indicates that a balanced weighting improves both intra-class compactness and inter-class separability, thereby im- proving the quality of distilled samples. Therefore, β = 0.5 is used in subsequent experiments. Impact of the Group Size N S in the Similarity Constraint. To determine an appropriate group size N S , we ablate N S with only the similarity constraint en- abled and IPC= 5. As shown in Tab. 6, N S = 4 achieves the best post-evaluation accuracy (66.1%). When N S increases to 5, the available discriminative cues be- come limited, and additional samples tend to focus on repeated regions. With only a few distilled samples, this repetition may cause the student model to overfit to these regions, which reduces performance (64.6%). Therefore, N S = 4 14H. Ma and G. Li et al. is selected. The distillation time under different N S settings is reported in the Appendix. Effectiveness of the Proposed Constraints. To evaluate the contribution of the proposed constraints, we compare different constraint combinations with IPC= 3. As shown in Tab. 7, both constraints improve performance over the no- constraint baseline (63.4%). The fine-grained characteristic constraint provides a slightly larger gain (64.8%) than the similarity constraint (64.6%). When both constraints are enabled, the highest accuracy (64.9%) is achieved, which con- firms their complementary effects. An additional ablation study on the relative weight λ is provided in the Appendix. 4.6 Visualization Attention Visualization under Similarity Constraint. As shown in Fig. 5, the sim- ilarity constraint encourages diverse attention patterns among distilled samples within the same group, allowing different samples to cover complementary dis- criminative regions. When N S = 5, later samples in the group (e.g., id IPC = 4/5) begin to attend to similar regions, which suggests that the benefit of similarity suppression becomes saturated under this setting. Visualization Comparison of Distilled Samples across Methods. Fig. 6 presents distilled samples from several CUB-200-2011 classes produced by different meth- ods. Samples generated by SRe 2 L++ and FADRM+ often exhibit noticeable Gaussian noise. In contrast, integrating FD 2 produces samples with clearer lo- cal structures and richer texture details. This observation indicates that the proposed method preserves fine-grained local cues more effectively and improves inter-class separability, which enhances the overall quality of the distilled dataset. N S =3 N S =2 N S =4 N S =5 id IPC =0id IPC =1id IPC =2id IPC =3id IPC =4 Fig. 5: Attention heatmap visualizations for Black-footed Albatross under different group sizes N S when only the similarity constraint is applied. SRe 2 L++ SRe 2 L++ FD 2 FADRM+ FADRM+ FD 2 Black-footed albatross Laysan albatross Sooty albatross Groove- billed ani Crested auklet Fig. 6: Visual comparison of distilled samples generated by different methods on five randomly selected classes. Fine-Grained Dataset Distillation15 5 Conclusion We propose FD 2 , a dedicated framework designed for fine-grained dataset dis- tillation. By jointly optimizing a fine-grained characteristic constraint and a similarity constraint together with the original distillation objective, FD 2 mit- igates unfavorable fine-grained characteristic and within-class homogenization while preserving the decoupled distillation pipeline. As an add-on module, FD 2 can be seamlessly integrated into decoupled methods and consistently improves performance across diverse fine-grained and general datasets in most settings. Future work will focus on further improving performance on general datasets and extending FD 2 to broader distillation paradigms and fine-grained tasks. References 1. Cazenavette, G., Wang, T., Torralba, A., Efros, A.A., Zhu, J.Y.: Dataset distil- lation by matching training trajectories. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR). p. 10718–10727 (2022) 2, 4 2. Chen, M., Huang, B., Lu, J., Li, B., Wang, Y., Cheng, M., Wang, W.: Dataset distillation via adversarial prediction matching. arXiv preprint arXiv:2312.08912 (2023) 4 3. Chen, Y., Welling, M., Smola, A.: Super-samples from kernel herding. In: Pro- ceedings of the Conference on Uncertainty in Artificial Intelligence (UAI) (2010) 22 4. Cui, J., Bi, X., Luo, Y., Zhao, X., Liu, J., Shen, Z.: FADRM: Fast and accurate data residual matching for dataset distillation. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) (2025) 2, 4, 9 5. Cui, J., Li, Z., Ma, X., Bi, X., Luo, Y., Shen, Z.: Dataset distillation via committee voting. arXiv preprint arXiv:2501.07575 (2025) 2, 3, 4, 8 6. Cui, J., Wang, R., Si, S., Hsieh, C.J.: DC-BENCH: Dataset condensation bench- mark. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) (2022) 2 7. Cui, J., Wang, R., Si, S., Hsieh, C.J.: Scaling up dataset distillation to imagenet-1k with constant memory. In: Proceedings of the International Conference on Machine Learning (ICML). p. 6565–6590 (2023) 2, 4 8. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). p. 248–255 (2009) 1, 2, 8 9. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: Proceedings of the International Conference on Learning Representations (ICLR) (2021) 1 10. Du, R., Chang, D., Bhunia, A.K., Xie, J., Ma, Z., Song, Y.Z., Guo, J.: Fine-grained visual classification via progressive multi-granularity training of jigsaw patches. In: Proceedings of the European Conference on Computer Vision (ECCV). p. 153– 168 (2020) 4 11. Fisher, R.A.: The use of multiple measurements in taxonomic problems. Annals of eugenics p. 179–188 (1936) 19 16H. Ma and G. Li et al. 12. Gidaris, S., Komodakis, N.: Dynamic few-shot visual learning without forgetting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 4367–4375 (2018) 22 13. Gu, J., Vahidian, S., Kungurtsev, V., Wang, H., Jiang, W., You, Y., Chen, Y.: Effi- cient dataset distillation via minimax diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 15793– 15803 (2024) 2, 4, 22 14. Guo, Z., Wang, K., Cazenavette, G., Li, H., Zhang, K., You, Y.: Towards lossless dataset distillation via difficulty-aligned trajectory matching. In: Proceedings of the International Conference on Learning Representations (ICLR) (2024) 4, 22 15. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). p. 770–778 (2016) 1, 12, 22 16. Hu, T., Qi, H.: See better before looking closer: Weakly supervised data augmen- tation network for fine-grained visual classification. ArXiv (2019) 20 17. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). p. 4700–4708 (2017) 12 18. Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3d object representations for fine- grained categorization. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW). p. 554–561 (2013) 8 19. Lee, S., Chun, S., Jung, S., Yun, S., Yoon, S.: Dataset condensation with con- trastive signals. In: Proceedings of the International Conference on Machine Learn- ing (ICML). p. 12352–12364 (2022) 4 20. Li, G., Togo, R., Ogawa, T., Haseyama, M.: Soft-label anonymous gastric x-ray image distillation. In: Proceedings of the IEEE International Conference on Image Processing (ICIP). p. 305–309 (2020) 2 21. Li, G., Togo, R., Ogawa, T., Haseyama, M.: Compressed gastric image generation based on soft-label dataset distillation for medical data sharing. Computer Methods and Programs in Biomedicine 227, 107189 (2022) 2 22. Li, G., Togo, R., Ogawa, T., Haseyama, M.: Importance-aware adaptive dataset distillation. Neural Networks 172, 106154 (2024) 2 23. Li, G., Zhao, B., Wang, T.: Awesome dataset distillation. https://github.com/Guang000/Awesome-Dataset-Distillation (2022) 2 24. Li, L., Li, G., Togo, R., Maeda, K., Ogawa, T., Haseyama, M.: Generative Dataset Distillation: Balancing global structure and local details. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Workshop. p. 7664–7671 (2024) 2 25. Li, M., Li, G., Mao, J., Ogawa, T., Haseyama, M.: Diversity-driven generative dataset distillation based on diffusion model with self-adaptive memory. In: IEEE International Conference on Image Processing (ICIP) (2024) 4 26. Li, M., Li, G., Mao, J., Ye, L., Ogawa, T., Haseyama, M.: Task-specific generative dataset distillation with difficulty-guided sampling. In: IEEE/CVF International Conference on Computer Vision (ICCV), Workshop (2025) 2 27. Li, W., Li, G., Maeda, K., Ogawa, T., Haseyama, M.: Decoupled audio-visual dataset distillation. arXiv preprint arXiv:2511.17890 (2025) 4 28. Li, W., Li, G., Maeda, K., Ogawa, T., Haseyama, M.: Hyperbolic dataset distilla- tion. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) (2025) 2 Fine-Grained Dataset Distillation17 29. Liu, S., Wang, K., Yang, X., Ye, J., Wang, X.: Dataset distillation via factoriza- tion. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) (2022) 4 30. Liu, Y., Gu, J., Wang, K., Zhu, Z., Jiang, W., You, Y.: DREAM: Efficient dataset distillation by representative matching. In: Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV). p. 17314–17324 (2023) 4 31. Luo, W., Yang, X., Mo, X., Lu, Y., Davis, L.S., Li, J., Yang, J., Lim, S.N.: Cross-x learning for fine-grained visual categorization. In: Proceedings of the IEEE/CVF international conference on computer vision (CVPR). p. 8242–8251 (2019) 4 32. van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of Machine Learning Research p. 2579–2605 (2008) 11 33. Maji, S., Rahtu, E., Kannala, J., Blaschko, M., Vedaldi, A.: Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 (2013) 8 34. Peebles, W., Xie, S.: Scalable diffusion models with transformers. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). p. 4195–4205 (2023) 22 35. Rao, Y., Chen, G., Lu, J., Zhou, J.: Counterfactual attention learning for fine-grained visual categorization and re-identification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). p. 1025–1034 (2021) 4, 6 36. Sajedi, A., Khaki, S., Amjadian, E., Liu, L.Z., Lawryshyn, Y.A., Plataniotis, K.N.: DataDAM: Efficient dataset distillation with attention matching. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). p. 17097– 17107 (2023) 4 37. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: In- verted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). p. 4510–4520 (2018) 12 38. Sener, O., Savarese, S.: Active learning for convolutional neural networks: A core- set approach. In: Proceedings of the International Conference on Learning Repre- sentations (ICLR) (2018) 22 39. Shao, S., Yin, Z., Zhou, M., Zhang, X., Shen, Z.: Generalized large-scale data condensation via various backbone and statistical matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 16709–16718 (2024) 4 40. Shao, S., Zhou, Z., Chen, H., Shen, Z.: Elucidating the design space of dataset condensation. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) (2024) 4 41. Su, D., Hou, J., Gao, W., Tian, Y., Tang, B.: D4M: Dataset distillation via dis- entangled diffusion model. In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR). p. 5809–5818 (2024) 4 42. Su, D., Hou, J., Li, G., Togo, R., Song, R., Ogawa, T., Haseyama, M.: Generative dataset distillation based on diffusion model. In: Proceedings of the European Conference on Computer Vision (ECCV), Workshop (2024) 2 43. Sun, P., Shi, B., Yu, D., Lin, T.: On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR). p. 9390–9399 (2024) 9 44. Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: Caltech-ucsd birds- 200-2011. Tech. rep., California Institute of Technology (2011) 8 18H. Ma and G. Li et al. 45. Wang, K., Zhao, B., Peng, X., Zhu, Z., Yang, S., Wang, S., Huang, G., Bilen, H., Wang, X., You, Y.: CAFE: Learning to condense dataset by aligning features. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. 12196–12205 (2022) 2, 4 46. Wang, S., Li, H., Wang, Z., Ouyang, W.: Dynamic position-aware network for fine- grained image recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). p. 2791–2799 (2021) 4 47. Wang, S., Wang, Z., Li, H., Ouyang, W.: Category-specific semantic coherency learning for fine-grained image recognition. In: Proceedings of the ACM Interna- tional Conference on Multimedia (ACM M). p. 174–183 (2020) 4 48. Wang, T., Zhu, J.Y., Torralba, A., Efros, A.A.: Dataset distillation. arXiv preprint arXiv:1811.10959 (2018) 2 49. Wen, Y., Zhang, K., Li, Z., Qiao, Y.: A discriminative feature learning approach for deep face recognition. In: Proceedings of the European Conference on Computer Vision (ECCV). p. 499–515 (2016) 19 50. Wu, H., Su, D., Hou, J., Li, G.: Dataset condensation with color compensation. Transactions on Machine Learning Research (2025) 4 51. Xiao, L., He, Y.: Are large-scale soft labels necessary for large-scale dataset distil- lation? In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) (2024) 4 52. Xue, E., Li, Y., Liu, H., Wang, P., Shen, Y., Wang, H.: Towards adversarially robust dataset distillation by curvature regularization. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) (2025) 4 53. Ye, L., Hamidi, S.M., Li, G., Ogawa, T., Haseyama, M., Plataniotis, K.N.: Information-guided diffusion sampling for dataset distillation. In: Advances in Neu- ral Information Processing Systems (NeurIPS), Workshop (2025) 4 54. Yin, Z., Shen, Z.: Dataset distillation in large data era. Transactions on Machine Learning Research (2024) 4 55. Yin, Z., Xing, E., Shen, Z.: Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) (2023) 2, 4, 8 56. Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: An extremely efficient convolu- tional neural network for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). p. 6848–6856 (2018) 12 57. Zhao, B., Bilen, H.: Dataset condensation with differentiable siamese augmenta- tion. In: Proceedings of the International Conference on Machine Learning (ICML). p. 12674–12685 (2021) 2, 4, 22 58. Zhao, B., Bilen, H.: Dataset condensation with gradient matching. In: Proceedings of the International Conference on Learning Representations (ICLR) (2021) 2, 4 59. Zhao, B., Bilen, H.: Synthesizing informative training samples with gan. In: Pro- ceedings of the Advances in Neural Information Processing Systems (NeurIPS), Workshop (2022) 2, 4 60. Zhao, B., Bilen, H.: Dataset condensation with distribution matching. In: Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). p. 6514–6523 (2023) 2, 4, 22 61. Zhong, X., Fang, H., Chen, B., Gu, X., Qiu, M., Qi, S., Xia, S.T.: Hierarchical features matter: A deep exploration of progressive parameterization method for dataset distillation. In: Proceedings of the Computer Vision and Pattern Recogni- tion Conference (CVPR). p. 30462–30471 (2025) 4 62. Zhou, B., Zhong, L., Chen, W.: Improve cross-architecture generalization on dataset distillation. arXiv preprint arXiv:2402.13007 (2024) 4 Fine-Grained Dataset Distillation19 Appendix Sec. 1 analyzes the effectiveness of FD 2 ; Sec. 2 evaluates fine-grained datasets with more distillation methods, and reports efficiency comparisons across other methods; Sec. 3 presents implementation details; Sec. 4 provides additional ab- lation studies; Sec. 5 visualizes distilled samples on CUB-200-2011. 1 Theoretical Analysis 1.1 Effectiveness of the Fine-grained Characteristic Constraint Preliminaries. Following the classical discriminative principle, recognition be- comes easier when features exhibit smaller intra-class variation and larger inter- class separation [11,49]. For sample i, let z i ∈ R d be its feature, y i ∈1,...,K its label, and μ k ∈ R d the feature center of class k. We define the intra-class deviation r i =∥z i − μ y i ∥ 2 , (1) and the average center-based margin ̄ ∆ i = E j̸=y i ∥z i − μ j ∥ 2 2 −∥z i − μ y i ∥ 2 2 .(2) Proposition 1. Let d ij =∥μ j − μ y i ∥ 2 , j ̸= y i .(3) Then the margin in Eq. (2) admits the lower bound ̄ ∆ i ≥ E j̸=y i d 2 ij − 2r i d ij . (4) Hence, decreasing r i or enlarging d ij j̸=y i increases ̄ ∆ i . In particular, if r i < P j̸=y i d 2 ij 2 P j̸=y i d ij , (5) then ̄ ∆ i > 0, implying that z i is, on average, closer to its own class center than to other class centers. Proof. For any j ̸= y i , ∥z i − μ j ∥ 2 2 −∥z i − μ y i ∥ 2 2 =∥μ j − μ y i ∥ 2 2 − 2⟨z i − μ y i , μ j − μ y i ⟩ ≥ d 2 ij − 2∥z i − μ y i ∥ 2 ∥μ j − μ y i ∥ 2 = d 2 ij − 2r i d ij ,(6) where the inequality follows from Cauchy–Schwarz. Averaging Eq. (6) over all j ̸= y i gives Eq. (4). The condition in Eq. (5) makes the right-hand side positive, hence ̄ ∆ i > 0.□ 20H. Ma and G. Li et al. In our method, we do not directly optimize the centers μ k . Instead, as de- scribed in Section 3.3, CAL maintains class prototypes c k as category-level repre- sentatives of discriminative features, which helps aggregate discriminative local information into representative vectors [16]. For distilled sample ̃x y,i , the proposed fine-grained characteristic constraint is L F ( ̃x y,i ) = β ℓ 2 (z y,i ,c y ) + (1− β) 1− E k̸=y ℓ 2 (z y,i ,c k ) , β ∈ [0, 1],(7) where ℓ 2 (u,v) = ∥u − v∥ 2 /(∥u∥ 2 + ∥v∥ 2 + ε). We introduce the normalized prototype-based discriminative score M (β) y,i = (1− β) E k̸=y ℓ 2 (z y,i ,c k ) − β ℓ 2 (z y,i ,c y ), (8) which increases when ̃x y,i becomes closer to c y and farther from c k k̸=y on average. Corollary 1. Assume that the maintained prototypes c k K k=1 approximate the discriminative class representatives in the learned feature space. Then mini- mizing Eq. (7) is equivalent to maximizing M (β) y,i . Consequently, the proposed fine-grained characteristic constraint improves intra-class compactness and inter- class separability in the normalized prototype space, thereby reducing the recog- nition difficulty of distilled samples. Proof. Rearranging Eq. (7) gives L F ( ̃x y,i ) = (1− β)−M (β) y,i .(9) Therefore, arg minL F ( ̃x y,i ) = arg maxM (β) y,i .(10) Under the assumption that c k approximate discriminative class representa- tives, maximizing M (β) y,i increases the margin between the target prototype and the other-class prototypes, which is consistent with Proposition 1.□ 1.2 Effectiveness of the Similarity Constraint Preliminaries. Existing decoupled methods generate the same-class samples sequentially. In the i-th iteration, a real image from class y is randomly selected from the original dataset as the initialization, and the i-th distilled sample is optimized under the same pipeline. The shared sample-wise generation process is written as A y,i = Ψ y x (0) y,i , (11) where A y,i ∈ R P denotes the final vectorized attention map, Ψ y denotes the common mapping for class y, x (0) y,i denotes the initialization image. If Ψ y is L y - Lipschitz around the initialization images and the solutions actually reached during optimization, then ∥A y,i − A y,j ∥ 2 ≤ L y ∥x (0) y,i − x (0) y,j ∥ 2 . i̸= j.(12) Fine-Grained Dataset Distillation21 For a same-class group of size N S , define ̄ A y = 1 N S N S X i=1 A y,i , Σ (y) A = 1 N S N S X i=1 (A y,i − ̄ A y )(A y,i − ̄ A y ) ⊤ .(13) Then tr Σ (y) A = 1 2N 2 S N S X i,j=1 ∥A y,i − A y,j ∥ 2 2 ,(14) which characterizes the overall dispersion of the same-class attention maps. Un- der linear attention pooling G y , h y,i = G ⊤ y A y,i , Σ (y) h = 1 N S N S X i=1 (h y,i − ̄ h y )(h y,i − ̄ h y ) ⊤ ,(15) we have Σ (y) h = G ⊤ y Σ (y) A G y , σ 2 min (G y ) tr Σ (y) A ≤ tr Σ (y) h ≤∥G y ∥ 2 2 tr Σ (y) A . (16) Proposition 2. Under the shared sample-wise generation process, same-class attention diversity satisfies tr Σ (y) A ≤ L 2 y 2N 2 S N S X i,j=1 ∥x (0) y,i − x (0) y,j ∥ 2 2 .(17) Hence, if same-class initialization images are close to each other, the resulting attention maps tend to focus on similar regions. Moreover, by Eq. (16), the diversity of the resulting representations is also limited. Proof. From Eq. (12), ∥A y,i − A y,j ∥ 2 2 ≤ L 2 y ∥x (0) y,i − x (0) y,j ∥ 2 2 .(18) Substituting Eq. (18) into Eq. (14) gives Eq. (17). Then Eq. (16) shows that smaller tr(Σ (y) A ) also restricts tr(Σ (y) h ).□ To counteract this tendency, we use the similarity constraint L S ( ̃x y,i ) = 1− E j<i [ℓ 2 (A y,i ,A y,j )],1 < i≤ N S ,(19) which serves as a penalty against repeated attention. Here, ℓ 2 (u,v) = ∥u − v∥ 2 /(∥u∥ 2 +∥v∥ 2 +ε) is the symmetrically normalized Euclidean metric. Assume 0 < η≤∥A y,i ∥ 2 +∥A y,j ∥ 2 + ε≤ ̄η, ∀j < i.(20) Then 1 ̄η ∥A y,i − A y,j ∥ 2 ≤ ℓ 2 (A y,i ,A y,j )≤ 1 η ∥A y,i − A y,j ∥ 2 .(21) 22H. Ma and G. Li et al. Corollary 2. Minimizing Eq. (19) increases intra-class attention diversity, and thus promotes more diverse discriminative regions and representations. Proof. From Eq. (19), minL S ( ̃x y,i ) ⇐⇒max E j<i [ℓ 2 (A y,i ,A y,j )].(22) By Eq. (21) and Jensen’s inequality, E j<i ∥A y,i − A y,j ∥ 2 2 ≥ (E j<i ∥A y,i − A y,j ∥ 2 ) 2 ≥ η 2 (E j<i [ℓ 2 (A y,i ,A y,j )]) 2 .(23) Moreover, from Eq. (14), tr Σ (y) A = 1 2N 2 S N S X p,q=1 ∥A y,p − A y,q ∥ 2 2 ≥ i− 1 N 2 S E j<i ∥A y,i − A y,j ∥ 2 2 . (24) Combining Eqs. (23) and (24) yields tr Σ (y) A ≥ i− 1 N 2 S η 2 (E j<i [ℓ 2 (A y,i ,A y,j )]) 2 . (25) Thus, minimizing L S ( ̃x y,i ) enlarges a lower bound on intra-class attention di- versity. By Eq. (16), this further promotes more diverse representations.□ 2 More Results and Discussions To further evaluate the effectiveness of FD 2 , we compare it with more SOTA dataset distillation methods. These methods mainly include coreset selection methods (Random, Herding [3], and K-Center [38]), DSA [57] (gradient match- ing), DM [60] (distribution matching), and DATM [14] (trajectory matching). We also attempted the generative method Minimax [13]. However, it failed to con- verge stably when fine-tuning the pretrained DiT [34] on fine-grained datasets, likely because fine-grained data require modeling subtle inter-class differences, while limited data hinder stable diffusion optimization and sufficient distribution coverage. Therefore, we do not include Minimax in Tab. 1 and Tab. 2. All experiments are conducted on CUB-200-2011, FGVC-Aircraft, and Stan- ford Cars, with all input images resized to 224 × 224. The coreset methods employ ResNet18 [15] as the student model; the other distillation methods fol- low their original papers and uniformly use ConvNet [12] for both distillation and post-evaluation. Accuracy of Other Methods. We further compare other methods on fine-grained datasets. As shown in Tab. 1, coreset selection yields relatively low accuracy because fine-grained data exhibit subtle inter-class differences and large intra- class variation. Among these methods, Herding achieves higher accuracy, while Random achieves lower accuracy. Dataset distillation methods perform better, Fine-Grained Dataset Distillation23 Table 1: Accuracy of coreset selection methods and other SOTA distillation methods on fine-grained datasets. DatasetIPC Coreset SelectionDataset Distillation Random Herding K-CenterDSA DM DATM SRe 2 L++ SRe 2 L++ FD 2 CUB-200-2011 11.51.51.43.6 1.8 4.353.456.4 32.02.22.26.5 4.5 6.660.064.9 52.63.02.910.2 7.5 8.763.567.0 FGVC-Aircraft 11.82.02.58.4 5.4 1.452.658.2 33.84.14.111.3 11.1 10.266.676.1 54.45.04.814.9 16.0 17.768.380.0 Stanford Cars 11.21.31.27.2 6.3 6.252.464.5 31.32.01.87.4 7.1 6.368.275.2 51.52.42.07.8 7.6 5.770.981.4 Table 2: Image optimization time per iteration and peak GPU memory of different methods on CUB-200-2011 at IPC=1. DSADMDATM SRe 2 L++ SRe 2 L++ FD 2 FADRM+ FADRM+ FD 2 Time Cost 3.9 s1.4 s2.7 s64.6 ms96.8 ms88.9 ms134.8 ms Peak GPU 68.4 GB 28.9 GB 50.0 GB 4.8 GB5.3 GB12.2 GB14.1 GB but DSA relies on gradient signals, DM on feature distribution signals, and DATM on weight trajectory signals. These signals are coarse-grained statistics over the whole dataset and thus cannot capture subtle but critical class dif- ferences. As the number of classes increases and inter-class distances shrink, such global signals are more easily dominated by shared patterns across classes, making class-specific patterns harder to preserve in distilled data. This limits discriminative representation learning and results in relatively low performance. SRe 2 L++ shows strong performance, while FD 2 further suppresses fine-grained characteristics and similarity, which reduces the difficulty for the student model to learn discriminative representations and ultimately improves accuracy. Distillation Efficiency of Different Methods. To evaluate distillation efficiency, we compare the image optimization time per iteration and peak GPU memory of different methods on CUB-200-2011 at IPC=1. Since RDED does not iteratively optimize images via gradient descent but distills samples by one-shot cropping, we do not include it in Tab. 2. As shown in Tab. 2, DSA, which relies on gradient signals, DM, which relies on feature distribution signals, and DATM, which relies on weight trajectory signals, generally require higher time and memory costs. In contrast, decoupled methods based on BN-layer global statistics are more efficient, with lower time and memory overhead, while also achieving better per- formance. Moreover, compared with SRe 2 L++ and FADRM+, FD 2 introduces only small additional time and memory costs, yet provides clear performance improvements. 24H. Ma and G. Li et al. Table 3: Optimal CAL settings of different models during pretraining and the corre- sponding accuracy. DatasetSettingsShuffleNetV2+CAL MobileNetV2+CAL DenseNet121+CAL ResNet18+CAL ResNet50+CAL CUB-200-2011 Num attention-maps 16161688 CAL Ratio α0.10.50.40.50.3 Backbone Acc55.172.377.471.676.1 CAL Acc59.775.381.276.179.6 FGVC-Aircraft Num attention-maps 1616163232 CAL Ratio α0.10.20.40.30.4 Backbone Acc71.484.387.583.987.3 CAL Acc70.384.888.484.687.7 Stanford Cars Num attention-maps 81632832 CAL Ratio α0.10.40.40.30.4 Backbone Acc71.686.788.685.289.2 CAL Acc76.788.891.6888.691.9 ImageNette Num attention-maps 81632832 CAL Ratio α0.30.10.10.30.2 Backbone Acc94.297.498.598.098.9 CAL Acc93.697.398.397.598.5 ImageWoof Num attention-maps 163232816 CAL Ratio α0.40.10.10.30.2 Backbone Acc84.292.492.492.394.1 CAL Acc87.493.393.392.693.8 3 Additional Implementation Details 3.1 Model Pretraining As shown in Tab. 3, we report the accuracy of different models integrated with CAL. On fine-grained datasets, ShuffleNetV2 is designed for lightweight com- putation and thus cannot provide sufficiently discriminative features for CAL, leading to lower accuracy than the other models. Moreover, a larger α further increases the optimization difficulty; in our experiments, ShuffleNetV2+CAL converges only at α = 0.1. These results indicate that, on fine-grained datasets, all models except ShuffleNetV2 are well compatible with CAL. In contrast, on the simpler ImageNette and ImageWoof datasets, all models show excellent com- patibility with CAL. 3.2 Distillation The Models Used in FADRM+ and FADRM+ FD 2 . We follow the FADRM+ set- ting and use four models for distillation. However, on fine-grained datasets, due to the poor performance of ShuffleNetV2+CAL, we use MobileNetV2, DenseNet121, ResNet18, and ResNet50 in both FADRM+ and FADRM+ FD 2 . For general datasets, we retain the original FADRM+ setting and use ShuffleNetV2, Mo- bileNetV2, DenseNet121, and ResNet18 for distillation. Image Initialization. Following the settings of SRe 2 ++ and FADRM+, SRe 2 ++ FD 2 and FADRM+ FD 2 also adopt 2× 2 and 1× 1 image initialization, respectively. Group-wise Distillation with Group Size N S . We optimize the IPC distilled images of each class in a group-wise manner, where each group contains at most Fine-Grained Dataset Distillation25 N S images, and each iteration corresponds to one image. Therefore, the total number of groups for each class is G = IPC N S . The first G− 1 groups all run for N S iterations, while the last group runs for N last = IPC− (G− 1)N S iterations. This strategy prevents the similarity constraint from introducing weakly discriminative regions into the distilled images while maintaining dis- tillation efficiency. 4 Addition ablation studies Effect of the Relative Weight λ between the Proposed Constraints. To determine a proper trade-off between the proposed constraints during distillation, we ab- late the relative weight λ with IPC= 5. As shown in Tab. 4, λ = 0.8 yields the higher post-evaluation Top-1 accuracy, while smaller values (0.2 and 0.4) lead to a performance drop. This suggests that assigning a higher weight to the fine-grained characteristic constraint improves the quality of distilled samples. Therefore, we set λ = 0.8 in the remaining experiments. Effect of the Group Size N S in the Similarity Constraint on the Distillation Time. We further compare the distillation time under different N S for the sim- ilarity constraint. Each group runs for N S iterations, and the reported time is the mean per-iteration cost within the group. As shown in 5, the time overhead increases with N S and reaches the highest value (1739.4 s) at N S = 5. This is be- cause more previously distilled samples are involved in the similarity constraint as the iterations proceed. Considering accuracy, N S = 4 is preferred. Table 4: On CUB-200-2011, the impact of the weight λ between the constraints on the post-evaluation Top-1 accuracy of dis- tilled samples. Methodλ Top-1 acc SRe 2 L++ FD 2 0.265.4 0.465.0 0.665.7 0.867.0 Table 5: Effect of the group size N S in the similarity constraint on distillation time on CUB-200-2011, where time denotes the mean time under the same group. Method N S Time Cost (s) SRe 2 L++ FD 2 21708.3 31709.0 41713.1 51739.4 5 More Visualizations At IPC=1, the fine-grained distilled samples obtained by FD 2 are shown in Figs. 1 to 4 for CUB-200-2011, Figs. 5 and 6 for FGVC-Aircraft, and Figs. 7 to 10 for Stanford Cars. 26H. Ma and G. Li et al. Fig. 1: Visualization of distilled samples from the first 100 classes on CUB-200-2011 obtained by SRe 2 ++ FD 2 at IPC= 1. Fine-Grained Dataset Distillation27 Fig. 2: Visualization of distilled samples from the last 100 classes on CUB-200-2011 obtained by SRe 2 ++ FD 2 at IPC= 1. 28H. Ma and G. Li et al. Fig. 3: Visualization of distilled samples from the first 100 classes on CUB-200-2011 obtained by FADRM+ FD 2 at IPC= 1. Fine-Grained Dataset Distillation29 Fig. 4: Visualization of distilled samples from the last 100 classes on CUB-200-2011 obtained by FADRM+ FD 2 at IPC= 1. 30H. Ma and G. Li et al. Fig. 5: Visualization of distilled samples on FGVC-Aircraft obtained by SRe 2 ++ FD 2 at IPC= 1. Fine-Grained Dataset Distillation31 Fig. 6: Visualization of distilled samples on FGVC-Aircraft obtained by FADRM+ FD 2 at IPC= 1. 32H. Ma and G. Li et al. Fig. 7: Visualization of distilled samples from the first 100 classes on Standford Cars obtained by SRe 2 ++ FD 2 at IPC= 1. Fine-Grained Dataset Distillation33 Fig. 8: Visualization of distilled samples from the last 96 classes on Standford Cars obtained by SRe 2 ++ FD 2 at IPC= 1. 34H. Ma and G. Li et al. Fig. 9: Visualization of distilled samples from the first 100 classes on Standford Cars obtained by FADRM+ FD 2 at IPC= 1. Fine-Grained Dataset Distillation35 Fig. 10: Visualization of distilled samples from the last 96 classes on Standford Cars obtained by FADRM+ FD 2 at IPC= 1.