Paper deep dive
Accelerating Large-Scale Dataset Distillation via Exploration-Exploitation Optimization
Muhammad J. Alahmadi, Peng Gao, Feiyi Wang, Dongkuan Xu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/21/2026, 4:16:24 AM
Summary
The paper introduces Exploration-Exploitation Distillation (E2D), a method for large-scale dataset distillation that bridges the accuracy-efficiency gap. E2D utilizes full-image initialization to preserve semantic integrity and a two-phase optimization strategy: an exploration phase for uniform updates to identify high-loss regions, and an exploitation phase to focus updates on these regions, thereby reducing redundancy and accelerating convergence. It achieves state-of-the-art accuracy on ImageNet-1K and ImageNet-21K while being significantly faster than existing methods.
Entities (11)
Relation Signals (10)
E2D → evaluatedon → ImageNet-1K
confidence 98% · We evaluate E2D on large-scale benchmarks ... on ImageNet-1K
E2D → evaluatedon → ImageNet-21K
confidence 98% · and on ImageNet-21K, our method substantially improves accuracy
E2D → consistsof → Exploration Phase
confidence 95% · It then uses a two-phase optimization strategy: an exploration phase ... and an exploitation phase
E2D → consistsof → Exploitation Phase
confidence 95% · It then uses a two-phase optimization strategy: ... an exploitation phase that focuses updates on these regions
E2D → uses → Full-image Initialization
confidence 95% · E2D ... begins with full-image initialization to preserve semantic integrity and feature diversity.
Exploitation Phase → focuseson → High-loss Regions
confidence 92% · an exploitation phase that focuses updates on these regions
Exploration Phase → identifies → High-loss Regions
confidence 92% · an exploration phase that performs uniform updates and identifies high-loss regions
E2D → outperforms → EDC
confidence 90% · surpassing the state-of-the-art on ImageNet-1K while being 18× faster
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Dataset distillation compresses the original data into compact synthetic datasets, reducing training time and storage while retaining model performance, enabling deployment under limited resources. Although recent decoupling-based distillation methods enable dataset distillation at large scale, they continue to face an efficiency gap: optimization-based decoupling methods achieve higher accuracy but demand intensive computation, whereas optimization-free decoupling methods are efficient but sacrifice accuracy. To overcome this trade-off, we propose Exploration--Exploitation Distillation (E$^2$D), a simple, practical method that minimizes redundant computation through an efficient pipeline that begins with full-image initialization to preserve semantic integrity and feature diversity. It then uses a two-phase optimization strategy: an exploration phase that performs uniform updates and identifies high-loss regions, and an exploitation phase that focuses updates on these regions to accelerate convergence. We evaluate E$^2$D on large-scale benchmarks, surpassing the state-of-the-art on ImageNet-1K while being $18\times$ faster, and on ImageNet-21K, our method substantially improves accuracy while remaining $4.3\times$ faster. These results demonstrate that targeted, redundancy-reducing updates, rather than brute-force optimization, bridge the gap between accuracy and efficiency in large-scale dataset distillation. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2602.15277v2
- Canonical: https://arxiv.org/abs/2602.15277v2
PDF not stored locally. Use the link above to view on the source site.
Full Text
55,510 characters extracted from source content.
Expand or collapse full text
Accelerating Large-Scale Dataset Distillation via Exploration–Exploitation Optimization Muhammad J. Alahmadi 1,2 Peng Gao 1 Feiyi Wang 3 Dongkuan (DK) Xu 1 1 North Carolina State University 2 King Abdulaziz University 3 Oak Ridge National Laboratory mjalahma,pgao5,dxu27@ncsu.edu fwang2@ornl.gov mjalahmadi@kau.edu.sa Abstract Dataset distillation compresses the original data into compact synthetic datasets, reducing training time and storage while retaining model performance, enabling deployment under limited resources.Although recent decoupling-based distillation methods enable dataset dis- tillation at large-scale, they continue to face an effi- ciency gap: optimization-based decoupling methods achieve higher accuracy but demand intensive computation, whereas optimization-free decoupling methods are efficient but sac- rifice accuracy. To overcome this trade-off, we propose Exploration–Exploitation Distillation (E 2 D), a simple, prac- tical method that minimizes redundant computation through an efficient pipeline that begins with full-image initializa- tion to preserve semantic integrity and feature diversity. It then uses a two-phase optimization strategy: an exploration phase that performs uniform updates and identifies high-loss regions, and an exploitation phase that focuses updates on these regions to accelerate convergence. We evaluate E 2 D on large-scale benchmarks, surpassing the state-of-the-art on ImageNet-1K while being 18×faster, and on ImageNet-21K, our method substantially improves accuracy while remain- ing 4.3×faster. These results demonstrate that targeted, redundancy-reducing updates, rather than brute-force opti- mization, bridge the gap between accuracy and efficiency in large-scale dataset distillation. Code is available at https://github.com/ncsu-dk-lab/E2D. 1. Introduction Dataset distillation, also known as dataset condensation, aims to distill the original data into compact synthetic datasets that retain the key information of the original data while being orders of magnitude smaller [2,33,34,42]. Distilled datasets train models faster and require less stor- age, enabling adaptability and real-world deployment under tight time and resource budgets [27,34,39]. Despite recent advances, dataset distillation remains constrained by high 050100150200 GPU Hours 30 35 40 45 50 55 60 Top-1 Accuracy (%) Image Per Class (IPC) IPC-10 IPC-50 Distillation Methods DWA RDED DELT EDC Ours Distillation Methods DWA RDED DELT EDC Ours Ours Figure 1. Comparison of Top-1 accuracy and synthesis time on ImageNet-1K using ResNet-18 for various dataset distillation meth- ods at IPC 10 and IPC 50. Synthesis time is measured on a single RTX A6000 GPU. Our method converges substantially faster and achieves the highest accuracy, leading to the best ac- curacy–efficiency trade-off. computational cost, with the most common bi-level distilla- tion methods requiring days to synthesize even moderately sized datasets (e.g., CIFAR-100) [12, 38, 42]. In response, a line of work on efficiency and scalability introduced the decoupled paradigm [1,37], which separates model training from synthetic data optimization to avoid costly bi-level updates. This paradigm emphasizes the gener- ation of soft labels to provide richer supervision, transferring much of the computational burden from optimization to label generation, as one study emphasis “a label is worth a thou- sand images” [20,35]. Building on this trend, RDED [29] advances the decoupled paradigm toward an optimization- free approach, eliminating iterative updates by directly ex- tracting and recombining representative patches from real images, reducing synthesis time from hours to minutes. Nevertheless, the drive toward faster distillation has not overcome the accuracy–efficiency trade-off. Optimization- free methods are fast but lose accuracy, since the teacher only guides patch selection without optimizing the syn- thetic data. In contrast, optimization-based methods typ- ically achieve higher accuracy, especially when combined with optimization-free strategies as an initialization step [25, 32], but remain computationally expensive. For instance, 1 arXiv:2602.15277v2 [cs.CV] 19 Feb 2026 EDC [25], a recent dataset distillation method, requires over 200 GPU hours to distill ImageNet-1K at 50 IPC. Our work aims to bridge this gap by asking two key questions. •Q1: How can we accelerate decoupling-based distillation to narrow the accuracy–efficiency gap? • Q2: Building on the efficiency focus, we ask whether decoupling-based distillation can reach its best accuracy early, such that further optimization not only adds cost but also degrade performance, challenging the assumption that more optimization always helps? We attribute inefficiency in recent decoupling-based dataset distillation methods to redundancy which we argue stems from two main sources: similar patches generated during initialization, and repeated low-value updates during optimization. Existing methods propagate redundancy by applying uniform gradient updates across all regions, regard- less of their contribution to loss reduction. This uniform treatment [7,25,37] inflates computation without improving representational quality. Unlike prior work that focuses only on diversity [7,24,26], we explicitly reduce redundancy, aligning efficiency and diversity to steer dataset distillation away from brute-force optimization toward leaner, more focused updates. To address Q1, we revisit patch-based initialization adopted in prior works [25,26,29]. This mechanism often produces clusters of similar patches, which reduces diver- sity and limits the representational coverage of the synthetic dataset, particularly when multiple patches originate from the same image, as shown in Fig. 3. Moreover, compact patch representations can distort features, compromising the quality of the synthetic data [44]. We instead use full-size image initialization, which ensures better initial represen- tations, reducing the need for extensive optimization. Re- markably, this simple change produces distilled data that already matches the accuracy of state-of-the-art methods before any optimization, substantially narrowing the effi- ciency–accuracy gap. Next, we propose a two-phase optimization strategy that reduces the redundant uniform updates of prior works. Rather than treating all regions in the synthetic data as equally valuable, the exploration phase broadly updates diverse regions to ensure coverage, before the exploitation phase concentrates updates on high-loss regions that provide stronger learning signals. This accelerates convergence and, with an accelerated student training schedule, delivers up to 20× efficiency gains (Fig. 1). Turning to Q2, we make the counter-traditional observa- tion that more optimization is not always beneficial. Pro- longed updates tend to reinforce redundant global dataset statistics while eroding both diversity and fine-grained fea- tures inherent in the original dataset. In contrast, our approach demonstrates that a focused optimization strat- egy can yield higher-quality distilled datasets with fewer 02004006008001000 Class Index 0.65 0.70 0.75 0.80 0.85 0.90 Cosine Similarity SRe²L RDED EDC DELT Ours Figure 2. Semantic cosine similarity across ImageNet-1K classes at IPC 50 using a ResNet-18 teacher. Lower values indicate greater di- versity and reduced redundancy; our method consistently achieves the lowest similarity. optimization steps. As illustrated in Fig. 2, our method achieves consistently lower semantic cosine similarity across ImageNet-1K classes, indicating richer diversity. At the same time, it achieves state-of-the-art accuracy, reaching peak ImageNet-1K performance with≈ 10×fewer opti- mization steps than EDC (Fig. 1). This challenges the con- ventional assumption that longer optimization is always bene- ficial and highlights the importance of efficient optimization. Our main contributions are: 1.We identify redundancy as a key inefficiency in recent decoupling-based dataset distillation. It arises from patch- based initialization, where similar crops dominate, and uniform optimization that ignores regional importance. While prior work assumes more optimization improves data, we find excessive updates amplify redundancy and degrade quality, reframing dataset distillation as an efficiency-driven, diversity-aligned process. 2.We propose the Exploration–Exploitation Distillation method (E 2 D) which departs from prior uniform opti- mization methods by integrating full-image initialization to preserve semantic integrity and diversity with a novel two-phase optimization strategy that identifies high-loss regions and concentrates updates there to reduce redun- dancy and accelerate convergence. 3.Extensive experiments validate the effectiveness of E 2 D on large-scale benchmarks. On ImageNet-1K, it sur- passes state-of-the-art methods while reducing synthesis time by up to 18×. On ImageNet-21K, it yields accu- racy gains of up to +9.6% while remaining 4.3× faster. These results show that E 2 D achieves a superior balance between accuracy and computational cost, making it a practical solution for large-scale dataset distillation. 2 2. Related Work Bi-Level Optimization Distillation. Early dataset distilla- tion methods often adopt bi-level optimization, where syn- thetic data updates are aligned with updates of models trained on the original dataset, in a batch-by-batch manner. Match- ing strategies include gradient matching [12,38,42], trajec- tory matching [2,4], and distribution matching [14,33,39, 41,45]. While distribution matching avoids costly bi-level optimization, it typically underperforms gradient or trajec- tory matching. To address this trade-off, Zhang et al. [38] accelerate gradient-based optimization through early-model augmentation and parameter perturbation. DREAM [15] accelerates optimization by matching only representative im- ages from the original dataset. TESLA [4] extends trajectory matching to ImageNet-1K with memory reduction and soft labels, but remains resource-intensive. Overall, these meth- ods achieve strong alignment but struggle to scale beyond small datasets. Decoupled-based Distillation.To improve scalability, decoupled-based approaches separate model training from synthetic data optimization.SRe 2 L [37] introduces a three-stage pipeline: Squeeze (pretrain with Batch Normal- ization (BN) statistics), Recover (independent optimization of synthetic data with global (BN) alignment), and Rela- bel (assign soft labels via a teacher model). CDA [36] scales to ImageNet-21K and further enhances performance with curriculum-based augmentation. RDED [29] follows a similar decoupled approach but removes the optimization stage, yielding several-fold efficiency improvement by se- lecting teacher-guided patches that balance diversity and realism [13]. Diversity in Decoupled-based Distillation. Several works address the limited diversity of global-statistics alignment in the decoupled-based distillation. DWA [7] promotes diver- sity by dynamically adjusting teacher weights and separating BN mean and variance terms to strengthen variance align- ment. G-VBSM [24] expands diversity by incorporating multiple backbones, layers, and statistics. More recently, EDC [25] builds on this framework to enhance both perfor- mance and efficiency through pipeline refinements across the distillation process. DELT [26] enhances diversity at the IPC level by assigning different optimization depths to each synthetic image, producing image-level variation to reduce uniformity in decoupled distillation Generative Distillation. A parallel line of work explores generative approaches such as diffusion-based distillation [8, 19,28,44] and GAN-guided synthesis [3,40,43] which shift from matching updates to training generative models that output synthetic data. However, they are orthogonal to optimization-based methods, as their generative outputs can be used as initialization for further refinement. (a) RDED (b) DELT (c) EDC (d) Ours Figure 3. Visual comparison of synthetic data generated by SRe 2 L, RDED, DELLT, EDC, and our method E 2 D, which produces more diverse, less redundant samples and preserves semantic integrity with full-size feature representations. 3. Methodology 3.1. Preliminaries LetD =(x i ,y i ) N i=1 be the original training dataset, where x i ∈ R H×W×C is an image, andy i ∈ 1,...,Lis its ground-truth label. Dataset distillation aims to synthesize a compact datasetS = ( ̃x j , ̃y j ) M j=1 withM ≪ N, such that a model trained onSachieves competitive performance compared to training onD, by effectively capturing essential and diverse information withinD. 3.2. Overview We study efficiency bottlenecks in recent decoupled dataset distillation methods and identify redundancy in two aspects of the pipeline: (i) suboptimal initialization that burdens the optimization with unnecessary corrective updates, and (i) prolonged optimization that increasingly reinforces re- dundant signals and constrains efficiency. While prior work has primarily focused on improving diversity through ar- chitectural choices or data sampling strategies, we observe that excessive optimization signals not only increase syn- thesis cost but can also degrade instance-level feature rich- ness, leading to negative returns. Our analysis shows that full-image initialization provides a substantially stronger starting point than the patch-based schemes commonly used in prior work [5,25,26,29,32]. Building on these insights, we propose a novel optimization strategy that mitigates re- 3 Full-size Image Initialization Optimize the crop 23 1 4 Optimization in Exploration Phase Optimization in Exploitation Phase Accelerated Learning Schedule in Evaluation Teacher model confidence Optimize high-loss crops until empty Add the high loss crop to the list Figure 4. Overview of our proposed method E 2 D. The pipeline consists of four components: (1) Full-size Image Initialization, which preserves the semantic and structural information of the original data, preventing distortion or redundancy; (2) Exploration Phase, which identifies challenging high-loss regions and ensure balanced optimization; (3) Exploitation Phase, which iteratively refines these challenging regions for efficient convergence; and (4) Accelerated Learning Schedule, applied during student training to further speed up convergence. Together, these components enable fast and effective dataset distillation with minimal redundancy. 0502004005002000 Optimization steps 0.65 0.75 0.85 0.95 Avg intra-class cosine similarity Ours (r = 0.930) EDC (r =0.706) Full-size init RDED (2×2 patch init) Random noise Figure 5. Cosine similarity trends across optimization steps. dundancy, preserves diversity and accelerate convergence. 3.3. Redundancy in Initialization and Optimization We treat redundancy as the degree of similarity between syn- thetic instances, viewing it as the opposite of diversity. Since diversity is inherently multi-dimensional, we adopt a practi- cal proxy based on pairwise cosine similarity of teacher fea- tures. Higher similarity indicates more redundant representa- tions, whereas lower similarity reflects greater instance-level diversity. Using this measure, we examine how teacher fea- ture representations evolve under patch-based and full-image initialization, and how subsequent optimization amplifies or reduces redundant signals. As shown in Fig. 5, patch-based initialization exhibits high redundancy and distorted feature representations, be- having closer to random noise than to the original data distri- bution captured by full-image initialization. In this regime, global-statistics optimization acts as a corrective step, reduc- ing redundancy by steering noisy and misaligned features toward meaningful class-level structure. In contrast, when optimization is applied to full-image initialization, where features already align with the data distribution, the same global objective behaves differently. Rather than correct- ing noise, continued global alignment progressively ampli- fies shared class-level signals, increasing redundancy and eroding instance-level diversity over time. This reveals a conditional behavior of global-statistics optimization: it is beneficial when representations are far from the target dis- tribution, but increasingly redundant once coarse alignment has been achieved. Motivated by these observations, we aim to accelerate synthesis by controlling redundancy during optimization while retaining strong class-level supervision. Rather than adding new objectives, we control when global updates are applied by allowing broad corrective optimization early and then limiting redundant updates once representations are aligned. This preserves instance-level diversity inherited from full-image initialization while improving synthesis effi- ciency. 3.4. Exploration–Exploitation Optimization Strat- egy We propose a two-phase optimization strategy that mitigates the redundancy of random multi-crop updates used in prior methods [7,25,37]. In conventional multi-crop optimiza- tion, each iteration performs in-place updates on randomly sampled local regions, repeatedly visiting similar patches and generating redundant learning signals. Inspired by the exploration–exploitation trade-off in reinforcement learn- ing [11], our method separates this process into distinct ex- ploration–exploitation phases to achieve efficient coverage 4 Algorithm 1: Exploration–Exploitation Distillation (E 2 D) Input: Original datasetD , teacher modelT , loss threshold ε, exploration iterations K, total iterations T Output: Updated synthetic datasetS 1 foreach (x i ,y i )∼S do / * Exploration Phase * / 2InitializeM i ← ∅ 3Initialize x i with a random image fromD 4for t = 1 to K do 5 ̃x i ← RandomResizedCrop(x i ) 6l i =L ce (T ( ̃x i ),y i ); 7if l i > ε then 8Store (crop coordinates,l i ) inM i ; 9Update ̃x i using the baseline distillation loss. / * Exploitation Phase * / 10for t = K + 1 to T do 11ifM i ̸=∅ then 12Sample (crop coordinates) fromM i via softmax over l i ; 13 ̃x i ← Crop(x i , crop coordinates); 14l ′ i =L ce (T ( ̃x i ),y i ); 15if l ′ i > ε then 16Update stored loss inM i ; 17else 18Remove crop fromM i ; 19else / * Early Stopping * / 20break 21Update ̃x i using the baseline distillation loss. and targeted refinement, directing computation toward re- gions with strong learning potential to increase information density and accelerate convergence. Our method is formally outlined in Algorithm 1. Exploration Phase. This phase performs random multi- crop optimization overKiterations to broadly update the synthetic data and record informative regions. For each synthetic imagex i , crops that yield high teacher loss L ce (T( ̃x i ),y i ) > εare stored with their coordinates and loss values in a per-image memory bufferM i . These buffers accumulate the teacher’s uncertainty landscape, identifying regions that are under-optimized or poorly aligned with class semantics. The same random crop is applied to all images in the batch to maintain computational efficiency. Exploitation Phase. AfterKexploration iterations, opti- mization shifts to focused refinement of the high-loss regions identified in the exploration phase. For each synthetic image x i , crops are sampled from its bufferM i with probabilities proportional to their stored losses: p ij = exp(ℓ ij ) P N i k=1 exp(ℓ ik ) (1) wherep ij denotes the probability of selecting thej-th high-loss crop in imagex i ,ℓ ij represents the loss of thej-th crop in imagex i , andN i indicates the total number of crops stored in the bufferM i . This softmax weighting prioritizes difficult regions while maintaining diversity among selected crops. Updated losses are re-evaluated and refreshed in memory; regions that fall below the loss threshold are discarded, preventing redundant computation on well-optimized areas. Optimization proceeds until all buffers are emptied or the maximum iteration budgetTis reached. Early stopping prevents over-optimization of the distilled data, preserving image features inherited from the original dataset. 4. Experiments To evaluate our method, we perform experiments on two large-scale datasets using multiple model architectures to assess its effectiveness and its ability to generalize across architectures. We conduct comprehensive ablations and convergence analyses, examining how varying acceleration factors, including optimization-free settings affect the effi- ciency–performance trade-off. 4.1. Experimental Setups Datasets. We evaluate our method on two large-scale and computationally intensive benchmarks: ImageNet-1K [6] and ImageNet-21K-P (referred to as ImageNet-21K through- out the paper) [22]. ImageNet-1K consists of 1,000 classes and 1.28 million images, while ImageNet-21K includes 10,450 classes and over 11 million images. We omit results on small-scale datasets (e.g., CIFAR-10/100), where our baseline EDC already achieves state-of-the-art performance with very few optimization iterations, which makes further acceleration provide negligible efficiency gains. Additional discussion is provided in the appendix. Network architectures. For Tab. 1 and Tab. 2, we re- port results using the widely adopted ResNet-18 archi- tecture as the evaluation model.Following previous works [25,36], and to ensure the distilled data are not over- fitted to any single architecture, we further assess cross- architecture generalization on diverse models including ResNet-50,101[9], MobileNet-V2 [23], EfficientNet- B0 [30], ConvNext-Tiny [16], ShuffleNet-V2 [18], DeiT- Tiny [31], RegNet-Y-8GF [21], and DenseNet-121 [10]. Baselines. We compare against recent state-of-the-art meth- ods in large-scale dataset distillation. For ImageNet-1K, we compare with SRe 2 L [37], CDA [36], DWA [7], G- VBSM [24], RDED [29], DELT [26], and EDC [25]. On 5 ResNet-18 DatasetIPCSRe 2 LCDADWARDEDDELTEDCOurs (optimization-free) Ours 10× ImageNet-1K1–6.6 ± 0.2–12.8 ± 0.111.5 ± 0.412.9 ± 0.4 1021.3 ± 0.631.4 ± 0.533.5 ± 0.337.9 ± 0.246.1 ± 0.448.6 ± 0.348.1 ± 0.250.0 ± 0.1 5046.8 ± 0.251.8 ± 0.453.5 ± 0.355.2 ± 0.259.2 ± 0.458.0 ± 0.258.4 ± 0.158.9 ± 0.1 Table 1. Comparison of dataset distillation methods on ImageNet-1K using ResNet-18 as the evaluation model. Our method is presented in two variants: with an acceleration factor of10×and an optimization-free variant. Our results are reported as mean ± standard deviation over five evaluation models, each randomly initialized and trained independently on the distilled dataset. Missing entries (–) indicate results not reported in the original work. ResNet-18 DatasetIPCSRe 2 LCDAD3SOurs (optimization-free) Ours 5× ImageNet-21K1018.5 ± 0.222.6 ± 0.226.9 ± 0.128.8 ± 0.0232.1 ± 0.4 2021.8 ± 0.126.4 ± 0.128.5 ± 0.133.9 ± 0.136.0 ± 0.1 Table 2. Comparison of dataset distillation methods on ImageNet-21K using ResNet-18 as the evaluation model. Our method is presented with an acceleration factor of5×, along with an optimization-free variant. Our results are reported as mean ± standard deviation over three evaluation models, each randomly initialized and trained independently on the distilled dataset. ImageNet-21K, we consider SRe 2 L, CDA, and D3S [17] as baselines. Implementation details. We adopt EDC as the implemen- tation backbone for our method in ImageNet-1K and CDA in ImageNet-21K. Both are accelerated using our Explo- ration–Exploitation strategy, allocating 70% of iterations to exploration and the remaining 30% to exploitation. Our method integrates seamlessly with state-of-the-art methods, demonstrating its adaptability cross both large-scale datasets and different distillation methods. EDC and CDA are op- timized for 2000 iterations, and our reported acceleration factors are measured relative to this budget. When extending our method to ImageNet-21K, we reused the optimization hyperparameters from prior work. Under this setup, even slight optimization caused performance to degrade below that of the optimization-free variant. We attribute this to a distributional mismatch between the teacher model and the synthesized data. To mitigate this, we reduced the learning rate and relaxed the batch-normalization regularization term α BN , which stabilized optimization and yielded smoother convergence curves. Additional implementation details and hyperparameter configurations are provided in the appendix. 4.2. Results Our method demonstrates strong performance as shown in Tab. 1 and Tab. 2. Despite being several folds more efficient, it consistently achieves superior accuracy compared to recent state-of-the-art methods. On ImageNet-1K, our method sets a new benchmark at the highly compressed IPC = 10 setting, reaching 50% accuracy. On the more challenging ImageNet-21K, it de- livers a substantial improvement by achieving 32.1% ac- curacy at IPC = 10. While the performance gap narrows as IPC increases, our method continues to outperform state-of-the-art methods, achieving 58.9% on ImageNet- 1K at IPC = 50 and 36% on ImageNet-21K at IPC = 20. Our optimization-free variant performs on par with the strongest prior method, EDC, on ImageNet-1K, despite re- quiring no optimization. Remarkably, on ImageNet-21K, our optimization-free variant outperforms all existing meth- ods while using only half the distillation budget (IPC = 10) compared to state-of-the-art methods trained at IPC = 20. 4.3. Cross-architecture Generalization A key property of distilled datasets is their ability to gen- eralize across diverse model architectures. We evaluate our method against its closest baselines on a range of architectures, as shown in Tab. 3 for ImageNet-1K and Tab. 4 for ImageNet-21K. Our method consistently outper- forms state-of-the-art methods across all models, demon- strating improved robustness and consistently superior cross- architecture performance. 4.4. Acceleration and Efficiency Acceleration of Optimization. We study how accelerat- ing our method influences performance relative to the EDC baseline on ImageNet-1K. As shown in Fig. 6, optimization- free settings generally perform increasingly close to the optimization-based ones as the IPC increases, a trend also observed in prior work [35]. Notably, our method in the optimization-free setting nearly matches the performance of the optimized version at high IPC, highlighting its efficiency given the substantial cost of optimization at large IPC. The 20×accelerated variant of our method results in less than 0.1% accuracy drop, striking an effective balance between speed and performance. The 10×accelerated version of 6 IPCMethod Evaluation model ResNet-18ResNet-50ResNet-101MobileNet-V2EfficientNet-B0ConvNext-TinyShuffleNet-V2 RDED42.046.048.334.442.848.319.4 10EDC48.654.151.745.051.154.429.8 Ours 10× 50.055.654.746.952.955.045.4 RDED56.663.761.253.957.665.430.9 50EDC58.064.364.957.860.966.645.7 Ours 10× 58.964.765.158.161.366.856.4 Table 3. Cross-architecture generalization performance on ImageNet-1K at IPC 10 and 50 across multiple evaluation models. IPCMethod Evaluation model ResNet-18ResNet-50ResNet-101DenseNet-121RegNet-Y-8GFConvNeXt-Tiny 20 SRe 2 L21.831.233.224.634.234.9 CDA26.435.336.128.636.136.3 D3S 28.535.436.031.936.4– Ours 5× 36.036.937.333.137.437.2 Table 4. Cross-architecture generalization performance on ImageNet-21K data at IPC 20 across multiple evaluation models. Missing entries (–) indicate unreported results with no publicly available codebase. our method is sufficient to reach convergence, while the 5× version does not provide additional gains. As the number of optimization steps approaches the standard 2000 iterations, both our method and the baseline converge in performance, suggesting that excessive optimization reinforces redundant signals and reduces instance-level diversity, supporting a counter-traditional view that more optimization is not neces- sarily better. Ours 20× Ours 10× EDC Synthesis Time (hrs)12.324.0229.8 Synthesis Speedup18.6×9.5× 1.0× (a) ImageNet-1K (IPC = 50) Ours 10× Ours 5× CDA Synthesis Time (hrs)38.867.7296.5 Synthesis Speedup7.6×4.3× 1.0× (b) ImageNet-21K (IPC = 20) Table 5. Synthesis time and acceleration factor comparison. Each subtable reports synthesis time (hours) and speedup over baseline on a single NVIDIA RTX A6000 GPU with ResNet-18, evaluated on different datasets. Efficiency Comparison. We compute the actual accelera- tion factor of our method, accounting for the overhead intro- duced by the Exploration–Exploitation strategy. To ensure fair comparison, all experiments are conducted on a single NVIDIA RTX A6000 GPU, and synthesis time is reported excluding data and model loading overhead. As shown in Tab. 5, our method accelerates EDC by over 18×while maintaining higher near-converged accuracy. The additional cost of the Exploration–Exploitation optimization strategy remains modest relative to the 2×improvement in synthesis efficiency (see Tab. 6 in the study). Beyond synthe- sis speed, we also address memory efficiency by applying Automatic Mixed Precision (AMP) to the forward pass of the bottleneck backbone architecture, EfficientNet-B0. This tweak reduces peak GPU memory consumption from 16.05 GB to 14 GB without any measurable loss in accuracy, al- lowing our method to fit comfortably on a single 16 GB GPU. For ImageNet-21K, our method also improves CDA effi- ciency, though the gain is smaller due to the dataset’s large- scale, which increases the cost of maintaining high-loss crop buffers compared to random cropping. Nevertheless, our method achieves consistently faster synthesis with stronger accuracy across both benchmarks. 4.5. Discussion Effectiveness of Exploration–Exploitation Strategy We evaluate the impact of our Exploration–Exploitation strategy against the random multi-crop optimization used in the baselines. As shown in Tab. 6, our method outperforms the baseline even when the latter is given twice the optimiza- tion budget, enabling up to 2× faster convergence during synthesis. Design Space of E 2 D. Toexaminethedesignspaceofourexplo- ration–exploitation method, we evaluate four variants in Tab. 7. (1) Exploit only. Image updates occur exclusively during exploitation, while exploration evaluates random crops using the teacher loss without performing any 7 0 (0) ×20 (100) ×10 (200) ×5 (400) ×1 (2000) Speed-Up (Optimization Steps) 44 46 48 50 Top-1 Accuracy (%) Ours EDC (a) ImageNet-1K (IPC 10). 0 (0) ×20 (100) ×10 (200) ×5 (400) ×1 (2000) Speed-Up (Optimization Steps) 55 56 57 58 59 Top-1 Accuracy (%) Ours EDC (b) ImageNet-1K (IPC 50). Figure 6. Top-1 accuracy vs. speedup for our method and baseline EDC under two IPC settings. Our method converges faster and starts from a stronger no-optimization starting point. In our method, extra optimization past convergence degrades performance, counter to the common assumption that more optimization is always better. IPCOurs 20× (R)Ours 20× (E 2 )Ours 10× (R) 1049.7±0.249.9±0.149.8±0.1 5058.4±0.158.8±0.0458.8±0.1 Table 6. Effectiveness of Exploration–Exploitation strategy. Ablation of our Exploration–Exploitation optimization (E 2 ) vs. random multi-crop optimization (R) on ImageNet-1K. E 2 improves accuracy and enables up to 2×faster convergence during synthesis. image-level updates. (2) GradCAM-guided probing. For each synthetic image, a GradCAM map is computed, and crops are sampled inversely to the activation magnitude so that low-activation regions are prioritized. (3) Alternating cycles. A periodic schedule of twenty steps in which exploration and exploitation alternate, replacing our two-stage procedure. (4) Random multi-crop optimization. The default strategy adopted in prior work, where crops are sampled uniformly at random throughout optimization. Across these variants, our method achieves the strongest performance. The drop in the exploit-only setting highlights the importance of uniform image-level updates, and the GradCAM variant further shows that semantic activation maps alone are insufficient, performing worse than simple random cropping. IPCExploit-OnlyGrad-CAMAlt. CyclesRandom CropE 2 D 1048.5±0.249.4±0.249.6±0.249.7±0.249.9±0.1 Table 7. Ablation of the design space of E 2 D under a×20 acceler- ation factor on ImageNet-1K (IPC 10). When to Switch to Exploitation Phase We analyze the effect of switching from exploration to exploitation at different points, controlled by hyperparame- terK. IfKis too small, the switch is premature and crop diversity suffers; if too large, it degenerates to random crop- ping. Our experiments show that performance remains stable across a broad range ofKvalues, suggesting robustness to the exact choice ofK. As shown in Tab. 8, allocating 70% of the iterations to exploration and 30% to exploitation yields the best accuracy, offering an effective balance between crop diversity and targeted refinement. K40607080 Ours 20× 49.7±0.249.8±0.149.9±0.149.7±0.1 Table 8. When to switch to exploitation. Ablation study on the number of exploration iterationsK(out of 100 total iterations) for our method accelerated by 20× on ImageNet-1K with IPC = 10. 5. Conclusion We introduced Exploration–Exploitation Distillation (E 2 D), a redundancy-reducing method for efficient large-scale dataset distillation. E 2 D integrates full-image initializa- tion, targeted Exploration–Exploitation optimization, and an accelerated student schedule to minimize redundant updates while preserving diversity.Evaluated on two large-scale benchmarks, ImageNet-1K and ImageNet-21K, E 2 D achieves higher accuracy with substantially lower syn- thesis cost. These findings show that greater accuracy does not require more optimization; rather, directing computation where it matters most leads to better accuracy–efficiency trade-offs in large-scale dataset distillation. Limitations and Future Work. Our method does not ac- count for the cost/efficiency of the relabeling stage or of constructing or selecting the pretrained models and collect- ing their statistics. We rely on simple random sampling for initialization, but more robust strategies may lead to more consistent performance. Furthermore, bridging the accu- racy–efficiency gap at higher IPC levels may require refined optimization and initialization strategies beyond existing approaches. 8 Acknowledgments This work was partially supported by the National Science Foundation (NSF) grants BCS-2416846, OAC-2417850, and DUE-2526340. Muhammad J. Alahmadi is supported in part by King Abdulaziz University (KAU). This research used resources of the Oak Ridge Leadership Computing Facility at Oak Ridge National Laboratory, supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. The views and conclusions are those of the authors and should not be interpreted as representing the official policies of the funding agencies or the government. References [1]Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. arXiv preprint arXiv:2006.08572, 2020. 1 [2] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Wearable imagenet: Syn- thesizing tileable textures via dataset distillation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2022. 1, 3 [3]George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Generalizing dataset distil- lation via deep generative prior. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 3 [4]Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet-1k with constant memory. In International Conference on Machine Learning (ICML), pages 6565–6590. PMLR, 2023. 3 [5]Jiacheng Cui, Zhaoyi Li, Xiaochen Ma, Xinyue Bi, Yaxin Luo, and Zhiqiang Shen. Dataset distillation via committee voting, 2025. 3, 1 [6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255, 2009. 5 [7] Jiawei Du, Xin Zhang, Juncheng Hu, Wenxin Huang, and Joey Tianyi Zhou. Diversity-driven synthesis: Enhancing dataset distillation through directed weight adjustment. In Ad- vances in Neural Information Processing Systems (NeurIPS), 2024. 2, 3, 4, 5, 1 [8]Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Hao- nan Wang, Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3 [9]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. 5 [10] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kil- ian Q. Weinberger. Densely connected convolutional net- works. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2261–2269, 2017. 5 [11]Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of Artifi- cial Intelligence Research, 4:237–285, 1996. 4 [12]Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data param- eterization. In International Conference on Machine Learning (ICML), 2022. 1, 3 [13]Guang Li, Bo Zhao, and Tongzhou Wang. Awesome dataset distillation github repo, 2022. 3 [14]Hongcheng Li, Yucan Zhou, Xiaoyan Gu, Bo Li, and Weiping Wang. Diversified semantic distribution matching for dataset distillation. In Proceedings of the 32nd ACM International Conference on Multimedia, page 7542–7550, New York, NY, USA, 2024. Association for Computing Machinery. 3 [15]Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. DREAM: Efficient dataset distillation by representative matching. arXiv preprint arXiv:2302.14416, 2023. 3 [16] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 5 [17] Noel Loo, Alaa Maalouf, Ramin Hasani, Mathias Lechner, Alexander Amini, and Daniel Rus. Large scale dataset distil- lation with domain shift. In Proceedings of the 41st Interna- tional Conference on Machine Learning (ICML). JMLR.org, 2024. 6 [18]Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architec- ture design. In Proceedings of the European Conference on Computer Vision (ECCV), 2018. 5 [19]Ding Qi, Jian Li, Junyao Gao, Shuguang Dou, Ying Tai, Jianlong Hu, Bo Zhao, Yabiao Wang, Chengjie Wang, and Cairong Zhao. Towards universal dataset distillation via task- driven diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10557–10566, 2025. 3 [20]Tian Qin, Zhiwei Deng, and David Alvarez-Melis. A label is worth a thousand images in dataset distillation. In Advances in Neural Information Processing Systems (NeurIPS), pages 131946–131971. Curran Associates, Inc., 2024. 1 [21]Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaim- ing He, and Piotr Dollar. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 5 [22] Tal Ridnik, Emanuel Ben Baruch, Asaf Noy, and Lihi Zelnik- Manor. Imagenet-21k pretraining for the masses. CoRR, abs/2104.10972, 2021. 5 [23]Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh- moginov, and Liang-Chieh Chen. MobileNetV2: Inverted Residuals and Linear Bottlenecks . In 2018 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 4510–4520, Los Alamitos, CA, USA, 2018. IEEE Com- puter Society. 5 [24]Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensation via 9 various backbone and statistical matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16709–16718, 2024. 2, 3, 5 [25]Shitong Shao, Zikai Zhou, Huanran Chen, and Zhiqiang Shen. Elucidating the design space of dataset condensation. In Ad- vances in Neural Information Processing Systems (NeurIPS), pages 99161–99201. Curran Associates, Inc., 2024. 1, 2, 3, 4, 5 [26] Zhiqiang Shen, Ammar Sherif, Zeyuan Yin, and Shitong Shao. Delt: A simple diversity-driven earlylate training for dataset distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 2, 3, 5, 1 [27] Rui Song, Dai Liu, Dave Zhenyu Chen, Andreas Festag, Carsten Trinitis, Martin Schulz, and Alois Knoll.Fed- erated learning via decentralized dataset distillation in resource-constrained edge environments. arXiv preprint arXiv:2208.11311, 2022. 1 [28] Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, and Bowen Tang. Dˆ4: Dataset distillation via disentangled diffusion model. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 5809– 5818, 2024. 3 [29]Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distilla- tion paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 1, 2, 3, 5 [30]Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR, abs/1905.11946, 2019. 5 [31]Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers and distillation through at- tention. In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 10347–10357. PMLR, 2021. 5 [32]Minh-Tuan Tran, Trung Le, Xuan-May Le, Thanh-Toan Do, and Dinh Phung. Enhancing dataset distillation via non- critical region refinement. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10015–10024, 2025. 1, 3 [33]Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12196–12205, 2022. 1, 3 [34] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros.Dataset distillation.arXiv preprint arXiv:1811.10959, 2018. 1 [35]Lingao Xiao, Songhua Liu, Yang He, and Xinchao Wang. Rethinking large-scale dataset compression: Shifting focus from labels to images. arXiv preprint arXiv:2502.06434, 2025. 1, 6 [36]Zeyuan Yin and Zhiqiang Shen. Dataset distillation via cur- riculum data synthesis in large data era. Transactions on Machine Learning Research, 2024. 3, 5 [37]Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, re- cover and relabel: Dataset condensation at imagenet scale from a new perspective. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 1, 2, 3, 4, 5 [38] Lei Zhang, Jie Zhang, Bowen Lei, Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, and Dongkuan Xu. Accelerating dataset distillation via model augmentation. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11950–11959, 2023. 1, 3 [39]Bo Zhao and Hakan Bilen. Dataset condensation with differ- entiable siamese augmentation. In International Conference on Machine Learning (ICML), 2021. 1, 3 [40] Bo Zhao and Hakan Bilen. Synthesizing informative training samples with gan. NeurIPS 2022 Workshop on Synthetic Data for Empowering ML Research, 2022. 3 [41]Bo Zhao and Hakan Bilen. Dataset condensation with dis- tribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023. 3 [42]Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Con- ference on Learning Representations, 2021. 1, 3 [43]Xinhao Zhong, Hao Fang, Bin Chen, Xulin Gu, Tao Dai, Meikang Qiu, and Shu-Tao Xia. Hierarchical features mat- ter: A deep exploration of gan priors for improved dataset distillation. arXiv preprint arXiv:2406.05704, 2024. 3 [44]Xinhao Zhong, Shuoyang Sun, Xulin Gu, Zhaoyang Xu, Yaowei Wang, Min Zhang, and Bin Chen. Efficient dataset distillation via diffusion-driven patch selection for improved generalization, 2025. 2, 3 [45]Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 3 10 Accelerating Large-Scale Dataset Distillation via Exploration–Exploitation Optimization Supplementary Material 6. Further Discussion Additional Cross-architecture Evaluation. To expand our evaluation of cross-architecture generaliza- tion, Tab. 11 reports results on transformer-based and hy- brid models, where our method consistently achieves the strongest performance. Full-size Image Initialization We adopt real image initial- ization sampled from the original dataset, which accelerates distillation compared to random noise, as observed in prior works [5,7,25,26,38]. Our closest baseline, EDC, relies on RDED [29] for fast, optimization-free initialization by com- posing each synthetic image from several high-confidence patches of real data. However, patch-based compression oversamples similar regions, which leads to redundancy, and it further distorts local features because of the com- pressed representation. We further posit that the subsequent relabeling stage, which functions as knowledge distillation, may be less effective under patch-based representations, as fragmented local regions provide limited global context for transferring semantic knowledge from teacher to student. In contrast, full-image initialization preserves global seman- tics and spatial coherence, improving generalization and convergence while simplifying deployment by removing the pre-synthesis step. At IPC = 1, we adopt RDED for its stronger performance under extreme data scarcity, but we downplay this case because single-image settings yield unstable accuracy across runs. Initialization Sensitivity in Extremely Compressed IPC Table 13 reports three runs of our method at IPC = 1 with different RDED initialization seeds. Performance varies by up to 2%, highlighting the instability caused by different initializations and the challenge of fair comparison in such an extreme compression setting. Full-size vs. Patch-Based Initialization We compare our full-size image initialization with EDC’s patch-based initialization. For IPC 10, EDC initializes each synthetic image using RDED with four crops per image setting, while our method directly samples full-size images. For IPC 50, to test an alternative setup and further validate our method’s effectiveness, we set EDC to be initialized from a single crop per image, a setting that empirically yields better accuracy, with SSRS learning schedule applied to both methods. All results are reported without the optimization stage to isolate the effect of initialization. As shown in Tab. 14, full-size initialization consistently yields higher accuracy than patch-based initialization, providing a stronger starting point that enables subsequent optimization to be more efficient. Relabeling and Post-Evaluation Strategies For relabeling, we follow the standard state-of-the-art approach for generating soft labels from the teacher model. During student training on distilled data, we adopt the Early Smoothing–Later Steep Learning Rate Schedule (SSRS) in- troduced in EDC [25]. SSRS combines the advantages of smooth cosine decay and MultiStep schedules by applying a gradual cosine reduction in the early phase, followed by a sharp decay in the final phase to accelerate convergence as described in this equation: μ(i) = 1 + cos iπ ζN 2 ,i≤ 5N 6 , 1 + cos 5π 6ζ 2 · 6N − 6i 6N , i > 5N 6 , (2) whereμ(i)is the learning rate at iterationi,Nis the to- tal number of training iterations, andζcontrols the cosine smoothness. Unlike EDC, which introduced SSRS only in the appendix and considered it in high-IPC settings, our method demonstrates that applying this schedule at mod- erate IPC also improves accuracy. Additionally, it reduces the synthesis time required to reach peak accuracy, high- lighting a connection between student training dynamics and distillation efficiency. Effectiveness of Early Smoothing-later Steep Learning Rate Schedule (SSRS) To evaluate the effectiveness of SSRS, we compare our method with and without it under various IPC settings. While performance drops significantly at extremely low IPC val- ues (e.g., IPC = 1), SSRS consistently improves results at higher IPCs by achieving better accuracy and requiring fewer optimization steps to reach convergence. As shown in Tab. 12, the benefits of SSRS become increasingly clear as IPC increases. Our method, accelerated20×with SSRS, achieves near-converged performance without additional computational cost. This observation is consistent with prior work [25], which shows that an effective learning schedule can yield substantial improvements with minimal overhead. Our findings further establish a connection between opti- mization efficiency and the design of the learning schedule, showing that accelerated learning scheduling can signifi- cantly reduce the need for extensive optimization. Synthetic Data Comparison in Logit Space Figure 7 presents t-SNE visualizations of logit embeddings for synthetic data generated by our method and recent state- 1 SynthesisRelabeling & Evaluation Iteration200Epochs300 OptimizerAdamW β 1 = 0.5, β 2 = 0.9 OptimizerAdamW Loss TypeMSE-GT + 0.025·CE Learning rate0.05Learning rate0.001 Batch Size80Batch Size100 K, ε140, 0.5LR ScheduleSSRS, ζ = 2 SmoothingLR at IPC = 1 InitializationRandom sampling (RDED at IPC = 1) AugmentationRandomResizedCrop RandomHorizontalFlip CutMix α BN 0.1EMA Rate0.99 Table 9. Hyperparameter settings for ImageNet-1K. SynthesisRelabeling & Evaluation Iteration400Epochs300 OptimizerAdamW β 1 = 0.5, β 2 = 0.9 OptimizerAdamW Loss TypeKL-Div Learning rate0.005Learning rate0.005 Batch Size80Batch Size100 K, ε280, 0.3LR ScheduleSSRS, ζ = 2 InitializationRandom samplingAugmentationRandomResizedCrop RandomHorizontalFlip CutMix α BN 0.01 Table 10. Hyperparameter settings for ImageNet-21K. IPCMethod Evaluation model DeiT-TinySwin-TinyConvNeXt-TinyViT-Small RDED14.029.248.3- 10EDC18.438.354.414.9 Ours 10× 23.642.755.019.8 RDED44.556.965.4- 50EDC55.063.366.654.5 Ours 10× 56.164.366.860.7 Table 11. Cross-architecture generalization on ImageNet-1K. MethodIPC = 10IPC = 50 w/o SSRSw/ SSRSw/o SSRSw/ SSRS Ours 10× 49.7± 0.250.0± 0.158.4± 0.158.9± 0.1 Ours 20× 49.4± 0.249.9± 0.158.0± 0.258.8± 0.04 Table 12. Ablation study on the effect of Early Smoothing–later Steep Learning Rate Schedule (SSRS) under different acceleration factors on ImageNet-1K. SSRS consistently improves performance and accelerates convergence across both IPC = 10 and IPC = 50. of-the-art baselines on ImageNet-1K. Our synthetic repre- sentations occupy a broader region of the embedding space, indicating higher diversity and stronger generalization. At Run 1Run 2Run 3 Ours 10× 10.8912.2912.90 Table 13. Performance across three independent runs of our method with different RDED initialization seeds on ImageNet-1K at IPC = 1 using a ResNet-18 architecture. Method IPC 1050 EDC44.557.1 Ours 47.958.4 Table 14.Effect of initialization strategy.Accuracy on ImageNet-1K when comparing the patch-based initialization of EDC with the full-image initialization adopted in our method (E 2 D) across two IPC settings. the same time, class-specific clusters remain clearly sepa- rated, reflecting preserved discriminability and semantically coherent structure. These observations show that our method generates synthetic data that is diverse and semantically aligned with class structure. 2 Figure 7. t-SNE visualization of logit embeddings for synthetic data generated by our method and state-of-the-art baselines on ImageNet-1K. Our method exhibits broader dispersion and clearer clusters, indicating better generalization and stronger class sepa- rability. Outlier points outside class clusters are highlighted by yellow circles. Acceleration on Low-Resolution Datasets. We exclude low-resolution datasets such as CIFAR-10 and CIFAR-100 from our primary focus, as EDC already achieves peak performance with minimal optimization, ren- dering further acceleration unnecessary. Table 15 exam- ines the effect of optimization iterations on CIFAR-100 using EDC. Although the default configuration uses 2000 iterations, our findings show that omitting optimization en- tirely preserves final accuracy for both IPC = 10 and IPC = 50 without noticeable degradation. This suggests that, for low-resolution datasets, the knowledge distillation process embedded in the relabeling stage alone effectively drives performance, making extra optimization of synthetic data redundant. Similarly, for CIFAR-10, the default EDC config- uration uses only 75 optimization steps, a setting that does not benefit from further acceleration. MethodIPC 10IPC 50 EDC (reported)63.7± 0.3 68.6± 0.2 EDC (reproduced) No optimization 65.1± 0.2 68.5± 0.2 With optimization 65.2± 0.1 68.8± 0.2 Table 15. Effect of optimization iterations in EDC on CIFAR- 100. The top row reports the value from the original paper; the last two rows are our reproduced results. Final accuracy remains stable even without optimization, indicating that the relabeling stage alone suffices for low-resolution datasets. 7. Implementation Details Loss Threshold The thresholdεspecifies the minimum loss during explo- ration stage that a crop must reach to be selected for ex- ploitation in our method. Conceptually,εacts as a filter that identifies which regions of the synthetic image provide sufficiently strong supervisory signal to justify further op- timization. The threshold controls the trade-off between optimization duration and computational overhead. Lower values increase the number of retained crops, which raises computational cost, while higher values risk premature stop- ping, leading to suboptimal optimization. Empirically, we found thatε = 0.5is effective for ImageNet-1K andε = 0.3 for ImageNet-21K. Hyperparameter Settings We report the hyperparameters used for each dataset experi- ment to facilitate reproducibility. Table 9 lists the configura- tion for ImageNet-1K, while Table 10 shows the settings for ImageNet-21K. 8. Additional Visualization We present additional visualizations of the synthetic data gen- erated by our method for ImageNet-1K and ImageNet-21K. By relying on full-size feature representations, our method avoids the distortions arising from spatially compressed fea- tures and produces images with higher structural fidelity and clearer class-specific patterns. Illustrative examples for ImageNet-1K are shown in Fig. 8, while corresponding ImageNet-21K examples are provided in Fig. 9. 3 Figure 8. Visualization of synthetic images produced by our method on ImageNet-1K. 4 Figure 9. Visualization of synthetic images produced by our method on ImageNet-21K. 5